Skip to main content

MS SQL Server Sources Failing with PKIX Certificate Error After Upgrading to Dremio v25.2.25

  • Updated

Summary

After upgrading to Dremio v25.2.25, MS SQL Server sources may display as red and fail to connect. This is caused by a change in the bundled MS SQL Server JDBC driver, which now enforces encrypted connections (encrypt=true) by default. As a result, Dremio performs full TLS certificate chain validation, and if the SQL Server's certificate or its signing CA is not present in Dremio's Java truststore, the connection fails with a PKIX path building error. The fix requires importing the appropriate certificates into the Java truststore on all Dremio nodes including coordinators and executors and re-enabling Verify Server Certificate on the affected sources.

Reported Issue

MS SQL Server sources in Dremio turned red after upgrading to v25.2.25. The following error is observed when attempting to save the source with Verify Server Certificate enabled:

Failure creating/updating this source: java.lang.Exception: Unavailable: The driver 
could not establish a secure connection to SQL Server by using Secure Sockets Layer 
(SSL) encryption. Error: "PKIX path building failed: 
sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid 
certification path to requested target"

NOTE: A temporary workaround can be applied by unchecking the Advanced Properties → Verify Server Certificate checkbox for the MS SQL Server Source. This restores connectivity but disables certificate validation. It is not recommended as a permanent solution.

Relevant Versions

Dremio v25.2.25 and later
Dremio AWSE (AWS Edition) with dynamic executors
MS SQL Server sources (self-signed or internal CA) and AWS RDS SQL Server

Troubleshooting Steps

1. Confirm the Dremio version change is the cause

Check whether the issue appeared immediately after upgrading to v25.2.25. If a parallel environment on an older version (e.g. v25.1.10) is connecting to the same SQL Server without issues, this confirms the JDBC driver change is the root cause.

2. Identify the configured truststore

Check the coordinator's dremio.conf to find the truststore path:

cat /opt/dremio/conf/dremio.conf | grep trustStore

The relevant lines will look like:

services.coordinator.client-endpoint.ssl.trustStore: "/etc/pki/java/cacerts"
services.coordinator.client-endpoint.ssl.trustStorePassword: "changeit"

3. Check whether the SQL Server certificate is already in the truststore

On the coordinator:

keytool -list -keystore /etc/pki/java/cacerts -storepass changeit | grep -i <cert-name>

If nothing is returned, the certificate is not imported and needs to be added.

4. Inspect the certificate the SQL Server is presenting

Run the following from the coordinator or an executor to identify the certificate issuer:

openssl s_client -connect <mssql-host>:1433 -starttls mssql 2>/dev/null | openssl x509 -noout -issuer -subject -dates

This confirms whether the cert is self-signed, issued by an internal CA, or issued by Amazon RDS CA (for RDS instances).

5. Verify executor truststore (AWSE)

SSH or Exec into an executor and run the same keytool -list check to confirm whether the cert is also missing there. Query execution failures despite a green source on the coordinator typically indicate the cert is missing on executors.

5. Carry out the "Steps to Resolve" described below.

Cause

Dremio v25.2.25 upgraded the bundled Microsoft SQL Server JDBC driver to a version where encrypt=true is enforced by default. In prior versions the default was encrypt=false, so unencrypted connections were accepted without any SSL configuration.

With encrypt=true active, the JDBC driver performs full TLS certificate chain validation (PKIX). If the SQL Server's certificate or the CA that signed it is not present in Dremio's Java truststore, the validation fails and the source cannot connect.

This affects both the coordinator (which handles source creation and metadata) and executor nodes (which handle query execution and also establish direct JDBC connections to the source). In AWSE deployments with dynamic executors, each executor boots with the default system truststore, so any custom certificates must be injected at startup time via the executor customization script.

Steps to Resolve

1. Obtain the correct certificate

On-premise SQL Server (self-signed or internal CA): Export the certificate from the SQL Server or Windows certificate store as a .crt file. Where possible, export the root CA certificate rather than the server certificate as this covers all certs issued by that CA and avoids breakage when the server cert is renewed.

AWS RDS SQL Server: Download the Amazon RDS CA bundle for the relevant region.

2. Import the certificate on the coordinator

Single certificate (on-premise):

keytool -importcert -alias <alias-name> \
  -file /path/to/cert.crt \
  -keystore /etc/pki/java/cacerts \
  -storepass changeit \
  -noprompt

RDS CA bundle (if applicable) (must be split into individual certs first):

 
awk 'BEGIN {c=0} /-----BEGIN CERTIFICATE-----/ {c++} {print > "/tmp/rds-cert-" c ".pem"}' \
  /var/dremio_efs/executor/certs/rds-ca-bundle.pem

for i in /tmp/rds-cert-*.pem; do
  keytool -importcert -alias "rds-ca-$(basename $i .pem)" \
    -file "$i" \
    -keystore /etc/pki/java/cacerts \
    -storepass changeit \
    -noprompt
  echo "Imported: $(basename $i)"
done

Verify the import:

keytool -list -keystore /etc/pki/java/cacerts -storepass changeit | grep <alias>

Restart Dremio on the coordinator.

3. Handle executors (AWSE with dynamic executors)

Place all certificate files on EFS so they are accessible to all executors at boot:

/var/dremio_efs/executor/certs/

Update the executor customization script to import all required certificates on every executor boot. The RDS bundle must be split inside the script so it runs locally on each executor so do not rely on pre-split files in /tmp from another machine.

Example Script (Adjust as Necessary):
#!/bin/bash
set -euo pipefail

DREMIO_ENV="/etc/dremio/dremio-env"
TRUSTSTORE="/etc/pki/java/cacerts"
TRUSTSTORE_PASS="changeit"

MEM_KB=$(awk '/MemTotal/ {print $2}' /proc/meminfo)

if [ "$MEM_KB" -gt 40000000 ]; then
  echo "big mem" > /tmp/memsize
  sed -i '/DREMIO_JAVA_SERVER_EXTRA_OPTS/d' "$DREMIO_ENV"
  echo 'DREMIO_JAVA_SERVER_EXTRA_OPTS="-Xmx16000m -XX:MaxDirectMemorySize=100000m"' >> "$DREMIO_ENV"
else
  echo "small mem" > /tmp/memsize
fi

import_cert() {
  local cert_file="$1"
  local alias_name="$2"

  if [ -f "$cert_file" ]; then
    echo "Checking certificate alias: $alias_name"
    if keytool -list \
      -keystore "$TRUSTSTORE" \
      -storepass "$TRUSTSTORE_PASS" \
      -alias "$alias_name" >/dev/null 2>&1; then
      echo "Certificate alias already exists. Replacing: $alias_name"
      keytool -delete \
        -keystore "$TRUSTSTORE" \
        -storepass "$TRUSTSTORE_PASS" \
        -alias "$alias_name"
    fi
    echo "Importing certificate: $cert_file"
    keytool -importcert \
      -alias "$alias_name" \
      -file "$cert_file" \
      -keystore "$TRUSTSTORE" \
      -storepass "$TRUSTSTORE_PASS" \
      -noprompt
    echo "Certificate imported successfully: $alias_name"
  else
    echo "WARNING: Certificate file not found: $cert_file"
  fi
}

# On-premise SQL Server certs
import_cert "/var/dremio_efs/executor/certs/mssql-server.crt" "mssql-server-cert"
import_cert "/var/dremio_efs/executor/certs/mp2-npdb02.crt" "mp2-npdb02"

# RDS CA bundle — split into individual certs on this executor and import
awk 'BEGIN {c=0} /-----BEGIN CERTIFICATE-----/ {c++} {print > "/tmp/rds-cert-" c ".pem"}' \
  /var/dremio_efs/executor/certs/rds-ca-bundle.pem

for i in /tmp/rds-cert-*.pem; do
  alias_name="rds-ca-$(basename $i .pem)"
  if keytool -list -keystore "$TRUSTSTORE" -storepass "$TRUSTSTORE_PASS" -alias "$alias_name" >/dev/null 2>&1; then
    keytool -delete -keystore "$TRUSTSTORE" -storepass "$TRUSTSTORE_PASS" -alias "$alias_name"
  fi
  keytool -importcert -alias "$alias_name" -file "$i" \
    -keystore "$TRUSTSTORE" -storepass "$TRUSTSTORE_PASS" -noprompt
  echo "Imported: $alias_name"
done

echo "Executor startup customization completed."


4. Once the certificates are confirmed in the truststore on both the coordinator and executors, re-enable Verify Server Certificate by checking the Advanced Properties → Verify Server Certificate checkbox for the MS SQL Server Source.
 

Additional Resources

Dremio AWSE Executor Customization: https://docs.dremio.com/25.x/get-started/cluster-deployments/deployment-models/amazon-deployments/aws/configure/aws-edition-executors/

AWS RDS SSL/TLS Certificate Documentation: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/UsingWithRDS.SSL.html

Using SSL with MS SQL Server on Amazon RDS: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/SQLServer.Concepts.General.SSL.Using.html

 


 

Was this article helpful?
0 out of 0 found this helpful
Return to top

Related articles