Unable to Connect to Hive Source – Certificate Mismatch on Hive Metastore Host – Dremio Support

Summary

A Hive source connection may fail in Dremio when there is a mismatch between the hostname used by Dremio and the hostname specified in the Hive Metastore (HMS) SSL certificate. This results in a TLS handshake failure and prevents Dremio from establishing a secure connection.

Symptoms and Troubleshooting

1.We would see the following error in the Dremio UI under Source settings:

Unavailable Suggested User Action: Could not connect to Hive source <source_name>, check your Hive credentials and network settings.: Failure connecting to source: java.net.SocketException: Connection or outbound has closed

2.We would see the following error messages in the Master logs:



ERROR c.dremio.exec.catalog.PluginsManager - Failure while starting plugin <source_name> after 2451ms.
java.util.concurrent.ExecutionException: java.lang.Exception: Unavailable
Suggested User Action: Could not connect to Hive source <source_name>, check your Hive credentials and network settings.: Failure connecting to source: java.net.SocketException: Connection or outbound has closed
at java.base/java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:395)
at java.base/java.util.concurrent.CompletableFuture.get(CompletableFuture.java:2005)
at com.dremio.exec.catalog.PluginsManager.start(PluginsManager.java:253)
at com.dremio.exec.catalog.CatalogServiceImpl.start(CatalogServiceImpl.java:230)
at com.dremio.service.SingletonRegistry$AbstractServiceReference.start(SingletonRegistry.java:154)
at com.dremio.service.ServiceRegistry.start(ServiceRegistry.java:89)
at com.dremio.service.SingletonRegistry.start(SingletonRegistry.java:43)
at com.dremio.dac.daemon.DACDaemon.startServices(DACDaemon.java:194)
at com.dremio.dac.daemon.DACDaemon.init(DACDaemon.java:200)
at com.dremio.dac.daemon.DremioDaemon.main(DremioDaemon.java:105)
Caused by: java.lang.Exception: Unavailable
Suggested User Action: Could not connect to Hive source <source_name>, check your Hive credentials and network settings.: Failure connecting to source: java.net.SocketException: Connection or outbound has closed
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$newStartSupplier$2(ManagedStoragePlugin.java:629)
at com.dremio.exec.catalog.ManagedStoragePlugin.lambda$nameSupplier$4(ManagedStoragePlugin.java:694)
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1700)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
at java.base/java.lang.Thread.run(Thread.java:829)

3.Basic TCP connectivity from the Dremio Master host/pod to the HMS port is successful:

$nc -vz <10.83.x.x/hmshostname> 9084
Ncat: Connected to 10.83.x.x :9084

Network connectivity can be tested using tools like "telnet" and "curl"

4.TLS handshake from Dremio to HMS fails when using the hostname but succeeds when using the IP address:

$ openssl s_client -connect hmshostname:9084 -showcerts
4067EC497A7F0000:error:10000002:BIO routines:BIO_lookup_ex:system lib:crypto/bio/bio_addr.c:758:Name or service not known
connect:errno=0
$

$ openssl s_client -connect 10.83.209.4:9084 -showcerts
Connecting to 10.83.x.x
CONNECTED(00000003)
Can't use SSL_get_servername
depth=1 C=CN, ST=GD, L=GZ, O=SC, Inc., CN=SC GDP Dev Root CA G1
verify error:num=19:self-signed certificate in certificate chain
verify return:1
depth=1 C=CN, ST=GD, L=GZ, O=SC, Inc., CN=SC GDP Dev Root CA G1
verify return:1
depth=0 C=CN, ST=GD, L=GZ, O=SC, OU=IT, CN=hms.gdp.prod
verify return:1
Certificate chain
 0 s:C=CN, ST=GD, L=GZ, O=SC, OU=IT, CN=hms.gdp.prod
   i:C=CN, ST=GD, L=GZ, O=SC, Inc., CN=SC GDP Dev Root CA G1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Oct 31 10:08:25 GMT; NotAfter: Oct 29 08:10:40 2035 GMT
-----BEGIN CERTIFICATE-----
MIIC9TCCAd2gAwIBAgIQFDD3R5s3jelwW77/3BD1qjUzCkwDQYJKoZIhvcNAQEL
BQAwYTELMAkGA1UEBhMCQ04xCzAJBgNVBAgMAkdEMQswCQYDVQQHDAJHWjEPMA0G
...
X509v3 Subject Alternative Name:
   DNS:hmshostname_new.svc.cluster.local,  DNS:localhost,   IP Address:127.0.0.1
...
-----END CERTIFICATE-----

5.Verify the Subject Alternative Name (SAN) in the local certificate:

$openssl x509 -in certificate.pem -noout -text | grep -A1 "Subject Alternative Name"
$X509v3 Subject Alternative Name:
   DNS:hmshostname.svc.cluster.local,  DNS:localhost,   IP Address:127.0.0.1

Root Cause

Steps 4 and 5 reveal a mismatch in the Subject Alternative Names (SANs) between the two certificates. This SAN mismatch causes the SSL handshake to fail.

Certificate 1 SAN: hmshostname_new.svc.cluster.local
Certificate 2 SAN: hmshostname.svc.cluster.local

Resolution

To resolve the SSL handshake failure, ensure that the SAN in either the HMS certificate or the Dremio certificate matches the correct hostname.

Related articles