PYArrow error - Exception has occurred: FlightInternalError Could not finish writing before closing

Summary

Connection to Dremio using pyarrow works from a local environment, but the same code fails when executed on a server. The following error occurs on this line:

bearer_token = client.authenticate_basic_token(USERNAME, PASSWORD)

Exception has occurred: FlightInternalError
Could not finish writing before closing
File "C:\Users\Dremio\myarrow.py", line 22, in <module> bearer_token = client.authenticate_basic_token(USERNAME, PASSWORD) pyarrow._flight.FlightInternalError: Could not finish writing before closing

 
Troubleshooting Steps

Verify server connectivity to the Dremio instance. You can use a cURL command to confirm if the required port is accessible. Ensure that the server is able to connect to the Dremio instance.

 
Steps to Resolve

Work with your network team to resolve any connectivity issues ( such as firewall) between client and server. Please find below a script showing how to enable certificate verification when using Arrow:
 

from pyarrow import flight

args = {
    "hostname": "your.server",
    "port": 32010,
    "username": "username",
    "password": "password",
    "query": "select * from sys.roles",
    "tls": True,
    "server_cert": "/path/to/your/certificate.crt",
    "disable_certificate_verification": False
}

if args['tls'] == True:
  scheme = "grpc+tls"
else:
  scheme = "grpc+tcp"

with open(args["server_cert"], "rb") as cert_file:
  cert = cert_file.read()

client = flight.FlightClient(f"{scheme}://{args['hostname']}:{args['port']}",disable_server_verification=args['disable_certificate_verification'],tls_root_certs=cert)
bearer_token = client.authenticate_basic_token(args['username'],args['password'])
options = flight.FlightCallOptions(headers=[bearer_token])

flight_info = client.get_flight_info(flight.FlightDescriptor.for_command(args['query']), options)
reader = client.do_get(flight_info.endpoints[0].ticket, options)
print(reader.read_pandas())