Summary:
When querying Dremio using an Arrow Flight–based client (for example, a Python application using the pyarrow library), the following error may appear in the job profile or client logs:
RpcException: Timeout while waiting for client to be in ready state.
This error occurs when Dremio attempts to send query results to the Arrow Flight client, but the client does not indicate readiness within the configured timeout period. The issue can occur intermittently and may appear after several minutes of query execution.
Relevant Versions
All Dremio Active Versions
Troubleshooting Steps
Review the job profile and confirm the failure occurs during result delivery over Arrow Flight.
Verify whether the issue is reproducible across different data sources to rule out source-specific performance problems.
-
Check the current value of the following support key:
flight.client.readiness.timeout.millis Temporarily increase the timeout value and re-run the query to confirm whether the issue is related to client readiness.
Cause
The flight.client.readiness.timeout.millis setting controls how long Dremio waits for an Arrow Flight client to indicate that it is ready to receive more data.
A timeout occurs when Dremio attempts to send query results, but the client does not respond within the configured time window. This typically indicates that the client is unable to keep up with result consumption rather than a failure on the Dremio side.
Common contributing factors include:
Network latency or bandwidth constraints between Dremio and the client environment
The Arrow Flight client being busy processing previously received data
CPU or memory constraints on the client machine
Large result sets requiring longer processing times on the client side
Steps to Resolve
Increase the Arrow Flight client readiness timeout.
Example configuration change:
Default:
50000msUpdated:
120000ms (2 minutes)
Re-run the affected queries and confirm that the timeout no longer occurs.
Impact of Increasing the Timeout
Increasing flight.client.readiness.timeout.millis is a valid mitigation, but the following considerations apply:
Dremio server resources remain allocated for a longer duration while waiting for the client.
Underlying performance or scalability issues on the client side may be masked rather than addressed.
Queries may run longer if result consumption is slow.
Best Practices
Ensure the Arrow Flight client processes incoming data efficiently (for example, by using batching or optimized memory handling).
Monitor CPU, memory, and network utilization on the client machine during query execution.
Reduce result set sizes by applying filters or limiting returned columns where possible.
Review network latency and bandwidth between the client environment and Dremio.
Increasing the timeout is appropriate for workloads that require additional client-side processing time; however, addressing performance bottlenecks will help ensure a more stable and efficient setup.
Additional Resources
https://docs.dremio.com/current/client-applications/drivers/arrow-flight-sql-jdbc-driver/