Summary
This article helps troubleshoot the "SYSTEM ERROR: IllegalStateException: AsyncHttpClient is closed" exception reported in the Dremio logs or query profiles.
Reported Issue
The following exceptions are displayed in the logs:
(java.lang.IllegalStateException) AsyncHttpClient is closed
com.dremio.plugins.async.utils.AsyncReadWithRetry.read()
com.dremio.plugins.gcs.GCSAsyncFileReader.readFully()
com.dremio.io.AsyncByteReader.versionedReadFully()
com.dremio.service.cachemanager.CacheAsyncByteReader.fetchDataFromPrimarySrcAsync()
com.dremio.service.cachemanager.CacheAsyncByteReader.readFully()
com.dremio.exec.store.dfs.LoggedFileSystem$LoggedAsyncByteReader.readFully()
com.dremio.io.AsyncByteReaderWithTimeout.readFully()
com.dremio.parquet.pages.async.ReadChunk.request()
com.dremio.parquet.pages.async.ReadChunk.request()
com.dremio.parquet.pages.async.ReadChunk.request()
com.dremio.parquet.pages.async.ReadChunk.request()
com.dremio.parquet.pages.async.SlidingWindowReader.ensureRange()
com.dremio.parquet.pages.async.SlidingWindowReader.readFully()
com.dremio.parquet.pages.BaseReaderIterator.readFully()
com.dremio.parquet.pages.SmartPRI.getPage()
com.dremio.parquet.pages.MemoizingPageIterator.getPage()
com.dremio.parquet.pages.PageIterator.nextPage()
com.dremio.parquet.pages.PageIterator.hasNextPage()
com.dremio.parquet.reader.column.generics.Float8DeltaReader.evalNextBatch()
com.dremio.parquet.reader.FilteringRowGroupReader.eval()
com.dremio.extra.exec.store.dfs.parquet.ParquetVectorizedReader.next()
com.dremio.exec.store.parquet.UnifiedParquetReader.readEnsuringReadersReturnSameNumberOfRecords()
...
(java.lang.IllegalStateException) AsyncHttpClient is closed
com.dremio.plugins.async.utils.AsyncReadWithRetry.read()
com.dremio.plugins.azure.AzureAsyncReader.read()
com.dremio.plugins.azure.AzureAsyncReader.readFully()
com.dremio.io.AsyncByteReader.versionedReadFully()
com.dremio.service.cachemanager.CacheAsyncByteReader.fetchDataFromPrimarySrcAsync()
com.dremio.service.cachemanager.CacheAsyncByteReader.readFully()
com.dremio.io.AsyncByteReaderWithTimeout.readFully()
com.dremio.io.AsyncByteReader.readFully()
com.dremio.parquet.pages.FooterReader.readFooterFuture()
com.dremio.parquet.pages.async.RowGroupReader.initialize()
com.dremio.extra.exec.store.dfs.parquet.SmartStreamFactory.create()
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.createInputStreamProvider()
Or the following error is displayed in the query profile:
SYSTEM ERROR: IllegalStateException: AsyncHttpClient is closed
Relevant Versions
This can happen in all Dremio releases.
Troubleshooting Steps
1. Check if the error is intermittent, or if it happens for a certain data source, or certain query.
2. In the error stack we can see AsyncByteReaderWithTimeout/AsyncTimeoutException timeout which may be caused by a slow source, so if the speed of that data source ca be increased, that should help.
3. Go to the Source settings, under Advanced Options, do you see the "Enable asynchronous access" option selected? If it is, please uncheck it, save the setting, try the query again to see if there is a problem with asynchronous access.
Cause
Generally this error is caused by a slow access to the data source.
Steps to Resolve
Follow the troubleshooting steps.
Generally slow connections to the data sources, if the speed can't be improved, is being worked around by increasing one of the following Support keys:dremio.iceberg.manifest_read.async.request.timeout_millis
store.parquet.async.request.timeout_millis