Summary
This article helps troubleshoot the "Failed to read row groups from block split" error reported in the profiles of failed queries.
Reported Issue
The following exceptions are displayed in the logs:
[Error Id: UUID on dremio-executor-*:0]
(java.lang.RuntimeException) Failed to read row groups from block split
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.initSplits()
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.filterIfNecessary()
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.next()
com.dremio.exec.store.parquet.ParquetSplitReaderCreatorIterator.next()
com.dremio.exec.store.dfs.PrefetchingIterator.next()
com.dremio.exec.store.dfs.PrefetchingIterator.next()
com.dremio.exec.store.parquet.ScanTableFunction.setupNextReader()
com.dremio.exec.store.parquet.ScanTableFunction.processRow()
com.dremio.sabot.op.tablefunction.TableFunctionOperator.outputData()
com.dremio.sabot.driver.SmartOp$SmartSingleInput.outputData()
com.dremio.sabot.driver.StraightPipe.pump()
com.dremio.sabot.driver.Pipeline.doPump()
com.dremio.sabot.driver.Pipeline.pumpOnce()
com.dremio.sabot.exec.fragment.FragmentExecutor$DoAsPumper.run()
com.dremio.sabot.exec.fragment.FragmentExecutor.run()
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run()
com.dremio.sabot.task.AsyncTaskWrapper.run()
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop()
com.dremio.sabot.task.slicing.SlicingThread.run()
Relevant Versions
This can happen in all Dremio releases.
Troubleshooting Steps
1. Check if there is another error following the stack trace above. If there is, then that becomes the error that needs to be troubleshooted.
2. Check if there are any disk/IO issues at the times this is reported.
3. Check if there were any changes to the environment that could have caused slow reads. If an AsyncTimeoutException is reported, please see this troubleshooting guide.
4. Check the logs of the executor node reported in the error for additional hints.
5. Check that the distributed storage is accessible and the credentials are still valid.
6. Find out after how many seconds the query fails. If it fails every time after the same amount of seconds, then there is a timeout that's being hit.
Cause
"Failed to read row groups from block split" is a generic error that is returned when one or more of the referred data files are absent. It either means that a value trying to be read couldn't be accessed/was not there, or that when it's trying to connect segments and a row group handoff marker is handing off an empty row, as a natural row end, and the new iterator is expecting something else, not expecting that it should initialize a new row. To identify the root cause of this error, additional troubleshooting needs to be performed.
Steps to Resolve
Follow the troubleshooting steps above.