Summary
This article discusses how to resolve the INVALID_DATASET_METADATA ERROR: Path /dremio/accelerator/../../metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json not found error when running PowerBI queries that are accelerated by reflections in Dremio. NOTE: This issue is only relevant to customer environments using a scale-out coordinator as part of the deployment.
Reported Issue
When attempting to run queries that used reflections through PowerBI, these would fail with the below error due to the query being sent to the scale out coordinator:
INVALID_DATASET_METADATA ERROR: Path /dremio/accelerator/26450972-7d99-444b-9e31-ba4eb65e915b/1993b45c-28c2-4962-a6b7-717252a04e3d_0/metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json not found.
However running the same query in the Dremio UI would succeed as the query in this case was being served by the master coordinator.
Relevant Versions
This issue applies to Dremio versions 25.2.2 and on wards.
Troubleshooting Steps
1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
2. After navigating to the "Raw Profile", under the Job Summary, verify that the query was run at the scale out coordinator per below example where the master is represented as dremio-master-0.dremio-cluster-pod.default.svc.cluster.local and the scale out as dremio-coordinator-0.dremio-cluster-pod.default.svc.cluster.local..
Job Summary
State: FAILED
Coordinator: dremio-coordinator-0.dremio-cluster-pod.default.svc.cluster.local
3. Still within the "Raw Profile", click on the "Acceleration" tab of the profile.
4. Confirm that the missing dataset ("unknown") is seen under the Substitution Info section per below example:
missing dataset ("unknown")
Raw Reflection (Type: raw, Refection Id: 26450972-7d99-444b-9e31-ba4eb65e915b): considered, matched, chosen.
5. Still within the "Raw Profile", click on the "Error" tab of the profile.
6. Confirm that the error is the same as mentioned in this article. As an example, the following error can be seen in the "Verbose" section of the "Error" tab of the job profile:
INVALID_DATASET_METADATA ERROR: Path /dremio/accelerator/26450972-7d99-444b-9e31-ba4eb65e915b/1993b45c-28c2-4962-a6b7-717252a04e3d_0/metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json not found.
SqlOperatorImpl ICEBERG_SUB_SCAN
Location 4:0:6
SqlOperatorImpl ICEBERG_SUB_SCAN
Location 4:0:6
ErrorOrigin: EXECUTOR
[Error Id: fe16bb9b-39cb-412c-b52b-2415885b9709 on dremio-executor-gda-4.dremio-cluster-pod.enterprise-dremio-platform-prod.svc.cluster.local:0]
(org.apache.iceberg.exceptions.NotFoundException) Path /dremio/accelerator/26450972-7d99-444b-9e31-ba4eb65e915b/1993b45c-28c2-4962-a6b7-717252a04e3d_0/metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json not found.
com.dremio.exec.store.iceberg.DremioInputFile.newStream():90
org.apache.iceberg.TableMetadataParser.read():279
org.apache.iceberg.TableMetadataParser.read():273
com.dremio.exec.store.iceberg.IcebergUtils.loadTableMetadata():1983
com.dremio.exec.store.iceberg.SupportsIcebergRootPointer.loadTableMetadata():108
com.dremio.exec.store.iceberg.IcebergManifestListRecordReader.setup():202
com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():410
com.dremio.sabot.op.scan.ScanOperator.setupReader():400
com.dremio.sabot.op.scan.ScanOperator.setup():364
com.dremio.sabot.driver.SmartOp$SmartProducer.setup():638
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():90
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():70
com.dremio.sabot.driver.SmartOp$SmartProducer.accept():609
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.Pipeline.setup():79
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():770
com.dremio.sabot.exec.fragment.FragmentExecutor.run():549
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1274
com.dremio.sabot.task.AsyncTaskWrapper.run():130
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():281
com.dremio.sabot.task.slicing.SlicingThread.run():186
Caused By (java.io.FileNotFoundException) Operation failed: "The specified path does not exist.", 404, HEAD, https://saazdstrprod5svincpovkzt.dfs.core.windows.net/dremio/accelerator/26450972-7d99-444b-9e31-ba4eb65e915b/1993b45c-28c2-4962-a6b7-717252a04e3d_0/metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json?upn=false&timeout=90
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.checkException():1028
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.open():168
com.dremio.plugins.util.ContainerFileSystem.open():350
org.apache.hadoop.fs.FileSystem.open():997
com.dremio.exec.hadoop.HadoopFileSystem.open():234
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.exec.store.dfs.LoggedFileSystem.open():89
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.exec.store.iceberg.SeekableInputStreamFactory.lambda$static$0():41
com.dremio.extra.exec.store.dfs.iceberg.AsyncSeekableInputStreamFactory.getStream():58
com.dremio.exec.store.iceberg.DremioInputFile.newStream():81
org.apache.iceberg.TableMetadataParser.read():279
org.apache.iceberg.TableMetadataParser.read():273
com.dremio.exec.store.iceberg.IcebergUtils.loadTableMetadata():1983
com.dremio.exec.store.iceberg.SupportsIcebergRootPointer.loadTableMetadata():108
com.dremio.exec.store.iceberg.IcebergManifestListRecordReader.setup():202
com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():410
com.dremio.sabot.op.scan.ScanOperator.setupReader():400
com.dremio.sabot.op.scan.ScanOperator.setup():364
com.dremio.sabot.driver.SmartOp$SmartProducer.setup():638
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():90
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():70
com.dremio.sabot.driver.SmartOp$SmartProducer.accept():609
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.Pipeline.setup():79
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():770
com.dremio.sabot.exec.fragment.FragmentExecutor.run():549
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1274
com.dremio.sabot.task.AsyncTaskWrapper.run():130
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():281
com.dremio.sabot.task.slicing.SlicingThread.run():186
Caused By (org.apache.hadoop.fs.azurebfs.contracts.exceptions.AbfsRestOperationException) Operation failed: "The specified path does not exist.", 404, HEAD, https://saazdstrprod5svincpovkzt.dfs.core.windows.net/dremio/accelerator/26450972-7d99-444b-9e31-ba4eb65e915b/1993b45c-28c2-4962-a6b7-717252a04e3d_0/metadata/00000-97c8c277-7fc4-4a54-a496-3d619d59f512.metadata.json?upn=false&timeout=90
org.apache.hadoop.fs.azurebfs.services.AbfsRestOperation.execute():135
org.apache.hadoop.fs.azurebfs.services.AbfsClient.getPathProperties():354
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystemStore.openFileForRead():370
org.apache.hadoop.fs.azurebfs.AzureBlobFileSystem.open():165
com.dremio.plugins.util.ContainerFileSystem.open():350
org.apache.hadoop.fs.FileSystem.open():997
com.dremio.exec.hadoop.HadoopFileSystem.open():234
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.exec.store.dfs.LoggedFileSystem.open():89
com.dremio.io.file.FilterFileSystem.open():54
com.dremio.exec.store.iceberg.SeekableInputStreamFactory.lambda$static$0():41
com.dremio.extra.exec.store.dfs.iceberg.AsyncSeekableInputStreamFactory.getStream():58
com.dremio.exec.store.iceberg.DremioInputFile.newStream():81
org.apache.iceberg.TableMetadataParser.read():279
org.apache.iceberg.TableMetadataParser.read():273
com.dremio.exec.store.iceberg.IcebergUtils.loadTableMetadata():1983
com.dremio.exec.store.iceberg.SupportsIcebergRootPointer.loadTableMetadata():108
com.dremio.exec.store.iceberg.IcebergManifestListRecordReader.setup():202
com.dremio.sabot.op.scan.ScanOperator.setupReaderAsCorrectUser():410
com.dremio.sabot.op.scan.ScanOperator.setupReader():400
com.dremio.sabot.op.scan.ScanOperator.setup():364
com.dremio.sabot.driver.SmartOp$SmartProducer.setup():638
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():90
com.dremio.sabot.driver.Pipe$SetupVisitor.visitProducer():70
com.dremio.sabot.driver.SmartOp$SmartProducer.accept():609
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.StraightPipe.setup():100
com.dremio.sabot.driver.Pipeline.setup():79
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():770
com.dremio.sabot.exec.fragment.FragmentExecutor.run():549
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1274
com.dremio.sabot.task.AsyncTaskWrapper.run():130
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():281
com.dremio.sabot.task.slicing.SlicingThread.run():186
4. Carry out the "Steps to Resolve" described below.
Cause
Queries that used reflections would fail and/or reference stale materializations due to the metadata in the materialization cache of the scale out coordinator being in an inconsistent state.
Steps to Resolve
Upgrade to Dremio v25.2.4 or higher.
Additional Resources
Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
v25.2.4 Release Notes: https://docs.dremio.com/current/release-notes/version-250-release#2524-march-2025