Overview
Disabling iceberg settings and attempting a refresh metadata - eg
ALTER PDS <PHYSICAL-DATASET-PATH> REFRESH METADATA FORCE UPDATE
can give an error in the UI as follows
IllegalStateException: Found a dataset with Iceberg metadata in non-iceberg refresh
Applies To
Dremio v18 and higher
Details
When disabling the following iceberg parameters
dremio.execution.support_unlimited_splits
dremio.iceberg.enabled
and then attempting a refresh eg
ALTER PDS <PHYSICAL-DATASET-PATH> REFRESH METADATA FORCE UPDATE
can give an IllegalStateException error as follows
IllegalStateException: Found a dataset with Iceberg metadata in non-iceberg refresh
the server.log shows for example ..
2022-04-12 16:44:54,273 [1daa537a-724c-e8e5-e14b-249f492e6d00/0:foreman-planning] ERROR c.d.s.commandpool.CommandWrapper - command 1daa537a-724c-e8e5-e14b-249f492e6d00/0:foreman-planning failed
java.lang.IllegalStateException: Found a dataset with Iceberg metadata in non-Iceberg refresh. Please forget metadata for dataset '{}' [Samples."samples.dremio.com"."Dremio University"."restaurant_reviews.parquet"]
at com.google.common.base.Preconditions.checkState(Preconditions.java:590)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:114)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:137)
at com.dremio.exec.catalog.SourceMetadataManager.saveDatasetAndMetadataInNamespace(SourceMetadataManager.java:717)
at com.dremio.exec.catalog.SourceMetadataManager.refreshDataset(SourceMetadataManager.java:656)
at com.dremio.exec.catalog.ManagedStoragePlugin.refreshDataset(ManagedStoragePlugin.java:841)
at com.dremio.exec.catalog.CatalogImpl.refreshDataset(CatalogImpl.java:982)
at com.dremio.exec.catalog.EnterpriseCatalogImpl.refreshDataset(EnterpriseCatalogImpl.java:211)
at com.dremio.exec.catalog.SourceAccessChecker.refreshDataset(SourceAccessChecker.java:254)
at com.dremio.exec.catalog.DelegatingCatalog.refreshDataset(DelegatingCatalog.java:242)
at com.dremio.exec.planner.sql.handlers.direct.RefreshTableHandler.toResult(RefreshTableHandler.java:104)
at com.dremio.exec.planner.sql.handlers.commands.DirectWriterCommand.plan(DirectWriterCommand.java:99)
at com.dremio.exec.work.foreman.AttemptManager.plan(AttemptManager.java:494)
at com.dremio.exec.work.foreman.AttemptManager.lambda$run$1(AttemptManager.java:387)
at com.dremio.service.commandpool.ReleasableBoundCommandPool.lambda$getWrappedCommand$3(ReleasableBoundCommandPool.java:138)
at com.dremio.service.commandpool.CommandWrapper.run(CommandWrapper.java:62)
at com.dremio.context.RequestContext.run(RequestContext.java:95)
at com.dremio.common.concurrent.ContextMigratingExecutorService.lambda$decorate$3(ContextMigratingExecutorService.java:199)
at com.dremio.common.concurrent.ContextMigratingExecutorService$ComparableRunnable.run(ContextMigratingExecutorService.java:180)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Cause
Dremio will attempt to find the original metadata to update, however, now the iceberg settings have been disabled, it is unable to find them.
Solution
Issue a FORGET METADATA first to forget the iceberg metadata - and then run a REFRESH METADATA FORCE UPDATE
ALTER PDS <PHYSICAL-DATASET-PATH> FORGET METADATA
ALTER PDS <PHYSICAL-DATASET-PATH> REFRESH METADATA FORCE UPDATE
Further Reading
Forgetting Physical Dataset Metadata
Refreshing Physical Dataset Metadata