Summary
When promoting a Delta Lake folder to a Physical Dataset (PDS) in Dremio, users may encounter a NullPointerException.
Reported Issue
The user may observe an error in the server.log such as below:
java.lang.NullPointerException: null
at org.apache.arrow.util.Preconditions.checkNotNull(Preconditions.java:784)
at com.dremio.exec.store.deltalake.DeltaLakeSchemaConverter.fromSchemaString(DeltaLakeSchemaConverter.java:70)
at com.dremio.exec.store.deltalake.DeltaLakeFormatDatasetAccessor$1.getRecordSchema(DeltaLakeFormatDatasetAccessor.java:131)
at com.dremio.exec.catalog.MetadataObjectsUtils.overrideExtended(MetadataObjectsUtils.java:106)
at com.dremio.exec.catalog.DatasetSaverImpl.saveUsingV1Flow(DatasetSaverImpl.java:263)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:121)
at com.dremio.exec.catalog.DatasetSaverImpl.save(DatasetSaverImpl.java:143)
at com.dremio.exec.catalog.EnterpriseDatasetSaver.save(EnterpriseDatasetSaver.java:83)
at com.dremio.exec.catalog.DatasetManager.createOrUpdateDataset(DatasetManager.java:512)
at com.dremio.exec.catalog.CatalogImpl.createOrUpdateDataset(CatalogImpl.java:1140)
at com.dremio.exec.catalog.DelegatingCatalog.createOrUpdateDataset(DelegatingCatalog.java:265)
at com.dremio.exec.catalog.SourceAccessChecker.createOrUpdateDataset(SourceAccessChecker.java:288)
at com.dremio.exec.catalog.DelegatingCatalog.createOrUpdateDataset(DelegatingCatalog.java:265)
at com.dremio.dac.service.source.SourceService.createPhysicalDataset(SourceService.java:598)
at com.dremio.dac.resource.SourceResource.saveFolderFormat(SourceResource.java:410)
... (truncated)
Relevant Versions
- Dremio 20.0 to 20.3
- Dremio 21.0 to 21.1
Troubleshooting Steps
N/A
Cause
This issue is due to a known defect outlined in internal JIRA DX-48275. It occurs when the commit information for the Delta Lake file is not located where Dremio expects it in the Delta log files, and the corresponding checkpoint file is missing.
Steps to Resolve
To resolve this issue, upgrade to Dremio version 21.2 or later.
Tips & Tricks
N/A
Best Practices
N/A
Recommendations
N/A
FAQ
N/A
Additional Resources
N/A