Summary
When using the iceberg parameters (dremio.execution.support_unlimited_splits and dremio.iceberg.enabled) users may see an error when creating reflections if they are not using a compatible type of storage underpinning the directory.
Reported Issue
The following error may be seen in the job profile:
SYSTEM ERROR: UnknownFormatConversionException: Conversion = 'Unknown format (pdfs) conversion for path /opt/dremio/data/pdfs/accelerator/6c16bb24-56c4-4358-a533-ffb1c0899523/07a33f5a-f5bb-4171-a48a-9cf978517335_0/metadata/7affb554-8916-4023-8609-35cbb4eecc92.avro Error Message : No File System scheme matches'
Relevant Versions
Dremio 18.0 and onwards
Troubleshooting Steps
N/A
Cause
Dremio will use a filesystem to store reflection data. If the user has enabled iceberg features to store this data then the filesystem needs to be the correct type to support this. iceberg support is controlled with the following support keys:
dremio.execution.support_unlimited_splits dremio.iceberg.enabled
Steps to Resolve
Iceberg format must be hosted on a distributed file store - supported types are as follows: ADLS - Hadoop, Hive catalogs GCS - Hadoop catalog HDFS - Hadoop catalog Hive (Recommended) - Hive catalog S3 - Hadoop, Hive catalogs **IMPORTANT**: When using Iceberg tables, Dremio recommends using the Hive catalog in production environments The user needs to use one of these in their `dist` setting in the `dremio.conf` file. See https://docs.dremio.com/deployment/dist-store-config/ for examples
Tips & Tricks
N/A
Best Practices
N/A
Recommendations
N/A
FAQ
N/A
Additional Resources
Support settings - https://docs.dremio.com/advanced-administration/support-settings/ Apache iceberg - http://docs.dremio.com/data-formats/apache-iceberg/