Here is the knowledge base article formatted using the provided section headings:
Summary
When trying to run a CTAS statement against Dremio in-order to create an iceberg table in AWS Glue, the following error is seen: SYSTEM ERROR: ValidationException: Cannot convert namespace <dremio db name> to Glue database name, because it must be 1-252 chars of lowercase letters, numbers, underscore
due to using hypens/dashes in the Dremio database, table, and column names.
Reported Issue
In addition to a failing CTAS job profile in Dremio, the following error stack may be seen in the Dremio server.log.
SYSTEM ERROR: ValidationException: Cannot convert namespace my-glue-db to Glue database name, because it must be 1-252 chars of lowercase letters, numbers, underscore
SqlOperatorImpl WRITER_COMMITTER
Location 0:0:8
SqlOperatorImpl WRITER_COMMITTER
Location 0:0:8
Fragment 0:0
[Error Id: 71806ef6-c325-4bca-ad86-fdec9462873e on 172.31.23.216:0]
(org.apache.iceberg.exceptions.ValidationException) Cannot convert namespace my-glue-db to Glue database name, because it must be 1-252 chars of lowercase letters, numbers, underscore
org.apache.iceberg.exceptions.ValidationException.check():49
org.apache.iceberg.aws.glue.IcebergToGlueConverter.validateNamespace():105
org.apache.iceberg.aws.glue.IcebergToGlueConverter.toDatabaseName():121
org.apache.iceberg.aws.glue.IcebergToGlueConverter.getDatabaseName():135
org.apache.iceberg.aws.glue.GlueTableOperations.<init>():100
org.apache.iceberg.aws.glue.DremioGlueTableOperations.<init>():25
com.dremio.plugins.awsglue.store.AWSGlueStoragePlugin.createIcebergTableOperations():319
com.dremio.exec.store.iceberg.glue.IcebergGlueModel.getIcebergCommand():59
com.dremio.exec.store.iceberg.model.IcebergBaseModel.getIcebergCommandWithMetricStat():83
com.dremio.exec.store.iceberg.model.IcebergBaseModel.getOptimizeCommitter():158
com.dremio.exec.store.iceberg.manifestwriter.IcebergCommitOpHelper.setup():176
com.dremio.sabot.op.writer.WriterCommitterOperator.setup():148
com.dremio.sabot.driver.SmartOp$SmartSingleInput.setup():282
com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():74
com.dremio.sabot.driver.Pipe$SetupVisitor.visitSingleInput():64
com.dremio.sabot.driver.SmartOp$SmartSingleInput.accept():227
com.dremio.sabot.driver.StraightPipe.setup():103
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.StraightPipe.setup():102
com.dremio.sabot.driver.Pipeline.setup():71
com.dremio.sabot.exec.fragment.FragmentExecutor.setupExecution():620
com.dremio.sabot.exec.fragment.FragmentExecutor.run():442
com.dremio.sabot.exec.fragment.FragmentExecutor.access$1700():107
com.dremio.sabot.exec.fragment.FragmentExecutor$AsyncTaskImpl.run():1000
com.dremio.sabot.task.AsyncTaskWrapper.run():122
com.dremio.sabot.task.slicing.SlicingThread.mainExecutionLoop():249
com.dremio.sabot.task.slicing.SlicingThread.run():171
Relevant Versions
All versions of Dremio
Troubleshooting Steps
1. Download the query profile and confirm the same SYSTEM ERROR: ValidationException error is seen
2. Visually check and confirm the glue catalog name from the Query tab of the Profile or the SQL runner tab in Dremio if currently trying to run the CTAS command again.
3. If there are hypens/dashes in the Dremio database, table, or column names then carry out the steps to resolve.
Cause
This error is entirely expected and part of the database name and table name validation when creating Dremio iceberg tables.
Per the AWS documentation (linked in this article), creating schema's on AWS Glue with dashes/hyphens goes against AWS Glue best practices and recommendations, so Dremio and the Iceberg requirements for naming, goes in-line with this.
Steps to Resolve
Create iceberg tables in AWS Glue via Dremio without using hyphens/dashes as the only acceptable characters for database names, table names, and column names are lowercase letters, numbers, and the underscore character.
Additional Resources
AWS Glue Best Practices:
https://docs.aws.amazon.com/athena/latest/ug/glue-best-practices.html#schema-names