Summary
This article discusses an issue that may occur when upgrading an AWSE cluster from 24.2.x to 24.3.1, where the upgrade fails with an error related to the "paths.copyintoerror" configuration parameter.
Reported Issue
When upgrading from 24.2.x to 24.3.1, the upgrade may fail with the following error message:
Starting Services Failed to start Services. Failure while starting services. Failure reading configuration file. The following properties were invalid: paths.copyintoerror
Relevant Versions
This issue specifically applies when migrating from version 24.2.x to 24.3.1.
Troubleshooting Steps
Check the sserver.log for a message along these lines
Jan 11 12:11:39 ip-172-31-35-49 dremio: Dremio is exiting. Failure while starting services Failure reading configuration file. The following properties were invalid:
Jan 11 12:11:39 ip-172-31-35-49 dremio: paths.copyintoerrors
Jan 11 12:11:39 ip-172-31-35-49 dremio: null
Jan 11 12:11:39 ip-172-31-35-49 dremio: [com.dremio.config.DremioConfig.check(DremioConfig.java:288), com.dremio.config.DremioConfig.<init>(DremioConfig.java:261), com.dremio.config.DremioConfig.create(DremioConfig.java:394), com.dremio.config.DremioConfig.create(DremioConfig.java:343), com.dremio.config.DremioConfig.create(DremioConfig.java:339), com.dremio.dac.server.DACConfig.newConfig(DACConfig.java:211), com.dremio.dac.daemon.AwsConfigurationModule.run(AwsConfigurationModule.java:60), com.dremio.dac.daemon.AwsDremioDaemon.main(AwsDremioDaemon.java:18)]
Jan 11 12:11:39 ip-172-31-35-49 dremio: java.lang.RuntimeException: Failure reading configuration file. The following properties were invalid:
Jan 11 12:11:39 ip-172-31-35-49 dremio: paths.copyintoerrors
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.config.DremioConfig.check(DremioConfig.java:288)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.config.DremioConfig.<init>(DremioConfig.java:261)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.config.DremioConfig.create(DremioConfig.java:394)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.config.DremioConfig.create(DremioConfig.java:343)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.config.DremioConfig.create(DremioConfig.java:339)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.dac.server.DACConfig.newConfig(DACConfig.java:211)
Jan 11 12:11:39 ip-172-31-35-49 dremio: at com.dremio.dac.daemon.AwsConfigurationModule.run(AwsConfigurationModule.java:60)
Cause
In 24.3.x, the COPY INTO feature has a new option to specify what to do (abort or continue) if an error is encountered during the loading process. This introduced a new dremio.conf configuration parameter, paths.copyintoerror, which existed but was unused in 24.2.x. In 24.3.x, this parameter was changed to paths.system_iceberg_tables, and any historical entry of paths.copyintoerror will cause a parsing error on startup.
Steps to Resolve
To resolve the issue, follow these steps:
- Restore back to 24.2.x using the provided restore steps.
- Edit the dremio.conf file and remove the line that starts with "paths.copyintoerrors = "dremioS3:///dremio-xxxxxxxxxx/dremio/copyintoerrors".
- Save the changes to the dremio.conf file.
- Retry the upgrade to 24.3.x.
If you do not have a backup to restore from, the article provides detailed steps to mount the volume to a temporary instance, edit the dremio.conf file directly, and then detach the volume before retrying the upgrade.
Next Steps
Once Dremio has recovered, no further steps are required.
Additional Resources