Issue
Query failed with the below error message while formatting the files
Sample Query:
SELECT * FROM NAS.*."0_0_0.parquet"
Error message:
The file format for 'NAS."*"."0_0_0.parquet"' could not be identified. In order for automatic format detection to succeed, files must include a file extension. Alternatively, manual promotion can be used to explicitly specify the format.
Relevant Versions
Applies to v24.3.0+
Background
In earlier versions (before 24.3.0), the queries containing wildcard in the file path used to work.
Example of the file structure:
Here is the successful query in v21.2.0:
Note: This query will only work when 'Automatically format files into tables when users issue queries' option is enabled in the source settings
Root Cause
Support for this type of query pattern has been removed from version 24.3.0 onwards. As such type of queries could cause various types of issues in the system like out of memory, inconsistent results, etc. Hence, this change was intentional.
Resolution
In the above example, simply format the `0_0_0.parquet` file under the subfolder by running below query or click on the button on the far right to promote the file.
SELECT * FROM NAS.subfolder."0_0_0.parquet"
For detailed steps on how to promote the files or folders in dremio, please refer to our documentation - here