Summary/Reported Issue
I am getting the below error.
VALIDATION ERROR: At most 800 columns including leaf level fields of complex type are allowed to be scanned, but the query is scanning 839. Please include the columns you want to be returned from the query and try again
Relevant Versions
Dremio 25.1.x
Other versions may be affected
Troubleshooting Steps
This error is triggered whenever one of the limits determined by both `store.plugin.max_leaf_columns_scanned` and `store.plugin.max_metadata_leaf_columns` is surpassed.
Both of these parameters together control the max number of columns in a table.
Both of these parameters were set to 800 in earlier versions of Dremio. However, in 25.x their default values should be:
- `store.plugin.max_leaf_columns_scanned`: 1500
- `store.plugin.max_metadata_leaf_columns`: 6400
Cause
Soft limits for max column number are being surpassed.
Steps to Resolve
If you're coming from an older version of Dremio to the latest, it's likely you had these values set when you were in a lower version of Dremio and simply kept it after the upgrade. Unless someone from Dremio has told you to reduce either of these values, you should be safe reverting them back to their defaults (as of 25.x).
If you're surpassing the default values of these parameters as of 25.x, we recommend checking with support before updating these values further.