Summary/Reported Issue
Reflection failing with the error "List field 'value' exceeded the maximum number of elements 4196, actual number of elements is 4457"
Relevant Versions
Reported in 25.0.9
Other versions may be affected
Troubleshooting Steps
The problematic file can be found as part of the error's stack trace:
UNSUPPORTED_OPERATION ERROR: List field 'value' exceeded the maximum number of elements 4196, actual number of elements is 4457
limit 4196
File path /here/you/will/find/the/path/to/the/problematic/file.parquet
Rowgroup index 0
SqlOperatorImpl TABLE_FUNCTION
Location 2:69:8
Fragment 2:0
Cause
The issue is being caused by hitting the soft limit set by the parameter `store.parquet.list_items.threshold`. This value controls the amount of items a List type column can have.
Steps to Resolve
To get rid of this issue you will need to increase this parameter to at least 4458 (or whatever the error reports in the "actual number of elements" + 1).
However, please be aware that this may affect the resource consumption and memory issues may occur due to raising the limits.