Summary
When querying the sys.jobs_recent table—especially to extract or filter based on error messages—users may encounter an error related to field size limits being exceeded. This prevents queries from completing and affects the ability to perform alerting or diagnostics based on job errors.
Relevant Versions
Dremio 24.x and later
Applies to engines enforcing limits.single_field_size_bytes
Troubleshooting Steps
- Field size limits are enforced at the engine level, before the query logic is processed.
- Because of this, using SQL functions like
LIKE,REGEXP_LIKE, orSUBSTRINGdoes not bypass the issue. - Even when attempting to isolate substrings, Dremio will reject fields that exceed the configured size before they reach query evaluation.
Example Use Case That Fails:
SELECT * FROM sys.jobs_recent WHERE error_msg LIKE '%OutOfMemory%';
This will still fail if error_msg exceeds the configured byte limit.
Cause
Dremio enforces a field-level size limit to prevent excessive memory consumption during query execution. This limit is applied before any expressions are evaluated, including LIKE or REGEXP_LIKE.
- Default limit may be too low for long error messages.
- Any field exceeding the configured byte size will trigger a failure during query planning.
Steps to Resolve
Temporary Workaround: Set Field Limit at Session Level
To temporarily increase the allowed size of a single field within your query session, use the following command:
ALTER SESSION SET limits.single_field_size_bytes = <required_size>;
- This setting applies only for the current session or query runner window.
- Ensure this
ALTER SESSIONstatement runs before your target query.
Confirming the Change
You can verify the session-level limit has been set by checking the query profile output.
Future Improvement (Planned):
Dremio engineering is actively working to transition from a field size limit to a row size limit, which will:
- Allow large fields as long as the total row size remains within safe limits.
- Reduce query failures caused by single-column limits.
Next Steps
- Use
ALTER SESSION SET limits.single_field_size_bytes = ...as a temporary measure. - Reattempt your query after applying the new session limit.