Summary
The article discusses an issue related to the handling of $date Unix epoch time formats in JSON files by Dremio versions 24.1.0 and above.
Reported Issue
Dremio versions 24.1.0 and above no longer allow the $date Unix epoch time formats from JSON files (e.g., "DateUpdated":{"$date":1553955293028}), which was supported in previous versions up to 24.0.2. This change results in an error when querying datasets with this time format.
Relevant Versions
This issue affects Dremio Software versions 24.1.0 and above, as well as Dremio Cloud.
Troubleshooting Steps
Add additional content.
Cause
Dremio no longer allows the $date Unix epoch time format in versions 24.1.0 and above. Querying a dataset with this time format results in the following error:
Invalid format for $date. Must be wrapped by $numberLong or follow any other valid
ISO-8601 format.
Steps to Resolve
There are two methods that can be implemented to resolve this issue:
- Replace the epoch time formats in the JSON files to a supported format: "YYYY-MM-DD"
- Disable the support flag:
planner.enforce_valid_json_date_format_enabled