Summary
This article provides steps to resolve an issue where the C3 cache in Dremio becomes corrupted, causing errors when running queries.
Reported Issue
Customers may encounter errors like below when running queries:
IOException: /presentation/data/commercial/emea/crm/eds_user_process_adoption_summary_daily/_delta_log/00000000000000000010.checkpoint.parquet
is not a Parquet file. expected magic number [80, 65, 82, 49] at tail, but found [0, 0, 0, 0]
Overview
Disabling the cache from the UI works temporarily, but the cache files remain in place. To fully resolve the issue, the C3 cache needs to be cleared following the provided steps.
Relevant Versions Tools and Integrations
All Dremio versions
Steps to Resolve
1/ Scale down executors to 0 with command:
kubectl scale statefulsets dremio-executor --replicas=0
PS: keep a note of the amount of replicas you have beforehand with command kubectl get pods | grep dremio-executor
)
You must clear the cache with all executors offline simultaneously. Failure to do so results in executors that have their cache cleared first starting to repopulate the cache from any new queries while work is still taking place on the other executors, leading to an unbalanced cache.
2/ Delete pvc's related to c3
First list them with command kubectl get pvc | grep executor-c3
You should see something like:
dremio-default-executor-c3-0-dremio-executor-0 Bound pvc-98092c11-f9b2-419c-a1f5-081cafeb584a 32Gi RWO default 17m
dremio-default-executor-c3-0-dremio-executor-1 Bound pvc-7a50c7c1-dd17-4af6-b7a1-075818431f78 32Gi RWO default 10d
It's very important not to confuse with your main PVCs. As you can see, the c3 pods are clearly labelled
Now you can delete the pvc's from above list with command below:
kubectl delete pvc dremio-default-executor-c3-0-dremio-executor-0 dremio-default-executor-c3-0-dremio-executor-1
3/ Verify that all the c3 related pvc's are deleted using command kubectl get pvc | grep executor-c3
This should also delete the pv's (and associated c3 files) which you can check with kubectl get pv | grep executor-c3
4/ Now scale up pods back up to original number using command:
kubectl scale statefulsets dremio-executor --replicas=2
(or 3 or whichever number of executor pods you found in step 1)
--> All the pods should now have new pv's attached and fresh c3 files created.
Next Steps
Verify by running some queries and see if c3 files are being updated on your executors with command:
ls -alrth /opt/dremio/cloudcache/c0/cm