Summary
This article covers an issue where users may encounter a "SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708" error when trying to access HDFS/Hive datasets in Dremio. It provides details on the reported issue, relevant versions affected, troubleshooting steps, the root cause, and steps to resolve the problem.
Reported Issue
When trying to access HDFS/Hive datasets, users may encounter the following error: SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708. This error will be visible in the Dremio UI through failing jobs/profiles. Additionally, other related errors may be logged in the Dremio server.log file.
Relevant Versions
This issue applies to Dremio releases prior to 22.1.1 and customer environments configured against HDFS/Hive sources.
Troubleshooting Steps
1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/
2. After navigating to the Raw Profile, click on the Error tab of the profile.
3. Confirm the verbose error stack is the same as below:
SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708 SqlOperatorImpl TABLE_FUNCTION Location 1:0:4 Fragment 1:0 [Error Id: 12ebdfa0-43ca-4089-b6e8-fad5b4759062 on dremio-executor-queries-2.dremio-cluster-pod.dremio.svc.cluster.local:0] (java.util.concurrent.CompletionException) java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 635708 com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():64 java.util.concurrent.CompletableFuture$AsyncRun.run():1640 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():750 Caused By (java.io.IOException) org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 635708 com.dremio.exec.hadoop.FSDataInputStreamWrapper.read():146 com.dremio.io.FilterFSInputStream.read():69 com.dremio.exec.store.dfs.FSInputStreamWithStatsWrapper.read():93 com.dremio.exec.hadoop.HadoopAsyncByteReader.readFully():108 com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():60 java.util.concurrent.CompletableFuture$AsyncRun.run():1640 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():750 Caused By (org.apache.hadoop.HadoopIllegalArgumentException) Invalid buffer, not of length 635708 org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers():137 org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>():48 org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode():86 org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode():170 org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer():433 org.apache.hadoop.hdfs.PositionStripeReader.decode():74 org.apache.hadoop.hdfs.StripeReader.readStripe():390 org.apache.hadoop.hdfs.DFSStripedInputStream.fetchBlockByteRange():507 org.apache.hadoop.hdfs.DFSInputStream.pread():1361 org.apache.hadoop.hdfs.DFSInputStream.read():1570 org.apache.hadoop.fs.FSDataInputStream.read():255 sun.reflect.GeneratedMethodAccessor28.invoke():-1 sun.reflect.DelegatingMethodAccessorImpl.invoke():43 java.lang.reflect.Method.invoke():498 com.dremio.exec.hadoop.FSDataInputStreamWrapper.read():140 com.dremio.io.FilterFSInputStream.read():69 com.dremio.exec.store.dfs.FSInputStreamWithStatsWrapper.read():93 com.dremio.exec.hadoop.HadoopAsyncByteReader.readFully():108 com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():60 java.util.concurrent.CompletableFuture$AsyncRun.run():1640 java.util.concurrent.ThreadPoolExecutor.runWorker():1149 java.util.concurrent.ThreadPoolExecutor$Worker.run():624 java.lang.Thread.run():750 SqlOperatorImpl TABLE_FUNCTION Location 1:0:4 Fragment 1:0 com.dremio.exec.hadoop.HadoopAsyncByteReader(HadoopAsyncByteReader.java:64) ...(:0)
Cause
The SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708 error was due to the HDFS 3.2.1 client shipped with Dremio 21.2.0 and other versions prior to 22.1.1, containing the bug mentioned in HDFS-14373.
Steps to Resolve
To resolve this issue, upgrade Dremio to version 22.1.1 or above (preferably the latest 22.x version at the time of reviewing this article due to added fixes and features).
The Dremio shipped HDFS client was upgraded from 3.2.1 to 3.3.2 from Dremio 22.1.1 on wards, resolving the issue.
Additional Resources
Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/