SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708 – Dremio Support

Summary

This article covers an issue where users may encounter a "SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708" error when trying to access HDFS/Hive datasets in Dremio. It provides details on the reported issue, relevant versions affected, troubleshooting steps, the root cause, and steps to resolve the problem.

Reported Issue

When trying to access HDFS/Hive datasets, users may encounter the following error: SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708. This error will be visible in the Dremio UI through failing jobs/profiles. Additionally, other related errors may be logged in the Dremio server.log file.

Relevant Versions

This issue applies to Dremio releases prior to 22.1.1 and customer environments configured against HDFS/Hive sources.

Troubleshooting Steps

1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/

2. After navigating to the Raw Profile, click on the Error tab of the profile.

3. Confirm the verbose error stack is the same as below:

         SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708

SqlOperatorImpl TABLE_FUNCTION
Location 1:0:4
Fragment 1:0

[Error Id: 12ebdfa0-43ca-4089-b6e8-fad5b4759062 on dremio-executor-queries-2.dremio-cluster-pod.dremio.svc.cluster.local:0]

  (java.util.concurrent.CompletionException) java.io.IOException: org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 635708
    com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():64
    java.util.concurrent.CompletableFuture$AsyncRun.run():1640
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():750
  Caused By (java.io.IOException) org.apache.hadoop.HadoopIllegalArgumentException: Invalid buffer, not of length 635708
    com.dremio.exec.hadoop.FSDataInputStreamWrapper.read():146
    com.dremio.io.FilterFSInputStream.read():69
    com.dremio.exec.store.dfs.FSInputStreamWithStatsWrapper.read():93
    com.dremio.exec.hadoop.HadoopAsyncByteReader.readFully():108
    com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():60
    java.util.concurrent.CompletableFuture$AsyncRun.run():1640
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():750
  Caused By (org.apache.hadoop.HadoopIllegalArgumentException) Invalid buffer, not of length 635708
    org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.checkOutputBuffers():137
    org.apache.hadoop.io.erasurecode.rawcoder.ByteBufferDecodingState.<init>():48
    org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode():86
    org.apache.hadoop.io.erasurecode.rawcoder.RawErasureDecoder.decode():170
    org.apache.hadoop.hdfs.StripeReader.decodeAndFillBuffer():433
    org.apache.hadoop.hdfs.PositionStripeReader.decode():74
    org.apache.hadoop.hdfs.StripeReader.readStripe():390
    org.apache.hadoop.hdfs.DFSStripedInputStream.fetchBlockByteRange():507
    org.apache.hadoop.hdfs.DFSInputStream.pread():1361
    org.apache.hadoop.hdfs.DFSInputStream.read():1570
    org.apache.hadoop.fs.FSDataInputStream.read():255
    sun.reflect.GeneratedMethodAccessor28.invoke():-1
    sun.reflect.DelegatingMethodAccessorImpl.invoke():43
    java.lang.reflect.Method.invoke():498
    com.dremio.exec.hadoop.FSDataInputStreamWrapper.read():140
    com.dremio.io.FilterFSInputStream.read():69
    com.dremio.exec.store.dfs.FSInputStreamWithStatsWrapper.read():93
    com.dremio.exec.hadoop.HadoopAsyncByteReader.readFully():108
    com.dremio.exec.hadoop.HadoopAsyncByteReader.lambda$readFully$0():60
    java.util.concurrent.CompletableFuture$AsyncRun.run():1640
    java.util.concurrent.ThreadPoolExecutor.runWorker():1149
    java.util.concurrent.ThreadPoolExecutor$Worker.run():624
    java.lang.Thread.run():750


SqlOperatorImpl TABLE_FUNCTION
Location 1:0:4
Fragment 1:0

com.dremio.exec.hadoop.HadoopAsyncByteReader(HadoopAsyncByteReader.java:64)
...(:0)

Cause

The SYSTEM ERROR: HadoopIllegalArgumentException: Invalid buffer, not of length 635708 error was due to the HDFS 3.2.1 client shipped with Dremio 21.2.0 and other versions prior to 22.1.1, containing the bug mentioned in HDFS-14373.

Steps to Resolve

To resolve this issue, upgrade Dremio to version 22.1.1 or above (preferably the latest 22.x version at the time of reviewing this article due to added fixes and features).

The Dremio shipped HDFS client was upgraded from 3.2.1 to 3.3.2 from Dremio 22.1.1 on wards, resolving the issue.

Additional Resources

Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/