OUT_OF_MEMORY ERROR with Unable to allocate buffer of size "dremio-client-connections" – Dremio Support

Summary

This article explains how to deal with JDBC client queries failing in Dremio with OUT_OF_MEMORY ERROR: Query was cancelled because it exceeded the memory limits set by the administrator. where the failure also reports "(cdjd.org.apache.arrow.memory.OutOfMemoryException) Unable to allocate buffer of size 2097152 (rounded from 1284096) due to memory limit. Current allocation: 65712128".

Reported Issue

When attempting to run queries from IBM Cognos via the Dremio JDBC driver, the query fails in the client as well as at Dremio where the failure job in Dremio reports the below error in the job profile indicating the query is reaching a 64 MB memory limit. NOTE: This can also happen with other IBM client applications such as IBM DataStage.

OUT_OF_MEMORY ERROR: Query was cancelled because it exceeded the memory limits set by the administrator.

Unable to allocate buffer of size 2097152 (rounded from 1284096) due to memory limit. Current allocation: 65712128
Allocation outcome details:
allocator[dremio-client-connections] reservation: 0 limit: 67108864 used: 65712128 requestedSize: 2097152 allocatedSize: 0 localAllocationStatus: fail

Allocator dominators:
Allocator(dremio-client-connections) 0/65712128/66588672/67108864 (res/actual/peak/limit) numChildAllocators:0

IBM Cognos would log a failure such as the below error:

Data source adapter error: java.sql.SQLException: IllegalStateException: Attempted to send a message when connection is no longer valid Failure sending message. Attempted to send a message when connection is no longer valid

Relevant Versions

This issue applies to Dremio versions 24.x and onwards.

Troubleshooting Steps

1. Navigate to the failing job profile as per the following documentation link steps: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/

2. After navigating to the "Raw Profile", click on the "Error" tab of the profile.

3. Confirm that the error is similar to the below where the memory allocator mentioned in the error is allocator[dremio-client-connections].

OUT_OF_MEMORY ERROR: Query was cancelled because it exceeded the memory limits set by the administrator.

Unable to allocate buffer of size 2097152 (rounded from 1284096) due to memory limit. Current allocation: 65712128
Allocation outcome details:
allocator[dremio-client-connections] reservation: 0 limit: 67108864 used: 65712128 requestedSize: 2097152 allocatedSize: 0 localAllocationStatus: fail

Allocator dominators:
Allocator(dremio-client-connections) 0/65712128/66588672/67108864 (res/actual/peak/limit) numChildAllocators:0


[Error Id: 5d468695-8593-4271-b95a-0360d73db569 ]

  (cdjd.org.apache.arrow.memory.OutOfMemoryException) Unable to allocate buffer of size 2097152 (rounded from 1284096) due to memory limit. Current allocation: 65712128
    cdjd.org.apache.arrow.memory.BaseAllocator.buffer():270
    cdjd.org.apache.arrow.memory.BaseAllocator.buffer():240
    cdjd.com.dremio.exec.rpc.MessageDecoder.decodeMessage():174
    cdjd.com.dremio.exec.rpc.MessageDecoder.decode():71
    cdjd.com.dremio.sabot.rpc.user.UserProtobufLengthDecoder.decode():35
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection():529
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.callDecode():468
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.channelRead():290
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():444
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():420
    cdjd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead():412
    cdjd.io.netty.handler.ssl.SslHandler.unwrap():1382
    cdjd.io.netty.handler.ssl.SslHandler.decodeJdkCompatible():1245
    cdjd.io.netty.handler.ssl.SslHandler.decode():1294
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection():529
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.callDecode():468
    cdjd.io.netty.handler.codec.ByteToMessageDecoder.channelRead():290
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():444
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():420
    cdjd.io.netty.channel.AbstractChannelHandlerContext.fireChannelRead():412
    cdjd.io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead():1410
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():440
    cdjd.io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead():420
    cdjd.io.netty.channel.DefaultChannelPipeline.fireChannelRead():919
    cdjd.io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read():166
    cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKey():788
    cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeysOptimized():724
    cdjd.io.netty.channel.nio.NioEventLoop.processSelectedKeys():650
    cdjd.io.netty.channel.nio.NioEventLoop.run():562
    cdjd.io.netty.util.concurrent.SingleThreadEventExecutor$4.run():997
    cdjd.io.netty.util.internal.ThreadExecutorMap$2.run():74
    java.lang.Thread.run():825

4. Carry out the "Steps to Resolve" described below.

Cause

This is a client-side configuration issue where the client has a direct memory setting (-XX:MaxDirectMemorySize) for which the value is at or around 64 MB which is being reached when the query's result set is large due to a combination of a high number of columns, and/or a large row size, along with a large enough number of rows to exceed the allocated memory buffer.

Steps to Resolve

Increase the direct memory setting (-XX:MaxDirectMemorySize) on the client and retry the query against Dremio.

Additional Resources

Viewing a Raw Profile: https://docs.dremio.com/current/sonar/monitoring/jobs/raw-profile/