Tuesday, June 28, 2016

Drill query fails with error "Unable to allocate buffer of size 256 due to memory limit. Current allocation: 268435456"

Symptom:

Drill query fails with error message like below:
[<SQL_ID>:foreman] WARN  o.a.d.e.p.l.partition.PruneScanRule - Exception while trying to prune partition.
org.apache.drill.exec.exception.OutOfMemoryException: Unable to allocate buffer of size 256 due to memory limit. Current allocation: 268435456
This error always shows in the foreman drillbit.log during planning phase.
In above example, it fails when doing partition pruning in planning phase.

Root Cause:

Starting in Drill 1.2, parameter "planner.memory_limit" defines the maximum amount of direct memory allocated to a query for planning. By default it is set to 256MB(268435456 bytes).
During partition pruning, if the query needs to filter more partitions, it needs more memory during planning.

Solution:

Increase planner.memory_limit to large enough per query. 
For example, set it to 512MB:
alter session set `planner.memory_limit`=536870912;


No comments:

Post a Comment

Popular Posts