Friday, September 11, 2015

How to enable debug logging for partition pruning phase in Drill

Goal:

How to enable debug logging for partition pruning phase in Drill.
This can help troubleshoot when the partition pruning is not happening due to some reason.

Env:

Drill 1.1

Solution:

1. Add below lines in logback.xml on all drill nodes. 

<logger name="org.apache.drill.exec.planner.logical.partition.PruneScanRule" additivity="false">
  <level value="debug" />
  <appender-ref ref="FILE" />
</logger>

2. Restart all drillbits.

3. Run a query which should utilize partition pruning.

For example:
> select * from dfs.`/drill/part1` where dir0='2015' and dir1='01';
+----------------+-------+-------+
|    columns     | dir0  | dir1  |
+----------------+-------+-------+
| ["1","'111'"]  | 2015  | 01    |
+----------------+-------+-------+
1 row selected (0.219 seconds)
Check the foreman drillbit.log, and below debug information should show:
DEBUG o.a.d.e.p.l.partition.PruneScanRule - Attempting to prune AND(=($1, '2015'), =($2, '01'))
DEBUG o.a.d.e.p.l.partition.PruneScanRule - Pruned [maprfs:/drill/part1/2015/01/01.csv, maprfs:/drill/part1/2015/02/02.csv, maprfs:/drill/part1/2015/03/03.csv, maprfs:/drill/part1/2014/11/11.csv, maprfs:/drill/part1/2014/12/12.csv] => [maprfs:/drill/part1/2015/01/01.csv]

No comments:

Post a Comment

Popular Posts