How to limit the Hive log size using RFA instead of default DRFA in Hive 2.x using log4j2

Env:

Hive 2.x on MapR

Goal:

By default, MapR Hive is using DRFA(Daily Rolling File Appender) for log4j2. The template for DRFA settings are in /opt/mapr/hive/hive-<version>/conf/hive-log4j2.properties.template
Administrators can copy hive-log4j2.properties.template to hive-log4j2.properties in "conf" directory and make the changes as they want.
However if the daily Hive log is too large and may potentially fill up all the disk space, we can use RFA(Rolling File Appender) instead to set a max size of each log and also the total number of logs.

Note:  Per HIVE-11304, Hive upgraded log4j 1.x to log4j2. So that the previous article is only for Hive 1.x.

Solution:

1. Copy /opt/mapr/hive/hive-<version>/conf/hive-log4j2.properties.template to hive-log4j2.properties in the same directory.

2.  Change the DRFA log4j2 settings to RFA ones.

 For example, if we want to limit the size of each hive log to 1MB, and number of hive logs to 3, below is a complete example:

status = INFO
name = HiveLog4j2
packages = org.apache.hadoop.hive.ql.log

# list of properties
property.hive.log.level = INFO
property.hive.root.logger = RFA
property.hive.log.dir = /opt/mapr/hive/hive-2.1/logs/${sys:user.name}
property.hive.log.file = hive.log
property.hive.perflogger.log.level = INFO

# list of all appenders
appenders = console, RFA

# console appender
appender.console.type = Console
appender.console.name = console
appender.console.target = SYSTEM_ERR
appender.console.layout.type = PatternLayout
appender.console.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n

# daily rolling file appender
appender.RFA.type = RollingRandomAccessFile
appender.RFA.name = RFA
appender.RFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
# Use %pid in the filePattern to append <process-id>@<host-name> to the filename if you want separate log files for different CLI session
appender.RFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
appender.RFA.layout.type = PatternLayout
appender.RFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
appender.RFA.policies.type = Policies
appender.RFA.policies.size.type = SizeBasedTriggeringPolicy
appender.RFA.policies.size.size = 1MB
appender.RFA.strategy.type = DefaultRolloverStrategy
appender.RFA.strategy.max = 3

Here are the difference:

$ diff hive-log4j2.properties hive-log4j2.properties.template
23c23
< property.hive.root.logger = RFA
---
> property.hive.root.logger = DRFA
29c29
< appenders = console, RFA
---
> appenders = console, DRFA
39,41c39,41
< appender.RFA.type = RollingRandomAccessFile
< appender.RFA.name = RFA
< appender.RFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
---
> appender.DRFA.type = RollingRandomAccessFile
> appender.DRFA.name = DRFA
> appender.DRFA.fileName = ${sys:hive.log.dir}/${sys:hive.log.file}
43,50c43,51
< appender.RFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
< appender.RFA.layout.type = PatternLayout
< appender.RFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
< appender.RFA.policies.type = Policies
< appender.RFA.policies.size.type = SizeBasedTriggeringPolicy
< appender.RFA.policies.size.size = 1MB
< appender.RFA.strategy.type = DefaultRolloverStrategy
< appender.RFA.strategy.max = 3
---
> appender.DRFA.filePattern = ${sys:hive.log.dir}/${sys:hive.log.file}.%d{yyyy-MM-dd}
> appender.DRFA.layout.type = PatternLayout
> appender.DRFA.layout.pattern = %d{ISO8601} %5p [%t] %c{2}: %m%n
> appender.DRFA.policies.type = Policies
> appender.DRFA.policies.time.type = TimeBasedTriggeringPolicy
> appender.DRFA.policies.time.interval = 1
> appender.DRFA.policies.time.modulate = true
> appender.DRFA.strategy.type = DefaultRolloverStrategy
> appender.DRFA.strategy.max = 30