How to limit the Hive log size using RFA instead of default DRFA

Env:

Hive 1.2 on MapR

Goal:

By default, MapR Hive is using DRFA(Daily Rolling File Appender) for log4j. The template for DRFA settings are in /opt/mapr/hive/hive-<version>/conf/hive-log4j.properties.template
Administrators can copy hive-log4j.properties.template to hive-log4j.properties in "conf" directory and make the changes as they want.
However if the daily Hive log is too large and may potentially fill up all the disk space, we can use RFA(Rolling File Appender) instead to set a max size of each log and also the total number of logs.

Solution:

1. Copy /opt/mapr/hive/hive-<version>/conf/hive-log4j.properties.template to hive-log4j.properties in the same directory.
2. Change the DRFA log4j settings to RFA ones.
For example, if we want to limit the size of each hive log to 1MB, and number of hive logs to 3, below is a complete example:

[root@v1 conf]# cat hive-log4j.properties
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
 
# Define some default values that can be overridden by system properties
hive.log.threshold=ALL
hive.root.logger=INFO,RFA
hive.log.dir=/opt/mapr/hive/hive-1.2/logs/${user.name}
hive.log.file=hive.log
 
# Define the root logger to the system property "hadoop.root.logger".
log4j.rootLogger=${hive.root.logger}, EventCounter
 
# Logging Threshold
log4j.threshold=${hive.log.threshold}
 
#
# Rolling File Appender - cap space usage at 1MB
#
log4j.appender.RFA=org.apache.log4j.RollingFileAppender
log4j.appender.RFA.File=${hive.log.dir}/${hive.log.file}
<span style="color: red;">log4j.appender.RFA.MaxFileSize=1MB
log4j.appender.RFA.MaxBackupIndex=3</span>
log4j.appender.RFA.layout=org.apache.log4j.PatternLayout
 
# Pattern format: Date LogLevel LoggerName LogMessage
# Debugging Pattern format
log4j.appender.RFA.layout.ConversionPattern=%d{ISO8601} %-5p [%t]: %c{2} (%F:%M(%L)) - %m%n
 
 
#
# console
# Add "console" to rootlogger above if you want to use this
#
 
log4j.appender.console=org.apache.log4j.ConsoleAppender
log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} [%t]: %p %c{2}: %m%n
log4j.appender.console.encoding=UTF-8
 
#custom logging levels
#log4j.logger.xxx=DEBUG
 
#
# Event Counter Appender
# Sends counts of logging messages at different severity levels to Hadoop Metrics.
#
log4j.appender.EventCounter=org.apache.hadoop.hive.shims.HiveEventCounter
 
 
log4j.category.DataNucleus=ERROR,RFA
log4j.category.Datastore=ERROR,RFA
log4j.category.Datastore.Schema=ERROR,RFA
log4j.category.JPOX.Datastore=ERROR,RFA
log4j.category.JPOX.Plugin=ERROR,RFA
log4j.category.JPOX.MetaData=ERROR,RFA
log4j.category.JPOX.Query=ERROR,RFA
log4j.category.JPOX.General=ERROR,RFA
log4j.category.JPOX.Enhancer=ERROR,RFA
 
 
# Silence useless ZK logs
log4j.logger.org.apache.zookeeper.server.NIOServerCnxn=WARN,RFA
log4j.logger.org.apache.zookeeper.ClientCnxnSocketNIO=WARN,RFA

Thursday, December 3, 2015