Showing posts with label Hadoop. Show all posts
Showing posts with label Hadoop. Show all posts
Wednesday, April 24, 2019

Some reducer output files are not moved into final output directory after a MapReduce Job with customized FileOutputCommitter is migrated from MRv1 to MRv2.

›
Symptom: After a MapReduce Job with customized FileOutputCommitter got migrated from MRv1 to MRv2, some reducer output files are not moved...
Tuesday, April 23, 2019

What is the difference between mapreduce.fileoutputcommitter.algorithm.version=1 and 2

›
Goal: This article explains the difference between mapreduce.fileoutputcommitter.algorithm.version=1 and 2 using a sample wordcount job.
2 comments:

How to customize FileOutputCommitter for MapReduce job by overwriting Output Format Class

›
Goal: This article explains how to customize FileOutputCommitter for MapReduce job by overwriting Output Format Class. This can be used t...
Wednesday, May 24, 2017

Hive on Tez : How to control the number of Mappers and Reducers

›
Goal: How to control the number of Mappers and Reducers in Hive on Tez.
9 comments:
Tuesday, May 23, 2017

Hive on Tez : How to identify which DAG ID for which Hive query in the same DAG Application Master

›
Goal: Hive on Tez : How to identify which DAG ID for which Hive query in the same DAG Application Master

Hive on Tez : How to identify the reused YARN containers

›
Goal: Hive on Tez : How to identify the reused YARN containers
Tuesday, May 2, 2017

Pig job fails with "org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120"

›
Symptom: Pig job fails with "org.apache.hadoop.mapreduce.counters.LimitExceededException: Too many counters: 121 max=120".
Wednesday, January 11, 2017

How to limit the size for each zookeeper transaction log

›
Goal: How to limit the size for each zookeeper transaction log.
1 comment:

How to purge Zookeeper Transaction Log and Snapshots

›
Goal: How to purge Zookeeper Transaction Log and Snapshots.

How to read or dump the Zookeeper Transaction Log and Snapshots

›
Goal: How to read or dump the Zookeeper Transaction Log and Snapshots.
Friday, October 7, 2016

How to use Drill to parse ResourceManager Rest API results

›
Goal: ResourceManager REST API provides all detailed information regarding YARN applications, metrics of YARN cluster, etc. This article...
1 comment:
Thursday, September 8, 2016

How to use Cgroups with YARN to limit the CPU utilization

›
Goal: YARN parameters like mapreduce.map.cpu.vcores/mapreduce.reduce.cpu.vcores can not hard limit the CPU utilization. This article expl...
2 comments:
Friday, September 2, 2016

How to override the Hive compression algorithm set in Hive

›
Goal: Hive users may set different customized "mapred.output.compression.codec"(same as "mapreduce.output.fileoutputformat....
Wednesday, June 29, 2016

How to troubleshoot Yarn job failure issue

›
Goal: This article explains the troubleshooting methodology for Yarn job failure issue for newcomers.
1 comment:
Thursday, September 3, 2015

ResourceManager fails to transition to Active mode with "InvalidResourceRequestException"

›
Env: Hadoop 2.5.1 Apache Hadoop ResourceManager HA enabled. Symptom: ResourceManager fails to transition to Active mode with "Inv...
Thursday, June 25, 2015

TaskTracker goes down with error "Comparison method violates its general contract"

›
Env: Hadoop version < 1.3 Symptom: TaskTracker goes down with below stacktrace in tasktracker and jobtracker logs:
Wednesday, June 24, 2015

Best practice for YARN resource management

›
The fundamental idea of MRv2(YARN) is to split up the two major functionalities -- resource management and job scheduling/monitoring, into...
Saturday, May 2, 2015

When will Application Master blacklist Node Manager and when will it ignore the blacklist?

›
Goal: This article explains below mechanism of MapReduceV2 job on YARN: When will Application Master(AM) blacklist one Node Manager(NM), ...
Wednesday, April 29, 2015

Need to restart Resource Manager when fair-scheduler.xml is firstly created.

›
Env: Hadoop 2.5.1 Symptom: 1. Once fair-scheduler.xml is created for the first time, Resource Manager(RM) can not load it every 10 seco...
Friday, April 17, 2015

Understand spill file index in MapReduce V2

›
Env Hadoop 2.x MapReduce V2 Goal This article explains when MapReduce V2 starts to generate spill file index files. We will explorer ...