Sunday, June 8, 2014

YARN memory configuration guideline

Refer to Determine YARN and MapReduce Memory Configuration Settings.
As a general recommendation, allowing for two Containers per disk and per core gives the best balance for cluster utilization.

1. Determine reserved memory.

Reserved Memory = Reserved for stack memory + Reserved for HBase memory (If HBase is on the same node).
Total Memory per NodeRecommended Reserved System MemoryRecommended Reserved HBase Memory
4 GB1 GB1 GB
8 GB2 GB1 GB
16 GB2 GB2 GB
24 GB4 GB4 GB
48 GB6 GB8 GB
64 GB8 GB8 GB
72 GB8 GB8 GB
96 GB12 GB16 GB
128 GB24 GB24 GB
256 GB32 GB32 GB
512 GB64 GB64 GB

2. Determine max number of Containers per node.

# of Containers = minimum of (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)

Total RAM per NodeRecommended Minimum Container Size
Less than 4 GB256 MB
Between 4 GB and 8 GB512 MB
Between 8 GB and 24 GB1024 MB
Above 24 GB2048 MB

3. Determine RAM per Container.

RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))

4. YARN and MapReduce configuration.

Configuration FileConfiguration SettingValue Calculation
yarn-site.xmlyarn.nodemanager.resource.memory-mb= Containers * RAM-per-Container
yarn-site.xmlyarn.scheduler.minimum-allocation-mb= RAM-per-Container
yarn-site.xmlyarn.scheduler.maximum-allocation-mb= Containers * RAM-per-Container RAM-per-Container
mapred-site.xmlmapreduce.reduce.memory.mb= 2 * RAM-per-Container 0.8 * RAM-per-Container 0.8 * 2 * RAM-per-Container
yarn-site.xml (check) 2 * RAM-per-Container
yarn-site.xml (check) 0.8 * 2 * RAM-per-Container

No comments:

Post a Comment

Popular Posts