Refer to
Determine YARN and MapReduce Memory Configuration Settings.
As a general recommendation, allowing for two Containers per disk and per core gives the best balance for cluster utilization.
1. Determine reserved memory.
Reserved Memory = Reserved for stack memory + Reserved for HBase memory (If HBase is on the same node).
| Total Memory per Node | Recommended Reserved System Memory | Recommended Reserved HBase Memory |
| 4 GB | 1 GB | 1 GB |
| 8 GB | 2 GB | 1 GB |
| 16 GB | 2 GB | 2 GB |
| 24 GB | 4 GB | 4 GB |
| 48 GB | 6 GB | 8 GB |
| 64 GB | 8 GB | 8 GB |
| 72 GB | 8 GB | 8 GB |
| 96 GB | 12 GB | 16 GB |
| 128 GB | 24 GB | 24 GB |
| 256 GB | 32 GB | 32 GB |
| 512 GB | 64 GB | 64 GB |
2. Determine max number of Containers per node.
# of Containers = minimum of (2*CORES, 1.8*DISKS, (Total available RAM) / MIN_CONTAINER_SIZE)
where MIN_CONTAINER_SIZE=
| Total RAM per Node | Recommended Minimum Container Size |
| Less than 4 GB | 256 MB |
| Between 4 GB and 8 GB | 512 MB |
| Between 8 GB and 24 GB | 1024 MB |
| Above 24 GB | 2048 MB |
3. Determine RAM per Container.
RAM-per-Container = maximum of (MIN_CONTAINER_SIZE, (Total Available RAM) / Containers))
4. YARN and MapReduce configuration.
| Configuration File | Configuration Setting | Value Calculation |
| yarn-site.xml | yarn.nodemanager.resource.memory-mb | = Containers * RAM-per-Container |
| yarn-site.xml | yarn.scheduler.minimum-allocation-mb | = RAM-per-Container |
| yarn-site.xml | yarn.scheduler.maximum-allocation-mb | = Containers * RAM-per-Container |
| mapred-site.xml | mapreduce.map.memory.mb | = RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.memory.mb | = 2 * RAM-per-Container |
| mapred-site.xml | mapreduce.map.java.opts | = 0.8 * RAM-per-Container |
| mapred-site.xml | mapreduce.reduce.java.opts | = 0.8 * 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.resource.mb | = 2 * RAM-per-Container |
| yarn-site.xml (check) | yarn.app.mapreduce.am.command-opts | = 0.8 * 2 * RAM-per-Container |
938DA4A79B
ReplyDeleteTakipçi Satın Al
İzlenme Hilesi
En İyi Aksiyon Filmleri
A solid guide that explains how YARN memory configuration impacts container allocation and overall cluster performance. The focus on balancing node resources and proper tuning of parameters like container size is especially useful. Very helpful for understanding Hadoop performance optimization basics.
ReplyDeletebrewery equipment for sale