YARN-site.xml相關:
yarn.scheduler.minimum-allocation-mb ? ?=1024
yarn.scheduler.maximum-allocation-mb ? ?=8192
yarn.nodemanager.vmem-pmem-ratio ? ? ? =2.1
yarn.nodemanager.resource.memory.mb ?=24576
Map-site.xml相關參數
mapreduce.map.java.opts ? ?=Xmx1024m
mapreduce.map.memory.mb ? ?=1536
mapreduce.reduce.java.opts ?=Xmx2560m
mapreduce.reduce.memory.mb ?=3072
map container的內存("mapreduce.map.memory.mb")被設置為1536mb 。am為其申請了2048m的內存,因為am的最小分配(yarn.scheduler.minimum-allocation-mb)是1024.如果map task對內存超過了2048m,nm會kill掉該task。reduce類似。
When a mapreduce job completes you will see several counters dumped at the end of the job.?
?The three memory counters below show how much physical memory was allocated vs virtual memory.?
默認情況下,yarn.nodemanager.vmem-pmem-ratio被設置為2.1,這意味著,每個map或者task任務只能使用2.1倍("mapreduce.reduce.memory.mb") or ("mapreduce.map.memory.mb") 大小的虛擬內存,否則,會被nm殺掉。
例如下面的的報錯信息:
Current usage: 2.1gb of 2.0gb physical memory used; 1.6gb of 3.15gb virtual memory used.?
Killing container.