Container is Running Beyond Physical Memory Limits

Problem

Every time a job gets executed an error is received: 

Running beyond physical memory limits. Current usage: 6.9 GB of 6.5 GB physical memory used - Container killed on request. Exit code is 143

WARN [<timestamp>] [ConcurrentJobExecutor-0] (DefaultMrJobClient.java:185) - attempt_<id>: Container [pid=<pid>,containerID=container_<id>] is running beyond physical memory limits. Current usage: 6.9 GB of 6.5 GB physical memory used; 10.1 GB of 13.6 GB virtual memory used. Killing container.
...
Container killed on request. Exit code is 143
Container exited with a non-zero exit code 143
...
Caused by: java.lang.RuntimeException: Task: Container killed by the ApplicationMaster.
...

Increasing the container memory doesn't solve the problem, as the limit will be hit later. 

Cause

Depending on the environment and cluster distribution, compression is activated. Snappy uses off-heap memory, so the percentage of memory allocated for the JVM must be lowered.

Solution

Starting with Datameer 5.6 the properties that should be use to control the memory of tasks are (with their default values):

das.job.map-task.memory=2048
das.job.reduce-task.memory=2048
das.job.application-manager.memory=2048
das.job.container.memory-heap-fraction=0.8 

In this case you want to reduce the das.job.container.memory-heap-fraction value to something like 0.7 or 0.6 because of Snappy's off-heap memory requirements.

You might also want to increase the container sizes as well, but try the percentage first. Increasing the container size without reducing the memory-heap-fraction probably results in an unsuccessful run.