Every time a job gets executed an error is received:
Running beyond physical memory limits. Current usage: 6.9 GB of 6.5 GB physical memory used - Container killed on request. Exit code is 143
WARN [<timestamp>] [ConcurrentJobExecutor-0] (DefaultMrJobClient.java:185) - attempt_<id>: Container [pid=<pid>,containerID=container_<id>] is running beyond physical memory limits. Current usage: 6.9 GB of 6.5 GB physical memory used; 10.1 GB of 13.6 GB virtual memory used. Killing container. ... Container killed on request. Exit code is 143 Container exited with a non-zero exit code 143 ... Caused by: java.lang.RuntimeException: Task: Container killed by the ApplicationMaster. ...
Increasing the container memory doesn't solve the problem, as the limit will be hit later.
Depending on the environment and cluster distribution, compression is activated. Snappy uses off-heap memory, so the percentage of memory allocated for the JVM must be lowered.
Starting with Datameer 5.6 the properties that should be use to control the memory of tasks are (with their default values):
In this case you want to reduce the
das.job.container.memory-heap-fraction value to something like
0.6 because of Snappy's off-heap memory requirements.
You might also want to increase the container sizes as well, but try the percentage first. Increasing the container size without reducing the
memory-heap-fraction probably results in an unsuccessful run.