Container is Running Beyond Physical Memory Limits

Problem

Every time a job is executed an error occurs:

WARN [<timestamp>] [ConcurrentJobExecutor-0] (DefaultMrJobClient.java:185) - attempt_<id>: Container [pid=<pid>,containerID=container_<id>] is running beyond physical memory limits. Current usage: <value> GB of <value> GB physical memory used. 

TEZ jobs may fail with Exit code 143.

Cause

Datameer version 5.6 uses new custom properties as an abstraction layer for several Execution Frameworks.

It may be necessary to configure memory related properties. Our default settings currently are: 

das.job.map-task.memory=2048 
das.job.reduce-task.memory=2048 
das.job.application-manager.memory=2048

Solution

If you receive a container memory error, slightly increase the following in gradual steps

das.job.map-task.memory=3072
das.job.reduce-task.memory=3072
das.job.application-manager.memory=4096

to

das.job.map-task.memory=4096
das.job.reduce-task.memory=4096
das.job.application-manager.memory=6144

To determine memory configuration settings, look into distribution specific documentation from Cloudera or Hortonworks. 

Remove the execution framework specific parameters from Datameer's configuration. I.e.:

mapred.map.child.java.opts=-Xmx<value>m 
mapred.reduce.child.java.opts=-Xmx<value>m 
mapred.job.map.memory.mb=<value> 
mapred.job.reduce.memory.mb=<value>

Further Information

If you are using Cloudera have a look into specific Snappy compression configurations and properties in general.