Problem
A Datameer environment with Smart Execution is connected to a Hadoop cluster (HDP2.1.2). Smart Execution is not working and generates error.
Error message
ERROR [ConcurrentJobExecutor-2] (ClusterSession.java:231) - Failed to generate file for 'GroupByOperation[uuid=<id>]' [7 mins, 44 sec] java.lang.RuntimeException: Job job_<id>_<id> failed! Failure info: NA at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:38) at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:30) ...
Troubleshooting steps
Check if Datameer settings are set correct in etc/das-env.sh for
export TMP=tmp
Review folders via
ls -al /tmp ls -al /<datameer-install-path>/tmp
Cause
The setting for TMP
in etc/das-env.sh
was different from standard.
export TMP=/tmp
versus
export TMP=tmp
The absolute path setting caused problems with Tez and SmallJob execution. The application expects the directory for temporary files within the Datameer directory structure and not under the root directory like /tmp
. Therefore it was necessary to set an relative path.
Solution
Correct the path variable for TMP
. In this case remove the slash (/
). Restart conductor.
Comments
0 comments
Please sign in to leave a comment.