Problem
When a job is run in a Kerberos Secured Hadoop Cluster the following error is displayed for all jobs that Datameer submits:
ERROR [2015-01-01 00:00:00.000] [JobScheduler thread-1] (JobScheduler.java:804) - Job 123 failed with exception. java.lang.RuntimeException: java.lang.RuntimeException: Failed to run cluster job for 'File upload job (123): FileUploadCSV#import(Identity)' at datameer.dap.common.graphv2.ConcurrentClusterSession$1.run(ConcurrentClusterSession.java:51) at datameer.dap.common.security.DatameerSecurityService$1.call(DatameerSecurityService.java:135) at datameer.dap.common.security.DatameerSecurityService$1.call(DatameerSecurityService.java:129) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617) at java.lang.Thread.run(Thread.java:745) Caused by: java.lang.RuntimeException: Failed to run cluster job for 'File upload job (123): FileUploadCSV#import(Identity)' at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:187) at datameer.dap.common.graphv2.ConcurrentClusterSession$1.run(ConcurrentClusterSession.java:48) ... 6 more Caused by: java.lang.RuntimeException: Failed to run job : Failed to renew token: Kind: MR_DELEGATION_TOKEN, Service: 10.10.1.123:10020, Ident: (owner=datameer@LAB.DATAMEER.COM, renewer=yarn, realUser=, issueDate=1449323693287, maxDate=1449928493287, sequenceNumber=6, masterKeyId=2) at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:49) at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:31) at datameer.dap.common.graphv2.hadoop.MrJob.runImpl(MrJob.java:228) at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:128) at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:181) ... 7 more Caused by: java.io.IOException: Failed to run job : Failed to renew token: Kind: MR_DELEGATION_TOKEN, Service: 10.10.1.123:10020, Ident: (owner=datameer@LAB.DATAMEER.COM, renewer=yarn, realUser=, issueDate=1449323693287, maxDate=1449928493287, sequenceNumber=6, masterKeyId=2) at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301) at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306) at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:564) at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:559) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671) at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:559) at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:550) at datameer.dap.common.job.mr.DefaultMrJobClient.submitJob(DefaultMrJobClient.java:312) at datameer.dap.common.job.mr.DefaultMrJobClient.runJobImpl(DefaultMrJobClient.java:79) at datameer.dap.common.job.mr.MrJobClient.runJob(MrJobClient.java:34) at datameer.dap.common.graphv2.hadoop.MrJob.runImpl(MrJob.java:216) ... 9 more
Cause
This problem is a Hadoop configuration issue. Specifically, the Job History daemon in the cluster is configured to bind to the IP address of 0.0.0.0 instead of the FQDN for its own host.
Solution
To resolve this issue, change all the binding addresses from 0.0.0.0 for any Hadoop daemon to the actual FQDN of the host that will be running any of those services.
More information might be available in this related community post: http://hortonworks.com/community/forums/topic/yarn-and-kerberos-issue/
If required, contact Datameer Support with further information.
Comments
0 comments
Please sign in to leave a comment.