Datameer Job Failure in Kerberos Environment - java.io.IOException: Failed to run job : Failed to renew token

Problem

When a job is run in a Kerberos Secured Hadoop Cluster the following error is displayed  for all jobs that Datameer submits:

ERROR [2015-01-01 00:00:00.000] [JobScheduler thread-1] (JobScheduler.java:804) - Job 123 failed with exception.
java.lang.RuntimeException: java.lang.RuntimeException: Failed to run cluster job for 'File upload job (123): FileUploadCSV#import(Identity)'
	at datameer.dap.common.graphv2.ConcurrentClusterSession$1.run(ConcurrentClusterSession.java:51)
	at datameer.dap.common.security.DatameerSecurityService$1.call(DatameerSecurityService.java:135)
	at datameer.dap.common.security.DatameerSecurityService$1.call(DatameerSecurityService.java:129)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
	at java.lang.Thread.run(Thread.java:745)
Caused by: java.lang.RuntimeException: Failed to run cluster job for 'File upload job (123): FileUploadCSV#import(Identity)'
	at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:187)
	at datameer.dap.common.graphv2.ConcurrentClusterSession$1.run(ConcurrentClusterSession.java:48)
	... 6 more
Caused by: java.lang.RuntimeException: Failed to run job : Failed to renew token: Kind: MR_DELEGATION_TOKEN, Service: 10.10.1.123:10020, Ident: (owner=datameer@LAB.DATAMEER.COM, renewer=yarn, realUser=, issueDate=1449323693287, maxDate=1449928493287, sequenceNumber=6, masterKeyId=2)
	at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:49)
	at datameer.dap.sdk.util.ExceptionUtil.convertToRuntimeException(ExceptionUtil.java:31)
	at datameer.dap.common.graphv2.hadoop.MrJob.runImpl(MrJob.java:228)
	at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:128)
	at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:181)
	... 7 more
Caused by: java.io.IOException: Failed to run job : Failed to renew token: Kind: MR_DELEGATION_TOKEN, Service: 10.10.1.123:10020, Ident: (owner=datameer@LAB.DATAMEER.COM, renewer=yarn, realUser=, issueDate=1449323693287, maxDate=1449928493287, sequenceNumber=6, masterKeyId=2)
	at org.apache.hadoop.mapred.YARNRunner.submitJob(YARNRunner.java:301)
	at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:536)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
	at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:564)
	at org.apache.hadoop.mapred.JobClient$1.run(JobClient.java:559)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
	at org.apache.hadoop.mapred.JobClient.submitJobInternal(JobClient.java:559)
	at org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:550)
	at datameer.dap.common.job.mr.DefaultMrJobClient.submitJob(DefaultMrJobClient.java:312)
	at datameer.dap.common.job.mr.DefaultMrJobClient.runJobImpl(DefaultMrJobClient.java:79)
	at datameer.dap.common.job.mr.MrJobClient.runJob(MrJobClient.java:34)
	at datameer.dap.common.graphv2.hadoop.MrJob.runImpl(MrJob.java:216)
	... 9 more

Cause

This problem is a Hadoop configuration issue. Specifically, the Job History daemon in the cluster is configured to bind to the IP address of 0.0.0.0 instead of the FQDN for its own host.

Solution

To resolve this issue, change all the binding addresses from 0.0.0.0 for any Hadoop daemon to the actual FQDN of the host that will be running any of those services.

More information might be available in this related community post: http://hortonworks.com/community/forums/topic/yarn-and-kerberos-issue/

If required, contact Datameer Support with further information.