Datameer Package for Cloudera 5.8 Running Against 5.8.3 Shows a Warning: No TokenRenewer defined for token kind kms-dt

Problem

A Datameer package for Cloudera 5.8, such as Datameer-6.1.18-cdh-5.8.0, running against a cluster with Cloudera 5.8.3 displays the following warning message during job execution:

No TokenRenewer defined for token kind kms-dt

Symptoms

When a Datameer environment is configured to use Key Management Server (KMS) to issue delegation tokens, jobs might fail when a token needs to be renewed. The following stack trace is logged in job.log:

WARN [2017-02-10 15:05:05.438] [MrPlanRunnerV2] (Token.java:365) - No TokenRenewer defined for token kind kms-dt
WARN [2017-02-10 15:05:05.438] [MrPlanRunnerV2] (Token.java:124) - Cannot find class for token kind kms-dt
WARN [2017-02-10 15:05:05.439] [MrPlanRunnerV2] (SecurityUtil.java:124) - Failed to cancel token Kind: kms-dt, Service: 192.168.225.172:16000, Ident: 00 0f 73 65 72 76 69 63 65 64 61 74 61 6d 65 65 72 04 79 61 72 6e 00 8a 01 5a 28 69 a9 48 8a 01 5a 4c 76 2d 48 8d 03 5a ad 61 because Token cancel is not supported  for kms-dt tokens
WARN [2017-02-10 15:05:05.467] [MrPlanRunnerV2] (Client.java:1455) - interrupted waiting to send rpc request to server
java.lang.InterruptedException
    at java.util.concurrent.FutureTask.awaitDone(FutureTask.java:404)
    at java.util.concurrent.FutureTask.get(FutureTask.java:191)
    at org.apache.hadoop.ipc.Client$Connection.sendRpcRequest(Client.java:1055)
    at org.apache.hadoop.ipc.Client.call(Client.java:1450)
    at org.apache.hadoop.ipc.Client.call(Client.java:1408)
    at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
    at com.sun.proxy.$Proxy145.cancelDelegationToken(Unknown Source)
    at org.apache.hadoop.yarn.api.impl.pb.client.ApplicationClientProtocolPBClientImpl.cancelDelegationToken(ApplicationClientProtocolPBClientImpl.java:336)
    at sun.reflect.GeneratedMethodAccessor607.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:256)
    at org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
    at com.sun.proxy.$Proxy146.cancelDelegationToken(Unknown Source)
    at org.apache.hadoop.yarn.security.client.RMDelegationTokenIdentifier$Renewer.cancel(RMDelegationTokenIdentifier.java:129)
    at org.apache.hadoop.security.token.Token.cancel(Token.java:395)
    at datameer.dap.common.graphv2.hadoop.SecurityUtil.finaliseDelegationTokenFile(SecurityUtil.java:122)
    at datameer.plugin.spark.cluster.SparkLauncherClusterJob.runImpl(SparkLauncherClusterJob.java:135)
    at datameer.dap.common.graphv2.ClusterJob.run(ClusterJob.java:125)
    at datameer.dap.common.graphv2.ClusterSession.execute(ClusterSession.java:206)
    at datameer.dap.common.graphv2.mixedframework.MixedClusterSession.execute(MixedClusterSession.java:48)
    at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:343)
    at datameer.dap.common.graphv2.MrPlanRunnerV2.run(MrPlanRunnerV2.java:129)
    at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
    at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:109)
    at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:186)
    at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34)
    at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1693)
    at datameer.dap.common.filesystem.Impersonator.doAs(Impersonator.java:31)
    at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30)

Cause

Cloudera made fixes related to token renewal to address underlying defects in maintenance release such as CDH 5.8.3. HADOOP-13155 is one of them.

Datameer packages such as Datameer-6.1.18-cdh-5.8.0 are bundled with JAR files from 5.8 distribution. Those JAR files don't have the latest implementation of DelegationTokenRenewer.

Solution

Implement the following steps:

Check Cloudera package version

[datameer@<host> ~]$ cat /usr/lib/hadoop/cloudera/cdh_version.properties

# Autogenerated build properties
version=2.6.0-cdh5.8.3
git.hash=<hash>
cloudera.hash=<hash>
cloudera.cdh.hash=<hash>
cloudera.cdh-packaging.hash=<hash>
cloudera.base-branch=cdh5-base-2.6.0
cloudera.build-branch=cdh5-2.6.0_5.8.3
cloudera.pkg.version=2.6.0+cdh5.8.3+<ver>
cloudera.pkg.release=1.cdh5.8.3.p<release>
cloudera.cdh.release=cdh5.8.3
cloudera.build.time=<buildTime>

cloudera.pkg.name=hadoop

Stop the Datameer service.
Locate the following JAR files in <Datameer>/webapps/conductor/WEB-INF/lib directory and move it outside of the Datameer directory structure as a backup:
hadoop-common-2.6.0-cdh5.8.0.jar
avro-1.7.6-cdh5.8.0.jar
hadoop-hdfs-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-shuffle-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-app-2.6.0-cdh5.8.0.jar
hadoop-openstack-2.6.0-cdh5.8.0.jar
hadoop-annotations-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-common-2.6.0-cdh5.8.0.jar
hadoop-yarn-api-2.6.0-cdh5.8.0.jar
hadoop-auth-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-core-2.6.0-cdh5.8.0.jar
hadoop-yarn-client-2.6.0-cdh5.8.0.jar
hadoop-aws-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-hs-2.6.0-cdh5.8.0.jar
hadoop-yarn-common-2.6.0-cdh5.8.0.jar
hadoop-common-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-hs-plugins-2.6.0-cdh5.8.0.jar
hadoop-yarn-server-common-2.6.0-cdh5.8.0.jar
hadoop-distcp-2.6.0-cdh5.8.0.jar
hadoop-mapreduce-client-jobclient-2.6.0-cdh5.8.0.jar
Locate all the corresponding JAR files from CDH 5.8.3 distribution and copy it to the above location.
Start the Datameer service.

Articles in this section

Comments

Articles in this section

Related articles