Articles in this section

See more

Retrieving Job Status from Job History - NullPointerException: HistoryClientService$HSClientProtocolHandler.getCounters

Joel Stewart
  • Updated
Follow

Environment

DM: 4.5 and higher, OS: Linux, DIST: CDH5.1, COM: Kerberos, YARN

This issue has been observed with CDH 5.1 clusters, but may also pertain to other cluster environments as well. 

Problem

After configuring Datameer to connect to a Kerberos enabled Hadoop cluster with YARN, the user attempts to run a job on the Datameer host (i.e. the Cluster Health job). The job completes successfully according to the YARN Job Tracker, but the job fails according to the Datameer Server. When reviewing the job logs, the following stack trace is logged:

WARN [2014-01-01 00:00:00.000] [MrPlanRunnerThread-0] (UserGroupInformation.java:1551) - PriviledgedActionException as:datameer@DATAMEER.COM (auth:KERBEROS) cause:java.io.IOException: org.apache.hadoop.ipc.RemoteException(java.lang.NullPointerException): java.lang.NullPointerException
	at org.apache.hadoop.mapreduce.v2.hs.HistoryClientService$HSClientProtocolHandler.getCounters(HistoryClientService.java:223)
	at org.apache.hadoop.mapreduce.v2.api.impl.pb.service.MRClientProtocolPBServiceImpl.getCounters(MRClientProtocolPBServiceImpl.java:159)
	at org.apache.hadoop.yarn.proto.MRClientProtocol$MRClientProtocolService$2.callBlockingMethod(MRClientProtocol.java:281)
	at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:585)
	at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1026)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1986)
	at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1982)
   	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:415)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1554)
	at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1980)

Cause

The cause is a settings issue between the Datameer Server and the Hadoop cluster.

Solution

To resolve this issue, follow these steps to add a custom Hadoop property to the Datameer settings.

  1. Check the current value of the yarn.app.mapreduce.am.staging-dir property on the Hadoop cluster.
  2. Using the value, add a line to the Custom Property settings of the Hadoop Cluster page on the Administration tab.
  3. Save these settings and then re-run the affect job to verify that the getCounters method no longer generates a NullPointerException.

 

 

Related articles

Comments

0 comments

Please sign in to leave a comment.