Articles in this section

See more

Operation category READ is not supported in state standby

Joel Stewart
  • Updated
Follow

Problem

In a Datameer environment that is connected to an highly-available (HA) Hadoop Namenode (NN) service, all Datameer jobs begin to fail after the Namenode fails over. The following stacktrace is observed in the logs:

Operation category READ is not supported in state standby
 at org.apache.hadoop.hdfs.server.namenode.ha.StandbyState.checkOperation(StandbyState.java:870)
 at org.apache.hadoop.hdfs.server.namenode.NameNode$NameNodeHAContext.checkOperation(NameNode.java:1408)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkOperation(FSNamesystem.java:969)
 at org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getFileInfo(FSNamesystem.java:3020)
 at org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getFileInfo(NameNodeRpcServer.java:694)
 at org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getFileInfo(ClientNamenodeProtocolServerSideTranslatorPB.java:646)
 at org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java:45002)
 at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453)
 at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1752)
 at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1748)
 at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
 at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1746)

Cause

This is mostly caused by incorrect settings within Datameers Hadoop configuration. 

Workaround

There are two possible work-arounds for this issue:

  • Datameer administrators may reconfigure the Hadoop Configuration to point to the active node.
  • or Hadoop administrators may fail the Namenode service back to the originally active node (if it is available again).

Solution

Review and correct the settings.

To investigate this in more detail, please contact Datameer Support and provide  the conductor logs from the Datameer instance and the logs from all namenode servers from the Hadoop cluster. 

Related articles

Comments

0 comments

Please sign in to leave a comment.