Export to Hiveserver2 fails

Comments

5 comments

  • Joel Stewart

    It appears that the root of the exception is this one: 

    Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.MRVersion

    Based on that, I'm inclined to take a close look at the classpath for the Datameer environment. This particular class is included in the hadoop-mapred-*.jar file. Can you review the class path that is defined within the Datameer Administration -> Hadoop Configuration page to identify if the underlying jar file exists within that path? 

     

    0
    Comment actions Permalink
  • Michael Ahn

    I'm using the auto configuration feature of Datameer for the Hadoop Cluster on CDH 5.11. The YARN classpath is set to

    $HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

    I found the missing class in $HADOOP_COMMON_HOME/lib/hadoop/client-0.20/hadoop-core.jar

    I changed the cluster configuration to Manual and added $HADOOP_COMMON_HOME/lib/hadoop/client-0.20/* to the YARN classpath

    Now it works. Looks like a bug in the cluster auto configuration for CDH.

     

    0
    Comment actions Permalink
  • Joel Stewart

    Thanks for the update Michael, I'm glad to hear that the adjustment to the classpath resolved the issue. With respect to the auto-configuration, did Datameer successfully match the YARN Classpath or the output from 'yarn classpath' as a command? This is the design intention of the auto-configuration.

    In this case, it seems that an additional class was unused in other CDH workflows and was now relevant to these Datameer jobs. 

    0
    Comment actions Permalink
  • Michael Ahn

    The result of yarn classpath command is:

    [datameer@dn183 ~]$ yarn classpath
    /etc/hadoop/conf:/etc/hadoop/conf:/etc/hadoop/conf:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH-5.12.0-1.cdh5.12.0.p0.29/lib/hadoop/libexec/../../hadoop-yarn/lib/*

    If I use this as Yarn Classpath in Datameer with manual cluster configuration the Hive connection is working. The Yarn classpath used in auto configuration (from job-conf.xml) is:

    $HADOOP_CLIENT_CONF_DIR,$HADOOP_CONF_DIR,$HADOOP_COMMON_HOME/*,$HADOOP_COMMON_HOME/lib/*,$HADOOP_HDFS_HOME/*,$HADOOP_HDFS_HOME/lib/*,$HADOOP_YARN_HOME/*,$HADOOP_YARN_HOME/lib/*

    This looks like the default value and is not specific for CDH.

    I found out that you can set the Yarn class path also in auto configuration mode by adding

    yarn.application.classpath=/etc/hadoop/conf,/opt/cloudera/parcels/CDH/lib/hadoop/*,/opt/cloudera/parcels/CDH/lib/hadoop/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/*,/opt/cloudera/parcels/CDH/lib/hadoop-hdfs/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop-yarn/*,/opt/cloudera/parcels/CDH/lib/hadoop-yarn/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/*,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*

    in the box "Hadoop Distribution Specific Properties:"

    0
    Comment actions Permalink
  • Joel Stewart

    Thanks Michael, I'll share this technical feedback with our Product Team so that we can improve the auto configuration in a future release. 

    0
    Comment actions Permalink

Please sign in to leave a comment.