UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z

Problem

Error Message

Error: Fatal Error cause TezChild exit.:java.lang.UnsatisfiedLinkError: org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy()Z
 at org.apache.hadoop.util.NativeCodeLoader.buildSupportsSnappy(Native Method)

Pre-Requirements

All Datameer jobs contains a file with the specific configuration properties from the Datameer application. It is called job-conf.xml. According that configuration file, Datameer was configured with

yarn.application.classpath

/etc/hadoop/conf:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop/.//*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/./:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/.//*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*:/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/.//*:/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/lib/* 

The classpath contains colons, but it should not and have comma separated values instead. To gather the correct YARN classpath for the Datameer client, you can use also the command

yarn classpath | tr ':' ','

on a data node *.

yarn.application.classpath

/etc/hadoop/conf,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop/.//*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/./,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-hdfs/.//*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/.//*,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/lib/*,/opt/cloudera/parcels/CDH/lib/hadoop-mapreduce/.//*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/.//*,/opt/cloudera/parcels/CDH/lib/hadoop/libexec/../../hadoop-yarn/lib/* 

Some configuration properties may contain the following path

LD_LIBRARY_PATH=$LD_LIBRARY_PATH: /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/ 

which contains a leading white space and a trailing slash, but it should not. In known good environments, the following is set and working 

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/opt/cloudera/parcels/CDH/lib/hadoop/lib/native 

Since the path settings are used to lookup and load libraries, wrong path settings can lead into unexpected behavior. Therefore it is recommended to correct the properties on a global level under tab Administration and Hadoop Cluster.

Cause

To make sure matching versions of Snappy are installed in Datameer and on Hadoop cluster it is recommend to verify the checksums from the library files. A version mismatch can result in erroneous behavior during codec loading or job execution.

Troubleshooting

Here you can see a short example. From the <datameer-install-path>

[datameer@datameer current]$ pwd
/opt/datameer/current

1) Check which Snappy libraries are loaded 

lsof -p <datameer pid> | grep snappy

or

lsof -u datameer | grep snappy

2) Execute

[datameer@datameer current]$ md5sum lib/native/libsnappy.so*
<md5sum>  lib/native/libsnappy.so
<md5sum>  lib/native/libsnappy.so.<version>
<md5sum>  lib/native/libsnappy.so.<version>

and from a data node execute

Example 1

[root@datanode ~]# md5sum /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/libsnappy.so*
<md5sum>  /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/libsnappy.so
<md5sum>  /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/libsnappy.so.<version>
<md5sum>  /opt/cloudera/parcels/CDH/lib/hadoop/lib/native/libsnappy.so.<version>

[root@datanode ~]# md5sum /usr/lib/hadoop/lib/native/libsnappy.so*
<md5sum>  /usr/lib/hadoop/lib/native/libsnappy.so
<md5sum>  /usr/lib/hadoop/lib/native/libsnappy.so.<version>
<md5sum>  /usr/lib/hadoop/lib/native/libsnappy.so.<version>

3) Check the page http://localhost:8080/dev/native-libs on the datameer node

4) Check native libraries as per Native Libraries Guide on a data node via

hadoop checknative -a

5) Check operating system version on data nodes and Datameer application host via

uname -a 
cat /etc/lsb-release
cat /etc/redhat-release

6) Check the $LD_LIBRARY_PATH and java.library.path on the involved nodes

Solution

• Check all involved path settings. 
• Correct them, if necessary. 
• Check if the Snappy versions and checksum of the files are the same.
• If not, synchronize the files.