Problem
After updating the GPFS Connector from 2.2.0 to 2.7.0, Datameer is unable to communicate with GPFS with errors similar to the following:
Caused by: java.io.IOException: BIGC00005E: Path related exception for: gpfs:/biginsights/hive/raw/ent_ns
at com.ibm.biginsights.fs.gpfs.GeneralParallelFileSystem.getFileStatus(GeneralParallelFileSystem.java:787)
at org.apache.hadoop.fs.FileSystem.globStatusInternal(FileSystem.java:1734)
Or
Caused by: java.io.IOException: BIGC00026E: failed to get gpfs details : uri = gpfs:/// conf = Configuration: core-default.xml, core-site.xml, mapred-default.xml, mapred-site.xml, hdfs-default.xml, hdfs-site.xml, pmr-default.xml, pmr-site.xml superGroup = supergrp user = hadoop userGroups = [Ljava.lang.String;@982e33f9 proxyUser = null proxyGroup = null isSuperUser = true
Cause
Datameer natively ships with the BigInsights libraries of the 2.2.0 GPFS connector. The 2.7.0 GPFS connector uses different libraries, jar files, and classes.
Solution
- Stop Datameer.
- Copy
/usr/lpp/mmfs/hadoop/hadoop-gpfs-2.7.0.jar
from a cluster node into the<datameer-install-dir>/webapps/conductor/WEB-INF/lib/
folder. - Delete the existing file
<datameer-install-dir>/lib/native/Linux-amd64-64/libgpfsbiginsights.so
. - Copy the new
libgpfshadoop.so.2.7.0
from a cluster node into the<Datameer-Install-Dir>/lib/native/Linux-amd64-64/
folder. - Start Datameer.
- Add
fs.gpfs.impl=org.apache.hadoop.fs.gpfs.GeneralParallelFileSystem
intoCustom Properties
of the cluster in Datameer.
Comments
0 comments
Please sign in to leave a comment.