Attempting to run a job against a cluster using Isilon as the storage backend fails with the following exceptions:
Diagnostics: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];
[system] WARN [2018-09-24 14:21:15.833] [ClusterMetadataUpdater thread-1] (Client.java:711) - Couldn't setup connection for user@REALM.COM to <hostname>.com/<IP Address:Port> javax.security.sasl.SaslException: No common protection layer between client and server
Isilon HDFS clusters require use_ip for tokens to be set to false for the whole cluster. When use_ip is set to false, all delegation tokens will be represented by hostnames rather than IPs. This is a requirement from the architecture of Isilon itself since the Isilon name node is "rolling" among a few servers.
However, due to a bug reported in MAPREDUCE-6565, in HDP environments, execution frameworks will always take the use_ip setting from core-site.xml from its local mr-framework/hadoop/etc/hadoop directory on distributed cache. In HDP's original distribution, core-site is left empty so the Application Master will use the default value (true) for use_ip (hadoop.security.token.service.use_ip). When a job is submitted from client with use_ip=false but the Application Master uses use_ip=false, the AM will not be able to initialize the SASL client with the name node.
Update the hadoop-site.xml file within the Datameer Tez plugin to ensure that the use_ip setting will be set to false.
1. Shut down the Datameer conductor. (./conductor.sh stop)
2. Navigate to <Datameer Home>/plugins and copy the plugin-tez-<version>.zip file to a temporary location.
3. Unzip the plugin file and edit /classes/hadoop-site.xml
4. Within the <configuration> tags add the following:
<description>Value for Isilon</description>
5. Save the xml file and then re-zip the plugin contents to create a new plugin-tez-<version>.zip
6. Replace the original plugin zip with the new modified copy.
7. Restart the Datameer conductor.
Please sign in to leave a comment.