Problem
Attempting to run a job against a cluster using Isilon as the storage backend fails with the following exceptions:
Diagnostics: Failed on local exception: java.io.IOException: org.apache.hadoop.security.AccessControlException: Client cannot authenticate via:[TOKEN, KERBEROS];
...
[system] WARN [2018-09-24 14:21:15.833] [ClusterMetadataUpdater thread-1] (Client.java:711) - Couldn't setup connection for user@REALM.COM to <hostname>.com/<IP Address:Port> javax.security.sasl.SaslException: No common protection layer between client and server
at com.sun.security.sasl.gsskerb.GssKrb5Client.doFinalHandshake(GssKrb5Client.java:251)
at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:186)
Cause
Isilon HDFS clusters require use_ip for tokens to be set to false for the whole cluster. When use_ip is set to false, all delegation tokens will be represented by hostnames rather than IPs. This is a requirement from the architecture of Isilon itself since the Isilon name node is "rolling" among a few servers.
However, due to a bug reported in MAPREDUCE-6565, in HDP environments, execution frameworks will always take the use_ip setting from core-site.xml from its local mr-framework/hadoop/etc/hadoop directory on distributed cache. In HDP's original distribution, core-site is left empty so the Application Master will use the default value (true) for use_ip (hadoop.security.token.service.use_ip). When a job is submitted from client with use_ip=false but the Application Master uses use_ip=false, the AM will not be able to initialize the SASL client with the name node.
Solution
Update the hadoop-site.xml file within the Datameer Tez plugin to ensure that the use_ip setting will be set to false.
1. Shut down the Datameer conductor. (./conductor.sh stop)
2. Navigate to <Datameer Home>/plugins and copy the plugin-tez-<version>.zip file to a temporary location.
3. Unzip the plugin file and edit /classes/hadoop-site.xml
4. Within the <configuration> tags add the following:
<property>
<name>hadoop.security.token.service.use_ip</name>
<value>false</value>
<description>Value for Isilon</description>
</property>
5. Save the xml file and then re-zip the plugin contents to create a new plugin-tez-<version>.zip
6. Replace the original plugin zip with the new modified copy.
7. Restart the Datameer conductor.
Comments
0 comments
Please sign in to leave a comment.