Error Connecting to Hadoop Cluster (DM Admin)

Problem

When attempting to connect a freshly upgraded Datameer to Cloudera 5.9.0, we received the following error message...

Error:

java.io.IOException: Connection reset by peer
    at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
    at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
    at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
    at sun.nio.ch.IOUtil.read(IOUtil.java:197)
    at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:380)
    at org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
    at org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:131)
    at org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:118)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at java.io.FilterInputStream.read(FilterInputStream.java:83)
    at org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed

Cause

When Datameer is attempting to connect to a secured Cluster environment, it is required to have all security layers declared in Datameer's custom properties. Once Datameer has connected to Hadoop, it can harvest the rest of the configuration details in order to build applications to be processed by Hadoop.

Solution

In this particular case, we were required to investigate the settings on the cluster and explicitly set the following in Datameer to match the cluster's configuration:

dfs.data.transfer.protection=privacy

Moving Forward

Please note that starting in Datameer 6, we have an auto configuration option available that is dependent upon having a valid Hadoop client installed on the Datameer host.  

Datameer will have direct access to all pertinent configurations through this channel and will require much less configuration from an Admin's point of view.