java.nio.channels.SocketChannel[connection-pending remote=/<IP>:<port>]

Problem

The application log file shows: 

30000 millis timeout while waiting for channel to be ready for connect. ch : java.nio.channels.SocketChannel[connection-pending remote=/192.168.1.1:50010]

All nodes are connected to each other via an internal switch, which is a subnet of 192.168.1.x. This network is not open to public access.

Cause

Each node in the Hadoop cluster has an internal IP (through an internal switch) and external IP address, used to communicate with clients and external apps. Datameer is trying to communicate with the nodes in the Hadoop cluster by using the internal IP addresses.

According the description this is caused by the multi-homed cluster. 

Solution

In this case, in the hdfs-site.xml file a property dfs.client.use.datanode.hostname is set. This is the parameter that should force a client to retrieve a hostname instead of an IP address and perform its own lookup of the hostname to get a routable path to that host. Since Datameer is instantiating its own client instance, it is not aware of this variable automatically. To solve this, add the following line into the Custom Properties for the Datameer Hadoop Cluster settings page:

dfs.client.use.datanode.hostname=true

Further Information

HDFS Support for Multihomed Networks