Workbook Fails With: Timeout Waiting for Connection From Pool

Problem

After converting to an EMR deployment of Datameer, some workbooks fail with the following exception:

awstasks.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool

Cause

These workbooks have complex structures that result in many task outputs to merge. In caching scenarios we cache these, and then attempt to merge them during job finalization. This mass merging of files causes the maximum number of S3 connections to be reached, and unreleased.

Workaround

This issue can be worked around by disabling the Hadoop S3 cache. Apply the following parameter to the job and it will run successfully:

fs.s3.impl.disable.cache=true

Engineering is working on a fix for this issue in internal project known as DAP-37249.