After converting to an EMR deployment of Datameer, some workbooks fail with the following exception:
awstasks.com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
These workbooks have complex structures that result in many task outputs to merge. In caching scenarios we cache these, and then attempt to merge them during job finalization. This mass merging of files causes the maximum number of S3 connections to be reached, and unreleased.
This issue can be worked around by disabling the Hadoop S3 cache. Apply the following parameter to the job and it will run successfully:
Engineering is working on a fix for this issue in internal project known as DAP-37249.