Symptom
Attempting to run a job that leverages S3 storage, the following error is thrown in the job.log
UnrecoverableException: java.io.InterruptedIOException: getFileStatus on s3://bucketname/Datameer/jobhistory/confID/jobID/job.log: com.amazonaws.SdkClientException: Unable to execute HTTP request: Timeout waiting for connection from pool
Problem
The connection pool can not release connections fast enough for the workloads being run.
Solution
Disable the S3 connection cache. This instructs Datameer jobs to release connections fully after use, instead of leaving them open for faster access.
Set the following property within the Datameer Cluster Configuration:
1. Navigate to theAdmin Tab
2. Click onCluster Properties
3. Within theCustom Properties
text area, add the following key/value pair:fs.s3.impl.disable.cache=true
Note: this may cause S3 related jobs to have a minor reduction in performance, due to needing to re-establish connections for every data retrieval from S3.
Comments
0 comments
Please sign in to leave a comment.