AWS Access Denied Error with Server Side Encryption (SSE) Enabled

Environment

DM: 5.x, DIST: HDP 2.1, OS: Linux, COM: -

Problem

Setting up a connection to Amazon S3 bucket failed with following error message:

AmazonS3Exception: Status Code: 403, AWS Service: Amazon S3, AWS Request ID: <id>, AWS Error Code: AccessDenied, AWS Error Message: Access Denied, S3 Extended Request ID: <id>
at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClient.java:686)
at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:350)
at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:202)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3066)
at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3037)
at com.amazonaws.services.s3.AmazonS3Client.listObjects(AmazonS3Client.java:533)
at datameer.dap.hadoop.filesystem.DatameerS3FileSystem$ListingIterator.computeNext(DatameerS3FileSystem.java:617)
at datameer.dap.hadoop.filesystem.DatameerS3FileSystem$ListingIterator.computeNext(DatameerS3FileSystem.java:605)
at datameer.com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at datameer.com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at datameer.dap.hadoop.filesystem.DatameerS3FileSystem.listStatus(DatameerS3FileSystem.java:282)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1483)
at org.apache.hadoop.fs.FileSystem.listStatus(FileSystem.java:1523)
at datameer.dap.sdk.cluster.filesystem.HadoopFileSystem.listStatus(HadoopFileSystem.java:124)
at datameer.dap.sdk.util.DatameerFsClient.listStatus(DatameerFsClient.java:53)
at datameer.dap.sdk.util.DatameerFsClient.listStatus(DatameerFsClient.java:46)
at datameer.dap.sdk.datastore.FileDataStoreModel.testConnect(FileDataStoreModel.java:56)
at datameer.dap.sdk.entity.DataStore.validate(DataStore.java:186)
...

Cause

Server Side Encryption (SSE) is required for to write. The job is attempting to do a test and is getting denied without SSE. 

The ability to implement AES 256 encryption in Hadoop was not added until the 2.5.0 distribution of Hadoop. Refer to Add S3 Server Side Encryption for background information. 

Apache Hadoop 2.6 release is supported in HDP 2.2 and beyond. 

Solution

Set the following value as either a Custom Property in Datameer or in the core-site.xml file in your Hadoop cluster: 

fs.s3n.server-side-encryption-algorithm=AES256

Workaround

Since this parameter must be set at the Apache Hadoop level, it is necessary to upgrade to HDP 2.2. As a workaround prior to the HDP 2.2 release, disable Server Side Encryption (SSE) on the specific S3 buckets that need to be accessed.