getObjectMetadata Action for S3 Bucket Policy

Problem

Attempting to create a connection via the S3 Connector to a bucket that doesn't allow the getObjectMetadata action fails with the exception:

WARN [2018-05-04 08:06:58.026] [qtp897697267-14] (DataStore.java:211) - connection fails: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8863CBFD8AD072D3)
awstasks.com.amazonaws.services.s3.model.AmazonS3Exception: Forbidden (Service: Amazon S3; Status Code: 403; Error Code: 403 Forbidden; Request ID: 8863CBFD8AD072D3), S3 Extended Request ID: 2SWqbBj2yI/ie1Mm3KvkBlyd4kArdqfP+oDT/ej+/OAqPyBQBHwdG/uKWaHKlb9uO+QrCxTaijg=
    at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.handleErrorResponse(AmazonHttpClient.java:1586)
    at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeOneRequest(AmazonHttpClient.java:1254)
    at awstasks.com.amazonaws.http.AmazonHttpClient$RequestExecutor.executeHelper(AmazonHttpClient.java:1035)

Solution

• As of Datameer 6.4 there are two S3 Connectors are available - S3 and S3 Native. They have some differences in implementation.
• The S3 connector uses getObjectMetadata to distinguish files and folders. This implementation is more extensive than most others clients as the implementation can be used as a Hadoop file system.
• The S3 Native connector doesn't use getObjectMetadata. However, at the moment it only allows for exporting files into S3 buckets.

According to the tests against the bucket whose policy doesn't allow getObjectMetadata:

• Attempting to create a connection via the S3 connector fails as getObjectMetadata is not allowed by the bucket policy.
• Attempting to create a connection via the S3 Native connector is successful.
• Export jobs created on this S3 Native connector run fine.

You could export data to the bucket whose policy doesn't allow getObjectMetadata via the S3 Native connector. The ability to import via this connector is expected in a future releases.