Discussion:
querying aws s3 objects that are server side encrypted
(too old to reply)
Gautam Parai
2017-05-23 23:48:16 UTC
Permalink
Raw Message
Hi Chandrakanth,


A quick google got me this result from AWS:


"We are sorry you have trouble getting this to work. The error is consistent with KMS requiring SigV4. We are still looking at why, by default, the s3 client for node.js doesn't do that. In the interim, you can explicitly set the signatureVersion when creating the s3 client object as follows:

var s3Obj = new AWS.S3({maxRetries: 10, "signatureVersion":"v4"});

Please let us know if this fixed or didn't fix your problem."

Can you try setting an equivalent S3 storage plugin configuration for `signatureVersion`? Please see here for an example https://drill.apache.org/docs/s3-storage-plugin/

Also, it would be good if you could share your S3 storage plugin configuration here.


Thanks,

Gautam

________________________________
From: Chandrakanth Akkinepalli <***@gmail.com>
Sent: Tuesday, May 23, 2017 1:34:25 PM
To: ***@drill.apache.org
Subject: querying aws s3 objects that are server side encrypted

Hi All,
I have configured aws s3 storage plugin to access data in my s3 bucket.

I have placed both un-encrypted and aws-kms sse objects in the bucket.

I am able to query the un-encrypted objects just fine via apache drill.
But i am getting following error when querying aws-kms encrypted objects.

SQL Error: VALIDATION ERROR: Status Code: 400, AWS Service: Amazon S3, AWS
Request ID: ABCDEFGH12345, AWS Error Code: InvalidArgument, AWS Error
Message: Requests specifying Server Side Encryption with AWS KMS managed
keys require AWS Signature Version 4.

SQL Query null


Did anyone else face this issue? How do i resolve it?

Thanks,
Chandra
Chandrakanth Akkinepalli
2017-05-24 12:29:37 UTC
Permalink
Raw Message
Gautam,
Thanks for your response, S3 plugin config is below.
I am not sure if there is an property in fs.s3a.* to set the signature
version ( in core-site.xml). Any recommendation on how to apply it.

{
"type": "file",
"enabled": true,
"connection": "s3a://abcd1234/",
"config": null,
"workspaces": {
"root": {
"location": "/EMR/Data/Test/",
"writable": false,
"defaultInputFormat": null
},
"omniture": {
"location": "/EMR/Data/Omniture_parquet/Omniture_global_data",
"writable": false,
"defaultInputFormat": null
},
"tmp": {
"location": "/tmp",
"writable": true,
"defaultInputFormat": null
}
},
"formats": {
"psv": {
"type": "text",
"extensions": [
"tbl"
],
"delimiter": "|"
},
"csv": {
"type": "text",
"extensions": [
"csv"
],
"delimiter": ","
},
"tsv": {
"type": "text",
"extensions": [
"tsv"
],
"delimiter": "\t"
},
"httpd": {
"type": "httpd",
"logFormat": "%h %t \"%r\" %>s %b \"%{Referer}i\"",
"timestampFormat": null
},
"parquet": {
"type": "parquet"
},
"json": {
"type": "json",
"extensions": [
"json"
]
},
"avro": {
"type": "avro"
},
"sequencefile": {
"type": "sequencefile",
"extensions": [
"seq"
]
},
"csvh": {
"type": "text",
"extensions": [
"csvh"
],
"extractHeader": true,
"delimiter": ","
}
}
}
Post by Gautam Parai
Hi Chandrakanth,
"We are sorry you have trouble getting this to work. The error is
consistent with KMS requiring SigV4. We are still looking at why, by
default, the s3 client for node.js doesn't do that. In the interim, you can
explicitly set the signatureVersion when creating the s3 client object as
var s3Obj = new AWS.S3({maxRetries: 10, "signatureVersion":"v4"});
Please let us know if this fixed or didn't fix your problem."
Can you try setting an equivalent S3 storage plugin configuration for
`signatureVersion`? Please see here for an example
https://drill.apache.org/docs/s3-storage-plugin/
Also, it would be good if you could share your S3 storage plugin configuration here.
Thanks,
Gautam
________________________________
Sent: Tuesday, May 23, 2017 1:34:25 PM
Subject: querying aws s3 objects that are server side encrypted
Hi All,
I have configured aws s3 storage plugin to access data in my s3 bucket.
I have placed both un-encrypted and aws-kms sse objects in the bucket.
I am able to query the un-encrypted objects just fine via apache drill.
But i am getting following error when querying aws-kms encrypted objects.
SQL Error: VALIDATION ERROR: Status Code: 400, AWS Service: Amazon S3, AWS
Request ID: ABCDEFGH12345, AWS Error Code: InvalidArgument, AWS Error
Message: Requests specifying Server Side Encryption with AWS KMS managed
keys require AWS Signature Version 4.
SQL Query null
Did anyone else face this issue? How do i resolve it?
Thanks,
Chandra
Loading...