After spending a day or two, I found the solution to the issue. We need to
add the following snippet in the core-site.xml
<property>
<name>fs.s3a.aws.credentials.provider</name>
<value>com.amazonaws.auth.InstanceProfileCredentialsProvider</value>
</property>
This explicitly tells drill to use that mode of fetching credentials
instead of relying on env variables or explicit access keys.
I would recommend adding it to the official document so that other folks
can benefit from this. If there is a way to do it myself, let me know (I
can raise a pull request etc too)
Thanks,
Nitin
Post by Nitin SHi ,
I did try that before and it did not work. I was not able to find it in
the documentation
Thanks
Nitin
Post by Khurram FaraazHi Nitin,
You can read the details from here -
http://drill-user.incubator.apache.narkive.com/mP5GE7fq/s3-using-iam-roles
A drill user hit the same issue in the past and he has documented the steps
for others to benefit. Hope that helps.
Regards,
Khurram
Post by Nitin SHello,
I am trying to setup drill in distributed mode and access some parquet
https://urldefense.proofpoint.com/v2/url?u=https-3A__drill.
apache.org_docs_s3-2Dstorage-2Dplugin_&d=DwIBaQ&c=
cskdkSMqhcnjZxdQVpwTXg&r=H5JEl9vb-mBIjic10QAbDD2vkUUKAxjO6wZO322RtdI&m=
I3BfnYvwCJajF5EHuHVLb6Pzlb1C_56Em6a6kgxv1Ac&s=
hQMUN2MEoN9AGwiPvEjbZR7mnh37WG9x2XWhPPm3ky8&e= do not provide any info
on what to do if we are using IAM roles for S3 (instead of static keys)
Kindly advise how to set that up.
Thanks,
Nitin