Discussion:
Performance issues while fetching data from Amazon S3
Walia, Jyotsana
2018-10-29 17:09:25 UTC
Permalink
Hi

We have Apache Drill and Zookeeper running in Kubernetes cluster. We are using Drill to fetch data from S3 storage. We are using the S3 plugin for this. We are able to successfully fetch the data but it’s taking way too long. The data size is not more than 2GB. What can we do to improve the performance? Would appreciate any help or pointers for the same.

Thanks
Jyotsana

This message may contain information that is confidential or privileged. If you are not the intended recipient, please advise the sender immediately and delete this message. See http://www.blackrock.com/corporate/en-us/compliance/email-disclaimers<http://www.blackrock.com/corporate/compliance/email-disclaimers> for further information. Please refer to http://www.blackrock.com/corporate/en-us/compliance/privacy-policy<http://www.blackrock.com/corporate/compliance/privacy-policy> for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see http://www.blackrock.com/corporate/about-us/contacts-locations.

© 2018 BlackRock, Inc. All rights reserved.
Pritesh Maker
2018-10-30 04:21:25 UTC
Permalink
Jyotsana

There was a similar issue reported recently -
https://issues.apache.org/jira/browse/DRILL-6814 - It could be related to
your use case as well. We are investigating the cause now.

Pritesh

On Mon, Oct 29, 2018 at 11:56 AM Walia, Jyotsana <
Post by Walia, Jyotsana
Hi
We have Apache Drill and Zookeeper running in Kubernetes cluster. We are
using Drill to fetch data from S3 storage. We are using the S3 plugin for
this. We are able to successfully fetch the data but it’s taking way too
long. The data size is not more than 2GB. What can we do to improve the
performance? Would appreciate any help or pointers for the same.
Thanks
Jyotsana
This message may contain information that is confidential or privileged.
If you are not the intended recipient, please advise the sender immediately
and delete this message. See
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_en-2Dus_compliance_email-2Ddisclaimers&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=jJkRysmeFkcKjmGT63_awG_tbtEpDskq5awRi_Ri2Ds&e=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_compliance_email-2Ddisclaimers&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=YIeO96lPXALztZajorU1aWcWc-PY89meeGgHmJUWfZo&e=>
for further information. Please refer to
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_en-2Dus_compliance_privacy-2Dpolicy&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=tAI_Zjt5nJMpmzDLR1D4Ze0yWYDFrdkRnWdorn0kWqo&e=
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_compliance_privacy-2Dpolicy&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=3Q_n3hZ4wUYPrMTY1ZxL4v22vbSn8CKs69LvjFBd6KA&e=>
for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_about-2Dus_contacts-2Dlocations&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=GaEr48Q7dybrZh_-UgySIEM0iuVFDmbk9pjvxn20kDE&e=
.
© 2018 BlackRock, Inc. All rights reserved.
Arina Yelchiyeva
2018-10-30 11:40:22 UTC
Permalink
Though in both cases the problem is the same (slow performance) but Drilll
setup is different.
Jyotsana, you can share more details (including setup details, query
profile etc) in the existing Jira or create new one and link both Jiras.

Kind regards,
Arina
Post by Walia, Jyotsana
Jyotsana
There was a similar issue reported recently -
https://issues.apache.org/jira/browse/DRILL-6814 - It could be related to
your use case as well. We are investigating the cause now.
Pritesh
On Mon, Oct 29, 2018 at 11:56 AM Walia, Jyotsana <
Post by Walia, Jyotsana
Hi
We have Apache Drill and Zookeeper running in Kubernetes cluster. We are
using Drill to fetch data from S3 storage. We are using the S3 plugin for
this. We are able to successfully fetch the data but it’s taking way too
long. The data size is not more than 2GB. What can we do to improve the
performance? Would appreciate any help or pointers for the same.
Thanks
Jyotsana
This message may contain information that is confidential or privileged.
If you are not the intended recipient, please advise the sender
immediately
Post by Walia, Jyotsana
and delete this message. See
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_en-2Dus_compliance_email-2Ddisclaimers&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=jJkRysmeFkcKjmGT63_awG_tbtEpDskq5awRi_Ri2Ds&e=
Post by Walia, Jyotsana
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_compliance_email-2Ddisclaimers&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=YIeO96lPXALztZajorU1aWcWc-PY89meeGgHmJUWfZo&e=
Post by Walia, Jyotsana
for further information. Please refer to
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_en-2Dus_compliance_privacy-2Dpolicy&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=tAI_Zjt5nJMpmzDLR1D4Ze0yWYDFrdkRnWdorn0kWqo&e=
Post by Walia, Jyotsana
<
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_compliance_privacy-2Dpolicy&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=3Q_n3hZ4wUYPrMTY1ZxL4v22vbSn8CKs69LvjFBd6KA&e=
Post by Walia, Jyotsana
for more information about BlackRock’s Privacy Policy.
For a list of BlackRock's office addresses worldwide, see
https://urldefense.proofpoint.com/v2/url?u=http-3A__www.blackrock.com_corporate_about-2Dus_contacts-2Dlocations&d=DwIGaQ&c=cskdkSMqhcnjZxdQVpwTXg&r=zySISmkmM4WNViCKijENtQ&m=T1vd7AGJWtdrS4q2oVJQhw98kscoUBu1yxLPqYcQ6H4&s=GaEr48Q7dybrZh_-UgySIEM0iuVFDmbk9pjvxn20kDE&e=
Post by Walia, Jyotsana
.
© 2018 BlackRock, Inc. All rights reserved.
Loading...