Discussion:
Timeout error querying S3 Apache Drill 11
(too old to reply)
Bob Carter
2017-08-03 19:35:58 UTC
Permalink
Raw Message
Error: SYSTEM ERROR: ConnectionPoolTimeoutException: Timeout waiting for connection from pool


[Error Id: c5d6ba1f-2868-48b4-8db0-63e027f15bd2 on db-migrate:31010]

(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (com.amazonaws.AmazonClientException) Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.http.AmazonHttpClient.executeHelper():454
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection():232
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection():199
sun.reflect.GeneratedMethodAccessor9.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():606
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70
com.amazonaws.http.conn.$Proxy58.getConnection():-1
org.apache.http.impl.client.DefaultRequestDirector.execute():456
org.apache.http.impl.client.AbstractHttpClient.execute():906
org.apache.http.impl.client.AbstractHttpClient.execute():805
com.amazonaws.http.AmazonHttpClient.executeHelper():384
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748 (state=,code=0)
0: jdbc:drill:zk=local>

I add the following to the core-site.xml but it does not help.


<property>
<name>fs.s3a.connection.maximum</name>
<value>100</value>
</property>
Padma Penumarthy
2017-08-04 03:13:21 UTC
Permalink
Raw Message
Can you try playing around with these properties and see if that helps ?


<property>
<name>fs.s3a.connection.establish.timeout</name>
<value>5000</value>
<description>Socket connection setup timeout in milliseconds.</description>
</property>

<property>
<name>fs.s3a.connection.timeout</name>
<value>50000</value>
<description>Socket connection timeout in milliseconds.</description>
</property>



________________________________
From: Bob Carter <***@battelleecology.org>
Sent: Thursday, August 3, 2017 12:35 PM
To: ***@drill.apache.org
Subject: Timeout error querying S3 Apache Drill 11

Error: SYSTEM ERROR: ConnectionPoolTimeoutException: Timeout waiting for connection from pool


[Error Id: c5d6ba1f-2868-48b4-8db0-63e027f15bd2 on db-migrate:31010]

(org.apache.drill.exec.work.foreman.ForemanException) Unexpected exception during fragment initialization: Unable to execute HTTP request: Timeout waiting for connection from pool
org.apache.drill.exec.work.foreman.Foreman.run():297
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (com.amazonaws.AmazonClientException) Unable to execute HTTP request: Timeout waiting for connection from pool
com.amazonaws.http.AmazonHttpClient.executeHelper():454
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748
Caused By (org.apache.http.conn.ConnectionPoolTimeoutException) Timeout waiting for connection from pool
org.apache.http.impl.conn.PoolingClientConnectionManager.leaseConnection():232
org.apache.http.impl.conn.PoolingClientConnectionManager$1.getConnection():199
sun.reflect.GeneratedMethodAccessor9.invoke():-1
sun.reflect.DelegatingMethodAccessorImpl.invoke():43
java.lang.reflect.Method.invoke():606
com.amazonaws.http.conn.ClientConnectionRequestFactory$Handler.invoke():70
com.amazonaws.http.conn.$Proxy58.getConnection():-1
org.apache.http.impl.client.DefaultRequestDirector.execute():456
org.apache.http.impl.client.AbstractHttpClient.execute():906
org.apache.http.impl.client.AbstractHttpClient.execute():805
com.amazonaws.http.AmazonHttpClient.executeHelper():384
com.amazonaws.http.AmazonHttpClient.execute():232
com.amazonaws.services.s3.AmazonS3Client.invoke():3528
com.amazonaws.services.s3.AmazonS3Client.invoke():3480
com.amazonaws.services.s3.AmazonS3Client.listObjects():604
org.apache.hadoop.fs.s3a.S3AFileSystem.getFileStatus():962
org.apache.hadoop.fs.s3a.S3AFileSystem.listStatus():734
org.apache.drill.exec.store.dfs.DrillFileSystem.listStatus():522
org.apache.drill.exec.store.dfs.WorkspaceSchemaFactory.accessible():160
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory$FileSystemSchema.<init>():77
org.apache.drill.exec.store.dfs.FileSystemSchemaFactory.registerSchemas():64
org.apache.drill.exec.store.dfs.FileSystemPlugin.registerSchemas():149
org.apache.drill.exec.store.StoragePluginRegistryImpl$DrillSchemaFactory.registerSchemas():396
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():110
org.apache.drill.exec.store.SchemaTreeProvider.createRootSchema():99
org.apache.drill.exec.ops.QueryContext.getRootSchema():164
org.apache.drill.exec.ops.QueryContext.getRootSchema():153
org.apache.drill.exec.ops.QueryContext.getNewDefaultSchema():139
org.apache.drill.exec.planner.sql.SqlConverter.<init>():111
org.apache.drill.exec.planner.sql.DrillSqlWorker.getQueryPlan():101
org.apache.drill.exec.planner.sql.DrillSqlWorker.getPlan():79
org.apache.drill.exec.work.foreman.Foreman.runSQL():1050
org.apache.drill.exec.work.foreman.Foreman.run():280
java.util.concurrent.ThreadPoolExecutor.runWorker():1145
java.util.concurrent.ThreadPoolExecutor$Worker.run():615
java.lang.Thread.run():748 (state=,code=0)
0: jdbc:drill:zk=local>

I add the following to the core-site.xml but it does not help.


<property>
<name>fs.s3a.connection.maximum</name>
<value>100</value>
</property>

Loading...