Discussion:
Failed to create schema tree when running Drill View Query
Divya Gehlot
2018-10-11 03:06:55 UTC
Permalink
Hi ,
At times I get below error when running the View query which fetches the
whole data set

org.apache.drill.common.exceptions.UserRemoteException: RESOURCE ERROR:
Failed to create schema tree.


The Strange behaviour is if I wait for sometime for say 2-3 minutes and run
the query again it works perfectly .


Appreciate if anybody can help me to get me the root cause for it !


Thanks,

Divya
Abhishek Girish
2018-10-11 03:14:57 UTC
Permalink
Hey Divha,

That's usually seen when the underlying FS cannot recognize the session
user when creating the schema tree. For example if impersonation is enabled
and no user is passed, Drill tries to use an "anonymous" user. I've seen
that in case of MapR-FS, that it cannot use such a user to proceed with
accessing the filesystem.

Can you try if (1) username is passed when launching your client such as
Sqlline, (2), the user (with same UID) exists on all nodes on the cluster.
Post by Divya Gehlot
Hi ,
At times I get below error when running the View query which fetches the
whole data set
Failed to create schema tree.
The Strange behaviour is if I wait for sometime for say 2-3 minutes and run
the query again it works perfectly .
Appreciate if anybody can help me to get me the root cause for it !
Thanks,
Divya
Divya Gehlot
2018-10-11 07:09:18 UTC
Permalink
Hi Abhishek,
Thanks for the response with clarifications !
When I jumped into the logs ..
May be I should provide some additional information regarding the Drill
cluster before sharing more information what I found in the logs
I have 6 nodes cluster where the 3 nodes are old and 3 are newly added
nodes
Old Nodes - dn1,dn2 and dn3
New Nodes - dn4,dn5,dn6
Whenever the query foreman is newly added nodes the error occurs , I don't
find this error in any of the old nodes drillbit.log
Even the user is Drill admin user I do see the above mentioned error .
I verified that Drill admin has same UID and GID across all the nodes .
I did check the uid is being passed while running the query
INFO o.a.d.exec.store.SchemaTreeProvider - User Error Occurred: Failed to
create schema tree. (Error getting user info for current user, 992521597)
Above error I could see in the drillbit.log files of dn4, dn5 and dn6 only
I dont see any error like this on dn1,dn2 or dn3 nodes.
Any thoughts why is it happening despite the same uid across all the nodes .



Thanks,
Divya
Hey Divha,
That's usually seen when the underlying FS cannot recognize the session
user when creating the schema tree. For example if impersonation is enabled
and no user is passed, Drill tries to use an "anonymous" user. I've seen
that in case of MapR-FS, that it cannot use such a user to proceed with
accessing the filesystem.
Can you try if (1) username is passed when launching your client such as
Sqlline, (2), the user (with same UID) exists on all nodes on the cluster.
Post by Divya Gehlot
Hi ,
At times I get below error when running the View query which fetches the
whole data set
Failed to create schema tree.
The Strange behaviour is if I wait for sometime for say 2-3 minutes and
run
Post by Divya Gehlot
the query again it works perfectly .
Appreciate if anybody can help me to get me the root cause for it !
Thanks,
Divya
Kunal Khatua
2018-10-11 23:43:02 UTC
Permalink
There is another possible explanation of why this is happening. 
The `createFullRootSchema` seems to be called as part of the initial schema creation required for a query, and that means initialization of the backend storage plugins for the new nodes.

You might need to disable some of the storage plugins to identify which one is taking time and causing `org.apache.drill.exec.store.SchemaFactory.registerSchemas(...)` [1] to take a long time.

I think, that should help resolve the issue.

[1] https://github.com/apache/drill/blob/master/exec/java-exec/src/main/java/org/apache/drill/exec/store/SchemaTreeProvider.java#L133. 
On 10/11/2018 12:10:00 AM, Divya Gehlot <***@gmail.com> wrote:
Hi Abhishek,
Thanks for the response with clarifications !
When I jumped into the logs ..
May be I should provide some additional information regarding the Drill
cluster before sharing more information what I found in the logs
I have 6 nodes cluster where the 3 nodes are old and 3 are newly added
nodes
Old Nodes - dn1,dn2 and dn3
New Nodes - dn4,dn5,dn6
Whenever the query foreman is newly added nodes the error occurs , I don't
find this error in any of the old nodes drillbit.log
Even the user is Drill admin user I do see the above mentioned error .
I verified that Drill admin has same UID and GID across all the nodes .
I did check the uid is being passed while running the query
INFO o.a.d.exec.store.SchemaTreeProvider - User Error Occurred: Failed to
create schema tree. (Error getting user info for current user, 992521597)
Above error I could see in the drillbit.log files of dn4, dn5 and dn6 only
I dont see any error like this on dn1,dn2 or dn3 nodes.
Any thoughts why is it happening despite the same uid across all the nodes .



Thanks,
Divya
Hey Divha,
That's usually seen when the underlying FS cannot recognize the session
user when creating the schema tree. For example if impersonation is enabled
and no user is passed, Drill tries to use an "anonymous" user. I've seen
that in case of MapR-FS, that it cannot use such a user to proceed with
accessing the filesystem.
Can you try if (1) username is passed when launching your client such as
Sqlline, (2), the user (with same UID) exists on all nodes on the cluster.
On Wed, Oct 10, 2018 at 8:07 PM Divya Gehlot
Post by Divya Gehlot
Hi ,
At times I get below error when running the View query which fetches the
whole data set
Failed to create schema tree.
The Strange behaviour is if I wait for sometime for say 2-3 minutes and
run
Post by Divya Gehlot
the query again it works perfectly .
Appreciate if anybody can help me to get me the root cause for it !
Thanks,
Divya
Loading...