yarn nodemanager doesn't start - Caused by: EPERM: Operation not permitted

-
I was dealing with this YarnRuntimeException recently and thought it might be useful for folks who encounter this situation.  

 Both Namenode and Datanode is up and running and it was time to start the yarn daemons. ResourceManager came up just fine while the nodemanager stalled with the following exception. 

 "
2014-04-19 03:46:37,595 INFO org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService failed in state INITED; cause: org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationService
org.apache.hadoop.yarn.exceptions.YarnRuntimeException: Failed to initialize LocalizationService
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.localizer.ResourceLocalizationService.serviceInit(ResourceLocalizationService.java:234)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.ContainerManagerImpl.serviceInit(ContainerManagerImpl.java:220)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.service.CompositeService.serviceInit(CompositeService.java:107)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.serviceInit(NodeManager.java:186)
        at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.initAndStartNodeManager(NodeManager.java:357)
        at org.apache.hadoop.yarn.server.nodemanager.NodeManager.main(NodeManager.java:404)
Caused by: EPERM: Operation not permitted
"

 Looking at the nodemanager logs and searching the internet for a while the initial impression was, yarn user doesn't have sufficient privileges to mkdir using "yarn.nodemanager.local-dirs". But, in the next couple of minutes my assumptions were wrong. 

 After digging into the source,  it turns out that, the log folder configured in the yarn-site.xml (yarn.nodemanager.log-dirs)  doesn't belong to yarn user group. 
Doing a chown -R yarn:yarn (assuming yarn being the name of the yarn user) on the log dir helped in bringing up the nodemanager. 

Comments

Anonymous said…
Thanks that helped
July 18, 2014 at 1:30:00 PM PDT
Anonymous said…
Helped me too! Thanks!
August 21, 2014 at 1:31:00 PM PDT
Tejuteju said…
Really Good blog post. provided a helpful information. I hope that you will post more updates like this Big Data Hadoop Online Course
September 18, 2018 at 2:49:00 AM PDT
Unknown said…
Nice and good article. It is very useful for me to learn and understand easily. Thanks for sharing your valuable information and time. Please keep updating Big data online training
October 15, 2019 at 2:58:00 AM PDT
Post a Comment

Popular posts from this blog

How to find the source branch of a branch in git

-
Build kept failing for one of the projects that I was trying to compile and I was told that the source branch ( meaning the original branch from where my working branch got created from) just compiles fine. I had this doubt in my mind, is the branch that I'm working on is really branched from of from the same source branch that I was told? How do I figure that out..  I know, gitk right??? But, I want to see if this is possible with git log. And it turns out , its possible with git log   git log --graph --color --decorate --oneline --all And finally I figured I was given a wrong branch to work with - *smile*  
Read more

gpg: cancelled by user

-
While signing a release tar.gz with gpg key, I got into this weird issue. After digging the internet I finally got the solution. Here is the error that I got and below is the solution to this issue.  [user@host tar]$ gpg --armor --output something.tar.gz.asc --detach-sig something- .tar.gz You need a passphrase to unlock the secret key for user: "user (somthing) " 4096-bit RSA key, ID 654321, created 2012-06-22 can't connect to `/home/user/.gnupg/S.gpg-agent': No such file or directory gpg-agent[7582]: command get_passphrase failed: Operation cancelled gpg: cancelled by user gpg: no default secret key: General error gpg: signing failed: General error Initially I thought I didn't start the gpg-agent [user@host tar]$      eval $(gpg-agent --daemon) I started the gpg-agent with the above eval [user@host tar]$ gpg --armor --output something.tar.gz.asc --detach-sig something- .tar.gz You need a passphrase to unlock the secret key for user: "
Read more