2 years ago

#72774

test-img

chenchen

hadoop3.3.1 start-yarn.sh failed

  • When I try to use Hadoop3.3.1, HDFS has been successfully run. When performing YARN services, I encountered the following issues, and the configuration file information is an error message as follows:

yarn-site.xml

<configuration>

<!-- Site specific YARN configuration properties -->
        <!-- 开启RM高可用 -->
        <!--property>
                <name>yarn.resourcemanager.ha.enabled</name>
                <value>true</value>
        </property-->
        
        <!-- 指定RM的cluster id -->
        <property>
                <name>yarn.resourcemanager.cluster-id</name>
                <value>tobias-yarn-test</value>
        </property>
        
        <!-- 指定RM的名字 -->
        <!--
        <property>
                <name>yarn.resourcemanager.ha.rm-ids</name>
                <value>rm1,rm2</value>
        </property>
        
         分别指定RM的地
        <property>
                <name>yarn.resourcemanager.hostname.rm1</name>
                <value>192.168.7.166</value>
        </property>
        
        <property>
                <name>yarn.resourcemanager.hostname.rm2</name>
                <value>192.168.7.116</value>
        </property>
        
         指定zk集群地址-->
        <!--property>
                <name>yarn.resourcemanager.zk-address</name>
                <value>192.168.8.118:2181</value>
        </property-->
        <property>
            <description>Comma separated list of Host:Port pairs. Each corresponds to a ZooKeeper server
   (e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002") to be used by the RM for storing RM state.
   This must be supplied when using org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore
   as the value for yarn.resourcemanager.store.class</description>
            <name>hadoop.zk.address</name>
            <value>192.168.8.118:2181</value>
         </property>
      
        <!-- 要运行MapReduce程序必须配置的附属服务-->
        <property>
                <name>yarn.nodemanager.aux-services</name>
                <value>mapreduce_shuffle</value>
        </property>
        
        <!-- 开启YARN集群的日志聚合功能 -->
        <property>
                <name>yarn.log-aggregation-enable</name>
                <value>true</value>
        </property>
        
        <!-- YARN集群的聚合日志最长保留时长 -->
        <property>
                <name>yarn.log-aggregation.retain-seconds</name>
                <value>86400</value>
        </property>
        
        <!-- 启用自动恢复 -->
        <property>
                <name>yarn.resourcemanager.recovery.enabled</name>
                <value>true</value>
        </property>
        
        <!-- 制定resourcemanager的状态信息存储在zookeeper集群上-->
        <property>
                <name>yarn.resourcemanager.store.class</name>
                <value>org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore</value>
        </property>

<!-- 关闭yarn内存检查 -->
 
        <property> 
                <name>yarn.nodemanager.pmem-check-enabled</name>
                <value>false</value>
        </property>
 
        <property>
                <name>yarn.nodemanager.vmem-check-enabled</name>
                <value>false</value>
        </property>
        <property>
                <name>yarn.scheduler.maximun-allocation-vcores</name>
                <value>4</value>
        </property>
                
        <property>
                <name>yarn.scheduler.minimum-allocation-mb</name>
                <value>1024</value>
        </property>     

                <property>
                <name>yarn.nodemanager.resource.memory-mb</name>
                <value>6144</value>
        </property>     
                <property>
                <name>yarn.scheduler.maximum-allocation-mb</name>
                <value>8192</value>
        </property>
        <property>
                <name>yarn.nodemanager.env-whitelist</name>
                <value>JAVA_HOME,HADOOP_COMMON_HOME,HADOOP_HDFS_HOME,HADOOP_CONF_DIR,CLASSPATH_PREPEND_DISTCACHE,HADOOP_YARN_HOME,HADOOP_HOME,PATH,LANG,TZ,HADOOP_MAPRED_HOME</value>
        </property>
</configuration>

错误内容

tail -100 ../../logs/hadoop-root-resourcemanager-CentOS7.log 
defaultAppPriorityPerQueue = 0
priority = 0
maxLifetime = -1 seconds
defaultLifetime = -1 seconds
2022-01-20 09:38:58,105 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized queue: root.default
2022-01-20 09:38:58,105 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized queue: root
2022-01-20 09:38:58,107 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacitySchedulerQueueManager: Initialized root queue root: numChildQueue= 1, capacity=1.0, absoluteCapacity=1.0, usedResources=<memory:0, vCores:0>usedCapacity=0.0, numApps=0, numContainers=0
2022-01-20 09:38:58,107 INFO org.apache.hadoop.yarn.server.resourcemanager.placement.UserGroupMappingPlacementRule: Initialized queue mappings, override: false
2022-01-20 09:38:58,107 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.WorkflowPriorityMappingsManager: Initialized workflow priority mappings, override: false
2022-01-20 09:38:58,107 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.placement.MultiNodeSortingManager: MultiNode scheduling is 'false', and configured policies are 
2022-01-20 09:38:58,107 INFO org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.CapacityScheduler: Initialized CapacityScheduler with calculator=class org.apache.hadoop.yarn.util.resource.DefaultResourceCalculator, minimumAllocation=<<memory:1024, vCores:1>>, maximumAllocation=<<memory:8192, vCores:4>>, asynchronousScheduling=false, asyncScheduleInterval=5ms,multiNodePlacementEnabled=false
2022-01-20 09:38:58,109 INFO org.apache.hadoop.conf.Configuration: dynamic-resources.xml not found
2022-01-20 09:38:58,113 INFO org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain: Initializing AMS Processing chain. Root Processor=[org.apache.hadoop.yarn.server.resourcemanager.DefaultAMSProcessor].
2022-01-20 09:38:58,113 INFO org.apache.hadoop.yarn.server.resourcemanager.ApplicationMasterService: disabled placement handler will be used, all scheduling requests will be rejected.
2022-01-20 09:38:58,113 INFO org.apache.hadoop.yarn.server.resourcemanager.AMSProcessingChain: Adding [org.apache.hadoop.yarn.server.resourcemanager.scheduler.constraint.processor.DisabledPlacementProcessor] tp top of AMS Processing chain. 
2022-01-20 09:38:58,118 INFO org.apache.hadoop.service.AbstractService: Service ResourceManager failed in state STARTED
org.apache.hadoop.service.ServiceStateException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /rmstore/ZKRMStateRoot/RMAppRoot/HIERARCHIES
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:935)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1420)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1461)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1457)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1457)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1508)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1699)
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /rmstore/ZKRMStateRoot/RMAppRoot/HIERARCHIES
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1637)
        at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1180)
        at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1156)
        at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1153)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:607)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:597)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:362)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:310)
        at org.apache.hadoop.util.curator.ZKCuratorManager.create(ZKCuratorManager.java:286)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.create(ZKRMStateStore.java:1436)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:409)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:825)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        ... 12 more
2022-01-20 09:38:58,130 ERROR org.apache.hadoop.security.token.delegation.AbstractDelegationTokenSecretManager: ExpiredTokenRemover received java.lang.InterruptedException: sleep interrupted
2022-01-20 09:38:58,137 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.w.WebAppContext@3337d04c{cluster,/,null,STOPPED}{jar:file:/fsmeeting/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-common-3.3.1.jar!/webapps/cluster}
2022-01-20 09:38:58,159 INFO org.eclipse.jetty.server.AbstractConnector: Stopped ServerConnector@ec0c838{HTTP/1.1, (http/1.1)}{0.0.0.0:8088}
2022-01-20 09:38:58,159 INFO org.eclipse.jetty.server.session: node0 Stopped scavenging
2022-01-20 09:38:58,160 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@7c22d4f{static,/static,jar:file:/fsmeeting/hadoop-3.3.1/share/hadoop/yarn/hadoop-yarn-common-3.3.1.jar!/webapps/static,STOPPED}
2022-01-20 09:38:58,163 INFO org.eclipse.jetty.server.handler.ContextHandler: Stopped o.e.j.s.ServletContextHandler@2484f433{logs,/logs,file:///fsmeeting/hadoop-3.3.1/logs/,STOPPED}
2022-01-20 09:38:58,187 INFO org.apache.hadoop.ipc.Server: Stopping server on 8033
2022-01-20 09:38:58,205 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioning to standby state
2022-01-20 09:38:58,206 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Transitioned to standby state
2022-01-20 09:38:58,208 ERROR org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: Error starting ResourceManager
org.apache.hadoop.service.ServiceStateException: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /rmstore/ZKRMStateRoot/RMAppRoot/HIERARCHIES
        at org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:203)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceStart(ResourceManager.java:935)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.startActiveServices(ResourceManager.java:1420)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1461)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$1.run(ResourceManager.java:1457)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:422)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1878)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.transitionToActive(ResourceManager.java:1457)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceStart(ResourceManager.java:1508)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.main(ResourceManager.java:1699)
Caused by: org.apache.zookeeper.KeeperException$UnimplementedException: KeeperErrorCode = Unimplemented for /rmstore/ZKRMStateRoot/RMAppRoot/HIERARCHIES
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:106)
        at org.apache.zookeeper.KeeperException.create(KeeperException.java:54)
        at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:1637)
        at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1180)
        at org.apache.curator.framework.imps.CreateBuilderImpl$17.call(CreateBuilderImpl.java:1156)
        at org.apache.curator.connection.StandardConnectionHandlingPolicy.callWithRetry(StandardConnectionHandlingPolicy.java:64)
        at org.apache.curator.RetryLoop.callWithRetry(RetryLoop.java:100)
        at org.apache.curator.framework.imps.CreateBuilderImpl.pathInForeground(CreateBuilderImpl.java:1153)
        at org.apache.curator.framework.imps.CreateBuilderImpl.protectedPathInForeground(CreateBuilderImpl.java:607)
        at org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:597)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:362)
        at org.apache.curator.framework.imps.CreateBuilderImpl$3.forPath(CreateBuilderImpl.java:310)
        at org.apache.hadoop.util.curator.ZKCuratorManager.create(ZKCuratorManager.java:286)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.create(ZKRMStateStore.java:1436)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.ZKRMStateStore.startInternal(ZKRMStateStore.java:409)
        at org.apache.hadoop.yarn.server.resourcemanager.recovery.RMStateStore.serviceStart(RMStateStore.java:825)
        at org.apache.hadoop.service.AbstractService.start(AbstractService.java:194)
        ... 12 more
2022-01-20 09:38:58,207 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server Responder
2022-01-20 09:38:58,206 INFO org.apache.hadoop.ipc.Server: Stopping IPC Server listener on 8033
2022-01-20 09:38:58,235 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceManager: SHUTDOWN_MSG:
  • This problem is based on Hadoop3.3.1 single-range pseudo-distributed existing, my ZooKeeper version is 3.4.9, is there a problem with such Yarn-Site.xml configuration, please help me Thank you

hadoop

hadoop-yarn

apache-zookeeper

0 Answers

Your Answer

Accepted video resources

Posts

Questions

Blogs

Jobs

Our Use of Cookies

We use cookies, including third-party cookies, to improve your experience and to show you personalised content and advertising.

To find out more, read our Privacy Policy and Cookie Policy.
My Options