Spark执行样例报警告：WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any-白红宇

Spark执行样例报警告：WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any

阅读量：6078 次

发布时间：2019-06-20

本文共 5197 字，大约阅读时间需要 17 分钟。

Spark执行样例报警告：WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any 博客分类： spark 异常

搭建Spark环境后，调测Spark样例时，出现下面的错误：

WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

[hadoop@gpmaster bin]$ ./run-example org.apache.spark.examples.SparkPi

15/10/01 08:59:33 INFO spark.SparkContext: Running Spark version 1.5.0

.......................

15/10/01 08:59:35 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://sparkDriver@192.168.1.128:17514]

15/10/01 08:59:35 INFO util.Utils: Successfully started service ‘sparkDriver‘ on port 17514.

.......................

15/10/01 08:59:36 INFO ui.SparkUI: Started SparkUI at http://192.168.1.128:4040

15/10/01 08:59:37 INFO spark.SparkContext: Added JAR file:/home/hadoop/spark/lib/spark-examples-1.5.0-hadoop2.6.0.jar at http://192.168.1.128:36471/jars/spark-examples-1.5.0-hadoop2.6.0.jar with timestamp 1443661177865

15/10/01 08:59:37 WARN metrics.MetricsSystem: Using default name DAGScheduler for source because spark.app.id is not set.

15/10/01 08:59:38 INFO client.AppClient$ClientEndpoint: Connecting to master spark://192.168.1.128:7077...

15/10/01 08:59:38 INFO cluster.SparkDeploySchedulerBackend: Connected to Spark cluster with app ID app-20151001085938-0000

.................................

15/10/01 08:59:40 INFO scheduler.TaskSchedulerImpl: Adding task set 0.0 with 2 tasks

15/10/01 08:59:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:00:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:00:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:00:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:00:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:01:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:01:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:01:40 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:01:55 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:02:10 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

15/10/01 09:02:25 WARN scheduler.TaskSchedulerImpl: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources

从警告信息大致可以知道：

初始化job时没有获取到任何资源；提示检查集群，确保workers可以被注册并有足够的内存资源。

可能的原因有几点，可以逐个排查：

1. 主机主机名和ip是否配置正确

先查看/etc/hosts文件配置是否正确

同时可以通过spark-shell查看SparkContext获取的上下文信息, 如下操作：

[hadoop@gpmaster bin]$ ./spark-shell

........

scala> sc.getConf.getAll.foreach(println)

(spark.fileserver.uri,http://192.168.1.128:34634)

(spark.app.name,Spark shell)

(spark.driver.port,25392)