Problem
When a Spark job is run it fails with an error and the following stack trace is logged in job.log
ERROR [2017-01-20 00:12:12.309] [JobScheduler thread-1] (JobScheduler.java:829) - Job 10710 failed with exception.
java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer
at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:187)
at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:70)
at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:76)
at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:473)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
at datameer.plugin.spark.SparkContextWrapper.initSparkContext(SparkContextWrapper.java:122)
at datameer.plugin.spark.SparkContextPool.createSparkContext(SparkContextPool.java:322)
at datameer.plugin.spark.SparkContextPool.access$100(SparkContextPool.java:43)
at datameer.plugin.spark.SparkContextPool$YarnClientModePool.getOrCreateSparkContext(SparkContextPool.java:92)
at datameer.plugin.spark.SparkClusterSession.<init>(SparkClusterSession.java:34)
at datameer.plugin.spark.SparkClientExecutionFramework.createSession(SparkClientExecutionFramework.java:42)
at datameer.plugin.spark.SparkClientExecutionFramework.createClusterSession(SparkClientExecutionFramework.java:31)
at datameer.plugin.spark.sx.SparkSxClusterJobFlowResolver.executableJobs(SparkSxClusterJobFlowResolver.java:53)
at datameer.dap.common.graphv2.ClusterSession.executableJobs(ClusterSession.java:245)
at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:337)
at datameer.dap.common.graphv2.MrPlanRunnerV2.run(MrPlanRunnerV2.java:129)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:109)
at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:186)
at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34)
at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at datameer.dap.common.filesystem.Impersonator.doAs(Impersonator.java:31)
at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30)
INFO [2017-01-20 00:12:12.332] [JobScheduler thread-1] (JobScheduler.java:904) - Computing after job completion operations for execution 10710 (type=NORMAL)
INFO [2017-01-20 00:12:12.332] [JobScheduler thread-1] (JobScheduler.java:908) - Finished computing after job completion operations for execution 10710 (type=NORMAL) [0 sec]
WARN [2017-01-20 00:12:12.341] [JobScheduler thread-1] (JobScheduler.java:759) - Job DapJobExecution{id=10710, type=NORMAL, status=ERROR} completed with status ERROR.
Cause
The cause of this error is that the Spark UI can't be enabled for jobs that are going to run via SparkClient. The below entry from job trace indicates that spark.ui.enabled
property is being added to cluster properties.
<property>
<name>spark.ui.enabled</name>
<value>true</value>
<source>programatically</source>
</property>
Solution
Delete the following files if they exist under in the Custom Properties either at the top level Hadoop Cluster configuration or at an individual level job configuration.
spark.ui.enabled=true
Comments
0 comments
Please sign in to leave a comment.