java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer

Problem

When a Spark job is run it fails with an error and the following stack trace is logged in job.log

ERROR [2017-01-20 00:12:12.309] [JobScheduler thread-1] (JobScheduler.java:829) - Job 10710 failed with exception.
java.lang.NoClassDefFoundError: com/sun/jersey/spi/container/servlet/ServletContainer
 at org.apache.spark.status.api.v1.ApiRootResource$.getServletHandler(ApiRootResource.scala:187)
 at org.apache.spark.ui.SparkUI.initialize(SparkUI.scala:70)
 at org.apache.spark.ui.SparkUI.<init>(SparkUI.scala:76)
 at org.apache.spark.ui.SparkUI$.create(SparkUI.scala:195)
 at org.apache.spark.ui.SparkUI$.createLiveUI(SparkUI.scala:146)
 at org.apache.spark.SparkContext.<init>(SparkContext.scala:473)
 at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2281)
 at org.apache.spark.SparkContext.getOrCreate(SparkContext.scala)
 at datameer.plugin.spark.SparkContextWrapper.initSparkContext(SparkContextWrapper.java:122)
 at datameer.plugin.spark.SparkContextPool.createSparkContext(SparkContextPool.java:322)
 at datameer.plugin.spark.SparkContextPool.access$100(SparkContextPool.java:43)
 at datameer.plugin.spark.SparkContextPool$YarnClientModePool.getOrCreateSparkContext(SparkContextPool.java:92)
 at datameer.plugin.spark.SparkClusterSession.<init>(SparkClusterSession.java:34)
 at datameer.plugin.spark.SparkClientExecutionFramework.createSession(SparkClientExecutionFramework.java:42)
 at datameer.plugin.spark.SparkClientExecutionFramework.createClusterSession(SparkClientExecutionFramework.java:31)
 at datameer.plugin.spark.sx.SparkSxClusterJobFlowResolver.executableJobs(SparkSxClusterJobFlowResolver.java:53)
 at datameer.dap.common.graphv2.ClusterSession.executableJobs(ClusterSession.java:245)
 at datameer.dap.common.graphv2.ClusterSession.runAllClusterJobs(ClusterSession.java:337)
 at datameer.dap.common.graphv2.MrPlanRunnerV2.run(MrPlanRunnerV2.java:129)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
 at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:109)
 at datameer.dap.common.security.DatameerSecurityService.runAsUser(DatameerSecurityService.java:186)
 at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:34)
 at datameer.dap.common.security.RunAsThread$1.run(RunAsThread.java:30)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
 at datameer.dap.common.filesystem.Impersonator.doAs(Impersonator.java:31)
 at datameer.dap.common.security.RunAsThread.run(RunAsThread.java:30)
 INFO [2017-01-20 00:12:12.332] [JobScheduler thread-1] (JobScheduler.java:904) - Computing after job completion operations for execution 10710 (type=NORMAL)
 INFO [2017-01-20 00:12:12.332] [JobScheduler thread-1] (JobScheduler.java:908) - Finished computing after job completion operations for execution 10710 (type=NORMAL) [0 sec]
 WARN [2017-01-20 00:12:12.341] [JobScheduler thread-1] (JobScheduler.java:759) - Job DapJobExecution{id=10710, type=NORMAL, status=ERROR} completed with status ERROR.

Cause

The cause of this error is that the Spark UI can't be enabled for jobs that are going to run via SparkClient. The below entry from job trace indicates that spark.ui.enabled property is being added to cluster properties.

<property>
  <name>spark.ui.enabled</name>
    <value>true</value>
  <source>programatically</source>
</property>

Solution

Delete the following files if they exist under in the Custom Properties either at the top level Hadoop Cluster configuration or at an individual level job configuration.

spark.ui.enabled=true