AccessControlException: Queue root.default already has 10000

Problem

Datameer can't schedule any new jobs. In the logs the following exception is shown:

Error Message

Failed to submit application_<id> to YARN : org.apache.hadoop.security.AccessControlException: Queue root.default already has 10000 applications, cannot accept submission of application: application_<id>

Cause

This error is from the Capacity Scheduler. Based on the following document Configuring YARN Capacity Scheduler Ambari, it appears the yarn.scheduler.capacity.maximum-applications limit are hit, which is set to 10,000 by default.

Solution 

1. Check the Hadoop cluster queue and clear out any unnecessary jobs.
2. Increase the limit in the YARN configuration to allow for more concurrent applications to be submitted.