Data Driven Jobs Not Started As Expected with Secure Impersonation

Symptoms

Datameer jobs that are configured to run when new data is available, also known as "Data Driven", do not start as expected -- even though new data is available. Within the Datameer conductor.log file, no errors, warnings nor exceptions are logged related to this symptom.

This symptom is reported in Datameer 6.4.x environments which have Impersonation activated.

The symptom may be triggered when an Import Job with a large number of historical imports (10,000 or more for example) has completed successfully in the environment. 

Cause

The cause of this issue is a software defect known internally as DAP-36458. 

Resolution

This issue is resolved in Datameer 6.4.7 and higher maintenance releases.

Potential Work-around

To reduce the likelihood this issue will occur, the Datameer Administrator may add the following property to the <INSTALLDIR>/conf/<MODE>.properties file (i.e. production.properties): 

• event.bus.async.threads=32

After editing this setting Datameer needs to be restarted to activate it. 

Adding this property activates Datameer to have more available thread capacity to prevent all available threads from being actively processing tasks other than the data driven scheduling. This is not a guarantee to prevent the issue but should reduce the frequency of the issue and should reduce the duration after Datameer is restarted before the issue recurs. 

Please contact Datameer Technical Support for further information.