Debugging Job Scheduler and Trigger Problems

Problem Description

If you experience the problem that scheduled jobs are not running anymore, but you can run them manually.

Step-by-step guide

1) Add within your <datameer-install-path>/conf/log4j-production.properties the configuration property

log4j.category.datameer.dap.conductor.trigger.StartJobTrigger=TRACE

It will be necessary to restart the Datameer service to make this change active. 

2) Watch or investigate the application log file at the times when a job should be scheduled. To do so catch the TRACE messages via 

grep -ir 'system.*trace' conductor.log.<id>
grep -ir 'cron4j::scheduler' conductor.log.<id>

The output form <datameer-install-path>/logs/conductor.log should than look like

[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>8a3257]] (StartJobTrigger.java:57) - Job '2' is going to be scheduled.
[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>c95b5a]] (StartJobTrigger.java:57) - Job '1' is going to be scheduled.
[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>c95b5a]] (StartJobTrigger.java:66) - Job '1' is scheduled succesfully.
[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>c95b5a]] (StartJobTrigger.java:72) - Job '1' triggering finished.
[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>8a3257]] (StartJobTrigger.java:66) - Job '2' is scheduled succesfully.
[system] TRACE [<date> <time>] [cron4j::scheduler[<id>]::executor[<id>8a3257]] (StartJobTrigger.java:72) - Job '2' triggering finished.

3) If the job became triggered catch the TRACE messages via 

grep -ir 'triggering.*finished' conductor.log.<id>

and check if it was executed later correctly! To do so catch the job execution via 

grep -ir 'DataSourceConfigurationImpl{id=<jobID>}' conductor.log.<id>

The output of <datameer-install-path>/logs/conductor.log should than look like 

[system]  INFO [<date> <time>] [JobScheduler thread-1] (JobScheduler.java:398) - [Job <id>] Preparing job in job scheduler thread for DataSourceConfigurationImpl{id=<jobID>}...
[system]  INFO [<date> <time>] [JobScheduler thread-1] (JobScheduler.java:401) - [Job <id>] Preparing job in job scheduler thread for DataSourceConfigurationImpl{id=<jobID>}... done (0 sec)
[system]  INFO [<date> <time>] [JobScheduler worker1-thread-134] (JobSchedulerJob.java:95) - [Job <id>] Preparing job for DataSourceConfigurationImpl{id=<jobID>}...
[system]  INFO [<date> <time>] [JobScheduler worker1-thread-134] (JobSchedulerJob.java:100) - [Job <id>] Preparing job for DataSourceConfigurationImpl{id=<jobID>}... done (0 sec)

4) Now, as you have the configuration and job IDs, you can proceed further with your investigation and check if the job were processed successful.

Useful Notes:

• Below property allows to log StartJobTrigger events for Scheduled artefacts only. DataDriven artefacts triggering won't be logged with a separate TRACE message.
  

  log4j.category.datameer.dap.conductor.trigger.StartJobTrigger=TRACE

• DataDriven Workbooks might not be triggered properly in case they have filter or sort introduced directly at the source sheet. To avoid such cases, we would recommend do not filter/sort data directly at the source sheets. This behaviour is expected to be improved in Datameer 6.1.25.