Post upgrade to 2.3.7, delayed_job for API fails to start - uninitialized constant StatusMonitorJob
I realized only today that delayed_job for API never starts up after I updated my local OBS instance to 2.3.7 (more than a month back)
There are 2 issues here:
1) Starting delayed_job.api run in foreground fails with the following error:
$ sudo RAILS_ENV=production /sbin/startproc -w -u wwwrun -g www -- /srv/www/obs/api/script/delayed_job.api run
** Erubis 2.6.6
Job failed to load: uninitialized constant StatusMonitorJob. Try to manually require the required file.
startproc: exit status of parent of /srv/www/obs/api/script/delayed_job.api: 1
2) "/etc/init.d/obsapidelayed status" incorrectly reports delayed_job.api to be "running".
Before I continue, I would like to specify that these two problems do not exist in a fresh install of OBS 2.3.x and are observed only in case of an update.
After grep'ing around, I figured out that delayed_job was looking at the database for pending jobs. StatusMonitorJob was removed back in February of 2012:
[api] don't rely on delayed_job for updating status
But during update of OBS, the existing jobs in delayed_jobs table for StatusMonitorJob aren't removed. To fix this, I had to fire up the console and delete all those jobs:
$ sudo /srv/www/obs/api/script/console production
Loading production environment (Rails 2.3.14)
** Erubis 2.6.6
>> Delayed::Job.find_each do |job|
?> job.delete if job.handler =~ /StatusMonitorJob/
=> Delayed::Job(id: integer, priority: integer, attempts: integer, handler: text, last_error: text, run_at: datetime, locked_at: datetime, failed_at: datetime, locked_by: string)
After this, "/srv/www/obs/api/script/delayed_job.api run" starts up just fine!
Coming to 2) issue I mentioned above: The problem here is obsapidelayed init script itself:
$ grep -A5 'status)' /etc/init.d/obsapidelayed
echo -n "Checking for service delayed OBS api jobs "
[ $? == $NUM ]
"rc_status" processes $? - exit code of previous command. Here [ $? == $NUM ] will always be true if checkproc can't find either the delayed_job.0 process or it's PID file - because in this case checkproc will return 3 and NUM is set to 3 :)
I zapped that condition out of that script and now "/etc/init.d/obsapidelayed status" responds with "unused" if there's no delayed_job.api processes.
Hope this helps anyone else who runs / manages a private instance of OBS updated from any previous version to 2.3.x