Post upgrade to 2.3.7, delayed_job for API fails to start - uninitialized constant StatusMonitorJob

classic Classic list List threaded Threaded
1 message Options
Reply | Threaded
Open this post in threaded view
|

Post upgrade to 2.3.7, delayed_job for API fails to start - uninitialized constant StatusMonitorJob

Srinidhi B
Hi,

I realized only today that delayed_job for API never starts up after I updated my local OBS instance to 2.3.7 (more than a month back)

There are 2 issues here:

1) Starting delayed_job.api run in foreground fails with the following error:
----
$ sudo RAILS_ENV=production /sbin/startproc -w -u wwwrun -g www -- /srv/www/obs/api/script/delayed_job.api run
mirror's password:
** Erubis 2.6.6
Job failed to load: uninitialized constant StatusMonitorJob. Try to manually require the required file.
startproc:  exit status of parent of /srv/www/obs/api/script/delayed_job.api: 1
----
2) "/etc/init.d/obsapidelayed status" incorrectly reports delayed_job.api to be "running".

Before I continue, I would like to specify that these two problems do not exist in a fresh install of OBS 2.3.x and are observed only in case of an update.

After grep'ing around, I figured out that delayed_job was looking at the database for pending jobs. StatusMonitorJob was removed back in February of 2012:

   [api] don't rely on delayed_job for updating status

But during update of OBS, the existing jobs in delayed_jobs table for StatusMonitorJob aren't removed. To fix this, I had to fire up the console and delete all those jobs:

----
$ sudo /srv/www/obs/api/script/console production
Loading production environment (Rails 2.3.14)
** Erubis 2.6.6
>> Delayed::Job.find_each do |job|
?> job.delete if job.handler =~ /StatusMonitorJob/
>> end
=> Delayed::Job(id: integer, priority: integer, attempts: integer, handler: text, last_error: text, run_at: datetime, locked_at: datetime, failed_at: datetime, locked_by: string)
----

After this, "/srv/www/obs/api/script/delayed_job.api run" starts up just fine!

Coming to 2) issue I mentioned above: The problem here is obsapidelayed init script itself:

----
$ grep -A5 'status)' /etc/init.d/obsapidelayed
    status)
        echo -n "Checking for service delayed OBS api jobs "
        checkproc delayed_job.0
        [ $? == $NUM ]
        rc_status -v
        ;;
----

"rc_status" processes $? - exit code of previous command. Here [ $? == $NUM ] will always be true if checkproc can't find either the delayed_job.0 process or it's PID file - because in this case checkproc will return 3 and  NUM is set to 3 :)

I zapped that condition out of that script and now "/etc/init.d/obsapidelayed status" responds with "unused" if there's no delayed_job.api processes.

Hope this helps anyone else who runs / manages a private instance of OBS updated from any previous version to 2.3.x

Srinidhi.

--
To unsubscribe, e-mail: [hidden email]
To contact the owner, e-mail: [hidden email]