Jobs queued but fail to run

Troubleshooting
1

When one of our users attempts to submit a job it is queued and then immediately fails. The number of jobs in the queue (as seen at the top of the window) is incremented but is not displayed with other jobs that are queued.

The meteor.stdout file shows this error when these jobs are submitted:
Job Submitted!
{ job_id: ‘CG9uR2dY9cWqbioGE’ }
{ sort: { createdAt: 1 }, skip: 0, limit: 100 }
{ value: ‘’ }
{ value: ‘’ }
{ value: ‘’ }
{ value: ‘2’ }
{ value: null }
Exception in setInterval callback: TypeError: Cannot read property ‘split’ of null
at job_queue_daemon_worker (server/modules/job-queue-daemon.js:113:58)
at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
at packages/meteor.js:445:45
at runWithEnvironment (packages/meteor.js:1176:24)

When restarting cryoSPARC we see a similar error:
----- JOB QUEUE DAEMON ------
Hardware Slots:
{ GPU: [ 0, 1, 2, 3 ],
RAM: [ 0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15 ] }
{ value: ‘’ }
{ value: ‘’ }
{ value: null }
Exception in setInterval callback: TypeError: Cannot read property ‘split’ of null
at job_queue_daemon_worker (server/modules/job-queue-daemon.js:113:58)
at [object Object]._.extend.withValue (packages/meteor.js:1122:17)
at packages/meteor.js:445:45
at runWithEnvironment (packages/meteor.js:1176:24)

Is something corrupted in the database? If so, how do we check and fix the database?

Thanks,
Steve

2

I was able to get this cleaned up after trying something from another post (All new jobs get queued, don't run):

eval $(cryosparc env)
source config.sh
mongo localhost:$CRYOSPARC_MONGO_PORT

Now in the database shell:

use meteor
db.jobs.update({status:"queued"}, {$set:{status:"killed"}})
exit

I had tried that earlier but failed to repeat the “db.jobs.update” command until all the queued jobs were marked as killed. Once that was done, we were able to once again queue and run jobs.

Steve

3