Jobs not appearing correctly in Resource Manager

Following a restart after our storage had to be restarted, we are seeing issues with jobs not appearing correctly in the resource manager. There are two symptoms:

  1. A few completed jobs are remaining in the list of current jobs, on their relevant lane. When the job is opened, they show as complete.

Job226 Completed 2021-10-08 124700

  1. New jobs are not appearing at all in the lane on which they are running (I have managed to catch then briefly flashing up when starting). I can see the jobs running from the cli:

In [7]: [(j[‘project_uid’], j[‘uid’], j[‘status’], j[‘job_type’]) for j in cli.get_jobs_by_status({ ‘$in’: [‘queued’, ‘launched’, ‘started’, ‘waiting’, ‘running’]})]
[(‘P14’, ‘J323’, ‘queued’, ‘ctf_refine_local’),
(‘P3’, ‘J988’, ‘running’, ‘nonuniform_refine_new’),
(‘P9’, ‘J251’, ‘running’, ‘new_local_refine’)]

There is nothing obvious when I check the logfiles. Any advice gratefully received.


Andy Purkiss

Another cryosparcm restart may have resolved the issue, but I’ll keep an eye on it.