Job leftover from deleting project

Troubleshooting
1

Hi all,

Recently a project (P242) was deleted from web GUI on our CS-master v4.0.
After the deletion command_core log show an error that “P242 J28 does not exist”:

2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    | Traceback (most recent call last):
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |   File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/commandcommon.py", line 200, in wrapper
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |     res = func(*args, **kwargs)
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |   File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 6949, in get_job_status
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |     return get_job(project_uid, job_uid, 'status')['status']
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |   File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/commandcommon.py", line 191, in wrapper
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |     return func(*args, **kwargs)
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |   File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 5552, in get_job
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    |     raise ValueError(f"{project_uid} {job_uid} does not exist.")
2023-01-13 09:44:26,027 COMMAND.COMMON       wrapper              ERROR    | ValueError: P242 J28 does not exist.
2023-01-13 09:44:26,051 COMMAND.COMMON       wrapper              ERROR    | JSONRPC ERROR at dump_workspaces

I have tried to delete job from icli as described here: Job/project forced removal for a deleted project directory

But with no luck.

Any ideas on how to proceed?

Best,
Jesper

2

Please can you check that:

  1. the web UI’s Current Jobs tab shows no jobs belonging to the deleted project
  2. no processes referring to P242 on master or worker(s):
    ps -eo args | grep P242
  3. there are no left-over Live sessions that have not either been completed or paused
cryosparcm icli
list(db.workspaces.find({'session_uid': {'$ne': None}, 'status': {'$nin': ['paused', 'completed']}}, {'project_uid': 1, 'session_uid': 1, 'title': 1, 'status': 1}))
3

Thanks @wtempel for getting back to me on this one.
The log is still going crazy and unfortunately nothing shows up from the suggested checks.
I even checked all the workers and also rebooted the master, nothing helps.

4

Any hint on how to stop this could be nice?
My command_core and command_rtp are still filling up fast with this error message and produces ~20 MB of log-files each hour.