Master has become unstable

Troubleshooting
1

Dear all,

For the second day in a row our cryoSPARC master v3.2 has crashed.

Webapp runs at a 100% CPU load and does not respond to any requests.

In the system log, it shows that:

cryosparc systemd: Starting Cleanup of Temporary Directories...

Right before the crash.

“cryosparcm log comand_core” runs in a internal circle of error messages, like this:

[JSONRPC ERROR  2021-11-10 15:49:04.239878  at  get_job ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    res = func(*args, **kwargs)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4043, in get_job
    return sanitize_id(com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs))
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 175, in sanitize_id
    if '_id' in obj:
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[JSONRPC ERROR  2021-11-10 15:49:04.240005  at  get_job_status ]
-----------------------------------------------------
Traceback (most recent call last):
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    res = func(*args, **kwargs)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4919, in get_job_status
    return get_job(project_uid, job_uid, 'status')['status']
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 133, in wrapper
    raise e
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 124, in wrapper
    res = func(*args, **kwargs)
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 4043, in get_job
    return sanitize_id(com.get_job(mongo.db, project_uid, job_uid, *args, **kwargs))
  File "/opt/bioxray/programs/cryosparc2/cryosparc2_master/cryosparc_command/command_core/__init__.py", line 175, in sanitize_id
    if '_id' in obj:
TypeError: argument of type 'NoneType' is not iterable
-----------------------------------------------------
[EXPORT_WORKSPACES] : Exporting all workspaces in P269..

Any idea what is going on and even better how to fix it?

Best,
Jesper

2

Hi @jelka,

Thanks for the post, this is quite unusual. Were you running any jobs (particularly interactive ones) before this happened? Did you delete any jobs recently? Is the machine running the cryoSPARC master component running fine (CPU, memory, network, disk)?

Could you please provide the following logs:

cryosparcm log command_core | tail -n 100
cryosparcm log database | tail -n 100
cryosparcm log webapp | tail -n 100

- Suhail