Patch Motion Correction not finishing

[MNSP]

Hello,

I’m trying to perform “patch motion correction” in CryoSPARC for (EMPIAR 11012). As far as I understand, this error likely means I’m missing some necessary input details.

PS: I say this is an error because this stays like this for days/weeks and nothing ever happens.

Project : P12
Workspace : W2
Job ID (Patch Motion Correction, that gives the error): J29
Job ID ( Import Movies Job before this one) : J2

Output of command: ps -eo pid,ppid,start,cmd | grep cryosparc
2716 23421 13:10:12 ssh -X -Y cryosparc_user@ban
2734 1 13:11:23 /opt/google/chrome/chrome_crashpad_handler --monitor-self --monitor-self-annotation=ptype=crashpad-handler --database=/home/cryosparc_user/.config/google-chrome/Crash Reports --metrics-dir=/
home/cryosparc_user/.config/google-chrome --url=https://clients2.google.com/cr/report --annotation=channel= --annotation=lsb-release=Fedora 24 (Workstation Edition) --annotation=plat=Linux --annotation=prod=Chro
me_Linux --annotation=ver=103.0.5060.114 --initial-client-fd=6 --shared-client-connection
2736 1 13:11:23 /opt/google/chrome/chrome_crashpad_handler --no-periodic-tasks --monitor-self-annotation=ptype=crashpad-handler --database=/home/cryosparc_user/.config/google-chrome/Crash Reports --url=http
s://clients2.google.com/cr/report --annotation=channel= --annotation=lsb-release=Fedora 24 (Workstation Edition) --annotation=plat=Linux --annotation=prod=Chrome_Linux --annotation=ver=103.0.5060.114 --initial-c
lient-fd=4 --shared-client-connection
3823 3652 Apr 29 vim /home/cryosparc_user/.bashrc
7958 4414 17:39:13 grep --color=auto cryosparc

Output of command: cryosparcm joblog P12 J29
================= CRYOSPARCW ======= 2024-05-07 16:37:55.503671 =========
Project P12 Job J29
Master ban.polikanovlab Port 39002

========= monitor process now starting main process at 2024-05-07 16:37:55.503763
Traceback (most recent call last):
File “”, line 1, in
File “cryosparc_master/cryosparc_compute/run.py”, line 172, in cryosparc_compute.run.run
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/jobs/init.py”, line 9, in
from . import runcommon # noqa
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 64, in
from … import dataset
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/dataset.py”, line 33, in
from cryosparc_tools.cryosparc.dataset import Dataset, Row, Spool
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 66, in
from .errors import DatasetLoadError
ModuleNotFoundError: No module named ‘cryosparc_tools.cryosparc.errors’
MAINPROCESS PID 21163
Process Process-1:
Traceback (most recent call last):
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “cryosparc_master/cryosparc_compute/run.py”, line 30, in cryosparc_compute.run.main
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/jobs/init.py”, line 9, in
from . import runcommon # noqa
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 64, in
from … import dataset
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_compute/dataset.py”, line 33, in
from cryosparc_tools.cryosparc.dataset import Dataset, Row, Spool
File “/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 66, in
from .errors import DatasetLoadError
ModuleNotFoundError: No module named ‘cryosparc_tools.cryosparc.errors’
Waiting for data… (interrupt to abort)

Thanks so much in advance,
Regards,
MNSP

[MNSP]

@wtempel , I mentioned you since I was unable to provide you the necessary information the last time I was facing similar issue and you had asked for this information. Looking forward to hearing from you!

[wtempel]

Please can you run the following commands on the ban.polikanovlab computer and post their outputs:

hostname -f
ps -eo pid,ppid,start,cmd | grep cryosparc_
/csb/work2/cryosparc_user/cryosparc/cryosparc_worker/bin/cryosparcw gpulist
cat /csb/work2/cryosparc_user/cryosparc/cryosparc_worker/version
cryosparcm status | grep -v LICENSE

[MNSP]

As I understand the commands you suggested are looking for specifics about the cryosparc version and nvidia drivers we have. Yesterday we figured out that we did have some incompatibilities. Therefore we rolled back our cryosparc versions and nvidia drivers to make them compatible to our graphic cards. Now we have cryosparc v4.3.1.
Although the process has started now, we are unable to parallelize it on multiple GPUs. Is there a way to do that? Thank you!

[wtempel]

@MNSP

we are unable to parallelize it on multiple GPUs. Is there a way to do that?

Some job types, like Patch Motion Correction, Patch CTF Estimation, Extract From Micrographs (GPU) or 2D Classification have a Number of GPUs to parallelize parameter that enable parallelization over a number of GPUs on the same host.

[MNSP]

I see, is this something done at the backend during set up of the cryosparc architecture or is it something the user does per job. I am unable to select multiple GPUs here , if that is what you are referring to?

[wtempel]

@MNSP

or is it something the user does per job.

The Number of GPUs to parallelize parameter needs to be defined for during job building, if the job type supports that parameter.

[MNSP]

Perfect! Thank you so much!

Indeed a silly mistake on my part! I had seen this but I thought the upper limit was 1 because it had 0/1 written next to the compute settings! Thank you!