V3.3.1 - job failure in patch motion correct

Troubleshooting
1

Hi, I recently upgraded a cryosparc installation to v3.3.1 (cluster mode). There were no errors with the installation. But when running through the test case as described at https://guide.cryosparc.com/processing-data/cryo-em-data-processing-in-cryosparc-introductory-tutorial with the T20S data set in step 5: motion correct. The job starts on the cluster worker node but then fails with:

[CPU: 212.7 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 402, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 61921 has terminated unexpectedly!

From the job.log:

Process Process-1:1:
Traceback (most recent call last):
  File "/uufs/chpc.utah.edu/sys/installdir/cryosparc/2.14.2/cryosparc_worker/deps/anacon
da/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py", line 297, in _bo
otstrap
    self.run()
  File "/uufs/chpc.utah.edu/sys/installdir/cryosparc/2.14.2/cryosparc_worker/deps/anacon
da/envs/cryosparc_worker_env/lib/python3.7/multiprocessing/process.py", line 99, in run
    self._target(*self._args, **self._kwargs)
  File "/uufs/chpc.utah.edu/sys/installdir/cryosparc/2.14.2/cryosparc_worker/cryosparc_c
ompute/jobs/pipeline.py", line 176, in process_work_simple
    process_setup(proc_idx) # do any setup you want on a per-process basis
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 81,
 in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.
process_setup
  File "/uufs/chpc.utah.edu/sys/installdir/cryosparc/2.14.2/cryosparc_worker/cryosparc_c
ompute/engine/__init__.py", line 8, in <module>
    from .engine import *  # noqa
  File "cryosparc_worker/cryosparc_compute/engine/engine.py", line 9, in init cryosparc_
compute.engine.engine
  File "cryosparc_worker/cryosparc_compute/engine/cuda_core.py", line 4, in init cryospa
rc_compute.engine.cuda_core
ModuleNotFoundError: No module named 'pycuda'
**** handle exception rc
set status to failed

Thanks,

Anita Orendt

2

Welcome to the forum @amorendt.
Assuming a functional cuda installation on your worker(s), you may first try on the worker(s):
cryosparcw newcuda <cuda-path>
Does this command resolve the problem? If not, please repeat the worker installation. If the problem remains unresolved, please report back in this forum thread.