Computing error during motion correction

Install, Configure and Update
1

Error occurred while processing J2/imported/002420043306325089619_FoilHole_24441263_Data_24440981_24440983_20250724_090040_Fractions.mrc
Traceback (most recent call last):
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec
return self.process(item)
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 213, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 216, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 245, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 222, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 292, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 301, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_master/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 157, in cryosparc_master.cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/gpu/gpuarray.py”, line 216, in fill
from .elementwise.fill import fill
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/gpu/elementwise/fill.py”, line 24, in
def fill(arr, x, out):
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/np/ufunc/decorators.py”, line 203, in wrap
guvec.add(fty)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/np/ufunc/deviceufunc.py”, line 475, in add
kernel = self._compile_kernel(fnobj, sig=tuple(outertys))
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/vectorizers.py”, line 241, in _compile_kernel
return cuda.jit(sig)(fnobj)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/decorators.py”, line 133, in _jit
disp.compile(argtypes)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/dispatcher.py”, line 934, in compile
kernel.bind()
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/dispatcher.py”, line 197, in bind
self._codelibrary.get_cufunc()
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/codegen.py”, line 195, in get_cufunc
cubin = self.get_cubin(cc=device.compute_capability)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/codegen.py”, line 166, in get_cubin
linker = driver.Linker.new(max_registers=self._max_registers, cc=cc)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 2601, in new
return CudaPythonLinker(max_registers, lineinfo, cc)
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numba/cuda/cudadrv/driver.py”, line 2888, in init
cc_enum = getattr(binding.CUjit_target,
File “/home/cryosparcuser/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/enum.py”, line 437, in getattr
raise AttributeError(name) from None
AttributeError: CU_TARGET_COMPUTE_120. Did you mean: ‘CU_TARGET_COMPUTE_20’?

Marking J2/imported/002420043306325089619_FoilHole_24441263_Data_24440981_24440983_20250724_090040_Fractions.mrc as incomplete and continuing…

2

Is it a mismatch between the new Nvidia drivers and the cuda version installed with cryosparc?

3

@boise What are the outputs of the commands

uname -a
cat /home/cryosparcuser/cryosparc/cryosparc_worker/version
nvidia-smi

on the CryoSPARC worker computer where the job ran (and failed)?

4

Hello @wtempel . Here is the output for the commands you asked for

Linux bigbend 5.14.0-503.40.1.el9_5.x86_64 #1 SMP PREEMPT_DYNAMIC Wed Apr 30 17:38:54 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
v4.7.1
Mon Jul 28 11:13:17 2025       
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 570.133.07             Driver Version: 570.133.07     CUDA Version: 12.8     |
|-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GeForce RTX 5070 Ti     Off |   00000000:41:00.0  On |                  N/A |
|  0%   48C    P5             34W /  300W |    1607MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GeForce RTX 5070 Ti     Off |   00000000:61:00.0 Off |                  N/A |
|  0%   38C    P8             18W /  300W |      34MiB /  16303MiB |      0%      Default |
|                                         |                        |                  N/A |
+-----------------------------------------+------------------------+----------------------+
                                                                                         
+-----------------------------------------------------------------------------------------+
| Processes:                                                                              |
|  GPU   GI   CI              PID   Type   Process name                        GPU Memory |
|        ID   ID                                                               Usage      |
|=========================================================================================|
|    0   N/A  N/A           67314      G   /usr/libexec/Xorg                       110MiB |
|    0   N/A  N/A           69557      G   /usr/libexec/Xorg                       269MiB |
|    0   N/A  N/A           69658      G   /usr/bin/gnome-shell                     65MiB |
|    0   N/A  N/A          106622      G   /usr/libexec/Xorg                       110MiB |
|    0   N/A  N/A          123450      G   /usr/lib64/firefox/firefox              326MiB |
|    1   N/A  N/A           67314      G   /usr/libexec/Xorg                         4MiB |
|    1   N/A  N/A           69557      G   /usr/libexec/Xorg                         4MiB |
|    1   N/A  N/A          106622      G   /usr/libexec/Xorg                         4MiB |
+-----------------------------------------------------------------------------------------+
5

Thanks for posting this information. The rtx5070ti card might not be supported in “standard” CryoSPARC v4.7.1. You may want to re-try the job after updating CryoSPARC to “special” v4.7.1-cuda12.

6

Thank you @wtempel. I am getting the following error.
bin]$ ./cryosparcm update —version=v4.7.1 -cuda12
Invalid argument: —version=v4.7.1

7

its working now. it is a syntax error. it should have been --version instead of -version in the command

8

The version string v4.7.1-cuda12 is continuous and should not include spaces:

cryosparcm update --version=v4.7.1-cuda12
1 Like