Motion Corr fails after update to v4.0.1

Troubleshooting
1

Hi,

Tried re-updating to v4.0.1 and forcing update of worker node again separately, however we’re still unable to use patch motion corr with the following error…

Error occurred while processing J1/imported/002355912254767308764_Micrograph_03981.tiff
Traceback (most recent call last):
File “/data/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/pipeline.py”, line 60, in exec
return self.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 177, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 180, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 182, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 264, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 125, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/cuda_kernels.py”, line 42, in cryosparc_compute.jobs.motioncorrection.cuda_kernels.do_accumframe_gpu
File “cryosparc_worker/cryosparc_compute/engine/cuda_core.py”, line 416, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/cuda_kernels.py”, line 37, in cryosparc_compute.jobs.motioncorrection.cuda_kernels.get_accumframe_gpu
File “/data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 137, in compile_plain
stderr=stderr.decode(“utf-8”, “replace”))
pycuda.driver.CompileError: nvcc compilation of /tmp/tmpzjjeqovg/kernel.cu failed
[command: nvcc --cubin -arch sm_61 -I/data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda kernel.cu]
[stderr:
cicc: /lib64/libstdc++.so.6: version GLIBCXX_3.4.21' not found (required by /data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/./libLerc.so) cicc: /lib64/libstdc++.so.6: version CXXABI_1.3.9’ not found (required by /data/cryosparc/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/./libLerc.so)
]

Marking J1/imported/002355912254767308764_Micrograph_03981.tiff as incomplete and continuing…

Any suggestions to fix this?

Thanks
Max

2

Looks as if we’re having a similar issue with other gpu based job types

3

Perhaps related to this:

… with a potential workaround further down in the thread, here.

Cheers,
Yang

4

@maxm Please can you post the following information:

5

CUDA toolkit path:
/usr/local/cuda

Version:

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Wed_Jun__2_19:15:15_PDT_2021
Cuda compilation tools, release 11.4, V11.4.48
Build cuda_11.4.r11.4/compiler.30033411_0

pycuda information:
(11, 4, 0)

output of final command:

Linux marufgpu 3.10.0-1160.36.2.el7.x86_64 #1 SMP Wed Jul 21 11:57:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

              total        used        free      shared  buff/cache   available
Mem:            125           4           4           0         117         120
Swap:            15           0          15

Wed Oct 12 16:01:02 2022       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.42.01    Driver Version: 470.42.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  Quadro P5000        Off  | 00000000:02:00.0 Off |                  Off |
| 26%   28C    P8     6W / 180W |      2MiB / 16278MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  Quadro P5000        Off  | 00000000:04:00.0 Off |                  Off |
| 26%   27C    P8     6W / 180W |     18MiB / 16278MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  Quadro P5000        Off  | 00000000:83:00.0 Off |                  Off |
| 26%   30C    P8     6W / 180W |      2MiB / 16278MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  Quadro P5000        Off  | 00000000:84:00.0 Off |                  Off |
| 26%   25C    P8     6W / 180W |      2MiB / 16278MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    1   N/A  N/A      2396      G   /usr/bin/X                          9MiB |
|    1   N/A  N/A      2763      G   /usr/bin/gnome-shell                5MiB |
+-----------------------------------------------------------------------------+
6

Hi Yang,

Thanks for this, does indeed seem to be the same issue, and it looks to be working now with this fix!

Max

7

@maxm @leetleyang
The CryoSPARC v4.0.1+221017 patch includes a fix for this problem. Before installing the fix, please undo the links that you may have configured as a workaround.