Error launching jobs using GPus

Troubleshooting
1

Hi, I get the following error when I launch any job that uses GPU. I have doubts if CS is compatible with the GPU architecture that I have.

[CPU: 215.2 MB] Error occurred while processing J3/imported/GridSquare_13802863_Data_FoilHole_14804672_Data_13808138_13808140_20210415_135321-3630_aligned_mic_DW.mrc Traceback (most recent call last):

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize return ctx_dict[cur_ctx][args]

KeyError: <pycuda._driver.Context object at 0x7ff6be18ee10> During handling of the above exception, another exception occurred: Traceback (most recent call last):

File “/programas/cryoSPARC/v3.2/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 59, in exec return self.process(item) File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 112, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 118, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py”, line 119, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 70, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File “cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py”, line 116, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill func = elementwise.get_fill_kernel(self.dtype)

File “”, line 2, in get_fill_kernel

*File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize result = func(args)

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel “fill”)

**File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel arguments, operation, name, keep, options, kwargs)

**File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types keep, options, kwargs)

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module options=options, keep=keep, no_extern_c=True)

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in init arch, code, cache_dir, include_dirs)

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile return compile_plain(source, options, keep, nvcc, cache_dir, target)

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 78, in compile_plain checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))

File"/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 55, in preprocess_source cmdline, stderr=stderr)

pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpuronti3t.cu failed [command: nvcc --preprocess -arch sm_80 -I/programas/cryoSPARC/v3.2/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmpuronti3t.cu --compiler-options -P] [stderr: b"nvcc

fatal : Value ‘sm_80’ is not defined for option ‘gpu-architecture’\n"] Marking J3/imported/GridSquare_13802863_Data_FoilHole_14804672_Data_13808138_13808140_20210415_135321-3630_aligned_mic_DW.mrc as incomplete and continuing…

2

Which CUDA version you are using? You need at least 11.0+ for A100 card.
You can check the list of allowed archs in nvcc --help

3

Hello Azazello, thank you for your quick response. I have a “sm_80” architecture and the card model is “NVidea A100-SXM4-40GB”. I have several cudas installed and I have tried all of them. With the cuda-11.x it gives me that error

4

Is sm_80 in in the list of allowed archs in nvcc --help?

5

Cuda compilation tools, release 11.2, V11.2.67
Build cuda_11.2.r11.2/compiler.29373293_0
root@amy:/programas/scipion/v3.0/config# nvcc --help |grep compute_80
‘compute_72’,‘compute_75’,‘compute_80’,‘compute_86’,‘lto_35’,‘lto_37’,‘lto_50’,
‘compute_72’,‘compute_75’,‘compute_80’,‘compute_86’,‘lto_35’,‘lto_37’,‘lto_50’,

Cuda compilation tools, release 10.2, V10.2.89
root@amy:/programas/scipion/v3.0/config# nvcc --help |grep compute_80
root@amy:/programas/scipion/v3.0/config#

6

Alright, was cryosparc worker installed & configured with CUDA 11.2?

7

Yes, I installed cryosparc with cuda11-2. I’ll reinstall again!