Hi,
I just installed CS in a new standalone machine with QuadroK4200 card (newer than what we were using). But after clean compilation of NVIDIA driver, CUDA11.4 and CS3.3.2 I’m getting this error while doing CTF estimation.
[CPU: 308.9 MB] Error occurred while processing J1/imported/006900796605051975576_000000815831687031299_FoilHole_29371650_Data_29355489_29355491_20220129_132947_fractions_patch_aligned_doseweighted.mrc
Traceback (most recent call last):
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f72ec185d50>
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/abhisek/cs/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 60, in exec
return self.process(item)
File "cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py", line 112, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process
File "cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py", line 118, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process
File "cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/run.py", line 119, in cryosparc_compute.jobs.ctf_estimation.run.run.ctfworker.process
File "cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py", line 70, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217
File "cryosparc_worker/cryosparc_compute/jobs/ctf_estimation/patchctf.py", line 116, in cryosparc_compute.jobs.ctf_estimation.patchctf.patchctf_v217
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File "<decorator-gen-13>", line 2, in get_fill_kernel
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
result = func(*args)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
"fill")
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
arch, code, cache_dir, include_dirs)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
File "/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 55, in preprocess_source
cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmps1sruke6.cu failed
[command: nvcc --preprocess -arch sm_30 -I/home/abhisek/cs/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmps1sruke6.cu --compiler-options -P]
[stderr:
b"nvcc fatal : Value 'sm_30' is not defined for option 'gpu-architecture'\n"]
Marking J1/imported/006900796605051975576_000000815831687031299_FoilHole_29371650_Data_29355489_29355491_20220129_132947_fractions_patch_aligned_doseweighted.mrc as incomplete and continuing...
I’m not sure what’s going on. I found no other CUDA in my system except recently installed 11.4. It’s odd for me because I haven’t seen this annyoing issue with other CS installation which is running on nvcc-10.1 but with GeForce 1080 card.
I tried this to resolve: Nvcc preprocessing failed
But no luck there.
Please advise what is wrong with compilation or driver ?
Thanks