How can I solve this problem?

Troubleshooting
1

[CPU: 369.1 MB] Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f612a2d0d50>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1722, in run_with_except_hook
run_old(*args, **kw)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 164, in thread_work
work = processor.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 260, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 117, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 433, in context_dependent_memoize
result = func(*args)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 498, in get_fill_kernel
“fill”)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmp3rl9jiur.cu failed
[command: nvcc --preprocess -arch sm_86 -I/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmp3rl9jiur.cu --compiler-options -P]
[stderr:
b"nvcc fatal : Value ‘sm_86’ is not defined for option ‘gpu-architecture’\n"]

[CPU: 369.1 MB] Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f612a2d0d50>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1722, in run_with_except_hook
run_old(*args, **kw)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 164, in thread_work
work = processor.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 260, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 117, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 433, in context_dependent_memoize
result = func(*args)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 498, in get_fill_kernel
“fill”)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpp7jdc9lr.cu failed
[command: nvcc --preprocess -arch sm_86 -I/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmpp7jdc9lr.cu --compiler-options -P]
[stderr:
b"nvcc fatal : Value ‘sm_86’ is not defined for option ‘gpu-architecture’\n"]

[CPU: 369.1 MB] Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f612a2d0d50>

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 1722, in run_with_except_hook
run_old(*args, **kw)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/threading.py”, line 870, in run
self._target(*self._args, **self._kwargs)
File “/opt/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 164, in thread_work
work = processor.process(item)
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 190, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 193, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 195, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.motionworker.process
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 251, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 260, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py”, line 117, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py”, line 549, in fill
func = elementwise.get_fill_kernel(self.dtype)
File “”, line 2, in get_fill_kernel
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 433, in context_dependent_memoize
result = func(*args)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 498, in get_fill_kernel
“fill”)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 163, in get_elwise_kernel
arguments, operation, name, keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 149, in get_elwise_kernel_and_types
keep, options, **kwargs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py”, line 76, in get_elwise_module
options=options, keep=keep, no_extern_c=True)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 291, in init
arch, code, cache_dir, include_dirs)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 254, in compile
return compile_plain(source, options, keep, nvcc, cache_dir, target)
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 78, in compile_plain
checksum.update(preprocess_source(source, options, nvcc).encode(“utf-8”))
File “/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py”, line 55, in preprocess_source
cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmp92f4uoqw.cu failed
[command: nvcc --preprocess -arch sm_86 -I/opt/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmp92f4uoqw.cu --compiler-options -P]
[stderr:
b"nvcc fatal : Value ‘sm_86’ is not defined for option ‘gpu-architecture’\n"]

[CPU: 225.0 MB] Outputting partial results now…
[CPU: 226.7 MB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 84, in cryosparc_compute.run.main
File “cryosparc_worker/cryosparc_compute/jobs/motioncorrection/run_patch.py”, line 402, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi
AssertionError: Child process with PID 70379 has terminated unexpectedly!

2

Hey Dali,
What graphics card are you using, and which CUDA version do you have installed?

3

I have this same error (Patch CTF error), using an RTX-3090 with CUDA 11.2. Has anyone got this combo to work, or is there a recommended version of CUDA to use with the 3090 cards?

4

Hi @olibclarke and @Dali,

I’m also having this issue following a reinstallation of our system and the cuda code. We have the 3090 RTX cards (2) running the newest driver version (I think its 465.27) and then the 10.2 version of cuda. If someone has any new advice since this ticket was opened, that would be greatly appreciated!

Best,
Justas

5

Hi @jr10, everything is working for me now with CUDA 11.2 and Nvidia driver version 460.67 on the latest cryosparc (3.2), hope that helps!

6

Hi @olibclarke,

Thanks for the update, I’m also moving to a newer version of cuda (11.3) since I found some info that suggests the error is due to 10.x cuda versions not supporting the new Ampere architecture.

Best,
Justas

7

yes, you will definitely need CUDA 11.x for this reason

I haven’t tried 11.3 yet, but 11.2 works

8

Hi,
@olibclarke @Dali I get a similar error in patch motion jobs, we have V3.2.0 cryosparc version and V11.4.100 Cuda version, and 470.57.02 Nvidia Driver Version on our system.
Error- ““File “/home/cryosparc/app/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py”, line 429, in context_dependent_memoize
return ctx_dict[cur_ctx][args]
KeyError: <pycuda._driver.Context object at 0x7f6b09369b10>””
Is there any way to address this?
Best,
Kapil

1 Like