CTF Correction dtype error

Greetings Cryosparc Community,

I have recently run into an error while running Cryosparc Live. I am quite new to this process, so any direction would be immensely appreciated.

When I run CTF correction I get the following output error:

Traceback (most recent call last):
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 429, in context_dependent_memoize
    return ctx_dict[cur_ctx][args]
KeyError: (dtype('float32'),)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 376, in cryosparc_compute.jobs.rtp_workers.run.rtp_worker
  File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 439, in cryosparc_compute.jobs.rtp_workers.run.process_movie
  File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 585, in cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
  File "cryosparc_worker/cryosparc_compute/jobs/rtp_workers/run.py", line 590, in cryosparc_compute.jobs.rtp_workers.run.do_patch_motion
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 255, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 264, in cryosparc_compute.jobs.motioncorrection.patchmotion.unbend_motion_correction
  File "cryosparc_worker/cryosparc_compute/jobs/motioncorrection/patchmotion.py", line 121, in cryosparc_compute.jobs.motioncorrection.patchmotion.prepare_movie_for_processing
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/gpuarray.py", line 549, in fill
    func = elementwise.get_fill_kernel(self.dtype)
  File "<decorator-gen-13>", line 2, in get_fill_kernel
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/tools.py", line 433, in context_dependent_memoize
    result = func(*args)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 498, in get_fill_kernel
    "fill")
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 163, in get_elwise_kernel
    arguments, operation, name, keep, options, **kwargs)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 149, in get_elwise_kernel_and_types
    keep, options, **kwargs)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/elementwise.py", line 76, in get_elwise_module
    options=options, keep=keep, no_extern_c=True)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 291, in __init__
    arch, code, cache_dir, include_dirs)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 254, in compile
    return compile_plain(source, options, keep, nvcc, cache_dir, target)
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 78, in compile_plain
    checksum.update(preprocess_source(source, options, nvcc).encode("utf-8"))
  File "/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/compiler.py", line 55, in preprocess_source
    cmdline, stderr=stderr)
pycuda.driver.CompileError: nvcc preprocessing of /tmp/tmpkx366p8_.cu failed
[command: nvcc --preprocess -arch sm_75 -I/home/cryosparcusr/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pycuda/cuda /tmp/tmpkx366p8_.cu --compiler-options -P]
[stderr:
b"In file included from /usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/cuda_runtime.h:83,\n                 from <command-line>:\n/usr/local/cuda-11.2/bin/../targets/x86_64-linux/include/crt/host_config.h:139:2: error: #error -- unsupported GNU version! gcc versions later than 10 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.\n  139 | #error -- unsupported GNU version! gcc versions later than 10 are not supported! The nvcc flag '-allow-unsupported-compiler' can be used to override this version check; however, using an unsupported host compiler may cause compilation failure or incorrect run time execution. Use at your own risk.\n      |  ^~~~~\n"]

The secondary exception I understand to be a version warning, it is the first error that is throwing me off. It appears that when sending context data to pycuda the software is throwing a datatype error.

If there is any additional information needed to help diagnose this information, I would be happy to provide it.

Thank you for your guidance,
Ali

Greetings and welcome to the forum @morteza.
How many worker hosts are connected to your CryoSPARC instance and what OS (with version) are they running?

Thank you for the kind welcome.

We are using Ubuntu 22-04-1 LTS, and are running just one worker host.

Best,
Ali,

CryoSPARC v3 is not supported on Ubuntu 22.04.
For an untested workaround, you may try:

  1. Install a version of the CUDA toolkit that’s supported on Ubuntu 22.04, such as CUDA-11.7.1.
eval $(/path/to/cryosparc_worker/bin/cryosparcw env)
conda update libstdcxx-ng -c conda-forge
cryosparcw newcuda /path/to/compatible/cuda

Does this fix the problem?

1 Like

Updating to the newest version of CUDA appears to have corrected the problem.

Thank you so much for your guidance.

Best,
Ali

1 Like