Failed to do patch CTF estimation

CTF Estimation
1

I just updated to v4.2.1 and found that the Patch CTF Estimation always got failed. The error message in the job.log mentioned something wrong with libcufft.so.10. I paste some of the message here:
Traceback (most recent call last):
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 200, in process_work_simple
process_setup(proc_idx) # do any setup you want on a per-process basis
File “cryosparc_master/cryosparc_compute/jobs/ctf_estimation/run.py”, line 79, in cryosparc_compute.jobs.ctf_estimation.run.run.process_setup
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 12, in init cryosparc_compute.engine.cuda_core
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py”, line 9, in
from . import gpufft
ImportError: libcufft.so.10: cannot open shared object file: No such file or directory
Process Process-1:2:
Traceback (most recent call last):
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 315, in _bootstrap
self.run()
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py”, line 108, in run
self._target(*self._args, **self._kwargs)
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py”, line 200, in process_work_simple
process_setup(proc_idx) # do any setup you want on a per-process basis
File “cryosparc_master/cryosparc_compute/jobs/ctf_estimation/run.py”, line 79, in cryosparc_compute.jobs.ctf_estimation.run.run.process_setup
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 12, in init cryosparc_compute.engine.cuda_core
File “/data1/share/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py”, line 9, in
from . import gpufft
ImportError: libcufft.so.10: cannot open shared object file: No such file or directory
Traceback (most recent call last):
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py”, line 245, in _feed
send_bytes(obj)
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py”, line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py”, line 411, in _send_bytes
self._send(header + buf)
File “/data1/share/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py”, line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe

I feel like it may be some version mismatch, but cannot figure it out. Please help!

2

Did you make sure that CUDA wasn’t in the PATH or LD_LIBRARY_PATH when updating?

What version of CUDA does nvcc --version report? Have you updated CUDA (CUDA 12 to my knowledge still does not work with CryoSPARC)?

3

Thank you for the quick replying. Our work station was down yesterday due to some package installation so I couldn’t check the info you mentioned. The CUDA version is 10. But I’m not quite sure about “make sure that CUDA wasn’t in the PATH or LD_LIBRARY_PATH when updating”, but here is what’s in .bashrc file. Do you think it’s the problem?

#CUDA10.0
export PATH=“/usr/local/cuda-10.0/bin”:$PATH
export LD_LIBRARY_PATH=/usr/local/cuda-10.0/lib64:$LD_LIBRARY_PATH

Added by cryoSPARC:

export PATH=“/data1/share/cryosparc/cryosparc_master/bin”:$PATH

4

It might well be. These entries are not required by CryoSPARC, and may interfere with CryoSPARC installation and operation.
I recommend, assuming

  • these .bashrc lines are not required by other programs
  • the nvidia driver version is at least 460.32.03 on your worker(s)

that you

  • comment out these lines
  • exit the shell
  • in a fresh shell, ensure
    which nvcc
    does not print any result
  • install 3DFlex prerequisites, which will (also) install and configure a CUDA toolkit for use with CryoSPARC (details)
cd /data1/share/cryosparc/cryosparc_worker/
./bin/cryosparcw install-3dflex
1 Like