No such file or directory: 'nvcc'

Install, Configure and Update
1

Hi,

I am trying to run jobs on Cryosparc but they all fail, giving the following error:

Traceback (most recent call last):
File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/init.py”, line 638, in _deco
return func._memoize_dic[args] # pylint: disable=protected-access
AttributeError: ‘function’ object has no attribute ‘_memoize_dic’

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/site-packages/pytools/prefork.py”, line 49, in call_capture_output
stderr=PIPE)
File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py”, line 800, in init
restore_signals, start_new_session)
File “/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/subprocess.py”, line 1551, in _execute_child
raise child_exception_type(errno_num, err_msg, err_filename)
FileNotFoundError: [Errno 2] No such file or directory: ‘nvcc’: ‘nvcc’

Does anyone have a fix for this?

Thanks!

2

Did it work before, or is this a new installation?

Have you updated CUDA?

nvcc is the CUDA compiler, and should not randomly go missing.

Check with which nvcc in a terminal, and check CUDA is added to $PATH and $LD_LIBRARY_PATH.

3

Before running this command, please execute (in the same terminal window)
eval $(/CryoSparc/CryoSparcV2/cryosparc2_worker/bin/cryosparcw env)

4

It worked before and we did not update cryosparc or CUDA.

This is what ‘which nvcc’ gives me:

/usr/bin/which: no nvcc in (/CryoSparc/CryoSparcV2/cryosparc2_worker/bin:/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/CryoSparc/CryoSparcV2/cryosparc2_worker/deps/anaconda/condabin:/usr/local/cuda-10.1/bin:/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/bin:/bin:/condabin:/CryoSparc/CryoSparcV2/cryosparc2_master/deps/external/mongodb/bin:/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/bin:/CryoSparc/CryoSparcV2/cryosparc2_master/bin:/CryoSparc/CryoSparcV2/cryosparr)

5

How many worker nodes does your cryoSPARC instance have?
Is it possible that on the worker where the job ran, nvcc does in fact not exist at the path
${CRYOSPARC_CUDA_PATH}/bin, as defined in /CryoSparc/CryoSparcV2/cryosparc2_worker/config.sh?
Please can you post the output of

eval $(/CryoSparc/CryoSparcV2/cryosparc2_worker/bin/cryosparcw env)
echo $CRYOSPARC_CUDA_PATH
ls -l ${CRYOSPARC_CUDA_PATH}/bin

on the worker node where the failed job ran.

6

There is one worker node.

This is the output on the worker node:

[cchana@quad cryosparc2_worker]$ eval $(/CryoSparc/CryoSparcV2/cryosparc2_worker/bin/cryosparcw env)
[cchana@quad cryosparc2_worker]$ echo $CRYOSPARC_CUDA_PATH
/usr/local/cuda-10.1
[cchana@quad cryosparc2_worker]$ ls -l ${CRYOSPARC_CUDA_PATH}/bin
total 43200
lrwxrwxrwx. 1 root root 4 Oct 18 2019 computeprof → nvvp
-rwxr-xr-x. 1 root root 8893000 Jul 29 2019 cuda-gdb
-rwxr-xr-x. 1 root root 581744 Jul 29 2019 cuda-gdbserver
-rwxr-xr-x. 1 root root 397472 Jul 29 2019 cuda-memcheck
-rwxr-xr-x. 1 root root 185896 Jul 29 2019 cuobjdump
-rwxr-xr-x. 1 root root 1148928 Jul 29 2019 gpu-library-advisor
-rwxr-xr-x. 1 root root 219 Jul 29 2019 nsight
-rwxr-xr-x. 1 root root 1580 Jul 29 2019 nsight_ee_plugins_manage.sh
-rwxr-xr-x. 1 root root 438 Jul 6 2019 nsight-sys
-rwxr-xr-x. 1 root root 524 Jul 6 2019 nsys
-rwxr-xr-x. 1 root root 524 Jul 6 2019 nsys-exporter
-rwxr-xr-x. 1 root root 22807192 Jul 29 2019 nvdisasm
-rwxr-xr-x. 1 root root 2583 Jul 2 2019 nv-nsight-cu
-rwxr-xr-x. 1 root root 2801 Jul 18 2019 nv-nsight-cu-cli
-rwxr-xr-x. 1 root root 10076848 Jul 29 2019 nvprof
-rwxr-xr-x. 1 root root 86680 Jul 29 2019 nvprune
-rwxr-xr-x. 1 root root 215 Jul 29 2019 nvvp

7

That’s looks unusual (to me).
Had it not been for the message

I would have expected nvcc to be included in the file listing. How was this toolkit installed? There may be CUDA toolkit distributions where nvcc would end up at a different path.
Assuming no other users or programs are affected by configuring another version of CUDA, I suggest installation of version 11.2 of the CUDA toolkit. For cryoSPARC to work with the newly installed toolkit, you would need to run (after toolkit installation or update, and confirmation that nvcc is present in /path/to/new_cuda_installation/bin/):
/CryoSparc/CryoSparcV2/cryosparc2_worker/bin/cryosparcw newcuda \ /path/to/new_cuda_installation
(details).

8

We had CUDA 11.6 installed (with nvcc present) so we changed the path from version 10.1 to 11.6.

This fixed the issue. Thanks for your help!