We recently updated our Cryosparc instance to v4.2.1 using the cryosparcm patch command. Afterwards, our v10 Cuda instance was accidentally overwritten by the Cuda v9 Ubuntu package. The Cuda v9 package was uninstalled/purged and the Cuda v10 reinstalled to working order using Nvidias .run script. However, now we are seeing the following error when we run a project job:
[command: nvcc --cubin -arch sm_61 -I/home/cryospuser/cryosparc3/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/cuda kernel.cu]
[stderr:
cc1plus: fatal error: cuda_runtime.h: No such file or directory
compilation terminated.
We have verified that the path to nvcc (in our case, /usr/local/cuda/bin) and the path to cuda_runtime.h (/usr/local/cuda/include) are part of the $PATH variable for the cryospuser. We’ve made sure to reboot our system and refresh our cryospuser login. the /usr/local/cuda paths are seen by cryosparc, this was verified by running cyrosparcm env.
Remove CUDA completely. Remove any environment variables related to it, and install CUDA 11.8. From 4.2.* you need CUDA 11. Then run: cryosparcw newcuda /path/to/cuda from the CryoSPARC worker directory.
Having these files in your $PATH in the user’s initial environment is not needed and may in fact interfere with CryoSPARC. CryoSPARC commands like cryosparcw manage inclusion in the environment of the relevant files “on-the-fly”.