Should installation require nvcc in path?

Install, Configure and Update
1

I recently installed cryoSPARC on a new machine in standalone mode. Installation failed unless I had nvcc in the PATH. This is not something I have had to do in the past. Is this intended? It’s not mentioned in the installation guide as far as I can see.

The machine was running Ubuntu 22.04.1 LTS, CUDA 11.8, nvidia driver version 520.61.05.

The installation error was 9549 lines long, but ended with

      gcc -pthread -B /local_slow/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/compiler_compat -Wno-unused-result -Wsign-compare -fwrapv -Wall -O3 -DNDEBUG -fPIC -DBOOST_ALL_NO_LIB=1 -DBOOST_THREAD_BUILD_DLL=1 -DBOOST_MULTI_INDEX_DISABLE_SERIALIZATION=1 -DBOOST_PYTHON_SOURCE=1 -Dboost=pycudaboost -DBOOST_THREAD_DONT_USE_CHRONO=1 -DPYGPU_PACKAGE=pycuda -DPYGPU_PYCUDA=1 -DHAVE_CURAND=1 -Isrc/cpp -Ibpl-subset/bpl_subset -I/local_slow/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/include -I/local_slow/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include/python3.8 -c src/cpp/cuda.cpp -o build/temp.linux-x86_64-cpython-38/src/cpp/cuda.o
      In file included from src/cpp/cuda.cpp:4:
      src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
         14 | #include <cuda.h>
            |          ^~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pycuda

uname -a output

Linux ljouwert 5.15.0-58-generic #64-Ubuntu SMP Thu Jan 5 11:43:13 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux

gcc -V

Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/11/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none:amdgcn-amdhsa
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: ../src/configure -v --with-pkgversion='Ubuntu 11.3.0-1ubuntu1~22.04' --with-bugurl=file:///usr/share/doc/gcc-11/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++,m2 --prefix=/usr --with-gcc-major-version-only --program-suffix=-11 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-plugin --enable-default-pie --with-system-zlib --enable-libphobos-checking=release --with-target-system-zlib=auto --enable-objc-gc=auto --enable-multiarch --disable-werror --enable-cet --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-nvptx/usr,amdgcn-amdhsa=/build/gcc-11-xKiWfi/gcc-11-11.3.0/debian/tmp-gcn/usr --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu --with-build-config=bootstrap-lto-lean --enable-link-serialization=2
Thread model: posix
Supported LTO compression algorithms: zlib zstd
gcc version 11.3.0 (Ubuntu 11.3.0-1ubuntu1~22.04)
2

It would be interesting to see:

  • output of
    env | grep PATH before you run installation
  • the full installation command

corresponding to this error.

3

Results from env:

MANPATH=:/opt/puppetlabs/puppet/share/man
PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/puppetlabs/bin

Installation command (partly redacted)

cd cryosparc_master
export LICENSE_ID="..." 
./install.sh --standalone --license $LICENSE_ID --worker_path /local_slow/cryosparc/cryosparc_worker --cudapath /usr/local/cuda-11.8 --ssdpath /local_fast/cryosparc --initial_email ... --initial_username ... --initial_firstname ... --initial_lastname ...
4

We received reports about, and confirmed, performance issues (unrelated to the installation problem) on CryoSPARC v4.1.2 instances that were configured with CUDA v11.8. We are working on a fix.
One possible workaround is configuring CryoSPARC with CUDA-11.7, which is automatically installed during
cryosparcw install-3dflex (for prerequisites, see Installing 3DFlex Dependencies - CryoSPARC Guide)
If instead you prefer to configure the CUDA toolkit separately, please read on, but keep in mind that the following suggestion may become obsolete with a future CryoSPARC software release.

I assume this error occurred during the (implicit for --standalone installation) call to cryosparc_worker/bin/install.sh, and that the preceding installation of the master package completed successfully.
We do not generally recommend including nvcc in the path because the presence of nvcc in the path may interfere with certain installations (for example for `cryosparcw install-3dflex, mentioned above). Unfortunately, under some circumstances, installation may fail unless certain CUDA-related environment variables are defined.
Under those circumstances, you may try setting for worker installation only:

export CUDA_PATH="/real/path/to/cuda"
export C_INCLUDE_PATH="$CUDA_PATH/include"
export CPLUS_INCLUDE_PATH="$C_INCLUDE_PATH"
export PATH=$CUDA_PATH/bin:$PATH

These definitions are for installation only, but should not be included in shell startup files.
If setting these variables results in a successful package installation, one should follow up with
cryosparc_worker/bin/cryosparcw connect ... (see guide) .

5

Hi @wtempel Do you have a recommendation for what to change when ‘which nvcc’ does return a path to nvcc (in my case, /usr/bin/nvcc)? I’m trying to follow the instructions about pre-reqs/dependencies for 3DFlex but I’m reluctant to break anything about my current setup.

When I originally installed CryoSPARC I remember getting an error about not finding nvcc in the CUDA path, and I fixed that with a softlink: ln -s /usr/bin/nvcc /usr/lib/cuda/bin/nvcc. This was back in version 2.15.

6

If this is the sole reason for the existence of /usr/bin/nvcc and the v2.15 installation is n longer needed, I suggest you remove the link (and any other links you may have created in /usr/bin or /usr/lib at that time for the same reason). Other mechanisms are in place for the current version of CryoSPARC to “find” components of the CUDA toolkit.

7

Pardon the long delay. I was able to clean out links and not break anything for CryoSPARC or other software. 3DFlex installs and runs now. Thanks!

1 Like