--nogpu still tries to install pycuda on v4.1.2 worker

UCBKurt · April 6, 2023, 9:53pm

We’re trying to install the CryoSPARC worker package on a CPU-only system with the --nogpu option. However, the install always fails at the following:

 In file included from src/cpp/cuda.cpp:4:
      src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
         14 | #include <cuda.h>
            |          ^~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pycuda

We have no need of pycuda on this machine since it doesn’t have any GPUs. Adding a GPU is not feasible since it is a 1U server with minimal space available.

UCBKurt · April 11, 2023, 6:34pm

Not to push, but the issue still occurs. Is there a way to exclude pycuda from a worker install?

rbs_sci · April 12, 2023, 3:07am

I have CUDA installed on a system with no nVidia GPU (you don’t need a GPU to run nvcc). It’s failing because it can’t find a CUDA header, if you install the CUDA toolkit (not the drivers) will installation proceed?

UCBKurt · April 12, 2023, 5:10pm

Same error, even after installing the cuda toolkit. Plus, installing cuda shouldn’t be necessary for a non-gpu install anyways.

wtempel · April 12, 2023, 10:13pm

As far as I am aware, --nogpu is not a supported option for the cryosparc_worker/install.sh command.

In my test after a runfile-based installation of CUDA-11.8 on a non-gpu node, I did not encounter

UCBKurt:

In file included from src/cpp/cuda.cpp:4:
      src/cpp/cuda.hpp:14:10: fatal error: cuda.h: No such file or directory
         14 | #include <cuda.h>
            |          ^~~~~~~~
      compilation terminated.
      error: command '/usr/bin/gcc' failed with exit code 1

when I ran crysparc_worker/install.sh ...

UCBKurt · April 12, 2023, 10:51pm

I was running CUDA 11.7.1 (offline runfile) on Ubuntu 22.04 (which is what our GPU nodes use). Removing --nogpu gives the same error.

rbs_sci · April 13, 2023, 12:13am

The error occurring regardless indicates a deeper underlying issue.

What are your $PATH and $LD_LIBRARY_PATH environment variables? Output of which nvcc? nvcc --version?

marinegor · May 24, 2023, 8:27pm

hey, I believe that the issue can be with cuda12, according to this:

github.com/inducer/pycuda

CUDA 12 has removed enum values CU_TARGET_COMPUTE_20 and CU_TARGET_COMPUTE_21

opened 03:49PM - 20 Dec 22 UTC

closed 09:00PM - 20 Dec 22 UTC

mmohrhard

bug

Currently compiling pycuda against CUDA 12 fails as CUDA 12 has removed the CU_T…ARGET_COMPUTE_21 and CU_TARGET_COMPUTE_20 enums entries from CUjit_target_enum To reproduce compile pycuda against CUDA 12.0. Otherwise, it can also be seen by downloading the cudart tarball from https://developer.download.nvidia.com/compute/cuda/redist/cuda_cudart/linux-x86_64/cuda_cudart-linux-x86_64-12.0.107-archive.tar.xz and check the cuda.h file. Here is the corresponding part of the file: ``` 1342 typedef enum CUjit_target_enum 1343 { 1344 CU_TARGET_COMPUTE_30 = 30, /**< Compute device class 3.0 */ 1345 CU_TARGET_COMPUTE_32 = 32, /**< Compute device class 3.2 */ 1346 CU_TARGET_COMPUTE_35 = 35, /**< Compute device class 3.5 */ 1347 CU_TARGET_COMPUTE_37 = 37, /**< Compute device class 3.7 */ 1348 CU_TARGET_COMPUTE_50 = 50, /**< Compute device class 5.0 */ 1349 CU_TARGET_COMPUTE_52 = 52, /**< Compute device class 5.2 */ 1350 CU_TARGET_COMPUTE_53 = 53, /**< Compute device class 5.3 */ 1351 CU_TARGET_COMPUTE_60 = 60, /**< Compute device class 6.0.*/ 1352 CU_TARGET_COMPUTE_61 = 61, /**< Compute device class 6.1.*/ 1353 CU_TARGET_COMPUTE_62 = 62, /**< Compute device class 6.2.*/ 1354 CU_TARGET_COMPUTE_70 = 70, /**< Compute device class 7.0.*/ 1355 CU_TARGET_COMPUTE_72 = 72, /**< Compute device class 7.2.*/ 1356 CU_TARGET_COMPUTE_75 = 75, /**< Compute device class 7.5.*/ 1357 CU_TARGET_COMPUTE_80 = 80, /**< Compute device class 8.0.*/ 1358 CU_TARGET_COMPUTE_86 = 86, /**< Compute device class 8.6.*/ 1359 CU_TARGET_COMPUTE_87 = 87, /**< Compute device class 8.7.*/ 1360 CU_TARGET_COMPUTE_89 = 89, /**< Compute device class 8.9.*/ 1361 CU_TARGET_COMPUTE_90 = 90, /**< Compute device class 9.0.*/ 1362 1363 /**< Compute device class 9.0. with accelerated features.*/ 1364 CU_TARGET_COMPUTE_90A = CU_COMPUTE_ACCELERATED_TARGET_BASE + CU_TARGET_COMPUTE_90 1365 } CUjit_target; ``` And the corresponding compile error: ``` src/wrapper/wrap_cudadrv.cpp:1034:26: error: ‘CU_TARGET_COMPUTE_20’ was not declared in this scope; did you mean ‘CU_TARGET_COMPUTE_90’? 1034 | .value("COMPUTE_20", CU_TARGET_COMPUTE_20) | ^~~~~~~~~~~~~~~~~~~~ | CU_TARGET_COMPUTE_90 src/wrapper/wrap_cudadrv.cpp:1037:26: error: ‘CU_TARGET_COMPUTE_21’ was not declared in this scope; did you mean ‘CU_TARGET_COMPUTE_61’? 1037 | .value("COMPUTE_21", CU_TARGET_COMPUTE_21) | ^~~~~~~~~~~~~~~~~~~~ | CU_TARGET_COMPUTE_61 ``` **Expected behavior** pycuda should compile against CUDA 12 **Environment (please complete the following information):** - OS: Linux - CUDA: 12 - pycuda: 2022.2

I tried installing cryosparc v4.1.2 and v4.2.1 on cuda12.1 and failed while installing pycuda with the same error as you. But when I discovered this issue and used cuda11.8, everything worked fine!