Error trying to exec 'cc1plus': execvp: No such file or directory\nnvcc fatal

Install, Configure and Update
1

Hi We have this error I tried to fix for few hours without success

2
21723×806 44.1 KB 



**$ cryosparcm status**
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryosparc/cryosparc_master
Current cryoSPARC version: v4.3.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 894421, uptime 0:27:15
app_api                          RUNNING   pid 894439, uptime 0:27:14
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 894141, uptime 0:27:28
command_rtp                      RUNNING   pid 894361, uptime 0:27:20
command_vis                      RUNNING   pid 894177, uptime 0:27:21
database                         RUNNING   pid 894040, uptime 0:27:31

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="xxxx"
export CRYOSPARC_MASTER_HOSTNAME="b01"
export CRYOSPARC_DB_PATH="/home/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true
export CRYOSPARC_FORCE_HOSTNAME=true

**$  uname -a && free -g**
Linux b01 5.15.0-70-generic #77-Ubuntu SMP Tue Mar 21 14:02:37 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
               total        used        free      shared  buff/cache   available
Mem:             754           9           7           0         737         740
Swap:              9           0           9

**$ env | grep PATH**
CRYOSPARC_PATH=/home/cryosparc/cryosparc_worker/bin
PYTHONPATH=/home/cryosparc/cryosparc_worker
CRYOSPARC_CUDA_PATH=/usr/local/cuda
LD_LIBRARY_PATH=/usr/local/cuda/lib64:/home/cryosparc/cryosparc_worker/deps/external/cudnn/lib
PATH=/usr/local/cuda/bin:/home/cryosparc/cryosparc_worker/bin:/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/cryosparc/cryosparc_worker/deps/anaconda/condabin:/usr/local/cuda:/home/cryosparc/cryosparc_master/bin:/progs/MC2/MotionCor2_1.4.7:/progs/anaconda3/bin:/progs/MC2/MotionCor2_1.4.7:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

**$ which nvcc**
/usr/local/cuda/bin/nvcc

**$ nvcc --version**
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

**$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"**
(11, 8, 0)

**$ /sbin/ldconfig -p | grep -i cuda**
        libnvrtc.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc.so.11.2
        libnvrtc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc.so
        libnvrtc-builtins.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc-builtins.so.11.8
        libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvrtc-builtins.so
        libnvjpeg.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvjpeg.so.11
        libnvjpeg.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvjpeg.so
        libnvblas.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvblas.so.11
        libnvblas.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvblas.so
        libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvToolsExt.so.1
        libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnvToolsExt.so
        libnpps.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnpps.so.11
        libnpps.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnpps.so
        libnppitc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppitc.so.11
        libnppitc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppitc.so
        libnppisu.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppisu.so.11
        libnppisu.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppisu.so
        libnppist.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppist.so.11
        libnppist.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppist.so
        libnppim.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppim.so.11
        libnppim.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppim.so
        libnppig.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppig.so.11
        libnppig.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppig.so
        libnppif.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppif.so.11
        libnppif.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppif.so
        libnppidei.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppidei.so.11
        libnppidei.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppidei.so
        libnppicc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppicc.so.11
        libnppicc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppicc.so
        libnppial.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppial.so.11
        libnppial.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppial.so
        libnppc.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppc.so.11
        libnppc.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libnppc.so
        libicudata.so.70 (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so.70
        libcusparse.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusparse.so.11
        libcusparse.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusparse.so
        libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolverMg.so.11
        libcusolverMg.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolverMg.so
        libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolver.so.11
        libcusolver.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcusolver.so
        libcurand.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcurand.so.10
        libcurand.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcurand.so
        libcuinj64.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcuinj64.so.11.8
        libcuinj64.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcuinj64.so
        libcufile_rdma.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile_rdma.so.1
        libcufile_rdma.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile_rdma.so
        libcufile.so.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile.so.0
        libcufile.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufile.so
        libcufftw.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufftw.so.10
        libcufftw.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufftw.so
        libcufft.so.10 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufft.so.10
        libcufft.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcufft.so
        libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so.11.0
        libcudart.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcudart.so
        libcudadebugger.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcudadebugger.so.1
        libcuda.so.1 (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /lib/x86_64-linux-gnu/libcuda.so
        libcublasLt.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so.11
        libcublasLt.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublasLt.so
        libcublas.so.11 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublas.so.11
        libcublas.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libcublas.so
        libaccinj64.so.11.8 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libaccinj64.so.11.8
        libaccinj64.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libaccinj64.so
        libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libOpenCL.so.1
        libOpenCL.so (libc6,x86-64) => /usr/local/cuda-11.8/targets/x86_64-linux/lib/libOpenCL.so

**$ nvidia-smi**
Thu Sep 14 16:10:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 520.61.05    Driver Version: 520.61.05    CUDA Version: 11.8     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:1B:00.0 Off |                  N/A |
|  0%   37C    P0   119W / 370W |      0MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:3E:00.0 Off |                  N/A |
|  0%   35C    P0   115W / 370W |      0MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:B2:00.0 Off |                  N/A |
|  0%   37C    P0   123W / 370W |      0MiB / 24576MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:DB:00.0 Off |                  N/A |
|  0%   35C    P0   108W / 370W |      0MiB / 24576MiB |      4%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

Thank you for your help!

2

Welcome to the forum @cfrodriguez.

Please can you post the text of the Traceback also, to make it easier for interested forum visitors to find this topic.

Please can you provide some additional background information:

  1. Is this a new installation of CryoSPARC, or an installation that has functioned normally before?
  2. Is this a combined, single-host master/worker installation?
  3. What is the output of thes commands on the worker:
    /home/cryosparc/cryosparc_worker/bin/cryosparcw call which g++
/home/cryosparc/cryosparc_worker/bin/cryosparcw call g++ --version
3

Hi @wtempel, thank you for your answer.

  1. This is an installation that used to work few months ago, maybe an upgrade broke it.
  2. I dont know how to answer this question, it is everything in the same machine, it is not a cluster. So probably the answer is yes?
$ /home/cryosparc/cryosparc_worker/bin/cryosparcw call which g++
/usr/bin/g++

$ /home/cryosparc/cryosparc_worker/bin/cryosparcw call g++ --version
g++ (Ubuntu 11.4.0-1ubuntu1~22.04) 11.4.0
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

Carlos

4

@cfrodriguez Please can you try:

  1. update the nvidia-driver to version 535 (but leave the CUDA toolkit version at 11.8)
  2. reboot the computer
  3. run the command
    /home/cryosparc/cryosparc_worker/bin/cryosparcw forcedeps
  4. if errors occur when running the command, please post them. Otherwise
  5. re-run the GPU-accelerated job that failed earlier
  6. if the job still fails, please post the output of
    nvidia-smi --query-gpu=name,driver_version --format=csv
/home/cryosparc/cryosparc_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"