Cryosparw update version 4.1.1

Install, Configure and Update
1

Hi,

I am getting the following error while trying to update cryosparc worker for version 4.1.1:

error: command '/usr/bin/gcc' failed with exit code 1
      [end of output]

   This error originates from a subprocess, and is likely not a problem with pip.
error: legacy-install-failure

× Encountered error while trying to install package.
╰─> pycuda

note: This is an issue with the package mentioned above, not pip.
hint: See above for output from the failure.
check_install_deps.sh: 66: ERROR: installing python failed.

Did anyone come across this?

Thanks

2

Please can you run these commands on the host where the error occurred and post the output:

  1. uname -a
  2. /usr/bin/gcc -v
3
  1. uname -a

Linux spgpu 4.15.0-200-generic #211-Ubuntu SMP Thu Nov 24 18:16:04 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

  1. /usr/bin/gcc -v

Using built-in specs.
COLLECT_GCC=/usr/bin/gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/x86_64-linux-gnu/7/lto-wrapper
OFFLOAD_TARGET_NAMES=nvptx-none
OFFLOAD_TARGET_DEFAULT=1
Target: x86_64-linux-gnu
Configured with: …/src/configure -v --with-pkgversion=‘Ubuntu 7.5.0-3ubuntu1~18.04’ --with-bugurl=file:///usr/share/doc/gcc-7/README.Bugs --enable-languages=c,ada,c++,go,brig,d,fortran,objc,obj-c++ --prefix=/usr --with-gcc-major-version-only --program-suffix=-7 --program-prefix=x86_64-linux-gnu- --enable-shared --enable-linker-build-id --libexecdir=/usr/lib --without-included-gettext --enable-threads=posix --libdir=/usr/lib --enable-nls --enable-bootstrap --enable-clocale=gnu --enable-libstdcxx-debug --enable-libstdcxx-time=yes --with-default-libstdcxx-abi=new --enable-gnu-unique-object --disable-vtable-verify --enable-libmpx --enable-plugin --enable-default-pie --with-system-zlib --with-target-system-zlib --enable-objc-gc=auto --enable-multiarch --disable-werror --with-arch-32=i686 --with-abi=m64 --with-multilib-list=m32,m64,mx32 --enable-multilib --with-tune=generic --enable-offload-targets=nvptx-none --without-cuda-driver --enable-checking=release --build=x86_64-linux-gnu --host=x86_64-linux-gnu --target=x86_64-linux-gnu
Thread model: posix
gcc version 7.5.0 (Ubuntu 7.5.0-3ubuntu1~18.04)

4

Thanks. May I also ask for the output of
/usr/bin/env

5 
CONDA_SHLVL=0
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.zst=01;31:*.tzst=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.wim=01;31:*.swm=01;31:*.dwm=01;31:*.esd=01;31:*.jpg=01;35:*.jpeg=01;35:*.mjpg=01;35:*.mjpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
LD_LIBRARY_PATH=/usr/local/lib:/opt/lib64:/opt/IMOD/lib:/usr/local/lib:/opt/lib64:
CONDA_EXE=/data/softwares/anaconda3/bin/conda
SSH_CONNECTION=160.39.20.199 60815 128.59.144.55 22
LESSCLOSE=/usr/bin/lesspipe %s %s
LANG=en_US.UTF-8
DISPLAY=localhost:10.0
OLDPWD=/data/emsoft/cryosparcV2/cryosparc2_worker
IMOD_JAVADIR=/usr/local/java
SCHRODINGER=/data/softwares/schrodinger2019-3/
SSH_AUTH_SOCK=/tmp/ssh-eRnCfKbsjW/agent.5744
_CE_M=
XDG_SESSION_ID=70
MODULES_CMD=/usr/lib/x86_64-linux-gnu/modulecmd.tcl
USER=shikhas
ENV=/usr/share/modules/init/profile.sh
PWD=/data/emsoft/cryosparcV2
FOR_DISABLE_STACK_TRACE=1
HOME=/home/shikhas
CONDA_PYTHON_EXE=/data/softwares/anaconda3/bin/python
IMOD_CALIB_DIR=/usr/local/ImodCalib
SSH_CLIENT=160.39.20.199 60815 22
CUDA_HOME=/usr/local/cuda-10.0
BASH_ENV=/usr/share/modules/init/bash
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
_CE_CONDA=
LOADEDMODULES=
SSH_TTY=/dev/pts/1
MAIL=/var/mail/shikhas
IMOD_PLUGIN_DIR=/opt/IMOD/lib/imodplug
TERM=xterm
SHELL=/bin/bash
IMOD_QTLIBDIR=/opt/IMOD/qtlib
SHLVL=1
IMOD_DIR=/opt/IMOD
MODULEPATH=/opt/sp/modulefiles:/opt/sp/modulefiles:
LOGNAME=shikhas
DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/1002/bus
XDG_RUNTIME_DIR=/run/user/1002
PATH=/home/shikhas/.local/bin:/data/softwares/anaconda3/condabin:/opt/sp/bin:/opt/cistem-1.0.0-beta:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/opt/IMOD/bin:/opt/sp/bin:/opt/cistem-1.0.0-beta:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/opt/EMAN2/bin:/opt/EMAN2/bin
MODULESHOME=/usr/share/modules
LESSOPEN=| /usr/bin/lesspipe %s
BASH_FUNC_module%%=() {  _moduleraw "$*" 2>&1
}
BASH_FUNC_switchml%%=() {  typeset swfound=1;
 if [ "${MODULES_USE_COMPAT_VERSION:-0}" = '1' ]; then
 typeset swname='main';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd.tcl ]; then
 typeset swfound=0;
 unset MODULES_USE_COMPAT_VERSION;
 fi;
 else
 typeset swname='compatibility';
 if [ -e /usr/lib/x86_64-linux-gnu/modulecmd-compat ]; then
 typeset swfound=0;
 MODULES_USE_COMPAT_VERSION=1;
 export MODULES_USE_COMPAT_VERSION;
 fi;
 fi;
 if [ $swfound -eq 0 ]; then
 echo "Switching to Modules $swname version";
 source /usr/share/modules/init/bash;
 else
 echo "Cannot switch to Modules $swname version, command not found";
 return 1;
 fi
}
BASH_FUNC__moduleraw%%=() {  unset _mlre _mlIFS _mlshdbg;
 if [ "${MODULES_SILENT_SHELL_DEBUG:-0}" = '1' ]; then
 case "$-" in
 *v*x*)
 set +vx;
 _mlshdbg='vx'
 ;;
 *v*)
 set +v;
 _mlshdbg='v'
 ;;
 *x*)
 set +x;
 _mlshdbg='x'
 ;;
 *)
 _mlshdbg=''
 ;;
 esac;
 fi;
 if [ -n "${IFS+x}" ]; then
 _mlIFS=$IFS;
 fi;
 IFS=' ';
 for _mlv in ${MODULES_RUN_QUARANTINE:-};
 do
 if [ "${_mlv}" = "${_mlv##*[!A-Za-z0-9_]}" -a "${_mlv}" = "${_mlv#[0-9]}" ]; then
 if [ -n "`eval 'echo ${'$_mlv'+x}'`" ]; then
 _mlre="${_mlre:-}${_mlv}_modquar='`eval 'echo ${'$_mlv'}'`' ";
 fi;
 _mlrv="MODULES_RUNENV_${_mlv}";
 _mlre="${_mlre:-}${_mlv}='`eval 'echo ${'$_mlrv':-}'`' ";
 fi;
 done;
 if [ -n "${_mlre:-}" ]; then
 _mlre="eval ${_mlre}";
 fi;
 eval `${_mlre:-}/usr/bin/tclsh /usr/lib/x86_64-linux-gnu/modulecmd.tcl bash $*`;
 _mlstatus=$?;
 if [ -n "${_mlIFS+x}" ]; then
 IFS=$_mlIFS;
 else
 unset IFS;
 fi;
 if [ -n "${_mlshdbg:-}" ]; then
 set -$_mlshdbg;
 fi;
 unset _mlre _mlv _mlrv _mlIFS _mlshdbg;
 return $_mlstatus
}
_=/usr/bin/env
6

Thanks. If there is an nvidia driver installed, what is its version?
nvidia-smi

7

Yes, nvidia driver is installed.
Driver Version: 440.33.01

8

If your cryosparc_worker tree is still in the state where

and you have the necessary system privileges, you may want to try:

  1. update your nvidia-driver to version 460.32.03 or newer (guide)
  2. reboot the computer
  3. open a shell under the account that runs the CryoSPARC instance and
unset LD_LIBRARY_PATH
export PATH=/usr/bin:/bin:/usr/sbin:/sbin
/path/to/cryosparc_worker/bin/cryosparcw install-3dflex 2>&1 | tee 3dflex_deps.log

If successful, installation of 3DFlex dependencies should overcome the problem of an initially failed pycuda installation.

9

Hi,

After following the instructions above, 3d-flex ran to completion, it did install pycuda.

However, now when I run a test job, a 2D classification which had run successfully earlier, to check if cryosparc is running fine I get this error:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 80, in cryosparc_compute.run.main
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/jobs/jobregister.py”, line 443, in get_run_function
runmod = importlib.import_module("…"+modname, name)
File “/data/emsoft/cryosparcV2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 1174, in exec_module
File “”, line 219, in _call_with_frames_removed
File “cryosparc_master/cryosparc_compute/jobs/class2D/run.py”, line 13, in init cryosparc_compute.jobs.class2D.run
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 12, in init cryosparc_compute.engine.cuda_core
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/skcuda_internal/fft.py”, line 27, in
from . import misc2 as misc
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/skcuda_internal/misc2.py”, line 32, in
from . import cuda
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/skcuda_internal/cuda.py”, line 17, in
from .cudadrv import *
File “/data/emsoft/cryosparcV2/cryosparc2_worker/cryosparc_compute/skcuda_internal/cudadrv.py”, line 39, in
raise OSError(‘CUDA driver library not found’)
OSError: CUDA driver library not found

10

Please can you show us the outputs of

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call which nvcc
/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call nvcc --version
/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"
nvidia-smi

and the contents of 3dflex_deps.log (recorded during install-3dflex).

11

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call which nvcc

/usr/local/cuda/bin/nvcc

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Mon_Oct_24_19:12:58_PDT_2022
Cuda compilation tools, release 12.0, V12.0.76
Build cuda_12.0.r12.0/compiler.31968024_0

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”

(11, 7, 0)

nvidia-smi

NVIDIA-SMI 525.60.13 Driver Version: 525.60.13 CUDA Version: 12.0

12

Got past all the installation errors after downgrading from Cuda 12.0 to Cuda 11.7
(3DFlex Dependencies; Building pycuda - #14 by rbs_sci)

Thank you for all the help.

13

Interesting. Have you confirmed that GPU jobs, lile the 2D classification you mentioned earlier, now run?
Please can you list the sequence of operations (CUDA downgrade, CryoSPARC-related installation commands) that got you to this point, and also document the current state of your system by running again these commands:

Thanks.

14

Have you confirmed that GPU jobs, lile the 2D classification you mentioned earlier, now run?
Yes.
Please can you list the sequence of operations (CUDA downgrade, CryoSPARC-related installation commands) that got you to this point,

  1. sudo apt-get --purge remove “cublas” “cuda*”
  2. sudo reboot
    .
    .
    .
  3. sudo apt-get -y install cuda-11-7
  4. sudo reboot
  5. ##for fixing the problem with cryosparcw:
    #start cryosparcm
    #uninstall the 3DFlex dependencies and return the worker to its original state:
    bin/cryosparcw forcedeps
    ##job ran to completion
    ##installed 3D-flex
    ##job ran to completion with this warning:
    PyTorch not installed correctly, or NVIDIA GPU not detected.

and also document the current state of your system by running again these commands:
/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call which nvcc
/usr/local/cuda/bin/nvcc

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0

/data/emsoft/cryosparcV2/cryosparc2_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”
(11, 7, 0)

nvidia-smi
NVIDIA-SMI 515.43.04 Driver Version: 515.43.04 CUDA Version: 11.7

15

Thanks for this information.

This message, in the absence of other error messages, should be harmless.