2d classification kernel error

sheff_diamond_em · May 2, 2023, 11:33am

Hi all,

Trying to run 2d classification on some particles (all preprocessing done in cryosparc, particles picked by deep train), and I keep getting this error:

Traceback (most recent call last):
  File "/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py", line 2061, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1082, in cryosparc_compute.engine.engine.process.work
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 320, in cryosparc_compute.engine.engine.EngineThread.compute_resid_pow
  File "cryosparc_master/cryosparc_compute/engine/cuda_kernels.py", line 1507, in cryosparc_compute.engine.cuda_kernels.compute_resid_pow
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 414, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
  File "cryosparc_master/cryosparc_compute/engine/cuda_kernels.py", line 1389, in cryosparc_compute.engine.cuda_kernels.get_compute_resid_pow_kernel
  File "/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py", line 294, in __init__
    self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error   : Binary format for key='0', ident='' is not recognized

Any ideas on how to address this?

Many thanks in advance,

Fred

wtempel · May 2, 2023, 2:10pm

Please can you post:

What hardware/software changes have occurred on the worker since the last successful run of a similar job.
What is the CryoSPARC version?
Details of the CryoSPARC worker environment

Output of the commands

/opt/cryosparc2/cryosparc2_worker/bin/cryosparcw call /usr/bin/env | grep -v LICENSE
cat /etc/ld.so.conf
ls /etc/ld.so.conf.d/
cat /etc/ld.so.conf.d/*.conf

sheff_diamond_em · May 2, 2023, 3:20pm

Hi,

Thanks for the response,

1.) I haven’t run 2d classification on this worker before; in the past I have imported particle stacks processed elsewhere for 3dva/3dflex etc, so this is the first time running this job type

2.) The version is 4.2.1, I updated it last week

3.) Outputs to the worker environment detail commands (command in bold, output in block quote):

env | grep PATH

LD_LIBRARY_PATH=/opt/cryosparc2/cryosparc2_worker/deps/external/cudnn/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64:/opt/IMOD/lib:/opt/relion-3.1-b/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64
PATH=/opt/cryosparc2/cryosparc2_worker/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/home/spuser/bin:/home/spuser/.local/bin:/opt/cryosparc2/cryosparc2_master/bin:/data/cryosparc_user/software/cryosparc/cryosparc2_master/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/opt/IMOD/bin:/opt/relion-3.1-b/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda-8.0/bin:/snap/bin:/home/spuser/.dotnet/tools:/usr/local/cuda-8.0/bin
MODULEPATH=/opt/sp/modulefiles:/opt/sp/modulefiles:/etc/environment-modules/modules:/usr/share/modules/versions:/usr/Modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
CRYOSPARC_PATH=/opt/cryosparc2/cryosparc2_worker/bin
PYTHONPATH=/opt/cryosparc2/cryosparc2_worker
CRYOSPARC_CUDA_PATH=/usr/local/cuda-10.1

which nvcc

/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/nvcc

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Jun__8_16:49:14_PDT_2022
Cuda compilation tools, release 11.7, V11.7.99
Build cuda_11.7.r11.7/compiler.31442593_0

python -c “import pycuda.driver; print(pycuda.driver.get_version())”

(11, 7, 0)

uname -a

Linux spgpu3 4.15.0-142-generic #146~16.04.1-Ubuntu SMP Tue Apr 13 09:27:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

free -g

              total        used        free      shared  buff/cache   available
Mem:            251         128         107           0          15         118
Swap:           255           0         255

nvidia-smi

Tue May  2 17:14:59 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.63.01    Driver Version: 470.63.01    CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:04:00.0 Off |                  N/A |
| 23%   32C    P8     7W / 250W |     21MiB / 11176MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 23%   29C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:08:00.0 Off |                  N/A |
| 23%   22C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 23%   27C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  Off  | 00000000:83:00.0 Off |                  N/A |
| 23%   26C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  Off  | 00000000:84:00.0 Off |                  N/A |
| 23%   30C    P8     8W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  Off  | 00000000:87:00.0 Off |                  N/A |
| 23%   28C    P8     7W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  Off  | 00000000:88:00.0 Off |                  N/A |
| 23%   34C    P8     7W / 250W |      2MiB / 11178MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      1834      G   /usr/lib/xorg/Xorg                 18MiB |
+-----------------------------------------------------------------------------+

4.) Output to the specified commands (command in bold, output in block quote):

/opt/cryosparc2/cryosparc2_worker/bin/cryosparcw call /usr/bin/env | grep -v LICENSE

LC_PAPER=de_DE.UTF-8
MODULE_VERSION_STACK=3.2.10
RELION_ALLOW_CHANGE_MINIMUM_DEDICATED=0
CRYOSPARC_USE_GPU=true
XDG_SESSION_ID=446
LC_ADDRESS=de_DE.UTF-8
LC_MONETARY=de_DE.UTF-8
IMOD_DIR=/opt/IMOD
SHELL=/bin/bash
TERM=xterm
IMOD_QTLIBDIR=/opt/IMOD/qtlib
PYTHONNOUSERSITE=true
SSH_CLIENT= # redacted #
CONDA_SHLVL=1
RELION_WARNING_LOCAL_MPI=12
CONDA_PROMPT_MODIFIER=(cryosparc_worker_env)
CRYOSPARC_CONDA_ENV=cryosparc_worker_env
LD_PRELOAD=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libpython3.8.so
LC_NUMERIC=de_DE.UTF-8
RELION_MOTIONCOR2_EXECUTABLE=/opt/motioncor2-bin-2.0.0/bin/MotionCor2
OLDPWD=/opt/cryosparc2/cryosparc2_worker
SSH_TTY=/dev/pts/8
RELION_PDFVIEWER_EXECUTABLE=evince
CRYOSPARC_DEVELOP=false
USER=spuser
CUDA_HOME=/usr/local/cuda
LC_TELEPHONE=de_DE.UTF-8
LD_LIBRARY_PATH=/opt/cryosparc2/cryosparc2_worker/deps/external/cudnn/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64:/opt/IMOD/lib:/opt/relion-3.1-b/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
CONDA_EXE=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/bin/conda
RELION_UNBLUR_EXECUTABLE=/opt/bin/unblur_openmp_7_17_15.exe
RELION_MINIMUM_DEDICATED=32
_CE_CONDA=
PATH=/opt/cryosparc2/cryosparc2_worker/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/home/spuser/bin:/home/spuser/.local/bin:/opt/cryosparc2/cryosparc2_master/bin:/data/cryosparc_user/software/cryosparc/cryosparc2_master/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/opt/IMOD/bin:/opt/relion-3.1-b/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda-8.0/bin:/snap/bin:/home/spuser/.dotnet/tools:/usr/local/cuda-8.0/bin
MAIL=/var/mail/spuser
MODULE_VERSION=3.2.10
MKL_NUM_THREADS=1
QT_QPA_PLATFORMTHEME=appmenu-qt5
IMOD_PLUGIN_DIR=/opt/IMOD/lib/imodplug
CONDA_PREFIX=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env
LC_IDENTIFICATION=de_DE.UTF-8
PWD=/opt/cryosparc2/cryosparc2_worker
CRYOSPARC_ROOT_DIR=/opt/cryosparc2/cryosparc2_worker
NUMEXPR_NUM_THREADS=1
_LMFILES_=/opt/sp/modulefiles/relion/3.1-b
RELION_RESMAP_EXECUTABLE=/opt/bin/ResMap
IMOD_JAVADIR=/usr/local/java
LANG=en_US.UTF-8
MODULEPATH=/opt/sp/modulefiles:/opt/sp/modulefiles:/etc/environment-modules/modules:/usr/share/modules/versions:/usr/Modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
LC_MEASUREMENT=de_DE.UTF-8
LOADEDMODULES=relion/3.1-b
FOR_DISABLE_STACK_TRACE=1
_CE_M=
RELION_CTFFIND_EXECUTABLE=/opt/ctffind-bin-4.1.5/bin/ctffind
HOME=/home/spuser
SHLVL=1
CRYOSPARC_PATH=/opt/cryosparc2/cryosparc2_worker/bin
RELION_SUMMOVIE_EXECUTABLE=/opt/bin/sum_movie_openmp_7_17_15.exe
PYTHONPATH=/opt/cryosparc2/cryosparc2_worker
CONDA_PYTHON_EXE=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/bin/python
LOGNAME=spuser
SSH_CONNECTION= # redacted #
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
MODULESHOME=/usr/share/modules
OMP_NUM_THREADS=1
CONDA_DEFAULT_ENV=cryosparc_worker_env
RELION_GCTF_EXECUTABLE=/opt/bin/Gctf
IMOD_CALIB_DIR=/usr/local/ImodCalib
LESSOPEN=| /usr/bin/lesspipe %s
RELION_PDFVIEWER_EXECUTABLEe=evince
DISPLAY=localhost:10.0
XDG_RUNTIME_DIR=/run/user/1001
LC_TIME=de_DE.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
CRYOSPARC_CUDA_PATH=/usr/local/cuda-10.1
LC_NAME=de_DE.UTF-8
BASH_FUNC_module%%=() {  eval `/usr/bin/modulecmd bash $*`
}

cat /etc/ld.so.conf

include /etc/ld.so.conf.d/*.conf

ls /etc/ld.so.conf.d/

cuda-10-1.conf cuda-11-2.conf fakeroot-x86_64-linux-gnu.conf i386-linux-gnu.conf libc.conf x86_64-linux-gnu.conf x86_64-linux-gnu_EGL.conf x86_64-linux-gnu_GL.conf

cat /etc/ld.so.conf.d/*.conf

/usr/local/cuda-10.1/targets/x86_64-linux/lib
/usr/local/cuda-11.2/targets/x86_64-linux/lib
/usr/lib/x86_64-linux-gnu/libfakeroot
# Multiarch support
/lib/i386-linux-gnu
/usr/lib/i386-linux-gnu
/lib/i686-linux-gnu
/usr/lib/i686-linux-gnu
# libc default configuration
/usr/local/lib
# Multiarch support
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/mesa-egl
/usr/lib/x86_64-linux-gnu/mesa

I hope that provides the information needed, if not please let me know what else would be helpful.

Thanks again!

Fred

wtempel · May 2, 2023, 6:18pm

Hi Fred,

Does this mean that you previously ran 3DVA and 3D Flex Train jobs on this worker? Before and/or after the update to v4.2.1?

As you seemingly have not updated 3D Flex dependencies (and, therefore, the CUDA toolkit) since the update to CryoSPARC v4.2.1, please can you try:

updating the nvidia driver to v520 or higher, then
cryosparcw forcedeps, then
cryosparcw install-3dflex
re-running the 2D classification

and report the outcome here.

sheff_diamond_em · May 4, 2023, 8:55am

Hi,

Thanks for the followup,

The nvidia driver has been upgraded and I followed your steps, the installation seemed to work but then I get the same error again when I try running the job:

Traceback (most recent call last):
File “/opt/cryosparc2/cryosparc2_worker/cryosparc_compute/jobs/runcommon.py”, line 2061, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1082, in cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 320, in cryosparc_compute.engine.engine.EngineThread.compute_resid_pow
File “cryosparc_master/cryosparc_compute/engine/cuda_kernels.py”, line 1507, in cryosparc_compute.engine.cuda_kernels.compute_resid_pow
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 414, in cryosparc_compute.engine.cuda_core.context_dependent_memoize.wrapper
File “cryosparc_master/cryosparc_compute/engine/cuda_kernels.py”, line 1389, in cryosparc_compute.engine.cuda_kernels.get_compute_resid_pow_kernel
File “/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pycuda/compiler.py”, line 294, in init
self.module = module_from_buffer(cubin)
pycuda._driver.LogicError: cuModuleLoadDataEx failed: device kernel image is invalid - error : Binary format for key=‘0’, ident=‘’ is not recognized

Best wishes,

Fred

sheff_diamond_em · May 4, 2023, 9:32am

Update: following a cuda update it is now running, thanks for the help!

Best wishes,

Fred

wtempel · May 4, 2023, 1:52pm

Hooray!
@sheff_diamond_em Would you mind repeating steps 3. and 4. from 2d classification kernel error - #2 by wtempel
This information may help us and other users when they encounter this error.
Thanks in advance.

sheff_diamond_em · May 5, 2023, 12:24pm

Yes of course, no problem:

Step 3:

env | grep PATH

LD_LIBRARY_PATH=/opt/cryosparc2/cryosparc2_worker/deps/external/cudnn/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64:/opt/IMOD/lib:/opt/relion-3.1-b/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64
PATH=/opt/cryosparc2/cryosparc2_worker/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/home/spuser/bin:/home/spuser/.local/bin:/opt/cryosparc2/cryosparc2_master/bin:/data/cryosparc_user/software/cryosparc/cryosparc2_master/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/opt/IMOD/bin:/opt/relion-3.1-b/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda-8.0/bin:/snap/bin:/home/spuser/.dotnet/tools:/usr/local/cuda-8.0/bin
MODULEPATH=/opt/sp/modulefiles:/opt/sp/modulefiles:/etc/environment-modules/modules:/usr/share/modules/versions:/usr/Modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
CRYOSPARC_PATH=/opt/cryosparc2/cryosparc2_worker/bin
PYTHONPATH=/opt/cryosparc2/cryosparc2_worker
CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.5

which nvcc

/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin/nvcc

nvcc --version

nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Wed_Sep_21_10:33:58_PDT_2022
Cuda compilation tools, release 11.8, V11.8.89
Build cuda_11.8.r11.8/compiler.31833905_0

/opt/cryosparc2/cryosparc2_worker/bin$ python -c “import pycuda.driver; print(pycuda.driver.get_version())”

(11, 8, 0)

/opt/cryosparc2/cryosparc2_worker/bin$ /sbin/ldconfig -p | grep -i cuda

        libnvrtc.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvrtc.so.11.2
        libnvrtc.so.10.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvrtc.so.10.1
        libnvrtc.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvrtc.so
        libnvrtc.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvrtc.so
        libnvrtc-builtins.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvrtc-builtins.so.11.2
        libnvrtc-builtins.so.10.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvrtc-builtins.so.10.1
        libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvrtc-builtins.so
        libnvrtc-builtins.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvrtc-builtins.so
        libnvjpeg.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvjpeg.so.11
        libnvjpeg.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvjpeg.so.10
        libnvjpeg.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvjpeg.so
        libnvjpeg.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvjpeg.so
        libnvgraph.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvgraph.so.10
        libnvgraph.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvgraph.so
        libnvblas.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvblas.so.11
        libnvblas.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvblas.so
        libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvToolsExt.so.1
        libnvToolsExt.so.1 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvToolsExt.so.1
        libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnvToolsExt.so
        libnvToolsExt.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnvToolsExt.so
        libnpps.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnpps.so.11
        libnpps.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnpps.so.10
        libnpps.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnpps.so
        libnpps.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnpps.so
        libnppitc.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppitc.so.11
        libnppitc.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppitc.so.10
        libnppitc.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppitc.so
        libnppitc.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppitc.so
        libnppisu.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppisu.so.11
        libnppisu.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppisu.so.10
        libnppisu.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppisu.so
        libnppisu.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppisu.so
        libnppist.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppist.so.11
        libnppist.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppist.so.10
        libnppist.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppist.so
        libnppist.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppist.so
        libnppim.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppim.so.11
        libnppim.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppim.so.10
        libnppim.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppim.so
        libnppim.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppim.so
        libnppig.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppig.so.11
        libnppig.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppig.so.10
        libnppig.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppig.so
        libnppig.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppig.so
        libnppif.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppif.so.11
        libnppif.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppif.so.10
        libnppif.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppif.so
        libnppif.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppif.so
        libnppidei.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppidei.so.11
        libnppidei.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppidei.so.10
        libnppidei.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppidei.so
        libnppidei.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppidei.so
        libnppicom.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppicom.so.10
        libnppicom.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppicom.so
        libnppicc.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppicc.so.11
        libnppicc.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppicc.so.10
        libnppicc.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppicc.so
        libnppicc.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppicc.so
        libnppial.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppial.so.11
        libnppial.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppial.so.10
        libnppial.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppial.so
        libnppial.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppial.so
        libnppc.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppc.so.11
        libnppc.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppc.so.10
        libnppc.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libnppc.so
        libnppc.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libnppc.so
        libicudata.so.55 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libicudata.so.55
        libicudata.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libicudata.so
        libcusparse.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusparse.so.11
        libcusparse.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcusparse.so.10
        libcusparse.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcusparse.so
        libcusparse.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusparse.so
        libcusolverMg.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusolverMg.so.11
        libcusolverMg.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusolverMg.so
        libcusolver.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusolver.so.11
        libcusolver.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcusolver.so.10
        libcusolver.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcusolver.so
        libcusolver.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcusolver.so
        libcurand.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcurand.so.10
        libcurand.so.10 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcurand.so.10
        libcurand.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcurand.so
        libcurand.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcurand.so
        libcuinj64.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcuinj64.so.11.2
        libcuinj64.so.10.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcuinj64.so.10.1
        libcuinj64.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcuinj64.so
        libcuinj64.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcuinj64.so
        libcufftw.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcufftw.so.10
        libcufftw.so.10 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcufftw.so.10
        libcufftw.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcufftw.so
        libcufftw.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcufftw.so
        libcufft.so.10 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcufft.so.10
        libcufft.so.10 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcufft.so.10
        libcufft.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcufft.so
        libcufft.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcufft.so
        libcudart.so.11.0 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudart.so.11.0
        libcudart.so.10.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so.10.1
        libcudart.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libcudart.so
        libcudart.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcudart.so
        libcudadebugger.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcudadebugger.so.1
        libcuda.so.1 (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so.1
        libcuda.so.1 (libc6) => /usr/lib/i386-linux-gnu/libcuda.so.1
        libcuda.so (libc6,x86-64) => /usr/lib/x86_64-linux-gnu/libcuda.so
        libcuda.so (libc6) => /usr/lib/i386-linux-gnu/libcuda.so
        libcublasLt.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublasLt.so.11
        libcublasLt.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublasLt.so
        libcublas.so.11 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublas.so.11
        libcublas.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libcublas.so
        libaccinj64.so.11.2 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libaccinj64.so.11.2
        libaccinj64.so.10.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libaccinj64.so.10.1
        libaccinj64.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libaccinj64.so
        libaccinj64.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libaccinj64.so
        libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libOpenCL.so.1
        libOpenCL.so.1 (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libOpenCL.so.1
        libOpenCL.so (libc6,x86-64) => /usr/local/cuda-10.1/targets/x86_64-linux/lib/libOpenCL.so
        libOpenCL.so (libc6,x86-64) => /usr/local/cuda-11.2/targets/x86_64-linux/lib/libOpenCL.so

uname -a

Linux spgpu3 4.15.0-142-generic #146~16.04.1-Ubuntu SMP Tue Apr 13 09:27:15 UTC 2021 x86_64 x86_64 x86_64 GNU/Linux

free -g

total        used        free      shared  buff/cache   available
Mem:            251         125          85           0          40         121

nvidia-smi

Fri May  5 14:19:04 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 525.105.17   Driver Version: 525.105.17   CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:04:00.0 Off |                  N/A |
| 21%   39C    P0    58W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:05:00.0 Off |                  N/A |
| 17%   35C    P0    58W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:08:00.0 Off |                  N/A |
| 17%   29C    P0    57W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:09:00.0 Off |                  N/A |
| 17%   36C    P0    60W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  Off  | 00000000:83:00.0 Off |                  N/A |
| 16%   33C    P0    58W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  Off  | 00000000:84:00.0 Off |                  N/A |
| 17%   36C    P0    59W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  Off  | 00000000:87:00.0 Off |                  N/A |
| 16%   33C    P0    56W / 250W |      0MiB / 11264MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  Off  | 00000000:88:00.0 Off |                  N/A |
| 20%   39C    P0    57W / 250W |      0MiB / 11264MiB |      1%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

And step 4:

opt/cryosparc2/cryosparc2_worker/bin/cryosparcw call /usr/bin/env | grep -v LICENSE

LC_PAPER=de_DE.UTF-8
MODULE_VERSION_STACK=3.2.10
RELION_ALLOW_CHANGE_MINIMUM_DEDICATED=0
CRYOSPARC_USE_GPU=true
XDG_SESSION_ID=532
LC_ADDRESS=de_DE.UTF-8
LC_MONETARY=de_DE.UTF-8
IMOD_DIR=/opt/IMOD
SHELL=/bin/bash
TERM=xterm
IMOD_QTLIBDIR=/opt/IMOD/qtlib
PYTHONNOUSERSITE=true
SSH_CLIENT= # redacted #
CONDA_SHLVL=1
RELION_WARNING_LOCAL_MPI=12
CONDA_PROMPT_MODIFIER=(cryosparc_worker_env)
CRYOSPARC_CONDA_ENV=cryosparc_worker_env
LD_PRELOAD=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libpython3.8.so
LC_NUMERIC=de_DE.UTF-8
RELION_MOTIONCOR2_EXECUTABLE=/opt/motioncor2-bin-2.0.0/bin/MotionCor2
OLDPWD=/opt/cryosparc2/cryosparc2_worker
SSH_TTY=/dev/pts/0
RELION_PDFVIEWER_EXECUTABLE=evince
CRYOSPARC_DEVELOP=false
USER=spuser
CUDA_HOME=/usr/local/cuda
LC_TELEPHONE=de_DE.UTF-8
LD_LIBRARY_PATH=/opt/cryosparc2/cryosparc2_worker/deps/external/cudnn/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64:/opt/IMOD/lib:/opt/relion-3.1-b/lib:/usr/local/lib:/opt/lib64:/usr/local/cuda/lib64:/usr/local/cuda-7.5/lib64
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
CONDA_EXE=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/bin/conda
RELION_UNBLUR_EXECUTABLE=/opt/bin/unblur_openmp_7_17_15.exe
RELION_MINIMUM_DEDICATED=32
_CE_CONDA=
PATH=/opt/cryosparc2/cryosparc2_worker/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/opt/cryosparc2/cryosparc2_worker/deps/anaconda/condabin:/home/spuser/bin:/home/spuser/.local/bin:/opt/cryosparc2/cryosparc2_master/bin:/data/cryosparc_user/software/cryosparc/cryosparc2_master/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/opt/IMOD/bin:/opt/relion-3.1-b/bin:/opt/cistem-1.0.0-beta:/opt/bin:/opt/sp/bin:/usr/lib64/openmpi/bin:/opt/motioncorr_v2.1/bin:/opt/frealign_v9.11/bin:/opt/bin:/usr/lib64/openmpi/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/local/cuda-8.0/bin:/snap/bin:/home/spuser/.dotnet/tools:/usr/local/cuda-8.0/bin
MAIL=/var/mail/spuser
MODULE_VERSION=3.2.10
MKL_NUM_THREADS=1
QT_QPA_PLATFORMTHEME=appmenu-qt5
IMOD_PLUGIN_DIR=/opt/IMOD/lib/imodplug
CONDA_PREFIX=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/envs/cryosparc_worker_env
LC_IDENTIFICATION=de_DE.UTF-8
PWD=/opt/cryosparc2/cryosparc2_worker
CRYOSPARC_ROOT_DIR=/opt/cryosparc2/cryosparc2_worker
NUMEXPR_NUM_THREADS=1
_LMFILES_=/opt/sp/modulefiles/relion/3.1-b
RELION_RESMAP_EXECUTABLE=/opt/bin/ResMap
IMOD_JAVADIR=/usr/local/java
LANG=en_US.UTF-8
MODULEPATH=/opt/sp/modulefiles:/opt/sp/modulefiles:/etc/environment-modules/modules:/usr/share/modules/versions:/usr/Modules/$MODULE_VERSION/modulefiles:/usr/share/modules/modulefiles
LC_MEASUREMENT=de_DE.UTF-8
LOADEDMODULES=relion/3.1-b
FOR_DISABLE_STACK_TRACE=1
_CE_M=
RELION_CTFFIND_EXECUTABLE=/opt/ctffind-bin-4.1.5/bin/ctffind
HOME=/home/spuser
SHLVL=1
CRYOSPARC_PATH=/opt/cryosparc2/cryosparc2_worker/bin
RELION_SUMMOVIE_EXECUTABLE=/opt/bin/sum_movie_openmp_7_17_15.exe
PYTHONPATH=/opt/cryosparc2/cryosparc2_worker
CONDA_PYTHON_EXE=/opt/cryosparc2/cryosparc2_worker/deps/anaconda/bin/python
LOGNAME=spuser
SSH_CONNECTION= # redacted #
XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop
MODULESHOME=/usr/share/modules
OMP_NUM_THREADS=1
CONDA_DEFAULT_ENV=cryosparc_worker_env
RELION_GCTF_EXECUTABLE=/opt/bin/Gctf
IMOD_CALIB_DIR=/usr/local/ImodCalib
LESSOPEN=| /usr/bin/lesspipe %s
RELION_PDFVIEWER_EXECUTABLEe=evince
DISPLAY=localhost:10.0
XDG_RUNTIME_DIR=/run/user/1001
LC_TIME=de_DE.UTF-8
LESSCLOSE=/usr/bin/lesspipe %s %s
CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.5
LC_NAME=de_DE.UTF-8
BASH_FUNC_module%%=() {  eval `/usr/bin/modulecmd bash $*`
}

cat /etc/ld.so.conf

include /etc/ld.so.conf.d/*.conf

ls /etc/ld.so.conf.d/

cuda-10-1.conf cuda-11-2.conf cuda-11-5.conf fakeroot-x86_64-linux-gnu.conf i386-linux-gnu.conf libc.conf x86_64-linux-gnu.conf x86_64-linux-gnu_EGL.conf x86_64-linux-gnu_GL.conf

cat /etc/ld.so.conf.d/*.conf

/usr/local/cuda-10.1/targets/x86_64-linux/lib
/usr/local/cuda-11.2/targets/x86_64-linux/lib
/usr/local/cuda-11.5/targets/x86_64-linux/lib
/usr/lib/x86_64-linux-gnu/libfakeroot

Multiarch support

/lib/i386-linux-gnu
/usr/lib/i386-linux-gnu
/lib/i686-linux-gnu
/usr/lib/i686-linux-gnu

libc default configuration

/usr/local/lib

Multiarch support

/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu/mesa-egl
/usr/lib/x86_64-linux-gnu/mesa

Let me know if there is anything else that would be helpful, and thanks again for the support!

Cheers,

Fred