Installing v4.6 failed GPU tests

zhenyuanliu · November 12, 2024, 8:48am

Hello, administrator. I have successfully run a V4.6 cryosparc in our cluster, a gradual upgrade from a lower version. Now I have some problems while re-installing a new cryosparc, I hope to wait for your help, thank you.

System information about the working node

$ uname -a
Linux agpu61 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ eval $(/Share/home/user/app/cryosparc_new_v4/cryosparc_worker/bin/cryosparcw env)
$ env | grep PATH
MANPATH=/Share/app/intel_other/2018u1/man/common:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mpi/man:/Share/app/intel_other/2018u1/documentation_2018/en/debugger//gdb-ia/man/:/Share/app/intel_other/2018u1/documentation_2018/en/debugger//gdb-igfx/man/:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mpi/man:::
LIBRARY_PATH=/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/ipp/lib/intel64:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64/gcc4.7:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64/gcc4.7:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/daal/lib/intel64_lin:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/daal/../tbb/lib/intel64_lin/gcc4.4:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/tbb/lib/intel64_lin/gcc4.7:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64_lin:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin
NUMBA_CUDA_INCLUDE_PATH=/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
LD_LIBRARY_PATH=
CPATH=/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/ipp/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/pstl/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/tbb/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/tbb/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/daal/include:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/include
NLSPATH=/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/compiler/lib/intel64/locale/%l_%t/%N:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin/locale/%l_%t/%N:/Share/app/intel_other/2018u1/debugger_2018/gdb/intel64/share/locale/%l_%t/%N:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/lib/intel64_lin/locale/%l_%t/%N
PATH=/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/bin:/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/condabin:/Share/home/user/app/cryosparc_new_v4/cryosparc_master/bin:/usr/local/bin:/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/lpp/mmfs/bin:/opt/ibutils/bin:/opt/slurm/bin:/Share/home/user/.local/bin:/Share/home/user/bin
MODULEPATH=/Share/app/module-5.0.1/modulefiles/devtools:/Share/app/module-5.0.1/modulefiles/compiler:/Share/app/module-5.0.1/modulefiles/apps:/Share/app/module-5.0.1/modulefiles/mpi:/Share/app/module-5.0.1/modulefiles/mathlib
LIBTBX_OPATH=
CRYOSPARC_PATH=/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/bin
PYTHONPATH=/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker
CLASSPATH=/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mpi/intel64/lib/mpi.jar:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/daal/lib/daal.jar:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mpi/intel64/lib/mpi.jar
PKG_CONFIG_PATH=/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/bin/pkgconfig:/Share/app/intel_other/2018u1/compilers_and_libraries_2018.1.163/linux/mkl/bin/pkgconfig
INFOPATH=/Share/app/intel_other/2018u1/documentation_2018/en/debugger//gdb-ia/info/:/Share/app/intel_other/2018u1/documentation_2018/en/debugger//gdb-igfx/info/
$ /sbin/ldconfig -p | grep -i cuda
        libicudata.so.50 (libc6,x86-64) => /lib64/libicudata.so.50
        libcudadebugger.so.1 (libc6,x86-64) => /lib64/libcudadebugger.so.1
        libcuda_wrapper.so.0 (libc6,x86-64) => /lib64/libcuda_wrapper.so.0
        libcuda_wrapper.so (libc6,x86-64) => /lib64/libcuda_wrapper.so
        libcuda.so.1 (libc6,x86-64) => /lib64/libcuda.so.1
        libcuda.so.1 (libc6) => /lib/libcuda.so.1
        libcuda.so (libc6,x86-64) => /lib64/libcuda.so
        libcuda.so (libc6) => /lib/libcuda.so
$ uname -a
Linux agpu61 3.10.0-957.el7.x86_64 #1 SMP Thu Oct 4 20:48:51 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
$ free -g
              total        used        free      shared  buff/cache   available
Mem:           1007          41         811           0         153         963
Swap:            63           0          63
$ nvidia-smi
Tue Nov 12 16:39:29 2024
+---------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.23.08              Driver Version: 545.23.08    CUDA Version: 12.3     |
|-----------------------------------------+----------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |         Memory-Usage | GPU-Util  Compute M. |
|                                         |                      |               MIG M. |
|=========================================+======================+======================|
|   0  NVIDIA A40                     On  | 00000000:17:00.0 Off |                    0 |
|  0%   27C    P8              21W / 300W |      4MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   1  NVIDIA A40                     On  | 00000000:65:00.0 Off |                    0 |
|  0%   25C    P8              20W / 300W |      4MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   2  NVIDIA A40                     On  | 00000000:CA:00.0 Off |                    0 |
|  0%   29C    P8              21W / 300W |      4MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+
|   3  NVIDIA A40                     On  | 00000000:E3:00.0 Off |                    0 |
|  0%   29C    P8              21W / 300W |      4MiB / 46068MiB |      0%      Default |
|                                         |                      |                  N/A |
+-----------------------------------------+----------------------+----------------------+

+---------------------------------------------------------------------------------------+
| Processes:                                                                            |
|  GPU   GI   CI        PID   Type   Process name                            GPU Memory |
|        ID   ID                                                             Usage      |
|=======================================================================================|
|  No running processes found                                                           |
+---------------------------------------------------------------------------------------+

zhenyuanliu · November 12, 2024, 8:49am

Work node installation output

******* CRYOSPARC SYSTEM: WORKER INSTALLER ***********************

 Installation Settings:
   License ID              : 51d21340-9d18-11ef-8c13-3b9a2f99e21d
   Root Directory          : /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker
   Standalone Installation : false
   Version                 : v4.6.0

******************************************************************

 NVIDIA driver check..
 Found nvidia-smi at /bin/nvidia-smi


******************************************************************

 Setting up hard-coded config.sh environment variables

******************************************************************

 Installing all dependencies.

Checking dependencies...
Dependencies for python have changed - reinstalling...
  ------------------------------------------------------------------------
  Installing anaconda python...
  ------------------------------------------------------------------------
PREFIX=/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda
Unpacking payload ...
Extracting _libgcc_mutex-0.1-conda_forge.tar.bz2
Extracting ca-certificates-2024.2.2-hbcca054_0.conda
Extracting ld_impl_linux-64-2.40-h41732ed_0.conda
Extracting libstdcxx-ng-13.2.0-h7e041cc_5.conda
Extracting pybind11-abi-4-hd8ed1ab_3.tar.bz2
Extracting python_abi-3.10-4_cp310.conda
Extracting tzdata-2024a-h0c530f3_0.conda
Extracting libgomp-13.2.0-h807b86a_5.conda
Extracting _openmp_mutex-4.5-2_gnu.tar.bz2
Extracting libgcc-ng-13.2.0-h807b86a_5.conda
Extracting bzip2-1.0.8-hd590300_5.conda
Extracting c-ares-1.28.1-hd590300_0.conda
Extracting fmt-10.2.1-h00ab1b0_0.conda
Extracting icu-73.2-h59595ed_0.conda
Extracting keyutils-1.6.1-h166bdaf_0.tar.bz2
Extracting libev-4.33-hd590300_2.conda
Extracting libffi-3.4.2-h7f98852_5.tar.bz2
Extracting libiconv-1.17-hd590300_2.conda
Extracting libnsl-2.0.1-hd590300_0.conda
Extracting libuuid-2.38.1-h0b41bf4_0.conda
Extracting libxcrypt-4.4.36-hd590300_1.conda
Extracting libzlib-1.2.13-hd590300_5.conda
Extracting lz4-c-1.9.4-hcb278e6_0.conda
Extracting lzo-2.10-h516909a_1000.tar.bz2
Extracting ncurses-6.4.20240210-h59595ed_0.conda
Extracting openssl-3.2.1-hd590300_1.conda
Extracting reproc-14.2.4.post0-hd590300_1.conda
Extracting xz-5.2.6-h166bdaf_0.tar.bz2
Extracting yaml-cpp-0.8.0-h59595ed_0.conda
Extracting libedit-3.1.20191231-he28a2e2_2.tar.bz2
Extracting libnghttp2-1.58.0-h47da74e_1.conda
Extracting libsolv-0.7.28-hfc55251_2.conda
Extracting libsqlite-3.45.2-h2797004_0.conda
Extracting libssh2-1.11.0-h0841786_0.conda
Extracting libxml2-2.12.6-h232c23b_1.conda
Extracting readline-8.2-h8228510_1.conda
Extracting reproc-cpp-14.2.4.post0-h59595ed_1.conda
Extracting tk-8.6.13-noxft_h4845f30_101.conda
Extracting zstd-1.5.5-hfc55251_0.conda
Extracting krb5-1.21.2-h659d440_0.conda
Extracting libarchive-3.7.2-h2aa1ff5_1.conda
Extracting python-3.10.14-hd12c33a_0_cpython.conda
Extracting libcurl-8.7.1-hca28451_0.conda
Extracting menuinst-2.0.2-py310hff52083_0.conda
Extracting archspec-0.2.3-pyhd8ed1ab_0.conda
Extracting boltons-24.0.0-pyhd8ed1ab_0.conda
Extracting brotli-python-1.1.0-py310hc6cd4ac_1.conda
Extracting certifi-2024.2.2-pyhd8ed1ab_0.conda
Extracting charset-normalizer-3.3.2-pyhd8ed1ab_0.conda
Extracting colorama-0.4.6-pyhd8ed1ab_0.tar.bz2
Extracting distro-1.9.0-pyhd8ed1ab_0.conda
Extracting idna-3.6-pyhd8ed1ab_0.conda
Extracting jsonpointer-2.4-py310hff52083_3.conda
Extracting libmamba-1.5.8-had39da4_0.conda
Extracting packaging-24.0-pyhd8ed1ab_0.conda
Extracting platformdirs-4.2.0-pyhd8ed1ab_0.conda
Extracting pluggy-1.4.0-pyhd8ed1ab_0.conda
Extracting pycosat-0.6.6-py310h2372a71_0.conda
Extracting pycparser-2.22-pyhd8ed1ab_0.conda
Extracting pysocks-1.7.1-pyha2e5f31_6.tar.bz2
Extracting ruamel.yaml.clib-0.2.8-py310h2372a71_0.conda
Extracting setuptools-69.5.1-pyhd8ed1ab_0.conda
Extracting truststore-0.8.0-pyhd8ed1ab_0.conda
Extracting wheel-0.43.0-pyhd8ed1ab_1.conda
Extracting cffi-1.16.0-py310h2fee648_0.conda
Extracting jsonpatch-1.33-pyhd8ed1ab_0.conda
Extracting libmambapy-1.5.8-py310h39ff949_0.conda
Extracting pip-24.0-pyhd8ed1ab_0.conda
Extracting ruamel.yaml-0.18.6-py310h2372a71_0.conda
Extracting tqdm-4.66.2-pyhd8ed1ab_0.conda
Extracting urllib3-2.2.1-pyhd8ed1ab_0.conda
Extracting requests-2.31.0-pyhd8ed1ab_0.conda
Extracting zstandard-0.22.0-py310h1275a96_0.conda
Extracting conda-package-streaming-0.9.0-pyhd8ed1ab_0.conda
Extracting conda-package-handling-2.2.0-pyh38be061_0.conda
Extracting conda-24.3.0-py310hff52083_0.conda
Extracting conda-libmamba-solver-24.1.0-pyhd8ed1ab_0.conda
Extracting mamba-1.5.8-py310h51d5547_0.conda

Installing base environment...

Transaction

  Prefix: /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda

  Updating specs:

   - conda-forge/linux-64::_libgcc_mutex==0.1=conda_forge[md5=d7c89558ba9fa0495403155b64376d81]
   - conda-forge/linux-64::ca-certificates==2024.2.2=hbcca054_0[md5=2f4327a1cbe7f022401b236e915a5fef]
   - conda-forge/linux-64::ld_impl_linux-64==2.40=h41732ed_0[md5=7aca3059a1729aa76c597603f10b0dd3]
   - conda-forge/linux-64::libstdcxx-ng==13.2.0=h7e041cc_5[md5=f6f6600d18a4047b54f803cf708b868a]
   - conda-forge/noarch::pybind11-abi==4=hd8ed1ab_3[md5=878f923dd6acc8aeb47a75da6c4098be]
   - conda-forge/linux-64::python_abi==3.10=4_cp310[md5=26322ec5d7712c3ded99dd656142b8ce]
   - conda-forge/noarch::tzdata==2024a=h0c530f3_0[md5=161081fc7cec0bfda0d86d7cb595f8d8]
   - conda-forge/linux-64::libgomp==13.2.0=h807b86a_5[md5=d211c42b9ce49aee3734fdc828731689]
   - conda-forge/linux-64::_openmp_mutex==4.5=2_gnu[md5=73aaf86a425cc6e73fcf236a5a46396d]
   - conda-forge/linux-64::libgcc-ng==13.2.0=h807b86a_5[md5=d4ff227c46917d3b4565302a2bbb276b]
   - conda-forge/linux-64::bzip2==1.0.8=hd590300_5[md5=69b8b6202a07720f448be700e300ccf4]
   - conda-forge/linux-64::c-ares==1.28.1=hd590300_0[md5=dcde58ff9a1f30b0037a2315d1846d1f]
   - conda-forge/linux-64::fmt==10.2.1=h00ab1b0_0[md5=35ef8bc24bd34074ebae3c943d551728]
   - conda-forge/linux-64::icu==73.2=h59595ed_0[md5=cc47e1facc155f91abd89b11e48e72ff]
   - conda-forge/linux-64::keyutils==1.6.1=h166bdaf_0[md5=30186d27e2c9fa62b45fb1476b7200e3]
   - conda-forge/linux-64::libev==4.33=hd590300_2[md5=172bf1cd1ff8629f2b1179945ed45055]
   - conda-forge/linux-64::libffi==3.4.2=h7f98852_5[md5=d645c6d2ac96843a2bfaccd2d62b3ac3]
   - conda-forge/linux-64::libiconv==1.17=hd590300_2[md5=d66573916ffcf376178462f1b61c941e]
   - conda-forge/linux-64::libnsl==2.0.1=hd590300_0[md5=30fd6e37fe21f86f4bd26d6ee73eeec7]
   - conda-forge/linux-64::libuuid==2.38.1=h0b41bf4_0[md5=40b61aab5c7ba9ff276c41cfffe6b80b]
   - conda-forge/linux-64::libxcrypt==4.4.36=hd590300_1[md5=5aa797f8787fe7a17d1b0821485b5adc]
   - conda-forge/linux-64::libzlib==1.2.13=hd590300_5[md5=f36c115f1ee199da648e0597ec2047ad]
   - conda-forge/linux-64::lz4-c==1.9.4=hcb278e6_0[md5=318b08df404f9c9be5712aaa5a6f0bb0]
   - conda-forge/linux-64::lzo==2.10=h516909a_1000[md5=bb14fcb13341b81d5eb386423b9d2bac]
   - conda-forge/linux-64::ncurses==6.4.20240210=h59595ed_0[md5=97da8860a0da5413c7c98a3b3838a645]
   - conda-forge/linux-64::openssl==3.2.1=hd590300_1[md5=9d731343cff6ee2e5a25c4a091bf8e2a]
   - conda-forge/linux-64::reproc==14.2.4.post0=hd590300_1[md5=82ca53502dfd5a64a80dee76dae14685]
   - conda-forge/linux-64::xz==5.2.6=h166bdaf_0[md5=2161070d867d1b1204ea749c8eec4ef0]
   - conda-forge/linux-64::yaml-cpp==0.8.0=h59595ed_0[md5=965eaacd7c18eb8361fd12bb9e7a57d7]
   - conda-forge/linux-64::libedit==3.1.20191231=he28a2e2_2[md5=4d331e44109e3f0e19b4cb8f9b82f3e1]
   - conda-forge/linux-64::libnghttp2==1.58.0=h47da74e_1[md5=700ac6ea6d53d5510591c4344d5c989a]
   - conda-forge/linux-64::libsolv==0.7.28=hfc55251_2[md5=535bafe1ed0a5bdd3f4c125ca05d378c]
   - conda-forge/linux-64::libsqlite==3.45.2=h2797004_0[md5=866983a220e27a80cb75e85cb30466a1]
   - conda-forge/linux-64::libssh2==1.11.0=h0841786_0[md5=1f5a58e686b13bcfde88b93f547d23fe]
   - conda-forge/linux-64::libxml2==2.12.6=h232c23b_1[md5=6853448e9ca1cfd5f15382afd2a6d123]
   - conda-forge/linux-64::readline==8.2=h8228510_1[md5=47d31b792659ce70f470b5c82fdfb7a4]
   - conda-forge/linux-64::reproc-cpp==14.2.4.post0=h59595ed_1[md5=715e1d720ec1a03715bebd237972fca5]
   - conda-forge/linux-64::tk==8.6.13=noxft_h4845f30_101[md5=d453b98d9c83e71da0741bb0ff4d76bc]
   - conda-forge/linux-64::zstd==1.5.5=hfc55251_0[md5=04b88013080254850d6c01ed54810589]
   - conda-forge/linux-64::krb5==1.21.2=h659d440_0[md5=cd95826dbd331ed1be26bdf401432844]
   - conda-forge/linux-64::libarchive==3.7.2=h2aa1ff5_1[md5=3bf887827d1968275978361a6e405e4f]
   - conda-forge/linux-64::python==3.10.14=hd12c33a_0_cpython[md5=2b4ba962994e8bd4be9ff5b64b75aff2]
   - conda-forge/linux-64::libcurl==8.7.1=hca28451_0[md5=755c7f876815003337d2c61ff5d047e5]
   - conda-forge/linux-64::menuinst==2.0.2=py310hff52083_0[md5=4837faab0d3e665df57fef662148c6a3]
   - conda-forge/noarch::archspec==0.2.3=pyhd8ed1ab_0[md5=192278292e20704f663b9c766909d67b]
   - conda-forge/noarch::boltons==24.0.0=pyhd8ed1ab_0[md5=61de176bd62041f9cd5bd4fcd09eb0ff]
   - conda-forge/linux-64::brotli-python==1.1.0=py310hc6cd4ac_1[md5=1f95722c94f00b69af69a066c7433714]
   - conda-forge/noarch::certifi==2024.2.2=pyhd8ed1ab_0[md5=0876280e409658fc6f9e75d035960333]
   - conda-forge/noarch::charset-normalizer==3.3.2=pyhd8ed1ab_0[md5=7f4a9e3fcff3f6356ae99244a014da6a]
   - conda-forge/noarch::colorama==0.4.6=pyhd8ed1ab_0[md5=3faab06a954c2a04039983f2c4a50d99]
   - conda-forge/noarch::distro==1.9.0=pyhd8ed1ab_0[md5=bbdb409974cd6cb30071b1d978302726]
   - conda-forge/noarch::idna==3.6=pyhd8ed1ab_0[md5=1a76f09108576397c41c0b0c5bd84134]
   - conda-forge/linux-64::jsonpointer==2.4=py310hff52083_3[md5=08ec1463dbc5c806a32fc431874032ca]
   - conda-forge/linux-64::libmamba==1.5.8=had39da4_0[md5=def669885dc103d8acb7ac2ac35e0b2f]
   - conda-forge/noarch::packaging==24.0=pyhd8ed1ab_0[md5=248f521b64ce055e7feae3105e7abeb8]
   - conda-forge/noarch::platformdirs==4.2.0=pyhd8ed1ab_0[md5=a0bc3eec34b0fab84be6b2da94e98e20]
   - conda-forge/noarch::pluggy==1.4.0=pyhd8ed1ab_0[md5=139e9feb65187e916162917bb2484976]
   - conda-forge/linux-64::pycosat==0.6.6=py310h2372a71_0[md5=0adaac9a86d59adae2bc86b3cdef2df1]
   - conda-forge/noarch::pycparser==2.22=pyhd8ed1ab_0[md5=844d9eb3b43095b031874477f7d70088]
   - conda-forge/noarch::pysocks==1.7.1=pyha2e5f31_6[md5=2a7de29fb590ca14b5243c4c812c8025]
   - conda-forge/linux-64::ruamel.yaml.clib==0.2.8=py310h2372a71_0[md5=dcf6d2535586c77b31425ed835610c54]
   - conda-forge/noarch::setuptools==69.5.1=pyhd8ed1ab_0[md5=7462280d81f639363e6e63c81276bd9e]
   - conda-forge/noarch::truststore==0.8.0=pyhd8ed1ab_0[md5=08316d001eca8854392cf2837828ea11]
   - conda-forge/noarch::wheel==0.43.0=pyhd8ed1ab_1[md5=0b5293a157c2b5cd513dd1b03d8d3aae]
   - conda-forge/linux-64::cffi==1.16.0=py310h2fee648_0[md5=45846a970e71ac98fd327da5d40a0a2c]
   - conda-forge/noarch::jsonpatch==1.33=pyhd8ed1ab_0[md5=bfdb7c5c6ad1077c82a69a8642c87aff]
   - conda-forge/linux-64::libmambapy==1.5.8=py310h39ff949_0[md5=37f8aa15b73c4691eeec15caf45aab25]
   - conda-forge/noarch::pip==24.0=pyhd8ed1ab_0[md5=f586ac1e56c8638b64f9c8122a7b8a67]
   - conda-forge/linux-64::ruamel.yaml==0.18.6=py310h2372a71_0[md5=50b7d9b39099cdbabf65bf27df73a793]
   - conda-forge/noarch::tqdm==4.66.2=pyhd8ed1ab_0[md5=2b8dfb969f984497f3f98409a9545776]
   - conda-forge/noarch::urllib3==2.2.1=pyhd8ed1ab_0[md5=08807a87fa7af10754d46f63b368e016]
   - conda-forge/noarch::requests==2.31.0=pyhd8ed1ab_0[md5=a30144e4156cdbb236f99ebb49828f8b]
   - conda-forge/linux-64::zstandard==0.22.0=py310h1275a96_0[md5=54698ba13cd3494547b289cd86a2176a]
   - conda-forge/noarch::conda-package-streaming==0.9.0=pyhd8ed1ab_0[md5=38253361efb303deead3eab39ae9269b]
   - conda-forge/noarch::conda-package-handling==2.2.0=pyh38be061_0[md5=8a3ae7f6318376aa08ea753367bb7dd6]
   - conda-forge/linux-64::conda==24.3.0=py310hff52083_0[md5=4187d17c4b75d8f7757820e835c507c9]
   - conda-forge/noarch::conda-libmamba-solver==24.1.0=pyhd8ed1ab_0[md5=304dc78ad6e52e0fd663df1d484c1531]
   - conda-forge/linux-64::mamba==1.5.8=py310h51d5547_0[md5=3b335eaa4894cbb5379a75f83a4d6b40]


  Package                         Version  Build               Channel         Size
─────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────

  + _libgcc_mutex                     0.1  conda_forge         conda-forge
  + ca-certificates              2024.2.2  hbcca054_0          conda-forge
  + ld_impl_linux-64                 2.40  h41732ed_0          conda-forge
  + libstdcxx-ng                   13.2.0  h7e041cc_5          conda-forge
  + pybind11-abi                        4  hd8ed1ab_3          conda-forge
  + python_abi                       3.10  4_cp310             conda-forge
  + tzdata                          2024a  h0c530f3_0          conda-forge
  + libgomp                        13.2.0  h807b86a_5          conda-forge
  + _openmp_mutex                     4.5  2_gnu               conda-forge
  + libgcc-ng                      13.2.0  h807b86a_5          conda-forge
  + bzip2                           1.0.8  hd590300_5          conda-forge
  + c-ares                         1.28.1  hd590300_0          conda-forge
  + fmt                            10.2.1  h00ab1b0_0          conda-forge
  + icu                              73.2  h59595ed_0          conda-forge
  + keyutils                        1.6.1  h166bdaf_0          conda-forge
  + libev                            4.33  hd590300_2          conda-forge
  + libffi                          3.4.2  h7f98852_5          conda-forge
  + libiconv                         1.17  hd590300_2          conda-forge
  + libnsl                          2.0.1  hd590300_0          conda-forge
  + libuuid                        2.38.1  h0b41bf4_0          conda-forge
  + libxcrypt                      4.4.36  hd590300_1          conda-forge
  + libzlib                        1.2.13  hd590300_5          conda-forge
  + lz4-c                           1.9.4  hcb278e6_0          conda-forge
  + lzo                              2.10  h516909a_1000       conda-forge
  + ncurses                  6.4.20240210  h59595ed_0          conda-forge
  + openssl                         3.2.1  hd590300_1          conda-forge
  + reproc                   14.2.4.post0  hd590300_1          conda-forge
  + xz                              5.2.6  h166bdaf_0          conda-forge
  + yaml-cpp                        0.8.0  h59595ed_0          conda-forge
  + libedit                  3.1.20191231  he28a2e2_2          conda-forge
  + libnghttp2                     1.58.0  h47da74e_1          conda-forge
  + libsolv                        0.7.28  hfc55251_2          conda-forge
  + libsqlite                      3.45.2  h2797004_0          conda-forge
  + libssh2                        1.11.0  h0841786_0          conda-forge
  + libxml2                        2.12.6  h232c23b_1          conda-forge
  + readline                          8.2  h8228510_1          conda-forge
  + reproc-cpp               14.2.4.post0  h59595ed_1          conda-forge
  + tk                             8.6.13  noxft_h4845f30_101  conda-forge
  + zstd                            1.5.5  hfc55251_0          conda-forge
  + krb5                           1.21.2  h659d440_0          conda-forge
  + libarchive                      3.7.2  h2aa1ff5_1          conda-forge
  + python                        3.10.14  hd12c33a_0_cpython  conda-forge
  + libcurl                         8.7.1  hca28451_0          conda-forge
  + menuinst                        2.0.2  py310hff52083_0     conda-forge
  + archspec                        0.2.3  pyhd8ed1ab_0        conda-forge
  + boltons                        24.0.0  pyhd8ed1ab_0        conda-forge
  + brotli-python                   1.1.0  py310hc6cd4ac_1     conda-forge
  + certifi                      2024.2.2  pyhd8ed1ab_0        conda-forge
  + charset-normalizer              3.3.2  pyhd8ed1ab_0        conda-forge
  + colorama                        0.4.6  pyhd8ed1ab_0        conda-forge
  + distro                          1.9.0  pyhd8ed1ab_0        conda-forge
  + idna                              3.6  pyhd8ed1ab_0        conda-forge
  + jsonpointer                       2.4  py310hff52083_3     conda-forge
  + libmamba                        1.5.8  had39da4_0          conda-forge
  + packaging                        24.0  pyhd8ed1ab_0        conda-forge
  + platformdirs                    4.2.0  pyhd8ed1ab_0        conda-forge
  + pluggy                          1.4.0  pyhd8ed1ab_0        conda-forge
  + pycosat                         0.6.6  py310h2372a71_0     conda-forge
  + pycparser                        2.22  pyhd8ed1ab_0        conda-forge
  + pysocks                         1.7.1  pyha2e5f31_6        conda-forge
  + ruamel.yaml.clib                0.2.8  py310h2372a71_0     conda-forge
  + setuptools                     69.5.1  pyhd8ed1ab_0        conda-forge
  + truststore                      0.8.0  pyhd8ed1ab_0        conda-forge
  + wheel                          0.43.0  pyhd8ed1ab_1        conda-forge
  + cffi                           1.16.0  py310h2fee648_0     conda-forge
  + jsonpatch                        1.33  pyhd8ed1ab_0        conda-forge
  + libmambapy                      1.5.8  py310h39ff949_0     conda-forge
  + pip                              24.0  pyhd8ed1ab_0        conda-forge
  + ruamel.yaml                    0.18.6  py310h2372a71_0     conda-forge
  + tqdm                           4.66.2  pyhd8ed1ab_0        conda-forge
  + urllib3                         2.2.1  pyhd8ed1ab_0        conda-forge
  + requests                       2.31.0  pyhd8ed1ab_0        conda-forge
  + zstandard                      0.22.0  py310h1275a96_0     conda-forge
  + conda-package-streaming         0.9.0  pyhd8ed1ab_0        conda-forge
  + conda-package-handling          2.2.0  pyh38be061_0        conda-forge
  + conda                          24.3.0  py310hff52083_0     conda-forge
  + conda-libmamba-solver          24.1.0  pyhd8ed1ab_0        conda-forge
  + mamba                           1.5.8  py310h51d5547_0     conda-forge

  Summary:

  Install: 78 packages

  Total download: 0 B

─────────────────────────────────────────────────────────────────────────────────────



Transaction starting
Linking _libgcc_mutex-0.1-conda_forge
Linking ca-certificates-2024.2.2-hbcca054_0
Linking ld_impl_linux-64-2.40-h41732ed_0
Linking libstdcxx-ng-13.2.0-h7e041cc_5
Linking pybind11-abi-4-hd8ed1ab_3
Linking python_abi-3.10-4_cp310
Linking tzdata-2024a-h0c530f3_0
Linking libgomp-13.2.0-h807b86a_5
Linking _openmp_mutex-4.5-2_gnu
Linking libgcc-ng-13.2.0-h807b86a_5
Linking bzip2-1.0.8-hd590300_5
Linking c-ares-1.28.1-hd590300_0
Linking fmt-10.2.1-h00ab1b0_0
Linking icu-73.2-h59595ed_0
Linking keyutils-1.6.1-h166bdaf_0
Linking libev-4.33-hd590300_2
Linking libffi-3.4.2-h7f98852_5
Linking libiconv-1.17-hd590300_2
Linking libnsl-2.0.1-hd590300_0
Linking libuuid-2.38.1-h0b41bf4_0
Linking libxcrypt-4.4.36-hd590300_1
Linking libzlib-1.2.13-hd590300_5
Linking lz4-c-1.9.4-hcb278e6_0
Linking lzo-2.10-h516909a_1000
Linking ncurses-6.4.20240210-h59595ed_0
Linking openssl-3.2.1-hd590300_1
Linking reproc-14.2.4.post0-hd590300_1
Linking xz-5.2.6-h166bdaf_0
Linking yaml-cpp-0.8.0-h59595ed_0
Linking libedit-3.1.20191231-he28a2e2_2
Linking libnghttp2-1.58.0-h47da74e_1
Linking libsolv-0.7.28-hfc55251_2
Linking libsqlite-3.45.2-h2797004_0
Linking libssh2-1.11.0-h0841786_0
Linking libxml2-2.12.6-h232c23b_1
Linking readline-8.2-h8228510_1
Linking reproc-cpp-14.2.4.post0-h59595ed_1
Linking tk-8.6.13-noxft_h4845f30_101
Linking zstd-1.5.5-hfc55251_0
Linking krb5-1.21.2-h659d440_0
Linking libarchive-3.7.2-h2aa1ff5_1
Linking python-3.10.14-hd12c33a_0_cpython
Linking libcurl-8.7.1-hca28451_0
Linking menuinst-2.0.2-py310hff52083_0
Linking archspec-0.2.3-pyhd8ed1ab_0
Linking boltons-24.0.0-pyhd8ed1ab_0
Linking brotli-python-1.1.0-py310hc6cd4ac_1
Linking certifi-2024.2.2-pyhd8ed1ab_0
Linking charset-normalizer-3.3.2-pyhd8ed1ab_0
Linking colorama-0.4.6-pyhd8ed1ab_0
Linking distro-1.9.0-pyhd8ed1ab_0
Linking idna-3.6-pyhd8ed1ab_0
Linking jsonpointer-2.4-py310hff52083_3
Linking libmamba-1.5.8-had39da4_0
Linking packaging-24.0-pyhd8ed1ab_0
Linking platformdirs-4.2.0-pyhd8ed1ab_0
Linking pluggy-1.4.0-pyhd8ed1ab_0
Linking pycosat-0.6.6-py310h2372a71_0
Linking pycparser-2.22-pyhd8ed1ab_0
Linking pysocks-1.7.1-pyha2e5f31_6
Linking ruamel.yaml.clib-0.2.8-py310h2372a71_0
Linking setuptools-69.5.1-pyhd8ed1ab_0
Linking truststore-0.8.0-pyhd8ed1ab_0
Linking wheel-0.43.0-pyhd8ed1ab_1
Linking cffi-1.16.0-py310h2fee648_0
Linking jsonpatch-1.33-pyhd8ed1ab_0
Linking libmambapy-1.5.8-py310h39ff949_0
Linking pip-24.0-pyhd8ed1ab_0
Linking ruamel.yaml-0.18.6-py310h2372a71_0
Linking tqdm-4.66.2-pyhd8ed1ab_0
Linking urllib3-2.2.1-pyhd8ed1ab_0
Linking requests-2.31.0-pyhd8ed1ab_0
Linking zstandard-0.22.0-py310h1275a96_0
Linking conda-package-streaming-0.9.0-pyhd8ed1ab_0
Linking conda-package-handling-2.2.0-pyh38be061_0
Linking conda-24.3.0-py310hff52083_0
Linking conda-libmamba-solver-24.1.0-pyhd8ed1ab_0
Linking mamba-1.5.8-py310h51d5547_0

Transaction finished

To activate this environment, use:

    micromamba activate /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda

Or to execute a single command in this environment, use:

    micromamba run -p /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda mycommand

installation finished.
  ------------------------------------------------------------------------
    Done.
    anaconda python installation successful.
  ------------------------------------------------------------------------
  Extracting all conda packages...
  ------------------------------------------------------------------------
.....................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................................
  ------------------------------------------------------------------------
    Done.
    conda packages installation successful.
  ------------------------------------------------------------------------
  Main dependency installation completed. Continuing...
  ------------------------------------------------------------------------
Completed.
Currently checking hash for ctffind
Dependencies for ctffind have changed - reinstalling...
  ------------------------------------------------------------------------
  ctffind 4.1.14 installation successful.
  ------------------------------------------------------------------------
Completed.
Currently checking hash for gctf
Dependencies for gctf have changed - reinstalling...
  ------------------------------------------------------------------------
  Gctf v1.06 installation successful.
  ------------------------------------------------------------------------
Completed.
Completed dependency check.

******* CRYOSPARC WORKER INSTALLATION COMPLETE *******************

 In order to run processing jobs, you will need to connect this
 worker to a cryoSPARC master.

******************************************************************

$ ./bin/cryosparcw connect --worker agpu61 --master alogin03 --port 32000 --gpus 0,1,2,3 --ssdpath=/ssdwork --newlane --lane agpu61
 ---------------------------------------------------------------
  CRYOSPARC CONNECT --------------------------------------------
 ---------------------------------------------------------------
  Attempting to register worker agpu61 to command alogin03:32002
  Connecting as unix user user
  Will register using ssh string: user@agpu61
  If this is incorrect, you should re-run this command with the flag --sshstr <ssh string>
 ---------------------------------------------------------------
  Connected to master.
 ---------------------------------------------------------------
  Current connected workers:
 ---------------------------------------------------------------
  Worker will be registered with 64 CPUs.
  Autodetecting available GPUs...
  Detected 4 CUDA devices.

   id           pci-bus  name
   ---------------------------------------------------------------
       0                23  NVIDIA A40
       1               101  NVIDIA A40
       2               202  NVIDIA A40
       3               227  NVIDIA A40
   ---------------------------------------------------------------
   Devices specified: 0, 1, 2, 3
   Devices 0, 1, 2, 3 will be enabled now.
   This can be changed later using --update
 ---------------------------------------------------------------
  Worker will be registered with SSD cache location /ssdwork
 ---------------------------------------------------------------
  Autodetecting the amount of RAM available...
  This machine has 1031.34GB RAM .
 ---------------------------------------------------------------
 ---------------------------------------------------------------
  Registering worker...
  Done.

  You can now launch jobs on the master node and they will be scheduled
  on to this worker node if resource requirements are met.
 ---------------------------------------------------------------
  Final configuration for agpu61
               cache_path :  /ssdwork
           cache_quota_mb :  None
         cache_reserve_mb :  10000
                     desc :  None
                     gpus :  [{'id': 0, 'mem': 47619112960, 'name': 'NVIDIA A40'}, {'id': 1, 'mem': 47619112960, 'name': 'NVIDIA A40'}, {'id': 2, 'mem': 47619112960, 'name': 'NVIDIA A40'}, {'id': 3, 'mem': 47619112960, 'name': 'NVIDIA A40'}]
                 hostname :  agpu61
                     lane :  agpu61
             monitor_port :  None
                     name :  agpu61
           resource_fixed :  {'SSD': True}
           resource_slots :  {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63], 'GPU': [0, 1, 2, 3], 'RAM': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39, 40, 41, 42, 43, 44, 45, 46, 47, 48, 49, 50, 51, 52, 53, 54, 55, 56, 57, 58, 59, 60, 61, 62, 63, 64, 65, 66, 67, 68, 69, 70, 71, 72, 73, 74, 75, 76, 77, 78, 79, 80, 81, 82, 83, 84, 85, 86, 87, 88, 89, 90, 91, 92, 93, 94, 95, 96, 97, 98, 99, 100, 101, 102, 103, 104, 105, 106, 107, 108, 109, 110, 111, 112, 113, 114, 115, 116, 117, 118, 119, 120, 121, 122, 123, 124, 125, 126, 127, 128]}
                  ssh_str :  user@agpu61
                    title :  Worker node agpu61
                     type :  node
          worker_bin_path :  /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/bin/cryosparcw
 ---------------------------------------------------------------

zhenyuanliu · November 12, 2024, 8:49am

Test output

$ cryosparcm test workers P1 --test gpu
Using project P1
Specifying gpu test
Running worker tests...
2024-11-12 16:27:26,094 log                  CRITICAL | Worker test results
2024-11-12 16:27:26,094 log                  CRITICAL | agpu61
2024-11-12 16:27:26,095 log                  CRITICAL |   ✕ GPU
2024-11-12 16:27:26,095 log                  CRITICAL |     Error: /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/cryosparc_compute/gpu/cryosparc_gpu.so: undefined symbol: cuMemFreeAsync
2024-11-12 16:27:26,095 log                  CRITICAL |     See P1 J2 for more information

Job log

$ cat job.log


================= CRYOSPARCW =======  2024-11-12 16:27:21.370340  =========
Project P1 Job J2
Master alogin03 Port 32002
===========================================================================
MAIN PROCESS PID 56241
========= now starting main process at 2024-11-12 16:27:21.370885
instance_testing.run cryosparc_compute.jobs.jobregister
/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float64'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:499: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  setattr(self, word, getattr(machar, word).flat[0])
/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.10/site-packages/numpy/core/getlimits.py:89: UserWarning: The value of the smallest subnormal for <class 'numpy.float32'> type is zero.
  return self._float_to_str(self.smallest_subnormal)
MONITOR PROCESS PID 56243
========= monitor process now waiting for main process
========= sending heartbeat at 2024-11-12 16:27:22.651484
***************************************************************
**** handle exception rc
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 116, in cryosparc_master.cryosparc_compute.run.main
  File "/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/cryosparc_compute/jobs/instance_testing/run.py", line 91, in run_gpu_job
    from ...gpu import driver, gpucore, gpuarray
  File "cryosparc_master/cryosparc_compute/gpu/gpucore.py", line 30, in init cryosparc_master.cryosparc_compute.gpu.gpucore
  File "/Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/cryosparc_compute/gpu/gpuarray.py", line 18, in <module>
    from . import cryosparc_gpu
ImportError: /Share2/home/user/app/cryosparc_new_v4/cryosparc_worker/cryosparc_compute/gpu/cryosparc_gpu.so: undefined symbol: cuMemFreeAsync
set status to failed
========= main process now complete at 2024-11-12 16:27:32.666232.
========= monitor process now complete at 2024-11-12 16:27:32.669198.

zhenyuanliu · November 12, 2024, 9:11am

I am sorry that the installation succeeded after I cancelled all the environment variables of the user, which should be in conflict with some software. Previously I only deleted PATH and LD_LIBRARY_PATH