3DFlex dependency installation error

Install, Configure and Update
1

Hi Everyone,

I have tried to install 3DFlex.
While I installed dependency, I got these error at the last.
After that, Cryosparc didn’t work.
When I did rollback to regular install condition (./bin/cryosparcw forcedeps)
Common processing works.

Please let me know what causes these error, still CUDA issue?
Thanks,

Jinseo

Successfully installed torch-1.13.1
ERROR: Exception:
Traceback (most recent call last):
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py”, line 160, in exc_logging_wrapper
status = run_func(*args)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py”, line 247, in wrapper
return func(self, options, args)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/commands/install.py”, line 344, in run
reqs = self.get_requirements(args, options, finder, session)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py”, line 411, in get_requirements
req_to_add = install_req_from_line(
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/req/constructors.py”, line 395, in install_req_from_line
return InstallRequirement(
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/req/req_install.py”, line 127, in init
self.local_file_path = self.link.file_path
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/models/link.py”, line 330, in file_path
return url_to_path(self.url)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/utils/urls.py”, line 43, in url_to_path
raise ValueError(
ValueError: non-local file URIs are not supported on this platform: ‘file://Programs/Cryosparc/cryosparc_worker/deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl’

2

Please can you confirm that your worker satisfies the installation prerequisites:

  1. nvidia-smi
  2. env | grep PATH

and then attempt again

cd /Programs/Cryosparc/cryosparc_worker/
./bin/cryosparcw 3dflex-install 2>&1 | tee 3dflex-install.log

Are there any additional errors recorded in 3dflex-install.log?

3

Thanks for answer,

Now I am running regular processing would be finished this night.
Tomorrow I would try again.

Here is some information.
$ nvidia-smi
Tue Jan 17 15:46:38 2023
±----------------------------------------------------------------------------+
| NVIDIA-SMI 460.84 Driver Version: 460.84 CUDA Version: 11.2 |
|-------------------------------±---------------------±---------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 GeForce RTX 3090 Off | 00000000:31:00.0 On | N/A |
| 41% 55C P2 146W / 350W | 1347MiB / 24265MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+
| 1 GeForce RTX 3090 Off | 00000000:4B:00.0 Off | N/A |
| 37% 28C P8 31W / 350W | 3MiB / 24268MiB | 0% Default |
| | | N/A |
±------------------------------±---------------------±---------------------+

$ env | grep PATH
SB_ORIG_LD_LIBRARY_PATH=/usr/local/relion-3.1/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local /cuda-11.0/lib:/usr/local/cuda-11.0/lib64:/usr/local/cuda-10.1/lib:/usr/local/cuda-10.1/lib64:/usr/loc al/cuda-10.0/lib:/usr/local/cuda-10.0/lib64:/usr/local/cuda-9.2/lib:/usr/local/cuda-9.2/lib64:/usr/loc al/cuda-9.1/lib:/usr/local/cuda-9.1/lib64:/usr/local/cuda-8.0/lib:/usr/local/cuda-8.0/lib64:/usr/local /cuda-7.5/lib:/usr/local/cuda-7.5/lib64:
LD_LIBRARY_PATH=/usr/local/relion-3.1/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/cuda-11 .0/lib:/usr/local/cuda-11.0/lib64:/usr/local/cuda-10.1/lib:/usr/local/cuda-10.1/lib64:/usr/local/cuda- 10.0/lib:/usr/local/cuda-10.0/lib64:/usr/local/cuda-9.2/lib:/usr/local/cuda-9.2/lib64:/usr/local/cuda- 9.1/lib:/usr/local/cuda-9.1/lib64:/usr/local/cuda-8.0/lib:/usr/local/cuda-8.0/lib64:/usr/local/cuda-7. 5/lib:/usr/local/cuda-7.5/lib64:

According to these,

GPU driver should be OK.
May be CUDA library from other PATH (SB_ORIG_LD_LIBRARY_PATH) may cause trouble?
I have been tested export command for LD_LIBRARY_PATH=.

After I try and will update post.

Thanks,

Jinseo

4

A colleague spotted something unusual in the paths inside the error trace. Please can you post the output of
/Programs/Cryosparc/cryosparc_worker/bin/cryosparcw env | grep -v LICENSE

Please clear LD_LIBRARY_PATH and shorten PATH before you attempt cryosparcw install-3dflex:

LD_LIBRARY_PATH=""
PATH=/usr/bin:/bin
export LD_LIBRARY_PATH PATH
5

Here it is.

$ /Programs/Cryosparc/cryosparc_worker/bin/cryosparcw env | grep -v LICENSE
export "CRYOSPARC_USE_GPU=true"
export "CRYOSPARC_CONDA_ENV=cryosparc_worker_env"
export "CRYOSPARC_DEVELOP=false"
export "CRYOSPARC_ROOT_DIR=/Programs/Cryosparc/cryosparc_worker"
export "CRYOSPARC_PATH=/Programs/Cryosparc/cryosparc_worker/bin"
export "CRYOSPARC_CUDA_PATH=//usr/local/cuda"
export "PATH=//usr/local/cuda/bin:/Programs/Cryosparc/cryosparc_worker/bin://Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin://Programs/Cryosparc/cryosparc_worker/deps/anaconda/condabin:/usr/local/EMAN_2.91/bin:/usr/local/EMAN_2.91/condabin:/programs/x86_64-linux/system/sbgrid_bin://Programs/Cryosparc/cryosparc_master/bin://Programs/Cryosparc/cryosparc_master/bin:/usr/local/relion-3.1/bin:/usr/local/mpich-3.2.1/bin:/usr/local/cuda/bin:/usr/local/IMOD/bin:/usr/local/bin:/usr/bin:/usr/local/sbin:/usr/sbin:/usr/local/motioncorr_v2.1/bin:/usr/local/Gctf_v1.06/bin:/usr/local/Gctf_v0.50/bin:/usr/local/ResMap:/usr/local/cistem-1.0.0-beta:/usr/local/EMAN_2.91/bin:/programs/share/bin:/programs/share/sbgrid/bin:/home/exx/.local/bin:/home/exx/bin"
export "LD_LIBRARY_PATH=//usr/local/cuda/lib64:/Programs/Cryosparc/cryosparc_worker/deps/external/cudnn/lib:/usr/local/relion-3.1/lib:/usr/local/cuda/lib:/usr/local/cuda/lib64:/usr/local/cuda-11.0/lib:/usr/local/cuda-11.0/lib64:/usr/local/cuda-10.1/lib:/usr/local/cuda-10.1/lib64:/usr/local/cuda-10.0/lib:/usr/local/cuda-10.0/lib64:/usr/local/cuda-9.2/lib:/usr/local/cuda-9.2/lib64:/usr/local/cuda-9.1/lib:/usr/local/cuda-9.1/lib64:/usr/local/cuda-8.0/lib:/usr/local/cuda-8.0/lib64:/usr/local/cuda-7.5/lib:/usr/local/cuda-7.5/lib64"
export "LD_PRELOAD=/Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/libpython3.8.so"
export "PYTHONPATH=/Programs/Cryosparc/cryosparc_worker"
export "PYTHONNOUSERSITE=true"
export "CONDA_SHLVL=1"
export "CONDA_PROMPT_MODIFIER=(cryosparc_worker_env)"
export "CONDA_EXE=//Programs/Cryosparc/cryosparc_worker/deps/anaconda/bin/conda"
export "CONDA_PREFIX=//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env"
export "CONDA_PYTHON_EXE=//Programs/Cryosparc/cryosparc_worker/deps/anaconda/bin/python"
export "CONDA_DEFAULT_ENV=cryosparc_worker_env"
6

I tried your suggestions.
But I still got same errors before.
==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 22.11.1

Please update conda by running

$ conda update -n base -c defaults conda

Found existing installation: pycuda 2020.1
Uninstalling pycuda-2020.1:
Successfully uninstalled pycuda-2020.1
Collecting torch
Using cached torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)
Installing collected packages: torch
Successfully installed torch-1.13.1
ERROR: Exception:
Traceback (most recent call last):
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/base_command.py”, line 160, in exc_logging_wrapper
status = run_func(*args)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py”, line 247, in wrapper
return func(self, options, args)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/commands/install.py”, line 344, in run
reqs = self.get_requirements(args, options, finder, session)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/cli/req_command.py”, line 411, in get_requirements
req_to_add = install_req_from_line(
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/req/constructors.py”, line 395, in install_req_from_line
return InstallRequirement(
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/req/req_install.py”, line 127, in init
self.local_file_path = self.link.file_path
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/models/link.py”, line 330, in file_path
return url_to_path(self.url)
File “//Programs/Cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/pip/_internal/utils/urls.py”, line 43, in url_to_path
raise ValueError(
ValueError: non-local file URIs are not supported on this platform: ‘file://Programs/Cryosparc/cryosparc_worker/deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl’

In the middle, There was notification about old conda. So I did update conda using suggested command.

==> WARNING: A newer version of conda exists. <==
current version: 4.12.0
latest version: 22.11.1

Please update conda by running

$ conda update -n base -c defaults conda

But my conda version still stuck like this.
$conda --version
conda 4.10.3

I am still looking for other ways.
Thanks,

Jinseo

7

Hi,

Finally solved issue without error.
The reason was this file “pycuda-2020.1-cp38-cp38-linux_x86_64.whl” wasn’t downloaded right size.
So I deleted that file and install 3dflex dependencies again.
It was done without any error.
Found existing installation: pycuda 2020.1
Uninstalling pycuda-2020.1:
Successfully uninstalled pycuda-2020.1
Collecting torch
Using cached torch-1.13.1-cp38-cp38-manylinux1_x86_64.whl (887.4 MB)
Installing collected packages: torch
Successfully installed torch-1.13.1
Collecting pycuda==2020.1
Downloading pycuda-2020.1.tar.gz (1.6 MB)
━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 1.6/1.6 MB 38.2 MB/s eta 0:00:00
Preparing metadata (setup.py): started
Preparing metadata (setup.py): finished with status ‘done’
Building wheels for collected packages: pycuda
Building wheel for pycuda (setup.py): started
Building wheel for pycuda (setup.py): still running…
Building wheel for pycuda (setup.py): finished with status ‘done’
Created wheel for pycuda: filename=pycuda-2020.1-cp38-cp38-linux_x86_64.whl size=640827 sha256=2120698940ae33046f049b6bf4dcd3885aba7f977c507286eeb72a8d9dfede82
Stored in directory: /tmp/pip-ephem-wheel-cache-0wujekss/wheels/e2/e4/43/1c836775732712d40ebfc34a6cd621e282ae157ab9833f764a
Successfully built pycuda
Installing collected packages: pycuda
Successfully installed pycuda-2020.1
3D Flex Refine dependencies installed successfully.

$ cryosparcm test workers P1 --test gpu --test-pytorch
didn’t show any critical error too.

I would continue my processing.
Thanks for your help,

Jinseo

8

Please ignore
==> WARNING: A newer version of conda exists. <==
in the context of CryoSPARC installation.
We wonder whether non-standard double-slashes // inside some of your environment variable definitions could confound the installation process, and where those // prefixes come from.
Please can you post the output of

grep -v LICENSE /Programs/Cryosparc/cryosparc_worker/config.sh
9

Here it is.

$ grep -v LICENSE /Programs/Cryosparc/cryosparc_worker/config.sh

export CRYOSPARC_USE_GPU=true
export CRYOSPARC_CUDA_PATH=“//usr/local/cuda”
export CRYOSPARC_DEVELOP=false

Does it need to be change from “//usr/local/cuda” to “/user/local/cuda”?
Commonally double slash means absolute path from root directory.
Does it cause potential error in the future?

Thanks

10

I would change it, just in case.

I thought a single leading / means exactly that, and double // is a “harmless” synonym under some (unix shell?), but possibly not all circumstances.

11

I think in some cases the double-slash appears to indicate a Samba path instead of a local filesystem?

12

Hi,

I changed cuda path with single slash using cryosparcw newcuda command.
Thanks,

Jinseo