Installing "3dflex" got failed

Praveen · April 13, 2023, 8:47pm

Hi Team, I was trying to install 3dflex, I followed the documentation as described in this URL Installing 3DFlex Dependencies - CryoSPARC Guide
But got stuck with the below error, th e installation was successful
when I ran the command “cryosparcm test workers P3 --test gpu --test-pytorch” to verify,I got this error. please find the below snippet.

@wtempel

wtempel · April 13, 2023, 9:13pm

Is there any additional information inside the event log or the job log of P3 J152?
Please paste log contents and terminal output as text.

Praveen · April 14, 2023, 2:49pm

Hi @wtempel ,
Job.log content is

===========================================================================
========= monitor process now starting main process
MAINPROCESS PID 11460
MAIN PID 11460
instance_testing.run cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
No sympy found
***************************************************************
**** handle exception rc
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/cryosparc_compute/jobs/instance_testing/run.py", line 183, in run_gpu_job
    import torch
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/__init__.py", line 1465, in <module>
    from . import _meta_registrations
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_meta_registrations.py", line 7, in <module>
    from torch._decomp import _add_op_to_registry, global_decomposition_table, meta_table
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_decomp/__init__.py", line 169, in <module>
    import torch._decomp.decompositions
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_decomp/decompositions.py", line 10, in <module>
    import torch._prims as prims
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_prims/__init__.py", line 33, in <module>
    from torch._subclasses.fake_tensor import FakeTensor, FakeTensorMode
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_subclasses/__init__.py", line 3, in <module>
    from torch._subclasses.fake_tensor import (
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_subclasses/fake_tensor.py", line 13, in <module>
    from torch._guards import Source
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_guards.py", line 78, in <module>
    class ShapeGuard(NamedTuple):
  File "/apps/cryosparc/v4.1.1_230110/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/torch/_guards.py", line 79, in ShapeGuard
    expr: sympy.Expr
NameError: name 'sympy' is not defined
set status to failed
========= main process now complete.
========= monitor process now complete.

I think this is a proble with “sympy” python package, eventhough I have installed this package with “pip install sympy” still it is not working. I need some help here? Thanks.

wtempel · April 14, 2023, 3:07pm

You should not have to install the sympy package directly; I suspect something went wrong during
cryosparcw install-3dflex.
You may want to

uninstall 3DFlex dependencies
cryosparcw forcedeps
ensure that in the shell where you run
cryosparcw install-3dflex and in the environment where CryoSPARC jobs run:
- no cuda executables or libraries are present, check:
  - $LD_LIBRARY_PATH
  - $PATH
  - output of /sbin/ldconfig -p
retry installation of 3DFlex dependencies, recording output
cryosparcw install-3dflex 2>&1 | tee install_3dflex.log
inspect recorded output for installation errors
retry the pytorch test

Praveen · April 14, 2023, 3:37pm

Thanks for the instant reply, let me test and update you. Thanks.

Praveen · May 10, 2023, 8:49pm

@wtempel

I have followed the above steps still, not able to run the “pytorch” test.
I did try couple of time, still not worked.

wtempel · May 11, 2023, 12:24pm

Please can you post the outputs of these commands:

path_to_cryosparc_worker=/path/to/cryosparc_worker # edit this variable definition
which nvcc
nvcc --version
${path_to_cryosparc_worker}/bin/cryosparcw call which nvcc
${path_to_cryosparc_worker}/bin/cryosparcw call nvcc --version
${path_to_cryosparc_worker}/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"
${path_to_cryosparc_worker}/bin/cryosparcw call python -c "import torch; print(torch.__version__)"

Praveen · May 11, 2023, 12:48pm

Hi @wtempel , Thanks for the reply. Please find the below snippet with the command output.

Thanks,
Praveen

Praveen · May 11, 2023, 1:23pm

Hi @wtempel could you please have a look. Thanks.

wtempel · May 11, 2023, 2:04pm

It is possible that cryosparcw install-3dflex has failed. You check the install-3dflex output for details.
To facilitate troubleshooting, you may also consider

updating CryoSPARC to the latest version and patch (v4.2.1+230427 as of May 11, 2023)
removing the old version of 3DFLex dependencies:
/path/to/cryosparc_worker/bin/cryosparcw forcedeps
ensuring there is no output when running (without cryosparcw call)
/usr/bin/which nvcc

installing the current version of 3DFlex dependencies

/path/to/cryosparc_worker/bin/cryosparcw install-3dflex 2>&1 | tee install_3dflex_421.log

Praveen · May 11, 2023, 5:01pm

@wtempel Now it is failing with the below error. Please have a look. Thanks

cryosparcm test workers P1 Using project P1
Specifying gpu test
Enabling PyTorch test
Running worker tests…
2023-05-11 16:59:27,088 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST 2023-05-11 16:59:27,089 WORKER_TEST --test gpu --test-pytorch
log CRITICAL | Worker test results
log CRITICAL | c6i-cpuonly-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error:
log CRITICAL | See P1 J102 for more information
log CRITICAL | g5-singlegpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J101 for more information
log CRITICAL | g4-singlegpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J104 for more information
log CRITICAL | g5-quadgpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J103 for more information
log CRITICAL | g4-quadgpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J105 for more information
log CRITICAL | g5-octagpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J107 for more information
log CRITICAL | g4-octagpu-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J108 for more information
log CRITICAL | g5-singlegpu-spot-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error: No module named ‘pycuda’
log CRITICAL | See P1 J106 for more information
log CRITICAL | c5n-cpuonly-queue
log CRITICAL | ✕ GPU
log CRITICAL | Error:
log CRITICAL | See P1 J109 for more information

wtempel · May 11, 2023, 5:11pm

Job and event logs may provide additional information

This error indicates a failed worker installation.
Please inspect the outputs (stdout, stderr) of the commands
cryosparcw forcedeps
and
cryosparcw install-3dflex

for specific errors.

Praveen · May 11, 2023, 5:37pm

It worked, well. Thanks @wtempel

cryosparcm test workers P1 --test gpu --test-pytorch
Using project P1
Specifying gpu test
Enabling PyTorch test
Running worker tests…
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | Worker test results
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | c6i-cpuonly-queue
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | ✕ GPU
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | Error:
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | See P1 J123 for more information
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | g5-singlegpu-queue
2023-05-11 17:36:18,286 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,288 WORKER_TEST log CRITICAL | g4-singlegpu-queue
2023-05-11 17:36:18,288 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,290 WORKER_TEST log CRITICAL | g5-quadgpu-queue
2023-05-11 17:36:18,290 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,292 WORKER_TEST log CRITICAL | g4-quadgpu-queue
2023-05-11 17:36:18,292 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,294 WORKER_TEST log CRITICAL | g5-octagpu-queue
2023-05-11 17:36:18,294 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,296 WORKER_TEST log CRITICAL | g4-octagpu-queue
2023-05-11 17:36:18,296 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,298 WORKER_TEST log CRITICAL | g5-singlegpu-spot-queue
2023-05-11 17:36:18,298 WORKER_TEST log CRITICAL | ✓ GPU
2023-05-11 17:36:18,299 WORKER_TEST log CRITICAL | c5n-cpuonly-queue
2023-05-11 17:36:18,299 WORKER_TEST log CRITICAL | ✕ GPU
2023-05-11 17:36:18,300 WORKER_TEST log CRITICAL | Error:
2023-05-11 17:36:18,300 WORKER_TEST log CRITICAL | See P1 J126 for more information