No module named 'pycuda' after 4.1 upgrade/3DFlex install

Hello, we’ve started to encounter a bug in 3D classification after upgrading to 4.1:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 80, in cryosparc_compute.run.main
  File "/home/cryosparc/cryosparc_worker/cryosparc_compute/jobs/jobregister.py", line 443, in get_run_function
    runmod = importlib.import_module(".."+modname, __name__)
  File "/home/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.7/importlib/__init__.py", line 127, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
  File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
  File "<frozen importlib._bootstrap>", line 983, in _find_and_load
  File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 1050, in exec_module
  File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
  File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 18, in init cryosparc_compute.jobs.class3D.run
  File "/home/cryosparc/cryosparc_worker/cryosparc_compute/engine/__init__.py", line 8, in <module>
    from .engine import *  # noqa
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 9, in init cryosparc_compute.engine.engine
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 4, in init cryosparc_compute.engine.cuda_core
ModuleNotFoundError: No module named 'pycuda'

This error occurs on both workers, which I’ve installed the 3DFlex requirements on as well. We’re running Ubuntu 20.04.

cryosparc@troll:/home/users/posert$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/troll/scratch/cryosparc/cryosparc_master
Current cryoSPARC version: v4.1.0
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 1197489, uptime 1 day, 0:00:52
app_api                          RUNNING   pid 1197522, uptime 1 day, 0:00:50
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 1197162, uptime 1 day, 0:01:04
command_rtp                      RUNNING   pid 1197232, uptime 1 day, 0:00:56
command_vis                      RUNNING   pid 1197216, uptime 1 day, 0:00:58
database                         RUNNING   pid 1197043, uptime 1 day, 0:01:08

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="{redacted}"
export CRYOSPARC_MASTER_HOSTNAME="troll"
export CRYOSPARC_DB_PATH="/troll/scratch/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

Worker 1, with 4x2080

cryosparc@troll:/home/users/posert$ nvidia-smi
Tue Dec 13 10:55:58 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+

Worker 2, with 4x 2080 Ti

cryosparc@askeladden:~$ nvidia-smi
Tue Dec 13 10:56:57 2022
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 495.29.05    Driver Version: 495.29.05    CUDA Version: 11.5     |
|-------------------------------+----------------------+----------------------+

More info: I think this is a result of the 3D Flex requirements installation. It actually affects all jobs, not just 3D classification, and I ran a few local refinements after upgrading to 4.1 and before installing 3D Flex reqs.

cryosparcw forcedeps fixes this issue.

3 Likes

Fixed by 3DFlex Dependencies; Building pycuda

We have just released CryoSPARC v4.1.1 to address this issue.