CryoSPARC on WSL2

hello @73km and @wtempel,

So is there a walk around this issue?

I am using a laptop with a single GPU. And I am getting the same error running patch-motion correction
My CUDA version is 11.7.

Thanks in advance

Kind regards,
Dmitry

Please can you provide some additional details:

  • Amount of GPU RAM
  • CryoSPARC version
  • how the CUDA toolkit was installed (type of package, Linux user)
  • outputs of these commands:
  • 6 Gb
  • 4.1.2
  • Ubuntu - /usr/local/cuda-11.7
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ cryosparcm env
export "CRYOSPARC_MASTER_HOSTNAME=DESKTOP-D1IHD96."
export "CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin"
export "CRYOSPARC_INSECURE=false"
export "CRYOSPARC_DB_ENABLE_AUTH_FLAG=--auth"
export "CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000"
export "CRYOSPARC_MONGO_CACHE_GB=4"
export "CRYOSPARC_COMMAND_VIS_PORT=39003"
export "CRYOSPARC_MONGO_FCV=3.6"
export "CRYOSPARC_COMMAND_RTP_PORT=39005"
export "CRYOSPARC_HTTP_APP_PORT=39000"
export "CRYOSPARC_ROOT_DIR=/home/david/cryosparc/cryosparc_master"
export "CRYOSPARC_FORCE_USER=false"
export "CRYOSPARC_HOSTNAME_CHECK=DESKTOP-D1IHD96."
export "CRYOSPARC_MONGO_PORT=39001"
export "CRYOSPARC_DB_PATH=/home/david/cryosparc/cryosparc_database"
export "CRYOSPARC_CLICK_WRAP=true"
export "CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT=39006"
export "CRYOSPARC_LIVE_ENABLED=true"
export "CRYOSPARC_COMMAND_CORE_PORT=39002"
export "CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-bef9962287b002917438d05439f545fe.sock"
export "CRYOSPARC_BASE_PORT=39000"
export "CRYOSPARC_LICENSE_ID=XXXXXXXXXXXXXX"
export "CRYOSPARC_HTTP_APP_LEGACY_PORT=39007"
export "CRYOSPARC_DEVELOP=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10"
export "CRYOSPARC_DB_ENABLE_AUTH=true"
export "CRYOSPARC_HEARTBEAT_SECONDS=60"
export "CRYOSPARC_CONDA_ENV=cryosparc_master_env"
export "CRYOSPARC_PROJECT_DIR_PREFIX=CS-"
export "CRYOSPARC_FORCE_HOSTNAME=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000"
export "PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin"
export "LD_LIBRARY_PATH=/usr/local/cuda/lib64:"
export "LD_PRELOAD=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.8.so"
export "PYTHONPATH=/home/david/cryosparc/cryosparc_master"
export "PYTHONNOUSERSITE=true"
export "CONDA_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/conda"
export "CONDA_PREFIX=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env"
export "CONDA_PROMPT_MODIFIER=(cryosparc_master_env)"
export "CONDA_SHLVL=1"
export "CONDA_PYTHON_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/python"
export "CONDA_DEFAULT_ENV=cryosparc_master_env"
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$


(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$



(base) david@DESKTOP-D1IHD96:~$ /usr/local/cuda-11.7/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
(base) david@DESKTOP-D1IHD96:~$


(base) david@DESKTOP-D1IHD96:~$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pycuda'
(base) david@DESKTOP-D1IHD96:~$


(base) david@DESKTOP-D1IHD96:~$ uname -a && free -g && nvidia-smi
Linux DESKTOP-D1IHD96 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
               total        used        free      shared  buff/cache   available
Mem:               7           1           1           0           5           6
Swap:              2           0           1
Thu Jan 26 17:02:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.92.01    Driver Version: 528.02       CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8    13W /  95W |      0MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        68      G   /Xwayland                       N/A      |
+-----------------------------------------------------------------------------+
(base) david@DESKTOP-D1IHD96:~$

ubuntu is running on WLS2

This probably failed because
eval $(/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)
had not been run before that.

1 Like

I now executed this command

(base) david@DESKTOP-D1IHD96:~ eval (/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)

(base) david@DESKTOP-D1IHD96:~$ env | grep PATH

CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_worker/bin

PYTHONPATH=/home/david/cryosparc/cryosparc_worker

CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.7

LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64:/home/david/cryosparc/cryosparc_worker/deps/external/cudnn/lib:/usr/local/cuda/lib64

PATH=/usr/local/cuda-11.7/bin:/home/david/cryosparc/cryosparc_worker/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin

(base) david@DESKTOP-D1IHD96:~$

  1. 6GB GPU-RAM may be too little, may or may not be related to the error you observe.
  2. I suspect that CUDA executables and libraries are present in your PATH and LD_LIBRARY_PATH definitions even before the CryoSPARC environment is loaded. This may interfere with CryoSPARC installation and function, for example, but probably not limited to, cryosparcw install-3dflex
  3. What is the output of
/home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"
  1. I hope we can see
  2. very well can be - but how to check?

(base) david@DESKTOP-D1IHD96:~$ /home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”
(11, 7, 0)

You could install CryoSPARC under a Linux account where you ensure (before installing and running CryoSPARC):

PATH=/usr/bin:/bin
LD_LIBRARY_PATH=""
export PATH LD_LIBRARY_PATH

and then (guide):
/path/to/cryosparc_worker/bin/cryosparcw install-3dflex
This will, among other things, install an independent CUDA environment that may work better under your circumstances.

1 Like

after running the abovementioned command the installation finished with the following error.


Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl Installing collected packages: pycuda Successfully installed pycuda-2020.1 PyTorch not installed correctly, or NVIDIA GPU not detected.

Just in case I tried to re-run the CS patch motion correction but the error remains the same.

What do you think about that?

Kind regards,

is harmless (in the absence of other errors).

Part of me wants to:

  • see the actual errors from the latest failed motion correction event and job logs
  • confirm that no /usr/local/cuda directories are inside $PATH and $LD_LIBRARY_PATH now, or were during installation
1 Like

Yes, both are sent to you privately.

Please can you also post the job log for that job.

1 Like

P1 J26: Patch Motion Correction

Project Details
±----------------±-----------------+
| DETAIL | VALUE |
±----------------±-----------------+
| Title | -------
±----------------±-----------------+
| Description | ------------------- |
±----------------±-----------------+
| ID | P1 |
±----------------±-----------------+
| Created by user | -------------- |
±----------------±-----------------+
Job Details
±----------------±-------------------------------------------------------------------+
| DETAIL | VALUE |
±----------------±-------------------------------------------------------------------+
| Title | New Job J26 |
±----------------±-------------------------------------------------------------------+
| Description | Enter a description. |
±----------------±-------------------------------------------------------------------+
| ID | J26 |
±----------------±-------------------------------------------------------------------+
| Job type | Patch Motion Correction |
±----------------±-------------------------------------------------------------------+
| Status | completed |
±----------------±-------------------------------------------------------------------+
| Created by user | ----------------- |
±----------------±-------------------------------------------------------------------+
| Created | Thu Jan 26 2023 21:19:18 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Queued | Thu Jan 26 2023 21:19:24 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Launched | Thu Jan 26 2023 21:19:25 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Started | Thu Jan 26 2023 21:19:29 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Waiting | - |
±----------------±-------------------------------------------------------------------+
| Killed | - |
±----------------±-------------------------------------------------------------------+
| Completed | Thu Jan 26 2023 21:20:01 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Failed | - |
±----------------±-------------------------------------------------------------------+
| Last accessed | Fri Jan 27 2023 10:16:40 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Size | 72.56 KB |
±----------------±-------------------------------------------------------------------+
Inputs
movies
J4.imported_movies
±--------------------------------+
| MOVIE_BLOB |
±--------------------------------+
| J4.imported_movies.movie_blob.F |
±--------------------------------+
±-----------------------------------+
| MSCOPE_PARAMS |
±-----------------------------------+
| J4.imported_movies.mscope_params.F |
±-----------------------------------+

Parameters
General settings
±------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±--------±--------±-----±---------+
| Make motion diagnostic plots | Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Number of movies to plot | 10 | X | X | |
±------------------------------±--------±--------±-----±---------+
| Only process this many movies | Not Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Low-memory mode | Not set | X | X | |
±------------------------------±--------±--------±-----±---------+
Motion correction
±---------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±---------------------------------±--------±--------±-----±---------+
| Maximum alignment resolution (A) | 5 | | | |
±---------------------------------±--------±--------±-----±---------+
| B-factor during alignment | 500 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Start frame (included, 0-based) | 0 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| End frame (excluded, 0-based) | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Output F-crop factor | 1 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Override e/A^2 | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Allow Variable Dose | Not set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Calibrated smoothing | 0.5 | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots Z | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+

±---------------------------------±--------±--------±-----±---------+
| Override knots Y | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots X | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Random seed (hot pixel replace) | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
Compute settings
±------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±------±--------±-----±---------+
| Number of GPUs to parallelize | 1 | X | X | |
±------------------------------±------±--------±-----±---------+

Outputs
micrographs
±-----------------------------+
| MICROGRAPH_BLOB_NON_DW |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_1X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_2X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------+
| BACKGROUND_BLOB |
±-----------------------+
| J26.exposure.stat_blob |
±-----------------------+
±--------------------+
| RIGID_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±--------------------+
| SPLINE_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
micrographs_incomplete
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
Target
DESKTOP-D1IHD96. (node)

[2023-01-26 21:19:25.65] License is valid.
[2023-01-26 21:19:25.65] Launching job on lane default target DESKTOP-D1IHD96. …
[2023-01-26 21:19:25.68] Running job on master node hostname DESKTOP-D1IHD96.
[2023-01-26 21:19:29.36] [CPU: 195.1 MB] [Avail: 6.23 GB] Job J26 Started
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Working in directory: /mnt/d/Cryosparc/Apo/CS-apo/J26
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Running on lane default
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Resources allocated:
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] Worker: DESKTOP-D1IHD96.
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] CPU : [0, 1, 2, 3, 4, 5]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] GPU : [0]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] RAM : [0]
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] SSD : False
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] --------------------------------------------------------------
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] Importing job module for job type patch_motion_correction_multi…
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] Job ready to run
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Job will process this many movies: 20
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Random seed: 195423761
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] parent process is 1770
[2023-01-26 21:19:31.47] [CPU: 166.0 MB] [Avail: 6.18 GB] Calling CUDA init from 1812
[2023-01-26 21:20:01.48] [CPU: 233.9 MB] [Avail: 6.20 GB] Child process with PID 1812 terminated unexpectedly with exit code

[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] --------------------------------------------------------------
[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] Compiling job outputs…
[2023-01-26 21:20:01.53] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs from input
group movies
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob_non_dw’,
‘micrograph_thumbnail_blob_1x’, ‘micrograph_thumbnail_blob_2x’,
‘micrograph_blob’, ‘background_blob’, ‘rigid_motion’,
‘spline_motion’]
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 0 items
[2023-01-26 21:20:01.56] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 0 items
[2023-01-26 21:20:01.58] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs_incomplete from
input group movies
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob’]
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 20 items
[2023-01-26 21:20:01.60] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 20 items
[2023-01-26 21:20:01.62] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs
[2023-01-26 21:20:01.65] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs_incomplete
[2023-01-26 21:20:01.69] [CPU: 234.1 MB] [Avail: 6.20 GB] Updating job size…
[2023-01-26 21:20:01.71] [CPU: 234.1 MB] [Avail: 6.19 GB] Exporting job and creating csg files…
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] Job complete. Total time 30.41s

Please can you post the job log
/mnt/d/Cryosparc/Apo/CS-apo/J26/job.log

1 Like
================= CRYOSPARCW =======  2023-01-26 21:19:26.183293  =========
Project P1 Job J26
Master DESKTOP-D1IHD96. Port 39002
===========================================================================
========= monitor process now starting main process at 2023-01-26 21:19:26.183316
MAINPROCESS PID 1770
MAIN PID 1770
motioncorrection.run_patch cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
***************************************************************
Running job on hostname %s DESKTOP-D1IHD96.
Allocated Resources :  {'fixed': {'SSD': False}, 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [0], 'RAM': [0]}, 'target': {'cache_path': None, 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 6441926656, 'name': 'NVIDIA GeForce RTX 3060 Laptop GPU'}], 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'monitor_port': None, 'name': 'DESKTOP-D1IHD96.', 'resource_fixed': {'SSD': False}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'GPU': [0], 'RAM': [0]}, 'ssh_str': 'david@DESKTOP-D1IHD96.', 'title': 'Worker node DESKTOP-D1IHD96.', 'type': 'node', 'worker_bin_path': '/home/david/cryosparc/cryosparc_worker/bin/cryosparcw'}}
Process Process-1:1:
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 200, in process_work_simple
    process_setup(proc_idx) # do any setup you want on a per-process basis
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 86, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/__init__.py", line 8, in <module>
    from .engine import *  # noqa
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 9, in init cryosparc_compute.engine.engine
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 12, in init cryosparc_compute.engine.cuda_core
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 27, in <module>
    from . import misc2 as misc
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/misc2.py", line 32, in <module>
    from . import cuda
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cuda.py", line 17, in <module>
    from .cudadrv import *
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cudadrv.py", line 39, in <module>
    raise OSError('CUDA driver library not found')
OSError: CUDA driver library not found
========= sending heartbeat at 2023-01-26 21:19:39.493177
========= sending heartbeat at 2023-01-26 21:19:49.510677
========= sending heartbeat at 2023-01-26 21:19:59.534035
***************************************************************
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
========= main process now complete at 2023-01-26 21:20:01.795774.
========= monitor process now complete at 2023-01-26 21:20:01.798105.

hello @wtempel - any suggestions for this case? Thanks in advance

I am not unsure why this happens. Possible causes to investigate:

  • existing CUDA toolkit files may not have been completely excluded from the environment during
    cryosparcw install-3dflex. What is the output (now) of
    env | grep PATH
    ?
  • WSL may be incompatible with the attempted installation method
1 Like

Output for the command env | grep PATH

(base) david@DESKTOP-D1IHD96:~$ env | grep PATH

LD_LIBRARY_PATH=/usr/local/cuda/lib64:

PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin