CryoSPARC on WSL2

Dmitry · January 26, 2023, 3:06pm

So is there a walk around this issue?

I am using a laptop with a single GPU. And I am getting the same error running patch-motion correction
My CUDA version is 11.7.

Thanks in advance

Kind regards,
Dmitry

wtempel · January 26, 2023, 3:51pm

Please can you provide some additional details:

Amount of GPU RAM
CryoSPARC version
how the CUDA toolkit was installed (type of package, Linux user)
outputs of these commands:

Dmitry · January 26, 2023, 3:58pm

6 Gb
4.1.2
Ubuntu - /usr/local/cuda-11.7

Dmitry · January 26, 2023, 4:02pm

(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ cryosparcm env
export "CRYOSPARC_MASTER_HOSTNAME=DESKTOP-D1IHD96."
export "CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin"
export "CRYOSPARC_INSECURE=false"
export "CRYOSPARC_DB_ENABLE_AUTH_FLAG=--auth"
export "CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000"
export "CRYOSPARC_MONGO_CACHE_GB=4"
export "CRYOSPARC_COMMAND_VIS_PORT=39003"
export "CRYOSPARC_MONGO_FCV=3.6"
export "CRYOSPARC_COMMAND_RTP_PORT=39005"
export "CRYOSPARC_HTTP_APP_PORT=39000"
export "CRYOSPARC_ROOT_DIR=/home/david/cryosparc/cryosparc_master"
export "CRYOSPARC_FORCE_USER=false"
export "CRYOSPARC_HOSTNAME_CHECK=DESKTOP-D1IHD96."
export "CRYOSPARC_MONGO_PORT=39001"
export "CRYOSPARC_DB_PATH=/home/david/cryosparc/cryosparc_database"
export "CRYOSPARC_CLICK_WRAP=true"
export "CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT=39006"
export "CRYOSPARC_LIVE_ENABLED=true"
export "CRYOSPARC_COMMAND_CORE_PORT=39002"
export "CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-bef9962287b002917438d05439f545fe.sock"
export "CRYOSPARC_BASE_PORT=39000"
export "CRYOSPARC_LICENSE_ID=XXXXXXXXXXXXXX"
export "CRYOSPARC_HTTP_APP_LEGACY_PORT=39007"
export "CRYOSPARC_DEVELOP=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10"
export "CRYOSPARC_DB_ENABLE_AUTH=true"
export "CRYOSPARC_HEARTBEAT_SECONDS=60"
export "CRYOSPARC_CONDA_ENV=cryosparc_master_env"
export "CRYOSPARC_PROJECT_DIR_PREFIX=CS-"
export "CRYOSPARC_FORCE_HOSTNAME=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000"
export "PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin"
export "LD_LIBRARY_PATH=/usr/local/cuda/lib64:"
export "LD_PRELOAD=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.8.so"
export "PYTHONPATH=/home/david/cryosparc/cryosparc_master"
export "PYTHONNOUSERSITE=true"
export "CONDA_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/conda"
export "CONDA_PREFIX=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env"
export "CONDA_PROMPT_MODIFIER=(cryosparc_master_env)"
export "CONDA_SHLVL=1"
export "CONDA_PYTHON_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/python"
export "CONDA_DEFAULT_ENV=cryosparc_master_env"
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$


(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$



(base) david@DESKTOP-D1IHD96:~$ /usr/local/cuda-11.7/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
(base) david@DESKTOP-D1IHD96:~$


(base) david@DESKTOP-D1IHD96:~$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pycuda'
(base) david@DESKTOP-D1IHD96:~$


(base) david@DESKTOP-D1IHD96:~$ uname -a && free -g && nvidia-smi
Linux DESKTOP-D1IHD96 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
               total        used        free      shared  buff/cache   available
Mem:               7           1           1           0           5           6
Swap:              2           0           1
Thu Jan 26 17:02:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.92.01    Driver Version: 528.02       CUDA Version: 12.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  On   | 00000000:01:00.0 Off |                  N/A |
| N/A   46C    P8    13W /  95W |      0MiB /  6144MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A        68      G   /Xwayland                       N/A      |
+-----------------------------------------------------------------------------+
(base) david@DESKTOP-D1IHD96:~$

Dmitry · January 26, 2023, 4:04pm

ubuntu is running on WLS2

wtempel · January 26, 2023, 4:16pm

Dmitry:

(base) david@DESKTOP-D1IHD96:~$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pycuda'

This probably failed because
eval $(/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)
had not been run before that.

Dmitry · January 26, 2023, 4:30pm

I now executed this command

(base) david@DESKTOP-D1IHD96:~ eval (/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)

(base) david@DESKTOP-D1IHD96:~$ env | grep PATH

CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_worker/bin

PYTHONPATH=/home/david/cryosparc/cryosparc_worker

CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.7

LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64:/home/david/cryosparc/cryosparc_worker/deps/external/cudnn/lib:/usr/local/cuda/lib64

PATH=/usr/local/cuda-11.7/bin:/home/david/cryosparc/cryosparc_worker/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin

(base) david@DESKTOP-D1IHD96:~$

wtempel · January 26, 2023, 4:43pm

6GB GPU-RAM may be too little, may or may not be related to the error you observe.
I suspect that CUDA executables and libraries are present in your PATH and LD_LIBRARY_PATH definitions even before the CryoSPARC environment is loaded. This may interfere with CryoSPARC installation and function, for example, but probably not limited to, cryosparcw install-3dflex
What is the output of

/home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"

Dmitry · January 26, 2023, 4:45pm

I hope we can see
very well can be - but how to check?

(base) david@DESKTOP-D1IHD96:~$ /home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”
(11, 7, 0)

wtempel · January 26, 2023, 4:59pm

You could install CryoSPARC under a Linux account where you ensure (before installing and running CryoSPARC):

PATH=/usr/bin:/bin
LD_LIBRARY_PATH=""
export PATH LD_LIBRARY_PATH

and then (guide):
/path/to/cryosparc_worker/bin/cryosparcw install-3dflex
This will, among other things, install an independent CUDA environment that may work better under your circumstances.

Dmitry · January 26, 2023, 9:11pm

after running the abovementioned command the installation finished with the following error.

…
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl Installing collected packages: pycuda Successfully installed pycuda-2020.1 PyTorch not installed correctly, or NVIDIA GPU not detected.

Just in case I tried to re-run the CS patch motion correction but the error remains the same.

What do you think about that?

Kind regards,

wtempel · January 26, 2023, 11:16pm

is harmless (in the absence of other errors).

Part of me wants to:

see the actual errors from the latest failed motion correction event and job logs
confirm that no /usr/local/cuda directories are inside $PATH and $LD_LIBRARY_PATH now, or were during installation

Dmitry · January 27, 2023, 9:38am

Yes, both are sent to you privately.

wtempel · January 27, 2023, 1:41pm

Please can you also post the job log for that job.

Dmitry · January 27, 2023, 1:49pm

P1 J26: Patch Motion Correction

Project Details
±----------------±-----------------+
| DETAIL | VALUE |
±----------------±-----------------+
| Title | -------
±----------------±-----------------+
| Description | ------------------- |
±----------------±-----------------+
| ID | P1 |
±----------------±-----------------+
| Created by user | -------------- |
±----------------±-----------------+
Job Details
±----------------±-------------------------------------------------------------------+
| DETAIL | VALUE |
±----------------±-------------------------------------------------------------------+
| Title | New Job J26 |
±----------------±-------------------------------------------------------------------+
| Description | Enter a description. |
±----------------±-------------------------------------------------------------------+
| ID | J26 |
±----------------±-------------------------------------------------------------------+
| Job type | Patch Motion Correction |
±----------------±-------------------------------------------------------------------+
| Status | completed |
±----------------±-------------------------------------------------------------------+
| Created by user | ----------------- |
±----------------±-------------------------------------------------------------------+
| Created | Thu Jan 26 2023 21:19:18 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Queued | Thu Jan 26 2023 21:19:24 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Launched | Thu Jan 26 2023 21:19:25 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Started | Thu Jan 26 2023 21:19:29 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Waiting | - |
±----------------±-------------------------------------------------------------------+
| Killed | - |
±----------------±-------------------------------------------------------------------+
| Completed | Thu Jan 26 2023 21:20:01 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Failed | - |
±----------------±-------------------------------------------------------------------+
| Last accessed | Fri Jan 27 2023 10:16:40 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Size | 72.56 KB |
±----------------±-------------------------------------------------------------------+
Inputs
movies
J4.imported_movies
±--------------------------------+
| MOVIE_BLOB |
±--------------------------------+
| J4.imported_movies.movie_blob.F |
±--------------------------------+
±-----------------------------------+
| MSCOPE_PARAMS |
±-----------------------------------+
| J4.imported_movies.mscope_params.F |
±-----------------------------------+

Parameters
General settings
±------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±--------±--------±-----±---------+
| Make motion diagnostic plots | Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Number of movies to plot | 10 | X | X | |
±------------------------------±--------±--------±-----±---------+
| Only process this many movies | Not Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Low-memory mode | Not set | X | X | |
±------------------------------±--------±--------±-----±---------+
Motion correction
±---------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±---------------------------------±--------±--------±-----±---------+
| Maximum alignment resolution (A) | 5 | | | |
±---------------------------------±--------±--------±-----±---------+
| B-factor during alignment | 500 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Start frame (included, 0-based) | 0 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| End frame (excluded, 0-based) | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Output F-crop factor | 1 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Override e/A^2 | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Allow Variable Dose | Not set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Calibrated smoothing | 0.5 | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots Z | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+

±---------------------------------±--------±--------±-----±---------+
| Override knots Y | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots X | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Random seed (hot pixel replace) | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
Compute settings
±------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±------±--------±-----±---------+
| Number of GPUs to parallelize | 1 | X | X | |
±------------------------------±------±--------±-----±---------+

Outputs
micrographs
±-----------------------------+
| MICROGRAPH_BLOB_NON_DW |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_1X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_2X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------+
| BACKGROUND_BLOB |
±-----------------------+
| J26.exposure.stat_blob |
±-----------------------+
±--------------------+
| RIGID_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±--------------------+
| SPLINE_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
micrographs_incomplete
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
Target
DESKTOP-D1IHD96. (node)

[2023-01-26 21:19:25.65] License is valid.
[2023-01-26 21:19:25.65] Launching job on lane default target DESKTOP-D1IHD96. …
[2023-01-26 21:19:25.68] Running job on master node hostname DESKTOP-D1IHD96.
[2023-01-26 21:19:29.36] [CPU: 195.1 MB] [Avail: 6.23 GB] Job J26 Started
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Working in directory: /mnt/d/Cryosparc/Apo/CS-apo/J26
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Running on lane default
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Resources allocated:
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] Worker: DESKTOP-D1IHD96.
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] CPU : [0, 1, 2, 3, 4, 5]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] GPU : [0]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] RAM : [0]
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] SSD : False
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] --------------------------------------------------------------
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] Importing job module for job type patch_motion_correction_multi…
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] Job ready to run
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Job will process this many movies: 20
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Random seed: 195423761
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] parent process is 1770
[2023-01-26 21:19:31.47] [CPU: 166.0 MB] [Avail: 6.18 GB] Calling CUDA init from 1812
[2023-01-26 21:20:01.48] [CPU: 233.9 MB] [Avail: 6.20 GB] Child process with PID 1812 terminated unexpectedly with exit code

[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] --------------------------------------------------------------
[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] Compiling job outputs…
[2023-01-26 21:20:01.53] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs from input
group movies
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob_non_dw’,
‘micrograph_thumbnail_blob_1x’, ‘micrograph_thumbnail_blob_2x’,
‘micrograph_blob’, ‘background_blob’, ‘rigid_motion’,
‘spline_motion’]
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 0 items
[2023-01-26 21:20:01.56] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 0 items
[2023-01-26 21:20:01.58] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs_incomplete from
input group movies
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob’]
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 20 items
[2023-01-26 21:20:01.60] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 20 items
[2023-01-26 21:20:01.62] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs
[2023-01-26 21:20:01.65] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs_incomplete
[2023-01-26 21:20:01.69] [CPU: 234.1 MB] [Avail: 6.20 GB] Updating job size…
[2023-01-26 21:20:01.71] [CPU: 234.1 MB] [Avail: 6.19 GB] Exporting job and creating csg files…
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] Job complete. Total time 30.41s

wtempel · January 27, 2023, 2:08pm

Please can you post the job log
/mnt/d/Cryosparc/Apo/CS-apo/J26/job.log

Dmitry · January 27, 2023, 3:36pm

================= CRYOSPARCW =======  2023-01-26 21:19:26.183293  =========
Project P1 Job J26
Master DESKTOP-D1IHD96. Port 39002
===========================================================================
========= monitor process now starting main process at 2023-01-26 21:19:26.183316
MAINPROCESS PID 1770
MAIN PID 1770
motioncorrection.run_patch cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
***************************************************************
Running job on hostname %s DESKTOP-D1IHD96.
Allocated Resources :  {'fixed': {'SSD': False}, 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [0], 'RAM': [0]}, 'target': {'cache_path': None, 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 6441926656, 'name': 'NVIDIA GeForce RTX 3060 Laptop GPU'}], 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'monitor_port': None, 'name': 'DESKTOP-D1IHD96.', 'resource_fixed': {'SSD': False}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'GPU': [0], 'RAM': [0]}, 'ssh_str': 'david@DESKTOP-D1IHD96.', 'title': 'Worker node DESKTOP-D1IHD96.', 'type': 'node', 'worker_bin_path': '/home/david/cryosparc/cryosparc_worker/bin/cryosparcw'}}
Process Process-1:1:
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
    self.run()
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 108, in run
    self._target(*self._args, **self._kwargs)
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 200, in process_work_simple
    process_setup(proc_idx) # do any setup you want on a per-process basis
  File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 86, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/__init__.py", line 8, in <module>
    from .engine import *  # noqa
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 9, in init cryosparc_compute.engine.engine
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 12, in init cryosparc_compute.engine.cuda_core
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 27, in <module>
    from . import misc2 as misc
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/misc2.py", line 32, in <module>
    from . import cuda
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cuda.py", line 17, in <module>
    from .cudadrv import *
  File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cudadrv.py", line 39, in <module>
    raise OSError('CUDA driver library not found')
OSError: CUDA driver library not found
========= sending heartbeat at 2023-01-26 21:19:39.493177
========= sending heartbeat at 2023-01-26 21:19:49.510677
========= sending heartbeat at 2023-01-26 21:19:59.534035
***************************************************************
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
    send_bytes(obj)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
    self._send_bytes(m[offset:offset + size])
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
    self._send(header + buf)
  File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
    n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
========= main process now complete at 2023-01-26 21:20:01.795774.
========= monitor process now complete at 2023-01-26 21:20:01.798105.

Dmitry · January 30, 2023, 11:42am

hello @wtempel - any suggestions for this case? Thanks in advance

wtempel · January 31, 2023, 3:14pm

I am not unsure why this happens. Possible causes to investigate:

existing CUDA toolkit files may not have been completely excluded from the environment during
cryosparcw install-3dflex. What is the output (now) of
env | grep PATH
?
WSL may be incompatible with the attempted installation method

Dmitry · January 31, 2023, 8:09pm

Output for the command env | grep PATH

(base) david@DESKTOP-D1IHD96:~$ env | grep PATH

LD_LIBRARY_PATH=/usr/local/cuda/lib64:

PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin