Hi!
I am running CryoSPARC in WSL2 Ubuntu 22.04. CryoSPARC run great in WSL2 until version 4.3.1, but updating to CryoSPARC version >= v4.4.0 results in jobs failing due to a CUDA error. The same problem occurs with a fresh CryoSPARC v.4.4.1 install.
I understand that WSL2 is not specifically supported, but I would appreciate any tips on how the issue can be solved!
When running extensive validation, the patch motion correction and patch ctf run correctly. Then I get an error when running the Blob picker job, the traceback of the error and the workstation configuration is below.
Thanks!
Genis
##########################################################################
[CPU: 417.4 MB Avail: 61.43 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 95, in cryosparc_master.cryosparc_compute.run.main
File “cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.py”, line 55, in cryosparc_master.cryosparc_compute.jobs.template_picker_gpu.run.run
File “cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.py”, line 93, in cryosparc_master.cryosparc_compute.jobs.template_picker_gpu.run.do_pick
File “cryosparc_master/cryosparc_compute/jobs/template_picker_gpu/run.py”, line 341, in cryosparc_master.cryosparc_compute.jobs.template_picker_gpu.run.do_pick
File “/home/genis/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py”, line 112, in init
self.handle = gpufft.gpufft_get_plan(
RuntimeError: cuda failure (driver API): cuCtxGetDevice(&device)
→ [unknown error code]
########################################################################
Type: single workstation
Version: v4.4.1
eval $(/home/genis/cryosparc/cryosparc_worker/bin/cryosparcw env)
env | grep PATH
CRYOSPARC_PATH=/home/genis/cryosparc/cryosparc_worker/bin
PYTHONPATH=/home/genis/cryosparc/cryosparc_worker
NUMBA_CUDA_INCLUDE_PATH=/home/genis/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/include
PATH=/home/genis/cryosparc/cryosparc_worker/bin:/home/genis/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/genis/cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/genis/cryosparc/cryosparc_master/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/bin:/mnt/c/Program Files/NVIDIA GPU Computing Toolkit/CUDA/v10.0/libnvvp:/mnt/c/Program Files/AdoptOpenJDK/jdk-11.0.10.9-hotspot/bin:/mnt/c/Anaconda:/mnt/c/Program Files/IMOD/bin:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files (x86)/NVIDIA Corporation/PhysX/Common:/mnt/c/Program Files/Git/cmd:/mnt/c/bin:/mnt/c/msys64/usr/bin:/mnt/c/Program Files/dotnet:/mnt/c/WINDOWS/system32:/mnt/c/WINDOWS:/mnt/c/WINDOWS/System32/Wbem:/mnt/c/WINDOWS/System32/WindowsPowerShell/v1.0:/mnt/c/WINDOWS/System32/OpenSSH:/mnt/c/Program Files/Tailscale:/mnt/c/Program Files/MATLAB/R2023a/bin:/mnt/c/Program Files (x86)/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/Intel/Intel(R) Management Engine Components/DAL:/mnt/c/Program Files/PowerShell/7:/mnt/c/Users/Genis Valentin/AppData/Local/Microsoft/WindowsApps:/mnt/c/Users/Genis Valentin/AppData/Local/Programs/Microsoft VS Code/bin:/mnt/c/Users/Genis Valentin/.dotnet/tools:/mnt/c/Users/Genis Valentin/AppData/Local/Programs/oh-my-posh/bin:/mnt/c/Program Files/Azure Data Studio/bin:/mnt/c/Users/Genis Valentin/AppData/Roaming/Warp:/snap/bin
nvcc --version
Command ‘nvcc’ not found, but can be installed with:
sudo apt install nvidia-cuda-toolkit
python -c “import pycuda.driver; print(pycuda.driver.get_version())”
Traceback (most recent call last):
File “”, line 1, in
ModuleNotFoundError: No module named ‘pycuda’
which python
/home/genis/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin/python
/sbin/ldconfig -p | grep -i cuda
libicudata.so.70 (libc6,x86-64) => /lib/x86_64-linux-gnu/libicudata.so.70
libcudadebugger.so.1 (libc6,x86-64) => /usr/lib/wsl/lib/libcudadebugger.so.1
libcuda.so.1 (libc6,x86-64) => /usr/lib/wsl/lib/libcuda.so.1
uname -a
Linux DESKTOP-A45DKHA 5.15.133.1-microsoft-standard-WSL2 #1 SMP Thu Oct 5 21:02:42 UTC 2023 x86_64 x86_64 x86_64 GNU/Linux
free -g
total used free shared buff/cache available
Mem: 62 25 2 0 34 36
Swap: 16 3 12
nvidia-smi
Thu Dec 7 13:03:06 2023
±--------------------------------------------------------------------------------------+
| NVIDIA-SMI 545.29.01 Driver Version: 546.01 CUDA Version: 12.3 |
|-----------------------------------------±---------------------±---------------------+
| GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|=========================================+======================+======================|
| 0 Quadro RTX 5000 On | 00000000:4F:00.0 Off | Off |
| 33% 29C P8 15W / 230W | 0MiB / 16384MiB | 0% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
| 1 Quadro RTX 5000 On | 00000000:91:00.0 Off | Off |
| 33% 30C P8 8W / 230W | 902MiB / 16384MiB | 9% Default |
| | | N/A |
±----------------------------------------±---------------------±---------------------+
±--------------------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=======================================================================================|
| 0 N/A N/A 14 G /Xwayland N/A |
| 0 N/A N/A 32 G /Xwayland N/A |
| 1 N/A N/A 14 G /Xwayland N/A |
| 1 N/A N/A 32 G /Xwayland N/A |
±--------------------------------------------------------------------------------------+