CryoSPARC on WSL2

What commands did you use, including any export commands you may have executed prior to installation?
What are the outputs of

env | grep PATH
echo '==============='
/path/to/cryosparc_worker/bin/cryosparcw env | grep -v CRYOSPARC_LICENSE
echo '==============='
find /usr -name libcuda.so

?

1 Like

@wtempel

here are the outputs

(base) david@DESKTOP-D1IHD96:~$ env | grep PATH

echo ‘===============’

/path/to/cryosparc_worker/bin/cryosparcw env | grep -v CRYOSPARC_LICENSE

echo ‘===============’

find /usr -name libcuda.so

LD_LIBRARY_PATH=/usr/local/cuda/lib64:

PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin

===============

-bash: /path/to/cryosparc_worker/bin/cryosparcw: No such file or directory

===============

/usr/local/cuda-11.7/targets/x86_64-linux/lib/stubs/libcuda.so

/usr/lib/wsl/lib/libcuda.so

  1. The outputs indicate that a CUDA toolkit-related directory still present in the “pre-CryoSPARC” LD_LIBRARY_PATH environment variable and may interfere with CryoSPARC installation and operation. See, for example, here for a general discussion of environment variables.
  2. I have not tested CryoSPARC on WSL2, but additional, compared to Linux on bare-metal, considerations may apply. Here are some links on running CUDA applications (not CryoSPARC specifically) on WSL2, which may provide some guidance in your further troubleshooting:
    Enabling GPU acceleration on Ubuntu on WSL2 with the NVIDIA CUDA Platform | Ubuntu
    How to set up CUDA and PyTorch in Ubuntu 22.04 on WSL2 - Willem's Fizzy Logic

This section of the PATH definition indicates

  1. that /home/david/cryosparc/cryosparc_master/bin has been added to the PATH multiple times. You may be able to avoid multiple occurrences by ensuring there is no more than one occurrence of
    export PATH=/home/david/cryosparc/cryosparc_master/bin:$PATH
    in ~/.bashrc.
  2. the presence of /usr/local/cuda/bin in your “pre-CryoSPARC” PATH, which may interfere with CryoSPARC installation and operation.
1 Like

hello @wtempel.
I still can not understand why does cuda not running.
the error keeps being the same after removing multiple occurrences.

Can it be an Idea to have this thread under the topic CS on WSL2?
Maybe someone can share the successful installation experience?

Kind regards,
Dmitry

Let’s try it. I have moved this discussion to a new topic.

1 Like

hello @wtempel during investigation I have found this topic.

No module named ‘pycuda’ after 4.1 upgrade/3DFlex install - Install, Configure and Update - CryoSPARC Discuss

The error looks very similar to mine (when try to run 2D classification).

My current error when running 2D classification →
[CPU: 195.1 MB Avail: 11.76 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 83, in cryosparc_compute.run.main
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/jobregister.py”, line 442, in get_run_function
runmod = importlib.import_module(“…”+modname, name)
File “/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 1174, in exec_module
File “”, line 219, in _call_with_frames_removed
File “cryosparc_master/cryosparc_compute/jobs/class2D/run.py”, line 13, in init cryosparc_compute.jobs.class2D.run
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 4, in init cryosparc_compute.engine.cuda_core
ModuleNotFoundError: No module named ‘pycuda’

What do you think?

Could you please explain the solution? “cryosparcw forcedeps` fixes this issue.”

Thank you

Dmitry

The command
/path/to/cryosparc_worker/cryosparcw forcedeps
configures CryoSPARC dependencies.
I have not tested whether this command will result in a functional worker installation under WSL2.

1 Like

@wtempel - interesting.
it seems that I don’t have it.

david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker$ cryosparcw forcedeps
cryosparcw: command not found
david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker$ ./cryosparcw forcedeps
-bash: ./cryosparcw: No such file or directory

all I have inside the cryosparc_worker folder similar are

david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker$ cry
cryosparcm cryptcatsvc.dll cryptext.dll cryptsp.dll cryptui.dll
crypt32.dll cryptdlg.dll cryptnet.dll cryptsvc.dll cryptuiwizard.dll
cryptbase.dll cryptdll.dll cryptngc.dll crypttpmeksvc.dll cryptxml.dll
david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker$ cryosparcm

Is it possible?

Sincerely,
D

You need to specify the full path to this command. In your case, this would be

/home/david/cryosparc/cryosparc_worker/bin/cryosparcw forcedeps
1 Like

The error remains but got modified (for 2D classification)

error before - cryosparcw forcedeps

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 83, in cryosparc_compute.run.main
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/jobregister.py”, line 442, in get_run_function
runmod = importlib.import_module(“…”+modname, name)
File “/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 1174, in exec_module
File “”, line 219, in _call_with_frames_removed
File “cryosparc_master/cryosparc_compute/jobs/class2D/run.py”, line 13, in init cryosparc_compute.jobs.class2D.run
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 4, in init cryosparc_compute.engine.cuda_core
ModuleNotFoundError: No module named ‘pycuda’

==================================================================
error after using - cryosparcw forcedeps

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 83, in cryosparc_compute.run.main
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/jobregister.py”, line 442, in get_run_function
runmod = importlib.import_module(“…”+modname, name)
File “/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/importlib/init.py”, line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File “”, line 1014, in _gcd_import
File “”, line 991, in _find_and_load
File “”, line 975, in _find_and_load_unlocked
File “”, line 671, in _load_unlocked
File “”, line 1174, in exec_module
File “”, line 219, in _call_with_frames_removed
File “cryosparc_master/cryosparc_compute/jobs/class2D/run.py”, line 13, in init cryosparc_compute.jobs.class2D.run
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/init.py”, line 8, in
from .engine import * # noqa
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 9, in init cryosparc_compute.engine.engine
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 12, in init cryosparc_compute.engine.cuda_core
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py”, line 27, in
from . import misc2 as misc
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/misc2.py”, line 32, in
from . import cuda
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cuda.py”, line 17, in
from .cudadrv import *
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cudadrv.py”, line 39, in
raise OSError(‘CUDA driver library not found’)
OSError: CUDA driver library not found

Hello everyone,

This seemed to be a problem with the CUDA driver and not with the toolkit. The problem is that cudadrv.py tries to locate the runtime library libcuda.so file and this is not found because WSL/WSL2 has a different location for it.

The solution was to append “/usr/lib/wsl/lib” to LD_LIBRARY_PATH (either in .bashrc or prior to execution should work). After that, the cuda driver management inside cryosparc can detect the library and everything works just fine both inside Scipion and in standalone.

The long term solution could be for cryosparc to detect it’s running inside of WSL2 and adding /usr/lib/wsl/lib to the path automatically, I guess.

Cheers,
Mikel.

2 Likes

thank you @Ratolon and @wtempel . The issue is solved!

Kind regards,
Dmitry

hello @wtempel,

I got a new error running 3D Homogenoius Refinement.

=================================================

Traceback (most recent call last):
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2061, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1084, in cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 346, in cryosparc_compute.engine.engine.EngineThread.compute_error
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 337, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory

====================================================

running the other commands

– lscpu && free -g && uname -a
– sudo dmidecode --type memory

I got the following results:

david@DESKTOP-D1IHD96:~$ lscpu && free -g && uname -a
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 16
On-line CPU(s) list: 0-15
Vendor ID: GenuineIntel
Model name: Intel(R) Core™ i7-10700 CPU @ 2.90GHz
CPU family: 6
Model: 165
Thread(s) per core: 2
Core(s) per socket: 8
Socket(s): 1
Stepping: 5
BogoMIPS: 5807.99
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse ss
e2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc arch_perfmon rep_good nopl xtopology cpuid p
ni pclmulqdq ssse3 fma cx16 pdcm pcid sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand hype
rvisor lahf_lm abm 3dnowprefetch invpcid_single ssbd ibrs ibpb stibp ibrs_enhanced fsgsbase bmi
1 avx2 smep bmi2 erms invpcid rdseed adx smap clflushopt xsaveopt xsavec xgetbv1 xsaves flush_l
1d arch_capabilities
Virtualization features:
Hypervisor vendor: Microsoft
Virtualization type: full
Caches (sum of all):
L1d: 256 KiB (8 instances)
L1i: 256 KiB (8 instances)
L2: 2 MiB (8 instances)
L3: 16 MiB (1 instance)
Vulnerabilities:
Itlb multihit: KVM: Mitigation: VMX unsupported
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Vulnerable: Clear CPU buffers attempted, no microcode; SMT Host state unknown
Retbleed: Mitigation; Enhanced IBRS
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl and seccomp
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer sanitization
Spectre v2: Mitigation; Enhanced IBRS, IBPB conditional, RSB filling, PBRSB-eIBRS SW sequence
Srbds: Unknown: Dependent on hypervisor status
Tsx async abort: Not affected
total used free shared buff/cache available
Mem: 13 1 6 0 5 12
Swap: 0 0 0
Linux DESKTOP-D1IHD96 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

david@DESKTOP-D1IHD96:~$ sudo dmidecode --type memory
[sudo] password for david:

dmidecode 3.3

Scanning /dev/mem for entry point.

No SMBIOS nor DMI entry point found, sorry.

Is there any way to fix that?

Thank you.
Sincerely,
D

This virtual(?) machine may not have enough system RAM for this job. We recommend a minimum of 32 GB system RAM (per GPU, guide).

1 Like

By default, WSL2 assigns a stupidly small amount of system memory. Well, it’s OK if you just want to “play” in a Linux VM but is far too little for anything serious. The Microsoft documentation shows that on recent builds, it will assign whichever is the smaller of 50% of system RAM or 8GB. So on a box with 64GB of RAM, my WSL2 VMs are assigned 8GB by default, which is nowhere near enough for CryoSPARC.

I would recommend following the instructions on this Microsoft page, although I would recommend assigning at least 40GB, probably more like 48GB to any WSL2 VM you want to run CryoSPARC in.

1 Like

hello @rbs_sci ,

Thank you for your explanations and suggestions! yes, indeed - originally WSL2 has setting up only 8GB of memory by default. So I manage to increase the usage visible by WSL2 to 14GB.

But as we see this is sill to little :slight_smile: I look forward for Ram upgrade - up to 64GB.
Has to be enough, isn’t it?

Kind regards,
Dmitry

I think you '@'ed the wrong person. :wink:

Yes, best bet is to upgrade to 64GB of RAM (or even 128GB) to give the WSL2 VM some more breathing room.

1 Like

thank you @rbs_sci, for useful recommendations.
Let see how it will play with more ram on board.

p.s.
the name was corrected accordingly.
Thank you for pointing out. :sweat_smile:

1 Like

dear @wtempel and @rbs_sci ,

I have upgraded the RAM to 64GB and allocated about 54GB for WSL2.

But the error I had before with not enough memory is still there.


Traceback (most recent call last):
File “/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/runcommon.py”, line 2061, in run_with_except_hook
run_old(*args, **kw)
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 1084, in cryosparc_compute.engine.engine.process.work
File “cryosparc_master/cryosparc_compute/engine/engine.py”, line 346, in cryosparc_compute.engine.engine.EngineThread.compute_error
File “cryosparc_master/cryosparc_compute/engine/cuda_core.py”, line 337, in cryosparc_compute.engine.cuda_core.EngineBaseThread.ensure_allocated
pycuda._driver.MemoryError: cuMemHostAlloc failed: out of memory


Can it be that CS does not see the RAM upgrade?
How to set it then?

Thank you in advance.

Sincerely,
Dmitry

Very likely. Please see Not all the RAM available after update to CryoSPARC4 for a suggestion.

1 Like