So is there a walk around this issue?
I am using a laptop with a single GPU. And I am getting the same error running patch-motion correction
My CUDA version is 11.7.
Thanks in advance
Kind regards,
Dmitry
So is there a walk around this issue?
I am using a laptop with a single GPU. And I am getting the same error running patch-motion correction
My CUDA version is 11.7.
Thanks in advance
Kind regards,
Dmitry
Please can you provide some additional details:
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ cryosparcm env
export "CRYOSPARC_MASTER_HOSTNAME=DESKTOP-D1IHD96."
export "CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin"
export "CRYOSPARC_INSECURE=false"
export "CRYOSPARC_DB_ENABLE_AUTH_FLAG=--auth"
export "CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000"
export "CRYOSPARC_MONGO_CACHE_GB=4"
export "CRYOSPARC_COMMAND_VIS_PORT=39003"
export "CRYOSPARC_MONGO_FCV=3.6"
export "CRYOSPARC_COMMAND_RTP_PORT=39005"
export "CRYOSPARC_HTTP_APP_PORT=39000"
export "CRYOSPARC_ROOT_DIR=/home/david/cryosparc/cryosparc_master"
export "CRYOSPARC_FORCE_USER=false"
export "CRYOSPARC_HOSTNAME_CHECK=DESKTOP-D1IHD96."
export "CRYOSPARC_MONGO_PORT=39001"
export "CRYOSPARC_DB_PATH=/home/david/cryosparc/cryosparc_database"
export "CRYOSPARC_CLICK_WRAP=true"
export "CRYOSPARC_HTTP_LIVEAPP_LEGACY_PORT=39006"
export "CRYOSPARC_LIVE_ENABLED=true"
export "CRYOSPARC_COMMAND_CORE_PORT=39002"
export "CRYOSPARC_SUPERVISOR_SOCK_FILE=/tmp/cryosparc-supervisor-bef9962287b002917438d05439f545fe.sock"
export "CRYOSPARC_BASE_PORT=39000"
export "CRYOSPARC_LICENSE_ID=XXXXXXXXXXXXXX"
export "CRYOSPARC_HTTP_APP_LEGACY_PORT=39007"
export "CRYOSPARC_DEVELOP=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10"
export "CRYOSPARC_DB_ENABLE_AUTH=true"
export "CRYOSPARC_HEARTBEAT_SECONDS=60"
export "CRYOSPARC_CONDA_ENV=cryosparc_master_env"
export "CRYOSPARC_PROJECT_DIR_PREFIX=CS-"
export "CRYOSPARC_FORCE_HOSTNAME=false"
export "CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000"
export "PATH=/home/david/cryosparc/cryosparc_master/deps/external/mongodb/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin:/home/david/cryosparc/cryosparc_master/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin"
export "LD_LIBRARY_PATH=/usr/local/cuda/lib64:"
export "LD_PRELOAD=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.8.so"
export "PYTHONPATH=/home/david/cryosparc/cryosparc_master"
export "PYTHONNOUSERSITE=true"
export "CONDA_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/conda"
export "CONDA_PREFIX=/home/david/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env"
export "CONDA_PROMPT_MODIFIER=(cryosparc_master_env)"
export "CONDA_SHLVL=1"
export "CONDA_PYTHON_EXE=/home/david/cryosparc/cryosparc_master/deps/anaconda/bin/python"
export "CONDA_DEFAULT_ENV=cryosparc_master_env"
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin
(base) david@DESKTOP-D1IHD96:~/cryosparc/cryosparc_worker/bin$
(base) david@DESKTOP-D1IHD96:~$ /usr/local/cuda-11.7/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2022 NVIDIA Corporation
Built on Tue_May__3_18:49:52_PDT_2022
Cuda compilation tools, release 11.7, V11.7.64
Build cuda_11.7.r11.7/compiler.31294372_0
(base) david@DESKTOP-D1IHD96:~$
(base) david@DESKTOP-D1IHD96:~$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
Traceback (most recent call last):
File "<string>", line 1, in <module>
ModuleNotFoundError: No module named 'pycuda'
(base) david@DESKTOP-D1IHD96:~$
(base) david@DESKTOP-D1IHD96:~$ uname -a && free -g && nvidia-smi
Linux DESKTOP-D1IHD96 5.15.79.1-microsoft-standard-WSL2 #1 SMP Wed Nov 23 01:01:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 7 1 1 0 5 6
Swap: 2 0 1
Thu Jan 26 17:02:40 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 527.92.01 Driver Version: 528.02 CUDA Version: 12.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... On | 00000000:01:00.0 Off | N/A |
| N/A 46C P8 13W / 95W | 0MiB / 6144MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 68 G /Xwayland N/A |
+-----------------------------------------------------------------------------+
(base) david@DESKTOP-D1IHD96:~$
ubuntu is running on WLS2
This probably failed because
eval $(/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)
had not been run before that.
I now executed this command
(base) david@DESKTOP-D1IHD96:~ eval (/home/david/cryosparc/cryosparc_worker/bin/cryosparcw env)
(base) david@DESKTOP-D1IHD96:~$ env | grep PATH
CRYOSPARC_PATH=/home/david/cryosparc/cryosparc_worker/bin
PYTHONPATH=/home/david/cryosparc/cryosparc_worker
CRYOSPARC_CUDA_PATH=/usr/local/cuda-11.7
LD_LIBRARY_PATH=/usr/local/cuda-11.7/lib64:/home/david/cryosparc/cryosparc_worker/deps/external/cudnn/lib:/usr/local/cuda/lib64
PATH=/usr/local/cuda-11.7/bin:/home/david/cryosparc/cryosparc_worker/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/bin:/home/david/cryosparc/cryosparc_worker/deps/anaconda/condabin:/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin
(base) david@DESKTOP-D1IHD96:~$
PATH
and LD_LIBRARY_PATH
definitions even before the CryoSPARC environment is loaded. This may interfere with CryoSPARC installation and function, for example, but probably not limited to, cryosparcw install-3dflex
/home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c "import pycuda.driver; print(pycuda.driver.get_version())"
(base) david@DESKTOP-D1IHD96:~$ /home/david/cryosparc/cryosparc_worker/bin/cryosparcw call python -c “import pycuda.driver; print(pycuda.driver.get_version())”
(11, 7, 0)
You could install CryoSPARC under a Linux account where you ensure (before installing and running CryoSPARC):
PATH=/usr/bin:/bin
LD_LIBRARY_PATH=""
export PATH LD_LIBRARY_PATH
and then (guide):
/path/to/cryosparc_worker/bin/cryosparcw install-3dflex
This will, among other things, install an independent CUDA environment that may work better under your circumstances.
after running the abovementioned command the installation finished with the following error.
…
Processing ./deps_bundle/python/python_packages/pip_packages/pycuda-2020.1-cp38-cp38-linux_x86_64.whl Installing collected packages: pycuda Successfully installed pycuda-2020.1 PyTorch not installed correctly, or NVIDIA GPU not detected.
Just in case I tried to re-run the CS patch motion correction but the error remains the same.
What do you think about that?
Kind regards,
is harmless (in the absence of other errors).
Part of me wants to:
/usr/local/cuda
directories are inside $PATH
and $LD_LIBRARY_PATH
now, or were during installationYes, both are sent to you privately.
P1 J26: Patch Motion Correction
Project Details
±----------------±-----------------+
| DETAIL | VALUE |
±----------------±-----------------+
| Title | -------
±----------------±-----------------+
| Description | ------------------- |
±----------------±-----------------+
| ID | P1 |
±----------------±-----------------+
| Created by user | -------------- |
±----------------±-----------------+
Job Details
±----------------±-------------------------------------------------------------------+
| DETAIL | VALUE |
±----------------±-------------------------------------------------------------------+
| Title | New Job J26 |
±----------------±-------------------------------------------------------------------+
| Description | Enter a description. |
±----------------±-------------------------------------------------------------------+
| ID | J26 |
±----------------±-------------------------------------------------------------------+
| Job type | Patch Motion Correction |
±----------------±-------------------------------------------------------------------+
| Status | completed |
±----------------±-------------------------------------------------------------------+
| Created by user | ----------------- |
±----------------±-------------------------------------------------------------------+
| Created | Thu Jan 26 2023 21:19:18 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Queued | Thu Jan 26 2023 21:19:24 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Launched | Thu Jan 26 2023 21:19:25 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Started | Thu Jan 26 2023 21:19:29 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Waiting | - |
±----------------±-------------------------------------------------------------------+
| Killed | - |
±----------------±-------------------------------------------------------------------+
| Completed | Thu Jan 26 2023 21:20:01 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Failed | - |
±----------------±-------------------------------------------------------------------+
| Last accessed | Fri Jan 27 2023 10:16:40 GMT+0100 (Central European Standard Time) |
±----------------±-------------------------------------------------------------------+
| Size | 72.56 KB |
±----------------±-------------------------------------------------------------------+
Inputs
movies
J4.imported_movies
±--------------------------------+
| MOVIE_BLOB |
±--------------------------------+
| J4.imported_movies.movie_blob.F |
±--------------------------------+
±-----------------------------------+
| MSCOPE_PARAMS |
±-----------------------------------+
| J4.imported_movies.mscope_params.F |
±-----------------------------------+
Parameters
General settings
±------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±--------±--------±-----±---------+
| Make motion diagnostic plots | Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Number of movies to plot | 10 | X | X | |
±------------------------------±--------±--------±-----±---------+
| Only process this many movies | Not Set | X | X | |
±------------------------------±--------±--------±-----±---------+
| Low-memory mode | Not set | X | X | |
±------------------------------±--------±--------±-----±---------+
Motion correction
±---------------------------------±--------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±---------------------------------±--------±--------±-----±---------+
| Maximum alignment resolution (A) | 5 | | | |
±---------------------------------±--------±--------±-----±---------+
| B-factor during alignment | 500 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Start frame (included, 0-based) | 0 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| End frame (excluded, 0-based) | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Output F-crop factor | 1 | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Override e/A^2 | Not Set | X | X | |
±---------------------------------±--------±--------±-----±---------+
| Allow Variable Dose | Not set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Calibrated smoothing | 0.5 | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots Z | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
±---------------------------------±--------±--------±-----±---------+
| Override knots Y | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Override knots X | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
| Random seed (hot pixel replace) | Not Set | X | X | X |
±---------------------------------±--------±--------±-----±---------+
Compute settings
±------------------------------±------±--------±-----±---------+
| PARAMETER | VALUE | DEFAULT | SPEC | ADVANCED |
±------------------------------±------±--------±-----±---------+
| Number of GPUs to parallelize | 1 | X | X | |
±------------------------------±------±--------±-----±---------+
Outputs
micrographs
±-----------------------------+
| MICROGRAPH_BLOB_NON_DW |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_1X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_THUMBNAIL_BLOB_2X |
±-----------------------------+
| J26.exposure.thumbnail_blob |
±-----------------------------+
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-----------------------+
| BACKGROUND_BLOB |
±-----------------------+
| J26.exposure.stat_blob |
±-----------------------+
±--------------------+
| RIGID_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±--------------------+
| SPLINE_MOTION |
±--------------------+
| J26.exposure.motion |
±--------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
micrographs_incomplete
±-----------------------------+
| MICROGRAPH_BLOB |
±-----------------------------+
| J26.exposure.micrograph_blob |
±-----------------------------+
±-------------------------+
| MOVIE_BLOB (PASSTHROUGH) |
±-------------------------+
| J26.exposure.movie_blob |
±-------------------------+
±----------------------------+
| MSCOPE_PARAMS (PASSTHROUGH) |
±----------------------------+
| J26.exposure.mscope_params |
±----------------------------+
Target
DESKTOP-D1IHD96. (node)
[2023-01-26 21:19:25.65] License is valid.
[2023-01-26 21:19:25.65] Launching job on lane default target DESKTOP-D1IHD96. …
[2023-01-26 21:19:25.68] Running job on master node hostname DESKTOP-D1IHD96.
[2023-01-26 21:19:29.36] [CPU: 195.1 MB] [Avail: 6.23 GB] Job J26 Started
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Working in directory: /mnt/d/Cryosparc/Apo/CS-apo/J26
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Running on lane default
[2023-01-26 21:19:29.37] [CPU: 195.3 MB] [Avail: 6.23 GB] Resources allocated:
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] Worker: DESKTOP-D1IHD96.
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] CPU : [0, 1, 2, 3, 4, 5]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] GPU : [0]
[2023-01-26 21:19:29.38] [CPU: 195.3 MB] [Avail: 6.23 GB] RAM : [0]
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] SSD : False
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] --------------------------------------------------------------
[2023-01-26 21:19:29.39] [CPU: 195.3 MB] [Avail: 6.23 GB] Importing job module for job type patch_motion_correction_multi…
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] Job ready to run
[2023-01-26 21:19:31.34] [CPU: 232.7 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Job will process this many movies: 20
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] Random seed: 195423761
[2023-01-26 21:19:31.44] [CPU: 233.2 MB] [Avail: 6.19 GB] parent process is 1770
[2023-01-26 21:19:31.47] [CPU: 166.0 MB] [Avail: 6.18 GB] Calling CUDA init from 1812
[2023-01-26 21:20:01.48] [CPU: 233.9 MB] [Avail: 6.20 GB] Child process with PID 1812 terminated unexpectedly with exit code
[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] --------------------------------------------------------------
[2023-01-26 21:20:01.52] [CPU: 234.0 MB] [Avail: 6.20 GB] Compiling job outputs…
[2023-01-26 21:20:01.53] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs from input
group movies
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob_non_dw’,
‘micrograph_thumbnail_blob_1x’, ‘micrograph_thumbnail_blob_2x’,
‘micrograph_blob’, ‘background_blob’, ‘rigid_motion’,
‘spline_motion’]
[2023-01-26 21:20:01.55] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 0 items
[2023-01-26 21:20:01.56] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.57] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 0 items
[2023-01-26 21:20:01.58] [CPU: 234.0 MB] [Avail: 6.20 GB] Passing through outputs for output group micrographs_incomplete from
input group movies
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] This job outputted results [‘micrograph_blob’]
[2023-01-26 21:20:01.59] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded output dset with 20 items
[2023-01-26 21:20:01.60] [CPU: 234.0 MB] [Avail: 6.20 GB] Passthrough results [‘movie_blob’, ‘mscope_params’]
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Loaded passthrough dset with 20 items
[2023-01-26 21:20:01.61] [CPU: 234.0 MB] [Avail: 6.20 GB] Intersection of output and passthrough has 20 items
[2023-01-26 21:20:01.62] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs
[2023-01-26 21:20:01.65] [CPU: 234.0 MB] [Avail: 6.20 GB] Checking outputs for output group micrographs_incomplete
[2023-01-26 21:20:01.69] [CPU: 234.1 MB] [Avail: 6.20 GB] Updating job size…
[2023-01-26 21:20:01.71] [CPU: 234.1 MB] [Avail: 6.19 GB] Exporting job and creating csg files…
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] ***************************************************************
[2023-01-26 21:20:01.75] [CPU: 234.1 MB] [Avail: 6.19 GB] Job complete. Total time 30.41s
Please can you post the job log
/mnt/d/Cryosparc/Apo/CS-apo/J26/job.log
================= CRYOSPARCW ======= 2023-01-26 21:19:26.183293 =========
Project P1 Job J26
Master DESKTOP-D1IHD96. Port 39002
===========================================================================
========= monitor process now starting main process at 2023-01-26 21:19:26.183316
MAINPROCESS PID 1770
MAIN PID 1770
motioncorrection.run_patch cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
***************************************************************
Running job on hostname %s DESKTOP-D1IHD96.
Allocated Resources : {'fixed': {'SSD': False}, 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'lane_type': 'node', 'license': True, 'licenses_acquired': 1, 'slots': {'CPU': [0, 1, 2, 3, 4, 5], 'GPU': [0], 'RAM': [0]}, 'target': {'cache_path': None, 'cache_quota_mb': None, 'cache_reserve_mb': 10000, 'desc': None, 'gpus': [{'id': 0, 'mem': 6441926656, 'name': 'NVIDIA GeForce RTX 3060 Laptop GPU'}], 'hostname': 'DESKTOP-D1IHD96.', 'lane': 'default', 'monitor_port': None, 'name': 'DESKTOP-D1IHD96.', 'resource_fixed': {'SSD': False}, 'resource_slots': {'CPU': [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15], 'GPU': [0], 'RAM': [0]}, 'ssh_str': 'david@DESKTOP-D1IHD96.', 'title': 'Worker node DESKTOP-D1IHD96.', 'type': 'node', 'worker_bin_path': '/home/david/cryosparc/cryosparc_worker/bin/cryosparcw'}}
Process Process-1:1:
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 315, in _bootstrap
self.run()
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/jobs/pipeline.py", line 200, in process_work_simple
process_setup(proc_idx) # do any setup you want on a per-process basis
File "cryosparc_master/cryosparc_compute/jobs/motioncorrection/run_patch.py", line 86, in cryosparc_compute.jobs.motioncorrection.run_patch.run_patch_motion_correction_multi.process_setup
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/engine/__init__.py", line 8, in <module>
from .engine import * # noqa
File "cryosparc_master/cryosparc_compute/engine/engine.py", line 9, in init cryosparc_compute.engine.engine
File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 12, in init cryosparc_compute.engine.cuda_core
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/fft.py", line 27, in <module>
from . import misc2 as misc
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/misc2.py", line 32, in <module>
from . import cuda
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cuda.py", line 17, in <module>
from .cudadrv import *
File "/home/david/cryosparc/cryosparc_worker/cryosparc_compute/skcuda_internal/cudadrv.py", line 39, in <module>
raise OSError('CUDA driver library not found')
OSError: CUDA driver library not found
========= sending heartbeat at 2023-01-26 21:19:39.493177
========= sending heartbeat at 2023-01-26 21:19:49.510677
========= sending heartbeat at 2023-01-26 21:19:59.534035
***************************************************************
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
Traceback (most recent call last):
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/queues.py", line 245, in _feed
send_bytes(obj)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 200, in send_bytes
self._send_bytes(m[offset:offset + size])
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 411, in _send_bytes
self._send(header + buf)
File "/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/multiprocessing/connection.py", line 368, in _send
n = write(self._handle, buf)
BrokenPipeError: [Errno 32] Broken pipe
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/fromnumeric.py:3372: RuntimeWarning: Mean of empty slice.
return _methods._mean(a, axis=axis, dtype=dtype,
/home/david/cryosparc/cryosparc_worker/deps/anaconda/envs/cryosparc_worker_env/lib/python3.8/site-packages/numpy/core/_methods.py:170: RuntimeWarning: invalid value encountered in double_scalars
ret = ret.dtype.type(ret / rcount)
========= main process now complete at 2023-01-26 21:20:01.795774.
========= monitor process now complete at 2023-01-26 21:20:01.798105.
I am not unsure why this happens. Possible causes to investigate:
cryosparcw install-3dflex
. What is the output (now) ofenv | grep PATH
Output for the command env | grep PATH
(base) david@DESKTOP-D1IHD96:~$ env | grep PATH
LD_LIBRARY_PATH=/usr/local/cuda/lib64:
PATH=/home/david/.local/bin:/home/david/cryosparc/cryosparc_master/bin:/home/david/cryosparc/cryosparc_master/bin:/usr/local/cuda/bin:/home/david/miniconda3/bin:/home/david/miniconda3/condabin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/usr/lib/wsl/lib:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32/WBEM:/mnt/c/Windows/system32:/mnt/c/Windows:/mnt/c/Windows/System32/Wbem:/mnt/c/Windows/System32/WindowsPowerShell/v1.0:/mnt/c/Windows/System32/OpenSSH:/mnt/c/Users/David/AppData/Local/Microsoft/WindowsApps:/snap/bin