(base) amax@amax-SYS-4029GP-TRT:/usr/local$ echo $CRYOSPARC_CUDA_PATH
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ eval $(~/software/cryosparc/cryosparc_worker/bin/cryosparcw env)
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ echo $CRYOSPARC_CUDA_PATH
/usr/local/cuda-11.2
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
(11, 7, 0)
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ uname -a && free -g && nvidia-smi
Linux amax-SYS-4029GP-TRT 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 1007 13 4 0 990 988
Swap: 1 1 0
Sun Jan 15 23:38:29 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03 Driver Version: 470.141.03 CUDA Version: 11.4 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
| | | MIG M. |
|===============================+======================+======================|
| 0 NVIDIA GeForce ... Off | 00000000:1A:00.0 Off | N/A |
| 30% 30C P8 21W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 1 NVIDIA GeForce ... Off | 00000000:1B:00.0 Off | N/A |
| 30% 29C P8 16W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 2 NVIDIA GeForce ... Off | 00000000:3D:00.0 Off | N/A |
| 30% 28C P8 30W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 3 NVIDIA GeForce ... Off | 00000000:3E:00.0 Off | N/A |
| 30% 29C P8 20W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 4 NVIDIA GeForce ... Off | 00000000:88:00.0 Off | N/A |
| 30% 26C P8 22W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 5 NVIDIA GeForce ... Off | 00000000:89:00.0 Off | N/A |
| 30% 28C P8 21W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 6 NVIDIA GeForce ... Off | 00000000:B1:00.0 Off | N/A |
| 30% 28C P8 22W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
| 7 NVIDIA GeForce ... Off | 00000000:B2:00.0 Off | N/A |
| 30% 27C P8 30W / 350W | 5MiB / 24268MiB | 0% Default |
| | | N/A |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: |
| GPU GI CI PID Type Process name GPU Memory |
| ID ID Usage |
|=============================================================================|
| 0 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 1 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 2 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 3 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 4 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 5 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 6 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
| 7 N/A N/A 3001 G /usr/lib/xorg/Xorg 4MiB |
+-----------------------------------------------------------------------------+
[CPU: 266.9 MB]
***************************************************************
[CPU: 7.61 GB]
Loading a ParticleStack with 68747 items...
[CPU: 7.61 GB]
SSD cache : cache successfully synced in_use
[CPU: 7.61 GB]
SSD cache : cache successfully synced, found 544301.90MB of files on SSD.
[CPU: 7.61 GB]
SSD cache : cache successfully requested to check 15794 files.
[CPU: 7.61 GB]
SSD cache : cache requires 0.00MB more on the SSD for files to be downloaded.
[CPU: 7.61 GB]
SSD cache : cache has enough available space.
[CPU: 7.61 GB]
SSD cache : cache starting transfers to SSD.
[CPU: 7.62 GB]
SSD cache : complete, all requested files are available on SSD.
[CPU: 7.64 GB]
Done.
[CPU: 7.64 GB]
Windowing particles
[CPU: 7.66 GB]
Done.
[CPU: 7.66 GB]
====== Gold Standard Split ======
[CPU: 7.66 GB]
Particles have input alignments3D connected, so reusing pre-existing split
[CPU: 7.66 GB]
Split A has 34373 particles
[CPU: 7.66 GB]
Split B has 34374 particles
[CPU: 7.66 GB]
====== Reconstruction ======
[CPU: 7.66 GB]
Input particles have box size 1240
[CPU: 7.66 GB]
Input particles have pixel size 0.7200
[CPU: 7.66 GB]
Volume refinement will be done with effective box size 1240
[CPU: 7.66 GB]
Volume refinement will be done with pixel size 0.7200
[CPU: 7.66 GB]
Particles will be zeropadded/truncated to size 1240 during backprojection
[CPU: 7.66 GB]
Particles will be backprojected with box size 1240
[CPU: 7.66 GB]
Reconstructing with C1 symmetry enforced
[CPU: 36.76 GB]
Engine Started.
[CPU: 40.92 GB]
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/refine/newrun.py", line 1086, in cryosparc_compute.jobs.refine.newrun.run_homo_reconstruct
File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 2980, in cryosparc_compute.engine.newengine.reconstruct
File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 3005, in cryosparc_compute.engine.newengine.reconstruct
File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 1823, in cryosparc_compute.engine.newengine.EngineThread.backproject
File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 313, in cryosparc_compute.engine.cuda_core.EngineBaseThread.toc
File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 309, in cryosparc_compute.engine.cuda_core.EngineBaseThread.wait
pycuda._driver.LogicError: cuStreamSynchronize failed: an illegal memory access was encountered
Hi,cryosparcteam:
I meet a problem when reconstruct particles with 1240 boxsize.It seems too large?My memory is 1T.And my GPU is 3090,CUDA version is11.2.How can I solve this err?Thanks very much.
ruirui