cuStreamSynchronize failed during reconstruction

(base) amax@amax-SYS-4029GP-TRT:/usr/local$ echo $CRYOSPARC_CUDA_PATH

(base) amax@amax-SYS-4029GP-TRT:/usr/local$ eval $(~/software/cryosparc/cryosparc_worker/bin/cryosparcw env)
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ echo $CRYOSPARC_CUDA_PATH
/usr/local/cuda-11.2
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ ${CRYOSPARC_CUDA_PATH}/bin/nvcc --version
nvcc: NVIDIA (R) Cuda compiler driver
Copyright (c) 2005-2021 NVIDIA Corporation
Built on Sun_Feb_14_21:12:58_PST_2021
Cuda compilation tools, release 11.2, V11.2.152
Build cuda_11.2.r11.2/compiler.29618528_0
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ python -c "import pycuda.driver; print(pycuda.driver.get_version())"
(11, 7, 0)
(base) amax@amax-SYS-4029GP-TRT:/usr/local$ uname -a && free -g && nvidia-smi
Linux amax-SYS-4029GP-TRT 5.15.0-52-generic #58~20.04.1-Ubuntu SMP Thu Oct 13 13:09:46 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
              total        used        free      shared  buff/cache   available
Mem:           1007          13           4           0         990         988
Swap:             1           1           0
Sun Jan 15 23:38:29 2023
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 470.141.03   Driver Version: 470.141.03   CUDA Version: 11.4     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|                               |                      |               MIG M. |
|===============================+======================+======================|
|   0  NVIDIA GeForce ...  Off  | 00000000:1A:00.0 Off |                  N/A |
| 30%   30C    P8    21W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   1  NVIDIA GeForce ...  Off  | 00000000:1B:00.0 Off |                  N/A |
| 30%   29C    P8    16W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   2  NVIDIA GeForce ...  Off  | 00000000:3D:00.0 Off |                  N/A |
| 30%   28C    P8    30W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   3  NVIDIA GeForce ...  Off  | 00000000:3E:00.0 Off |                  N/A |
| 30%   29C    P8    20W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   4  NVIDIA GeForce ...  Off  | 00000000:88:00.0 Off |                  N/A |
| 30%   26C    P8    22W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   5  NVIDIA GeForce ...  Off  | 00000000:89:00.0 Off |                  N/A |
| 30%   28C    P8    21W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   6  NVIDIA GeForce ...  Off  | 00000000:B1:00.0 Off |                  N/A |
| 30%   28C    P8    22W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+
|   7  NVIDIA GeForce ...  Off  | 00000000:B2:00.0 Off |                  N/A |
| 30%   27C    P8    30W / 350W |      5MiB / 24268MiB |      0%      Default |
|                               |                      |                  N/A |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                                  |
|  GPU   GI   CI        PID   Type   Process name                  GPU Memory |
|        ID   ID                                                   Usage      |
|=============================================================================|
|    0   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    1   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    2   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    3   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    4   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    5   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    6   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
|    7   N/A  N/A      3001      G   /usr/lib/xorg/Xorg                  4MiB |
+-----------------------------------------------------------------------------+
[CPU: 266.9 MB]
***************************************************************

[CPU: 7.61 GB]
Loading a ParticleStack with 68747 items...

[CPU: 7.61 GB]
 SSD cache : cache successfully synced in_use

[CPU: 7.61 GB]
 SSD cache : cache successfully synced, found 544301.90MB of files on SSD.

[CPU: 7.61 GB]
 SSD cache : cache successfully requested to check 15794 files.

[CPU: 7.61 GB]
 SSD cache : cache requires 0.00MB more on the SSD for files to be downloaded.

[CPU: 7.61 GB]
 SSD cache : cache has enough available space.

[CPU: 7.61 GB]
 SSD cache : cache starting transfers to SSD.

[CPU: 7.62 GB]
 SSD cache : complete, all requested files are available on SSD.

[CPU: 7.64 GB]
  Done.

[CPU: 7.64 GB]
Windowing particles

[CPU: 7.66 GB]
  Done.

[CPU: 7.66 GB]
====== Gold Standard Split ======

[CPU: 7.66 GB]
  Particles have input alignments3D connected, so reusing pre-existing split

[CPU: 7.66 GB]
  Split A has 34373 particles 

[CPU: 7.66 GB]
  Split B has 34374 particles 

[CPU: 7.66 GB]
====== Reconstruction ======

[CPU: 7.66 GB]
  Input particles have box size 1240

[CPU: 7.66 GB]
  Input particles have pixel size 0.7200

[CPU: 7.66 GB]
  Volume refinement will be done with effective box size 1240

[CPU: 7.66 GB]
  Volume refinement will be done with pixel size 0.7200

[CPU: 7.66 GB]
  Particles will be zeropadded/truncated to size 1240 during backprojection

[CPU: 7.66 GB]
  Particles will be backprojected with box size 1240

[CPU: 7.66 GB]
  Reconstructing with C1 symmetry enforced

[CPU: 36.76 GB]
Engine Started.

[CPU: 40.92 GB]
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/refine/newrun.py", line 1086, in cryosparc_compute.jobs.refine.newrun.run_homo_reconstruct
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 2980, in cryosparc_compute.engine.newengine.reconstruct
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 3005, in cryosparc_compute.engine.newengine.reconstruct
  File "cryosparc_master/cryosparc_compute/engine/newengine.py", line 1823, in cryosparc_compute.engine.newengine.EngineThread.backproject
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 313, in cryosparc_compute.engine.cuda_core.EngineBaseThread.toc
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 309, in cryosparc_compute.engine.cuda_core.EngineBaseThread.wait
pycuda._driver.LogicError: cuStreamSynchronize failed: an illegal memory access was encountered

Hi,cryosparcteam:

I meet a problem when reconstruct particles with 1240 boxsize.It seems too large?My memory is 1T.And my GPU is 3090,CUDA version is11.2.How can I solve this err?Thanks very much.

ruirui

Thank you for reporting this problem. We have taken note and suggest that you re-attempt this job with downsampled particles.