"illegal memory access" error during Initial Model

Troubleshooting
1 
Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/run.c:3954)
  File "cryosparc2_worker/cryosparc2_compute/jobs/abinit/run.py", line 273, in cryosparc2_compute.jobs.abinit.run.run_homo_abinit (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/abinit/run.c:9284)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1113, in cryosparc2_compute.engine.engine.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:33381)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1114, in cryosparc2_compute.engine.engine.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:33316)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1082, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:29494)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 673, in cryosparc2_compute.engine.engine.EngineThread.backproject (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:22508)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 270, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.toc (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:8877)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 266, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.wait (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:8650)
LogicError: cuStreamSynchronize failed: an illegal memory access was encountered

Version: 2.9.0
iteration 100, 1000 particles, box size 100, pixel size 2.091 Å

Any ideas?

Wim Hagen

2

Hi @WimH,
Was symmetry enforced? What is the GPU model/CUDA version?

3

CUDA 10.0.130, O-symmetry, we have a mix of P100, 1080 and 2080 GPU’s. Need to reproduce to find which GPU model exactly.

4

Maybe flush the GPU memory? (here you have some tips about it) because probably you tried rebooting the server…if you have more users, maybe they locked some of the memory…