"illegal memory access" error during Initial Model

Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 78, in cryosparc2_compute.run.main (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/run.c:3954)
  File "cryosparc2_worker/cryosparc2_compute/jobs/abinit/run.py", line 273, in cryosparc2_compute.jobs.abinit.run.run_homo_abinit (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/jobs/abinit/run.c:9284)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1113, in cryosparc2_compute.engine.engine.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:33381)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1114, in cryosparc2_compute.engine.engine.process (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:33316)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 1082, in cryosparc2_compute.engine.engine.process.work (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:29494)
  File "cryosparc2_worker/cryosparc2_compute/engine/engine.py", line 673, in cryosparc2_compute.engine.engine.EngineThread.backproject (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/engine.c:22508)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 270, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.toc (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:8877)
  File "cryosparc2_worker/cryosparc2_compute/engine/cuda_core.py", line 266, in cryosparc2_compute.engine.cuda_core.EngineBaseThread.wait (/home/installtest/deps_manage/cryosparc2_package/deploy/stage/cryosparc2_worker/cryosparc2_compute/engine/cuda_core.c:8650)
LogicError: cuStreamSynchronize failed: an illegal memory access was encountered

Version: 2.9.0
iteration 100, 1000 particles, box size 100, pixel size 2.091 Å

Any ideas?

Wim Hagen

Hi @WimH,
Was symmetry enforced? What is the GPU model/CUDA version?

CUDA 10.0.130, O-symmetry, we have a mix of P100, 1080 and 2080 GPU’s. Need to reproduce to find which GPU model exactly.

Maybe flush the GPU memory? (here you have some tips about it) because probably you tried rebooting the server…if you have more users, maybe they locked some of the memory…