Illegal memory access in multirefine

Troubleshooting
1

My multirefine jobs are failing quickly, but after several iterations, with a CUDA memory access error like the one below. Has anyone else seen this, or been able to work around it?

Traceback (most recent call last):
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/sparc/streamlog.py", line 321, in run_with_except_hook
    run_old(*args, **kw)
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/engine/cuda_core.py", line 86, in run
    self.target(*self.args, dev=self.dev, thidx=self.thidx)
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/engine/engine.py", line 626, in work
    ET.compute_resid_pow() # do this even if not do_align because we have to compare the different structures
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/engine/engine.py", line 264, in compute_resid_pow
    self.toc('compute_resid_pow')
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/engine/engine.py", line 42, in toc
    self.wait()
  File "/mnt/cache/cryosparc/cryosparc/cryosparc-compute/engine/engine.py", line 38, in wait
    self.stream.synchronize()
LogicError: cuStreamSynchronize failed: an illegal memory access was encountered
2

I believe this bug is triggered when multirefine is launched with structures and/or particles from multiple experiments. If structures and particles only from one experiment are selected, my job completes. If a structure or particles are also taken from another experiment (same dataset), the job fails.

3

Just as another data point, I haven’t taken particles from multiple datasets, but I’ve used structures from multiple datasets in multirefine without seeing this

Cheers
Oli

4

@spunjani Let me know what other information I could give. I’ve had trouble replicating the error with new jobs, but clones of jobs that had the error always fail the same way.

5