My Heterogeneous Refinement job with 3 classes keep dying with “illegal memory access error”.
I am using the default parameters and have close to half a million particles.
The error message is as follows:
Traceback (most recent call last):
File “/home_local/hpc/cryosparc/cryosparc-compute/sparc/streamlog.py”, line 438, in run_with_except_hook
run_old(*args, **kw)
File “/home_local/hpc/cryosparc/cryosparc-compute/engine/cuda_core.py”, line 86, in run
self.target(*self.args, dev=self.dev, thidx=self.thidx)
File “/home_local/hpc/cryosparc/cryosparc-compute/engine/engine.py”, line 822, in work
ET.setup_current_data_and_ctf(radwn_max_backproject, ctf_phase_flip_only=False)
File “/home_local/hpc/cryosparc/cryosparc-compute/engine/engine.py”, line 190, in setup_current_data_and_ctf
self.ensure_allocated(‘C’, (self.N_D, self.N_TT_aligned), n.float32, cpu=False)
File “/home_local/hpc/cryosparc/cryosparc-compute/engine/engine.py”, line 52, in ensure_allocated
new = cuda_core.allocate_gpu(shape, dtype, curr)
File “/home_local/hpc/cryosparc/cryosparc-compute/engine/cuda_core.py”, line 109, in allocate_gpu
ret = gpuarray.empty(shape, dtype=dtype)
File “/home_local/hpc/cryosparc/anaconda2/lib/python2.7/site-packages/pycuda/gpuarray.py”, line 209, in init
self.gpudata = self.allocator(self.size * self.dtype.itemsize)
LogicError: cuMemAlloc failed: an illegal memory access was encountered
Did anybody face the same issue before or have any suggestions?