Error in 3D classification

3D Classification
1

Classification ran fine until the end of the first full iteration, whereupon it died during the outputting data step. 80 class classification, previous run with slightly different focus mask ran fine. Only thing I can think of is that these were very downsampled particles, and the target resolution was set higher than Nyquist in the downsampled box. Could this be the cause?

Cheers
Oli

Outputting data...

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 988, in cryosparc_master.cryosparc_compute.jobs.class3D.run.run_class_3D
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/volume_multi_utils.py", line 91, in convert_to_alignment3D_multi
    multi_ds = ds.copy().add_fields(alignment3D_multi_fields)
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 1043, in add_fields
    self._data.addcol_array(name, TYPE_TO_DSET_MAP[dt.base.type], dt.shape)
  File "cryosparc/core.pyx", line 124, in cryosparc.core.Data.addcol_array
MemoryError: Could not add column to dataset

EDIT:
Tried rerunning with exactly the same target res and number of classes as a previous successful run, it still fails. The only difference is the focus mask. Error from log:

========= sending heartbeat at 2024-06-20 03:14:14.082696
dataset.more_memory: out of memory (errno 12: Cannot allocate memory)
could not allocate more array heap space (adding column alignments3D_multi/resid_pow) (errno 12: Cannot allocate memory)
HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
**** handle exception rc
Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 988, in cryosparc_master.cryosparc_compute.jobs.class3D.run.run_class_3D
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_compute/volume_multi_utils.py", line 91, in convert_to_alignment3D_multi
    multi_ds = ds.copy().add_fields(alignment3D_multi_fields)
  File "/home/exx/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py", line 1043, in add_fields
    self._data.addcol_array(name, TYPE_TO_DSET_MAP[dt.base.type], dt.shape)
  File "cryosparc/core.pyx", line 124, in cryosparc.core.Data.addcol_array
MemoryError: Could not add column to dataset
set status to failed
========= sending heartbeat at 2024-06-20 03:14:24.094144

I guess maybe it is just running out of RAM… perhaps I had only one job running during the full iterations of the previous run. In these cases it would be very useful to have checkpoints, so a failed (very long running!) job could be continued from a previous iteration.

2

Hi @olibclarke, yes I can confirm that these messages definitely indicate that the job is running out of RAM. Sorry!

1 Like