Error: could not load dataset from file

Hi,

I have a issue when I tried to output the inspected particles.

shown in the event log: “MemoryError: Could not add rows to dataset”
“cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /xxx/xxx/CryoSparcUserData/xxx/J27/picked_particles.cs”

I would appreciate if you help me solve the problem.

@liujiani Please can you post details about the job:

cspid="P99"
csjid="J199"
cyosparcm cli "get_job('$cspid', '$csjid', 'job_type', 'version', 'params_spec', 'instance_information', 'input_slot_groups', 'status')" 
cryosparcm eventlog "$cspid" "$csjid"
cryosparcm joblog "$cspid" "$csjid" | tail -n 40

where you replace P99 and J199 with the failed job’s project and job IDs, respectively.

Hi, here are the output:

cryosparcm eventlog "$cspid" "$csjid"
[Thu, 25 Jul 2024 11:02:19 GMT]  License is valid.
[Thu, 25 Jul 2024 11:02:19 GMT]  Running job on master node
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Job J30 Started
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Master running v4.5.3, worker running v4.5.3
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Working in directory: /xx/J30
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Running on lane None
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Resources allocated:
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB]   Worker:  puhti-login15.bullx
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] --------------------------------------------------------------
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Importing job module for job type inspect_picks_v2...
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] Job ready to run
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] ***************************************************************
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] Interactive job running on port 33821
[Thu, 25 Jul 2024 11:04:54 GMT] [CPU RAM used: 2708 MB] Loaded info for 12256 micrographs
[Thu, 25 Jul 2024 11:04:54 GMT] [CPU RAM used: 2708 MB] Loaded info for 10119545 particles
[Thu, 25 Jul 2024 11:04:56 GMT] [CPU RAM used: 2708 MB] Calibrating NCC scores to defocus..
[Thu, 25 Jul 2024 11:05:01 GMT]  NCC calibration
[Thu, 25 Jul 2024 11:08:27 GMT] [CPU RAM used: 2676 MB] Calibrating Power scores to defocus..
[Thu, 25 Jul 2024 11:08:28 GMT]  Power calibration
[Thu, 25 Jul 2024 11:12:51 GMT] [CPU RAM used: 4124 MB] (DF) Loading particle pick data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading CTF data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading CTF stats data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading rigid motion data...
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4117 MB] Done in 260.10s
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4117 MB] uid                         object
idx                          int64
fname                       object
project_uid                 object
job_uid                     object
num_particles                int64
num_particles_threshold      int64
pick_ncc_median            float32
pick_pow_median            float32
df_avg                     float32
ctf_fit_to_A               float32
df_ast                     float32
ctf_fit_cc                 float32
phase_shift_deg            float32
df_range                   float32
df_tilt_normal             float32
ice_thickness_rel          float32
motion_total_pix           float64
motion_curvature           float64
dtype: object
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4118 MB] [{'name': 'idx', 'title': 'Index', 'short': 'Index'}, {'name': 'num_particles', 'title': 'Number of picked particles', 'short': 'Num. particles'}, {'name': 'num_particles_threshold', 'title': 'Number of picked particles matching threshold', 'short': 'Num. particles threshold'}, {'name': 'pick_ncc_median', 'title': 'Median pick cc score', 'short': 'Pick CC', 'round': 2}, {'name': 'pick_pow_median', 'title': 'Median pick power score', 'short': 'Pick power', 'round': 2}, {'name': 'df_avg', 'title': 'Average defocus (A)', 'short': 'DF Avg', 'round': 0}, {'name': 'ctf_fit_to_A', 'title': 'CTF fit resolution (A)', 'short': 'CTF Fit', 'round': 2}, {'name': 'df_ast', 'title': 'Astigmatism (A)', 'short': 'Astig'}, {'name': 'df_range', 'title': 'Defocus Range', 'short': 'DF Range'}, {'name': 'df_tilt_normal', 'title': 'Defocus Tilt Angle (degrees)', 'short': 'Tilt Angle'}, {'name': 'ice_thickness_rel', 'title': 'Relative Ice Thickness', 'short': 'Rel Ice Thick.'}, {'name': 'motion_total_pix', 'title': 'Total full-frame motion distance (pixels)', 'short': 'Motion dist.'}, {'name': 'motion_curvature', 'title': 'Full-frame motion curvature', 'short': 'Motion curv.'}, {'name': 'uid', 'title': 'UID', 'short': 'UID'}, {'name': 'fname', 'title': 'File name', 'short': 'File'}]
[Thu, 25 Jul 2024 11:19:52 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 220
[Thu, 25 Jul 2024 11:20:09 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 189
[Thu, 25 Jul 2024 11:20:24 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 155
[Thu, 25 Jul 2024 11:20:39 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 87
[Thu, 25 Jul 2024 11:20:56 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:21:12 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0.42, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:21:52 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0.93, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:08 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 0.95, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:24 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.02, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:31 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.14, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:40 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.14, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:46 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.19, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:54 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.08, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:24:01 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.48, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] --------------------------------------------------------
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] Outputting particles that have:
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB]   NCC score       >  1.480
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB]   Local power     >  4.000
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB]   Local power     <  68.000
[Thu, 25 Jul 2024 11:24:20 GMT]  Pick stats and final thresholds
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] 3191013 particles included, 6928532 particles excluded.
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] --------------------------------------------------------
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:31 GMT]  Extracted coordinates on J3/motioncorrected/010929021455981439625_FoilHole_18255919_Data_18134883_0_20240618_235540_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT]  Extracted coordinates on J3/motioncorrected/005438301115846311077_FoilHole_18255920_Data_18134892_1_20240618_235624_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT]  Extracted coordinates on J3/motioncorrected/009765333374663387756_FoilHole_18255921_Data_18134901_6_20240618_235704_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT]  Extracted coordinates on J3/motioncorrected/010835682593195539140_FoilHole_18255923_Data_18134889_8_20240618_235745_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT]  Extracted coordinates on J3/motioncorrected/012629020083643638179_FoilHole_18255924_Data_18134898_4_20240618_235826_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT]  Extracted coordinates on J3/motioncorrected/010325923352975725172_FoilHole_18255926_Data_18134886_2_20240618_235907_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT]  Extracted coordinates on J3/motioncorrected/018347129054301543381_FoilHole_18255927_Data_18134895_8_20240619_000204_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT]  Extracted coordinates on J3/motioncorrected/011816606238871428378_FoilHole_18255929_Data_18134883_7_20240619_000848_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:34 GMT]  Extracted coordinates on J3/motioncorrected/015959081893009637711_FoilHole_18255930_Data_18134892_4_20240619_000929_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:34 GMT] [CPU RAM used: 4160 MB]   Loading micrograph..
[Thu, 25 Jul 2024 11:24:34 GMT]  Extracted coordinates on J3/motioncorrected/006207268421323697032_FoilHole_18255931_Data_18134901_8_20240619_001010_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] ==== Completed. Extracted 3191013 particles.
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Interactive backend shutting down.
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] --------------------------------------------------------------
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Compiling job outputs...
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Passing through outputs for output group micrographs from input group micrographs
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] This job outputted results ['micrograph_blob']
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB]   Loaded output dset with 12256 items
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] Passthrough results ['ctf', 'ctf_stats', 'rigid_motion', 'spline_motion', 'mscope_params', 'background_blob', 'micrograph_blob_denoised', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'micrograph_blob_non_dw', 'micrograph_blob_non_dw_AB', 'movie_blob', 'gain_ref_blob']
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB]   Loaded passthrough dset with 12256 items
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB]   Intersection of output and passthrough has 12256 items
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Output dataset contains:  ['ctf_stats', 'micrograph_thumbnail_blob_1x', 'gain_ref_blob', 'rigid_motion', 'micrograph_thumbnail_blob_2x', 'micrograph_blob_non_dw', 'movie_blob', 'spline_motion', 'mscope_params', 'micrograph_blob_denoised', 'background_blob', 'ctf', 'micrograph_blob_non_dw_AB']
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result ctf
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result ctf_stats
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result rigid_motion
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result spline_motion
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result mscope_params
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result background_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result micrograph_blob_denoised
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result micrograph_thumbnail_blob_1x
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result micrograph_thumbnail_blob_2x
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result micrograph_blob_non_dw
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result micrograph_blob_non_dw_AB
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result movie_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB]   Outputting passthrough result gain_ref_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Passing through outputs for output group particles from input group particles
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB] This job outputted results ['location']
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB]   Loaded output dset with 3191013 items
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB] Passthrough results ['pick_stats', 'ctf']
[Thu, 25 Jul 2024 11:24:47 GMT] [CPU RAM used: 7049 MB] Traceback (most recent call last):
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 567, in load
    dset = cls(indata)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 770, in __init__
    self._data.addrows(nrows)
  File "cryosparc/core.pyx", line 108, in cryosparc.core.Data.addrows
MemoryError: Could not add rows to dataset

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 119, in cryosparc_master.cryosparc_compute.run.main
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1276, in passthrough_outputs
    dset = load_input_group(input_group_name, passthrough_result_names, allow_passthrough=True, memoize=True)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in load_input_group
    dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in <listcomp>
    dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 648, in load_input_connection_slots
    dset = load_input_connection_single_slot(input_group_name, slot_name, connection_idx, allow_passthrough=allow_passthrough, memoize=memoize)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 638, in load_input_connection_single_slot
    d = load_output_result_dset(_project_uid, output_result, slotconnection['version'], slot_name, memoize=memoize)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 586, in load_output_result_dset
    d = load_dataset_cached(abspath).copy()
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 572, in load_dataset_cached
    return dataset.Dataset.load(path)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 605, in load
    raise DatasetLoadError(f"Could not load dataset from file {file}") from err
cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /xx/J27/picked_particles.cs
cryosparcm joblog "$cspid" "$csjid" | tail -n 40
min: -59.327230 max: 58.863390
min: -58.905836 max: 58.467871
min: -60.135975 max: 59.660972
min: -60.533772 max: 60.051444
min: -58.625220 max: 58.173891
min: -58.849793 max: 58.339229
min: -62.444555 max: 62.013399
**** handle exception rc
Traceback (most recent call last):
  File "/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 567, in load
    dset = cls(indata)
  File "/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 770, in __init__
    self._data.addrows(nrows)
  File "cryosparc/core.pyx", line 108, in cryosparc.core.Data.addrows
MemoryError: Could not add rows to dataset

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 119, in cryosparc_master.cryosparc_compute.run.main
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1276, in passthrough_outputs
    dset = load_input_group(input_group_name, passthrough_result_names, allow_passthrough=True, memoize=True)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in load_input_group
    dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in <listcomp>
    dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 648, in load_input_connection_slots
    dset = load_input_connection_single_slot(input_group_name, slot_name, connection_idx, allow_passthrough=allow_passthrough, memoize=memoize)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 638, in load_input_connection_single_slot
    d = load_output_result_dset(_project_uid, output_result, slotconnection['version'], slot_name, memoize=memoize)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 586, in load_output_result_dset
    d = load_dataset_cached(abspath).copy()
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 572, in load_dataset_cached
    return dataset.Dataset.load(path)
  File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 605, in load
    raise DatasetLoadError(f"Could not load dataset from file {file}") from err
cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /xx/J27/picked_particles.cs
set status to failed
========= main process now complete at 2024-07-25 14:24:57.235271.
========= monitor process now complete at 2024-07-25 14:24:57.239357.

Your master computer may have run out of RAM.
How much RAM does the computer have? You can find out with the command:

free -h

To display top memory “consumers” (processes), you can run the command

top -n 1 -w 120 -o %MEM -bc | head -n 20

Hi, the RAM looks fine?

free -h
              total        used        free      shared  buff/cache   available
Mem:          170Gi        50Gi        51Gi       1.3Gi        68Gi       114Gi
Swap:            0B          0B          0B
top -n 1 -w 120 -o %MEM -bc | head -n 20
top - 09:00:09 up 32 days, 47 min, 66 users,  load average: 9.75, 6.01, 4.14
Tasks: 1390 total,   6 running, 1312 sleeping,  67 stopped,   5 zombie
%Cpu(s): 33.1 us, 12.0 sy,  0.0 ni, 52.9 id,  0.0 wa,  0.6 hi,  1.4 si,  0.0 st
MiB Mem : 175014.3 total,  52720.5 free,  52029.0 used,  70264.9 buff/cache
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 117422.2 avail Mem

@liujiani To overcome the issue of limited RAM you want to try:

  1. Partition the exposures into multiple, smaller sets using a Exposure Sets Tool job.
  2. Partition the particles into corresponding smaller sets by connecting the smaller exposure sets, one smaller set for each job, from the previous step and the particles to a corresponding number of Manually Curate Exposures jobs. Each job should output a subset of particles that corresponds to the connected smaller set of exposures.
  3. Connect one smaller set of exposures and its corresponding subset of particles to an Inspect Picks job. Determine and apply suitable acceptance/rejection criteria.
  4. Apply the criteria determined in the previous step interactively or non-interactively using additional Inspect Picks jobs to which you respectively connected the remaining pairs of smaller sets of exposures and particles.