Losing particles upon re-extraction w/o downsampling

Particle Picking
1

I am running into a problem when re-extracting particles. My box size is 512 but during initial processing steps I down sample 2x by setting the “Fourier-crop to box size” to 256. After some pre-processing, I have ~400,000 particles I want to re-extract without downsampling. It set up the job with my particle set and all the micrographs I used originally, with box size set to 512. But the job only returns ~300,000 particles.

I went back through the processing and looked for duplicated particles and went so far as to re-process the data from the beginning. But, I still ended up the the same issue of mysteriously dropped particles upon re-extraction.

I can’t figure out why I am losing ~25% of my particles or if it’s user error, an issue with CryoSparc, or and issue with data placement/our server.

2

Welcome to the forum @segarka. Please can you post the outputs of these commands for both downsampled and non-downsampled cases:

csprojectid=P99 # replace with actual project ID
csjobid=J199 # replace with id of the extract job
cryosparcm cli "get_job('$csprojectid', '$csjobid', 'job_type', 'version', 'status',  'params_spec', 'errors_run', 'input_slot_groups')"
cryosparcm eventlog $csprojectid $csjobid | tail -n 60
3

Hello! Sorry for the delay, we’ve been moving severs.

Results for downsampled:

{'_id': '680921f0cd49ce5425f0bbb4', 'errors_run': [], 'input_slot_groups': [{'connections': [{'group_name': 'micrographs', 'job_uid': 'J1640', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1640', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Micrographs for particle extraction', 'name': 'micrographs', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'micrograph_blob', 'optional': False, 'title': 'Raw micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'background_blob', 'optional': True, 'title': 'Estimated background data', 'type': 'exposure.stat_blob'}, {'description': '', 'name': 'mscope_params', 'optional': False, 'title': 'Microscope parameters', 'type': 'exposure.mscope_params'}, {'description': '', 'name': 'ctf', 'optional': True, 'title': 'CTF parameters', 'type': 'exposure.ctf'}], 'title': 'Micrographs', 'type': 'exposure'}, {'connections': [{'group_name': 'particles', 'job_uid': 'J1640', 'slots': [{'group_name': 'particles', 'job_uid': 'J1640', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1640', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1640', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 0, 'description': 'Particles to extract', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}, {'description': '', 'name': 'alignments2D', 'optional': True, 'title': 'Particle 2D alignments', 'type': 'particle.alignments2D'}, {'description': '', 'name': 'alignments3D', 'optional': True, 'title': 'Particle 3D alignments', 'type': 'particle.alignments3D'}], 'title': 'Particles', 'type': 'particle'}], 'job_type': 'extract_micrographs_multi', 'params_spec': {'bin_size_pix': {'value': 256}, 'box_size_pix': {'value': 512}, 'compute_num_gpus': {'value': 2}}, 'project_uid': 'P5', 'status': 'completed', 'uid': 'J1703', 'version': 'v4.6.2'}

[Sun, 02 Mar 2025 23:05:57 GMT] [CPU RAM used: 16935 MB] (48887 of 48890) Finished processing micrograph 48887.
[Sun, 02 Mar 2025 23:05:57 GMT] [CPU RAM used: 16935 MB] (48888 of 48890) Finished processing micrograph 48888.
[Sun, 02 Mar 2025 23:05:58 GMT] [CPU RAM used: 16935 MB] (48889 of 48890) Finished processing micrograph 48889.
[Sun, 02 Mar 2025 23:05:58 GMT] [CPU RAM used: 16935 MB] (48890 of 48890) Finished processing micrograph 48890.
[Sun, 02 Mar 2025 23:06:13 GMT] [CPU RAM used: 17309 MB] ==== Completed. Extracted 3828131 particles in 28091.98s.
[Sun, 02 Mar 2025 23:06:14 GMT] [CPU RAM used: 16379 MB] --------------------------------------------------------------
[Sun, 02 Mar 2025 23:06:14 GMT] [CPU RAM used: 16379 MB] Compiling job outputs...
[Sun, 02 Mar 2025 23:06:14 GMT] [CPU RAM used: 16379 MB] Passing through outputs for output group micrographs from input group micrographs
[Sun, 02 Mar 2025 23:06:15 GMT] [CPU RAM used: 16379 MB] This job outputted results ['micrograph_blob']
[Sun, 02 Mar 2025 23:06:15 GMT] [CPU RAM used: 16379 MB] Loaded output dset with 46305 items
[Sun, 02 Mar 2025 23:06:15 GMT] [CPU RAM used: 16379 MB] Passthrough results ['background_blob', 'mscope_params', 'ctf', 'ctf_stats', 'rigid_motion', 'spline_motion', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'movie_blob', 'micrograph_blob_non_dw', 'gain_ref_blob']
[Sun, 02 Mar 2025 23:06:18 GMT] [CPU RAM used: 16444 MB] Loaded passthrough dset with 48890 items
[Sun, 02 Mar 2025 23:06:18 GMT] [CPU RAM used: 16453 MB] Intersection of output and passthrough has 46305 items
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Output dataset contains: ['mscope_params', 'ctf_stats', 'ctf', 'gain_ref_blob', 'micrograph_thumbnail_blob_1x', 'rigid_motion', 'movie_blob', 'micrograph_thumbnail_blob_2x', 'spline_motion', 'background_blob', 'micrograph_blob_non_dw']
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result background_blob
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result mscope_params
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result ctf
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result ctf_stats
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result rigid_motion
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result spline_motion
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_thumbnail_blob_1x
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_thumbnail_blob_2x
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result movie_blob
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_blob_non_dw
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result gain_ref_blob
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Passing through outputs for output group micrographs_incomplete from input group micrographs
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] This job outputted results ['micrograph_blob']
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Loaded output dset with 0 items
[Sun, 02 Mar 2025 23:06:19 GMT] [CPU RAM used: 16466 MB] Passthrough results ['background_blob', 'mscope_params', 'ctf', 'ctf_stats', 'rigid_motion', 'spline_motion', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'movie_blob', 'micrograph_blob_non_dw', 'gain_ref_blob']
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Loaded passthrough dset with 48890 items
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Intersection of output and passthrough has 0 items
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Output dataset contains: ['mscope_params', 'ctf_stats', 'ctf', 'gain_ref_blob', 'micrograph_thumbnail_blob_1x', 'rigid_motion', 'movie_blob', 'micrograph_thumbnail_blob_2x', 'spline_motion', 'background_blob', 'micrograph_blob_non_dw']
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result background_blob
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result mscope_params
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result ctf
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result ctf_stats
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result rigid_motion
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result spline_motion
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_thumbnail_blob_1x
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_thumbnail_blob_2x
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result movie_blob
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result micrograph_blob_non_dw
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Outputting passthrough result gain_ref_blob
[Sun, 02 Mar 2025 23:06:20 GMT] [CPU RAM used: 16466 MB] Passing through outputs for output group particles from input group particles
[Sun, 02 Mar 2025 23:06:31 GMT] [CPU RAM used: 16951 MB] This job outputted results ['blob']
[Sun, 02 Mar 2025 23:06:31 GMT] [CPU RAM used: 16951 MB] Loaded output dset with 3828131 items
[Sun, 02 Mar 2025 23:06:31 GMT] [CPU RAM used: 16951 MB] Passthrough results ['location', 'ctf', 'pick_stats']
[Sun, 02 Mar 2025 23:06:55 GMT] [CPU RAM used: 18628 MB] Loaded passthrough dset with 4855644 items
[Sun, 02 Mar 2025 23:06:58 GMT] [CPU RAM used: 18397 MB] Intersection of output and passthrough has 3828131 items
[Sun, 02 Mar 2025 23:07:04 GMT] [CPU RAM used: 18453 MB] Output dataset contains: ['ctf', 'pick_stats', 'location']
[Sun, 02 Mar 2025 23:07:04 GMT] [CPU RAM used: 18453 MB] Outputting passthrough result location
[Sun, 02 Mar 2025 23:07:04 GMT] [CPU RAM used: 18453 MB] Outputting passthrough result ctf
[Sun, 02 Mar 2025 23:07:04 GMT] [CPU RAM used: 18453 MB] Outputting passthrough result pick_stats
[Sun, 02 Mar 2025 23:07:04 GMT] [CPU RAM used: 17650 MB] Checking outputs for output group micrographs
[Sun, 02 Mar 2025 23:07:06 GMT] [CPU RAM used: 17743 MB] Checking outputs for output group micrographs_incomplete
[Sun, 02 Mar 2025 23:07:06 GMT] [CPU RAM used: 17743 MB] Checking outputs for output group particles
[Sun, 02 Mar 2025 23:07:30 GMT] [CPU RAM used: 18422 MB] Updating job size...
[Sun, 02 Mar 2025 23:07:38 GMT] [CPU RAM used: 18423 MB] Exporting job and creating csg files...
[Sun, 02 Mar 2025 23:07:44 GMT] [CPU RAM used: 18423 MB] ***************************************************************
[Sun, 02 Mar 2025 23:07:44 GMT] [CPU RAM used: 18423 MB] Job complete. Total time 28194.84s

And results for non-downsapled:

{'_id': '68092357cd49ce5425f74332', 'errors_run': [], 'input_slot_groups': [{'connections': [{'group_name': 'micrographs', 'job_uid': 'J1620', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'micrograph_blob_non_dw_AB', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J1620', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}, {'group_name': 'micrographs', 'job_uid': 'J857', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J857', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}, {'group_name': 'micrographs', 'job_uid': 'J856', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J856', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}, {'group_name': 'micrographs', 'job_uid': 'J882', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J882', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}, {'group_name': 'micrographs', 'job_uid': 'J888', 'slots': [{'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'micrograph_blob', 'result_type': 'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name': 'background_blob', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name': 'mscope_params', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'ctf', 'result_type': 'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'micrograph_thumbnail_blob_2x', 'result_type': 'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'rigid_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'micrograph_blob_non_dw', 'result_type': 'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J888', 'result_name': 'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Micrographs for particle extraction', 'name': 'micrographs', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'micrograph_blob', 'optional': False, 'title': 'Raw micrograph data', 'type': 'exposure.micrograph_blob'}, {'description': '', 'name': 'background_blob', 'optional': True, 'title': 'Estimated background data', 'type': 'exposure.stat_blob'}, {'description': '', 'name': 'mscope_params', 'optional': False, 'title': 'Microscope parameters', 'type': 'exposure.mscope_params'}, {'description': '', 'name': 'ctf', 'optional': True, 'title': 'CTF parameters', 'type': 'exposure.ctf'}], 'title': 'Micrographs', 'type': 'exposure'}, {'connections': [{'group_name': 'particles', 'job_uid': 'J1752', 'slots': [{'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'location', 'result_type': 'particle.location', 'slot_name': 'location', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'alignments2D', 'result_type': 'particle.alignments2D', 'slot_name': 'alignments2D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'alignments3D', 'result_type': 'particle.alignments3D', 'slot_name': 'alignments3D', 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'ctf', 'result_type': 'particle.ctf', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'blob', 'result_type': 'particle.blob', 'slot_name': None, 'version': 'F'}, {'group_name': 'particles', 'job_uid': 'J1752', 'result_name': 'pick_stats', 'result_type': 'particle.pick_stats', 'slot_name': None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 0, 'description': 'Particles to extract', 'name': 'particles', 'repeat_allowed': False, 'slots': [{'description': '', 'name': 'location', 'optional': False, 'title': 'Particle locations', 'type': 'particle.location'}, {'description': '', 'name': 'alignments2D', 'optional': True, 'title': 'Particle 2D alignments', 'type': 'particle.alignments2D'}, {'description': '', 'name': 'alignments3D', 'optional': True, 'title': 'Particle 3D alignments', 'type': 'particle.alignments3D'}], 'title': 'Particles', 'type': 'particle'}], 'job_type': 'extract_micrographs_multi', 'params_spec': {'box_size_pix': {'value': 512}, 'compute_num_gpus': {'value': 2}, 'update_alignments2D': {'value': True}, 'update_alignments3D': {'value': True}, 'update_location': {'value': True}}, 'project_uid': 'P5', 'status': 'completed', 'uid': 'J1754', 'version': 'v4.5.3'


[Tue, 11 Mar 2025 23:46:10 GMT] [CPU RAM used: 12430 MB] (48886 of 48890) Finished processing micrograph 48881. It had no particles in it.
[Tue, 11 Mar 2025 23:46:10 GMT] [CPU RAM used: 12430 MB] (48887 of 48890) Finished processing micrograph 48883. It had no particles in it.
[Tue, 11 Mar 2025 23:46:10 GMT] [CPU RAM used: 12430 MB] (48888 of 48890) Finished processing micrograph 48885. It had no particles in it.
[Tue, 11 Mar 2025 23:46:10 GMT] [CPU RAM used: 12430 MB] (48889 of 48890) Finished processing micrograph 48887. It had no particles in it.
[Tue, 11 Mar 2025 23:46:10 GMT] [CPU RAM used: 12430 MB] (48890 of 48890) Finished processing micrograph 48889. It had no particles in it.
[Tue, 11 Mar 2025 23:46:28 GMT] [CPU RAM used: 12553 MB] ==== Completed. Extracted 308935 particles in 22749.79s.
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB] --------------------------------------------------------------
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB] Compiling job outputs...
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB] Passing through outputs for output group micrographs from input group micrographs
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB] This job outputted results ['micrograph_blob']
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB]  Loaded output dset with 29382 items
[Tue, 11 Mar 2025 23:46:29 GMT] [CPU RAM used: 12137 MB] Passthrough results ['background_blob', 'mscope_params', 'ctf', 'movie_blob', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'ctf_stats', 'rigid_motion', 'spline_motion', 'micrograph_blob_non_dw', 'gain_ref_blob']
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12198 MB]  Loaded passthrough dset with 48890 items
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12198 MB]  Intersection of output and passthrough has 29382 items
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Output dataset contains: ['micrograph_thumbnail_blob_2x', 'ctf', 'micrograph_thumbnail_blob_1x', 'mscope_params', 'rigid_motion', 'gain_ref_blob', 'ctf_stats', 'spline_motion', 'micrograph_blob_non_dw', 'movie_blob', 'background_blob']
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result background_blob
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result mscope_params
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result ctf
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result movie_blob
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_thumbnail_blob_1x
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_thumbnail_blob_2x
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result ctf_stats
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result rigid_motion
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result spline_motion
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_blob_non_dw
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result gain_ref_blob
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB] Passing through outputs for output group micrographs_incomplete from input group micrographs
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB] This job outputted results ['micrograph_blob']
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB]  Loaded output dset with 0 items
[Tue, 11 Mar 2025 23:46:32 GMT] [CPU RAM used: 12199 MB] Passthrough results ['background_blob', 'mscope_params', 'ctf', 'movie_blob', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'ctf_stats', 'rigid_motion', 'spline_motion', 'micrograph_blob_non_dw', 'gain_ref_blob']
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Loaded passthrough dset with 48890 items
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Intersection of output and passthrough has 0 items
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Output dataset contains: ['micrograph_thumbnail_blob_2x', 'ctf', 'micrograph_thumbnail_blob_1x', 'mscope_params', 'rigid_motion', 'gain_ref_blob', 'ctf_stats', 'spline_motion', 'micrograph_blob_non_dw', 'movie_blob', 'background_blob']
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result background_blob
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result mscope_params
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result ctf
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result movie_blob
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_thumbnail_blob_1x
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_thumbnail_blob_2x
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result ctf_stats
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result rigid_motion
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result spline_motion
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result micrograph_blob_non_dw
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB]  Outputting passthrough result gain_ref_blob
[Tue, 11 Mar 2025 23:46:33 GMT] [CPU RAM used: 12199 MB] Passing through outputs for output group particles from input group particles
[Tue, 11 Mar 2025 23:46:35 GMT] [CPU RAM used: 12312 MB] This job outputted results ['blob', 'location', 'alignments2D', 'alignments3D']
[Tue, 11 Mar 2025 23:46:35 GMT] [CPU RAM used: 12312 MB]  Loaded output dset with 308935 items
[Tue, 11 Mar 2025 23:46:35 GMT] [CPU RAM used: 12312 MB] Passthrough results ['ctf', 'pick_stats']
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12600 MB]  Loaded passthrough dset with 401161 items
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12549 MB]  Intersection of output and passthrough has 308935 items
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12549 MB]  Output dataset contains: ['ctf', 'pick_stats']
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12549 MB]  Outputting passthrough result ctf
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12549 MB]  Outputting passthrough result pick_stats
[Tue, 11 Mar 2025 23:46:38 GMT] [CPU RAM used: 12477 MB] Checking outputs for output group micrographs
[Tue, 11 Mar 2025 23:46:39 GMT] [CPU RAM used: 12533 MB] Checking outputs for output group micrographs_incomplete
[Tue, 11 Mar 2025 23:46:39 GMT] [CPU RAM used: 12533 MB] Checking outputs for output group particles
[Tue, 11 Mar 2025 23:46:42 GMT] [CPU RAM used: 12535 MB] Updating job size...
[Tue, 11 Mar 2025 23:46:47 GMT] [CPU RAM used: 12535 MB] Exporting job and creating csg files...
[Tue, 11 Mar 2025 23:47:05 GMT] [CPU RAM used: 12535 MB] ***************************************************************
[Tue, 11 Mar 2025 23:47:05 GMT] [CPU RAM used: 12535 MB] Job complete. Total time 22795.63s
4

Hi @segarka!

Is it possible that your particles are being rejected for being too close to the edge of the micrograph? At the end of the Extract From Micrographs job, there’s a message like this which will tell you how many particles were rejected for this reason:

Number of particles rejected near micrograph edges: 9146

5

Thanks for your comment! I have considered that. However, because these particles are being re-extracted after initial extraction with downsampling, I would’ve thought that those particles that were too close to the edge would’ve been ignored when I extracted the first time. The box size is the same between both extractions, just the Fourier-crop box size changed. But I could be misunderstanding extractions…

6

Hi @segarka, could you post the message I mentioned above from your extract job? That would let us know how many particles were rejected for being near the edges of the micrograph