Micrograph Denoiser job fails when Curate Exposures job precedes it

josh.cofsky · August 21, 2024, 10:14pm

Hi,

I’m using the following pipeline in CryoSPARC v4.5.3:

Import Movies
Patch Motion Correction (with default settings regarding generating denoiser training data)
Patch CTF
Curate Exposures (discard ~10% of micrographs due to poor CTF fit, thick ice, excessive motion, etc.—note: I want to run this now so that the Denoiser is not trained on poor-quality movies)
Micrograph Denoiser

The Micrograph Denoiser job fails with the following error message:

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 115, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/micrograph_analysis/run_denoise.py", line 97, in cryosparc_master.cryosparc_compute.jobs.micrograph_analysis.run_denoise.run_train
AssertionError: Input micrographs do not contain denoiser training data. 
        In order to train a denoiser model specific to this dataset, you must re-run Patch Motion correction 
        on a subset of the movies to produce training data. By default, denoiser training requires only 100 movies 
        with training data. 
        Please run Patch motion correction, setting the "Only process this many movies" parameter to 100, 
        and ensure the "Output denoiser training data" parameter is turned on. Then, connect the resulting 
        micrographs to this job to train a new model.
        Alternatively, you can use a denoiser model that you have trained on a different data by connecting to 
        the Denoiser model input of this job, or you can turn on the "Use pretrained model" parameter of this job 
        to use a generic pretrained denoiser model, without the need for training data.

It seems the training data generated during Patch Motion Correction is not carried through the Curate Exposures job. My workaround is as follows:

Import Movies
Patch Motion Correction (with default settings regarding generating denoiser training data)
Patch CTF
Curate Exposures
Re-run Patch Motion Correction on 100 of the accepted exposures
Micrograph Denoiser on re-Motion-Corrected exposures from #5 to generate model
Micrograph Denoiser on the exposures from #4 using the model from #6
(similar to the protocol recommended for denoising micrographs that were originally motion-corrected using earlier versions of cryoSPARC).

Is this failure a bug or am I misunderstanding something?

Thanks!
-Josh

wtempel · August 26, 2024, 5:35pm

@josh.cofsky Please can you post the outputs of the following commands

csproject=P99
for job in J197 J198 J199 J200
do
    cryosparcm cli "get_job('$csproject', '$job', 'job_type', 'version', 'params_spec', 'input_slot_groups')"
done

where P99 should be replaced by the actual CryoSPARC project ID, and J197 through P200 by the actual IDs of the “original” Patch Motion Correction, Patch CTF Estimation, Curate Exposures and (failed) Micrograph Denoiser jobs, respectively.

josh.cofsky · August 26, 2024, 8:25pm

$ for job in J200 J195 J196 J221
> do
>     cryosparcm cli "get_job('$csproject', '$job', 'job_type', 'version',
> 'params_spec', 'input_slot_groups')"
> done
{'_id': '66c4dd0899ee63d549ee62bb', 'input_slot_groups':
[{'connections': [{'group_name': 'imported_movies', 'job_uid': 'J37',
'slots': [{'group_name': 'imported_movies', 'job_uid': 'J37',
'result_name': 'movie_blob', 'result_type': 'exposure.movie_blob',
'slot_name': 'movie_blob', 'version': 'F'}, {'group_name':
'imported_movies', 'job_uid': 'J37', 'result_name': 'gain_ref_blob',
'result_type': 'exposure.gain_ref_blob', 'slot_name': 'gain_ref_blob',
'version': 'F'}, {'group_name': 'imported_movies', 'job_uid': 'J37',
'result_name': 'mscope_params', 'result_type':
'exposure.mscope_params', 'slot_name': 'mscope_params', 'version':
'F'}]}], 'count_max': inf, 'count_min': 1, 'description': 'Movies for
motion correction', 'name': 'movies', 'repeat_allowed': False,
'slots': [{'description': '', 'name': 'movie_blob', 'optional': False,
'title': 'Raw movie data', 'type': 'exposure.movie_blob'},
{'description': '', 'name': 'gain_ref_blob', 'optional': True,
'title': 'Raw movie data', 'type': 'exposure.gain_ref_blob'},
{'description': '', 'name': 'mscope_params', 'optional': False,
'title': 'Exposure parameters', 'type': 'exposure.mscope_params'}],
'title': 'Movies', 'type': 'exposure'}, {'connections': [],
'count_max': 1, 'count_min': 0, 'description': 'Dose weights from
reference-based motion correction', 'name': 'doseweights',
'repeat_allowed': False, 'slots': [{'description': '', 'name':
'refmotion_doseweights', 'optional': True, 'title': 'Empirical dose
weight parameters', 'type': 'hyperparameter.refmotion_doseweights'}],
'title': 'Dose weights', 'type': 'hyperparameter'}], 'job_type':
'patch_motion_correction_multi', 'params_spec': {'compute_num_gpus':
{'value': 4}, 'output_f16': {'value': True}}, 'project_uid': 'P181',
'uid': 'J200', 'version': 'v4.5.3'}
{'_id': '66c49d9999ee63d549e64c96', 'input_slot_groups':
[{'connections': [{'group_name': 'micrographs', 'job_uid': 'J200',
'slots': [{'group_name': 'micrographs', 'job_uid': 'J200',
'result_name': 'micrograph_blob_non_dw', 'result_type':
'exposure.micrograph_blob', 'slot_name': 'micrograph_blob_non_dw',
'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J200',
'result_name': 'micrograph_blob', 'result_type':
'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version':
'F'}, {'group_name': 'micrographs', 'job_uid': 'J200', 'result_name':
'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name':
'mscope_params', 'version': 'F'}, {'group_name': 'micrographs',
'job_uid': 'J200', 'result_name': 'micrograph_blob_non_dw_AB',
'result_type': 'exposure.micrograph_blob', 'slot_name': None,
'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J200',
'result_name': 'micrograph_thumbnail_blob_1x', 'result_type':
'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'micrographs', 'job_uid': 'J200', 'result_name':
'micrograph_thumbnail_blob_2x', 'result_type':
'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'micrographs', 'job_uid': 'J200', 'result_name':
'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name': None,
'version': 'F'}, {'group_name': 'micrographs', 'job_uid': 'J200',
'result_name': 'background_blob', 'result_type': 'exposure.stat_blob',
'slot_name': None, 'version': 'F'}, {'group_name': 'micrographs',
'job_uid': 'J200', 'result_name': 'rigid_motion', 'result_type':
'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name':
'micrographs', 'job_uid': 'J200', 'result_name': 'spline_motion',
'result_type': 'exposure.motion', 'slot_name': None, 'version': 'F'},
{'group_name': 'micrographs', 'job_uid': 'J200', 'result_name':
'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name':
None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1,
'description': 'Micrographs for CTF estimation', 'name': 'exposures',
'repeat_allowed': False, 'slots': [{'description': '', 'name':
'micrograph_blob_non_dw', 'optional': True, 'title': 'Non-doseweighted
micrograph data', 'type': 'exposure.micrograph_blob'}, {'description':
'', 'name': 'micrograph_blob', 'optional': True, 'title': 'Micrograph
data (dose-weighted)', 'type': 'exposure.micrograph_blob'},
{'description': '', 'name': 'mscope_params', 'optional': False,
'title': 'Exposure parameters', 'type': 'exposure.mscope_params'}],
'title': 'Micrographs', 'type': 'exposure'}], 'job_type':
'patch_ctf_estimation_multi', 'params_spec': {'compute_num_gpus':
{'value': 4}}, 'project_uid': 'P181', 'uid': 'J195', 'version':
'v4.5.3'}
{'_id': '66c4d59499ee63d549ecb375', 'input_slot_groups':
[{'connections': [{'group_name': 'exposures', 'job_uid': 'J195',
'slots': [{'group_name': 'exposures', 'job_uid': 'J195',
'result_name': 'mscope_params', 'result_type':
'exposure.mscope_params', 'slot_name': 'mscope_params', 'version':
'F'}, {'group_name': 'exposures', 'job_uid': 'J195', 'result_name':
'movie_blob', 'result_type': 'exposure.movie_blob', 'slot_name':
'movie_blob', 'version': 'F'}, {'group_name': 'exposures', 'job_uid':
'J195', 'result_name': 'micrograph_blob', 'result_type':
'exposure.micrograph_blob', 'slot_name': 'micrograph_blob', 'version':
'F'}, {'group_name': 'exposures', 'job_uid': 'J195', 'result_name':
'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name':
'background_blob', 'version': 'F'}, {'group_name': 'exposures',
'job_uid': 'J195', 'result_name': 'micrograph_thumbnail_blob_1x',
'result_type': 'exposure.thumbnail_blob', 'slot_name':
'micrograph_thumbnail_blob_1x', 'version': 'F'}, {'group_name':
'exposures', 'job_uid': 'J195', 'result_name':
'micrograph_thumbnail_blob_2x', 'result_type':
'exposure.thumbnail_blob', 'slot_name':
'micrograph_thumbnail_blob_2x', 'version': 'F'}, {'group_name':
'exposures', 'job_uid': 'J195', 'result_name': 'ctf', 'result_type':
'exposure.ctf', 'slot_name': 'ctf', 'version': 'F'}, {'group_name':
'exposures', 'job_uid': 'J195', 'result_name': 'ctf_stats',
'result_type': 'exposure.ctf_stats', 'slot_name': 'ctf_stats',
'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J195',
'result_name': 'rigid_motion', 'result_type': 'exposure.motion',
'slot_name': 'rigid_motion', 'version': 'F'}, {'group_name':
'exposures', 'job_uid': 'J195', 'result_name': 'spline_motion',
'result_type': 'exposure.motion', 'slot_name': 'spline_motion',
'version': 'F'}, {'group_name': 'exposures', 'job_uid': 'J195',
'result_name': 'micrograph_blob_non_dw', 'result_type':
'exposure.micrograph_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'exposures', 'job_uid': 'J195', 'result_name':
'gain_ref_blob', 'result_type': 'exposure.gain_ref_blob', 'slot_name':
None, 'version': 'F'}]}], 'count_max': inf, 'count_min': 1,
'description': 'Exposures', 'name': 'exposures', 'repeat_allowed':
False, 'slots': [{'description': '', 'name': 'mscope_params',
'optional': False, 'title': 'Microscope parameters', 'type':
'exposure.mscope_params'}, {'description': '', 'name': 'movie_blob',
'optional': True, 'title': 'Raw Movie Blob', 'type':
'exposure.movie_blob'}, {'description': '', 'name': 'micrograph_blob',
'optional': True, 'title': 'Raw micrograph data', 'type':
'exposure.micrograph_blob'}, {'description': '', 'name':
'background_blob', 'optional': True, 'title': 'Estimated background
data', 'type': 'exposure.stat_blob'}, {'description': '', 'name':
'micrograph_blob_denoised', 'optional': True, 'title': 'Denoised
micrograph data', 'type': 'exposure.micrograph_blob'}, {'description':
'', 'name': 'micrograph_blob_thumb', 'optional': True, 'title':
'Micrograph preview thumbnail (from Live)', 'type':
'exposure.micrograph_blob'}, {'description': '', 'name':
'micrograph_thumbnail_blob_1x', 'optional': True, 'title': 'Micrograph
preview thumbnail (1x)', 'type': 'exposure.thumbnail_blob'},
{'description': '', 'name': 'micrograph_thumbnail_blob_2x',
'optional': True, 'title': 'Micrograph preview thumbnail (2x)',
'type': 'exposure.thumbnail_blob'}, {'description': '', 'name': 'ctf',
'optional': True, 'title': 'CTF parameters', 'type': 'exposure.ctf'},
{'description': '', 'name': 'ctf_plotdata', 'optional': True, 'title':
'CTF plot data', 'type': 'exposure.ctf_plotdata'}, {'description': '',
'name': 'ctf_stats', 'optional': True, 'title': 'CTF stats', 'type':
'exposure.ctf_stats'}, {'description': '', 'name': 'rigid_motion',
'optional': True, 'title': 'Global Motion', 'type':
'exposure.motion'}, {'description': '', 'name': 'spline_motion',
'optional': True, 'title': 'Spline Motion', 'type':
'exposure.motion'}, {'description': '', 'name': 'annotation_blob',
'optional': True, 'title': 'Annotation data', 'type':
'exposure.annotation_blob'}], 'title': 'Exposures', 'type':
'exposure'}, {'connections': [], 'count_max': inf, 'count_min': 0,
'description': '', 'name': 'particles', 'repeat_allowed': False,
'slots': [{'description': '', 'name': 'location', 'optional': False,
'title': 'Particle locations', 'type': 'particle.location'},
{'description': '', 'name': 'pick_stats', 'optional': True, 'title':
'Particle picking statistics', 'type': 'particle.pick_stats'},
{'description': '', 'name': 'motion', 'optional': True, 'title':
'Particle local trajectories', 'type': 'particle.motion'}], 'title':
'Particles', 'type': 'particle'}], 'job_type': 'curate_exposures_v2',
'params_spec': {}, 'project_uid': 'P181', 'uid': 'J196', 'version':
'v4.5.3'}
{'_id': '66ccc9e299ee63d54985fc73', 'input_slot_groups':
[{'connections': [{'group_name': 'exposures_accepted', 'job_uid':
'J196', 'slots': [{'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'micrograph_blob_non_dw', 'result_type':
'exposure.micrograph_blob', 'slot_name': 'micrograph_blob_non_dw',
'version': 'F'}, {'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'ctf', 'result_type': 'exposure.ctf',
'slot_name': 'ctf', 'version': 'F'}, {'group_name':
'exposures_accepted', 'job_uid': 'J196', 'result_name':
'micrograph_blob', 'result_type': 'exposure.micrograph_blob',
'slot_name': None, 'version': 'F'}, {'group_name':
'exposures_accepted', 'job_uid': 'J196', 'result_name':
'mscope_params', 'result_type': 'exposure.mscope_params', 'slot_name':
None, 'version': 'F'}, {'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'movie_blob', 'result_type':
'exposure.movie_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'exposures_accepted', 'job_uid': 'J196', 'result_name':
'background_blob', 'result_type': 'exposure.stat_blob', 'slot_name':
None, 'version': 'F'}, {'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'micrograph_thumbnail_blob_1x', 'result_type':
'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'exposures_accepted', 'job_uid': 'J196', 'result_name':
'micrograph_thumbnail_blob_2x', 'result_type':
'exposure.thumbnail_blob', 'slot_name': None, 'version': 'F'},
{'group_name': 'exposures_accepted', 'job_uid': 'J196', 'result_name':
'ctf_stats', 'result_type': 'exposure.ctf_stats', 'slot_name': None,
'version': 'F'}, {'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'rigid_motion', 'result_type':
'exposure.motion', 'slot_name': None, 'version': 'F'}, {'group_name':
'exposures_accepted', 'job_uid': 'J196', 'result_name':
'spline_motion', 'result_type': 'exposure.motion', 'slot_name': None,
'version': 'F'}, {'group_name': 'exposures_accepted', 'job_uid':
'J196', 'result_name': 'gain_ref_blob', 'result_type':
'exposure.gain_ref_blob', 'slot_name': None, 'version': 'F'}]}],
'count_max': inf, 'count_min': 1, 'description': 'Micrographs to
denoise', 'name': 'exposures', 'repeat_allowed': False, 'slots':
[{'description': '', 'name': 'micrograph_blob_non_dw', 'optional':
False, 'title': 'Raw micrograph data', 'type':
'exposure.micrograph_blob'}, {'description': '', 'name':
'micrograph_blob_non_dw_AB', 'optional': True, 'title': 'Raw
micrograph data diff pair', 'type': 'exposure.micrograph_blob'},
{'description': '', 'name': 'ctf', 'optional': False, 'title': 'CTF
data', 'type': 'exposure.ctf'}], 'title': 'Exposures', 'type':
'exposure'}, {'connections': [], 'count_max': 1, 'count_min': 0,
'description': '', 'name': 'denoise_model', 'repeat_allowed': False,
'slots': [{'description': '', 'name': 'checkpoint', 'optional': False,
'title': '', 'type': 'denoise_model.checkpoint'}], 'title': 'Denoise
model', 'type': 'denoise_model'}], 'job_type': 'denoise_train',
'params_spec': {}, 'project_uid': 'P181', 'uid': 'J221', 'version':
'v4.5.3'}

wtempel · August 26, 2024, 9:22pm

It seems that your Curate Exposures job J196 is missing the

J195.exposures.micrograph_blob_non_dw_AB inputs. Can you confirm whether that slot is in the outputs of J195?

Please can you also post the outputs of

for job in J200 J195 J196
do
    cryosparcm cli "get_job('P181', '$job', 'job_type', 'version', 'created_at', 'completed_at')"
done

josh.cofsky · August 27, 2024, 6:50pm

J195.exposures.micrograph_blob_non_dw_AB is in the output of J195:

but not in the input to J196:

Here is the output to the command you posted:

$ for job in J200 J195 J196; do cryosparcm cli "get_job('P181',
'$job', 'job_type', 'version','created_at', 'completed_at')"; done
{'_id': '66c4dd0899ee63d549ee62bb', 'completed_at': 'Wed, 21 Aug 2024
02:38:35 GMT', 'created_at': 'Tue, 20 Aug 2024 18:14:32 GMT',
'job_type': 'patch_motion_correction_multi', 'project_uid': 'P181',
'uid': 'J200', 'version': 'v4.5.3'}
{'_id': '66c49d9999ee63d549e64c96', 'completed_at': 'Wed, 21 Aug 2024
17:00:18 GMT', 'created_at': 'Tue, 20 Aug 2024 13:43:53 GMT',
'job_type': 'patch_ctf_estimation_multi', 'project_uid': 'P181',
'uid': 'J195', 'version': 'v4.5.3'}
{'_id': '66c4d59499ee63d549ecb375', 'completed_at': 'Wed, 21 Aug 2024
17:18:32 GMT', 'created_at': 'Tue, 20 Aug 2024 17:42:44 GMT',
'job_type': 'curate_exposures_v2', 'project_uid': 'P181', 'uid':
'J196', 'version': 'v4.5.3'}

wtempel · August 28, 2024, 6:35pm

@josh.cofsky Can you try creating a new (not clone an existing) Curate Exposures job, and connecting to it the J195 exposures as input. Does the J195.exposures.micrograph_blob_non_dw_AB slot show up in the inputs of that newly created Curate Exposures job?

josh.cofsky · August 28, 2024, 7:03pm

That fixed it! If I create a new Curate Exposures job by making a clone of the existing one, it fails to pull the J195.exposures.micrograph_blob_non_dw_AB.F passthrough from the Patch CTF job. If I create a new Curate Exposures job from scratch, it successfully pulls the AB passthrough. We recently upgraded our CryoSPARC installation, so I must’ve cloned a Curate Exposures job from pre-upgrade when I originally created J196.

All makes sense now. Thanks @wtempel !