Merging particle sets and missing 'K' error

Particle Curation
1

Hi,

I’ve now run into this error in a few different places when merging more than 2 particle sets:

RuntimeError: Could not initialize missing alignments3D_multi (particle.alignments3D_multi) fields in dataset due to missing dtype param ‘K’. This may indicate that an input or output dataset is empty or missing expected fields.

I think this should be treated as a bug, as the normal behavior, IMO, should be to drop the fields it doesn’t know how to interpret (so either an upstream job is not setting K and it should be, or a downstream job should be ignoring alignments3D_multi if it doesn’t have enough metadata to interpret it).

But this also reminds me of a feature request that I think would be useful in general for composing jobs: the ability to filter out/remove passthrough fields.

2

@craigyk Please can you post the CryoSPARC version and the job type that produced this error.

3

v5.0.2, I’ve seen it feeding Particles selected out of Ref Auto Select 3D into several jobs, but the job that is most useful in the context of creating generic workflows, is aggregating and splitting outputs via the Particle Sets job. I can see the alignments3D_multi field in the passthrough category for this job and kind of wish I could just strip the field out at that point.

4

Thanks @craigyk. We made a note of this issue and will discuss it internally. In the meantime, assuming RuntimeError occurs near the end of the job run, it may be possible to mark the job as complete and use its outputs, with the caveat that some passthrough items may not be available downstream.

5

yup, can confirm the jobs mark complete OK

6

I’d like to +1 taking a look at dealing with the error below:

it happens whenever you merge two different paths that had different numbers of classes applied during ab-initio or heterogeneous refinement… I still think it would be perfectly acceptable to auto-strip the particle.alignments3D_multi field in the case of mismatched # of classes, maybe with a warning that it did not get passthrough

Traceback (most recent call last): File "cli/run.py", line 113, in cli.run.run_job File "/opt/cryosparc/pcs/cryosparc_worker/core/jobs.py", line 1581, in passthrough_outputs job = passthrough_input(job, input_name) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/cryosparc/pcs/cryosparc_worker/core/jobs.py", line 1641, in passthrough_input save_output( File "/opt/cryosparc/pcs/cryosparc_worker/core/jobs.py", line 2085, in save_output added_count = create_missing_fields_in_dataset(dataset, result.name, result.get_known_dtype(output.type)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/opt/cryosparc/pcs/cryosparc_worker/models/datasets.py", line 55, in create_missing_fields_in_dataset raise RuntimeError( RuntimeError: Could not initialize missing alignments3D_multi (particle.alignments3D_multi) fields in dataset due to missing dtype param 'K'. This may indicate that an input or output dataset is empty or missing expected fields.

7

Thanks @craigyk ! that is indeed the cause, and the fix we have planned!

1 Like
8

Hi @craigyk, this should be fixed in the latest CryoSPARC v5.0.6 update: alignments3D_multi results are no longer passed through for jobs where two or more particle inputs are connected. Note that jobs connected to an incorrectly passed-through output that was produced prior to this update may still fail, please clone and re-run the relevant jobs if you need to use those outputs. Thanks and let me know if you run into any further issue with this!