When running 3D-VA display with many input components, it only processes the first 20 - any of the ones after 20 are not processed and end up in the passthrough.
I realize such large numbers of components are not often necessary, but I think for symmetric systems they probably will be (in this case the 20th component was still quite informative).
I tried to work around this manually by running a second 3D-VA display job with just the missing components (by replacing them in the inputs), but it failed with the attached error:
[CPU: 434.8 MB]
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 96, in cryosparc_compute.run.main
File "cryosparc_master/cryosparc_compute/jobs/var3D/run_disp.py", line 99, in cryosparc_compute.jobs.var3D.run_disp.run
File "/home/user/software/cryosparc/cryosparc2_worker/cryosparc_tools/cryosparc/dataset.py", line 683, in __getitem__
return Column(get_data_field(self._data, key), self._data)
File "/home/user/software/cryosparc/cryosparc2_worker/cryosparc_tools/cryosparc/dtype.py", line 113, in get_data_field
return makefield(field, get_data_field_dtype(data, field))
File "/home/user/software/cryosparc/cryosparc2_worker/cryosparc_tools/cryosparc/dtype.py", line 119, in get_data_field_dtype
raise KeyError(f"Unknown dataset field {field} or field type {t}")
KeyError: 'Unknown dataset field components_mode_4/value or field type 0'
EDIT2: Also, in this case, the order of the the component images in 3D-VA display no longer matches the naming of the volume series, which is confusing - in the originating 3D-VA job, the order is correct, in the 3D-VA display job they are incorrect and no longer match the features in the volume series.
Thanks for reporting – right now a limitation of the job is only processing 20 components. However, you should be able to use the individual component output groups from 3DVA for both particles and volumes.
E.g. Here’s a 3DVA Display job’s inputs where I connected the input component slots for components 0, 1, and 2 to intake the 3DVA’s components 1, 2, and 5 (for both particles and volumes). Could you check that the 3DVA Display job you’re referring to in your edit has an analogous input tab to this?
Hi Michael - yes sorry that was my mistake, I had only replaced the particle and not the volume inputs.
However more of an issue is what I mentioned in “EDIT2” - the images of the eigenvolumes in 3D-VA display are sometimes wrong, they are mixed up.
Some of them correspond to the correct volume series, and some of them do not - this happens with less than 20 components also.
E.g. for example, let’s say I see a component corresponding to one subunit disappearing as mode 4 in the output of 3D-VA - this component image might then be listed as component 12 in 3D-VA display, but the volume series for component 4 is the one that matches the image that is now listed as component 12. Does that make sense?
One note about the 3DVA job is that after the final iteration, the latent coordinates and 3D components are transformed through a linear transform that ensures that the jth latent coordinate spans the data covariance’s jth largest magnitude eigenvector. The final variability subspace that 3DVA finds is preserved, but all variability components are intermixed by this process. Given this, if drawing inferences based on the component slice plots, its best to look at the final set of slice plots in the 3DVA Job. These plots are titled Result Mode 0x for the xth component, rather than just Mode 0x.
If the above doesn’t explain the discrepancy, could you also post the un-collapsed inputs tab (particle & volumes inputs) for the 3DVA Display job that shows this mismatch?
Yes, I am looking at the final set of slice plots, and this does not explain it.
The inputs are posted below, but this does not show the mismatch.
The issue is that the final slice plots in 3D-VA are correct, and consistent with the density features of the volume series in 3D-VA display, while the slice plots in 3D-VA display are wrong - the order is mixed up, and does not match the features observed in the volume series. This does not happen for all the components, but is very obvious for some of them.
To follow-up, we identified an issue with 3DVA Display that caused the order of output components to be scrambled when there are more than 10 components. We will aim to fix this in an upcoming release. Once fixed, it should be possible to use 3DVA Display as intended to process any subset of components as long as the particle and volume slots are in agreement.
