It was my understanding that when force hard classification is switched on in Class3D, the per-particle ESS values should be 1.0 by definition - this has certainly been the case in runs so far.
However, in a recent run with force hard classification switched on, I see particles with ESS of either 1 or 2, and a final average ESS of 1.8 - is this expected under certain circumstances, or is it a bug?
Nothing especially unusual about the parameters, with the possible exception of a high-ish target resolution (3Å) and non-default number of epochs (10) and batch size (10000).
Perhaps of note, there are <1000 particles in class 5, but the “effective size” of the class is given as >100k, which seems hard to reconcile with the per-particle ESS values:
Also weird - the number of particles in classes initially numbered 4 & 5 (but no others) changes in the final step where classes are reordered by size. Is this expected?
@olibclarke – This is very strange. You are right that ESS should be 1.0 by definition. I tried for a bit to reproduce this behaviour to no avail :(. Based on your description, it seems like the class posteriors are somehow in a state where they sum to 2.0 rather than to 1.0.
The “effective size” is the sum of the posteriors across all particles for each class, so class 5 must have most of this superfluous probability mass (i.e., a particle may have a posterior that looks like [0,1,0,0,0,1]; it is assigned to class 1 but contributes ‘size’ to class 1 and 5).
Definitely not expected. When we re-order classes, we recompute the assignments via an argmax call on the class posteriors. I suspect that some particles have class posteriors that are ‘two hot’ (as above) and upon reordering get switched (since argmax will return the first index if there are numerically identical values).
One quick follow up: do you see ESS values of 2 from the very first O-EM iteration?
It is quite reproducible even with different parameters using this set of particle/mask inputs, so I can send them if you would like to reproduce/debug?
