Exposure Group CSLive duplicate

[CryoEM2]

v4.4.1, after using live to process 5000 micrographs, I pause the session. I then add an additional exposure group pointing to different micrograph location and start session. CSlive then “finds” the original 5000micrographs and starts reprocessing them. This has happened multiple times. It should ignore the 5000 that were done and be searching for the new ones to add on and continue. Curiously, the new exposure group 2 says “5020 micrographs found” despite that folder location having only the 20. And if I start a new session and point to the 20 then it finds only the 20 (so the workaround is to run two sessions and merge particles. unfortunately I would much rather continue one session for datakeeping and to utilize well-established picking strategy and 2D templates in live2D).

And after, I can’t export the particles (or use the ones I already exported):

AssertionError: Dataset was the wrong length! Got 1482 expected 1481, Exposure UID: 1

[ChaiG]

I have yet to try adding an extra exposure group in CS Live, but, would the 'Ignore exposures from group’ toggle need to be switched on to only allow exposures from the 2nd group to be processed?

Apologies if you already tried this.

Best,

Chai

[wtempel]

@CryoEM2 Please can you post (redacting confidential info like target names) for each exposure group

• Directory to watch
• File name wildcard filter
• whether Search Recursively was enabled

[CryoEM2]

//incoming/2024/outgoing_20240729_060_1/Images-Disc1 5000 found
//incoming/2024/outgoing_20240729_060_1again/Images-Disc1 5020 found

live session starts churning over the 10020 (reloading the first set along with the second as if they are newly found).

searching for .eer
yes, recursively

Moving to new workspace and providing //incoming/2024/outgoing_20240729_060_1again/Images-Disc1 yields 20 found.

[wtempel]

@CryoEM2 Unfortunately, we were unable to reproduce the behavior. Please let us know when you encounter this issue again and you discover any patterns that might help us reproduce the issue.

[CryoEM2]

btw, for the first time I am having no problem circumventing the issue by first starting the session again, then adding the new exposure group.

[wtempel]

@CryoEM2

for the first time I am having no problem circumventing the issue by first starting the session again, then adding the new exposure group.

@CryoEM2 Please can you confirm that this workaround allows you to

@CryoEM2

continue one session for datakeeping and to utilize well-established picking strategy and 2D templates in live2D

?

[CryoEM2]

yes, restarting a completed session, then adding a new exposure group (more images on the same grid collected at a later date), allows the session to grow and the particles are automatically sorted according to the existing 2D classes as desired. I can’t recall if I manually selected “attempt resume”.

Done Full Iteration 1 took 20194.584s for 8590816 images

[2024-09-25 0:59:01.57]
[CPU: 25.89 GB Avail: 314.64 GB]
Identifying duplicate particle locations, and rejecting based on NCC score…

[2024-09-25 1:01:49.51]
[CPU: 32.26 GB Avail: 308.39 GB]
Done in 167.94s. 8585899 particles accepted and 4917 particles rejected

[2024-09-25 1:02:04.38]
[CPU: 32.26 GB Avail: 308.26 GB]
Loaded info for 200 classes in P199 S5

[2024-09-25 1:03:16.00]
[CPU: 34.62 GB Avail: 305.90 GB]
Loaded info for 8585899 particles in P199 S5

[2024-09-25 1:05:57.22]
[CPU: 33.45 GB Avail: 307.09 GB]

Checking for new particles at 09/25/2024 04:03:21
Allowed exposure groups are [1, 2]
Found 8338 matching exposures
Found 12399252 available particles
Particle box size:64
Particle pixel size:4.18
Found 12399252 matching particles
Final particle counts: 0 removed, 8590816 repeat, 3808436 added
Performing streaming classification, 12398816 particles ingested of which 3808000 particles are new and 436 particles remain
Continuing to next iteration

[2024-09-25 1:06:16.04]
[CPU: 32.31 GB Avail: 308.21 GB]
Streaming Iteration 2 ==================================================================

[2024-09-25 1:06:26.34]
[CPU: 37.02 GB Avail: 303.62 GB]
Using random seed of 941960476

[2024-09-25 1:06:28.40]
[CPU: 38.35 GB Avail: 302.29 GB]
Loading a ParticleStack with 12398816 items…

[2024-09-25 1:09:24.19]
[CPU: 36.20 GB Avail: 304.34 GB]
Done.

[2024-09-25 1:09:24.20]
[CPU: 36.20 GB Avail: 304.34 GB]
Windowing particles

[2024-09-25 1:09:24.20]
[CPU: 36.20 GB Avail: 304.34 GB]
Done.