Multiple exposure groups with CLI

I was going to post this question, but then I figured it out by looking at the …/cryosparc_command/command_rtp/init.py so now I am just going to post this for future reference.

My issue was that I wanted to add more exposure groups to a live session being created from a python script. Existing documentation didn’t seem to cover the answer (tutorial shows how to set up parameters of the one exposure created with the live session, but not how to add more).

So here it is: if you’d like to add more exposure groups via rtp, use this assuming that

rtp = client.CommandClient(host=host, port=command_rtp_port) // the usual

newid = rtp.exposure_group_add_new(project_uid, session_uid)

which returns the (integer) id of the new group that is then used in calls to rtp.exposure_group_update_value etc.

I have not tried that yet, but exposure groups can also be deleted with rtp.exposure_group_delete(project_uid, session_uid, exp_group_id). As far I can see following down the rabbit hole of reading the cryosparc python code, the exp_group_id assumes max(“existing exp_group_id”)+1 when a new group is added. So if you delete group #5 out of existing 9, next one will be #10 and #5 will never be reused. However, if you delete the last one (#9), its id will be reused.

Again, this ain’t really a question, so I am breaking some rules most likely, but I figured this could save time to someone else who will be, like me, trying to create multiple exposure groups via cli. Until documentation is updated, of course.

Does setting a different exposure group ID not work?

rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, **exp_group_id=1**, name='file_engine_watch_path_abs', value=file_engine_watch_path_abs)

Not for me:

Traceback (most recent call last):
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 268, in
main()
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 265, in main
methodsaction
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 236, in create_session
that.build_session()
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 171, in build_session
self.enable_datasets()
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 149, in enable_datasets
v.update(self.rtp, self.pid, self.session_id, k+1)
File “/ibbr/epo-scripts/cryingem/epycryingem/cryosparc/cslive.py”, line 107, in update
rtp.exposure_group_update_value(project_uid=pid, session_uid=sid, exp_group_id=gid, name=k, value=v)
File “/opt/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py”, line 121, in func
raise CommandError(
cryosparc_tools.cryosparc.errors.CommandError: *** (http://cryo-gpu-live:39005, code 400) Encountered ServerError from JSONRPC function “exposure_group_update_value” with params {‘project_uid’: ‘P255’, ‘session_uid’: ‘S1’, ‘exp_group_id’: 2, ‘name’: ‘file_engine_watch_path_abs’, ‘value’: ‘/mnt/beegfs/CryoEM/epozhars/R829_S3920/RAW/R829_S3920_20240204_211540/sem/micrographs/set_20240204_225704’}:
ServerError: Exposure group with ID 2 not found!
Traceback (most recent call last):
File “/opt/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py”, line 195, in wrapper
res = func(*args, **kwargs)
File “/opt/cryosparc/cryosparc_master/cryosparc_command/command_rtp/init.py”, line 625, in exposure_group_update_value
eg = rc.com.query(sess[‘exposure_groups’], lambda eg : eg[‘exp_group_id’]==exp_group_id, error=“Exposure group with ID %d not found!” % exp_group_id)
File “/opt/cryosparc/cryosparc_master/cryosparc_compute/jobs/common.py”, line 714, in query
assert res != default, error
AssertionError: Exposure group with ID 2 not found!

Hello,

I ran into the same issue. Thanks for this post, it works very well.

To add on this topic since documentation is missing on this:

More information here: Search exposures recursively via CLI - #2 by stephan

In order to create an exposure group with continuous import and searching recursively, I wrote this for loop that goes through a dictionnary (containing all path and filter information):

for key, array in ExpGrpDic.items():
    file_engine_watch_path_abs = array[0]
    file_engine_filter = array[1]
    gainref_path = array[2]
    newid = cs.rtp.exposure_group_add_new(project_uid, session_uid)
    cs.rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=int(newid), name='file_engine_watch_path_abs', value=file_engine_watch_path_abs)
    cs.rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=int(newid), name='file_engine_filter', value=file_engine_filter)
    cs.rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=int(newid), name='gainref_path', value=gainref_path)
    cs.rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=int(newid), name='file_engine_recursive', value=True)
    cs.rtp.exposure_group_update_value(project_uid=project_uid, session_uid=session_uid, exp_group_id=int(newid), name='file_engine_enable', value=True)
    print(f"Exposure group {newid} created:\nPath: {file_engine_watch_path_abs}\nType: {file_engine_filter}\nGain: {gainref_path}")
1 Like

Thanks @pozharski @ccgauvin94 @wjnicol for your posts. Please could you share what the relationships, similarities and differences are between the the exposure groups that you add to the same Live session.

In my case it’s always that datasets are located in folders with the common root that would include other unrelated datasets. Either this is because - usually due to user sloppiness - multiple data collection sessions were used or when datasets collected with different pixel sizes are in the same branch of directories (e.g. I could have a small test dataset with larger pixel size, then a regular dataset, and then another one collected with tilt). There are surely other ways to skin this cat (e.g. by placing symbolic links to multiple folders with micrographs into the same subdirectory).

In our case, 99% of the time, one exposure group is a grid dataset. When doing detergent screening or screening one system in different sample prep’ conditions I like pooling all the data in one session because it’s convenient to do and allows easier comparison when data processing within one workspace.