Scripting Interactive Jobs

Features and Functionality
,
#1

Hello,

I am wanting to use a curate exposures job but I am not sure on how to do this via a script. I looked through the T20S extensive workflow script (curate exposures section is below) and I am not sure how/where I would give values for ‘total motion’ or ‘CTF fit to A’ as a parameter to filter the micrographs .

Can you offer some advice on how to do this?

 # connect rigid motion and ctf estimation to exposure curation
juids['curate_exposures'] = cli.make_job(
    job_type='curate_exposures',
    title='bench/curate_exposures{}/{}'.format(suffix, run_count),
    project_uid=puid,
    workspace_uid=wuid,
    user_id=uuid,
    created_by_job_uid=workflow_juid,
    input_group_connects={'exposures': parent_input_group},
    enable_bench=True)
time.sleep(0.3)
cli.enqueue_job(puid, juids['curate_exposures'], lane)

# ---- interactive exposure curation
# wait for job to be ready
jstatus = rc.wait_job_status(puid, juids['curate_exposures'], ['waiting'], timeout=120)
assert jstatus == 'waiting', "Curate exposures did not start within 120 seconds!"
# the following will error if the response is not JSON-able:
fieldsres = cli.interactive_post(puid, juids['curate_exposures'], 'get_fields_and_thresholds', {})
cli.interactive_post(puid, juids['curate_exposures'], 'modify_all', {'value': False})
cli.interactive_post(puid, juids['curate_exposures'], 'modify_selection', {'value': True})
fields = fieldsres['fields']
field = rc.com.query(fields, lambda f: f['name'] == 'idx')
field['active'] = True
field['thresholds'][1] = 9
cli.interactive_post(puid, juids['curate_exposures'], 'set_thresholds', {'fields':fields})
counts = cli.interactive_post(puid, juids['curate_exposures'], 'get_counts', {})
assert counts['total'] == 20
assert counts['total_accepted'] == 20
assert counts['total_selected'] == 10
# now finish job
cli.interactive_post(puid, juids['curate_exposures'], 'shutdown', {})
jstatus = rc.wait_job_status(puid, juids['curate_exposures'], ['completed'], timeout=120)
assert jstatus == 'completed', "Curate exposures did not finish within 120 seconds!"

@wtempel

Thanks,
Kye

#2

I also would like to understand how to interact with jobs that use a wrapper such as motion corr or Topaz in the same scripting manner as the T20S workflow.

#3

A colleague suggested (after substituting actual project and job IDs for the P#m J# placeholders):

fieldsres = cli.interactive_post('P#', 'J#', 'get_fields_and_thresholds', {})
fields = fieldsres['fields']

motion_field = [f for f in fields if f['name'] == 'motion_total_pix'][0]
motion_field['thresholds'] = [10.0, 40.0]  # SET DESIRED THRESHOLD HERE
motion_field['active'] = True

ctf_field = [f for f in fields if f['name'] == 'ctf_fit_to_A'][0]
ctf_field['thresholds'] = [3.0, 3.5]  # SET DESIRED THRESHOLD HERE
ctf_field['active'] = True

cli.interactive_post('P#', 'J#', 'set_thresholds', {'fields': fields})

cli.interactive_post() can also be accessed via cryosparc-tools like (ensure correct parameters):

from cryosparc.tools import CryoSPARC
cs = CryoSPARC(<correct params here>)
cs.cli.interactive_post(<correct params here>)
1 Like
#4

Thanks for the response @wtempel!

This worked for setting the thresholds but for some reason I cannot get the interactive job to shutdown. Here is the error I am currently getting:

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-8-6a1a7e3452d1> in <module>
----> 1 cli.interactive_post('P138', 'J242', 'shutdown', {})

/shared/cryosparc/cryosparc_master/cryosparc_tools/cryosparc/command.py in func(*args, **kwargs)
    110
    111             assert res, f'JSON response not received for method "{key}" with params {params}'
--> 112             assert "error" not in res, f'Error for "{key}" with params {params}:\n' + format_server_error(res["error"])
    113             return res["result"]
    114

AssertionError: Error for "interactive_post" with params ('P138', 'J242', 'shutdown', {}):
ServerError: Expecting value: line 1 column 1 (char 0)
Traceback (most recent call last):
  File "/shared/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 200, in wrapper
    res = func(*args, **kwargs)
  File "/shared/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 6817, in interactive_post
    return r.json()
  File "/shared/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/requests/models.py", line 898, in json
    return complexjson.loads(self.text, **kwargs)
  File "/shared/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/shared/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/shared/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)```
#5

On CryoSPARC v4 and later, you may have to use 'shutdown_interactive' instead.

cli.interactive_post('P138', 'J242', 'shutdown_interactive', {})`

What is your CryoSPARC version?

#6

Currently using v4.1.0

Are there any other important endpoints that are useful with interactive_post? I can’t find any documentation for interactive_post on the cli reference or the cryoSPARC tools pages.

EDIT - UPDATE: ‘shutdown_interactive’ ended up working.

#8

I also would like to ask about another “issue” I’m facing.

I have found that when I use a script that queues 20+ jobs where an interactive job is in the middle and proceeding steps take a long time (i.e. motion corr or patch CTF) my terminal will time out waiting for the interactive job to run. Subsequent steps will also not queue or run because of the terminal timing out.

Is there a way to submit an interactive job with parameters already set? Or a different way to run a workflow script that would prevent this from happening?

I have resorted to running the script from the cryoSPARC icli while using the linux screen feature to allow for me to close the terminal without the terminal timing out.

#9

We do not currently host documentation for these endpoints. We have discussed your scripting observations within our team and wondered if it might be better to use cryosparc-tools to more directly manipulate dataset files than to attempt “scripting” an interactive job. For the case where one wishes to only retain only row/entries of the input that satisfy a certain condition, the Dataset.mask() method may be useful.

#10

So what you are referring too would be using the Dataset.mask feature to curate particles based on the CTF fit resolution outside the use of a curate exposures job?

#11

Yes. I was assuming that acceptance criteria would be known at the time the script is run.