How to add a custom workflow?

Features and Functionality
1

I’d like to add a custom workflow similar to the basic workflow under job builder: workflows. Does anyone know how to do it?

2

I don’t think this is possible right now but it would be a very useful feature

Oli

2 Likes
3

Hi @mturk and @olibclarke,

Support for custom workflows is in the pipeline! I’ll update this post when it’s released.

- Suhail

4 Likes
4

Hi,
Not sure if this functionality exists already. It would be quite helpful if we can run the entire pipeline (10 jobs) with a single queue click. For example, I’m importing particles from another program and I’d like to filter the particles according to some parameter before importing them into cryoSPARC. Given that I’d be trying multiple iterations of filtering, I would I’d like to reduce the number of queuing clicks. Scenario:
Filter 1: Import particles, click queue, automatically run 10 jobs, get resolution.
Filter 2: Import particles, click queue, automatically run 10 jobs, get resolution.
Filter N: Import particles, click queue, automatically run 10 jobs, get resolution.

This flow would overwrite prior filter jobs to minimize the number of jobs seen in the tree and created in the database. I realize the detailed note keeping is being lost at this point, but who wants to clutter their workflow with hundreds of jobs? Thank you.

Best wishes,
Reza

5

Hi @rkhayat, we don’t yet have a way to do this in the cryoSPARC interface. However, if you’re willing to write some Python code then you can set this up with cryoSPARC’s interactive CLI.

First check out the Python code for the Extensive Workflow at /path/to/cryosparc_master/cryosparc_compute/jobs/workflows/buildrun_bench.py

Look for the run_extensive_workflow(job) function. It’s code automatically queues and runs all the jobs for the T20S Tutorial. You may use snippets from this code directly in cryoSPARC’s interactive CLI mode (the cryosparcm icli shell command):

$ cryosparcm icli
Python 2.7.15 |Anaconda, Inc.| (default, Oct 10 2018, 21:32:13) 
Type "copyright", "credits" or "license" for more information.

IPython 5.1.0 -- An enhanced Interactive Python.
?         -> Introduction and overview of IPython's features.
%quickref -> Quick reference.
help      -> Python's own help system.
object?   -> Details about 'object', use 'object??' for extra details.
^[[A
 connecting to cryosparc:39002 ...
 cli, rtp, db, gfs and tools ready to use

In [1]: puid = 'P2'  # Substitute with your project ID
   ...: wuid = 'W1'  # Substitute with your workspace ID
   ...: uuid = db.users.find_one({'name': 'YOUR NAME'})['_id'] # Substitute with your cryoSPARC user name
   ...: 

In [2]: movies_juid = cli.make_job(
   ...:     job_type='import_movies',
   ...:     project_uid=puid,
   ...:     workspace_uid=wuid,
   ...:     user_id=uuid,
   ...:     params={ # Substitute each parameter with your dataset parameters
   ...:         'blob_paths': '/path/to/movies/*.tif',
   ...:         'gainref_path': '/path/to/movies/gainref.mrc',
   ...:         'psize_A': 0.6575,
   ...:         'accel_kv': 300,
   ...:         'cs_mm': 2.7,
   ...:         'total_dose_e_per_A2': 53
   ...:     }
   ...: )

In [3]: cli.enqueue_job(puid, movies_juid, 'default')  # Substitute last arg with a worker lane name
Out[3]: u'launched'

In [4]:

The code also shows examples for connecting job inputs and even interacting with Interactive jobs.

Hope that helps, let me know if you run into trouble with any of that!

Nick

3 Likes
6

Since a year has passed since this topic was created, I kindly want to ask if you are still planning to enable custom workflows. It would be great to just run a pre-defined batch procedure for routine analysis.

7

Hi @tarek,

Have you seen the post above (How to add a custom workflow?) that describes how to run custom workflows via a Python script?

Does it suit your needs? Could you explain in some more detail what types of workflows you are looking to produce?

- Suhail

8

Hello!

I would also be very interested in this feature, especially during the initial stages of processing where essentially one does not change many settings. So for example have a category of ‘Job Workflow Templates’ where one could queue up several jobs from Import to 2D selection and with one click set this up for several datasets.

Thank you!

1 Like
9

Hi!

Is this feature still planned to be implemented or will we have to do with the Python script?

Thank you!

1 Like
10

Hi @sdawood & @nfrasser,

If I were to make a custom workflow script similar to the benchmark workflow one, would there be a way to get that to be a line item in the job builder section of the GUI where I could run the custom workflow just as the benchmark workflow?

Thanks,
Kye

1 Like