All data the workflow export

Data Management
1

Can I export all job types and their sequences and even random seed used in the every job?

So maybe one day when I need to reproduce the process, I just need to put my raw data and gain in the original folder?

2

@J6H55 There are several CryoSPARC features that might be relevant in this context.

  1. CryoSPARC workflows. You may

    • select the job producing the final result along with its ancestors
    • create a workflow
    • export the workflow as a text file

    Be aware that the workflows feature alone may not preserve all the information needed, particularly if:

    • the job pipeline includes interactive jobs
    • the job pipeline involves the selective connection of some multi-class results, like from heterogeneous refinement
    • the default rather than a custom random seed was used anywhere in the job chain

  2. The data cleanup tool allows for the selective clearing of deterministic pre-processing jobs, that can be easily rerun. Clearing these jobs, and potentially clearing

    should significantly reduce the storage footprint of the project directory and make it more feasible to preserve and reuse the project directory in the future.

  3. One can use CryoSPARC tools to script a sequence of processing steps. Such a script could be used to repeat these steps in the future. CryoSPARC tools can also be used to extract relevant job information from a selection of completed CryoSPARC jobs.

    project_uid = "P296"  # extract from this project's jobs
output_dir = "~/"  # write output file to home dir

master_hostname = "localhost"  # should match hostname portion of login token
base_port = 62000  # should match port portion of login token

import cryosparc.tools

cs = cryosparc.tools.CryoSPARC(f"http://{master_hostname}:{base_port}")
assert cs.test_connection()
api = cs.api

import gzip
import pathlib
from datetime import datetime

import yaml

jobs_metadata = {
    job["uid"]: {
        "type": job["spec"]["type"],
        "cs_version": job["version"],
        "params": job["spec"]["params"],
        "inputs": job["spec"]["inputs"],
        "outputs": job["spec"]["outputs"],
        "events": [
            e.text.strip()
            for e in filter(
                lambda x: x.type == "text",
                api.jobs.get_event_logs(project_uid, job["uid"]),
            )
        ],
    }
    for job in sorted(
        map(
            lambda x: x.model_dump(),
            filter(
                # only include completed jobs
                lambda x: x.status == "completed",
                api.jobs.find(project_uid=project_uid),
            ),
        ),
        key=lambda j: int(j["uid"][1:]),
    )
}

output_path = (
    pathlib.Path(output_dir).expanduser()
    / f"{project_uid}_jobinfo_{datetime.now().strftime('%Y%m%d%H%M%S')}.yaml.gz"
)


with gzip.open(output_path, "wt") as ofh:
    yaml.dump(jobs_metadata, ofh, sort_keys=False)

print(f"Saved job info to {output_path}.")

    Text from the event logs is included to capture user inputs to interactive jobs. A YAML file like the one created by the code can be used for reference by humans (to look up job connections and parameters) or as a data source for other scripts.

Whichever method or combination of methods you choose needs to be tested and refined to ensure suitability for your use case. Do not delete any processing results until you have confirmed that your reproduction strategy and raw data can recreate the original processing results.