Cancelling all running or waiting jobs

Features and Functionality
1

Hey!

Just wondering if there are any quick cli commands to kill all jobs across all projects and workspaces in the current states: [‘queued’, ‘launched’, ‘started’, ‘waiting’, ‘running’]

Will this terminate the jobs submitted to a Slurm scheduler cleanly?

Cheers!

2

Keeping in mind that the behavior or availability of CryoSPARC cli commands is subject to change in future versions of CryoSPARC, in CryoSPARC v4.6.2 you should be able to save this script

import cryosparc_compute.client
import os

cli = cryosparc_compute.client.CommandClient(
        host=os.environ['CRYOSPARC_MASTER_HOSTNAME'],
        port=os.environ['CRYOSPARC_COMMAND_CORE_PORT'],
        service='command_core')

for status in ['queued', 'launched', 'started', 'waiting', 'running']:
    for job in cli.get_jobs_by_status(status):
        try:
            cli.kill_job(job['project_uid'], job['uid'])
            print(f"kill_job() was sent to {job['project_uid']} {job['uid']} ({status} {job['job_type']}).")
            continue
        except:
            pass
        try: # kill_job() does not work for queued jobs
            cli.clear_job(job['project_uid'], job['uid'])
            print(f"clear_job() was sent to {job['project_uid']} {job['uid']} ({status} {job['job_type']}).")
            continue
        except:
            print(f"Giving up on trying to terminate job {job['project_uid']} {job['uid']} ({status} {job['job_type']})")

to a file, like terminate_cryosparc_jobs.py, and run it

cryosparcm call python /path/to/terminate_cryosparc_jobs.py
3

Thank you @wtempel! Worked perfectly.

4

@wtempel bringing this one up again for v5. This seems to work but would it be the best way to do it? Works on the host running Cryosparc through cryosparcm.

import cli.setup_client as cc

for status in ['queued', 'launched', 'started', 'waiting', 'running']:
    for job in cc.api.jobs.find(status=[status]):
        try:
            cc.api.jobs.kill(job.project_uid, job.uid)
            cc.api.jobs.add_event_log(job.project_uid, job.uid, f"Killed {job.project_uid} {job.uid}.", type=['error'])
            continue
        except:
            pass
        try: 
            cc.api.jobs.clear(job.project_uid, job.uid)
            cc.api.jobs.add_event_log(job.project_uid, job.uid, f"Cleared {job.project_uid} {job.uid}", type=['warning'])
            continue
        except:
            print(f"Unable to terminate job {job.project_uid} {job.uid} ({status} {job.title}.)")
5

Hi @vatese, this should work as expected in this latest v5.0 version of CryoSPARC, though you can also now do this with a cryosparc-tools script, which should not require changes following new CryoSPARC releases. I definitely recommend switching to that if you can. The following script should work similarly:

from cryosparc.tools import CryoSPARC

cs = CryoSPARC("http://localhost:61000")

for job in cs.find_jobs(status='queued'):
    try:
        job.clear()
        job.log(f"Cleared {job.project_uid} {job.uid}", level='warning')
    except:
        print(f"Unable to clear job {job.project_uid} {job.uid} ({job.status} {job.title}.)")

for job in cs.find_jobs(status=['launched', 'started', 'waiting', 'running']):
    try:
        job.kill()
        job.log(f"Killed {job.project_uid} {job.uid}.", level='error')
    except:
        print(f"Unable to terminate job {job.project_uid} {job.uid} ({job.status} {job.title}.)")

Replace http://localhost:61000 with the URL you use to access CryoSPARC from your browser, (or just replace 61000 with the CRYOSPARC_BASE_PORT configured in cryosparc_master/config.sh if running locally on the same machine that hosts your CryoSPARC instance).

I recommend setting up a dedicated Python environment for running cryosparc-tools scripts instead of using CryoSPARC’s environment via cryosparcm call. Note you’ll also have to “log in” from your terminal before running the script.

Hope that helps, let me know if you have any questions or run into any issues!

1 Like