From v3 to v4.4.1.: Recalculate the size of all jobs in a project

Data Management
1

Hi everyone,

I have noticed that some of my projects which I have started processing in one of the v3 versions of cryoSPARC, at some point imported to the current machine while also some subversion of v3 was installed and now two years later I was planning on clearing out some space to archive while the instance were the projects exist is now in the current v4.4.1 version.

Going through the website documentation I found the following cryoSPARC cli command: update_all_job_sizes(asynchronous=True, sleep_time=0)

Is this the job I should run given the current scenario or should I run one of the update_project commands documented on the same page? If the former, as I see no project is specified… so it will update every single job for each project in the instance?

Cheers,
André

2

To help us answer your question, please can you describe your overall plan, including, but not limited to, details such as

  • the granularity at which you want to identify data that you may remove from “hot” storage (entire projects?, certain job types?)
  • the selection criteria for data that can be removed
  • the method of archiving and data removal
3

Hi Wolfram,

Thanks for reaching out!
At this point I do not want to archive the project just yet, but what I want is to restore the job size estimation for all jobs within a project to decide if I want or do not want to clear them. So it is a per job decision within a specific project (or two or three, but not all the projects in the instance are affected, or in need of clearing up).

Thanks,
André

4

@AndreGraca
Thanks for your patience while we were working on a response (and the code).
The update_all_job_sizes() was not intended to be manually run, but it may be help in your specific case.
You will want to ensure

  1. The current CryoSPARC version. At the time of writing, this is v4.4.1 with patch 240110 for v4.4.1 applied. The 240110 includes an update of the update_all_job_sizes() function and requires a CryoSPARC restart after patching.
  2. Confirm with reference to the command_core log
    cryosparcm log command_core | grep -e "update_all_job_sizes + \| Finished updating all"
    that the automatic run of update_all_job_sizes at the CryoSPARC startup is complete, based on the timestamp. Look for a log entry like
    2024-01-11 22:52:29,322 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 0 projects updated)
    that is newer than the latest CryoSPARC restart.
  3. Open a terminal for monitoring the command_core log, in which you run
    cryosparcm log command_core
    whose output should automatically be updated as new command_core log entries are recorded.
  4. In another terminal, run
    cryosparcm cli "update_all_job_sizes()"
    This command should immediately “go into the background”, but may take a long time to run.
  5. Monitor, in the former terminal window, the progress and completion of the command you entered in the latter terminal window.

[Edited (again) 2024-01-11]