Dear Cryosparc Team,
We are currently developing our pipeline for in-house users. So far it is still in the developmental stage.
We are starting to run cryosparc jobs launched via scripts using cryosparc tools API.
All jobs are created and executed inside the same dedicated project.
Most runs consist of 3 jobs linked together (import particles, import reference, run refinement).
Once the job finishes the results are copied to a different folder outside of the cryosparc partiotion and the cryosparc jobs are deleted.
So far we reached job 1000 and the execution works like a charm.
Are there any drawbacks in keeping one project for these automatic jobs which is likely to reach >10000 ones we open it up to users?
On a similar note, I can see that the front end always attempts to show me all old and new jobs in that project when clicking “card view” or “table view”. After some loading time all old/delete jobs have disappeared again. Is there a way to update the data base for this project to really remove all older jobs and not have them pop up anymore?
We are running version 4.2 (cryosparc and API) at the moment and will update to 4.4. soon.
Thanks
Ferdinand
Not CS team, but I find that once a project exceeds 500 jobs (normally I only get to about 150-250 max, even on larger projects, but a couple of recent ones have contained extreme heterogeneity of what was considered a fairly homogeneous sample - which is of course interesting for other reasons!) the UI starts to get a lot slower to respond in the card view, sometimes taking tens of seconds to load all jobs as I scroll through.
I shudder to think what 10,000+ jobs would be like in a single project.
Never had a deleted job pop up before disappearing, though. That’s new. Hm…
The number of jobs in a given projects is not limited per se, but you may want to limit the number of jobs for practical reasons:
We do not expect the CryoSPARC app to gather information on deleted jobs, but the speed of some database queries can be affected by the total number of jobs in a project, including deleted jobs.
To ensure consistency, projects are locked during job deletion. The lock may block subsequent project actions for a considerable time, for example when a job with many associated files is deleted.
Given the project locking and the given use case of programatic job creation and deletion, you may want to aim for fewer than approx. 1000 jobs, including deleted jobs, per project.
To restrain the number of jobs in a single projects, you could create jobs in separate projects according to criteria that make sense under your specific circumstances, such as projects specific to