Memory issue on the master lane

Install, Configure and Update
1

Hi there, we have a cryosparc instance hosted on our university computing facility. The setup uses a virtual machine to host the cryosparc master, and the Slurm job scheduler to distribute the computing requests to different cryosparc workers. The virtual machine is configured with 8 CPU and 24GB of memory. Things are working quite well in general; however, we experience memory issues on the cryosparc master periodically.

The issues involve interactive job types running on only the master lane, most notably “inspect particle picks”, “select 2D classes”, and “manually curate exposures” because it is quite common for us to have 10k+ micrographs and 10M+ particles :grin:.

Below are two examples of the error message:

image
image1291×466 92.3 KB

image
image2543×982 347 KB

When facing this type of error, our solution is to restart the VM. After all, that was how NASA fixed the Hubble gyroscope.

image
image900×965 61.5 KB

Admittedly, this is not satisfying. And to improve the situation, here are the questions for the cryosparc dev team.

  1. It seems that cryosparc master always ask for 1 CPU and 24 GB of memory for these interactive job, which is based on job queuing page (pasted below). But how cryosparc master manages the available memory? As you can see from the error message above, the available memory can be between 3 GB and 10 GB.

    Screenshot from 2025-02-24 10-38-21
    Screenshot from 2025-02-24 10-38-21826×764 33.2 KB

  2. Is it possible to make these interactive jobs run on cryosparc workers as well, as the workers usually have access to much more memory?

Thank you for your help!

2

@sunch The 24 GB RAM shown on the Queue dialog

  • is a job type-specific, rough estimate that does not take into account the size and complexity of the connected inputs and therefore may underestimate actual RAM requirements
  • has no affect in the case of interactive jobs

CryoSPARC interactive jobs must run on the CryoSPARC master host and cannot be launched on another (worker) host.
In your specific case, we recommend using a VM with more RAM.

3

@wtempel, thank you for confirming the Queue dialog RAM info is job specific. As you suggestd, we did implement the increase of RAM from 24GB to 36 GB, which appears to be enough for handling 3 to 5 million particles for select 2D jobs.

But the question how cryosparc master manages the memory remains. Does that mean the cryosparc master simply uses all available memory at its disposal? In addition, it would be a more general solution to make interactive jobs run on cryosparc workers.

4

@sunch

  1. The memory use of CryoSPARC master processes may grow unboundedly within the constraints of the system. More generally, CryoSPARC does not by itself constrain jobs to the preallocated memory slots, but the system or cluster workload manager may, based on configuration.
  2. There are currently architectural limitations that mean it is not possible to run interactive jobs on CryoSPARC workers. We have noted this request.
5

Thank you for the explanation and taking the time to note the request!