Configure CryoSPARC with unlimited workers on HPC

Install, Configure and Update
1

Hi,

I’m trying to set up ‘Master-Worker‘ pattern of CryoSPARC on a large HPC, and the workload will only need a small subset of the HPC’s nodes, say 4 nodes out of 200. As I understand from CryoSPARC Guide, I need to do the followings on each of the 4 dedicated nodes:

  1. install worker program
  2. configure local cache storage ssdpath
  3. run cryosparcw connect --worker <WORKER_HOSTNAME> --master <MASTER_HOSTNAME> ... to connect to the master
  4. ensure it has TCP access to the designated communication ports of the master node

Then I configure cluster_info.json and cluster_script.sh correspondingly (ensure jobs go to only these 4 nodes) and run cryosparcm cluster connect ... to register them as a cluster lane in CryoSPARC.

But the HPC admin told me “the parallel file system (shared by all HPC nodes) should be faster than the local nvmes (on each node)”, so I can put cache and worker program (Step 1 and 2) on the shared file system, which may allow CryoSPARC to grab any node from the entire HPC through Slurm. So I wanted to ask if this idea is possible or discouraged even if possible?

  • Is there any technical requirement (other than Step 4) that would prevent this from working?
  • is it not a good idea to use a shared and large storage for caching? Can CryoSPARC manage the cache efficiently, assuming the configured cache size is 60TB and peak workload is 30-40 jobs running simultaneously? Will this slow down the job performance, compared with distributing the cache in multiple locations?
  • I’m aware that in Step 3 all workers will be registered into CryoSPARC database, will it be necessary to register all 200 nodes? Since the user won’t need to care about which node their job will be allocated, if all the nodes have identical specification with the only difference being hostnames, is it possible to just register one as a representative to display in the web UI?

Thank you in advance!
Dandan

2

A given worker or cluster should be connected to CryoSPARC using one of the

cryosparc_worker/bin/cryosparcw connect

or

cryosparc_master/bin/cryosparcm cluster connect

commands, not both. For example, connecting a cluster lane with cryosparcm cluster connect and then also connecting a node from that same cluster with cryosparcw connect would likely lead to malfunction. If your cluster is under the control of a workload manager like slurm or pbs, you might prefer the cryosparcm cluster connect method.

Please can you share details of the local nvme devices and the parallel file system available on your cluster, and how/whether the performance of the parallel file system was measured in comparison to the local nvmes performance.

If the shared filesystem is reliable, you may use it for a shared cryosparc_worker/ installation.
Will CryoSPARC project directories be stored on a shared filesystem other than the one you consider for caching? If so, is the shared file system you plan to use for caching significantly faster than the file system used for CryoSPARC project directories?
If you decide to use a shared filesystem for caching, please check whether you need to customize the CRYOSPARC_CACHE_LOCK_STRATEGY setting inside cryosparc_worker/config.sh.

3

Hi @wtempel Thank you for the reply – this really helps me sort things out.

When using cryosparc_worker/bin/cryosparcw connect, users have the chance to select a specific node for their jobs and make use of the cached particles in a later processing step if they select the same node, while using cryosparc_master/bin/cryosparcm cluster connect the node allocation will be completely determined by the scheduling policy of the workload manager, which may not guarantee a node where particle cache has existed and thus lead to an extra step of copy of particles. Is this correct?

If the shared filesystem is reliable, you may use it for a shared cryosparc_worker/ installation.

What are the considerations here for the filesystem to be reliable enough to install cryosparc_worker/ ? Do you mean a highly reliable network to access cryosparc_worker/?

If we set the issue of cache configuration aside for now and a reliable shared installation of cryosparc_worker/ is possible, do you foresee any other issues that may prevent the idea of connecting the entire cluster as worker nodes from working?

4

With a suitably configured cluster lane, one can optionally select a specific slurm node with this construct in the script template, which uses

#SBATCH --job-name cryosparc_{{ project_uid }}_{{ job_uid }}
{%- if specific_node is defined %}
#SBATCH --nodelist={{ specific_node }},
#SBATCH --nodes=1-1
{%- endif %}
#SBATCH --chdir={{ job_dir_abs }}

I mentioned reliability because there is a trade-off between reliability, endurance and speed. For example, a raid0, local nvme file system may be a plausible and speedy choice for caching, but its lifetime and reliability are shortened due to frequent cache writes and the lack of redundancy. Such a filesystem would be a poor choice for a software installation. The balance may be more favorable on a parallel filesystem, depending on implementation specifics.

Performance- or policy-related issues may arise, depending on your institution’s specific circumstances. But I think the general idea is sound and potential issues can likely be addressed by refining the configuration.