Hi everyone,
I am looking for advice on a deployment strategy for our lab.
We recently set up a local workstation in our lab (equipped with a single GPU, RTX 5060 Ti). While it can handle small tasks, it lacks the power for processing larger datasets.
We want to offload the heavy processing jobs to our University’s HPC cluster while potentially keeping the project management or Master node local (if that solves the issue below).
The Challenge (IT Requirements):
I contacted our university IT to install CryoSPARC on the cluster. They are hesitant about hosting a persistent Master process because the policy strictly favors “batch jobs” with no interaction.
Here is the feedback from our IT support:
“So, for a batch job there is no interaction. It may be possible to split the workload into several, independent jobs for each stage… It seems there is a Dockerfile, so creating an apptainer image for HPC should be straightforward.”
Here are my questions:
-
Given the IT restrictions on persistent services, would it be better to host the CryoSPARC Master on our local lab workstation and configure the HPC cluster purely as a “Lane” (Worker)?
-
Does this setup work well if the HPC requires jobs to be submitted via Apptainer/Singularity containers?
-
Are there specific network requirements (e.g., ports to open) for the local Master to communicate with the remote HPC Slurm scheduler?
-
-
If we must deploy everything on the HPC, how do other users handle the “Batch Job” requirement? Do you typically request a special interactive node for the Master, or is there a way to submit the Master itself as a long-running job?
-
Since IT plans to build an Apptainer image for the cluster, are there any known pitfalls when connecting a local non-containerized Master to a remote containerized Worker?
Any advice on this “Local Master + Remote HPC” setup would be incredibly helpful.
Thanks!