Master and worker instances

Install, Configure and Update
#1

Hi,

A quick question. Is there any way that a computer could be a worker, but at the same time a self-contained master/worker? What I mean that, given two computers, A and B, master A would be sending jobs to B, but there would also be a possibility for B to be a master and the worker on its own?

Please let me know,

Best,

Jean-Paul

#2

Hi @jparmache_psu, to clarify, do you want to run two cryoSPARC master instances (including a dedicated UI and database) on both machines A and B? Or are A and B both GPU machines and you’d like to be able to queue jobs to both? Or something else entirely?

You can install cryoSPARC on B as a combined master/worker with the Single Workstation installation instructions.

You can also connect the master A as a worker the same way you would when you’re connecting B to A as a worker, just using localhost for the --master hostname:

#3

Hi @nfrasser,

Would that require separate folders for independent worker configuration? I am asking about submitting to multiple computers when there is a need for computationally intensive, but have individual workstations for individual projects. I was wondering if it is possible to configure computer A to be able to calculate on computer A, but also submit to computer B, C and D (and each of these workstations to also have an instance that would make them independent) - but that all, without reinstallations.

Please let me know,

Best,

Jean-Paul

#4

@jparmache_psu this is possible with a few considerations:

  • Each workstation must run the same version of master and worker
  • The UNIX user account that runs cryoSPARC must have passwordless SSH access to each machine it submits jobs to
  • Each workstation must be on the same network and have certain networking ports open and accessible to the other workstations
  • Each workstation must have access to a shared file system where cryoEM data and processing results are stored
  • I do not recommend installing multiple instances of the cryoSPARC master package (includes the database and web interface)
    • Instead, pick a single workstation to host the master and use separate cryoSPARC user accounts on each workstation to isolate projects
    • If the workstations are set-up with password-less SSH, you can access the master from each one via SSH port forwarding
    • If you do choose to run a master instance on each workstation, do not install it on a shared file system and run it from multiple workstations. These will conflict with each other. You will have to install it once on each workstation’s non-shared drive.
  • You may install the cryoSPARC worker bundle only once by installing it into the shared file system without negative effects
    • If you do this, make sure identical Nvidia Driver and CUDA versions are installed on each workstation

Take a read through the installation prerequisites and installation instructions, where we cover this in more detail.

Hope this helps!

Run CS v3.1 master process on "non-master' node