When we install cryosparc on a node in our cluster, the worker is installed separately on a node with NVidia k80 GPUs. However, cryosparc jobs are subsequently run on a variety of GPU node types (k20x, k80, p100, v100, a100). Does it matter what kind of GPU on which the cryosparc worker is installed? The CUDA version is always the same (11.2.2). Is there anything about the GPU device that is hard-coded onto disk, or does cryosparc dynamically determine GPU attributes for each job?
Hi,
we are currently running a slurm cluster that has nodes with 2080Tis, a 3090, a 4090, 2070 super and quadro 8000s without issues (each node only has one type of GPU).
Cuda version (11.8) and nvidia driver version (525) are the same on each node.
The cryosparc worker executable is the same.
We have different lanes in cryosparc to account for the different performance level of each node type, but cryosparc recognises the GPU it is running on even when a single queue is used.
Information on installed GPU devices of CryoSPARC-managed workers (unlike cluster workers described in GPU type for worker installation - #2 by Andrea)
- is stored inside the CryoSPARC database (which in turn is usually stored outside the
cryosparc_workersoftware directory) - can be queried with the command
cryosparcm cli "get_scheduler_targets()" - is updated for accessible workers during CryoSPARC startup or when
cryosparcw connect ...is run, but not updated for individual jobs