Topaz Denoise job not using requested GPUs

Particle Picking
1

Dear cryosparc team,

I sent a Topaz Denoise job with the requesting 4 threads / 4 GPUs. However, the topaz launch command only requested one GPU.

Starting Topaz process using version 0.2.5a...

Using Topaz provided pretrained model.

Beginning Topaz denoising command by running command /mnt/DATA/other/topaz.sh denoise [4492 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /[REDACTED]/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet


Distributing over 4 processes...

Unless I am mistaken, --device 0 shoud be --device 0,1,2,3 (or whatever is correct in topaz syntax)
Indeed, only one GPU is used if checked via nvidia-smi on the assigned node

Might this be a bug in the function that maps GUI parameters to the topaz script? Or I am doing something wrong?

P.S.
the topaz.s script is:

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH

source /soft/anaconda/etc/profile.d/conda.sh
conda activate topaz
export CUDA_VISIBLE_DEVICES=$SLURM_JOB_GPUS
exec topaz $@
2

Does it work correctly outside of CryoSPARC if you run Topaz directly?

3

Yes, good point. From a quick look, it seems like topaz can use only one GPU (i.e. only accept a single device id as the --device parameter)
Then, at what stage does the “Number of GPUs for parallel processes” parameter come into play?

4

The Number of GPUs for parallel processes was made available in the CryoSPARC job builder for various Topaz jobs even when the respective underlying Topaz command may not support it.