Discrete GPU usage

[wtempel]

CUDA_VISIBLE_DEVICES?
See SLURM Job Submission: Configuring node count
for an example were idle GPUs are assigned to CUDA_VISIBLE_DEVICES

@hansenbry

available_devs=""
for devidx in $(seq 0 15);
do
    if [[ -z $(nvidia-smi -i $devidx --query-compute-apps=pid --format=csv,noheader) ]] ; then
        if [[ -z "$available_devs" ]] ; then
            available_devs=$devidx
        else
            available_devs=$available_devs,$devidx
        fi
    fi
done
export CUDA_VISIBLE_DEVICES=$available_devs

and another user’s opinion on the subject:

@mcahn

Incidentally, I don’t bother to set CUDA_VISIBLE_DEVICES. SLURM will make it look like there are only as many devices as you requested, and then CryoSPARC will use those.