Lanes added after update do not submit jobs. Cluster configuration

[wtempel]

Please can you post additional information:

1. the output of the command

   cryosparcm cli "get_scheduler_targets()"
   

   and indicate the names of the lanes that function and malfunction, respectively.
2. Are the project directories available (under matching paths) on the new GPU nodes?
3. Post any messages related to the jobs’ submission failures from logs or the UI, such as from
   • jobs’ Event Logs
   • command_core log
     cryosparcm log command_core
   • slurm logs

Possibly unrelated to your immediate concern of jobs not being submitted:

@mruetter

ml 2021b
ml CUDA/11.6.0

Are you sure these modules are needed for (and do not interfere with) CryoSPARC jobs, as CryoSPARC v4.4 includes its own CUDA dependencies.

@mruetter

available_devs=""
for devidx in $(seq 0 15);
do
    if [[ -z $(nvidia-smi -i $devidx --query-compute-apps=pid --format=csv,noheader) ]] ; then
        if [[ -z "$available_devs" ]] ; then
            available_devs=$devidx
        else
            available_devs=$available_devs,$devidx
        fi
    fi
done
export CUDA_VISIBLE_DEVICES=$available_devs

This code block may fail to properly assign/restrict GPU resources to jobs. For a more robust alternative, consider CUDA_ERROR_NO_DEVICE - but only when AF2 is running! - #9 by wtempel.