Cryosparc V4.1.0 is able to recognize both GPUs in workstation, but only GPU0 is used

Troubleshooting
1

Hi Cryosparc community,

I recently updated the Cryosparc version of one of workstations to V4.1.0. Everything seemed to went really smooth as the GUI displays both GPUs in the machine as available. However, when I set a query to be parallelized in both GPUs, only one of them is allocated. As you can see in this screenshot from a CTF estimation job

image
image1135×355 37.3 KB

I was able to use both cards in this machine before as you can see in this log from a CTF estimation job that I run using CryoSPARC V 3.2

*License is valid.*

*Launching job on lane default target kiribath ...*

*Running job on master node hostname kiribath*
*[CPU: 80.5 MB]*

*Project P2 Job J44 Started*
*[CPU: 80.5 MB]*

*Master running v3.3.1, worker running v3.3.1*
*[CPU: 81.0 MB]*
*Working in directory: /data1/cryosparc_projects/P2/J44*
*[CPU: 81.0 MB]*
*Running on lane default*
*[CPU: 81.0 MB]*

*Resources allocated: *
*[CPU: 81.0 MB]*
*  Worker:  kiribath*
*[CPU: 81.0 MB]*
*  CPU   :  [0, 1, 2, 3]*
*[CPU: 81.0 MB]*
*  GPU   :  [0, 1]*
*[CPU: 81.0 MB]*
*  RAM   :  [0]*
*[CPU: 81.0 MB]*
*  SSD   :  False*

this is my output from nvidia-smi, which shows that both GPUs are properly behaving in my system.

*Fri Jan  6 11:44:17 2023       *
*+-----------------------------------------------------------------------------+*
*| NVIDIA-SMI 470.161.03   Driver Version: 470.161.03   CUDA Version: 11.4     |*
*|-------------------------------+----------------------+----------------------+*
*| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |*
*| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |*
*|                               |                      |               MIG M. |*
*|===============================+======================+======================|*
*|   0  NVIDIA GeForce ...  Off  | 00000000:1A:00.0  On |                  N/A |*
*| 30%   41C    P8    36W / 350W |    881MiB / 24259MiB |     38%      Default |*
*|                               |                      |                  N/A |*
*+-------------------------------+----------------------+----------------------+*
*|   1  NVIDIA GeForce ...  Off  | 00000000:68:00.0 Off |                  N/A |*
*| 62%   60C    P2   153W / 350W |   1700MiB / 24268MiB |     14%      Default |*
*|                               |                      |                  N/A |*
*+-------------------------------+----------------------+----------------------+*

Have you seen this issue before? Please let me know if you need additional information from my end

Thank you in advance :slight_smile:

2

Please can you post the following information:

  1. the "params_spec": section from /data1/cryosparc_projects/CS-get3d-get4-5/J7/job.json
  2. the output of
    /path/to/cryospar_worker/bin/cryosparcw gpulist
  3. the output of
    cryosparcm cli "get_scheduler_targets()