The cryoSPARC version is 5.0.6. When I try to connect all the GPUs, it tells me that some of the GPUs are not available. Please refer to the screenshot below.
commad : ./bin/cryosparcw connect --master spgpu --worker spgpu --port 39000 --ssdpath /ssd/cryosparc_cache/ --update --gpus 0,1,2,3,4,5,6,7
The current system is equipped with 8 RTX 3090 GPUs, but cryoSPARC can only recognize 6 of them. NO 6 and 7 cannot recognition.
[image redacted by moderator because it contained confidential information]
nvidia-smi output:
I tried some of the methods I found, but none of them worked. Do you have any suggestions?
@saberli Please can you test CUDA device initialization outside CryoSPARC with these commands
cd $(mktemp -d)
/spshared/apps/cryosparc4/cryosparc_worker/bin/cryosparcw call python -m venv cudatest
. cudatest/bin/activate
pip install cuda-python~=12.8.0
# paste and run this multiline command
python <<EOF
from cuda.bindings import driver as cuda
# Try to initialize CUDA
try:
result = cuda.cuInit(0)[0]
if result == cuda.CUresult.CUDA_SUCCESS:
print("CUDA initialized successfully!")
device_count = cuda.cuDeviceGetCount()[1]
print(f"Found {device_count} CUDA device(s)")
else:
print(f"CUDA initialization failed with error code: {result}")
except Exception as e:
print(f"Error during CUDA initialization: {e}")
EOF
# record output of command above
# then clean up
deactivate
rm cudatest/ -r
and post the output.
This is the output, it found 8 GPUs.
