Topaz could not find GPU

Troubleshooting
#1

Hi, I am trying Topaz Train inside CryoSPARC but for some reason Topaz couldn’t find my GPU.
I’ve already followed the suggestion in this link to provide ~/topaz.sh as the path of the Topaz executable file. However, I still got this message, and the job will take a long time to finish.

Any suggestion is greatly appreciated!
I have GTX 3090 card, NVIDIA-SMI 455.45.01 Driver Version: 455.45.01 CUDA Version: 11.1

Screen Shot 2022-05-13 at 3.11.08 PM
Screen Shot 2022-05-13 at 3.11.08 PM1400×542 119 KB

#2

Hi,
could you please tell more about how you installed topaz? does the topaz run on GPU without calling from cryosparc?

#3

Hi, I followed this instruction to install topaz

I tried to run the command line outside cryosparc and got the same error message (fall back to CPU).

/data/topaz.sh train --train-images /data/cryosparc_dir/sample/P19/J631/image_list_train.txt --train-targets /data/cryosparc_dir//P19/J631/topaz_particles_processed_train.txt --test-images /data/cryosparc_dir/sample/P19/J631/image_list_test.txt --test-targets /data/cryosparc_dir/sample/P19/J631/topaz_particles_processed_test.txt --num-particles 100 --learning-rate 0.0002 --minibatch-size 128 --num-epochs 10 --method GE-binomial --slack -1 --autoencoder 0 --l2 0.0 --minibatch-balance 0.0625 --epoch-size 5000 --model resnet8 --units 32 --dropout 0.0 --bn on --unit-scaling 2 --ngf 32 --num-workers 8 --cross-validation-seed 520603428 --radius 3 --num-particles 100 --device 0 --no-pretrained --save-prefix=/data/cryosparc_dir/sample/P19/J631/models/model -o /data/cryosparc_dir/sample/P19/J631/train_test_curve.txt

#4

From your error, It seems that problems occurs at PyTorch, which cannot talk to CUDA. Could it be that that PyTorch which you have installed, is a CPU only version? Or for older CUDA version?
Copy your topaz.sh to other file, exchange the last line

exec topaz @

to the

python -c “import torch;print(torch.version);print(torch.version.cuda);print(torch.cuda.is_available())”

and execute. It will give you, PyTorch version, CUDA version and True if CUDA is avaliable

#5

Hi @arinkin Thank you so much for the reply!
I had problem running the command line:
(topaz) ruiz@sp:~$ /data/topaz.sh
/data/topaz.sh: line 19: syntax error near unexpected token torch.version' /data/topaz.sh: line 19: python3.6 -c “import torch;print(torch.version);print(torch.version.cuda);print(torch.cuda.is_available())”’

But here is the result by doing it manually inside python3.6
(topaz) ruiz@sp:~$ python3.6
Python 3.6.15 | packaged by conda-forge | (default, Dec 3 2021, 18:49:41)
[GCC 9.4.0] on linux
Type “help”, “copyright”, “credits” or “license” for more information.

import torch
print(torch.version)
<module ‘torch.version’ from ‘/data/bin2/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/version.py’>

print(torch.version.cuda)
None

print(torch.cuda.is_available())
False

#6

It seems during topaz installation, if I change cudatoolkit=11.2 to cudatoolkit=11.0.3, it works :thinking:

conda install topaz=0.2.4 cudatoolkit=11.0.3 -c tbepler -c pytorch -c conda-forge