Topaz GPU issue

Install, Configure and Update
1

License is valid.

Launching job on lane default target BIOCHEM-7048GR-TR …

Running job on master node hostname BIOCHEM-7048GR-TR

[CPU: 195.6 MB Avail: 77.54 GB]
Job J406 Started

[CPU: 195.6 MB Avail: 77.54 GB]
Master running v4.3.1, worker running v4.3.1

[CPU: 195.6 MB Avail: 77.54 GB]
Working in directory: /data/RajivS/20220719_1444_pep/P7/J406

[CPU: 195.6 MB Avail: 77.55 GB]
Running on lane default

[CPU: 195.6 MB Avail: 77.55 GB]
Resources allocated:

[CPU: 195.6 MB Avail: 77.55 GB]
Worker: BIOCHEM-7048GR-TR

[CPU: 195.6 MB Avail: 77.54 GB]
CPU : [0, 1, 2, 3]

[CPU: 195.6 MB Avail: 77.54 GB]
GPU : [0, 1, 2, 3]

[CPU: 195.6 MB Avail: 77.54 GB]
RAM : [0]

[CPU: 195.6 MB Avail: 77.54 GB]
SSD : False

[CPU: 195.6 MB Avail: 77.54 GB]

[CPU: 195.6 MB Avail: 77.54 GB]
Importing job module for job type topaz_denoise…

[CPU: 230.1 MB Avail: 77.52 GB]
Job ready to run

[CPU: 230.1 MB Avail: 77.52 GB]

[CPU: 230.1 MB Avail: 77.52 GB]
Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
Citations:

  • Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
  • Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920

Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE.

[CPU: 230.3 MB Avail: 77.42 GB]
Starting Topaz process using version 0.2.5a…

[CPU: 230.3 MB Avail: 77.42 GB]
Using Topaz provided pretrained model.

[CPU: 230.3 MB Avail: 77.42 GB]

Beginning Topaz denoising command by running command /home/spuser/miniconda3/envs/topaz/bin/topaz denoise [39 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /data/RajivS/20220719_1444_pep/P7/J406/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet unet-small fcnn affine

[CPU: 230.3 MB Avail: 77.42 GB]
Distributing over 4 processes…

[CPU: 230.5 MB Avail: 76.89 GB]
CudaWarning: module ‘torch._C’ has no attribute ‘_cuda_setDevice’

[CPU: 230.5 MB Avail: 76.89 GB]
Falling back to CPU.

[CPU: 230.5 MB Avail: 76.88 GB]

using device=1 with cuda=False

and this is not progressing!

Please help to troubleshoot.

2

Do you remember the full conda install command you used for topaz installation?
Does

still occur when you prepare a wrapper script and point the job’s Path to Topaz executable parameter to that script?

3

Yes, this is
conda install topaz=0.2.5 cudatoolkit=11 -c tbepler -c pytorch
Topaz env has cudatoolkit-11.8.0

However, cryosparc worker has cuda-11.5
Nvidia SMI reflects cuda 11.7
All other jobs runs fine except when Path to topaz given in cryosparc topaz job.

I tried wrapper script and given Path to Topaz executable parameter to that scrip, however its showing same error.

4

You may want to retry with a new environment:

conda create -n topaz2 python=3.6
conda activate topaz2
conda install topaz -c tbepler -c pytorch

which closely follows instructions here, gave cudatoolkit-11.3.1 and pytorch-1.10.2. Caveat: I am using a newer nvidia driver (v535), which may or may not matter.

1 Like
5

This worked! Thank you!

Shall I delete earlier topaz env and clean up?

6

I do not see a reason not to, but am unaware of your specific local circumstances.

1 Like