TOPAZ denoise "succeeds" but no output denoised micrographs

Particle Picking
1

hello,

I run a TOPAZ denoising which says it succeeds and has “completed micrographs” values for subsequent jobs, yet all those subsequent jobs fail. I checked the denoised_micrographs folder and it’s empty. This is the same TOPAZ we have successfully been using for training and extraction. It usually gives that CUDA error (see my other post) but runs anyways.

Starting Topaz process using version 0.2.4...

[CPU: 244.4 MB]

Using Topaz provided pretrained model.

[CPU: 244.4 MB]

Beginning Topaz denoising command by running command /nfs/scistore14/schurgrp/cryoschurgrp/topaz.sh denoise [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /nfs/scistore14/schurgrp/cryoschurgrp/P6/J1132/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet

[CPU: 244.4 MB]

Distributing over 4 processes...

[CPU: 244.6 MB]

CudaWarning: module 'torch._C' has no attribute '_cuda_setDevice'

[CPU: 244.6 MB]

CudaWarning: module 'torch._C' has no attribute '_cuda_setDevice'

[CPU: 244.6 MB]

Falling back to CPU.

[CPU: 244.6 MB]

CudaWarning: module 'torch._C' has no attribute '_cuda_setDevice'

[CPU: 244.6 MB]

CudaWarning: module 'torch._C' has no attribute '_cuda_setDevice'

[CPU: 244.6 MB]

# using device=0 with cuda=False

[CPU: 244.6 MB]

Falling back to CPU.

[CPU: 244.6 MB]

Falling back to CPU.

[CPU: 244.6 MB]

# using device=0 with cuda=False

[CPU: 244.6 MB]

# Loading model: unet

[CPU: 244.6 MB]

# using device=0 with cuda=False

[CPU: 244.6 MB]

# Loading model: unet

[CPU: 244.6 MB]

# Loading model: unet

[CPU: 244.6 MB]

Falling back to CPU.

[CPU: 244.6 MB]

# using device=0 with cuda=False

[CPU: 244.6 MB]

# Loading model: unet

[CPU: 244.6 MB]

[CPU: 244.6 MB]

[CPU: 244.6 MB]

[CPU: 170.7 MB]

Topaz denoising command complete in 29.767s.

[CPU: 170.7 MB]

Converting Topaz outputs to cryoSPARC outputs...

[CPU: 173.4 MB]

Finished Topaz process in 29.83s

[CPU: 173.5 MB]

--------------------------------------------------------------

[CPU: 173.5 MB]

Compiling job outputs...

[CPU: 173.5 MB]

Passing through outputs for output group denoised_micrographs from input group micrographs

[CPU: 175.4 MB]

This job outputted results ['micrograph_blob', 'micrograph_blob_denoised']

[CPU: 175.4 MB]

Loaded output dset with 1 items

[CPU: 175.4 MB]

Passthrough results ['ctf', 'ctf_stats', 'micrograph_blob_non_dw', 'mscope_params']

[CPU: 175.5 MB]

Loaded passthrough dset with 1 items

[CPU: 176.2 MB]

Intersection of output and passthrough has 1 items

[CPU: 176.2 MB]

Checking outputs for output group denoised_micrographs

[CPU: 176.4 MB]

Updating job size...

[CPU: 176.4 MB]

Exporting job and creating csg files...

[CPU: 176.5 MB]

***************************************************************

[CPU: 176.5 MB]

Job complete. Total time 47.22s
2

How many micrographs are you providing, and what parameters in terms of threads etc? We have found that it silently fails if it runs out of memory - usually if we run it with just one thread it works

3

Hi Oli, thanks for responding. I actually just tried it on a single micrograph for testing purposes. Also found that if I run TOPAZ standalone in terminal it succeeded in denoising, so that’s good to know. However, it would be nice to be able to denoise within topaz as well.

1 Like
4

Please can you post the contents of /nfs/scistore14/schurgrp/cryoschurgrp/topaz.sh.
Did you use the topaz command directly or /nfs/scistore14/schurgrp/cryoschurgrp/topaz.sh for that standalone run?