Topaz denoise don't work in 2.15.0

Elad · July 17, 2020, 3:06pm

Hi,

Topaz denoise 0.2.3 don’t work in v2.15.0. Job is finish as completed but there is not images when it finish and the output can’t be used in any down stream job.

This is the denoise log (there is some cude error) topaz picking work OK:

License is valid.

Launching job on lane default target mamba ...

Running job on master node hostname mamba

[CPU: 90.4 MB]   Project P3 Job J48 Started

[CPU: 90.4 MB]   Master running v2.15.0, worker running v2.15.0

[CPU: 90.7 MB]   Running on lane default

[CPU: 90.7 MB]   Resources allocated: 

[CPU: 90.7 MB]     Worker:  mamba

[CPU: 90.7 MB]     CPU   :  [0]

[CPU: 90.7 MB]     GPU   :  [0, 1]

[CPU: 90.7 MB]     RAM   :  [0]

[CPU: 90.7 MB]     SSD   :  False

[CPU: 90.7 MB]   --------------------------------------------------------------

[CPU: 90.7 MB]   Importing job module for job type topaz_denoise...

[CPU: 162.5 MB]  Job ready to run

[CPU: 162.5 MB]  ***************************************************************

[CPU: 162.5 MB]  Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
Citations:
- Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
- Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920

Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE.
***************************************************************


[CPU: 162.6 MB]  Starting Topaz process using version 0.2.3...

[CPU: 162.6 MB]  Using Topaz provided pretrained model.

[CPU: 162.6 MB]  
Beginning Topaz denoising command by running command /programs/x/topaz/0.2.3/bin.capsules/topaz denoise [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /data2/P4/J48/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --deconvolve --model unet


[CPU: 162.6 MB]  Distributing over 2 processes...

[CPU: 162.6 MB]  # using device=1 with cuda=True

[CPU: 162.6 MB]  # Loading model: unet

[CPU: 162.6 MB]  Traceback (most recent call last):

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/topaz/bin/topaz", line 11, in <module>

[CPU: 162.6 MB]      load_entry_point('topaz-em==0.2.3', 'console_scripts', 'topaz')()

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/main.py", line 146, in main

[CPU: 162.6 MB]      args.func(args)

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/commands/denoise.py", line 531, in main

[CPU: 162.6 MB]      , use_cuda=use_cuda

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/commands/denoise.py", line 285, in denoise_image

[CPU: 162.6 MB]      x = dn.correct_spatial_covariance(x, patch=deconv_patch)

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 304, in correct_spatial_covariance

[CPU: 162.6 MB]      f,cov = estimate_unblur_filter(x, width=width, s=s)

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 194, in estimate_unblur_filter

[CPU: 162.6 MB]      cov = spatial_covariance(x, n=width, s=s)

[CPU: 162.6 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 147, in spatial_covariance

[CPU: 162.6 MB]      cov = F.conv2d(x.unsqueeze(0).unsqueeze(1), x_c.unsqueeze(0).unsqueeze(1)).squeeze()

[CPU: 162.6 MB]  RuntimeError: CUDA error: invalid argument

[CPU: 162.7 MB]  # using device=0 with cuda=True

[CPU: 162.7 MB]  # Loading model: unet

[CPU: 162.7 MB]  Traceback (most recent call last):

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/topaz/bin/topaz", line 11, in <module>

[CPU: 162.7 MB]      load_entry_point('topaz-em==0.2.3', 'console_scripts', 'topaz')()

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/main.py", line 146, in main

[CPU: 162.7 MB]      args.func(args)

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/commands/denoise.py", line 531, in main

[CPU: 162.7 MB]      , use_cuda=use_cuda

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/commands/denoise.py", line 285, in denoise_image

[CPU: 162.7 MB]      x = dn.correct_spatial_covariance(x, patch=deconv_patch)

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 304, in correct_spatial_covariance

[CPU: 162.7 MB]      f,cov = estimate_unblur_filter(x, width=width, s=s)

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 194, in estimate_unblur_filter

[CPU: 162.7 MB]      cov = spatial_covariance(x, n=width, s=s)

[CPU: 162.7 MB]    File "/programs/x86_64-linux/topaz/0.2.3/lib/python3.7/site-packages/topaz/denoise.py", line 147, in spatial_covariance

[CPU: 162.7 MB]      cov = F.conv2d(x.unsqueeze(0).unsqueeze(1), x_c.unsqueeze(0).unsqueeze(1)).squeeze()

[CPU: 162.7 MB]  RuntimeError: CUDA error: invalid argument

[CPU: 162.7 MB]  Topaz denoising command complete in 14.529s.


[CPU: 162.7 MB]  Converting Topaz outputs to cryoSPARC outputs...

[CPU: 162.7 MB]  Finished Topaz process in 14.57s

[CPU: 162.7 MB]  --------------------------------------------------------------

[CPU: 162.7 MB]  Compiling job outputs...

[CPU: 162.7 MB]  Passing through outputs for output group denoised_micrographs from input group micrographs

[CPU: 162.9 MB]  This job outputted results ['micrograph_blob', 'micrograph_blob_denoised']

[CPU: 162.9 MB]    Loaded output dset with 237 items

[CPU: 162.9 MB]  Passthrough results ['ctf', 'ctf_stats', 'micrograph_blob_non_dw', 'mscope_params']

[CPU: 162.9 MB]    Loaded passthrough dset with 237 items

[CPU: 163.0 MB]    Intersection of output and passthrough has 237 items

[CPU: 163.0 MB]  Checking outputs for output group denoised_micrographs

[CPU: 163.0 MB]  Updating job size...

[CPU: 163.0 MB]  Exporting job and creating csg files...

[CPU: 163.0 MB]  ***************************************************************

[CPU: 163.0 MB]  Job complete. Total time 26.55s

alexjamesnoble · July 17, 2020, 9:09pm

Hi Elad,

This may primarily be a CUDA error. My guess is your current combination of cuda drivers/cuda toolkit is not what Topaz is expecting.

To test this, first try running Topaz Denoise in Cryosparc with ‘CPU only’. Let me know if that works.

Best,
-Alex

Elad · July 20, 2020, 4:28pm

I have try run denoise with CPU only option.
The job mark as complete without any error in the log but there is no images in the log and it can not open in the manual picking. Before update to 2.15.0 it was work OK.

License is valid.

Launching job on lane default target mamba ...

Running job on master node hostname mamba

[CPU: 90.4 MB]   Project P3 Job J66 Started

[CPU: 90.4 MB]   Master running v2.15.0, worker running v2.15.0

[CPU: 90.7 MB]   Running on lane default

[CPU: 90.7 MB]   Resources allocated: 

[CPU: 90.7 MB]     Worker:  mamba

[CPU: 90.7 MB]     CPU   :  [0]

[CPU: 90.7 MB]     GPU   :  []

[CPU: 90.7 MB]     RAM   :  [0]

[CPU: 90.7 MB]     SSD   :  False

[CPU: 90.7 MB]   --------------------------------------------------------------

[CPU: 90.7 MB]   Importing job module for job type topaz_denoise...

[CPU: 162.5 MB]  Job ready to run

[CPU: 162.5 MB]  ***************************************************************

[CPU: 162.5 MB]  Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble.
Citations:
- Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8
- Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920

Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE.
***************************************************************


[CPU: 162.6 MB]  Starting Topaz process using version 0.2.3...

[CPU: 162.6 MB]  Using Topaz provided pretrained model.

[CPU: 162.6 MB]  
Beginning Topaz denoising command by running command /programs/x/topaz/0.2.3/bin.capsules/topaz denoise [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device -1 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /data2/P4/J66/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --deconvolve --model unet


[CPU: 162.6 MB]  Distributing over 20 processes...

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24852 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24773 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24703 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24868 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24898 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24704 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24720 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24902 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24760 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24958 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24711 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24957 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24934 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24734 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24833 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24825 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.2 MB]  /programs/share/capsules/lib/job.sh: line 120: 24922 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.3 MB]  /programs/share/capsules/lib/job.sh: line 120: 24881 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.3 MB]  /programs/share/capsules/lib/job.sh: line 120: 24955 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.3 MB]  /programs/share/capsules/lib/job.sh: line 120: 24956 Segmentation fault      "$SB_EXECFILE" "$@"

[CPU: 163.3 MB]  Topaz denoising command complete in 76.338s.


[CPU: 163.3 MB]  Converting Topaz outputs to cryoSPARC outputs...

[CPU: 163.3 MB]  Finished Topaz process in 76.38s

[CPU: 163.3 MB]  --------------------------------------------------------------

[CPU: 163.3 MB]  Compiling job outputs...

[CPU: 163.3 MB]  Passing through outputs for output group denoised_micrographs from input group micrographs

[CPU: 163.5 MB]  This job outputted results ['micrograph_blob', 'micrograph_blob_denoised']

[CPU: 163.5 MB]    Loaded output dset with 237 items

[CPU: 163.5 MB]  Passthrough results ['ctf', 'ctf_stats', 'micrograph_blob_non_dw', 'mscope_params']

[CPU: 163.5 MB]    Loaded passthrough dset with 237 items

[CPU: 163.6 MB]    Intersection of output and passthrough has 237 items

[CPU: 163.6 MB]  Checking outputs for output group denoised_micrographs

[CPU: 163.6 MB]  Updating job size...

[CPU: 163.6 MB]  Exporting job and creating csg files...

[CPU: 163.6 MB]  ***************************************************************

[CPU: 163.6 MB]  Job complete. Total time 92.67s

olibclarke · July 24, 2020, 6:45pm

Hi Elad - I’ve seen this as well - for me, running with a single GPU gives a CUDA error (and no outputs are created, even though the job is marked as completed not failed). If I run with multiple GPUs it works fine.

olibclarke · July 24, 2020, 6:51pm

More particularly - I need to set the number of GPUs the same as the number of threads, and then it runs fine

Elad · July 24, 2020, 8:54pm

Thanks I’ll try this

Elad · July 24, 2020, 9:19pm

that work great beside if I try to use “Apply deconvolution”

alexjamesnoble · July 27, 2020, 12:07am

Thanks for responding with a solution here, Oli. I was baffled.

-Alex