Hi,
I tried to follow the tutorial to run Topaz denoise in CS. At step 2 of create denoising job, I input the exposures output from the completed CTF estimationjob into the micrographs input, the job is marked as complete, without generating side-by-side micrograph comparisons. I copied the event log as below. Can anybody give some suggestions? Thanks!
[CPU: 84.1 MB]
Job J59 Started
[CPU: 84.1 MB]
Master running v4.0.1, worker running v4.0.1
[CPU: 84.1 MB]
Working in directory: /mnt/atlantis/homes/yy314/20221010-TalosArctica/relion_1011/CS_1011/P17/J59
[CPU: 84.1 MB]
Running on lane proteus-8
[CPU: 84.1 MB]
Resources allocated:
[CPU: 84.1 MB]
Worker: proteus-8
[CPU: 84.1 MB]
CPU : [0]
[CPU: 84.3 MB]
GPU : [0, 1]
[CPU: 84.3 MB]
RAM : [0]
[CPU: 84.3 MB]
SSD : False
[CPU: 84.3 MB]
--------------------------------------------------------------
[CPU: 84.3 MB]
Importing job module for job type topaz_denoise...
[CPU: 212.9 MB]
Job ready to run
[CPU: 212.9 MB]
***************************************************************
[CPU: 212.9 MB]
Topaz is a particle detection tool created by Tristan Bepler and Alex J. Noble. Citations: - Bepler, T., Morin, A., Rapp, M. et al. Positive-unlabeled convolutional neural networks for particle picking in cryo-electron micrographs. Nat Methods 16, 1153-1160 (2019) doi:10.1038/s41592-019-0575-8 - Bepler, T., Noble, A.J., Berger, B. Topaz-Denoise: general deep denoising models for cryoEM. bioRxiv 838920 (2019) doi: https://doi.org/10.1101/838920 Structura Biotechnology Inc. and cryoSPARC do not license Topaz nor distribute Topaz binaries. Please ensure you have your own copy of Topaz licensed and installed under the terms of its GNU General Public License v3.0, available for review at: https://github.com/tbepler/topaz/blob/master/LICENSE. ***************************************************************
[CPU: 213.1 MB]
Starting Topaz process using version 0.2.4...
[CPU: 213.1 MB]
Using Topaz provided pretrained model.
[CPU: 213.1 MB]
Beginning Topaz denoising command by running command /mnt/atlantis/homes/yy314/miniconda3/envs/topaz/bin/topaz denoise [MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /mnt/atlantis/homes/yy314/20221010-TalosArctica/relion_1011/CS_1011/P17/J59/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet
[CPU: 213.1 MB]
Distributing over 2 processes...
[CPU: 215.2 MB]
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp line=405 error=8 : invalid device function
[CPU: 215.2 MB]
# using device=0 with cuda=True
[CPU: 215.2 MB]
# Loading model: unet
[CPU: 215.2 MB]
THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp line=405 error=8 : invalid device function
[CPU: 215.2 MB]
# using device=1 with cuda=True
[CPU: 215.2 MB]
# Loading model: unet
[CPU: 215.2 MB]
Traceback (most recent call last):
[CPU: 215.2 MB]
Traceback (most recent call last):
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/bin/topaz", line 11, in <module>
[CPU: 215.2 MB]
load_entry_point('topaz-em==0.2.4', 'console_scripts', 'topaz')()
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/bin/topaz", line 11, in <module>
[CPU: 215.2 MB]
load_entry_point('topaz-em==0.2.4', 'console_scripts', 'topaz')()
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main
[CPU: 215.2 MB]
args.func(args)
[CPU: 215.2 MB]
args.func(args)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 547, in main
[CPU: 215.2 MB]
, use_cuda=use_cuda
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 547, in main
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 292, in denoise_image
[CPU: 215.2 MB]
, use_cuda=use_cuda
[CPU: 215.2 MB]
mic += dn.denoise(model, x, patch_size=patch_size, padding=padding)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 292, in denoise_image
[CPU: 215.2 MB]
mic += dn.denoise(model, x, patch_size=patch_size, padding=padding)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 68, in denoise
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 68, in denoise
[CPU: 215.2 MB]
return denoise_patches(model, x, patch_size, padding=padding)
[CPU: 215.2 MB]
return denoise_patches(model, x, patch_size, padding=padding)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 92, in denoise_patches
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 92, in denoise_patches
[CPU: 215.2 MB]
yij = model(xij).squeeze() # denoise the patch
[CPU: 215.2 MB]
yij = model(xij).squeeze() # denoise the patch
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 476, in forward
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 476, in forward
[CPU: 215.2 MB]
p1 = self.enc1(x)
[CPU: 215.2 MB]
p1 = self.enc1(x)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward
[CPU: 215.2 MB]
input = module(input)
[CPU: 215.2 MB]
input = module(input)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
result = self.forward(*input, **kwargs)
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
[CPU: 215.2 MB]
File "/mnt/atlantis/homes/yy314/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward
[CPU: 215.2 MB]
self.padding, self.dilation, self.groups)
[CPU: 215.2 MB]
self.padding, self.dilation, self.groups)
[CPU: 215.2 MB]
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
[CPU: 215.2 MB]
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
[CPU: 213.3 MB]
Topaz denoising command complete in 17.754s.
[CPU: 213.3 MB]
Converting Topaz outputs to cryoSPARC outputs...
[CPU: 213.4 MB]
Finished Topaz process in 17.90s
[CPU: 213.4 MB]
--------------------------------------------------------------
[CPU: 213.4 MB]
Compiling job outputs...
[CPU: 213.4 MB]
Passing through outputs for output group denoised_micrographs from input group micrographs
[CPU: 213.7 MB]
This job outputted results ['micrograph_blob', 'micrograph_blob_denoised']
[CPU: 213.7 MB]
Loaded output dset with 358 items
[CPU: 213.7 MB]
Passthrough results ['mscope_params', 'movie_blob', 'background_blob', 'ctf', 'ctf_stats', 'rigid_motion', 'spline_motion', 'micrograph_blob_non_dw', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x']
[CPU: 214.3 MB]
Loaded passthrough dset with 358 items
[CPU: 214.5 MB]
Intersection of output and passthrough has 358 items
[CPU: 214.5 MB]
Checking outputs for output group denoised_micrographs
[CPU: 214.5 MB]
Updating job size...
[CPU: 214.5 MB]
Exporting job and creating csg files...
[CPU: 214.5 MB]
***************************************************************
[CPU: 214.5 MB]
Job complete. Total time 31.62s