Could not get image for micrograph

Application
1

I’m supporting a few researchers who are running Cryosparc. We’re running Cryosparc 4.2.1 currently, but we were seeing this same issue in 4.1.2 as well.

When running a “manual picker” job which is using topaz denoised micrographs with particles from an “inspect picks” job - when attempting to interactively select micrographs I receive an " Could not get image for micrograph: xxxxxxxxxxxx" error.

When looking at the request via the browser inspect tools, I’m see a 500 error for https://myhost.example.com/api/cmd/interactive/myhost.example.com/41235/get_micrograph_data

with a response of “Cannot connect to CryoSPARC Command Interactive API”

I really have no idea when this issue started appearing, but from what the researchers tell me, this has worked for them in the past.

Looking for any help that you could provide.

2

Are you actually running with a certificate/https, or should that request be to http?

What is the output of curl http://myhost.example.com:39002 vs curl https://myhost.example.com:39002 ?

Edit: NVM, after thinking about it more, if other jobs are working, it’s highly unlikely that’s the culprit.

3

If you hammer through a small tutorial dataset, do you see the same issue? Perhaps check system logs for any disk related errors, memory errors etc., because it sounds like a file is missing or damaged.

4

Thanks - we are using an nginx proxy and do have a cert.

5

THanks @rbs_sci - this led me to look a the topaz denoise job. At first glance it would appear that the denoise job ran successfully, however, it appears that the python errors encountered during the job aren’t interpreted in a way that makes it obvious to the cryosparc user that something is wrong.

When looking at the job directory, the “denoised_micrographs” folder is empty, which I assume should be populated with micrographs. I have not investigated this much further yet, but here is the trace:

Starting Topaz process using version 0.2.5a...

Using Topaz provided pretrained model.


Beginning Topaz denoising command by running command /user/r/c/rallycat/miniconda3/envs/topaz/bin/topaz denoise [107 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --normalize --patch-size 1536 --patch-padding 256 --output /netfiles/rallylab/cryosparc/CS-temp/J150/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet


Distributing over 4 processes...

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp line=405 error=8 : invalid device function

# using device=0 with cuda=True

# Loading model: unet

Traceback (most recent call last):

File "/user/r/c/rallycat/miniconda3/envs/topaz/bin/topaz", line 33, in <module>

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main

args.func(args)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 547, in main

, use_cuda=use_cuda

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/denoise.py", line 292, in denoise_image

mic += dn.denoise(model, x, patch_size=patch_size, padding=padding)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 68, in denoise

return denoise_patches(model, x, patch_size, padding=padding)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 92, in denoise_patches

yij = model(xij).squeeze() # denoise the patch

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__

result = self.forward(*input, **kwargs)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/denoise.py", line 499, in forward

p1 = self.enc1(x)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__

result = self.forward(*input, **kwargs)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/container.py", line 92, in forward

input = module(input)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/module.py", line 489, in __call__

result = self.forward(*input, **kwargs)

File "/user/r/c/rallycat/miniconda3/envs/topaz/lib/python3.6/site-packages/torch/nn/modules/conv.py", line 320, in forward

self.padding, self.dilation, self.groups)

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED

THCudaCheck FAIL file=/opt/conda/conda-bld/pytorch_1544174967633/work/aten/src/THC/THCGeneral.cpp line=405 error=8 : invalid device function

# using device=0 with cuda=True

# Loading model: unet
6

So Topaz is flat out failing.

Does it work outside of CryoSPARC? That’s the next thing to check/test…

I haven’t used Topaz much so can’t help very much on that front. :frowning:

1 Like