Hi guys,
I find the topaz excutable path is the /usr/local/EMAN_2.91/bin/topaz instead of /home/cxz604/anaconda3/envs/topaz/bin/topaz after i finished the installation. I thought everything is good. However, when i run topaz training , I got some errors below. Does anyone have an idea what is going on ?
[CPU: 115.2 MB Avail: 322.62 GB]
Traceback (most recent call last):
[CPU: 115.2 MB Avail: 322.62 GB]
File “/usr/local/EMAN_2.91/bin/topaz”, line 11, in
[CPU: 115.2 MB Avail: 322.62 GB]
load_entry_point(‘topaz-em==0.2.4’, ‘console_scripts’, ‘topaz’)()
[CPU: 115.2 MB Avail: 322.62 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/main.py”, line 148, in main
[CPU: 115.2 MB Avail: 322.62 GB]
args.func(args)
[CPU: 115.2 MB Avail: 322.62 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/commands/train.py”, line 695, in main
[CPU: 115.2 MB Avail: 322.62 GB]
, save_prefix=save_prefix, use_cuda=use_cuda, output=output)
[CPU: 115.2 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/commands/train.py”, line 577, in fit_epochs
[CPU: 115.2 MB Avail: 322.63 GB]
, use_cuda=use_cuda, output=output)
[CPU: 115.2 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/commands/train.py”, line 557, in fit_epoch
[CPU: 115.2 MB Avail: 322.63 GB]
metrics = step_method.step(X, Y)
[CPU: 115.2 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/methods.py”, line 103, in step
[CPU: 115.2 MB Avail: 322.63 GB]
score = self.model(X).view(-1)
[CPU: 115.2 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
[CPU: 115.2 MB Avail: 322.63 GB]
result = self.forward(*input, **kwargs)
[CPU: 115.2 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/model/classifier.py”, line 28, in forward
[CPU: 115.4 MB Avail: 322.63 GB]
z = self.features(x)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
[CPU: 115.4 MB Avail: 322.63 GB]
result = self.forward(*input, **kwargs)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/model/features/resnet.py”, line 54, in forward
[CPU: 115.4 MB Avail: 322.63 GB]
z = self.features(x)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
[CPU: 115.4 MB Avail: 322.63 GB]
result = self.forward(*input, **kwargs)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/container.py”, line 117, in forward
[CPU: 115.4 MB Avail: 322.63 GB]
input = module(input)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
[CPU: 115.4 MB Avail: 322.63 GB]
result = self.forward(*input, **kwargs)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/topaz/model/features/resnet.py”, line 272, in forward
[CPU: 115.4 MB Avail: 322.63 GB]
y = self.bn(y)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/module.py”, line 727, in _call_impl
[CPU: 115.4 MB Avail: 322.63 GB]
result = self.forward(*input, **kwargs)
[CPU: 115.4 MB Avail: 322.63 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/modules/batchnorm.py”, line 136, in forward
[CPU: 115.4 MB Avail: 322.63 GB]
self.weight, self.bias, bn_training, exponential_average_factor, self.eps)
[CPU: 115.4 MB Avail: 322.64 GB]
File “/usr/local/EMAN_2.91/lib/python3.7/site-packages/torch/nn/functional.py”, line 2058, in batch_norm
[CPU: 115.4 MB Avail: 322.67 GB]
training, momentum, eps, torch.backends.cudnn.enabled
[CPU: 115.4 MB Avail: 322.70 GB]
RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
[CPU: 116.8 MB Avail: 369.00 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 115, in cryosparc_master.cryosparc_compute.run.main
File “/home/cxz604/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 384, in run_topaz_wrapper_train
utils.run_process(train_command)
File “/home/cxz604/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 99, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/usr/local/EMAN_2.91/bin/topaz train --train-images /data/zcp/cryoEM/double-mutant/non-activated-data/CS-folder/CS-double-mutant-non-activate/J867/image_list_train.txt --train-targets /data/zcp/cryoEM/double-mutant/non-activated-data/CS-folder/CS-double-mu…)