Topaz error when connected through cryosparc

Install, Configure and Update
1

I am having some issues with running topaz in cryoSPARC. It is generating an error that I do not get when I run topaz on its own, using the same command.

I am attempting to install topaz on a cluster which uses a batch submission system. I have installed topaz version 0.2.5a in a conda environment. We have cryosparc v4.5.3 installed.

Here is the batch submission script I use to run topaz on its own (which runs successfully):

#!/bin/sh -l
#SBATCH --account=cryoem:ca105
#SBATCH --partition=turing
#SBATCH --job-name=topaz
#SBATCH --mem-per-cpu=3800m
#SBATCH --time=0-24:00:00
#SBATCH --ntasks-per-node=4
#SBATCH --nodes=1
#SBATCH --gpus=4
#SBATCH --error=/sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/imports/J194_denoised/topaz.err
#SBATCH --output=/sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/imports/J194_denoised/topaz.out

source activate topaz
topaz denoise /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/exports/jobs/J194_exposure_sets/J194_split_0/*/*/*patch_aligned_doseweighted.mrc --output /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/J200/denoised_micrographs

To use topaz with cryosparc, I use the following wrapper script:

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH

if [[ $@ == '--version' ]]; then
        echo '0.2.5'
        exit 0
fi

source /sdf/group/cryoem/sw/conda/etc/profile.d/conda.sh
conda activate topaz

topaz $@

conda deactivate

Finally, when I run denoising in cryosparc, here is the output:

 [CPU:  235.8 MB  Avail: 113.53 GB]

Starting Topaz process using version 0.2.5...
[CPU:  235.8 MB  Avail: 113.53 GB]

Using Topaz provided pretrained model.
[CPU:  235.8 MB  Avail: 113.53 GB]


Beginning Topaz denoising command by running command /sdf/home/g/gnye88/topaz.sh denoise [300 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY] --device 0 --format mrc --patch-size 1024 --patch-padding 500 --output /sdf/group/cryoem/g/CA105/GPN_CS/CS-fse/J205/denoised_micrographs --lowpass 1 --gaussian 0 --inv-gaussian 0 --deconv-patch 1 --pixel-cutoff 0 --model unet

[CPU:  235.8 MB  Avail: 113.53 GB]

Distributing over 2 processes...
[CPU:  235.8 MB  Avail: 113.49 GB]

/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
[CPU:  235.8 MB  Avail: 113.49 GB]

/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py:138: UserWarning: mkl-service package failed to import, therefore Intel(R) MKL initialization ensuring its correct out-of-the box operation under condition when Gnu OpenMP had already been loaded by Python process is not assured. Please install mkl-service package, see http://github.com/IntelPython/mkl-service
[CPU:  235.8 MB  Avail: 113.49 GB]

from . import _distributor_init
[CPU:  235.8 MB  Avail: 113.49 GB]

from . import _distributor_init
[CPU:  235.8 MB  Avail: 113.49 GB]

Traceback (most recent call last):
[CPU:  235.8 MB  Avail: 113.49 GB]

Traceback (most recent call last):
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/bin/topaz", line 33, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/bin/topaz", line 33, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
[CPU:  235.8 MB  Avail: 113.49 GB]

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 60, in main
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 60, in main
[CPU:  235.8 MB  Avail: 113.49 GB]

import topaz.commands.train
[CPU:  235.8 MB  Avail: 113.49 GB]

import topaz.commands.train
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 8, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/topaz/commands/train.py", line 8, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

import numpy as np
[CPU:  235.8 MB  Avail: 113.49 GB]

import numpy as np
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py", line 230, in <module>
[CPU:  235.8 MB  Avail: 113.49 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/numpy/__init__.py", line 230, in <module>
[CPU:  235.8 MB  Avail: 113.51 GB]

import mkl
[CPU:  235.8 MB  Avail: 113.53 GB]

import mkl
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 48, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 48, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

with RTLD_for_MKL():
[CPU:  235.8 MB  Avail: 113.53 GB]

with RTLD_for_MKL():
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 33, in __enter__
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/site-packages/mkl/__init__.py", line 33, in __enter__
[CPU:  235.8 MB  Avail: 113.53 GB]

import ctypes
[CPU:  235.8 MB  Avail: 113.53 GB]

import ctypes
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/ctypes/__init__.py", line 7, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

File "/sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/lib/python3.6/ctypes/__init__.py", line 7, in <module>
[CPU:  235.8 MB  Avail: 113.53 GB]

from _ctypes import Union, Structure, Array
[CPU:  235.8 MB  Avail: 113.53 GB]

from _ctypes import Union, Structure, Array
[CPU:  235.8 MB  Avail: 113.53 GB]

ImportError: libffi.so.6: cannot open shared object file: No such file or directory
[CPU:  235.8 MB  Avail: 113.53 GB]

ImportError: libffi.so.6: cannot open shared object file: No such file or directory
[CPU:  236.0 MB  Avail: 113.52 GB]

Topaz denoising command complete in 2.766s.
2

in my hands (and bash) produces this output

-bash: activate: No such file or directory

Maybe the topaz command in your slurm script uses a different installation of topaz than the one used in your CryoSPARC job?

This code block in the wrapper script may give the impression of a functional topaz installation even if the installation is defunct.
You may want to test the wrapper script outside CryoSPARC.

3

Thanks for your reply. As for your bash error, I am not sure why you are seeing that as even the topaz documentation recommends using this command (see here - topaz/README.md at master · tbepler/topaz · GitHub).

Yes, I have tested using this wrapper script outside of cryosparc (for example, in the first batch submission script I showed above, replacing the command: topaz denoise with /sdf/home/g/gnye88/topaz.sh denoise) and it has run successfully.

As for the chunk you pointed out, this is to fix an error that I saw initially which has been reported here - Version string rejected by cryoSPARC · Issue #177 · tbepler/topaz · GitHub
Is there a different way you would recommend handling this error?

I am not sure if it is running a different version because the error messages are showing the path to the same topaz install that was run previously.

4

I just found that

can be avoided if I run source activate ... from the conda base environment. The base environment is not active by default in my shell.
May I ask what the outputs of these commands are:

conda activate topaz 
conda list | grep -e tbepler -e pytorch
find /sdf/group/cryoem/g/CA105/GPN_test/conda/envs/topaz/ -name "libffi*"