cryoSPARC v.3.1.0 and Topaz

menmt · April 28, 2021, 2:17pm

After upgrading from version 2.15.0 to version 3.1.0 I get the following error message when I use ‘Topaz Train’. I have run ‘Topaz Train’ successfully using the version 2.15.0 with the same path to the Topaz executable.

AssertionError: Cannot determine topaz version, command “/home/cryosparc_user/anaconda3/envs/topaz/bin/topaz --version” did not produce valid output: “Current thread 0x00007fc9ba20e740 (most recent call first):”

When I run the command “/home/cryosparc_user/anaconda3/envs/topaz/bin/topaz --version” in the terminal I get: “TOPAZ 0.2.3”

I also tested Topaz 0.2.4, but got the same error.

Any suggestions?

Thank you.

jelka · February 7, 2021, 10:23pm

@menmt I had same error, but figured out it was due to cryoSPARCs move from python2 to python3.
You also need to reinstall Topaz under a python3 env:

conda create -n topaz python=3
conda activate topaz
conda install topaz=0.2.3 cudatoolkit=11.0 -c tbepler -c pytorch

Now Topaz should run.
In my case, although with some error messages, but it still completes.

//Jesper

jelka · February 8, 2021, 8:16am

The errors I see are:

ERROR: ld.so: object 'cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object 'cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded: ignored.
UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)

The first two errors, seems to be related to jobs being submitted to a standalone worker and it looks for cryosparc_master path on this worker.

The last error beats me. Could be conda repo incompatibility between PyTorch and NumPy?

menmt · February 8, 2021, 3:44pm

I re-installed Topaz under a python3 env, but still had the same error…

nfrasser · February 8, 2021, 6:41pm

@jelka @menmt can you try making a bash file that activates your topaz conda environment before invoking topaz? It’ll look something like this:

#!/usr/bin/env bash
# source ~/anaconda/etc/profile.d/conda.sh
conda activate topaz
topaz $@

Optionally remove the # from the beginning of the second line if your conda environment is not automatically activated when the shell starts (then specify the correct path to conda.sh if anaconda is not installed at ~/anaconda)

Save this in your home directory as topaz.sh, then give it execute permissions via command line:

chmod +x ~/topaz.sh

Then in cryoSPARC specify ~/topaz.sh for the “Path to Topaz executable” field and run your job.

If this still leads to the same type of errors, add the following to the top of the topaz.sh, under the #!/usr/bin/env bash line:

if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH

This will deactivate cryoSPARC’s Anaconda environment before activating the Topaz environment you installed.

Let me know how that goes.

menmt · February 8, 2021, 10:19pm

Now it works although I have to deactivate cryoSZPARC’s Anaconda env before activating Topaz env.
Thak you!

Adrian · February 17, 2021, 7:00pm

Hello . I am having this error, similar to one posted here.

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370116979/work/torch/csrc/utils/tensor_numpy.cpp:141.)

When trying the bash file I got the following error:

AssertionError: Cannot determine topaz version, command “/home/adrian/topaz.sh --version” did not produce valid output: “/home/adrian/topaz.sh: línea 4: `fi’”

Do you have any insight about this?

Thank you

nfrasser · February 18, 2021, 9:47pm

It looks like you have an error in your topaz.sh file. Can you send me the full contents? Please also run the file with the following command:

/home/adrian/topaz.sh --version

And send me the full output.

Adrian · February 24, 2021, 10:33pm

Hello, I found in topaz github the following so I suppose there is no problem.

“This is a warning that new versions of pytorch started raising because our mrc micrograph/tomogram arrays are read-only. The warning is annoying, but, to my knowledge, doesn’t cause any actual errors and can safely be ignored.”

This is my full topaz.sh file

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
   conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
# source ~/anaconda/etc/profile.d/conda.sh
conda activate topaz_cryosparc
topaz $@

I will try the /home/adrian/topaz.sh --version

jelka · February 22, 2021, 12:13pm

This worked for first two problems I had.

But the my last problem which seems similar to @Adrian’s is still there:

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)

topaz --version says “TOPAZ 0.2.3”

Thanks

//Jesper

eMKiso · February 25, 2021, 8:49pm

Hi,
is it OK if we use the v0.2.4 (or v0.2.5) Topaz version with CS v3.1.0?

Thanks!

nfrasser · February 26, 2021, 5:12pm

@jelka @Adrian I’m not sure what causes the warning but it should not affect the output.

@eMKiso yes all newer versions of topaz should be supported by v3.1

zybo · April 29, 2021, 2:20pm

Hello everyone, I’m having a similar trouble, with latest version of CryoSparc and Topaz.
here is the traceback:

    **[CPU: 191.6 MB]  Traceback (most recent call last):**
**  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 115, in run_topaz_wrapper_train**
**    topaz_version = utils.get_topaz_version(topaz_exec_path)**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 101, in get_topaz_version**
**    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'**
**AssertionError: Cannot determine topaz version, command "/app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/bin/topaz --version" did not produce valid output: "ImportError: /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/lib/python3.6/site-packages/torch/lib/libmkldnn.so.0: undefined symbol: cblas_sgemm_alloc"**

nfrasser · April 29, 2021, 2:24pm

Hi @zybo, this error indicates that something went wrong with the Topaz installation. It is not related to cryoSPARC. I suggest you remove the Anaconda environment at /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz and attempt a fresh installation of Topaz from the instructions here: https://github.com/tbepler/topaz#installation