cryoSPARC v.3.1.0 and Topaz

menmt · February 4, 2021, 10:52pm

After upgrading from version 2.15.0 to version 3.1.0 I get the following error message when I use ‘Topaz Train’. I have run ‘Topaz Train’ successfully using the version 2.15.0 with the same path to the Topaz executable.

AssertionError: Cannot determine topaz version, command “/home/cryosparc_user/anaconda3/envs/topaz/bin/topaz --version” did not produce valid output: “Current thread 0x00007fc9ba20e740 (most recent call first):”

When I run the command “/home/cryosparc_user/anaconda3/envs/topaz/bin/topaz --version” in the terminal I get: “TOPAZ 0.2.3”

I also tested Topaz 0.2.4, but got the same error.

Any suggestions?

Thank you.

jelka · February 7, 2021, 10:23pm

@menmt I had same error, but figured out it was due to cryoSPARCs move from python2 to python3.
You also need to reinstall Topaz under a python3 env:

conda create -n topaz python=3
conda activate topaz
conda install topaz=0.2.3 cudatoolkit=11.0 -c tbepler -c pytorch

Now Topaz should run.
In my case, although with some error messages, but it still completes.

//Jesper

jelka · February 7, 2021, 10:28pm

The errors I see are:

ERROR: ld.so: object 'cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so' from LD_PRELOAD cannot be preloaded: ignored.
ERROR: ld.so: object 'cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so' from LD_PRELOAD cannot be preloaded: ignored.
UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)

The first two errors, seems to be related to jobs being submitted to a standalone worker and it looks for cryosparc_master path on this worker.

The last error beats me. Could be conda repo incompatibility between PyTorch and NumPy?

menmt · February 8, 2021, 3:44pm

I re-installed Topaz under a python3 env, but still had the same error…

nfrasser · February 8, 2021, 6:41pm

@jelka @menmt can you try making a bash file that activates your topaz conda environment before invoking topaz? It’ll look something like this:

#!/usr/bin/env bash
# source ~/anaconda/etc/profile.d/conda.sh
conda activate topaz
topaz $@

Optionally remove the # from the beginning of the second line if your conda environment is not automatically activated when the shell starts (then specify the correct path to conda.sh if anaconda is not installed at ~/anaconda)

Save this in your home directory as topaz.sh, then give it execute permissions via command line:

chmod +x ~/topaz.sh

Then in cryoSPARC specify ~/topaz.sh for the “Path to Topaz executable” field and run your job.

If this still leads to the same type of errors, add the following to the top of the topaz.sh, under the #!/usr/bin/env bash line:

if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH

This will deactivate cryoSPARC’s Anaconda environment before activating the Topaz environment you installed.

Let me know how that goes.

menmt · February 8, 2021, 10:19pm

Now it works although I have to deactivate cryoSZPARC’s Anaconda env before activating Topaz env.
Thak you!

Adrian · February 17, 2021, 7:00pm

Hello . I am having this error, similar to one posted here.

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370116979/work/torch/csrc/utils/tensor_numpy.cpp:141.)

When trying the bash file I got the following error:

AssertionError: Cannot determine topaz version, command “/home/adrian/topaz.sh --version” did not produce valid output: “/home/adrian/topaz.sh: línea 4: `fi’”

Do you have any insight about this?

Thank you

nfrasser · February 18, 2021, 9:47pm

It looks like you have an error in your topaz.sh file. Can you send me the full contents? Please also run the file with the following command:

/home/adrian/topaz.sh --version

And send me the full output.

Adrian · February 19, 2021, 3:20pm

Hello, I found in topaz github the following so I suppose there is no problem.

“This is a warning that new versions of pytorch started raising because our mrc micrograph/tomogram arrays are read-only. The warning is annoying, but, to my knowledge, doesn’t cause any actual errors and can safely be ignored.”

This is my full topaz.sh file

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
   conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
# source ~/anaconda/etc/profile.d/conda.sh
conda activate topaz_cryosparc
topaz $@

I will try the /home/adrian/topaz.sh --version

jelka · February 22, 2021, 11:01am

This worked for first two problems I had.

But the my last problem which seems similar to @Adrian’s is still there:

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)

topaz --version says “TOPAZ 0.2.3”

Thanks

//Jesper

eMKiso · February 25, 2021, 8:17pm

Hi,
is it OK if we use the v0.2.4 (or v0.2.5) Topaz version with CS v3.1.0?

Thanks!

nfrasser · February 26, 2021, 5:12pm

@jelka @Adrian I’m not sure what causes the warning but it should not affect the output.

@eMKiso yes all newer versions of topaz should be supported by v3.1

zybo · April 28, 2021, 2:16pm

Hello everyone, I’m having a similar trouble, with latest version of CryoSparc and Topaz.
here is the traceback:

    **[CPU: 191.6 MB]  Traceback (most recent call last):**
**  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 115, in run_topaz_wrapper_train**
**    topaz_version = utils.get_topaz_version(topaz_exec_path)**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 101, in get_topaz_version**
**    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'**
**AssertionError: Cannot determine topaz version, command "/app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/bin/topaz --version" did not produce valid output: "ImportError: /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/lib/python3.6/site-packages/torch/lib/libmkldnn.so.0: undefined symbol: cblas_sgemm_alloc"**

nfrasser · April 29, 2021, 2:24pm

Hi @zybo, this error indicates that something went wrong with the Topaz installation. It is not related to cryoSPARC. I suggest you remove the Anaconda environment at /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz and attempt a fresh installation of Topaz from the instructions here: https://github.com/tbepler/topaz#installation

KiSchnelle · September 2, 2021, 8:45am

Just wanted to reply since i just had the same problem as zybo, and i think its cause of the pytorch/cuda version. I could solve it by using the nvidia channel instead of confa-forge.

conda install topaz cudatoolkit=11.1 -c tbepler -c pytorch -c nvidia

One thing i though see is that extract aswell as train is using all available CPUs, so even though cryosparc just tell 0-8 on htop i see all 48 threads on 100% util. Also for train 64GB of Ram is apparently not enough and as soon its full the job will be stuck at preprocessing micrographs with CPU util getting 0%

Omid · October 21, 2021, 6:12pm

Dear @KiSchnelle and @zybo ,

I had similar issues as well. For me the I can install the latest version of Topaz using
conda install topaz cudatoolkit=9.0 -c tbepler -c pytorch -c conda-forge.

For pytorch cuda toolkit 11 or higher requires the NVidia option as you have indicated, but that causes the problem of Topaz using all available resources and crashing my system. So the downgrading to toolkit 9,0 seem to be a better compromise for me.

But for larger datasets i.e. more than 5000 movies topaz extract (or training jobs) runs really slow or freezes cryosparc entirely. It doesn’t use all the resources as before, so the computer is fine but cryosparc stops responding. Has anyone else experienced this? When I look at what is running, it seems that topaz is still going but I lose access to the browser and user interface.

Best,
Omid

KiSchnelle · October 21, 2021, 7:23pm

Oh so thats why Topaz takes so much resources:D i thought its just well Topaz:D i just fixed that by adding more RAM or splitting the dataset.
Iam actually not sure if i can downgrade the toolkit with RTX30XX cards. I think they need 11, dont they.

Are you using standalone installation? then maybe its still crashing the pc somehow, with master/worker i never had the master server/gui of cryosparc freezing just the topaz working node crashing.

You could maybe try combining pytorch version 1.7.1 with toolkit 9.2 or another combination

cheers
Kilian

amorin · March 16, 2022, 10:46pm

I have the same problem launching Topaz from within cryosparc, however I have installed topaz from source and am not running it in a virtual environment.

The error I get from cryosparc:

[CPU: 228.2 MB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 917, in run_topaz_wrapper_extract
topaz_version = utils.get_topaz_version(topaz_exec_path)
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 126, in get_topaz_version
f’Cannot determine topaz version, command “{topaz_exec_path} --version” did not produce valid output: “{topaz_version}”’
AssertionError: Cannot determine topaz version, command “/home/ubuntu/.local/bin/topaz --version” did not produce valid output: “ModuleNotFoundError: No module named ‘topaz’”

Yet, when I type “topaz --version” into my terminal, I correctly get:

TOPAZ 0.2.5a

Setting the path explicitly in the job does not help (see error above), nor does setting it using the cryosparc cli on a per project basis, nor does making a bash file deactivating the cryosparc conda environment as suggested above.

Any insights to get this working would be appreciated.

Best,

-Andrew

wtempel · March 18, 2022, 3:17pm

Welcome to the forum @amorin.
Please can you paste:

the content of the bash script you mentioned,
the output of <path-to-bash-script> --version
error messages when you run a cryoSPARC job involving topaz for which you specified <path-to-bash-script> in the Path to Topaz executable field.

amorin · March 18, 2022, 4:19pm

Hi wtempel,
Thanks for responding.

The bash script I’m using is:

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
conda deactivate > /dev/null 2>&1 || true # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
#/home/ubuntu/.local/bin/topaz
topaz $@

where, if I uncomment each of the last two lines in turn, I get slightly different output of:

[CPU: 226.7 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
    topaz_version = utils.get_topaz_version(topaz_exec_path)
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "ModuleNotFoundError: No module named 'topaz'"

or

[CPU: 227.7 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
    topaz_version = utils.get_topaz_version(topaz_exec_path)
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "/home/ubuntu/topaz/topaz.sh: line 16: topaz: command not found"

Notice the “No module” vs “command not found” output in the last line, respectively. (if that helps at all).

If I run that script from the command line directly - “sh topaz.sh --version”, then I get the proper output of “TOPAZ 0.2.5a”.

Thanks for your continuing help.

-Andrew