cryoSPARC v.3.1.0 and Topaz

jelka · February 22, 2021, 11:01am

This worked for first two problems I had.

But the my last problem which seems similar to @Adrian’s is still there:

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)

topaz --version says “TOPAZ 0.2.3”

Thanks

//Jesper

eMKiso · February 25, 2021, 8:17pm

Hi,
is it OK if we use the v0.2.4 (or v0.2.5) Topaz version with CS v3.1.0?

Thanks!

nfrasser · February 26, 2021, 5:12pm

@jelka @Adrian I’m not sure what causes the warning but it should not affect the output.

@eMKiso yes all newer versions of topaz should be supported by v3.1

zybo · April 28, 2021, 2:16pm

Hello everyone, I’m having a similar trouble, with latest version of CryoSparc and Topaz.
here is the traceback:

    **[CPU: 191.6 MB]  Traceback (most recent call last):**
**  File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 115, in run_topaz_wrapper_train**
**    topaz_version = utils.get_topaz_version(topaz_exec_path)**
**  File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 101, in get_topaz_version**
**    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'**
**AssertionError: Cannot determine topaz version, command "/app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/bin/topaz --version" did not produce valid output: "ImportError: /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/lib/python3.6/site-packages/torch/lib/libmkldnn.so.0: undefined symbol: cblas_sgemm_alloc"**

nfrasser · April 29, 2021, 2:24pm

Hi @zybo, this error indicates that something went wrong with the Topaz installation. It is not related to cryoSPARC. I suggest you remove the Anaconda environment at /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz and attempt a fresh installation of Topaz from the instructions here: https://github.com/tbepler/topaz#installation

KiSchnelle · September 2, 2021, 8:45am

Just wanted to reply since i just had the same problem as zybo, and i think its cause of the pytorch/cuda version. I could solve it by using the nvidia channel instead of confa-forge.

conda install topaz cudatoolkit=11.1 -c tbepler -c pytorch -c nvidia

One thing i though see is that extract aswell as train is using all available CPUs, so even though cryosparc just tell 0-8 on htop i see all 48 threads on 100% util. Also for train 64GB of Ram is apparently not enough and as soon its full the job will be stuck at preprocessing micrographs with CPU util getting 0%

Omid · October 21, 2021, 6:12pm

Dear @KiSchnelle and @zybo ,

I had similar issues as well. For me the I can install the latest version of Topaz using
conda install topaz cudatoolkit=9.0 -c tbepler -c pytorch -c conda-forge.

For pytorch cuda toolkit 11 or higher requires the NVidia option as you have indicated, but that causes the problem of Topaz using all available resources and crashing my system. So the downgrading to toolkit 9,0 seem to be a better compromise for me.

But for larger datasets i.e. more than 5000 movies topaz extract (or training jobs) runs really slow or freezes cryosparc entirely. It doesn’t use all the resources as before, so the computer is fine but cryosparc stops responding. Has anyone else experienced this? When I look at what is running, it seems that topaz is still going but I lose access to the browser and user interface.

Best,
Omid

KiSchnelle · October 21, 2021, 7:23pm

Oh so thats why Topaz takes so much resources:D i thought its just well Topaz:D i just fixed that by adding more RAM or splitting the dataset.
Iam actually not sure if i can downgrade the toolkit with RTX30XX cards. I think they need 11, dont they.

Are you using standalone installation? then maybe its still crashing the pc somehow, with master/worker i never had the master server/gui of cryosparc freezing just the topaz working node crashing.

You could maybe try combining pytorch version 1.7.1 with toolkit 9.2 or another combination

cheers
Kilian

amorin · March 16, 2022, 10:46pm

I have the same problem launching Topaz from within cryosparc, however I have installed topaz from source and am not running it in a virtual environment.

The error I get from cryosparc:

[CPU: 228.2 MB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 917, in run_topaz_wrapper_extract
topaz_version = utils.get_topaz_version(topaz_exec_path)
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 126, in get_topaz_version
f’Cannot determine topaz version, command “{topaz_exec_path} --version” did not produce valid output: “{topaz_version}”’
AssertionError: Cannot determine topaz version, command “/home/ubuntu/.local/bin/topaz --version” did not produce valid output: “ModuleNotFoundError: No module named ‘topaz’”

Yet, when I type “topaz --version” into my terminal, I correctly get:

TOPAZ 0.2.5a

Setting the path explicitly in the job does not help (see error above), nor does setting it using the cryosparc cli on a per project basis, nor does making a bash file deactivating the cryosparc conda environment as suggested above.

Any insights to get this working would be appreciated.

Best,

-Andrew

wtempel · March 18, 2022, 3:17pm

Welcome to the forum @amorin.
Please can you paste:

the content of the bash script you mentioned,
the output of <path-to-bash-script> --version
error messages when you run a cryoSPARC job involving topaz for which you specified <path-to-bash-script> in the Path to Topaz executable field.

amorin · March 18, 2022, 4:19pm

Hi wtempel,
Thanks for responding.

The bash script I’m using is:

#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
conda deactivate > /dev/null 2>&1 || true # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
#/home/ubuntu/.local/bin/topaz
topaz $@

where, if I uncomment each of the last two lines in turn, I get slightly different output of:

[CPU: 226.7 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
    topaz_version = utils.get_topaz_version(topaz_exec_path)
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "ModuleNotFoundError: No module named 'topaz'"

or

[CPU: 227.7 MB]  Traceback (most recent call last):
  File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
    topaz_version = utils.get_topaz_version(topaz_exec_path)
  File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
    f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "/home/ubuntu/topaz/topaz.sh: line 16: topaz: command not found"

Notice the “No module” vs “command not found” output in the last line, respectively. (if that helps at all).

If I run that script from the command line directly - “sh topaz.sh --version”, then I get the proper output of “TOPAZ 0.2.5a”.

Thanks for your continuing help.

-Andrew

wtempel · March 18, 2022, 5:13pm

@amorin Could the problem be that topaz.sh is not executable by the unix user that owns the cryoSPARC instance?
ls -l /home/ubuntu/topaz/topaz.sh
and possibly
chmod u+x /home/ubuntu/topaz/topaz.sh.
What is the output of /home/ubuntu/topaz/topaz.sh --version without using the sh command?

amorin · March 18, 2022, 5:29pm

No. Its set executable

ubuntu@ip:~/topaz$ ./topaz.sh --version
TOPAZ 0.2.5a
ubuntu@ip:~/topaz$ /home/ubuntu/.local/bin/topaz --version
TOPAZ 0.2.5a
ubuntu@ip:~/topaz$ ls -l topaz.sh
-rwxrwxr-x 1 ubuntu ubuntu 372 Mar 18 16:10 topaz.sh

-Andrew

wtempel · March 18, 2022, 6:17pm

@amorin Your script resembles the guide example except the missing call to
conda activate <topaz-conda-env>
Could that be the problem?

amorin · March 18, 2022, 6:29pm

Topaz is installed from source without conda, so there is no topaz conda environment to activate.
I don’t even have conda installed - except for whatever comes with cryosparc.

Might not the difference in output between the two versions of the topaz.sh script be a clue? And also confirmation that the issue is not about it being executable. One seems to be looking for a python module. Just a guess.

-Andrew

wtempel · March 23, 2022, 8:37pm

@amorin For use with cryoSPARC, I would suggest installing topaz in its dedicated conda environment and using a wrapper script (recently updated). Miniconda should be sufficient to initially establish the virtual environment.

Dmitry · March 3, 2023, 12:18pm

dear @nfrasser and @wtempel

In attempt to reproduce the following suggestions and get Topaz installed

(Optional) Create a topaz.sh wrapper script

https://guide.cryosparc.com/processing-data/all-job-types-in-cryosparc/deep-picking/topaz#optional-create-a-topaz.sh-wrapper-script

I run the following script -

But then I the terminal dissapeared and I have no clue where to look for topaz.sh

Could you please advice -
a) what went wrong - bash: : command not found…
b) was the topaz.sh created and if yes - where to find it?

Thank you in advance.

Sincerely,
Dmitry

nfrasser · March 3, 2023, 1:36pm

@Dmitry you must create the topaz.sh file yourself and insert the contents. If you’d like to create the file from the command line, here are the commands that would do that:

cat > topaz.sh << END
#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
    conda deactivate > /dev/null 2>&1 || true  # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate topaz
exec topaz $@
END
chmod +x topaz.sh

Dmitry · May 25, 2023, 8:04pm

Dear colleagues,

We installed topaz and currently running the Topaz training.

In the progress log window I see the following –

UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /tmp/pip-req-build-vflcj7wg/torch/csrc/utils/tensor_numpy.cpp:180.)

Since it is already running quite a long time - for about 7 hours without further update I suppose there is an issue.

Could you please advice how to fix that?

Thank you in advance.

Sincerely,
Dmitry

wtempel · May 25, 2023, 9:22pm

A similar UserWarning was discussed under NumPy array Warning · Issue #68 · tbepler/topaz · GitHub.

You may want to

confirm in the Event Log that Topaz has recognized your GPU device and is not “Falling back to CPU” (as it does in TOPAZ not using GPU)
confirm the GPU worker has not running out of free RAM or CPU resources
possibly reduce the Number of parallel processes and Number of CPUs values from their defaults (as discussed in Topaz Preprocessing very slow - #2 by ccgauvin94)