It looks like you have an error in your topaz.sh file. Can you send me the full contents? Please also run the file with the following command:
/home/adrian/topaz.sh --version
And send me the full output.
Hello, I found in topaz github the following so I suppose there is no problem.
“This is a warning that new versions of pytorch started raising because our mrc micrograph/tomogram arrays are read-only. The warning is annoying, but, to my knowledge, doesn’t cause any actual errors and can safely be ignored.”
This is my full topaz.sh file
#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
conda deactivate > /dev/null 2>&1 || true # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
# source ~/anaconda/etc/profile.d/conda.sh
conda activate topaz_cryosparc
topaz $@
I will try the /home/adrian/topaz.sh --version
1 Like
This worked for first two problems I had.
But the my last problem which seems similar to @Adrian’s is still there:
- UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /opt/conda/conda-bld/pytorch_1607370172916/work/torch/csrc/utils/tensor_numpy.cpp:141.)
topaz --version says “TOPAZ 0.2.3”
Thanks
//Jesper
Hi,
is it OK if we use the v0.2.4 (or v0.2.5) Topaz version with CS v3.1.0?
Thanks!
@jelka @Adrian I’m not sure what causes the warning but it should not affect the output.
@eMKiso yes all newer versions of topaz should be supported by v3.1
Hello everyone, I’m having a similar trouble, with latest version of CryoSparc and Topaz.
here is the traceback:
**[CPU: 191.6 MB] Traceback (most recent call last):**
** File "cryosparc_worker/cryosparc_compute/run.py", line 84, in cryosparc_compute.run.main**
** File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 115, in run_topaz_wrapper_train**
** topaz_version = utils.get_topaz_version(topaz_exec_path)**
** File "/app/apps/rhel7/cryosparc/cryosparc2_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 101, in get_topaz_version**
** f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'**
**AssertionError: Cannot determine topaz version, command "/app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/bin/topaz --version" did not produce valid output: "ImportError: /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz/lib/python3.6/site-packages/torch/lib/libmkldnn.so.0: undefined symbol: cblas_sgemm_alloc"**
Hi @zybo, this error indicates that something went wrong with the Topaz installation. It is not related to cryoSPARC. I suggest you remove the Anaconda environment at /app/apps/rhel7/anaconda-python3/2020-11/envs/topaz and attempt a fresh installation of Topaz from the instructions here: https://github.com/tbepler/topaz#installation
1 Like
Just wanted to reply since i just had the same problem as zybo, and i think its cause of the pytorch/cuda version. I could solve it by using the nvidia channel instead of confa-forge.
conda install topaz cudatoolkit=11.1 -c tbepler -c pytorch -c nvidia
One thing i though see is that extract aswell as train is using all available CPUs, so even though cryosparc just tell 0-8 on htop i see all 48 threads on 100% util. Also for train 64GB of Ram is apparently not enough and as soon its full the job will be stuck at preprocessing micrographs with CPU util getting 0%
Dear @KiSchnelle and @zybo ,
I had similar issues as well. For me the I can install the latest version of Topaz using
conda install topaz cudatoolkit=9.0 -c tbepler -c pytorch -c conda-forge.
For pytorch cuda toolkit 11 or higher requires the NVidia option as you have indicated, but that causes the problem of Topaz using all available resources and crashing my system. So the downgrading to toolkit 9,0 seem to be a better compromise for me.
But for larger datasets i.e. more than 5000 movies topaz extract (or training jobs) runs really slow or freezes cryosparc entirely. It doesn’t use all the resources as before, so the computer is fine but cryosparc stops responding. Has anyone else experienced this? When I look at what is running, it seems that topaz is still going but I lose access to the browser and user interface.
Best,
Omid
Oh so thats why Topaz takes so much resources:D i thought its just well Topaz:D i just fixed that by adding more RAM or splitting the dataset.
Iam actually not sure if i can downgrade the toolkit with RTX30XX cards. I think they need 11, dont they.
Are you using standalone installation? then maybe its still crashing the pc somehow, with master/worker i never had the master server/gui of cryosparc freezing just the topaz working node crashing.
You could maybe try combining pytorch version 1.7.1 with toolkit 9.2 or another combination
cheers
Kilian
I have the same problem launching Topaz from within cryosparc, however I have installed topaz from source and am not running it in a virtual environment.
The error I get from cryosparc:
[CPU: 228.2 MB] Traceback (most recent call last):
File “cryosparc_worker/cryosparc_compute/run.py”, line 85, in cryosparc_compute.run.main
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 917, in run_topaz_wrapper_extract
topaz_version = utils.get_topaz_version(topaz_exec_path)
File “/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 126, in get_topaz_version
f’Cannot determine topaz version, command “{topaz_exec_path} --version” did not produce valid output: “{topaz_version}”’
AssertionError: Cannot determine topaz version, command “/home/ubuntu/.local/bin/topaz --version” did not produce valid output: “ModuleNotFoundError: No module named ‘topaz’”
Yet, when I type “topaz --version” into my terminal, I correctly get:
TOPAZ 0.2.5a
Setting the path explicitly in the job does not help (see error above), nor does setting it using the cryosparc cli on a per project basis, nor does making a bash file deactivating the cryosparc conda environment as suggested above.
Any insights to get this working would be appreciated.
Best,
-Andrew
Welcome to the forum @amorin.
Please can you paste:
- the content of the bash script you mentioned,
- the output of
<path-to-bash-script> --version - error messages when you run a cryoSPARC job involving topaz for which you specified
<path-to-bash-script>in the Path to Topaz executable field.
Hi wtempel,
Thanks for responding.
The bash script I’m using is:
#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
conda deactivate > /dev/null 2>&1 || true # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
#/home/ubuntu/.local/bin/topaz
topaz $@
where, if I uncomment each of the last two lines in turn, I get slightly different output of:
[CPU: 226.7 MB] Traceback (most recent call last):
File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
topaz_version = utils.get_topaz_version(topaz_exec_path)
File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "ModuleNotFoundError: No module named 'topaz'"
or
[CPU: 227.7 MB] Traceback (most recent call last):
File "cryosparc_worker/cryosparc_compute/run.py", line 85, in cryosparc_compute.run.main
File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 917, in run_topaz_wrapper_extract
topaz_version = utils.get_topaz_version(topaz_exec_path)
File "/home/ubuntu/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 126, in get_topaz_version
f'Cannot determine topaz version, command "{topaz_exec_path} --version" did not produce valid output: "{topaz_version}"'
AssertionError: Cannot determine topaz version, command "/home/ubuntu/topaz/topaz.sh --version" did not produce valid output: "/home/ubuntu/topaz/topaz.sh: line 16: topaz: command not found"
Notice the “No module” vs “command not found” output in the last line, respectively. (if that helps at all).
If I run that script from the command line directly - “sh topaz.sh --version”, then I get the proper output of “TOPAZ 0.2.5a”.
Thanks for your continuing help.
-Andrew
@amorin Could the problem be that topaz.sh is not executable by the unix user that owns the cryoSPARC instance?
ls -l /home/ubuntu/topaz/topaz.sh
and possibly
chmod u+x /home/ubuntu/topaz/topaz.sh.
What is the output of /home/ubuntu/topaz/topaz.sh --version without using the sh command?
No. Its set executable
ubuntu@ip:~/topaz$ ./topaz.sh --version
TOPAZ 0.2.5a
ubuntu@ip:~/topaz$ /home/ubuntu/.local/bin/topaz --version
TOPAZ 0.2.5a
ubuntu@ip:~/topaz$ ls -l topaz.sh
-rwxrwxr-x 1 ubuntu ubuntu 372 Mar 18 16:10 topaz.sh
-Andrew
@amorin Your script resembles the guide example except the missing call to
conda activate <topaz-conda-env>
Could that be the problem?
Topaz is installed from source without conda, so there is no topaz conda environment to activate.
I don’t even have conda installed - except for whatever comes with cryosparc.
Might not the difference in output between the two versions of the topaz.sh script be a clue? And also confirmation that the issue is not about it being executable. One seems to be looking for a python module. Just a guess.
-Andrew
@amorin For use with cryoSPARC, I would suggest installing topaz in its dedicated conda environment and using a wrapper script (recently updated). Miniconda should be sufficient to initially establish the virtual environment.
In attempt to reproduce the following suggestions and get Topaz installed
(Optional) Create a topaz.sh wrapper script
I run the following script -
But then I the terminal dissapeared and I have no clue where to look for topaz.sh
Could you please advice -
a) what went wrong - bash: : command not found…
b) was the topaz.sh created and if yes - where to find it?
Thank you in advance.
Sincerely,
Dmitry
@Dmitry you must create the topaz.sh file yourself and insert the contents. If you’d like to create the file from the command line, here are the commands that would do that:
cat > topaz.sh << END
#!/usr/bin/env bash
if command -v conda > /dev/null 2>&1; then
conda deactivate > /dev/null 2>&1 || true # ignore any errors
conda deactivate > /dev/null 2>&1 || true # ignore any errors
fi
unset _CE_CONDA
unset CONDA_DEFAULT_ENV
unset CONDA_EXE
unset CONDA_PREFIX
unset CONDA_PROMPT_MODIFIER
unset CONDA_PYTHON_EXE
unset CONDA_SHLVL
unset PYTHONPATH
unset LD_PRELOAD
unset LD_LIBRARY_PATH
source $HOME/miniconda3/etc/profile.d/conda.sh
conda activate topaz
exec topaz $@
END
chmod +x topaz.sh
1 Like