Having trouble installing topaz

Particle Picking
1

The documentation here states the following:

As cryoSPARC only wraps installed Topaz instances rather than providing Topaz instances directly, Topaz must be first installed in the Python environment within cryoSPARC.

It is not recommended to use the Anaconda Python environment that is installed with cryoSPARC. This environment is subject to be destroyed upon an update to cryoSPARC. It is best to use your own Anaconda Python (2.7) environment, or a new one if one doesn’t already exist. Please ensure the Topaz executable is accessible by the cryoSPARC user account, and accessible on each worker node.

So the back to back combination of “Topaz must be first installed in the Python environment within cryoSPARC.” and “It is not recommended to use the Anaconda Python environment that is installed with cryoSPARC.” is somewhat confusing.

I interpreted this to mean to use the anaconda installation found here:
~cryosparc_user/cryosparc2_worker/deps/anaconda

and create a separate topaz anaconda environment within this base on the worker nodes, since I’m guessing that’s where the code will run. (We have 2 workstations serving as cryosparc workers and a server running the master software.)

So In this anaconda, I proceeded to activate base, and create a new topaz environment:

(One oddity which I don’t understand: I had to try and run ~deps/anaconda/bin/conda before conda.sh could be sourced effectively. Not sure what that’s all about, as I’m reasonably familiar with anaconda.)

$ . /local/home/cryosparc_user/cryosparc2_worker/deps/anaconda/etc/profile.d/conda.sh
$ source activate
$ conda create --name topaz python=2.7
$ conda activate topaz
$ conda install topaz -c tbepler -c pytorch

The tutorial then suggests finding the location of the topaz executable, which I did:
~cryosparc_user/cryosparc2_worker/deps/anaconda/envs/topaz/bin/topaz

which I then sent to the cryosparc administrator. He tried inserting this executable path in the configuration, but cryosparc complained that it didn’t exist, suggesting that topaz also needed to be installed on cryosparc2_master as well, which I did. Hence, problem: the absolute path to the topaz executable on the master is necessarily different than it is on the worker nodes, so when he attempts to use topaz, the executable can’t be found:

File "cryosparc2_compute/jobs/topaz/topaz_utils.py", line 75, in get_topaz_version
    raise OSError('Input Topaz executable path does not exist.')
OSError: Input Topaz executable path does not exist.

One troubling thing in all of this is that it appears you’re running the topaz executable without activating the topaz environment. As far as I know, that’s not how anaconda is intended to be used, and if this works, it’s only working accidentally, since topaz isn’t being run in the environment that was created for it but rather in the cryosparc environment.

Anyway, if anyone has any insights into what I need to do to get this to work, please share.

Thanks!

2

@sdawood @nfrasser @stephan Any suggestions or additional documentation? We would love to start using Topaz within cryoSPARC.

Thanks
Jason

1 Like
3

Hello @pgoetz and @Jason,

I apologize for the confusion, we’ll clarify the statements you mentioned in the documentation. Regarding the installation itself, the intended method is to install Topaz in an Anaconda environment outside of cryoSPARC and then using the executable path from that environment. Please try installing Topaz in a separate Anaconda environment and then using that path. It should also be noted that Topaz does not need to be available on the master node, it only needs to be available on the worker nodes. Please let me know if the problem persists.

Regards,
Jay Yoo

4

Following up on this after looking at the updated documentation on using topaz within cryosparc.

It should also be noted that Topaz does not need to be available on the master node, it only needs to be available on the worker nodes. Please let me know if the problem persists.

Unfortunately this does not appear to be correct? If I just install topaz on just the worker nodes, and attempt to add the path to topaz using the cyrosparc administrative interface, this fails, giving a something along the lines of a “file not found” error message. The only workaround we’ve been able to identify is to also install topaz on the cryosparc master. Not only that, the path must be the same as it is on the workers, or the workers can’t find topaz.

Next, the instructions provided here don’t seem to correspond with any reality that I’m aware of. In particular, this:

Finding Default Topaz Executable Path

The following is one possible method of finding this path in Linux. This method assumes that Topaz was installed using Anaconda or Pip.

  1. If you have not yet already, follow the instructions in the Topaz GitHub Repository to install Topaz using Anaconda.
  2. Open a Terminal and enter:

which topaz

This is not how miniconda works. In order to find an executable in a miniconda environment, you must first activate that environment:

$ conda activate topaz
$ which topaz

In fact, doing anything other than working within a activated environment is not supported:

If you choose not to activate your environment, then loading and setting environment variables to activate scripts will not happen. We only support activation.
https://docs.conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html

Of course one can’t do a miniconda activation of topaz from within cryosparc (or can one?). So here are the details of what I did to install topaz on the cryosparc worker nodes and the master. First I installed a generic miniconda3 environment in /usr/local. (Using miniconda2 vs. miniconda3 shouldn’t matter.)

Then:

 # conda create --name topaz python=2.7
 # conda activate topaz
 # conda install topaz -c tbepler -c pytorch

The topaz executable is then found here:

root@javelina:~# conda activate topaz
(topaz) root@javelina:~# which topaz
/usr/local/miniconda3/envs/topaz/bin/topaz

Entering that path into the cryosparc configuration results in an error message when one attempts to use topaz within cryosparc:

[CPU: 249.7 MB]  Starting Topaz process using version ImportError: No module named nn...
[CPU: 249.7 MB]  Traceback (most recent call last):
  File "cryosparc2_worker/cryosparc2_compute/run.py", line 85, in cryosparc2_compute.run.main
  File "cryosparc2_compute/jobs/topaz/run_topaz.py", line 120, in run_topaz_wrapper_train
    ver_023_plus = utils.check_topaz_version(topaz_version)
  File "cryosparc2_compute/jobs/topaz/topaz_utils.py", line 66, in check_topaz_version
    if (int(version_vals[0]) > 0) or (int(version_vals[1]) > 2) or (int(version_vals[1]) == 2 and int(version_vals[2]) >= 3):
ValueError: invalid literal for int() with base 10: ''

As expected, topaz can’t find its python modules because the topaz miniconda enviornment hasn’t been activated.

As far as I can tell, the only way to solve this problem is to turn topaz into an appimage so that the entire operating environment can be presented to cyrosparc as a single executable. I’m attempting to build this appImage now, but have a question: does anyone know how I can test topaz outside of cryosparc? I need to have some way of testing that it’s autonomously working before running it from cryosparc.

Thanks.

5

Hi @pgoetz,

Regarding the first concern you mentioned with the master node, this does appear to be a mistake on our end. As you said, the Topaz executable path must be available to master and the path must match on the worker nodes.

A basic litmus test to verify whether Topaz is working would be to run a basic Topaz command using the executable directly in Terminal with the environment activated. One such basic Topaz command is

topaz --version

If this is unsufficient in verifying Topaz’s functionality, Topaz can be tested on a Jupyter notebook by installing Jupyter and matplotlib onto the same environment that Topaz is installed in then running one of Topaz’s provided tutorials. More details can be found in the following link under the Tutorial section: https://github.com/tbepler/topaz

Upon testing further, we found that your issue with Topaz being unable to import modules is reproducible when attempting to use the latest release of Topaz. We also found that it is resolved by using Topaz version 0.2.3 instead. Could you try following the same Anaconda Topaz installation instructions except instead of performing

conda install topaz -c tbepler -c pytorch

try using

conda install topaz=0.2.3 -c tbepler -c pytorch

instead? Once Topaz version 0.2.3 is installed, could you try running a Topaz job with the path to that executable instead? The job and setup that created the traceback you sent earlier should suffice.

We’ll further update the documentation related to the master node accordingly. Please let us know if installing Topaz 0.2.3 helps alleviate your issue.

Regards,
Jay Yoo

6

Hi Jay -

Thanks for this helpful response. Yes, there is clearly a problem with topaz 0.2.4! I just tried executing the simple test you suggest above, and:

pgoetz@hobo:~$ conda activate topaz
(topaz) pgoetz@hobo:~$ topaz --version
Traceback (most recent call last):
  File "/opt/miniconda/envs/topaz/bin/topaz", line 11, in <module>
    load_entry_point('topaz-em==0.2.4', 'console_scripts', 'topaz')()
  File "/opt/miniconda/envs/topaz/lib/python2.7/site-packages/topaz/main.py", line 68, in main
    import topaz.commands.denoise
  File "/opt/miniconda/envs/topaz/lib/python2.7/site-packages/topaz/commands/denoise.py", line 77, in <module>
    import topaz.denoise as dn
  File "/opt/miniconda/envs/topaz/lib/python2.7/site-packages/topaz/denoise.py", line 7, in <module>
    import torch.nn as nn
ImportError: No module named nn

So even when the topaz environment is activated it still can’t find the nn module! I’ll try downgrading to topaz 0.2.3 and will report back on whether or not this fixes our problem. However meanwhile I’m going to continue to work on an AppImage solution, as we keep running into issues with trying to interface distinct miniconda environments.

7

Hey @pgoetz,

You’re right about the environment activation piece- the only reason this works at all is probably because of chance! I think an Appimage might be a nice solution- is building one easy to automate?

8

Regarding AppImage, I’m not sufficiently expert yet to offer a definitive opinion, but it should be easy to automate creating these things. Other people are doing it, and in fact it can even be used with continuous integration systems like Travis CI and Jenkins (which I know next to nothing about, just know that this is a thing).

9

Hi @pgoetz, are you still having trouble installing Topaz? If so, please post an issue here:

Thanks!
-Alex

1 Like
10

We downgraded to Topaz 0.2.3 and that seems to be working fine.

1 Like