Topaz training failed after update

Particle Picking
1

Hi everyone,
somehow the topaz train does not work after i update but it worked well before. it gives error messages below. I also checked other posters with similar issues. I tried split the micrographs but it still does not work. Does anyone have an idea what is potential problem?

[CPU: 222.2 MB Avail: 367.72 GB]
WARNING: no micrograph found matching image name “014178270276145150691_CP_xxx_tilt30_1648_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.71 GB]
WARNING: no micrograph found matching image name “014774187030274709097_xx_xxx_tilt30_0419_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.71 GB]
WARNING: no micrograph found matching image name “008732402471452487208_xx_xxx_tilt30_1328_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.70 GB]
WARNING: no micrograph found matching image name “010283681879661460872_xx_xxx_tilt30_0892_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.69 GB]
WARNING: no micrograph found matching image name “004999302769548460318_xx_xx_tilt30_3424_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.69 GB]
WARNING: no micrograph found matching image name “014297745618647049179_xxxx_tilt30_0917_patch_aligned_doseweighted”. Skipping it.

[CPU: 222.2 MB Avail: 367.68 GB]
Traceback (most recent call last):

[CPU: 222.2 MB Avail: 367.67 GB]
File “/home/xx/miniconda3/envs/topaz/bin/topaz”, line 11, in

[CPU: 222.2 MB Avail: 367.66 GB]
load_entry_point(‘topaz-em==0.2.4’, ‘console_scripts’, ‘topaz’)()

[CPU: 222.2 MB Avail: 367.66 GB]
File “/home/xx/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/main.py”, line 148, in main

[CPU: 222.2 MB Avail: 367.65 GB]
args.func(args)

[CPU: 222.2 MB Avail: 367.64 GB]
File “/home/CP/miniconda3/envs/topaz/lib/python3.6/site-packages/topaz/commands/train_test_split.py”, line 128, in main

[CPU: 222.2 MB Avail: 367.63 GB]
image_list_train = pd.DataFrame({‘image_name’: image_names_train, ‘path’: paths_train})

[CPU: 222.2 MB Avail: 367.63 GB]
File “/home/xx/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/core/frame.py”, line 468, in init

[CPU: 222.2 MB Avail: 367.62 GB]
mgr = init_dict(data, index, columns, dtype=dtype)

[CPU: 222.2 MB Avail: 367.61 GB]
File “/home/xx/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py”, line 283, in init_dict

[CPU: 222.2 MB Avail: 367.61 GB]
return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)

[CPU: 222.2 MB Avail: 367.60 GB]
File “/home/xx/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py”, line 78, in arrays_to_mgr

[CPU: 222.2 MB Avail: 367.59 GB]
index = extract_index(arrays)

[CPU: 222.2 MB Avail: 367.58 GB]
File “/home/xx/miniconda3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py”, line 397, in extract_index

[CPU: 222.2 MB Avail: 367.58 GB]
raise ValueError(“arrays must all be same length”)

[CPU: 222.2 MB Avail: 367.57 GB]
ValueError: arrays must all be same length

[CPU: 222.2 MB Avail: 367.55 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main
File “/home/xx/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 332, in run_topaz_wrapper_train
utils.run_process(split_command)
File “/home/xx/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 99, in run_process
assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/home/xx/miniconda3/envs/topaz/bin/topaz train_test_split --number 251 --seed 229024607 --image-dir /data/zcp/xx-msp1d1-3muts/CS-sxxx-go-titlet-30/J75/preprocessed /data/zcp/xxx-msp1d1-3muts/CS-xxx-go-titlet-30/J75/topaz_parti…)

2

Did you update CryoSPARC, Topaz or both? What were the version before the update, and what are the currently installed versions?

3

Hi，
I am sorry to reply so late. current version of cryosparc is v4.6.2. I restalled the topaz and installation is successful. I tried topaz training again but it give diffferent error messages below. I have no idea what is going on with that

Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main
File “/home/CP/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 118, in run_topaz_wrapper_train
topaz_version = utils.get_topaz_version(topaz_exec_path)
File “/home/CP/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 136, in get_topaz_version
assert semver.VersionInfo.isvalid(topaz_version_for_validation),
AssertionError: Cannot determine topaz version, command “/home/CP/miniconda3/envs/topaz/bin/topaz --version” did not produce valid output: “pkg_resources.DistributionNotFound: The ‘h5py’ distribution was not found and is required by topaz-em”

4

I even tried to create topaz.sh based on other posts with similiar issues. However, issue is still there. I really need your help for this issue.
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 129, in cryosparc_master.cryosparc_compute.run.main
File “/home/CP/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 118, in run_topaz_wrapper_train
topaz_version = utils.get_topaz_version(topaz_exec_path)
File “/home/CP/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py”, line 136, in get_topaz_version
assert semver.VersionInfo.isvalid(topaz_version_for_validation),
AssertionError: Cannot determine topaz version, command “/home/topaz.sh --version” did not produce valid output: “pkg_resources.DistributionNotFound: The ‘topaz-em==0.3.5’ distribution was not found and is required by the application”

5

You may have installed a newer version of Topaz that we have not tested within CryoSPARC.

I tested within CryoSPARC a Topaz v0.2.5 installation. For this purpose, I modified the instructions at the Topaz repository to force installation of specific versions for Topaz and the mkl dependency.

conda create -n topaz-0.2.5-20241217 python=3.6 
conda activate topaz-0.2.5-20241217
conda install topaz=0.2.5 pytorch=1.10.2=py3.6_cuda11.3_cudnn8.2.0_0 mkl=2024.0.0 -c tbepler -c pytorch

I ensured that the conda environment was stored on a share that was accessible to the CryoSPARC worker and defined Path to Topaz executable as

/path/to/envs/topaz-0.2.5-20241217/bin/topaz
6

Hi wtempel,
Thank you very much for this troubleshooting. I run these commands and it gives error message like this below. it seems that - mkl=2024.0.0* is not available.
(topaz-0.2.5-20241217) [CP@c113963 ~]$ conda install topaz=0.2.5 pytorch=1.10.2=py3.6_cuda11.3_cudnn8.2.0_0 mkl=2024.0.0 -c tbepler -c pytorch
/home/CP/miniconda3/lib/python3.11/site-packages/conda/base/context.py:201: FutureWarning: Adding ‘defaults’ to channel list implicitly is deprecated and will be removed in 25.3.

To remove this warning, please choose a default channel explicitly with conda’s regular configuration system, e.g. by adding ‘defaults’ to the list of channels:

conda config --add channels defaults

For more information see Using the .condarc conda configuration file — conda 24.11.1 documentation

deprecated.topic(
/home/CP/miniconda3/lib/python3.11/site-packages/conda/base/context.py:201: FutureWarning: Adding ‘defaults’ to channel list implicitly is deprecated and will be removed in 25.3.

To remove this warning, please choose a default channel explicitly with conda’s regular configuration system, e.g. by adding ‘defaults’ to the list of channels:

conda config --add channels defaults

For more information see Using the .condarc conda configuration file — conda 24.11.1 documentation

deprecated.topic(
Channels:

  • tbepler
  • pytorch
  • defaults
    Platform: linux-64
    Collecting package metadata (repodata.json): done
    Solving environment: failed

PackagesNotFoundError: The following packages are not available from current channels:

  • mkl=2024.0.0*

Current channels:

To search for alternate channels that may provide the conda package you’re
looking for, navigate to

https://anaconda.org

and use the search bar at the top of the page.

7

I have not been able to test the following suggestion, but have you tried inside the topaz-0.2.5-20241217 conda environment appending
-c conda-forge to the installation command:

conda install topaz=0.2.5 pytorch=1.10.2=py3.6_cuda11.3_cudnn8.2.0_0 mkl=2024.0.0 -c tbepler -c pytorch -c conda-forge

?

8

Hi wtempel,
Really appreciate it for your support. it works well now after i append -c conda-forge to the installation command as you suggested. Thank you very much. :+1: :+1: :+1: :+1: :+1: :+1: