Topaz train job fails: Arrays must all be same length

Hello,

I tried several different workflows for Topaz Train, but they all failed. Did anyone ever stumbled upon this? Or does anyone know how to work around this?

Thank you,

Trial 1: Blob Picker → Inspect Picks → Extract Micrographs → 2D Classification → Select 2D Classes → Topaz Train

  • Inputs to Topaz Train: Mics from Inspect Picks and ptcls from 2D.

  • Result: Failed.

  • Topaz executable: /home/cryosparcuser/.conda/envs/topaz/bin/topaz

Trial 2: Inputs to Topaz Train: Mics from Extract Micrographs and ptcls from 2D Classes.

  • Result: Failed.

  • Topaz executable (I reinstalled Topaz again): /home/cryosparcuser/topaz.sh

Trial 3: Inputs to Topaz Train: Both Mics and ptcls from Extract Micrographs.

  • Result: Failed.

  • The error message is as below:

\[CPU:  724.5 MB  Avail: 499.77 GB\]

WARNING: no micrograph found matching image name "004206278573885948794_m26jul01d_NY1626_g3_00007GrK3_00072sq_v02_00005hl_00014enn.frames_patch_aligned_doseweighted". Skipping it.

\[CPU:  724.5 MB  Avail: 499.77 GB\]

WARNING: no micrograph found matching image name "004094380918364751086_m26jul01d_NY1626_g3_00028GrK3_00046sq_v02_00004hl_00017enn.frames_patch_aligned_doseweighted". Skipping it.

\[CPU:  724.5 MB  Avail: 499.77 GB\]

WARNING: no micrograph found matching image name "016343720720140038759_m26jul01d_NY1626_g3_00022GrK3_00047sq_v03_00003hl_00068enn.frames_patch_aligned_doseweighted". Skipping it.

\[CPU:  724.5 MB  Avail: 499.77 GB\]

WARNING: no micrograph found matching image name "015512745986933156503_m26jul01d_NY1626_g3_00022GrK3_00047sq_v02_00002hl_v01_00035enn.frames_patch_aligned_doseweighted". Skipping it.

\[CPU:  724.5 MB  Avail: 499.77 GB\]

Traceback (most recent call last):

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/bin/topaz", line 33, in <module>

\[CPU:  724.5 MB  Avail: 499.77 GB\]

sys.exit(load_entry_point('topaz-em==0.2.5', 'console_scripts', 'topaz')())

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/topaz/main.py", line 148, in main

\[CPU:  724.5 MB  Avail: 499.77 GB\]

args.func(args)

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/topaz/commands/train_test_split.py", line 128, in main

\[CPU:  724.5 MB  Avail: 499.77 GB\]

image_list_train = pd.DataFrame({'image_name': image_names_train, 'path': paths_train})

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/pandas/core/frame.py", line 468, in \__init_\_

\[CPU:  724.5 MB  Avail: 499.77 GB\]

mgr = init_dict(data, index, columns, dtype=dtype)

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 283, in init_dict

\[CPU:  724.5 MB  Avail: 499.77 GB\]

return arrays_to_mgr(arrays, data_names, index, columns, dtype=dtype)

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 78, in arrays_to_mgr

\[CPU:  724.5 MB  Avail: 499.77 GB\]

index = extract_index(arrays)

\[CPU:  724.5 MB  Avail: 499.77 GB\]

File "/home/cryosparcuser/miniforge3/envs/topaz/lib/python3.6/site-packages/pandas/core/internals/construction.py", line 397, in extract_index

\[CPU:  724.5 MB  Avail: 499.77 GB\]

raise ValueError("arrays must all be same length")

\[CPU:  724.5 MB  Avail: 499.77 GB\]

ValueError: arrays must all be same length

\[CPU:  724.5 MB  Avail: 499.77 GB\]

**Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 116, in cryosparc_master.cryosparc_compute.run.main
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py", line 332, in run_topaz_wrapper_train
    utils.run_process(split_command)
  File "/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/topaz_utils.py", line 99, in run_process
    assert process.returncode == 0, f"Subprocess exited with status {process.returncode} ({str_command})"
AssertionError: Subprocess exited with status 1 (/home/cryosparcuser/topaz.sh train_test_split --number 7475 --seed 1430494252 --image-dir /data/CS/20260701d_XXXXXX/CS-XXXXXX-J13/preprocessed /data/CS/20260701d_XXXXXX/CS-XXXXXX/J13/topaz_particles_processed.txt)**

@abhipsa Please can you let us know:

  1. the CryoSPARC version
  2. the non-default parameters of the Topaz Train job

The version is: v4.6.0

Estimated particle diameter (A)= 180
Expected number of particles= 100

Rest everything is “default”

(I don’t think cryosparc version is an issue, because topaz train on a different project runs fine but on a different cluster)

This error looks familiar, what step is it failing on, says it can’t do the training test split but did the preprocessing downsampling step work? Such are the downsampled images made, the .mrc files (mine are each 1mb for my downsampling) in the topaz train /processed folder? If those aren’t getting made, perhaps trying a reassign particles to micrographs or full motion/ctf on small subset of the micrographs.
If the preprocessed are there confirm the file names match up in .txt files to the .mrc files. Also the referenced images have an odd enn.frames_patch_aligned_doseweighted, while mine are all XXXXXX_EER_patch_aligned_doseweighted.mrc so maybe the “.” is causing an issue.

The event log says preprocessing is complete as below:

CPU: 569.8 MB Avail: 473.22 GB

Starting Topaz process using version 0.2.5a…

CPU: 569.8 MB Avail: 473.22 GB

Random seed used is 987956712

CPU: 572.8 MB Avail: 478.95 GB

--------------------------------------------------------------

CPU: 572.8 MB Avail: 478.97 GB

Starting preprocessing…

CPU: 572.8 MB Avail: 479.00 GB

Using a downsampling factor of 9

CPU: 572.8 MB Avail: 479.03 GB

Starting micrograph preprocessing by running command /home/cryosparcuser/topaz.sh preprocess --scale 9 --niters 200 --num-workers 4 -o /data/CS/20260701XXXXXXX/CS-2026-0701-XXXXXXX/J17/preprocessed [37371 MICROGRAPH PATHS EXCLUDED FOR LEGIBILITY]

CPU: 572.8 MB Avail: 479.04 GB

Preprocessing over 2 processes…

CPU: 580.6 MB Avail: 479.52 GB

Inverting negative staining…

CPU: 590.5 MB Avail: 483.07 GB

Inverting negative staining complete.

CPU: 590.5 MB Avail: 483.08 GB

Micrograph preprocessing command complete.

But when I go inside the /preprocessed folder, its empty.

@abhipsa Training a model on tens of thousands exposures is time consuming, may involve (under the hood) very long command lines and might not be needed for this use case.

For troubleshooting, may we suggest

  1. Connecting curated particles (such as from the latest Select 2D Classes job) and the exposures from which curated particles were picked to a [Curate Exposures] job.
  2. Selecting approx. 100 exposures (selection criteria not important for troubleshooting, very important after troubleshooting for “production” training)
  3. Connecting exposures and particle outputs from the Curate Exposures job to a new Topaz Train job
  4. Letting us know any errors observed when running that Topaz Train job

After troubleshooting is complete, for the eventual “production” run of Topaz Train, the selection criteria and size for the subset of exposures need to be carefully considered to ensure an unbiased representation of the full exposure set from which the trained model will eventually pick particles.