V4.7.1 — recovering an exact particle set by re-extracting from re-imported movies after a scratch purge (re-associating micrograph UIDs?)

Import and Export
1

Hi all,

After a scratch auto-purge on our cluster I lost the preprocessing and extracted particle stacks for a project, but I still have (a) the original movies backed up off-scratch and (b) the particle .cs from a downstream refinement job, which includes the refined poses. I’d like to recover the exact same particle set — and preserve a split I made into two volumes via 3D classification — while making sure I don’t damage the parts of the project that are still intact.

Setup

  • cryoSPARC v4.7.1, master + worker on an HPC cluster
  • Project was on cluster scratch (automatically purged, not backed up); persistent storage is available at /user/work/…
  • The purged data includes particle stacks under e.g. J1262/reconstructed/…_particles.mrc; the surviving particle .cs (e.g. J1346_000_particles.cs) still has the membership + refined poses

I’ve read “Re-extract particles after losing correspondence to original micrographs” and the .cs-manipulation guide. My intended plan, in a fresh project on /user/work, is:
Import movies → Patch Motion → Patch CTF (same params/pixel size) → re-associate the surviving particle .cs to the new micrographs by matching the micrograph path → Import Result Group → Extract from Micrographs (same box/pixel size) → re-refine.

My questions for v4.7.1

  1. Is that still the recommended workflow, or is there now a GUI job that re-associates particles to newly imported micrographs, so I can avoid manually editing the .cs?
  2. If manual editing is still needed, what’s the current dataset-API call? The snippet in the older thread uses from cryosparc_compute import dataset, which I assume has changed.
  3. Will this recover exactly the same particles (matching by micrograph filename), and carry through the refined poses and my classification-based volume split, so I can resume refinement rather than re-classify? Is there anything I must keep identical (box size, pixel size, fractional coordinates) for the poses to stay valid?
  4. Separately: on a failed 3D Classification restart, cryoSPARC auto-queued all ancestor jobs and they began re-running from Patch Motion Correction (still running). If I kill that cascade to stop further writes to scratch, will it affect any of the intact branches/jobs in the project?

Error that started it

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 129, in cryosparc_master.cryosparc_compute.run.main
  File "cryosparc_master/cryosparc_compute/jobs/class3D/run.py", line 101, in cryosparc_master.cryosparc_compute.jobs.class3D.run.run_class_3D
  File "/user/work/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/particles.py", line 131, in read_blobs
    with open(abs_path, 'rb') as mrc_file:
FileNotFoundError: [Errno 2] No such file or directory: '/scratch/cryosparcuser/aa17341/CS-ffar1-ns-20241204/CS-p1-ffar1-bril2-fab-nb-2025/J1262/reconstructed/57944304851460488877_particles.mrc'

Thanks very much for any guidance.

2

You may try (disclaimer: I have not tested this sequence)

  1. converting the cs file to the start format, with a utility like pyem
  2. importing particles
    • enable Ignore raw data
    • connect newly imported exposures
    • use exposure file/rlnMicrographName prefix/suffix settings as needed to ensure correct particle-exposure associations

Regarding potentially needed scripted adjustments

This functionality is now available in cryosparc-tools.

1 Like