Database corrupted after outage — restore backup or rebuild DB?

Troubleshooting
1

Hi CryoSPARC team and community,

I’m looking for advice on the safest recovery path after a storage/outage issue that appears to have corrupted our CryoSPARC database.

Our current situation is:

  • CryoSPARC was updated recently (I ran cryosparcm update --version v4.7.0; previously I had also used v4.3.1 during troubleshooting).

  • After a storage issue/outage, CryoSPARC stopped starting correctly.

  • We saw database startup problems and app spawn errors.

  • I have older CryoSPARC database backups (.archive), but the newest one is from 2024-04-19 (so it is significantly outdated relative to current project activity).

  • The active database path is large (~3.2 TB) and likely damaged. /data is nearly full, so space is a constraint.

I found a previous forum thread suggesting that restoring an outdated DB backup may overwrite newer metadata and potentially cause inconsistencies with project directories, and that a safer approach may be:

  1. Start a new blank database at a new path. Are there version-specific concerns attaching older project directories into a new DB on CryoSPARC v4.7.0? Is there any safe way to salvage partial metadata from the damaged DB path (without risking project corruption), or should I avoid touching it beyond copying files?
  2. Recreate users / reconnect workers. Also, when attaching projects to a fresh DB, what metadata is typically lost (job history details, user permissions, scheduler state, workspace organization)?
  3. Attach existing project directories (possibly removing cs.lock in project folders if needed).If I must remove cs.lock for attachment, is there a recommended bulk safe workflow to do this only when necessary?

This sounds like the best direction, but I’d like confirmation and best practices/suggestions/options before proceeding.

Thanks for your time reading this. If anyone has done this exact recovery path after a storage outage/corrupted WiredTiger DB, I’d really appreciate guidance.

2

Welcome to the forum @David66. Bias alert: I posted the instructions to which you linked. The warning against restoring an outdated database backup still applies, as does the recommendation to build a new database. The rebuilding may be an opportunity to attach only projects that are still needed, which should result in a leaner database.