Bug: Cross-platform migration failure to v5.0.x results in empty collection catalog mapping, replica set limbo (NotMaster), and forced archive restoration

System Environment Details

  • Host System 1 (Current): AlmaLinux release 10.2 (Lavender Lion) | Kernel: 6.12.0-211.20.1.el10_2.x86_64

  • Host System 2 (Prior): Ubuntu 24.04.4 LTS (Noble Numbat) | Kernel: 6.8.0-49-generic (cuba.cryoem.zi.columbia.edu)

  • Target Software Version: CryoSPARC v5.0.6 (Upgraded from v4.x legacy database files)

  • Database Engine: Embedded MongoDB v4.0.28 (WiredTiger Storage Engine)

  • Frequency: This is the second standalone CryoSPARC instance on our network displaying this exact behavior across completely different Linux distribution families, indicating a platform-independent bug in the v5 core upgrade handler.

Bug Description

During an upgrade path from a legacy v4 instance to v5.0.6, the database subsystem failed to execute or validate the structural schema migrations over an existing ~200 GB database data path (/opt/apps/cryosparc_database). Instead of processing the historical records, the application layer spun up a fallback instance, stamped the internal version metadata as 5.0.6, and generated a clean database catalog file mapping.

As a result, checking db.projects.count_documents({}) returned exactly 10 template/dummy records, while the underlying physical collection-*.wt allocation tables remained on disk but completely unmapped by the MongoDB catalog.

Manual remediation using native database administration routines was blocked by severe internal database lockouts:

  1. The engine was initialized into a read-only replication state (SECONDARY) without cluster administrative permissions assigned to the cryosparcm mongo application user, preventing manual drops of stale version markers (not authorized on admin to execute command { replSetInitiate... }).

  2. Dropping the replica state tracking via unauthenticated standalone maintenance mode (--noauth, dropping local DB) caused the managed supervisord database invocation to enter an existential recovery state (NotPrimaryError: node is not in primary or recovering state).

  3. Running cryosparcm upgrade or cryosparcm upgrade --force failed to pick up the orphan records due to fixed validation parameters or version parsing limits.

Final Resolution Block: The system could only be salvaged by cleanly stopping the instance, completely blowing away/renaming the physical /opt/apps/cryosparc_database path, and running a cold binary ingestion via cryosparcm database restore --file=<archive> from a pre-upgrade .archive dump.

Steps to Reproduce / Technical Timeline

  1. Initial State: Run a managed upgrade on a large historical database block (~193 GB physical collection data) on either an AlmaLinux 10 or Ubuntu 24.04 host.

  2. Symptom: Open Web UI or CLI and check project tracking totals:

    cryosparcm cli "db.projects.count_documents({})"
    # Output: 10 (Historical user project list wiped from catalog view)
    
  3. Attempt Forced Upgrade: Attempting to force manual schema processing:

   cryosparcm start database
   cryosparcm upgrade
   # Output: ⚠ Previous version was 5.0.6; database upgrade is not required for this version.
  1. Bypass Verification / Auth Deadlock: Launching standalone MongoDB without authentication flags to drop tracking metadata resulted in a permanent Master/Slave authentication deadlock upon return to the managed supervisor layer:

    "errmsg" : "not master and slaveOk=false", "code" : 13435, "codeName" : "NotMasterNoSlaveOk"
    
  2. Final Workaround:

    cryosparcm stop
   mv /opt/apps/cryosparc_database /opt/apps/cryosparc_database_blank_bak
   mkdir /opt/apps/cryosparc_database
   cryosparcm database restore --file=/path/to/backup.archive

Requested Log Bundles for Engineering Analysis

To help track down why the v5 schema transformation script silently decoupled from the existing storage files, the logs and conf are uploaded here:

  1. Core Database Runtime Log (Crucial Trace Data):

    • Path: /opt/apps/cryosparc_master/run/database.log

    • Note on File State: This log file actively grew in size during the runtime restoration process as the background utility stream mapped incoming collections. The early timestamps (prior to the restoration block) capture the precise, historical NotMasterNoSlaveOk initialization errors and replica-set identity crisis loops that caused the upgrade script to stall.

  2. The Command API Stack Logs:

    • Path: /opt/apps/cryosparc_master/run/command_core.log

    • Why: Logs the initial schema validation exceptions (CSModelException or validation array serialization failures) during the first v5 boot sequence.

  3. Supervisor Configuration Matrix:

    • Path: /opt/apps/cryosparc_master/config/supervisord.conf

    • Why: Verifies the exact flag architecture passed to the replica sets on modern Linux deployment spaces.

So let me know how I can provide the log files!

Thanks @RobK for the detailed report.

Please can you describe the upgrade procedure performed:

  1. Context: Were commands run in an interactive shell session?
  2. Commands and prompt interactions:
    • full commands
    • prompts encountered
    • responses to these prompts

Please can you provide details (as above) for the managed upgrade procedure

Yes, the commands were executed sequentially within an interactive SSH shell session as the root user directly on the cluster management/master node (engram-36).

Commands, Prompts, and Interactions:

1. Shutting Down the Pre-Existing CryoSPARC Instance

Before attempting the cross-platform metadata migration, the active CryoSPARC instance was stopped cleanly to freeze the database state.

  • Command Run:

    cryosparcm stop
    

    Prompts Encountered: None. The core processes (command_core, command_vis, command_rtp) and the underlying MongoDB instance shut down asynchronously.

  • Responses / Outcome: Terminal output confirmed all processes were successfully stopped.

2. Executing the CryoSPARC v5.0.x Upgrade Command

The interactive update script was triggered to pull the new version binaries and initiate the internal database schema migrations.

  • Command Run:

    cryosparcm update
    
  • Prompts Encountered: The script paused mid-execution to verify backup compliance before touching the database collections:

    Are you sure you want to update to the latest version? (y/n)
    
  • Responses to Prompts: Manually entered y and pressed Enter.

  • Prompts Encountered (Database Warning): A secondary interactive prompt was encountered warning about the database migration step and confirming a recent independent DB archive existed:

    Have you backed up your database before proceeding? (y/n)
    
  • Responses to Prompts: Manually entered y and pressed Enter.

3. Post-Upgrade Database Start & Failure State

Following completion of the script binaries deployment, the instance was started to complete the catalog mapping, which immediately triggered the replica set notmaster lock/limbo state.

  • Command Run:

    Bash

    cryosparcm start
    
    
  • Prompts Encountered: None.

  • Responses / Outcome: The standard startup sequence failed out during the MongoDB validation check, resulting in an unrecoverable empty collection catalog mapping. The terminal threw continuous connection loops reflecting the database being trapped in a standalone/replica set mismatch state, forcing an administrative archive restoration.

Thanks @RobK for the additional details. Please can you prepare an archive

cd $(mktemp -d)
tar zcvf engram-36-cs-logs.tgz \ 
  /opt/apps/cryosparc_master/config.sh \ 
  /opt/apps/cryosparc_master/run/
# note path for retrieval
realpath engram-36-cs-logs.tgz

and let me know, via a personal forum message, whether you can share the engram-36-cs-logs.tgz using a private link on your institution’s filesharing platform or we should make sharing arrangements on our end.

Thanks @RobK for sharing the requested information. A review of the shared files suggest that shared logs correspond to a different CryoSPARC master host, not engram-36 as you mentioned earlier. Could there have been a mixup?

No I definitely provided logs from the server in question it has 2 hostnames one that starts with cryoem11 and the other starts with sn (serial number).

Thanks for the clarification.

Yes, the cryoem11 hostname matches the logs. A first look at the logs suggests that records for 11 projects were upgraded successful during the update at 2026-06-23 12:37.
What were the timestamps of the events, like

?
What is the relationship between
the root user/engram-36 node

and the non-root user that should have run the commands on cryoem11?

Nothing I must’ve been logged in there and it’s a Dell Isilon

nothing at all

on node.

The logs indicate that the database filesystem ran out of space on 2026-06-09. What actions (and when) were performed to recover from the No space left on device event?

The only thing that comes to mind was deleting the cache that was written to disk I changed that to use TMPFS after the upgrade.

What is the current state of the CryoSPARC instance? Is it in use, or is it currently down?

It’s up and running after doing a restore of the DB.

I can provide logs from another server that we had to do the same.

Please can you confirm for this instance:

  • What was the CryoSPARC version when the DB backup was created?
  • What was the CryoSPARC version when the DB backup was restored?
  • Was it ensured that no projects or jobs were modified between the backup and the restoration (see caution)?

Please do, as before (cryosparc_master/config.sh plus all of cryosparc_master/run/.
Additionally, please provide timestamps for key events, such as software updates, database backups, database restoration, etc. for both servers/instances, to help us navigate the logs.

4.7

I don’t recall.

yes no jobs open/running at all

PM’d you the link

@RobK We suspect that the No space left on device on cryoem11 left CryoSPARC-related database records in an inconsistent state. We recommend continuous monitoring of the filesystem holding the database to ensure it never runs out of space.
On the second CryoSPARC server, a CryoSPARC startup record at 2026-05-22 09:59:06, predating the update to v5 by a few minutes, indicates that the database already was missing some expected CryoSPARC-related records.
These findings suggest underlying causes other than the CryoSPARC update to v5 for the database problems on these CryoSPARC instances.