Frequency: This is the second standalone CryoSPARC instance on our network displaying this exact behavior across completely different Linux distribution families, indicating a platform-independent bug in the v5 core upgrade handler.
Bug Description
During an upgrade path from a legacy v4 instance to v5.0.6, the database subsystem failed to execute or validate the structural schema migrations over an existing ~200 GB database data path (/opt/apps/cryosparc_database). Instead of processing the historical records, the application layer spun up a fallback instance, stamped the internal version metadata as 5.0.6, and generated a clean database catalog file mapping.
As a result, checking db.projects.count_documents({}) returned exactly 10 template/dummy records, while the underlying physical collection-*.wt allocation tables remained on disk but completely unmapped by the MongoDB catalog.
Manual remediation using native database administration routines was blocked by severe internal database lockouts:
The engine was initialized into a read-only replication state (SECONDARY) without cluster administrative permissions assigned to the cryosparcm mongo application user, preventing manual drops of stale version markers (not authorized on admin to execute command { replSetInitiate... }).
Dropping the replica state tracking via unauthenticated standalone maintenance mode (--noauth, dropping local DB) caused the managed supervisord database invocation to enter an existential recovery state (NotPrimaryError: node is not in primary or recovering state).
Running cryosparcm upgrade or cryosparcm upgrade --force failed to pick up the orphan records due to fixed validation parameters or version parsing limits.
Final Resolution Block: The system could only be salvaged by cleanly stopping the instance, completely blowing away/renaming the physical /opt/apps/cryosparc_database path, and running a cold binary ingestion via cryosparcm database restore --file=<archive> from a pre-upgrade .archive dump.
Steps to Reproduce / Technical Timeline
Initial State: Run a managed upgrade on a large historical database block (~193 GB physical collection data) on either an AlmaLinux 10 or Ubuntu 24.04 host.
Symptom: Open Web UI or CLI and check project tracking totals:
cryosparcm cli "db.projects.count_documents({})"
# Output: 10 (Historical user project list wiped from catalog view)
Attempt Forced Upgrade: Attempting to force manual schema processing:
cryosparcm start database
cryosparcm upgrade
# Output: ⚠ Previous version was 5.0.6; database upgrade is not required for this version.
Bypass Verification / Auth Deadlock: Launching standalone MongoDB without authentication flags to drop tracking metadata resulted in a permanent Master/Slave authentication deadlock upon return to the managed supervisor layer:
To help track down why the v5 schema transformation script silently decoupled from the existing storage files, the logs and conf are uploaded here:
Core Database Runtime Log (Crucial Trace Data):
Path: /opt/apps/cryosparc_master/run/database.log
Note on File State: This log file actively grew in size during the runtime restoration process as the background utility stream mapped incoming collections. The early timestamps (prior to the restoration block) capture the precise, historical NotMasterNoSlaveOk initialization errors and replica-set identity crisis loops that caused the upgrade script to stall.
Yes, the commands were executed sequentially within an interactive SSH shell session as the root user directly on the cluster management/master node (engram-36).
Commands, Prompts, and Interactions:
1. Shutting Down the Pre-Existing CryoSPARC Instance
Before attempting the cross-platform metadata migration, the active CryoSPARC instance was stopped cleanly to freeze the database state.
Command Run:
cryosparcm stop
Prompts Encountered: None. The core processes (command_core, command_vis, command_rtp) and the underlying MongoDB instance shut down asynchronously.
Responses / Outcome: Terminal output confirmed all processes were successfully stopped.
2. Executing the CryoSPARC v5.0.x Upgrade Command
The interactive update script was triggered to pull the new version binaries and initiate the internal database schema migrations.
Command Run:
cryosparcm update
Prompts Encountered: The script paused mid-execution to verify backup compliance before touching the database collections:
Are you sure you want to update to the latest version? (y/n)
Responses to Prompts: Manually entered y and pressed Enter.
Prompts Encountered (Database Warning): A secondary interactive prompt was encountered warning about the database migration step and confirming a recent independent DB archive existed:
Have you backed up your database before proceeding? (y/n)
Responses to Prompts: Manually entered y and pressed Enter.
3. Post-Upgrade Database Start & Failure State
Following completion of the script binaries deployment, the instance was started to complete the catalog mapping, which immediately triggered the replica set notmaster lock/limbo state.
Command Run:
Bash
cryosparcm start
Prompts Encountered: None.
Responses / Outcome: The standard startup sequence failed out during the MongoDB validation check, resulting in an unrecoverable empty collection catalog mapping. The terminal threw continuous connection loops reflecting the database being trapped in a standalone/replica set mismatch state, forcing an administrative archive restoration.
Thanks @RobK for the additional details. Please can you prepare an archive
cd $(mktemp -d)
tar zcvf engram-36-cs-logs.tgz \
/opt/apps/cryosparc_master/config.sh \
/opt/apps/cryosparc_master/run/
# note path for retrieval
realpath engram-36-cs-logs.tgz
and let me know, via a personal forum message, whether you can share the engram-36-cs-logs.tgz using a private link on your institution’s filesharing platform or we should make sharing arrangements on our end.
Thanks @RobK for sharing the requested information. A review of the shared files suggest that shared logs correspond to a different CryoSPARC master host, not engram-36 as you mentioned earlier. Could there have been a mixup?
No I definitely provided logs from the server in question it has 2 hostnames one that starts with cryoem11 and the other starts with sn (serial number).
Yes, the cryoem11 hostname matches the logs. A first look at the logs suggests that records for 11 projects were upgraded successful during the update at 2026-06-23 12:37.
What were the timestamps of the events, like
?
What is the relationship between
the root user/engram-36 node
and the non-root user that should have run the commands on cryoem11?
The logs indicate that the database filesystem ran out of space on 2026-06-09. What actions (and when) were performed to recover from the No space left on device event?
What was the CryoSPARC version when the DB backup was created?
What was the CryoSPARC version when the DB backup was restored?
Was it ensured that no projects or jobs were modified between the backup and the restoration (see caution)?
Please do, as before (cryosparc_master/config.sh plus all of cryosparc_master/run/.
Additionally, please provide timestamps for key events, such as software updates, database backups, database restoration, etc. for both servers/instances, to help us navigate the logs.
@RobK We suspect that the No space left on device on cryoem11 left CryoSPARC-related database records in an inconsistent state. We recommend continuous monitoring of the filesystem holding the database to ensure it never runs out of space.
On the second CryoSPARC server, a CryoSPARC startup record at 2026-05-22 09:59:06, predating the update to v5 by a few minutes, indicates that the database already was missing some expected CryoSPARC-related records.
These findings suggest underlying causes other than the CryoSPARC update to v5 for the database problems on these CryoSPARC instances.