Database fails to start after upgrade from v4.7.1 to v5.0.4: No such file or directory: 'mongo'

Install, Configure and Update
1

CryoSPARC instance information

  • Type: cluster

  • Software version from cryosparcm status: v5.0.4

  • Output ofuname -a && free -g on master node:
    Linux ``cryosparc.somehost.com`` 6.8.0-107-generic #107-Ubuntu SMP PREEMPT_DYNAMIC Fri Mar 13 19:51:50 UTC 2026 x86_64 x86_64 x86_64 GNU/Linux
    total used free shared buff/cache available
    Mem: 31 2 27 0 1 28
    Swap: 0 0 0

CryoSPARC worker environment

  • deemed irrelevant for the reported issue

Issue

  • Type of Issue: Database startup error after cryosparc upgrade

  • Explanation: After upgrading to v5.0.4 using an ansible playbook, cryosparc was starting without trouble. Only when stopping / starting the application again afterwards on manually on the node through systemd, the cryosparc master failed to start. Also when starting it manually the startup failed. A node reboot did not help.

  • Include the full error message/traceback, if you received any

  • Relevant logs: see next section

  • backend: not running

  • jobs: none

  • webapp: not running

  • database: not running / see next section for logs

Steps to Reproduce

  • Upgrade cryosparc master and worker from v4.7.1 to v5.0.4 using ansible executing the commands cryosparcm update --version=v5.0.4 and ssh -i ~/.ssh/id_rsa@cluster /path/to/cryosparc/update_worker_from_cluster.sh

  • Ensure cryosparc is down:
    $ cryosparcm stop
    $ ps -eo user,pid,ppid,start,command | grep -e cryosparc_ -e mongo |grep -v grep
    $
    $ ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
    ls: cannot access ‘/tmp/cryosparc*.sock’: No such file or directory
    ls: cannot access ‘/tmp/mongo*.sock’: No such file or directory

  • Attempt to start the database

    $ cryosparcm start database
    Starting CryoSPARC database…
    ✕ ExecError: supervisorctl -c /cluster/project/scopem/cryosparc/install_path/cryosparc_master/config/supervisord.conf start ‘database:*’: Command failed (code 1)
    Output: database: ERROR (no such file)

    Error:

  • Changed base port following Change port after update to 5.0 - #3 by wtempel , and also because that has also helped to resolve a similar issue on another cryosparc v5 instance, and because the previous port that we had used was reported to be no longer a valid choice (Port range 39000 to 39010 overlaps with ephemeral port range (32768-60999) - the latter may also be due to a recent OS upgrade from Ubuntu 20.04 LTS to Ubuntu 24.04 LTS ):

    • $ cryosparcm changeport 61000
      Requested change base CryoSPARC port from 3900 to 61000. This will perform the following actions:
      Stop cryoSPARC
      Set the CRYOSPARC_BASE_PORT variable in config.sh to the new port
      Reconfigure the database to use the new port
      Start cryoSPARC
      Please ensure you have a recent database backup before proceeding
      Change base port? [y/N]: y
      Continuing.
      Stopping CryoSPARC …
      Shut down
      Wrote changes to config.sh
      Starting CryoSPARC database…
      ✕ ExecError: supervisorctl -c /cluster/project/scopem/cryosparc/install_path/cryosparc_master/config/supervisord.conf start ‘database:*’: Command failed (code 1)
      Output: database: ERROR (no such file)

      Error:

  • Checking logs:
    No new logs appended to the database log - the last log seems to date back to the point in time when cryosparc was still running (after the upgrade to 5.0.4):
    $ cryosparcm log database |tail -n 40
    2026-04-10T10:19:11.527+0200 I NETWORK [listener] connection accepted from 129.132.35.39:37478 #1457 (15 connections now open)
    2026-04-10T10:19:11.528+0200 I NETWORK [listener] connection accepted from 129.132.35.39:37496 #1458 (16 connections now open)
    2026-04-10T10:19:11.528+0200 I NETWORK [conn1457] received client metadata from 129.132.35.39:37478 conn1457: { driver: { name: “nodejs|Meteor”, version: “6.16.0|METEOR@3.3.2” }, platform: “Node.js v22.19.0, LE”, os: { name: “linux”, architecture: “x64”, version: “6.8.0-101-generic”, type: “Linux” } }
    2026-04-10T10:19:11.529+0200 I NETWORK [conn1458] received client metadata from 129.132.35.39:37496 conn1458: { driver: { name: “nodejs|Meteor”, version: “6.16.0|METEOR@3.3.2” }, platform: “Node.js v22.19.0, LE”, os: { name: “linux”, architecture: “x64”, version: “6.8.0-101-generic”, type: “Linux” } }
    2026-04-10T10:19:11.532+0200 I NETWORK [conn1458] end connection 129.132.35.39:37496 (15 connections now open)
    2026-04-10T10:19:11.533+0200 I NETWORK [conn1457] end connection 129.132.35.39:37478 (14 connections now open)
    2026-04-10T10:19:12.035+0200 I NETWORK [listener] connection accepted from 129.132.35.39:37512 #1459 (15 connections now open)
    2026-04-10T10:19:12.036+0200 I NETWORK [conn1459] received client metadata from 129.132.35.39:37512 conn1459: { driver: { name: “nodejs|Meteor”, version: “6.16.0|METEOR@3.3.2” }, platform: “Node.js v22.19.0, LE”, os: { name: “linux”, architecture: “x64”, version: “6.8.0-101-generic”, type: “Linux” } }
    2026-04-10T10:19:12.036+0200 I NETWORK [listener] connection accepted from 129.132.35.39:37514 #1460 (16 connections now open)
    2026-04-10T10:19:12.036+0200 I NETWORK [conn1460] received client metadata from 129.132.35.39:37514 conn1460: { driver: { name: “nodejs|Meteor”, version: “6.16.0|METEOR@3.3.2” }, platform: “Node.js v22.19.0, LE”, os: { name: “linux”, architecture: “x64”, version: “6.8.0-101-generic”, type: “Linux” } }
    2026-04-10T10:19:12.037+0200 I NETWORK [conn1459] end connection 129.132.35.39:37512 (15 connections now open)
    2026-04-10T10:19:12.038+0200 I NETWORK [conn1460] end connection 129.132.35.39:37514 (14 connections now open)
    2026-04-10T10:19:16.654+0200 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
    2026-04-10T10:19:16.654+0200 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
    2026-04-10T10:19:16.654+0200 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
    2026-04-10T10:19:16.655+0200 I REPL [signalProcessingThread] shutting down replication subsystems
    2026-04-10T10:19:16.655+0200 I REPL [signalProcessingThread] Stopping replication reporter thread
    2026-04-10T10:19:16.656+0200 I REPL [signalProcessingThread] Stopping replication fetcher thread
    2026-04-10T10:19:16.656+0200 I REPL [signalProcessingThread] Stopping replication applier thread
    2026-04-10T10:19:16.656+0200 I NETWORK [conn20] end connection 129.132.35.39:45930 (13 connections now open)
    2026-04-10T10:19:16.656+0200 I NETWORK [conn21] end connection 129.132.35.39:45946 (12 connections now open)
    2026-04-10T10:19:16.674+0200 I NETWORK [conn23] end connection 127.0.0.1:53790 (11 connections now open)
    2026-04-10T10:19:16.674+0200 I NETWORK [conn26] end connection 127.0.0.1:53820 (10 connections now open)
    2026-04-10T10:19:16.674+0200 I NETWORK [conn24] end connection 127.0.0.1:53802 (8 connections now open)
    2026-04-10T10:19:16.674+0200 I NETWORK [conn25] end connection 127.0.0.1:53818 (9 connections now open)
    2026-04-10T10:19:16.816+0200 I NETWORK [conn16] end connection 129.132.35.39:42352 (7 connections now open)
    2026-04-10T10:19:16.816+0200 I NETWORK [conn17] end connection 129.132.35.39:42362 (6 connections now open)
    2026-04-10T10:19:16.820+0200 I NETWORK [conn19] end connection 129.132.35.39:42388 (5 connections now open)
    2026-04-10T10:19:16.820+0200 I NETWORK [conn14] end connection 129.132.35.39:42332 (4 connections now open)
    2026-04-10T10:19:16.854+0200 I NETWORK [conn15] end connection 129.132.35.39:42338 (3 connections now open)
    2026-04-10T10:19:16.854+0200 I NETWORK [conn18] end connection 129.132.35.39:42360 (2 connections now open)
    2026-04-10T10:19:16.883+0200 I NETWORK [conn138] end connection 129.132.35.39:37756 (1 connection now open)
    2026-04-10T10:19:16.883+0200 I NETWORK [conn139] end connection 129.132.35.39:37770 (0 connections now open)
    2026-04-10T10:19:17.223+0200 I REPL [signalProcessingThread] Stopping replication storage threads
    2026-04-10T10:19:17.226+0200 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
    2026-04-10T10:19:17.245+0200 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
    2026-04-10T10:19:17.245+0200 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
    2026-04-10T10:19:17.873+0200 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
    2026-04-10T10:19:17.879+0200 I CONTROL [signalProcessingThread] now exiting
    2026-04-10T10:19:17.879+0200 I CONTROL [signalProcessingThread] shutting down with code:0

  • I tried to reuse a recipe that has helped on another cryosparc instance to resolve a similar issue after upgrade, described here:
    $ cryosparcm database fixport
    […]
    FileNotFoundError: [Errno 2] No such file or directory: 'mongo'

  • I thus appeded the path to this binary to my current PATH var and retried to start cryosparc again, giving yet another error:
    $ export PATH=”${PATH}:/xyz/cryosparc/install_path/cryosparc_master/deps_bundle/external/mongodb/bin”
    $ cryosparcm start
    OperationFailure: Our replica set config is invalid or we are not a member of it, full error: {‘ok’: 0.0, ‘errmsg’: ‘Our replica set config is invalid or we are not a member of it’, ‘code’: 93, ‘codeName’: ‘InvalidReplicaSetConfig’}
    2026-04-10T13:45:44.637559 Error configuring database. Most recent database log lines:
    2026-04-10T13:45:36.439+0200 I STORAGE [signalProcessingThread] Shutting down the PeriodicThreadToAbortExpiredTransactions
    2026-04-10T13:45:36.439+0200 I REPL [signalProcessingThread] Shutting down the ReplicationCoordinator
    2026-04-10T13:45:36.439+0200 I REPL [signalProcessingThread] shutting down replication subsystems
    2026-04-10T13:45:36.439+0200 I ASIO [Replication] Killing all outstanding egress activity.
    2026-04-10T13:45:36.439+0200 I SHARDING [signalProcessingThread] Shutting down the ShardingInitializationMongoD
    2026-04-10T13:45:36.439+0200 I COMMAND [signalProcessingThread] Killing all open transactions
    2026-04-10T13:45:36.439+0200 I - [signalProcessingThread] Killing all operations for shutdown
    2026-04-10T13:45:36.440+0200 I NETWORK [signalProcessingThread] Shutting down the ReplicaSetMonitor
    2026-04-10T13:45:36.440+0200 I ASIO [ReplicaSetMonitor-TaskExecutor] Killing all outstanding egress activity.
    2026-04-10T13:45:36.440+0200 I CONNPOOL [ReplicaSetMonitor-TaskExecutor] Dropping all pooled connections to localhost:39001 due to ShutdownInProgress: Shutting down the connection pool
    2026-04-10T13:45:36.440+0200 I REPL [signalProcessingThread] Shutting down the LogicalTimeValidator
    2026-04-10T13:45:36.440+0200 I CONTROL [signalProcessingThread] Shutting down free monitoring
    2026-04-10T13:45:36.440+0200 I FTDC [signalProcessingThread] Shutting down full-time data capture
    2026-04-10T13:45:36.440+0200 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
    2026-04-10T13:45:36.447+0200 I STORAGE [signalProcessingThread] Shutting down the HealthLog
    2026-04-10T13:45:36.447+0200 I STORAGE [signalProcessingThread] Shutting down the storage engine
    2026-04-10T13:45:36.448+0200 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
    2026-04-10T13:45:36.448+0200 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
    2026-04-10T13:45:36.453+0200 I STORAGE [signalProcessingThread] Shutting down session sweeper thread
    2026-04-10T13:45:36.453+0200 I STORAGE [signalProcessingThread] Finished shutting down session sweeper thread
    2026-04-10T13:45:36.504+0200 I STORAGE [signalProcessingThread] Downgrading WiredTiger datafiles.
    2026-04-10T13:45:38.900+0200 I STORAGE [signalProcessingThread] WiredTiger message [1775821538:900839][8454:0x7881099ff6c0], txn-recover: Main recovery loop: starting at 30952/170624 to 30953/256
    2026-04-10T13:45:39.004+0200 I STORAGE [signalProcessingThread] WiredTiger message [1775821539:4466][8454:0x7881099ff6c0], txn-recover: Recovering log 30952 through 30953
    2026-04-10T13:45:39.092+0200 I STORAGE [signalProcessingThread] WiredTiger message [1775821539:92034][8454:0x7881099ff6c0], txn-recover: Recovering log 30953 through 30953
    2026-04-10T13:45:39.139+0200 I STORAGE [signalProcessingThread] WiredTiger message [1775821539:139388][8454:0x7881099ff6c0], txn-recover: Set global recovery timestamp: 0
    2026-04-10T13:45:39.291+0200 I STORAGE [signalProcessingThread] Starting to check the table logging settings for existing WiredTiger tables
    2026-04-10T13:45:44.068+0200 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
    2026-04-10T13:45:44.075+0200 I - [signalProcessingThread] Dropping the scope cache for shutdown
    2026-04-10T13:45:44.075+0200 I CONTROL [signalProcessingThread] now exiting
    2026-04-10T13:45:44.075+0200 I CONTROL [signalProcessingThread] shutting down with code:0

  • Re-ran the changeport and database fixport commands again:
    $ cryosparcm changeport 61000
    $ cryosparcm start database
    $ cryosparcm database fixport

  • This finally resolved the issue, I was able to start cryosparc again.

Question

Is there any explanation why cryosparc suddenly fails to discover the mongo binary unless we explicitly add the path to where it’s stored to the user account that’s supposed to run the application?
What’s making things particularly hard to understand is why the application was running initially after upgrade, when started via ansible, but after a single restart the startup did not succeed any more.

Thanks for your help and insights!