Master node not updating

Install, Configure and Update
1

I am trying to update to v4.2.1 using cryosparcm update. I get the following message:

===================================================
Successfully updated master to version v4.2.1.
===================================================
 
Starting cryoSPARC System master process..
CryoSPARC is not already running.
^[[ATraceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/cryosparc_compute/database_management.py", line 48, in configure_mongo
    initialize_replica_set()
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/cryosparc_compute/database_management.py", line 87, in initialize_replica_set
    admin_db = try_get_pymongo_admin_db(mongo_client)
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/cryosparc_compute/database_management.py", line 249, in try_get_pymongo_admin_db
    admin_db.command(({'serverStatus': 1}))
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 828, in command
    return self._command(
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 703, in _command
    return sock_info.command(
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 740, in command
    return command(
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/network.py", line 177, in command
    helpers._check_command_response(
  File "/CryoSparc/CryoSparcV2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/helpers.py", line 180, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: there are no users authenticated, full error: {'operationTime': Timestamp(1682432559, 1), 'ok': 0.0, 'errmsg': 'there are no users authenticated', 'code': 13, 'codeName': 'Unauthorized', '$clusterTime': {'clusterTime': Timestamp(1682432559, 1), 'signature': {'hash': b'S\xa2A\xb8/\x9c\xc7\xd4\x9c\x1fnw\xb8\xe3\xe4\xfd\xd1\x14\xf0\x85', 'keyId': 7192753418186784769}}}
configuring database
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
[2023-04-25T10:22:40-0400] Error configuring database. Most recent database log lines:
2023-04-25T10:12:53.153-0400 I COMMAND  [conn84] command meteor.jobs command: find { find: "jobs", filter: { deleted: { $ne: true }, status: { $in: [ "completed", "failed", "killed" ] }, project_uid: { $in: [ "P3", "P5", "P14", "P15", "P27", "P33", "P34", "P35", "P36", "P37", "P41", "P46", "P48", "P49", "P51", "P52", "P53", "P55", "P56", "P57", "P58", "P59", "P60", "P62", "P64", "P65", "P66", "P68", "P69", "P70", "P71", "P72", "P74", "P75", "P77", "P78", "P79", "P80", "P87", "P104", "P105", "P106", "P111", "P112", "P113" ] } }, sort: { completed_at: -1 }, projection: { children: true, completed_at: true, created_at: true, created_by_job_uid: true, created_by_user_id: true, deleted: true, failed_at: true, interactive: true, job_type: true, killed_at: true, launched_at: true, output_group_images: true, output_result_groups: true, params_spec: true, parents: true, project_uid: true, project_uid_num: true, queued_at: true, queued_to_lane: true, resources_allocated: true, running_at: true, starred: true, started_at: true, status: true, tags: true, title: true, ui_tile_images: true, ui_tile_width: true, uid: true, uid_num: true, workspace_uids: true }, limit: 20, lsid: { id: UUID("e4d1c240-2a69-4688-ab2a-1f13b940e8de") }, $clusterTime: { clusterTime: Timestamp(1682431973, 1), signature: { hash: BinData(0, E013FB1D7AD46AB1DBB1C1BEB84EFE8FF528CE83), keyId: 7192753418186784769 } }, $db: "meteor" } planSummary: IXSCAN { project_uid: 1, uid: 1 } keysExamined:4454 docsExamined:4435 hasSortStage:1 cursorExhausted:1 numYields:35 nreturned:20 reslen:642109 locks:{ Global: { acquireCount: { r: 72 } }, Database: { acquireCount: { r: 36 } }, Collection: { acquireCount: { r: 36 } } } protocol:op_msg 106ms
2023-04-25T10:16:39.849-0400 I COMMAND  [LogicalSessionCacheRefresh] command config.$cmd command: update { update: "system.sessions", ordered: false, allowImplicitCollectionCreation: false, writeConcern: { w: "majority", wtimeout: 15000 }, $db: "config" } numYields:0 reslen:229 locks:{ Global: { acquireCount: { r: 26, w: 26 } }, Database: { acquireCount: { w: 26 } }, Collection: { acquireCount: { w: 13 } }, oplog: { acquireCount: { w: 13 } } } protocol:op_msg 3487ms
2023-04-25T10:21:42.273-0400 I COMMAND  [LogicalSessionCacheRefresh] command config.$cmd command: update { update: "system.sessions", ordered: false, allowImplicitCollectionCreation: false, writeConcern: { w: "majority", wtimeout: 15000 }, $db: "config" } numYields:0 reslen:229 locks:{ Global: { acquireCount: { r: 12, w: 12 } }, Database: { acquireCount: { w: 12 } }, Collection: { acquireCount: { w: 6 } }, oplog: { acquireCount: { w: 6 } } } protocol:op_msg 5911ms
2023-04-25T10:22:29.871-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:59182 #2744 (156 connections now open)
2023-04-25T10:22:29.872-0400 I NETWORK  [conn2744] received client metadata from 127.0.0.1:59182 conn2744: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-04-25T10:22:29.874-0400 I NETWORK  [listener] connection accepted from 127.0.0.1:59184 #2745 (157 connections now open)
2023-04-25T10:22:29.874-0400 I NETWORK  [conn2745] received client metadata from 127.0.0.1:59184 conn2745: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-04-25T10:22:39.889-0400 I ACCESS   [conn2745] Unauthorized: not authorized on admin to execute command { endSessions: [ { id: UUID("0506b1ec-e587-4c2a-8505-cc5485a02c64") } ], $clusterTime: { clusterTime: Timestamp(1682432559, 1), signature: { hash: BinData(0, 53A241B82F9CC7D49C1F6E77B8E3E4FDD114F085), keyId: 7192753418186784769 } }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }

Then when I try to run a job I get this error message:


AssertionError: Version mismatch! Worker version (v4.2.1) and Master version (v4.1.2) are not the same. Please update.

How can I update the master node only to v4.2.1?

2

I suspect that a “zombie” database process prevented proper startup of updated master services. This problem may also have prevented update of any workers.
For recovery, I suggest (run commands under Linux account that “owns” the CryoSPARC instance and its processes):

  1. cryosparcm stop
  2. Identify CryoSPARC processes that were not terminated as expected:
    ps -u $USER -opid,ppid,cmd | grep -e cryosparc -e mongo
  3. Determine the process IDs of potentially remaining supervisord and mongod processes and, if such processes exist, terminate them
    kill -TERM <process id>
    (but not kill -KILL or kill -9), wait 10 seconds and confirm that no CryoSPARC related processes, such as supervisord or mongod remain:
    ps -u $USER -opid,ppid,cmd | grep -e cryosparc -e mongo (again)
  4. cryosparcm start

If CryoSPARC startup succeeds, manually update the worker(s) according to “cluster” instructions. Otherwise, report any error messages here.