Dear CryoSPARC community,
This is related to version 4.0.1 installed on a single workstation with Ubuntu 20.04.1 LTS. Everything worked fine till one day we got a hardware (mainboard) replacement. Before the workstation was connected back to the internet, somehow the system time was set to a future date in the year 2034. CryoSPARC started OK. After internet access got recovered, the system time was also automatically corrected. But then cryosparcm start gave the following error messages:
Starting cryoSPARC System master process…
CryoSPARC is not already running.
configuring database
Traceback (most recent call last):
File “”, line 1, in
File “/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 41, in configure_mongo
initializeReplicaSet()
File “/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_compute/database_management.py”, line 82, in initializeReplicaSet
admin_db.command(({‘serverStatus’: 1}))
File “/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/database.py”, line 761, in command
codec_options, session=session, **kwargs)
File “/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/database.py”, line 652, in _command
client=self.__client)
File “/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/pool.py”, line 721, in command
exhaust_allowed=exhaust_allowed)
File “/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/network.py”, line 163, in command
parse_write_concern_error=parse_write_concern_error)
File “/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/pymongo/helpers.py”, line 167, in _check_command_response
raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: New cluster time, 2034865471, is too far from this node’s wall clock time, 1725445269., full error: {‘operationTime’: Timestamp(2034865471, 8), ‘ok’: 0.0, ‘errmsg’: “New cluster time, 2034865471, is too far from this node’s wall clock time, 1725445269.”, ‘code’: 205, ‘codeName’: ‘ClusterTimeFailsRateLimiter’, ‘$clusterTime’: {‘clusterTime’: Timestamp(2034865471, 9), ‘signature’: {‘hash’: b’\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00\x00’, ‘keyId’: 0}}}
cryosparcm status gave the following:
CryoSPARC System master node installed at
/home/cryosparcuser/cryosparc/cryosparc_master
Current cryoSPARC version: v4.0.1
CryoSPARC process status:
app STOPPED Not started
app_api STOPPED Not started
app_api_dev STOPPED Not started
app_legacy STOPPED Not started
app_legacy_dev STOPPED Not started
command_core STOPPED Not started
command_rtp STOPPED Not started
command_vis STOPPED Not started
database STOPPED Not started
Connection Refused: Server at http://amax:39002/api is not accepting new connections. Is the server running?
An error ocurred while checking license status
Could not get license verification status. Are all CryoSPARC processes RUNNING?
And this is the output from cryosparcm log database:
2024-09-03T15:00:42.919+0800 I NETWORK [listener] connection accepted from 127.0.0.1:43282 340 (10 connections now open)
2024-09-03T15:00:42.920+0800 I NETWORK [conn340] received client metadata from 127.0.0.1:43282 conn340: { driver: { name: “nodejs”, version: “4.3.1” }, os: { type: “Linux”, name: “linux”, architecture: “x64”, version: “5.15.0-119-generic” }, platform: “Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)” }
2024-09-03T15:00:42.923+0800 I NETWORK [conn340] end connection 127.0.0.1:43282 (9 connections now open)
2024-09-03T15:00:43.323+0800 I NETWORK [conn332] end connection 127.0.0.1:53752 (8 connections now open)
2024-09-03T15:00:43.324+0800 I NETWORK [listener] connection accepted from 127.0.0.1:35002 #341 (9 connections now open)
2024-09-03T15:00:43.324+0800 I NETWORK [conn341] received client metadata from 127.0.0.1:35002 conn341: { driver: { name: “PyMongo”, version: “3.12.2” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.15.0-119-generic” }, platform: “CPython 3.7.12.final.0” }
2024-09-03T15:00:44.290+0800 I NETWORK [conn333] end connection 127.0.0.1:49444 (8 connections now open)
2024-09-03T15:00:45.747+0800 I NETWORK [conn334] end connection 127.0.0.1:53754 (7 connections now open)
2024-09-03T15:00:45.747+0800 I NETWORK [conn335] end connection 127.0.0.1:53768 (6 connections now open)
2024-09-03T15:00:45.747+0800 I NETWORK [listener] connection accepted from 127.0.0.1:35004 #342 (7 connections now open)
2024-09-03T15:00:45.748+0800 I NETWORK [conn342] received client metadata from 127.0.0.1:35004 conn342: { driver: { name: “PyMongo”, version: “3.12.2” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.15.0-119-generic” }, platform: “CPython 3.7.12.final.0” }
2024-09-03T15:00:46.416+0800 I NETWORK [listener] connection accepted from 127.0.0.1:35012 #343 (8 connections now open)
2024-09-03T15:00:46.416+0800 I NETWORK [conn343] received client metadata from 127.0.0.1:35012 conn343: { driver: { name: “PyMongo”, version: “3.12.2” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.15.0-119-generic” }, platform: “CPython 3.7.12.final.0” }
2024-09-03T15:00:46.418+0800 I ACCESS [conn343] Successfully authenticated as principal cryosparc_user on admin from client 127.0.0.1:35012
2024-09-03T15:00:47.934+0800 I NETWORK [conn336] end connection 127.0.0.1:43242 (7 connections now open)
2024-09-03T15:00:48.672+0800 I NETWORK [listener] connection accepted from 127.0.0.1:35824 #344 (8 connections now open)
2024-09-03T15:00:48.672+0800 I NETWORK [conn344] received client metadata from 127.0.0.1:35824 conn344: { driver: { name: “nodejs”, version: “4.3.1” }, os: { type: “Linux”, name: “linux”, architecture: “x64”, version: “5.15.0-119-generic” }, platform: “Node.js v14.19.3, LE (unified)|Node.js v14.19.3, LE (unified)” }
2024-09-03T15:00:52.333+0800 I NETWORK [conn337] end connection 127.0.0.1:34988 (7 connections now open)
2024-09-03T15:00:52.334+0800 I NETWORK [listener] connection accepted from 127.0.0.1:51510 #345 (8 connections now open)
2024-09-03T15:00:52.334+0800 I NETWORK [conn345] received client metadata from 127.0.0.1:51510 conn345: { driver: { name: “PyMongo”, version: “3.12.2” }, os: { type: “Linux”, name: “Linux”, architecture: “x86_64”, version: “5.15.0-119-generic” }, platform: “CPython 3.7.12.final.0” }
2024-09-03T15:00:52.413+0800 I NETWORK [conn338] end connection 127.0.0.1:43252 (7 connections now open)
2024-09-03T15:00:52.471+0800 I CONTROL [signalProcessingThread] got signal 15 (Terminated), will terminate after current cmd ends
2024-09-03T15:00:52.471+0800 I NETWORK [signalProcessingThread] shutdown: going to close listening sockets…
2024-09-03T15:00:52.471+0800 I NETWORK [signalProcessingThread] removing socket file: /tmp/mongodb-39001.sock
2024-09-03T15:00:52.472+0800 I REPL [signalProcessingThread] shutting down replication subsystems
2024-09-03T15:00:52.472+0800 I REPL [signalProcessingThread] Stopping replication reporter thread
2024-09-03T15:00:52.472+0800 I REPL [signalProcessingThread] Stopping replication fetcher thread
2024-09-03T15:00:52.472+0800 I REPL [signalProcessingThread] Stopping replication applier thread
2024-09-03T15:00:52.491+0800 I NETWORK [conn20] end connection 127.0.0.1:42388 (6 connections now open)
2024-09-03T15:00:52.491+0800 I NETWORK [conn344] end connection 127.0.0.1:35824 (5 connections now open)
2024-09-03T15:00:52.617+0800 I REPL [signalProcessingThread] Stopping replication storage threads
2024-09-03T15:00:52.617+0800 I FTDC [signalProcessingThread] Shutting down full-time diagnostic data capture
2024-09-03T15:00:52.626+0800 I STORAGE [WTOplogJournalThread] oplog journal thread loop shutting down
2024-09-03T15:00:52.627+0800 I STORAGE [signalProcessingThread] WiredTigerKVEngine shutting down
2024-09-03T15:00:52.650+0800 I STORAGE [signalProcessingThread] shutdown: removing fs lock…
2024-09-03T15:00:52.650+0800 I CONTROL [signalProcessingThread] now exiting
2024-09-03T15:00:52.650+0800 I CONTROL [signalProcessingThread] shutting down with code:0
Could anyone give any suggestion on how to solve this problem? All the files are still there, it seems to be just the timestamp mismatch. I have not backed up my database (will definitely do that on a regular basis in the future). Would a CryoSPARC update be able to fix the problem?
Thank you very much!