pymongo.errors.ServerSelectionTimeoutError: localhost:XXXXX: [Errno 111] Connection refused

Kellie · March 16, 2019, 12:08am

I suddenly encountered this error with starting cryosparc, could someone help me?

Starting cryoSPARC System master process..
CryoSPARC is not already running.
database: started
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 1149, in database_names
    "listDatabases")["databases"]]
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/database.py", line 491, in command
    with client._socket_for_reads(read_preference) as (sock_info, slave_ok):
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 859, in _socket_for_reads
    with self._get_socket(read_preference) as sock_info:
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/contextlib.py", line 17, in __enter__
    return self.gen.next()
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/mongo_client.py", line 823, in _get_socket
    server = self._get_topology().select_server(selector)
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 214, in select_server
    address))
  File "/data/CRYOSPARC/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/topology.py", line 189, in select_servers
    self._error_message(selector))
pymongo.errors.ServerSelectionTimeoutError: localhost:38001: [Errno 111] Connection refused

stephan · March 18, 2019, 4:26pm

Hi @Kellie,

Did you try restarting again? Also, was cryoSPARC running before this incident?

Kellie · March 18, 2019, 4:45pm

Hi @stephan,
Yes it was running, I was ssh’ing in from other location so there is a chance there was a mishap separate from me.
I tried all that I could think of to correct the error (stopping, checking tmp files/processes, etc…), I eventually gave up and reinstalled using a different port.

Thanks!
Kellie

donghuachen · March 23, 2022, 2:32am

@stephan,

Sometimes I can successfully run “cryosparcm restart” and access localhost:39002 from Chrome browser, but a couple or few minutes later I could not access localhost:39002 anymore, and I had the following similar error when I run “cryosparcm restart”, any suggestions? Thanks! I am running cryoSPARC v3.3.1+220315.

pymongo.errors.ServerSelectionTimeoutError: localhost:39003: [Errno 111] Connection refused
Timeout: 20.0s
Topology Description: <TopologyDescription id: 6238e81ec57b834bd520e413
topology_type: Single
servers: [<ServerDescription ('localhost', 39003)
server_type: Unknown
rtt: None
error=AutoReconnect('localhost:39003: [Errno 111] Connection refused')>]>```

wtempel · March 23, 2022, 9:47pm

@donghuachen If this cryoSPARC instance is explicitly configured with GUI access at port 39002
and
there are no other applications, like an additional cryoSPARC instance, using ports in the range 39000…39012
you may try this sequence (as the Linux user who “owns” the cryoSPARC instance)

cryosparcm stop
ps x | grep mongo
- if there is no mongod process, proceed to next step
- else
  - note the mongod PID (<mongo-pid> hereafter)
  - kill -2 <mongo-pid>
  - confirm there is no more mongod process: ps x | grep mongo
rm /tmp/mongodb-*.sock
Does cryosparcm start now complete without error, is the GUI accessible?

donghuachen · March 23, 2022, 10:23pm

Yes. After cryosparcm start, I could access GUI but only for a couple of minutes. I checked its status showing “database RUNNING”. After a couple of minutes, I could not access GUI anymore and the status was “database EXITED” (also posted here https://discuss.cryosparc.com/t/disconnecting-of-the-database/4613/4).

wtempel · March 24, 2022, 1:44pm

@donghuachen Please can you show us the output of:
cryosparcm log database

donghuachen · March 24, 2022, 3:55pm

Here is the partial output from cryosparcm log database (I have changed the port 39002 to 39200 recently):

2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] MongoDB starting : pid=12287 port=39201 dbpath=/oak/stanford/groups/kornberg/donghuac/cryosparc2_database 64-bit host=sh02-15n13.int
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] db version v3.4.10
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] allocator: tcmalloc
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] modules: none
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] build environment:
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten]     distarch: x86_64
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten]     target_arch: x86_64
2022-03-23T10:40:39.991-0700 I CONTROL  [initandlisten] options: { net: { port: 39201 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/oak/stanford/groups/kornberg/donghuac/cryosparc2_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2022-03-23T10:40:39.992-0700 W -        [initandlisten] Detected unclean shutdown - /oak/stanford/groups/kornberg/donghuac/cryosparc2_database/mongod.lock is not empty.
2022-03-23T10:40:40.010-0700 I -        [initandlisten] Detected data files in /oak/stanford/groups/kornberg/donghuac/cryosparc2_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2022-03-23T10:40:40.010-0700 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2022-03-23T10:40:40.010-0700 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2022-03-23T10:40:40.557-0700 I STORAGE  [initandlisten] Starting WiredTigerRecordStoreThread local.oplog.rs
2022-03-23T10:40:40.557-0700 I STORAGE  [initandlisten] The size storer reports that the oplog contains 68300 records totaling to 68192707 bytes
2022-03-23T10:40:40.557-0700 I STORAGE  [initandlisten] Sampling from the oplog between Feb 25 17:25:31:b and Mar 23 10:18:20:4 to determine where to place markers for truncation
2022-03-23T10:40:40.557-0700 I STORAGE  [initandlisten] Taking 101 samples and assuming that each section of oplog contains approximately 6722 records totaling to 6711440 bytes
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 25 19:53:55:1
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 25 23:26:00:1
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 26 08:23:21:1
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 26 09:41:56:7
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 26 10:05:50:6
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Feb 27 16:22:25:1
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Mar  1 12:28:18:1
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Mar 12 14:32:53:666
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Mar 12 16:22:34:e69
2022-03-23T10:40:40.565-0700 I STORAGE  [initandlisten] Placing a marker at optime Mar 22 10:38:22:1
2022-03-23T10:40:40.851-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.851-0700 I CONTROL  [initandlisten] ** WARNING: Access control is not enabled for the database.
2022-03-23T10:40:40.851-0700 I CONTROL  [initandlisten] **          Read and write access to data and configuration is unrestricted.
2022-03-23T10:40:40.851-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] ** WARNING: You are running on a NUMA machine.
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] **          We suggest launching mongod like this to avoid performance problems:
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] **              numactl --interleave=all mongod [other options]
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] ** WARNING: soft rlimits too low. rlimits set to 10240 processes, 131072 files. Number of processes should be at least 65536 : 0.5 times number of files.
2022-03-23T10:40:40.852-0700 I CONTROL  [initandlisten] 
2022-03-23T10:40:40.870-0700 I FTDC     [initandlisten] Initializing full-time diagnostic data capture with directory '/oak/stanford/groups/kornberg/donghuac/cryosparc2_database/diagnostic.data'
2022-03-23T10:40:40.871-0700 I NETWORK  [thread1] waiting for connections on port 39201
2022-03-23T10:40:40.871-0700 I REPL     [replExecDBWorker-0] New replica set config in use: { _id: "meteor", version: 40670, protocolVersion: 1, members: [ { _id: 0, host: "localhost:39201", arbiterOnly: false, buildIndexes: true, hidden: false, priority: 1.0, tags: {}, slaveDelay: 0, votes: 1 } ], settings: { chainingAllowed: true, heartbeatIntervalMillis: 2000, heartbeatTimeoutSecs: 10, electionTimeoutMillis: 10000, catchUpTimeoutMillis: 60000, getLastErrorModes: {}, getLastErrorDefaults: { w: 1, wtimeout: 0 }, replicaSetId: ObjectId('5e48e184a4a76b872de0136b') } }
2022-03-23T10:40:40.871-0700 I REPL     [replExecDBWorker-0] This node is localhost:39201 in the config
2022-03-23T10:40:40.871-0700 I REPL     [replExecDBWorker-0] transition to STARTUP2
2022-03-23T10:40:40.871-0700 I REPL     [replExecDBWorker-0] Starting replication storage threads
2022-03-23T10:40:40.872-0700 I REPL     [replExecDBWorker-0] Starting replication fetcher thread
2022-03-23T10:40:40.872-0700 I REPL     [replExecDBWorker-0] Starting replication applier thread
2022-03-23T10:40:40.872-0700 I REPL     [replExecDBWorker-0] Starting replication reporter thread
2022-03-23T10:40:40.872-0700 I REPL     [rsSync] transition to RECOVERING
2022-03-23T10:40:40.873-0700 I REPL     [rsSync] transition to SECONDARY
2022-03-23T10:40:40.873-0700 I REPL     [rsSync] conducting a dry run election to see if we could be elected
2022-03-23T10:40:40.873-0700 I REPL     [ReplicationExecutor] dry election run succeeded, running for election
2022-03-23T10:40:41.378-0700 I REPL     [ReplicationExecutor] election succeeded, assuming primary role in term 360
2022-03-23T10:40:41.378-0700 I REPL     [ReplicationExecutor] transition to PRIMARY
2022-03-23T10:40:41.378-0700 I REPL     [ReplicationExecutor] Entering primary catch-up mode.
2022-03-23T10:40:41.378-0700 I REPL     [ReplicationExecutor] Exited primary catch-up mode.
2022-03-23T10:40:42.003-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:47308 #1 (1 connection now open)
2022-03-23T10:40:42.004-0700 I NETWORK  [conn1] received client metadata from 127.0.0.1:47308 conn1: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:40:42.005-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:47310 #2 (2 connections now open)
2022-03-23T10:40:42.005-0700 I NETWORK  [conn2] received client metadata from 127.0.0.1:47310 conn2: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:40:42.512-0700 I -        [conn1] end connection 127.0.0.1:47308 (2 connections now open)
2022-03-23T10:40:42.512-0700 I -        [conn2] end connection 127.0.0.1:47310 (2 connections now open)
2022-03-23T10:40:42.578-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:47312 #3 (1 connection now open)
2022-03-23T10:40:42.578-0700 I NETWORK  [conn3] received client metadata from 127.0.0.1:47312 conn3: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "CentOS Linux release 7.9.2009 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.59.1.el7.x86_64" } }
2022-03-23T10:40:42.579-0700 I REPL     [conn3] replSetInitiate admin command received from client
2022-03-23T10:40:42.581-0700 I -        [conn3] end connection 127.0.0.1:47312 (1 connection now open)
2022-03-23T10:40:42.873-0700 I REPL     [rsSync] transition to primary complete; database writes are now permitted
2022-03-23T10:40:43.652-0700 I NETWORK  [thread1] connection accepted from 127.0.0.1:47314 #4 (1 connection now open)
2022-03-23T10:40:43.653-0700 I NETWORK  [conn4] received client metadata from 127.0.0.1:47314 conn4: { application: { name: "MongoDB Shell" }, driver: { name: "MongoDB Internal Client", version: "3.4.10" }, os: { type: "Linux", name: "CentOS Linux release 7.9.2009 (Core)", architecture: "x86_64", version: "Kernel 3.10.0-1160.59.1.el7.x86_64" } }
2022-03-23T10:40:43.654-0700 I -        [conn4] end connection 127.0.0.1:47314 (1 connection now open)
2022-03-23T10:40:47.914-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37784 #5 (1 connection now open)
2022-03-23T10:40:47.914-0700 I NETWORK  [conn5] received client metadata from 10.18.15.13:37784 conn5: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:40:47.915-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37786 #6 (2 connections now open)
2022-03-23T10:40:47.915-0700 I NETWORK  [conn6] received client metadata from 10.18.15.13:37786 conn6: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:40:48.179-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37788 #7 (3 connections now open)
2022-03-23T10:40:48.179-0700 I NETWORK  [conn7] received client metadata from 10.18.15.13:37788 conn7: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:40:54.221-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37806 #8 (4 connections now open)
2022-03-23T10:40:54.221-0700 I NETWORK  [conn8] received client metadata from 10.18.15.13:37806 conn8: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:41:03.080-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37850 #9 (5 connections now open)
2022-03-23T10:41:03.080-0700 I NETWORK  [conn9] received client metadata from 10.18.15.13:37850 conn9: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:41:05.139-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37852 #10 (6 connections now open)
2022-03-23T10:41:05.140-0700 I NETWORK  [conn10] received client metadata from 10.18.15.13:37852 conn10: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:41:05.141-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37854 #11 (7 connections now open)
2022-03-23T10:41:05.141-0700 I NETWORK  [conn11] received client metadata from 10.18.15.13:37854 conn11: { driver: { name: "PyMongo", version: "3.11.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "CPython 3.7.8.final.0" }
2022-03-23T10:41:07.604-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37856 #12 (8 connections now open)
2022-03-23T10:41:07.609-0700 I NETWORK  [conn12] received client metadata from 10.18.15.13:37856 conn12: { driver: { name: "nodejs", version: "2.2.34" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE, mongodb-core: 2.1.18" }
2022-03-23T10:41:07.685-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37858 #13 (9 connections now open)
2022-03-23T10:41:07.685-0700 I NETWORK  [conn13] received client metadata from 10.18.15.13:37858 conn13: { driver: { name: "nodejs", version: "2.2.34" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE, mongodb-core: 2.1.18" }
2022-03-23T10:41:07.686-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37860 #14 (10 connections now open)
2022-03-23T10:41:07.687-0700 I NETWORK  [conn14] received client metadata from 10.18.15.13:37860 conn14: { driver: { name: "nodejs", version: "2.2.34" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE, mongodb-core: 2.1.18" }
2022-03-23T10:41:09.498-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37862 #15 (11 connections now open)
2022-03-23T10:41:09.501-0700 I NETWORK  [conn15] received client metadata from 10.18.15.13:37862 conn15: { driver: { name: "nodejs-core", version: "3.2.7" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v14.15.1, LE" }
2022-03-23T10:41:09.517-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37864 #16 (12 connections now open)
2022-03-23T10:41:09.517-0700 I NETWORK  [conn16] received client metadata from 10.18.15.13:37864 conn16: { driver: { name: "nodejs-core", version: "3.2.7" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v14.15.1, LE" }
2022-03-23T10:41:09.519-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37866 #17 (13 connections now open)
2022-03-23T10:41:09.519-0700 I NETWORK  [conn17] received client metadata from 10.18.15.13:37866 conn17: { driver: { name: "nodejs-core", version: "3.2.7" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v14.15.1, LE" }
2022-03-23T10:41:11.033-0700 W FTDC     [ftdc] Uncaught exception in 'FileNotOpen: Failed to open interim file /oak/stanford/groups/kornberg/donghuac/cryosparc2_database/diagnostic.data/metrics.interim.temp' in full-time diagnostic data capture subsystem. Shutting down the full-time diagnostic data capture subsystem.
2022-03-23T10:41:12.894-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37868 #18 (14 connections now open)
2022-03-23T10:41:12.906-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37870 #19 (15 connections now open)
2022-03-23T10:41:12.911-0700 I NETWORK  [conn19] received client metadata from 10.18.15.13:37870 conn19: { driver: { name: "nodejs", version: "3.3.5" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE" }
2022-03-23T10:41:16.667-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37876 #20 (16 connections now open)
2022-03-23T10:41:16.673-0700 I NETWORK  [conn20] received client metadata from 10.18.15.13:37876 conn20: { driver: { name: "nodejs", version: "3.6.2" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "'Node.js v14.15.1, LE (unified)" }
2022-03-23T10:41:16.675-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37878 #21 (17 connections now open)
2022-03-23T10:41:16.675-0700 I NETWORK  [conn21] received client metadata from 10.18.15.13:37878 conn21: { driver: { name: "nodejs-core", version: "3.2.7" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v14.15.1, LE" }
2022-03-23T10:41:28.705-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37884 #22 (18 connections now open)
2022-03-23T10:41:28.706-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37886 #23 (19 connections now open)
2022-03-23T10:41:28.708-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37888 #24 (20 connections now open)
2022-03-23T10:41:28.838-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37890 #25 (21 connections now open)
2022-03-23T10:41:28.838-0700 I NETWORK  [conn25] received client metadata from 10.18.15.13:37890 conn25: { driver: { name: "nodejs", version: "3.3.5" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE" }
2022-03-23T10:41:28.857-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37892 #26 (22 connections now open)
2022-03-23T10:41:28.858-0700 I NETWORK  [conn26] received client metadata from 10.18.15.13:37892 conn26: { driver: { name: "nodejs", version: "3.3.5" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE" }
2022-03-23T10:41:28.859-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37894 #27 (23 connections now open)
2022-03-23T10:41:28.859-0700 I NETWORK  [conn27] received client metadata from 10.18.15.13:37894 conn27: { driver: { name: "nodejs", version: "3.3.5" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE" }
2022-03-23T10:41:28.860-0700 I NETWORK  [thread1] connection accepted from 10.18.15.13:37896 #28 (24 connections now open)
2022-03-23T10:41:28.861-0700 I NETWORK  [conn28] received client metadata from 10.18.15.13:37896 conn28: { driver: { name: "nodejs", version: "3.3.5" }, os: { type: "Linux", name: "linux", architecture: "x64", version: "3.10.0-1160.59.1.el7.x86_64" }, platform: "Node.js v8.9.4, LE" }
2022-03-23T10:41:28.982-0700 I COMMAND  [conn27] command meteor.fs.chunks command: find { find: "fs.chunks", filter: { files_id: ObjectId('5ea558e4f50da502542e6048') }, sort: { n: 1 }, returnKey: false, showRecordId: false } planSummary: IXSCAN { files_id: 1, n: 1 } keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:119249 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 118ms
2022-03-23T10:41:28.982-0700 I COMMAND  [conn28] command meteor.fs.chunks command: find { find: "fs.chunks", filter: { files_id: ObjectId('5e98c9ebf50da57a89466325') }, sort: { n: 1 }, returnKey: false, showRecordId: false } planSummary: IXSCAN { files_id: 1, n: 1 } keysExamined:1 docsExamined:1 cursorExhausted:1 numYields:1 nreturned:1 reslen:153995 locks:{ Global: { acquireCount: { r: 4 } }, Database: { acquireCount: { r: 2 } }, Collection: { acquireCount: { r: 2 } } } protocol:op_query 119ms
2022-03-23T10:42:00.747-0700 E STORAGE  [thread2] WiredTiger error (5) [1648057320:747508][12287:0x7fd05f1b0700], file:WiredTiger.wt, WT_SESSION.checkpoint: /oak/stanford/groups/kornberg/donghuac/cryosparc2_database/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1029 bytes at offset 0: Input/output error
2022-03-23T10:42:00.749-0700 E STORAGE  [thread2] WiredTiger error (0) [1648057320:749037][12287:0x7fd05f1b0700], file:WiredTiger.wt, WT_SESSION.checkpoint: WiredTiger.turtle: encountered an illegal file format or internal value
2022-03-23T10:42:00.749-0700 E STORAGE  [thread2] WiredTiger error (-31804) [1648057320:749065][12287:0x7fd05f1b0700], file:WiredTiger.wt, WT_SESSION.checkpoint: the process must exit and restart: WT_PANIC: WiredTiger library panic
2022-03-23T10:42:00.749-0700 I -        [thread2] Fatal Assertion 28558 at src/mongo/db/storage/wiredtiger/wiredtiger_util.cpp 361
2022-03-23T10:42:00.749-0700 I -        [thread2] 

***aborting after fassert() failure


2022-03-23T10:42:00.759-0700 F -        [thread2] Got signal: 6 (Aborted).

 0x555b61049ac1 0x555b61048cd9 0x555b610491bd 0x7fd063019630 0x7fd062c72387 0x7fd062c73a78 0x555b6031de97 0x555b60d80b66 0x555b60327b46 0x555b60327d62 0x555b60327fc4 0x555b619efb90 0x555b619edc66 0x555b619ec5ef 0x555b619ecd2f 0x555b61a3ae54 0x555b61a3bbd9 0x555b61a3cf6e 0x555b61a3d229 0x555b61a29f81 0x555b619a175d 0x7fd063011ea5 0x7fd062d3ab0d
----- BEGIN BACKTRACE -----
{"backtrace":[{"b":"555B5FB16000","o":"1533AC1","s":"_ZN5mongo15printStackTraceERSo"},{"b":"555B5FB16000","o":"1532CD9"},{"b":"555B5FB16000","o":"15331BD"},{"b":"7FD06300A000","o":"F630"},{"b":"7FD062C3C000","o":"36387","s":"gsignal"},{"b":"7FD062C3C000","o":"37A78","s":"abort"},{"b":"555B5FB16000","o":"807E97","s":"_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj"},{"b":"555B5FB16000","o":"126AB66"},{"b":"555B5FB16000","o":"811B46","s":"__wt_eventv"},{"b":"555B5FB16000","o":"811D62","s":"__wt_err"},{"b":"555B5FB16000","o":"811FC4","s":"__wt_panic"},{"b":"555B5FB16000","o":"1ED9B90","s":"__wt_turtle_update"},{"b":"555B5FB16000","o":"1ED7C66","s":"__wt_metadata_update"},{"b":"555B5FB16000","o":"1ED65EF"},{"b":"555B5FB16000","o":"1ED6D2F","s":"__wt_meta_ckptlist_set"},{"b":"555B5FB16000","o":"1F24E54"},{"b":"555B5FB16000","o":"1F25BD9","s":"__wt_checkpoint"},{"b":"555B5FB16000","o":"1F26F6E"},{"b":"555B5FB16000","o":"1F27229","s":"__wt_txn_checkpoint"},{"b":"555B5FB16000","o":"1F13F81"},{"b":"555B5FB16000","o":"1E8B75D"},{"b":"7FD06300A000","o":"7EA5"},{"b":"7FD062C3C000","o":"FEB0D","s":"clone"}],"processInfo":{ "mongodbVersion" : "3.4.10", "gitVersion" : "078f28920cb24de0dd479b5ea6c66c644f6326e9", "compiledModules" : [], "uname" : { "sysname" : "Linux", "release" : "3.10.0-1160.59.1.el7.x86_64", "version" : "#1 SMP Wed Feb 23 16:47:03 UTC 2022", "machine" : "x86_64" }, "somap" : [ { "b" : "555B5FB16000", "elfType" : 3, "buildId" : "D9AB5C91FBC6F740604F4BC28348FE33EC87DEC2" }, { "b" : "7FD0649D3000", "elfType" : 3, "buildId" : "240A9FC7345A61CFA8DC368663D7E2FED99D364F" }, { "b" : "7FD063B4A000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libpython3.7m.so", "elfType" : 3 }, { "b" : "7FD06404E000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/libtiff.so", "elfType" : 3 }, { "b" : "7FD063942000", "path" : "/lib64/librt.so.1", "elfType" : 3, "buildId" : "3E44DF7055942478D052E40FDD1F5B7862B152B0" }, { "b" : "7FD06373E000", "path" : "/lib64/libdl.so.2", "elfType" : 3, "buildId" : "7F2E9CB0769D7E57BD669B485A74B537B63A57C4" }, { "b" : "7FD06343C000", "path" : "/lib64/libm.so.6", "elfType" : 3, "buildId" : "7615604EAF4A068DFAE5085444D15C0DEE93DFBD" }, { "b" : "7FD063226000", "path" : "/lib64/libgcc_s.so.1", "elfType" : 3, "buildId" : "EDF51350C7F71496149D064AA8B1441F786DF88A" }, { "b" : "7FD06300A000", "path" : "/lib64/libpthread.so.0", "elfType" : 3, "buildId" : "E10CC8F2B932FC3DAEDA22F8DAC5EBB969524E5B" }, { "b" : "7FD062C3C000", "path" : "/lib64/libc.so.6", "elfType" : 3, "buildId" : "8DA9707D518E88179B1B2903CD846C82102D84C1" }, { "b" : "7FD063EB4000", "path" : "/lib64/ld-linux-x86-64.so.2", "elfType" : 3, "buildId" : "62C449974331341BB08DCCE3859560A22AF1E172" }, { "b" : "7FD062A39000", "path" : "/lib64/libutil.so.1", "elfType" : 3, "buildId" : "FF2196BD22A8443054C83031E0E76EB01BA1219C" }, { "b" : "7FD063FB6000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/./libwebp.so.7", "elfType" : 3 }, { "b" : "7FD063EEA000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/./libzstd.so.1", "elfType" : 3 }, { "b" : "7FD062A10000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/./liblzma.so.5", "elfType" : 3 }, { "b" : "7FD0629D2000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/./libjpeg.so.9", "elfType" : 3 }, { "b" : "7FD0629B8000", "path" : "/home/groups/kornberg/donghuac/cryosparc/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/lib/./libz.so.1", "elfType" : 3 } ] }}
 mongod(_ZN5mongo15printStackTraceERSo+0x41) [0x555b61049ac1]
 mongod(+0x1532CD9) [0x555b61048cd9]
 mongod(+0x15331BD) [0x555b610491bd]
 libpthread.so.0(+0xF630) [0x7fd063019630]
 libc.so.6(gsignal+0x37) [0x7fd062c72387]
 libc.so.6(abort+0x148) [0x7fd062c73a78]
 mongod(_ZN5mongo32fassertFailedNoTraceWithLocationEiPKcj+0x0) [0x555b6031de97]
 mongod(+0x126AB66) [0x555b60d80b66]
 mongod(__wt_eventv+0x3D7) [0x555b60327b46]
 mongod(__wt_err+0x9D) [0x555b60327d62]
 mongod(__wt_panic+0x2E) [0x555b60327fc4]
 mongod(__wt_turtle_update+0x150) [0x555b619efb90]
 mongod(__wt_metadata_update+0xE6) [0x555b619edc66]
 mongod(+0x1ED65EF) [0x555b619ec5ef]
 mongod(__wt_meta_ckptlist_set+0x29F) [0x555b619ecd2f]
 mongod(+0x1F24E54) [0x555b61a3ae54]
 mongod(__wt_checkpoint+0xA9) [0x555b61a3bbd9]
 mongod(+0x1F26F6E) [0x555b61a3cf6e]
 mongod(__wt_txn_checkpoint+0xD9) [0x555b61a3d229]
 mongod(+0x1F13F81) [0x555b61a29f81]
 mongod(+0x1E8B75D) [0x555b619a175d]
 libpthread.so.0(+0x7EA5) [0x7fd063011ea5]
 libc.so.6(clone+0x6D) [0x7fd062d3ab0d]
-----  END BACKTRACE  -----

wtempel · March 29, 2022, 4:21pm

I have not seen a combination of errors and warnings quite like this, and am not sure how or if they may be related, for example:

Uncaught exception in 'FileNotOpen: Failed to open interim file
WARNING: soft rlimits too low. rlimits set to 10240 processes, 131072 files. Number of processes should be at least 65536 : 0.5 times number of files.
WiredTiger error (5) [1648057320:747508][12287:0x7fd05f1b0700], file:WiredTiger.wt, WT_SESSION.checkpoint: /oak/stanford/groups/kornberg/donghuac/cryosparc2_database/WiredTiger.turtle.set: handle-write: pwrite: failed to write 1029 bytes at offset 0: Input/output error

My suggestion at this point:

Ensure the directory /oak/stanford/groups/kornberg/donghuac/cryosparc2_database
- (and its content) is owned and writeable by Linux user that “owns” the cryoSPARC instance
- did not/does not become full or otherwise unavailable (network issues?)
Create two kinds of database backups (because each kind has its limitations)
- using cryosparcm backup
- using rsync (modify source and target directories according to your circumstances). This method requires that the database is not being modified during the backup (MongoDB Backup Methods — MongoDB Manual). Ensure cryoSPARC and the associated mongod process(es) are stopped.
Repair the database (see forum post, mongo docs)
Attempt cryosparcm start