MongoDB: "OperationFailure: node is not in primary or recovering state" when changing port

I need to create a new instance of cryosparc (on a different port) that uses a copy of my current db.
When I copy the database and start using a new port , I get the following erro

[JSONRPC ERROR  2021-04-27 14:31:44.724015  at  get_num_active_licenses ]
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 115, in wrapper
    res = func(*args, **kwargs)
  File "cryosparc2_command/command_core/__init__.py", line 1520, in get_num_active_licenses
    for j in jobs_running:
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 928, in __send_message
    helpers._check_command_response(doc['data'][0])
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
OperationFailure: node is not in primary or recovering state
-----------------------------------------------------
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 200, in background_worker
    concurrent_job_monitor()
  File "cryosparc2_command/command_core/__init__.py", line 1527, in concurrent_job_monitor
    current_concurrent_licenses_deque.append(get_num_active_licenses())
  File "cryosparc2_command/command_core/__init__.py", line 124, in wrapper
    raise e
OperationFailure: node is not in primary or recovering state
Traceback (most recent call last):
  File "cryosparc2_command/command_core/__init__.py", line 205, in background_worker
    heartbeat_manager()
  File "cryosparc2_command/command_core/__init__.py", line 1571, in heartbeat_manager
    active_jobs = get_active_licenses()
  File "cryosparc2_command/command_core/__init__.py", line 1536, in get_active_licenses
    for j in jobs_running:
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1114, in next
    if len(self.__data) or self._refresh():
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 1036, in _refresh
    self.__collation))
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/cursor.py", line 928, in __send_message
    helpers._check_command_response(doc['data'][0])
  File "/stornext/System/data/structbio/cryosparc/cryosparc-v2/cryosparc2_master/deps/anaconda/lib/python2.7/site-packages/pymongo/helpers.py", line 210, in _check_command_response
    raise OperationFailure(msg % errmsg, code, response)
OperationFailure: node is not in primary or recovering state
****** Concurrent job monitor failed ****
****** Instance heartbeat failed ****

Hi @jiskander,

The first thing you should do is keep a backup (copy) of your database at a different location.

The next thing is to re-configure the replica set:

  1. First, make sure there are no other cryoSPARC instances running on the system
  2. Navigate to the new cryoSPARC instance’s cryosparc_master folder
  3. Then, start the new instance’s database: ./bin/cryosparcm start database
  4. Then, open up a mongo shell and get information about the replica set by running the commands:
    ./bin/cryosparcm mongo
    rs.conf()
    
    You should see something like this:
    "members" : [
                {
                        "_id" : 0,
                        "host" : "localhost:63501",
                        "arbiterOnly" : false,
                        "buildIndexes" : true,
                        "hidden" : false,
                        .
                        .
                        .
    
    If that port number is the old port, run the following command:
    rs.reconfig({
        "_id": "meteor",
        "members": [
            {
                "_id": 0,
                "host": "localhost:<NEW_BASE_PORT_NUMBER+1>"
            }
        ]
        })
    
    
1 Like