Error configuring database

Install, Configure and Update
1

I updated cryosparc lately, and when it start , the error shows,

`===================================================
Successfully updated master to version v4.3.0.
===================================================

Starting cryoSPARC System master process..
CryoSPARC is not already running.
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 48, in configure_mongo
    initialize_replica_set()
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 87, in initialize_replica_set
    admin_db = try_get_pymongo_db(mongo_client)
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 250, in try_get_pymongo_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 828, in command
    return self._command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 703, in _command
    return sock_info.command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 740, in command
    return command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/network.py", line 177, in command
    helpers._check_command_response(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/helpers.py", line 180, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: there are no users authenticated, full error: {'operationTime': Timestamp(1692348340, 1), 'ok': 0.0, 'errmsg': 'there are no users authenticated', 'code': 13, 'codeName': 'Unauthorized', '$clusterTime': {'clusterTime': Timestamp(1692348340, 1), 'signature': {'hash': b'\xfaAab\x19\x8bh.\xd8\t@\x84_\xcc\xfa\x83@\xddO\x98', 'keyId': 7202630923844583426}}}
configuring database
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
[2023-08-18T16:45:45+0800] Error configuring database. Most recent database log lines:
2023-08-18T16:45:05.075+0800 I COMMAND  [LogicalSessionCacheReap] command config.$cmd command: delete { delete: "transactions", ordered: false, writeConcern: { w: "majority", wtimeout: 15000 }, $db: "config" } numYields:0 reslen:214 locks:{ Global: { acquireCount: { r: 2, w: 2 } }, Database: { acquireCount: { w: 2 } }, Collection: { acquireCount: { w: 1 } }, oplog: { acquireCount: { w: 1 } } } protocol:op_msg 431ms
2023-08-18T16:45:21.365+0800 I COMMAND  [conn12] command local.oplog.rs command: find { find: "oplog.rs", filter: { ns: /^(?:meteor\.|admin\.\$cmd)/, $or: [ { op: { $in: [ "i", "u", "d" ] } }, { op: "c", o.drop: { $exists: true } }, { op: "c", o.dropDatabase: 1 }, { op: "c", o.applyOps: { $exists: true } } ], ts: { $gt: Timestamp(1692348260, 3) } }, tailable: true, awaitData: true, lsid: { id: UUID("c158f549-d413-4403-88ed-eeb374703094") }, $clusterTime: { clusterTime: Timestamp(1692348304, 8), signature: { hash: BinData(0, 34B94189D691143641657F5DE5E32C312C7EF785), keyId: 7202630923844583426 } }, $db: "local" } planSummary: COLLSCAN cursorid:12425569266 keysExamined:0 docsExamined:140019 numYields:1104 nreturned:0 reslen:212 locks:{ Global: { acquireCount: { r: 2210 } }, Database: { acquireCount: { r: 1105 } }, oplog: { acquireCount: { r: 1105 } } } protocol:op_msg 878ms
2023-08-18T16:45:34.652+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:46666 #1640 (128 connections now open)
2023-08-18T16:45:34.665+0800 I NETWORK  [conn1640] received client metadata from 127.0.0.1:46666 conn1640: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-08-18T16:45:34.672+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:46668 #1641 (129 connections now open)
2023-08-18T16:45:34.691+0800 I NETWORK  [conn1641] received client metadata from 127.0.0.1:46668 conn1641: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-08-18T16:45:44.711+0800 I ACCESS   [conn1641] Unauthorized: not authorized on admin to execute command { endSessions: [ { id: UUID("c50fe6f9-06cf-4392-bae6-636c162ac0d3") } ], $clusterTime: { clusterTime: Timestamp(1692348340, 1), signature: { hash: BinData(0, FA416162198B682ED80940845FCCFA8340DD4F98), keyId: 7202630923844583426 } }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }
2023-08-18T16:45:44.711+0800 I NETWORK  [conn1640] end connection 127.0.0.1:46666 (128 connections now open)
2023-08-18T16:45:44.711+0800 I NETWORK  [conn1641] end connection 127.0.0.1:46668 (127 connections now open)
th code:100
/home/yuexin/cryosparc/cryosparc_master/bin/cryosparcm: line 2205: syntax error near unexpected token `fi'`

i checked the status , and it shows :

`(base) [yuexin@yxtaiyi ~]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/yuexin/cryosparc/cryosparc_master
Current cryoSPARC version: v4.3.0
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_api                          STOPPED   Not started
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     STOPPED   Not started
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         STOPPED   Not started

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="??????"
export CRYOSPARC_MASTER_HOSTNAME="localhost"
export CRYOSPARC_DB_PATH="/home/yuexin/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true`

what is going on and how should i fix this ? any instructions will be helpful

2

i de-update the version to the older one which is 4.2.1, and i could access the project and queue jobs, but when cryosparcm restart, the error is still there

`(base) [yuexin@yxtaiyi ~]$ cryosparcm restart
CryoSPARC is running.
Stopping cryoSPARC
Shut down
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 48, in configure_mongo
    initialize_replica_set()
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 87, in initialize_replica_set
    admin_db = try_get_pymongo_admin_db(mongo_client)
  File "/home/yuexin/cryosparc/cryosparc_master/cryosparc_compute/database_management.py", line 249, in try_get_pymongo_admin_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 828, in command
    return self._command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/database.py", line 703, in _command
    return sock_info.command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 740, in command
    return command(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/network.py", line 177, in command
    helpers._check_command_response(
  File "/home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/helpers.py", line 180, in _check_command_response
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: there are no users authenticated, full error: {'operationTime': Timestamp(1692356584, 21), 'ok': 0.0, 'errmsg': 'there are no users authenticated', 'code': 13, 'codeName': 'Unauthorized', '$clusterTime': {'clusterTime': Timestamp(1692356584, 21), 'signature': {'hash': b'9\xefG\x00 \xc8\xb2\x1e\xdf\xa6\x97\x04\x08\xe0\xb3\x9f\xd1\xec\x1d\xd2', 'keyId': 7202630923844583426}}}
[2023-08-18T19:03:05+0800] Error configuring database. Most recent database log lines:
2023-08-18T19:02:12.229+0800 I COMMAND  [conn12] command local.oplog.rs command: find { find: "oplog.rs", filter: { ns: /^(?:meteor\.|admin\.\$cmd)/, $or: [ { op: { $in: [ "i", "u", "d" ] } }, { op: "c", o.drop: { $exists: true } }, { op: "c", o.dropDatabase: 1 }, { op: "c", o.applyOps: { $exists: true } } ], ts: { $gt: Timestamp(1692356199, 4) } }, tailable: true, awaitData: true, lsid: { id: UUID("c158f549-d413-4403-88ed-eeb374703094") }, $clusterTime: { clusterTime: Timestamp(1692356531, 1), signature: { hash: BinData(0, 69C5B5D1EB175A23B6B1202EC544DA608F3F88BF), keyId: 7202630923844583426 } }, $db: "local" } planSummary: COLLSCAN cursorid:10198405158 keysExamined:0 docsExamined:142664 numYields:1123 nreturned:0 reslen:212 locks:{ Global: { acquireCount: { r: 2248 } }, Database: { acquireCount: { r: 1124 } }, oplog: { acquireCount: { r: 1124 } } } protocol:op_msg 943ms
2023-08-18T19:02:42.246+0800 I COMMAND  [conn12] command local.oplog.rs command: find { find: "oplog.rs", filter: { ns: /^(?:meteor\.|admin\.\$cmd)/, $or: [ { op: { $in: [ "i", "u", "d" ] } }, { op: "c", o.drop: { $exists: true } }, { op: "c", o.dropDatabase: 1 }, { op: "c", o.applyOps: { $exists: true } } ], ts: { $gt: Timestamp(1692356199, 4) } }, tailable: true, awaitData: true, lsid: { id: UUID("c158f549-d413-4403-88ed-eeb374703094") }, $clusterTime: { clusterTime: Timestamp(1692356561, 1), signature: { hash: BinData(0, 6BFAD134D7F70BC0D3F2C5F7B6BF84D92815F26E), keyId: 7202630923844583426 } }, $db: "local" } planSummary: COLLSCAN cursorid:11073736806 keysExamined:0 docsExamined:142667 numYields:1124 nreturned:0 reslen:212 locks:{ Global: { acquireCount: { r: 2250 } }, Database: { acquireCount: { r: 1125 } }, oplog: { acquireCount: { r: 1125 } } } protocol:op_msg 989ms
2023-08-18T19:02:54.912+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:53992 #1648 (128 connections now open)
2023-08-18T19:02:54.931+0800 I NETWORK  [conn1648] received client metadata from 127.0.0.1:53992 conn1648: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-08-18T19:02:54.933+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:53994 #1649 (129 connections now open)
2023-08-18T19:02:54.951+0800 I NETWORK  [conn1649] received client metadata from 127.0.0.1:53994 conn1649: { driver: { name: "PyMongo", version: "3.13.0" }, os: { type: "Linux", name: "Linux", architecture: "x86_64", version: "3.10.0-957.el7.x86_64" }, platform: "CPython 3.8.15.final.0" }
2023-08-18T19:03:04.965+0800 I ACCESS   [conn1649] Unauthorized: not authorized on admin to execute command { endSessions: [ { id: UUID("eabbf2dd-d1d9-4147-9c87-de1fde54d066") } ], $clusterTime: { clusterTime: Timestamp(1692356584, 21), signature: { hash: BinData(0, 39EF470020C8B21EDFA6970408E0B39FD1EC1DD2), keyId: 7202630923844583426 } }, $db: "admin", $readPreference: { mode: "primaryPreferred" } }
2023-08-18T19:03:04.965+0800 I NETWORK  [conn1648] end connection 127.0.0.1:53992 (128 connections now open)
2023-08-18T19:03:04.965+0800 I NETWORK  [conn1649] end connection 127.0.0.1:53994 (127 connections now open)
 code:100`

so is the cryosparcm status:

`(base) [yuexin@yxtaiyi ~]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/yuexin/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_api                          STOPPED   Not started
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     STOPPED   Not started
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         STOPPED   Not started

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="??????"
export CRYOSPARC_MASTER_HOSTNAME="?????"
export CRYOSPARC_DB_PATH="/home/yuexin/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true`
3

I have no idea what is going on, like : why in the version 4.3.0 i can not get access to the project and when in the version 4.2.1 i can somehow get access? and in the both versions the database configuration is error.

4

Some CryoSPARC-related processes may have escaped during
cryosparcm stop
or
cryosparcm restart
Please can you post the output of the commands

cryosparcm stop
ps -eouser,pid,ppid,cmd | grep -e cryosparc_ -e mongo -e supervisord
5

thanks for your reply, the output is

(base) [yuexin@yxtaiyi ~]$ ps -eouser,pid,ppid,cmd | grep -e cryosparc_ -e mongo -e supervisord
yuexin    35034      1 python /home/yuexin/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/yuexin/cryosparc/cryosparc_master/supervisord.conf
yuexin    35247  35034 mongod --auth --dbpath /home/yuexin/cryosparc/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --nojournal --wiredTigerCacheSizeGB 4 --bind_ip_all
yuexin    35352  35034 python -c import cryosparc_command.command_core as serv; serv.start(port=39002)
yuexin    35395  35034 python -c import cryosparc_command.command_vis as serv; serv.start(port=39003)
yuexin    35419  35034 python -c import cryosparc_command.command_rtp as serv; serv.start(port=39005)
yuexin    35514  35034 /home/yuexin/cryosparc/cryosparc_master/cryosparc_app/api/nodejs/bin/node ./bundle/main.js
yuexin   163919 158082 grep --color=auto -e cryosparc_ -e mongo -e supervisord
6

Thanks @MiaoXiaoPu1121. Some processes seem to have “survived” the CryoSPARC shutdown, and may have caused pymongo.errors.OperationFailure following the update to CryoSPARC v4.3.
May I suggest you

  1. run the command
    kill 35034
    to terminate the supervisord process. Do not use kill -9 with CryoSPARC processes.
  2. wait 10 seconds
  3. confirm, with the command
    ps -eouser,pid,ppid,cmd | grep -e cryosparc_ -e mongo -e supervisord
    that all CryoSPARC-related processes have been terminated
  4. try
    cryosparcm start
7

Thanks to your reply, i want to refresh the situation: after i de-update to 4.2.1, the error showed when i imported the eer file, said the master version and worker version does not match, and ask me to update.

i used

cryosparcm patch --force

to update a patch , and the eer can be imported with no error, and motion cor/CTF/2D class runs well.

i would try your valuable advice later , cause some data need to be processed with urgently.

Addition, will the database error affect the result or speed of processing ?

Thanks for your patient and advice ~

8

Please can you post the the current output of

cryosparcm status | grep -v LICENSE

I do not know. The fact that you could run jobs suggests that there is still some sort of database access.
It is possible that updated/downgraded CryoSPARC software interacts with a database process that started before the update/downgrade. CryoSPARC behavior is undefined/untested under these circumstances.

When did

  1. you upgrade to v4.3
  2. you downgrade to v4.2.1
  3. run cryosparcm patch --force

What is the output of the command

ps -eouser,pid,ppid,start,cmd | grep mongo
9

we are running import job now , and the out put of cyrosparc status is

(base) [yuexin@yxtaiyi ~]$ cryosparcm status | grep -v LICENSE
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/yuexin/cryosparc/cryosparc_master
Current cryoSPARC version: v4.2.1+230621
----------------------------------------------------------------------------

CryoSPARC process status:

app                              STOPPED   Not started
app_api                          STOPPED   Not started
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     STOPPED   Not started
command_rtp                      STOPPED   Not started
command_vis                      STOPPED   Not started
database                         STOPPED   Not started

----------------------------------------------------------------------------
License is valid
----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_MASTER_HOSTNAME="localhost"
export CRYOSPARC_DB_PATH="/home/yuexin/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

the output of ps is

(base) [yuexin@yxtaiyi ~]$ ps -eouser,pid,ppid,start,cmd | grep mongo
yuexin    35247  35034   Jul 11 mongod --auth --dbpath /home/yuexin/cryosparc/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --nojournal --wiredTigerCacheSizeGB 4 --bind_ip_all
yuexin   327870 324561 01:17:25 grep --color=auto mongo
10

Sorry to bother, will this be complicate to fix ? Will a re-install will be available and easier to get to the normal situation ?

11

I recommend:

  1. cryosparcm stop
  2. search for left-over CryoSPARC-related processes
    ps -eouser,pid,ppid,cmd | grep -e cryosparc_ -e mongo -e supervisord
  3. kill the supervisord process, if there is any.
    (Do not use kill -9.)
  4. wait 10 seconds.
  5. confirm there are no more CryoSPARC-related processes (repeat ps command above)
  6. cryosparcm update
  7. (only if needed: non-automatic cryosparcw update)

After these steps (tested in August 2023), the CryoSPARC instance should be at v4.3 and running.

12

Thank you, i will try later

13

Hi, i tryed your recommendation, and seems work well with no more Traceback. Thank you.