Cannot restart cryosparc, database error

Hi everyone,

I seems to have run into cryosparc database issue on my single workstation cryoSPARC. The app is very much alive, jobs can still be sent and run to completion, but when I switch to Tree viewing, the page starts to freeze after 5 seconds. Then I try seeing if my instance is still alive with ‘cryosparcm status’. The first time it returns this

CryoSPARC System master node installed at
/home/cryosparcuser/cryosparc/cryosparc_master
Current cryoSPARC version: v4.6.0
----------------------------------------------------------------------------

CryoSPARC is not running.

----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID="XXXXXXXXXXX"
export CRYOSPARC_MASTER_HOSTNAME="10.xxxxxxxx"
export CRYOSPARC_FORCE_HOSTNAME=true
export CRYOSPARC_DB_PATH="/home/cryosparcuser/cryosparc/cryosparc_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DB_CONNECTION_TIMEOUT_MS=20000
export CRYOSPARC_INSECURE=false
export CRYOSPARC_DB_ENABLE_AUTH=true
export CRYOSPARC_CLUSTER_JOB_MONITOR_INTERVAL=10
export CRYOSPARC_CLUSTER_JOB_MONITOR_MAX_RETRIES=1000000
export CRYOSPARC_PROJECT_DIR_PREFIX='CS-'
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_CLICK_WRAP=true

Which is weird because all the usual “cryosparc process status” are not showing. I tried “cryosparcm restart” then this pops out

CryoSPARC is not already running.
If you would like to restart, use cryosparcm restart
Starting CryoSPARC System master process...
CryoSPARC is not already running.
configuring database...
Warning: Could not get database status (attempt 1/3)
Warning: Could not get database status (attempt 2/3)
Warning: Could not get database status (attempt 3/3)
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_compute/database_manageme                                                   nt.py", line 47, in configure_mongo
    initialize_replica_set()
  File "/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_compute/database_manageme                                                   nt.py", line 84, in initialize_replica_set
    admin_db = try_get_pymongo_db(mongo_client)
  File "/home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_compute/database_manageme                                                   nt.py", line 251, in try_get_pymongo_db
    admin_db.command(({'serverStatus': 1}))
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/_csot.py", line 108, in csot_wrapper
    return func(self, *args, **kwargs)
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/database.py", line 897, in command
    return self._command(
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/database.py", line 737, in _command
    return conn.command(
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/helpers.py", line 342, in inner
    return func(*args, **kwargs)
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/pool.py", line 989, in command
    return command(
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/network.py", line 212, in command
    helpers._check_command_response(
  File "/home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master                                                   _env/lib/python3.10/site-packages/pymongo/helpers.py", line 248, in _check_command_respons                                                   e
    raise OperationFailure(errmsg, code, response, max_wire_version)
pymongo.errors.OperationFailure: there are no users authenticated, full error: {'operation                                                   Time': Timestamp(1733972676, 1), 'ok': 0.0, 'errmsg': 'there are no users authenticated',                                                    'code': 13, 'codeName': 'Unauthorized', '$clusterTime': {'clusterTime': Timestamp(17339726                                                   76, 1), 'signature': {'hash': b'E\xe2\xad#\x02\x9ej\xdb\\\x17"\x9a\xb8\x95\x816\x1f\x18\x1                                                   2\xed', 'keyId': 7401057373537697799}}}
[2024-12-12T11:04:41+08:00] Error configuring database. Most recent database log lines:
2024-12-12T11:04:28.282+0800 I STORAGE  [initandlisten] exception in initAndListen: DBPath                                                   InUse: Unable to lock the lock file: /home/cryosparcuser/cryosparc/cryosparc_database/mong                                                   od.lock (Resource temporarily unavailable). Another mongod instance is already running on                                                    the /home/cryosparcuser/cryosparc/cryosparc_database directory, terminating
2024-12-12T11:04:28.282+0800 I CONTROL  [initandlisten] now exiting
2024-12-12T11:04:28.282+0800 I CONTROL  [initandlisten] shutting down with code:100
2024-12-12T11:04:31.253+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:3672                                                   0 #682 (135 connections now open)
2024-12-12T11:04:31.253+0800 I NETWORK  [conn682] received client metadata from 127.0.0.1:                                                   36720 conn682: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name:                                                    "Linux", architecture: "x86_64", version: "5.14.0-362.13.1.el9_3.x86_64" }, platform: "CP                                                   ython 3.10.14.final.0" }
2024-12-12T11:04:31.254+0800 I NETWORK  [listener] connection accepted from 127.0.0.1:3673                                                   0 #683 (136 connections now open)
2024-12-12T11:04:31.254+0800 I NETWORK  [conn683] received client metadata from 127.0.0.1:                                                   36730 conn683: { driver: { name: "PyMongo", version: "4.8.0" }, os: { type: "Linux", name:                                                    "Linux", architecture: "x86_64", version: "5.14.0-362.13.1.el9_3.x86_64" }, platform: "CP                                                   ython 3.10.14.final.0" }
2024-12-12T11:04:41.269+0800 I ACCESS   [conn683] Unauthorized: not authorized on admin to                                                    execute command { endSessions: [ { id: UUID("3e3fcfbd-dcd9-4366-a78f-90291ef748dc") } ],                                                    $clusterTime: { clusterTime: Timestamp(1733972676, 1), signature: { hash: BinData(0, 45E2A                                                   D23029E6ADB5C17229AB89581361F1812ED), keyId: 7401057373537697799 } }, $db: "admin", $readP                                                   reference: { mode: "primaryPreferred" } }
2024-12-12T11:04:41.270+0800 I NETWORK  [conn683] end connection 127.0.0.1:36730 (135 conn                                                   ections now open)
2024-12-12T11:04:41.270+0800 I NETWORK  [conn682] end connection 127.0.0.1:36720 (134 conn                                                   ections now open)

I’m not sure what is currently happening. Can someone help me?
Thanks and kind regards,
Khoa.

when looking into /tmp/ there are two following .sock files

[cryosparcuser@XXX tmp]$ ls
cryosparc-supervisor-2c23a496597238f50b722d927bce9930.sock
mongodb-39001.sock

Should i try removing them and restart?

Your CryoSPARC instance seems to be in an inconsistent state. You may want to

  1. collect some information that may help you understand how the situation arose. Collect the outputs of these commands:
    ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
    ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_ -e mongo
    
  2. perform a thorough and confirmed shutdown of CryoSPARC

Please do not remove the *.sock files before you have confirmed with a suitably parameterized ps command that all processes related to those files have in fact exited.

The command returns this:

[cryosparcuser@AÂ ~]$ ls -l /tmp/cryosparc*.sock /tmp/mongo*.sock
srwx------ 1 cryosparcuser cryosparcuser 0 Dec 12 17:05 /tmp/cryosparc-supervisor-2c23a496597238f50b722d927bce9930.sock
srwx------ 1 cryosparcuser cryosparcuser 0 Nov 22 12:49 /tmp/mongodb-39001.sock

This command returns this:

[cryosparcuser@AGG ~]$ ps -eo user:12,pid,ppid,start,command | grep -e cryosparc_ -e mongo
cryosparcus+  707817    3498   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryosparcuser/cryosparc/cryosparc_master/supervisord.conf
cryosparcus+  707928  707817   Nov 22 mongod --auth --dbpath /home/cryosparcuser/cryosparc/cryosparc_database --port 39001 --oplogSize 64 --replSet meteor --wiredTigerCacheSizeGB 4 --bind_ip_all
cryosparcus+  708039  707817   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708040  708039   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn -n command_core -b 0.0.0.0:39002 cryosparc_command.command_core:start() -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708072  707817   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708086  708072   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_vis:app -n command_vis -b 0.0.0.0:39003 -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708119  707817   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708120  708119   Nov 22 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/gunicorn cryosparc_command.command_rtp:start() -n command_rtp -b 0.0.0.0:39005 -c /home/cryosparcuser/cryosparc/cryosparc_master/gunicorn.conf.py
cryosparcus+  708154  707817   Nov 22 /home/cryosparcuser/cryosparc/cryosparc_master/cryosparc_app/nodejs/bin/node ./bundle/main.js
cryosparcus+ 1810198       1 17:05:23 python /home/cryosparcuser/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryosparcuser/cryosparc/cryosparc_master/supervisord.conf
cryosparcus+ 1935255 1935055 10:07:11 grep --color=auto -e cryosparc_ -e mongo

After following the Complete shutdown protocol, I was able to restart cryoSPARC. Thanks a lot!
However, this is happening more than I expect. Any possible ways to diagnose the cause of this?

Thank you for your assitance!

The gap between the *.sock files’ timestamps is unexpected:

A possible explanation could be the “incorrect” (not via cryosparcm restart) removal of the
cryosparc-supervisor-2c23a496597238f50b722d927bce9930.sock. The removal could occur automatically due to some incorrectly configured process that is supposed to cleanup unneeded temporary files, possibly in the context of a cron job or system service, such as systemd-tmpfiles.

1 Like