Cryosparc 4.0.2 not restarting (mongodb permission)

Tru · June 9, 2023, 7:34am

Hello

I have been called to rescue a no-longer starting cryosparc setup (single machine, CentOS-7),
with a possible hostname/IP address confusion,
I would like to cleanly fix the current setup before upgrading to 4.2.1

starting clean (no process running)

[cryoem@ld20-1013 tmp]$ ps -ef|grep cryoem
cryoem    21944  21943  0 Jun08 pts/2    00:00:00 -bash
cryoem    34409  21944  0 09:03 pts/2    00:00:00 ps -ef
cryoem    34410  21944  0 09:03 pts/2    00:00:00 grep --color=auto cryoem

[cryoem@ld20-1013 tmp]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryoem/Applications/cryoSPARC2/cryosparc2_master
Current cryoSPARC version: v4.0.2
----------------------------------------------------------------------------

CryoSPARC is not running.

----------------------------------------------------------------------------

global config variables:
export CRYOSPARC_LICENSE_ID=".."
export CRYOSPARC_MASTER_HOSTNAME="localhost.localdomain"
export CRYOSPARC_FORCE_HOSTNAME=true
export CRYOSPARC_DB_PATH="/home/cryoem/Applications/cryoSPARC2/cryosparc2_database"
export CRYOSPARC_BASE_PORT=39000
export CRYOSPARC_DEVELOP=false
export CRYOSPARC_INSECURE=false
export CRYOSPARC_CLICK_WRAP=true

nothing in /tmp:

[cryoem@ld20-1013 tmp]$ ls -lart
<...>
drwxrwxrwt.  2 root   root    4096 May 14  2020 .font-unix
drwxrwxrwt.  2 root   root    4096 May 14  2020 .XIM-unix
drwxrwxrwt.  2 root   root    4096 May 14  2020 .Test-unix
drwx------.  2 cryoem cryoem  4096 Dec 14 09:57 pymp-xjZ1Cn
drwx------.  2 cryoem cryoem  4096 Jun  8 13:45 tracker-extract-files.3599
drwx------.  2 cryoem cryoem  4096 Jun  8 15:06 .esd-3599
-rw-r--r--.  1 root   root      73 Jun  8 15:09 lua_xJV5tr
drwxr-xr-x.  2 root   root    4096 Jun  8 15:09 hsperfdata_root
drwxrwxrwt. 16 root   root   28672 Jun  9 09:02 .

Hostname and IP not matching (long story with our IT service), so I forced localhost.localdomain in the config file…

[cryoem@ld20-1013 tmp]$ hostname
ld20-1013.bim.pasteur.fr

[cryoem@ld20-1013 tmp]$ ip -4 a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
3: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
    inet 10.20.38.12/24 brd 10.20.38.255 scope global noprefixroute dynamic eth0
       valid_lft 63438sec preferred_lft 63438sec

[cryoem@ld20-1013 tmp]$ host localhost.localdomain
localhost.localdomain has address 127.0.0.1
localhost.localdomain has IPv6 address ::1
[cryoem@ld20-1013 tmp]$ cryosparcm start
Starting cryoSPARC System master process..
CryoSPARC is not already running.
configuring database
            configuration complete
database: started
exception: connect failed
ERROR: Could not check database configuration - is the database running?
Start the database with the following command:

    /home/cryoem/Applications/cryoSPARC2/cryosparc2_master/bin/cryosparcm restart

Then re-run checkdb
[cryoem@ld20-1013 tmp]$ cryosparcm checkdb
    database configuration is OK.

[cryoem@ld20-1013 tmp]$ ps -ef|grep ^cryoem
cryoem    21944  21943  0 Jun08 pts/2    00:00:00 -bash
cryoem    35434      1  0 09:10 ?        00:00:00 python /home/cryoem/Applications/cryoSPARC2/cryosparc2_master/deps/anaconda/envs/cryosparc_master_env/bin/supervisord -c /home/cryoem/Applications/cryoSPARC2/cryosparc2_master/supervisord.conf
cryoem    35548  35434  8 09:10 ?        00:00:16 mongod --auth --dbpath /home/cryoem/Applications/cryoSPARC2/cryosparc2_database --port 39001 --oplogSize 64 --replSet meteor --nojournal --wiredTigerCacheSizeGB 4 --bind_ip_all
cryoem    35789  21944  0 09:13 pts/2    00:00:00 ps -ef
cryoem    35790  21944  0 09:13 pts/2    00:00:00 grep --color=auto ^cryoem

mongodb status

[cryoem@ld20-1013 tmp]$ eval $(cryosparcm env)
[cryoem@ld20-1013 tmp]$ mongo_uri=$(python -c "from cryosparc_compute import database_management; print(database_management.get_mongo_uri('admin'))")
[cryoem@ld20-1013 tmp]$ mongo $mongo_uri
MongoDB shell version v3.6.23
connecting to: mongodb://localhost.localdomain:39001/admin?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") }
MongoDB server version: 3.6.23
Welcome to the MongoDB shell.
For interactive help, type "help".
For more comprehensive documentation, see
        http://docs.mongodb.org/
Questions? Try the support group
        http://groups.google.com/group/mongodb-user
meteor:PRIMARY>  db.runCommand({compact:'projects', force: true})
{
        "operationTime" : Timestamp(1686294946, 1),
        "ok" : 0,
        "errmsg" : "not authorized on admin to execute command { compact: \"projects\", force: true, lsid: { id: UUID(\"356b5fd6-7915-46f7-800b-7b6055ff57ca\") }, $clusterTime: { clusterTime: Timestamp(1686294946, 1), signature: { hash: BinData(0, BCD8801BCD7F94C00B986431024A3AD27D5A7FA8), keyId: 7189609347507290115 } }, $db: \"admin\" }",
        "code" : 13,
        "codeName" : "Unauthorized",
        "$clusterTime" : {
                "clusterTime" : Timestamp(1686294946, 1),
                "signature" : {
                        "hash" : BinData(0,"vNiAG81/lMALmGQxAko60n1af6g="),
                        "keyId" : NumberLong("7189609347507290115")
                }
        }
}

meteor:PRIMARY>
bye

forcing “admin”:

[cryoem@ld20-1013 tmp]$ mongo_uri=$(python -c "from cryosparc_compute import database_management; print(database_management.get_mongo_uri('admin', admin=True))")
[cryoem@ld20-1013 tmp]$ mongo $mongo_uri
MongoDB shell version v3.6.23
connecting to: mongodb://localhost.localdomain:39001/admin?authSource=admin&gssapiServiceName=mongodb
Implicit session: session { "id" : UUID("xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx") }
MongoDB server version: 3.6.23
Server has startup warnings:
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten]
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/enabled is 'always'.
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten]
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] ** WARNING: /sys/kernel/mm/transparent_hugepage/defrag is 'always'.
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] **        We suggest setting it to 'never'
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten]
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] ** WARNING: Running wiredTiger with the --nojournal option in a replica set
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten] **          is deprecated and subject to be removed in a future version.
2023-06-09T09:10:34.659+0200 I CONTROL  [initandlisten]
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0]
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] ** WARNING: This replica set node is running without journaling enabled but the
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          writeConcernMajorityJournalDefault option to the replica set config
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          is set to true. The writeConcernMajorityJournalDefault
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          option to the replica set config must be set to false
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          or w:majority write concerns will never complete.
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          In addition, this node's memory consumption may increase until all
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0] **          available free RAM is exhausted.
2023-06-09T09:10:34.686+0200 I REPL     [replexec-0]
meteor:PRIMARY>

Thanks for reading until herr!

Tru

wtempel · June 9, 2023, 12:28pm

Hi @Tru,
Please can you email us the tgz created by the command
cryosparcm snaplogs.

Tru · June 9, 2023, 2:14pm

Hi,

a colleage has just upgraded the version to 4.2.1, and that has “fixed” the mongodb issue!

[cryoem@ld20-1013 tmp]$ cryosparcm status
----------------------------------------------------------------------------
CryoSPARC System master node installed at
/home/cryoem/Applications/cryoSPARC2/cryosparc2_master
Current cryoSPARC version: v4.2.1
----------------------------------------------------------------------------

CryoSPARC process status:

app                              RUNNING   pid 65244, uptime 0:06:20
app_api                          RUNNING   pid 65263, uptime 0:06:18
app_api_dev                      STOPPED   Not started
app_legacy                       STOPPED   Not started
app_legacy_dev                   STOPPED   Not started
command_core                     RUNNING   pid 65044, uptime 0:06:33
command_rtp                      RUNNING   pid 65143, uptime 0:06:26
command_vis                      RUNNING   pid 65074, uptime 0:06:27
database                         RUNNING   pid 64934, uptime 0:06:37

----------------------------------------------------------------------------
License is valid

Thanks

Tru

wtempel · June 21, 2023, 3:48pm