Please advise: database: ERROR (spawn error)

zhangrui_wustl · April 9, 2021, 5:24am

Hi, I mistakenly moved the cryosparc folder to another location, created a soft link at its original location, then launched cryosparc web GUI, saw it is not responding, then moved the cryosparc folder back to its original location.

However, when I run “ryosparcm restart”, I got “database: ERROR (spawn error)”

Here is log info after running “cryosparcm log database”
Please help! Thanks!

2021-04-09T00:09:18.127-0500 E STORAGE  [initandlisten] WiredTiger error (-31802) [1617944958:127375][138724:0x7f823a719d00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2021-04-09T00:09:18.127-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944958:127394][138724:0x7f823a719d00], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2021-04-09T00:09:18.127-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944958:127398][138724:0x7f823a719d00], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2021-04-09T00:09:18.127-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944958:127401][138724:0x7f823a719d00], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2021-04-09T00:09:18.127-0500 I -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 276
2021-04-09T00:09:18.127-0500 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2021-04-09T00:09:18.127-0500 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2021-04-09T00:09:18.127-0500 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-39002.sock
2021-04-09T00:09:18.127-0500 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2021-04-09T00:09:18.127-0500 I CONTROL  [initandlisten] now exiting
2021-04-09T00:09:18.127-0500 I CONTROL  [initandlisten] shutting down with code:100
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] MongoDB starting : pid=138727 port=39002 dbpath=/home/zhangrui/bin/cryosparc/cryosparc_database 64-bit host=cns
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] db version v3.4.10
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] allocator: tcmalloc
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] modules: none
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] build environment:
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten]     distarch: x86_64
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten]     target_arch: x86_64
2021-04-09T00:09:21.199-0500 I CONTROL  [initandlisten] options: { net: { port: 39002 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/home/zhangrui/bin/cryosparc/cryosparc_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2021-04-09T00:09:21.199-0500 W -        [initandlisten] Detected unclean shutdown - /home/zhangrui/bin/cryosparc/cryosparc_database/mongod.lock is not empty.
2021-04-09T00:09:21.223-0500 I -        [initandlisten] Detected data files in /home/zhangrui/bin/cryosparc/cryosparc_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2021-04-09T00:09:21.223-0500 W STORAGE  [initandlisten] Recovering data from the last clean checkpoint.
2021-04-09T00:09:21.223-0500 I STORAGE  [initandlisten]
2021-04-09T00:09:21.223-0500 I STORAGE  [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-04-09T00:09:21.223-0500 I STORAGE  [initandlisten] **          See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-04-09T00:09:21.223-0500 I STORAGE  [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2021-04-09T00:09:21.231-0500 E STORAGE  [initandlisten] WiredTiger error (-31802) [1617944961:231309][138727:0x7fe6d8170d00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2021-04-09T00:09:21.231-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944961:231328][138727:0x7fe6d8170d00], file:WiredTiger.wt, connection: WiredTiger has failed to open its metadata
2021-04-09T00:09:21.231-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944961:231331][138727:0x7fe6d8170d00], file:WiredTiger.wt, connection: This may be due to the database files being encrypted, being from an older version or due to corruption on disk
2021-04-09T00:09:21.231-0500 E STORAGE  [initandlisten] WiredTiger error (0) [1617944961:231335][138727:0x7fe6d8170d00], file:WiredTiger.wt, connection: You should confirm that you have opened the database with the correct options including all encryption and compression options
2021-04-09T00:09:21.231-0500 I -        [initandlisten] Assertion: 28595:-31802: WT_ERROR: non-specific WiredTiger error src/mongo/db/storage/wiredtiger/wiredtiger_kv_engine.cpp 276
2021-04-09T00:09:21.231-0500 I STORAGE  [initandlisten] exception in initAndListen: 28595 -31802: WT_ERROR: non-specific WiredTiger error, terminating
2021-04-09T00:09:21.231-0500 I NETWORK  [initandlisten] shutdown: going to close listening sockets...
2021-04-09T00:09:21.231-0500 I NETWORK  [initandlisten] removing socket file: /tmp/mongodb-39002.sock
2021-04-09T00:09:21.231-0500 I NETWORK  [initandlisten] shutdown: going to flush diaglog...
2021-04-09T00:09:21.231-0500 I CONTROL  [initandlisten] now exiting
2021-04-09T00:09:21.231-0500 I CONTROL  [initandlisten] shutting down with code:100

stephan · April 9, 2021, 2:34pm

Hi @zhangrui_wustl,

If cryoSPARC is back to it’s original location, you should do the following:

Restart your workstation
Repair your database
Or, optionally, restore from backup (if you have one)

First, make sure cryoSPARC is turned off (cryosparcm stop) and no zombie processes are running (follow the instructions here after the line Check that no ghost processes are running on all nodes)

Then, restart your workstation if you can.

Once that’s done, run the commands to repair your database:

eval $(cryosparcm env)
cd cryosparc2_database
mongod --dbpath ./ --repair

For more information on mongoDB repairs, see this article.

zhangrui_wustl · April 9, 2021, 2:53pm

Hi, I followed your instructions but the repair didn’t seem to work.
Please see the error message:

2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] MongoDB starting : pid=24101 port=27017 dbpath=./ 64-bit host=cns
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] db version v3.4.10
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] git version: 078f28920cb24de0dd479b5ea6c66c644f6326e9
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] allocator: tcmalloc
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] modules: none
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] build environment:
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] distarch: x86_64
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] target_arch: x86_64
2021-04-09T09:45:14.310-0500 I CONTROL [initandlisten] options: { repair: true, storage: { dbPath: “./” } }
2021-04-09T09:45:14.325-0500 I - [initandlisten] Detected data files in ./ created by the ‘wiredTiger’ storage engine, so setting the active storage engine to ‘wiredTiger’.
2021-04-09T09:45:14.325-0500 I STORAGE [initandlisten]
2021-04-09T09:45:14.325-0500 I STORAGE [initandlisten] ** WARNING: Using the XFS filesystem is strongly recommended with the WiredTiger storage engine
2021-04-09T09:45:14.325-0500 I STORAGE [initandlisten] ** See http://dochub.mongodb.org/core/prodnotes-filesystem
2021-04-09T09:45:14.325-0500 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=257306M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),log=(enabled=false),
2021-04-09T09:45:14.342-0500 E STORAGE [initandlisten] WiredTiger error (-31802) [1617979514:342979][24101:0x7fd149123d00], file:WiredTiger.wt, connection: unable to read root page from file:WiredTiger.wt: WT_ERROR: non-specific WiredTiger error
2021-04-09T09:45:14.343-0500 E STORAGE [initandlisten] WiredTiger error (0) [1617979514:342999][24101:0x7fd149123d