Yesterday I have updated to recent version 2.15. I have tried to reset the password of a user using the Token available in the Admin page, but without requesting for a reset password in the login page. Actually it worked but when user logged in he could not see his recent runs but only older ones. But no other user could log in. I imagine changing the password without reset request corrupted the database ( only my guess).
We have a backup system running every morning. So I took the database backup ( was with version 2.14.2) and tried start crysosparc ( version 2.15) and that is giving me Database Spawn error. Was there a change of db handling from 2.14.2 to 2.15 ? My guess is that if I switch to 2.14.2 with DB back up it could work. How do I install a fresh instance of 2.14.2 ( I will use the backed up database ). ?
Sorry for all the trouble. it was a bad coincidence
I have downgraded to 2.14.2 and used a database snapshot of older ( by 2 days ) database, but it still gives
database: ERROR (spawn error)
cryosparcm log database gives the following
2020-08-21T16:27:26.595+0200 I CONTROL [initandlisten] MongoDB starting : pid=4402 port=39001 dbpath=/fs/fs02/lv14/pool/pool-conti-em2/cryosparc_database 64-bit host=alderaan
2020-08-21T16:27:26.595+0200 I CONTROL [initandlisten] db version v3.4.10-4-g67ee356c6b
2020-08-21T16:27:26.595+0200 I CONTROL [initandlisten] git version: 67ee356c6be377cda547d16423daef5beb4e8377
2020-08-21T16:27:26.595+0200 I CONTROL [initandlisten] OpenSSL version: OpenSSL 1.0.2p 14 Aug 2018
2020-08-21T16:27:26.595+0200 I CONTROL [initandlisten] allocator: tcmalloc
2020-08-21T16:27:26.596+0200 I CONTROL [initandlisten] modules: none
2020-08-21T16:27:26.596+0200 I CONTROL [initandlisten] build environment:
2020-08-21T16:27:26.596+0200 I CONTROL [initandlisten] distarch: x86_64
2020-08-21T16:27:26.596+0200 I CONTROL [initandlisten] target_arch: x86_64
2020-08-21T16:27:26.596+0200 I CONTROL [initandlisten] options: { net: { port: 39001 }, replication: { oplogSizeMB: 64, replSet: "meteor" }, storage: { dbPath: "/fs/fs02/lv14/pool/pool-conti-em2/cryosparc_database", journal: { enabled: false }, wiredTiger: { engineConfig: { cacheSizeGB: 4.0 } } } }
2020-08-21T16:27:26.599+0200 W - [initandlisten] Detected unclean shutdown - /fs/fs02/lv14/pool/pool-conti-em2/cryosparc_database/mongod.lock is not empty.
2020-08-21T16:27:26.625+0200 I - [initandlisten] Detected data files in /fs/fs02/lv14/pool/pool-conti-em2/cryosparc_database created by the 'wiredTiger' storage engine, so setting the active storage engine to 'wiredTiger'.
2020-08-21T16:27:26.625+0200 W STORAGE [initandlisten] Recovering data from the last clean checkpoint.
2020-08-21T16:27:26.626+0200 I STORAGE [initandlisten] wiredtiger_open config: create,cache_size=4096M,session_max=20000,eviction=(threads_min=4,threads_max=4),config_base=false,statistics=(fast),log=(enabled=true,archive=true,path=journal,compressor=snappy),file_manager=(close_idle_time=100000),checkpoint=(wait=60,log_size=2GB),statistics_log=(wait=0),,log=(enabled=false),
2020-08-21T16:27:26.709+0200 E STORAGE [initandlisten] WiredTiger error (-31802) [1598020046:709343][4402:0x7f8f81a8fd40], file:sizeStorer.wt, WT_SESSION.open_cursor: unable to read root page from file:sizeStorer.wt: WT_ERROR: non-specific WiredTiger error
2020-08-21T16:27:26.711+0200 I - [initandlisten] Invariant failure: ret resulted in status UnknownError: -31802: WT_ERROR: non-specific WiredTiger error at src/mongo/db/storage/wiredtiger/wiredtiger_size_storer.cpp 69
2020-08-21T16:27:26.711+0200 I - [initandlisten]
***aborting after invariant() failure
Is it possible to recover the database ?
Any help in this regard would be very much helpful
It looks like there may have been a mongoDB instance already running when you tried to start the database up with the older database files. I think the best course of action at this point would be to turn off anything related to cryoSPARC (including the database, and killing processes gracefully if you have to), then running a repair command.
For example:
cryosparcm stop
ps -ax | grep “supervisord” (kill only the process that is running from your cryosparc2 install)
ps -ax | grep “cryosparc2_command” (kill all the matching processes related to your cryosparc2 instance)
ps -ax | grep “mongod” (kill only the process running your cryosparc2 database)
e.g. kill 82681
Ensure you have a backup of your database
eval $(cryosparcm env)
cd cryosparc2_database
mongod --dbpath ./ --repair
Thanks for the tip. That rebuilt the index and I could rescue the database. There are many users and complete database folder is about 280 GB… Is there anyway to make it smaller ? Every time something or other happens it take very long time to either rescue or pull it back from backup. Thanks.
Hi @apunjani@stephan
I have another question. In case if I delete a user from the database, will all the projects corresponding to the user will also be deleted or retained ? Especially the directories in the hard disk… Thanks
It is possible to reduce the size of the MongoDB database by using the compact command, but please note this is a blocking command; cryoSPARC will not work while this command is running.
MongoDB does not release storage space back to the OS when you delete documents, which is why the storage size remains the same.
First, create a backup of your database.
Then, in a shell on the master node:
cryosparcm restart
eval $(cryosparcm env)
mongo localhost:$CRYOSPARC_MONGO_PORT # will create a db shell
@stephan thanks for this, but I seem to have trouble getting this to work.
Running v3.0.1 and database is now ~170 GB.
I have cleaned up by deleted many projects.
Although the database does not shrink upon compacting. Se below.
Any hints?
Looking further into this: It seems as backup -> restore can solve the problem?
//Jesper
-bash-4.2$ mongo localhost:39001
MongoDB shell version v3.4.10
connecting to: localhost:39001
MongoDB server version: 3.4.10
Server has startup warnings:
2021-01-18T07:18:44.267+0100 I CONTROL [initandlisten]
2021-01-18T07:18:44.267+0100 I CONTROL [initandlisten] ** WARNING: Access control is not enabled for the database.
2021-01-18T07:18:44.267+0100 I CONTROL [initandlisten] ** Read and write access to data and configuration is unrestricted.
2021-01-18T07:18:44.267+0100 I CONTROL [initandlisten]
meteor:PRIMARY> use meteor
switched to db meteor
meteor:PRIMARY> db.stats()
{
“db” : “meteor”,
“collections” : 17,
“views” : 0,
“objects” : 16942594,
“avgObjSize” : 10063.272572251923,
“dataSize” : 170497941503,
“storageSize” : 185287651328,
“numExtents” : 0,
“indexes” : 58,
“indexSize” : 743878656,
“ok” : 1
}
meteor:PRIMARY> db.runCommand({compact:‘projects’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘workspaces’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘jobs’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘users’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘cache_files’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘events’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘file_index’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘fs.chunks’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘fs.files’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘int_curate_exposures’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.runCommand({compact:‘notifications’, force: true})
{ “ok” : 1 }
meteor:PRIMARY> db.stats()
{
“db” : “meteor”,
“collections” : 17,
“views” : 0,
“objects” : 16942594,
“avgObjSize” : 10063.272572251923,
“dataSize” : 170497941503,
“storageSize” : 185264324608,
“numExtents” : 0,
“indexes” : 58,
“indexSize” : 737787904,
“ok” : 1
}
I have a quota on a shared cluster for the cryosparc database. Now I reached the size limit. Is there a simple way to reduce the database size? Thanks.