I maintain multiple cryoSPARC stand-alone servers. Before updating to the latest version, I would have users report that cryoSPARC had stopped working. I couldn’t start it again because the database threw a spawn error. If I rebooted the server, then I could start cryoSPARC normally. However, a reboot stopped fixing the problem on two servers. When I upgraded from v3.3.1+patch to V3.3.2 (& 05/18 patch on one) last Friday, I had to restore the database from backup. Over the weekend both servers give the error that the database exited too quickly. A reboot still doesn’t allow cryoSPARC to be restarted. The mongodb repair steps fail too (on both servers). The hardware checks out good. How can we further troubleshoot the software?
Thanks,
Irvin
@irv555
The database and command_core logs may provide some clues.
You can capture them with the commands:
cryosparcm log command_core > core_20220524_server1.out
and
cryosparcm log database > db_20220524_server1.out
Another question: Are the servers’ /tmp directories cleared during reboot? /tmp/mongodb-*.sock and /tmp/cryosparc-supervisor-*.sock files that survive a reboot would interfere we cryoSPARC startup.
That’s a lot of information in the files. Can I give you a link where the two output files are at?
XXXXXXXXXXXXXXXXXX
I have tried to recover both ways (with and without) the tmp files. Deleting the supervisor sock file has helped when I get that type of error, but messing with the db sock file has never made a difference. Only a reboot would usually fix this. However, not this time.
Thanks.
The database log included illegal file format or internal value errors. If there is a cluster of such errors on multiple instance, one may suspect a common cause, such as
- “unclean” shutdown(s) of the instances
- unreliable storage media or connections to storage
Even if a --repair operation appears to succeed, I am not sure whether that operation might discard data, as the documentation for a newer version of mongodb indicates (cryoSPARC currently uses an older version of the database software).
It would therefore preferable to restore a recent backup of the database.
If a recent backup is not available, and another attempt at mongod [...] --repair fails, there is the option of starting from a fresh database. The process could be like this:
- ensure the instance is properly shutdown:
cryosparcm stop
- create a new directory for the database and update the
export CRYOSPARC_DB_PATH=...
line in cryosparc_master/config.sh accordingly.
- restart the instance:
cryosparcm start
The process would initialize a “blank” database that lacks (1) user information, (2) worker configuration, (3) project data. You would use the command line to restore (1) and (2). For (3), you may import existing project directories.