Sorry this seems a recurring event. We tried repairing the database but failed. Unfortunately we don’t have a MongodDB backup and would really appreciate your help to restore running CS. The cryosparcm log database and mongod repair messages are very long; I can send them by email for trouble shooting. The repair log seems pointing to an error when we prematurely removed a job before the next job was completed like indicated below.
2023-06-16T14:12:15.932-0400 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.salvage 3200
2023-06-16T14:12:15.950-0400 I STORAGE [initandlisten] WiredTiger progress WT_SESSION.salvage 3300
2023-06-16T14:12:16.120-0400 E STORAGE [initandlisten] WiredTiger error (5) [1686939136:120935][32270:0x7fd2a46ccd40], file:collection-81--4522566990421786315.wt, WT_SESSION.salvage: .//collection-81--4522566990421786315.wt: handle-read: pread: failed to read 1593344 bytes at offset 532168704: Input/output error
2023-06-16T14:12:16.204-0400 E STORAGE [initandlisten] WiredTiger error (5) [1686939136:204395][32270:0x7fd2a46ccd40], file:collection-81--4522566990421786315.wt, WT_SESSION.salvage: .//collection-81--4522566990421786315.wt: handle-read: pread: failed to read 4096 bytes at offset 532168704: Input/output error
2023-06-16T14:12:16.212-0400 I - [initandlisten] Invariant failure rs.get() src/mongo/db/catalog/database.cpp 195
2023-06-16T14:12:16.212-0400 I - [initandlisten]
***aborting after invariant() failure
2023-06-16T14:12:16.226-0400 F - [initandlisten] Got signal: 6 (Aborted).
0x5571d1954ac1 0x5571d1953cd9 0x5571d19541bd 0x7fd2a380b630 0x7fd2a3464387 0x7fd2a3465a78 0x5571d0c28bd8 0x5571d0e077d0 0x5571d0e0e1ef 0x5571d0e11e54 0x5571d130bfee 0x5571d0c13195 0x5571d0c15e70 0x5571d0c3407b 0x7fd2a3450555 0x5571d0c8ec41
----- BEGIN BACKTRACE -----
After identifying and eliminating the cause (hardware error? unsuitable filesystem choice?) of the Input/output error, you may want to try the recovery procedure outlined in Could not get database status - #13 by wtempel.
Thanks for your instructions. They are very helpful. But it was not clear to me how to attach the project in CS 3.3.2. The instructions you sent works in CS 4.0+. Do I need to update to CS4.0+ in order to attach the projects created in CS 3.3.2? Many thanks.
I apologize for my mistake. You are correct: projects can be imported, not attached, in v 3.3.2.
The downtime imposed by the database error may be a good opportunity to update and patch your CryoSPARC instance. Before you go ahead, please ensure you understand what caused the database error, and how you can avoid such error in the future.
The import function in CS 3.3.2 seems to import the exported jobs only. But we did not export any jobs before CS crashed. Is there anyway to get the projects and jobs back without exported jobs?
We had got the CS 3.3.2 running with a new mongod database. We can update our current CS 3.3.2 to the latest version. Is it possible to attach the project created in CS 3.3.2 to CS 4.0+ then?
Many thank!
This is not the expected/documented behavior. Please can you inspect /path/to/cryosparc_master/run/command_core.log for events and failures during the import. The log can be accessed and browsed with the command cryosparcm log command_core
My statement “The import function in CS 3.3.2 seems to import the exported jobs only” was incorrect. CS3.3.2 did import projects. The progress bar showed import was completed but the workplaces showed no jobs. I’ve sent you the command_core.log file by email. The log file showed several errors.
This email has not (yet?) found its way into my inbox. Please can you post relevant/representative errors in this forum topic (after concealing confidential information as needed).
command_core.log is the most recent one. Command_core.log.1 was created five day ago when CS was not set up properly. I don’t think the log.1 file would help with troubleshooting unless you think otherwise.
Maybe the grep commands I provided are not quite accurate. Alternatively you could manually browse command_core.log to investigate the lines that surround the lines output by your earlier grep search Database spawn error ver 3.3.2 - #15 by haomingz. Look for specific file or directory paths, project or job identifiers and other information that could help in a more precise problem definition.
I combed through the log file but didn’t find anything meaningful. Thanks to your help, I recovered the most of what I need. We’ll keep daily backup after updating CS to the latest version. Your help is very much appreciated!!