Import project and regenerate database after database failure

[dzyla]

Hi,

One of our cryosparc instances crashed, and the database file got damaged (over 180GB total). I tried the mongo repair database command, but it showed that one of the files was corrupt and exited in error. I restored the old database backup, but multiple jobs are missing. In previous cryosparc versions, it was possible to import a project, and it “regenerated” the project structure. However, with the new cryosparc v4 attaching the project to a new database generates an empty project. Could you suggest some solution to it? We will increase the frequency of the backups, but still, there should be some additional solutions in cases like this. Thanks!

[wtempel]

@dzyla

I restored the old database backup, but multiple jobs are missing.

Are only such jobs missing that were created after the backup that you just restored?

@dzyla

attaching the project to a new database generates an empty project.

This behavior is not expected.

1. Was this an attempt to attach the project to a new database, or to the database that was restored from backup?
2. Did you observe any error messages during the attachment attempt?
3. Was the project whose attachment you attempted previously part of the CryoSPARC instance that crashed and whose database was damaged?
4. If you wish, you may email us a compressed copy of
   /path/to/cryosparc_master/run/command_core.log, also specifying the path of project directory to be imported.

[dzyla]

@wtempel

Are only such jobs missing that were created after the backup that you just restored?

Yes, only jobs are missing that were created after the backup date.

@wtempel

1. Was this an attempt to attach the project to a new database, or to the database that was restored from backup?

Yes, we tried this. Empty, the new database and attach the project from the folder. The project ends up empty without any workspaces/jobs inside.

@wtempel

2. Did you observe any error messages during the attachment attempt?

No, the attachment went smoothly.

@wtempel

3. Was the project whose attachment you attempted previously part of the CryoSPARC instance that crashed and whose database was damaged?

Yes. It was a project from the same cryosparc instance that just dated back.

@wtempel

If you wish, you may email us a compressed copy of
/path/to/cryosparc_master/run/command_core.log, also specifying the path of the project directory to be imported.

Sure. It generated over 10 command_core.log logs, most showing the same message. Happy to provide them for further investigation.

Thanks!

[wtempel]

@dzyla

Yes, we tried this. Empty, the new database and attach the project from the folder. The project ends up empty without any workspaces/jobs inside.

Likely, creation of the fresh database entailed creation of a new “instance ID”. Attachment of a project directory would fail if the project directory was still locked to an older instance ID. I noticed this problem inside a log file you provided.
We generally discourage manipulation of cs.lock files. An exception may be warranted when an (outdated) database backup is missing information on many newer jobs, and you may prefer starting from a fresh database:

• recreate your CryoSPARC users
• re-connect your worker host(s) or cluster(s)
• attach project directories after removing existing cs.lock files

I strongly recommend more frequent backups going forward to avoid having to resort to this tedious procedure.