Import project and regenerate database after database failure

Data Management
1

Hi,

One of our cryosparc instances crashed, and the database file got damaged (over 180GB total). I tried the mongo repair database command, but it showed that one of the files was corrupt and exited in error. I restored the old database backup, but multiple jobs are missing. In previous cryosparc versions, it was possible to import a project, and it “regenerated” the project structure. However, with the new cryosparc v4 attaching the project to a new database generates an empty project. Could you suggest some solution to it? We will increase the frequency of the backups, but still, there should be some additional solutions in cases like this. Thanks!

2

Are only such jobs missing that were created after the backup that you just restored?

This behavior is not expected.

  1. Was this an attempt to attach the project to a new database, or to the database that was restored from backup?
  2. Did you observe any error messages during the attachment attempt?
  3. Was the project whose attachment you attempted previously part of the CryoSPARC instance that crashed and whose database was damaged?
  4. If you wish, you may email us a compressed copy of
    /path/to/cryosparc_master/run/command_core.log, also specifying the path of project directory to be imported.
3

Yes, only jobs are missing that were created after the backup date.

Yes, we tried this. Empty, the new database and attach the project from the folder. The project ends up empty without any workspaces/jobs inside.

No, the attachment went smoothly.

Yes. It was a project from the same cryosparc instance that just dated back.

Sure. It generated over 10 command_core.log logs, most showing the same message. Happy to provide them for further investigation.

Thanks!

4

Likely, creation of the fresh database entailed creation of a new “instance ID”. Attachment of a project directory would fail if the project directory was still locked to an older instance ID. I noticed this problem inside a log file you provided.
We generally discourage manipulation of cs.lock files. An exception may be warranted when an (outdated) database backup is missing information on many newer jobs, and you may prefer starting from a fresh database:

  • recreate your CryoSPARC users
  • re-connect your worker host(s) or cluster(s)
  • attach project directories after removing existing cs.lock files

I strongly recommend more frequent backups going forward to avoid having to resort to this tedious procedure.

5

I had a similar issue after a hardware failure, I had to move jobs to another machine but could not export them. I simply changed the name of cs.lock to cs.old (deleted the file effectively) and was able to import the projects to a new instance of cryosparc that had no projects in it so there would be no overlapping project numbers and this worked fine.

6

@schiracha
One may attach projects to CryoSPARC instances that already contain projects.
During project attachment, CryoSPARC will assign the next available project ID to the project, thereby avoiding conflicts with the IDs of projects that already exist on the instance. The newly attached project’s previous ID plays no role in the selection of the new project ID; preservation of the project ID throughout a sequence of detaching and attaching a project would be a coincidence.

7

Yes, a problem occurs when you cant detatch the project because of a hardware failure. When I tried to do this with a project already in the space with teh same number and a cs file deleted, I got an error that the project already existed

8

Could, by any chance, the error message have said that the project directory already exists?