Json files are lost. Can we still resurrect (attach) our project?

Import and Export
1

Hello!

We have a backup of our project (not archived) including all the Jobs, but not the three json files (project.json, job_manifest.json, workspaces.json). Can we still recreate the json files by producing synthetic analogs and then attach the project? Although the original project folder is now empty (purged after we made the backup) the GUI still shows the results. Can this information help in making the missing json files?

Thank you in advance,

Anton

2

Welcome to the forum @Anton .
You can try:

  1. On the CryoSPARC master installation whose GUI still shows the results, assuming the project ID was P99, run these commands
    cryosparcm icli # from the shell to the interactive cli
with open('/tmp/tmpMjafg1Mvbr.json', 'w') as output_handle:
    json.dump(cli.get_project('P99'), output_handle, indent=2)
exit() # back to the shell
    and copy /tmp/tmpMjafg1Mvbr.json as project.json into the project directory you wish to attach.
  2. Create a job_manifest.json using the build_manifest.py script in Corrupted project json files after faulty reattaching - #2 by wtempel and copy it into the project directory you wish to attach. Do not manually create workspaces.json.
  3. Try attaching the project and check the command core log for potential errors:
    cryosparcm log command_core
    If the log shows any errors related to the attachment attempt:
    • post errors here
    • do not proceed to next steps
      Due to the absence of workspaces.json, no jobs or workspaces are expected to show up in the GUI (yet) even if attachment did not encounter any errors.
  4. Jobs should begin appearing as you manually create workspaces in the project. Continue creating workspaces until all jobs appear.

Caution: Do not attempt creating new jobs in the project until all existing jobs have been imported successfully. Otherwise, existing job directories may become corrupted.

3

Hello, I have a cryosparc problem that seems to be related to this one.

I have a project made of 6 workspaces, 549 jobs and 11 tera that has been performed and that we wanted to transfer to another server. Once the transfer has been performed, I have reattached it with cryosparc on the new server, it seems to work, but I can only see the first 424 jobs.

The folders for the missing jobs do exist, they seem to contain all the expected files. I can open the .mrc map they contain in Chimera so the jobs seem to be defintely ok, excepted that I cannot see them in cryosparc.

I had a detailed look at the files and I am wondering whether it could be related to one of the three files at the root of this project : job_manifest.json, workspaces.json and project.json. The first two files have been last modified on February 25th, while the last one was last modified last thursday when we attached the project to the new server. The 549 jobs have been generated until march 10 and it happends that the job 424 (the last job that is seen by cryosparc) was generated on February 25th, shortly before the job_manifest.json and workspaces.json. Moreover, the job_manifest.json file contains the job list until job 424.

So I am wondering if the trouble I have could be related with an outdated version of job_manifest.json (and possibly workspaces.json). I have no idea why this file has not been updated each time a new job has been performed. I have no “up to date” version of job_manifest.json.

I am unsure what these files are precisely doing but it seems they have something to do with my problem. Would it be sufficient to “manually” edit job_manifest.json to add the missing jobs ? Are there other files that also have to be modified ? I want to be sure that by doing so I am not going to disrupt the entire project.

4

Welcome to the forum @Ste91

What is or was the version of CryoSPARC running on the new server when you attached the project?
Do you still have the original copy of the project directory, which has not been attached to the new server?

5

The project was initially started on version 5.0.1. It was reattached on version 5.0.4.

I do not have a copy of the full directory. However I have a copy of job_manifest.json, workspaces.json and project.json and I think it was before from before they have been attached.

Something else, I do not know whether it could be part of the problem. When I first reattached the project on the new server, there has been a connection trouble during import, resulting in an import failure. So I de-attached and re-attached the project, this time it went smoothly and there was no mistake.

6

Thanks @Ste91 for these details. please can you email us

  1. copies of the older and the current job_manifest.json files
  2. the tgz file created by the command
    cryosparcm snaplogs
  3. a file project_dir_ls.txt.gz with the directory listing of the project directory, created with these commands:
    cd $(mktemp -d) # create and change to a temporary directory
# replace /path/of/project_dir with actual path
ls -al /path/of/project_dir > project_dir_ls.txt
gzip project_dir_ls.txt
realpath project_dir_ls.txt.gz # note path for retrieval

I’ll send you a personal message with the e-mail address.

7

@Ste91 Thanks for emailing us the logs. May I ask for some clarification regarding

  1. Was the project attached to the “old” server until at least March 10?
  2. Was the project directory properly detached from the “old” server prior tp transfer?
  3. Are the database records for that project still preserved on the “old” server? In other words, is the project still visible on the “old” server when one enables the display of detached projects?
    image
    image1438×186 16.2 KB

    If so, what is the project ID of that project on the “old” server?