Incomplete project import

Data Management
1

Hi,

I am trying to import my projects from one machine to another. Both the origin and destination installations run CS v3.2. After copying the project folders (including links) to the new machine, I applied the required patch to Cryosparc and ran the import. For some reason, one of the projects was properly imported, while the other has all the workspaces but is missing many of the jobs. Both of the projects have many workspaces and many jobs. The only difference between them is that the faulty one is older and has been used in the past with older versions of Cryosparc.

As suggested elsewhere, I ran “grep -w IMPORT_PROJECT ./run/command_core.log > import_project_logs.txt” from the master folder. In the logfile created I see at the end of the import the following line:

[IMPORT_PROJECT] : Inserting streamlogs into jobs…
[IMPORT_PROJECT] : Unable to import project from /data/cryosparc/…/P8: not enough data for a BSON document

Please let me know if you have an idea about what causes this problem.

Thanks,
Lior

2

@lalmagor Is the failed project directory P8 missing metadata that should be there after a successful (previous) migration from pre-2.11?

Please can you cd into the directory that contains the dysfunctional P8 directory and run

find ./P8/ -name "*son" -o -name "*_data" -o -name "*.log"

What’s the command’s output?

1 Like
3

Thanks for replying.

When I run the command on my project folder I can see that it finds the four files in each of the job folders. For e.g. job 204 has:

/P8/J204/gridfs_data
./P8/J204/job.json
./P8/J204/job.log
./P8/J204/events.bson

The same four files appear in jobs that do or do not show in the imported project’s workspace.

There are also folders that are missing the job.log file, but checking with the original project, these are in folders that belong to deleted jobs that are not showing in the original workspace anymore.

4

@lalmagor Please could you share with us

  • the file ./P8/job_manifest.json
  • a list of job directories that correspond to missing jobs

?

5

Hi wtempel,

Please see below.

./P8/job_manifest.json:
job_manifest.json

Missing projects:
missing.txt

Thanks

6

@lalmagor
Assuming that missing.txt is a complete listing of “missing” jobs, the import failure pattern deviates from the expected behavior when a job import fails: once a job fails to be imported, jobs subsequent to that job in job_manifest.json should also fail to be imported.
Something else must have happened, but I do not know what it was.

In case you suspect that metadata were corrupted during the copying from the old machine, you could retry (taking care to always maintain a safe copy of your data):

  • examine the source of your data for corruption (for example, did you ever run out of space during previous copy operations involving those data?)
  • copy the data again, but to a path on the destination host that differs from your previous copy attempt’s path
  • repeat the project import using the new destination path

Alternatively, you can attempt importing individual jobs of interest if the old cryoSPARC instance is still functional.

You could also share with us (via direct message) the complete run/command_core.log file that contained the error message you reported earlier. Log rotation may have occurred since you first reported the problem, and a digit may have been appended to the appropriate file, for example command_core.log.1.

7

Hi,

My project on the old machine is intact and working properly.
I will try to copy my files again to another location on the new machine in case something going wrong last time I did it was the cause for the corruption.

Thank you so much,
Lior