After 'Detach project' in the old cryosparc, the directory was copied to another path, but the 'Attach project' operation failed

Data Management
1

Thank you all for your support. Now, I will present my problem description.

After ‘Detach project’ in v4.6.2, I failed to ‘Attach project’ in v4.7.1, and received the error message ‘Unable to import project from /xxx/xxx/xxx: Ran out of input’. Later, when I tried ‘Attach project’ again in v4.6.2, the same error occurred.

On the web page, one can see the new project being generated. After entering, one can also see Workspaces, but the jobs inside are missing.

image
image1628×900 60.9 KB

Entering the “workspaces” section, there is no content.
workspaces
workspaces1593×708 18.9 KB

Notifications
Notifications1413×147 7.22 KB

2

@zhenyuanliu Please can you try the following command

cryosparcm log command_core | grep -A 60 "Created project P629"

and post the output here.

3

Thank you for your assistance. Here is the output result of the command.

[cryosparc@alogin05 ~]$ cryosparcm log command_core | grep -A 100 "Created project P629"
2025-07-31 14:47:28,482 import_project       INFO     | Created project P629
2025-07-31 14:47:28,497 import_workspaces    INFO     | Created workspace W1 in P629
2025-07-31 14:47:28,498 import_workspaces    INFO     | Created workspace W2 in P629
2025-07-31 14:47:28,499 import_workspaces    INFO     | Created workspace W3 in P629
2025-07-31 14:47:28,500 import_workspaces    INFO     | Created workspace W4 in P629
2025-07-31 14:47:28,500 import_workspaces    INFO     | Created workspace W5 in P629
2025-07-31 14:47:28,501 import_workspaces    INFO     | Created workspace W6 in P629
2025-07-31 14:47:28,502 import_workspaces    INFO     | Created workspace W7 in P629
2025-07-31 14:47:28,502 import_workspaces    INFO     | Created workspace W8 in P629
2025-07-31 14:47:28,503 import_workspaces    INFO     | Created workspace W9 in P629
2025-07-31 14:47:28,504 import_workspaces    INFO     | Created workspace W10 in P629
2025-07-31 14:47:28,504 import_workspaces    INFO     | Created workspace W11 in P629
2025-07-31 14:47:28,505 import_workspaces    INFO     | Created workspace W12 in P629
2025-07-31 14:47:28,506 import_workspaces    INFO     | Created workspace W13 in P629
2025-07-31 14:47:28,506 import_workspaces    INFO     | Created workspace W14 in P629
2025-07-31 14:47:28,507 import_workspaces    INFO     | Created workspace W15 in P629
2025-07-31 14:47:28,508 import_workspaces    INFO     | Created workspace W16 in P629
2025-07-31 14:47:28,508 import_workspaces    INFO     | Created workspace W17 in P629
2025-07-31 14:47:28,509 import_workspaces    INFO     | Created workspace W18 in P629
2025-07-31 14:47:28,510 import_workspaces    INFO     | Created workspace W19 in P629
2025-07-31 14:47:28,511 import_workspaces    INFO     | Created workspace W20 in P629
2025-07-31 14:47:28,511 import_workspaces    INFO     | Created workspace W21 in P629
2025-07-31 14:47:28,512 import_workspaces    INFO     | Created workspace W22 in P629
2025-07-31 14:47:28,513 import_workspaces    INFO     | Created workspace W23 in P629
2025-07-31 14:47:28,513 import_workspaces    INFO     | Created workspace W24 in P629
2025-07-31 14:47:28,514 import_workspaces    INFO     | Created workspace W25 in P629
2025-07-31 14:47:28,515 import_workspaces    INFO     | Created workspace W26 in P629
2025-07-31 14:47:28,515 import_workspaces    INFO     | Created workspace W27 in P629
2025-07-31 14:47:28,537 import_workspaces    INFO     | Created workspace W28 in P629
2025-07-31 14:47:28,538 import_workspaces    INFO     | Created workspace W29 in P629
2025-07-31 14:47:28,539 import_workspaces    INFO     | Created workspace W30 in P629
2025-07-31 14:47:28,540 import_workspaces    INFO     | Created workspace W31 in P629
2025-07-31 14:47:28,541 import_workspaces    INFO     | Created workspace W32 in P629
2025-07-31 14:47:28,541 import_workspaces    INFO     | Created workspace W33 in P629
2025-07-31 14:47:28,542 import_workspaces    INFO     | Created workspace W35 in P629
2025-07-31 14:47:28,543 import_workspaces    INFO     | Created workspace W36 in P629
2025-07-31 14:47:28,544 import_workspaces    INFO     | Created workspace W37 in P629
2025-07-31 14:47:28,544 import_workspaces    INFO     | Created workspace W38 in P629
2025-07-31 14:47:28,545 import_workspaces    INFO     | Created workspace W39 in P629
2025-07-31 14:47:28,546 import_workspaces    INFO     | Created workspace W40 in P629
2025-07-31 14:47:28,546 import_workspaces    INFO     | Created workspace W41 in P629
2025-07-31 14:47:28,547 import_workspaces    INFO     | Created workspace W42 in P629
2025-07-31 14:47:28,548 import_workspaces    INFO     | Created workspace W43 in P629
2025-07-31 14:47:28,548 import_workspaces    INFO     | Created workspace W44 in P629
2025-07-31 14:47:28,549 import_workspaces    INFO     | Created workspace W45 in P629
2025-07-31 14:47:28,550 import_workspaces    INFO     | Created workspace W46 in P629
2025-07-31 14:47:28,550 import_workspaces    INFO     | Created workspace W47 in P629
2025-07-31 14:47:28,551 import_workspaces    INFO     | Created workspace W48 in P629
2025-07-31 14:47:28,552 import_workspaces    INFO     | Created workspace W49 in P629
2025-07-31 14:47:28,552 import_workspaces    INFO     | Created workspace W50 in P629
2025-07-31 14:47:28,553 import_workspaces    INFO     | Created workspace W51 in P629
2025-07-31 14:47:28,554 import_workspaces    INFO     | Created workspace W52 in P629
2025-07-31 14:47:28,554 import_workspaces    INFO     | Created workspace W53 in P629
2025-07-31 14:47:28,575 import_jobs          INFO     | Inserting jobs into project...
2025-07-31 14:47:28,595 import_jobs          INFO     | Uploading image data for J1...
2025-07-31 14:47:28,634 import_project_run   ERROR    | Unable to import project from /Share3/cryosparc/cryosparc_projects/new_sunke_P775
2025-07-31 14:47:28,634 import_project_run   ERROR    | Traceback (most recent call last):
2025-07-31 14:47:28,634 import_project_run   ERROR    |   File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4631, in import_project_run
2025-07-31 14:47:28,634 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-07-31 14:47:28,634 import_project_run   ERROR    |   File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4850, in import_jobs
2025-07-31 14:47:28,634 import_project_run   ERROR    |     gridfs_data = pickle.load(openfile)
2025-07-31 14:47:28,634 import_project_run   ERROR    | EOFError: Ran out of input
2025-07-31 14:47:28,634 dump_project         INFO     | Exporting project P629
2025-07-31 14:47:28,635 dump_project         INFO     | Exported project P629 to /Share3/cryosparc/cryosparc_projects/new_sunke_P775/project.json in 0.00s
2025-07-31 14:47:28,663 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-07-31 14:47:28,663 run                  ERROR    | Traceback (most recent call last):
2025-07-31 14:47:28,663 run                  ERROR    |   File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-07-31 14:47:28,663 run                  ERROR    |     self.target(*self.args)
2025-07-31 14:47:28,663 run                  ERROR    |   File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4631, in import_project_run
2025-07-31 14:47:28,663 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-07-31 14:47:28,663 run                  ERROR    |   File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4850, in import_jobs
2025-07-31 14:47:28,663 run                  ERROR    |     gridfs_data = pickle.load(openfile)
2025-07-31 14:47:28,663 run                  ERROR    | EOFError: Ran out of input
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2304, in run_with_except_hook
    run_old(*args, **kw)
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4631, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4850, in import_jobs
    gridfs_data = pickle.load(openfile)
EOFError: Ran out of input

Traceback (most recent call last):
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2304, in run_with_except_hook
    run_old(*args, **kw)
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4631, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/Share2/cryosparc/.software/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4850, in import_jobs
    gridfs_data = pickle.load(openfile)
EOFError: Ran out of input
4

Hello, based on the error message, I temporarily moved the ‘J1’ directory to another directory in an attempt to skip it. After moving multiple jobs, I found that I still needed to address the dependencies between the jobs. May I ask if there is a way to restore the project that has been ‘Detached Project’ to be usable again? Or is it possible to prevent Cryosparc from resolving the dependencies and successfully import some normal jobs?

5

Some files in the new copy of the project directory may be corrupted. To confirm possible corruption, and to determine whether corruption was present before the copy operation, may I suggest comparing the copy of the J1/ directory to the J1/ directory inside the original, detached project directory. Please can you post the outputs of these two commands for both copies of the J1/ directory, respectively:

find /path/to/J1 -ls
find /path/to/J1 -exec sha256sum {} \;
6

I’m truly sorry. I have already deleted the original data. However, I made the backup using ‘rsync -avP’, and there were no errors reported in the backup log.

7

May I ask if I can restore a project that has been ‘Detached’ to a working state again?

8

If I decide not to restore the original job dependency completely and only import the jobs into the new project, is there a command that allows me to perform batch operations? Previously, with 4000 jobs, it took a very long time to export each job and then re-import them one by one.

9

I have not tested using the rsync -P flag in the context of transferring a CryoSPARC project, but am concerned about the risk that the --partial option, which the -P option implies, might increase the risk of incomplete files in the copy.

If an intact copy of the project directory is available, it is expected that attaching that copy to a CryoSPARC instance will result in a working CryoSPARC project. An intact copy of the project directory seems, unfortunately, unavailable in the current case.

In CryoSPARC v4.7.1, cli commands can be used to export jobs from an attached project and import (a copy of) a given job’s export directory to a CryoSPARC project. Ordinarily, the import of individual jobs, outside the context of a project attachment, requires preceding, corresponding job exports. Do I understand correctly that jobs have not been individually exported?