Import project after migrating files between filesystems (on different cryosparc installations)

Hi all,

I migrated a project from one filesystem to another, and I’d like to import the full project into a different installation of cryoSPARC on the new filesystem. When I try to complete the import, I get the following error:
Unable to import project from /cove/em_processing/abiester/ACDS_20220927/P12: Ran out of input

Does anyone have insights on resolving this error?

Thank you,
Alison

Welcome to the forum @abiester.
What are the CryoSPARC versions of the “old” and the “receiving” CryoSPARC instances, respectively?

What command(s) did you use for the migration?
Did you confirm that /cove/em_processing/abiester/ACDS_20220927/P12 holds a complete copy of the project directory?

The cryoSPARC instances are both v3.3.2. The command I used for the migration was:
rsync -avS abiester@satori-login-003.mit.edu:/nobackup/users/abiester/cryosparc/ACDS_20220927 ./

I ran that command standing in this directory:
/net/cd-cove-10g/cove/em_processing/abiester

I believe that path holds a complete copy of the project directory based on the output of the rsync command.

May I ask:
Did you inspect the output of
cryosparcm log command_core
for error messages related to this attempted import?

Did ACDS_20220927 contain and did you attempt to import any other project directories in addition to P12?
Is the difference between

and

consistent with any filesystem unmount-mount sequence(s) that may have followed the rsync operation?

Here is the output of the command core log:

The directory ACDS_20220927 does not contain projects other than P12. I have tried multiple times to complete the import of P12 and have also tried importing other projects unsuccessfully, but those were in different directories moved with different rsync commands.

I don’t know what a filesystem unmount-mount sequence is. Can you explain further to help me answer that question?

Are you following this procedure:
Guide: Data Management in CryoSPARC (≤v3.3) - CryoSPARC Guide?

Yes, that is the procedure I have been trying.

I was referring to (un-)mount in this context, and
wanted to understand why, in your case, /net/cd-cove-10g was included in your working directory 's path during rsync, but not included in the path definition during import.

Please can you post the command_core log excerpt from above as text, and include (another 20 or so) additional lines that precede that excerpt and are relevant to the import attempt.

Oh I see, yes /net/cd-cove-10g/cove and /cove lead to the same place in the filesystem. I’m not entirely sure if this redundancy is related to (un-)mounting. When I navigate to /cove/em_processing/abiester and then print my path, it always will say /net/cd-cove-10g/cove/em_processing/abiester. Would you suspect that to be related to this issue?

I think I’ve gotten the relevant lines for the import this time. If I’m incorrect, please let me know:

2023-01-12 12:34:24,790 COMMAND.SCHEDULER    get_gpu_info         INFO     | UPDATING WORKER GPU INFO
2023-01-12 12:34:24,790 COMMAND.JOBS         update_all_job_sizes INFO     | UPDATING ALL JOB SIZES IN 10s
2023-01-12 12:34:24,791 COMMAND.DATA         export_all_projects  INFO     | EXPORTING ALL PROJECTS IN 60s...
2023-01-12 12:38:50,165 COMMAND.DATA         import_project_run   INFO     | Importing project from /cove/em_processing/abiester/ACDS_20220927/P12
2023-01-12 12:38:50,522 COMMAND.DATA         import_project_run   INFO     | Created project P10
2023-01-12 12:38:50,701 COMMAND.DATA         import_project_run   INFO     | Created workspace W1 in P10
2023-01-12 12:38:50,746 COMMAND.DATA         import_project_run   INFO     | Created workspace W2 in P10
2023-01-12 12:38:50,878 COMMAND.DATA         import_project_run   INFO     | Created workspace W3 in P10
2023-01-12 12:38:50,948 COMMAND.DATA         import_project_run   INFO     | Created workspace W4 in P10
2023-01-12 12:38:51,026 COMMAND.DATA         import_project_run   INFO     | Inserting jobs into project...
2023-01-12 12:38:51,187 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:38:51,340 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.14s
2023-01-12 12:38:51,556 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.35s...
2023-01-12 12:38:51,562 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:38:51,751 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.18s...
2023-01-12 12:38:51,771 COMMAND.DATA         import_project_run   INFO     | Imported J1 into P10 in 0.61s...
2023-01-12 12:38:51,862 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:38:51,935 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.07s
2023-01-12 12:38:51,983 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.12s...
2023-01-12 12:38:51,991 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:38:52,088 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 1 streamlogs in 0.09s...
2023-01-12 12:38:52,088 COMMAND.DATA         import_project_run   INFO     | Imported J10 into P10 in 0.28s...
2023-01-12 12:38:52,169 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:38:58,038 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 59 files in 5.85s
2023-01-12 12:38:58,131 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 5.94s...
2023-01-12 12:38:58,145 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:38:59,315 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 45 streamlogs in 1.16s...
2023-01-12 12:38:59,316 COMMAND.DATA         import_project_run   INFO     | Imported J100 into P10 in 7.21s...
2023-01-12 12:38:59,317 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:38:59,317 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.00s
2023-01-12 12:38:59,337 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.02s...
2023-01-12 12:38:59,337 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:38:59,338 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.00s...
2023-01-12 12:38:59,338 COMMAND.DATA         import_project_run   INFO     | Imported J101 into P10 in 0.02s...
2023-01-12 12:38:59,370 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:38:59,454 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.07s
2023-01-12 12:38:59,592 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.21s...
2023-01-12 12:38:59,601 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:38:59,724 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.09s...
2023-01-12 12:38:59,730 COMMAND.DATA         import_project_run   INFO     | Imported J102 into P10 in 0.39s...
2023-01-12 12:38:59,803 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:00,572 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 4 files in 0.77s
2023-01-12 12:39:00,899 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 1.10s...
2023-01-12 12:39:00,916 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:03,130 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 26 streamlogs in 2.20s...
2023-01-12 12:39:03,153 COMMAND.DATA         import_project_run   INFO     | Imported J103 into P10 in 3.37s...
2023-01-12 12:39:03,238 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:03,290 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.03s
2023-01-12 12:39:03,413 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.16s...
2023-01-12 12:39:03,427 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:03,581 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.15s...
2023-01-12 12:39:03,581 COMMAND.DATA         import_project_run   INFO     | Imported J104 into P10 in 0.36s...
2023-01-12 12:39:03,614 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:03,673 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.04s
2023-01-12 12:39:03,749 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.12s...
2023-01-12 12:39:03,760 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:04,156 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 23 streamlogs in 0.40s...
2023-01-12 12:39:04,167 COMMAND.DATA         import_project_run   INFO     | Imported J105 into P10 in 0.57s...
2023-01-12 12:39:04,231 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:04,289 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.05s
2023-01-12 12:39:04,361 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.12s...
2023-01-12 12:39:04,361 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:04,362 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.00s...
2023-01-12 12:39:04,362 COMMAND.DATA         import_project_run   INFO     | Imported J106 into P10 in 0.18s...
2023-01-12 12:39:04,388 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:04,454 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.07s
2023-01-12 12:39:04,720 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.33s...
2023-01-12 12:39:04,733 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:04,827 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.09s...
2023-01-12 12:39:04,841 COMMAND.DATA         import_project_run   INFO     | Imported J107 into P10 in 0.48s...
2023-01-12 12:39:05,015 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:05,218 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 0 files in 0.15s
2023-01-12 12:39:05,561 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 0.50s...
2023-01-12 12:39:05,570 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:05,636 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 0 streamlogs in 0.05s...
2023-01-12 12:39:05,644 COMMAND.DATA         import_project_run   INFO     | Imported J108 into P10 in 0.65s...
2023-01-12 12:39:05,708 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:09,602 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 516 files in 3.81s
2023-01-12 12:39:09,618 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 3.83s...
2023-01-12 12:39:09,618 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:12,000 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 8993 streamlogs in 2.38s...
2023-01-12 12:39:12,001 COMMAND.DATA         import_project_run   INFO     | Imported J109 into P10 in 6.34s...
2023-01-12 12:39:12,024 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:13,322 COMMAND.DATA         import_project_run   INFO     | Done. Uploaded 7 files in 1.30s
2023-01-12 12:39:13,529 COMMAND.DATA         import_project_run   INFO     | Inserted job document in 1.50s...
2023-01-12 12:39:13,539 COMMAND.DATA         import_project_run   INFO     | Inserting streamlogs into jobs...
2023-01-12 12:39:14,259 COMMAND.DATA         import_project_run   INFO     | Done. Inserted 50 streamlogs in 0.67s...
2023-01-12 12:39:14,259 COMMAND.DATA         import_project_run   INFO     | Imported J11 into P10 in 2.26s...
2023-01-12 12:39:14,333 COMMAND.DATA         import_project_run   INFO     | Uploading project image data...
2023-01-12 12:39:14,460 COMMAND.DATA         import_project_run   ERROR    | Unable to import project from /cove/em_processing/abiester/ACDS_20220927/P12
2023-01-12 12:39:14,460 COMMAND.DATA         import_project_run   ERROR    | Traceback (most recent call last):
2023-01-12 12:39:14,460 COMMAND.DATA         import_project_run   ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
2023-01-12 12:39:14,460 COMMAND.DATA         import_project_run   ERROR    |     gridfs_data = pickle.load(openfile)
2023-01-12 12:39:14,460 COMMAND.DATA         import_project_run   ERROR    | EOFError: Ran out of input
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    | Traceback (most recent call last):
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    |     self.target(*self.args)
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    |     gridfs_data = pickle.load(openfile)
2023-01-12 12:39:14,462 COMMAND.MAIN         run                  ERROR    | EOFError: Ran out of input
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1837, in run_with_except_hook
    run_old(*args, **kw)
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
    self.target(*self.args)
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
    gridfs_data = pickle.load(openfile)
EOFError: Ran out of input

The data inside the job directory that alphabetically follows J11 (J110 ?) may be corrupt.
Please can you confirm which job alphabetically succeeds J11 inside that project directory, using the ls command.
You may wish to, in order to see where the (suspected) corruption first occurred,

  • export that putatively corrupt job from the “source” CryoSPARC instance and, therefore, from the “source” job directory
  • copy the exported job’s directory hierarchy for subsequent import to the “target” instance
  • import the job to the “target” instance

Do you observe any errors during that trial?

1 Like

I went through that trial, it does appear that J110 is the subsequent job alphabetically. I exported that job and rsync’d the exported job’s directory hierarchy to the import folder in the “target” instance. Then I attempted to import the job in the target instance. Below is the error message I got (from the command core logs). Do you have any insights on what I should do next?

2023-01-19 16:36:23,542 COMMAND.DATA         import_job           INFO     | Importing job into P10 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
2023-01-19 16:36:23,593 COMMAND.DATA         import_job           INFO     | Inserted job document into P10 J110 in 0.03s...
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_job_run
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    | Traceback (most recent call last):
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |     self.target(*self.args)
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4735, in import_job_run
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |     rc.import_job(project_uid, new_uid, abs_path_export_job_dir, logger=lambda msg: job_send_streamlog(project_uid, new_uid, msg), notification_id=notification_id)
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1329, in import_job
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    |     assert False, "[IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of %s %s from %s"%(project_uid, job_uid, abs_path_export_job_dir)
2023-01-19 16:38:52,493 COMMAND.MAIN         run                  ERROR    | AssertionError: [IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of P10 J110 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1837, in run_with_except_hook
    run_old(*args, **kw)
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
    self.target(*self.args)
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4735, in import_job_run
    rc.import_job(project_uid, new_uid, abs_path_export_job_dir, logger=lambda msg: job_send_streamlog(project_uid, new_uid, msg), notification_id=notification_id)
  File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1329, in import_job
    assert False, "[IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of %s %s from %s"%(project_uid, job_uid, abs_path_export_job_dir)
AssertionError: [IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of P10 J110 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine

Could there be broken links? Please can you run

find /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine -type l -ls

Yes, in fact the import is full of broken links. I don’t think there is an easy way to re-do those links because there is no equivalent (curated) list of files contained anywhere else in the filesystem. The folders contain subsets of particles from an extraction job, but they link to a folder containing all of the files from that extraction job. So I’m not sure how to re-link the files in any simple way. Do you know what I should do next?

I am having a hard time imagining the state of your filesystems beyond “it’s complicated”. It is therefore possible that the following suggestions do not apply to your situation:
If you have access to the original copy of the project directory and links in that location refer to the actual files, you could use (while requiring a lot of additional space) an archiving parameter that dereferences links, such as gnu-tar’s --dereference option in conjunction with --create or the rsync --copy-links option.
If you have access to the actual files, but the way symlinks are broken follows a pattern, you may find this use case, and the functions mentioned therein, interesting.

With the help of our tech person, I was able to remake the symbolic links, so they now point to an appropriate path and are not broken.

I went back to cryosparc and deleted the failed J110 import, then went to re-do the import. Overall, it seemed to work, but it changed the job number to J111, which I imagine may be problematic.

What would you recommend doing next? Command core log below for deleting previous and re-doing import:

2023-02-01 16:01:33,539 COMMAND.DATA         delete_job           INFO     | Deleting P10 J110
2023-02-01 16:01:33,579 COMMAND.DATA         delete_job           INFO     |      Now clearing job...
2023-02-01 16:02:07,138 COMMAND.DATA         dump_job_database    INFO     | Request to export P10 J110
2023-02-01 16:02:07,137 COMMAND.CORE         run                  INFO     | Received task dump_job_database with 2 args and 0 kwargs
2023-02-01 16:02:07,392 COMMAND.DATA         dump_job_database    INFO     |    Exporting job to /cove/em_processing/abiester/ACDS_20220927/P12/J110
2023-02-01 16:02:07,497 COMMAND.DATA         dump_job_database    INFO     |    Exporting all of job's images in the database to /cove/em_processing/abiester/ACDS_20220927/P12/J110/gridfs_data...
2023-02-01 16:02:07,543 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 0 images in 0.05s
2023-02-01 16:02:07,603 COMMAND.DATA         dump_job_database    INFO     |    Exporting all job's streamlog events...
2023-02-01 16:02:07,785 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 1 files in 0.17s
2023-02-01 16:02:07,785 COMMAND.DATA         dump_job_database    INFO     |    Exporting job metafile...
2023-02-01 16:02:07,880 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported in 0.09s
2023-02-01 16:02:07,923 COMMAND.DATA         dump_job_database    INFO     |    Updating job manifest...
2023-02-01 16:02:08,236 COMMAND.DATA         dump_job_database    INFO     |    Done. Updated in 0.31s
2023-02-01 16:02:08,236 COMMAND.DATA         dump_job_database    INFO     | Exported P10 J110 in 1.10s
2023-02-01 16:02:08,268 COMMAND.CORE         run                  INFO     | Completed task in 1.12996244430542 seconds
2023-02-01 16:02:08,268 COMMAND.CORE         run                  INFO     | Received task dump_job_database with 2 args and 0 kwargs
2023-02-01 16:02:08,268 COMMAND.DATA         dump_job_database    INFO     | Request to export P10 J110
2023-02-01 16:02:08,566 COMMAND.DATA         dump_job_database    INFO     |    Exporting job to /cove/em_processing/abiester/ACDS_20220927/P12/J110
2023-02-01 16:02:08,707 COMMAND.DATA         dump_job_database    INFO     |    Exporting all of job's images in the database to /cove/em_processing/abiester/ACDS_20220927/P12/J110/gridfs_data...
2023-02-01 16:02:08,777 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 0 images in 0.05s
2023-02-01 16:02:08,786 COMMAND.DATA         dump_job_database    INFO     |    Exporting all job's streamlog events...
2023-02-01 16:02:09,105 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported 1 files in 0.27s
2023-02-01 16:02:09,174 COMMAND.DATA         dump_job_database    INFO     |    Exporting job metafile...
2023-02-01 16:02:09,274 COMMAND.DATA         dump_job_database    INFO     |    Done. Exported in 0.07s
2023-02-01 16:02:09,287 COMMAND.DATA         dump_job_database    INFO     |    Updating job manifest...
2023-02-01 16:02:10,059 COMMAND.DATA         dump_job_database    INFO     |    Done. Updated in 0.72s
2023-02-01 16:02:10,059 COMMAND.DATA         dump_job_database    INFO     | Exported P10 J110 in 1.79s
2023-02-01 16:02:10,088 COMMAND.CORE         run                  INFO     | Completed task in 1.8202462196350098 seconds
2023-02-01 16:02:10,089 COMMAND.CORE         run                  INFO     | Received task layout_tree with 4 args and 0 kwargs
2023-02-01 16:02:10,286 COMMAND.DATA         import_job           INFO     | Importing job into P10 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    | JSONRPC ERROR at layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    |     res = func(*args, **kwargs)
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    |     treegap=treegap)
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN         wrapper              ERROR    | KeyError: 'J99'
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | Encountered exception while running background task
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |     res = func(*args, **kwargs)
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |     treegap=treegap)
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | KeyError: 'J99'
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | 
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | The above exception was the direct cause of the following exception:
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | 
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 926, in cryosparc_command.command_core.core.background_tasks_worker
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    |     raise servererr from e
2023-02-01 16:02:10,459 COMMAND.CORE         run                  ERROR    | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:02:10,515 COMMAND.CORE         run                  INFO     | Received task layout_tree with 4 args and 0 kwargs
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    | JSONRPC ERROR at layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    |     res = func(*args, **kwargs)
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    |     treegap=treegap)
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN         wrapper              ERROR    | KeyError: 'J99'
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | Encountered exception while running background task
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |     res = func(*args, **kwargs)
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |     treegap=treegap)
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | KeyError: 'J99'
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | 
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | The above exception was the direct cause of the following exception:
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | 
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | Traceback (most recent call last):
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 926, in cryosparc_command.command_core.core.background_tasks_worker
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    |     raise servererr from e
2023-02-01 16:02:10,673 COMMAND.CORE         run                  ERROR    | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:02:11,487 COMMAND.DATA         import_job           INFO     | Inserted job document into P10 J111 in 0.33s...
2023-02-01 16:05:51,232 COMMAND.CORE         heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/0ec2e09c-3da8-11ed-b1f1-631efd722674 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1d94cbd6d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-02-01 16:05:51,233 COMMAND.CORE         heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    | JSONRPC ERROR at layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    | Traceback (most recent call last):
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    |     res = func(*args, **kwargs)
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    |     treegap=treegap)
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN         wrapper              ERROR    | KeyError: 'J99'
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | Failed laying out tree in P10
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | Traceback (most recent call last):
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |     res = func(*args, **kwargs)
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |     treegap=treegap)
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |   File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | KeyError: 'J99'
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | 
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | The above exception was the direct cause of the following exception:
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | 
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | Traceback (most recent call last):
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4742, in import_job_run
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |     layout_tree(project_uid, None, None, project_uid)
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |   File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  |     raise servererr from e
2023-02-01 16:11:16,310 COMMAND.DATA         import_job_run       WARNING  | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:11:16,311 COMMAND.DATA         import_job_run       INFO     | Imported P10 J111 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine in 545.15s

What steps were involved? Did you separately export J110 from the “source” instance before trying to import this single job?

I used the exported J110 described in this message, that came from the source instance, to try to import the job.

After the failed import due to the symbolic links, in that same imported J110, I fixed the links. Then, in the GUI, I deleted the J110 where the import failed. Then, I used the “Import Job” function in the GUI to import J110 with the fixed links. This import was successful, but the job number changed to J111.

Jobs individually imported, as opposed to included in a project import, are not guaranteed to maintain their original job uid, but are assigned the next available job uid, given jobs (not necessarily from import) already existing in the destination project.

Okay, I see. What would you recommend that I do in terms of the larger issue? Does this result suggest to you that J110 was in fact corrupted? How can I complete the project import?

Aside from adjusting symbolic links, was the J110 directory altered following the rsync that you described earlier: Import project after migrating files between filesystems (on different cryosparc installations) - #3 by abiester? Did you adjust the links in that copy or in a new copy of the J110 directory?