The cryoSPARC instances are both v3.3.2. The command I used for the migration was:
rsync -avS abiester@satori-login-003.mit.edu:/nobackup/users/abiester/cryosparc/ACDS_20220927 ./
I ran that command standing in this directory:
/net/cd-cove-10g/cove/em_processing/abiester
I believe that path holds a complete copy of the project directory based on the output of the rsync command.
May I ask:
Did you inspect the output of
cryosparcm log command_core
for error messages related to this attempted import?
Did ACDS_20220927 contain and did you attempt to import any other project directories in addition to P12?
Is the difference between
and
consistent with any filesystem unmount-mount sequence(s) that may have followed the rsync operation?
Here is the output of the command core log:
The directory ACDS_20220927 does not contain projects other than P12. I have tried multiple times to complete the import of P12 and have also tried importing other projects unsuccessfully, but those were in different directories moved with different rsync commands.
I don’t know what a filesystem unmount-mount sequence is. Can you explain further to help me answer that question?
Yes, that is the procedure I have been trying.
I was referring to (un-)mount in this context, and
wanted to understand why, in your case, /net/cd-cove-10g was included in your working directory 's path during rsync, but not included in the path definition during import.
Please can you post the command_core log excerpt from above as text, and include (another 20 or so) additional lines that precede that excerpt and are relevant to the import attempt.
Oh I see, yes /net/cd-cove-10g/cove and /cove lead to the same place in the filesystem. I’m not entirely sure if this redundancy is related to (un-)mounting. When I navigate to /cove/em_processing/abiester and then print my path, it always will say /net/cd-cove-10g/cove/em_processing/abiester. Would you suspect that to be related to this issue?
I think I’ve gotten the relevant lines for the import this time. If I’m incorrect, please let me know:
2023-01-12 12:34:24,790 COMMAND.SCHEDULER get_gpu_info INFO | UPDATING WORKER GPU INFO
2023-01-12 12:34:24,790 COMMAND.JOBS update_all_job_sizes INFO | UPDATING ALL JOB SIZES IN 10s
2023-01-12 12:34:24,791 COMMAND.DATA export_all_projects INFO | EXPORTING ALL PROJECTS IN 60s...
2023-01-12 12:38:50,165 COMMAND.DATA import_project_run INFO | Importing project from /cove/em_processing/abiester/ACDS_20220927/P12
2023-01-12 12:38:50,522 COMMAND.DATA import_project_run INFO | Created project P10
2023-01-12 12:38:50,701 COMMAND.DATA import_project_run INFO | Created workspace W1 in P10
2023-01-12 12:38:50,746 COMMAND.DATA import_project_run INFO | Created workspace W2 in P10
2023-01-12 12:38:50,878 COMMAND.DATA import_project_run INFO | Created workspace W3 in P10
2023-01-12 12:38:50,948 COMMAND.DATA import_project_run INFO | Created workspace W4 in P10
2023-01-12 12:38:51,026 COMMAND.DATA import_project_run INFO | Inserting jobs into project...
2023-01-12 12:38:51,187 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:38:51,340 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.14s
2023-01-12 12:38:51,556 COMMAND.DATA import_project_run INFO | Inserted job document in 0.35s...
2023-01-12 12:38:51,562 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:38:51,751 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.18s...
2023-01-12 12:38:51,771 COMMAND.DATA import_project_run INFO | Imported J1 into P10 in 0.61s...
2023-01-12 12:38:51,862 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:38:51,935 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.07s
2023-01-12 12:38:51,983 COMMAND.DATA import_project_run INFO | Inserted job document in 0.12s...
2023-01-12 12:38:51,991 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:38:52,088 COMMAND.DATA import_project_run INFO | Done. Inserted 1 streamlogs in 0.09s...
2023-01-12 12:38:52,088 COMMAND.DATA import_project_run INFO | Imported J10 into P10 in 0.28s...
2023-01-12 12:38:52,169 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:38:58,038 COMMAND.DATA import_project_run INFO | Done. Uploaded 59 files in 5.85s
2023-01-12 12:38:58,131 COMMAND.DATA import_project_run INFO | Inserted job document in 5.94s...
2023-01-12 12:38:58,145 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:38:59,315 COMMAND.DATA import_project_run INFO | Done. Inserted 45 streamlogs in 1.16s...
2023-01-12 12:38:59,316 COMMAND.DATA import_project_run INFO | Imported J100 into P10 in 7.21s...
2023-01-12 12:38:59,317 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:38:59,317 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.00s
2023-01-12 12:38:59,337 COMMAND.DATA import_project_run INFO | Inserted job document in 0.02s...
2023-01-12 12:38:59,337 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:38:59,338 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.00s...
2023-01-12 12:38:59,338 COMMAND.DATA import_project_run INFO | Imported J101 into P10 in 0.02s...
2023-01-12 12:38:59,370 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:38:59,454 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.07s
2023-01-12 12:38:59,592 COMMAND.DATA import_project_run INFO | Inserted job document in 0.21s...
2023-01-12 12:38:59,601 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:38:59,724 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.09s...
2023-01-12 12:38:59,730 COMMAND.DATA import_project_run INFO | Imported J102 into P10 in 0.39s...
2023-01-12 12:38:59,803 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:00,572 COMMAND.DATA import_project_run INFO | Done. Uploaded 4 files in 0.77s
2023-01-12 12:39:00,899 COMMAND.DATA import_project_run INFO | Inserted job document in 1.10s...
2023-01-12 12:39:00,916 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:03,130 COMMAND.DATA import_project_run INFO | Done. Inserted 26 streamlogs in 2.20s...
2023-01-12 12:39:03,153 COMMAND.DATA import_project_run INFO | Imported J103 into P10 in 3.37s...
2023-01-12 12:39:03,238 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:03,290 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.03s
2023-01-12 12:39:03,413 COMMAND.DATA import_project_run INFO | Inserted job document in 0.16s...
2023-01-12 12:39:03,427 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:03,581 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.15s...
2023-01-12 12:39:03,581 COMMAND.DATA import_project_run INFO | Imported J104 into P10 in 0.36s...
2023-01-12 12:39:03,614 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:03,673 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.04s
2023-01-12 12:39:03,749 COMMAND.DATA import_project_run INFO | Inserted job document in 0.12s...
2023-01-12 12:39:03,760 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:04,156 COMMAND.DATA import_project_run INFO | Done. Inserted 23 streamlogs in 0.40s...
2023-01-12 12:39:04,167 COMMAND.DATA import_project_run INFO | Imported J105 into P10 in 0.57s...
2023-01-12 12:39:04,231 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:04,289 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.05s
2023-01-12 12:39:04,361 COMMAND.DATA import_project_run INFO | Inserted job document in 0.12s...
2023-01-12 12:39:04,361 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:04,362 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.00s...
2023-01-12 12:39:04,362 COMMAND.DATA import_project_run INFO | Imported J106 into P10 in 0.18s...
2023-01-12 12:39:04,388 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:04,454 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.07s
2023-01-12 12:39:04,720 COMMAND.DATA import_project_run INFO | Inserted job document in 0.33s...
2023-01-12 12:39:04,733 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:04,827 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.09s...
2023-01-12 12:39:04,841 COMMAND.DATA import_project_run INFO | Imported J107 into P10 in 0.48s...
2023-01-12 12:39:05,015 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:05,218 COMMAND.DATA import_project_run INFO | Done. Uploaded 0 files in 0.15s
2023-01-12 12:39:05,561 COMMAND.DATA import_project_run INFO | Inserted job document in 0.50s...
2023-01-12 12:39:05,570 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:05,636 COMMAND.DATA import_project_run INFO | Done. Inserted 0 streamlogs in 0.05s...
2023-01-12 12:39:05,644 COMMAND.DATA import_project_run INFO | Imported J108 into P10 in 0.65s...
2023-01-12 12:39:05,708 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:09,602 COMMAND.DATA import_project_run INFO | Done. Uploaded 516 files in 3.81s
2023-01-12 12:39:09,618 COMMAND.DATA import_project_run INFO | Inserted job document in 3.83s...
2023-01-12 12:39:09,618 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:12,000 COMMAND.DATA import_project_run INFO | Done. Inserted 8993 streamlogs in 2.38s...
2023-01-12 12:39:12,001 COMMAND.DATA import_project_run INFO | Imported J109 into P10 in 6.34s...
2023-01-12 12:39:12,024 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:13,322 COMMAND.DATA import_project_run INFO | Done. Uploaded 7 files in 1.30s
2023-01-12 12:39:13,529 COMMAND.DATA import_project_run INFO | Inserted job document in 1.50s...
2023-01-12 12:39:13,539 COMMAND.DATA import_project_run INFO | Inserting streamlogs into jobs...
2023-01-12 12:39:14,259 COMMAND.DATA import_project_run INFO | Done. Inserted 50 streamlogs in 0.67s...
2023-01-12 12:39:14,259 COMMAND.DATA import_project_run INFO | Imported J11 into P10 in 2.26s...
2023-01-12 12:39:14,333 COMMAND.DATA import_project_run INFO | Uploading project image data...
2023-01-12 12:39:14,460 COMMAND.DATA import_project_run ERROR | Unable to import project from /cove/em_processing/abiester/ACDS_20220927/P12
2023-01-12 12:39:14,460 COMMAND.DATA import_project_run ERROR | Traceback (most recent call last):
2023-01-12 12:39:14,460 COMMAND.DATA import_project_run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
2023-01-12 12:39:14,460 COMMAND.DATA import_project_run ERROR | gridfs_data = pickle.load(openfile)
2023-01-12 12:39:14,460 COMMAND.DATA import_project_run ERROR | EOFError: Ran out of input
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | POST-RESPONSE-THREAD ERROR at import_project_run
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | Traceback (most recent call last):
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | self.target(*self.args)
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | gridfs_data = pickle.load(openfile)
2023-01-12 12:39:14,462 COMMAND.MAIN run ERROR | EOFError: Ran out of input
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1837, in run_with_except_hook
run_old(*args, **kw)
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
self.target(*self.args)
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 3590, in import_project_run
gridfs_data = pickle.load(openfile)
EOFError: Ran out of input
The data inside the job directory that alphabetically follows J11 (J110 ?) may be corrupt.
Please can you confirm which job alphabetically succeeds J11 inside that project directory, using the ls command.
You may wish to, in order to see where the (suspected) corruption first occurred,
Do you observe any errors during that trial?
I went through that trial, it does appear that J110 is the subsequent job alphabetically. I exported that job and rsync’d the exported job’s directory hierarchy to the import folder in the “target” instance. Then I attempted to import the job in the target instance. Below is the error message I got (from the command core logs). Do you have any insights on what I should do next?
2023-01-19 16:36:23,542 COMMAND.DATA import_job INFO | Importing job into P10 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
2023-01-19 16:36:23,593 COMMAND.DATA import_job INFO | Inserted job document into P10 J110 in 0.03s...
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | POST-RESPONSE-THREAD ERROR at import_job_run
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | Traceback (most recent call last):
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | self.target(*self.args)
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4735, in import_job_run
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | rc.import_job(project_uid, new_uid, abs_path_export_job_dir, logger=lambda msg: job_send_streamlog(project_uid, new_uid, msg), notification_id=notification_id)
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1329, in import_job
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | assert False, "[IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of %s %s from %s"%(project_uid, job_uid, abs_path_export_job_dir)
2023-01-19 16:38:52,493 COMMAND.MAIN run ERROR | AssertionError: [IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of P10 J110 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1837, in run_with_except_hook
run_old(*args, **kw)
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 177, in run
self.target(*self.args)
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4735, in import_job_run
rc.import_job(project_uid, new_uid, abs_path_export_job_dir, logger=lambda msg: job_send_streamlog(project_uid, new_uid, msg), notification_id=notification_id)
File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1329, in import_job
assert False, "[IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of %s %s from %s"%(project_uid, job_uid, abs_path_export_job_dir)
AssertionError: [IMPORT_JOB] : Unable to find data referenced by cs files. Aborting import of P10 J110 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
Could there be broken links? Please can you run
find /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine -type l -ls
Yes, in fact the import is full of broken links. I don’t think there is an easy way to re-do those links because there is no equivalent (curated) list of files contained anywhere else in the filesystem. The folders contain subsets of particles from an extraction job, but they link to a folder containing all of the files from that extraction job. So I’m not sure how to re-link the files in any simple way. Do you know what I should do next?
I am having a hard time imagining the state of your filesystems beyond “it’s complicated”. It is therefore possible that the following suggestions do not apply to your situation:
If you have access to the original copy of the project directory and links in that location refer to the actual files, you could use (while requiring a lot of additional space) an archiving parameter that dereferences links, such as gnu-tar’s --dereference option in conjunction with --create or the rsync --copy-links option.
If you have access to the actual files, but the way symlinks are broken follows a pattern, you may find this use case, and the functions mentioned therein, interesting.
With the help of our tech person, I was able to remake the symbolic links, so they now point to an appropriate path and are not broken.
I went back to cryosparc and deleted the failed J110 import, then went to re-do the import. Overall, it seemed to work, but it changed the job number to J111, which I imagine may be problematic.
What would you recommend doing next? Command core log below for deleting previous and re-doing import:
2023-02-01 16:01:33,539 COMMAND.DATA delete_job INFO | Deleting P10 J110
2023-02-01 16:01:33,579 COMMAND.DATA delete_job INFO | Now clearing job...
2023-02-01 16:02:07,138 COMMAND.DATA dump_job_database INFO | Request to export P10 J110
2023-02-01 16:02:07,137 COMMAND.CORE run INFO | Received task dump_job_database with 2 args and 0 kwargs
2023-02-01 16:02:07,392 COMMAND.DATA dump_job_database INFO | Exporting job to /cove/em_processing/abiester/ACDS_20220927/P12/J110
2023-02-01 16:02:07,497 COMMAND.DATA dump_job_database INFO | Exporting all of job's images in the database to /cove/em_processing/abiester/ACDS_20220927/P12/J110/gridfs_data...
2023-02-01 16:02:07,543 COMMAND.DATA dump_job_database INFO | Done. Exported 0 images in 0.05s
2023-02-01 16:02:07,603 COMMAND.DATA dump_job_database INFO | Exporting all job's streamlog events...
2023-02-01 16:02:07,785 COMMAND.DATA dump_job_database INFO | Done. Exported 1 files in 0.17s
2023-02-01 16:02:07,785 COMMAND.DATA dump_job_database INFO | Exporting job metafile...
2023-02-01 16:02:07,880 COMMAND.DATA dump_job_database INFO | Done. Exported in 0.09s
2023-02-01 16:02:07,923 COMMAND.DATA dump_job_database INFO | Updating job manifest...
2023-02-01 16:02:08,236 COMMAND.DATA dump_job_database INFO | Done. Updated in 0.31s
2023-02-01 16:02:08,236 COMMAND.DATA dump_job_database INFO | Exported P10 J110 in 1.10s
2023-02-01 16:02:08,268 COMMAND.CORE run INFO | Completed task in 1.12996244430542 seconds
2023-02-01 16:02:08,268 COMMAND.CORE run INFO | Received task dump_job_database with 2 args and 0 kwargs
2023-02-01 16:02:08,268 COMMAND.DATA dump_job_database INFO | Request to export P10 J110
2023-02-01 16:02:08,566 COMMAND.DATA dump_job_database INFO | Exporting job to /cove/em_processing/abiester/ACDS_20220927/P12/J110
2023-02-01 16:02:08,707 COMMAND.DATA dump_job_database INFO | Exporting all of job's images in the database to /cove/em_processing/abiester/ACDS_20220927/P12/J110/gridfs_data...
2023-02-01 16:02:08,777 COMMAND.DATA dump_job_database INFO | Done. Exported 0 images in 0.05s
2023-02-01 16:02:08,786 COMMAND.DATA dump_job_database INFO | Exporting all job's streamlog events...
2023-02-01 16:02:09,105 COMMAND.DATA dump_job_database INFO | Done. Exported 1 files in 0.27s
2023-02-01 16:02:09,174 COMMAND.DATA dump_job_database INFO | Exporting job metafile...
2023-02-01 16:02:09,274 COMMAND.DATA dump_job_database INFO | Done. Exported in 0.07s
2023-02-01 16:02:09,287 COMMAND.DATA dump_job_database INFO | Updating job manifest...
2023-02-01 16:02:10,059 COMMAND.DATA dump_job_database INFO | Done. Updated in 0.72s
2023-02-01 16:02:10,059 COMMAND.DATA dump_job_database INFO | Exported P10 J110 in 1.79s
2023-02-01 16:02:10,088 COMMAND.CORE run INFO | Completed task in 1.8202462196350098 seconds
2023-02-01 16:02:10,089 COMMAND.CORE run INFO | Received task layout_tree with 4 args and 0 kwargs
2023-02-01 16:02:10,286 COMMAND.DATA import_job INFO | Importing job into P10 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | JSONRPC ERROR at layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | res = func(*args, **kwargs)
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | treegap=treegap)
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,121 COMMAND.MAIN wrapper ERROR | KeyError: 'J99'
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | Encountered exception while running background task
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | res = func(*args, **kwargs)
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | treegap=treegap)
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | KeyError: 'J99'
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR |
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | The above exception was the direct cause of the following exception:
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR |
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 926, in cryosparc_command.command_core.core.background_tasks_worker
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | raise servererr from e
2023-02-01 16:02:10,459 COMMAND.CORE run ERROR | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:02:10,515 COMMAND.CORE run INFO | Received task layout_tree with 4 args and 0 kwargs
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | JSONRPC ERROR at layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | res = func(*args, **kwargs)
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | treegap=treegap)
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,606 COMMAND.MAIN wrapper ERROR | KeyError: 'J99'
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | Encountered exception while running background task
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | res = func(*args, **kwargs)
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | treegap=treegap)
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | KeyError: 'J99'
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR |
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | The above exception was the direct cause of the following exception:
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR |
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | Traceback (most recent call last):
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 926, in cryosparc_command.command_core.core.background_tasks_worker
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | raise servererr from e
2023-02-01 16:02:10,673 COMMAND.CORE run ERROR | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:02:11,487 COMMAND.DATA import_job INFO | Inserted job document into P10 J111 in 0.33s...
2023-02-01 16:05:51,232 COMMAND.CORE heartbeat_manager ERROR | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/0ec2e09c-3da8-11ed-b1f1-631efd722674 (Caused by NewConnectionError('<urllib3.connection.HTTPSConnection object at 0x7f1d94cbd6d0>: Failed to establish a new connection: [Errno -2] Name or service not known'))
2023-02-01 16:05:51,233 COMMAND.CORE heartbeat_manager WARNING | Error connecting to cryoSPARC license server during instance heartbeat.
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | JSONRPC ERROR at layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | Traceback (most recent call last):
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | res = func(*args, **kwargs)
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | treegap=treegap)
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:11:16,308 COMMAND.MAIN wrapper ERROR | KeyError: 'J99'
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | Failed laying out tree in P10
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | Traceback (most recent call last):
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 146, in wrapper
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | res = func(*args, **kwargs)
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 5371, in layout_tree
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | treegap=treegap)
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | File "cryosparc_master/cryosparc_command/command_core/core.py", line 470, in cryosparc_command.command_core.core.layout_tree
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | KeyError: 'J99'
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING |
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | The above exception was the direct cause of the following exception:
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING |
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | Traceback (most recent call last):
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4742, in import_job_run
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | layout_tree(project_uid, None, None, project_uid)
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | File "/net/cd-cove2-10g/home/drennan/abiester/cryosparc/cd-kraken/62576/cryosparc_master/cryosparc_command/command_core/__init__.py", line 153, in wrapper
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | raise servererr from e
2023-02-01 16:11:16,310 COMMAND.DATA import_job_run WARNING | flask_jsonrpc.exceptions.ServerError
2023-02-01 16:11:16,311 COMMAND.DATA import_job_run INFO | Imported P10 J111 from /cove/em_processing/abiester/ACDS_20220927/P12/imports/jobs/P12_J110_hetero_refine in 545.15s
What steps were involved? Did you separately export J110 from the “source” instance before trying to import this single job?
I used the exported J110 described in this message, that came from the source instance, to try to import the job.
After the failed import due to the symbolic links, in that same imported J110, I fixed the links. Then, in the GUI, I deleted the J110 where the import failed. Then, I used the “Import Job” function in the GUI to import J110 with the fixed links. This import was successful, but the job number changed to J111.
Jobs individually imported, as opposed to included in a project import, are not guaranteed to maintain their original job uid, but are assigned the next available job uid, given jobs (not necessarily from import) already existing in the destination project.
Okay, I see. What would you recommend that I do in terms of the larger issue? Does this result suggest to you that J110 was in fact corrupted? How can I complete the project import?
Aside from adjusting symbolic links, was the J110 directory altered following the rsync that you described earlier: Import project after migrating files between filesystems (on different cryosparc installations) - #3 by abiester? Did you adjust the links in that copy or in a new copy of the J110 directory?
The J110 directory that imported successfully was a different copy of J110 than the copy described on Jan 13. The original copy of J110 described on Jan 13 did not contain broken links. The links were created when I completed the export of J110 in the “source” cryosparc instance.
You could again attempt to import the entire project:
rsync -S option.Alternatively, you could import individual jobs as needed into an existing project as needed. This method has the advantage that the import of all a project’s jobs would not be interrupted by an error that may occur for a job that may not be of interest anyway.