Missing jobs after transfering project

Data Management
1

This topic has been raised before, but I could not find a solution, so I am asking for help. My situation is similar to this post.

We had a workstation mounted to a shared filesystem that crashed. I then installed cryosparc 4.7 on a different workstation that is mounted to the same shared filesystem, and then I deleted the cs.lock file in my project folder and attached it to the new one. I did not move those files. The project is over 20 TB in size and has 867 jobs, but when I attached it, there are only 64 jobs there. All the workspaces are present, but some of them have no jobs. When I originally attached the project, there was a loading bar for about 30 minutes and then it was marked complete. Others have noted that a project this big can take a long time to attach, so I waited for 2 days now, but there are still only 64 jobs visible. Can you please help me get my project back?

I also ran the command cryosparcm filterlog command_core -l ERROR and this was the result:

[marcell@msg-app4 ~]$ cryosparcm filterlog command_core -l ERROR
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
2025-05-09 13:52:29,359 import_project_run   ERROR    | Traceback (most recent call last):
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:29,359 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:29,359 import_project_run   ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return loads(fp.read(),
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:29,359 import_project_run   ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:29,359 import_project_run   ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 13:52:32,247 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-05-09 13:52:32,247 run                  ERROR    | Traceback (most recent call last):
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-05-09 13:52:32,247 run                  ERROR    |     self.target(*self.args)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:32,247 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:32,247 run                  ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:32,247 run                  ERROR    |     return loads(fp.read(),
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:32,247 run                  ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:32,247 run                  ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:32,247 run                  ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-12 19:44:15,145 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:45:25,345 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:49:40,454 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:50:50,627 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2

@Marcell Please can you inspect the unfiltered command_core log for additional (including non-ERROR) lines that may reveal additional details, such as a specific job ID, relevant to the JSONDecodeError around 13:52 on May 9. You can browse the log with the command

cryosparcm log command_core | less

or try

cryosparcm log command_core | grep '2025-05-09 13:52:'
3

Thanks for looking into this. I think this is the relevant piece. I included a few jobs that were successfully loaded as well in case that helps.

2025-05-09 13:52:28,394 import_jobs          INFO     | Inserted job document in 0.67s...
2025-05-09 13:52:28,394 import_jobs          INFO     | Inserting streamlogs into jobs...
2025-05-09 13:52:28,561 import_jobs          INFO     | Done. Inserted 299 streamlogs in 0.17s...
2025-05-09 13:52:28,561 import_jobs          INFO     | Imported J165 into P1 in 0.83s...
2025-05-09 13:52:28,566 import_jobs          INFO     | Uploading image data for J166...
2025-05-09 13:52:29,282 import_jobs          INFO     | Done. Uploaded 197 files in 0.72s
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
2025-05-09 13:52:29,359 import_project_run   ERROR    | Traceback (most recent call last):
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:29,359 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:29,359 import_project_run   ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return loads(fp.read(),
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:29,359 import_project_run   ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:29,359 import_project_run   ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 13:52:29,383 dump_project         INFO     | Exporting project P1
2025-05-09 13:52:29,417 dump_project         INFO     | Exported project P1 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.03s
2025-05-09 13:52:32,247 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-05-09 13:52:32,247 run                  ERROR    | Traceback (most recent call last):
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-05-09 13:52:32,247 run                  ERROR    |     self.target(*self.args)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:32,247 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:32,247 run                  ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:32,247 run                  ERROR    |     return loads(fp.read(),
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:32,247 run                  ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:32,247 run                  ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:32,247 run                  ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2306, in run_with_except_hook
    run_old(*args, **kw)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
    job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)

Traceback (most recent call last):
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2306, in run_with_except_hook
    run_old(*args, **kw)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
    job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 14:44:14,610 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-09 15:04:35,935 update_project_size  INFO     | Starting PostResponseThread for update project size for P1
2025-05-09 15:04:35,937 update_project_size_run INFO     | Beginning update project size for P1
2025-05-09 15:37:20 info                 INFO     | Handling signal: quit
2025-05-09 15:37:20 info                 INFO     | Worker exiting (pid: 1193651)
4

It seems to me like it’s getting stuck at J166 or J167 (I really don’t know what I am talking about though). Would it be helpful if I uploaded the .json files for those jobs?

5

Possibly. Do you have a way of sharing the files? Otherwise, I can send you a direct message about sharing arrangements.

6

Here is a folder with some json files. I put in the project, workspace and job_manifest from the project level, and I added the json file from jobs 166 and 167. I renamed them, in their respective folders they are still named job.json.

This download link only seems to work if it’s copy-pasted into the browser, let me know if there is an issue.

https://limewire.com/d/TgAcG#OehdmTR3c4

7

Thanks @Marcell I fetched the files.

1 Like
8

Have there been multiple attempts to attach the project to the newly installed, v4.7 CryoSPARC instance?

9

I did try it a second time. After the first attempt, I detached, then reattached. In both cases, it got stuck on the same job.

10

Was the second attempt performed before or after

2025-05-09 13:52:28,566 import_jobs          INFO     | Uploading image data for J166...
2025-05-09 13:52:29,282 import_jobs          INFO     | Done. Uploaded 197 files in 0.72s
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
11

It was performed after that, i.e. this was from the first attempt. I just looked at the log from the second attempt, and there may be another clue:

2025-05-12 13:36:35,150 import_jobs          INFO     | Uploading image data for J165...
2025-05-12 13:36:35,656 import_jobs          INFO     | Done. Uploaded 176 files in 0.51s
2025-05-12 13:36:35,661 import_jobs          INFO     | Inserted job document in 0.51s...
2025-05-12 13:36:35,661 import_jobs          INFO     | Inserting streamlogs into jobs...
2025-05-12 13:36:35,790 import_jobs          INFO     | Done. Inserted 299 streamlogs in 0.13s...
2025-05-12 13:36:35,790 import_jobs          INFO     | Imported J165 into P2 in 0.64s...
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J159 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J159 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J148 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J148 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J138 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J138 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J129 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J129 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J119 parents include non-existent job J77
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J119 parents include non-existent job J99
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J100 parents include non-existent job J99
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J15 parents include non-existent job J5
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J10 parents include non-existent job J9
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J127 parents include non-existent job J21
2025-05-12 13:36:36,732 import_project_run   WARNING  | Failed laying out tree in P2: 'J5'
2025-05-12 13:36:36,733 import_project_run   WARNING  | Imported project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2 as P2 in 3976.71s with errors.
2025-05-12 13:36:36,760 dump_project         INFO     | Exporting project P2
2025-05-12 13:36:36,764 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 13:36:36,764 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 14:03:16,141 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 14:10:30,089 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-12 14:10:30,107 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 14:12:30,633 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-12 14:12:30,697 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 15:03:17,392 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 15:25:53,512 dump_project         INFO     | Exporting project P2
2025-05-12 15:25:53,514 dump_project         INFO     | Exporting project P1
2025-05-12 15:25:53,516 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:25:53,516 update_project_size_run INFO     | Finished updating project size for P2 in 6556.75s
2025-05-12 15:25:53,517 dump_project         INFO     | Exported project P1 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:25:53,517 update_project_size_run INFO     | Finished updating project size for P1 in 12152.48s
2025-05-12 15:25:53,517 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 1 projects updated)
2025-05-12 15:35:47,875 dump_project         INFO     | Exporting project P2
2025-05-12 15:35:47,878 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:35:47,878 update_project_size_run INFO     | Finished updating project size for P2 in 4997.18s
2025-05-12 15:35:47,878 dump_project         INFO     | Exporting project P2
2025-05-12 15:35:47,881 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:35:47,881 update_project_size_run INFO     | Finished updating project size for P2 in 5117.77s
2025-05-12 16:03:17,467 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 17:03:18,411 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 18:03:18,520 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 19:03:18,699 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 19:44:15,145 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:44:15,145 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:45:25,345 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:45:25,345 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:49:40,454 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:49:40,454 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:50:50,627 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:50:50,627 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 20:03:19,119 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 21:03:19,333 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 22:03:19,737 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 23:03:20,505 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 00:03:21,457 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 01:03:22,425 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 02:03:22,729 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 03:03:22,878 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 04:03:23,793 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 05:03:24,603 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 06:03:24,744 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 07:03:24,846 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 08:03:25,624 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 09:03:25,829 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 10:03:26,034 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 11:03:26,253 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 12:03:26,882 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 12:21:12,438 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-13 12:21:12,439 update_project_size_run INFO     | Beginning update project size for P2
2025-05-13 13:03:28,040 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 14:03:29,002 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 14:38:54,881 dump_project         INFO     | Exporting project P2
2025-05-13 14:38:54,887 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.01s
2025-05-13 14:38:54,887 update_project_size_run INFO     | Finished updating project size for P2 in 8262.45s
2025-05-13 15:03:29,289 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 16:03:30,112 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 17:03:30,264 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 18:03:30,667 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 19:03:30,887 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 20:03:31,143 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 21:03:31,960 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 22:03:32,236 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 23:03:32,385 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 00:03:33,189 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 01:03:33,309 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 02:03:34,099 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 03:03:34,368 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 04:03:35,249 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 05:03:36,003 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 06:03:36,664 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 07:03:37,577 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 08:03:37,739 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 09:03:38,599 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 10:03:39,170 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 11:03:39,676 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 12:03:39,715 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 13:03:39,932 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 14:03:40,186 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 14:10:04 info                 INFO     | Handling signal: quit
2025-05-14 14:10:04 info                 INFO     | Worker exiting (pid: 2856084)
2025-05-14 14:10:04,925 interrupt            INFO     |  === EXITED === 
2025-05-14 14:10:05 info                 INFO     | Shutting down: Master

Those jobs do exist in the project folder, and there are files in the respective folders.

12

Hi I got the same problem. :frowning: After attaching a project (here P6), I got at first 69 jobs, but when I refresh it, it turned to be 0. Nothing showed up after attaching. I checked the command_core.log file, and I did get an error but do not quite know how to solve it. Could anyone give me a hand? Thanks!

2025-10-07 14:40:06,422 export_project       INFO     | Exporting J80 in P6
2025-10-07 14:40:06,422 dump_job_database    INFO     | Request to export P6 J80
2025-10-07 14:40:06,423 dump_job_database    INFO     |    Exporting job to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J80
2025-10-07 14:40:06,424 dump_job_database    INFO     |    Exporting all of job's images in the database to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J80/gridfs_data...
2025-10-07 14:40:06,617 dump_job_database    INFO     |    Writing 232 database images to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J80/gridfs_data/gridfsdata_0
2025-10-07 14:40:06,618 dump_job_database    INFO     |    Done. Exported 232 images in 0.19s
2025-10-07 14:40:06,618 dump_job_database    INFO     |    Exporting all job's streamlog events...
2025-10-07 14:40:06,628 dump_job_database    INFO     |    Done. Exported 1 files in 0.01s
2025-10-07 14:40:06,628 dump_job_database    INFO     |    Exporting job metafile...
2025-10-07 14:40:06,629 dump_job_database    INFO     |    Creating .csg file for particles
2025-10-07 14:40:06,632 dump_job_database    INFO     |    Creating .csg file for class_averages
2025-10-07 14:40:06,635 dump_job_database    INFO     |    Done. Exported in 0.01s
2025-10-07 14:40:06,635 dump_job_database    INFO     |    Updating job manifest...
2025-10-07 14:40:06,635 dump_job_database    INFO     |    Done. Updated in 0.00s
2025-10-07 14:40:06,635 dump_job_database    INFO     | Exported P6 J80 in 0.21s
2025-10-07 14:40:06,637 export_project       INFO     | Exporting J9 in P6
2025-10-07 14:40:06,637 dump_job_database    INFO     | Request to export P6 J9
2025-10-07 14:40:06,638 dump_job_database    INFO     |    Exporting job to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J9
2025-10-07 14:40:06,638 dump_job_database    INFO     |    Exporting all of job's images in the database to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J9/gridfs_data...
2025-10-07 14:40:06,668 dump_job_database    INFO     |    Writing 16 database images to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/J9/gridfs_data/gridfsdata_0
2025-10-07 14:40:06,668 dump_job_database    INFO     |    Done. Exported 16 images in 0.03s
2025-10-07 14:40:06,668 dump_job_database    INFO     |    Exporting all job's streamlog events...
2025-10-07 14:40:06,762 dump_job_database    INFO     |    Done. Exported 1 files in 0.09s
2025-10-07 14:40:06,762 dump_job_database    INFO     |    Exporting job metafile...
2025-10-07 14:40:06,764 dump_job_database    INFO     |    Creating .csg file for particles
2025-10-07 14:40:06,766 dump_job_database    INFO     |    Creating .csg file for micrographs
2025-10-07 14:40:06,769 dump_job_database    INFO     |    Creating .csg file for templates
2025-10-07 14:40:06,772 dump_job_database    INFO     |    Done. Exported in 0.01s
2025-10-07 14:40:06,772 dump_job_database    INFO     |    Updating job manifest...
2025-10-07 14:40:06,773 dump_job_database    INFO     |    Done. Updated in 0.00s
2025-10-07 14:40:06,773 dump_job_database    INFO     | Exported P6 J9 in 0.14s
2025-10-07 14:40:06,781 dump_project         INFO     | Exporting project P6
2025-10-07 14:40:06,782 dump_project         INFO     | Exported project P6 to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/project.json in 0.00s
2025-10-07 14:40:06,782 dump_workspaces      INFO     | Exporting all workspaces in P6...
2025-10-07 14:40:06,783 dump_workspaces      INFO     | Exported all workspaces in P6 to /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/workspaces.json in 0.00s
2025-10-07 14:40:06,784 export_project       WARNING  | Exported jobs in P6 with errors
2025-10-07 15:04:15,997 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 16:04:16,443 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 17:04:16,816 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 18:04:17,695 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 19:04:17,716 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 20:04:18,684 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 21:04:18,806 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 22:04:19,545 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-07 23:04:20,290 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 00:04:21,031 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 01:04:21,302 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 02:04:21,986 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 03:04:22,730 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 04:04:23,280 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 05:04:24,131 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 06:04:24,610 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 07:04:25,080 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 08:04:25,609 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 09:04:26,346 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 10:04:27,081 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 11:04:27,847 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 12:04:28,654 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 13:04:28,782 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 14:04:29,462 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 15:04:30,298 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 16:04:30,971 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 17:04:31,768 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 18:04:32,627 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-10-08 18:12:59,293 wrapper              ERROR    | JSONRPC ERROR at set_user_viewed_project
2025-10-08 18:12:59,293 wrapper              ERROR    | Traceback (most recent call last):
2025-10-08 18:12:59,293 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
2025-10-08 18:12:59,293 wrapper              ERROR    |     res = func(*args, **kwargs)
2025-10-08 18:12:59,293 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1208, in set_user_viewed_project
2025-10-08 18:12:59,293 wrapper              ERROR    |     update_project(project_uid, {'last_accessed' : {'name' : get_username_by_id(user_id), 'accessed_at' : datetime.datetime.utcnow()}}, operation='$set', export=False)
2025-10-08 18:12:59,293 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 187, in wrapper
2025-10-08 18:12:59,293 wrapper              ERROR    |     return func(*args, **kwargs)
2025-10-08 18:12:59,293 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 250, in wrapper
2025-10-08 18:12:59,293 wrapper              ERROR    |     assert os.path.isfile(
2025-10-08 18:12:59,293 wrapper              ERROR    | AssertionError: validation error: lock file for P5 at /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/CS-caeel-ad318-mo-graphene-grid-anti-gfp-3/cs.lock absent or otherwise inaccessible. 
2025-10-08 18:13:00,435 wrapper              ERROR    | JSONRPC ERROR at set_user_viewed_workspace
2025-10-08 18:13:00,435 wrapper              ERROR    | Traceback (most recent call last):
2025-10-08 18:13:00,435 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
2025-10-08 18:13:00,435 wrapper              ERROR    |     res = func(*args, **kwargs)
2025-10-08 18:13:00,435 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1234, in set_user_viewed_workspace
2025-10-08 18:13:00,435 wrapper              ERROR    |     update_workspace(project_uid, workspace_uid, {'last_accessed' : {'name' : get_username_by_id(user_id), 'accessed_at' : datetime.datetime.utcnow()}}, operation='$set', export=False)
2025-10-08 18:13:00,435 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 187, in wrapper
2025-10-08 18:13:00,435 wrapper              ERROR    |     return func(*args, **kwargs)
2025-10-08 18:13:00,435 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 250, in wrapper
2025-10-08 18:13:00,435 wrapper              ERROR    |     assert os.path.isfile(
2025-10-08 18:13:00,435 wrapper              ERROR    | AssertionError: validation error: lock file for P5 at /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/CS-caeel-ad318-mo-graphene-grid-anti-gfp-3/cs.lock absent or otherwise inaccessible. 
2025-10-08 18:13:02,372 wrapper              ERROR    | JSONRPC ERROR at set_user_viewed_project
2025-10-08 18:13:02,372 wrapper              ERROR    | Traceback (most recent call last):
2025-10-08 18:13:02,372 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
2025-10-08 18:13:02,372 wrapper              ERROR    |     res = func(*args, **kwargs)
2025-10-08 18:13:02,372 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 1208, in set_user_viewed_project
2025-10-08 18:13:02,372 wrapper              ERROR    |     update_project(project_uid, {'last_accessed' : {'name' : get_username_by_id(user_id), 'accessed_at' : datetime.datetime.utcnow()}}, operation='$set', export=False)
2025-10-08 18:13:02,372 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 187, in wrapper
2025-10-08 18:13:02,372 wrapper              ERROR    |     return func(*args, **kwargs)
2025-10-08 18:13:02,372 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 250, in wrapper
2025-10-08 18:13:02,372 wrapper              ERROR    |     assert os.path.isfile(
2025-10-08 18:13:02,372 wrapper              ERROR    | AssertionError: validation error: lock file for P5 at /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/CS-caeel-ad318-mo-graphene-grid-anti-gfp-3/cs.lock absent or otherwise inaccessible. 
2025-10-08 18:13:05,622 wrapper              ERROR    | JSONRPC ERROR at request_delete_project
2025-10-08 18:13:05,622 wrapper              ERROR    | Traceback (most recent call last):
2025-10-08 18:13:05,622 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
2025-10-08 18:13:05,622 wrapper              ERROR    |     res = func(*args, **kwargs)
2025-10-08 18:13:05,622 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 250, in wrapper
2025-10-08 18:13:05,622 wrapper              ERROR    |     assert os.path.isfile(
2025-10-08 18:13:05,622 wrapper              ERROR    | AssertionError: validation error: lock file for P5 at /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/CS-caeel-ad318-mo-graphene-grid-anti-gfp-3/cs.lock absent or otherwise inaccessible. 
2025-10-08 18:13:15,659 wrapper              ERROR    | JSONRPC ERROR at archive_project
2025-10-08 18:13:15,659 wrapper              ERROR    | Traceback (most recent call last):
2025-10-08 18:13:15,659 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 196, in wrapper
2025-10-08 18:13:15,659 wrapper              ERROR    |     res = func(*args, **kwargs)
2025-10-08 18:13:15,659 wrapper              ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 250, in wrapper
2025-10-08 18:13:15,659 wrapper              ERROR    |     assert os.path.isfile(
2025-10-08 18:13:15,659 wrapper              ERROR    | AssertionError: validation error: lock file for P5 at /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/CS-caeel-ad318-mo-graphene-grid-anti-gfp-3/cs.lock absent or otherwise inaccessible. 
2025-10-08 18:13:44,820 import_project       INFO     | Importing project from /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2
2025-10-08 18:13:45,921 import_project       INFO     | Created project P8
2025-10-08 18:13:45,926 import_workspaces    INFO     | Created workspace W1 in P8
2025-10-08 18:13:46,081 import_jobs          INFO     | Inserting jobs into project...
2025-10-08 18:13:46,081 import_jobs          INFO     | Uploading image data for J1...
2025-10-08 18:13:46,081 import_jobs          INFO     | Done. Uploaded 0 files in 0.00s
2025-10-08 18:13:46,086 import_jobs          INFO     | Inserted job document in 0.01s...
2025-10-08 18:13:46,086 import_jobs          INFO     | Inserting streamlogs into jobs...
2025-10-08 18:13:46,088 import_project_run   ERROR    | Unable to import project from /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2
2025-10-08 18:13:46,088 import_project_run   ERROR    | Traceback (most recent call last):
2025-10-08 18:13:46,088 import_project_run   ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-10-08 18:13:46,088 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-10-08 18:13:46,088 import_project_run   ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4904, in import_jobs
2025-10-08 18:13:46,088 import_project_run   ERROR    |     events_doc_data = BSON(openfile.read()).decode()
2025-10-08 18:13:46,088 import_project_run   ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1446, in decode
2025-10-08 18:13:46,088 import_project_run   ERROR    |     return decode(self, codec_options)
2025-10-08 18:13:46,088 import_project_run   ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1078, in decode
2025-10-08 18:13:46,088 import_project_run   ERROR    |     return cast("Union[dict[str, Any], _DocumentType]", _bson_to_dict(data, opts))
2025-10-08 18:13:46,088 import_project_run   ERROR    | bson.errors.InvalidBSON: not enough data for a BSON document
2025-10-08 18:13:46,090 dump_project         INFO     | Exporting project P8
2025-10-08 18:13:46,092 dump_project         INFO     | Exported project P8 to /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/project.json in 0.00s
2025-10-08 18:13:46,094 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-10-08 18:13:46,094 run                  ERROR    | Traceback (most recent call last):
2025-10-08 18:13:46,094 run                  ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-10-08 18:13:46,094 run                  ERROR    |     self.target(*self.args)
2025-10-08 18:13:46,094 run                  ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-10-08 18:13:46,094 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-10-08 18:13:46,094 run                  ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4904, in import_jobs
2025-10-08 18:13:46,094 run                  ERROR    |     events_doc_data = BSON(openfile.read()).decode()
2025-10-08 18:13:46,094 run                  ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1446, in decode
2025-10-08 18:13:46,094 run                  ERROR    |     return decode(self, codec_options)
2025-10-08 18:13:46,094 run                  ERROR    |   File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1078, in decode
2025-10-08 18:13:46,094 run                  ERROR    |     return cast("Union[dict[str, Any], _DocumentType]", _bson_to_dict(data, opts))
2025-10-08 18:13:46,094 run                  ERROR    | bson.errors.InvalidBSON: not enough data for a BSON document
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2306, in run_with_except_hook
    run_old(*args, **kw)
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4904, in import_jobs
    events_doc_data = BSON(openfile.read()).decode()
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1446, in decode
    return decode(self, codec_options)
  File "/usr/local/cryosparc4/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/site-packages/bson/__init__.py", line 1078, in decode
    return cast("Union[dict[str, Any], _DocumentType]", _bson_to_dict(data, opts))
bson.errors.InvalidBSON: not enough data for a BSON document
13

@HanW Please can you

  1. confirm whether you are concerned about the project at
    /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
    and or the project at
    /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
  2. describe the relationship between these two directories that have the same name, but different absolute paths?
  3. post the outputs of the commands
    df -Th /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
df -Th /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
14

Thanks for the reply!

The project was originally at, and from where I detached the project

/data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/

But since this disk was almost full so I decided to move and copy it to

/data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/

So when I tried to attach the copied project in the second directory (in the same instance), I got 0 jobs. And I also tried to attach the project in the first directory to see if I missed something, but even the project in the first directory could not be attached correctly…..

And the outputs of the commands:

df -Th /data/8TBssd1/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sdb1      ext4  7,3T  6,9T  4,0K 100% /data/8TBssd1
df -Th /data/14TBdisk2/cryosparcuser/CS-caeel-ad318-mo-graphene-grid-anti-gfp-2/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/sde       ext4   13T   11T  1,2T  91% /data/14TBdisk2