Missing jobs after transfering project

Install, Configure and Update
1

This topic has been raised before, but I could not find a solution, so I am asking for help. My situation is similar to this post.

We had a workstation mounted to a shared filesystem that crashed. I then installed cryosparc 4.7 on a different workstation that is mounted to the same shared filesystem, and then I deleted the cs.lock file in my project folder and attached it to the new one. I did not move those files. The project is over 20 TB in size and has 867 jobs, but when I attached it, there are only 64 jobs there. All the workspaces are present, but some of them have no jobs. When I originally attached the project, there was a loading bar for about 30 minutes and then it was marked complete. Others have noted that a project this big can take a long time to attach, so I waited for 2 days now, but there are still only 64 jobs visible. Can you please help me get my project back?

I also ran the command cryosparcm filterlog command_core -l ERROR and this was the result:

[marcell@msg-app4 ~]$ cryosparcm filterlog command_core -l ERROR
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
2025-05-09 13:52:29,359 import_project_run   ERROR    | Traceback (most recent call last):
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:29,359 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:29,359 import_project_run   ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return loads(fp.read(),
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:29,359 import_project_run   ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:29,359 import_project_run   ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 13:52:32,247 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-05-09 13:52:32,247 run                  ERROR    | Traceback (most recent call last):
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-05-09 13:52:32,247 run                  ERROR    |     self.target(*self.args)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:32,247 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:32,247 run                  ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:32,247 run                  ERROR    |     return loads(fp.read(),
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:32,247 run                  ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:32,247 run                  ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:32,247 run                  ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-12 19:44:15,145 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:45:25,345 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:49:40,454 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:50:50,627 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/XXX (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2

@Marcell Please can you inspect the unfiltered command_core log for additional (including non-ERROR) lines that may reveal additional details, such as a specific job ID, relevant to the JSONDecodeError around 13:52 on May 9. You can browse the log with the command

cryosparcm log command_core | less

or try

cryosparcm log command_core | grep '2025-05-09 13:52:'
3

Thanks for looking into this. I think this is the relevant piece. I included a few jobs that were successfully loaded as well in case that helps.

2025-05-09 13:52:28,394 import_jobs          INFO     | Inserted job document in 0.67s...
2025-05-09 13:52:28,394 import_jobs          INFO     | Inserting streamlogs into jobs...
2025-05-09 13:52:28,561 import_jobs          INFO     | Done. Inserted 299 streamlogs in 0.17s...
2025-05-09 13:52:28,561 import_jobs          INFO     | Imported J165 into P1 in 0.83s...
2025-05-09 13:52:28,566 import_jobs          INFO     | Uploading image data for J166...
2025-05-09 13:52:29,282 import_jobs          INFO     | Done. Uploaded 197 files in 0.72s
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
2025-05-09 13:52:29,359 import_project_run   ERROR    | Traceback (most recent call last):
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:29,359 import_project_run   ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:29,359 import_project_run   ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return loads(fp.read(),
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:29,359 import_project_run   ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:29,359 import_project_run   ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:29,359 import_project_run   ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:29,359 import_project_run   ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 13:52:29,383 dump_project         INFO     | Exporting project P1
2025-05-09 13:52:29,417 dump_project         INFO     | Exported project P1 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.03s
2025-05-09 13:52:32,247 run                  ERROR    | POST-RESPONSE-THREAD ERROR at import_project_run
2025-05-09 13:52:32,247 run                  ERROR    | Traceback (most recent call last):
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
2025-05-09 13:52:32,247 run                  ERROR    |     self.target(*self.args)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
2025-05-09 13:52:32,247 run                  ERROR    |     warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
2025-05-09 13:52:32,247 run                  ERROR    |     job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
2025-05-09 13:52:32,247 run                  ERROR    |     return loads(fp.read(),
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
2025-05-09 13:52:32,247 run                  ERROR    |     return cls(**kw).decode(s)
2025-05-09 13:52:32,247 run                  ERROR    |   File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
2025-05-09 13:52:32,247 run                  ERROR    |     raise JSONDecodeError("Extra data", s, end)
2025-05-09 13:52:32,247 run                  ERROR    | json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
**custom thread exception hook caught something
**** handle exception rc
Traceback (most recent call last):
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2306, in run_with_except_hook
    run_old(*args, **kw)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
    job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)

Traceback (most recent call last):
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 2306, in run_with_except_hook
    run_old(*args, **kw)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/commandcommon.py", line 73, in run
    self.target(*self.args)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4618, in import_project_run
    warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4861, in import_jobs
    job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 293, in load
    return loads(fp.read(),
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/__init__.py", line 359, in loads
    return cls(**kw).decode(s)
  File "/wynton/home/chenglab/marcell/cryosparc_install/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.10/json/decoder.py", line 340, in decode
    raise JSONDecodeError("Extra data", s, end)
json.decoder.JSONDecodeError: Extra data: line 6080 column 2 (char 241418)
2025-05-09 14:44:14,610 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-09 15:04:35,935 update_project_size  INFO     | Starting PostResponseThread for update project size for P1
2025-05-09 15:04:35,937 update_project_size_run INFO     | Beginning update project size for P1
2025-05-09 15:37:20 info                 INFO     | Handling signal: quit
2025-05-09 15:37:20 info                 INFO     | Worker exiting (pid: 1193651)
4

It seems to me like it’s getting stuck at J166 or J167 (I really don’t know what I am talking about though). Would it be helpful if I uploaded the .json files for those jobs?

5

Possibly. Do you have a way of sharing the files? Otherwise, I can send you a direct message about sharing arrangements.

6

Here is a folder with some json files. I put in the project, workspace and job_manifest from the project level, and I added the json file from jobs 166 and 167. I renamed them, in their respective folders they are still named job.json.

This download link only seems to work if it’s copy-pasted into the browser, let me know if there is an issue.

https://limewire.com/d/TgAcG#OehdmTR3c4

7

Thanks @Marcell I fetched the files.

1 Like
8

Have there been multiple attempts to attach the project to the newly installed, v4.7 CryoSPARC instance?

9

I did try it a second time. After the first attempt, I detached, then reattached. In both cases, it got stuck on the same job.

10

Was the second attempt performed before or after

2025-05-09 13:52:28,566 import_jobs          INFO     | Uploading image data for J166...
2025-05-09 13:52:29,282 import_jobs          INFO     | Done. Uploaded 197 files in 0.72s
2025-05-09 13:52:29,359 import_project_run   ERROR    | Unable to import project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2
11

It was performed after that, i.e. this was from the first attempt. I just looked at the log from the second attempt, and there may be another clue:

2025-05-12 13:36:35,150 import_jobs          INFO     | Uploading image data for J165...
2025-05-12 13:36:35,656 import_jobs          INFO     | Done. Uploaded 176 files in 0.51s
2025-05-12 13:36:35,661 import_jobs          INFO     | Inserted job document in 0.51s...
2025-05-12 13:36:35,661 import_jobs          INFO     | Inserting streamlogs into jobs...
2025-05-12 13:36:35,790 import_jobs          INFO     | Done. Inserted 299 streamlogs in 0.13s...
2025-05-12 13:36:35,790 import_jobs          INFO     | Imported J165 into P2 in 0.64s...
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J159 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J159 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J148 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J148 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J138 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J138 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J129 parents include non-existent job J65
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J129 parents include non-existent job J94
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J119 parents include non-existent job J77
2025-05-12 13:36:36,329 layout_tree          WARNING  | Job P2 J119 parents include non-existent job J99
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J100 parents include non-existent job J99
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J15 parents include non-existent job J5
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J10 parents include non-existent job J9
2025-05-12 13:36:36,330 layout_tree          WARNING  | Job P2 J127 parents include non-existent job J21
2025-05-12 13:36:36,732 import_project_run   WARNING  | Failed laying out tree in P2: 'J5'
2025-05-12 13:36:36,733 import_project_run   WARNING  | Imported project from /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2 as P2 in 3976.71s with errors.
2025-05-12 13:36:36,760 dump_project         INFO     | Exporting project P2
2025-05-12 13:36:36,764 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 13:36:36,764 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 14:03:16,141 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 14:10:30,089 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-12 14:10:30,107 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 14:12:30,633 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-12 14:12:30,697 update_project_size_run INFO     | Beginning update project size for P2
2025-05-12 15:03:17,392 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 15:25:53,512 dump_project         INFO     | Exporting project P2
2025-05-12 15:25:53,514 dump_project         INFO     | Exporting project P1
2025-05-12 15:25:53,516 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:25:53,516 update_project_size_run INFO     | Finished updating project size for P2 in 6556.75s
2025-05-12 15:25:53,517 dump_project         INFO     | Exported project P1 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:25:53,517 update_project_size_run INFO     | Finished updating project size for P1 in 12152.48s
2025-05-12 15:25:53,517 update_all_job_sizes_run INFO     | Finished updating all job sizes (0 jobs updated, 1 projects updated)
2025-05-12 15:35:47,875 dump_project         INFO     | Exporting project P2
2025-05-12 15:35:47,878 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:35:47,878 update_project_size_run INFO     | Finished updating project size for P2 in 4997.18s
2025-05-12 15:35:47,878 dump_project         INFO     | Exporting project P2
2025-05-12 15:35:47,881 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.00s
2025-05-12 15:35:47,881 update_project_size_run INFO     | Finished updating project size for P2 in 5117.77s
2025-05-12 16:03:17,467 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 17:03:18,411 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 18:03:18,520 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 19:03:18,699 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 19:44:15,145 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:44:15,145 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:45:25,345 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:45:25,345 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:49:40,454 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:49:40,454 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 19:50:50,627 heartbeat_manager    ERROR    | HTTPSConnectionPool(host='get.cryosparc.com', port=443): Max retries exceeded with url: /heartbeat/2d62cbc6-2c0b-11f0-880c-732536936247 (Caused by ProxyError('Cannot connect to proxy.', TimeoutError('timed out')))
2025-05-12 19:50:50,627 heartbeat_manager    WARNING  | Error connecting to cryoSPARC license server during instance heartbeat.
2025-05-12 20:03:19,119 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 21:03:19,333 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 22:03:19,737 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-12 23:03:20,505 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 00:03:21,457 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 01:03:22,425 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 02:03:22,729 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 03:03:22,878 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 04:03:23,793 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 05:03:24,603 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 06:03:24,744 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 07:03:24,846 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 08:03:25,624 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 09:03:25,829 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 10:03:26,034 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 11:03:26,253 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 12:03:26,882 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 12:21:12,438 update_project_size  INFO     | Starting PostResponseThread for update project size for P2
2025-05-13 12:21:12,439 update_project_size_run INFO     | Beginning update project size for P2
2025-05-13 13:03:28,040 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 14:03:29,002 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 14:38:54,881 dump_project         INFO     | Exporting project P2
2025-05-13 14:38:54,887 dump_project         INFO     | Exported project P2 to /wynton/group/cheng/marcell/Janelia_2/CS-janelia-2/project.json in 0.01s
2025-05-13 14:38:54,887 update_project_size_run INFO     | Finished updating project size for P2 in 8262.45s
2025-05-13 15:03:29,289 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 16:03:30,112 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 17:03:30,264 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 18:03:30,667 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 19:03:30,887 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 20:03:31,143 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 21:03:31,960 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 22:03:32,236 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-13 23:03:32,385 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 00:03:33,189 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 01:03:33,309 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 02:03:34,099 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 03:03:34,368 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 04:03:35,249 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 05:03:36,003 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 06:03:36,664 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 07:03:37,577 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 08:03:37,739 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 09:03:38,599 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 10:03:39,170 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 11:03:39,676 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 12:03:39,715 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 13:03:39,932 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 14:03:40,186 background_worker    INFO     | License does not have telemetry enabled; will re-check license in 1 hour.
2025-05-14 14:10:04 info                 INFO     | Handling signal: quit
2025-05-14 14:10:04 info                 INFO     | Worker exiting (pid: 2856084)
2025-05-14 14:10:04,925 interrupt            INFO     |  === EXITED === 
2025-05-14 14:10:05 info                 INFO     | Shutting down: Master

Those jobs do exist in the project folder, and there are files in the respective folders.