Hi, all:
Due to the crash of database, we created a new database, and then attached the projects to the new database. But after attaching, we found that the project missed a lot of jobs. Is there any suggestion to deal with this problem?
Regards
Samuel
Hi,all:
I also met the same question. While attaching projects, database exited unexpected, and some jobs were not imported.
Maybe I should detach and reattach project? Is there someone know how to fix this?
Sincerely
Luis
Please can you post relevant messages from the database log:
cryosparcm log database
Please can you check for errors related to the project attachment:
cryosparcm filterlog command_core -l ERROR
Actually I tried detach and attach many times, these jobs can not be found.
This time I run the cryosparcm filterlog command_core -l ERROR , and got this:
2024-01-13 00:45:24,448 import_project_run ERROR | Unable to import project from /work/caolab/yu.cao/SYVN1
2024-01-13 00:45:24,448 import_project_run ERROR | Traceback (most recent call last):
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4479, in import_project_run
2024-01-13 00:45:24,448 import_project_run ERROR | warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4722, in import_jobs
2024-01-13 00:45:24,448 import_project_run ERROR | job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/__init__.py", line 293, in load
2024-01-13 00:45:24,448 import_project_run ERROR | return loads(fp.read(),
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/__init__.py", line 370, in loads
2024-01-13 00:45:24,448 import_project_run ERROR | return cls(**kw).decode(s)
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/decoder.py", line 337, in decode
2024-01-13 00:45:24,448 import_project_run ERROR | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2024-01-13 00:45:24,448 import_project_run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/decoder.py", line 355, in raw_decode
2024-01-13 00:45:24,448 import_project_run ERROR | raise JSONDecodeError("Expecting value", s, err.value) from None
2024-01-13 00:45:24,448 import_project_run ERROR | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
2024-01-13 00:45:24,555 run ERROR | POST-RESPONSE-THREAD ERROR at import_project_run
2024-01-13 00:45:24,555 run ERROR | Traceback (most recent call last):
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 72, in run
2024-01-13 00:45:24,555 run ERROR | self.target(*self.args)
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4479, in import_project_run
2024-01-13 00:45:24,555 run ERROR | warning = import_jobs(jobs_manifest, abs_path_export_project_dir, new_project_uid, owner_user_id, notification_id) or warning
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4722, in import_jobs
2024-01-13 00:45:24,555 run ERROR | job_doc_data = json.load(openfile, object_hook=json_util.object_hook)
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/__init__.py", line 293, in load
2024-01-13 00:45:24,555 run ERROR | return loads(fp.read(),
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/__init__.py", line 370, in loads
2024-01-13 00:45:24,555 run ERROR | return cls(**kw).decode(s)
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/decoder.py", line 337, in decode
2024-01-13 00:45:24,555 run ERROR | obj, end = self.raw_decode(s, idx=_w(s, 0).end())
2024-01-13 00:45:24,555 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/json/decoder.py", line 355, in raw_decode
2024-01-13 00:45:24,555 run ERROR | raise JSONDecodeError("Expecting value", s, err.value) from None
2024-01-13 00:45:24,555 run ERROR | json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)
Thanks a lot.
Please can you post the outputs of the commands
stat -f /work/caolab/yu.cao/SYVN1
ls -al /work/caolab/yu.cao/SYVN1
File: “/work/caolab/yu.cao/SYVN1”
ID: ef0009600000002 Namelen: 255 Type: gpfs
Block size: 1048576 Fundamental block size: 1048576
Blocks: Total: 4120398720 Free: 127826782 Available: 127826782
Inodes: Total: 536870912 Free: 251750949
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J42
drwxrwxr-x 3 cylab cylab 32768 Aug 18 22:23 J420
drwxrwxr-x 3 cylab cylab 4096 Aug 19 14:23 J421
drwxrwxr-x 3 cylab cylab 32768 Aug 19 14:35 J422
drwxrwxr-x 3 cylab cylab 4096 Aug 19 14:27 J423
drwxrwxr-x 3 cylab cylab 32768 Aug 19 14:41 J424
drwxrwxr-x 3 cylab cylab 32768 Aug 19 22:14 J425
drwxrwxr-x 4 cylab cylab 4096 Aug 19 22:02 J426
drwxrwxr-x 3 cylab cylab 4096 Aug 19 22:00 J427
drwxrwxr-x 4 cylab cylab 4096 Aug 19 22:02 J428
drwxrwxr-x 3 cylab cylab 4096 Aug 19 22:01 J429
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J43
drwxrwxr-x 3 cylab cylab 4096 Aug 19 22:02 J430
drwxrwxr-x 3 cylab cylab 32768 Aug 19 22:32 J431
drwxrwxr-x 3 cylab cylab 32768 Aug 20 00:38 J432
drwxrwxr-x 3 cylab cylab 4096 Aug 20 00:11 J433
drwxrwxr-x 3 cylab cylab 4096 Aug 20 00:12 J434
drwxrwxr-x 3 cylab cylab 32768 Aug 20 00:37 J435
drwxrwxr-x 3 cylab cylab 32768 Aug 20 09:25 J436
drwxrwxr-x 3 cylab cylab 32768 Aug 20 09:48 J437
drwxrwxr-x 3 cylab cylab 4096 Aug 20 09:14 J438
drwxrwxr-x 3 cylab cylab 32768 Aug 20 09:31 J439
drwxrwxr-x 4 cylab cylab 4096 Nov 26 2021 J44
drwxrwxr-x 3 cylab cylab 4096 Aug 20 11:12 J440
drwxrwxr-x 3 cylab cylab 32768 Aug 20 11:53 J441
drwxrwxr-x 4 cylab cylab 4096 Aug 30 22:44 J442
drwxrwxr-x 3 cylab cylab 32768 Aug 30 23:12 J443
drwxrwxr-x 4 cylab cylab 4096 Aug 30 22:49 J444
drwxrwxr-x 3 cylab cylab 32768 Aug 30 23:14 J445
drwxrwxr-x 4 cylab cylab 4096 Sep 1 10:57 J446
drwxrwxr-x 3 cylab cylab 4096 Sep 1 10:58 J447
drwxrwxr-x 3 cylab cylab 32768 Sep 1 13:12 J448
drwxrwxr-x 4 cylab cylab 4096 Sep 7 15:30 J449
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J45
drwxrwxr-x 3 cylab cylab 32768 Sep 7 15:56 J450
drwxrwxr-x 4 cylab cylab 4096 Sep 7 20:46 J451
drwxrwxr-x 3 cylab cylab 32768 Sep 7 21:36 J452
drwxrwxr-x 4 cylab cylab 4096 Sep 8 19:26 J453
drwxrwxr-x 4 cylab cylab 4096 Sep 8 19:26 J454
drwxrwxr-x 3 cylab cylab 32768 Sep 8 19:59 J455
drwxrwxr-x 3 cylab cylab 32768 Sep 8 20:00 J456
drwxrwxr-x 3 cylab cylab 4096 Sep 8 20:03 J457
drwxrwxr-x 3 cylab cylab 4096 Sep 8 20:05 J458
drwxrwxr-x 3 cylab cylab 32768 Sep 8 20:33 J459
drwxrwxr-x 4 cylab cylab 4096 Nov 26 2021 J46
drwxrwxr-x 4 cylab cylab 4096 Sep 8 20:08 J460
drwxrwxr-x 3 cylab cylab 4096 Sep 10 22:13 J461
drwxrwxr-x 3 cylab cylab 32768 Sep 8 22:27 J462
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:07 J463
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:37 J464
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:37 J465
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:52 J466
drwxrwxr-x 4 cylab cylab 4096 Sep 8 22:53 J467
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:56 J468
drwxrwxr-x 3 cylab cylab 32768 Sep 8 23:10 J469
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J47
drwxrwxr-x 3 cylab cylab 4096 Sep 8 22:58 J470
drwxrwxr-x 3 cylab cylab 4096 Sep 8 23:59 J471
drwxrwxr-x 3 cylab cylab 4096 Sep 9 00:02 J472
drwxrwxr-x 3 cylab cylab 4096 Sep 9 00:04 J473
drwxrwxr-x 3 cylab cylab 4096 Sep 8 23:59 J474
drwxrwxr-x 3 cylab cylab 4096 Sep 9 00:03 J475
drwxrwxr-x 3 cylab cylab 4096 Sep 9 00:03 J476
drwxrwxr-x 3 cylab cylab 4096 Sep 9 19:03 J477
drwxrwxr-x 3 cylab cylab 32768 Sep 9 00:16 J478
drwxrwxr-x 3 cylab cylab 32768 Sep 9 19:25 J479
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J48
drwxrwxr-x 3 cylab cylab 4096 Sep 9 19:10 J480
drwxrwxr-x 3 cylab cylab 32768 Sep 9 19:32 J481
drwxrwxr-x 3 cylab cylab 4096 Sep 9 22:25 J482
drwxrwxr-x 3 cylab cylab 32768 Sep 9 22:34 J483
drwxrwxr-x 3 cylab cylab 4096 Sep 9 22:27 J484
drwxrwxr-x 3 cylab cylab 32768 Sep 10 06:04 J485
drwxrwxr-x 4 cylab cylab 4096 Sep 10 22:19 J486
drwxrwxr-x 3 cylab cylab 32768 Sep 10 22:31 J487
drwxrwxr-x 3 cylab cylab 4096 Sep 11 13:19 J488
drwxrwxr-x 3 cylab cylab 32768 Sep 11 13:36 J489
drwxrwxr-x 4 cylab cylab 4096 Nov 26 2021 J49
drwxrwxr-x 3 cylab cylab 4096 Sep 15 08:51 J490
drwxrwxr-x 3 cylab cylab 32768 Sep 15 08:58 J491
drwxrwxr-x 3 cylab cylab 4096 Nov 26 2021 J5
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J50
drwxrwxr-x 4 cylab cylab 4096 Nov 26 2021 J51
drwxrwxr-x 3 cylab cylab 4096 Oct 11 2021 J52
drwxrwxr-x 3 cylab cylab 4096 Oct 11 2021 J53
drwxrwxr-x 3 cylab cylab 4096 Oct 11 2021 J54
drwxrwxr-x 3 cylab cylab 4096 Oct 11 2021 J55
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J56
drwxrwxr-x 3 cylab cylab 4096 Oct 11 2021 J57
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J58
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J59
drwxrwxr-x 3 cylab cylab 4096 Nov 26 2021 J6
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J60
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J61
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J62
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J63
drwxrwxr-x 3 cylab cylab 32768 Nov 26 2021 J64
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J65
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J66
drwxrwxr-x 3 cylab cylab 4096 Oct 12 2021 J67
just some parts of the results
And during another project attaching, database exited again, and this is the result of
cryosparcm filterlog command_core -l ERROR
2024-01-12 23:18:02,677 background_worker ERROR | Job Heartbeat check failed
2024-01-12 23:18:02,677 background_worker ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1606, in _retryable_read
2024-01-12 23:18:02,677 background_worker ERROR | server = self._select_server(read_pref, session, address=address)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1436, in _select_server
2024-01-12 23:18:02,677 background_worker ERROR | server = topology.select_server(server_selector)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 250, in select_server
2024-01-12 23:18:02,677 background_worker ERROR | return random.choice(self.select_servers(selector, server_selection_timeout, address))
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 211, in select_servers
2024-01-12 23:18:02,677 background_worker ERROR | server_descriptions = self._select_servers_loop(selector, server_timeout, address)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
2024-01-12 23:18:02,677 background_worker ERROR | raise ServerSelectionTimeoutError(
2024-01-12 23:18:02,677 background_worker ERROR | pymongo.errors.ServerSelectionTimeoutError: shipmhpc:45101: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65a13393995ee018df22100e, topology_type: Single, servers: [<ServerDescription ('shipmhpc', 45101) server_type: Unknown, rtt: None, error=AutoReconnect('shipmhpc:45101: [Errno 111] Connection refused')>]>
2024-01-12 23:18:02,677 background_worker ERROR |
2024-01-12 23:18:02,677 background_worker ERROR | During handling of the above exception, another exception occurred:
2024-01-12 23:18:02,677 background_worker ERROR |
2024-01-12 23:18:02,677 background_worker ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 165, in background_worker
2024-01-12 23:18:02,677 background_worker ERROR | check_heartbeats()
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 2587, in check_heartbeats
2024-01-12 23:18:02,677 background_worker ERROR | overdue_jobs = list(mongo.db['jobs'].find({'status' : {'$in' : com.job_alive_statuses},
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
2024-01-12 23:18:02,677 background_worker ERROR | if len(self.__data) or self._refresh():
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py", line 1195, in _refresh
2024-01-12 23:18:02,677 background_worker ERROR | self.__send_message(q)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py", line 1078, in __send_message
2024-01-12 23:18:02,677 background_worker ERROR | response = client._run_operation(
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1515, in _run_operation
2024-01-12 23:18:02,677 background_worker ERROR | return self._retryable_read(
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1623, in _retryable_read
2024-01-12 23:18:02,677 background_worker ERROR | raise last_error
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1617, in _retryable_read
2024-01-12 23:18:02,677 background_worker ERROR | return func(session, server, sock_info, secondary_ok)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1511, in _cmd
2024-01-12 23:18:02,677 background_worker ERROR | return server.run_operation(
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/server.py", line 114, in run_operation
2024-01-12 23:18:02,677 background_worker ERROR | reply = sock_info.receive_message(request_id)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 795, in receive_message
2024-01-12 23:18:02,677 background_worker ERROR | self._raise_connection_failure(error)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 969, in _raise_connection_failure
2024-01-12 23:18:02,677 background_worker ERROR | _raise_connection_failure(self.address, error)
2024-01-12 23:18:02,677 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 266, in _raise_connection_failure
2024-01-12 23:18:02,677 background_worker ERROR | raise AutoReconnect(msg)
2024-01-12 23:18:02,677 background_worker ERROR | pymongo.errors.AutoReconnect: shipmhpc:45101: [Errno 104] Connection reset by peer
2024-01-12 23:18:02,678 run ERROR | POST-RESPONSE-THREAD ERROR at import_project_run
2024-01-12 23:18:02,678 run ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 549, in load
2024-01-12 23:18:02,678 run ERROR | with bopen(file, "rb") as f:
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/contextlib.py", line 113, in __enter__
2024-01-12 23:18:02,678 run ERROR | return next(self.gen)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_tools/cryosparc/util.py", line 228, in bopen
2024-01-12 23:18:02,678 run ERROR | with open(file, mode) as f:
2024-01-12 23:18:02,678 run ERROR | FileNotFoundError: [Errno 2] No such file or directory: '/work/caolab/yu.cao/P15/J465/J465_class_02_00100_volume_sharp.cs'
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | The above exception was the direct cause of the following exception:
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4499, in import_project_run
2024-01-12 23:18:02,678 run ERROR | update_project_size(new_project_uid, use_prt=False)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2024-01-12 23:18:02,678 run ERROR | return func(*args, **kwargs)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 232, in wrapper
2024-01-12 23:18:02,678 run ERROR | return func(*args, **kwargs)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4384, in update_project_size
2024-01-12 23:18:02,678 run ERROR | return update_project_size_run()
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4354, in update_project_size_run
2024-01-12 23:18:02,678 run ERROR | [calculate_intermediate_results_size(project_uid, job['uid'], use_prt=False) for job in missing_intermediate_results_size_jobs]
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4354, in <listcomp>
2024-01-12 23:18:02,678 run ERROR | [calculate_intermediate_results_size(project_uid, job['uid'], use_prt=False) for job in missing_intermediate_results_size_jobs]
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2024-01-12 23:18:02,678 run ERROR | return func(*args, **kwargs)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 232, in wrapper
2024-01-12 23:18:02,678 run ERROR | return func(*args, **kwargs)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 6830, in calculate_intermediate_results_size
2024-01-12 23:18:02,678 run ERROR | calculate_intermediate_results_size_run()
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 6822, in calculate_intermediate_results_size_run
2024-01-12 23:18:02,678 run ERROR | total_size_bytes = rc.calculate_intermediate_results_size(project_uid, job_uid, always_keep_final)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 854, in calculate_intermediate_results_size
2024-01-12 23:18:02,678 run ERROR | associated_files = [find_associated_files(project_uid, job_uid, path) for path in intermediate_result_files]
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 854, in <listcomp>
2024-01-12 23:18:02,678 run ERROR | associated_files = [find_associated_files(project_uid, job_uid, path) for path in intermediate_result_files]
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 878, in find_associated_files
2024-01-12 23:18:02,678 run ERROR | d = dataset.Dataset.load(abs_cs_file_path)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 592, in load
2024-01-12 23:18:02,678 run ERROR | raise DatasetLoadError(f"Could not load dataset from file {file}") from err
2024-01-12 23:18:02,678 run ERROR | cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /work/caolab/yu.cao/P15/J465/J465_class_02_00100_volume_sharp.cs
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | During handling of the above exception, another exception occurred:
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1553, in _retry_internal
2024-01-12 23:18:02,678 run ERROR | server = self._select_server(writable_server_selector, session)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1436, in _select_server
2024-01-12 23:18:02,678 run ERROR | server = topology.select_server(server_selector)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 250, in select_server
2024-01-12 23:18:02,678 run ERROR | return random.choice(self.select_servers(selector, server_selection_timeout, address))
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 211, in select_servers
2024-01-12 23:18:02,678 run ERROR | server_descriptions = self._select_servers_loop(selector, server_timeout, address)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
2024-01-12 23:18:02,678 run ERROR | raise ServerSelectionTimeoutError(
2024-01-12 23:18:02,678 run ERROR | pymongo.errors.ServerSelectionTimeoutError: shipmhpc:45101: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65a13393995ee018df22100e, topology_type: Single, servers: [<ServerDescription ('shipmhpc', 45101) server_type: Unknown, rtt: None, error=AutoReconnect('shipmhpc:45101: [Errno 111] Connection refused')>]>
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | During handling of the above exception, another exception occurred:
2024-01-12 23:18:02,678 run ERROR |
2024-01-12 23:18:02,678 run ERROR | Traceback (most recent call last):
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 72, in run
2024-01-12 23:18:02,678 run ERROR | self.target(*self.args)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 4504, in import_project_run
2024-01-12 23:18:02,678 run ERROR | com.error_notification(mongo.db, notification_id, "Unable to import project from %s: %s"%(abs_path_export_project_dir, str(e)))
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_compute/jobs/common.py", line 879, in error_notification
2024-01-12 23:18:02,678 run ERROR | update_notification(db, notification_id, {'message':message, 'status':'danger', 'icon':'error', 'progress_pct' : None})
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_compute/jobs/common.py", line 850, in update_notification
2024-01-12 23:18:02,678 run ERROR | db['notifications'].update_one({'_id' : objectid.ObjectId(notification_id)}, {operation : data})
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/collection.py", line 1132, in update_one
2024-01-12 23:18:02,678 run ERROR | self._update_retryable(
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/collection.py", line 961, in _update_retryable
2024-01-12 23:18:02,678 run ERROR | return self.__database.client._retryable_write(
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1644, in _retryable_write
2024-01-12 23:18:02,678 run ERROR | return self._retry_with_session(retryable, func, s, None)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1532, in _retry_with_session
2024-01-12 23:18:02,678 run ERROR | return self._retry_internal(retryable, func, session, bulk)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1571, in _retry_internal
2024-01-12 23:18:02,678 run ERROR | raise last_error
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1565, in _retry_internal
2024-01-12 23:18:02,678 run ERROR | return func(session, sock_info, retryable)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/collection.py", line 942, in _update
2024-01-12 23:18:02,678 run ERROR | return self._update(
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/collection.py", line 898, in _update
2024-01-12 23:18:02,678 run ERROR | result = sock_info.command(
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 769, in command
2024-01-12 23:18:02,678 run ERROR | self._raise_connection_failure(error)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 969, in _raise_connection_failure
2024-01-12 23:18:02,678 run ERROR | _raise_connection_failure(self.address, error)
2024-01-12 23:18:02,678 run ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/pool.py", line 266, in _raise_connection_failure
2024-01-12 23:18:02,678 run ERROR | raise AutoReconnect(msg)
2024-01-12 23:18:02,678 run ERROR | pymongo.errors.AutoReconnect: shipmhpc:45101: [Errno 104] Connection reset by peer
2024-01-12 23:18:32,752 background_worker ERROR | Concurrent Job Monitor failed
2024-01-12 23:18:32,752 background_worker ERROR | Traceback (most recent call last):
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 170, in background_worker
2024-01-12 23:18:32,752 background_worker ERROR | concurrent_job_monitor()
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 2172, in concurrent_job_monitor
2024-01-12 23:18:32,752 background_worker ERROR | current_concurrent_licenses_deque.append(get_num_active_licenses())
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/commandcommon.py", line 186, in wrapper
2024-01-12 23:18:32,752 background_worker ERROR | return func(*args, **kwargs)
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/cryosparc_command/command_core/__init__.py", line 2165, in get_num_active_licenses
2024-01-12 23:18:32,752 background_worker ERROR | for j in jobs_running:
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py", line 1280, in next
2024-01-12 23:18:32,752 background_worker ERROR | if len(self.__data) or self._refresh():
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/cursor.py", line 1165, in _refresh
2024-01-12 23:18:32,752 background_worker ERROR | self.__session = self.__collection.database.client._ensure_session()
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2027, in _ensure_session
2024-01-12 23:18:32,752 background_worker ERROR | return self.__start_session(True, causal_consistency=False)
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 1976, in __start_session
2024-01-12 23:18:32,752 background_worker ERROR | server_session = self._get_server_session()
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/mongo_client.py", line 2013, in _get_server_session
2024-01-12 23:18:32,752 background_worker ERROR | return self._topology.get_server_session()
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 525, in get_server_session
2024-01-12 23:18:32,752 background_worker ERROR | session_timeout = self._check_session_support()
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 507, in _check_session_support
2024-01-12 23:18:32,752 background_worker ERROR | self._select_servers_loop(
2024-01-12 23:18:32,752 background_worker ERROR | File "/cm/shared/apps/cryosparc/cylab/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.8/site-packages/pymongo/topology.py", line 226, in _select_servers_loop
2024-01-12 23:18:32,752 background_worker ERROR | raise ServerSelectionTimeoutError(
2024-01-12 23:18:32,752 background_worker ERROR | pymongo.errors.ServerSelectionTimeoutError: shipmhpc:45101: [Errno 111] Connection refused, Timeout: 30s, Topology Description: <TopologyDescription id: 65a13393995ee018df22100e, topology_type: Single, servers: [<ServerDescription ('shipmhpc', 45101) server_type: Unknown, rtt: None, error=AutoReconnect('shipmhpc:45101: [Errno 111] Connection refused')>]>
2024-01-12 23:18:32,753 run ERROR | POST-RESPONSE-THREAD ERROR at import_project_run
Thanks a lot.