Metadata | Log tab
================= CRYOSPARCW ======= 2023-01-27 00:45:44.539597 =========
Project P3 Job J129
Master bioinform01 Port 39002
========= monitor process now starting main process at 2023-01-27 00:45:44.539637
MAINPROCESS PID 675548
========= monitor process now waiting for main process
MAIN PID 675548
topaz.run_topaz cryosparc_compute.jobs.jobregister
========= sending heartbeat at 2023-01-27 00:45:58.398959
========= sending heartbeat at 2023-01-27 00:46:08.832713
========= sending heartbeat at 2023-01-27 00:46:19.106545
========= sending heartbeat at 2023-01-27 00:46:29.541990
========= sending heartbeat at 2023-01-27 00:46:39.981597
========= sending heartbeat at 2023-01-27 00:46:50.541644
========= sending heartbeat at 2023-01-27 00:47:00.773666
========= sending heartbeat at 2023-01-27 00:47:11.089586
========= sending heartbeat at 2023-01-27 00:47:21.426765
========= sending heartbeat at 2023-01-27 00:47:31.865655
========= sending heartbeat at 2023-01-27 00:47:42.409606
========= sending heartbeat at 2023-01-27 00:47:53.014593
========= sending heartbeat at 2023-01-27 00:48:03.398787
uid: invalid handle 1, wrong generation counter (given 0, expected 32768) (errno 1: Operation not permitted)
add column: invalid handle 1, wrong generation counter (given 0, expected 32768) (errno 1: Operation not permitted)
Running job on hostname %s bioinform01
Allocated Resources : {‘fixed’: {‘SSD’: False}, ‘hostname’: ‘bioinform01’, ‘lane’: ‘default’, ‘lane_type’: ‘node’, ‘license’: False, ‘licenses_acquired’: 0, ‘slots’: {‘CPU’: [14, 15, 16, 17, 18, 19, 20, 21], ‘GPU’: [1], ‘RAM’: [3]}, ‘target’: {‘cache_path’: ‘/scratch/cryosparc_cache’, ‘cache_quota_mb’: None, ‘cache_reserve_mb’: 10000, ‘desc’: None, ‘gpus’: [{‘id’: 0, ‘mem’: 25396576256, ‘name’: ‘Quadro RTX 6000’}, {‘id’: 1, ‘mem’: 25396969472, ‘name’: ‘Quadro RTX 6000’}, {‘id’: 2, ‘mem’: 25396969472, ‘name’: ‘Quadro RTX 6000’}], ‘hostname’: ‘bioinform01’, ‘lane’: ‘default’, ‘monitor_port’: None, ‘name’: ‘bioinform01’, ‘resource_fixed’: {‘SSD’: True}, ‘resource_slots’: {‘CPU’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26, 27, 28, 29, 30, 31, 32, 33, 34, 35, 36, 37, 38, 39], ‘GPU’: [0, 1, 2], ‘RAM’: [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22, 23]}, ‘ssh_str’: ‘cryosparc_user@bioinform01’, ‘title’: ‘Worker node bioinform01’, ‘type’: ‘node’, ‘worker_bin_path’: ‘/cryosparc/cryosparc_worker/bin/cryosparcw’}}
**** handle exception rc
Traceback (most recent call last):
File “/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 256, in run_topaz_wrapper_train
curr_mic = micrographs_dset.query({‘uid’: curr_original_mic_uid})
File “/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 1202, in query
return self.mask(self.query_mask(query))
File “/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 1265, in mask
return type(self)([(f, self[f][mask]) for f in self])
File “/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 638, in init
self.add_fields([entry[0] for entry in populate])
File “/cryosparc/cryosparc_worker/cryosparc_tools/cryosparc/dataset.py”, line 931, in add_fields
assert self._data.addcol_scalar(
AssertionError: Could not add (‘uid’, ‘<u8’) with dtype uint64
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 96, in cryosparc_compute.run.main
File “/cryosparc/cryosparc_worker/cryosparc_compute/jobs/topaz/run_topaz.py”, line 260, in run_topaz_wrapper_train
assert False, “Particles do not originate from input micrographs.”
AssertionError: Particles do not originate from input micrographs.
set status to failed
========= main process now complete at 2023-01-27 00:48:04.451273.
========= monitor process now complete at 2023-01-27 00:48:04.457013.
