Template picker (J20) ran for ~7hrs, completing >9000 of 14,500 images, before stopping abnormally. Twice. I marked the job complete, but no files were output (queued extract job immediately failed with no input). Any ideas? Re-downloading micrographs now in case of damaged files. I have not repeated the job since restarting cryoSPARC, but that did not recover the missing outputs.
Copying path to any output files yields:
Unable to get result path: ServerError: list index out of range
End of picking Event Log:
[CPU: 12.37 GB Avail: 42.55 GB]
Completed 9639 of 14573 : J3/imported/003646671211009403497_FoilHole_23840554_Data_23836515_23836517_20221223_165300_fractions.mrc
Picked 557 particles in 4.41s (27489.94s total)
[CPU: 12.36 GB Avail: 42.57 GB]Completed 9640 of 14573 : J3/imported/008234090351039803370_FoilHole_23840554_Data_23836518_23836520_20221223_165303_fractions.mrc
Picked 565 particles in 4.42s (27494.36s total)
[CPU: 173.2 MB Avail: 54.91 GB]====== Job process terminated abnormally.
[CPU: 977.8 MB Avail: 54.94 GB]Finalizing Job…
[CPU: 977.8 MB Avail: 54.94 GB]Passing through outputs for output group micrographs from input group micrographs
[CPU: 977.8 MB Avail: 54.94 GB]This job outputted results [‘micrograph_blob’]
[CPU: 977.8 MB Avail: 54.94 GB]Loaded output dset with 0 items
[CPU: 977.8 MB Avail: 54.94 GB]Passthrough results [‘ctf’, ‘mscope_params’, ‘ctf_stats’, ‘micrograph_blob_non_dw’]
[CPU: 987.1 MB Avail: 54.93 GB]Loaded passthrough dset with 14573 items
[CPU: 987.1 MB Avail: 54.93 GB]Intersection of output and passthrough has 0 items
[CPU: 987.1 MB Avail: 54.93 GB]Checking outputs for output group micrographs
[CPU: 988.0 MB Avail: 54.93 GB]Done
**** Job marked as completed by James Kizziah (ID: 63d80a8a2dcf5a7bd3934552) ****
Extraction traceback:
[CPU: 201.3 MB Avail: 54.41 GB]
Traceback (most recent call last):
File “cryosparc_master/cryosparc_compute/run.py”, line 96, in cryosparc_compute.run.main
File “/home/cryosparcuser/cryosparc/cryosparc_worker/cryosparc_compute/jobs/extract/run.py”, line 344, in run_extract_micrographs_multi
assert micrographs_dset is not None, ‘Could not load micrographs input group’
AssertionError: Could not load micrographs input group
Relevant section of command_core log:
2023-02-03 08:15:24,345 COMMAND.JOBS set_job_status INFO | Status changed for P1.J20 from launched to started
2023-02-03 08:15:27,504 COMMAND.JOBS set_job_status INFO | Status changed for P1.J20 from started to running
2023-02-03 08:25:37,155 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 09:25:37,929 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 10:25:38,856 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 11:25:39,635 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 12:25:40,158 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 13:25:40,376 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 14:25:40,638 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 15:25:41,334 COMMAND.BG_WORKER background_worker INFO | License does not have telemetry enabled; will re-check license in 1 hour.
2023-02-03 15:53:58,885 COMMAND.JOBS set_job_status INFO | Status changed for P1.J20 from running to failed
Couldn’t find anything this license notice. Is that the problem?
End of J20 job log:
========= sending heartbeat at 2023-02-03 15:53:44.135831
========= sending heartbeat at 2023-02-03 15:53:54.149356
========= main process now complete at 2023-02-03 15:53:58.872155.
========= monitor process now complete at 2023-02-03 15:53:58.948209.
Nothing in database log immediately before time of error.
Single workstation, cryoSPARC v4.1.2
Linux 3.10.0-1160.81.1.el7.x86_64 #1 SMP Fri Dec 16 17:29:43 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
total used free shared buff/cache available
Mem: 62 5 19 0 37 56
Swap: 31 0 31