Hi, here are the output:
cryosparcm eventlog "$cspid" "$csjid"
[Thu, 25 Jul 2024 11:02:19 GMT] License is valid.
[Thu, 25 Jul 2024 11:02:19 GMT] Running job on master node
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Job J30 Started
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Master running v4.5.3, worker running v4.5.3
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Working in directory: /xx/J30
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Running on lane None
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Resources allocated:
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Worker: puhti-login15.bullx
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] --------------------------------------------------------------
[Thu, 25 Jul 2024 11:03:03 GMT] [CPU RAM used: 83 MB] Importing job module for job type inspect_picks_v2...
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] Job ready to run
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] ***************************************************************
[Thu, 25 Jul 2024 11:03:42 GMT] [CPU RAM used: 218 MB] Interactive job running on port 33821
[Thu, 25 Jul 2024 11:04:54 GMT] [CPU RAM used: 2708 MB] Loaded info for 12256 micrographs
[Thu, 25 Jul 2024 11:04:54 GMT] [CPU RAM used: 2708 MB] Loaded info for 10119545 particles
[Thu, 25 Jul 2024 11:04:56 GMT] [CPU RAM used: 2708 MB] Calibrating NCC scores to defocus..
[Thu, 25 Jul 2024 11:05:01 GMT] NCC calibration
[Thu, 25 Jul 2024 11:08:27 GMT] [CPU RAM used: 2676 MB] Calibrating Power scores to defocus..
[Thu, 25 Jul 2024 11:08:28 GMT] Power calibration
[Thu, 25 Jul 2024 11:12:51 GMT] [CPU RAM used: 4124 MB] (DF) Loading particle pick data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading CTF data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading CTF stats data...
[Thu, 25 Jul 2024 11:12:53 GMT] [CPU RAM used: 4124 MB] Loading rigid motion data...
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4117 MB] Done in 260.10s
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4117 MB] uid object
idx int64
fname object
project_uid object
job_uid object
num_particles int64
num_particles_threshold int64
pick_ncc_median float32
pick_pow_median float32
df_avg float32
ctf_fit_to_A float32
df_ast float32
ctf_fit_cc float32
phase_shift_deg float32
df_range float32
df_tilt_normal float32
ice_thickness_rel float32
motion_total_pix float64
motion_curvature float64
dtype: object
[Thu, 25 Jul 2024 11:17:13 GMT] [CPU RAM used: 4118 MB] [{'name': 'idx', 'title': 'Index', 'short': 'Index'}, {'name': 'num_particles', 'title': 'Number of picked particles', 'short': 'Num. particles'}, {'name': 'num_particles_threshold', 'title': 'Number of picked particles matching threshold', 'short': 'Num. particles threshold'}, {'name': 'pick_ncc_median', 'title': 'Median pick cc score', 'short': 'Pick CC', 'round': 2}, {'name': 'pick_pow_median', 'title': 'Median pick power score', 'short': 'Pick power', 'round': 2}, {'name': 'df_avg', 'title': 'Average defocus (A)', 'short': 'DF Avg', 'round': 0}, {'name': 'ctf_fit_to_A', 'title': 'CTF fit resolution (A)', 'short': 'CTF Fit', 'round': 2}, {'name': 'df_ast', 'title': 'Astigmatism (A)', 'short': 'Astig'}, {'name': 'df_range', 'title': 'Defocus Range', 'short': 'DF Range'}, {'name': 'df_tilt_normal', 'title': 'Defocus Tilt Angle (degrees)', 'short': 'Tilt Angle'}, {'name': 'ice_thickness_rel', 'title': 'Relative Ice Thickness', 'short': 'Rel Ice Thick.'}, {'name': 'motion_total_pix', 'title': 'Total full-frame motion distance (pixels)', 'short': 'Motion dist.'}, {'name': 'motion_curvature', 'title': 'Full-frame motion curvature', 'short': 'Motion curv.'}, {'name': 'uid', 'title': 'UID', 'short': 'UID'}, {'name': 'fname', 'title': 'File name', 'short': 'File'}]
[Thu, 25 Jul 2024 11:19:52 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 220
[Thu, 25 Jul 2024 11:20:09 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 189
[Thu, 25 Jul 2024 11:20:24 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 155
[Thu, 25 Jul 2024 11:20:39 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 87
[Thu, 25 Jul 2024 11:20:56 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:21:12 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0.42, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:21:52 GMT] [CPU RAM used: 4159 MB] Thresholds changed: NCC : 0.93, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:08 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 0.95, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:24 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.02, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:31 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.14, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:40 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.14, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:46 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.19, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:22:54 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.08, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:24:01 GMT] [CPU RAM used: 4158 MB] Thresholds changed: NCC : 1.48, Power Min: 4, Power Max: 68
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] --------------------------------------------------------
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] Outputting particles that have:
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] NCC score > 1.480
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] Local power > 4.000
[Thu, 25 Jul 2024 11:24:17 GMT] [CPU RAM used: 4158 MB] Local power < 68.000
[Thu, 25 Jul 2024 11:24:20 GMT] Pick stats and final thresholds
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] 3191013 particles included, 6928532 particles excluded.
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] --------------------------------------------------------
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:31 GMT] Extracted coordinates on J3/motioncorrected/010929021455981439625_FoilHole_18255919_Data_18134883_0_20240618_235540_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:31 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT] Extracted coordinates on J3/motioncorrected/005438301115846311077_FoilHole_18255920_Data_18134892_1_20240618_235624_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT] Extracted coordinates on J3/motioncorrected/009765333374663387756_FoilHole_18255921_Data_18134901_6_20240618_235704_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT] Extracted coordinates on J3/motioncorrected/010835682593195539140_FoilHole_18255923_Data_18134889_8_20240618_235745_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:32 GMT] Extracted coordinates on J3/motioncorrected/012629020083643638179_FoilHole_18255924_Data_18134898_4_20240618_235826_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:32 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT] Extracted coordinates on J3/motioncorrected/010325923352975725172_FoilHole_18255926_Data_18134886_2_20240618_235907_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT] Extracted coordinates on J3/motioncorrected/018347129054301543381_FoilHole_18255927_Data_18134895_8_20240619_000204_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:33 GMT] Extracted coordinates on J3/motioncorrected/011816606238871428378_FoilHole_18255929_Data_18134883_7_20240619_000848_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:33 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:34 GMT] Extracted coordinates on J3/motioncorrected/015959081893009637711_FoilHole_18255930_Data_18134892_4_20240619_000929_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:34 GMT] [CPU RAM used: 4160 MB] Loading micrograph..
[Thu, 25 Jul 2024 11:24:34 GMT] Extracted coordinates on J3/motioncorrected/006207268421323697032_FoilHole_18255931_Data_18134901_8_20240619_001010_Fractions_patch_aligned_doseweighted.mrc
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] ==== Completed. Extracted 3191013 particles.
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Interactive backend shutting down.
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] --------------------------------------------------------------
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Compiling job outputs...
[Thu, 25 Jul 2024 11:24:37 GMT] [CPU RAM used: 4151 MB] Passing through outputs for output group micrographs from input group micrographs
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] This job outputted results ['micrograph_blob']
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] Loaded output dset with 12256 items
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] Passthrough results ['ctf', 'ctf_stats', 'rigid_motion', 'spline_motion', 'mscope_params', 'background_blob', 'micrograph_blob_denoised', 'micrograph_thumbnail_blob_1x', 'micrograph_thumbnail_blob_2x', 'micrograph_blob_non_dw', 'micrograph_blob_non_dw_AB', 'movie_blob', 'gain_ref_blob']
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] Loaded passthrough dset with 12256 items
[Thu, 25 Jul 2024 11:24:38 GMT] [CPU RAM used: 4151 MB] Intersection of output and passthrough has 12256 items
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Output dataset contains: ['ctf_stats', 'micrograph_thumbnail_blob_1x', 'gain_ref_blob', 'rigid_motion', 'micrograph_thumbnail_blob_2x', 'micrograph_blob_non_dw', 'movie_blob', 'spline_motion', 'mscope_params', 'micrograph_blob_denoised', 'background_blob', 'ctf', 'micrograph_blob_non_dw_AB']
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result ctf
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result ctf_stats
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result rigid_motion
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result spline_motion
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result mscope_params
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result background_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result micrograph_blob_denoised
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result micrograph_thumbnail_blob_1x
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result micrograph_thumbnail_blob_2x
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result micrograph_blob_non_dw
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result micrograph_blob_non_dw_AB
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result movie_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Outputting passthrough result gain_ref_blob
[Thu, 25 Jul 2024 11:24:39 GMT] [CPU RAM used: 4151 MB] Passing through outputs for output group particles from input group particles
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB] This job outputted results ['location']
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB] Loaded output dset with 3191013 items
[Thu, 25 Jul 2024 11:24:41 GMT] [CPU RAM used: 4154 MB] Passthrough results ['pick_stats', 'ctf']
[Thu, 25 Jul 2024 11:24:47 GMT] [CPU RAM used: 7049 MB] Traceback (most recent call last):
File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 567, in load
dset = cls(indata)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 770, in __init__
self._data.addrows(nrows)
File "cryosparc/core.pyx", line 108, in cryosparc.core.Data.addrows
MemoryError: Could not add rows to dataset
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 119, in cryosparc_master.cryosparc_compute.run.main
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1276, in passthrough_outputs
dset = load_input_group(input_group_name, passthrough_result_names, allow_passthrough=True, memoize=True)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in load_input_group
dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in <listcomp>
dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 648, in load_input_connection_slots
dset = load_input_connection_single_slot(input_group_name, slot_name, connection_idx, allow_passthrough=allow_passthrough, memoize=memoize)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 638, in load_input_connection_single_slot
d = load_output_result_dset(_project_uid, output_result, slotconnection['version'], slot_name, memoize=memoize)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 586, in load_output_result_dset
d = load_dataset_cached(abspath).copy()
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 572, in load_dataset_cached
return dataset.Dataset.load(path)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 605, in load
raise DatasetLoadError(f"Could not load dataset from file {file}") from err
cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /xx/J27/picked_particles.cs
cryosparcm joblog "$cspid" "$csjid" | tail -n 40
min: -59.327230 max: 58.863390
min: -58.905836 max: 58.467871
min: -60.135975 max: 59.660972
min: -60.533772 max: 60.051444
min: -58.625220 max: 58.173891
min: -58.849793 max: 58.339229
min: -62.444555 max: 62.013399
**** handle exception rc
Traceback (most recent call last):
File "/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 567, in load
dset = cls(indata)
File "/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 770, in __init__
self._data.addrows(nrows)
File "cryosparc/core.pyx", line 108, in cryosparc.core.Data.addrows
MemoryError: Could not add rows to dataset
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "cryosparc_master/cryosparc_compute/run.py", line 119, in cryosparc_master.cryosparc_compute.run.main
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 1276, in passthrough_outputs
dset = load_input_group(input_group_name, passthrough_result_names, allow_passthrough=True, memoize=True)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in load_input_group
dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 686, in <listcomp>
dsets = [load_input_connection_slots(input_group_name, keep_slot_names, idx, allow_passthrough=allow_passthrough, memoize=memoize, force_innerjoin=force_innerjoin) for idx in range(num_connections)]
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 648, in load_input_connection_slots
dset = load_input_connection_single_slot(input_group_name, slot_name, connection_idx, allow_passthrough=allow_passthrough, memoize=memoize)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 638, in load_input_connection_single_slot
d = load_output_result_dset(_project_uid, output_result, slotconnection['version'], slot_name, memoize=memoize)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 586, in load_output_result_dset
d = load_dataset_cached(abspath).copy()
File “/xx/cryoSPARC/cryosparc_master/cryosparc_compute/jobs/runcommon.py", line 572, in load_dataset_cached
return dataset.Dataset.load(path)
File “/xx/cryoSPARC/cryosparc_master/cryosparc_tools/cryosparc/dataset.py", line 605, in load
raise DatasetLoadError(f"Could not load dataset from file {file}") from err
cryosparc_tools.cryosparc.errors.DatasetLoadError: Could not load dataset from file /xx/J27/picked_particles.cs
set status to failed
========= main process now complete at 2024-07-25 14:24:57.235271.
========= monitor process now complete at 2024-07-25 14:24:57.239357.