Using extracted particles from an interrupted local motion correction job

Hi everyone!

This is not the first time similar thing happens and that I question myself:

I have a local motion correction job that has terminated due to an unknown error half-way. For the moment I don’t mind proceeding with the subset of particles that it did extract up to the moment the job terminated, however the metadata file is probably only output at the end of the job, therefore missing from the directory. In this way I cannot import that particle stack…

Does anyone know if there is a way one could still use the already extracted particles, or is the only solution to restart the job and wait until it finishes?

Cheers,
André

Please can you post the final lines of the Event Log and any errors that may be present in the job log (Metadata|Log).

1 Like

I have just made a suggestion not long ago regarding this issue.:

I think there is no way you can recover the meta informations, because they only exist in the worker’s memory during the job run.

(Another problem of holding a result list in memory without periodic saving is that it may cause out-of-memory errors if the dataset is very large.)

One temporary solution is to always chop your data into smaller chunks, anticipating such failures.

1 Like

Thank you both for your answers.

@wtempel, the log file is huge and repeating often the same, I will paste it at the end of this message (I ended up truncating parts that look exactly the same). However I wanted to say that for me the aim with this topic was not to solve this error and I was sure this was an isolate error that would not repeat if the job was restarted (and so it was, meanwhile tested). It would be nice to have a way to use extracted particles up to that moment,

@ZhijieLi thanks for also making this suggestion! Also thank you for the tips.

Metadata:

{“id”:“64f565ea8329d4b9760692f8”,“children”:[],“cloned_from”:null,“completed_at”:null,“created_at”:“2023-09-04T05:06:50.372Z”,“created_by_user_id”:“61b3622b120e121db5492b2e”,“deleted”:false,“description”:“Enter a description.”,“failed_at”:“2023-09-04T18:17:45.794Z”,“interactive”:false,“interactive_hostname”:“R2D2”,“interactive_port”:null,“is_ancestor_of_final_result”:false,“is_final_result”:false,“job_type”:“local_motion_correction”,“killed_at”:null,“last_exported”:null,“launched_at”:“2023-09-04T12:10:24.773Z”,“output_group_images”:{“particles”:“64f5c94557bf11733ce562c6”},“output_result_groups”:[{“uid”:“J82-G0”,“type”:“particle”,“name”:“particles”,“title”:“Particles extracted”,“description”:“”,“contains”:[{“uid”:“J82-R0”,“type”:“particle.blob”,“group_name”:“particles”,“name”:“blob”,“passthrough”:false},{“uid”:“J82-R1”,“type”:“particle.motion”,“group_name”:“particles”,“name”:“motion”,“passthrough”:false},{“uid”:“J82-R2”,“type”:“particle.location”,“group_name”:“particles”,“name”:“location”,“passthrough”:true},{“uid”:“J82-R3”,“type”:“particle.alignments2D”,“group_name”:“particles”,“name”:“alignments2D”,“passthrough”:true},{“uid”:“J82-R4”,“type”:“particle.ctf”,“group_name”:“particles”,“name”:“ctf”,“passthrough”:true},{“uid”:“J82-R5”,“type”:“particle.pick_stats”,“group_name”:“particles”,“name”:“pick_stats”,“passthrough”:true}],“passthrough”:“particles”,“num_items”:0,“summary”:{}},{“uid”:“J82-G1”,“type”:“exposure”,“name”:“movies”,“title”:“Movies”,“description”:“”,“contains”:[{“uid”:“J82-R6”,“type”:“exposure.movie_blob”,“group_name”:“movies”,“name”:“movie_blob”,“passthrough”:false},{“uid”:“J82-R7”,“type”:“exposure.gain_ref_blob”,“group_name”:“movies”,“name”:“gain_ref_blob”,“passthrough”:true},{“uid”:“J82-R8”,“type”:“exposure.mscope_params”,“group_name”:“movies”,“name”:“mscope_params”,“passthrough”:true},{“uid”:“J82-R9”,“type”:“exposure.stat_blob”,“group_name”:“movies”,“name”:“background_blob”,“passthrough”:true},{“uid”:“J82-R10”,“type”:“exposure.motion”,“group_name”:“movies”,“name”:“rigid_motion”,“passthrough”:true},{“uid”:“J82-R11”,“type”:“exposure.ctf”,“group_name”:“movies”,“name”:“ctf”,“passthrough”:true},{“uid”:“J82-R12”,“type”:“exposure.micrograph_blob”,“group_name”:“movies”,“name”:“micrograph_blob”,“passthrough”:true},{“uid”:“J82-R13”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies”,“name”:“micrograph_thumbnail_blob_1x”,“passthrough”:true},{“uid”:“J82-R14”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies”,“name”:“micrograph_thumbnail_blob_2x”,“passthrough”:true},{“uid”:“J82-R15”,“type”:“exposure.ctf_stats”,“group_name”:“movies”,“name”:“ctf_stats”,“passthrough”:true},{“uid”:“J82-R16”,“type”:“exposure.motion”,“group_name”:“movies”,“name”:“spline_motion”,“passthrough”:true},{“uid”:“J82-R17”,“type”:“exposure.micrograph_blob”,“group_name”:“movies”,“name”:“micrograph_blob_non_dw”,“passthrough”:true}],“passthrough”:“movies”,“num_items”:0,“summary”:{}},{“uid”:“J82-G2”,“type”:“exposure”,“name”:“movies_incomplete”,“title”:“Movies incomplete”,“description”:“”,“contains”:[{“uid”:“J82-R18”,“type”:“exposure.movie_blob”,“group_name”:“movies_incomplete”,“name”:“movie_blob”,“passthrough”:false},{“uid”:“J82-R19”,“type”:“exposure.gain_ref_blob”,“group_name”:“movies_incomplete”,“name”:“gain_ref_blob”,“passthrough”:true},{“uid”:“J82-R20”,“type”:“exposure.mscope_params”,“group_name”:“movies_incomplete”,“name”:“mscope_params”,“passthrough”:true},{“uid”:“J82-R21”,“type”:“exposure.stat_blob”,“group_name”:“movies_incomplete”,“name”:“background_blob”,“passthrough”:true},{“uid”:“J82-R22”,“type”:“exposure.motion”,“group_name”:“movies_incomplete”,“name”:“rigid_motion”,“passthrough”:true},{“uid”:“J82-R23”,“type”:“exposure.ctf”,“group_name”:“movies_incomplete”,“name”:“ctf”,“passthrough”:true},{“uid”:“J82-R24”,“type”:“exposure.micrograph_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_blob”,“passthrough”:true},{“uid”:“J82-R25”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_thumbnail_blob_1x”,“passthrough”:true},{“uid”:“J82-R26”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_thumbnail_blob_2x”,“passthrough”:true},{“uid”:“J82-R27”,“type”:“exposure.ctf_stats”,“group_name”:“movies_incomplete”,“name”:“ctf_stats”,“passthrough”:true},{“uid”:“J82-R28”,“type”:“exposure.motion”,“group_name”:“movies_incomplete”,“name”:“spline_motion”,“passthrough”:true},{“uid”:“J82-R29”,“type”:“exposure.micrograph_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_blob_non_dw”,“passthrough”:true}],“passthrough”:“movies”,“num_items”:0,“summary”:{}}],“params_spec”:{“res_max_align”:{“value”:2},“box_size_pix”:{“value”:720},“bin_size_pix”:{“value”:180}},“parents”:[“J52”,“J81”],“priority”:0,“project_uid”:“P23”,“project_uid_num”:23,“queue_message”:null,“queued_at”:“2023-09-04T12:10:23.434Z”,“resources_allocated”:{“lane”:“RTX3090”,“lane_type”:“node”,“hostname”:“R2D2”,“target”:{“cache_path”:“/mnt/scratch_large/cryosparc_cache”,“cache_quota_mb”:null,“cache_reserve_mb”:10000,“desc”:null,“gpus”:[{“id”:0,“name”:“NVIDIA GeForce RTX 3090”,“mem”:25447170048}],“hostname”:“R2D2”,“lane”:“RTX3090”,“monitor_port”:null,“name”:“R2D2”,“resource_fixed”:{“SSD”:true},“resource_slots”:{“CPU”:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31,32,33,34,35,36,37,38,39,40,41,42,43,44,45,46,47],“GPU”:[0],“RAM”:[0,1,2,3,4,5,6,7,8,9,10,11,12,13,14,15,16,17,18,19,20,21,22,23,24,25,26,27,28,29,30,31]},“ssh_str”:“angr5008@R2D2”,“title”:“Worker node R2D2”,“type”:“node”,“worker_bin_path”:“/home/angr5008/Software/cryosparc/cryosparc_worker/bin/cryosparcw”},“slots”:{“CPU”:[0,1,2,3,4,5],“GPU”:[0],“RAM”:[0,1]},“fixed”:{“SSD”:false},“license”:true,“licenses_acquired”:1},“running_at”:“2023-09-04T12:10:30.269Z”,“started_at”:“2023-09-04T12:10:28.394Z”,“status”:“failed”,“status_num”:40,“title”:“New Job J82”,“ui_tile_height”:1,“ui_tile_images”:[{“name”:“img0”,“fileid”:“64f5c94557bf11733ce562c4”,“num_rows”:1,“num_cols”:1},{“name”:“img1”,“fileid”:“64f5c94f57bf11733ce562d8”,“num_rows”:1,“num_cols”:1}],“ui_tile_width”:2,“uid”:“J82”,“uid_num”:82,“waiting_at”:null,“workspace_uids”:[“W2”],“input_slot_groups”:[{“type”:“exposure”,“name”:“movies”,“title”:“Movies”,“description”:“Movies for motion correction, already full-frame aligned”,“count_min”:1,“count_max”:null,“repeat_allowed”:false,“slots”:[{“type”:“exposure.movie_blob”,“name”:“movie_blob”,“title”:“Raw movie data”,“description”:“”,“optional”:false},{“type”:“exposure.gain_ref_blob”,“name”:“gain_ref_blob”,“title”:“Raw movie data”,“description”:“”,“optional”:true},{“type”:“exposure.mscope_params”,“name”:“mscope_params”,“title”:“Exposure parameters”,“description”:“”,“optional”:false},{“type”:“exposure.stat_blob”,“name”:“background_blob”,“title”:“Background estimates”,“description”:“”,“optional”:false},{“type”:“exposure.motion”,“name”:“rigid_motion”,“title”:“Full-frame motion estimates”,“description”:“”,“optional”:false},{“type”:“exposure.ctf”,“name”:“ctf”,“title”:“CTF parameters”,“description”:“”,“optional”:true}],“connections”:[{“job_uid”:“J52”,“group_name”:“exposures_accepted”,“slots”:[{“slot_name”:“movie_blob”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“movie_blob”,“result_type”:“exposure.movie_blob”,“version”:“F”},{“slot_name”:“gain_ref_blob”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“gain_ref_blob”,“result_type”:“exposure.gain_ref_blob”,“version”:“F”},{“slot_name”:“mscope_params”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“mscope_params”,“result_type”:“exposure.mscope_params”,“version”:“F”},{“slot_name”:“background_blob”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“background_blob”,“result_type”:“exposure.stat_blob”,“version”:“F”},{“slot_name”:“rigid_motion”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“rigid_motion”,“result_type”:“exposure.motion”,“version”:“F”},{“slot_name”:“ctf”,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“ctf”,“result_type”:“exposure.ctf”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“micrograph_blob”,“result_type”:“exposure.micrograph_blob”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“micrograph_thumbnail_blob_1x”,“result_type”:“exposure.thumbnail_blob”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“micrograph_thumbnail_blob_2x”,“result_type”:“exposure.thumbnail_blob”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“ctf_stats”,“result_type”:“exposure.ctf_stats”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“spline_motion”,“result_type”:“exposure.motion”,“version”:“F”},{“slot_name”:null,“job_uid”:“J52”,“group_name”:“exposures_accepted”,“result_name”:“micrograph_blob_non_dw”,“result_type”:“exposure.micrograph_blob”,“version”:“F”}]}]},{“type”:“particle”,“name”:“particles”,“title”:“Particles”,“description”:“Particle locations at which to perform local motion correction”,“count_min”:0,“count_max”:null,“repeat_allowed”:false,“slots”:[{“type”:“particle.location”,“name”:“location”,“title”:“Particle locations”,“description”:“”,“optional”:false}],“connections”:[{“job_uid”:“J81”,“group_name”:“particles_selected”,“slots”:[{“slot_name”:“location”,“job_uid”:“J81”,“group_name”:“particles_selected”,“result_name”:“location”,“result_type”:“particle.location”,“version”:“F”},{“slot_name”:null,“job_uid”:“J81”,“group_name”:“particles_selected”,“result_name”:“blob”,“result_type”:“particle.blob”,“version”:“F”},{“slot_name”:null,“job_uid”:“J81”,“group_name”:“particles_selected”,“result_name”:“alignments2D”,“result_type”:“particle.alignments2D”,“version”:“F”},{“slot_name”:null,“job_uid”:“J81”,“group_name”:“particles_selected”,“result_name”:“ctf”,“result_type”:“particle.ctf”,“version”:“F”},{“slot_name”:null,“job_uid”:“J81”,“group_name”:“particles_selected”,“result_name”:“pick_stats”,“result_type”:“particle.pick_stats”,“version”:“F”}]}]}],“ui_layouts”:{“P23W2”:{“show”:true,“floater”:false,“top”:3668,“left”:1383,“width”:298,“height”:192,“groups”:[]}},“created_by_job_uid”:null,“errors_build_inputs”:{},“errors_build_params”:{},“intermediate_results_size_bytes”:0,“intermediate_results_size_last_updated”:“2023-09-04T18:17:45.862Z”,“job_dir”:“J82”,“job_dir_size”:27541001061,“job_dir_size_last_updated”:“2023-09-04T18:17:45.862Z”,“last_accessed”:{“name”:“Stormtrooper”,“accessed_at”:“2023-09-09T08:31:56.326Z”},“last_intermediate_data_cleared_amount”:0,“last_intermediate_data_cleared_at”:null,“output_results”:[{“uid”:“J82-R0”,“type”:“particle.blob”,“group_name”:“particles”,“name”:“blob”,“title”:“Particle raw data”,“description”:“”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”],[“sign”,“f4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:false},{“uid”:“J82-R1”,“type”:“particle.motion”,“group_name”:“particles”,“name”:“motion”,“title”:“Particle motion data”,“description”:“”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“idx”,“u4”],[“frame_start”,“u4”],[“frame_end”,“u4”],[“zero_shift_frame”,“u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:false},{“uid”:“J82-R2”,“type”:“particle.location”,“group_name”:“particles”,“name”:“location”,“title”:“Passthrough location”,“description”:“Passthrough from input particles.location (slot_name)”,“min_fields”:[[“micrograph_uid”,“u8”],[“exp_group_id”,“u4”],[“micrograph_path”,“O”],[“micrograph_shape”,“2u4”],[“center_x_frac”,“f4”],[“center_y_frac”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R3”,“type”:“particle.alignments2D”,“group_name”:“particles”,“name”:“alignments2D”,“title”:“Passthrough alignments2D”,“description”:“Passthrough from input particles.alignments2D (result_name)”,“min_fields”:[[“split”,“u4”],[“shift”,“2f4”],[“pose”,“f4”],[“psize_A”,“f4”],[“error”,“f4”],[“error_min”,“f4”],[“resid_pow”,“f4”],[“slice_pow”,“f4”],[“image_pow”,“f4”],[“cross_cor”,“f4”],[“alpha”,“f4”],[“alpha_min”,“f4”],[“weight”,“f4”],[“pose_ess”,“f4”],[“shift_ess”,“f4”],[“class_posterior”,“f4”],[“class”,“u4”],[“class_ess”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R4”,“type”:“particle.ctf”,“group_name”:“particles”,“name”:“ctf”,“title”:“Passthrough ctf”,“description”:“Passthrough from input particles.ctf (result_name)”,“min_fields”:[[“type”,“O”],[“exp_group_id”,“u4”],[“accel_kv”,“f4”],[“cs_mm”,“f4”],[“amp_contrast”,“f4”],[“df1_A”,“f4”],[“df2_A”,“f4”],[“df_angle_rad”,“f4”],[“phase_shift_rad”,“f4”],[“scale”,“f4”],[“scale_const”,“f4”],[“shift_A”,“2f4”],[“tilt_A”,“2f4”],[“trefoil_A”,“2f4”],[“tetra_A”,“4f4”],[“anisomag”,“4f4”],[“bfactor”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R5”,“type”:“particle.pick_stats”,“group_name”:“particles”,“name”:“pick_stats”,“title”:“Passthrough pick_stats”,“description”:“Passthrough from input particles.pick_stats (result_name)”,“min_fields”:[[“ncc_score”,“f4”],[“power”,“f4”],[“template_idx”,“u4”],[“angle_rad”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R6”,“type”:“exposure.movie_blob”,“group_name”:“movies”,“name”:“movie_blob”,“title”:“Movie data”,“description”:“”,“min_fields”:[[“path”,“O”],[“shape”,“3u4”],[“psize_A”,“f4”],[“is_gain_corrected”,“u4”],[“format”,“O”],[“has_defect_file”,“u4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:false},{“uid”:“J82-R7”,“type”:“exposure.gain_ref_blob”,“group_name”:“movies”,“name”:“gain_ref_blob”,“title”:“Passthrough gain_ref_blob”,“description”:“Passthrough from input movies.gain_ref_blob (slot_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“flip_x”,“u4”],[“flip_y”,“u4”],[“rotate_num”,“u4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R8”,“type”:“exposure.mscope_params”,“group_name”:“movies”,“name”:“mscope_params”,“title”:“Passthrough mscope_params”,“description”:“Passthrough from input movies.mscope_params (slot_name)”,“min_fields”:[[“accel_kv”,“f4”],[“cs_mm”,“f4”],[“total_dose_e_per_A2”,“f4”],[“phase_plate”,“u4”],[“neg_stain”,“u4”],[“exp_group_id”,“u4”],[“defect_path”,“O”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R9”,“type”:“exposure.stat_blob”,“group_name”:“movies”,“name”:“background_blob”,“title”:“Passthrough background_blob”,“description”:“Passthrough from input movies.background_blob (slot_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“binfactor”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R10”,“type”:“exposure.motion”,“group_name”:“movies”,“name”:“rigid_motion”,“title”:“Passthrough rigid_motion”,“description”:“Passthrough from input movies.rigid_motion (slot_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“idx”,“u4”],[“frame_start”,“u4”],[“frame_end”,“u4”],[“zero_shift_frame”,“u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R11”,“type”:“exposure.ctf”,“group_name”:“movies”,“name”:“ctf”,“title”:“Passthrough ctf”,“description”:“Passthrough from input movies.ctf (slot_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“exp_group_id”,“u4”],[“accel_kv”,“f4”],[“cs_mm”,“f4”],[“amp_contrast”,“f4”],[“df1_A”,“f4”],[“df2_A”,“f4”],[“df_angle_rad”,“f4”],[“phase_shift_rad”,“f4”],[“cross_corr_ctffind4”,“f4”],[“ctf_fit_to_A”,“f4”],[“fig_of_merit_gctf”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R12”,“type”:“exposure.micrograph_blob”,“group_name”:“movies”,“name”:“micrograph_blob”,“title”:“Passthrough micrograph_blob”,“description”:“Passthrough from input movies.micrograph_blob (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”],[“format”,“O”],[“is_background_subtracted”,“u4”],[“vmin”,“f4”],[“vmax”,“f4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R13”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies”,“name”:“micrograph_thumbnail_blob_1x”,“title”:“Passthrough micrograph_thumbnail_blob_1x”,“description”:“Passthrough from input movies.micrograph_thumbnail_blob_1x (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“format”,“O”],[“binfactor”,“u4”],[“micrograph_path”,“O”],[“vmin”,“f4”],[“vmax”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R14”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies”,“name”:“micrograph_thumbnail_blob_2x”,“title”:“Passthrough micrograph_thumbnail_blob_2x”,“description”:“Passthrough from input movies.micrograph_thumbnail_blob_2x (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“format”,“O”],[“binfactor”,“u4”],[“micrograph_path”,“O”],[“vmin”,“f4”],[“vmax”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R15”,“type”:“exposure.ctf_stats”,“group_name”:“movies”,“name”:“ctf_stats”,“title”:“Passthrough ctf_stats”,“description”:“Passthrough from input movies.ctf_stats (result_name)”,“min_fields”:[[“type”,“O”],[“ctf_fit_to_A”,“f4”],[“cross_corr”,“f4”],[“df_range”,“2f4”],[“df_tilt_normal”,“2f4”],[“ice_thickness_rel”,“f4”],[“spectrum_dim”,“u4”],[“diag_image_path”,“O”],[“fit_data_path”,“O”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R16”,“type”:“exposure.motion”,“group_name”:“movies”,“name”:“spline_motion”,“title”:“Passthrough spline_motion”,“description”:“Passthrough from input movies.spline_motion (result_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“idx”,“u4”],[“frame_start”,“u4”],[“frame_end”,“u4”],[“zero_shift_frame”,“u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R17”,“type”:“exposure.micrograph_blob”,“group_name”:“movies”,“name”:“micrograph_blob_non_dw”,“title”:“Passthrough micrograph_blob_non_dw”,“description”:“Passthrough from input movies.micrograph_blob_non_dw (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”],[“format”,“O”],[“is_background_subtracted”,“u4”],[“vmin”,“f4”],[“vmax”,“f4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R18”,“type”:“exposure.movie_blob”,“group_name”:“movies_incomplete”,“name”:“movie_blob”,“title”:“Movie data”,“description”:“”,“min_fields”:[[“path”,“O”],[“shape”,“3u4”],[“psize_A”,“f4”],[“is_gain_corrected”,“u4”],[“format”,“O”],[“has_defect_file”,“u4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:false},{“uid”:“J82-R19”,“type”:“exposure.gain_ref_blob”,“group_name”:“movies_incomplete”,“name”:“gain_ref_blob”,“title”:“Passthrough gain_ref_blob”,“description”:“Passthrough from input movies.gain_ref_blob (slot_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“flip_x”,“u4”],[“flip_y”,“u4”],[“rotate_num”,“u4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R20”,“type”:“exposure.mscope_params”,“group_name”:“movies_incomplete”,“name”:“mscope_params”,“title”:“Passthrough mscope_params”,“description”:“Passthrough from input movies.mscope_params (slot_name)”,“min_fields”:[[“accel_kv”,“f4”],[“cs_mm”,“f4”],[“total_dose_e_per_A2”,“f4”],[“phase_plate”,“u4”],[“neg_stain”,“u4”],[“exp_group_id”,“u4”],[“defect_path”,“O”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R21”,“type”:“exposure.stat_blob”,“group_name”:“movies_incomplete”,“name”:“background_blob”,“title”:“Passthrough background_blob”,“description”:“Passthrough from input movies.background_blob (slot_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“binfactor”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R22”,“type”:“exposure.motion”,“group_name”:“movies_incomplete”,“name”:“rigid_motion”,“title”:“Passthrough rigid_motion”,“description”:“Passthrough from input movies.rigid_motion (slot_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“idx”,“u4”],[“frame_start”,“u4”],[“frame_end”,“u4”],[“zero_shift_frame”,“u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R23”,“type”:“exposure.ctf”,“group_name”:“movies_incomplete”,“name”:“ctf”,“title”:“Passthrough ctf”,“description”:“Passthrough from input movies.ctf (slot_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“exp_group_id”,“u4”],[“accel_kv”,“f4”],[“cs_mm”,“f4”],[“amp_contrast”,“f4”],[“df1_A”,“f4”],[“df2_A”,“f4”],[“df_angle_rad”,“f4”],[“phase_shift_rad”,“f4”],[“cross_corr_ctffind4”,“f4”],[“ctf_fit_to_A”,“f4”],[“fig_of_merit_gctf”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R24”,“type”:“exposure.micrograph_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_blob”,“title”:“Passthrough micrograph_blob”,“description”:“Passthrough from input movies.micrograph_blob (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”],[“format”,“O”],[“is_background_subtracted”,“u4”],[“vmin”,“f4”],[“vmax”,“f4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R25”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_thumbnail_blob_1x”,“title”:“Passthrough micrograph_thumbnail_blob_1x”,“description”:“Passthrough from input movies.micrograph_thumbnail_blob_1x (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“format”,“O”],[“binfactor”,“u4”],[“micrograph_path”,“O”],[“vmin”,“f4”],[“vmax”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R26”,“type”:“exposure.thumbnail_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_thumbnail_blob_2x”,“title”:“Passthrough micrograph_thumbnail_blob_2x”,“description”:“Passthrough from input movies.micrograph_thumbnail_blob_2x (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“format”,“O”],[“binfactor”,“u4”],[“micrograph_path”,“O”],[“vmin”,“f4”],[“vmax”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R27”,“type”:“exposure.ctf_stats”,“group_name”:“movies_incomplete”,“name”:“ctf_stats”,“title”:“Passthrough ctf_stats”,“description”:“Passthrough from input movies.ctf_stats (result_name)”,“min_fields”:[[“type”,“O”],[“ctf_fit_to_A”,“f4”],[“cross_corr”,“f4”],[“df_range”,“2f4”],[“df_tilt_normal”,“2f4”],[“ice_thickness_rel”,“f4”],[“spectrum_dim”,“u4”],[“diag_image_path”,“O”],[“fit_data_path”,“O”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R28”,“type”:“exposure.motion”,“group_name”:“movies_incomplete”,“name”:“spline_motion”,“title”:“Passthrough spline_motion”,“description”:“Passthrough from input movies.spline_motion (result_name)”,“min_fields”:[[“type”,“O”],[“path”,“O”],[“idx”,“u4”],[“frame_start”,“u4”],[“frame_end”,“u4”],[“zero_shift_frame”,“u4”],[“psize_A”,“f4”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true},{“uid”:“J82-R29”,“type”:“exposure.micrograph_blob”,“group_name”:“movies_incomplete”,“name”:“micrograph_blob_non_dw”,“title”:“Passthrough micrograph_blob_non_dw”,“description”:“Passthrough from input movies.micrograph_blob_non_dw (result_name)”,“min_fields”:[[“path”,“O”],[“idx”,“u4”],[“shape”,“2u4”],[“psize_A”,“f4”],[“format”,“O”],[“is_background_subtracted”,“u4”],[“vmin”,“f4”],[“vmax”,“f4”],[“import_sig”,“u8”]],“versions”:[],“metafiles”:[],“num_items”:[],“passthrough”:true}],“params_base”:{“do_plots”:{“type”:“boolean”,“value”:true,“title”:“Make motion diagnostic plots”,“desc”:“Whether or not to make diagnostic plots.”,“order”:0,“section”:“general_settings”,“advanced”:false,“hidden”:false},“num_plots”:{“type”:“number”,“value”:10,“title”:“Number of movies to plot”,“desc”:“Only make plots for the first this many movies.”,“order”:1,“section”:“general_settings”,“advanced”:false,“hidden”:false},“random_num”:{“type”:“number”,“value”:null,“title”:“Only process this many movies”,“desc”:“Randomly select this many movies to process. Helpful for tweaking params.”,“order”:2,“section”:“general_settings”,“advanced”:false,“hidden”:false},“res_max_align”:{“type”:“number”,“value”:5,“title”:“Maximum alignment resolution (A)”,“desc”:“Maximum resolution (in A) to consider when aligning frames. Generally, betwen 5A and 3A is best.”,“order”:3,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“bfactor”:{“type”:“number”,“value”:500,“title”:“B-factor during alignment”,“desc”:“B-factor that blurs frames before aligning. Generally 500 to 100 is best.”,“order”:4,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“smooth_lambda_cal”:{“type”:“number”,“value”:20,“title”:“Calibrated smoothing lambda”,“desc”:“Calibrated smoothing constant applied to trajectories (None to autotune)”,“order”:5,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“smooth_lambda”:{“type”:“number”,“value”:null,“title”:“Raw smoothing lambda”,“desc”:“Smoothing constant applied to trajectories (None to autotune)”,“order”:6,“section”:“motion_settings”,“advanced”:false,“hidden”:true},“smooth_backoff”:{“type”:“number”,“value”:10,“title”:“Smoothing backoff”,“desc”:“”,“order”:7,“section”:“motion_settings”,“advanced”:false,“hidden”:true},“do_spatial_smooth”:{“type”:“boolean”,“value”:true,“title”:“Spatial smoothing”,“desc”:“Whether or not to smooth particle trajectories by averaging over space.”,“order”:8,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“spatial_smooth_sigma_A”:{“type”:“number”,“value”:400,“title”:“Spatial smoothing stddev (A)”,“desc”:“Standard deviation of spatial smoothing weight. Particles closer to each other than this distance will count most strongly towards eachothers trajectories when smoothing.”,“order”:9,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“use_rigid_init”:{“type”:“boolean”,“value”:true,“title”:“Use rigid as initial trajectory”,“desc”:“Use the rigid frame trajectory as an initial guess for particle trajectories. Not recommended to change.”,“order”:10,“section”:“motion_settings”,“advanced”:true,“hidden”:false},“integer_shifts”:{“type”:“boolean”,“value”:false,“title”:“Force integer shifts”,“desc”:“Force trajectories to be interger-valued, to avoid aliasing artefacts.”,“order”:11,“section”:“motion_settings”,“advanced”:true,“hidden”:false},“do_doseweight”:{“type”:“boolean”,“value”:true,“title”:“Apply exposure weighting”,“desc”:“Whether to apply exposure weighting based on pre-computed dose weights.”,“order”:12,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“override_expframe”:{“type”:“number”,“value”:null,“title”:“Override e/A^2/frame”,“desc”:“Override the dose rate that was given at import time but can be overridden here.”,“order”:13,“section”:“motion_settings”,“advanced”:false,“hidden”:false},“box_size_pix”:{“type”:“number”,“value”:null,“title”:“Extraction box size (pix)”,“desc”:“Box size at which to extract particles. Box size should be at least twice the particle width.”,“order”:14,“section”:“extract_params”,“advanced”:false,“hidden”:false},“bin_size_pix”:{“type”:“number”,“value”:null,“title”:“Fourier crop to box size (pix)”,“desc”:“None means use the same as the extraction box size”,“order”:15,“section”:“extract_params”,“advanced”:false,“hidden”:false},“flip_x”:{“type”:“boolean”,“value”:false,“title”:“Flip mic. in x before extract?”,“desc”:“”,“order”:16,“section”:“extract_params”,“advanced”:true,“hidden”:false},“flip_y”:{“type”:“boolean”,“value”:false,“title”:“Flip mic. in y before extract?”,“desc”:“”,“order”:17,“section”:“extract_params”,“advanced”:true,“hidden”:false},“scale_const_override”:{“type”:“number”,“value”:null,“title”:“Scale constant (override)”,“desc”:“”,“order”:18,“section”:“extract_params”,“advanced”:true,“hidden”:true},“overall_normalizer_override”:{“type”:“number”,“value”:null,“title”:“Overall normalizer (override)”,“desc”:“”,“order”:19,“section”:“extract_params”,“advanced”:true,“hidden”:true}},“params_secs”:{“general_settings”:{“title”:“General settings”,“desc”:“”,“order”:0},“motion_settings”:{“title”:“Motion correction”,“desc”:“”,“order”:1},“extract_params”:{“title”:“Particle Extraction”,“desc”:“”,“order”:2}},“resources_needed”:{“slots”:{“CPU”:6,“GPU”:1,“RAM”:2},“fixed”:{“SSD”:false}},“PID_main”:232269,“PID_monitor”:232171,“PID_workers”:[],“bench”:{},“cluster_job_custom_vars”:{},“cluster_job_id”:null,“cluster_job_monitor_event_id”:null,“cluster_job_monitor_last_run_at”:null,“cluster_job_monitor_retries”:0,“cluster_job_status”:null,“cluster_job_status_code”:null,“cluster_job_submission_script”:null,“completed_count”:0,“enable_bench”:false,“errors_run”:[{“message”:“Job process terminated abnormally.”,“warning”:false}],“experiment_worker_path”:null,“generate_intermediate_results”:false,“heartbeat_at”:“2023-09-04T18:17:44.726Z”,“instance_information”:{“platform_node”:“R2D2”,“platform_release”:“5.15.0-79-generic”,“platform_version”:“#86~20.04.2-Ubuntu SMP Mon Jul 17 23:27:17 UTC 2023”,“platform_architecture”:“x86_64”,“cpu_model”:“AMD Ryzen Threadripper 3960X 24-Core Processor”,“physical_cores”:24,“total_memory”:“251.55GB”,“available_memory”:“239.46GB”,“used_memory”:“9.89GB”,“ofd_soft_limit”:1024,“ofd_hard_limit”:1048576,“gpu_info”:[{“id”:0,“name”:“NVIDIA GeForce RTX 3090”,“mem”:25447170048}],“CUDA_version”:“11.2.0”},“is_experiment”:false,“last_scheduled_at”:null,“last_updated”:“2023-09-09T08:31:56.327Z”,“queue_index”:null,“queue_status”:null,“queued_job_hash”:null,“queued_to_lane”:“RTX3090”,“run_as_user”:null,“tokens_acquired_at”:1693829424.7666337,“tokens_requested_at”:null,“type”:“local_motion_correction”,“version”:“v4.3.0”,“no_check_inputs_ready”:false,“queued_to_gpu”:false,“queued_to_hostname”:null,“num_tokens”:1,“progress”:[]}

I am not sure if there is a best way to share this files with you. If you prefer it in another format, let me know how.

Log file (truncated):

================= CRYOSPARCW ======= 2023-09-04 14:10:25.403371 =========
Project P23 Job J82
Master R2D2 Port 39002
===========================================================================
========= monitor process now starting main process at 2023-09-04 14:10:25.403406
MAINPROCESS PID 232269
========= monitor process now waiting for main process
MAIN PID 232269
motioncorrection.run_local cryosparc_compute.jobs.jobregister
ElectronCountedFramesDecompressor: reading using TIFF-EER mode.
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
========= sending heartbeat at 2023-09-04 14:10:38.493541
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
T
TIFFReadDirectory: Unknown field with tag 65002 (0xfdea) encountered
ElectronCountedFramesDecompressor::prepareRead: found
Unknown field with tag 65100 (0xfe4c) encounteredUnknown field with tag 65100 (0xfe4c) encounteredUnknown field with tag 65100 (0xfe4c) encounteredUnknown field with tag 65100 (0xfe4c) encounteredUnknown field with tag 65100 (0xfe4c) encounteredgpufft: creating new cufft plan (plan id 0 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 32
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 1 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 8
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 2 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 2
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 3 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 1
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 4 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 32
type C2R
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 5 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 8
type C2R
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 6 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 2
type C2R
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 7 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 1
type C2R
wkspc automatic
Python traceback:
Unknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encountered========= sending heartbeat at 2023-09-04 14:10:59.321507
Unknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredgpufft: creating new cufft plan (plan id 8 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 64
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 9 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 64
type C2R
wkspc automatic
Python traceback:
Unknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encountered
========= sending heartbeat at 2023-09-04 14:11:09.381501
Unknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encountered
gpufft: creating new cufft plan (plan id 10 pid 232269)
gpu_id 0
ndims 2
dims 720 720 0
inembed 720 720 0
istride 1
idist 518400
onembed 720 361 0
ostride 1
odist 259920
batch 128
type R2C
wkspc automatic
Python traceback:
gpufft: creating new cufft plan (plan id 11 pid 232269)
gpu_id 0
ndims 2
dims 180 180 0
inembed 180 91 0
istride 1
idist 16380
onembed 180 180 0
ostride 1
odist 32400
batch 128
type C2R
wkspc automatic
Python traceback:
Unknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encounteredUnknown field with tag 65002 (0xfdea) encountered========= sending heartbeat at 2023-09-04 20:14:24.436290
========= sending heartbeat at 2023-09-04 20:14:34.450769
========= sending heartbeat at 2023-09-04 20:14:44.465635
========= sending heartbeat at 2023-09-04 20:14:54.477943
========= sending heartbeat at 2023-09-04 20:15:04.492406
========= sending heartbeat at 2023-09-04 20:15:14.507160
========= sending heartbeat at 2023-09-04 20:15:24.521821
========= sending heartbeat at 2023-09-04 20:15:34.536988
========= sending heartbeat at 2023-09-04 20:15:44.551440
========= sending heartbeat at 2023-09-04 20:15:54.565528
========= sending heartbeat at 2023-09-04 20:16:04.579981
========= sending heartbeat at 2023-09-04 20:16:14.593514
========= sending heartbeat at 2023-09-04 20:16:24.608384
========= sending heartbeat at 2023-09-04 20:16:34.623140
========= sending heartbeat at 2023-09-04 20:16:44.637787
========= sending heartbeat at 2023-09-04 20:16:54.649506
========= sending heartbeat at 2023-09-04 20:17:04.664116
========= sending heartbeat at 2023-09-04 20:17:14.677950
========= sending heartbeat at 2023-09-04 20:17:24.692518
========= sending heartbeat at 2023-09-04 20:17:34.709042
========= sending heartbeat at 2023-09-04 20:17:44.723698
========= main process now complete at 2023-09-04 20:17:45.787492.
========= monitor process now complete at 2023-09-04 20:17:45.865941.

This may already work for some types of error, but possibly not for the type of error you encountered. We would therefore be interested in the specific error you encountered.

Is this the log file for the instance of the job that failed, or log file after (successful) restart of the job? If the job is still in its failed stated, or if you encounter the error again, please can you also post the end of the Event Log, including any Traceback.
image

Hi @wtempel,

Yes, it’s the log file for the job that I did not restart. In fact to retry I did not restarted, but I decided to clone it and run the clone job, keeping the original failed job intact so we could continue to troubleshoot it.

Here you have all that was printed in the event log.

[CPU: 4.61 GB Avail: 176.99 GB]
Processed 2300 of 9804 movies in 21806.39s
[CPU: 4.61 GB Avail: 176.99 GB]
(2312 of 9804) Processing J49/imported/006767267461558170245_FoilHole_726995_Data_31752623_31752625_20230901_065834_EER.eer
Motion correcting and extracting 60 particles (0 rejected near edges)
[CPU: 4.61 GB Avail: 176.99 GB]
Loading raw movie data from J49/imported/006767267461558170245_FoilHole_726995_Data_31752623_31752625_20230901_065834_EER.eer …
[CPU: 4.57 GB Avail: 176.91 GB]
Done in 6.90s
[CPU: 4.57 GB Avail: 176.91 GB]
Processing …Done in 1.46s
[CPU: 4.03 GB Avail: 177.60 GB]
Curvature 2564037.0 Smooth lambda cal 20.0
[CPU: 4.03 GB Avail: 177.60 GB]
Writing out particles… Done in 0.01s
[CPU: 4.03 GB Avail: 177.59 GB]
(2313 of 9804) Processing J49/imported/011088365768576908942_FoilHole_726996_Data_31752606_31752608_20230901_065334_EER.eer
Motion correcting and extracting 65 particles (0 rejected near edges)
[CPU: 4.03 GB Avail: 177.59 GB]
Loading raw movie data from J49/imported/011088365768576908942_FoilHole_726996_Data_31752606_31752608_20230901_065334_EER.eer …
[CPU: 4.03 GB Avail: 177.58 GB]
Done in 7.74s
[CPU: 4.03 GB Avail: 177.58 GB]
Processing …Done in 1.46s
[CPU: 4.25 GB Avail: 177.28 GB]
Curvature 2536692.5 Smooth lambda cal 20.0
[CPU: 4.25 GB Avail: 177.27 GB]
Writing out particles… Done in 0.01s
[CPU: 4.25 GB Avail: 177.26 GB]
(2314 of 9804) Processing J49/imported/015807790047606116986_FoilHole_726996_Data_31752614_31752616_20230901_065339_EER.eer
Motion correcting and extracting 80 particles (0 rejected near edges)
[CPU: 4.25 GB Avail: 177.26 GB]
Loading raw movie data from J49/imported/015807790047606116986_FoilHole_726996_Data_31752614_31752616_20230901_065339_EER.eer …
[CPU: 199.8 MB Avail: 181.54 GB]
====== Job process terminated abnormally.

We cannot, unfortunately, suggest a way to use the partial results in this case.

1 Like