Error while importing movies in CryoSPARC v4.0.1

Hi

Can anyone help me with the issues I am having importing *.tiff and *.mrc files? Our in-house GPU system is recently upgraded to cryoSPARCv4.0.1 from 3.3
Thanks

[2022-11-07 16:07:21.61] License is valid.
[2022-11-07 16:07:21.63] Running job on master node
[2022-11-07 16:07:23.28] [CPU: 96.3 MB] Job J1 Started
[2022-11-07 16:07:23.29] [CPU: 96.3 MB] Master running v4.0.1, worker running v4.0.1
[2022-11-07 16:07:23.31] [CPU: 96.3 MB] Working in directory: /cem1data0/cryosparcdata0/jsistla/CS-bnl-data-set/J1
[2022-11-07 16:07:23.31] [CPU: 96.3 MB] Running on lane default
[2022-11-07 16:07:23.31] [CPU: 96.3 MB] Resources allocated:
[2022-11-07 16:07:23.32] [CPU: 96.3 MB] Worker: sbcem1
[2022-11-07 16:07:23.33] [CPU: 96.3 MB] --------------------------------------------------------------
[2022-11-07 16:07:23.33] [CPU: 96.3 MB] Importing job module for job type import_moviesā€¦
[2022-11-07 16:07:32.51] [CPU: 263.7 MB] Job ready to run
[2022-11-07 16:07:32.51] [CPU: 263.7 MB] ***************************************************************
[2022-11-07 16:07:32.54] [CPU: 263.7 MB] Importing movies from /cem1data0/cryosparcdata0/jsistla/combined data/igr3
data_BNL/*.tiff
[2022-11-07 16:07:32.58] [CPU: 265.4 MB] Importing 7466 files
[2022-11-07 16:07:33.07] [CPU: 270.0 MB] Import paths were unique at level -1
[2022-11-07 16:07:33.08] [CPU: 270.2 MB] Importing 7467 files
[2022-11-07 16:07:33.08] [CPU: 270.2 MB] Reading header for each exposureā€¦
[2022-11-07 16:07:33.10] [CPU: 272.5 MB] Spawning worker processes to read headers in parallelā€¦
[2022-11-07 16:07:33.10] [CPU: 272.5 MB] Processed 5457 headersā€¦
[2022-11-07 16:07:40.19] [CPU: 98.3 MB] ====== Job process terminated abnormally.

Welcome to the forum @JSistla.
Please can you post the contents of
/cem1data0/cryosparcdata0/jsistla/CS-bnl-data-set/J1/job.log.

Hi
Do you mean the input parameters?
Thanks

Those may be of interest later. For now, it would be interesting to see if there are any errors in
/cem1data0/cryosparcdata0/jsistla/CS-bnl-data-set/J1/job.log
The file can be accessed via the command line or, alternatively, under the Metadata tab, Log ā€œsubtabā€.
image

Hi Please find below the content data
Thanks

{
  "id": "63692cb75e31bf16e198db93",
  "children": [],
  "cloned_from": null,
  "completed_at": null,
  "created_at": "2022-11-07T16:05:11.180Z",
  "created_by_user_id": "635ad351e6d20c6049a2d0b6",
  "deleted": false,
  "description": "Enter a description.",
  "failed_at": "2022-11-07T16:07:40.186Z",
  "interactive": false,
  "interactive_hostname": "sbcem1",
  "interactive_port": null,
  "job_type": "import_movies",
  "killed_at": null,
  "last_exported": "2022-11-07T16:07:04.388Z",
  "launched_at": "2022-11-07T16:07:21.616Z",
  "output_group_images": {},
  "output_result_groups": [
    {
      "uid": "J1-G0",
      "type": "exposure",
      "name": "imported_movies",
      "title": "Imported movies",
      "description": "Imported movies.",
      "contains": [
        {
          "uid": "J1-R0",
          "type": "exposure.movie_blob",
          "group_name": "imported_movies",
          "name": "movie_blob",
          "passthrough": false
        },
        {
          "uid": "J1-R1",
          "type": "exposure.gain_ref_blob",
          "group_name": "imported_movies",
          "name": "gain_ref_blob",
          "passthrough": false
        },
        {
          "uid": "J1-R2",
          "type": "exposure.mscope_params",
          "group_name": "imported_movies",
          "name": "mscope_params",
          "passthrough": false
        }
      ],
      "passthrough": false,
      "num_items": 0,
      "summary": {}
    },
    {
      "uid": "J1-G1",
      "type": "exposure",
      "name": "failed_movies",
      "title": "Failed movies",
      "description": "Movies that failed to import successfully.",
      "contains": [
        {
          "uid": "J1-R3",
          "type": "exposure.movie_blob",
          "group_name": "failed_movies",
          "name": "movie_blob",
          "passthrough": false
        }
      ],
      "passthrough": false,
      "num_items": 0,
      "summary": {}
    }
  ],
  "params_spec": {
    "blob_paths": {
      "value": "/cem1data0/cryosparcdata0/jsistla/combined data/igr3 data_BNL/*.tiff"
    },
    "gainref_path": {
      "value": "/cem1data0/cryosparcdata0/jsistla/combined data/igr3 data_BNL/K3-Gain-20220606-308690-105Kx.mrc"
    },
    "psize_A": {"value": 0.825},
    "accel_kv": {"value": 300},
    "cs_mm": {"value": 2.7},
    "total_dose_e_per_A2": {"value": 40}
  },
  "parents": [],
  "priority": 0,
  "project_uid": "P5",
  "project_uid_num": 5,
  "queue_message": null,
  "queued_at": "2022-11-07T16:07:20.969Z",
  "resources_allocated": {
    "lane": null,
    "lane_type": null,
    "hostname": "sbcem1",
    "target": null,
    "slots": {},
    "fixed": {},
    "license": false,
    "licenses_acquired": 0
  },
  "running_at": "2022-11-07T16:07:32.536Z",
  "started_at": "2022-11-07T16:07:23.277Z",
  "status": "failed",
  "status_num": 40,
  "title": "New Job J1",
  "ui_tile_height": 1,
  "ui_tile_images": [],
  "ui_tile_width": 3,
  "uid": "J1",
  "uid_num": 1,
  "waiting_at": null,
  "workspace_uids": ["W1"],
  "errors_build_inputs": {},
  "errors_build_params": {},
  "input_slot_groups": [],
  "job_dir": "J1",
  "job_dir_size": 0,
  "last_accessed": {
    "name": "cryosparcuser",
    "accessed_at": "2022-11-07T17:10:32.558Z"
  },
  "output_results": [
    {
      "uid": "J1-R0",
      "type": "exposure.movie_blob",
      "group_name": "imported_movies",
      "name": "movie_blob",
      "title": "Raw movie data",
      "description": "",
      "min_fields": [
        ["path", "O"],
        ["shape", "3u4"],
        ["psize_A", "f4"],
        ["is_gain_corrected", "u4"],
        ["format", "O"],
        ["has_defect_file", "u4"],
        ["import_sig", "u8"]
      ],
      "versions": [],
      "metafiles": [],
      "num_items": [],
      "passthrough": false
    },
    {
      "uid": "J1-R1",
      "type": "exposure.gain_ref_blob",
      "group_name": "imported_movies",
      "name": "gain_ref_blob",
      "title": "Gain reference",
      "description": "",
      "min_fields": [
        ["path", "O"],
        ["idx", "u4"],
        ["shape", "2u4"],
        ["flip_x", "u4"],
        ["flip_y", "u4"],
        ["rotate_num", "u4"]
      ],
      "versions": [],
      "metafiles": [],
      "num_items": [],
      "passthrough": false
    },
    {
      "uid": "J1-R2",
      "type": "exposure.mscope_params",
      "group_name": "imported_movies",
      "name": "mscope_params",
      "title": "Exposure parameters",
      "description": "",
      "min_fields": [
        ["accel_kv", "f4"],
        ["cs_mm", "f4"],
        ["total_dose_e_per_A2", "f4"],
        ["phase_plate", "u4"],
        ["neg_stain", "u4"],
        ["exp_group_id", "u4"],
        ["defect_path", "O"]
      ],
      "versions": [],
      "metafiles": [],
      "num_items": [],
      "passthrough": false
    },
    {
      "uid": "J1-R3",
      "type": "exposure.movie_blob",
      "group_name": "failed_movies",
      "name": "movie_blob",
      "title": "Raw movie data",
      "description": "",
      "min_fields": [
        ["path", "O"],
        ["shape", "3u4"],
        ["psize_A", "f4"],
        ["is_gain_corrected", "u4"],
        ["format", "O"],
        ["has_defect_file", "u4"],
        ["import_sig", "u8"]
      ],
      "versions": [],
      "metafiles": [],
      "num_items": [],
      "passthrough": false
    }
  ],
  "params_base": {
    "blob_paths": {
      "type": "path",
      "value": null,
      "title": "Movies data path",
      "desc": "Absolute path, wildcard-expression (e.g. /mount/data/somewhere/*.mrcs) that will be imported. MRC (mrc, mrcs, stk) and TIFF format supported.",
      "order": 0,
      "section": "import",
      "advanced": false,
      "hidden": false,
      "path_dir_allowed": false,
      "path_file_allowed": true,
      "path_glob_allowed": true,
      "skip_path_validation": false
    },
    "gainref_path": {
      "type": "path",
      "value": null,
      "title": "Gain reference path",
      "desc": "Absolute path to a single gain reference for all the raw data, in MRC format. Leave blank if data is already gain-corrected.",
      "order": 1,
      "section": "import",
      "advanced": false,
      "hidden": false,
      "path_dir_allowed": false,
      "path_file_allowed": true,
      "path_glob_allowed": false,
      "skip_path_validation": false
    },
    "defect_path": {
      "type": "path",
      "value": null,
      "title": "Defect file path",
      "desc": "Absolute path to a defect file for all the raw data. This should be a .txt file. Leave blank if not applicable.",
      "order": 2,
      "section": "import",
      "advanced": false,
      "hidden": false,
      "path_dir_allowed": false,
      "path_file_allowed": true,
      "path_glob_allowed": false,
      "skip_path_validation": false
    },
    "gainref_flip_x": {
      "type": "boolean",
      "value": false,
      "title": "Flip gain ref & defect file in X?",
      "desc": "Flip gain ref and defect file left-to-right (in X axis)",
      "order": 3,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "gainref_flip_y": {
      "type": "boolean",
      "value": false,
      "title": "Flip gain ref & defect file in Y?",
      "desc": "Flip gain ref and defect file top-to-bottom (in Y axis)",
      "order": 4,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "gainref_rotate_num": {
      "type": "number",
      "value": 0,
      "title": "Rotate gain ref?",
      "desc": "Rotate gain ref counter-clockwise by 90 degrees this many times",
      "order": 5,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "psize_A": {
      "type": "number",
      "value": null,
      "title": "Raw pixel size (A)",
      "desc": "Pixel size of the raw movie data in Angstroms",
      "order": 6,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "accel_kv": {
      "type": "number",
      "value": null,
      "title": "Accelerating Voltage (kV)",
      "desc": "",
      "order": 7,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "cs_mm": {
      "type": "number",
      "value": null,
      "title": "Spherical Aberration (mm)",
      "desc": "",
      "order": 8,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "total_dose_e_per_A2": {
      "type": "number",
      "value": null,
      "title": "Total exposure dose (e/A^2)",
      "desc": "",
      "order": 9,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "negative_stain_data": {
      "type": "boolean",
      "value": false,
      "title": "Negative Stain Data",
      "desc": "If Negative Stain Data is on, this indicates that there are light particles on dark background. If it's off, this indicates the movies have dark particles on light background (cryo-em data).",
      "order": 10,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "phase_plate_data": {
      "type": "boolean",
      "value": false,
      "title": "Phase Plate Data",
      "desc": "",
      "order": 11,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "override_exp_group_id": {
      "type": "number",
      "value": null,
      "title": "Override Exposure Group ID",
      "desc": "",
      "order": 12,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "skip_header_check": {
      "type": "boolean",
      "value": false,
      "title": "Skip Header Check",
      "desc": "Skip reading of every header file to increase import speed. WARNING: this assumes exposure shapes and extensions are consistent across the entire dataset.",
      "order": 13,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "eer_num_fractions": {
      "type": "number",
      "value": 40,
      "title": "EER Number of Fractions",
      "desc": "Number of fractions to make out of the EER input data.",
      "order": 14,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "eer_upsamp_factor": {
      "type": "number",
      "value": 2,
      "title": "EER Upsampling Factor",
      "desc": "Upsampling factor when decoding EER input data. Note that the pixel size you provide should be the raw pixel size at the nominal 4k sensor, not the pixel size after EER upsampling. ",
      "order": 15,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "output_constant_ctf": {
      "type": "boolean",
      "value": false,
      "title": "Output Constant CTF",
      "desc": "Output constant CTF so that subsequent steps will ignore CTF entirely.",
      "order": 16,
      "section": "import",
      "advanced": false,
      "hidden": false
    },
    "compute_num_cpus": {
      "type": "number",
      "value": 4,
      "title": "Number of CPUs to parallelize",
      "desc": "Use this many CPUs to read headers in parallel",
      "order": 17,
      "section": "compute_settings",
      "advanced": false,
      "hidden": false
    }
  },
  "params_secs": {
    "import": {
      "title": "Movies",
      "desc": "Metadata and raw data for movies.",
      "order": 0
    },
    "compute_settings": {"title": "Compute settings", "desc": "", "order": 1}
  },
  "resources_needed": {},
  "PID_main": 969886,
  "PID_monitor": 969885,
  "PID_workers": [],
  "bench": {},
  "cluster_job_id": null,
  "cluster_job_monitor_event_id": null,
  "cluster_job_monitor_last_run_at": null,
  "cluster_job_monitor_retries": 0,
  "cluster_job_status": null,
  "cluster_job_status_code": null,
  "completed_count": 0,
  "created_by_job_uid": null,
  "enable_bench": false,
  "errors_run": [
    {"message": "Job process terminated abnormally.", "warning": false}
  ],
  "experiment_worker_path": null,
  "heartbeat_at": "2022-11-07T16:07:33.608Z",
  "instance_information": {
    "platform_node": "sbcem1",
    "platform_release": "5.4.0-126-generic",
    "platform_version": "#142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022",
    "platform_architecture": "x86_64",
    "physical_cores": 32,
    "max_cpu_freq": 2100,
    "total_memory": "125.77GB",
    "available_memory": "119.79GB",
    "used_memory": "4.95GB"
  },
  "is_experiment": false,
  "last_scheduled_at": null,
  "last_updated": "2022-11-07T17:10:32.560Z",
  "queue_index": null,
  "queue_status": null,
  "queued_job_hash": null,
  "queued_to_lane": "default",
  "run_as_user": null,
  "run_on_master_direct": true,
  "tokens_acquired_at": 1667837241.6133823,
  "tokens_requested_at": null,
  "type": "import_movies",
  "version": "v4.0.1",
  "ui_layouts": {
    "P5": {
      "show": true,
      "floater": true,
      "top": 0,
      "left": 0,
      "width": 444,
      "height": 192,
      "groups": []
    },
    "P5W1": {
      "show": true,
      "floater": true,
      "top": 0,
      "left": 0,
      "width": 444,
      "height": 192,
      "groups": []
    }
  },
  "no_check_inputs_ready": false,
  "queued_to_gpu": false,
  "queued_to_hostname": null,
  "num_tokens": 0
}

That presumably is the content of the Data ā€œsubtabā€. Please can you share the output of the Log subtab.

Here you go.

Log

================= CRYOSPARCW ======= 2022-11-07 16:07:22.410093 =========
Project P5 Job J1
Master sbcem1 Port 39002

========= monitor process now starting main process
MAINPROCESS PID 969886
MAIN PID 969886
imports.run cryosparc_compute.jobs.jobregister
========= monitor process now waiting for main process
========= sending heartbeat
========= main process now complete.
========= monitor process now complete.

I am having the same error with v4.0.3 with .tif files. I have tried cloning and running again a couple times, and each time it completed processing one more header than the previous.

@JSistla @zlwatson
Please can you try

  1. Enable Skip Header Check during Import Movies.
  2. A follow-up task, like motion correction.

Do these work?

I have exactly the same error and I have tried both the points:

  1. Skipping the header check did not help
  2. motion correction does not start as it says: waiting because inputs are not ready

Hi, I was not able to import even after enabling the Skip header

Please can you re-try the import while specifying a single tiff file. Does that work?

I did try just importing a single .tiff file, with and without header check and still did not work, the job starts and fails immediately after.

License is valid.
Running job on master node
[CPU: 98.0 MB]
Job J11 Started
[CPU: 98.0 MB]
Master running v4.0.2, worker running v4.0.2
[CPU: 98.2 MB]
Working in directory: /data/gridC/gcollu/Processing_2/CS-tx/J11
[CPU: 98.2 MB]
Running on lane default
[CPU: 98.2 MB]
Resources allocated:
[CPU: 98.2 MB]
Worker: grid.lumcnet.prod.intern
[CPU: 98.2 MB]


[CPU: 98.2 MB]
Importing job module for job type import_moviesā€¦
[CPU: 268.9 MB]
Job ready to run
[CPU: 268.9 MB]


[CPU: 269.0 MB]
Importing movies from /data/gridC/gcollu/Processing_2/data/FoilHole_26764507_Data_26757469_26757471_20221026_165043_fractions.tiff
[CPU: 269.0 MB]
Importing 1 files
[CPU: 269.0 MB]
Import paths were unique at level -1
[CPU: 269.0 MB]
Importing 3 files
[CPU: 269.1 MB]
ā€˜Skip Header Checkā€™ parameter enabled, checking first header only
[CPU: 99.7 MB]
====== Job process terminated abnormally.

Thank you for trying that. Would you be willing to share with us:

  • job.json from that failed jobā€™s directory
  • FoilHole_26764507_Data_26757469_26757471_20221026_165043_fractions.tiff
  • any other input file(s) used in this job as applicable (gain reference?)

If needed, we can make arrangements for you to upload the data to our server.

Sure, I can do that. Thanks!
How should I share the files with you?

I do have an additional information: I have tried importing .tiff files collected at the Titan Krios instead of at the Arctica, and in that case works. So something is wrong meanwhile importing .tiff files collected on our Arctica.

Dear wtempel,

Iā€™d like to report exactly the same error.
Iā€™m using the latest version v4.1.1 to import a single particle dataset collected at Krios G3.
Skipping the header check or specifying a single .tif file didnā€™t work.

Thank you!

Welcome to the forum @Minghao.
Did you apply patch 230104 to that CryoSPARC instance?
Would you be willing to share job.json as well as one of the failing movies with us?
We can provide an upload link if needed.

Hi wtempel,
I have tried to apply the latest patch 230110 but it didnā€™t help.
Is it necessary to apply the specific patch 230104?

I would like to send you the files, please give me a link for uploading.
Thank you.