Local Refinement Beta Job Fail

merandamasse · June 26, 2020, 7:15pm

I have been trying to run a local refinement with some masks that I made, and the job keeps crashing CryoSPARC and then is showing up as failed once I restart the program. I am working on a CentOS 7 computer and updated to the newest version of CryoSPARC (2.15).

For this job, I imported the particles and volume that I had gotten from my NU-Refinement, and then imported the masks I had created in Chimera after running the “import volume” job to get the mask. I keep getting this error: Job is unresponsive - no heartbeat received in 30 seconds.

I’m not really sure what to do at this point, so any help would be appreciated.

mmclean · June 29, 2020, 7:45pm

Hi @merandamasse,

Thanks for reporting this; a few questions:

How far does the job get before it fails? Are there any printouts in the stream log or does it fail immediately?
Would it be possible for you to post (or directly message) screenshots of the job stream log from the beginning up until the point of failure?

Thank you,
Michael

merandamasse · June 29, 2020, 7:58pm

Hi Michael,

It will usually get some way through the job before it fails. I’ve actually tried to run some other jobs too (blob picker) and it will get some way through and then crash. I’ll DM you the job stream from the NU Refinement and the blob picker.

Thanks!
Meranda

mmclean · July 2, 2020, 2:42pm

Hi Meranda,

Thank you for providing the job log; for the local refinement, do you happen to remember the box size that you used? Occasionally with larger box sizes, and especially when multiple jobs are running on the same worker at once, running out of memory may cause cryoSPARC to crash as you have observed.

If you had multiple jobs running concurrently during either the local refinement or the blob picker, it might help to try running one job at a time (with a smaller box sizes or a fewer number of micrographs, respectively) if you haven’t already tried these.

Let me know how it goes!
Michael

merandamasse · July 7, 2020, 4:43pm

Hi @mmclean,

I’ve tried adjusting my box size, I’ve tried changing the heartbeat to 180 seconds (this didn’t take in the shell script), I’ve tried a fresh install and I’ve tried to back-install to version 2.13, and no matter what I keep getting this heartbeat error. I have plenty of memory on the computer, and I’ve been getting this error on NU refinement, homogeneous refinement, particle extraction… I’m not really sure what to do at this point. Is there anything else I can send over that might help? I’m also wondering if there is something wrong with the hardware of my workstation?

Thanks!
Meranda

mmclean · July 13, 2020, 4:21pm

Hi Meranda,

This certainly sounds like it could be a hardware issue; have jobs previously worked without crashing cryoSPARC in version 2.13, or did this only begin to occur when you updated to 2.15?

To help us figure out the issue, could you run any of the homogeneous or NU-refinement jobs that have previously crashed cryoSPARC and failed, and then send us the job log for those as well, so we could see at what point the job is failing? As well, if the job causes cryoSPARC to crash again, could you try running a ps aux command to see if any of the job processes are still running as zombie processes?

Best regards,
Michael

merandamasse · July 13, 2020, 4:50pm

Hi Michael,

I’ll get those to you ASAP. It’s worth noting that I am working with Exxact to look at some hardware issues. I most recently moved around one of the DIMM memory cards since it was showing a lot of errors, but while that got rid of the errors didn’t solve the issues with the jobs. I’m hoping to check the GPU functions and some other hardware issues and get back to you on that.

Best,
Meranda

merandamasse · July 13, 2020, 5:04pm

Hi @mmclean,

I recently ran an “Extract from Micrograph” job with a template, and this was the output from the same job:

{
  "_id": {
    "_str": "5f0793251b455744cf1d2dc5"
  },
  "children": [
    "J184"
  ],
  "created_at": "2020-07-09T21:59:01.876Z",
  "created_by_job_uid": null,
  "created_by_user_id": "5e62824d062638166165d702",
  "interactive": false,
  "job_type": "extract_micrographs_multi",
  "launched_at": "2020-07-09T21:59:11.553Z",
  "parents": [
    "J157",
    "J4"
  ],
  "project_uid": "P3",
  "queue_message": null,
  "queued_at": "2020-07-09T21:59:11.022Z",
  "started_at": "2020-07-09T21:59:12.379Z",
  "status": "failed",
  "title": "New Job J185",
  "uid": "J185",
  "waiting_at": null,
  "workspace_uids": [
    "W8"
  ],
  "cloned_from": "J183",
  "deleted": false,
  "type": "extract_micrographs",
  "ui_tile_height": 1,
  "ui_tile_images": [
    {
      "num_rows": 1,
      "num_cols": 1,
      "name": "mic",
      "fileid": "5f0793430626381fcf5caa5d"
    },
    {
      "num_rows": 1,
      "num_cols": 1,
      "name": "particles",
      "fileid": "5f0793440626381fcf5caa61"
    }
  ],
  "ui_tile_width": 2,
  "completed_at": null,
  "description": "Enter a description.",
  "failed_at": "2020-07-09T22:13:42.358Z",
  "job_dir_size": 10821529007,
  "killed_at": null,
  "is_experiment": false,
  "experiment_worker_path": null,
  "output_group_images": {
    "micrographs": "5f0793430626381fcf5caa5f",
    "particles": "5f0793440626381fcf5caa63"
  },
  "queue_status": null,
  "project_uid_num": 3,
  "interactive_port": null,
  "params_spec": {
    "box_size_pix": {
      "value": 512
    },
    "compute_num_gpus": {
      "value": 4
    },
    "bin_size_pix": {
      "value": 256
    }
  },
  "uid_num": 185,
  "errors_build_inputs": {},
  "params_base": {
    "compute_num_gpus": {
      "title": "Number of GPUs to parallelize (0 for CPU-only)",
      "section": "compute_settings",
      "value": 1,
      "hidden": false,
      "type": "number",
      "order": 0,
      "advanced": false,
      "desc": "Micrographs will be split between the GPUs."
    },
    "scale_const_override": {
      "title": "Scale constant (override)",
      "section": "extract_params",
      "value": null,
      "hidden": false,
      "type": "number",
      "order": 6,
      "advanced": true,
      "desc": "Override the scale constant used to scale the micrographs"
    },
    "flip_x": {
      "title": "Flip mic. in x before extract?",
      "section": "extract_params",
      "value": false,
      "hidden": false,
      "type": "boolean",
      "order": 4,
      "advanced": false,
      "desc": "Flip the micrograph in the x axis before extraction"
    },
    "flip_y": {
      "title": "Flip mic. in y before extract?",
      "section": "extract_params",
      "value": false,
      "hidden": false,
      "type": "boolean",
      "order": 5,
      "advanced": false,
      "desc": "Flip the micrograph in the y axis before extraction"
    },
    "overall_normalizer_override": {
      "title": "Overall normalizer (override)",
      "section": "extract_params",
      "value": null,
      "hidden": false,
      "type": "number",
      "order": 7,
      "advanced": true,
      "desc": "Override the normalizer used to normalize micrographs"
    },
    "num_extract": {
      "title": "Number of mics to extract",
      "section": "extract_params",
      "value": null,
      "hidden": false,
      "type": "number",
      "order": 3,
      "advanced": false,
      "desc": "Only extract a certain number of micrographs. None means all"
    },
    "box_size_pix": {
      "title": "Extraction box size (pix)",
      "section": "extract_params",
      "value": 256,
      "hidden": false,
      "type": "number",
      "order": 1,
      "advanced": false,
      "desc": "Size of box to be extracted from micrograph."
    },
    "bin_size_pix": {
      "title": "Fourier crop to box size (pix)",
      "section": "extract_params",
      "value": null,
      "hidden": false,
      "type": "number",
      "order": 2,
      "advanced": false,
      "desc": "Size of particle boxes after they have been extracted. None means use the same as the extraction box size"
    }
  },
  "resources_allocated": {
    "lane": "default",
    "fixed": {
      "SSD": false
    },
    "target": {
      "monitor_port": null,
      "lane": "default",
      "name": "c107925",
      "title": "Worker node c107925",
      "resource_slots": {
        "GPU": [
          0,
          1,
          2,
          3
        ],
        "RAM": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47
        ],
        "CPU": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47
        ]
      },
      "hostname": "c107925",
      "worker_bin_path": "/home/cryosparc_user/software/cryosparc/cryosparc2_worker/bin/cryosparcw",
      "cache_path": "/scr/cryosparc_scratch/",
      "cache_quota_mb": null,
      "resource_fixed": {
        "SSD": true
      },
      "gpus": [
        {
          "mem": 11554324480,
          "id": 0,
          "name": "GeForce RTX 2080 Ti"
        },
        {
          "mem": 11554717696,
          "id": 1,
          "name": "GeForce RTX 2080 Ti"
        },
        {
          "mem": 11554717696,
          "id": 2,
          "name": "GeForce RTX 2080 Ti"
        },
        {
          "mem": 11554717696,
          "id": 3,
          "name": "GeForce RTX 2080 Ti"
        }
      ],
      "desc": null,
      "type": "node",
      "ssh_str": "cryosparc_user@c107925",
      "cache_reserve_mb": 10000
    },
    "license": true,
    "slots": {
      "GPU": [
        0,
        1,
        2,
        3
      ],
      "RAM": [
        0,
        1
      ],
      "CPU": [
        0,
        1,
        2,
        3,
        4,
        5,
        6,
        7
      ]
    },
    "hostname": "c107925",
    "lane_type": "default",
    "licenses_acquired": 4
  },
  "job_dir": "J185",
  "errors_run": [
    {
      "message": "Job is unresponsive - no heartbeat received in 30 seconds.",
      "warning": false
    }
  ],
  "queued_to_lane": "default",
  "run_on_master_direct": false,
  "version": "v2.13.2",
  "output_result_groups": [
    {
      "num_items": 0,
      "uid": "J185-G0",
      "title": "Micrographs picked",
      "passthrough": "micrographs",
      "description": "",
      "contains": [
        {
          "passthrough": false,
          "type": "exposure.micrograph_blob",
          "uid": "J185-R0",
          "name": "micrograph_blob",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.stat_blob",
          "uid": "J185-R1",
          "name": "background_blob",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.mscope_params",
          "uid": "J185-R2",
          "name": "mscope_params",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.ctf",
          "uid": "J185-R3",
          "name": "ctf",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.ctf_stats",
          "uid": "J185-R4",
          "name": "ctf_stats",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.micrograph_blob",
          "uid": "J185-R5",
          "name": "micrograph_blob_non_dw",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.thumbnail_blob",
          "uid": "J185-R6",
          "name": "micrograph_thumbnail_blob_1x",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.thumbnail_blob",
          "uid": "J185-R7",
          "name": "micrograph_thumbnail_blob_2x",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.motion",
          "uid": "J185-R8",
          "name": "rigid_motion",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.motion",
          "uid": "J185-R9",
          "name": "spline_motion",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.movie_blob",
          "uid": "J185-R10",
          "name": "movie_blob",
          "group_name": "micrographs"
        },
        {
          "passthrough": true,
          "type": "exposure.gain_ref_blob",
          "uid": "J185-R11",
          "name": "gain_ref_blob",
          "group_name": "micrographs"
        }
      ],
      "type": "exposure",
      "name": "micrographs"
    },
    {
      "num_items": 0,
      "uid": "J185-G1",
      "title": "Particles extracted",
      "passthrough": "particles",
      "description": "",
      "contains": [
        {
          "passthrough": false,
          "type": "particle.blob",
          "uid": "J185-R12",
          "name": "blob",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.location",
          "uid": "J185-R13",
          "name": "location",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.ctf",
          "uid": "J185-R14",
          "name": "ctf",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.alignments3D",
          "uid": "J185-R15",
          "name": "alignments3D",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.components",
          "uid": "J185-R16",
          "name": "components_mode_0",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.components",
          "uid": "J185-R17",
          "name": "components_mode_1",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.components",
          "uid": "J185-R18",
          "name": "components_mode_2",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.alignments2D",
          "uid": "J185-R19",
          "name": "alignments2D",
          "group_name": "particles"
        },
        {
          "passthrough": true,
          "type": "particle.pick_stats",
          "uid": "J185-R20",
          "name": "pick_stats",
          "group_name": "particles"
        }
      ],
      "type": "particle",
      "name": "particles"
    }
  ],
  "run_as_user": null,
  "errors_build_params": {},
  "PID_main": 8143,
  "PID_monitor": 8137,
  "last_scheduled_at": null,
  "input_slot_groups": [

merandamasse · July 13, 2020, 5:04pm

Here is the rest:

{
      "name": "micrographs",
      "title": "Micrograph",
      "repeat_allowed": false,
      "connections": [
        {
          "slots": [
            {
              "job_uid": "J4",
              "slot_name": "micrograph_blob",
              "version": "F",
              "result_name": "micrograph_blob",
              "group_name": "exposures",
              "result_type": "exposure.micrograph_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": "background_blob",
              "version": "F",
              "result_name": "background_blob",
              "group_name": "exposures",
              "result_type": "exposure.stat_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": "mscope_params",
              "version": "F",
              "result_name": "mscope_params",
              "group_name": "exposures",
              "result_type": "exposure.mscope_params"
            },
            {
              "job_uid": "J4",
              "slot_name": "ctf",
              "version": "F",
              "result_name": "ctf",
              "group_name": "exposures",
              "result_type": "exposure.ctf"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "ctf_stats",
              "group_name": "exposures",
              "result_type": "exposure.ctf_stats"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "micrograph_blob_non_dw",
              "group_name": "exposures",
              "result_type": "exposure.micrograph_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "micrograph_thumbnail_blob_1x",
              "group_name": "exposures",
              "result_type": "exposure.thumbnail_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "micrograph_thumbnail_blob_2x",
              "group_name": "exposures",
              "result_type": "exposure.thumbnail_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "rigid_motion",
              "group_name": "exposures",
              "result_type": "exposure.motion"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "spline_motion",
              "group_name": "exposures",
              "result_type": "exposure.motion"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "movie_blob",
              "group_name": "exposures",
              "result_type": "exposure.movie_blob"
            },
            {
              "job_uid": "J4",
              "slot_name": null,
              "version": "F",
              "result_name": "gain_ref_blob",
              "group_name": "exposures",
              "result_type": "exposure.gain_ref_blob"
            }
          ],
          "job_uid": "J4",
          "group_name": "exposures"
        }
      ],
      "count_max": null,
      "slots": [
        {
          "type": "exposure.micrograph_blob",
          "title": "Raw micrograph data",
          "optional": false,
          "description": "",
          "name": "micrograph_blob"
        },
        {
          "type": "exposure.stat_blob",
          "title": "Estimated background data",
          "optional": true,
          "description": "",
          "name": "background_blob"
        },
        {
          "type": "exposure.mscope_params",
          "title": "Microscope parameters",
          "optional": false,
          "description": "",
          "name": "mscope_params"
        },
        {
          "type": "exposure.ctf",
          "title": "CTF parameters",
          "optional": true,
          "description": "",
          "name": "ctf"
        }
      ],
      "type": "exposure",
      "count_min": 1,
      "description": "Micrograph for particle extraction"
    },
    {
      "name": "particles",
      "title": "Particles",
      "repeat_allowed": false,
      "connections": [
        {
          "slots": [
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "location",
              "slot_name": "location",
              "result_type": "particle.location"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "blob",
              "slot_name": null,
              "result_type": "particle.blob"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "ctf",
              "slot_name": null,
              "result_type": "particle.ctf"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "alignments3D",
              "slot_name": null,
              "result_type": "particle.alignments3D"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "components_mode_0",
              "slot_name": null,
              "result_type": "particle.components"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "components_mode_1",
              "slot_name": null,
              "result_type": "particle.components"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "components_mode_2",
              "slot_name": null,
              "result_type": "particle.components"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "alignments2D",
              "slot_name": null,
              "result_type": "particle.alignments2D"
            },
            {
              "job_uid": "J157",
              "group_name": "particles_cluster_3",
              "version": "F",
              "result_name": "pick_stats",
              "slot_name": null,
              "result_type": "particle.pick_stats"
            }
          ],
          "job_uid": "J157",
          "group_name": "particles_cluster_3"
        }
      ],
      "count_max": null,
      "slots": [
        {
          "type": "particle.location",
          "title": "Particle locations",
          "optional": false,
          "description": "",
          "name": "location"
        }
      ],
      "type": "particle",
      "count_min": 0,
      "description": "Particles to extract"
    }
  ],
  "resources_needed": {
    "slots": {
      "GPU": 4,
      "RAM": 2,
      "CPU": 8
    },
    "fixed": {
      "SSD": false
    }
  },
  "tokens_requested_at": null,
  "PID_workers": [
    8187,
    8188,
    8189,
    8190
  ],
  "queued_job_hash": null,
  "params_secs": {
    "compute_settings": {
      "title": "Compute settings",
      "order": 0,
      "desc": "General settings"
    },
    "extract_params": {
      "title": "Particle Extraction",
      "order": 1,
      "desc": ""
    }
  },
  "queue_index": null,
  "output_results": [
    {
      "num_items": [],
      "uid": "J185-R0",
      "passthrough": false,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "format",
          "O"
        ],
        [
          "is_background_subtracted",
          "u4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "",
      "name": "micrograph_blob",
      "title": "Raw micrograph data",
      "versions": [],
      "type": "exposure.micrograph_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R1",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "binfactor",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "psize_A",
          "f4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.background_blob (slot_name)",
      "name": "background_blob",
      "title": "Passthrough background_blob",
      "versions": [],
      "type": "exposure.stat_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R2",
      "passthrough": true,
      "min_fields": [
        [
          "accel_kv",
          "f4"
        ],
        [
          "cs_mm",
          "f4"
        ],
        [
          "total_dose_e_per_A2",
          "f4"
        ],
        [
          "phase_plate",
          "u4"
        ],
        [
          "neg_stain",
          "u4"
        ],
        [
          "exp_group_id",
          "u4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.mscope_params (slot_name)",
      "name": "mscope_params",
      "title": "Passthrough mscope_params",
      "versions": [],
      "type": "exposure.mscope_params"
    },
    {
      "num_items": [],
      "uid": "J185-R3",
      "passthrough": true,
      "min_fields": [
        [
          "type",
          "O"
        ],
        [
          "path",
          "O"
        ],
        [
          "exp_group_id",
          "u4"
        ],
        [
          "accel_kv",
          "f4"
        ],
        [
          "cs_mm",
          "f4"
        ],
        [
          "amp_contrast",
          "f4"
        ],
        [
          "df1_A",
          "f4"
        ],
        [
          "df2_A",
          "f4"
        ],
        [
          "df_angle_rad",
          "f4"
        ],
        [
          "phase_shift_rad",
          "f4"
        ],
        [
          "cross_corr_ctffind4",
          "f4"
        ],
        [
          "ctf_fit_to_A",
          "f4"
        ],
        [
          "fig_of_merit_gctf",
          "f4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.ctf (slot_name)",
      "name": "ctf",
      "title": "Passthrough ctf",
      "versions": [],
      "type": "exposure.ctf"
    },
    {
      "num_items": [],
      "uid": "J185-R4",
      "passthrough": true,
      "min_fields": [
        [
          "type",
          "O"
        ],
        [
          "ctf_fit_to_A",
          "f4"
        ],
        [
          "cross_corr",
          "f4"
        ],
        [
          "df_range",
          "2f4"
        ],
        [
          "df_tilt_normal",
          "2f4"
        ],
        [
          "ice_thickness_rel",
          "f4"
        ],
        [
          "spectrum_dim",
          "u4"
        ],
        [
          "diag_image_path",
          "O"
        ],
        [
          "fit_data_path",
          "O"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.ctf_stats (result_name)",
      "name": "ctf_stats",
      "title": "Passthrough ctf_stats",
      "versions": [],
      "type": "exposure.ctf_stats"
    },
    {
      "num_items": [],
      "uid": "J185-R5",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "format",
          "O"
        ],
        [
          "is_background_subtracted",
          "u4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.micrograph_blob_non_dw (result_name)",
      "name": "micrograph_blob_non_dw",
      "title": "Passthrough micrograph_blob_non_dw",
      "versions": [],
      "type": "exposure.micrograph_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R6",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "format",
          "O"
        ],
        [
          "binfactor",
          "u4"
        ],
        [
          "micrograph_path",
          "O"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.micrograph_thumbnail_blob_1x (result_name)",
      "name": "micrograph_thumbnail_blob_1x",
      "title": "Passthrough micrograph_thumbnail_blob_1x",
      "versions": [],
      "type": "exposure.thumbnail_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R7",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "format",
          "O"
        ],
        [
          "binfactor",
          "u4"
        ],
        [
          "micrograph_path",
          "O"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.micrograph_thumbnail_blob_2x (result_name)",
      "name": "micrograph_thumbnail_blob_2x",
      "title": "Passthrough micrograph_thumbnail_blob_2x",
      "versions": [],
      "type": "exposure.thumbnail_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R8",
      "passthrough": true,
      "min_fields": [
        [
          "type",
          "O"
        ],
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "frame_start",
          "u4"
        ],
        [
          "frame_end",
          "u4"
        ],
        [
          "zero_shift_frame",
          "u4"
        ],
        [
          "psize_A",
          "f4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.rigid_motion (result_name)",
      "name": "rigid_motion",
      "title": "Passthrough rigid_motion",
      "versions": [],
      "type": "exposure.motion"
    },
    {
      "num_items": [],
      "uid": "J185-R9",
      "passthrough": true,
      "min_fields": [
        [
          "type",
          "O"
        ],
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "frame_start",
          "u4"
        ],
        [
          "frame_end",
          "u4"
        ],
        [
          "zero_shift_frame",
          "u4"
        ],
        [
          "psize_A",
          "f4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.spline_motion (result_name)",
      "name": "spline_motion",
      "title": "Passthrough spline_motion",
      "versions": [],
      "type": "exposure.motion"
    },
    {
      "num_items": [],
      "uid": "J185-R10",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "shape",
          "3u4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "is_gain_corrected",
          "u4"
        ],
        [
          "format",
          "O"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.movie_blob (result_name)",
      "name": "movie_blob",
      "title": "Passthrough movie_blob",
      "versions": [],
      "type": "exposure.movie_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R11",
      "passthrough": true,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "flip_x",
          "u4"
        ],
        [
          "flip_y",
          "u4"
        ],
        [
          "rotate_num",
          "u4"
        ]
      ],
      "group_name": "micrographs",
      "metafiles": [],
      "description": "Passthrough from input micrographs.gain_ref_blob (result_name)",
      "name": "gain_ref_blob",
      "title": "Passthrough gain_ref_blob",
      "versions": [],
      "type": "exposure.gain_ref_blob"
    },
    {
      "num_items": [],
      "uid": "J185-R12",
      "passthrough": false,
      "min_fields": [
        [
          "path",
          "O"
        ],
        [
          "idx",
          "u4"
        ],
        [
          "shape",
          "2u4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "sign",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "",
      "name": "blob",
      "title": "Particle raw data",
      "versions": [],
      "type": "particle.blob"
    },
    {
      "num_items": [],
      "uid": "J185-R13",
      "passthrough": true,
      "min_fields": [
        [
          "micrograph_uid",
          "u8"
        ],
        [
          "exp_group_id",
          "u4"
        ],
        [
          "micrograph_path",
          "O"
        ],
        [
          "micrograph_shape",
          "2u4"
        ],
        [
          "center_x_frac",
          "f4"
        ],
        [
          "center_y_frac",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.location (slot_name)",
      "name": "location",
      "title": "Passthrough location",
      "versions": [],
      "type": "particle.location"
    },
    {
      "num_items": [],
      "uid": "J185-R14",
      "passthrough": true,
      "min_fields": [
        [
          "type",
          "O"
        ],
        [
          "exp_group_id",
          "u4"
        ],
        [
          "accel_kv",
          "f4"
        ],
        [
          "cs_mm",
          "f4"
        ],
        [
          "amp_contrast",
          "f4"
        ],
        [
          "df1_A",
          "f4"
        ],
        [
          "df2_A",
          "f4"
        ],
        [
          "df_angle_rad",
          "f4"
        ],
        [
          "phase_shift_rad",
          "f4"
        ],
        [
          "scale",
          "f4"
        ],
        [
          "scale_const",
          "f4"
        ],
        [
          "shift_A",
          "2f4"
        ],
        [
          "tilt_A",
          "2f4"
        ],
        [
          "trefoil_A",
          "2f4"
        ],
        [
          "tetra_A",
          "4f4"
        ],
        [
          "anisomag",
          "4f4"
        ],
        [
          "bfactor",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.ctf (result_name)",
      "name": "ctf",
      "title": "Passthrough ctf",
      "versions": [],
      "type": "particle.ctf"
    },
    {
      "num_items": [],
      "uid": "J185-R15",
      "passthrough": true,
      "min_fields": [
        [
          "split",
          "u4"
        ],
        [
          "shift",
          "2f4"
        ],
        [
          "pose",
          "3f4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "error",
          "f4"
        ],
        [
          "error_min",
          "f4"
        ],
        [
          "resid_pow",
          "f4"
        ],
        [
          "slice_pow",
          "f4"
        ],
        [
          "image_pow",
          "f4"
        ],
        [
          "cross_cor",
          "f4"
        ],
        [
          "alpha",
          "f4"
        ],
        [
          "alpha_min",
          "f4"
        ],
        [
          "weight",
          "f4"
        ],
        [
          "pose_ess",
          "f4"
        ],
        [
          "shift_ess",
          "f4"
        ],
        [
          "class_posterior",
          "f4"
        ],
        [
          "class",
          "u4"
        ],
        [
          "class_ess",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.alignments3D (result_name)",
      "name": "alignments3D",
      "title": "Passthrough alignments3D",
      "versions": [],
      "type": "particle.alignments3D"
    },
    {
      "num_items": [],
      "uid": "J185-R16",
      "passthrough": true,
      "min_fields": [
        [
          "component",
          "u4"
        ],
        [
          "value",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.components_mode_0 (result_name)",
      "name": "components_mode_0",
      "title": "Passthrough components_mode_0",
      "versions": [],
      "type": "particle.components"
    },
    {
      "num_items": [],
      "uid": "J185-R17",
      "passthrough": true,
      "min_fields": [
        [
          "component",
          "u4"
        ],
        [
          "value",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.components_mode_1 (result_name)",
      "name": "components_mode_1",
      "title": "Passthrough components_mode_1",
      "versions": [],
      "type": "particle.components"
    },
    {
      "num_items": [],
      "uid": "J185-R18",
      "passthrough": true,
      "min_fields": [
        [
          "component",
          "u4"
        ],
        [
          "value",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.components_mode_2 (result_name)",
      "name": "components_mode_2",
      "title": "Passthrough components_mode_2",
      "versions": [],
      "type": "particle.components"
    },
    {
      "num_items": [],
      "uid": "J185-R19",
      "passthrough": true,
      "min_fields": [
        [
          "split",
          "u4"
        ],
        [
          "shift",
          "2f4"
        ],
        [
          "pose",
          "f4"
        ],
        [
          "psize_A",
          "f4"
        ],
        [
          "error",
          "f4"
        ],
        [
          "error_min",
          "f4"
        ],
        [
          "resid_pow",
          "f4"
        ],
        [
          "slice_pow",
          "f4"
        ],
        [
          "image_pow",
          "f4"
        ],
        [
          "cross_cor",
          "f4"
        ],
        [
          "alpha",
          "f4"
        ],
        [
          "alpha_min",
          "f4"
        ],
        [
          "weight",
          "f4"
        ],
        [
          "pose_ess",
          "f4"
        ],
        [
          "shift_ess",
          "f4"
        ],
        [
          "class_posterior",
          "f4"
        ],
        [
          "class",
          "u4"
        ],
        [
          "class_ess",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.alignments2D (result_name)",
      "name": "alignments2D",
      "title": "Passthrough alignments2D",
      "versions": [],
      "type": "particle.alignments2D"
    },
    {
      "num_items": [],
      "uid": "J185-R20",
      "passthrough": true,
      "min_fields": [
        [
          "ncc_score",
          "f4"
        ],
        [
          "power",
          "f4"
        ],
        [
          "template_idx",
          "u4"
        ],
        [
          "angle_rad",
          "f4"
        ]
      ],
      "group_name": "particles",
      "metafiles": [],
      "description": "Passthrough from input particles.pick_stats (result_name)",
      "name": "pick_stats",
      "title": "Passthrough pick_stats",
      "versions": [],
      "type": "particle.pick_stats"
    }
  ],
  "token_acquired_at": null,
  "cluster_job_id": null,
  "interactive_hostname": "c107925",
  "running_at": "2020-07-09T21:59:16.135Z",
  "last_exported": "2020-07-09T21:59:02.055Z",
  "ui_layouts": {
    "P3": {
      "show": true,
      "top": 5572,
      "height": 192,
      "width": 298,
      "groups": [
        {
          "lane": 5,
          "rightendcol": -435,
          "single": true,
          "len": 1,
          "subtree": 0,
          "startrow": 21,
          "start": "J185",
          "startcol": -435,
          "leftendcol": -435,
          "conns": [
            {
              "origin": [
                1790,
                5764
              ],
              "end": "J184",
              "endcol": -435,
              "subtree": 0,
              "startrow": 21,
              "start": "J185",
              "startcol": -435,
              "coords": [
                [
                  0,
                  0
                ],
                [
                  0,
                  120
                ],
                [
                  10,
                  120
                ],
                [
                  10,
                  160
                ]
              ],
              "extent": [
                10,
                160
              ],
              "coloridx": 36,
              "endrow": 22
            }
          ],
          "endrow": 22
        }
      ],
      "floater": false,
      "left": 1641
    },
    "P3W8": {
      "show": true,
      "top": 5112,
      "height": 192,
      "width": 298,
      "groups": [
        {
          "lane": 2,
          "rightendcol": -96,
          "single": true,
          "len": 1,
          "subtree": 0,
          "startrow": 21,
          "start": "J185",
          "startcol": 73,
          "leftendcol": -96,
          "conns": [
            {
              "origin": [
                597,
                5304
              ],
              "end": "J184",
              "endcol": -96,
              "subtree": 0,
              "startrow": 21,
              "start": "J185",
              "startcol": 73,
              "coords": [
                [
                  159,
                  0
                ],
                [
                  159,
                  60
                ],
                [
                  0,
                  60
                ],
                [
                  0,
                  80
                ]
              ],
              "extent": [
                159,
                80
              ],
              "coloridx": 23,
              "endrow": 22
            }
          ],
          "endrow": 22
        }
      ],
      "floater": false,
      "left": 607
    }
  },
  "queued_to_hostname": false,
  "queued_to_gpu": false,
  "no_check_inputs_ready": false,
  "num_tokens": 4,
  "job_sig": "358456195652047346155394767988713030953556654085432455890825361761340233915740651792429361024162856619906907340890793061144428891210051547664920703322928692133813287026884402609934304503111163187091564042652406545836511112039288280125213862501940880876746395718755194013755645056090551866166007903423934144704668586095034906628479873071348205362830480680794594104212660853788550177309763525140051867915235081598026341991976003908795687552452532560997092414211676843716476866839996513499830920378454118526856199600340437360055158435174456073651443247566848262038306172536583632817229505934545939208048362681778258694",
  "tokens_acquired_at": 1594331951.529313,
  "status_num": 40
}

merandamasse · July 13, 2020, 5:05pm

Sorry I had to split things up because of the character limit.

I also have frequently run a ps aux command to check for anything running in the background, and have never seen anything.

I have also gotten this “heartbeat” error in version 2.0, 2.13 and 2.14.

Thanks!
Meranda

mmclean · July 14, 2020, 3:42pm

Hi Meranda,

Apologies for the miscommunication – could you also provide us with the job.log file for any recent refinement/ab-initio jobs you have ran? The project directory can be found under the “Details” sidebar of a project in the UI, and the job directory will be within this. For example if it was Project 2 and Job 95, the log would be in P2/J95/job.log), so it is not directly accessible from the UI. This could help us see where in the job the error is actually occurring, and if it has anything to do with memory, etc.

Thank you and best regards,
Michael

merandamasse · July 14, 2020, 4:28pm

Hi @mmclean,

so sorry, but I can’t find this file. Is it going to be somewhere in the cryosparc_worker directory?

Thanks!
Meranda

mmclean · July 14, 2020, 9:47pm

Hi Meranda,

The location of a project directory generally won’t be within the cryosparc_worker directory – Are you able to first view the project directory through the UI? If you first go to your cryosparc “Projects” page by clicking the Project button on the lower left-hand corner, then navigate to your project of interest as done in the GIF below, you can find the project directory, and the job directory of interest will be a direct subdirectory of the project directory. Are you then able to locate the project directory in your system’s filesystem?

merandamasse · August 20, 2020, 2:22pm

Hi @mmclean, sorry for not getting back sooner. After playing around some and calling Exxact, it looks like I had a faulty DIMM chip, so no issues were found with CryoSPARC. Thank you for your help