Duplicate rejection bug in Class 2D in v5.02?

2D Classification
1

Hi,

I think there is an issue with duplicate rejection during Class2D in v5.

I have a job with an input stack of 95k particles, coming from blob picker (200-400Å) with default minimum separation parameters.

Class2D in previous versions rejected zero particles; the same job, cloned and run in v5.02, is rejecting 25k particles (according to the log because they fail to meet the minimum separation criterion of 20 Å). I am almost certain that there are no true duplicates being identified here..

Best,

Oli

EDIT:

Partial metadata for the job with duplicate rejections, in case it is useful.

{
  "id": "6991a0fa6970b118db092153",
  "children": [],
  "completed_at": "2026-02-15T11:24:00.504Z",
  "created_at": "2026-02-15T10:33:30.000Z",
  "created_by_job_uid": "J12",
  "created_by_user_id": "67638623f30c28a0a6ed322a",
  "deleted": false,
  "failed_at": null,
  "interactive": false,
  "job_type": "class_2D_new",
  "killed_at": null,
  "launched_at": "2026-02-15T10:33:35.813Z",
  "output_group_images": {
    "class_averages": "6991accda5944cd35ea12f6d",
    "particles": "6991a11ba5944cd35ea12028"
  },
  "output_result_groups": [
    {
      "uid": "J321-G0",
      "type": "particle",
      "name": "particles",
      "title": "Particles",
      "description": "Particles that were successfully processed, including alignments across all classes.",
      "contains": [
        {
          "uid": "J321-R0",
          "name": "blob",
          "type": "particle.blob",
          "group_name": "particles",
          "passthrough": false
        },
        {
          "uid": "J321-R1",
          "name": "ctf",
          "type": "particle.ctf",
          "group_name": "particles",
          "passthrough": false
        },
        {
          "uid": "J321-R2",
          "name": "alignments2D",
          "type": "particle.alignments2D",
          "group_name": "particles",
          "passthrough": false
        },
        {
          "uid": "J321-R3",
          "name": "pick_stats",
          "type": "particle.pick_stats",
          "group_name": "particles",
          "passthrough": true
        },
        {
          "uid": "J321-R4",
          "name": "location",
          "type": "particle.location",
          "group_name": "particles",
          "passthrough": true
        }
      ],
      "passthrough": "particles",
      "num_items": 71275,
      "summary": {
        "alignments2D/psize_A": 1.5975000858306885,
        "blob/psize_A": 1.5975000858306885,
        "blob/shape": [400, 400],
        "blob/sign": -1,
        "ctf/accel_kv": 300,
        "ctf/amp_contrast": 0.10000000149011612,
        "ctf/anisomag": [0, 0, 0, 0],
        "ctf/bfactor": 0,
        "ctf/cs_mm": 0.0010000000474974513,
        "ctf/phase_shift_rad": 0,
        "ctf/scale": 1,
        "ctf/shift_A": [0, 0],
        "ctf/tetra_A": [0, 0, 0, 0],
        "ctf/tilt_A": [0, 0],
        "ctf/trefoil_A": [0, 0],
        "ctf/type": "spline",
        "location/micrograph_psize_A": 1.065000057220459,
        "location/micrograph_shape": [4092, 5760]
      },
      "latest_summary_stats": {
        "mean_class_ESS": 2.4064605236053467,
        "version": 89
      }
    },
    {
      "uid": "J321-G1",
      "type": "particle",
      "name": "particles_rejected",
      "title": "Rejected particles",
      "description": "Particles rejected based on a set minimum separation distance",
      "contains": [
        {
          "uid": "J321-R5",
          "name": "blob",
          "type": "particle.blob",
          "group_name": "particles_rejected",
          "passthrough": false
        },
        {
          "uid": "J321-R6",
          "name": "ctf",
          "type": "particle.ctf",
          "group_name": "particles_rejected",
          "passthrough": false
        },
        {
          "uid": "J321-R7",
          "name": "alignments2D",
          "type": "particle.alignments2D",
          "group_name": "particles_rejected",
          "passthrough": false
        },
        {
          "uid": "J321-R8",
          "name": "pick_stats",
          "type": "particle.pick_stats",
          "group_name": "particles_rejected",
          "passthrough": true
        },
        {
          "uid": "J321-R9",
          "name": "location",
          "type": "particle.location",
          "group_name": "particles_rejected",
          "passthrough": true
        }
      ],
      "passthrough": "particles",
      "num_items": 23419,
      "summary": {
        "alignments2D/psize_A": 1.5975000858306885,
        "blob/psize_A": 1.5975000858306885,
        "blob/shape": [400, 400],
        "blob/sign": -1,
        "ctf/accel_kv": 300,
        "ctf/amp_contrast": 0.10000000149011612,
        "ctf/anisomag": [0, 0, 0, 0],
        "ctf/bfactor": 0,
        "ctf/cs_mm": 0.0010000000474974513,
        "ctf/phase_shift_rad": 0,
        "ctf/scale": 1,
        "ctf/shift_A": [0, 0],
        "ctf/tetra_A": [0, 0, 0, 0],
        "ctf/tilt_A": [0, 0],
        "ctf/trefoil_A": [0, 0],
        "ctf/type": "spline",
        "location/micrograph_psize_A": 1.065000057220459,
        "location/micrograph_shape": [4092, 5760]
      }
    },
    {
      "uid": "J321-G2",
      "type": "template",
      "name": "class_averages",
      "title": "2D class averages",
      "description": "2D class average templates",
      "contains": [
        {
          "uid": "J321-R10",
          "name": "blob",
          "type": "template.blob",
          "group_name": "class_averages",
          "passthrough": false
        }
      ],
      "passthrough": false,
      "num_items": 100,
      "summary": {
        "blob/psize_A": 2.496093988418579,
        "blob/res_A": 25.78557777404785,
        "blob/shape": [256, 256]
      }
    }
  ],
  "params_spec": {
    "class2D_K": {"value": 100},
    "class2D_sigma_init_factor": {"value": 1},
    "class2D_num_full_iter": {"value": 10},
    "class2D_num_full_iter_batch": {"value": 80},
    "random_seed_default": {"value": 396013774}
  },
  "parents": ["J10"],
  "project_uid": "P20",
  "project_uid_num": 20,
  "queued_at": "2026-02-15T10:33:34.116Z",
  "queued_to_lane": "default",
  "resources_allocated": {
    "lane": "default",
    "lane_type": "node",
    "hostname": "ANES002074D",
    "slots": {"CPU": [0, 1, 2, 3], "GPU": [1], "RAM": [1, 2, 3]},
    "fixed": {"SSD": true},
    "licenses_acquired": 1,
    "target": {
      "cache_path": "/scratch/cryosparc_cache",
      "cache_reserve_mb": 10000,
      "cache_quota_mb": null,
      "lane": "default",
      "name": "ANES002074D",
      "title": "Worker ANES002074D",
      "desc": null,
      "hostname": "ANES002074D",
      "worker_bin_path": "/home/user/software/cryosparc/cryosparc_worker/bin/cryosparcw",
      "type": "node",
      "ssh_str": "user@ANES002074D",
      "resource_slots": {
        "CPU": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63
        ],
        "GPU": [0, 1, 2, 3],
        "RAM": [
          0,
          1,
          2,
          3,
          4,
          5,
          6,
          7,
          8,
          9,
          10,
          11,
          12,
          13,
          14,
          15,
          16,
          17,
          18,
          19,
          20,
          21,
          22,
          23,
          24,
          25,
          26,
          27,
          28,
          29,
          30,
          31,
          32,
          33,
          34,
          35,
          36,
          37,
          38,
          39,
          40,
          41,
          42,
          43,
          44,
          45,
          46,
          47,
          48,
          49,
          50,
          51,
          52,
          53,
          54,
          55,
          56,
          57,
          58,
          59,
          60,
          61,
          62,
          63
        ]
      },
      "resource_fixed": {"SSD": true},
      "monitor_port": null,
      "gpus": [
        {"id": 0, "name": "NVIDIA RTX 6000 Ada Generation", "mem": 51527024640},
        {"id": 1, "name": "NVIDIA RTX 6000 Ada Generation", "mem": 51527024640},
        {"id": 2, "name": "NVIDIA RTX 6000 Ada Generation", "mem": 51527024640},
        {"id": 3, "name": "NVIDIA RTX 6000 Ada Generation", "mem": 51527024640}
      ]
    }
  },
  "running_at": "2026-02-15T10:33:40.289Z",
  "starred_by": [],
  "started_at": "2026-02-15T10:33:36.969Z",
  "status": "completed",
  "status_num": 35,
  "tags": [],
  "title": "",
  "ui_tile_images": [
    {
      "name": "classes",
      "fileid": "6991accda5944cd35ea12f6b",
      "num_rows": 1,
      "num_cols": 3
    }
  ],
  "ui_tile_width": 3,
  "uid": "J321",
  "uid_num": 321,
  "workspace_uids": ["W1"],
  "updated_at": "2026-02-15T17:06:02.798Z",
  "dumped_at": "2026-02-15T11:24:08.983Z",
  "last_dumped_version": "v5.0.2",
  "autodump": false,
  "spec": {
    "type": "class_2D_new",
    "params": {
      "prepare_window_dataset": true,
      "prepare_window_inner_radius": 0.85,
      "prepare_window_outer_radius": 0.99,
      "class2D_K": 100,
      "class2D_max_res": 6,
      "class2D_max_res_align": null,
      "class2D_sigma_init_factor": 1,
      "class2D_window": true,
      "class2D_window_inner_A": null,
      "class2D_window_outer_A": null,
      "class2D_recenter": true,
      "class2D_recenter_thresh": 0.2,
      "class2D_recenter_binary": false,
      "class2D_estimate_in_plane_pose": false,
      "class2D_output_filaments": false,
      "class2D_remove_duplicate_particles": true,
      "class2D_min_dist_A": 20,
      "class2D_mic_psize_A": null,
      "class2D_clamp": false,
      "class2D_do_ctf": true,
      "class2D_min_res_align": null,
      "class2D_sort_classes_by_size": true,
      "class2D_sort_method": "similarity",
      "class2D_hard_class_last_iter": false,
      "class2D_do_alignment": true,
      "class2D_force_max": true,
      "class2D_ctf_phase_flip_only": false,
      "class2D_random_perturb_rs": null,
      "class2D_num_full_iter": 10,
      "class2D_num_full_iter_batch": 80,
      "class2D_num_full_iter_batchsize_per_class": 100,
      "class2D_init_scale": 1,
      "class2D_zp_factor": 2,
      "class2D_ignore_dc": true,
      "class2D_min_over_scale": false,
      "class2D_nonneg": false,
      "class2D_use_frc_reg": true,
      "class2D_use_frc_reg_full": true,
      "class2D_sigma_init_iter": 2,
      "class2D_sigma_num_anneal_iters": 15,
      "class2D_sigma_use_white": false,
      "intermediate_plots": true,
      "class2D_scale_bars": true,
      "random_seed": 396013774,
      "random_seed_default": 396013774,
      "compute_use_ssd": true,
      "compute_num_gpus": 1,
      "generate_intermediate_results": false
    },
    "inputs": {
      "particles": {
        "type": "particle",
        "title": "Particle stacks",
        "description": "Particle stacks to use. Multiple stacks will be concatenated.",
        "slots": [
          {"name": "blob", "dtype": "blob", "required": true},
          {"name": "ctf", "dtype": "ctf", "required": false},
          {"name": "filament", "dtype": "filament", "required": false},
          {"name": "pick_stats", "dtype": "pick_stats", "required": false},
          {"name": "location", "dtype": "location", "required": false},
          {"name": "alignments3D", "dtype": "alignments3D", "required": false}
        ],
        "count_min": 1,
        "count_max": "inf",
        "repeat_allowed": false,
        "connections": [
          {
            "job_uid": "J10",
            "output": "particles",
            "results": [
              {
                "name": "blob",
                "dtype": "blob",
                "job_uid": "J10",
                "output": "particles",
                "result": "blob",
                "version": "F"
              },
              {
                "name": "ctf",
                "dtype": "ctf",
                "job_uid": "J10",
                "output": "particles",
                "result": "ctf",
                "version": "F"
              },
              {
                "name": "pick_stats",
                "dtype": "pick_stats",
                "job_uid": "J10",
                "output": "particles",
                "result": "pick_stats",
                "version": "F"
              },
              {
                "name": "location",
                "dtype": "location",
                "job_uid": "J10",
                "output": "particles",
                "result": "location",
                "version": "F"
              }
            ]
          }
        ]
      }
    },
    "outputs": {
      "particles": {
        "type": "particle",
        "title": "Particles",
        "description": "Particles that were successfully processed, including alignments across all classes.",
        "slots": [
          {"name": "blob", "dtype": "blob"},
          {"name": "ctf", "dtype": "ctf"},
          {"name": "alignments2D", "dtype": "alignments2D"}
        ],
        "passthrough": "particles",
        "results": [
          {
            "name": "blob",
            "dtype": "blob",
            "versions": [89],
            "metafiles": ["J321/J321_089_particles.cs"],
            "num_items": [71275],
            "passthrough": false
          },
          {
            "name": "ctf",
            "dtype": "ctf",
            "versions": [89],
            "metafiles": ["J321/J321_089_particles.cs"],
            "num_items": [71275],
            "passthrough": false
          },
          {
            "name": "alignments2D",
            "dtype": "alignments2D",
            "versions": [89],
            "metafiles": ["J321/J321_089_particles.cs"],
            "num_items": [71275],
            "passthrough": false
          },
          {
            "name": "pick_stats",
            "dtype": "pick_stats",
            "versions": [0],
            "metafiles": ["J321/J321_passthrough_particles.cs"],
            "num_items": [71275],
            "passthrough": true
          },
          {
            "name": "location",
            "dtype": "location",
            "versions": [0],
            "metafiles": ["J321/J321_passthrough_particles.cs"],
            "num_items": [71275],
            "passthrough": true
          }
        ],
        "num_items": 71275,
        "image": "6991a11ba5944cd35ea12028",
        "summary": {
          "alignments2D/psize_A": 1.5975000858306885,
          "blob/psize_A": 1.5975000858306885,
          "blob/shape": [400, 400],
          "blob/sign": -1,
          "ctf/accel_kv": 300,
          "ctf/amp_contrast": 0.10000000149011612,
          "ctf/anisomag": [0, 0, 0, 0],
          "ctf/bfactor": 0,
          "ctf/cs_mm": 0.0010000000474974513,
          "ctf/phase_shift_rad": 0,
          "ctf/scale": 1,
          "ctf/shift_A": [0, 0],
          "ctf/tetra_A": [0, 0, 0, 0],
          "ctf/tilt_A": [0, 0],
          "ctf/trefoil_A": [0, 0],
          "ctf/type": "spline",
          "location/micrograph_psize_A": 1.065000057220459,
          "location/micrograph_shape": [4092, 5760]
        },
        "latest_summary_stats": {
          "mean_class_ESS": 2.4064605236053467,
          "version": 89
        }
      },
      "particles_rejected": {
        "type": "particle",
        "title": "Rejected particles",
        "description": "Particles rejected based on a set minimum separation distance",
        "slots": [
          {"name": "blob", "dtype": "blob"},
          {"name": "ctf", "dtype": "ctf"},
          {"name": "alignments2D", "dtype": "alignments2D"}
        ],
        "passthrough": "particles",
        "results": [
          {
            "name": "blob",
            "dtype": "blob",
            "versions": [89],
            "metafiles": ["J321/J321_particles_rejected.cs"],
            "num_items": [23419],
            "passthrough": false
          },
          {
            "name": "ctf",
            "dtype": "ctf",
            "versions": [89],
            "metafiles": ["J321/J321_particles_rejected.cs"],
            "num_items": [23419],
            "passthrough": false
          },
          {
            "name": "alignments2D",
            "dtype": "alignments2D",
            "versions": [89],
            "metafiles": ["J321/J321_particles_rejected.cs"],
            "num_items": [23419],
            "passthrough": false
          },
          {
            "name": "pick_stats",
            "dtype": "pick_stats",
            "versions": [0],
            "metafiles": ["J321/J321_passthrough_particles_rejected.cs"],
            "num_items": [23419],
            "passthrough": true
          },
          {
            "name": "location",
            "dtype": "location",
            "versions": [0],
            "metafiles": ["J321/J321_passthrough_particles_rejected.cs"],
            "num_items": [23419],
            "passthrough": true
          }
        ],
        "num_items": 23419,
        "image": null,
        "summary": {
          "alignments2D/psize_A": 1.5975000858306885,
          "blob/psize_A": 1.5975000858306885,
          "blob/shape": [400, 400],
          "blob/sign": -1,
          "ctf/accel_kv": 300,
          "ctf/amp_contrast": 0.10000000149011612,
          "ctf/anisomag": [0, 0, 0, 0],
          "ctf/bfactor": 0,
          "ctf/cs_mm": 0.0010000000474974513,
          "ctf/phase_shift_rad": 0,
          "ctf/scale": 1,
          "ctf/shift_A": [0, 0],
          "ctf/tetra_A": [0, 0, 0, 0],
          "ctf/tilt_A": [0, 0],
          "ctf/trefoil_A": [0, 0],
          "ctf/type": "spline",
          "location/micrograph_psize_A": 1.065000057220459,
          "location/micrograph_shape": [4092, 5760]
        },
        "latest_summary_stats": {}
      },
      "class_averages": {
        "type": "template",
        "title": "2D class averages",
        "description": "2D class average templates",
        "slots": [{"name": "blob", "dtype": "blob"}],
        "passthrough": null,
        "results": [
          {
            "name": "blob",
            "dtype": "blob",
            "versions": [89],
            "metafiles": ["J321/J321_089_class_averages.cs"],
            "num_items": [100],
            "passthrough": false
          }
        ],
        "num_items": 100,
        "image": "6991accda5944cd35ea12f6d",
        "summary": {
          "blob/psize_A": 2.496093988418579,
          "blob/res_A": 25.78557777404785,
          "blob/shape": [256, 256]
        },
        "latest_summary_stats": {}
      }
    },
    "ui_tile_width": 3,
    "ui_tile_height": 1,
    "resource_spec": {"cpu": 4, "gpu": 1, "ram": 3, "ssd": true}
  },
  "job_dir": "J321",
  "job_dir_size": 532235436,
  "job_dir_size_last_updated": "2026-02-15T11:23:59.453Z",
  "run_as_user": null,
  "description": "",
  "cloned_from": "J12",
  "waiting_at": null,
  "heartbeat_at": "2026-02-15T11:23:58.932Z",
  "tokens_acquired_at": "2026-02-15T10:33:35.808Z",
  "tokens_requested_at": null,
  "last_scheduled_at": "2026-02-15T10:33:34.586Z",
  "last_accessed": {
    "name": "clarkelab",
    "accessed_at": "2026-02-15T17:06:02.798Z"
  },
  "has_error": false,
  "has_warning": false,
  "version_created": "v5.0.2",
  "version": "v5.0.2",
  "priority": 0,
  "deleting": false,
  "queued_by_user_id": "67638623f30c28a0a6ed322a",
  "queued_to_hostname": null,
  "queued_to_gpu": null,
  "queue_status": null,
  "queue_message": null,
  "queued_job_hash": null,
  "num_tokens": 1,
  "errors_run": [],
  "interactive_port": null,
  "PID_monitor": 3123981,
  "PID_main": 3123970,
  "PID_workers": [],
  "cluster_job_id": null,
  "cluster_job_status": null,
  "cluster_job_status_code": null,
  "cluster_job_monitor_event_id": null,
  "cluster_job_monitor_retries": 0,
  "cluster_job_monitor_last_run_at": null,
  "cluster_job_submission_script": null,
  "cluster_job_custom_vars": {},
  "is_experiment": false,
  "enable_bench": false,
  "bench": {},
  "bench_timings": {},
  "completed_count": 1,
  "instance_information": {
    "CUDA_version": "12.8",
    "available_memory": "481.84GB",
    "cpu_model": "AMD Ryzen Threadripper PRO 7975WX 32-Cores",
    "driver_version": "12.8",
    "gpu_info": [
      {
        "id": 1,
        "name": "NVIDIA RTX 6000 Ada Generation",
        "mem": 51527024640,
        "bus_id": "0000:02:00",
        "compute_mode": "Default",
        "persistence_mode": "Enabled",
        "power_limit": 300,
        "sw_power_limit": "Not Active",
        "hw_power_limit": "Not Active",
        "max_pcie_link_gen": 4,
        "current_pcie_link_gen": 1,
        "temperature": 37,
        "gpu_utilization": 0,
        "memory_utilization": 0,
        "driver_version": "570.172.08"
      }
    ],
    "max_cpu_freq": 5352,
    "nvrtc_version": "12.8",
    "ofd_hard_limit": 1048576,
    "ofd_soft_limit": 1024,
    "physical_cores": 32,
    "platform_architecture": "x86_64",
    "platform_node": "ANES002074D",
    "platform_release": "6.8.0-59-generic",
    "platform_version": "#61-Ubuntu SMP PREEMPT_DYNAMIC Fri Apr 11 23:16:11 UTC 2025",
    "toolkit_version": "12.8",
    "total_memory": "503.19GB",
    "used_memory": "17.31GB",
    "version": "v5.0.2"
  },
  "last_intermediate_data_cleared_at": null,
  "last_intermediate_data_cleared_amount": 0,
  "intermediate_results_size_bytes": 0,
  "intermediate_results_size_last_updated": "2026-02-15T11:23:59.453Z",
  "is_final_result": false,
  "is_ancestor_of_final_result": false,
  "no_check_inputs_ready": false,
  "progress": [
2

Hey @olibclarke – thanks for reporting, we’re looking into this now. Which version of CryoSPARC did the previous (zero rejected particles) job use?

3

Hi Valentin,

The previous job was run with v5.0.0-privatebeta.5 - thanks for taking a look!

Cheers

Oli

1 Like
4

Hey @olibclarke ,

Could you provide us with the full event log and job log for this job?

Thanks, Michael

1 Like
5

Hi Michael,

Sure thing - they should all be in this zip file:

Cheers

Oli

1 Like
6

Dear Oli,

Thank you for the job log and for bearing with my delay. Unfortunately I couldn’t locate sufficient information in the job reports, as 2D classification doesn’t produce all of the same logging info as the standalone Remove Duplicate Particles job does.

Are you able to provide the job reports for the following jobs:

  • Remove Duplicate Particles on the particles from the blob picker job, with the same 20Å separation distance, no micrographs input, and no manual pixel size parameter set?
  • Remove Duplicate Particles on the combined particles and rejected particles outputs from the erroneous 2D Classification job, with the same parameters?

Thank you again – these would help us figure out what’s going wrong.
Michael

1 Like
7

Hi Michael,

Looking at this closer it is a little more complicated than I realized initially:

  • The extracted particle stack input to Class2D was actually derived from 2 blob pick jobs (let’s call them pickA and pickB) with different template size range parameters. PickA had 43737 particles after inspect picks, and pickB had 77041 particles after inspect picks (both derived from the same micrograph set. The extraction of both sets (in v5-pb5) gave 94,694 particles, presumably less than the sum of the inputs due to near-edge rejection. Cloning and rerunning the extraction job in 5.02 still gives 94,694 particles, so nothing has changed there.
  • Running Remove Duplicates on pick A and pickB together gives 94948 particles - almost the same number seen in extraction of both sets together. However, if these are then extracted, the number of particles remaining is 72k, presumably due to near-edge rejection.
  • Running Remove Duplicates on the extraction job (prior to class2D) gives 72,134 particles accepted, and 22,560 particles rejected.
  • Running Remove Duplicates on the combined accepted/rejected outputs from the erroneous class2D job gives 71,275 accepted, 23419 rejected.
  • Running Remove Duplicates on the output of the other 2D job gives 67,788 accepted, 26,906 rejected.

Based on the above, what would be the most helpful logs to provide? I guess maybe based on this the issue is actually that Class2D duplicate rejection was not working properly in the v5-pb but is working now?

Cheers

Oli

8

Thanks @olibclarke for the detailed look into this. I think based on your description, the number of particles retained after combined extraction + standalone remove duplicates is always ~70k. There is variance here from:

  • 2D classification changing particle shifts, this affects which/how many particles fall below the minimum separation threshold (can impact subsequent remove duplicates jobs)
  • possible changes in box size (could affect near-edge particles)

Between v4.7 and v5.0.2 , there were changes to:

  • standalone remove duplicates (particularly ensuring the correct pixel size field was being used)
  • 2D classification’s default sorting method (similarity sort which can affect the centering of classes and hence can affect particle shifts) – though in general 2D classification will not produce the same number of accepted/rejected particles between any two runs

Given this, I think the behaviour described in your five bullet points above sounds expected and is consistent with there being ~70k true non-duplicate particles and ~25k true rejected particles.

If you encounter unexpected/wrong behaviour in a pipeline that used either of 2D class and/or remove duplicates (both run in v5.0.0+, neither run in any version before v5.0.0 so we can be sure that bugs in previous versions are not confusing the analysis) then please continue to report to us

Best,
Michael

9

Hi Michael,

Ok! just to clarify, the previous job was run in v5-pb5, not 4.7. The odd thing was that the cloned 2D jobs gave completely different numbers of output particles, despite the same inputs. So I guess it was a bug that has now been fixed? I don’t think going from 0 rejected to 25k is due to a variance in particle shifts..

Cheers

Oli

10

Dear Oli,

Ah yes – sorry I just re-read my post and I didn’t mention. There were indeed changes to both the standalone remove duplicates job, and the implementation of it in 2D classification, that happened between pb6 (last version with this bug) and pb7 (first version with the hopeful fix). But if you observe large inconsistencies on the order of >25% of the particle size, please definitely report them to us :slight_smile:

Michael

1 Like
11

Gotcha will do, thanks Michael!