Check for corrupt particles job should check for infinities

Features and Functionality
1

Hello. I was processing EMPIAR 10255 and I was seeing Nan’s in the ab initio job.

I checked for corrupt particles and nothing came back as problematic.

It seems like there are particles here with some pixels read in as infinities. When I removed the offending particles myself with a script in python the jobs reconstruct just fine.

It would be useful if the find corrupt particles job could detected infinities as well as NaNs.

using version cryosparc 4.6.

2

Hi @Miro-Astore. The optional “nan-check” feature actually does check for infinities in addition to NaNs. You’re right that the naming doesn’t correctly indicate this, though. How did you identify the problematic particles and what exactly did you find? For example, if you read the MRC files and got IEEE-754 Inf values, but the nan-check feature of the check for corrupt particles job did not flag those same values, I’ll be rather puzzled…
–Harris

3

Hi Harris, Thank for picking this up.

I used the following script to check for infinities and write a new star file. I’m happy to send you a slice of the problematic particles. What I found was that some of them had infinity values for their pixels.

Let me know if you have more questions.


import tqdm
import mrcfile
import pandas as pd 
import numpy as np
import starfile


# === Parameters ===
input_path = 'output.mrcs'
output_path = 'cleaned.mrc'
output_star = 'no_infs.star'
batch_size = 1000  # Number of particles per batch, adjust as needed

# === Load STAR file ===
starfile_rows = starfile.read('output.star')
good_starfile_rows = pd.DataFrame()

# === First pass: Check the shape and total number of particles ===
with mrcfile.mmap(input_path, mode='r') as mrc:
    data_shape = mrc.data.shape  # Should be (N, H, W)
    N, H, W = data_shape
    print(f"Found {N} particles of shape {H}x{W}")

# === Process batches of particles ===
print('Beginning batch cleaning...')
cleaned_particles = []

with mrcfile.mmap(input_path, permissive=True) as mrc:
    for i in tqdm.tqdm(range(0, N , batch_size)):
        batch = mrc.data[i:i+batch_size]
        star_batch = starfile_rows['particles'][i:i+batch_size]
        
        # Find good (non-inf) particles
        is_finite = np.isfinite(batch).all(axis=(1, 2))
        new_rows = star_batch[is_finite]
        good_starfile_rows = pd.concat([good_starfile_rows, new_rows], ignore_index=True)
        good_particles = batch[is_finite]
        cleaned_particles.append(good_particles)

# === Final integrity checks ===
if len(cleaned_particles) == 0:
    raise ValueError("No clean particles found! Check the input MRC data for NaNs or Infs.")

cleaned_data = np.concatenate(cleaned_particles, axis=0)
good_starfile_rows = good_starfile_rows.reset_index(drop=True)

if good_starfile_rows.empty:
    raise ValueError("Resulting STAR particle table is empty — nothing to write.")

if 'optics' not in starfile_rows:
    raise ValueError("No 'optics' block found in original STAR file.")

print("Final STAR blocks:")
print("Optics block shape:", starfile_rows['optics'].shape)
print("Particles block shape:", good_starfile_rows.shape)

# === Sanitize DataFrame columns ===
def sanitize_column(col):
    # Flatten any list-like or array-like values and cast to string
    return col.apply(lambda x: str(x) if np.isscalar(x) or isinstance(x, str) else str(np.array(x).tolist()))

for colname in good_starfile_rows.columns:
    if good_starfile_rows[colname].dtype == 'object':
        try:
            print(f"Sanitizing column: {colname}")
            good_starfile_rows[colname] = sanitize_column(good_starfile_rows[colname])
        except Exception as e:
            print(f"Warning: could not sanitize column {colname}: {e}")

# === Write STAR file ===
blocks = {
    'optics': starfile_rows['optics'].reset_index(drop=True),
    'particles': good_starfile_rows.reset_index(drop=True)
}

starfile.write(blocks, output_star)
print(f"Wrote cleaned STAR file to: {output_star}")

# === Write cleaned MRC stack ===
with mrcfile.new(output_path, overwrite=True) as mrc_out:
    mrc_out.set_data(cleaned_data.astype(np.float32))

print(f"Saved {cleaned_data.shape[0]} clean particles to {output_path}")
4

@Miro-Astore (with apologies for the delay) in that case I would be interested in getting the file and having a look at why this wasn’t caught. If you have a convenient way to share the mrc with me, please DM me with retrieval instructions. If you prefer I can send you instructions on how you can scp/rsync it to us.
– Harris