Curate exposures - Maximum allowed size exceeded

Exposure Curation
1

Dear all,

we recently updated to CS v4.0.3.
We sometimes get crazy high numbers from job Patch motion correction in the field Total full-frame motion distance (pixels). This is probably an artefact and usually it just a small number of exposures with this issue. In CS v3 we just filtered those out and that was it.
In v4.0.3 the job Manually curate exposures unfortunately fails when it encounters number that are too high. See the example below:

Total full-frame motion distance (pixels):
Min: 0.000	Max: 372585674088067777070075248755343360.000
Average: 193141664385286199290019702636544.000	Median: 17.191

Traceback (most recent call last):
  File "cryosparc_master/cryosparc_compute/run.py", line 93, in cryosparc_compute.run.main
  File "/d/hpc/software/cryosparc/cryosparc_master/cryosparc_compute/jobs/interactive/run_curate_exposures_v2.py", line 248, in run
    fig = plotutil.plot_hist(data_finite, 'auto', xlabel=f'{field["title"]}', ylabel='Number of exposures')
  File "/d/hpc/software/cryosparc/cryosparc_master/cryosparc_compute/plotutil.py", line 1448, in plot_hist
    ax.hist(vals,bins=bins,range=range, rwidth=rwidth, density=density, cumulative=cumulative)
  File "/d/hpc/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/matplotlib/__init__.py", line 1414, in inner
    return func(ax, *map(sanitize_sequence, args), **kwargs)
  File "/d/hpc/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/matplotlib/axes/_axes.py", line 6641, in hist
    m, bins = np.histogram(x[i], bins, weights=w[i], **hist_kwargs)
  File "<__array_function__ internals>", line 6, in histogram
  File "/d/hpc/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/numpy/lib/histograms.py", line 792, in histogram
    bin_edges, uniform_bins = _get_bin_edges(a, bins, range, weights)
  File "/d/hpc/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/numpy/lib/histograms.py", line 448, in _get_bin_edges
    endpoint=True, dtype=bin_type)
  File "<__array_function__ internals>", line 6, in linspace
  File "/d/hpc/software/cryosparc/cryosparc_master/deps/anaconda/envs/cryosparc_master_env/lib/python3.7/site-packages/numpy/core/function_base.py", line 128, in linspace
    y = _nx.arange(0, num, dtype=dt).reshape((-1,) + (1,) * ndim(delta))
ValueError: Maximum allowed size exceeded

Is there a way we can circumvent this or is this a bug you can perhaps fix?

Thanks!

2

I can also confirm that we see the same problem / phenomenon.
Manual curate exposures routinely crashes.

1 Like
3

Hi all,

Thanks for reporting – we have triaged this and a fix to the Exposure Curation ValueError should be included in the next release.

Best,
Michael

1 Like
4

Great thank you @mmclean and the cryoSPARC team!

All the best!

5

Dear @mmclean and colleagues,

Running Curate exposures I received the following error (but the protocol finishes ok) -

Warnings:
Jun 9, 2024, 8:05:15 PM

Histogram for Total full-frame motion distance (pixels) failed due to ValueError: Maximum allowed size exceeded
Jun 9, 2024, 8:05:15 PM
Histogram for Full-frame motion curvature failed due to ValueError: Maximum allowed size exceeded

Any ideas on how to fix that?

Thank you

Kind regards,
Dmitry

P.s. CS v 4.5.3

6

These messages concern the generation of histogram plots, and do not indicate a failure of the job as a whole. Exposures are still accepted or rejected according to the configured thresholds.

1 Like