5.0.6: ab-initio reconstruction loop

biocit · June 23, 2026, 9:03am

Hi

Since the upgrade to Cryosparc V5 some users are complaining that ab initio reconstruction jobs occasionally go into a loop where they keep running but never produce any output.

This issue appears to occur when an ab initio reconstruction job is missing a class.
I’d say this problem of endless running always coincides with this appearance that a class from the workspace-tile remains empty/dark. I have the feeling that it is stuck somewhere before finishing the first iteration. But what exactly causes what, I cannot see as the log doesn’t update anymore (last two lines from a job that is stuck since yesterday):

[2026-06-22 22:56:16.15] ----------- Iteration 0 (epoch 0.000). radwn 10.00 resolution 33.28A minisize 90 beta 0.10

[2026-06-22 22:56:18.40] Estimating noise model to meet target of 98.17 poses.. Current ESS R is 1116.56 poses.. Current sigma is 0.10 New sigma is 0.10

These jobs are fresh and not copied from jobs generated in v4

Any idea?

olibclarke · June 23, 2026, 9:18am

Usually in my experience this happens because users have unintentionally set minimum alignment resolution (highpass) rather than initial resolution - this parameter name should be changed, I think

biocit · June 23, 2026, 9:32am

Thanks. The job has these metadata:

    "abinit_min_res": {
      "type": "number",
      "value": null,
      "title": "Minimum alignment resolution (Angstroms)",
      "desc": "High-pass filter for the purpose of alignment",
      "order": 6,
      "section": "abinit",
      "advanced": true,
      "hidden": false
    }
    "abinit_init_res": {
      "type": "number",
      "value": 35,
      "title": "Initial resolution (Angstroms)",
      "desc": "Starting frequency to consider",
      "order": 7,
      "section": "abinit",
      "advanced": false,
      "hidden": false
    }

As far as i can see both values are set. So it would be better to remove the highpass value?

olibclarke · June 23, 2026, 9:34am

Hmmm no this seems like a different issue then, because the minimum alignment res here is set to “null”, meaning it shouldn’t be applied..

biocit · June 23, 2026, 10:35am

I got now the info that sometimes it helps to clone the job again and run it. Strange.

olibclarke · June 23, 2026, 10:55am

Looks like somehow one of the initial volumes is ending up empty. Cloning changes the random seed which would explain why that helps, but it shouldn’t initialize from an empty volume to begin with, that is odd

wtempel · June 23, 2026, 4:29pm

@biocit Thanks for your initial post and the additional information. What is the THP setting on the worker node, viewable via the following command:

cat /sys/kernel/mm/transparent_hugepage/enabled

biocit · June 23, 2026, 7:54pm

cat /sys/kernel/mm/transparent_hugepage/enabled
always madvise [never]

wtempel · June 25, 2026, 9:18pm

@biocit Please can you test the following

Check the input particles for corruption, including Check for NaN values.
Run a clone of job J3184
- disable Enforce non-negativity (advanced parameter)
- disable Cache particle images on SSD
Run 2D classification with the same input particles. Check the result for classes with blank density that have particles assigned

and let us know the results of these tests.

biocit · June 29, 2026, 2:13pm

@wtempel

I got this feedback:

Run 2D classification with the same input particles. Check the result for classes with blank density that have particles assigned

They submitted 2D classification jobs with 50 and 300 classes from the exact same subset. However, these particles already went through 2D classification and I didn’t notice any “blank” classes yet.

Both jobs contains particles in every 2D class.

Check the input particles for corruption, including Check for NaN values.

No error messages. But I found a Warning with a variable “my_nan_count” in the stack. But the same was also in a healty job.

This is the log part from an stuck Ab-Initio Reconstruction

gpufft: creating new cufft plan (plan id 0   pid 1443632) 
	gpu_id  2 
	ndims   2 
	dims    128 128 0 
	inembed 128 128 0 
	istride 1 
	idist   16384 
	onembed 128 128 0 
	ostride 1 
	odist   16384 
	batch   10 
	type    C2C 
	wkspc   automatic 
	Python traceback:

HOST ALLOCATION FUNCTION: using numba.cuda.pinned_array
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/site-packages/numba/cuda/cudadrv/nvrtc.py:257: UserWarning: NVRTC log messages whilst compiling kernel:

kernel(35): warning #68-D: integer conversion resulted in a change of sign
                  my_nan_count += __shfl_xor_sync(-1, my_nan_count, x);
                                                  ^

Remark: The warnings can be suppressed with "-diag-suppress <warning-number>"

kernel(44): warning #68-D: integer conversion resulted in a change of sign
                      my_nan_count += __shfl_xor_sync(-1, my_nan_count, x);
                                                      ^

kernel(17): warning #177-D: variable "N_I" was declared but never referenced
              unsigned N_I = gridDim.x;
                       ^
  warnings.warn(msg)
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/threading.py:1075: RuntimeWarning: divide by zero encountered in scalar divide
  self.run()
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/threading.py:1075: RuntimeWarning: invalid value encountered in scalar divide
  self.run()
/opt/cryosparc/cryosparc_worker/cli/cryosparcw.py:290: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected
  return run(conf)
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/threading.py:1075: RuntimeWarning: invalid value encountered in scalar divide
  self.run()
/opt/cryosparc/cryosparc_worker/.pixi/envs/worker/lib/python3.12/threading.py:1075: RuntimeWarning: divide by zero encountered in scalar divide
  self.run()
/opt/cryosparc/cryosparc_worker/cli/cryosparcw.py:290: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected
  return run(conf)
WARNING: io_uring support disabled (supported by kernel, but initialization failed), I/O performance may degrade
/opt/cryosparc/cryosparc_worker/compute/sigproc.py:387: RuntimeWarning: invalid value encountered in divide
  absM /= n.sum(absM)
2026-06-28 16:23:53,147 core                 heartbeat        INFO   | ========= Updating heartbeat
gpufft: creating new cufft plan (plan id 1   pid 1443632) 
	gpu_id  2 
	ndims   2 
	dims    128 128 0 
	inembed 128 128 0 
	istride 1 
	idist   16384 
	onembed 128 128 0 
	ostride 1 
	odist   16384 
	batch   90 
	type    C2C 
	wkspc   automatic 
	Python traceback:

/opt/cryosparc/cryosparc_worker/cli/cryosparcw.py:290: RuntimeWarning: divide by zero encountered in scalar divide
  return run(conf)
/opt/cryosparc/cryosparc_worker/cli/cryosparcw.py:290: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected
  return run(conf)
2026-06-28 16:24:03,157 core                 heartbeat        INFO   | ========= Updating heartbeat
2026-06-28 16:24:13,168 core                 heartbeat        INFO   | ========= Updating heartbeat
/opt/cryosparc/cryosparc_worker/cli/cryosparcw.py:290: RuntimeWarning: invalid value encountered in scalar divide
  return run(conf)

Run a clone of job J3184
disable Enforce non-negativity (advanced parameter)
disable Cache particle images on SSD

They deactivated both parameters on one job. But didn’t help.

biocit · June 29, 2026, 4:50pm

Check For Corrupt Particles

[CPU: 208.9 MB Avail: 496.15 GB] No corruption detected.

hsnyder · July 6, 2026, 2:57pm

Hi @biocit,

Thanks for making us aware of this - two questions:

Have you seen this behavior on other datasets, or just one?
Given the suspicion that this might be v5-related do you still have a v4 instance that you could test the same particle stack with?

Thanks

–Harris

biocit · July 6, 2026, 3:17pm

Hi @hsnyder

Have you seen this behavior on other datasets, or just one?

Currently one group complains for several datasets. I haven’t heard something from the other groups.

Given the suspicion that this might be v5-related do you still have a v4 instance that you could test the same particle stack with?

Unfortunately not. The complains started as soon as we switched to V5. I could ask if they run a job on an older dataset that they processed before successfully with V4. If that would help.

hsnyder · July 6, 2026, 6:02pm

Hi @biocit,

No problem - that being the case, would the group be willing to share the problematic data with Structura, confidentially? That may be the easiest way for us to go about debugging this.

–Harris

biocit · July 10, 2026, 10:51am

Quick Update:
The groups are not so open to share data. We try now if we can reproduce with an EMPIAR Testset