Hello,
Ab-Initio Reconstruction never progresses past iteration 0. Noise model estimation appears stuck:
Iteration 0 (epoch 0.001). radwn 10.10 resolution 35.00A minisize 100 beta 0.10
Estimating noise model to meet target of 98.17 poses.. Current ESS R is 8926.19 poses.. Current sigma is 0.10 New sigma is 0.10
ESS never converges toward the target and sigma stays at 0.10 (left running >9 h once, no progress; sigma never updates).
On startup, before iteration 0, job.log shows:
…/threading.py:1075: RuntimeWarning: invalid value encountered in scalar divide
…/threading.py:1075: RuntimeWarning: divide by zero encountered in scalar divide
…/cryosparcw.py:290: UserWarning: Cannot manually free CUDA array; will be freed when garbage collected
and an NVRTC warning while compiling a kernel (note: NaN-counting kernel):
kernel(35): warning #68-D: integer conversion resulted in a change of sign
my_nan_count += __shfl_xor_sync(-1, my_nan_count, x);
Already ruled out (extensive testing):
-Data are clean: input particles have blob/psize_A = [0.93], blob/shape = [380], no NaN in psize; ctf/df1_A, df2_A sensible (~14000 Å, zero zeros), accel_kv = 300, cs_mm = 2.7. The particles give excellent 2D class averages.
-Independent of number of classes (identical with 6 and with 1).
-Independent of particle count (identical with 200k and with 10k).
-Independent of window settings (identical with spherical window and with No window).
-Independent of noise model (identical with symmetric and with white).
-Otherwise default parameters.
-The target of 98.17 poses value is identical across all runs.
-Not the SLURM environment: reproduces both on our SLURM cluster lane AND on a directly-connected node lane (clean base environment). On that node cryosparcw gpulist runs cleanly with no CUDA errors.
Environment:
-cryoSPARC v5.0.4
-Worker built via pixi (.pixi/envs/worker, Python 3.12)
-A100-SXM4-40GB, NVIDIA driver 570.86.15 (CUDA 12.8); submission script loads CUDA/12.4.0
-2D Classification and other GPU jobs run fine on this installation; only Ab-Initio shows this.
Questions:
-Is the divide by zero / invalid value in scalar divide during noise model estimation a known issue in v5.0.4, particularly on A100 with the pixi worker build?
-Could the NVRTC warning in the NaN-counting kernel indicate a miscompiled CUDA kernel? Would clearing the numba kernel cache / rebuilding the worker help?
-Which version would you recommend for a stable A100 / pixi setup? (We previously ran v4.7.1 but had separate issues there; we’d like to roll back to a version where both 2D and Ab-Initio are stable.)
Thanks for any guidance.
