Old project throws errors when running jobs on updated CryoSPARC

Troubleshooting
1

Dear CryoSPARC team,

I had reason to go back to an old project today (old being “run on earlier CryoSPARC version”), and where before everything worked (2D, 3D, 3DVA, homo/hetero refine, NU refine) now I get the following error:

Traceback (most recent call last):
  File "/home/cryosparcer/bin/cryosparc_worker/cryosparc_compute/jobs/runcommon.py", line 2061, in run_with_except_hook
    run_old(*args, **kw)
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 131, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/cuda_core.py", line 132, in cryosparc_compute.engine.cuda_core.GPUThread.run
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 1101, in cryosparc_compute.engine.engine.process.work
  File "cryosparc_master/cryosparc_compute/engine/engine.py", line 390, in cryosparc_compute.engine.engine.EngineThread.find_and_set_best_pose_shift
  File "<__array_function__ internals>", line 5, in unravel_index
ValueError: index 1056227885 is out of bounds for array with size 336

The error is the same between all job types, but the ValueError variables change.

New projects (since the update) appear unaffected… one a similar complex and one completely different have run happily so far.

I’ve got a few more things to try (reextract particles, maybe the stack has been corrupted somehow?) but thought I’d see if anyone else has reported anything similar. The iteration the error occurs on varies, which is why I suspect corrupted particle(s) may be the cause. No filesystem or memory errors reported, though.

Thanks in advance.

2

hi, I’m not sure if it’s useful but there’s a “Check For Corrupt Particles” job in CryoSPARC – it might help understanding if it’s actually the problem.

For me, even switching from 3.x to 4.x didn’t cause problems like that, so I’d assume the particles are indeed the case. You might also check who modified the file last via ls -ltrah the_folder_with_particles – if someone messed with the files, a particular file should be different from others.

1 Like
3

Thanks. I did the corruption check. No filesystem related errors in dmesg, nothing unusual with file timestamps etc. I’ve got a re-extraction run queued so I’ll see what that throws up (or not, as the case may be)…

I guess we’ll see. :slight_smile:

4

Just to tidy this up; the server had two bad sticks of RAM - I presume that they were faulty from the day it arrived but it took time to manifest in a way that ECC wasn’t able to correct. They were replaced and the issue appears resolved.

I guess it was just (bad?) luck that it manifest the way it did. Cheers to @marinegor for the input!

1 Like