SIGSEGV in core/ioengine/core.so across multiple job types

Troubleshooting
1

Hi

We’re seeing a random segmentation fault on our cryoSPARC installation that appears to originate from the native I/O engine. After several rounds of testing, the issue no longer seems tied to a specific dataset, worker, GPU, or SSD cache configuration.

Environment

  • cryoSPARC version: v5.0.6+260710
  • Multiple worker nodes
  • GPUs tested:
    • RTX 2080 Ti
    • RTX 4090
    • RTX A5000

Crash signature

All failures end with the same native stack trace:

Received SIGSEGV (addr=0000000000000000)

core/ioengine/core.so(get_npy_dtype_from_str+0x18)
core/ioengine/core.so(wrap_array_from_iocmd+0x60)

The register dump is also consistent across all crashes, with:

rdi = 0000000000000000

indicating that get_npy_dtype_from_str() is being called with a null pointer.

What we’ve tested

1. Local CTF Refinement

The same job crashes reproducibly on:

  • RTX 2080 Ti worker
  • RTX 4090 worker

The crash occurs with the identical stack trace.

2. SSD cache

We tested all three scenarios:

  • cold SSD cache
  • fully populated SSD cache
  • SSD cache completely disabled (FixedResourceSlots(SSD=False) )

The crash still occurs with the same stack trace.

3. Different job type

We also observed the same native crash in an unrelated Live Processing worker (rtp_worker ).

In that case the stack reaches the I/O engine through:

blobio/prefetch
live_worker/rtp_exposure_cache

instead of Local CTF Refinement, but ultimately crashes in the same functions:

get_npy_dtype_from_str()
wrap_array_from_iocmd()

All our workers are on the latest RockyLinux 9. Any idea?

Thanks

2

Thanks @biocit for this information.
Please can you email us job reports for an example failed job of type local ctf refinement and live processing worker.

3

Email is sent. Thanks

4

Hi @biocit ,

I believe we’ve finally figured out what’s going on here. There’s a bug in the same error handling code path that produces the WARNING: io_uring support disabled (supported by kernel, but initialization failed), I/O performance may degrade message. That situation doesn’t occur on the systems that we have access to, so it was difficult to identify the exact problem.

In an upcoming release we will fix the bug so that this won’t crash the job and will instead fall back to the no-io_uring path, as was intended. For now, if this issue is blocking you, you can disable io_uring via the CRYOSPARC_IO_URING environment variable ( Environment Variables (v5.0+) | CryoSPARC Guide )

–Harris

5.0.6: 2d classification crash
5.0.6: ab-initio reconstruction loop
5

Hi Harris

Thanks. I implemented the hotfix on one lane. If it still crashes i will report back.

Best

6

So far no complains. Seams to work.