Hi,
I am currently having a problem with CryoSPARC v5.0.6 on a workstation with two NVIDIA RTX PRO 4000 Blackwell GPUs.
The problem mainly happens during 2D Classification and Ab-initio Reconstruction. Other jobs, especially Non-uniform Refinement, seem to run without problems.
The CryoSPARC jobs fail with errors like:
CUDA_ERROR_LAUNCH_TIMEOUT
For example:
RuntimeError: cuStreamSynchronize(...):
CUDA ERROR: (CUDA_ERROR_LAUNCH_TIMEOUT)
the launch timed out and was terminated
I have also seen:
numba.cuda.cudadrv.driver.CudaAPIError:
[702] Call to cuMemcpyHtoDAsync results in CUDA_ERROR_LAUNCH_TIMEOUT
and:
[702] Call to cuMemGetInfo results in CUDA_ERROR_LAUNCH_TIMEOUT
At the same time, the Linux kernel reports an NVIDIA Xid 8 error:
NVRM: krcWatchdog_IMPL: RC watchdog: GPU is probably locked! Notify Timeout Seconds: 7
NVRM: GPU at PCI:0000:01:00
NVRM: Xid (PCI:0000:01:00): 8, pid=5937, name=python, channel 0x0000000e
What makes this a bit strange is that it happens on both GPUs, not only one card.
From the kernel logs I found several Xid 8 events:
GPU 0 / PCI 01:00.0:
2026-08-04 Xid 8
2026-08-04 Xid 8
2026-08-06 Xid 8
GPU 1 / PCI 61:00.0:
2026-08-10 Xid 8
2026-08-16 Xid 8
Our system:
Ubuntu 24.04.4 LTS
Kernel: 6.17.0-1032-oem
CryoSPARC: v5.0.6
2 × NVIDIA RTX PRO 4000 Blackwell
24 GB VRAM each
595.84 open
CRYOSPARC_NO_PAGELOCK=true
THP: always [madvise] never
I first thought this could be a driver problem, so I updated from NVIDIA 580.173.02 to 595.84.
Unfortunately, the same problem still happens:
580.173.02 -> Xid 8 / CUDA_ERROR_LAUNCH_TIMEOUT
595.84 -> Xid 8 / CUDA_ERROR_LAUNCH_TIMEOUT
I also repeated one of the problematic 2D Classification jobs after the driver update.
The job was running only on GPU 0:
GPU=[0]
MAIN PROCESS PID 5937
Just before the crash, the CryoSPARC log showed:
DIE: cuMemsetD32Async(...):
CUDA ERROR: (CUDA_ERROR_LAUNCH_TIMEOUT)
the launch timed out and was terminated
Received SIGSEGV
At exactly the same time, the kernel reported:
NVRM: RC watchdog: GPU is probably locked!
NVRM: Xid (PCI:0000:01:00): 8, pid=5937, name=python
So the Xid is coming from the same CryoSPARC Python process.
I also checked that the CRYOSPARC_NO_PAGELOCK=true setting is actually being used. The job log contains:
HOST ALLOCATION FUNCTION: using n.empty
(CRYOSPARC_NO_PAGELOCK==true)
I do not see obvious signs of a hardware problem. After the crash there were no PCIe replay errors, no thermal throttling and no memory remapping errors. The GPU was also not close to running out of VRAM. We are also running other GPU heavy applications on that workstation without any issues.
I would be happy if anyone has an idea how to fix it.
I can also provide the full CryoSPARC job log and NVIDIA bug report if that would help.
Thanks!