I’m currently in the process of replacing Rocky 8 by Ubuntu 26.04 LTS on all our Linux computers. To my surprise, I saw your recent warning “As of June 2026, CryoSPARC versions up to 5.0.6 are incompatible with the version 7 kernel that is the default for Ubuntu 26.04.” on your operating system requirement site. Indeed, on the first compute node with Ubuntu 26.04 LTS, a 2D classification job in CryoSPARC version 5.0.6 died with “Job is unresponsive - no heartbeat received in 180 seconds”. These are bad news for me.
Can you estimate, how soon a CryoSPARC update will be released, that will be compatible with the 7.x kernel in Ubuntu 26.04 LTS or any other new Linux distribution?
We decided to modernize our Linux installations and do the switch from RHEL clones, like Rocky, to Ubuntu LTS or Debian stable. On one of our servers, I will test the current Debian stable “Trixie” which comes with the 6.12 LTS kernel. However, since kernel “7” was an arbitrary naming decision by Linus Torvalds, it’s also not clear to me with which 6.x kernel version CryoSPARC starts having problems.
An “older” Ubuntu (24.04) is also happily using kernel 7.0.0 with CryoSPARC, so I don’t think it’s actually kernel related.
@wtempel: On one system, I’m running Linux Mint based on Ubuntu 24.04 with kernel 7.0.0-14-generic, the system has been up for nearly 40 days without a reboot on that kernel (so, June 3rd was update/reboot day) and have a full processing run from Import to NU refine (about to start RBMC) of a large dataset which has been 100% happy.
This was the system which was a pain a while ago with nvidia-595 and Blackwell GPUs, if you remember that thread. With 580.159, it’s been solid as a rock again.
In my experimentation with 26.04, I’ve found a few of my scripts really don’t like the new Rust-based uutils core utils.
If there are any logs I could send you to help, please let me know.
Thanks for the feedback @dirk@rbs_sci . In cases where you did (or did not) observe problems with a v7 kernel, where CryoSPARC jobs queued to node directly using the CryoSPARC scheduler, or through an external workload manager, like slurm?
They’re not cryo-EM related. I got caught by two of the well publicised issues with uutils (dd causing corrupt backups, ln screwing up non-UTF-8 characters).
Meanwhile I ran a couple of tests on three different servers with the tasks 2D Classification, Ab-Initio Reconstruction, Heterogenous Refinement of all classes, Homogenous Refinement of the best class.
The three servers have the following configuration:
To my surprise, none of my test jobs showed any problems! Even server No. 3, which failed in a 2D Classification of a colleague with “… no heartbeat received …” (see above), finished all jobs without any error. I am puzzled.
I will now put these servers back into our regular SLURM queues and watch out for any problems by our users, which I will report here.
Scant evidence suggests that multiple simultaneous jobs on the same host could trigger the problem for some, but not all, of the simultaneously running jobs.
Almost half of our nodes run Ubuntu (one Debian as a control), the other half run Rocky 8. We are still searching for a pattern of failure/success. However, we see now Ab-Initio jobs that reproducibly fail on nodes running Ubuntu 26.04 LTS, and their cloned jobs succeed on nodes running Rocky 8. This happens both on older hardware and on more recent hardware. The failed jobs are stalled at -/0 particles and are usually killed after several hours.
Our nodes running Rocky 8 all have kernel 4.18 (but Red Hat patches its kernels a lot) and the same Nvidia driver 575.57 installed, the nodes running Ubuntu 26.04 LTS all have kernel 7.0 and Nvidia drivers installed that still support the graphics cards (580.173 for “Pascal” generation, 595.71 for newer generation).
We have one example of cloned Ab-Initio jobs that failed/succeeded on the same hardware with 4x GTX1080 and 256 GB RAM, running Ubuntu 26.04 LTS/Rocky 8. The failed job was killed after 17 hours stalled at 0 particles, the succeeded job finished in less than 2 hours with 250000 particles. On both nodes, the job was the only one running (SLURM job scheduler).
The kernel log on the node with the failed job did not show any OOM or CUDA messages or any other suspicious messages (as far as I can see).
Here is a “cryosparcm job events … | tail -50” for the failed job:
[2026-07-22 14:31:13] [836 MB] Using random seed for initialization of 2113629888
[2026-07-22 14:31:13] [836 MB] Generating random initial densities.
[2026-07-22 14:31:13] [836 MB] Generating random initial density for class 0
[2026-07-22 14:31:16] [1080 MB] Generating random initial density for class 1
[2026-07-22 14:31:16] [1083 MB] Generating random initial density for class 2
[2026-07-22 14:31:17] [1087 MB] Generating random initial density for class 3
[2026-07-22 14:31:17] [1088 MB] Generating random initial density for class 4
[2026-07-22 14:31:17] [1094 MB] Done in 4.288s.
[2026-07-22 14:31:18] [FIGURE] Generated initialization!
[asset file=“J179_generated_initialization.png”
id=“6a60d4355d782fcad0c3e78b”]
[asset file=“J179_generated_initialization.pdf”
id=“6a60d4365d782fcad0c3e78d”]
[2026-07-22 14:31:18] [1100 MB] Applying spherical window to 237.29A diameter (falloff
to 276.38A)
[2026-07-22 14:31:18] [1100 MB] ( Radius 0.85 to 0.99 )
2026-07-22 14:31:18
[2026-07-22 14:31:20] [FIGURE] Structure for Class 000 Iteration 0
[asset file=“J179_structure_for_class_000_iteration_0.png”
id=“6a60d4385d782fcad0c3e7a9”]
[asset file=“J179_structure_for_class_000_iteration_0.pdf”
id=“6a60d4385d782fcad0c3e7ab”]
[2026-07-22 14:31:21] [FIGURE] Structure for Class 001 Iteration 0
[asset file=“J179_structure_for_class_001_iteration_0.png”
id=“6a60d4385d782fcad0c3e7ae”]
[asset file=“J179_structure_for_class_001_iteration_0.pdf”
id=“6a60d4395d782fcad0c3e7b0”]
[2026-07-22 14:31:21] [FIGURE] Structure for Class 002 Iteration 0
[asset file=“J179_structure_for_class_002_iteration_0.png”
id=“6a60d4395d782fcad0c3e7b3”]
[asset file=“J179_structure_for_class_002_iteration_0.pdf”
id=“6a60d4395d782fcad0c3e7b5”]
[2026-07-22 14:31:21] [FIGURE] Structure for Class 003 Iteration 0
[asset file=“J179_structure_for_class_003_iteration_0.png”
id=“6a60d4395d782fcad0c3e7b8”]
[asset file=“J179_structure_for_class_003_iteration_0.pdf”
id=“6a60d4395d782fcad0c3e7ba”]
[2026-07-22 14:31:22] [FIGURE] Structure for Class 004 Iteration 0
[asset file=“J179_structure_for_class_004_iteration_0.png”
id=“6a60d43a5d782fcad0c3e7bd”]
[asset file=“J179_structure_for_class_004_iteration_0.pdf”
id=“6a60d43a5d782fcad0c3e7bf”]
[2026-07-22 14:31:22] [1151 MB] ----------- Iteration 0 (epoch 0.000). radwn 10.00
resolution 27.92A minisize 90 beta 0.10
[2026-07-22 14:31:23] [1195 MB] Estimating noise model to meet target of 98.17 poses..
Current ESS R is 1785.79 poses..
Current sigma is 0.10
New sigma is 0.10
[2026-07-23 07:28:10] [254 MB] **** Kill signal sent by Katja (ID:
5c08e7c94f423c0b61e82f18) ****
If I can help any further to shed some light on this issue, please, tell me!
Thanks for sharing these observations. For the failed, ubuntu-26.04 job whose event log you posted, please can you also post the outputs of these commands
project_uid=P99 # replace with actual ID
job_uid=J199 # replace with actual ID
cryosparcm job log "$project_uid" "$job_uid" | grep -e ^Allocated -e ^HOST -e ^Transparent
cryosparcm job log "$project_uid" "$job_uid" | tail -n 20
Any news on this issue? The updating guide now says “As of June 2026, CryoSPARC versions up to 5.0.6 are incompatible with the version 7 kernel that is the default for Ubuntu 26.04.” - does that mean, version 5.0.7 is now compatible with kernel 7.x and Ubuntu 26.04? I couldn’t find anything regarding the kernel version or Ubuntu version in the release notes of version 5.0.7.