Hi all,
The latest performance boost in v4.6 on classification and refinement jobs, seems to relate to particles on SSD cache and the use of the io_uring kernel module.
Although I have updated kernel to 5.4 and kernel config says “CONFIG_IO_URING=y”, I still get the new v4.6.1 warning “io_uring support disabled (not supported by kernel), I/O performance may degrade”.
What else is needed to enable io_uring usage in CS jobs?
I have the same issue:
uname -r
5.14.0-427.37.1.el9_4.x86_64
grep CONFIG_IO_URING /boot/config-$(uname -r)
CONFIG_IO_URING=y
But cryosparc complains. I am using RockyLinux 9.
Next question could be:
Why is a optional optimization setting show as warning when not used?
io_uring is a bit controversial - security wise:
https://en.wikipedia.org/wiki/Io_uring
In June 2023, Google’s security team reported that 60% of the exploits submitted to their bug bounty program in 2022 were exploits of the Linux kernel’s io_uring vulnerabilities. As a result, io_uring was disabled for apps in Android, and disabled entirely in ChromeOS as well as Google servers.[11] Docker also consequently disabled io_uring from their default seccomp profile.[12]
Maybe a warning should be shown when io_uring is used?
I agree it should not be shown as a warning to not use this setting
I came here to report the same issue. Non-uniform Refinement gave the following error…
“[CPU: 89.1 MB Avail: 246.56 GB] WARNING: io_uring support disabled (not supported by kernel), I/O performance may degrade”
My workstations is on the Centos Stream 9 and the kernel is 5.14.0-529.el9.x86_64
As per bioct’s comment, I performed thsi command…
grep CONFIG_IO_URING /boot/config-$(uname -r)
CONFIG_IO_URING=y
Hi @jelka, @biocit, @stefanoliver,
On a given system, io_uring may be disabled at runtime, even if it is supported by the kernel, via sysctl. I’d recommend checking if this is the case on your system. See this link for details: io - io_uring with `fio` fails on Rocky 9.3 w/kernel 5.14.0-362.18.1.el9_3.x86_64 - Unix & Linux Stack Exchange
Regarding the security aspects of io_uring - thank you for raising the question. While io_uring (and many other kernel/software subcomponents) can be a security issue for applications that are hosted on the internet or are accessed by untrusted users, we operate under the requirement that CryoSPARC should always be deployed within a trusted environment, with access and authentication provided through external systems (e.g. VPN, reverse proxy, etc). This requirement is unchanged regardless of whether io_uring is used.
– Harris
I activated now on our systems. To make it persistent i did:
sudo nano /etc/sysctl.d/io_uring.conf
adding:
kernel.io_uring_disabled=0
save it. Then
sudo sysctl --system
Thanks. This did the trick.
Okay thanks @hsnyder
Although I still run Rocky Linux 8 (w. elrepo kernel-lt 5.4) and here io_uring does not want to be controlled through sysctl.
io_uring seems enabled nevertheless:
grep io_uring_setup /proc/kallsyms
ffffffffb27575f0 t io_uring_setup
ffffffffb27576b0 T __x64_sys_io_uring_setup
ffffffffb27576d0 T __ia32_sys_io_uring_setup
ffffffffb3b007e0 d event_exit__io_uring_setup
ffffffffb3b00880 d event_enter__io_uring_setup
ffffffffb3b00920 d __syscall_meta__io_uring_setup
ffffffffb3b00960 d args__io_uring_setup
ffffffffb3b00970 d types__io_uring_setup
ffffffffb41f18b8 t __event_exit__io_uring_setup
ffffffffb41f18c0 t __event_enter__io_uring_setup
ffffffffb41f36a8 t __p_syscall_meta__io_uring_setup
ffffffffb41f6f20 t _eil_addr___ia32_sys_io_uring_setup
ffffffffb41f6f30 t _eil_addr___x64_sys_io_uring_setup
Even original Rocky Linux 8 w. kernel-4.18 seems to have some io_uring features backported:
grep io_uring_setup /proc/kallsyms
ffffffff985214f0 W __ia32_sys_io_uring_setup
ffffffff985214f0 W __x64_sys_io_uring_setup
I am just wondering how does cryoSPARC worker detect if io_uring is enabled or not?
Best,
Jesper
To detect io_uring support, we call io_uring_get_probe() from liburing. This allows us to check what io_uring features are supported by the running kernel. It appears that some old kernels (the first few that supported io_uring) don’t actually support this kind of probing. So it is possible that your kernel has early io_uring support, but doesn’t being interrogated in this way. If that’s the case, then the only workaround at this time would be to update your kernel.
–Harris
Dear all,
I am using cryoparc v 4.7.1.
I have the “WARNING: io_uring support disabled (not supported by kernel), I/O performance may degrade”, and tried to Biocit’s tricks “[biocit, post:7, topic:15564”] kernel.io_uring_disabled=0
[/quote], but did not work out.
By the way, does this warning really affect the refinement performance if I do not use SSD?
Thank you for your time and advise.
Lan
What’s the output of uname -a? io_uring has been supported since kernel 2.6 (at least with async support)…
Not supported by kernel implies you’re running a really old install, have a kernel compiled without io_uring support, or it’s been disabled by boot flags.
Hi rbs_sic:
Thank you for suggestions.
here is the output of uname -a
Linux r16763 4.18.0-372.13.1.el8_6.x86_64 #1 SMP Wed Jun 29 17:21:09 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux
kernel 4.18 was installed 2 years ago when the cryosparc updated to require a updated NVIDIA dirver.
what should I do the next ?
Lan
If you really want io_uring, you’ll need to update to a kernel which supports it. I’m not familiar with RedHat kernel naming schemes (after some bad experiences I’ve always avoided RHEL and it’s downstream distros) but if the reporting is correct, you’re running a three year old kernel! I’d really recommend a major update.
I use the Rocky 8.4.
By the way, I am thinking of temporarily downgrading CryoSPARC. Which version do you recommend?
Thanks,
Lan
I wouldn’t recommend downgrading CryoSPARC unless absolutely necessary. Earliest I would go back to would probably be 4.4… some versions (I forget which) change the database version and so are incompatible.
Rocky 8.4 has been unsupported for a long time. Update to 8.10 or the latest 9.x release urgently.
Hi rbs,
A correction, my rocky is 8.9. Lan
which version of kernel you suggest? Lan
HI res_sci,
Another change I made was to store the processed CS jobs on an external hard drive due to limited internal space. Could this be the major problem for the very slow processing?
Thanks,
Lan
Well, yes. External drives (I presume USB as Firewire and eSATA have effectively died out) have a lot of limitations. It’s difficult to break 90MB/s via USB HDDs (sustained) and USB has significantly higher latency than SATA.