FS#31187 - [linux] - disable NUMA from config files
Attached to Project:
Arch Linux
Opened by John (graysky) - Sunday, 19 August 2012, 02:42 GMT
Last edited by Tobias Powalowski (tpowa) - Friday, 12 October 2012, 10:52 GMT
Opened by John (graysky) - Sunday, 19 August 2012, 02:42 GMT
Last edited by Tobias Powalowski (tpowa) - Friday, 12 October 2012, 10:52 GMT
|
Details
Disabling NUMA is beneficial for desktop systems. I offer
attached results of a gcc/make-based benchmark whose code is
referenced at the bottom of this request on github. Here you
see the mainline "3.5.2-1-ARCH" vs. two different BFS
patched kernels. One has NUMA enabled per the Arch Linux
defaults and the other has it disabled:
http://s19.postimage.org/a8mk5gxgz/3770k.jpg There is a clear and statistically significant difference in compile times (n=28) with the median gain through disabling NUMA being 344 ms. From my research, unless the hardware has >1 PHYSICAL CPU -- not cores but physical processors -- it is advantageous to disable NUMA as measured by this non-latency endpoint. Since the majority of Arch users run workstation/laptop setups, I would ask that the default configs disable NUMA. Link to benchmark script: https://github.com/graysky2/bin/blob/master/bench |
This task depends upon
Closed by Tobias Powalowski (tpowa)
Friday, 12 October 2012, 10:52 GMT
Reason for closing: Won't implement
Additional comments about closing: 1 second is not worth disabling it
Friday, 12 October 2012, 10:52 GMT
Reason for closing: Won't implement
Additional comments about closing: 1 second is not worth disabling it
Note that every other major distro enables NUMA for x86_64.
Although you are right that other major distros do enable NUMA in their vanilla configs, this is NOT a reason to do it for Arch Linux. I contend that the overwhelming super majority of Arch users have only 1 physical CPU (again workstation and laptop users) and thus are impacted detrimentally by this option. In other words, if 99.99999 % of ARCH users do not have >1 CPU, why in the world would we enable the option in the kernel package that benefits the 0.00001 % of users that do? Just because our peer groups do it is no justification that it is a data-driven and sound decision.
I am tired and am going to bed now, but will compile via ABS the 3.5.2-1-ARCH linux package tomorrow morning repeating this study without any non-standard patches to address point.
1. wider testing across more hardware than just an Ivy Bridge processor.
2. wider testing across workloads beyond just Kbuild.
> Although you are right that other major distros do enable NUMA in their vanilla configs, this is NOT a reason to do it for Arch Linux.
There's a lot of things we could potentially disable/enable in our default config to make it a faster kernel. I'm hard pressed to believe that, given enough evidence that this is a good idea™, it's going to make a bigger difference than disabling a lot of the debug options we've recently enabled. The hivemind of the _desktop_ distros clearly has this enabled for a reason.
http://s19.postimage.org/gp566f1pv/8_19_2012_2_06_53_AM.jpg
To your points raised in your new post:
1) I have repeated this results using the -ck kernel on a core2 processor (X3360) with the rank order being the same. More testing on older hardware or on hardware with less cores could be done, but my suspicion is that there would be no surprises given that all represent single CPU machines.
2) Agreed that more workloads would be interesting, but given what this particular option does per the manpage (http://www.kernel.org/doc/man-pages/online/pages/man5/numa_maps.5.html), I don't see the need to identify and run these to prove the point.
I can't comment on the debugging options but with respect to the NUMA option, I feel that it enabling it on mono CPU systems is detrimental based on these results. Thank you for your consideration.
3) I respectfully disagree. If all major distros do something because their peer group did it, how is this a reason to justify doing it for Arch -- particularly considering the data showing that it is a performance regression for those users with only one physical CPU?
"...the overwhelming super majority of Arch users have only 1 physical CPU (again workstation and laptop users) and thus are impacted detrimentally by this option. In other words, if 99.99999 % of ARCH users do not have >1 CPU, why in the world would we enable the option in the kernel package that benefits the 0.00001 % of users that do? Just because our peer groups do it is no justification that it is a data-driven and sound decision."
We're talking about a 0.2% speedup by disabling NUMA here. Is that a valid reason to disable NUMA support completely? For me it is not.
Have you benchmarked performance with numa=off?
Nope, you can't do that.
Conditions:
1) Stock kernel
2) Stock kernel booting with numa=off
3) Compiled stock kernel from ABS with NUMA disabled prior to compiling. For example `% zgrep NUMA /proc/config.gz` returned: '# CONFIG_NUMA is not set' on the running kernel. This was the only modification I did to it.
Results for single socket machine:
*Booting with 'numa=off' gave the longest compile times.
*The stock kernel gave longer compile times than the same kernel compiled with NUMA disabled in the .config. Confirming my initial report.
*The fastest compile times were observed with NUMA disabled in the .config prior to building.
Statistical significance on all three tests.
Machine 1 = 3770K @ 4.5 GHz
Total CPUs = 1
Physical cores = 4 and hyperthreaded cores = 4
Total cores (physical+virtual) = 8
http://s19.postimage.org/x5qgb1gqb/3770_ARCH.jpg
Results for dual socket machine:
*Booting with 'numa=off' gave the longest compile times.
*The same kernel compiled with NUMA disabled in the .config gave longer compile times than the stock kernel.
*The fastest compile times were observed with the stock kernel.
Statistical significance on all three tests.
Machine 2 = Dual 5620 @ 2.4 GHz
Total CPUs = 2
Physical cores = 8 and hyperthreaded cores = 8
Total cores (physical+virtual) = 8
http://s19.postimage.org/mwxz57soj/dual_5620.jpg
So there you have it! I still subscribe to the thinking I outlined above, but understand that I am not a developer. Thanks for the consideration.