FS#69333 - [kodi] build with -DUSE_LTO=$(nproc)

Attached to Project: Community Packages
Opened by John (graysky) - Friday, 15 January 2021, 09:51 GMT
Last edited by Ike Devolder (BlackEagle) - Saturday, 27 February 2021, 14:13 GMT
Task Type Feature Request
Category Packages
Status Closed
Assigned To Ike Devolder (BlackEagle)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Building with this switch reduced installed size reported by pacman vs the identical commit (kodi-x11 from master not version 18.9 but should give similar results) built without native LTO enabled is approx 10% smaller:

Total Installed Size: 81.43 MiB
Net Upgrade Size: -8.50 MiB

In addition to the size savings, in general, there known gains in efficiency and speed in the executables/DSOs compiled with LTO. See the following and some of the links therein: https://fedoraproject.org/wiki/LTOByDefault
This task depends upon

Closed by  Ike Devolder (BlackEagle)
Saturday, 27 February 2021, 14:13 GMT
Reason for closing:  Implemented
Additional comments about closing:  implemented in upcoming kodi 19
Comment by Doug Newgard (Scimmia) - Friday, 15 January 2021, 12:51 GMT
What, exactly, does $(nproc) do here? On the surface, it sounds like a terrible idea to use that for a binary package.
Comment by Eli Schwartz (eschwartz) - Friday, 15 January 2021, 13:18 GMT
makepkg shall only increase the parallelism of of builds via makepkg.conf, where coincidentally you may set CFLAGS/LDFLAGS including LTO. The same logic here applies equally to all packages.

I would guess that LTO does not care how many processes you use to do it... But if it did care then this is completely unacceptable as we would automatically have to reject any suggestion that makes the Reproducible Builds status of a package dependent on the number of cores possessed by the build server.
Comment by John (graysky) - Friday, 15 January 2021, 15:22 GMT
I was thinking the $(nproc) piece would allow the build system to share the number of cores it has when building to minimize build time. Upstream cmake can also use: -DUSE_LTO=ON
Comment by Eli Schwartz (eschwartz) - Friday, 15 January 2021, 15:30 GMT
There is nothing upstream cmake about this. USE_LTO is custom code in https://github.com/xbmc/xbmc/blob/master/cmake/scripts/linux/ArchSetup.cmake#L50

Which does nothing other than add -flto=2 (or -flto=$(nproc) with your suggestion) and -fno-fat-lto-objects to C/XX FLAGS.

Again, if this is so great then the best place to do it is in makepkg.conf rather than cherry-picking one application to build with LTO using custom code.
Comment by John (graysky) - Friday, 15 January 2021, 18:26 GMT
@eschwartz - "Upstream" in my comment was confusing. I did not mean cmake, I meant Kodi as "upstream." Sorry for the confusion there.

Note that I tried building with -flto in my makepkg.conf[1] in a clean chroot, but that ended in errors[2]. Enabling lto via the cmake switch builds successfully but only with the distro default CFLAGS.

1. CFLAGS="-march=x86-64 -O2 -pipe -fno-plt -flto" ; CXXFLAGS="${CFLAGS}"
2. https://gist.github.com/graysky2/951bb3620c8a42782dd238c75ba99dd3
Comment by John (graysky) - Friday, 15 January 2021, 19:25 GMT
As a test, I tried build without the -DUSE_LTO=ON switch but instead with both of the switches it sets and got the same errors on linking as I reported in my last reply.

CFLAGS+=" -flto=8 -fno-fat-lto-objects"
Comment by Eli Schwartz (eschwartz) - Friday, 15 January 2021, 19:39 GMT
-flto should be set in LDFLAGS too...
Comment by John (graysky) - Friday, 15 January 2021, 19:49 GMT
@eschwartz - I can try that but the code you referenced (ArchSetup.cmake) only seems to modify CFLAGS/CXXFLAGS.
Comment by Eli Schwartz (eschwartz) - Friday, 15 January 2021, 20:01 GMT
Well, LTO requires passing it to the linker too... and the cmake file in question also defines CMAKE_INTERPROCEDURAL_OPTIMIZATION so I presume that makes cmake internally set up the linker flags.
Comment by John (graysky) - Friday, 15 January 2021, 20:13 GMT
Same linking error with:

CFLAGS="-march=x86-64 -O2 -pipe -fno-plt -fdiagnostics-color -flto=8 -fno-fat-lto-objects"
CXXFLAGS="${CFLAGS}"
LDFLAGS="-Wl,-O1,--sort-common,--as-needed,-z,relro,-z,now -flto=8 -fno-fat-lto-objects"

So the internal USE_LTO does something special.
Comment by loqs (loqs) - Friday, 15 January 2021, 20:23 GMT
Try make VERBOSE=1 or pass -DCMAKE_VERBOSE_MAKEFILE:BOOL=ON to cmake to enable verbose output from make to see what options are passed to gcc e.t.c..
Comment by John (graysky) - Friday, 15 January 2021, 23:43 GMT
@loqs - Interestingly, when I grepped for '-flto' in the build log with -DVERBOSE=ON, I did not see anything. Log: https://gist.github.com/graysky2/a685b916618f7c13b89c3804b9acd271
Comment by loqs (loqs) - Saturday, 16 January 2021, 00:58 GMT
Strange works here. The issue with adding -flto to *FLAGS I believe is that gets passed to the internal ffmpeg build which fails.
Using DUSE_LTO:BOOL=ON only sets the options for the kodi build itself. You should be able to trigger ffmpeg failing with similar output by adding -flto to *FLAGS.
Edit:
With ffmpeg change --enable-lto to --disable-lto add -flto to * FLAGS.

Poor examples showing
PKGBUILD1.diff enable LTO in ffmpeg's configure
PKGBUILD2.diff disable LTO when build ffmpeg
Edit2:
ffmpeg's configure script disables inline_asm_direct_symbol_refs when using the lto option but not when -flto is used in FLAGS used, passing --disable-x86asm to configure allows building with the -flto in FLAGS.
Comment by loqs (loqs) - Tuesday, 02 February 2021, 23:23 GMT
LTO.patch passes --enable-lto to ffmpeg's configure script if kodi's CMAKE has CMAKE_INTERPROCEDURAL_OPTIMIZATION set which would be from using -DUSE_LTO=ON.

Not figured out how to fix ffmpeg's configure script to disable inline_asm_direct_symbol_refs when -flto is used in FLAGS.
Comment by Ike Devolder (BlackEagle) - Saturday, 27 February 2021, 14:13 GMT
After looking around I have enabled -DUSE_LTO=ON in kodi 19, the ffmpeg part is left default and "it works on my machine ™"

Loading...