FS#59413 - [devtools] - Building with makechrootpkg doesn't use distcc correctly

Attached to Project: Arch Linux
Opened by John (graysky) - Saturday, 21 July 2018, 15:21 GMT
Last edited by Andreas Radke (AndyRTR) - Tuesday, 04 May 2021, 12:38 GMT
Task Type Bug Report
Category Arch Projects
Status Closed
Assigned To Pierre Schmitz (Pierre)
Eli Schwartz (eschwartz)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

In the past, I have been able to build packages in a clean chroot using distcc, but recently (past month or so), the build from within the root fails to distribute out the other distcc nodes at the same rate as before. I based this statement on the build time for the same package being much longer than before. For example:

Building the linux kernel (just make bzImage) in the build function:
* Running makepkg on the localhost only (no distccd) takes 3m 25s
* Running makechrootpkg setup to build on the localhost only (no distccd) takes 3m 22s
Conclusion: both give the same result

* Running makepkg configured to use distccd takes 2m 19s
* Running makechrootpkg configured to use distccd (as described below) takes 14m 52s
Conclusion: something is wrong

Steps to reproduce:
1) Have distccd started and configured to run properly on the host OS and on other nodes on the network.
2) Run: mkarchroot ~/.chroot64/root base-devel distcc
3) Edit ~/.chroot64/root/etc/makepkg.conf to a) enable distcc in BUILDENV array b) adjust MAKEFLAGS and c) define DISTCC_HOSTS
4) Try to build for example the linux package: makechrootpkg -c -r ~/.chroot64
This task depends upon

Closed by  Andreas Radke (AndyRTR)
Tuesday, 04 May 2021, 12:38 GMT
Reason for closing:  None
Additional comments about closing:  see last comment
Comment by Eli Schwartz (eschwartz) - Sunday, 22 July 2018, 04:15 GMT
Uh, can we have something a little more concrete than "it feels wrong"?

1) Are you sure the makepkg.conf is -- still -- properly configured?
2) What does g/cc actually resolve to inside the chroot? This is trivially confirmed by running 'type -a gcc' in the PKGBUILD build function.
3) Set distcc's own $DISTCC_VERBOSE environment variable to like actually get information about its running.
4) Try to provide some vaguely minimal form of useful information in your bug reports other than "it doesn't work".
Comment by John (graysky) - Sunday, 22 July 2018, 09:13 GMT
Thanks for the reply and feedback, eschwartz.

1) Yes, I am sure. The attached is ~/.chroot64/facade/etc/makepkg.conf (filename=makepkg.conf)
2) Adding `type -a gcc` into the PKGBUILD returns the following and run from the makechrootpkg command:
gcc is /usr/lib/distcc/bin/gcc
gcc is /usr/bin/gcc

For reference, I also am attaching the PKGBUILD I am using to reproduce the issue (filename=PKGBUILD)

3) I edited the server's /etc/conf.d/distccd to contain: DISTCC_ARGS="--allow 10.9.8.0/24 --log-file /tmp/distccd.log --verbose" and also added `export DISTCC_VERBOSE=1` in the PKGBUILD itself. When running the build, I am seeing tons of messages about distcc lock files being busy with the verbose switch. Find the output attached (filename=linux-4.17.8-1-x86_64-build.log.xz)
As well, I added the server log which interestingly doesn't contain any info beyond the initial entries. (filename=distccd.log)
4) Without the verbose switches you recommended, I found nothing diagnostic in dmesg/journalctl or in distccd.log, so the best I could do is report the differences in compile time. Thanks again for the direction and helpful suggestions.
Comment by John (graysky) - Sunday, 22 July 2018, 20:42 GMT
Is there anything the devtool scripts would be setting up that would block or redirect calls to the localhost distccd?
Comment by Eli Schwartz (eschwartz) - Monday, 23 July 2018, 05:19 GMT
I cannot imagine so, it just runs systemd-nspawn with some directories bind-mounted plus --register=no --keep-unit --as-pid2

The interesting thing is distcc in the makepkg execution is reporting:
distcc[4501] (dcc_parse_hosts) found localhost token "localhost/9"
distcc[4501] (dcc_parse_hosts) found tcp token "waste/9"
distcc[4501] (dcc_parse_hosts) found tcp token "relic/5"
distcc[4501] (dcc_parse_hosts) found tcp token "myth2/5"

Which indicates it correctly detected your build slaves, or at least that's what I assume a token is.

And then there *are* lines like:

distcc[5411] exec on waste/9: gcc -Wall -Wundef -Wstrict-prototypes -Wno-trigraphs -fno-strict-aliasing -fno-common -fshort-wchar -Werror-implicit-function-declaration -Wno-format-security -std=gnu89 -fno-PIE -mno-sse -mno-mmx -mno-sse2 -mno-3dnow -mno-avx -m64 -falign-jumps=1 -falign-loops=1 -mno-80387 -mno-fp-ret-in-387 -mpreferred-stack-boundary=3 -mskip-rax-setup -mtune=generic -mno-red-zone -mcmodel=kernel -funit-at-a-time -pipe -Wno-sign-compare -fno-asynchronous-unwind-tables -mindirect-branch=thunk-extern -mindirect-branch-register -fno-delete-null-pointer-checks -Wno-frame-address -Wno-format-truncation -Wno-format-overflow -Wno-int-in-bool-context -O2 --param=allow-store-data-races=0 -Wframe-larger-than=2048 -fstack-protector-strong -Wno-unused-but-set-variable -Wno-unused-const-variable -fno-var-tracking-assignments -pg -mfentry -Wdeclaration-after-statement -Wno-pointer-sign -fno-strict-overflow -fno-merge-all-constants -fmerge-constants -fno-stack-check -fconserve-stack -Werror=implicit-int -Werror=strict-prototypes -Werror=date-time -Werror=incompatible-pointer-types -Werror=designated-init -fmacro-prefix-map=./= -Wno-packed-not-aligned -c -o arch/x86/platform/efi/.tmp_quirks.o arch/x86/platform/efi/quirks.c

But they're outnumbered by the ones to localhost...
Comment by John (graysky) - Monday, 23 July 2018, 16:18 GMT
@eschwartz - Agreed. It's definitely something in the chroot. I can reproduce it on any node on the network: 1) building outside of the chroot works as expected and 2) building in the chroot gives tons of "(dcc_lock_host) /build/.distcc/lock/cpu_xxx is busy" lines and 5-6x slow down in compile times. What ever is causing it, it is somewhat recent (ie past 4-6 weeks). I have been building in the chroot with distcc for years without issue.
Comment by John (graysky) - Tuesday, 04 September 2018, 19:55 GMT
I think I tracked this issue to down to the fact that hostname resolution is broken from within the chroot. If I define the DISTCC_HOSTS variable using IP addresses rather than hostnames, distcc works as expected. Can someone else confirm this to rule out something funny my router is doing?
Comment by John (graysky) - Saturday, 06 March 2021, 11:05 GMT
Probably safe to close. Solution to building linux kernel in a clean chroot is to add the following line to the PKGBUILD before building:

sed -i '/HAVE_GCC_PLUGINS/d' arch/x86/Kconfig

Loading...