Arch Linux

Please read this before reporting a bug:

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!

FS#70954 - [gcc] [toolchain] build order / reproducibility issue

Attached to Project: Arch Linux
Opened by Toolybird (Toolybird) - Thursday, 20 May 2021, 02:32 GMT
Last edited by Allan McRae (Allan) - Thursday, 01 July 2021, 01:08 GMT
Task Type General Gripe
Category Packages: Core
Status Assigned
Assigned To Giancarlo Razzolini (grazzolini)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No


(I'm sure "the powers that be" are fully aware of any issues raised here, but it still might be good to have this documented somewhere. Feel free to shut this down if not appropriate for the bug tracker.)

Whenever Arch upgrades the toolchain with a major new GCC version (e.g. GCC-10 -> GCC11), a situation arises where the toolchain becomes (kind of) unreproducible. This is evidenced by (as of this writing) new entries appearing on the the Arch Repro Status Page[1] for toolchain components (in particular; binutils, gcc and gcc-libs).

Part of the cause is quite obvious when considering the current toolchain build order. Glibc startfiles compiled with the previous GCC are linked into the final binaries for both binutils and gcc. For example (as of this writing):

$ strings /usr/bin/ld | grep GCC:
GCC: (GNU) 10.2.0
GCC: (GNU) 11.1.0
$ strings /usr/bin/gcc | grep GCC:
GCC: (GNU) 10.2.0
GCC: (GNU) 11.1.0

Code from the previous toolchain is leaking into the current. This will of course sort itself out as toolchain components get rebuilt for minor revisions. But it would be nice if everything "just worked" from the get-go.

One way to fix this would be a slight tweak to the current toolchain build order. The status quo has clearly served Arch well over the years (albeit with this tiny flaw) so this is merely a suggestion from the peanut gallery :)



Because GCC is such a beast to compile, it would make sense (and is perfectly acceptable IMHO) for the first GCC to be compiled with `--disable-bootstrap' (in fact I would advocate for the both GCC's to be non-bootstrapped, but that's a separate, possibly controversial, topic for another day).

Part of my thinking is based on experience building cross toolchains. Sidenote: there is a python script in the glibc sources `' which IMHO represents state-of-the-art methodology for building cross toolchains. If you haven't already, check it out, it's awesome!

Anyway, just throwing it out there for comment.

This task depends upon

Comment by Allan McRae (Allan) - Thursday, 20 May 2021, 03:30 GMT
Hrm... I'm trying to work out if this also suggests the need to rebuild glibc for more minor gcc bumps (e.g. 11.1.0 to 11.2.0, or even 11.1.0 to 11.1.1) which we have not done in the past. e.g. binutils built after a minor gcc bump is still reproducible, but will contain references to multiple gcc versions.
Comment by Eli Schwartz (eschwartz) - Thursday, 20 May 2021, 04:07 GMT
There shouldn't be any reproducibility problems except when bootstrapping using packages that never get published.

Comment by Toolybird (Toolybird) - Thursday, 20 May 2021, 05:19 GMT
> except when bootstrapping using packages that never get published

Bingo! Because that's *exactly* what happens now in the Arch toolchain bootstrap. There are intermediate packages along the way.

There's got to be a way to express these dependencies properly in terms of the package manager (e.g.: glibc-first, binutils-stage1 etc.) but so far I am yet to come up with a clean solution.
Comment by Allan McRae (Allan) - Thursday, 20 May 2021, 06:55 GMT
As I said in my comment, deciding whether to do a full bootstrap on more minor version is not about reproducibility. It was about whether we should ensure there is only a reference into a single gcc version in these files.
Comment by Emil (xexaxo) - Thursday, 20 May 2021, 18:40 GMT
Humble request - please document (as you get the chance of course) our current rebuild order.
A while ago Allan pointed me to LFS for more details about the sequence, yet they recommend/use what Toolybird is proposing just above.
Comment by Giancarlo Razzolini (grazzolini) - Friday, 21 May 2021, 01:35 GMT Comment by Toolybird (Toolybird) - Friday, 09 July 2021, 21:20 GMT
While this bug is about the Arch toolchain bootstrap procedure, it turns out there is also an upstream GCC 11 bug affecting reproducibility.

I dug around a bit and filed an upstream bug report[1].

A fix has been proposed[2].

Until the fix is committed, the patch can be grabbed from here[3].


sorry about the last link, flyspray has mangled it..
Comment by Toolybird (Toolybird) - Sunday, 11 July 2021, 21:47 GMT
Also on the rebuilderd status page is gcc-go. I have made some progress on this one.

Filed an upstream bug report[1] but no response so far. Folks interested in repro should take a look.

Comment by Toolybird (Toolybird) - Thursday, 15 July 2021, 19:59 GMT
Upstream have committed fixes for both bugs. Yay, GCC is reproducible again. The first will be included in upcoming 11.2. The gcc-go fix is only on mainline but is an easy cherry-pick.

Still mulling on how to fix the "bootstrap whole Arch toolchain reproducibly" issue...