Release Engineering

This project is intented for all release related issues (isos, installer, etc), under the umbrella of the ArchLinux Release Engineers

FS#63683 - [archiso] reproducible arch iso image

Attached to Project: Release Engineering
Opened by Jelle van der Waa (jelly) - Saturday, 07 September 2019, 12:08 GMT
Last edited by Jelle van der Waa (jelly) - Saturday, 07 September 2019, 12:17 GMT
Task Type Bug Report
Category ArchISO
Status Researching
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No


For the reproducible builds effort within Arch, we also want to make our iso reproducible. A bunch of patches have been posted on the mailing list. This ticket should provide an overview of the issues:

- mirrorlist is not reproducible
offending line:
curl -o ${work_dir}/x86_64/airootfs/etc/pacman.d/mirrorlist ''

fixed this by just copying the system mirrorlist for now; we should
probably just use pacman-mirrorlist

- install.txt generation is not stable over time
offending line:
lynx -dump -nolist '' >> ${work_dir}/x86_64/airootfs/root/install.txt

fixed this by removing it for now; can we just version it in archiso
monthly or something?

- various instances of `cp` which do not copy timestamps
think this has been fixed now by tweaking initramfs

- work/x86_64/airootfs/var/log/pacman.log
different timestamps inside the file itself; hack this out by deleting it
do we really need this in the release archiso?

- install dates etc; can we replace INSTALLDATE in package desc with a sed macro (just set all to e.g. 1, or the ISO date with 0 time)
--- releng/work/x86_64/airootfs/var/lib/pacman/local/licenses-20181104-1/desc 2019-09-04 15:01:15.244798335 +0100
+++ releng2/work/x86_64/airootfs/var/lib/pacman/local/licenses-20181104-1/desc 2019-09-04 15:04:59.893817680 +0100
@@ -20,7 +20,7 @@


- there are a few gzip commands with no -n
syslinux modalias.gz; fixed now

- ldconfig aux-cache
pretty sure we can just delete this, hackfixed by deleting for now

- mkinitcpio does not create reproducible initramfs
mkinitcpio: Add --reproducible flag #1
adds an additional dependency on cpio
for now I have a local pkg with --reproducible hardcoded on
this is because the mkinitcpio hooks trigger from linux.preset
where to change this?

- systemd journal catalog is not reproducible
journal: Make the output of --update-catalog deterministic #13482
for now I have a local pkg with a systemd-git build, seems to work

- java cacerts
the trust tool responds to SOURCE_DATE_EPOCH so just do a final run there

- efiboot.img
the fat filesystem has a partition ID, set that to 0
the timestamps within the fs need to be 1'd or SOURCE_DATE_EPOCH'd
This task depends upon

Comment by Daniel Edgecumbe (esotericnonsense) - Saturday, 07 September 2019, 12:13 GMT
FWIW you can check out a patchset that fixes some of these issues on the arch-releng ML

and I have a branch here

(esotericnonsense/20190906repro is the patchset, reproducible is a WIP branch)
Comment by Jelle van der Waa (jelly) - Saturday, 07 September 2019, 12:17 GMT
Some more notes about downloading things:

- mirrorlist, some people argue that running reflector once network connection has been established.

- wiki, as an alternative to downloading the install.txt file, maybe we can put arch-wiki-lite with a link? On the archiso. The downside is that arch-wiki-docs is big due to HTML and arch-wiki-lite isn't automatically generated we should take that up with keenerd.

- EFI images, these as far as I can see are always downloaded from master. Maybe we can actually build and package these images in our repo?
Comment by Daniel Edgecumbe (esotericnonsense) - Saturday, 07 September 2019, 18:12 GMT
Quick comment on EFI images:

I think that longer term it may make sense to build them, but in the short term we can probably link to a specific revision rather than master. That way the verification won't fail.
Comment by Jelle van der Waa (jelly) - Sunday, 09 February 2020, 19:14 GMT
Another requirement for reproducible arch iso is to keep a BUILDINFO file with the installed packages and versions otherwise it would be impossible to reproduce.