FS#48253 - [archiso] archiso broken by latest syslinux-6.03-4 release

Attached to Project: Arch Linux
Opened by Jeff Hodd (jghodd) - Friday, 19 February 2016, 18:53 GMT
Last edited by Anatol Pomozov (anatolik) - Sunday, 28 February 2016, 17:14 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Pierre Schmitz (Pierre)
Gerardo Exequiel Pozzi (djgera)
Anatol Pomozov (anatolik)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:

The latest release of syslinux - v6.03-4 - breaks archiso builds, making them unbootable. On boot, the system displays "ISOLINUX 6.03 6.03* EHDD" and hangs.

I noticed the breakage a couple of days ago after an archiso build that downloaded the most recent syslinux package version. I had been repeatedly building for the 3 days prior to that without any issues. None of the configuration changed between builds since I'm working on detailing the kde installation. Yesterday, I repackaged syslinux-6.03-3 as syslinux-6.03-5 and loaded it into my own testing repo - the issue went away after an archiso build using the repackaged syslinux-6.03-3. The problem is definitely caused by the '-4' version.

Additional info:
* package version(s)
- archiso-23-1
- syslinux-6.03-4

* config and/or log files etc.


Steps to reproduce:

Create an arch iso build using archiso and write it to a flash drive - try to boot it up. It errors out with the message "ISOLINUX 6.03 6.03* EHDD" and hangs.
This task depends upon

Closed by  Anatol Pomozov (anatolik)
Sunday, 28 February 2016, 17:14 GMT
Reason for closing:  Fixed
Additional comments about closing:  6.04-5
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 06:43 GMT
What filesystem type do you use for your /boot partition?
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 07:19 GMT
Here's the fdisk listing of the partitions after dd'ing the iso image to the flash drive:

Disk /dev/sdd: 58.5 GiB, 62774706176 bytes, 122606848 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x427b7ac4

Device Boot Start End Sectors Size Id Type
/dev/sdd1 * 0 7864319 7864320 3.8G 0 Empty
/dev/sdd2 164 63651 63488 31M ef EFI (FAT-12/16/32)

Otherwise, I don't use a separate /boot partition - /boot is included in the root partition.

I did note that no changes were made to any of the configuration where syslinux is concerned. I use what;s basically provided in the archiso package's 'releng' directory with some archiso config file customisations for my build (kernel params). What has been working for 3 years - and I do rolling releases, so I build out sometimes 5-6 times a month - suddenly stopped working with the arrival of syslinux-6.03-4.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 07:21 GMT
I don't use btrfs, if that's what you need to know. The squashed root partition is formatted as ext4.
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 09:00 GMT
Ok it fails at the time of booting from USB key.



Indeed the syslinux package contains 2 changes in the last release
1) in syslinux-install_update https://projects.archlinux.org/svntogit/packages.git/commit/trunk?h=packages/syslinux&id=7bfa5296e2db18f4cddd77dac7c8aa4fa53d200c and as far as I see archiso does not use the script
2) An upstream bugfix for btrfs as a /boot partition http://repo.or.cz/syslinux.git/patch/548386049cd41e887079cdb904d3954365eb28f3?hp=721a0af2f0ba111c31685c5f6c5481eb25346971 (related to the recent format change). It does not sounds like your case neither.

So it sounds weird that it fails. I can't test archiso right now so let's try to check if there are errors in your archiso build log?

Do you try to boot in EFI mode? I wonder if gnu-efi-libs pulled during syslinux build has changed since last syslinux build. To test it could you *build* previous syslinux (6.03-3) and use instead what we have in the repo. Does it still fail?

Could you try to boot in BIOS mode?
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 18:46 GMT
I always boot in BIOS mode - that's where the issue is coming up.

Check out this task thread - https://bugs.archlinux.org/task/48247

When I originally created this bug report, I created it as a core:syslinux issue which was found by the Manjaro folks, who also discovered the syslinux bug. They've been able to determine that the issue is the btrfs boot patch. I'm building their version now (no btrfs boot patch + latest git commit) and will also build the current version (6.03-4) without the btrfs boot patch, and will test both during the course of the day. I'll update you once I've tested both versions.
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 19:11 GMT
I confirm that image built with archiso does not boot from my USB key. The same computer works with syslinux 6.03-4 fine. So there is something special about USB book process that does let to use the latest syslinux.

https://bugs.archlinux.org/task/48247 claims that the problem is btrfs patch but I don't understand how it can be. The patch touches only codepath for booting from btrfs /boot partition... Weird. I am going to build syslinux without the btrfs patch and create an archiso image.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 19:19 GMT
Yeah, I'll be doing the same thing. I'm also a little surprised that this issue would be due to the btrfs boot patch - I noticed that it only affects btrfs.c and btrfs.h. Shouldn't impact a non-btrfs partition.

I noticed that you closed the other task. Should this one now be considered a syslinux vs an archiso bug?
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 19:33 GMT
I have merged another bug here to avoid duplication for debugging effort. I'll change title once it become clear the real reason of the problem.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 19:46 GMT
Ok. Thanks, Anatol.

I'm about to start a new archiso build using the 6.03-4 commit without the btrfs patch. It'll take a while, but I'll keep an eye on new information from you in the meantime.
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 20:37 GMT
I built syslinux without btrfs patch, I also reverted syslinux sources to 6.03-3 and rebuilt but in both cases USB boot hangs. Could anybody confirm/deny it? If it is confirmed then it is probably because of environment.

I see upstream discusses binutils-2.26 issue
http://www.syslinux.org/archives/2016-February/024794.html
https://sourceware.org/bugzilla/show_bug.cgi?id=19538

but we do not have binutils in [stable] yet....
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 21:09 GMT
ok. i guess there's not much point in testing different syslinux builds then. I guess for now I'll revert to my repackaged 6.03-3 and wait for the new binutils package. Apparently building the new binutils is a little daunting since it requires building the new/test versions of glibc linux-headers and gcc... at least.

Since syslinux-6.03-4 does not appear to be ready for prime-time, can 6.03-4 be backed out of the core repository and replaced with the older 6.03-3?
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 21:43 GMT
> not much point in testing different syslinux builds then

Please do test your version of syslinux like you mentioned in #comment144182. It is very important to make sure that the problem is not btrfs patch but the build environment. 2 people have the same result will confirm that the problem is somewhere else.

> can 6.03-4 be backed out of the core repository and replaced with the older 6.03-3?
Assuming that syslinux is broken because of build environment (binutils or gcc) then rebuild 6.03-3 will not help. We need the binaries built with the old environment (exactly like 6.03-3). To go this route I need confirmation from other people that the problem is not because of syslinux changes.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 21:49 GMT
No problem - I'll test both then. My build takes between 4 and 5 hours though, so it'll be awhile.
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 21:52 GMT
Here are changes for gcc5 compatibility

http://www.syslinux.org/archives/2015-September/024317.html
http://www.syslinux.org/archives/2015-September/024318.html
http://www.syslinux.org/archives/2015-September/024319.html

These changes look suspicious. I am going to test them as well.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 21:55 GMT
I've built both the current arch version without the btrfs patch, and the manjaro version which is the most recent syslinux git commit minus the btrfs patch. I'm trying out the arch version without the patch first.
Comment by Allan McRae (Allan) - Saturday, 20 February 2016, 21:57 GMT
Why do you think this is a binutils issue? The bug appears in binutils>=2.26 which landed in [testing] yesterday. Syslinux was built several daya before that...

In fact... our binutils-2.26 package in [testing] has that fix.
Comment by Jeff Hodd (jghodd) - Saturday, 20 February 2016, 21:58 GMT
um, yeah... those look really suspicious.
Comment by Anatol Pomozov (anatolik) - Saturday, 20 February 2016, 22:04 GMT
Allan, I am not telling that is because of binutils, at least I have no confirmation for it yet. I added you to make you aware that there are some syslinux problems with binutils and gcc5. You expertise in debugging this issue will be helpful.

Our binutils 2.25.1 pulls additional commits from upstream. I do not know the original commit that introduces the binutils problem but I wonder if it can be in our 2.25.1 already?
Comment by Allan McRae (Allan) - Saturday, 20 February 2016, 22:07 GMT
It should not be - the 2.25 branch appear unaffected.
Comment by Dan (midfingr) - Saturday, 20 February 2016, 22:49 GMT
I can confirm this affects builds using archiso which do not boot via usb key or virtual machine (qemu, vbox).
Comment by Anatol Pomozov (anatolik) - Sunday, 21 February 2016, 01:08 GMT
Adding the gcc patches allows me to build syslinux that boots from USB key. I am going to push it to [testing] so you can verify it.

Comment by Jeff Hodd (jghodd) - Sunday, 21 February 2016, 01:27 GMT
Nice catch, Anatol.

I did a 'git log' on the arch 6.04-4 commit and compared it to the the latest from repo.or.cz/syslinux.git - quite a difference. The arch version is missing quite a few updates between Oct. 2014 and Feb. 2016, including a gcc patch. At some point are the two versions going to converge? A 1.5 year difference in the code base is quite a large one.
Comment by Anatol Pomozov (anatolik) - Sunday, 21 February 2016, 02:48 GMT
All these changes will be available to Arch users once upstream makes a new release.
Comment by Jeff Hodd (jghodd) - Sunday, 21 February 2016, 04:04 GMT
Ok. I understand. I see that the last 'tagged' release was 16 months ago. Anyway, I'll download your testing version and test it out. It'll likely be tomorrow before I can report back.
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 08:00 GMT
Well, I tested commit c1cc1bcef1e23ecb9f650c75f9dafc7e61101218 on our end with Manjaro. It works. The btrfs-patch is included in our version as it already exists in the source. I didn't yet tested the current 6.03-4 without the btrfs-patch.

So followed tarball seems to be fine (6.04-dev): http://repo.or.cz/syslinux.git/snapshot/c1cc1bcef1e23ecb9f650c75f9dafc7e61101218.tar.gz
You either try to find out which extra patches are needed our you use a newer git-snapshot which works.
Comment by Anatol Pomozov (anatolik) - Sunday, 21 February 2016, 14:08 GMT
Philip, have you tested 6.04-5?
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 16:22 GMT
You can follow my work here: https://github.com/manjaro/packages-extra/issues/52
I'm currently building another ISO image. As soon as I've feedback I'll post it here.
Comment by Philip Müller (philm) - Sunday, 21 February 2016, 16:42 GMT
Yep, I can confirm that 6.04-5 works for me.
Comment by Jeff Hodd (jghodd) - Sunday, 21 February 2016, 17:30 GMT
Ok, Anatol - 6.03-5 works fine for me. Thanks much, bud.
Comment by Plague (centuryplague) - Sunday, 21 February 2016, 19:00 GMT
Hello, I can confirm this problem has blocked my live ISO build since Feb 16-17, I did not have a chance to create a bug but this is exactly the issue. The ISOs do not boot in physical machine or virtual machine. It happened whether using my custom ISO or the stock releng profile.
   archiso_livecd_virtmanager_bo... (17.9 KiB)
Comment by Jeff Hodd (jghodd) - Sunday, 21 February 2016, 19:15 GMT
Plague - use the syslinux-6.03-5 package currently in testing (I had to create my own testing repository to use it with my archiso builds). It fixes the issue.
Comment by Anatol Pomozov (anatolik) - Sunday, 21 February 2016, 21:53 GMT
syslinux-6.03-5 has been moved to stable. Please report if you see any issues with it.
Comment by Jeff Hodd (jghodd) - Sunday, 21 February 2016, 21:59 GMT
Thank you for your rapid response and your insightful solution, Anatol. It's much appreciated.
Comment by Chris Plock (plockc) - Sunday, 21 February 2016, 22:26 GMT
What timing, I had this issue today with archiso after not touching it for a couple weeks, although I believe for me it said "ISOLINUX 6.03 6.03* ETCD". It now works, thanks!
Comment by Plague (centuryplague) - Monday, 22 February 2016, 18:13 GMT
Appears fixed.

Loading...