Arch Linux

Qemu 2.7.0-2 command line boots from PXE on both bios mode and efi mode, without qemu helper such as libvirt or virt manager.

This is the command line to launch qemu to boot from network. The host bridge is qemu0 and the tap interface is vm0, attached to qemu0.

qemu-system-x86_64 -vga std -m 512M -smp cores=1 -enable-kvm -cpu host -drive file=/tmp/disk.img,format=raw,index=0,media=disk,if=virtio -net nic,model=virtio,macaddr=00:aa:bb:cc:dd:ee,vlan=0 -net tap,ifname=vm0,script=no,downscript=no -boot menu=on -boot n

DHCP, DNS, and TFTP server are handled by dnsmasq listening on qemu0 interface. Using built in Qemu ipxe for bios PXEBOOT and aur iPXE for efi PXEBOOT[1]

[1] https://aur.archlinux.org/packages/ipxe-git/

Thank you for this, I'll have to check it out.

I have a bridge network device setup via systemd (br1). Through virt-manager I've always been able to use my 'br1' device where 'specify shared device' is requested and everything has previously worked fine. Everything actually still does work fine with 2.7, with the exception of a PXE boot.

Alif, the above command (though removing the drive parameter) gives me the same exact failure on 2.7 as through virt-manager/libvirt. (immediate crash on executing PXE rom). It works fine on 2.6.1 however.

I don't rely on systemd-network on the above bridge and tap configuration. I only use ip link and tuntap to set up the network.

Have you tried using qemu's built in tftp server? I've successfully booted OpenBSD 6.0 pxeboot and bsd.rd using the built in PXE server. No need to setup bridge with qemu's built in tftp server for network boot.

This command serves tftp files on /tmp/tftp

qemu-system-x86_64 -boot n -net nic,model=virtio,vlan=0 -net user,vlan=0,tftp=/tmp/tftp,bootfile=/pxeboot

If this fails for you, you can blame qemu's built in iPXE rom, but it works for me.

Thank you, This does work fine for me, so maybe it's not the qemu iPXE rom.

I just wish I could figure out what changed between 2.6.1 and 2.7 that crashes the machine when iPXE attempts to do anything related to my physical network.

Currently I have my tftp server setup on another Arch metal machine acting as a NAS. Other devices (including virtual machines) are directed to it via
'dhcp-boot=pxelinux.0,nas,192.168.9.103' dnsmasq option on my tomato router.

This has worked perfectly for years until now (It still works for physical hosts).

Perhaps then this is an issue with libvirt/qemu interaction. For me, no matter what the network setup (even NAT, host only) as soon as the boot process gets to PXE rom, the VM crashes immediately. No network activity takes place at all. All I see is this:

Booting from ROM...
iPXE (PCI 00:03.0) starting execution

I guess I have some more digging to do.

sudo qemu-system-x86_64 -boot n -net nic,model=virtio,vlan=0 -net bridge,vlan=0,br=br1 -drive file=/tmp/qc2.img,format=qcow2,index=0,media=disk -m 1024

The above command works perfectly.

As soon as I add -enable-kvm, this triggers the immediate crash mentioned above when attempting the PXE boot.

As long as kvm is not enabled, my normal configuration works fine, but slowly.

The same is true for your command that serves the pxe files from /tmp via tftp protocol. When run with -enable-kvm, I get the immediate crash.

I suspect qemu 2.7 introduces regression with KVM on Bloomfield processors. You may find the bad commit using git bisect or ask qemu developers to do it.

Alif, many thanks for your help in troubleshooting this. I will follow up with the qemu team.

Hi,

im having a crash with ipxe netboot on 2.7 with a Intel(R) Core(TM) i7 CPU 920 bit not on i7-4710HQ

I did a git bisect on qemu running a command to attempt a pxe boot as a test
good -> boot attempted
bad -> hang
anything else -> skip

Looks like the problems is with the efi-*rom files commit c52125ab9280733b8d265195f6ffe9c76772b0a5 (not sure why these are used for a bios boot though).


```
[peter@station30 qemu]$ git diff c52125ab9280733b8d265195f6ffe9c76772b0a5 c52125ab9280733b8d265195f6ffe9c76772b0a5^1
diff --git a/pc-bios/efi-e1000.rom b/pc-bios/efi-e1000.rom
index 4e61f9b..4bc89a3 100644
Binary files a/pc-bios/efi-e1000.rom and b/pc-bios/efi-e1000.rom differ
diff --git a/pc-bios/efi-e1000e.rom b/pc-bios/efi-e1000e.rom
deleted file mode 100644
index 192a437..0000000
Binary files a/pc-bios/efi-e1000e.rom and /dev/null differ
diff --git a/pc-bios/efi-eepro100.rom b/pc-bios/efi-eepro100.rom
index 66c5226..85b7f9b 100644
Binary files a/pc-bios/efi-eepro100.rom and b/pc-bios/efi-eepro100.rom differ
diff --git a/pc-bios/efi-ne2k_pci.rom b/pc-bios/efi-ne2k_pci.rom
index 8c3e5fd..ebafd84 100644
Binary files a/pc-bios/efi-ne2k_pci.rom and b/pc-bios/efi-ne2k_pci.rom differ
diff --git a/pc-bios/efi-pcnet.rom b/pc-bios/efi-pcnet.rom
index 802e225..6f19723 100644
Binary files a/pc-bios/efi-pcnet.rom and b/pc-bios/efi-pcnet.rom differ
diff --git a/pc-bios/efi-rtl8139.rom b/pc-bios/efi-rtl8139.rom
index 8827181..086551b 100644
Binary files a/pc-bios/efi-rtl8139.rom and b/pc-bios/efi-rtl8139.rom differ
diff --git a/pc-bios/efi-virtio.rom b/pc-bios/efi-virtio.rom
index 2fc0497..140c680 100644
Binary files a/pc-bios/efi-virtio.rom and b/pc-bios/efi-virtio.rom differ
diff --git a/pc-bios/efi-vmxnet3.rom b/pc-bios/efi-vmxnet3.rom
deleted file mode 100644
index 3d42635..0000000
Binary files a/pc-bios/efi-vmxnet3.rom and /dev/null differ


[peter@station30 qemu]$ git bisect bad
c52125ab9280733b8d265195f6ffe9c76772b0a5 is the first bad commit
commit c52125ab9280733b8d265195f6ffe9c76772b0a5
Author: Gerd Hoffmann <kraxel@redhat.com>
Date: Fri Jun 24 14:18:19 2016 +0200

ipxe: update prebuilt binaries

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>

:040000 040000 1cbe7b99f5fd330fab98804444b0d06f0c029b00 c3f45812b9f74080e08718c1923050e304ccc768 M pc-bios


[peter@station30 qemu]$ git bisect log
git bisect start
# good: [e8ad4d16808690e9c0d68b140218ca466c9309fc] ide: move headers to include folder
git bisect good e8ad4d16808690e9c0d68b140218ca466c9309fc
# bad: [d9e0dfa2462e32cc5c6c49401ad7bff36453f75c] block: Split bdrv_merge_limits() from bdrv_refresh_limits()
git bisect bad d9e0dfa2462e32cc5c6c49401ad7bff36453f75c
# skip: [1ec20c2a3aa5b90522d15fccf7f052a90f70ddaa] Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging
git bisect skip 1ec20c2a3aa5b90522d15fccf7f052a90f70ddaa
# good: [d7f30403576f04f1f3a5fb5a1d18cba8dfa7a6d2] cputlb: don't cpu_abort() if guest tries to execute outside RAM or RAM
git bisect good d7f30403576f04f1f3a5fb5a1d18cba8dfa7a6d2
# good: [13f5e8003e7b67039cb7a19e41b4f7f7ac669cb3] qmp: fix spapr example of query-hotpluggable-cpus
git bisect good 13f5e8003e7b67039cb7a19e41b4f7f7ac669cb3
# good: [3173a1fd549b7fa0f7029b2c6a6b86ba6efa92aa] Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20160704' into staging
git bisect good 3173a1fd549b7fa0f7029b2c6a6b86ba6efa92aa
# good: [8662d7db392f906c7808014051b278ad1542db93] Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-2.7-20160705' into staging
git bisect good 8662d7db392f906c7808014051b278ad1542db93
# bad: [202204717a7e73971cccebd38c5d8ac4b0bfcef8] nbd: Advertise realistic limits to block layer
git bisect bad 202204717a7e73971cccebd38c5d8ac4b0bfcef8
# bad: [45027808cddf42ce002fb62824204d275d06504d] ipxe: add new roms to BLOBS
git bisect bad 45027808cddf42ce002fb62824204d275d06504d
# good: [1676103dc25e8b7d5bbd2ee7ae0bab08a02c1c88] e1000e: add boot rom
git bisect good 1676103dc25e8b7d5bbd2ee7ae0bab08a02c1c88
# good: [43716de6b3b24f949f8b95a0ae9e2e263dd4f1f0] vmxnet3: add boot rom
git bisect good 43716de6b3b24f949f8b95a0ae9e2e263dd4f1f0
# bad: [c52125ab9280733b8d265195f6ffe9c76772b0a5] ipxe: update prebuilt binaries
git bisect bad c52125ab9280733b8d265195f6ffe9c76772b0a5
# first bad commit: [c52125ab9280733b8d265195f6ffe9c76772b0a5] ipxe: update prebuilt binaries

[peter@station30 qemu]$ git bisect bad
c52125ab9280733b8d265195f6ffe9c76772b0a5 is the first bad commit
commit c52125ab9280733b8d265195f6ffe9c76772b0a5
Author: Gerd Hoffmann <kraxel@redhat.com>
Date: Fri Jun 24 14:18:19 2016 +0200

ipxe: update prebuilt binaries

Signed-off-by: Gerd Hoffmann <kraxel@redhat.com>

:040000 040000 1cbe7b99f5fd330fab98804444b0d06f0c029b00 c3f45812b9f74080e08718c1923050e304ccc768 M pc-bios


[peter@station30 qemu]$ git bisect log
git bisect start
# good: [e8ad4d16808690e9c0d68b140218ca466c9309fc] ide: move headers to include folder
git bisect good e8ad4d16808690e9c0d68b140218ca466c9309fc
# bad: [d9e0dfa2462e32cc5c6c49401ad7bff36453f75c] block: Split bdrv_merge_limits() from bdrv_refresh_limits()
git bisect bad d9e0dfa2462e32cc5c6c49401ad7bff36453f75c
# skip: [1ec20c2a3aa5b90522d15fccf7f052a90f70ddaa] Merge remote-tracking branch 'remotes/bonzini/tags/for-upstream' into staging
git bisect skip 1ec20c2a3aa5b90522d15fccf7f052a90f70ddaa
# good: [d7f30403576f04f1f3a5fb5a1d18cba8dfa7a6d2] cputlb: don't cpu_abort() if guest tries to execute outside RAM or RAM
git bisect good d7f30403576f04f1f3a5fb5a1d18cba8dfa7a6d2
# good: [13f5e8003e7b67039cb7a19e41b4f7f7ac669cb3] qmp: fix spapr example of query-hotpluggable-cpus
git bisect good 13f5e8003e7b67039cb7a19e41b4f7f7ac669cb3
# good: [3173a1fd549b7fa0f7029b2c6a6b86ba6efa92aa] Merge remote-tracking branch 'remotes/pmaydell/tags/pull-target-arm-20160704' into staging
git bisect good 3173a1fd549b7fa0f7029b2c6a6b86ba6efa92aa
# good: [8662d7db392f906c7808014051b278ad1542db93] Merge remote-tracking branch 'remotes/dgibson/tags/ppc-for-2.7-20160705' into staging
git bisect good 8662d7db392f906c7808014051b278ad1542db93
# bad: [202204717a7e73971cccebd38c5d8ac4b0bfcef8] nbd: Advertise realistic limits to block layer
git bisect bad 202204717a7e73971cccebd38c5d8ac4b0bfcef8
# bad: [45027808cddf42ce002fb62824204d275d06504d] ipxe: add new roms to BLOBS
git bisect bad 45027808cddf42ce002fb62824204d275d06504d
# good: [1676103dc25e8b7d5bbd2ee7ae0bab08a02c1c88] e1000e: add boot rom
git bisect good 1676103dc25e8b7d5bbd2ee7ae0bab08a02c1c88
# good: [43716de6b3b24f949f8b95a0ae9e2e263dd4f1f0] vmxnet3: add boot rom
git bisect good 43716de6b3b24f949f8b95a0ae9e2e263dd4f1f0
# bad: [c52125ab9280733b8d265195f6ffe9c76772b0a5] ipxe: update prebuilt binaries
git bisect bad c52125ab9280733b8d265195f6ffe9c76772b0a5
# first bad commit: [c52125ab9280733b8d265195f6ffe9c76772b0a5] ipxe: update prebuilt binaries
```

ipxe is setup as a git submodule
so I pulled that down
then to get older ipxebuilds to wokr its necessary to fixup parserom.pl and compile with make NO_WERROR=1

in ipxe

diff --git a/src/util/parserom.pl b/src/util/parserom.pl
index 28df606..5a849a5 100755
--- a/src/util/parserom.pl
+++ b/src/util/parserom.pl
@@ -157,7 +157,7 @@ sub process_isa_rom {

# Output Makefile rules for the specified ROM declarations
sub print_make_rules {
- my ( $state, my $image, my $desc, my $vendor, my $device, my $dup ) = @_;
+ my ( $state, $image, $desc, $vendor, $device, $dup ) = @_;
unless ( $state->{'is_header_printed'} ) {
print "# NIC\t\n";
print "# NIC\tfamily\t$state->{family}\n";

in qemu
diff --git a/roms/Makefile b/roms/Makefile
index 88b3709..86e967e 100644
--- a/roms/Makefile
+++ b/roms/Makefile
@@ -126,11 +126,13 @@ efi-rom-%: build-pxe-roms build-efi-roms
build-pxe-roms:
$(MAKE) -C ipxe/src CONFIG=qemu \
CROSS_COMPILE=$(x86_64_cross_prefix) \
+ NO_WERROR=1 \
$(patsubst %,bin/%.rom,$(pxerom_targets))

build-efi-roms: build-pxe-roms
$(MAKE) -C ipxe/src CONFIG=qemu \
CROSS_COMPILE=$(x86_64_cross_prefix) \
+ NO_WERROR=1 \
$(patsubst %,bin-i386-efi/%.efidrv,$(pxerom_targets)) \
$(patsubst %,bin-x86_64-efi/%.efidrv,$(pxerom_targets))

git checkout 4e03af8ec2d497e725566a91fd5c19dd604c18a6
git bisect start
git bisect good
git bisect bad 0418631918

make clean ;make -j 8 efirom |& tee build.log
sudo cp ../pc-bios/efi-*.rom /usr/share/qemu/
test vm ipxeboot
git bisect good|bad
...

git bisect start
# good: [4e03af8ec2d497e725566a91fd5c19dd604c18a6] [autoboot] Display image information as part of the default control flow
git bisect good 4e03af8ec2d497e725566a91fd5c19dd604c18a6
# bad: [04186319181298083ef28695a8309028b26fe83c] [thunderx] Fix compilation with older versions of gcc
git bisect bad 04186319181298083ef28695a8309028b26fe83c
# good: [7544763626d67b352d4b1440336080d63c0be0b7] [infiniband] Avoid multiple calls to ib_cmrc_shutdown()
git bisect good 7544763626d67b352d4b1440336080d63c0be0b7
# good: [97c3f6e55a941a3caf3a0a36346d2e89f5a7b283] [iscsi] Include DHCP server address in iBFT
git bisect good 97c3f6e55a941a3caf3a0a36346d2e89f5a7b283
# bad: [17c6f322eef5e0a2250a89b140486cf07598d2fa] [arm] Add support for 64-bit ARM (Aarch64)
git bisect bad 17c6f322eef5e0a2250a89b140486cf07598d2fa
# good: [b696a5063e80aa68cb064a8ac0098edd31632555] [image] Skip misleading "format not recognised" error message
git bisect good b696a5063e80aa68cb064a8ac0098edd31632555
# bad: [65b32a0b7000f70a5bb1d33190d40f9b04c93172] [sis190] Fix building with GCC 6
git bisect bad 65b32a0b7000f70a5bb1d33190d40f9b04c93172
# bad: [9f91df422b9ec7f1db493ea2fe333d8e609b58f9] [build] Remove unnecessary dependency on zlib
git bisect bad 9f91df422b9ec7f1db493ea2fe333d8e609b58f9
# good: [fe62f3c8312c3747c1a6c375b502cffa65d60f63] [tg3] Fix _tg3_flag() for 64-bit builds
git bisect good fe62f3c8312c3747c1a6c375b502cffa65d60f63
# bad: [efd5cf9aadcaf36f45db5d1c3059197a8479567c] [efi] Eliminate use of libbfd
git bisect bad efd5cf9aadcaf36f45db5d1c3059197a8479567c
# bad: [71560d185475117b10994d839afe059577e7768c] [librm] Preserve FPU, MMX and SSE state across calls to virt_call()
git bisect bad 71560d185475117b10994d839afe059577e7768c
# first bad commit: [71560d185475117b10994d839afe059577e7768c] [librm] Preserve FPU, MMX and SSE state across calls to virt_call()

git revert 71560d185475117b10994d839afe059577e7768c
make clean ;make -j 8 efirom |& tee build.log
sudo cp ../pc-bios/efi-*.rom /usr/share/qemu/

All is good for me but this probably breaks the reason for the patch.

It looks like this will be fixed for the next release of qemu.
Upstream bug: https://bugs.launchpad.net/qemu/+bug/1623276/

Peter Pickford specifically thanked by the qemu team for the extremely helpful git bisect!