FS#61966 - systemd/udevd: eth0 configuration fails after update - autonegotiation is unset or enabled

Attached to Project: Arch Linux
Opened by Stefan Förster (HotblackDesiato) - Sunday, 10 March 2019, 07:54 GMT
Last edited by Dave Reisner (falconindy) - Sunday, 08 December 2019, 11:18 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To Dave Reisner (falconindy)
Christian Hesse (eworm)
Architecture x86_64
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

Description:

I am running a VPS with Arch Linux 64 bit. Yesterday I did a system upgrade and rebooted. Since then the VPS was unreachable, no ping or ssh. I started the emergency system that the host provides and could look at some logs. I am not an expert, but after many hours of reading posts in the internet I believe the kernel 5.0, which has also been updated, has a different behaviour on how to initialise the eth0 device and udevd/networkd later struggle to re-initialise the device or systemd, which has also been updated, has a different behaviour.

Here is the relevant part of the system.journal:

journalctl -b -u systemd-networkd.service -u systemd-udevd.service --no-hostname --file system.journal
-- Logs begin at Sun 2019-03-10 06:47:29 UTC, end at Sun 2019-03-10 07:11:44 UTC. --
Mar 10 07:11:36 systemd[1]: Starting udev Kernel Device Manager...
Mar 10 07:11:36 systemd[1]: Started udev Kernel Device Manager.
Mar 10 07:11:36 systemd[1]: Starting Network Service...
Mar 10 07:11:36 systemd-udevd[296]: link_config: autonegotiation is unset or enabled, the speed and duplex are n
Mar 10 07:11:37 systemd-networkd[269]: Enumeration completed
Mar 10 07:11:37 systemd-networkd[269]: lo: Removing non-existent address: ::1/128 (valid forever), ignoring
Mar 10 07:11:37 systemd[1]: Started Network Service.
Mar 10 07:11:37 systemd-udevd[279]: Using default interface naming scheme 'v240'.
Mar 10 07:11:37 systemd-networkd[269]: eth0: Gained carrier
Mar 10 07:11:37 systemd-networkd[269]: eth0: Could not bring up interface: Invalid argument
Mar 10 07:11:37 systemd-networkd[269]: eth0: could not set address: Permission denied

Compared to older journal logs before the update, the error "link_config: autonegotiation is unset or enabled, the speed and duplex are n" is new. The same for "Using default interface naming scheme 'v240'" and of course "eth0: Could not bring up interface: Invalid argument".

No change to the link file:
[Match]
MACAddress=XX:XX:XX:XX:XX:XX
[Link]
Name=eth0

I added "AutoNegotiation=false", but no change.

There is also a file "eth0_STATIC.network:

[Match]
Name=eth0
[Network]
Address=XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX/64
Gateway=XXXX::1
DNS=XXXX:XXXX::X:XX
DNS=XXXX:XXXX::X:XX
Address=XX.XXX.XXX.20/24
Gateway=XX.XXX.XXX.1
DNS=79.143.183.251
DNS=79.143.183.252

I was able to setup an rc-local.service. rc.local runs as first service after start of systemd. An 'ifconfig' command in rc.local reveals that eth0 is up before udevd/networkd:

eth0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 80.241.217.20 netmask 255.255.255.0 broadcast 80.241.217.255
ether 00:50:56:3e:14:4c txqueuelen 1000 (Ethernet)
RX packets 1606 bytes 127435 (124.4 KiB)
RX errors 0 dropped 0 overruns 0 frame 0
TX packets 0 bytes 0 (0.0 B)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0

The problem comes later when udevd/networkd try to reconfigure eth0.

From pacman.log:
[2019-03-09 16:42] [ALPM] installed systemd-libs (241.7-2)
[2019-03-09 16:42] [ALPM] upgraded e2fsprogs (1.44.5-1 -> 1.44.6-1)
[2019-03-09 16:42] [ALPM] upgraded openssl (1.1.1.a-1 -> 1.1.1.b-1)
[2019-03-09 16:42] [ALPM] upgraded pcre (8.42-2 -> 8.43-1)
[2019-03-09 16:42] [ALPM] upgraded libxau (1.0.8-3 -> 1.0.9-1)
[2019-03-09 16:42] [ALPM] upgraded at-spi2-core (2.30.0-2 -> 2.30.1-1)
[2019-03-09 16:42] [ALPM] upgraded at-spi2-atk (2.30.0-1 -> 2.30.1-1)
[2019-03-09 16:42] [ALPM] upgraded bind-tools (9.13.5-4 -> 9.13.7-1)
[2019-03-09 16:42] [ALPM] upgraded pixman (0.36.0-1 -> 0.38.0-2)
[2019-03-09 16:42] [ALPM] upgraded harfbuzz (2.3.1-1 -> 2.3.1-2)
[2019-03-09 16:42] [ALPM] upgraded cairo (1.16.0-1 -> 1.16.0-2)
[2019-03-09 16:42] [ALPM] upgraded cryptsetup (2.0.6-1 -> 2.1.0-1)
[2019-03-09 16:42] [ALPM] upgraded libpsl (0.20.2-3 -> 0.20.2-5)
[2019-03-09 16:42] [ALPM] upgraded kmod (25-1 -> 26-2)
[2019-03-09 16:42] [ALPM] upgraded libelf (0.175-1 -> 0.176-1)
[2019-03-09 16:42] [ALPM] upgraded systemd (240.34-3 -> 241.7-2)
[2019-03-09 16:42] [ALPM] upgraded curl (7.64.0-1 -> 7.64.0-9)
[2019-03-09 16:42] [ALPM] upgraded sqlite (3.26.0-2 -> 3.27.2-1)
[2019-03-09 16:42] [ALPM] upgraded mariadb-libs (10.3.12-5 -> 10.3.13-3)
[2019-03-09 16:42] [ALPM] upgraded dovecot (2.3.4.1-1 -> 2.3.5-1)
[2019-03-09 16:42] [ALPM] upgraded file (5.35-1 -> 5.36-1)
[2019-03-09 16:42] [ALPM] upgraded libjpeg-turbo (2.0.1-1 -> 2.0.2-1)
[2019-03-09 16:42] [ALPM] upgraded gdk-pixbuf2 (2.38.0-1 -> 2.38.1-1)
[2019-03-09 16:42] [ALPM] upgraded getmail (5.9-1 -> 5.13-1)
[2019-03-09 16:42] [ALPM] upgraded git (2.20.1-1 -> 2.21.0-1)
[2019-03-09 16:42] [ALPM] upgraded libassuan (2.5.2-1 -> 2.5.3-1)
[2019-03-09 16:42] [ALPM] upgraded gnutls (3.6.6-1 -> 3.6.6-3)
[2019-03-09 16:42] [ALPM] upgraded libsecret (0.18.7-1 -> 0.18.8-2)
[2019-03-09 16:42] [ALPM] upgraded gnupg (2.2.12-2 -> 2.2.13-1)
[2019-03-09 16:42] [ALPM] upgraded librsvg (2:2.44.12-1 -> 2:2.44.13-1)
[2019-03-09 16:42] [ALPM] upgraded libxkbcommon (0.8.3-1 -> 0.8.4-1)
[2019-03-09 16:42] [ALPM] upgraded mesa (18.3.3-2 -> 18.3.4-1)
[2019-03-09 16:42] [ALPM] upgraded tar (1.31-2 -> 1.32-1)
[2019-03-09 16:42] [ALPM] upgraded google-chrome (72.0.3626.96-1 -> 72.0.3626.121-1)
[2019-03-09 16:42] [ALPM] upgraded gpgme (1.12.0-1 -> 1.12.0-2)
[2019-03-09 16:42] [ALPM] upgraded iputils (20180629.f6aac8d-3 -> 20180629.f6aac8d-4)
[2019-03-09 16:42] [ALPM] upgraded libatomic_ops (7.6.8-1 -> 7.6.10-1)
[2019-03-09 16:42] [ALPM] upgraded libfontenc (1.1.3-2 -> 1.1.4-1)
[2019-03-09 16:43] [ALPM] upgraded linux-firmware (20190118.a8b75ca-1 -> 20190212.28f5f7d-1)
[2019-03-09 16:43] [ALPM] upgraded mkinitcpio-busybox (1.29.3-1 -> 1.30.1-1)
[2019-03-09 16:43] [ALPM] upgraded linux (4.20.7.arch1-1 -> 5.0.arch1-1)
[2019-03-09 16:43] [ALPM] upgraded man-db (2.8.5-1 -> 2.8.5-2)
[2019-03-09 16:43] [ALPM] upgraded man-pages (4.16-2 -> 5.00-1)
[2019-03-09 16:43] [ALPM] upgraded mariadb-clients (10.3.12-5 -> 10.3.13-3)
[2019-03-09 16:43] [ALPM] upgraded mariadb (10.3.12-5 -> 10.3.13-3)
[2019-03-09 16:43] [ALPM] upgraded mesa-vdpau (18.3.3-2 -> 18.3.4-1)
[2019-03-09 16:43] [ALPM] upgraded netctl (1.19-1 -> 1.20-1)
[2019-03-09 16:43] [ALPM] upgraded pacman (5.1.2-2 -> 5.1.3-1)
[2019-03-09 16:43] [ALPM] upgraded php (7.3.2-1 -> 7.3.3-1)
[2019-03-09 16:43] [ALPM] upgraded php-cgi (7.3.2-1 -> 7.3.3-1)
[2019-03-09 16:43] [ALPM] upgraded php-fpm (7.3.2-1 -> 7.3.3-1)
[2019-03-09 16:43] [ALPM] upgraded pigeonhole (0.5.4-3 -> 0.5.4-4)
[2019-03-09 16:43] [ALPM] upgraded reflector (2018-2 -> 2019.3-1)
[2019-03-09 16:43] [ALPM] upgraded run-parts (4.8.6-1 -> 4.8.6.1-1)
[2019-03-09 16:43] [ALPM] upgraded s-nail (14.9.11-2 -> 14.9.11-3)
[2019-03-09 16:43] [ALPM] upgraded sshfs (3.5.1-1 -> 3.5.1-2)
[2019-03-09 16:43] [ALPM] upgraded systemd-sysvcompat (240.34-3 -> 241.7-2)
[2019-03-09 16:43] [ALPM] upgraded xorg-mkfontscale (1.1.3-1 -> 1.2.0-2)
[2019-03-09 16:43] [ALPM] upgraded ttf-liberation (2.00.4-1 -> 2.00.5-1)
[2019-03-09 16:43] [ALPM] upgraded unrar (1:5.7.1-1 -> 1:5.7.3-1)
[2019-03-09 16:43] [ALPM] upgraded vim-runtime (8.1.0877-1 -> 8.1.0996-1)
[2019-03-09 16:43] [ALPM] upgraded vim (8.1.0877-1 -> 8.1.0996-1)
[2019-03-09 16:43] [ALPM] upgraded whois (5.4.0-2 -> 5.4.1-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-common (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-devel (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-xdmx (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-xephyr (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-xnest (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-xvfb (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-server-xwayland (1.20.3-1 -> 1.20.4-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xrdb (1.1.1-1 -> 1.2.0-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xdm (1.1.11+50+gb685338-1 -> 1.1.12-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xev (1.2.2-2 -> 1.2.3-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xhost (1.0.7-2 -> 1.0.8-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xmodmap (1.0.9-2 -> 1.0.10-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xinit (1.4.0-3 -> 1.4.1-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xlsatoms (1.1.2-2 -> 1.1.3-1)
[2019-03-09 16:43] [ALPM] upgraded xorg-xvinfo (1.1.3-2 -> 1.1.4-1)

Additional info:
systemd version 241.7-2

Steps to reproduce:
Not sure whether or how to repro this. Arch Linux is running on a virtual private server and has been updated (systemd & linux kernel).
This task depends upon

Closed by  Dave Reisner (falconindy)
Sunday, 08 December 2019, 11:18 GMT
Reason for closing:  Fixed
Additional comments about closing:  systemd 242.29-3
Comment by Benoit Izac (benizac) - Friday, 22 March 2019, 07:33 GMT
Hello,

I have a similar problem since the same date, my connection does not work on startup because the ethernet interface is too long to bring up:

-- Subject: A start job for unit netctl@ethernet\x2ddhcp.service has begun execution
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- A start job for unit netctl@ethernet\x2ddhcp.service has begun execution.
--
-- The job identifier is 686.
Mar 22 08:13:13 boa network[1207]: Starting network profile 'ethernet-dhcp'...
Mar 22 08:13:13 boa kernel: RTL8211DN Gigabit Ethernet r8169-2600:00: attached PHY driver [RTL8211DN Gigabit Ethernet] (mii_bus:phy_addr=r8169-2600:00, irq=IGNORE)
Mar 22 08:13:13 boa kernel: r8169 0000:26:00.0 eth0: Link is Down
Mar 22 08:13:19 boa network[1207]: No connection found on interface 'eth0' (timeout)
Mar 22 08:13:19 boa network[1207]: Failed to bring the network up for profile 'ethernet-dhcp'
Mar 22 08:13:19 boa systemd[1]: netctl@ethernet\x2ddhcp.service: Main process exited, code=exited, status=1/FAILURE
-- Subject: Unit process exited
-- Defined-By: systemd
-- Support: https://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- An ExecStart= process belonging to unit netctl@ethernet\x2ddhcp.service has exited.
--
-- The process' exit code is 'exited' and its exit status is 1.


I have to do 3 or 4 successives ``netctl restart ethernet-dhcp'' to have the Ethernet interface working.
Comment by Stefan Förster (HotblackDesiato) - Saturday, 23 March 2019, 01:52 GMT
Dear Benoit,

Thanks for your reply, interesting. Unfortunately, I can't do further tests since my VPS is a remote machine and I have no physical access to it. After one week I decided to reset the VPS and reinstall everything. I installed Debian stable as it seems that Arch is too cutting edge and unstable. Hence I no longer can do any testing. But since you have the same issue, I suggest to keep the bug report open. This should be further investigated. I am running Arch on my desktop and haven't updated since the bug occurred, I'm too scared.

Cheers, Stefan
Comment by Herbert M. (seppel) - Friday, 03 May 2019, 09:53 GMT
Same problem here after system updating on 2. of may.

Mai 02 19:35:19 vserver systemd-networkd[355]: Assertion 'link->network' failed at ../systemd-stable/src/network/networkd-link.c:1858, function link_acquire_ipv4_conf(). Aborting.


With following dumped core

Mai 02 19:35:19 vserver systemd-coredump[377]: Process 355 (systemd-network) of user 981 dumped core.

Stack trace of thread 355:
#0 0x00007f367ade082f raise (libc.so.6)
#1 0x00007f367adcb672 abort (libc.so.6)
#2 0x00007f367ab5802c n/a (libsystemd-shared-242.so)
#3 0x0000555a2de313b2 n/a (systemd-networkd)
#4 0x0000555a2de2a8b8 n/a (systemd-networkd)
#5 0x0000555a2de5982d n/a (systemd-networkd)
#6 0x00007f367ab95963 sd_netlink_process (libsystemd-shared-242.so)
#7 0x00007f367ab95f14 n/a (libsystemd-shared-242.so)
#8 0x00007f367ab888ce n/a (libsystemd-shared-242.so)
#9 0x00007f367ab88ba3 sd_event_dispatch (libsystemd-shared-242.so)
#10 0x00007f367ab88d69 sd_event_run (libsystemd-shared-242.so)
#11 0x00007f367ab88f8c sd_event_loop (libsystemd-shared-242.so)
#12 0x0000555a2ddfe423 n/a (systemd-networkd)
#13 0x00007f367adccce3 __libc_start_main (libc.so.6)
#14 0x0000555a2ddfef6e n/a (systemd-networkd)

Network does not come up on boot. Manual intervention is needed to bring the network up with netctl.

This happend on a VPS at https://www.netcup.de.

If i revert to the system state to 25. April the problem is gone. Seems to be a problem with systemd.networkd or current kernel.

Best regards.
Comment by loqs (loqs) - Friday, 03 May 2019, 11:28 GMT
@seppel https://github.com/systemd/systemd/pull/12466
Comment by Gene (GeneC) - Sunday, 09 June 2019, 22:43 GMT
Best I can tell the fix [1] is in systemd master around a month ago but no on the v242 tag which is I think what arch is using.

Any chance to please get this fix in to the arch systemd package - its obviously a very nasty bug.

thank ya.

[1] https://github.com/systemd/systemd/commit/6450ee3f8c841dce72d8d5b986813bf2cf7dbdf8
Comment by Gene (GeneC) - Monday, 10 June 2019, 01:33 GMT
Reading through the package file - backports() - has this one (drop invalid assertion)
# network: drop invalid assertion
'51aba17b88617515e037e8985d3a4ea871ac47fe'

This one fixes #12422

and
# network: fix assertion when link get carrier
'b9ea3d2e47b3741f3f46ba9c5e19640136933b71

which fixes #124522
These look to fix the issue - unless someone thinks otherwise - so it's in arch systemd and we should be all good now and this can be closed.



Comment by Gene (GeneC) - Sunday, 16 June 2019, 21:03 GMT
Well its not really fixed - while it works with 5.1.9 kernel it does NOT work with 5.2-rc5 kernel - I still get error from systemd-networkd

systemd-networkd[759]: enp3s0: Could not bring up interface: Invalid argument

At this point there are no interfaces configured other than lo.
A laptop running network manager + iwd actually works fine - its all the servers which are using systemd-networkd that fail.


(i'm running systemd 242.29-2)

Comment by Gene (GeneC) - Sunday, 16 June 2019, 21:11 GMT
Looks like this bug:
https://github.com/systemd/systemd/issues/12784
which is fixed in git - looks to me like we meed additional patches for systemd

Can we add these 2 patches to the backports() in systemd please?

4eb086a38712ea98faf41e075b84555b11b54362
9f6e82e6eb3b6e73d66d00d1d6eee60691fb702f
Comment by Gene (GeneC) - Sunday, 16 June 2019, 22:23 GMT
Additional Info:

I built the arch version of systemd with the above 2 patches - I took the trunk PKGBUILD and modified the backports() array with above 2 patches.

When I tried to install systemd, systemd-libs and systemd-sysvcompat things hung/crashed during install of systemd after installing systemd-libs.

The machine at this point was hosed and offline - as systemd now in a bad state. Don't yet know which of the 2 packages caused the problem.

Open to suggestions on how to proceed - will restore machine back to original state. Don't know if one or both patches are a problem or if there is a conflict with existing patches in backports().
Best I can tell everything applied cleanly.

Don't know if this is relevant but I have testing installed and so gcc is 9.1.0-1


I could also try building latest git snapshot but not sure how much that helps with the arch package.

Comment by Gene (GeneC) - Sunday, 16 June 2019, 23:04 GMT
I tried building git master (v242-923-g4b381a9ef6) and I have same problem - systemd-libs updated fine, then it crashes on systemd. Machine is hosed same as before.

I wonder if I am not building the package correctly?
Comment by Jan Alexander Steffens (heftig) - Tuesday, 18 June 2019, 16:30 GMT
If you build the trunk package as-is, does the problem still occur?
Comment by Gene (GeneC) - Tuesday, 18 June 2019, 17:29 GMT
yeh thats good suggestion - I haven't had time to try that yet. -Obviously if that fails its my build that's the root of my problem - unclear how I can mess that up (compiler differences aside)
I will report back on that soon as I can try it out.
Comment by Gene (GeneC) - Wednesday, 19 June 2019, 18:53 GMT
Same problem - clearly my build is bad. I tried both
asp checkout; cd trunk
and
asp export

They look the same- changed pkgrel from 2 to 3 and did makepkg.

Unable to install using pacman -U on resulting packages.

Other than gcc 9.1 compiler I'm at a bit of a loss as to why the systemd package is not building correctly.
Suggestions?

Also, if someone else can produce a working systemd package with the 2 additional backports() I'd be happy to test those with 5.2-rc5.

Thanks and sorry I am unable to validate further without help.
Comment by Christian Hesse (eworm) - Friday, 21 June 2019, 09:43 GMT
Anybody wants to test systemd 242.29-3 from testing?
Comment by Gene (GeneC) - Friday, 21 June 2019, 12:33 GMT
I confirm this fixes the problems - tested with 5.1.12, 5.1.14 and 5.2-rc5/6 kernels - package installs properly and no issues booting either kernel.

Thank you!!

Loading...