FS#60832 - Linux 4.19.x Breaks My Networking

Attached to Project: Arch Linux
Opened by Alberto Díaz López (Takuya) - Friday, 16 November 2018, 23:19 GMT
Last edited by Doug Newgard (Scimmia) - Sunday, 09 December 2018, 21:27 GMT
Task Type Bug Report
Category Packages: Core
Status Closed
Assigned To No-one
Architecture x86_64
Severity Critical
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description: Linux 4.19.x breaks networking, i have to restart a few times to have the network working, but if i restart again, it brokes again and i have to restart a few times more. It's a wired network.


Additional info:
* package version(s) 4.19.2
* config and/or log files etc.


Steps to reproduce:
This task depends upon

Closed by  Doug Newgard (Scimmia)
Sunday, 09 December 2018, 21:27 GMT
Reason for closing:  Upstream
Comment by Doug Newgard (Scimmia) - Saturday, 17 November 2018, 07:40 GMT
Logs? Setup? Anything useful at all? As-is, this ticket is totally useless.
Comment by Alberto Díaz López (Takuya) - Saturday, 17 November 2018, 13:19 GMT
Yes, i'm looking for that.

This is my Ethernet network information.

03:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 11).

The motherboard is an ASUS Z97-P. I'm pretty sure that's a problem with the drivers.

I suppose i can't edit the ticket after being opened.
Comment by gerald (bronbunckus) - Monday, 19 November 2018, 18:08 GMT
i have this exact same issue with my ethernet after 4.19.x kernel update. i have to reboot about 5 times to get network working, and if the computer is suspended the network is broken on wake and i have to reboot another 5+ times.
if i launch into an x11 session when the internet is not working the networkmanager icon in kde's system tray will have a yellow icon. if the computer was suspended, on wake the kde networkmanager system tray icon is greyed out.

network card: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 0c)
mobo: MSI Z97 PC Mate (MS-7850)
the network card is onboard

not sure what other information to give or where relevant logs would be
Comment by loqs (loqs) - Monday, 19 November 2018, 18:11 GMT
dmesg from a 4.19.Y boot with the issue and for comparison dmesg from 4.19.Y and 4.18.Y without the issue.
Comment by gerald (bronbunckus) - Monday, 19 November 2018, 18:45 GMT
see attachment

note that ive never had to rollback before and i must have fudged it a bit because after rolling back to 4.18.16 x11 would not start so something about video drivers may show up in the 4.18.16 log, but networking was working fine as expected so i assume it is ok

i didnt include 4.19.1 but it has the same issue as 4.19.2. this issue started with 4.19.1 but i waited for 4.19.2 in case that solved my issue
Comment by loqs (loqs) - Monday, 19 November 2018, 19:55 GMT
Please add r8169.debug=16 to the https://wiki.archlinux.org/index.php/Kernel_parameters which will enable full logging for the module and please post updated good and bad dmesg's for 4.19.2.

4.18.16 shows
[ 4.281922] r8169 0000:03:00.0 enp3s0: link down
[ 4.281924] r8169 0000:03:00.0 enp3s0: link down
[ 4.281990] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 6.825247] r8169 0000:03:00.0 enp3s0: link up
[ 6.825261] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready

4.19.2 bad shows
[ 9.752643] r8169 0000:03:00.0 enp3s0: Link is Down
[ 12.291561] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 12.291576] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
[ 12.956853] r8169 0000:03:00.0 enp3s0: Link is Down

4.19.2 good shows
[ 12.148609] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
Comment by gerald (bronbunckus) - Monday, 19 November 2018, 20:22 GMT
see attachment
Comment by infine (infine) - Thursday, 22 November 2018, 06:50 GMT
Same problem on Lenovo Y520 with 4.19.1

04:00.0 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 10)

Ethernet controller does not wake up from suspend, ip link shows no carrier. Reloading r8169 module appears to fix the problem without rebooting.

I don't have r8169.debug enabled since I'm kinda in a middle of work, but as it stands after resuming from suspend but before reloading the module I have just (skipping other devices)

[37188.676093] r8169 0000:04:00.0 enp4s0: Link is Down
[37189.598093] IPv6: ADDRCONF(NETDEV_UP): enp4s0: link is not ready
[37189.598497] Generic PHY r8169-400:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-400:00, irq=IGNORE)
[37189.724585] IPv6: ADDRCONF(NETDEV_UP): enp4s0: link is not ready

After reloading the module I get

[37293.525027] libphy: r8169: probed
[37293.525329] r8169 0000:04:00.0 eth0: RTL8168g/8111g, 54:e1:ad:8e:5a:76, XID 50900880, IRQ 136
[37293.525331] r8169 0000:04:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[37293.527475] r8169 0000:04:00.0 enp4s0: renamed from eth0
[37293.550214] IPv6: ADDRCONF(NETDEV_UP): enp4s0: link is not ready
[37293.550640] Generic PHY r8169-400:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-400:00, irq=IGNORE)
[37293.671789] IPv6: ADDRCONF(NETDEV_UP): enp4s0: link is not ready
[37295.242124] r8169 0000:04:00.0 enp4s0: Link is Up - 100Mbps/Full - flow control rx/tx
[37295.242133] IPv6: ADDRCONF(NETDEV_CHANGE): enp4s0: link becomes ready
Comment by infine (infine) - Thursday, 22 November 2018, 08:13 GMT
EDIT: duplicate comment, sorry
Comment by gerald (bronbunckus) - Thursday, 22 November 2018, 12:03 GMT
can confirm reloading the module fixes the issue without reboot, but i had to do it twice

first boot:

[ 4.692198] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 4.693091] r8169 0000:03:00.0 enp3s0: Link is Down
[ 7.225659] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 7.225679] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
[ 9.975526] r8169 0000:03:00.0 enp3s0: Link is Down

reload #1:

[ 327.995292] libphy: r8169: probed
[ 327.995574] r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:cb:8a:c2:ab:dc, XID 4c000880, IRQ 28
[ 327.995576] r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 327.997876] r8169 0000:03:00.0 enp3s0: renamed from eth0
[ 328.022015] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 328.022286] Generic PHY r8169-300:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-300:00, irq=IGNORE)
[ 328.148757] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 330.810033] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 330.810044] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
[ 332.225058] r8169 0000:03:00.0 enp3s0: Link is Down

reload #2:

[ 375.725115] libphy: r8169: probed
[ 375.725380] r8169 0000:03:00.0 eth0: RTL8168g/8111g, d8:cb:8a:c2:ab:dc, XID 4c000880, IRQ 28
[ 375.725382] r8169 0000:03:00.0 eth0: jumbo features [frames: 9200 bytes, tx checksumming: ko]
[ 375.727315] r8169 0000:03:00.0 enp3s0: renamed from eth0
[ 375.761673] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 375.762015] Generic PHY r8169-300:00: attached PHY driver [Generic PHY] (mii_bus:phy_addr=r8169-300:00, irq=IGNORE)
[ 375.888752] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 378.444944] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control rx/tx
[ 378.444962] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready

i have r8169.debug=16 enabled but dont seem to have any extra output
Comment by Alberto Díaz López (Takuya) - Thursday, 22 November 2018, 23:15 GMT
It doesn't happened last time, i updated some more packages, included "netctl" & "hwids", maybe one of those solved the bug. It's true that i only rebooted once since i updated, i'll have to check it tomorrow, but looks like this bug it's already solved.
I'll update the status tomorrow.
Comment by Alberto Díaz López (Takuya) - Friday, 23 November 2018, 12:37 GMT
Update. The networking at bootup is already fixed, as i said before, but i forgot to check what happened when i suspended the system, i just tried it before and surprise, when i restore the system from the suspension, the networking is disabled, but no like before when i started the system, before was like it was connected but something it was wrong, the danger signal at the plasma-nm at the systray, when i restore it from suspension now, (i don't really remember if i tried before, when the bug was still at the start, if it happened what happens now at the restoration from suspension, i mean), the plasma-nm at systray is with a red cross, indicating that there's no active connection detected. What a painful status. But looks like it's not all the "fault" from the Linux kernel, because with the same version (4.19.2), updating some packages (like i said i think it was the package "hwids" which solved the start issue), changed the behavior at booting, so i really don't know what would be the problem, but even now i can boot normally, i can't suspend the system without losing access to the network after restore it from the suspension.
Comment by Alberto Díaz López (Takuya) - Saturday, 24 November 2018, 12:09 GMT
Update 2.
I just continued using normally the system, but since yesterday evening/night i noted that it's not working again, sometimes it boots everything alright, some other times, it boots with the networking disabled. Looks like it was an error.
How can i cancel the request of task closure?
Comment by Alberto Díaz López (Takuya) - Monday, 26 November 2018, 16:58 GMT
Update 3.
Tested again on Linux 4.19.4 and the same issues. I also add some dmesg logs when the networking is not working after booting.

4.19.4 Network Not Working After Booting

[ 2.081972] r8169 0000:03:00.0 enp3s0: renamed from eth0
[ 2.687627] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 2.815219] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 2.815524] r8169 0000:03:00.0 enp3s0: Link is Down
[ 6.411025] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control off
[ 6.411036] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
[ 7.467569] r8169 0000:03:00.0 enp3s0: Link is Down

4.19.4 Networking Working Normally After Booting

[ 2.066610] r8169 0000:03:00.0 enp3s0: renamed from eth0
[ 2.587504] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 2.715280] IPv6: ADDRCONF(NETDEV_UP): enp3s0: link is not ready
[ 5.585957] r8169 0000:03:00.0 enp3s0: Link is Up - 1Gbps/Full - flow control off
[ 5.585970] IPv6: ADDRCONF(NETDEV_CHANGE): enp3s0: link becomes ready
Comment by Jolan Luff (jolan) - Monday, 26 November 2018, 17:59 GMT
I hit this too on one machine but 3 identical machines have been fine so far. Part of this bug seems timing dependent.

There's discussion about this on the LKML:

https://marc.info/?t=154279781800003&r=1&w=2

There's a patch that seems to workaround the issue here:

https://marc.info/?l=linux-kernel&m=154291066330748&w=2

But the root cause is still being investigated/discussed.

Bugzilla entry to keep tabs on too:

https://bugzilla.redhat.com/show_bug.cgi?id=1650984
Comment by loqs (loqs) - Tuesday, 04 December 2018, 00:03 GMT Comment by Doug Newgard (Scimmia) - Sunday, 09 December 2018, 16:49 GMT
So is this fixed now?
Comment by Alberto Díaz López (Takuya) - Sunday, 09 December 2018, 19:12 GMT
When the Linux package reaches out Core repository, after the Testing, i would lovely close this bug, just after check everything is alright, but of course, if this keeps going like this, it will be launched Linux 4.20 on the official stream before that day comes, because it's been 4.19.7 on Testing, but before it reached out the Core repository, 4.19.8 was launched, so it become the one that's on Testing actually. I really hope i can check it out soon and close this bug and my headaches about networking on Arch Linux, at least for now.
Comment by loqs (loqs) - Sunday, 09 December 2018, 19:18 GMT
If you are not using any out of tree kernel modules you could just try linux 4.19.8 from testing without a partial update risk due to the separation of kernel and user space.

Loading...