FS#75491 - [linux] 5.18.15-arch1-1 breaks ax88179_178a USB NIC

Attached to Project: Arch Linux
Opened by Andreas (poison) - Tuesday, 02 August 2022, 11:03 GMT
Last edited by Toolybird (Toolybird) - Monday, 29 August 2022, 20:41 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Jan Alexander Steffens (heftig)
David Runge (dvzrv)
Architecture x86_64
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
After upgrading from 5.18.14 to 5.18.15 my ax88179_178a USB NICs cause problems:
LEDs indicate link, but there's no more connection and no log entry.
Only (much) later there's a dmesg entry:

Aug 02 02:28:16 firestarter kernel: ------------[ cut here ]------------
Aug 02 02:28:16 firestarter kernel: NETDEV WATCHDOG: netUSBLan0 (ax88179_178a): transmit queue 0 timed out
Aug 02 02:28:16 firestarter kernel: WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x20b/0x220
Aug 02 02:28:16 firestarter kernel: Modules linked in: rfcomm bluetooth ecdh_generic rfkill tun xt_LOG nf_log_syslog xt_limit xt_addrtype veth ext4 crc16 mbcache jbd2 dm_crypt cbc encrypted_keys trusted asn1_encoder tee xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_recent xt_tcpudp xt_conntrack iptable_mangle iptable>
Aug 02 02:28:16 firestarter kernel: ucsi_acpi typec_ucsi typec roles vfat fat tpm_crb tpm_tis tpm_tis_core tpm rng_core acpi_pad acpi_tad igen6_edac mac_hid wmi usbip_host usbip_core crypto_user fuse bpf_preload ip_tables x_tables usbhid btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_mod sdhci_pci>
Aug 02 02:28:16 firestarter kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: G U 5.18.15-arch1-1 #1 9ff3be2e7813d5f2c07119812e1642852fe6c646
Aug 02 02:28:16 firestarter kernel: Hardware name: Intel(R) Client Systems NUC11PAHi5/NUC11PABi5, BIOS PATGL357.0046.2022.0608.1909 06/08/2022
Aug 02 02:28:16 firestarter kernel: RIP: 0010:dev_watchdog+0x20b/0x220
Aug 02 02:28:16 firestarter kernel: Code: ff e9 40 ff ff ff 48 89 df c6 05 56 00 3f 01 01 e8 ea 74 f9 ff 44 89 e9 48 89 de 48 c7 c7 60 d0 d6 bc 48 89 c2 e8 b2 60 19 00 <0f> 0b e9 22 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3
Aug 02 02:28:16 firestarter kernel: RSP: 0018:ffffa216002a8e90 EFLAGS: 00010282
Aug 02 02:28:16 firestarter kernel: RAX: 0000000000000000 RBX: ffff8e3d8777e000 RCX: 0000000000000027
Aug 02 02:28:16 firestarter kernel: RDX: ffff8e41207616a8 RSI: 0000000000000001 RDI: ffff8e41207616a0
Aug 02 02:28:16 firestarter kernel: RBP: ffff8e3d8777e4c8 R08: 0000000000000000 R09: ffffa216002a8ca0
Aug 02 02:28:16 firestarter kernel: R10: 0000000000000003 R11: ffffffffbd4caa08 R12: ffff8e3d8777e41c
Aug 02 02:28:16 firestarter kernel: R13: 0000000000000000 R14: ffffffffbc1d5a80 R15: ffff8e3d8777e4c8
Aug 02 02:28:16 firestarter kernel: FS: 0000000000000000(0000) GS:ffff8e4120740000(0000) knlGS:0000000000000000
Aug 02 02:28:16 firestarter kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 02 02:28:16 firestarter kernel: CR2: 000055732bc99b80 CR3: 000000016d57c004 CR4: 0000000000f70ee0
Aug 02 02:28:16 firestarter kernel: PKRU: 55555554
Aug 02 02:28:16 firestarter kernel: Call Trace:
Aug 02 02:28:16 firestarter kernel: <IRQ>
Aug 02 02:28:16 firestarter kernel: ? pfifo_fast_reset+0x140/0x140
Aug 02 02:28:16 firestarter kernel: call_timer_fn+0x24/0x130
Aug 02 02:28:16 firestarter kernel: __run_timers+0x21c/0x2a0
Aug 02 02:28:16 firestarter kernel: run_timer_softirq+0x1d/0x40
Aug 02 02:28:16 firestarter kernel: __do_softirq+0xd0/0x2c9
Aug 02 02:28:16 firestarter kernel: ? sched_clock_cpu+0xd/0xb0
Aug 02 02:28:16 firestarter kernel: __irq_exit_rcu+0x8e/0xc0
Aug 02 02:28:16 firestarter kernel: sysvec_apic_timer_interrupt+0x72/0x90
Aug 02 02:28:16 firestarter kernel: </IRQ>
Aug 02 02:28:16 firestarter kernel: <TASK>
Aug 02 02:28:16 firestarter kernel: asm_sysvec_apic_timer_interrupt+0x19/0x20
Aug 02 02:28:16 firestarter kernel: RIP: 0010:cpuidle_enter_state+0xdc/0x380
Aug 02 02:28:16 firestarter kernel: Code: 00 00 31 ff e8 85 b5 7e ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 92 02 00 00 31 ff e8 ca a2 84 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 ce 48 8d 04 49 48 8d 04 81 49 8d
Aug 02 02:28:16 firestarter kernel: RSP: 0018:ffffa2160018be90 EFLAGS: 00000246
Aug 02 02:28:16 firestarter kernel: RAX: ffff8e4120772cc0 RBX: 0000000000000002 RCX: 0000000000000000
Aug 02 02:28:16 firestarter kernel: <TASK>
Aug 02 02:28:16 firestarter kernel: asm_sysvec_apic_timer_interrupt+0x19/0x20
Aug 02 02:28:16 firestarter kernel: RIP: 0010:cpuidle_enter_state+0xdc/0x380
Aug 02 02:28:16 firestarter kernel: Code: 00 00 31 ff e8 85 b5 7e ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 92 02 00 00 31 ff e8 ca a2 84 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 ce 48 8d 04 49 48 8d 04 81 49 8d
Aug 02 02:28:16 firestarter kernel: RSP: 0018:ffffa2160018be90 EFLAGS: 00000246
Aug 02 02:28:16 firestarter kernel: RAX: ffff8e4120772cc0 RBX: 0000000000000002 RCX: 0000000000000000
Aug 02 02:28:16 firestarter kernel: RDX: 00000b1386b41081 RSI: fffffffdd9cf14c7 RDI: 0000000000000000
Aug 02 02:28:16 firestarter kernel: RBP: ffff8e412077e100 R08: 0000000000000000 R09: 0000000034e8f93a
Aug 02 02:28:16 firestarter kernel: R10: 0000000000000018 R11: 0000000000006903 R12: ffffffffbd54be00
Aug 02 02:28:16 firestarter kernel: R13: 00000b1386b41081 R14: 0000000000000002 R15: 0000000000000000
Aug 02 02:28:16 firestarter kernel: cpuidle_enter+0x2d/0x40
Aug 02 02:28:16 firestarter kernel: do_idle+0x1ba/0x220
Aug 02 02:28:16 firestarter kernel: cpu_startup_entry+0x1d/0x20
Aug 02 02:28:16 firestarter kernel: start_secondary+0x11c/0x140
Aug 02 02:28:16 firestarter kernel: secondary_startup_64_no_verify+0xd5/0xdb
Aug 02 02:28:16 firestarter kernel: </TASK>
Aug 02 02:28:16 firestarter kernel: ---[ end trace 0000000000000000 ]---

Aug 02 10:07:20 firestarter kernel: ------------[ cut here ]------------
Aug 02 10:07:20 firestarter kernel: NETDEV WATCHDOG: netUSBInet0 (ax88179_178a): transmit queue 0 timed out
Aug 02 10:07:20 firestarter kernel: WARNING: CPU: 5 PID: 0 at net/sched/sch_generic.c:529 dev_watchdog+0x20b/0x220
Aug 02 10:07:20 firestarter kernel: Modules linked in: ch341 tun xt_LOG nf_log_syslog xt_limit xt_addrtype veth ext4 crc16 mbcache jbd2 dm_crypt cbc encrypted_keys trusted asn1_encoder tee xt_MASQUERADE ipt_REJECT nf_reject_ipv4 xt_recent xt_tcpudp xt_conntrack iptable_mangle iptable_nat nf_nat nf_conntrack nf_def>
Aug 02 10:07:20 firestarter kernel: tpm_tis vfat fat tpm_tis_core tpm rng_core mac_hid acpi_tad acpi_pad usbip_host usbip_core crypto_user fuse bpf_preload ip_tables x_tables usbhid btrfs blake2b_generic libcrc32c crc32c_generic xor raid6_pq dm_mod sdhci_pci cqhci sdhci nvme xhci_pci mmc_core crc32c_intel nvme_co>
Aug 02 10:07:20 firestarter kernel: CPU: 5 PID: 0 Comm: swapper/5 Tainted: G U 5.18.15-arch1-1 #1 9ff3be2e7813d5f2c07119812e1642852fe6c646
Aug 02 10:07:20 firestarter kernel: Hardware name: Intel(R) Client Systems NUC11PAHi5/NUC11PABi5, BIOS PATGL357.0046.2022.0608.1909 06/08/2022
Aug 02 10:07:20 firestarter kernel: RIP: 0010:dev_watchdog+0x20b/0x220
Aug 02 10:07:20 firestarter kernel: Code: ff e9 40 ff ff ff 48 89 df c6 05 56 00 3f 01 01 e8 ea 74 f9 ff 44 89 e9 48 89 de 48 c7 c7 60 d0 d6 84 48 89 c2 e8 b2 60 19 00 <0f> 0b e9 22 ff ff ff 66 66 2e 0f 1f 84 00 00 00 00 00 0f 1f 00 f3
Aug 02 10:07:20 firestarter kernel: RSP: 0018:ffffab62802a8e90 EFLAGS: 00010282
Aug 02 10:07:20 firestarter kernel: RAX: 0000000000000000 RBX: ffff95a60145a000 RCX: 000000000000083f
Aug 02 10:07:20 firestarter kernel: RDX: 0000000000000000 RSI: 00000000000000f6 RDI: 000000000000083f
Aug 02 10:07:20 firestarter kernel: RBP: ffff95a60145a4c8 R08: 0000000000000000 R09: ffffab62802a8ca0
Aug 02 10:07:20 firestarter kernel: R10: 0000000000000003 R11: ffffffff854caa08 R12: ffff95a60145a41c
Aug 02 10:07:20 firestarter kernel: R13: 0000000000000000 R14: ffffffff841d5a80 R15: ffff95a60145a4c8
Aug 02 10:07:20 firestarter kernel: FS: 0000000000000000(0000) GS:ffff95a9a0740000(0000) knlGS:0000000000000000
Aug 02 10:07:20 firestarter kernel: CS: 0010 DS: 0000 ES: 0000 CR0: 0000000080050033
Aug 02 10:07:20 firestarter kernel: CR2: 000055f54006f748 CR3: 0000000369e10006 CR4: 0000000000f70ee0
Aug 02 10:07:20 firestarter kernel: PKRU: 55555554
Aug 02 10:07:20 firestarter kernel: Call Trace:
Aug 02 10:07:20 firestarter kernel: <IRQ>
Aug 02 10:07:20 firestarter kernel: ? pfifo_fast_reset+0x140/0x140
Aug 02 10:07:20 firestarter kernel: call_timer_fn+0x24/0x130
Aug 02 10:07:20 firestarter kernel: __run_timers+0x21c/0x2a0
Aug 02 10:07:20 firestarter kernel: run_timer_softirq+0x1d/0x40
Aug 02 10:07:20 firestarter kernel: __do_softirq+0xd0/0x2c9
Aug 02 10:07:20 firestarter kernel: ? sched_clock_cpu+0xd/0xb0
Aug 02 10:07:20 firestarter kernel: __irq_exit_rcu+0x8e/0xc0
Aug 02 10:07:20 firestarter kernel: sysvec_apic_timer_interrupt+0x72/0x90
Aug 02 10:07:20 firestarter kernel: </IRQ>
Aug 02 10:07:20 firestarter kernel: <TASK>
Aug 02 10:07:20 firestarter kernel: asm_sysvec_apic_timer_interrupt+0x19/0x20
Aug 02 10:07:20 firestarter kernel: RIP: 0010:cpuidle_enter_state+0xdc/0x380
Aug 02 10:07:20 firestarter kernel: Code: 00 00 31 ff e8 85 b5 7e ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 92 02 00 00 31 ff e8 ca a2 84 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 ce 48 8d 04 49 48 8d 04 81 49 8d
Aug 02 10:07:20 firestarter kernel: RSP: 0018:ffffab628018be90 EFLAGS: 00000246
Aug 02 10:07:20 firestarter kernel: RAX: ffff95a9a0772cc0 RBX: 0000000000000002 RCX: 0000000000000000
Aug 02 10:07:20 firestarter kernel: RDX: 000001fdcd77c61e RSI: fffffffdf7e09a11 RDI: 0000000000000000
Aug 02 10:07:20 firestarter kernel: sysvec_apic_timer_interrupt+0x72/0x90
Aug 02 10:07:20 firestarter kernel: __irq_exit_rcu+0x8e/0xc0
Aug 02 10:07:20 firestarter kernel: sysvec_apic_timer_interrupt+0x72/0x90
Aug 02 10:07:20 firestarter kernel: </IRQ>
Aug 02 10:07:20 firestarter kernel: <TASK>
Aug 02 10:07:20 firestarter kernel: asm_sysvec_apic_timer_interrupt+0x19/0x20
Aug 02 10:07:20 firestarter kernel: RIP: 0010:cpuidle_enter_state+0xdc/0x380
Aug 02 10:07:20 firestarter kernel: Code: 00 00 31 ff e8 85 b5 7e ff 45 84 ff 74 16 9c 58 0f 1f 40 00 f6 c4 02 0f 85 92 02 00 00 31 ff e8 ca a2 84 ff fb 0f 1f 44 00 00 <45> 85 f6 0f 88 25 01 00 00 49 63 ce 48 8d 04 49 48 8d 04 81 49 8d
Aug 02 10:07:20 firestarter kernel: RSP: 0018:ffffab628018be90 EFLAGS: 00000246
Aug 02 10:07:20 firestarter kernel: RAX: ffff95a9a0772cc0 RBX: 0000000000000002 RCX: 0000000000000000
Aug 02 10:07:20 firestarter kernel: RDX: 000001fdcd77c61e RSI: fffffffdf7e09a11 RDI: 0000000000000000
Aug 02 10:07:20 firestarter kernel: RBP: ffff95a9a077e100 R08: 0000000000000000 R09: 0000000034e8f93a
Aug 02 10:07:20 firestarter kernel: R10: 0000000000000018 R11: 0000000000000457 R12: ffffffff8554be00
Aug 02 10:07:20 firestarter kernel: R13: 000001fdcd77c61e R14: 0000000000000002 R15: 0000000000000000
Aug 02 10:07:20 firestarter kernel: cpuidle_enter+0x2d/0x40
Aug 02 10:07:20 firestarter kernel: do_idle+0x1ba/0x220
Aug 02 10:07:20 firestarter kernel: cpu_startup_entry+0x1d/0x20
Aug 02 10:07:20 firestarter kernel: start_secondary+0x11c/0x140
Aug 02 10:07:20 firestarter kernel: secondary_startup_64_no_verify+0xd5/0xdb
Aug 02 10:07:20 firestarter kernel: </TASK>
Aug 02 10:07:20 firestarter kernel: ---[ end trace 0000000000000000 ]---


Downgrading to 5.18.14 fixes the issue

Additional info:
* package version(s)
linux-5.18.15.arch1-1

* config and/or log files etc.

* link to upstream bug report, if any
no

Steps to reproduce:
Happens after a few minutes to a few hours, traffic seems to increase the chance.
   klog.txt (99.1 KiB)
This task depends upon

Closed by  Toolybird (Toolybird)
Monday, 29 August 2022, 20:41 GMT
Reason for closing:  Fixed
Additional comments about closing:  linux 5.19.2.arch1-1
Comment by Jay Mann (jmandawg) - Wednesday, 03 August 2022, 17:58 GMT
I'm having the same issue. I posted in the forums as well:

https://bbs.archlinux.org/viewtopic.php?id=278526
Comment by Andreas (poison) - Wednesday, 03 August 2022, 18:20 GMT
Could it really be the commit that claims to solve this issue is actually causing it?
https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=36a15e1cb134c0395261ba1940762703f778438c
Comment by Jay Mann (jmandawg) - Wednesday, 03 August 2022, 21:10 GMT
Tough to say, when i look at the archlinux source code on github for ax88179_178a.c on branch 5.18.15 there is a weird meesage at the top that states:

This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.

https://github.com/archlinux/linux/blob/v5.18.15-arch1/drivers/net/usb/ax88179_178a.c
Comment by loqs (loqs) - Wednesday, 03 August 2022, 21:28 GMT
You could try the kernel from https://bbs.archlinux.org/viewtopic.php?pid=2050150#p2050150 which is prebuilt with the commit reverted.
Comment by Andreas (poison) - Thursday, 04 August 2022, 10:46 GMT
lsusb -vvv
   lsusb.txt (39.5 KiB)
Comment by Jay Mann (jmandawg) - Thursday, 04 August 2022, 11:52 GMT
@loqs your kernel with reverted patch is still working without issues.

I'm wondering why these 2 commits from j-alonso were added to archlinux 5.18 kernel but are not in mainline kernel 5.18:

Arch:
https://github.com/archlinux/linux/commits/v5.18.15-arch1/drivers/net/usb/ax88179_178a.c

Mainline:
https://github.com/torvalds/linux/commits/v5.18/drivers/net/usb/ax88179_178a.c

They are in mainline 5.19, but there is a preceeding commit that may be relevant, which is missing from Arch 5.18.

Comment by Mark Blakeney (bulletmark) - Thursday, 04 August 2022, 12:14 GMT
I experience a problem similar to this as I report in the forum thread https://bbs.archlinux.org/viewtopic.php?pid=2050279#p2050279. My problem is also fixed by logs modified kernel.
Comment by loqs (loqs) - Thursday, 04 August 2022, 18:35 GMT
I would suggest opening a bug report upstream on [1] Product Drivers Component Network be sure to list the commit that introduced the issue and add Jose Alonso <joalonsof@gmail.com> to the CC list.

The commit in mainline is [1] which was backported to 5.18.15 as [2] and 5.15.58 as [3] from a checkout of linux stable you can cross check this with:

$ git tag --contains f8bfce2177cdd64357dd57b8f43154f236d25f51
v5.18.15
v5.18.16
$ git tag --contains b34229f4b212367196d787170b02be6f31802622
v5.15.58
v5.15.59

Edit:
You could also try linux-mainline 5.19 which you can obtain prebuilt from [5].

[1] https://bugzilla.kernel.org/
[2] https://github.com/torvalds/linux/commit/36a15e1cb134c0395261ba1940762703f778438c
[3] https://github.com/gregkh/linux/commit/f8bfce2177cdd64357dd57b8f43154f236d25f51
[4] https://github.com/gregkh/linux/commit/b34229f4b212367196d787170b02be6f3180262
[5] https://wiki.archlinux.org/title/Unofficial_user_repositories#miffe
Comment by Mark Blakeney (bulletmark) - Friday, 05 August 2022, 00:46 GMT
@loqs, not sure if you mean the OP Andreas, or Jay, or myself to raise that bug? I am hesitant to do that while the last question from Jay above is outstanding. He is querying whether this may be due to Arch applying those two 5.19 commits prematurely to 5.18, or because Arch is missing commit https://github.com/torvalds/linux/commit/ee8b7a1156f357613646d6c69d07ac5a087a1071.
Comment by loqs (loqs) - Friday, 05 August 2022, 01:57 GMT
The commits are not part of 5.18 they are part of 5.18.15 and 5.18.16 as I tried to demonstrate in my last post links showed the backports were committed by gregkh and part of upstream tags signed by gregkh. You can find the same commit f8bfce2177cdd64357dd57b8f43154f236d25f51 listed in [1] and b34229f4b212367196d787170b02be6f31802622 listed in [2] as published on kernel.org. [3] shows the additional commits added for 5.18.15-arch1 and [4] for 5.18.16-arch1 which does not contain anything related to ax88179_178a

[1] https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.18.15
[2] https://cdn.kernel.org/pub/linux/kernel/v5.x/ChangeLog-5.15.58
[3] https://github.com/archlinux/linux/compare/3740a5da82ebec7a6d8f3a6deea77b8129c8c2ee...v5.18.15-arch1
[4] https://github.com/archlinux/linux/compare/8843bf1f0737ecea456d2bbd19d4263d49f2d110...v5.18.16-arch1
Comment by Mark Blakeney (bulletmark) - Friday, 05 August 2022, 06:31 GMT
Sorry guys, my issue is not fixed by loqs kernel as my testing was invalid. So my issue is independent to the issue described here.
Comment by Andreas (poison) - Friday, 05 August 2022, 07:27 GMT
Problem still exists in 5.18.16.

Can confirm that reverting commit f8bfce2177cdd64357dd57b8f43154f236d25f51 in 5.18.15/5.18.16 fixes the issue.
   klog-5.18.16.txt (96.2 KiB)
Comment by loqs (loqs) - Friday, 05 August 2022, 07:50 GMT
Upstream bug report https://bugzilla.kernel.org/show_bug.cgi?id=216327
Comment by Toolybird (Toolybird) - Thursday, 11 August 2022, 05:52 GMT
It appears "drivers/net/usb/ax88179_178a.c" has received some commits on linus mainline (but not in 5.19.arch1-1) so if someone affected is up for the challenge....you know what to do.
Comment by loqs (loqs) - Monday, 15 August 2022, 19:32 GMT
The revert is queued for 5.19.2 https://git.kernel.org/pub/scm/linux/kernel/git/stable/stable-queue.git/commit/queue-5.19/revert-net-usb-ax88179_178a-needs-flag_send_zlp.patch?id=4cb76883a4e5618020a6d05de297d1a4650d0655
Comment by loqs (loqs) - Sunday, 28 August 2022, 03:33 GMT
Anyone affected by the issue able to confirm it has been resolved with 5.19.2?
Comment by Andreas (poison) - Monday, 29 August 2022, 19:55 GMT
@loqs Cannot reproduce it anymore after a week, so I assume it's fixed :)

Loading...