FS#63397 - [networkmanager] v1.20 Connecting to WiFi fails with reason=WRONG_KEY

Attached to Project: Arch Linux
Opened by François Guerraz (kubrick) - Wednesday, 07 August 2019, 07:50 GMT
Last edited by freswa (frederik) - Saturday, 22 February 2020, 20:47 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

Description:
After upgrading to v1.20.0, connecting to WPA PSK network fails with reason=WRONG_KEY. Downgrading to 1.18.2 fixes the problem.

Hardware is DELL XPS 9350 with BCM4350.
Up to date gnome & kernel 5.2.7.
This task depends upon

Closed by  freswa (frederik)
Saturday, 22 February 2020, 20:47 GMT
Reason for closing:  Upstream
Additional comments about closing:  https://bugzilla.kernel.org/show_bug.cgi ?id=206315
Comment by Thomas Haller (thom311) - Thursday, 08 August 2019, 13:01 GMT
Can you provide a level=TRACE file showing the issue?

See the comment at https://cgit.freedesktop.org/NetworkManager/NetworkManager/tree/contrib/fedora/rpm/NetworkManager.conf#n28 for how to get the logfile. Note also the comment about private data and journal's rate-limiting.

Thank you.
Comment by François Guerraz (kubrick) - Thursday, 08 August 2019, 13:52 GMT
Here is the log with TRACE level when I try to connect to the COMPANY network.
I hope it helps.
Comment by Thomas Haller (thom311) - Thursday, 08 August 2019, 14:25 GMT
> Config: added 'key_mgmt' value 'WPA-PSK WPA-PSK-SHA256 FT-PSK'

It might be related to NM now enabling fast BSS transition.

Can you provide debug logs from wpa_supplicant?

Not sure how to enable that best... I'd probably edit wpa_supplicant.service file and add "-ddd" to the arguments, then restart supplicant and retry.
Comment by François Guerraz (kubrick) - Thursday, 08 August 2019, 15:44 GMT
I have rebuilt wpa_supplicant with CONFIG_IEEE80211R=n and that fixed the issue so fast BSS transition is indeed the culprit.
Comment by Thomas Haller (thom311) - Thursday, 08 August 2019, 15:52 GMT
still, NM should handle that gracefully.

In your log you see

<debug> [1565271673.0273] supplicant: FT is supported

so, NM was thinking that supplicant supports it.

Before rebuild, what did supplicant say about "Capabilities"?

busctl get-property fi.w1.wpa_supplicant1 /fi/w1/wpa_supplicant1 fi.w1.wpa_supplicant1 Capabilities

Comment by François Guerraz (kubrick) - Thursday, 08 August 2019, 16:11 GMT
[edited, I had also inadvertently downgraded wpa_supplicant]

before the rebuild:
as 6 "ap" "ibss-rsn" "p2p" "interworking" "pmf" "ft"

after:
as 5 "ap" "ibss-rsn" "p2p" "interworking" "pmf"
Comment by François Guerraz (kubrick) - Friday, 09 August 2019, 08:45 GMT
Do you have a valid GPG key that I can use to send you the wpa_supplicant debug log?
Comment by Thomas Haller (thom311) - Friday, 09 August 2019, 08:59 GMT
Please use https://tynq.net/thaller-at-redhat.com.asc (with the respective e-mail address).
It is the same key, with which I signed the last NetworkManager 1.20.0 release: https://gitlab.freedesktop.org/NetworkManager/NetworkManager/-/tags/1.20.0
Comment by Thomas Haller (thom311) - Monday, 19 August 2019, 13:57 GMT
@François, thank you for the logfile.

it's not clear why association fails.


I presume this is still about "Hardware is DELL XPS 9350 with BCM4350.".

It's a bit of a stretch, but that driver had problems in the past with changing the MAC address.
Would it be possible to retry without NetworkManager ever changing the MAC address?

For that, drop a file /etc/NetworkManager/conf.d/99-don-t-touch-mac.conf

[connection-99-don-t-touch-mac]
wifi.scan-rand-mac-address=no
wifi.cloned-mac-address=preserve

and ensure that no relevant profile overrides the global default:

$ for UUID in $(nmcli -g TYPE,UUID connection show | sed -n 's/^802-11-wireless://p'); do nmcli -g 802-11-wireless.cloned-mac-address connection show uuid "$UUID" | grep -q '.' || continue; echo ">>> profile $UUID override cloned-mac-address: "; nmcli -f connection.uuid,connection.id,802-11-wireless.cloned-mac-address connection show "$UUID"; done

Then reboot and reproduce. Usually reboot is not necessary for such changes, in this case it's useful to ensure that NM never touched the MAC address since the machine is running. If you grep the debug log for "set-hw-addr:", it would confirm that NM indeed does not touch the MAC address.




Also, do you know more about this AP? What model/vendor?


Thank you for bearing with me :) !!

Comment by François Guerraz (kubrick) - Monday, 19 August 2019, 20:41 GMT
I'll try tomorrow but I think I already disabled it (because it's inconvenient).
Comment by Thomas Haller (thom311) - Tuesday, 20 August 2019, 14:28 GMT
Could you also show the output of

# busctl -j get-property fi.w1.wpa_supplicant1 /fi/w1/wpa_supplicant1/Interfaces/1 fi.w1.wpa_supplicant1.Interface Capabilities | cat

(where you need to specify the right interface name on the command line, see `busctl tree fi.w1.wpa_supplicant1` for that).


The question is which "KeyMgnt" capabilities get advertised for the device...
Comment by François Guerraz (kubrick) - Wednesday, 28 August 2019, 14:54 GMT
Apologies for the late reply:

Result of the Capabilities query:
{
"type": "a{sv}",
"data": {
"Pairwise": {
"type": "as",
"data": [
"ccmp",
"tkip"
]
},
"Group": {
"type": "as",
"data": [
"ccmp",
"tkip",
"wep104",
"wep40"
]
},
"GroupMgmt": {
"type": "as",
"data": [
"aes-128-cmac"
]
},
"KeyMgmt": {
"type": "as",
"data": [
"none",
"ieee8021x",
"wpa-eap",
"wpa-ft-eap",
"wpa-eap-sha256",
"wpa-psk",
"wpa-ft-psk",
"wpa-psk-sha256",
"wps"
]
},
"Protocol": {
"type": "as",
"data": [
"rsn",
"wpa"
]
},
"AuthAlg": {
"type": "as",
"data": [
"open",
"shared",
"leap"
]
},
"Scan": {
"type": "as",
"data": [
"active",
"passive",
"ssid"
]
},
"Modes": {
"type": "as",
"data": [
"infrastructure",
"ad-hoc",
"ap",
"p2p"
]
},
"MaxScanSSID": {
"type": "i",
"data": 10
}
}
}
Comment by Thomas Haller (thom311) - Wednesday, 28 August 2019, 14:57 GMT
> "KeyMgmt": {
> "type": "as",
> "data": [
> "none",
> "ieee8021x",
> "wpa-eap",
> "wpa-ft-eap",
> "wpa-eap-sha256",
> "wpa-psk",
> "wpa-ft-psk",
> "wpa-psk-sha256",
> "wps"
> ]
> },

OK, that means https://cgit.freedesktop.org/NetworkManager/NetworkManager/commit/?id=2f8a4e90f0fd0f900996e3081d49f8799bba4c6f would *not* fix your issue...
Comment by François Guerraz (kubrick) - Wednesday, 28 August 2019, 15:14 GMT
also, I can confirm that the problem persists with disabling random mac addresses...
Comment by François Guerraz (kubrick) - Tuesday, 03 September 2019, 10:24 GMT
Any more clue? Do you suspect a driver issue? Considering that the firmware on linux hasn't been updated in ages I wouldn't be surprised if there was an issue there as well...
Comment by Thomas Haller (thom311) - Tuesday, 03 September 2019, 14:21 GMT
What we got so far:

- disabling FT avoids the issue (either by downgrading to 1.18 or recompiling supplicant with CONFIG_IEEE80211R=n)

- supplicant announces that FT can be used, consequently NM 1.20 wants to use it.

This points to an issue in wpa_supplicant, the driver or the firmware. Possibly it's even the AP's fault, but then supplicant probably should still have a workaround for it.


From NetworkManager's point of view, the question is what the exact cause is, and whether NM can do something to workaround it. We would rather not add an option to disable FT, because it just increases up the number of configuration options for something that should work without explicit configuration. Also, how would an affecter user know how to set such a hypothetical option?
Comment by François Guerraz (kubrick) - Wednesday, 04 September 2019, 07:11 GMT
should we reassign to wpa_supplicant and just go down the stack?
Comment by Beniamino Galvani (bengal) - Wednesday, 04 September 2019, 14:38 GMT
Hi,

which driver are you using? Can you paste the relevant part of lspci and lsmod output?

Also, is there anything related to wifi in the 'dmesg' output when the authentication fails?

Thank you.
Comment by François Guerraz (kubrick) - Tuesday, 24 September 2019, 08:39 GMT
3a:00.0 Network controller: Broadcom Inc. and subsidiaries BCM4350 802.11ac Wireless Network Adapter (rev 08)

Using brcmfmac

But the firmware is antique
[ 14.186006] brcmfmac: brcmf_c_preinit_dcmds: Firmware: BCM4350/8 wl0: Oct 22 2015 06:16:26 version 7.35.180.119 (r594535) FWID 01-e791c176

That might be the problem...
Comment by Matteo Fortini (metiu) - Saturday, 12 October 2019, 07:10 GMT
I can confirm the issue is triggered by NM adding any FT-XXX options to key_mgmt

I attach the output of wpa_supplicant -d

This is also on an XPS 9350 with wpa_supplicant 2.9.3 and Linux 5.3

Configuration which doesn't work:

network={
ssid="myNetwork"
scan_ssid=1
proto=RSN
key_mgmt=WPA-EAP WPA-EAP-SHA256 FT-EAP FT-EAP-SHA384
pairwise=CCMP
auth_alg=OPEN
bgscan="simple:30:-65:300"
eap=PEAP
identity="myuser"
password="*******"
fragment_size=1266
proactive_key_caching=1
}


Configuration which works:

network={
ssid="myNetwork"
scan_ssid=1
proto=RSN
key_mgmt=WPA-EAP
pairwise=CCMP
auth_alg=OPEN
bgscan="simple:30:-65:300"
eap=PEAP
identity="myuser"
password="*******"
fragment_size=1266
proactive_key_caching=1
}


Comment by Daniel Shchur (docquantum) - Wednesday, 06 November 2019, 21:06 GMT
This still exists in 1.20.0-6
Comment by François Guerraz (kubrick) - Monday, 16 December 2019, 16:25 GMT
Hello, can we please reassign to wpa_supplicant? Then if it's not wpa_supplicant, we'll follow up with kernel people?
Comment by François Guerraz (kubrick) - Sunday, 26 January 2020, 16:45 GMT
I have enabled 802.11R on my home network (a couple of openwrt/hostapd based routers) and it does make the association fail (although it works brilliantly with my other devices)
Comment by François Guerraz (kubrick) - Sunday, 26 January 2020, 16:52 GMT Comment by Matteo Fortini (metiu) - Monday, 10 February 2020, 09:22 GMT
For FT networks the patch by Matthew Wang fixes the problem.
https://w1.fi/cgit/hostap/commit/?id=23dc196fde951b3d508f367a603cddffbd053490

Loading...