FS#64189 - [cockpit] Browser uses a lot of CPU and cockpit is unresponsive

Attached to Project: Community Packages
Opened by rainer (raneon) - Sunday, 20 October 2019, 16:46 GMT
Last edited by Massimiliano Torromeo (mtorromeo) - Saturday, 26 October 2019, 15:51 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Massimiliano Torromeo (mtorromeo)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Since the upgrade to cockpit 205 the webinterface is extremely unresponsive and it consumes a lot of CPU. I've tested this with Firefox and Falkon, both have the same issues. Closing Firefox is not enough, you need to even kill it's web processes.


Steps to reproduce:
Login to cockpit
And the e.g. try to enter ssh key path or password
This task depends upon

Closed by  Massimiliano Torromeo (mtorromeo)
Saturday, 26 October 2019, 15:51 GMT
Reason for closing:  Fixed
Additional comments about closing:  cockpit-205.1-1
Comment by Massimiliano Torromeo (mtorromeo) - Wednesday, 23 October 2019, 16:44 GMT
It's not clear to me if it's been reported on cockpit-dashboard because the dashboard plugin is unresponsive or if it's cockpit in general to expose the behavior. Can you test it again without cockpit-dashboard installed?
Comment by rainer (raneon) - Wednesday, 23 October 2019, 21:52 GMT
What would I test without the cockpit-dashboard in a browser like Firefox? I see this issue only if I'm using the web front-end. It's quite impossible for me to tell if there is an issue in cockpit itself or not. I've tested it again an Firefox got completely unresponsive directly after login and it consumed 100% of 5 CPU cores. Do you have the same issue or is your installation working fine?
Comment by rainer (raneon) - Wednesday, 23 October 2019, 22:00 GMT
Is this maybe related to some security hardening mentioned in the blog post for cockpit 205? Maybe user interaction is required to transfer existing installs?
Comment by Massimiliano Torromeo (mtorromeo) - Wednesday, 23 October 2019, 22:05 GMT
As I thought you are misinterpreting the functionality provided by the cockpit-dashboard plugin. It is not the whole cockpit interface. You can run cockpit in the browser without it just fine. It is a multi-server dashboard. Basically this[1], but it doesn't look exactly like that anymore.

I can't test it right now but I will probably tomorrow.

[1] https://cockpit-project.org/blog/cockpit-multi-server-dashboard.html
Comment by rainer (raneon) - Thursday, 24 October 2019, 16:22 GMT
I have one system that is running the dashboard. And to this server 20 machines are connected. So yes, if I login to cockpit then the first thing I see is this multiserver dashboard. I then enter a ssh key and password to connect to the other machines. Usually I would be able to select then a single machine to check the disk or use the terminal, but all this doesn't work anymore.

To make cockpit and the dashboard work again, I had to downgrade both packages. So something is seriously wrong with this upgrade. I will check in the cockpit bug tracker if I can find some additional info.
Comment by Massimiliano Torromeo (mtorromeo) - Thursday, 24 October 2019, 18:15 GMT
My cockpit is responsive both in firefox and in chromium. Is there some specific action required to trigger the behavior?
Also, it's very unclear what process specifically is using the CPU.
Comment by rainer (raneon) - Thursday, 24 October 2019, 19:05 GMT
Did you test cockpit with the dashboard and some remote machines? And is your setup migrated from cockpit 204 to 205 with the "cockpit.socket" enabled systemd service?

Let me quote myself again: "Steps to reproduce:
Login to cockpit
And then e.g. try to enter ssh key path or password"

So there is nothing special that I have to do... I just use the dashboard with several machines that are connected but they need the ssh key and password.

Next quote: "Closing Firefox is not enough, you need to even kill it's web processes" => web is a process of Firefox.

Comment by rainer (raneon) - Thursday, 24 October 2019, 19:15 GMT
I did a test with chromium, it works a little better, 130% CPU Usage, but I was as well not able to enter my ssh key and connect to a remote machine. The local cockpit interface was partially unresponsive too, cockpit 204 and 205 are a night and day difference.
Comment by rainer (raneon) - Thursday, 24 October 2019, 19:38 GMT
Did some further testing. Cockpit 204 and cockpit-dashboard 204 work perfectly fine. If I upgrade cockpit-dashboard to 205 but keep cockpit 204 everything still seems to work fine. Only when I upgrade cockpit to 205 then I have the mentioned issues. In journalctl I found some error messages:

Okt 24 21:24:14 pc1 cockpit-ws[284713]: Error sending data: Broken pipe
Okt 24 21:24:14 pc1 cockpit-ws[284713]: WebSocket from (null) for session closed
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received out of order middle message fragment
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received out of order inital message fragment
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received fragmented control frame
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received fragmented control frame
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received fragmented control frame
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received out of order inital message fragment
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received unfragmented message when fragment was expected
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received fragmented control frame
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: received out of order inital message fragment
Okt 24 21:24:31 pc1 cockpit-ws[284713]: Received invalid WebSocket response from the server
Okt 24 21:24:31 pc1 cockpit-ws[284713]: WebSocket from (null) for session closed

Could this be related to the security hardending mentioned here? https://cockpit-project.org/blog/cockpit-205.html
Maybe an existing installs require user intervention...
Comment by Massimiliano Torromeo (mtorromeo) - Thursday, 24 October 2019, 20:37 GMT
cockpit-205-1 did not even implement the security hardening changes that upstream did in their package.
The option is behind a configure flag, meaning it's optional and it's supposed to work just as before.
I did intend to follow upstream anyway, so I just did, and I released cockpit-205-2.
Feel free to test it but I doubt it will make any difference. It works fine for me though.
I suggest to report upstream if you still experience those issues.
Comment by rainer (raneon) - Friday, 25 October 2019, 17:56 GMT
Cockpit 205-2 didn't change anything, all the browsers I've tested still behave badly when opening cockpit.

I understand that the focus here is on packaging. I assume that your setup is a single client installation, probably that's why it works. Thanks you
Comment by Massimiliano Torromeo (mtorromeo) - Friday, 25 October 2019, 18:01 GMT
I tested with multiple servers (not 20 though) and it still works fine for me.
Comment by rainer (raneon) - Saturday, 26 October 2019, 09:55 GMT
I did some more tests, now on other systems to avoid that my setup is broken. What I've found out is that cockpit (activated with "systemctl start cockpit.socket") works fine in Firefox, if I open it via localhost:9090. In this case it operates normally and I can use all the functionalities.

But as soon as I access it via the IP (for example 192.168.1.101:9090) it starts to behave strange. I login and will be kicked out again. So I press the button to reconnect, but same behavior, I will be kicked out again.

I've tried this with IPv4 and IPv6, same issue.

Would you please do me a favor and test this as well? With this single machine setup I do not believe I'm doing anything special now.
Comment by rainer (raneon) - Saturday, 26 October 2019, 10:54 GMT
I've opened a bug report for cockpit: https://github.com/cockpit-project/cockpit/issues/13042
Comment by Massimiliano Torromeo (mtorromeo) - Saturday, 26 October 2019, 11:09 GMT
With your latest details I was able to reproduce the issue.

Loading...