FS#76735 - [monero] monerod: high cpu usage

Attached to Project: Community Packages
Opened by Leon Möller (jkhsjdhjs) - Sunday, 04 December 2022, 00:21 GMT
Last edited by kpcyrd (kpcyrd) - Sunday, 15 October 2023, 20:02 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To kpcyrd (kpcyrd)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
Some time after startup (usually 1-3 days), monerod starts to consume a high amount of CPU time on two threads (each thread 80+% of a Ryzen 5950X CPU thread).
To at least get any info on what is causing it I built the package myself with debug symbols enabled by adding the options !strip and debug to the PKGBUILD and changing CMAKE_BUILD_TYPE from release to debug.
Output of perf top -a and htop: https://gist.github.com/jkhsjdhjs/03a1f5a8c7198de2463ca7c9068898bd

Since the perf top output contains some SSL related stuff I thought that it might be related to this package being linked against OpenSSL 3, while the official binaries are linked against OpenSSL 1.1. Thus I built monerod with OpenSSL 1.1 myself by exporting PKG_CONFIG_PATH=/usr/lib/openssl-1.1/pkgconfig in the build() function of the PKGBUILD.
But the same issue appeared: https://gist.github.com/jkhsjdhjs/4f46c15961f8163f41405696539682db
This package is also linked against unbound, which is currently built with OpenSSL 3, so in the second build monerod was linked against OpenSSL 3 and OpenSSL 1.1. I'm not sure if I'd also have to rebuild unbound with OpenSSL 1.1 to make sure it's not a compatibility issue with OpenSSL 3.

The log file doesn't indicate anything useful. My config is available here: https://gist.github.com/jkhsjdhjs/d6d14560f9e697ed595749946cd49dbc

I haven't been able to reproduce this issue with the official binary (yet, maybe I didn't wait long enough in the last attempt). I'm currently trying to do exactly that again.


Additional info:
* package version(s) 0.18.1.2-2
This task depends upon

Closed by  kpcyrd (kpcyrd)
Sunday, 15 October 2023, 20:02 GMT
Reason for closing:  Fixed
Additional comments about closing:  0.18.3.1-3
Comment by Toolybird (Toolybird) - Sunday, 04 December 2022, 00:51 GMT Comment by Leon Möller (jkhsjdhjs) - Sunday, 04 December 2022, 01:01 GMT
I haven't yet, and it looks like it could be the same issue. I'll try to reproduce it with the official binary and if I'm able to, hopefully provide useful debugging info upstream.
I also just noticed that I leaked my RPC password when I posted my config :D
Comment by Leon Möller (jkhsjdhjs) - Sunday, 18 December 2022, 23:19 GMT
I've been running the official binary for over 2 weeks now and can conclude, that this issue does not occur with the official binary. I'm not sure what could cause the issue with the package though, it has to be some difference during the build.
Comment by Toolybird (Toolybird) - Monday, 19 December 2022, 02:15 GMT
It should be possible to attach gdb to the out-of-control process and see what it's doing. The PM might be willing to build the Arch pkg with debug symbols on?
Comment by Leon Möller (jkhsjdhjs) - Monday, 19 December 2022, 20:50 GMT
I already built the package myself with debug symbols (see initial post). When I ran monerod with debug symbols I only used perf top though, as I don't really know what I'd have to do with gdb. If you can help me using gdb to figure out what's going wrong I'd appreciate that. The perf top results only indicated that something was going wrong with openssl (see issue description).
Comment by Leon Möller (jkhsjdhjs) - Thursday, 22 December 2022, 18:16 GMT Comment by Toolybird (Toolybird) - Monday, 08 May 2023, 21:19 GMT
So it looks like a fix has been proposed...but not applied yet. What's the latest?
Comment by Leon Möller (jkhsjdhjs) - Monday, 08 May 2023, 22:48 GMT
I'm not aware of any further development. The patch works fine for me, no negative reviews on the PR. Upstream generally seems to take its time for merging PRs. Maybe the patch can be applied in the PKGBUILD until it has been merged and incorporated in a release?
Comment by Leon Möller (jkhsjdhjs) - Tuesday, 27 June 2023, 16:53 GMT Comment by Leon Möller (jkhsjdhjs) - Sunday, 15 October 2023, 19:58 GMT
0.18.3.1 has the fix, this can be closed.

Loading...