FS#61433 - [mariadb] galera_recovery killed by OOM with systemd >= 240

Attached to Project: Arch Linux
Opened by Cristian Bradiceanu (cbredi) - Wednesday, 16 January 2019, 20:18 GMT
Last edited by Christian Hesse (eworm) - Thursday, 24 January 2019, 14:49 GMT
Task Type Bug Report
Category Upstream Bugs
Status Closed
Assigned To Christian Hesse (eworm)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

Description:

After upgrading to systemd-240 mariadb would not start as galera_recovery is killed by OOM.
Tried to set OOMScoreAdjust=-900 for mariadb.service but same problem persists. With OOMScoreAdjust=-999 the system becomes unstable or crashes.
Downgrading to systemd-239.370 fixes this issue.

Additional info:
Confirmed affected on two 3-server mariadb + galera clusters with systemd 240.0-3, 240.34-2 and 240.34-3

Steps to reproduce:
Upgrade to systemd-240, after reboot mariadb (+galera) would not start:
/usr/bin/galera_recovery: line 71: 8976 Killed ./bin/mysqld --user=mysql --wsrep_recover --disable-log-error'

Downgrade to systemd 239.370 and mariadb (+galera) will start successfully.

I have added libsystemd systemd systemd-sysvcompat to IgnorePkg, this causes various systemd binaries to stop working with error while loading shared libraries: libidn2.so.0: cannot open shared object file: No such file or directory
Downgrading to libidn2-2.0.5-1 fixes systemd-239.370 but then pacman stopped working with pacman: error while loading shared libraries: libidn2.so.4: cannot open shared object file: No such file or directory so I ended up copying libidn2.so.0.3.4 from libidn2-2.0.5-1 to /lib to keep both systemd and pacman working after upgrading to libidn2-2.1.0-1
This task depends upon

Closed by  Christian Hesse (eworm)
Thursday, 24 January 2019, 14:49 GMT
Reason for closing:  Fixed
Additional comments about closing:  mariadb 10.3.12-4
Comment by Anthony K. (crt.011) - Saturday, 19 January 2019, 23:49 GMT
Thank you for reporting this. I was also running into this same problem after the recent systemd updates. I can confirm mariadb + galera starts after downgrading systemd.
Comment by AK (Andreaskem) - Sunday, 20 January 2019, 09:51 GMT Comment by Cristian Bradiceanu (cbredi) - Sunday, 20 January 2019, 16:00 GMT
Thank you AK for your comment, can’t seem to find a way to revert to systemd’s old behaviour to try to confirm your guess.
Comment by AK (Andreaskem) - Sunday, 20 January 2019, 16:19 GMT
Not sure, but as with "OOMScoreAdjust" there is a Parameter "LimitNOFILE". Maybe try setting it to 4096?
Comment by Cristian Bradiceanu (cbredi) - Sunday, 20 January 2019, 16:32 GMT
Thank you again AK for your help, LimitNOFILE=16364 is already in mariadb.service.
With LimitNOFILE=4096 mariadb + galera still won’t start under systemd 240.
Comment by loqs (loqs) - Sunday, 20 January 2019, 16:58 GMT
Have you tried bisecting systemd between v239 and v240 to find the causal commit?
Comment by Cristian Bradiceanu (cbredi) - Monday, 21 January 2019, 11:09 GMT
I have also raised this issue at https://github.com/systemd/systemd/issues/11510
Comment by Johannes Ernst (jernst) - Wednesday, 23 January 2019, 01:03 GMT
I ran into something similar on new installs: https://bbs.archlinux.org/viewtopic.php?pid=1828172#p1828172
Comment by loqs (loqs) - Wednesday, 23 January 2019, 13:42 GMT Comment by Christian Hesse (eworm) - Wednesday, 23 January 2019, 15:25 GMT
Created a bug report for MariaDB:
https://jira.mariadb.org/browse/MDEV-18360
Comment by Christian Hesse (eworm) - Wednesday, 23 January 2019, 23:36 GMT
Please try mariadb 10.3.12-4 from [testing].
Comment by Cristian Bradiceanu (cbredi) - Thursday, 24 January 2019, 09:31 GMT
mariadb 10.3.12-4 from [testing] + galera now successfully starts on systemd 240, thank you very much

Loading...