FS#59795 - [ejabberd] systemctl stop causes crash, manual shutdown is clean

Attached to Project: Community Packages
Opened by Uwe Sauter (UweSauter) - Friday, 24 August 2018, 16:28 GMT
Last edited by Sergej Pupykin (sergej) - Monday, 29 October 2018, 09:36 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sergej Pupykin (sergej)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:

Every time ejabberd 18.06-2 is stopped via "systemctl stop ejabberd" it crashes and an Erlang dump file is saved in /var/log/ejabberd. If ejabberd is started and stopped manually it shuts down clearly.
An update to 18.06-3 causes ejabberd to crash when being started (by systemctl and manually), producing dump files.

Additional info:
* package version(s)
ejabberd 18.06-2


* config and/or log files etc.
This task depends upon

Closed by  Sergej Pupykin (sergej)
Monday, 29 October 2018, 09:36 GMT
Reason for closing:  Fixed
Additional comments about closing:  rebuilt with upstream .service file
Comment by Robert Kormann (RoKoInfo) - Tuesday, 04 September 2018, 10:28 GMT
I can confirm this behavior on a different machine, however, it seems that the crashes when stopping exist for a longer time. The first one I could find on my system is from Sep 2017:

=erl_crash_dump:0.3
Sun Sep 10 19:16:01 2017
Slogan: erl_child_setup closed
System version: Erlang/OTP 20 [erts-9.0.1] [source] [64-bit] [smp:2:2] [ds:2:2:10] [async-threads:10] [hipe] [kernel-poll:true]
Compiled: Fri Jul 21 13:18:12 2017
Taints: ets_cache,asn1rt_nif,jid,stringprep,fxml_stream,fxml,fast_tls,p1_sha,fast_yaml,crypto,erl_tracer
Atoms: 30793
Comment by Uwe Sauter (UweSauter) - Tuesday, 02 October 2018, 18:29 GMT
Updated to 18.09-1 today and the issue still occurs.

After a reboot it produces a new erl_crash file and the actual server process doesn't run, though systemd says the service were running:

# systemctl status ejabberd
● ejabberd.service - A distributed, fault-tolerant Jabber/XMPP server
Loaded: loaded (/usr/lib/systemd/system/ejabberd.service; enabled; vendor preset: disabled)
Active: active (running) since Tue 2018-10-02 20:11:26 CEST; 1min 19s ago
Process: 923 ExecStart=/usr/bin/ejabberdctl start (code=exited, status=0/SUCCESS)
Tasks: 1 (limit: 4680)
Memory: 4.9M
CGroup: /system.slice/ejabberd.service
└─969 /usr/lib/erlang/erts-9.3.3/bin/epmd -daemon

Oct 02 20:11:26 uweb systemd[1]: Starting A distributed, fault-tolerant Jabber/XMPP server...
Oct 02 20:11:26 uweb systemd[1]: Started A distributed, fault-tolerant Jabber/XMPP server.


Also, no ejabberd.log file is created.

Had to downgrade to 18.06-2 to get a running server.
Comment by Sergej Pupykin (sergej) - Wednesday, 03 October 2018, 09:36 GMT
I don't understand yet why, but I was able to start it as root (yes, it is bad idea).
Comment by A. Bosch (progandy) - Wednesday, 03 October 2018, 18:20 GMT
For the crash on stop, the official ejabberd.service template defines a timeout period for the stop action. Maybe the arch service should do that too?

https://github.com/processone/ejabberd/blob/master/ejabberd.service.template
https://git.archlinux.org/svntogit/community.git/tree/trunk/ejabberd.service?h=packages/ejabberd

TimeoutStopSec=200

That has no effect on the startup, though. Maybe a start timeout is necessary as well?
Comment by A. Bosch (progandy) - Wednesday, 03 October 2018, 18:48 GMT
This unit seems to run without crashes with 18.09-1. I got an ejabberd.log, but no data in error.log or crash.log.
Comment by Uwe Sauter (UweSauter) - Monday, 08 October 2018, 18:46 GMT
Using a slightly different ejabberd.service [1] I can start and stop 18.06.2 just fine. But 18.09-1 still crashes on startup. I tried running a live session but that crashes, too.

# HOME=/var/lib/ejabberd sudo -u jabber ejabberdctl live
--------------------------------------------------------------------

IMPORTANT: ejabberd is going to start in LIVE (interactive) mode.
All log messages will be shown in the command shell.
You can interact with the ejabberd node if you know how to use it.
Please be extremely cautious with your actions,
and exit immediately if you are not completely sure.

To exit this LIVE mode and stop ejabberd, press:
q(). and press the Enter key

--------------------------------------------------------------------
To bypass permanently this warning, add to ejabberdctl.cfg the line:
EJABBERD_BYPASS_WARNINGS=true
Press return to continue
/usr/bin/ejabberdctl: line 169: read: read error: 0: Resource temporarily unavailable

Erlang/OTP 20 [erts-9.3.3] [source] [64-bit] [smp:4:4] [ds:4:4:10] [async-threads:10] [hipe] [kernel-poll:true]

{"init terminating in do_boot",{undef,[{ejabberd,start,[],[]},{init,start_em,1,[]},{init,do_boot,3,[]}]}}
init terminating in do_boot ({undef,[{ejabberd,start,[],[]},{init,start_em,1,[]},{init,do_boot,3,[]}]})

Crash dump is being written to: /var/log/ejabberd/erl_crash_20181008-204220.dump...done


Is it possible that new dependencies were added to 18.09 that are missing on my system? BTW: I'm running on an old Atom D525 with 4GB memory. Might this be too small/slow?


[1] ejabberd.service
[Unit]
Description=A distributed, fault-tolerant Jabber/XMPP server
After=network.target

[Service]
Type=forking
User=jabber
Group=jabber
Environment=HOME=/var/lib/ejabberd
LimitNOFILE=65535
Restart=on-failure
RestartSec=5
ExecStart=/usr/bin/ejabberdctl start
ExecStartPost=/usr/bin/ejabberdctl started
ExecStop=/usr/bin/ejabberdctl stop
ExecStopPost=/usr/bin/ejabberdctl stopped
ExecReload=/usr/bin/ejabberdctl reload_config
PrivateDevices=true
TimeoutSec=300
KillMode=mixed

[Install]
WantedBy=multi-user.target
Comment by A. Bosch (progandy) - Thursday, 11 October 2018, 20:53 GMT
There is something strange with your live session. You seem to be running Erlang/OTP 20 [erts-9.3.3].
In my tests, the package version 18.09-1 uses Erlang/OTP 21 [erts-10.1], though. Did you use the wrong erlang version?

> Is it possible that new dependencies were added to 18.09 that are missing on my system?
I don't know. I tried it with the default configuration from the ejabberd package on my regular laptop.
This uses the sqlite database. Postgresql is not installed, but postgresql-libs is.

> BTW: I'm running on an old Atom D525 with 4GB memory. Might this be too small/slow?
Not really. My laptop has 4GB memory as well and I am running the i3-350M with ~1GHz (powersave profile).
Comment by Uwe Sauter (UweSauter) - Friday, 12 October 2018, 06:49 GMT
Which erlang version is necessary? Currently I have the following installed, purely as dependencies from ejabberd:

# pacman -Qs erlang
local/ejabberd 18.06-2
Jabber server written in Erlang
local/erlang-nox-20 20.3.8-2
General-purpose concurrent functional programming language developed by Ericsson (headless version)
local/erlang-unixodbc 21.1-1
Unixodbc support for Erlang


# pacman -Qi erlang-nox-20
Name : erlang-nox-20
Version : 20.3.8-2
Description : General-purpose concurrent functional programming language developed by Ericsson (headless
version)
Architecture : x86_64
URL : http://www.erlang.org/
Licenses : Apache
Groups : None
Provides : erlang-nox
Depends On : ncurses openssl
Optional Deps : erlang-unixodbc: database support [installed]
java-environment: for Java support
lksctp-tools: for SCTP support
Required By : ejabberd erlang-unixodbc
Optional For : None
Conflicts With : erlang erlang-nox
Replaces : None
Installed Size : 103.72 MiB
Packager : Alexander Rødseth <rodseth@gmail.com>
Build Date : Tue 03 Jul 2018 12:33:12 PM CEST
Install Date : Thu 19 Jul 2018 04:00:24 PM CEST
Install Reason : Installed as a dependency for another package
Install Script : No
Validated By : Signature


The currently installed 18.06-2 has erlang-nox-20 as dependency defined:

# pacman -Qi ejabberd
Name : ejabberd
Version : 18.06-2
Description : Jabber server written in Erlang
Architecture : x86_64
URL : http://www.ejabberd.im/
Licenses : GPL
Groups : None
Provides : None
Depends On : expat openssl zlib erlang-nox-20 pam iproute2 erlang-unixodbc libyaml sqlite gd
Optional Deps : None
Required By : None
Optional For : None
Conflicts With : None
Replaces : None
Installed Size : 22.36 MiB
Packager : Sergej Pupykin <pupykin.s+arch@gmail.com>
Build Date : Tue 17 Jul 2018 04:34:21 PM CEST
Install Date : Mon 08 Oct 2018 08:48:35 PM CEST
Install Reason : Explicitly installed
Install Script : No
Validated By : None


If I try to upgrade ejabberd it will happily install without telling that it needs Erlang/OTP 21, though pacman lists erlang-nox and not erlang-nox-20 as dependency:

# pacman -Qip ejabberd-18.09-1-x86_64.pkg.tar.xz
Name : ejabberd
Version : 18.09-1
Description : Jabber server written in Erlang
Architecture : x86_64
URL : http://www.ejabberd.im/
Licenses : GPL
Groups : None
Provides : None
Depends On : expat openssl zlib erlang-nox pam iproute2 erlang-unixodbc libyaml sqlite gd
Optional Deps : None
Conflicts With : None
Replaces : None
Compressed Size : 11.92 MiB
Installed Size : 22.81 MiB
Packager : Sergej Pupykin <pupykin.s+arch@gmail.com>
Build Date : Mon 01 Oct 2018 03:56:34 PM CEST
Install Script : No
Validated By : None
Signatures : None


Checking erlang-nox-20 it reports that it conflicts with erlang-nox (Conflicts With : erlang erlang-nox).

Is dependency checking not working correctly?

Should I manually upgrade from erlang-nox-20 to erlang-nox?

Thanks in advance.
Comment by Robert Kormann (RoKoInfo) - Saturday, 13 October 2018, 10:30 GMT
@ Uwe Sauter: I tried the update you asked for (pacman -Rdd erlang-nox-20) and replaced it with erlang-nox, and no glitches up to now. Seems to work like a charm for an hour or so, will keep an eye on it.

Robert
Comment by Uwe Sauter (UweSauter) - Saturday, 13 October 2018, 12:10 GMT
I removed erlang-nox-20 as well and upgraded to 18.09-1. With this version and using the systemd unit form above I no longer get crash dumps when stoping (18.06) or starting (18.09) ejabberd. 18.09 also solved the TLSv1.3 related problem I had.

Thanks everyone.

@sergej: Please consider replacing the current systemd unit file shipped with the Arch package with something more like the upstream version, like I did manually.

I'm happy now and this can be closed from my side.

Loading...