FS#42976 - [salt-raet] salt-master systemd script fails

Attached to Project: Community Packages
Opened by Rob Smith (kormoc) - Monday, 01 December 2014, 19:07 GMT
Last edited by Daniel Wallace (gtmanfred) - Tuesday, 17 February 2015, 02:33 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sébastien Luttringer (seblu)
Daniel Wallace (gtmanfred)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
When attempting to run salt-master via systemctl, it starts but fails to report in and thus gets terminated by systemd.

Additional info:

* package version(s)

salt-raet 2014.7.0-1

* config and/or log files etc.

[root@clamps ~]# journalctl -u salt-master -l
-- Logs begin at Sat 2014-07-26 20:45:55 PDT, end at Mon 2014-12-01 11:05:36 PST
Nov 30 20:45:52 clamps systemd[1]: salt-master.service start operation timed out
Nov 30 20:45:52 clamps systemd[1]: Failed to start The Salt Master Server.
Nov 30 20:45:52 clamps systemd[1]: Unit salt-master.service entered failed state
Nov 30 20:45:52 clamps systemd[1]: salt-master.service failed.

Steps to reproduce:
This task depends upon

Closed by  Daniel Wallace (gtmanfred)
Tuesday, 17 February 2015, 02:33 GMT
Reason for closing:  Upstream
Additional comments about closing:  this is a blocker for 2015.2

if it isn't solved then, I will reopen this.

https://github.com/saltstack/salt/issues /20531
Comment by Daniel Wallace (gtmanfred) - Tuesday, 02 December 2014, 00:13 GMT
Can you provide the output of your /var/log/salt/master log file around the time when starting.

Be sure to sanitize anything that may be sensitive.
Comment by Rob Smith (kormoc) - Tuesday, 02 December 2014, 00:19 GMT
so I run systemctl start salt-master and it just hangs. While it hangs, in another terminal window, I run systemctl status salt-master, I see

[root@clamps ~]# systemctl status salt-master
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled)
Active: activating (start) since Mon 2014-12-01 16:15:27 PST; 55s ago
Main PID: 9575 (salt-master)
CGroup: /system.slice/salt-master.service
├─9575 /usr/bin/python2 /usr/bin/salt-master
├─9578 /usr/bin/python2 /usr/bin/salt-master
├─9579 /usr/bin/python2 /usr/bin/salt-master
├─9580 /usr/bin/python2 /usr/bin/salt-master
├─9581 /usr/bin/python2 /usr/bin/salt-master
├─9582 /usr/bin/python2 /usr/bin/salt-master
└─9583 /usr/bin/python2 /usr/bin/salt-master
[root@clamps ~]#

Once finally fails with:
[root@clamps ~]# systemctl start salt-master
Job for salt-master.service failed. See "systemctl status salt-master.service" and "journalctl -xe" for details.

I run status, it shows

[root@clamps ~]# systemctl status salt-master
● salt-master.service - The Salt Master Server
Loaded: loaded (/usr/lib/systemd/system/salt-master.service; enabled)
Active: failed (Result: timeout) since Mon 2014-12-01 16:16:57 PST; 30s ago
Process: 9575 ExecStart=/usr/bin/salt-master (code=killed, signal=TERM)
Main PID: 9575 (code=killed, signal=TERM)

Dec 01 16:16:57 clamps systemd[1]: salt-master.service start operation time...g.
Dec 01 16:16:57 clamps systemd[1]: Failed to start The Salt Master Server.
Dec 01 16:16:57 clamps systemd[1]: Unit salt-master.service entered failed ...e.
Dec 01 16:16:57 clamps systemd[1]: salt-master.service failed.
Hint: Some lines were ellipsized, use -l to show in full.
[root@clamps ~]#

journalctl shows:
Dec 01 16:16:57 clamps systemd[1]: salt-master.service start operation timed out. Terminating.
Dec 01 16:16:57 clamps systemd[1]: Failed to start The Salt Master Server.
Dec 01 16:16:57 clamps systemd[1]: Unit salt-master.service entered failed state.
Dec 01 16:16:57 clamps systemd[1]: salt-master.service failed.

journalctl -xe shows:
Dec 01 16:16:57 clamps systemd[1]: salt-master.service start operation timed out
Dec 01 16:16:57 clamps systemd[1]: Failed to start The Salt Master Server.
-- Subject: Unit salt-master.service has failed
-- Defined-By: systemd
-- Support: http://lists.freedesktop.org/mailman/listinfo/systemd-devel
--
-- Unit salt-master.service has failed.
--
-- The result is failed.
Dec 01 16:16:57 clamps systemd[1]: Unit salt-master.service entered failed state
Dec 01 16:16:57 clamps systemd[1]: salt-master.service failed.

Master log doesn't show anything at all when run via systemctl.

When run by hand via salt-master -l debug, it shows plenty of information, but nothing pertinent.
Comment by Daniel Wallace (gtmanfred) - Tuesday, 02 December 2014, 01:35 GMT
It sounds like it isn't sending the systemd.daemon.notify correctly.

We had a long discussion about this. I am going to have to follow up with how they changed the threading process system for the raet stuff, cause it was located elsewhere. But I bet that is what it is.

In the mean time, you can drop

/etc/systemd/system/salt-master.service.d/forking.conf
[Service]
ExecStart=
ExecStart=/usr/bin/salt-master -d
Type=forking
PIDFile=/var/run/salt-master.pid

I will have to follow up with Tom on why https://github.com/saltstack/salt/pull/16163 doesn't fix this.
Comment by Daniel Wallace (gtmanfred) - Tuesday, 17 February 2015, 02:32 GMT

Loading...