FS#49815 - [opensmtpd] Fails to start during boot

Attached to Project: Community Packages
Opened by Pavel (psp) - Thursday, 23 June 2016, 08:19 GMT
Last edited by Johannes Löthberg (demize) - Wednesday, 01 November 2017, 23:05 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Lukas Fleischer (lfleischer)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Opensmtpd fails during the boot with the message: fatal: smtpd: bind: Cannot assign requested address.

Starting from the command line with systemctl start smtpd after boot works without any problems.

The problem is the start during the boot worked until recently, on Jun 8 boot smtpd was still starting during the boot fine. There must have been updates to some libraries or other packages that caused different behaviour of network environment between Jun 8, 10:40 and now (June 22)(sorry, do not boot too frequently). I tried to changed the target in smtpd unit to network-online.target but the result was the same.

info:

Additional info:
* Version: 5.9.2p1
* no changes to /etc/smtpd/smtpd.conf during the period, standard listen directive.

listen on eth0 port 25 ...

Steps to reproduce:

Have smtpd.service enabled and reboot.

Network configured with dhcpcd (static ip) during boot.


Workaround: replace network interface with 0.0.0.0

listen on eth0 port 25 ... -> listen on 0.0.0.0 port 25 ...
This task depends upon

Closed by  Johannes Löthberg (demize)
Wednesday, 01 November 2017, 23:05 GMT
Reason for closing:  Fixed
Additional comments about closing:  6.0.2p1-6 requires and orders after network-online.target like it should. And for things that depend on the networking being set up to work overall you need to have network-online.target set up properly. (*-wait-online.service)
Comment by Pavel (psp) - Thursday, 23 June 2016, 08:23 GMT
Forgot to add: architecture x64.
Comment by Wilhelm Schuster (wlhlm) - Sunday, 21 August 2016, 07:02 GMT
This has been reported on the opensmtpd mailing list before [1] and seems to be a caused by the combination opemsmtpd+systemd. Moreover this hasn't been a problem only recently, but has been happening for me since I've been using opensmtpd on Archlinux (more than two years).

[1]: https://marc.info/?l=opensmtpd-misc&m=144442204714169
Comment by Tobias Stolzmann (teiesti) - Tuesday, 18 October 2016, 23:56 GMT
I have the same problem but something is weird: If I replace the line "After=network.target" with "After=network-online.target" in /usr/lib/systemd/system/smtpd.service, everything works well for me. If I understand the previous posts correctly, this is exactly what Pavel tried -- but he had no success.

Some thoughts:

1) In general: This looks like a timing issue. Timing issues usually come from race conditions. At boot time, setting up the network races against starting OpenSMTPD. Assuming that, we have an explanation why there is no problem to start OpenSMTPD by hand: When we try by hand, the race is over because the network is online.

2) Using "After=network.target" causes (one part?) of the race condition. If you read [1] and [2] you'll see, that interfaces might not be "online" when network.target is reached. You need to use "After=network-online.target" to ensure this.

3) Thought 1 and 2 can explain my problems. But they can't explain Parvel's. Therefore I think, we have (at least) two problems which are connected. There must be a difference in configuration.

4) When I read the OpenSMTPD mailing list, I noticed that people where using DHCP. I am using a static configuration. Since DHCP takes quite a long time, this can make trouble. @Parvel: Are you using DHCP?

All in all, I strongly recommend to replace "After=network.target" with "After=network-online.target" in the unit file. It seems to fix some issues and causes for sure no harm. We can see for other problem after we did this. (By the way: Software like BIND or ISC DHCP uses "After=network-online.target".)


[1]: https://marc.info/?l=opensmtpd-misc&m=144445668220970&w=2
[2]: https://wiki.freedesktop.org/www/Software/systemd/NetworkTarget/
Comment by Pavel (psp) - Wednesday, 19 October 2016, 07:55 GMT
Tobias: ad 4) Yes, I use DHCP static configuration. It is an IPv4 server in both IPv4 and IPv6 network, with noipv6rs and noipv6 parameters set in dhcpcd.conf.
What is weird that Opensmtpd used to start during boot normally until June 8 and two weeks later the problem appeared. There were no changes to the configuration except the usual upgrades with pacman -Syu.

There have been further releases of opensmtpd, time to retest when I have time.
Comment by Tobias Stolzmann (teiesti) - Wednesday, 19 October 2016, 13:37 GMT
> I use DHCP static configuration.
There is nothing that is called a DHCP static configuration. For a single interface, you can either use DHCP or a static configuration! Read [1] to see the difference.

> There have been further releases of opensmtpd, time to retest when I have time.
Are you using the latest version of OpenSMTPD?

Once again something about "After=network.target" vs. "After=network-online.target":
In my last post, I explained why "After=network.target" cannot work. I have a running example that it does not work. "After=network-online.target" resolves the issue in my environment. Comparable packages (eg. bind) are using "After=network-online.target". The patch does not cause new problems and is very easy. That are a lot of reasons. Therefore, I would really like to see the package maintainer including the patch before we continue to think about other reasons.


[1]: http://superuser.com/questions/194473/difference-between-static-and-dhcp
Comment by Wilhelm Schuster (wlhlm) - Wednesday, 19 October 2016, 14:38 GMT
@teiesti network-online.target does not work for me either.
Comment by Wilhelm Schuster (wlhlm) - Wednesday, 19 October 2016, 15:19 GMT
Ok, a workaround for me is to listen on a domain instead of an interface. This way (+network-online.target) smtpd comes up at boot.
Comment by Tobias Stolzmann (teiesti) - Wednesday, 19 October 2016, 15:48 GMT
Mhmm.... I am listening on an interface. That's weird.

I'll provide my configuration file in the attachment. Maybe, you can find a difference. Would you provide (a censored version of) yours? I would like to reproduce your situation.
Comment by Wilhelm Schuster (wlhlm) - Wednesday, 19 October 2016, 17:57 GMT
> Would you provide (a censored version of) yours?

Sure.

I noticed that I have maybe have kind of the same setup as @psp: The server gets static IPv4 configuration via DHCP (meaning DHCP always returns the same address, just that I don't have to manually punch in addresses and gateways). systemd-networkd is used for interface management. My .network file basically looks like this:

[Match]
name=ens3

[Network]
DHCP=v4

Address=<static IPv6 configuration>
...

I changed the smtpd service to "After=network-online.target" and enabled "systemd-networkd-wait-online.service" and smtpd failed to start up with:

Oct 19 14:48:33 smtpd[191]: invalid virtual ip or interface: ens3
Comment by Wilhelm Schuster (wlhlm) - Saturday, 22 October 2016, 07:05 GMT
Never mind my workaround, it's still racy and won't come up on boot occasionally.

Loading...