Arch Linux

- Overview:

The ipset.service is specified as "Before=(...) iptables.service", however, this is insufficient. There is a race condition on ipset restoring fast enough. If an ipset doesn't exist, the entire iptables-restore fails.

This can essentially take down a major part of the system firewall in some cases, as it did in mine: I used simple hand-written iptables rules with ipset to whitelist a small number of hosts and otherwise reject everything.

- Package versions and random details details:

core/iptables-nft 1:1.8
extra/ipset 7.13.1
extra/linux-hardened 6.4.10.hardened1-1
I believe this will occur on earlier versions as well on the legacy iptables.
BTRFS filesystem on an SSD, for what it's worth.


- Steps to reproduce:

(essentially what is on the ArchWiki pages for iptables and ipset)

create some ipsets, and iptables rules that use them:
# ipset --create ...
# iptables -A ...

persist the changes:

# ipset save > /etc/ipset.conf
# iptables-save > /etc/iptables/iptables.conf
$ systemctl enable ipset iptables

... then reboot N hundred times until one day you're unlucky, the server is suffering for unknown reasons, and then

$ systemctl status iptables

reports a failure due to (roughly) "ipset BLABLABLA doesn't exist", despite

# systemctl status ipset

showing that it loaded successfully.

Apologies, I've only experienced this 'naturally' once, and didn't save the printed status of iptables.service before rebooting to mitigate damage. For ease of explanation, I replicated it by deleting one of the ipsets from /etc/ipset.conf. The message of the original bug was identical to the below, to my memory:

# systemctl status iptables [root@hostname | 21:23:42]
× iptables.service - IPv4 Packet Filtering Framework
Loaded: loaded (/usr/lib/systemd/system/iptables.service; enabled; preset: disabled)
Active: failed (Result: exit-code) since Tue 2023-09-12 21:22:28 SAST; 1min 17s ago
Process: 320 ExecStart=/usr/bin/iptables-restore /etc/iptables/iptables.rules (code=exited, status=2)
Main PID: 320 (code=exited, status=2)

Sep 12 21:22:28 hostname systemd[1]: Starting IPv4 Packet Filtering Framework...
Sep 12 21:22:28 hostname iptables-restore[320]: iptables-restore v1.8.9 (nf_tables): Set MY-CUSTOM-IPSET-NAME doesn't exist.
Sep 12 21:22:28 hostname iptables-restore[320]: Error occurred at line: 15
Sep 12 21:22:28 hostname iptables-restore[320]: Try `iptables-restore -h' or 'iptables-restore --help' for more information.
Sep 12 21:22:28 hostname systemd[1]: iptables.service: Main process exited, code=exited, status=2/INVALIDARGUMENT
Sep 12 21:22:28 hostname systemd[1]: iptables.service: Failed with result 'exit-code'.
Sep 12 21:22:28 hostname systemd[1]: Failed to start IPv4 Packet Filtering Framework.

When I then inspected output of

# iptables-save

it showed that all my hand-written rules were missing, I presume iptables-restore is atomic. Only rules added dynamically like fail2ban ('redundant' security :-) and docker are present.

- Possible solution? :

(Forgive me for the crudenss, I am not a systemd wizard.) I thought of putting a 100ms sleep before attempting to run iptables-restore, i.e. add the directive

ExecStartPre=/bin/sleep 0.1

to the [Service] section of '/usr/lib/systemd/system/iptables.service'

It may be necessary to do this (or another, better change) to nftables and ip6tables services as well, I haven't delved any deeper at this point.