FS#18157 - [filesystem] symlink /var/{run,lock} to /run/{,lock}

Attached to Project: Arch Linux
Opened by Thomas Bächler (brain0) - Thursday, 04 February 2010, 13:05 GMT
Last edited by Tom Gundersen (tomegun) - Tuesday, 13 December 2011, 00:05 GMT
Task Type Feature Request
Category System
Status Closed
Assigned To Tobias Powalowski (tpowa)
Jan de Groot (JGC)
Aaron Griffin (phrakture)
Thomas Bächler (brain0)
Dan McGee (toofishes)
Tom Gundersen (tomegun)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 24
Private No

Details

Many things changed since this request was first opened, the original problems are solved elsewhere, but there are still reasons to implement this:

The new tmpfs-mounted /run folder allows programs like udev, lvm and mdadm to keep runtime data from initrd until shutdown.

We can move all runtime data to these folders by simply creating symlinks in /var. In addition to consolidating all the runtime data in one place, this will remove needless writes to disk, as well as remove the need to cleanup the directories on boot.

Implementing this is still blocked by packages that (mistakenly) rely on data in /var/{run,lock} being persistent after reboot (as described below).

Original feature request:
========================
In another bug report, I got the impression that mounting tmpfs for /var/run and /var/lock very early in the initialization process is a good idea and solves some stupid problems when running applications while /var is still readonly.

Problems:

1) People might have /var on a separate partition from /
2) Some programs don't work if their /var/{run,lock}/whatever folder is missing - I would consider these bugs, but we need to think about it. We would also have to stop putting /var/{lock,run}/* files or directories into packages. On my system:
$ grep -e ^var/run -e ^var/lock /var/lib/pacman/local/*/files | cut -d: -f2|sort -u
var/lock/
var/lock/sane/
var/run/
var/run/bftpd/
var/run/ConsoleKit/
var/run/dbus/
var/run/hald/
var/run/hald/hald-local/
var/run/hald/hald-runner/
var/run/httpd/
var/run/lirc/
var/run/mpd/
var/run/mysqld/
var/run/network/
var/run/network/interfaces/
var/run/network/profiles/
var/run/openntpd/
var/run/PolicyKit/
var/run/samba/

I assigned a few people to this who might have an opinion and useful input here, feel free to remove yourself if you don't care.
This task depends upon

Closed by  Tom Gundersen (tomegun)
Tuesday, 13 December 2011, 00:05 GMT
Reason for closing:  Implemented
Additional comments about closing:  This is now in initscripts in git, please give it a try and reports problems to the ML.
Comment by Allan McRae (Allan) - Thursday, 04 February 2010, 14:59 GMT
Potentially stupid question follows...

From this comment "solves some stupid problems when running applications while /var is still readonly", I get the impression that these programs are wanting to write something in /var/{lock,run}. How would it be handled when these files disappear when the tmpfs is removed and the real /var is mounted rw?
Comment by Jan de Groot (JGC) - Thursday, 04 February 2010, 15:11 GMT
I don't like to have this on tmpfs. So far not a single distribution out there is crazy enough to mount /var/run and /var/lock on tmpfs. The only distribution that does this is a modified version of archlinux which I'm running on a readonly mounted ext2-formatted compactflash card to avoid any writes to the filesystem.

From the directories above, some don't come with initscripts, which means we'll have to hack the software ourselves. Examples of this are PolicyKit and ConsoleKit, both are daemons launched from dbus.
Comment by Thomas Bächler (brain0) - Thursday, 04 February 2010, 15:28 GMT
@Allan: Yes, that IS a problem. Mounting /var/lock or /var/run early is a good idea if /var is on /, but if /var is separate, we would have to mount --move the tmpfs-/var/{lock,run} to some "safe" location, mount everything and mount --move them back. Suffice it to say there probably is a proper solution for this issue, we just have to figure it out.

@JGC: From what I remember, Debian is planning to do this in their next major release, so at least one other distro is "crazy enough". IMO, software should create necessary /var/{run,lock}/whatever directories when needed, if it doesn't it can be considered a bug. Having these folders on tmpfs doesn't sound too crazy to me anyway. Tell me, how else would you solve  FS#18153 ? It wants to write PID files and/or lock files very early in the boot process, at a time when access to certain file systems might be impossible because they are on LVM.


What about adding this as an optional "experimental" feature to rc.sysinit/rc.conf and letting crazy people like me test it and see how it goes?
Comment by Michael Trunner (trunneml) - Thursday, 04 February 2010, 16:24 GMT
I think it is a good idea to put lock and run on an tmpfs. Some servers running on my university sometimes have problems with old lock and pid files after a crash or unclean shutdown. That problem will be fixed with an tmpfs for /var/(run|lock), too.

A workaround for the problem of separated /var patition could be to mount the early tmpfs somewhere else for example /dev/shm and use a bind mount at boot stage and in fstab. I believe that should work for both separated /var partitions and /var on root.

Or just use the the tmpfs on boot time and when the partitions are mounted copy the stuff to the harddisk. I know that isn't a wounderful solution. :-D

Comment by Jan de Groot (JGC) - Thursday, 04 February 2010, 17:37 GMT
Debian already implemented it for lenny, but it's experimental and turned off by default because it breaks a lot of their packages. They accept bugs for packages that don't work with /var/run and /var/lock on tmpfs, but if the maintainer doesn't like to fix it, the bug won't get fixed.

As for dmeventd, it's a logging daemon that records devicemapper events to syslog. Is it really needed to run it that early on bootup?
Comment by Gerardo Exequiel Pozzi (djgera) - Thursday, 04 February 2010, 23:29 GMT
This is a list of advantages that I can see:

Regardless of /var/run is tmpfs or not:
* Making rc-scripts compatible with empty /var/run (variable run-time data), makes more robust and elegant.
* Makes rc-scripts more FHS-complaint.
* pacman does not handle empty dirs at all (I refer to "/var/run/program-name")

If /var/run is tmpfs:
* Makes unnecessary the cleanup at rc.sysinit
* Some netbooks users, can benefit from this change.

If /var/lock is tmpfs:
* Since is (chmod 1777) mounting as tmpfs can control space usage.


Some rc-scripts are currently compatible with empty /var/run, these automatically create the necessary directories inside (for example hald)
I am talking about /var/run because the usage is much more than /var/lock, but this also applies to /var/lock.


http://www.pathname.com/fhs/pub/fhs-2.3.html#VARRUNRUNTIMEVARIABLEDATA
http://www.pathname.com/fhs/pub/fhs-2.3.html#VARLOCKLOCKFILES
Comment by Jim Pryor (Profjim) - Saturday, 06 February 2010, 22:49 GMT
I've been doing this on my laptop for about 6 months, with some minor glitches but no serious problems. I do also have /var on another partition, and mount the tmpfs in my /etc/fstab after the /var is mounted. I can see that strategy wouldn't work for what's being envisaged here; you want the /var/{run,lock} available sooner. brain0's proposal in the 3rd comment is a little complex but doesn't sound so bad to me. I'll be (pleasantly) surprised if there's a simpler way to get the same effect.

One potential gotcha I had to work around: the umount command in /etc/rc.shutdown doesn't unmount tmpfs. You could change the switches to make it do so. Or, if you've mounted /var/{run,lock} on top of a separate /var partition, you'll have to move them again so that /var can be umounted.
Comment by Nicola Tuveri (romen) - Thursday, 18 November 2010, 13:53 GMT
Is there any news on this?
 FS#18153 , which is blocked by this, get serious if you have mirrored or snapshot logical volumes, as the boot time heavily increases.
Comment by Allan McRae (Allan) - Thursday, 18 November 2010, 13:57 GMT
FYI - it will be a Fedora 15 feature: http://fedoraproject.org/wiki/Features/var-run-tmpfs
Comment by Nicola Tuveri (romen) - Thursday, 18 November 2010, 14:07 GMT
@Allan: do you know how do they plan to solve the mentioned problem of people having a separate /var partition ? I read through your link, and they don't mention it, maybe they do not allow to mount /var as a separate volume?
Comment by Tom Gundersen (tomegun) - Wednesday, 01 December 2010, 10:22 GMT
Further to Allan's comment, SuSE and Ubuntu also will (or alreday do?) have /var/run on tmpfs.

Here is a description of how Fedora will implement it: <http://www.spinics.net/lists/fedora-devel/msg146048.html>. They intend to use a systemd feature to work around bugs in packages (packages that are not able to create the needed directories themselves). However, we could easily also implement this workaround independently of systemd (it is described in the email). If there is interest in this, I'd be happy to help with the implementation.

It should be noted that, to the best of my knowledge, Fedora (and probably others) will not mount /var/run before /var. I believe the longterm goal is to support a read-only root, in that case / (ro) and /var (rw) will be on separate partitions and /var will be mounted before /var/run (tmpfs).

I'm not using LVM, but I'm pretty certain  FS#18153  can be solved without moving mounts around.
Comment by Nicola Tuveri (romen) - Wednesday, 01 December 2010, 10:44 GMT
@tomegun The point with  FS#18153  is that /var might be a logical volume (over LVM), starting LVM with snapshots or mirroring requires rw access to /var/{lock,run} (as per current default configuration), hence we need /var/{lock,run} before /var is mounted.

Having tmpfs for /var/{run,lock} will help, but I cannot see a way to avoid moving mounts around because of this issue.
Comment by Tom Gundersen (tomegun) - Wednesday, 01 December 2010, 11:07 GMT
@romen: I added a comment about this to the relevant bug (it seems to have been solved upstream).
Comment by Tom Gundersen (tomegun) - Monday, 28 March 2011, 13:02 GMT
I'm revisiting this after a long time:

I think it is worthwhile to support /var/run and /var/lock being tmpfs. However, I think we should follow the example of udev/dracut/systemd/... and implement it as follows:
1) Make /run and /run/lock tmpfs and mount them early in rc.sysinit and/or in initrd.
2) Make /var/run and /var/lock symlinks to the above directories.
3) Implement <http://0pointer.de/public/systemd-man/tmpfiles.d.html> to recreate the required directories/files on boot.

The last point is the biggest blocker as it would require us to create a daemon (which we do not have the manpower to do I think), or use someone else's (we could easily use the systemd one, but that would be a bit strange packaging-wise). Secondly, every package that relies on files in /var/run and /var/lock would need to come with a config file to create the relevant files/folders. This we should get for free at some point in the future as systemd is requiring this anyway.

What do you think?

I guess we should discuss this further on the mailinglist.
Comment by Michael Trunner (trunneml) - Monday, 28 March 2011, 20:17 GMT
@tomegun: Why did we need such a daemon to create empty dirs and files? I think that every startup rc can and should do that. I can't find any reason for an extra application or even daemon, did I miss something? :-?
Comment by Tom Gundersen (tomegun) - Monday, 28 March 2011, 20:56 GMT
@Michael: I should have said application, not daemon. We should add directory creation to every rc script (I would at least be in favor of any such patch), but that would not cover the cases of programs with no script in rc.d (such as consolekit and other daemons started via dbus activation).

There would need to be some tool to create the needed dirs with the right permissions on boot for the programs that cannot do it themselves, and there would have to be some configuration files to indicate that this tool should create a certain directory. Since many packages probably will come with files on the tmpfiles.d format soon anyway, I figured it would make sense to reuse that standard rather than to make up a new one.

It should be fairly simple to make such a tool (we would only need to implement a minimal subset of the above standard, enough to cover our needs), so if anyone is interested in this I think it is definitely doable and I'd be happy to help. The tool should be able to work both when the dirs are mounted on a persistent device as well as on tmpfs, and it should be able to replace the relevant snippet in rc.sysinit.

What do you think?
Comment by Michael Trunner (trunneml) - Monday, 28 March 2011, 22:19 GMT
@tomegun:
Okay, I see :-)
But are there really any programs, that haven't an rc.d file an need folders or files created?
And it shouldn't be that hard to create an small sh script that can create that dirs and files based on some small conf-files.
Comment by Tom Gundersen (tomegun) - Monday, 28 March 2011, 22:59 GMT
@Michael: There are not many, but a few. I believe lvm is one of them. Also, if we are going to do /var/run and /var/lock, then we might as well do /tmp, in which case there will also be a few files that need to be created on boot to avoid security holes (like we do now in rc.sysinit).

As you say this should be very easy. Just make a script that parses some conf files, reads off the directories and creates them with the correct permissions and ownerships.

The standard I linked to above is really very simple. We would have to go through the files in /etc/tmpfiles.d/*.conf which have space separated lines of the format:
d /var/run/screens 1777 root root 10d
I suggest we ignore the last entry (Age) and directives of type "x" (possibly also "r" and "R"), but except for that implement the standard.

Are you interested in working on this? Any patches sent to the mailinglist would be greatly appreciated :-)

A further benefit of doing this is that if /var/run and /var/lock are tmpfs, we can move (bind-mount) them to /run and /run/lock as some upstream projects have started doing, and we simplify our boot/shutdown process quite a bit.
Comment by Greg (dolby) - Wednesday, 30 March 2011, 14:28 GMT Comment by Michael Trunner (trunneml) - Wednesday, 30 March 2011, 15:10 GMT
/run sounds good for me. I will give it a try, and patch my system for it. And even try to create that folder and directory creating script.
Comment by Tom Gundersen (tomegun) - Wednesday, 30 March 2011, 15:49 GMT
@Greg: thanks for the reference. @Michael: great! I'm using /run myself, and haven't seen any problems.
Comment by Sergej Pupykin (sergej) - Thursday, 31 March 2011, 12:28 GMT
/var/run and /var/lock also should be removed from the valid dirs list in namcap/rules/fhs.py
Comment by Sergej Pupykin (sergej) - Thursday, 31 March 2011, 12:30 GMT
Full list of packages with /var/run:

community/bind-geodns-9.4.1-3
community/cherokee-1.2.2-2
community/collectd-4.10.3-1
community/couchdb-1.0.2-1
community/courier-imap-4.8.1-1
community/courier-mta-0.65.3-1
community/dspam-3.9.0-3
community/fail2ban-0.8.4-2
community/fcron-3.0.6-4
community/freeradius-2.1.10-4
community/inn-2.5.2-4
community/ircservices-5.1.23-1
community/jabberd14-1.6.1.1-8
community/lastfmsubmitd-1.0.6-4
community/libinfinity-0.4.2-1
community/libvirt-0.8.8-1
community/minbif-1.0.4-1
community/miredo-1.2.3-1
community/murmur-1.2.3-1
community/nginx-0.8.54-2
community/openntpd-3.9p1-13
community/p3scan-2.3.2-4
community/privoxy-3.0.17-1
community/prosody-0.7.0-4
community/tinyproxy-1.8.2-2
community/ultimate-ircd-3.0.2-2
community/wdm-1.28-4
community/xl2tpd-1.2.8-1
core/dbus-core-1.4.1-1
coreystem-2010.12-1/files
core/netcfg-2.5.4-1
extra/apache-2.2.17-1
extra/avahi-0.6.29-1
extra/bftpd-3.1-1
extra/bind-9.8.0-1
extra/bitlbee-3.0.2-1
extra/consolekit-0.4.3-2
extra/cups-1.4.6-1
extra/cyrus-sasl-2.1.23-4
extra/dansguardian-2.10.1.1-2
extra/dovecot-2.0.11-2
extra/fetchmail-6.3.19-1
extra/gdm-2.32.1-1
extra/hal-0.5.14-6
extra/iptraf-3.0.0-2
extra/laptop-mode-tools-1.57-1
extra/lighttpd-1.4.28-4
extra/lxdm-0.3.0-1
extra/munin-1.4.5-2
extra/munin-node-1.4.5-2
extra/networkmanager-0.8.3-0.20110113
extra/php-fpm-5.3.6-1
extra/proftpd-1.3.3d-2
extra/samba-3.5.8-2
extra/udisks-1.0.2-2
extra/wesnoth-1.8.5-3

Full list of packages with /var/lock:

community/ejabberd-2.1.6-1
core/dmraid-1.0.0.rc16+CVS-2
extra/hylafax-6.0.4-1
extra/sane-1.0.22-1
Comment by Tom Gundersen (tomegun) - Thursday, 31 March 2011, 13:02 GMT
@Sergej: Thanks for the full list (the list I posted was obviously incomplete). The list could be filtered further though, as some (e.g. avahi) only contain /var/run or /var/lock, which is fine as long as they do not contain anything else (like /var/run/avahi-daemon).
Comment by Gerardo Exequiel Pozzi (djgera) - Friday, 01 April 2011, 16:03 GMT
Great seems that this will become a reality soon.
See some patches required for [extra]:  FS#13204   FS#13211   FS#13225   FS#13226   FS#13227   FS#13228   FS#13229   FS#13230   FS#13231   FS#13232   FS#13233   FS#13234   FS#13235 . Some was reject and others are currently implemented.
I did nothing for [community] because two years ago nobody interested in implementing this.
Some notes about few packages can be interested:arch-general@archlinux.org/msg03581.html"> http://www.mail-archive.com/arch-general@archlinux.org/msg03581.html
Comment by Sergej Pupykin (sergej) - Monday, 04 April 2011, 11:03 GMT
I created todo for community packages: http://www.archlinux.org/todo/70/
Comment by Sergej Pupykin (sergej) - Monday, 04 April 2011, 11:05 GMT
updated core/extra packages list:

dbus-core-1.4.1-1
dmraid-1.0.0.rc16+CVS-2
netcfg-2.5.4-1
apache-2.2.17-1
bftpd-3.1-1
bind-9.8.0-1
bitlbee-3.0.2-1
consolekit-0.4.3-2
cups-1.4.6-1
cyrus-sasl-2.1.23-4
dovecot-2.0.11-2
fetchmail-6.3.19-1
gdm-2.32.1-1
hal-0.5.14-6
iptraf-3.0.0-2
laptop-mode-tools-1.57-1
lighttpd-1.4.28-4
lxdm-0.3.0-1
munin-1.4.5-2
munin-node-1.4.5-2
networkmanager-0.8.3-0.20110113
php-fpm-5.3.6-1
samba-3.5.8-2
sane-1.0.22-1
udisks-1.0.2-2
wesnoth-1.8.5-3
Comment by Pierre Schmitz (Pierre) - Saturday, 09 April 2011, 10:35 GMT
You should add these pacakges to the todo list; otherwise their maintainers will never know about this.
Comment by Kyle Keen (keenerd) - Sunday, 10 April 2011, 13:04 GMT
I've "fixed" netcfg. Turns out it already makes directories on demand, see line 51 of /usr/lib/network/network. Network is sourced by all relevant scripts, so it should be fine. (Bash and zsh completions use hard coded /var/run instead of $STATE_DIR, but that is a different problem.)

The only correction needed is the makefile. Remove line 9, "$(DESTDIR)/var/run/network/{interfaces,profiles} \"

Is there a better place for this report?
Comment by Greg (dolby) - Sunday, 10 April 2011, 14:48 GMT
Send a patch on arch-projects mailing list.
Comment by Tom Gundersen (tomegun) - Saturday, 23 April 2011, 15:12 GMT
In initscripts we mount /run (and I intend to create /run/lock) during early boot. After all our packages support this, we could symlink /var/run to /run and /var/lock to /run/lock. I'm using this at the moment (even without patching any packages) and everything works fine.

If anyone want to try it out, create the symlinks and apply this patch to initscripts:
<https://github.com/teg/initscripts-arch/commit/b688326d1d1f1b1903ff90c0d60ed441157ecdb1>.
Comment by Tom Gundersen (tomegun) - Tuesday, 26 April 2011, 15:39 GMT
With initscripts from git, we can now symlink /var/{run,lock} to /run/{,lock}. I have not detected any side effects of doing this (even with many of the packages in the TODO list), but we should probably assess the core/extra packages in the list before adding the symlinks to [filesystem].

I updated the bug description to indicate the current state of affairs.

What do you guys think?
Comment by Pierre Schmitz (Pierre) - Saturday, 07 May 2011, 08:04 GMT
I can only add those symlinks to the fileystem package if no other package has files/directories in /var/{run,lock}. Afaik pacamn will fail to update otherwise.
Comment by Tom Gundersen (tomegun) - Saturday, 07 May 2011, 10:01 GMT
@Pierre: we could either wait for all packages to remove their files/directories, or at some point the symlinks could be created on boot. Either way we'd better wait until all the core/extra packages (at least) have been confirmed to work without persistent /var/{run,lock}.
Comment by Andreas Radke (AndyRTR) - Friday, 17 June 2011, 19:11 GMT
Maybe [ -d /var/lock/NAME ] || mkdir -p /var/lock/NAME and [ -d /var/run/NAME ] || mkdir -p /var/run/NAME conditionals should be added to the /usr/share/pacman/rc-script.proto (part of abs pkg)?
Comment by Tom Gundersen (tomegun) - Friday, 17 June 2011, 22:46 GMT
@Andy: sounds reasonable (if rc-script.proto is meant for what I think it is).
Comment by Gerardo Exequiel Pozzi (djgera) - Friday, 17 June 2011, 23:10 GMT
For packages that does not have any rc.d script, we can have a "rc.d/make-directories" that can be configurable from "conf.d/make-directories.d" in initscripts. So in each package that needs this, can install a file on conf.d/make-directories.d/$pkgname. Just an idea ;)
Comment by Tom Gundersen (tomegun) - Saturday, 18 June 2011, 08:52 GMT
@djgera: we probably need something like that. I'd suggest we implement this standard though (or a small subset of it): <http://0pointer.de/public/systemd-man/tmpfiles.d.html>. Rather than inventing our own.
Comment by Heiko Baums (cyberpatrol) - Saturday, 18 June 2011, 10:36 GMT
@Tom: I find it interesting that Lennart Poettering's ideas and implementations are meanwhile at once called standard? Is he the new Linux/Posix standardization board? I just say pulseaudio (and ice1712 cards). And his tmpfiles.d is for his systemd which is not used in Arch Linux.
Comment by Tom Gundersen (tomegun) - Saturday, 18 June 2011, 10:59 GMT
@Heiko: don't be ridiculous, that's not at all what I wrote, keep your personal issues out of technical discussions.

If you have technical reasons why my proposal is not good, I'd be interested to heard them, but problems you have had with your sound card hardly counts.
Comment by Heiko Baums (cyberpatrol) - Saturday, 18 June 2011, 11:40 GMT
@Tom: But what counts is that you call Lennart's idea and implementation a standard. And that Lennart's software pulseaudio, which is definitely not working correctly at least on some audio cards, is also called and used as a standard by several distros and software developers (Gnome 3 depends on it as far as I know) is a technical reason and not just my personal issue.

And I just doubt that the ideas and implementations of someone whose software is not working for everybody (I'm not talking about a few bugs) shall be called and used as a standard at least not without discussing this across all distros and the major upstream projects.

Lennart is not a standardization board, he's just one developer.
Comment by Tom Gundersen (tomegun) - Saturday, 18 June 2011, 12:00 GMT
@Heiko:

Allow me to clarify: I'm only referring to the specification (the manpage), not any particular implementation of it. What I meant by calling it "a standard" is that it is an implementation independent specification that was developed as a collaboration and agreed upon by many (if not most) of the relevant parties. I did not mean to imply that it has anything to do with standardization bodies.

The specification I linked to is being implemented by Fedora, SuSE and Debian at least, and the systemd team that developed that document consists of people from all those distro's (and several others, including Arch).

If the systemd project succeeds in pushing these "tmpfiles" into the relevant upstream projects, then Arch would benefit from implementing the same specification so we would not have to maintain these things ourselves, but they would come bundled with upstream packages. If the systemd project fails in this goal, then we have not lost anything, as we would anyway have to implement something along these lines.
Comment by Tom Gundersen (tomegun) - Monday, 25 July 2011, 21:40 GMT
initscripts in [testing] now has the ability to create folders/files in /run at boot. it will read .conf files from /usr/lib/tmpfiles.d and it accepts "Lennart's Standard (TM)", with the exception that we ignore the Age parameter.

Some example files are shipped with initscripts, and the script /usr/lib/initscripts/arch-tmpfiles contains comments explaining how the different entries work.

Loading...