FS#41863 - [pacman] Basic support for systemd factory reset/atomic upgrades

Attached to Project: Pacman
Opened by Tobias Hunger (hunger) - Sunday, 07 September 2014, 18:51 GMT
Last edited by Allan McRae (Allan) - Sunday, 04 December 2022, 08:01 GMT
Task Type Feature Request
Category Packages: Core
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version 4.1.2
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 7
Private No



Systemd offers a factory reset feature for a while now. See http://0pointer.net/blog/projects/stateless.html for a discussion of the feature.

It would be nice if that was simple to use on arch Linux.

To support this feature the following changes would be needed on pacman (AFAICT:-):

* Move the pacman DB into /usr, maybe with a fallback for the current location in /var. I have mine in /usr/lib/pacman/db.
* Move the default pacman.conf and mirrorlist into /usr/share/factory/etc.
* Provide a tempfiles.d to copy those files to /etc if not already there and create the directories pacman needs to function.

With that a user should be able to make use of systemd's factory reset. These changes will also help a lot with the atomic update proposal Lennart made recently ( http://0pointer.net/blog/revisiting-how-we-put-together-linux-systems.html ).
This task depends upon

Closed by  Allan McRae (Allan)
Sunday, 04 December 2022, 08:01 GMT
Reason for closing:  Works for me
Additional comments about closing:  Configuration options are available to move database to wherever. Other parts of this request were never really clear. Open a new bug with specific details of the request if still needed.
Comment by Dave Reisner (falconindy) - Sunday, 07 September 2014, 19:04 GMT
So, I don't really see this as being something that would be generally useful. That said...

> With that a user should be able to make use of systemd's factory reset.
You can package up all of all "that" in a package for the AUR, with the exception of the migration of pacman's DB. I'd suggest doing this, and finding out who's interested, and where the missing pieces are.
Comment by Tobias Hunger (hunger) - Sunday, 07 September 2014, 19:16 GMT
I admit that I do not care too much for the factory reset. That is indeed mostly relevant for embedde systems. It would be nice to have for people working on embedded systems. Not sure there are too many of those that are based on arch though.

The atomic upgrade thing is a killer feature though!

Do a snapshot of your current /usr subvolume, run pacman -Syu in there and reboot (or start it in a VM:-). That gets you into your new system. If something is broken: You can go back to your old state with another reboot. That is super awesome and I am using these atomic upgrades (with ostree, not btrfs-snapshots though) for a couple of month now.

I do not want to go back to the "old" ways anymore:-)
Comment by Pablo Lezaeta (Jristz) - Sunday, 07 September 2014, 20:58 GMT
Apackage in AUr could be used to many tast but there are some like the DB and the pacman related that need land in pacman.
I think support them will not only alivianate future things if some user or future developer want integrate that enveded support, we not know the way that systemd will take in the future what if the factory reset become obligatory? we need to be prepared too.

And the pacman side of think will help not only for users that want the factory reset but that could help future profs or features.

for the pacnew thigs could be like:
ld-factory=X, etc=X, factory=X
Do nothing
old-factory=X, etc=X, factory=Y
copy new factory settings to /etc
old-factory=X, etc=Y, factory=X
do nothing
old-factory=X, etc=Y, factory=Y
do nothing
old-factory=X, etc=Y, factory=Z
warn user, optionally pacdiff dialog
Comment by Tobias Hunger (hunger) - Monday, 08 September 2014, 21:38 GMT
I do not think that merging old/new factory and /etc is something that needs to be tackled at this point. Getting the basics first would be greatly appreciated:-)
Comment by Pablo Lezaeta (Jristz) - Tuesday, 09 September 2014, 23:28 GMT
I add a feature for abs #41899 and mkinitcpio #41900 since both are arch projects and have config files on /etc/*.conf and this proposal could be extended to them too
Comment by Yamakaky (Yamakaky) - Wednesday, 10 September 2014, 19:36 GMT
I agree with the pacnew proposal but the last part could be changed to "warn user, put a pacnew file", which is the current behavior.

I was going to open a bug report for moving the database in /usr, but it's already in this one ^^
Comment by Allan McRae (Allan) - Monday, 15 September 2014, 23:30 GMT
  • Field changed: Attached to Project (Arch Linux → Pacman)
  • Assignment removed
Moving to the pacman section, because I'd guess this requires changes on pacman's end.

Unassigning everyone, because I am fairly sure no-one assigned cares that much...
Comment by Andrew Gregory (andrewgregory) - Thursday, 13 April 2017, 14:48 GMT
Does pacman actually need any changes for this? I don't see anything here that pacman can't already handle.
Comment by Tobias Hunger (hunger) - Thursday, 13 April 2017, 17:58 GMT
I am running a stateless arch setup for a long time now and pacman handles the task fine with the exception of the issues pointed out in my original report.

Moving the Pacman DB is perfectly possible right now with a simple change to pacman.conf. But I think it should be default to have the Pacman DB in /usr -- together with the files managed with the help of that db. No change for Pacman code required for this.

Having a systemd-ish configuration system with the defaults in /usr and the admin changes in /etc would be great though and does probably require changes to Pacman. The tmpfiles are just a way to work around the missing override feature.

Feel free to contact me if you want to know more about stateless arch Linux systems:-) They work great.
Comment by Yamakaky (Yamakaky) - Thursday, 13 April 2017, 18:00 GMT
Shouldn't the pacman db be in /var?

I'm interested ^^ Did you follow some articles or was it all by yourself? If so, you could add a page in the wiki ;)
Comment by Tobias Hunger (hunger) - Thursday, 13 April 2017, 19:37 GMT
For the stateless systems as proposed by Lennart, there is one /usr snapshot for each version of the system. Everything else is either created from data found there (e.g. using systemd-tmpfiles.d) or thrown away on reboot.

So with pacman storing its DB in /var that becomes problematic, even when you keep it around: There are potentially lots of /usr-snapshots, all potentially created from different sets of packages. Which of these is reflected by the pacman DB in /var? Keeping the pacman DB in /usr fixes this ambiguity -- and makes sure you can do "pacman -Ql foo" and will get results for exactly the "foo" currently installed in the running /usr-snapshot.

I did find a volunteer that wanted to go over my notes and repositories (https://gitlab.com/hunger/cleanroom is the script I use to install up all my arch systems *daily*) and wanted to turn that chaos into something that can be put into the arch wiki. Need to poke him:-)

It is a pretty simple process: I have my build server run the script for each system. That script generates an image with what will be /usr of the target system. That image is pushed onto the target system and made read-only there (simple btrfs snapshots). Then the signed kernel/initrd/commandline/EFI signature combo is taken from its place in the /usr-image and copied into /boot/EFI/Linux and then I reboot into the new system snapshot.
Comment by Yamakaky (Yamakaky) - Thursday, 13 April 2017, 19:40 GMT
What scope do you use it for? PC, a few servers, a datacenter?

In fact, it sounds very similar to http://coreos.com/ ^^
Comment by Doug Newgard (Scimmia) - Friday, 14 April 2017, 02:18 GMT
Yes, the database should be in /var except in this insane setup
Comment by Pablo Lezaeta (Jristz) - Friday, 14 April 2017, 02:59 GMT
If we add the option to configure the location of the config files, databases, and that stuff to fix this it also colateraly fix a few request and inconsistencies with locations and also add a tool for other pacman users (since pacman is not archlinux esclusive) to configure it at they need.

also for the database, why not ask to the systemd guys where a set of files like that should be located to support factory reseting?
Comment by Tobias Hunger (hunger) - Friday, 14 April 2017, 07:46 GMT
@Yamakaky: Yes, coreOS is one inspiration:-)

@Doug: Why should the DB be in /var anderen Why is my setup insane? Besides the usual "We always did it like that":-)
Comment by Jakub Klinkovský (lahwaacz) - Friday, 14 April 2017, 08:02 GMT
file-hierarchy(7) says:

/usr Vendor-supplied operating system resources. [...]
/var Persistent, variable system data. [...]

The pacman database is not vendor-supplied, it actually varies. That is assuming that you can actually use pacman in your system to install/remove packages. If that's not possible with the "stateless" system, why do you even bother with pacman? If it is possible, does the "factory reset" drop newly installed packages and restore deleted packages?
Comment by Eli Schwartz (eschwartz) - Friday, 14 April 2017, 14:30 GMT
It seems to me that "stateless" /usr images are meant to be created with pacstrap and never updated, merely recreated. They are, after all, supposed to be pristine base images, and you aren't supposed to use them long-term *at all*.

The only utility in providing the pacman databases with such images, is for the sake of incremental updates of the image. I don't see why is necessary though. And as lahwaacz pointed out, supporting that would violate the filesystem hierarchy. For no sufficiently justified reason, at all.
Comment by Andrew Gregory (andrewgregory) - Friday, 14 April 2017, 14:44 GMT
Setting aside the feasibility and usefulness of stateless systems, am I correct that the only change you actually need in pacman is the ability to fall back to a vendor-supplied config file if /etc/pacman.conf does not exist?
Comment by Tobias Hunger (hunger) - Friday, 14 April 2017, 15:45 GMT
@lahwaacz and @eschwartz: Pacman is a pretty useful tool to query installed software. That is why I keep it and it's package DB around. Also makes it easy to show which package versions changed between images. But yes, on my systems /usr is read-only and contains the pacman DB, so updating the DB or the 0ackages themselves is not possible on my machines.

The FHS is a recommendation, I follow it where it makes sense, but here it does not. A stateless system is way outside what is possible with that standard.

@andrewgregory: A stateless system is possible right now. Having the pacman DB in /usr would make setting up such a system slightly easier, as would having the settings nicely split between system defaults and admin overrides. The later would be nice in any context though:-) I did get used to having systemd-style config overrides really fast:-)

Comment by Andrew Gregory (andrewgregory) - Friday, 14 April 2017, 16:02 GMT
You are already free to move the DB into /usr either individually at run-time via pacman.conf or together with the cache and log at build-time via configure; so I don't know what you want to change. As somebody who doesn't use systemd, I also don't know what you mean by systemd-style overrides.
Comment by Eli Schwartz (eschwartz) - Friday, 14 April 2017, 16:18 GMT

Basically that would mean instead of using Include = to inline-include a file, you would include /etc/pacman.d/*.conf automatically and make them take priority over everything.

Of course, one could simply ship a modified pacman.conf and apply it via systemd tmpfiles.d and have done with it... no need for pacman to do anything...
Comment by Tobias Hunger (hunger) - Friday, 14 April 2017, 17:38 GMT
@andrewgregory: I'd like to see pacman put it's DB somewhere into /usr by default for new installs. Doing so makes a admittedly not very common use case simpler and does not effect other users.

In addition to that I'd like a better configuration handling:-)

All this is not critical to enable stateless systems though, so from my point of view this is a nice to have but not critical.
Comment by Yardena Cohen (yardenac) - Friday, 14 April 2017, 18:20 GMT
"The pacman database is not vendor-supplied, it actually varies."

You could argue that pacman *is* the vendor, so this is a grey area. The contents of /usr and the contents of the pacman db directly reflect each other. They move together and change together. I think it's perfectly logical to have them in the same place.
Comment by Jakub Klinkovský (lahwaacz) - Friday, 14 April 2017, 19:04 GMT
> The FHS is a recommendation, I follow it where it makes sense, but here it does not. A stateless system is way outside what is possible with that standard.

That file-hierarchy(7) manual I was referring to is part of systemd: https://www.freedesktop.org/software/systemd/man/file-hierarchy.html (The manual corresponding to the FHS is hier(7).)

So the structure outlined in file-hierarchy(7) should take into account all crazy ideas of the systemd developers. In particular, it is perfectly capable of handling the *stateless* system setup. The problem with a pacman database is that it holds a *state*, which was not accounted for in the Lennart's blog post. I don't deny the usefulness to keep the DB around, but that's not a stateless system - so there is no proposal (let alone standard) regarding how much state should be kept around and where.

> I'd like to see pacman put it's DB somewhere into /usr by default for new installs. Doing so makes a admittedly not very common use case simpler and does not effect other users.

That is a big false assumption. Firstly, if you change the DB location of only the new installs, but don't instruct users to migrate existing systems accrodingly, the documentation will be havoc. If you instruct to do the migration, then - if nothing else - it affects the configuration of backup scripts of many people. I don't se how the uncommon (perhaps even personal, given the lack of standards for partially-stateless systems) benefits can outweigh that and breaking the file-hierarchy/FHS.
Comment by Tobias Hunger (hunger) - Friday, 14 April 2017, 19:48 GMT
@lahwaacz: Systemd does not mention package databases anywhere in the document you link to.

Yes, the pacman DB is state, but so is the sum of all installed packages:-)
Comment by Eli Schwartz (eschwartz) - Friday, 14 April 2017, 20:02 GMT
Perhaps it would be useful at this time, to point out that RPM (the package management software used by the distro responsible for systemd) stores its own database in /var/lib/rpm/ ...

What does Red Hat/Fedora do to resolve the "need" of an installation database in the context of stateless systems? ;)
Comment by Andrew Gregory (andrewgregory) - Friday, 14 April 2017, 20:03 GMT
I fail to see how the default db location is relevant. This feature request has nothing to do with how pacman is packaged, only pacman itself. If you build pacman yourself you can change the default by setting $localstatedir to whatever you like; if you don't build pacman yourself, you're stuck with whatever the packager chose which has nothing to do with pacman.

So far, the only request that I can see that is related to pacman itself is for a change in how configuration is handled, but I'm still not sure exactly what you want or how it would help you achieve your goal.