FS#66967 - pacman fails to handle out of space /tmp properly

Attached to Project: Pacman
Opened by Paul Hargreaves (yetanotheruser) - Wednesday, 10 June 2020, 13:43 GMT
Last edited by Allan McRae (Allan) - Wednesday, 09 December 2020, 00:39 GMT
Task Type Bug Report
Category General
Status Unconfirmed
Assigned To No-one
Architecture All
Severity Medium
Priority Normal
Reported Version 5.2.1
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No


Summary and Info:
/tmp fills up during a pacman run, and when it does pacman continues without handling the issue or aborting. It then incorrectly assumes the packages have been upgraded.

Steps to Reproduce:
1. Fill /tmp (e.g. dd if=/dev/zero of=/tmp/whoops bs=1M)
2. pacman -Syu with packages such as linux that need updated.
3. Observe failure of package installation and lots of errors.
4. Then pacman -Syu and observe nothing to be done.

(10/13) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
/usr/lib/initcpio/functions: line 692: printf: write error: No space left on dev
==> Starting build: 5.6.15-arch1-1
-> Running build hook: [base]
install: error writing '/tmp/mkinitcpio.MiDHjT/root/bin/busybox': No space left
on device

Edit: pacman does use /tmp for extracting install scripts. So I think this bug is still valid, just not for the reason submitted...
This task depends upon

Comment by Doug Newgard (Scimmia) - Wednesday, 10 June 2020, 13:51 GMT
That's not pacman, that's a hook, and the update itself IS successful. I don't see a pacman problem here.
Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 14:27 GMT
The package installed correctly and there are no traces of the old package left anywhere on your system, but its PostTransaction hook failed with "error: command failed to execute correctly".

> At this point, I'd expect any packages that had been extracted correctly to have been installed (none, in my case)

You updated 16 packages, of which 4 raised install scriptlet errors, and 1 post transaction hook likewise raised errors.

That's 11 packages which installed flawlessly without a hitch, and 3 packages (git, systemd, and openssh) which failed to run an install scriptlet which currently checks for versions you don't have and performs migrations you don't need

> and those that had failed be able to be installed again.

... and 1 package (archlinux-keyring) which was supposed to run:

pacman-key --populate archlinux

Plus the mkinitcpio hook, which you can solve by running mkinitcpio --allpresets.

What do you expect pacman to do about this? Why do you think you cannot install them again?

sudo pacman -S archlinux-keyring linux

It will reinstall the packages and run the commands which previously failed. Assuming it can, since /tmp is broken for a reason.

Also assuming the scripts actually do anything now, because they are called with positional parameters for the old and new versions of the package, and many install scriptlets only do something when the old version was a certain version. You may need to inspect them manually using pacscripts $pkgname.

Install script errors should be recoverable ones, but it does require user intervention. If errors are logged, investigate them.

You can also wrap pacman in a btrfs snapshot (see the 'snap-pac' package) to roll back your entire filesystem on errors, and try to repeat the upgrade. pacman will not incorporate this feature into itself automatically, and it will only work for filesystems which you set up with snapshotting capabilities.
Comment by Paul Hargreaves (yetanotheruser) - Wednesday, 10 June 2020, 14:56 GMT
My system is fine, it's a VM that's sitting on zfs with snapshots so recovery wasn't an issue, and in my case was a simple rollback, make sure /tmp was sized approprately, then re-ran.

What I'd expect pacman to do is re-run whatever failed again, until success. If that isn't reasonable, then maybe I'd expect pacman to stop on the first error encountered, since debugging a single broken thing is probably easier than attempting to debug lots of broken things. Assuming they're broken, which I'd need to figure out somewhere.

Assuming that isn't pacman's job, where in the docs should I be looking? e.g. https://wiki.archlinux.org/index.php/Pacman#Troubleshooting section doesn't cover this case particularly well.

Where does pacscripts come from?

Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 15:20 GMT
It comes from the pacman-contrib package.

And aborting mid transaction would be a lot more dangerous because then you cannot even rely on programs having the shared libraries they need, so the only way to recover is using the installation media.

Failing install scriptlets or hooks should never cause the command prompt to stop working. Aborted transactions can and do.
Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 15:22 GMT
BTW one thing which might help is to teach mkinitcpio to fail earlier if it cannot write to /tmp. This would result in far fewer error logs to read through in order to figure out everything that failed and needs to be repeated.
Comment by Allan McRae (Allan) - Wednesday, 09 December 2020, 00:38 GMT
pacman does use /tmp for extracting install scripts. So I think this bug is still valid, just not for the reason submitted...