Pacman

Welcome to the Pacman bug tracker. Please search the current bugs and feature requests before filing a new one! Use advanced search and select "Search in Comments".

* Please select the correct category and version.
* Write a descriptive summary, background info, and provide a reproducible test case whenever possible.
Tasklist

FS#66967 - pacman fails to handle out of space /tmp properly

Attached to Project: Pacman
Opened by Paul Hargreaves (yetanotheruser) - Wednesday, 10 June 2020, 13:43 GMT
Task Type Bug Report
Category General
Status Unconfirmed
Assigned To No-one
Architecture All
Severity Medium
Priority Normal
Reported Version 5.2.1
Due in Version Undecided
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

Summary and Info:
/tmp fills up during a pacman run, and when it does pacman continues without handling the issue or aborting. It then incorrectly assumes the packages have been upgraded.


Steps to Reproduce:
1. Fill /tmp (e.g. dd if=/dev/zero of=/tmp/whoops bs=1M)
2. pacman -Syu with packages such as linux that need updated.
3. Observe failure of package installation and lots of errors.
4. Then pacman -Syu and observe nothing to be done.


sudo pacman -Syu
:: Synchronising package databases...
downloading core.db...
downloading extra.db...
downloading community.db...
:: Starting full system upgrade...
resolving dependencies...
looking for conflicting packages...

Packages (16) amd-ucode-20200519.8ba6fa6-1 archlinux-keyring-20200603-1 ca-cer
tificates-mozilla-3.53-1 cryptsetup-2.3.3-1 device-mapper-2.02.187-3 git-2.27
.0-1 iana-etc-20200601-1 libgpg-error-1.38-1 libksba-1.4.0-1 linux-5.6.15.ar
ch1-1 linux-firmware-20200519.8ba6fa6-1 openssh-8.3p1-1 sqlite-3.32.1-1 syst
emd-245.6-2 systemd-libs-245.6-2 systemd-sysvcompat-245.6-2

Total Download Size: 189.99 MiB
Total Installed Size: 707.00 MiB
Net Upgrade Size: 28.98 MiB

:: Proceed with installation? [Y/n] :: Retrieving packages...
downloading amd-ucode-20200519.8ba6fa6-1-any.pkg.tar.zst...
downloading archlinux-keyring-20200603-1-any.pkg.tar.zst...
downloading iana-etc-20200601-1-any.pkg.tar.zst...
downloading libgpg-error-1.38-1-x86_64.pkg.tar.zst...
downloading iana-etc-20200601-1-any.pkg.tar.zst...
downloading libgpg-error-1.38-1-x86_64.pkg.tar.zst...
downloading systemd-libs-245.6-2-x86_64.pkg.tar.zst...
downloading ca-certificates-mozilla-3.53-1-x86_64.pkg.tar.zst...
downloading device-mapper-2.02.187-3-x86_64.pkg.tar.zst...
downloading cryptsetup-2.3.3-1-x86_64.pkg.tar.zst...
downloading libksba-1.4.0-1-x86_64.pkg.tar.zst...
downloading systemd-245.6-2-x86_64.pkg.tar.zst...
downloading linux-5.6.15.arch1-1-x86_64.pkg.tar.zst...
downloading linux-firmware-20200519.8ba6fa6-1-any.pkg.tar.zst...
downloading openssh-8.3p1-1-x86_64.pkg.tar.zst...
downloading sqlite-3.32.1-1-x86_64.pkg.tar.zst...
downloading systemd-sysvcompat-245.6-2-x86_64.pkg.tar.zst...
downloading git-2.27.0-1-x86_64.pkg.tar.zst...
checking keyring...
checking package integrity...
loading package files...
checking for file conflicts...
checking available disk space...
:: Running pre-transaction hooks...
(1/1) Removing linux initcpios...
:: Processing package changes...
upgrading amd-ucode...
upgrading archlinux-keyring...
warning: warning given when extracting .INSTALL (Write failed)
error: could not copy tempfile to /tmp/alpm_HvbjOE/.INSTALL (No space left on de
vice)
upgrading iana-etc...
upgrading libgpg-error...
upgrading systemd-libs...
upgrading ca-certificates-mozilla...
upgrading device-mapper...
upgrading cryptsetup...
upgrading git...
warning: warning given when extracting .INSTALL (Write failed)
upgrading libksba...
upgrading systemd...
warning: warning given when extracting .INSTALL (Write failed)
error: could not copy tempfile to /tmp/alpm_FGpQIG/.INSTALL (No space left on de
vice)
upgrading linux...
upgrading linux-firmware...
upgrading openssh...
warning: warning given when extracting .INSTALL (Write failed)
error: could not copy tempfile to /tmp/alpm_xO9ghI/.INSTALL (No space left on de
vice)
upgrading sqlite...
upgrading systemd-sysvcompat...
:: Running post-transaction hooks...
( 1/13) Creating system user accounts...
( 2/13) Updating journal message catalog...
( 3/13) Reloading system manager configuration...
( 4/13) Updating udev hardware database...
( 5/13) Applying kernel sysctl settings...
( 6/13) Creating temporary files...
( 7/13) Reloading device manager configuration...
( 8/13) Arming ConditionNeedsUpdate...
( 9/13) Updating module dependencies...
(10/13) Updating linux initcpios...
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'default'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux.img
/usr/lib/initcpio/functions: line 692: printf: write error: No space left on dev
ice
==> Starting build: 5.6.15-arch1-1
-> Running build hook: [base]
install: error writing '/tmp/mkinitcpio.MiDHjT/root/bin/busybox': No space left
on device
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
install: error writing '/tmp/mkinitcpio.MiDHjT/root/usr/sbin/kmod': No space lef
t on device
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
install: error writing '/tmp/mkinitcpio.MiDHjT/root/usr/sbin/blkid': No space le
ft on device
[snip]
install: error writing '/tmp/mkinitcpio.MiDHjT/root/init_functions': No space le
ft on device
install: error writing '/tmp/mkinitcpio.MiDHjT/root/init': No space left on devi
ce
-> Running build hook: [udev]
install: error writing '/tmp/mkinitcpio.MiDHjT/root/usr/lib/systemd/systemd-udev
d': No space left on device
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
[snip]
-> Running build hook: [block]
/usr/lib/initcpio/functions: line 386: cannot create temp file for here-document
: No space left on device
/usr/lib/initcpio/functions: line 374: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 374: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 403: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 403: _addedmodules[]: bad array subscript
==> ERROR: module not found: `'
/usr/lib/initcpio/functions: line 386: cannot create temp file for here-document
: No space left on device
[snip]
==> ERROR: module not found: `'
/usr/lib/initcpio/functions: line 386: cannot create temp file for here-document
: No space left on device
/usr/lib/initcpio/functions: line 374: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 374: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 403: _addedmodules[]: bad array subscript
/usr/lib/initcpio/functions: line 403: _addedmodules[]: bad array subscript
[...snip]
cp: error writing '/tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel
/hid-roccat-koneplus.ko.xz': No space left on device
cp: error writing '/tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel
/hid-roccat-arvo.ko.xz': No space left on device
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-zydacron.k
o.xz: File format not recognized
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-roccat-kon
epure.ko.xz: File format not recognized
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-aureal.ko.
xz: File format not recognized
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-petalynx.k
o.xz: File format not recognized
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-tmff.ko.xz
: File format not recognized
[snip]
xz: /tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/kernel/hid-roccat-arv
o.ko.xz: File format not recognized
==> Generating module dependencies
install: error writing '/tmp/mkinitcpio.MiDHjT/root/lib/modules/5.6.15-arch1-1/m
odules.builtin': No space left on device
awk: cmd. line:1: (FILENAME=/lib/modules/5.6.15-arch1-1/modules.order FNR=200) f
atal: print to "standard output" failed: No space left on device
depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.MiDHjT/root/lib/modul
es/5.6.15-arch1-1/kernel/hid-tmff.ko.xz: Invalid argument
depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.MiDHjT/root/lib/modul
es/5.6.15-arch1-1/kernel/hid-multitouch.ko.xz: Invalid argument
depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.MiDHjT/root/lib/modul
es/5.6.15-arch1-1/kernel/hid-generic.ko.xz: Invalid argument
depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.MiDHjT/root/lib/modul
es/5.6.15-arch1-1/kernel/hid-redragon.ko.xz: Invalid argument
depmod: ERROR: failed to load symbols from /tmp/mkinitcpio.MiDHjT/root/lib/modul
es/5.6.15-arch1-1/kernel/hid-xinmo.ko.xz: Invalid argument
[snip]
==> WARNING: errors were encountered during the build. The image may not be comp
lete.
==> Building image from preset: /etc/mkinitcpio.d/linux.preset: 'fallback'
-> -k /boot/vmlinuz-linux -c /etc/mkinitcpio.conf -g /boot/initramfs-linux-fal
lback.img -S autodetect
/usr/bin/mkinitcpio: line 380: cannot create temp file for here-document: No spa
ce left on device
/usr/lib/initcpio/functions: line 692: printf: write error: No space left on dev
ice
==> Starting build: 5.6.15-arch1-1
-> Running build hook: [base]
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/bin/busybox': No space left
on device
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/sbin/kmod': No space lef
t on device
[snip]
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/50-udev
-default.rules': No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/60-pers
istent-storage.rules': No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/64-btrf
s.rules': No space left on device
[snip]
/usr/lib/initcpio/functions: line 574: cannot create temp file for here-document
: No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/50-udev
-default.rules': No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/60-pers
istent-storage.rules': No space left on device
install: error writing '/tmp/mkinitcpio.3A6Ogq/root/usr/lib/udev/rules.d/64-btrf
s.rules': No space left on device
[snip]
depmod: ERROR: Could not create index 'modules.dep'. Output is truncated: No spa
ce left on device
==> Creating gzip-compressed initcpio image: /boot/initramfs-linux-fallback.img
==> WARNING: errors were encountered during the build. The image may not be comp
lete.
error: command failed to execute correctly
(11/13) Reloading system bus configuration...
(12/13) Updating the info directory file...
(13/13) Rebuilding certificate stores...

At this point, I'd expect any packages that had been extracted correctly to have been installed (none, in my case) and those that had failed be able to be installed again.

However....

pacman -Syu
:: Synchronising package databases...
core is up to date
extra is up to date
community is up to date
:: Starting full system upgrade...
there is nothing to do


So pacman has assumed success where there was none.

Logs are being added to /var/log/pacman, e.g.
[2020-06-05T08:01:17+0100] [ALPM-SCRIPTLET] depmod: ERROR: failed to load symbol
s from /tmp/mkinitcpio.3A6Ogq/root/lib/modules/5.6.15-arch1-1/kernel/hid-microso
ft.ko.xz: Invalid argument
[2020-06-05T08:01:17+0100] [ALPM-SCRIPTLET] ==> WARNING: errors were encountered
during the build. The image may not be complete.
etc.


This task depends upon

2020-06-10: A task closure has been requested. Reason for request: Per others, not a pacman issue?
Comment by Doug Newgard (Scimmia) - Wednesday, 10 June 2020, 13:51 GMT
That's not pacman, that's a hook, and the update itself IS successful. I don't see a pacman problem here.
Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 14:27 GMT
The package installed correctly and there are no traces of the old package left anywhere on your system, but its PostTransaction hook failed with "error: command failed to execute correctly".

> At this point, I'd expect any packages that had been extracted correctly to have been installed (none, in my case)

You updated 16 packages, of which 4 raised install scriptlet errors, and 1 post transaction hook likewise raised errors.


That's 11 packages which installed flawlessly without a hitch, and 3 packages (git, systemd, and openssh) which failed to run an install scriptlet which currently checks for versions you don't have and performs migrations you don't need

> and those that had failed be able to be installed again.

... and 1 package (archlinux-keyring) which was supposed to run:

pacman-key --populate archlinux

Plus the mkinitcpio hook, which you can solve by running mkinitcpio --allpresets.

What do you expect pacman to do about this? Why do you think you cannot install them again?

sudo pacman -S archlinux-keyring linux

It will reinstall the packages and run the commands which previously failed. Assuming it can, since /tmp is broken for a reason.

Also assuming the scripts actually do anything now, because they are called with positional parameters for the old and new versions of the package, and many install scriptlets only do something when the old version was a certain version. You may need to inspect them manually using pacscripts $pkgname.

Install script errors should be recoverable ones, but it does require user intervention. If errors are logged, investigate them.

You can also wrap pacman in a btrfs snapshot (see the 'snap-pac' package) to roll back your entire filesystem on errors, and try to repeat the upgrade. pacman will not incorporate this feature into itself automatically, and it will only work for filesystems which you set up with snapshotting capabilities.
Comment by Paul Hargreaves (yetanotheruser) - Wednesday, 10 June 2020, 14:56 GMT
My system is fine, it's a VM that's sitting on zfs with snapshots so recovery wasn't an issue, and in my case was a simple rollback, make sure /tmp was sized approprately, then re-ran.

What I'd expect pacman to do is re-run whatever failed again, until success. If that isn't reasonable, then maybe I'd expect pacman to stop on the first error encountered, since debugging a single broken thing is probably easier than attempting to debug lots of broken things. Assuming they're broken, which I'd need to figure out somewhere.

Assuming that isn't pacman's job, where in the docs should I be looking? e.g. https://wiki.archlinux.org/index.php/Pacman#Troubleshooting section doesn't cover this case particularly well.

Where does pacscripts come from?

Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 15:20 GMT
It comes from the pacman-contrib package.

And aborting mid transaction would be a lot more dangerous because then you cannot even rely on programs having the shared libraries they need, so the only way to recover is using the installation media.

Failing install scriptlets or hooks should never cause the command prompt to stop working. Aborted transactions can and do.
Comment by Eli Schwartz (eschwartz) - Wednesday, 10 June 2020, 15:22 GMT
BTW one thing which might help is to teach mkinitcpio to fail earlier if it cannot write to /tmp. This would result in far fewer error logs to read through in order to figure out everything that failed and needs to be repeated.

Loading...