Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines
Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.
REPEAT: Do NOT report bugs for outdated packages!
https://wiki.archlinux.org/title/Bug_reporting_guidelines
Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.
REPEAT: Do NOT report bugs for outdated packages!
FS#73352 - [pacman] "pacman -Syu" has overwritten all system files with empty files
Attached to Project:
Arch Linux
Opened by andrew (andrew-ld) - Thursday, 13 January 2022, 20:19 GMT
Last edited by Balló György (City-busz) - Wednesday, 07 June 2023, 14:50 GMT
Opened by andrew (andrew-ld) - Thursday, 13 January 2022, 20:19 GMT
Last edited by Balló György (City-busz) - Wednesday, 07 June 2023, 14:50 GMT
|
DetailsDescription:
today while running a "pacman -Syu" I noticed that ldconfig was starting to complain that some shared libraries were empty files. not paying much attention to this problem I restarted the computer and it could no longer load some system modules, I noticed that the initramfs was not updated so I tried to do it manually but the .preset file was empty, doing some analysis I noticed that every single updated package overwrote both the configuration and binaries with empty files. Additional info: package version: Pacman v6.0.1 - libalpm v13.0.1 - I honestly have no idea how something of this caliber could have happened, I have no modified system configuration, I keep the system as standard as possible with defaults. - I tried to reinstall all system packages (I cleared pacman cache first), after this massive reinstall the system binaries are restored. system configurations, on the other hand, remained empty files. - I don't have any third party repo and i haven't updated any packages from aur, i don't have any unofficial pacman hooks. - I checked the system logs with journalctl, I fscked the system filesystem but there is no sign of corruption which could indicate that the problem has existed since before this update. Steps to reproduce: pacman -Syu |
This task depends upon
Other adjacent issues were due to bad RAM causing filesystem corruption, and when /var/cache/pacman/pkg is a symlink and an update to pacman causes the package cache to vanish part-way through the update.
I'm not sure there is anything we can do here. This appears unique in millions of pacman updates, so without further information there is nothing we can do.
https://bugs.archlinux.org/task/73426
When building with `makepkg`, it's there, but when building with extra-x86_64-build from devtools 20211129-1 and installing the package with `pacman -U`, it's empty.
I tried different compilation flags for gcc, and stripping and not stripping, but I get the same result.
Could this be connected to devtools somehow?
It did indeed overwrite multiple files with empty files- pacman will not error after it fails to write one file.
when disk was full during full system upgrade, the pacman replaced lots of files with empty ones rendering system useless or unbootable.
back then the solution was to chack for free space before proceeding with upgrade. which helped a lot at the time.
but also was a workaround in some sense. because the operation is not atomic. the free space can disappear between the check and actual package upgrade.
What about adding additional free space checks?
I had a similar problem when running out of disk space sometime during an upgrade. I noticed the problem while `pacman` was running, so killed the process before much damage was done. Ran out of space because `/var/cache/pacman` had numerous old packages and `/usr/lib/modules` had dozens of no longer used folders. I ran `pacman -Sc` and installed `kernel-modules-hook` to clean up the folders.
Same problem. I think pacman should reserve the space beforehand in some smart way.
Eg. unpack/decompress the files first (to the same fs/dir with some .suffix !) and then mv them to final location (overwriting original files and only after that removing the rest).
Note this approach would require twice the free space, since every package has to be unpacked twice for a little while, which is not that great idea, especialy for packages with size of several GBs...
BTW How are Debian/apt and other distros solving this problem?
The problem with free space checks is they are not accurate on all file systems. My systems frequently have compression settings enabled and BTRFS and other modern file systems have de-duplication, sometimes COW, etc. These mean free space calculations are only even projected estimates, not hard values.
- if the new file is larger than the present file on the filesystem, it should be resized first with fallocate.
- if the new file is less large than the file present on the filesystem it should be resized only after writing the new content to it.
in either case the file to be overwritten should be opened in append mode to prevent it from becoming 0 bytes and some other program that found more space on the disk starts writing new files and there is no more space available for the package to be updated.
as an additional solution in combination with the previous one or an alternative would be to stop any operation after the one that went wrong because of insufficient space.
It's anyway better than breaking the system, in my opinion, as it gives chances to manually restore normal operation.
It's also better than completely stop transaction in the middle, as it will require non-trivial (I think) actions from admin to fix the system.
Behavior could be configurable of course, like:
on_nospace = stop-transaction | pause-to-ask | ignore