Release Engineering

This project is intented for all release related issues (isos, installer, etc), under the umbrella of the ArchLinux Release Engineers
Tasklist

FS#45618 - [archiso] pacman -Syu crashes liveiso

Attached to Project: Release Engineering
Opened by Ingo Albrecht (indigo) - Saturday, 11 July 2015, 21:18 GMT
Last edited by Gerardo Exequiel Pozzi (djgera) - Tuesday, 29 September 2015, 00:15 GMT
Task Type Bug Report
Category ArchISO
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

This originates from a wiki discussion in the pacman article.[1]

Booting the live medium and executing pacman -Syu leads to the system crashing. As pointed out by Lahwaacz in the discussion, also a "dd if=/dev/zero of=testfile bs=1M count=1000" leads to the same crash.

A booted iso df -h on my system shows:
/dev/mapper/arch_airootfs 32GB size
cowspace 5.9GB size

Obviously the 32GB is nonsense. Can it be that this confuses pacman and dd?

Ideally pacman's default checksize option should warn the user and abort the upgrade, but it does not. Likewise, dd should just error out with "no space left".

Obviously it is arguable in which cases one would want to run an upgrade on a booted liveiso, but there are reasons. The user reporting it wanted to repair a borked pacman database, others will have other reasons. In any case the iso should not crash like this.

[1] https://wiki.archlinux.org/index.php/Talk:Pacman#Problem_with_section_.22pacman_crashes_during_an_upgrade.22
This task depends upon

Closed by  Gerardo Exequiel Pozzi (djgera)
Tuesday, 29 September 2015, 00:15 GMT
Reason for closing:  None
Additional comments about closing:  switched to overlayfs in archiso-23 (available in snapshoot 2015.10)
Comment by Gerardo Exequiel Pozzi (djgera) - Monday, 13 July 2015, 23:50 GMT
"Not a bug". Yes, is not the best thing, but nothing can be done, please let me explain...

/ is the "sum" of a ext4 image of 32GB (read-only) [sparse] + cow-file (256MB by default) (read-write) [sparse] inside a limited tmpfs (75% of your RAM).

When cowfile is full...boom!. You can change cow-file size, and also you can store it where you want, and idealy set a size of 32GB.

Sure, I can create an ext4 image (RO) with the just space needed + some percent for metadata, but you can not extent it, in a easy way later, if you need (resize2fs, more cow space needed on resize operation for new bitmaps and inodes tables, etc)...
Comment by Jakub Klinkovsk√Ĺ (lahwaacz) - Tuesday, 14 July 2015, 05:43 GMT
Is there any chance to set the cow-file size dynamically based on the amount of RAM available?
Comment by Gerardo Exequiel Pozzi (djgera) - Tuesday, 14 July 2015, 22:54 GMT
depends what you call dynamic. As said before, you can set the cow-image to 32GB, and when tmpfs is full...boom!
Comment by Ingo Albrecht (indigo) - Thursday, 16 July 2015, 14:24 GMT
Thanks for explaining.
In absence of an easy way to make it dynamic, maybe a medium iso size? To keep it simple for both archs: Booting i686 comes up with 1.8GB tmpfs here. As long as the ext4 arch_aifsroot image is kept smaller than that, pacman's checksize should figure it and error out gracefully? Would that leave enough spare for complex install/partition operations? What do you think?

Any other alternatives to have pacman error out than crash? Extra wrapping it on the iso perhaps. Not elegant but if it is a simpler universal work-around - why not. A wrapper like "Warning, the installation ISO itself is not designed to be updated dynamically and may crash the livesystem due to ram constraints. You can run the command from the livesystem to update a mounted installed system anytime." should suffice (imo). Just because it is an Arch showcase command.
Comment by Gerardo Exequiel Pozzi (djgera) - Thursday, 16 July 2015, 23:06 GMT
The ISO size is just what is needed.

Some time ago, ext4 image was build based on space usage + estimation of a 50% free space. So if the ext4 usage was ~1G, free space of ~1G, and the ext4 of 2G (sparse) and compressed inside an squashfs image. So the COW image size (ideal) should be 2G, if you RAM was below and tmpfs becomes full...boom!

A fixed size of 32G at build time (NOW) instead of a dynamic size at build time lots of things are simplified and much more for systemd units in a future.

The situation can be better using overlayfs (implemented some time ago in archiso, but not used by default, maybe is time to test again, and bring the idea for next iso snapshot), in this case the "free space" reported is from the upperdir (tmpfs).

Nothing to do with pacman, absolutely unrelated to it. Documentation can be made in the wiki.

https://projects.archlinux.org/archiso.git/commit/?id=c62a9d748b908565426857fb35bbbe83154202fe [archiso] Always use a 32GiB filesystem size for airootfs
https://projects.archlinux.org/archiso.git/commit/?id=36459f3acc217ff6a0b4ad1b6049d68a8da413e2 [archiso] Drop aitab support
https://projects.archlinux.org/archiso.git/commit/?id=a637bdb8574168ab4cddef13d5de7f458650bda5 [archiso] Add optional OverlayFS support

Comment by Gerardo Exequiel Pozzi (djgera) - Saturday, 18 July 2015, 21:19 GMT
https://lists.archlinux.org/pipermail/arch-releng/2015-July/003569.html [arch-releng] [RFC] Switch from dm-snapshot to overlay
Comment by Ingo Albrecht (indigo) - Thursday, 23 July 2015, 23:26 GMT
(sorry, no time to reply earlier) So, you already have another alternative implemented! Thanks, looking forward to the result of the rfc. As suggested, I have added an item with your above info in the pacman article [1] and will update it with this flyspray's result later.

For the regular install guide/Beginner guide in the wiki, there is no real place to mention the current limitation because readers are guided towards using pacstrap anyway. Nonetheless, other places do mention pacman -Syu with less context - prominently, for example [2]. One can't really blame new users trying out an Arch iso first time to check how pacman works from the live console and this issue easily leaves an odd impression, would be nice to have it dealt with.

[1] https://wiki.archlinux.org/index.php?title=Pacman&type=revision&diff=387253&oldid=385601
[2] https://www.archlinux.org/download/
Comment by Gerardo Exequiel Pozzi (djgera) - Friday, 24 July 2015, 02:22 GMT
> If you ''require'' an install media with persistent dataspace, the [[Archiso]] build script can be used to create an image using an overlay filesystem

Maybe I understand bad, but this phrase sounds like the only way for persistence is using overlay. As you know, in archiso, since I implemented dm-snapshot this is posible using "cow_label=/cow_device=", Indeed you can have multiple persistence "snapshots" using "cow_directory=" ;)
And for space issues, since the cow-space is by default tmpfs -> you can still assign big space (cow_spacesize=), and active a swap space, zram... ;)Lots of options...
Comment by Ingo Albrecht (indigo) - Saturday, 25 July 2015, 18:24 GMT
Well, yes - very flexible and plenty options (!), but most don't help for the official media at boot/runtime for this issue. The intent of that sentence was mainly to point to the fact that updating the booted official media at runtime generally is nonsense anyway. You are right, I've added a sentence about swap to work-around and shortened respective sentence.[1] It now reads:
> If you require an install media with persistent dataspace, the Archiso build script can be used to create one along with its boot options.

[1] https://wiki.archlinux.org/index.php?title=Pacman&type=revision&diff=387767&oldid=387253
Comment by Gerardo Exequiel Pozzi (djgera) - Sunday, 26 July 2015, 06:20 GMT
They do not help? Just increase the cow-file to a reasonable size and you can full update the medium if you want, even reinstalling all packages (~750M installed + 200M downloaded packages). So a cow_filesize=1G is just sufficient. There is no need to swap, zram, maybe for extreme cases where you want to install lots of things at runtime.
Comment by Ingo Albrecht (indigo) - Sunday, 26 July 2015, 10:29 GMT
Before my last comment I simply booted the 201507 iso i686 with cow_spacesize=32G (following your first comment) and it made no change (unsurprisingly, yes). Now I did it again with cow_spacesize=1G (assuming your cow_filesize=1G was a typo) and:
pacman -Syu --ignore linux (~63MB download) indeed worked without running out of memory (it crashes without the boot option).
Reinstalling (pacman -Qnq | pacman -S -), however, crashed just again (download size 216M, (re-)installed size 703M). free cowspace before this crash reinstall was 1017M.
So cow_spacesize=1G helped, but I would not call it a viable work-around, more playing dice with the sympton.

The booted default official iso environment should allow pacman to just error out when there is no space to perform the operation.
Comment by Gerardo Exequiel Pozzi (djgera) - Sunday, 26 July 2015, 22:02 GMT
1G was an approximation, the real size should be (tested it, just for fun) cow_spacesize=1200M, after pacman -Sy $(pacman -Qq) in x86_64 this leaves tmpfs "/run/archiso/cowspace" with 100M free. If your system crash, then you does not have sufficient memory.

Sorry, there is nothing to do, other than understand how it works. If you need more space, then increase it.

Sure the "correct" way is to create a filesystem of just the space used plus minor "free space", but has a lots of limitations. Or use overlay filesystem, and live with it, is not a "real filesystem", some crazy things happens, but at least, people will not complain about reported free space != real free space.

By default is set to 256M because releng iso is not supposed to be update. If I increase the size and the cowfile is located on a filesystem that does not support sparse files, will use more space than needed.

What I can do however, is increase default of cow_spacesize to 2G, for people who want to reinstall all packages in the live-enviroment plus install some few packages, since many does not know archiso boot options and how these works. In some way this "revert back" from fixed 256M. Looks like lots of people wants to do these funny things, or newbie people who do not follow the correct install process and instead of installing Arch in the target media they does on live-medium, and complains about crash!.

Comment by Ingo Albrecht (indigo) - Monday, 27 July 2015, 20:09 GMT
Thanks for looking into it again. I've not built an iso with the tools yet, so I cannot comment on the pros/cons (what crazy things happen?) of using overlayfs for it yet. It now sounds like you want to give it more time before deciding to switch or not. Also working around the issue should not make the process to create the images more complicated of course.

Now your idea of setting cow_spacesize=2G sounds like a pragmatic way to have a work-around for most users. The only ones left out are machines with very little memory. But if I see it correctly even these will not encounter more crashes with it. The rest, with enough memory for cowspace 2GB, should only benefit. Theoretically, they could even update a six month old ISO (e.g. booted for recovery) without it crashing. So, this sounds like a good idea to deal with it, if you ask me.

> Looks like lots of people wants to do these funny things, or newbie people who do not follow the correct install process and instead of installing Arch in the target media they does on live-medium, and complains about crash!.

I don't know how many users encounter the issue, but the point is that nowhere it states that the install media is not meant to be updated and the space management does not allow for all regular operations. How are users supposed to know it? Couple of ideas:
1. the media root has the converted installation guide as "install.txt". We could mention it in there, for example at the end of https://wiki.archlinux.org/index.php/Installation_guide#Download
I have made a proposal in the wiki for this already. If it gets added, you can merge it to the "install.txt" so that we have a mention of the general constraint in the media itself.
2. You could add an /etc/issue statement for the official install iso. This could read "Welcome to the Arch Linux installation media! Be aware the media is designed to enable a new installation or recovery of an existing system. Due to resource constraints (memory) you cannot perform all regular operations, e.g. software updating, to the live system at runtime."
Comment by Gerardo Exequiel Pozzi (djgera) - Monday, 27 July 2015, 21:50 GMT
See overlayfs.txt more specifically [Non-standard behavior] ;) Yes, should be delayed, next ISO will contain linux-4.1, and seems to be buggy according to users reports around, I do not want to mix two possible issues ;)

Docs says, what should be done ;) Why docs should say what should _not_ be done? Leave these lots of things if users want to experiment. :D
Comment by Ingo Albrecht (indigo) - Wednesday, 29 July 2015, 19:39 GMT
Why? Because the live environment's memory "management" imposes usage restrictions that are not mentioned anywhere. But anyhow, glad we figured some alternatives to potentially improve it. So, it's ok, leave restrictions undocumented and let the users crash. I have no heartblood in this ;)
Comment by Gerardo Exequiel Pozzi (djgera) - Wednesday, 29 July 2015, 22:33 GMT
Depends the point of view of who imposes the storage restriction. default settings? user settings? system settings?

I missed to mention, an obvious case when I say: Sure the "correct" way is to create a filesystem of just the space used plus minor "free space". (Only and only if you have sufficient memory, then you can change all files). Example:
* ext4: total 1.3G / used 1.2G / free 0.1G | cow-file: 1.3G
If you want to upgrade all files (upgrade all packages), you can with a max net size of ~100M, like on normal system. If you have at least 2G of RAM: nice will work!, less system will run out-of-memory. Maybe crash inmediatelly, due no swap.

Documentation can say something like: "WARNING: Due design limitation, all writes in the the root filesystem, are limited to the space available in /run/archiso/cowspace, not only in the root filesystem. Also the space available in "cowspace" is limited to the available page cache". (page-cache -> cowspace -> filesystem)

You can enable swap, since tmpfs can use swap, but, but...I do not recommend it at all. Put the cow-file on disk directly!.
Comment by Ingo Albrecht (indigo) - Friday, 31 July 2015, 16:27 GMT
Combining your warning with drafts in the wiki[1], I propose a doc mention to be something like:
"Note that due to design limitations, all writes to the root filesystem of the live system are limited to the space available in /run/archiso/cowspace. Once the space is depleted from resource intensive tasks, for example updating packages, the live system will crash."

[1] https://wiki.archlinux.org/index.php/Talk:Installation_guide#addition_for_awareness_of_installation_iso_constraints
Comment by Gerardo Exequiel Pozzi (djgera) - Friday, 31 July 2015, 19:23 GMT
Yes, but is important that "cowspace", as said before, is not the only indication, because you can have "cowspace" (tmpfs) of 2G but your system has 1G RAM. Under this scenario if you try to fill it, the most probably things that happens is a kernel deadlock, OOM-killer everywhere.
Comment by Ingo Albrecht (indigo) - Friday, 31 July 2015, 21:26 GMT
Yeah, but OOM is OOM - regardless. The more details one wants to bring in, the longer it gets. Basic intention was to warn about the restriction, too many details confuse. So the message for install.txt "don't try to update the iso and cry if it does not play". If users do require to know more, they should indeed look into the tools/docs.

Anyway, here is another,slightly longer version (avoiding a specific 2G and "cow_spacesize=" parameter mention) to choose from:

"Note that due to design limitations, all writes to the root filesystem of the live system are limited to the real memory available for /run/archiso/cowspace, which has a default regardless of available memory. Once it is depleted from resource intensive tasks, for example updating packages, the kernel will lock."
Comment by Gerardo Exequiel Pozzi (djgera) - Sunday, 02 August 2015, 17:19 GMT
The install.txt file just references the wiki, nothing to do here. Anyway such message is not releated to installation, nothing to change @ "Installation_guide" (btw: I do not have perms to change it).
You are free to modify the wiki as you want. I will add something to README.knowissues and nothing more. If you want to do in "motd", send a RFC patch to arch-releng.
Comment by Ingo Albrecht (indigo) - Sunday, 02 August 2015, 23:33 GMT
Considering it again the "motd" idea would be overdoing it for this issue. Adding it to README.knownissues is good, but that's not what the user booting the iso sees. That's why I have already made a proposal to add it to the installation guidey (see link above). If it does not get added into the installation guide because of the "do/dont" paradigm, so be it - Arch Way. Since it is the livesystem it is unlikely a user looses data because of such a crash. Thanks again for discussing it at this length. Please close this issue anytime you are content with the result.

Loading...