Arch Linux

Could you provide us with:

a) the actual output you're talking about (dmesg)
b) the --debug output
c) a way to replicate this bug

Thanks

boot with mem=64M and try to install all core packages to a root directory

Thanks, adding it to my list of things to look into this weekend.

additional info isntall environmnt size is 20MB + 8 mb ramdisk + 1.7 mb kernel.
so 30 megs for pacman running available.

When I was testing this scenario with pacman 3.1, I never saw it using more than 23 or so MB of memory. There could have been some changes since the 3.0.X split that account for this, but I'm not sure. I haven't tested it in a restricted memory environment however.

i think it depends on how many packages are installed at once.

For your info, I've done at the moment a test installation on a VM with only 48MB RAM. Install with the new ISO and lowmem initrd works like a charm.
Also never have a pacman problem, neither during installation nor when installing for ex. xorg-server and it's dependencies.
During this operation the system never use swap space. Maybe only a problem if one have low memory (X running, etc) during pacman usage.

how many packages did you install?

Sorry, at the moment i could not say it exactly, but
- during installation base+support (0 byte swap)
- xorg-server, xterm (and deps) (0 byte swap)
- xfce4-group (and deps) (about 25MB Swap, xorg+icewm running)
Maybe about 15-20 packages per run.
If it make a sense i could try tomorrow for ex. a complete install of kde.

i tested only with all packages selected at once.

I've tested now with:
pacman -Sv kde gnome (would work dep check etc., i don't do the real install)

Then something one would never do in real life:
pacman -Sl extra | cut -d " " -f1 | xargs pacman -Sv
and
pacman -Sl extra | cut -d " " -f1 | xargs pacman -Sdfv
So pacman check for dependencies conflicts and inter-conflicts for about ~2300 packages.
Cause there ARE conflicts, above never got installed.
But i think these are the actions were pacman needs most resources and oom-killer would come.
I've never hat more swap than ~25MB, pacman itself uses about 10-15MB VIRT.

the oom kill doesnt happen during intitialisation it comes while installing packages, suddenly

tpowa: if you want our tests to be the same, can you provide us with the EXACT list of packages? I was under the assumption the installer simply did a "pacman -S base" because that makes more sense to me - if it does not, can you provide us with a list?

it installs everything from packages.txt, because i selected all packages.
it also happens with quickinst so it's nothing setup specific.
quickinst only installs base

tpowa, can you try with pacman.static? I'm curious if this is loaded libraries causing the excess memory overhead.

pacman.static also fails, i used no swap.

This should be fixed by our master (3.2 development) branch, as we just made a lot of changes related to backend memory allocation (static to dynamic) as well reducing duplication of entire pmpkg_t structs in memory (90% of these dups were unnecessary).

tpowa- if i give you a pacman.static from our development tree, could you test it out? It should have no problem even installing all of [core] (base, base-devel, etc. all at once).

I think we can safely say this should be fixed in 3.2. I blogged about it, actually, as massif changed a bit and I wanted to rediscuss it.

http://toofishes.net/blog/valgrind-330-and-new-massif/

Woot, go Dan!

sounds pretty good, dan when will it be ready or is it backported to 3.1 branch?
new isos should come finally with 2.6.24 kernel and with this fix it would be so great.

greetings
tpowa

Definitely NOT backportable. This is a huge change that needs a lot more testing to ensure we don't have any memory allocation issues.

Speaking of new ISOs, did you ever see the archiso scripts that Simo and I got working quite nicely?
http://archlinux.org/pipermail/arch-dev-public/2007-October/002616.html
http://projects.archlinux.org/git/?p=archiso.git;a=summary

Maybe this can be closed now that archiso is in use, pacman 3.2 is close to release, and the next official iso in preparation might have it.

From my view: close it.
Maybe i/we could make some tests with lowmem machines with new iso's and then reopen or file a new report if this become important.

Sorry for warming up this thread, but i have found some interesting news about this task.
I have had the need for testing again such a enviroment (VM with 64 MB RAM) and with our current install-iso.

By kernelparameter we use 75% from RAM for ramdisk/unionfs later as root-fs(/). With 64 MB RAM this leads to a
Root with araound 54MB.
Package selection and installation give error during copying (BTW: also with error the installer says: Coping/installing successfully at the end - sounds like a installer bug, i investigate this later)

Next try i used a ramdisk size from 90%. This fails also. The reason is easy - the base package files (and other selected by the user) are copied to /tmp (cause /var/cache/pacman/pkg isn't availabe).
But base occupies alone ~110 MB in /tmp.
And this is the problem - / is getting full.

So after a little thinking i found a solution.
After partitioning and before package selection/installing:
mkdir /mnt/tmp
chmod 1777 /mnt/tmp
mount -o bind /mnt/tmp /tmp

So the package installer has enough space for saving downloaded or copied packages and the installation goes
on without errors. The system and installer itself needs during the whole installation process not more than
~15-17MB.
So this i a chance to install Arch now on machines with very little RAM. I don't know if this method were also possible with the on-topic 2007 ISO ("tpowas"), but with the new ISO it is definitely no problem anymore.

If you agree this could be a hint in the installation guide.

Well, I think the issue here is that the cache directory needs to be created on the host system before installing. Can you try this:

After partitioning try: mkdir -p /mnt/var/cache/pacman/pkg/
(Note: this may be /media and not /mnt, I can't remember)

The install may then use that dir as cache (and cache downloaded packages on the new disk. That's the way it SHOULD work

For the next installer, this patch should help. It might apply to the current installer too:
http://code.neotuli.net/gitweb/?p=installer.git;a=commitdiff;h=b677b3e44624608e684de6fb4a5a9b2531ac1257;hp=6cd6408df9686534e8c9a8f05d77e4eec2ba32ab

The new pacman (3.2.0) will definitely be better in the memory usage department as it does everything dynamically, which cuts memory usage by a good 50% on most operations over the 3.1.x series.

Sorry for warming up this thread, but i have found some interesting news about this task.
I have had the need for testing again such a enviroment (VM with 64 MB RAM) and with our current install-iso.

By kernelparameter we use 75% from RAM for ramdisk/unionfs later as root-fs(/). With 64 MB RAM this leads to a
Root with araound 54MB.
Package selection and installation give error during copying (BTW: also with error the installer says: Coping/installing successfully at the end - sounds like a installer bug, i investigate this later)

Next try i used a ramdisk size from 90%. This fails also. The reason is easy - the base package files (and other selected by the user) are copied to /tmp (cause /var/cache/pacman/pkg isn't availabe).
But base occupies alone ~110 MB in /tmp.
And this is the problem - / is getting full.

So after a little thinking i found a solution.
After partitioning and before package selection/installing:
mkdir /mnt/tmp
chmod 1777 /mnt/tmp
mount -o bind /mnt/tmp /tmp

So the package installer has enough space for saving downloaded or copied packages and the installation goes
on without errors. The system and installer itself needs during the whole installation process not more than
~15-17MB.
So this i a chance to install Arch now on machines with very little RAM. I don't know if this method were also possible with the on-topic 2007 ISO ("tpowas"), but with the new ISO it is definitely no problem anymore.

If you agree this could be a hint in the installation guide.

Oh, uups posted my comment double? Sorry ;-) I hunted a beasty fly and must have pushed some keys on keyboard....

@Aaron:
mkdir -p /mnt/var/cache/pacman/pkg/ doesn't solved it. Still says: couldn't create package cache, using /tmp instead.

It's this in the current installer i think:
-----------
if [ "$delpkg" = "yes" ]; then

- # set up a symlink to fool pacman so it doesn't copy all the packages to the cache

- ln -sf /src/core/pkg /var/cache/pacman/pkg >/dev/null 2>&1

- else
-------------
I used the option "delete packages", so the symlink is created to an ro-only filesystem. If choosing "keep packages" this symlink goes to /mnt/var/.......

But as Dan mentioned: This part is modified in new installer.

This bug is marked as due for 3.2.0
Is it fixed?

What Dan said on 23 July :
"I'd like to push this to Simo and others to actually do some testing
once we do get 3.2 released, so for now leave it open."