Arch Linux

I agree that we should at least get the warnings back if they were there before.

This is a really sticky issue that is hard to deal with unless we implement transactions in pacman. Otherwise there is no real way to roll back or get yourself out of this mess.

Virtual machines? Transactions? Let's keep it simple!
I propose to check the filesystem mounted on '/' on available disk space before installing. If it's not enough for the package tarballs + extracted packages + some overhead (keeping in mind all deps of course), warn the user and ask if he/she wants to continue. (some tips on how to free disk space, such as clearing the package tarball cache could also be shown)

He/she might have other filesystems mounted separately ( /usr, /var,...) so the counting will in that case be not totally accurate, but I think this is the easiest, most 'KISS' and still pretty effective approach

The freespace check code was disgusting, even with a "simple" implementation:

http://projects.archlinux.org/?p=pacman.git;a=commitdiff;h=ed13ac2cc8dd15d8a1
http://projects.archlinux.org/?p=pacman.git;a=commitdiff;h=5e774d72b0ac

It is pretty easy to make a shell script program like the following:
#!/bin/sh
# spaceman - prevent user mistakes by showing freespace before pacman operation
df -h
pacman $@

I would call this natural selection. You should always check if you have enough free space yourself. If you didn't and you hosed your system, go complain to Darwin.

I had similar problem...

Description: If theres no space left on root filesystem during upgrading/installing packages, pacman screws whole system and leave packages installed just partially or uninstalled without reporting any error or warning. This should damage/remove system binaries and other files which are needed to boot, etc...

theres possibility of some another errors caused by pacman when no space left...

In my case that made unusable binaries like: pacman (so i cant repair the system), halt, and many nescessary things...


Steps to reproduce:
install or upgrade few packages with little space left on /

How to fix the bug?
- check diskspace before proceeding whole upgrade
- check diskspace before installing each single package

if you quickly want to do something, you can do it Dan McGee's (toofishes) way by adding alias to your bashrc:
alias pacman="df -h; pacman"

btw similar problem here: http://bugs.archlinux.org/task/7692?project=3

I've just had an office server rendered inoperable due to this bug, I was delighted to see that the bug already exists with a high priority on this bug-tracker. Shame that instead of fixing it people are offering work-arounds or blaming the user for not checking disk-space!

Anyway, I had a look through libalpm/add.c and found this suspect line of code in the function commit_single_pkg(..):

741 for(i = 0; archive_read_next_header(archive, &entry) == ARCHIVE_OK; i++)

So what happens if archive_read_next_header returns a result other than ARCHIVE_OK? The for() loop is terminated, and we drop down to this line:

781 if(errors) {

That would be great, only the errors variable wasn't incremented in this case, so pacman continues installing packages regardless of the fact the disk is full, and what pacman is actually doing is replacing vital system files with truncated or zero sized files. What should happen is that Pacman aborts the install, with a nice helpful error message. I also noticed a few warnings in the log file about failed attempts to update the pacman database, but yet again these weren't enough to cause Pacman to stop its trail of destruction.

Devs: Please fix this bug before it destroys anyone else's system!

PS, another popular package manager unpacks the archive to a temporary directory before mv'ing the files into root - I think this would prevent partial installs due to disk full errors.

Just had a closer look at the source code, I don't think the return code from archive_read_next_header(..) is that important, but I did notice that upon encountering an error a log message is sent to the logger and the commit_single_pkg(..) function returns -1. The function is called from within the alpm_add_commit(..) function:

874: commit_single_pkg(newpkg, pkg_current, pkg_count, trans, db);

See what it does with the -1 return value? Absolutely nothing!

Still not sure why I have no error log messages, perhaps they're not written to pacman.log? No errors were shown on the console either.

Ehm... anybody knows, if this was fixed in pacman 3?

No, I don't open bugs for fixed/untested issues.

sorry i was misleaded about latest pacman version (i tought that pacman 3 is brand new...)
But this is pretty dangerous bug! and nobody wants to get his arch killed by one of core packages.

Frankly I think this is a typical PEBKAC. You should know how full your / is, and 2 GB is asking for trouble. I'd rather have a too small ~ partition than a too small / one, since a full / will still let you boot and log in (as root at least), and a full / won't.

I think we have enough handholding already in Arch :).

This isn't PEBKAC, this is one of the most basic must-haves for any package manager - making sure not to trash the system. And yes, pacman fails this one, probably since day one. No, this won't stop me advocating Arch, and I know very well how to take care of my systems, but please stop questioning the seriousness of this bug. You could have over a gig free disk space, run a -Syu every two weeks for months and all of a sudden experience this, just because of a few new dependencies.
If the real fix is too hard to implement right away, add at least an additional check which outputs "Not enough space left!" after the "Total Size Installed: ..." line.

Borromini: This is serious problem... you probably have enough time to remember left disk space on all of your systems, but there are other people too. If you don't like it and you have lot of free time, you should make Linux From Scratch. Why pacman should destroy .so files? pacman should upgrade or NOTupgrade, but nothing else! are there some pacman developers, or its unsupported software?!! It's one of most important packages (maybe most important) of Archlinux so it doesn't looks nice to be unfixed for so long time... im trying to get through the code, but it will take some time for me, bacause i never saw how the pacman works...

Please stop the needless back-and-forth.

We know this is a problem. We have also stated it is not at all easy to address without some serious rethinking about the way we install files.

Harvie: please refrain from making such insinuating, even trollish statements.

I agree empty .so files are a problem, especially if pacman makes it look like it went alright, and doesn't say anything at all. Jens is right about that. That definitely is a problem. I'll leave it at that.

can you please add this issue to 3.3 roadmap or at least to 4th version?

pacman-3.3.1-1
pacman (yaourt) begins to eat off space on your hard disk, if the package is downloaded for a long time.
Internet connection speed: 128/128 kbit/s
Filled folder: /var/cache/pacman/pkg/
Many times when trying to download eclipse-cdt.
Helps only:
rm -r /var/cache/pacman/pkg/*
No problem, if in /etc/pacman.conf use:
XferCommand = /usr/bin/wget --passive-ftp -c -O %o %u

kidoz: http://bugs.archlinux.org/task/16359
(different bug, fixed in the next update)

The same issue with pacman v3.3.2-1. After installation failed pacman doesn't delete installed files. It doesn't delete files even on `remove package' command. Freeing disk space and reinstalling with --force flag helps.

i have made a dirty hack. that basically follows byte's suggestion. Attached is the patch file formatted with git-format-patch as per instructions in the "submitting-patches.txt" file.

I can confirm the results. This happened to me today, too. I updated the system in a “desultorily” way (i.e. started the process and minimized the terminal emulator window). After a while i reopened it and the whole window was full of warnings about no diskspace left an all packages including pacman itself were broken.

Because i was not willing to fix 70 packages (containing about 50 percent core packages) because of pacmans stupidity i simply reinstalled the system (it’s a data-media-backup-musicplayer-homeserver system with only a few tools installed and all important things on other file systems). In the end it wasn’t THAT bad, but imagine this happens to a machine not used passively but as only available machine.

This bug is almost two years old now and pacman REALLY should have a functionality to check if there is enough disk space to perform the update in a successful manner.

Oh, and about that PEBKAC discussion from January 2009: It’s ridiculous! A package manager should not break the system and itself because of disk usage.

dsohler: to fix your system chroot to it from install live-cd (after setting up network connection and reinstalling pacman manualy using alternative root parameter to live-cd pacman), try to save some space, then make script to list all installed packages, reinstall each package ang clean every package from cache right after it's (re-)installed (to save more space). then check grub configuration and reboot.

harvie: I tried several solutions for this, but after about one hour i decided to simply reinstall Arch, which took 20 minutes (including packages download). Now i’m going to write a pacman wrapper checking disk space before update and comparing it with pacmans output of the expected disk usage.

The better way were to (re-)implemet that feature.

Please do, and when you get something that works we would appreciate if you contributed it back to us and attached it here. And when you realize it isn't near as easy as it sounds (I'm interested in seeing what you think about someone that has an /etc, /var, /srv, /boot, and / partition), maybe you will also understand why this problem hasn't been solved yet and why telling people that aren't paid to develop software haven't yet done this.

btw when you are going to reimplement whole package instalation procedure, please keep in mind some feature to add hooks to be done whenl all packages are installed. eg.: when i want to install 10 ttf fonts or 10 desktop applications i will have to wait for fontcache (or desktop icon database) update after each package is installed, but it should be enough to run this process only once when all fonts (or desktop entries) are installed. by now it makes installations and updates take much longer than they really need.

Is there way to get this task to roadmap? I think it should be there...

Question/idea:
when upgrading the system, pacman shows something like this:
----------------------------------
resolving dependencies...
looking for inter-conflicts...

Targets (1): aria2-1.9.5-1

Total Download Size: 0.00 MB
Total Installed Size: 4.17 MB

Proceed with installation? [Y/n]
----------------------------------

Now, instead of reimplementing stuff/writing complicated disk space checking code, why not just have pacman call 'df -h' and display the output (after a little processing)?
It could look something like this:

----------------------------------
resolving dependencies...
looking for inter-conflicts...

Targets (1): aria2-1.9.5-1

Total Download Size: 0.00 MB
Total Installed Size: 4.17 MB

Free space as reported by df:
/: 4.1 GB
/boot: 60 MB
/var: 5 GB

Proceed with installation? [Y/n]
----------------------------------

And let the user decide.

I totally support cantabile’s idea with using df (or even included in libalpm) for showing the disk usage before installation. In fact i created an alias for doing so:

alias pm='df -h ; pacman'

ok, we all already know that you can make alias, so there's no need to add such feature to pacman source.

i can suggest better workaround:
we know 3 numbers: Total Download Size, Total Installed Size, df /
so we can use such simple heuristic workaround:

if( (TotalDownloadSize + TotalInstalledSize*2) >= (df /) ) abort;

it is just educated guessing, but it will work in most cases until it will be fixed with better granurality (check file space for each package and each filesystem that package installs to)

btw i've just realized another possible issue which we should be aware of.

we will probably get df of every filesystem in some array and we'll be decreasing the number with size of each file in package which installs to that filesystem/directory. and we will not allow installing package if one of those numbers will be <= 0 (or <= than some bigger positive number). so we will be checking the paths of installed files against the paths of filesystems. the problem is that we should be aware of SYMLINKS. (i can have lot of space on /, but /usr/share can be symlink to directory on some almost full partition - well it shouldn't but it's not impossible). we should check if all directories are really on filesystem corresponding to their path, or there are physicaly stored on other path on different filesystem.

Don't forget that this needs to be portable to other platforms than just Arch. Pacman isn't an Arch-specific project. You also need to consider that there are files currently on the filesystem that will be removed before the new package is installed, so that adjusts the size estimate as well. It also needs to be scalable. Consider an entire transaction at once, not just an individual package. It may touch thousands of files across multiple partitions. Are you going to check them all individually? Speed is a concern. If it makes every transaction take minutes to check the filesystem, that's not useful.

Trust me, this isn't a low-hanging fruit. If it had a simple solution it would already be done.

Daenyth: it's just very simple but working workaround and it's better to have false positively disabled upgrading than bricked system...

I guess, the simplest solution is the best. Just use df or create a function in libalpm to output disk usage for all mounted file systems, before starting update letting the user decide if he wants to run the update.

Oh, and @Thomas Mudrunka: One can avoid problems with full disk during update by manually (or with an alias) running df before pacman, but it should be fixed/implemented in pacman too, maybe as option in /etc/pacman.conf.

Dirk: why do you still think it should be in pacman?
echo "alias pacman='df -h; pacman'" > /etc/profile.d/pacman.sh; chmod +x /etc/profile.d/pacman.sh
you can also ship this file in package... there is REALLY no need for messing with pacman code itself.


Daenyth. maybe that another simple solution is try to unpack whole package to temporary directories (on desired partitions) and if all of them were succesfully uncompressed, we can safely move them to their intended location. (we can make some temporary directories like .pacmantemp-xDS87dsh on each desired partition and delete this directory once package upgrade is successfuly finished or aborted). All we need to make this is to have portable way for getting list of partitions...

One package can be split across multiple filesystems. What about files created or modified during post_install? What if the system is being filled by another process as pacman runs? This is not solvable. The only real solution here is to have a real transaction rather than just unpacking files, and if it fails, roll back so that the system is not in an inconsistent state.

Daenyth: have you read my suggestion?

> What if the system is being filled by another process as pacman runs? This is not solvable.
that's why i say "unpack all files before replacing files". this will securely reserve space on filesystems. that's what every polite software (without journal) does: make temporary working copy somewhere and if you are sure that everything is working (and it's already on disk and will not need more disk space after MVing), safely mv it to desired location.

> One package can be split across multiple filesystems.
you just need to have multiple temporary directories (one on each filesystem) where you will unpack files before actually installing them


you can also use reverse approach... backup every file from installed package (pacman -Ql package) somewhere and then try to install new version of package. if something fails, just remove all installed files from new package and give back the old versions from backup. backup can be deleted imediately after instalation or keeped for few days. btw there is already script "bacman" from pacman-contrib, which can make backup installable package from any package already installed on filesystem... (backward instalation :-)

> What about files created or modified during post_install?
well... this is an issue, anyway most of packages are not generating any big files in post_install and it's quite easy to reinstall package if pacman didn't screwed up all libraries during update... (ok mkinitcpio should check free space too...) so it's not such dangerous as problems during installing real package contents.
what you are talking about seems to me like have whole / directory in GIT repository. it's needed to be able to track file changes, but it's overload.

Re: Daenyth, I agree if you want a "proper" solution, you would use transactions. but since btrfs will get stable one day, the user could make a btrfs snapshot before running pacman, so he can always revert to the earlier state.

For a quick and simple solution, I agree with the bash alias.

Either way, no need to complicate libalpm/pacman code.

There is still an issue with extracting all files to a temporary directory and then copying them:

> pacman -Si | grep "Installed Size :" | cut -f2 -d":" | sort -h | tail -n4
496268.00 K
523148.00 K
559024.00 K
870868.00 K

That last package would require the user to have an extra ~870M free to upgrade the package. That is an extreme example, but think of someone who has not updated in a long time. The total required space for all the extraction into temporary directories may be multi-gigabyte in size.

Allan: even when doing this per-package? 870*2 < 2GB, which can be handled i guess... now i someone who has not updated in a long time and have no disk space tries to update he will end up with unusable system...

If you do this on a per package basis and pacman aborts due to lack of space, you can also end with an unusable system (especially on a rolling release distro). But it would be better shape than currently...

So patches welcome for that approach. Remember it will need to work on all the platforms pacman currently works (and is used) on (linux, BSD, MacOSX, cygwin, hurd...) so the approach can not be platform specific.

As a quick solution, could we please focus on cantabile’s idea (2010-07-02, 10:23:35) for a second? :)

No, we can't. You already wrote a perfectly acceptable workaround that implements his suggestion! Please read the rest of the comments and please stop assuming this is an easy problem to solve- it is not even close, and everyone that regularly contributes to pacman is not happy with half-baked solutions because we will just be tasked with fixing their shortcomings later. Better yet, listen to Allan and his calls for patches. It helps not one bit for people to keep chiming in here without offering *solutions*, not just ideas.

I patched Clyde for this problem. Using two config variables for /'s mount and /var's mount covers most cases. Eventually I'll get it to work in C, but please don't hold your breath.

How did you implement this in clyde? Specifically how do you calculate the space needed in each partition for the upgrade to complete?

This is the patch for Clyde: http://dpaste.com/265951

There are three out-of-disk points of failure in Pacman. First, when syncing, running out of space to extract the repo tarballs is really bad. However I did not fix this as the tarballs are small (unlikely to max out the disk) and the sync process is entirely contained in libalpm (I did not want to mess with the deeper magic just yet). And with Pacman 3.5, it will be even less a problem. My idea here is to check for a meg or two of space, and abort otherwise.

Second, when downloading the packages. Third, when extracting packages. Beforehand, you know the total download and total install. If you have a single root partition, it sums up download+install and compares against free root. If you have a seperate /var partition, it compares download against free var and install against free root. You set these mountpoints in the config, with a default of single root on /.

This is not foolproof, but any fool with a fancier partition scheme may step forward.

From an IM convo I had with Xavier, just for posterity:

Xavier: seriously now, I always felt like we should put back the old code, no matter how crappy and not portable it is. never felt strongly enough about it to look into it
Xavier: but are you completely opposed to that ?
Dan: it was awful
Dan: 1. not portable (not even the biggest of my concerns)
2. summed up ALL partitions on system and used that as the "free space available" number
Dan: so i don't even think it would have caught this guys case!
Xavier: ok 2 was very bad then
Xavier: this is probably quite common to have small / but spaces elsewhere

So any solution brought forward (which Xavier might work on, he says) would have to (1) not cause problems if the system didn't support whatever method we used, so probably just act as if there is no space checking, and (2) not half-ass the partition logic. There are plenty of systems with "fancier" layouts- /, /boot (quite important, actually), /srv, /var, /tmp, /home, /opt all have different requirements as far as package installs go.

Here is my plan on how to implement this:

1) get fs layout using either getmntent (Linux, Cygwin, old BSDs) or getmntinfo (OSX, new BSDs)
2) calculate size needed on filesystem using archive_entry_size to get size from new package and current package size using stat calls
3) check that amount of space is available using statvfs

That should be portable across all our usual targets. #1 is the real tricky point for portability but those two functions cover the vast majority of systems likely to use pacman (according to all the #ifdefs in coreutils lib/mountlist.c).

Implementation queries:
- Should we do the calculation for the whole transaction or on a per package basis? I'd think whole transaction as it is generally a bad thing to stop mid transaction in an update... but when doing it on a whole transaction basis, if a package being updated is significantly reduced in size, other packages installed before it could hit a full filesystem. That may be an edge case we can ignore?
- I would be tempted to only enable it if a configuration option is given as I personally would not use it. Does that sound reasonable?
- I am ignoring calculations of space needed for package download as issues with that will be caught when checksumming the package files.
- if neither getmntent or getmntinfo are available, we could disable size checking during configuring and print an error telling the person to report to pacman-dev so it can be added.

Does that sound reasonable? If so, I can take a stab at implementation...

As far as my first query goes, it should step through the packages in the sorted order that they will be installed and keep track of the maximum amount of space required at each step. My guess is that most of the time this will be roughly equal to the final difference in space used but it will catch the edge case I mentioned above.

Good, and possibly workable, idea, Allan. That's help in the right direction, and much appreciated.

Here's my take on the issue as I've read thru all this:
Regardless of being paid or not, programmers are reasonable expected to do good work. We will assume of such being done here. They should also be the most familiar with issues like this. We can assume that such is also the case here.

Nothing wrong with any of that. But if programmers want help, or desire to just alleviate problems, there should be a clear logic tree posted somewhere so that everyone can read it, become familiar with the thinking behind the code and determine for themselves if it really conforms to the KISS method or not. Without that, we're all just trying to do our best in the dark. Point us to that, and we've really got something. This is definitely an issue that would benefit from that info.

Perhaps if we start with a few facts, we can come up with a better arrangement of ideas, and maybe reach a reasonable consensus:
1. The free space on a given partition should be readily available.
2. Pacman should be able to get that info from the OS.
3. Pacman should know the sizes of the compressed packages which are going to be downloaded.
4. These sizes could be totaled.
5. A very liberal calculation could then be offered for the expanded size of all these packages in total.
6. The available space and the estimated needed space could be compared. The first being smaller than the second would automatically halt pacman and display a warning message.
7. We need to remember that the extracted program files will replace the files already existing.
8. Only the additional files, along with the difference in size of existing packages, will require further space.
9. Pacman could offer an option to download and install only one package at a time, with #6 happening if size was an issue. The user could then select to skip that particular package and dependencies.
10. Downloaded (cached) packages could individually be deleted after expansion and installation as a pacman option prior to further installs. This would free up space.
11. Just download the delta's?

Would any or all of these options solve the problems? Or just help a little?

I will state that I too had to waste a great deal of time with this problem happening to me. That is not the Arch Way or the KISS method.

Perhaps folks like Glenn are correct. I agree that I should know everything in the world, including the space on my hard drive on a moment by moment basis. Sadly, I'm just not that good. Still, if I could find that out, I would still need pacman to tell me how much it needs for each part of the operation. So I guess that wouldn't work anyway.

As an aside, for perspective:
I frankly expect my computer and its OS to work in the background without failure. While we may have gotten to the point where we expect some problems, there is no excuse for the huge numbers we are seeing in all OSes. (Anyone can start up DR-DOS, open WordPerfect and begin creating a document. That is rock solid and dependable. I'm just saying.) Perhaps Arch isn't for productivity. Fine. But let us not belittle the real frustration users have, nor the unacceptable nature of many shortcomings in software. (Keeping things close to the vest creates the assumption that coders don't need help. At which point is is reasonable to expect no problems, and no bug discussion area.)

Yes, I know, Allan, that last bit isn't particularly appropriate for bug reporting areas. But perhaps we can use these comments as a means to get back to the very real issue.

Thoughts anyone?

My thoughts are that your post does not provide anything that we don't already know.

If you want to really help, then review or test the first step that Allan made :
http://mailman.archlinux.org/pipermail/pacman-dev/2010-October/011916.html

Or start coding the next steps, which Allan already presented better than you did.

First, I broke it down into simpler and more easily understood steps. (Missing a step in programming is disasterous.) Your implied assumption that everyone understood it was incorrect.

I don't see where the user is given an option to install only what will fit. If this isn't done, then the user must do a trial and error method, picking one package at a time after writing down all of the packages found for upgrade. I am sure that would be unacceptable to most commentors here. Further points were not clarified either.

Also, you gave no instructions as to "how" to test the script. And let us not forget that the code is not commented for those of us who don't program. I have no way, therefore, to review it. That would be a place where you could be helpful, if you really want to.

Thanks.

I can't take credit for this idea (thanks loonyphoenix!) but one of the absolute simplest fixes for this would be to loop on the error. In pseudo code:

while (archive_read_extract(...)) {
print("Error! Press enter if you've fixed it or ^C to abort.");
scanf();
}

And do that for every extraction. This method can't go wrong. No false negatives, no false positives. Waiting for the user to fix the problem makes more sense than counting the number of errors.

Or... we could do it properly and check for free space before doing anything and not abort mid install...

See http://projects.archlinux.org/users/allan/pacman.git/log/?h=diskspace for actual code in progress.

I agree. After looking at the proposed code however, I have to admit I didn't understand the steps involved. Allan, could you please explain them for us dumbbunnies? Thanks.

It does exactly the three steps I gave in https://bugs.archlinux.org/task/11639#comment67966

Then, with regard to your "Implementation queries:", we run into my other concerns. As a user, I definitely would want to be able to install what I could, rather than nothing. How that is implemented is secondary to being able to get the job done, of course. We need to remember that the calculaton used, based upon the space available, needs to keep in mind that there may be plenty of space available if one program is installed at a time and the package deleted before the next is downloaded. To be clear:
1. Calculate if all packages can exist in given space and also be extracted in that same space.
2. If that cannot be done, the process must calculate which, if any, of the downloads, or group of downloads can work with the given space.

Pacman will fail if there is not enough space for the entire transaction. The user then can make up their own mind on how to proceed. Either clearing space or installing only certain packages at a time. I will not be implementing anything that automatically performs a partial upgrade as those can be almost as bad as what we are trying to fix...

It is unclear what you meant by the concern you implied in the last sentence. My response would be that a full upgrade does not necessarily mean that only installing some of the packages is in any way going to break anything. A user could try for an upgrade at any point in time. Many packages are upgraded without any other package being involved. Some users will go a longer space of time between upgrades than others, and therefore have many more programs to install. Etc., etc..

The whole point in computerization is to have the computer do things that are tedious for the user. Ludditism is counterproductive in this regard, and must be avoided by all programmers everywhere.

If the user can pick and choose the packages to install, then the computer can do it even faster based upon safe criteria. That is exactly what must be automated.

What we are speaking about here is a lack of space, and not a problem with inappropriate install order, as appears to be implied. The inappropriate installation order is assumed to be handled automatically by pacman, else it is a failure in part or in whole. I'm just wanting to make sure that user doesn't have to jump thru manual hoops to accomplish a computerized task, as is the case now.

As far as I am concerned, a package manager should do exactly what it is told and if it can not do that then it should fail without doing anything.

So in this case, pacman either aborts due to lack of space or proceeds with the whole transaction. I will not implement anything in the middle ground.

Well I do understand your position. However, I am equally certain that most people, by a significantly wide margin, would disagree with it.

I really can't imagine anyone saying that they prefer your "all or nothing approach" that would require yet another program to handle the missing functionality. At the end of the day all we end up with is a tiny kernel of process with a myriad of wrappers to complete the needed parts. We already have wrappers for colorization, download speedup, more useful feedback and terminal layout, for other repository access and automating third-party package creation, etc., etc..

To make your decision worse, the problem lies with a very basic process function. When the program doesn't know what to do with a shortage of space and simply gives up, it just isn't finished. I'm actually surprised that this issue was not forseen from the outset.

@ Mr. K.: implementing a middle road is just dangerous. What if you have packageX, which depends on a specific version of packageY. The upgrade process decides: I'm only going to install packageX because I don't have enough space for the rest.

Immediate result: package Y will stop working.

However, if pacman would have decided to abort the whole transaction, your system would still be in a consistent state. Imagine the messy code you'd have to write to have this "middle road" work, you'd be busy tackling edge cases instead of real proper work.

Another case where implementing the above would cause problems: imagine pacman filling up your disk space to the brim, ie leaving 0 bytes of space left. Your system may be in a consistent state, I'd really like to see the result of the bootup process: the rc.d scripts wouldn't even be able to create lockfiles anymore. If you're not seeing where I'm going with this, then maybe you shouldn't be involved in the decision process either.

The real problem is not pacman, it's the lack of disk space. It is: 1) structural and 2) underlying. It is not pacman's fault you are running out of disk space, it is your fault.

This discussion is going nowhere. I will implement the disk space checking and have pacman abort when there is not enough space. That is a vast improvement over the current system destroying issues when not enough space is available. I will not implement pacman falling back to installing only some of the packages asked for in a transaction where disk space is an issue as I think that idea is stupid.

As I do not get the final say on features added to pacman, this may be a case of "patches welcome". But given the bug has been open for two years so far and my work is the first real progress on this issue, I will not be holding my breath waiting for more patches to arrive...

Ya, please put in a check if space is needed before download and install takes place. I ran into this problem and now I'm unable to upgrade as hundreds of '<file> already exists in filesystem' error comes.

I like the check before installing everything approach. I only suggested the loop solution because I was under the impression that a more thorough solution was difficult to implement, and it was better than the current approach of "ignore the error, trash the system". It seemed easy to implement to me, a kind of temporary workaround to let those that have run into the problem the chance to solve it gracefully. Certainly a halfway-installed upgrade is dangerous and inferior to checking if the whole upgrade can be completed.

However, I have a question. If you check free space before installing the packages, and there's enough, but by the time the whole upgrade is completed, a partition fills up some more so there isn't enough, is the old 0-byte issue going to surface? I understand that it's a rare occurrence, but it IS possible. So the protection is not robust. Maybe a back-up plan of catching a disk error while installing a package and looping on it might be a good idea after all?

No, no, no Glenn. The program already does dependency checks. That simply couldn't happen, nor was it advocated. It is a non-issue. I can only shudder to think if the basic code of pacman were messed with in any way.

Regarding other's comments, I believe any narrow view that fails to see the alternatives is the most stupid position to take. There are any number of acceptable ways to do something, and saying that a problem can't be solved is totally unacceptable. The shortcoming is in the doer, and not in the possibilities.

Victor is on to part of the problem, but most still don't seem get it. If there is enough space to add the needed parts of existing files, the problem is an inefficient use of the available space.

Here's the simple scenario example:
1. The partition has 500 MB of space free.
2. The space needed by all the changes to existing files is 300 MB.
3. The space needed by all the downloads is 700 MB. This is the problem. It exists in part because the program packages hold the whole damn program and not just the delta difference, plus the fact that there is some processing overhead in space requirements. (It can also be because of limited free space, but I would guess that such is not the case in 80% of the time.)
4. This problem is eliminated by a package-by-package install. First, the first package is installed and deleted afterward. Then the next package, and so on, until all are installed.

Here's the next simple scenario example:
1. The partition has 500 MB of space free.
2. The space needed by all the changes to existing files is 550 MB.
3. It doesn't matter the download size, since there isn't enough for the finished product in the first place.
4. However, it is found that one of the programs being upgraded has no relationship with the other programs being upgraded. Therefore, the one program can be upgraded because it will fit. This is obviously better than nothing. The user would then select to upgrade that one program. The odds are that there are others that will fall into that category as well. (For instance, just because a kernel is upgraded does not mean that every program needs to be as well.)

Some people seem to be working off of the incorrect assumption that all upgrades affect all other programs. That is stupid. Worse, they forget that there is no reason to expect 2 or 3 GB of programs (which is a hell of a lot) would require a half a gig of space just for an upgrade. That is outrageous! Whenever anyone sees such a situation they should immediately recognize that something is wrong, and it most likely has to do with the level of efficiency of the process, and certainly not the actual space needed. And that is why our systems get trashed.

@KitchM: you seem to think that the package manager should have a built-in workaround (let me stress again, workaround, a Bad Thing™) to low disk space. That's not how we think of it. First of all, delta upgrades have been tested in the past and as it turned out it didn't make much of a difference if you count everything together. Second, just must keep in mind that pacman itself DOES NOT track versioned libraries (shared objects). Pacman will ONLY recognize such an event when the PKGBUILD explicitly states that the package dependencies must be greater or equal to a certain version. Unknowing users may choose to upgrade only libraries, producing the same failure that might occur when users do partial system upgrades, since not all PKGBUILDs enforce certain versions on their dependencies. Third, if you really want partial upgrades you can -S the individual packages by yourself. Fort: do not No no no me, or any of us. I've seen some of your comments and some of them are way below acceptable behavior. We have the last call, not you. You will stop calling other peoples opinions stupid (which is basically what you've been doing - subtle as it might be). You've had your say, we read it and some rejected it on solid grounds.

@Victor: this is a very good point (that we already discussed in private but not here afaik).
It might be better to abort right away on the first libarchive warning that happens (Dan made a patch doing just that a long time ago).
There are a few open questions however :
1) in which cases are libarchive warnings triggered (other than disk full situations)
2) do we abort on first file that triggers a warning or do we attempt to keep extracting the current package (and abort after)
3) when it fails, the current package will be half-extracted. do we keep a local database entry for it or not

Otherwise, maybe some more thinking could be done on FS#8585

@Xavier:

I'm thinking that it should be possible not to abort, but to pause the transaction and wait for the user to manually free up some space. Once the user indicates that the transaction should be continued, pacman would reinstall the package it stopped with and continue on. In this case there should be no problem with the local database, too.

Since there would be another defense, the one that checks free space before the transaction is started, free space must have been reduced after pacman started the transaction. It would mean that another process was responsible for it, which means that there is data that can be safely removed by the user by reversing the process's transaction.

There is clearly a lot of confusion in the minds of some. I will attempt to further explain. First, a workaround was never advocated by me. I simply stated once that Allan's idea was good, but I hope it was clear that I viewed it as a stop-gap measure.

For what it's worth, and to clarify for the confused, I hate workarounds. Any workaround is second-best. It is the same as a patch, an add-on, an extension or a wrapper. (This does not apply to "wrappers" that add a GUI to a program.) It only indicates that a program was not properly designed at the outset and needed to be fixed in a patch-like manner. I would never, ever suggest that it be done, except as an interim solution until the program is redesigned properly.

Further, the writer failed to identify the "we". But it doesn't really matter to me any more than it would to any end user of the product. "We" would just want the product to work correctly. There is little excuse when it doesn't. The fact of the matter is that pacman fails to properly do its job by first finding if there are enough resources to accomplish the task. That is so foundational that overlooking that boggles the mind. (And please believe that I am stating this fact in the nicest manner possible, since trying to ignore that failure would be an act of enabling.)

Also, a more careful reading of my post would have made it possible to realize that I never promoted the idea of using deltas only. (Another strawman argument.) It was simply a comparative usage, and nothing more. I have no knowledge whether or not "versioned libraries" = "shared objects" as was implied, nor if indeed that has anything to do with the subject. Regardless, and to cut to the chase of the matter, pacman must be designed to know which packages are needed by another package, and which are not. I would have thought that was part of the process, and I'll bet that most users would have thought the same.

It also appears we need to repeat that the user does not want to do things manually. Please note that was "does not". The purpose of the computer is to automate, not force the user to jump thru hoops. Therefore, the program must list the possibilities which will come the closest to the needs of the user, not the needs of the program. Calculating the space available and the space needed is not rocket science. Nor is the next step; to figure what could safely work within the parameters available.

The "fort" item that was written is just too easy to address. It always explains why some people think everything is about them. Any reader can do a search on this topic for the word "stupid". Please note who first used it, and thereby authorized its use. It is interesting to note that he was not so attacked as I. (I'm beginning to understand the "we". It is sadly humorous. Sociopathic behavior always strikes me as such.)

I hope he understands that some of us don't take well to threats. He will have to stop his selfish attacks on others, and that goes for the rest of the "we". We've had enough. "We" will just be ignoring him now.

tl;dr

Excuse me, but can somebody briefly explain how this was implemented?
There is some new "CheckSpace" flag in the pacman.conf
how this all works? it's not documented in manpage...

CheckSpace does the following:
1) figures out your partition layout
2) approximately calculates how much additional space is needed on each partition the install/upgrade
3) compares that to the available space

There is also a second line of defense where pacman aborts on package extraction errors (primarily caused by no space to extract the files) which can happen if something is busy fulling up the hard-drive.

Cool. THX.
BTW what means "pacman aborts"? that means that transactions are implemented?