FS#14353 - [solution suggested] Pacman causes troubles with half-updated ArchLinux mirrors

Attached to Project: Pacman
Opened by Tomas Mudrunka (harvie) - Friday, 17 April 2009, 23:55 GMT
Last edited by Dan McGee (toofishes) - Friday, 05 June 2009, 13:02 GMT
Task Type Feature Request
Category General
Status Closed
Assigned To No-one
Architecture All
Severity High
Priority Normal
Reported Version 3.2.2
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 3
Private No

Details

== Summary and Info:
Sometimes when executing full system upgrade (pacman -Syu) and lot of new packages are available (new GNOME, etc...) there can be problems, when local repositories in my country have old versions of packages which are not strictly required (optional dependencies) by other packages which are already in mirror.

Optional dependendencies should cause problems if the new version of package is not compatible with old version of opt. depend package. So Pacman should also look for newest packages in few other mirrors.

Here is better description of problems that are caused by half-synchronized mirrors:
http://bugs.archlinux.org/task/14243


== Steps to Reproduce:
Do full system upgrade from mirror, which is partialy out-of-sync.


== Possible solution:
Modify pacman to check multiple mirrors and select the one with latest packages.
For example apt-get based systems are doing upgrade this way... So i can download 90% of packages from half-synced mirror in my country, other 9% from bit more updated mirror, which is in nearer country and for example the left 1% which is not on any mirror yet from original (throtled) archlinux repository. This fallback method should protect the system from being bricked by slow mirror.
This task depends upon

Closed by  Dan McGee (toofishes)
Friday, 05 June 2009, 13:02 GMT
Reason for closing:  Won't implement
Additional comments about closing:  See comments. Let's not go back and forth and discuss this to death. It shouldn't be a problem with properly synced mirrors *and* properly dependent packages, neither of which pacman can or will enforce.
Comment by Sven-Hendrik Haase (Svenstaro) - Thursday, 23 April 2009, 04:36 GMT
I was bitten by this when Gnome updated, so I vote for this.
Comment by Brendan (watricky) - Wednesday, 03 June 2009, 06:52 GMT
I'm having difficulty understanding this bug - it seems improbable. Maybe the language is just ambiguous? Can someone please elaborate?

From my understanding of the pacman database, if the database is current but files are unavailable, pacman will choose the next mirror in "/etc/pacman.d/mirrorlist", a failover, which the bug seems to suggest would be part of the solution.
Pacman usually quits if not all packages can be downloaded. The report at http://bugs.archlinux.org/task/14243 mentions a specific Server being used, possibly without the use of the mirrorlist.

As pacman supports failover, having a single server listed on its own is nulling the failover functionality making this bug silly. Perhaps that is not the case however and pacman isn't finding this package elsewhere but is somehow still committing upgrades in spite of conflicts. Does pacman really allow this or is the user overriding pacman's default behaviour with --nodeps or --force ?
Comment by Tomas Mudrunka (harvie) - Thursday, 04 June 2009, 01:46 GMT
there is no --nodeps or --force, just pacman -Syu

maybe there is problem with package lists (index of all packages in the repository, which is placed on the repository server next to the packages). all dependend packages should be added to the list (database) at the same time. im not really sure how pacman and archlinux repositories works, but i have experienced issue like in this model situation:

gnome-1.0-1 was updated to gnome-1.0-2,
but gnome-dependency-1.0-2 wasn't in my mirror yet
gnome-1.0-2 is dependent on gnome-dependency-1.0, so it installed (and maybe worked) also with 1.0-1, but there were several issues (which can be fatal in some future cases)
after a while (hour or two) i fixed my broken gnome, when gnome-dependency-1.0-2 appeared on the mirror.

so we need to update all dependent files in repository at the same time (or make sure if they will not make problems with older releases/issues). maybe more strict dependency specification should help (also adding release), but it can make users downloading new gnome each time some dependency is changed (which is bad idea).

but i think the only problem was in mirror and pacman should merge package indexes from all mirrors in pacman.conf to get URLs of latest packages. for example.

on my prefered local mirror there is gnome-dependency-1.0-1, but on switzerland mirror is gnome-dependency-1.0-2 at this time, so pacman should download all new files from my local mirror and gnome-dependency from switzerland (because i can't get latest version from local mirror).

hope it is clear... ;)
Comment by Xavier (shining) - Thursday, 04 June 2009, 09:50 GMT
does that mean that mirrors can sync from the origin while a big upload is being done, and can get a database is a not perfectly consistent state?

Maybe "adding N packages to the database" should be made atomic then, to prevent this.
So that the new database is really added only after all the gnome packages have been added to it.
Comment by Tomas Mudrunka (harvie) - Thursday, 04 June 2009, 12:19 GMT
Xavier: pacman should not suppose repositories/mirrors failproof. it should check if other mirrors are more up to date.
Comment by Xavier (shining) - Thursday, 04 June 2009, 13:56 GMT
So all users end up to overload the most up-to-date mirror, rather than their fast local mirrors which is maybe just a few hours behind?
That sounds like a bad idea...
Comment by Dan McGee (toofishes) - Thursday, 04 June 2009, 15:05 GMT
-1 on this, we already have a fallback system and it is the mirrors that are not consistent, which is awfully hard to enforce in the package manager.
Comment by Brendan (watricky) - Thursday, 04 June 2009, 15:07 GMT
This reads more like the problem was that the gnome package's dependencies were not specified correctly.

If they were specified correctly then it would not have updated gnome-1.0-1 to gnome-1.0-2 until gnome-dependency-1.0-2 was already available. Also, it would have then installed gnome-dependency-1.0-2 simultaneously as part of the update.

In a case like that, the issue needs to be reported to the package maintainer so that it can be fixed.
Comment by Xavier (shining) - Thursday, 04 June 2009, 15:20 GMT
> Maybe "adding N packages to the database" should be made atomic then, to prevent this.
> So that the new database is really added only after all the gnome packages have been added to it.

Dan just told me this operation is completely atomic. That means that if the packager/developer uses it correctly, the new database with the new gnome can always be perfectly consistent and have all the needed packages/dependencies.

The only thing which can happen then, is that a mirror picked the new database, but is still fetching all the new packages. In this case, the current mirror fallback system in pacman will work perfectly.

The other possibility is that all the packages were not added in the same time in one atomic operation, and versioned dependencies were not used (the first problem is more important).
In any cases, this is a package maintaining problem, not a tool problem.

I will probably close this as Invalid then.
Comment by Allan McRae (Allan) - Friday, 05 June 2009, 06:32 GMT
> The only thing which can happen then, is that a mirror picked the
> new database, but is still fetching all the new packages.

This is the major source of all these problems.

So I agree with Xavier's assessment that this can be closed
Comment by Tomas Mudrunka (harvie) - Friday, 05 June 2009, 06:33 GMT
yes, but what if that eg. gnome-dependency is optional dependency? gnome have lot of optional dependencies (like pm-utils), so you can still end up with crippled gnome for some reason. i also remember that i wasn't able to start X server (for about an hour till next update) after reboot (because of old/incompatible version of videocard driver or something similar). so maybe pacman should look for versions of optional dependencies (if they are installed) and wait for correct version (im not sure if pacman does this at the time). also package maintainers should be more carefull...
Comment by Xavier (shining) - Friday, 05 June 2009, 07:53 GMT
No matter whether the dependencies are optional or not, all related packages can be added together, so that you get all of them on -Su operation, and avoid any problem.

In case of non-optional dependencies, the maintainer can also specify versioned dependencies, which is probably a good practice, but it isn't strictly required.

Please just report the specific gnome / xorg problems to their maintainers when they happen, this would be more useful than this current bug report.

Loading...