Arch Linux

And how could pacman check that?

We probably shouldn't use partial files at all when it comes to DB downloads. It makes sense for packages where the content of a specific filename is immutable, but it doesn't make sense for something like the database.

> And how could pacman check that?
When starting new download check if dbfile last update time as specified by server is newer than that of local file or .part file.

It's easier to just go with Dan's idea though.
Then such check won't be needed (even for packages, I cannot imagine a situation when package file gets updated without changing its name).

So do we need to add a "int resume" parameter to all of our download functions?

- int _alpm_download_single_file(const char *filename,
alpm_list_t *servers, const char *localpath,
time_t mtimeold, time_t *mtimenew)
- handle->fetchcb(url, localpath, mtimeold, mtimenew) (+ the callback itself)
- int _alpm_download_files(alpm_list_t *files,
alpm_list_t *servers, const char *localpath)
- static int download(const char *url, const char *localpath,
time_t mtimeold, time_t *mtimenew) {
- download_internal(url, localpath, mtimeold, mtimenew)

?

That was my initial thought, although now I am looking closer at our download_internal logic. I noticed we don't first to a fetchStat operation first, which might allow us to do some more careful things with mtime and size checks that we aren't currently doing.

The funny thing with the mtime stuff right now is we only use it for databases. If we didn't, our logic would be completely broken as passing an identical mtime in aborts the download early without any regard for whether the download is finished.

I would have tried to address this before 3.3 if it was an easy problem, but it might have to wait until 3.3.1 or even 3.4 as it will probably require an API change.

Xavier- don't know how far you got on this but let me look around for some work I might have started here.

I did absolutely nothing yet, but I wanted to look into it so I added it to my task list to not forget.
I will wait for your initial work :)

I should say that this does not apply to db alone. If a package download is interrupted, same thing happens. All this while I've been blaming my mobile connection (to an extent it _is_ at fault, because it frequently stops transmission for a short while when it goes from 3G to EDGE and vice-versa).

Workaround there is to remove the cached (partial) download, and restart. If that does not work, wget && pacman -U =p

Wait, what ? I interrupt and resume package download all the time, this works fine. Packages have an unique filename so don't change.

However, the database changes all the time while keeping the same name. This is what gives us some troubles.

So do you mean

a) package downloads should never error out like:

error: failed retrieving file 'foobar-1.0-1-i686.pkg.tar.gz' from
foo.bar.org : Requested Range Not Satisfiable

-OR-

b) package downloads can error out like that (if an update is interrupted and then resumed with an -Syyu for instance) but it's because the database changes all the time

-OR-

c) I should not stop cursing my ISP

I mean that :
- this should never happen for package download (unless you are hit by some very weird and more serious bug like  FS#16359 )
- this might happen for database download, because of a limitation of pacman (not a bug), which is what this whole report is about

So if your situation is not the same as  FS#16359 , please open a new bug report, with as much details as possible.

Before that, let me just clarify this. By interruption, I meant network interruption (physical or not; cable unplug, congestion, whatever). A ^C and then resuming will always work (even with db). I simulated a disconnection in the following:

[schiv@v3000 ~]$ sudo pacman -S go-openoffice
resolving dependencies...
looking for inter-conflicts...

Targets (1): go-openoffice-3.1.1.2-1

Total Download Size: 166.99 MB
Total Installed Size: 361.03 MB

Proceed with installation? [Y/n] y
:: Retrieving packages from extra...
^Co-openoffice-3.1.1... 1392.5K 82.6K/s 00:34:12 [---------] 0%
Interrupt signal received

[schiv@v3000 ~]$ sudo pacman -S go-openoffice
resolving dependencies...
looking for inter-conflicts...

Targets (1): go-openoffice-3.1.1.2-1

Total Download Size: 166.99 MB
Total Installed Size: 361.03 MB

Proceed with installation? [Y/n] y
:: Retrieving packages from extra...
error: failed retrieving file 'go-openoffice-3.1.1.2-1-i686.pkg.tar.gz' from mirror.internode.on.net : Requested Range Not Satisfiable
^C
Interrupt signal received

I believe it's related, which then as a by-product is leading to  FS#16359 . So, in the words of this task's summary:

"The bug here is that pacman does not seem to even check if the file was changed and goes straight to continuing the download of <pkg>.part"

But alas, I wouldn't be able to attach the .part file even if I interrupted it at 10K, because it'd be over 1G by the time I do a ^C! (moving on to  FS#16359 )

I still don't get it. You say the file was changed. How was the file changed ? A given package never changes on the remote server.

Another thing I don't get : in the first download, how did you simulate network disconnection ? And when you did that simulation, the download did not stop ?
And you had to to ctrl+C after the simulated interruption ?

I stand corrected - my issue is actually borked before even resuming. But then again, libfetch (if that's the default downloader) is the problem (I will be commenting on this in the other task), so you may want to watch that thing with regards to this bug.

As for the simulation, simply networkmanager gui > disconnect, or killall $dialer where in my case is pppd. One could simply reproduce this by pulling out the network cable as well, or bringing down the interface, or any other brutal method (hint: please try).

Correct - the download did not stop. It just paused with no visual feedback, but in the background the file gets filled with crap at the speed of sound.

I request re-opening this task.

Using "Pacman v3.4.1 - libalpm v5.0.1", I recently had the same problem; for each mirror, I was getting "error: failed retrieving file 'extra.db.tar.gz' ... Requested Range Not Satisfiable" and finally "error: failed to update extra (Requested Range Not Satisfiable)".

The solution was to do the following: sudo rm /var/lib/pacman/extra.db.tar.gz.part

After this, everything went swimmingly.