FS#35050 - [makepkg] support for incremental builds with VCS sources

Attached to Project: Pacman
Opened by Lukas Jirkovsky (6xx) - Wednesday, 01 May 2013, 17:45 GMT
Last edited by Allan McRae (Allan) - Monday, 17 November 2014, 07:19 GMT
Task Type Feature Request
Category makepkg
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version 4.1.0
Due in Version 4.2.0
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

It would be great if there was a support for incremental builds when using VCS sources.

Right now, when building a package, the $srcdir seems to be removed and then updated repository content from the $SRCDEST is then exported to the $srcdir. This means that whenever package is built using makepkg, everything is rebuilt.

Because the packages with VCS sources are likely to be used by people who follow development closely and thus they are likely to be build the package very often, it would make sense to (optionally?) just update the $srcdir content, so the incremental build works as this can save quite a few hours.
This task depends upon

Closed by  Allan McRae (Allan)
Monday, 17 November 2014, 07:19 GMT
Reason for closing:  Implemented
Additional comments about closing:  In git
Comment by KaiSforza (KaiSforza) - Wednesday, 01 May 2013, 18:23 GMT
This can be done manually quite easily:

1) update git sources (use git pull origin <some branch> in the $srcdir repo)
1a) possibly run some cleanup on the repo to remove stale built files (git clean or make clean, maybe?)
2) run makepkg -e (will use the existing source tree)

But honestly, this shouldn't be too much of an issue. The only part of the whole 'clone to srcdir' bit that actually takes any time is the checkout of the files. If you have $SRCDEST and $srcdir on the same filesystem, then git will use hardlinking (see the git-clone(1) man page). You shouldn't really notices much of a change, honestly, unless you're using a repo that's >400M or so. And even that won't be too much of a hassle. I currently build the linux-mainline package using a git source, and that is the only one I could think of taking more than a couple of seconds to update, clone, and start building.
Comment by Allan McRae (Allan) - Wednesday, 01 May 2013, 21:49 GMT
Build outside of the source directory...
Comment by Lukas Jirkovsky (6xx) - Friday, 03 May 2013, 11:29 GMT
KaiSforza: sure this approach is possible, but it is not very convenient and when you forget to do it, you will find out that you are rebuilding everything. BTW, the problem is not the time to export the sources, but the time needed for the rebuild. Eg. if the git pull results in a change of one file, I want the incremental build to work so only this file is rebuilt (takes few seconds), not everything (may take a few hours).

Allan: I'm not sure what you mean. Do you mean skip making package and just use the "make install" or something similar? That would not be very clean (and I don't want to put my system into risk of some problem caused by problem in the buildsystem).

Before pacman 4.1 all I needed to do for the incremental build to work was to omit the removal of the build directory. Now that the makepkg handles the VCS sources (which I really like), it would be nice to have an option that basically does the steps mentioned by KaiSforza (ie. update sources in $srcdir and build the package without cleaning the srcdir). I think this would be a nice addition for developers.
Comment by Allan McRae (Allan) - Friday, 03 May 2013, 12:09 GMT
No, I meant:

mkdir $srcdir/build
cd $srcdir/build
../$pkgname-$pkgver/configure ....

Comment by Lukas Jirkovsky (6xx) - Friday, 03 May 2013, 14:38 GMT
Allan: that's quite good solution. At least it doesn't involve some harakiri like updating repo manually. I didn't thought of that (which is funny/sad, given that pretty much all older packages use it).
Comment by Lukas Jirkovsky (6xx) - Friday, 03 May 2013, 15:02 GMT
EDIT: I'm afraid that it wont work (at least when the projects is built using GNU make), because the timestamps of all files in the repository are changed, so everything is rebuilt anyway.
Comment by Vladimír Vondruš (mosra) - Friday, 17 May 2013, 13:29 GMT
I found a way to work around the cloning/timestamp issue by changing the output directory and adding the repository to `noextract` array, e.g.:

_svnname=blender
source=("src/${_svnname}::svn+https://svn.blender.org/svnroot/bf-blender/trunk/blender")
noextract=("src/${_svnname}")

This way the SVN repository is cloned into the same location as originally and `noextract` prevents makepkg to destroy it (currently makepkg deletes `src/blender` and then tries to copy `src/blender` to `src/blender`, which obviously fails). Then the repository can be used as before, all timestamps are preserved and there are no unnecessary copies around.

However this works only with SVN, Git repositories are cloned as "bare" and they cannot be used this way, as there is no working copy to work with. In my opinion the easiest way to support this is to have some option (e.g. `!bare`) which will clone the repository with working copy.

On the other hand this will work only if the repository isn't touched in any way (no in-source building, patching etc.), as subsequent updates might fail with conflicts (and there is no way to clean up the repository before updating, as `prepare()` is called after that).
Comment by Lukas Jirkovsky (6xx) - Sunday, 29 September 2013, 15:01 GMT
In case someone, who is interested in this being fixed, isn't watching pacman-dev: I've made a proof of concept patch that adds support for incremental builds. It currently works only with GIT. The related email to pacman-dev is this:
https://mailman.archlinux.org/pipermail/pacman-dev/2013-September/017874.html

Any comments are welcome. If everything is fine, I would like to extend this functionality to other VCSs, too. I'm also attaching a patch that can be applied directly against /usr/bin/makepkg from pacman 4.1.2 to make this easier to test.

Loading...