FS#43498 - [pacman] makepkg VCS rebuilding - new flag(s) needed

Attached to Project: Pacman
Opened by nfnty (nfnty) - Sunday, 18 January 2015, 19:21 GMT
Last edited by Allan McRae (Allan) - Monday, 01 February 2016, 00:28 GMT
Task Type Feature Request
Category General
Status Unconfirmed   Reopened
Assigned To Allan McRae (Allan)
Architecture All
Severity Low
Priority Normal
Reported Version 4.2.1
Due in Version 5.0.0
Due Date Undecided
Percent Complete 0%
Votes 0
Private No

Details

I've been having two problems with --noextract when rebuilding modified VCS sources, specifically with git bisecting.

1: --noextract can't be run with prepare(). It is necessary to run prepare() again when checking out new commits.

2: pkgver() is not run. $pkgver is not updated and --force has to be used when building and therefore discarding the old package. Making it hard to keep track of what commit the package was built with and not being able to compare packages.

I'm proposing two things. Second proposal is also beneficial for non-VCS sources.

1: Adding the flag --vcsrebuild
Like --noextract only with pkgver() and prepare(). --noprepare should work as is.

2: Adding the flag --prepare
OR
2: --noextract calls prepare()
--noprepare explicitly called when needed
This task depends upon

Comment by Allan McRae (Allan) - Sunday, 18 January 2015, 21:20 GMT
Why do you even need --noextract?
Comment by nfnty (nfnty) - Sunday, 18 January 2015, 23:01 GMT
Please elaborate.

After consideration only one general flag --rebuild (short form -B) is needed. Basically a flag that leaves the srcdir completely intact and runs the normal build process again, including pkgver() and prepare(). This flag would be beneficial for VCS and non-VCS builds alike.

--noprepare can be explicitly appended if needed. Maybe --nopkgver should also be added.

This also makes --noextract more or less unnecessary.
Comment by Allan McRae (Allan) - Monday, 19 January 2015, 00:40 GMT
makepkg does leave the source directory intact. If you want the VCS sources not to be updated, use --holdver.
Comment by nfnty (nfnty) - Monday, 19 January 2015, 00:48 GMT
No it doesn't. --holdver only makes sure that new commits are not fetched. Makepkg makes a new branch from HEAD called 'makepkg' and builds from there.
Comment by Allan McRae (Allan) - Monday, 19 January 2015, 00:59 GMT
Are you using pacman-4.2? Check again...
Comment by nfnty (nfnty) - Monday, 19 January 2015, 01:35 GMT
Yes, 4.2.0-5. With only --holdver a new branch called 'makepkg' is created from 'origin/HEAD' and the repo is not updated (with `git fetch --all -p`). With --noextract everything is working except no pkgver() and prepare().

You can check yourself what --holdver is doing on line 564-577 in scripts/makepkg.sh.in or makepkg for that matter.
Comment by Allan McRae (Allan) - Monday, 19 January 2015, 01:44 GMT
You want to look at the "extract_git()" function (line 580) and pay attention to the "updating" variable.
Comment by nfnty (nfnty) - Monday, 19 January 2015, 02:25 GMT
Yes, it's always issuing `git fetch` if there is a directory already existing and `git checkout --force --no-track -B makepkg origin/HEAD` if "$fragment" is zero length. Meaning that it won't build from the modified current branch. --holdver is only stopping makepkg to update all remotes and pruning afterwards.

Something which is stopped with --noextract. Also --holdver is useless with --noextract.

What's your point?
Comment by nfnty (nfnty) - Monday, 19 January 2015, 02:30 GMT
I've created a rough draft (can't seem to get pacman to build, tests failing) of -B, --rebuild. Attached patch.
Comment by Allan McRae (Allan) - Monday, 19 January 2015, 02:34 GMT
I have reread your original report several times now, and I have no idea what you are doing.

You mention "git bisecting", but given your responses, I have absolutely no idea how you are doing that. Can you please provide a step-by-step guide to your workflow?

Comment by nfnty (nfnty) - Monday, 19 January 2015, 03:15 GMT
This is something I use when finding regressions fast.

[first terminal]
1. cd "$srcdir/$srcname"
2. git bisect start
3. git bisect good "$oldpkg_commit"
4. git bisect bad HEAD
[second terminal]
5. makepkg --install --noconfirm --log --syncdeps --rmdeps --force --noextract
6. Test package
[first terminal]
if bad:
7. git bisect bad
if good:
7. git bisect good
[second terminal]
8. makepkg --install --noconfirm --log --syncdeps --rmdeps --force --noextract
9. Test package
...
...
...
continue until done.

Basically when updating packages and I hit a regression I go into $srcdir and start a git bisect. Therefore makepkg should do everything as usual minus the extraction(updating) and with pkgver() otherwise I have to overwrite the current package.
Comment by Peter Lewis (petelewis) - Monday, 17 August 2015, 13:00 GMT
I've been trying to do the same thing and just found this. I also can't get makepkg (with -e) to build from the sources as they are (i.e. after checking out a specific commit in $srcdir). It always seems to build from HEAD. To reproduce, follow the workflow above. To check, take a look in $pkgdir after building -- the contents are different from the currently checked out commit in $srcdir.
Comment by Allan McRae (Allan) - Monday, 17 August 2015, 13:35 GMT
I just build a git package, went and edited $srcdir/<source> such that it should get a compile error and ran "makepkg -e". It did not recheck out the source. So that is fine.

Then I tried a bisect in the $srcdir/<source> and running the build manually. It build from head:

allan@arya /home/arch/build/pacman-git/src/pacman ((76fc724...)|BISECTING)
> ./src/pacman/pacman -V

.--. Pacman v4.2.0-234-g76fc7 - libalpm v9.0.1

Note the version difference. So I can replicate the issue, but not for the reason described here. Someone with more git knowledge will need to look at this.


Note that you can bisect in the git checkout in the directory with the PKGBUILD and use the "#commit=...." fragment to work around this.
Comment by Peter Lewis (petelewis) - Wednesday, 19 August 2015, 09:04 GMT
There does seem to be an issue here, but I was a bit quick with my comment. I realised that the issue I was facing is due to the source tree being dirtied with some temp files by building. Since there were some stale things lying around specific to the previous version, it wasn't rebuilding them. I also didn't notice these, since the .gitignore was setup to not show them in 'git status'. I suppose the "advice" is just to ensure a clean $srcdir following the checkout during the bisect.

Is that the same thing causing your issue Allan, or something else?
Comment by Allan McRae (Allan) - Wednesday, 19 August 2015, 09:34 GMT
That does not appear to be the cause...

I make a clean build directory and just extract the source (makepkg -Co). Then do a git bisect and manually build from within that directory. It still builds from head. Bisecting just is not working for some reason.
Comment by Andrew Gregory (andrewgregory) - Wednesday, 19 August 2015, 14:20 GMT
> allan@arya /home/arch/build/pacman-git/src/pacman ((76fc724...)|BISECTING)
> > ./src/pacman/pacman -V
>
> .--. Pacman v4.2.0-234-g76fc7 - libalpm v9.0.1
>
> Note the version difference.

Am I missing something? Those version numbers match and `makepkg -e` appears to be building whatever commit I checkout.
Comment by Allan McRae (Allan) - Wednesday, 19 August 2015, 21:49 GMT
Oh crap - stupid "g" in front of the version number! So everything works for me...

Back to not understanding this bug then.
Comment by Andrew Gregory (andrewgregory) - Thursday, 20 August 2015, 03:14 GMT
OP wants to be able to tell makepkg to use $srcdir in its current state and also run pkgver() and prepare(). At the moment, with --noextract makepkg will not run pkgver() and prepare(); without --noextract makepkg will always run `git checkout origin/HEAD` for existing repos, obliterating local changes.
Comment by Allan McRae (Allan) - Monday, 14 December 2015, 14:51 GMT
  • Field changed: Category (Packages: Core → General)
  • Field changed: Reported Version ( → 4.2.1)
  • Field changed: Due in Version (Undecided → 5.0.0)
  • Field changed: Architecture (All → All)
  • Task assigned to Allan McRae (Allan)
The fix I posted for  FS#46800  also does this.
Comment by Evangelos Foutras (foutrelis) - Sunday, 31 January 2016, 14:45 GMT
Running prepare() when --noextract is used no longer allows running 'makepkg -o && makepkg -e' with any PKGBUILD that applies patches in prepare(). [1]

Sure there's --noprepare which restores the old behavior, but that's a lot of extra typing for what I believe is a much more common use of --noextract.

For OP's use case of doing git bisects, you can specify the commit in the source array and thus skip --noextract since makepkg will checkout the correct commit each time.

[1] I often extract the sources using 'makepkg -o', manually edit some source files, and then use 'makepkg -e' to package it (while possibly repeating the edit/package steps).
Comment by Allan McRae (Allan) - Monday, 01 February 2016, 00:25 GMT
I agree git bisects should be done by changing the commit ID in the PKGBUILD.

We can suppress the prepare() function with --noprepare, but changing this back provides no way to tell prepare() to run unless we add a --prepare flag.

If someone provides that patch, I will look at it...
Comment by Allan McRae (Allan) - Monday, 01 February 2016, 00:29 GMT
Reopened for the moment.
Comment by Evangelos Foutras (foutrelis) - Wednesday, 17 February 2016, 07:20 GMT
Proposed partial reversion: https://lists.archlinux.org/pipermail/pacman-dev/2016-February/020875.html -- would be nice to have this included in 5.0.1.

I don't see the need for a --prepare switch to be used together with --noextract (I consider prepare() tightly coupled to source extraction); its usefulness would be minimal at best.

If someone really wants to call prepare() when not extracting sources (for some special testing/bisect scenario), they can call it from build().

Loading...