FS#16394 - Split Packages in AUR
Attached to Project:
AUR web interface
Opened by Harley Laue (losinggeneration) - Tuesday, 29 September 2009, 17:43 GMT
Last edited by Lukas Fleischer (lfleischer) - Tuesday, 27 May 2014, 18:47 GMT
Opened by Harley Laue (losinggeneration) - Tuesday, 29 September 2009, 17:43 GMT
Last edited by Lukas Fleischer (lfleischer) - Tuesday, 27 May 2014, 18:47 GMT
Closed by Lukas Fleischer (lfleischer)
Tuesday, 27 May 2014, 18:47 GMT
Reason for closing: Implemented
Additional comments about closing: Implemented in 3.0.0.
Tuesday, 27 May 2014, 18:47 GMT
Reason for closing: Implemented
Additional comments about closing: Implemented in 3.0.0.
-----------
Base Package Details:
$basename
$url
...
etc
----------
Split Package Details:
$splitname1
$depends
...
etc
---------
$splitname2
$depends
...
etc
---------
[comments]
I don't know if the terms Base/Split should be used or not, but that's the general idea of how it could be done.
I maintain kernel26-fbcondecor which is based on kernel26 from [core]/abs which is now a split package. It builds and installs fine with makepkg but I can't upload it to AUR.
I guess the problem is that AUR parses the PKGBUILD's variable $pkgname to generate the package name. In split packages $pkgname is not a variable but an array with the names of every resulting binary packages. The name of the package itself is stored in the variable $pkgbase.
So AUR should either parse $pkgname for normal packages or $pkgbase for split packges to generate the package name. It could e.g. look if there's a variable $pkgbase in the PKGBUILD or not or if $pkgname is a variable or an array to determine the type of the package.
- move the url, pkgdesc, groups, backup, depends, optdepends, replaces and install variables from package_kernel26() function before the build() function;
- combine the contents of the package_kernel26-headers() with build();
- remove package_kernel26(), package_kernel26-headers() and package_kernel26-firmware() function definitions;
I'm the maintainer of fbsplash-themes-arch-banner and had to trick the AUR in various ways to get that split package uploaded. The package is split to give the users a chance to keep the initcpio a bit smaller.
First of all it creates a much bigger overhead for the user then one package PKGBUILDs. For example let's take a look at xapian-bindings (xapian-{python,ruby,php,etc}-bindings). For penguintv a user needs only the xapian-python-bindings. Now (s)he can only dl the PKGBUILD for this package, build it and install it. In case of split PKGBUILDs one has either to edit the PKGBUILD to get only the needed package or delete the not needed packages after building. In both cases not KISS anyway - or at least much unnecessary work.
Then the maintainer: On the first glimpse it seems to be less work to create multiple packages from one PKGBUILD, but then one needs to maintain much more packages, for which one cannot guarantee that they actually work, simply because they are not needed. Once again xapian: Some time ago I wrote a split PKGBUILD for xapian-bindings, which ended in almost ten packages from which I needed and used only one. For now I have only to take care of the python-bindings and that's it. In case of split PKGBUILDs I have to check each built package if it's ok and so on. Way too much work compared to single PKGBUILDs.
Official packages are provided as binary packages, so it is not important for the user how these were built. He/she can only dl the packages she/he needs and that's it. On the AUR the situation is different.
I think it's clearer and more straightforward for noobs and people who don't care about package technicalities (though experienced): one dependency - one PKGBUILD.
The same applies to other packages like docs or headers/firmware, etc.
PS: I sometimes don't understand what the need is for split kernel packages in the AUR. The idea of omitting the kernel26-headers is that all packages built upon these headers are provided as binaries, so they don't need to be rebuild. For the AUR kernels you mostly need the headers to build your own modules, etc. So not adding the headers to the package is rather a disadvantage.
PPS: On the other hand a clearly commented PKGBUILD (eg, kernel section, header section, firmware section) provides a similar functionality - if one wants to edit the PKGBUILD.
PPPS: unvote
Though it's convenient for package builders, things get more cluttered for users when getting to AUR.
And as I said above, I don't think it makes AUR PKGBUILD maintaining easier.
IMO, ABS and AUR are two different pairs of shoes.
Btw, haven't found a makepkg option to build only certain packages from split PKGBUILDs.
"First the maintainer still has to maintain only one single PKGBUILD and therefore only one single package in AUR as before."
That's not really true. Once a split PKGBUILD providing for example the xapian-python-bindigs is in AUR the single PKGBUILD is obsolete. And here it starts to get cluttered (eg, duplicates).
"By default makepkg builds and installs every resulting binary package," - sometimes this is _not_ intended!
Sure, flexibility to do it either this or that way is a reason which speaks for introducing split PKGBUILDs to AUR.
PS: Personally, I have no problem with split PKGBUILDs in the AUR. But I think it diverges from current "intuitive" use of the AUR.
PPS: "The PKGBUILDs wouldn't need to be reversed to single packages anymore as it has to be done now." Depends on how you see it: "less" work on side, "more" work on the other... Merging more package functions into one is not that hard.
And split packages in general indeed make something simpler. See e.g. cdrdao, which is primarily a CLI tool, but has also a GUI named gcdmaster in its official source package included. Both are built and installed in one step with the same ./configure && make && make install procedure. It was just possible to disable the GUI by setting a ./configure switch. The official binary package of cdrdao in [extra] didn't install gcdmaster (had set this ./configure switch), because including gcdmaster would need GTK dependencies for a CLI tool. Most users only want the CLI tool and would have been forced to install GTK if gcdmaster would have been included. Now with split packages it is possible to add gcdmaster to the official binary package with the advantage that users who only want the CLI tool can only install cdrdao without the GTK dependencies and users who want the GUI, too, can additionally install gcdmaster with the GTK dependencies.
I don't know xapian, but I know the kernel packages. I'm the maintainer of kernel26-fbcondecor. Split packages don't mean that the AUR packages are split. Only the resulting binary packages are split. For kernel26-fbcondecor this means that there's still only one package kernel26-fbcondecor in AUR and not three packages. But this one AUR package results in three binary packages kernel26-fbcondecor, kernel26-fbcondecor-headers and kernel26-fbcondecor-firmware. That's the same as with the stock kernel, which also has only one PKGBUILD kernel26 which results in three binary packages. So there's no need for cluttered duplicates in AUR.
Btw., if AUR supports split packages it doesn't mean that the maintainers are forced to split their packages.
""By default makepkg builds and installs every resulting binary package," - sometimes this is _not_ intended!"
Then you have the possibility to uninstall the unneeded packages or tell makepkg to only build the needed packages as Daenyth explained.
"PS: Personally, I have no problem with split PKGBUILDs in the AUR. But I think it diverges from current "intuitive" use of the AUR."
The current "intuitive" use still remains. For the user it doesn't change anything except for a better flexibility. If a user wants to install kernel26-fbcondecor he now has to download and build the package kernel26-fbcondecor, which now results in one binary package kernel26-fbcondecor which includes everything. With split packages the user still downloads and builds only one package kernel26-fbcondecor but then gets three binary packages kernel26-fbcondecor, kernel26-fbcondecor-headers and kernel26-fbcondecor-firmware which also include everything by default.
And it's not confusing, because the official stock kernel is now also split into these three binary packages. The current situation is more confusing as the user needs to install different kernels by different methods and new users could easily be confused by not having the same packages for kernel26-fbcondecor as they get for kernel26.
If splitting the official kernel26 package itself makes sense is a different question. I don't like the splitting of this package much, but I also see the possible advantages. But my kernel26-fbcondecor is based on kernel26 and I'd like to keep kernel26-fbcondecor as similar to kernel26 as possible. That's primarily not a question of more or less work but of consistency.
"PPS: "The PKGBUILDs wouldn't need to be reversed to single packages anymore as it has to be done now." Depends on how you see it: "less" work on side, "more" work on the other... Merging more package functions into one is not that hard."
Usually this is correct, but not in this case. In case of split packages it's less work for the maintainer and not more work for the user. See the kernel example.
How dependencies are handled is another question which need to be implemented in AUR, ABS and/or pacman. But I don't see a big problem here.
Even though I don't like this feature, it should be included because PKGBUILDs with
split packages are valid. The AUR should be able to accept any valid packages. The
biggest barrier is parsing these PKGBUILDs.
Bash is an executable format, it's not for parsing data out of. The results from
running a PKGBUILD can differ from system to system, but what we need is consistency.
We need the same results whether you're running i686, x86_64, or whatnot. Furthermore,
executing arbitrary scripts is not only a security risk but a serious security flaw.
We need an easily parsable format for metadata in source packages, maybe like what is
provided with binary packages (PKGINFO).
If you want current and advanced features like this I urge you to pressure upstream
(pacman/makepkg) to provide the needed formats to make this possible.
I read your ML proposition of adding a SRCINFO file to the src.tar.gz when running "makepkg --source".
It is a elegant and consistent way without much parsing. - Also very useful for third party applications.
Is this a planned feature or only a loose idea for now?
louipc: And BTW there is already bash parser. We are parsing bash script so we definetely SHOULD use bash. You say the bash is security flaw? I say all we need to do is strip bash to minimal version, delete all calls that are executing another binary or creating new process and we have secure bash parser in form of patch to bash or some similar shell (busybox?).
I'll take a look how hard it could be to make some shell secure for AUR purposes and i'll let you know if i succeeded.
And i've just remembered that there is restricted mode in bash, which can be great start (if it's not sufficient right now).
Technically speaking binary packages minus the PKGINFO could contain most the 'metadata'.
You could write some sophisticated namcap like tool to extract out everything (arch, license,
pkgname, pkgver, etc) from the executables, and included docs. In the end it's much easier to
output an easily parsable PKGINFO.
In the case of source packages the PKGBUILD is the executable. Do we really want to have a
complicated parser to extract interesting data from the executable, or do we just want that data
included by the packager in an easily parsable format?
Anyways this discussion is something that should take place on the pacman mailing list, or pacman
bug tracker. A sophisticated PKGBUILD parser will not be implemented in the AUR as long as I am
the maintainer. Cheers.
louipc: PLEASE! i don't want to see archlinux package system bloated same way as the debian's one. Behave like a real men and make the parser.
In 30 minutes i've created simple parser which can be modified to be more secure and to be able to run in patched bash (currently i am not even able to run it in restricted bash, but it can be done). Look at the code: http://aur.pastebin.com/mhc43SVu
And output?:
<?php
$packages=array(
'ldns-lib',
'ldns-tools',
'ldns-drill',
);
$pkgdescs['ldns-lib']='libldns DNSSEC enabled DNS library';
$pkgdescs['ldns-lib']='Some utilities using libldns';
$pkgdescs['ldns-lib']='Drill DNSSEC utility from LDNS package (ala enabled Bind's dig)';
This is the way to go.
<?php
$packages=array(
'ldns-lib',
'ldns-tools',
'ldns-drill',
);
$pkgdescs['ldns-lib']='libldns DNSSEC enabled DNS library';
$pkgdescs['ldns-tools']='Some utilities using libldns';
$pkgdescs['ldns-drill']='Drill DNSSEC utility from LDNS package (ala enabled Bind's dig)';
ArchWay you remember? (KEEP IT SIMPLE STUPID!) Actually i thing that we should get rid of .PKGINFO too. PKGBUILD should be in some easily parsable (but bash-compatible) format, so we can just COPY-AND-PASTE exact PKGBUILD section to .PKGINFO without any expensive processing (well we should trim whitespaces, validate against regexes, etc...). There are many ways to write code that is easily executable by several programming languages (eg.: code which has meaning in C,PHP and BASH: http://en.wikipedia.org/wiki/Polyglot_(computing) ). This sort of PKGBUILD format can be invented with little effort.
and BTW modified bash is not complex parser. all we need is to add some header files. i will look at it few days later when i will get back from holydays.
and face it: bash is the only good parser for parsing bash code.
Changing the PKGBUILD format won't happen, either. Too much work.
pacman-dev, or the pacman bug tracker, or elsewhere. Please see that it does
find its way there.
If you want to work on a bash parser or even on a new implementation of the
AUR, that's awesome and I encourage you to pursue what you belive in. Just
please don't be too pushy.
Thank you.
(but unfortunately without executing PKGBUILD in sandbox, so it's bit incomplete)
It's about supporting split packages in AUR. In other words, pacman and makepkg can handle split packages, but split packages can't be uploaded to AUR, because AUR rejects those packages. So what sense does it make to discuss a bash parser or whatever on the pacman-dev mailing list or the pacman bug tracker?
It's a missing feature of AUR, not pacman or makepkg, so it has to be discussed here or on the aur-general mailing list.
Not pacman needs to be fix and to be able to parse the PKGBUILDs of split packages - it can already do this - AUR needs to be fixed and to be able to parse and accept PKGBUILDs of split packages.
And why should bash be patched just to be able to parse PKGBUILDs, .SRCINFO files or whatever?
1. I see no debian like overbloat since nothing changes for the package and packager. Providing some useful metainformation doesn't "bloat" anything. It is simply a text file added to other textfiles.
On the other hand to create a bash parser for each application which might need some information out of a PKGBUILD is senseless duplicated work.
2. It's totally senseless to change/patch/whatever bash just to get some infos out of a textfile (WTF????).
3. Sourcing a PKGBUILD after making the source tarball is quite dangerous. And there must be a regex parser anyway. Think of the vars inside the package_*() functions. There is no way to get this information by sourcing only. You have to _execute_ them to get the local vars. This has lots of side effects and simply not needed for some purposes.
@Heiko:
I disagree.
We should think about how pacman/makepkg provides information. This information _is_ a big part of the package and thus of the package manager. PKGBUILDs are the domain of makepkg. Imo, because makepkg can create source tarballs, it also should provide some further information about this source package as it does for binary package - at least some basics. This information file (eg, .SRCINFO) is not intended as a replacement for the buildfile.
However, I think it is more efficient and also sane to create a SRCINFO file during the build process of the source tarball.
First of all no duplicate work.
Second, it _is_ safer.
BTW, here are some thoughts of mine about the SRCINFO file:
http://mailman.archlinux.org/pipermail/pacman-dev/2010-August/011461.html
Vlad
And about the PKGBUILD:
It is not so easy to change the format of a PKGBUILD. There are thousands of them in ABS/AUR. All of them have to be replaced. Too much (senseless) work.
PKGBUILDs are written in bash. So, theoretically, they should support all of bash's features/syntax. And they should be easily understandable. Such restrictions like "only this is allowed" are anti-productive.
DonVla:
ad> Think of the vars inside the package_*() functions. There is no way to get this information by sourcing only.
did you at least briefly looked at my code???!
get_function() {
export -f "$1" || exit 1;
bash -c 'env' | tr '\n' '\0' | sed 's/.*'"$1"'=() {\(.*\)}.*/\1/g' | tr '\0' '\n'
}
ad> debian overbloat
sorry, but storing the same information on two different places (especialy in the same tarball) IS OVERBLOAT.
changing the PKGBUILD format was extreme example, but it's still better than adding .SRCINFO (which is not needed at all, so there's no point for this).
ad> "only this is allowed"
well i guess you will not need to execute "rm -rf /*" (or any other command) to get pkgname, pkgdesc and depends and if you will, you should change the PKGBUILD. if pkgname (or something similar) is variable, it can be probably handled by BASH's internal processing features.
My conclusion: I am strightly against patching makepkg or pacman to support some .SRCINFO bloats. Again: It's AUR problem and AUR should fix it.
This looks to me like when Canonical thought that kernel release engineering will get changed to comply with Ubuntu releases. Linus said them to go away (actually something more rude i guess).
It is not (only) a parsing problem. It is not only about getting, but also about providing information.
Once again, I see no overbloat in making a strict distinction between build and info file. As Louipc said, you can omit the PKGINFO file and extract infos out the bins. But this is too much overhead and senseless. Same applies imo to the src.tar.gz packs.
And what is this:
"PKGBUILDs are in BASH, so only BASH can parse them well, but it's insecure to execute user contributed scripts, so we need to patch bash to be enough secure to execute untrusted scripts." ???
Don't get me wrong, but this is total ... <silence - no offense>. I totally see no need in patching bash.
However, if you got some _real_ and coherent suggestions how to do this, then mail them to the aur-general list.
In your code you're doing stuff which has already been done dozens of times. Take some time and have a look how third party apps get infos out of PKGBUILDs.
"I am strightly against patching makepkg or pacman to support some .SRCINFO bloats."
But you want to patch bash then???
this patched bash will be single binary installed only on single AUR machine
$ du -hs /bin/bash
592K /bin/bash
it will not be a patch to bash package itself. let's say it will be package called bash-parser. it will no longer be bash, it will be just part of AUR used to parse PKGBUILDs. simple utility.
.SRCINFOs will be lots of files with redundant informations stored all along the AUR - one per package and in every src package.
If we can (and yes - we can) parse the existing PKGBUILDs directly in AUR without converting them to some new .SRCINFO file WE SHOULD DO IT. Why should makepkg or pacman care that AUR is not able to parse PKGBUILDs? There can be secure PKGBUILD parsing tool or library, but makepkg should stay unafected.
PKGBUILD definition or a customised bash parser.
Open a new ticket if you want to discuss that.
FS#15043- Need better parsing of PKGBUILDsFS#15043. Both are somehow related butFS#15043is not about supporting split packages like this feature request.Maybe we can just comment out the conditions that are preventing invalid PKGBUILDs from being uploaded or disable them only in cases where we'll found that pkgname is array... (simple regex)
http://aur.archlinux.org/packages/kernel-netbook/kernel-netbook/PKGBUILD
With this method also with yaourt, the only thing that one user has to do to get a splitted package is set SPLITTED=y....
@Thomas: No, there's no workaround. There already have been made many approaches, but none of them worked.
Heiko: omg. even GIT or anonymous FTP would be better...
Here you are: http://aur.archlinux.org/packages/splittest/splittest/PKGBUILD
Do not even try to fix it in AUR parser. Since now it's not a bug. It's a feature :-)
Well your splittest hasn't been tried, yet. Anything else has failed. And it was not only me who tried to fool AUR.
Nevertheless it's a workaround. Split packages should be supported by AUR without such a hack in the PKGBUILD.
> Well your splittest hasn't been tried, yet.
What do you mean? i have successfully uploaded it to AUR and i have succesfully installed both packages from it using yaourt. If such packages will become popular and moved to [community] then we can remove the hack. That hack can be simply added (or removed) to packages.
I think that development of ArchLinux is WAY more important than respecting some AUR parser. I would rather replace AUR by GIT repository than preventing people from improving ArchLinux itself. AUR is just tool and i want to make packages not use AUR. I think that AUR would be more usefull if we have removed the parser checks and let people upload whatewer they want (no matter if we manage to parse it or not) until it will get fixed...
Now we have to implement this without workarounds.
"Now we have to implement this without workarounds."
So the AUR parser has to be fixed. And, btw., of course it is important to develop Arch Linux. But that doesn't mean, that AUR must not be improved. AUR belongs to Arch Linux as anything else.
"AUR belongs to Arch Linux as anything else"
Well... i don't think so. I'd prefer ArchLinux without AUR to ArchLinux without well-maintained user-submitted packages...
So AUR definitely IS a part of Arch Linux regardless of your opinion about that. You don't need to use it, if you don't want to.
It's time to stop spamming :-) i am glad that we have both the same opinion :-)
Right now when a user tries to submit a split package, he faces the strange message:
'Invalid name: only lowercase letters are allowed'
It can take a while before he figures what's going on.
The following patch at least tries to detect the situation and sends a somewhat better
error message:
'Uploading splitted packages to AUR is not allowed'
(at least until #15043 is fixed)
BTW, I really like the idea of having 'makepkg --source' generating machine-readable metada,
I'll look into it.
The "true &&" "hack" doesn't affect neither makepkg or even the info fields in the package home page.
FS#15043(which blocks this from being implemented) was closed on May 3rd anyway:"Reason for closing: Won't implement
Additional comments about closing: Please use .AURINFO if the PKGBUILD parser doesn't work for you."
Also, .AURINFO doesn't help here either:
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n84
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n263
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n285
Basically it just overwrites what was already found in the PKGBUILD and errors on split packages.
If it's too hard to support something and we have an extremely simple and efficient workaround, why are you mocking the workaround?
I'm not really sure what you're asking from the AUR anyway. To show all the package names in the relevant 'home page' and/or to be able to build/install/update just part of a package?
Or just to ignore the error when uploading your tarballs without the "dirty" hack?
I'm not mocking it, but pointing out that it's a workaround and not a proper fix. There /is/ a difference.
For the rest, read all previous responses. They pretty well cover the rest of your questions.
I've been trying to explain to you the entire time why the "workaround" is the only thing you're ever gonna have, yet you still keep on insisting on complaining how "dirty" it is and that Lukas should just come up with a better parser. He _won't_ because it's too hard (which ironicly in turn is too hard for you).
And had you actually read the discussion yourself you'd know all this.
FS#15043.Also I wonder why does it always magically make the other person troll when you're losing the argument.
FS#15043isn't really relevant for this. It was "fixed" by other means (2 was fixed outright, 1&4 can be handled by .AURINFO, and 3 was basically never implemented.) .AURINFO doesn't help here and the issue isn't "fixed" but does have a known workaround to the problem at hand.Det: please seek professional help.
Also the reason why I mentioned
FS#15043was that the general idea is about having a better parser, which in turn would allow split packages, which in turn, after closing, also blocks this from being implemented (see the "This task depends upon" note).The real problem here stems from what to do with it. For instance, how should the package(s) be displayed, searched for, etc. A final solution to these questions were never agreed on, thus this issue lived on.
FS#15043was only /really/ relevant here if the parser was reworked to better support more/all of the features BASH supports (which was eventually decided against as it can depend on things other than just BASH.) The real meat of this issue though wasn't about the parser at all, which is why I keep telling you that related issue isn't really all that relevant here.That's it. I'm done talking to you on this issue as that should be more than enough to quite anyone who's not trolling *hint hint*
But the thing is that the discussion here for the last 2-3 years has been about _parsers_ and only the first two posts about what to do, if the problem didn't exist (since it wasn't brought up yet). You don't seem to really understand the connection that the parser/AURINFO issue _is_ the root of this problem - from there, once that's fixed, you could decide how to provide that info to the user in the web interface.
And just a FYI, this is even categorized as 'PKGBUILD Parser'.
[1] https://mailman.archlinux.org/pipermail/aur-dev/2014-April/002676.html
Just awesome work.