FS#16394 - Split Packages in AUR

Attached to Project: AUR web interface
Opened by Harley Laue (losinggeneration) - Tuesday, 29 September 2009, 17:43 GMT
Last edited by Lukas Fleischer (lfleischer) - Tuesday, 27 May 2014, 18:47 GMT
Task Type Feature Request
Category Backend
Status Closed
Assigned To Lukas Fleischer (lfleischer)
Architecture All
Severity Medium
Priority Normal
Reported Version 1.5.6.4
Due in Version 3.0.0
Due Date Undecided
Percent Complete 100%
Votes 74
Private No

Details

With makepkg from pacman 3.3, packages can be split. It would be nice to have this feature exposed for AUR.

References (for those who somehow don't know):
http://www.archlinux.org/news/455/
http://wiki.archlinux.org/index.php/KDE_Packages
http://wiki.archlinux.org/index.php/DeveloperWiki:Splitting_KDE
This task depends upon
 FS#15043 - Need better parsing of PKGBUILDs 

Closed by  Lukas Fleischer (lfleischer)
Tuesday, 27 May 2014, 18:47 GMT
Reason for closing:  Implemented
Additional comments about closing:  Implemented in 3.0.0.
Comment by Xilon (Xilon) - Monday, 19 October 2009, 19:09 GMT
How are these packages shown? Do we have one page for the pkgbase and then list the split packages and their dependencies etc, or have a page for each package?
Comment by Harley Laue (losinggeneration) - Monday, 19 October 2009, 19:23 GMT
I like the one page listing the split packages and their dependencies myself. I think it'd be less confusing than having each have their own page (since they'd all be sharing the same PKGBUILD.) Perhaps have the common portions at the top, then show each of the split's details in a box below. Something like:

-----------
Base Package Details:
$basename
$url
...
etc
----------
Split Package Details:
$splitname1
$depends
...
etc
---------
$splitname2
$depends
...
etc
---------
[comments]

I don't know if the terms Base/Split should be used or not, but that's the general idea of how it could be done.
Comment by Heiko Baums (cyberpatrol) - Saturday, 02 January 2010, 11:18 GMT
I'd also like to see support for split packages in AUR. Or let say I need it.
I maintain kernel26-fbcondecor which is based on kernel26 from [core]/abs which is now a split package. It builds and installs fine with makepkg but I can't upload it to AUR.

I guess the problem is that AUR parses the PKGBUILD's variable $pkgname to generate the package name. In split packages $pkgname is not a variable but an array with the names of every resulting binary packages. The name of the package itself is stored in the variable $pkgbase.

So AUR should either parse $pkgname for normal packages or $pkgbase for split packges to generate the package name. It could e.g. look if there's a variable $pkgbase in the PKGBUILD or not or if $pkgname is a variable or an array to determine the type of the package.
Comment by Heiko Baums (cyberpatrol) - Saturday, 02 January 2010, 11:22 GMT
There's nothing more to do at AUR because makepkg -i installs every binary package which is built by a split package.
Comment by Nicky726 (Nicky726) - Monday, 11 January 2010, 17:57 GMT
I as a maintainer of kernel26-selinux, would also like to see this implemented, as I want my package to be as KISS as possible, that is the same as kernel26 in [core] only with necesary changes.
Comment by (MajorTom) - Wednesday, 27 January 2010, 15:50 GMT
I maintain kernel26-pae (32bit) and because of AUR's inability to accept split packages I always need to make significant modifications to PKGBUILD:
- move the url, pkgdesc, groups, backup, depends, optdepends, replaces and install variables from package_kernel26() function before the build() function;
- combine the contents of the package_kernel26-headers() with build();
- remove package_kernel26(), package_kernel26-headers() and package_kernel26-firmware() function definitions;
Comment by kujub (kujub) - Wednesday, 10 February 2010, 12:49 GMT
+1 for this
I'm the maintainer of fbsplash-themes-arch-banner and had to trick the AUR in various ways to get that split package uploaded. The package is split to give the users a chance to keep the initcpio a bit smaller.
Comment by Hervé (herve) - Wednesday, 17 February 2010, 16:52 GMT
I also need it for a documentation package that is split into HTML and PDF versions.
Comment by Diego Ferigo (dieghen89) - Wednesday, 19 May 2010, 22:57 GMT
I maintein the kernel-netbook in aur and it will be great have the split option also in aur...+1!
Comment by Vlad George (DonVla) - Thursday, 20 May 2010, 11:39 GMT
In the beginning I supported the idea of having PKGBUILDs for split packages in the AUR. But the more I think about the more I disagree with this.
First of all it creates a much bigger overhead for the user then one package PKGBUILDs. For example let's take a look at xapian-bindings (xapian-{python,ruby,php,etc}-bindings). For penguintv a user needs only the xapian-python-bindings. Now (s)he can only dl the PKGBUILD for this package, build it and install it. In case of split PKGBUILDs one has either to edit the PKGBUILD to get only the needed package or delete the not needed packages after building. In both cases not KISS anyway - or at least much unnecessary work.
Then the maintainer: On the first glimpse it seems to be less work to create multiple packages from one PKGBUILD, but then one needs to maintain much more packages, for which one cannot guarantee that they actually work, simply because they are not needed. Once again xapian: Some time ago I wrote a split PKGBUILD for xapian-bindings, which ended in almost ten packages from which I needed and used only one. For now I have only to take care of the python-bindings and that's it. In case of split PKGBUILDs I have to check each built package if it's ok and so on. Way too much work compared to single PKGBUILDs.
Official packages are provided as binary packages, so it is not important for the user how these were built. He/she can only dl the packages she/he needs and that's it. On the AUR the situation is different.
I think it's clearer and more straightforward for noobs and people who don't care about package technicalities (though experienced): one dependency - one PKGBUILD.
The same applies to other packages like docs or headers/firmware, etc.

PS: I sometimes don't understand what the need is for split kernel packages in the AUR. The idea of omitting the kernel26-headers is that all packages built upon these headers are provided as binaries, so they don't need to be rebuild. For the AUR kernels you mostly need the headers to build your own modules, etc. So not adding the headers to the package is rather a disadvantage.

PPS: On the other hand a clearly commented PKGBUILD (eg, kernel section, header section, firmware section) provides a similar functionality - if one wants to edit the PKGBUILD.

PPPS: unvote
Comment by Gavin Bisesi (Daenyth) - Thursday, 20 May 2010, 13:02 GMT
I think you're quite wrong, Vlad. For one, you can tell makepkg to build only the packages you care about. For two, it will make it simpler to maintain packages. Furthermore, why should we arbitrarily not support something in AUR that is in use everywhere else?
Comment by Heiko Baums (cyberpatrol) - Thursday, 20 May 2010, 13:26 GMT
Vlad, I agree with Daenyth. You're completely wrong. It doesn't create any overhead. First the maintainer still has to maintain only one single PKGBUILD and therefore only one single package in AUR as before. Not the PKGBUILD and the AUR package are split, but only the resulting binary package. For the user it also doesn't make many differences, on the contrary it gives the user more flexibility. By default makepkg builds and installs every resulting binary package, which is likely be done by the AUR scripts like yaourt and aurbuild. So the user gets the same as with a single package. But with a split package he has the flexibility to be able to uninstall single binary packages he doesn't need or want. And as Daenyth said, it makes maintaining packages a lot easier. See the various kernel packages. The PKGBUILDs wouldn't need to be reversed to single packages anymore as it has to be done now.
Comment by Vlad George (DonVla) - Thursday, 20 May 2010, 13:46 GMT
Yep. Daenyth, you might be right, but as I see it this makes things "complicated" (sure, it's not rocket science).
Though it's convenient for package builders, things get more cluttered for users when getting to AUR.
And as I said above, I don't think it makes AUR PKGBUILD maintaining easier.
IMO, ABS and AUR are two different pairs of shoes.

Btw, haven't found a makepkg option to build only certain packages from split PKGBUILDs.
Comment by Vlad George (DonVla) - Thursday, 20 May 2010, 13:54 GMT
Hi Heiko,

"First the maintainer still has to maintain only one single PKGBUILD and therefore only one single package in AUR as before."
That's not really true. Once a split PKGBUILD providing for example the xapian-python-bindigs is in AUR the single PKGBUILD is obsolete. And here it starts to get cluttered (eg, duplicates).
"By default makepkg builds and installs every resulting binary package," - sometimes this is _not_ intended!
Sure, flexibility to do it either this or that way is a reason which speaks for introducing split PKGBUILDs to AUR.

PS: Personally, I have no problem with split PKGBUILDs in the AUR. But I think it diverges from current "intuitive" use of the AUR.

PPS: "The PKGBUILDs wouldn't need to be reversed to single packages anymore as it has to be done now." Depends on how you see it: "less" work on side, "more" work on the other... Merging more package functions into one is not that hard.
Comment by Heiko Baums (cyberpatrol) - Thursday, 20 May 2010, 14:04 GMT
It makes AUR PKGBUILD maintaining easier. See the various kernel packages as I've written before. Now I have to revert every kernel package from the split package from [core] to a single package. This isn't hard to do but makes more work. And for the user it doesn't make anything more complicated. The user just has to get used to split packages and learn how they work, which is btw. also not that hard.

And split packages in general indeed make something simpler. See e.g. cdrdao, which is primarily a CLI tool, but has also a GUI named gcdmaster in its official source package included. Both are built and installed in one step with the same ./configure && make && make install procedure. It was just possible to disable the GUI by setting a ./configure switch. The official binary package of cdrdao in [extra] didn't install gcdmaster (had set this ./configure switch), because including gcdmaster would need GTK dependencies for a CLI tool. Most users only want the CLI tool and would have been forced to install GTK if gcdmaster would have been included. Now with split packages it is possible to add gcdmaster to the official binary package with the advantage that users who only want the CLI tool can only install cdrdao without the GTK dependencies and users who want the GUI, too, can additionally install gcdmaster with the GTK dependencies.
Comment by Heiko Baums (cyberpatrol) - Thursday, 20 May 2010, 14:38 GMT
"That's not really true. Once a split PKGBUILD providing for example the xapian-python-bindigs is in AUR the single PKGBUILD is obsolete. And here it starts to get cluttered (eg, duplicates)."

I don't know xapian, but I know the kernel packages. I'm the maintainer of kernel26-fbcondecor. Split packages don't mean that the AUR packages are split. Only the resulting binary packages are split. For kernel26-fbcondecor this means that there's still only one package kernel26-fbcondecor in AUR and not three packages. But this one AUR package results in three binary packages kernel26-fbcondecor, kernel26-fbcondecor-headers and kernel26-fbcondecor-firmware. That's the same as with the stock kernel, which also has only one PKGBUILD kernel26 which results in three binary packages. So there's no need for cluttered duplicates in AUR.

Btw., if AUR supports split packages it doesn't mean that the maintainers are forced to split their packages.

""By default makepkg builds and installs every resulting binary package," - sometimes this is _not_ intended!"

Then you have the possibility to uninstall the unneeded packages or tell makepkg to only build the needed packages as Daenyth explained.

"PS: Personally, I have no problem with split PKGBUILDs in the AUR. But I think it diverges from current "intuitive" use of the AUR."

The current "intuitive" use still remains. For the user it doesn't change anything except for a better flexibility. If a user wants to install kernel26-fbcondecor he now has to download and build the package kernel26-fbcondecor, which now results in one binary package kernel26-fbcondecor which includes everything. With split packages the user still downloads and builds only one package kernel26-fbcondecor but then gets three binary packages kernel26-fbcondecor, kernel26-fbcondecor-headers and kernel26-fbcondecor-firmware which also include everything by default.

And it's not confusing, because the official stock kernel is now also split into these three binary packages. The current situation is more confusing as the user needs to install different kernels by different methods and new users could easily be confused by not having the same packages for kernel26-fbcondecor as they get for kernel26.

If splitting the official kernel26 package itself makes sense is a different question. I don't like the splitting of this package much, but I also see the possible advantages. But my kernel26-fbcondecor is based on kernel26 and I'd like to keep kernel26-fbcondecor as similar to kernel26 as possible. That's primarily not a question of more or less work but of consistency.

"PPS: "The PKGBUILDs wouldn't need to be reversed to single packages anymore as it has to be done now." Depends on how you see it: "less" work on side, "more" work on the other... Merging more package functions into one is not that hard."

Usually this is correct, but not in this case. In case of split packages it's less work for the maintainer and not more work for the user. See the kernel example.

How dependencies are handled is another question which need to be implemented in AUR, ABS and/or pacman. But I don't see a big problem here.
Comment by Loui Chang (louipc) - Wednesday, 09 June 2010, 17:08 GMT
There really should be no debate on whether or not it should be included.
Even though I don't like this feature, it should be included because PKGBUILDs with
split packages are valid. The AUR should be able to accept any valid packages. The
biggest barrier is parsing these PKGBUILDs.

Bash is an executable format, it's not for parsing data out of. The results from
running a PKGBUILD can differ from system to system, but what we need is consistency.
We need the same results whether you're running i686, x86_64, or whatnot. Furthermore,
executing arbitrary scripts is not only a security risk but a serious security flaw.
We need an easily parsable format for metadata in source packages, maybe like what is
provided with binary packages (PKGINFO).

If you want current and advanced features like this I urge you to pressure upstream
(pacman/makepkg) to provide the needed formats to make this possible.
Comment by Vlad George (DonVla) - Wednesday, 09 June 2010, 17:25 GMT
@Loui: Yep, you're right. Quite senseless discussion. Sry I started it :D !
I read your ML proposition of adding a SRCINFO file to the src.tar.gz when running "makepkg --source".
It is a elegant and consistent way without much parsing. - Also very useful for third party applications.
Is this a planned feature or only a loose idea for now?
Comment by Tomas Mudrunka (harvie) - Thursday, 05 August 2010, 22:40 GMT
DonVla: IMHO it's very bad idea to add some redundant data to source packages. It can't be too hard to parse and if you can generate SRCINFO with some script, you can unzip PKGBUILD and run the SRCINFO script on server side.

louipc: And BTW there is already bash parser. We are parsing bash script so we definetely SHOULD use bash. You say the bash is security flaw? I say all we need to do is strip bash to minimal version, delete all calls that are executing another binary or creating new process and we have secure bash parser in form of patch to bash or some similar shell (busybox?).

I'll take a look how hard it could be to make some shell secure for AUR purposes and i'll let you know if i succeeded.
Comment by Gavin Bisesi (Daenyth) - Thursday, 05 August 2010, 22:49 GMT
Thomas, you're trying to be helpful, but trust me when I say you have no clue how hard it is to parse bash. What you suggest will not work.
Comment by Tomas Mudrunka (harvie) - Thursday, 05 August 2010, 23:37 GMT
Daenyth: I don't want to parse bash. I just want to source the PKGBUILD (in crippled version of bash) and read the values - just like makepkg does. So i will use bash internal parser - not my own parser.

And i've just remembered that there is restricted mode in bash, which can be great start (if it's not sufficient right now).
Comment by Gavin Bisesi (Daenyth) - Thursday, 05 August 2010, 23:42 GMT
The problem is that to parse bash you need to run it. You can't blacklist everything bad, and you can't whitelist everything needed. It doesn't work
Comment by Loui Chang (louipc) - Friday, 06 August 2010, 00:15 GMT
It might be possible, but I think it would just be easier to let makepkg take care of the parsing.

Technically speaking binary packages minus the PKGINFO could contain most the 'metadata'.
You could write some sophisticated namcap like tool to extract out everything (arch, license,
pkgname, pkgver, etc) from the executables, and included docs. In the end it's much easier to
output an easily parsable PKGINFO.

In the case of source packages the PKGBUILD is the executable. Do we really want to have a
complicated parser to extract interesting data from the executable, or do we just want that data
included by the packager in an easily parsable format?

Anyways this discussion is something that should take place on the pacman mailing list, or pacman
bug tracker. A sophisticated PKGBUILD parser will not be implemented in the AUR as long as I am
the maintainer. Cheers.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 00:33 GMT
Daenyth: you can just comment any line in bash source which can execute something. this can't be so hard. maybe some #defines overriding syscalls and stdio.h functions can do the thing.

louipc: PLEASE! i don't want to see archlinux package system bloated same way as the debian's one. Behave like a real men and make the parser.

In 30 minutes i've created simple parser which can be modified to be more secure and to be able to run in patched bash (currently i am not even able to run it in restricted bash, but it can be done). Look at the code: http://aur.pastebin.com/mhc43SVu

And output?:

<?php
$packages=array(
'ldns-lib',
'ldns-tools',
'ldns-drill',
);
$pkgdescs['ldns-lib']='libldns DNSSEC enabled DNS library';
$pkgdescs['ldns-lib']='Some utilities using libldns';
$pkgdescs['ldns-lib']='Drill DNSSEC utility from LDNS package (ala enabled Bind's dig)';


This is the way to go.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 00:37 GMT
Sorry, little fix: http://aur.pastebin.com/B6x1tfV1

<?php
$packages=array(
'ldns-lib',
'ldns-tools',
'ldns-drill',
);
$pkgdescs['ldns-lib']='libldns DNSSEC enabled DNS library';
$pkgdescs['ldns-tools']='Some utilities using libldns';
$pkgdescs['ldns-drill']='Drill DNSSEC utility from LDNS package (ala enabled Bind's dig)';
Comment by Jan Alexander Steffens (heftig) - Friday, 06 August 2010, 00:39 GMT
sourcing the PKGBUILD is an absolute no-no due to the possibility of injecting code.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 00:48 GMT
heftig: have you read all the comments? i am going to modify bash, so sourcing will be secure. calm down.
Comment by Jan Alexander Steffens (heftig) - Friday, 06 August 2010, 00:50 GMT
I still think it's a bad idea. No need to build a complex parser (with modified bash -_-) when makepkg can simply write an easily-parsable .SRCINFO into the source package.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 01:10 GMT
heftig: and two years later we will add .SRCINFOv2 because of another lazy developer who cannot parse the original .SRCINFO because he needs to treat packages in another way. it's absolutely stupid idea. i think we should rather modify PKGBUILD definition (and modify all PKGBUILDs we currently have) than storing redundant data.

ArchWay you remember? (KEEP IT SIMPLE STUPID!) Actually i thing that we should get rid of .PKGINFO too. PKGBUILD should be in some easily parsable (but bash-compatible) format, so we can just COPY-AND-PASTE exact PKGBUILD section to .PKGINFO without any expensive processing (well we should trim whitespaces, validate against regexes, etc...). There are many ways to write code that is easily executable by several programming languages (eg.: code which has meaning in C,PHP and BASH: http://en.wikipedia.org/wiki/Polyglot_(computing) ). This sort of PKGBUILD format can be invented with little effort.

and BTW modified bash is not complex parser. all we need is to add some header files. i will look at it few days later when i will get back from holydays.

and face it: bash is the only good parser for parsing bash code.
Comment by Jan Alexander Steffens (heftig) - Friday, 06 August 2010, 01:34 GMT
Bash is the only good parser for bash code. So we leave parsing the PKGBUILD to the bash-implemented makepkg and use it to create a simple SRCINFO that won't need bash to parse. Safe. Simple. Arch Way.

Changing the PKGBUILD format won't happen, either. Too much work.
Comment by Loui Chang (louipc) - Friday, 06 August 2010, 04:53 GMT
There's a good amount of discussion here that really should be directed to
pacman-dev, or the pacman bug tracker, or elsewhere. Please see that it does
find its way there.

If you want to work on a bash parser or even on a new implementation of the
AUR, that's awesome and I encourage you to pursue what you belive in. Just
please don't be too pushy.

Thank you.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 05:10 GMT
louipc: :-) BTW i've already seen two AUR reimplementations: http://github.com/sebnow/aur2 (can't remember where the second one was)
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 05:12 GMT
and BTW it seems it uses BASH parser too: http://github.com/sebnow/aur2/blob/master/archlinux/aur/Package/parsepkgbuild.sh
(but unfortunately without executing PKGBUILD in sandbox, so it's bit incomplete)
Comment by Heiko Baums (cyberpatrol) - Friday, 06 August 2010, 05:26 GMT
Sorry, one question. What has this bug report got to do with pacman or bash?

It's about supporting split packages in AUR. In other words, pacman and makepkg can handle split packages, but split packages can't be uploaded to AUR, because AUR rejects those packages. So what sense does it make to discuss a bash parser or whatever on the pacman-dev mailing list or the pacman bug tracker?
It's a missing feature of AUR, not pacman or makepkg, so it has to be discussed here or on the aur-general mailing list.

Not pacman needs to be fix and to be able to parse the PKGBUILDs of split packages - it can already do this - AUR needs to be fixed and to be able to parse and accept PKGBUILDs of split packages.

And why should bash be patched just to be able to parse PKGBUILDs, .SRCINFO files or whatever?
Comment by Vlad George (DonVla) - Friday, 06 August 2010, 09:20 GMT
@Thomas:
1. I see no debian like overbloat since nothing changes for the package and packager. Providing some useful metainformation doesn't "bloat" anything. It is simply a text file added to other textfiles.
On the other hand to create a bash parser for each application which might need some information out of a PKGBUILD is senseless duplicated work.
2. It's totally senseless to change/patch/whatever bash just to get some infos out of a textfile (WTF????).
3. Sourcing a PKGBUILD after making the source tarball is quite dangerous. And there must be a regex parser anyway. Think of the vars inside the package_*() functions. There is no way to get this information by sourcing only. You have to _execute_ them to get the local vars. This has lots of side effects and simply not needed for some purposes.

@Heiko:
I disagree.
We should think about how pacman/makepkg provides information. This information _is_ a big part of the package and thus of the package manager. PKGBUILDs are the domain of makepkg. Imo, because makepkg can create source tarballs, it also should provide some further information about this source package as it does for binary package - at least some basics. This information file (eg, .SRCINFO) is not intended as a replacement for the buildfile.

However, I think it is more efficient and also sane to create a SRCINFO file during the build process of the source tarball.
First of all no duplicate work.
Second, it _is_ safer.

BTW, here are some thoughts of mine about the SRCINFO file:
http://mailman.archlinux.org/pipermail/pacman-dev/2010-August/011461.html

Vlad
Comment by Vlad George (DonVla) - Friday, 06 August 2010, 09:38 GMT
@Thomas
And about the PKGBUILD:
It is not so easy to change the format of a PKGBUILD. There are thousands of them in ABS/AUR. All of them have to be replaced. Too much (senseless) work.
PKGBUILDs are written in bash. So, theoretically, they should support all of bash's features/syntax. And they should be easily understandable. Such restrictions like "only this is allowed" are anti-productive.
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 10:54 GMT
cyberpatrol: Exactly! that's what i say. it's problem of AUR and only AUR should be fixed - anything else. bit i think you've misunderstood the patched bash concept. This patched bash will be only on server running AUR to help AUR parse the PKGBUILDs without using any additional and redundant bloatware like .SRCINFO. PKGBUILDs are in BASH, so only BASH can parse them well, but it's insecure to execute user contributed scripts, so we need to patch bash to be enough secure to execute untrusted scripts.

DonVla:
ad> Think of the vars inside the package_*() functions. There is no way to get this information by sourcing only.
did you at least briefly looked at my code???!
get_function() {
export -f "$1" || exit 1;
bash -c 'env' | tr '\n' '\0' | sed 's/.*'"$1"'=() {\(.*\)}.*/\1/g' | tr '\0' '\n'
}

ad> debian overbloat
sorry, but storing the same information on two different places (especialy in the same tarball) IS OVERBLOAT.

changing the PKGBUILD format was extreme example, but it's still better than adding .SRCINFO (which is not needed at all, so there's no point for this).
ad> "only this is allowed"
well i guess you will not need to execute "rm -rf /*" (or any other command) to get pkgname, pkgdesc and depends and if you will, you should change the PKGBUILD. if pkgname (or something similar) is variable, it can be probably handled by BASH's internal processing features.



My conclusion: I am strightly against patching makepkg or pacman to support some .SRCINFO bloats. Again: It's AUR problem and AUR should fix it.

This looks to me like when Canonical thought that kernel release engineering will get changed to comply with Ubuntu releases. Linus said them to go away (actually something more rude i guess).
Comment by Vlad George (DonVla) - Friday, 06 August 2010, 11:45 GMT
Heh? It seems you don't get the point.
It is not (only) a parsing problem. It is not only about getting, but also about providing information.
Once again, I see no overbloat in making a strict distinction between build and info file. As Louipc said, you can omit the PKGINFO file and extract infos out the bins. But this is too much overhead and senseless. Same applies imo to the src.tar.gz packs.
And what is this:
"PKGBUILDs are in BASH, so only BASH can parse them well, but it's insecure to execute user contributed scripts, so we need to patch bash to be enough secure to execute untrusted scripts." ???
Don't get me wrong, but this is total ... <silence - no offense>. I totally see no need in patching bash.
However, if you got some _real_ and coherent suggestions how to do this, then mail them to the aur-general list.
In your code you're doing stuff which has already been done dozens of times. Take some time and have a look how third party apps get infos out of PKGBUILDs.
Comment by Vlad George (DonVla) - Friday, 06 August 2010, 11:46 GMT
One last remark - missed this at first reading:
"I am strightly against patching makepkg or pacman to support some .SRCINFO bloats."
But you want to patch bash then???
Comment by Tomas Mudrunka (harvie) - Friday, 06 August 2010, 12:49 GMT
> But you want to patch bash then???
this patched bash will be single binary installed only on single AUR machine
$ du -hs /bin/bash
592K /bin/bash

it will not be a patch to bash package itself. let's say it will be package called bash-parser. it will no longer be bash, it will be just part of AUR used to parse PKGBUILDs. simple utility.

.SRCINFOs will be lots of files with redundant informations stored all along the AUR - one per package and in every src package.

If we can (and yes - we can) parse the existing PKGBUILDs directly in AUR without converting them to some new .SRCINFO file WE SHOULD DO IT. Why should makepkg or pacman care that AUR is not able to parse PKGBUILDs? There can be secure PKGBUILD parsing tool or library, but makepkg should stay unafected.
Comment by Loui Chang (louipc) - Friday, 06 August 2010, 22:55 GMT
I'm deleting any more posts that mention SRCINFO, PKGINFO, changing the
PKGBUILD definition or a customised bash parser.
Open a new ticket if you want to discuss that.
Comment by Tomas Mudrunka (harvie) - Wednesday, 11 August 2010, 20:15 GMT
louipc: No need for opening new ticket:  FS#15043  - Need better parsing of PKGBUILDs
Comment by Heiko Baums (cyberpatrol) - Wednesday, 11 August 2010, 21:33 GMT
But this feature request is not a duplicate of  FS#15043 . Both are somehow related but  FS#15043  is not about supporting split packages like this feature request.
Comment by Tomas Mudrunka (harvie) - Monday, 01 November 2010, 15:05 GMT
BTW is there possibility to have some simple workaround, so current AUR will be able to accept split-packages? I think it's more important to be able to upload the split-pkgs to AUR than showing the proper informations. We can just take the first package from split-pkg and ignore the others. So users will be able to share their split-pkgs until this get properly fixed...

Maybe we can just comment out the conditions that are preventing invalid PKGBUILDs from being uploaded or disable them only in cases where we'll found that pkgname is array... (simple regex)
Comment by Diego Ferigo (dieghen89) - Monday, 01 November 2010, 16:50 GMT
I have found a workaround for the splitted pkgbuilds in aur. You can find an example in kernel-netbook PKGBUILD:

http://aur.archlinux.org/packages/kernel-netbook/kernel-netbook/PKGBUILD

With this method also with yaourt, the only thing that one user has to do to get a splitted package is set SPLITTED=y....
Comment by Heiko Baums (cyberpatrol) - Monday, 01 November 2010, 17:24 GMT
@Diego: This is not the way to go. Don't create two packages/alternatives in one PKGBUILD between which the user has to choose by manually editing the PKGBUILD and commenting or uncommenting parts of the PKGBUILD. KISS! Keep it simple, stupid! Either build a single or a split package, but not both in one PKGBUILD. Such an "alternative" split package doesn't make sense in AUR anyway. Btw., it is "split" not "splitted".

@Thomas: No, there's no workaround. There already have been made many approaches, but none of them worked.
Comment by Tomas Mudrunka (harvie) - Monday, 01 November 2010, 22:11 GMT
Diego: well i believe we can find some way to fool-out AUR parser using advanced bash constructs so it will think that package is not splitpkg while it is... i will try to find something :-)

Heiko: omg. even GIT or anonymous FTP would be better...
Comment by Tomas Mudrunka (harvie) - Monday, 01 November 2010, 22:27 GMT
WORKAROUND! Splitpkg in AUR?
Here you are: http://aur.archlinux.org/packages/splittest/splittest/PKGBUILD

Do not even try to fix it in AUR parser. Since now it's not a bug. It's a feature :-)
Comment by Heiko Baums (cyberpatrol) - Monday, 01 November 2010, 23:49 GMT
@Thomas: What has GIT and FTP to do with AUR and split packages?

Well your splittest hasn't been tried, yet. Anything else has failed. And it was not only me who tried to fool AUR.
Nevertheless it's a workaround. Split packages should be supported by AUR without such a hack in the PKGBUILD.
Comment by Tomas Mudrunka (harvie) - Tuesday, 02 November 2010, 00:12 GMT
Heiko: You can upload splitpackages to 10 years old FTP server but you can't upload them to AUR :-)

> Well your splittest hasn't been tried, yet.
What do you mean? i have successfully uploaded it to AUR and i have succesfully installed both packages from it using yaourt. If such packages will become popular and moved to [community] then we can remove the hack. That hack can be simply added (or removed) to packages.

I think that development of ArchLinux is WAY more important than respecting some AUR parser. I would rather replace AUR by GIT repository than preventing people from improving ArchLinux itself. AUR is just tool and i want to make packages not use AUR. I think that AUR would be more usefull if we have removed the parser checks and let people upload whatewer they want (no matter if we manage to parse it or not) until it will get fixed...
Comment by John (graysky) - Sunday, 07 November 2010, 13:39 GMT
Actually you CAN in fact update a split PKGBUILD to the AUR. Have a look at kernel26-ck which I based on your brilliant implementation of a split package for apparmor :p The secret seems to be the true && pkgname=('foo' 'bar') line. Also note that you have to relocate the pkgdesc to the bottom of the PKGBUILD or else it doesn't get read properly.
Comment by Tomas Mudrunka (harvie) - Sunday, 07 November 2010, 21:44 GMT
John: RTFF(lyspray) https://bugs.archlinux.org/task/16394#comment68053
Now we have to implement this without workarounds.
Comment by Heiko Baums (cyberpatrol) - Sunday, 07 November 2010, 22:44 GMT
Thomas:
"Now we have to implement this without workarounds."

So the AUR parser has to be fixed. And, btw., of course it is important to develop Arch Linux. But that doesn't mean, that AUR must not be improved. AUR belongs to Arch Linux as anything else.
Comment by Tomas Mudrunka (harvie) - Monday, 08 November 2010, 00:22 GMT
Heiko:
"AUR belongs to Arch Linux as anything else"

Well... i don't think so. I'd prefer ArchLinux without AUR to ArchLinux without well-maintained user-submitted packages...
Comment by Heiko Baums (cyberpatrol) - Monday, 08 November 2010, 01:26 GMT
Thomas: This is really off-topic for this bug report, but can it be, that Arch Linux is not the right distro for you? AUR is one of the main advantages over other distributions. I'd also prefer if there were more packages in the official repos, but AUR makes it possible and easy to build packages which are not in the repos and make them accessible by other users. And there are packages which won't never be added to the official repos for comprehensible reasons.

So AUR definitely IS a part of Arch Linux regardless of your opinion about that. You don't need to use it, if you don't want to.
Comment by Tomas Mudrunka (harvie) - Monday, 08 November 2010, 01:51 GMT
Heiko: I think that you have misunderstood me. Packages in AUR (and ability to submit them) are way more important than AUR (the webapp) itself.
Comment by Heiko Baums (cyberpatrol) - Monday, 08 November 2010, 01:55 GMT
Thomas: Well, then I don't know why you have a problem with improving AUR, because a fix is always preferable to a workaround.
Comment by Tomas Mudrunka (harvie) - Monday, 08 November 2010, 03:15 GMT
Who sais that i have problem? :-D I said "Now we have to implement this without workarounds."
It's time to stop spamming :-) i am glad that we have both the same opinion :-)
Comment by Yannick LM (yannick_lm) - Saturday, 05 May 2012, 17:46 GMT
Just a quick thought here.

Right now when a user tries to submit a split package, he faces the strange message:
'Invalid name: only lowercase letters are allowed'

It can take a while before he figures what's going on.

The following patch at least tries to detect the situation and sends a somewhat better
error message:
'Uploading splitted packages to AUR is not allowed'

(at least until #15043 is fixed)

BTW, I really like the idea of having 'makepkg --source' generating machine-readable metada,
I'll look into it.
Comment by Heiko Baums (cyberpatrol) - Wednesday, 22 August 2012, 11:32 GMT
This is simply not true except for the makepkg -i part. Split packages can only be uploaded to AUR with a very dirty workaround. AUR can't find any dependencies to subpackages of a split package. So AUR does not support split packages and there's a lot to be done.
Comment by (Det) - Wednesday, 26 June 2013, 11:31 GMT
Is "true && pkgname=('blah' 'blahblah')" a "very dirty" workaround?
Comment by Harley Laue (losinggeneration) - Wednesday, 26 June 2013, 14:21 GMT
Yes, but it's been being used for a couple years. IMO, it's not ideal, but it does (still) work.
Comment by (Det) - Wednesday, 26 June 2013, 22:18 GMT
A "dirty workaround" is something like 10-20 lines that has to be customized separately for each package.

The "true &&" "hack" doesn't affect neither makepkg or even the info fields in the package home page.

 FS#15043  (which blocks this from being implemented) was closed on May 3rd anyway:

"Reason for closing: Won't implement
Additional comments about closing: Please use .AURINFO if the PKGBUILD parser doesn't work for you."
Comment by Harley Laue (losinggeneration) - Wednesday, 26 June 2013, 22:46 GMT
That's your opinion (about whether or not you want to consider it a dirty work around.) In any case, it /is/ a workaround and that's what really matters here.

Also, .AURINFO doesn't help here either:
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n84
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n263
https://projects.archlinux.org/aur.git/tree/web/html/pkgsubmit.php#n285
Basically it just overwrites what was already found in the PKGBUILD and errors on split packages.
Comment by (Det) - Wednesday, 26 June 2013, 23:13 GMT
Overrides*. The overwriting's a wrong term.

If it's too hard to support something and we have an extremely simple and efficient workaround, why are you mocking the workaround?

I'm not really sure what you're asking from the AUR anyway. To show all the package names in the relevant 'home page' and/or to be able to build/install/update just part of a package?

Or just to ignore the error when uploading your tarballs without the "dirty" hack?
Comment by Harley Laue (losinggeneration) - Wednesday, 26 June 2013, 23:25 GMT
First, I was not wrong. My description was about the linked code. Variables are overwritten, classes & functions are overridden. Also, such non-sense (basically everything you've submitted to this bug report, as well as this entire response) doesn't belong in the bug report. This isn't a forum, please don't treat it as such.

I'm not mocking it, but pointing out that it's a workaround and not a proper fix. There /is/ a difference.

For the rest, read all previous responses. They pretty well cover the rest of your questions.
Comment by (Det) - Thursday, 27 June 2013, 09:13 GMT
Which is where the irony kicks in, because you just can't seem to put it together that it can't be done.

I've been trying to explain to you the entire time why the "workaround" is the only thing you're ever gonna have, yet you still keep on insisting on complaining how "dirty" it is and that Lukas should just come up with a better parser. He _won't_ because it's too hard (which ironicly in turn is too hard for you).

And had you actually read the discussion yourself you'd know all this.
Comment by Harley Laue (losinggeneration) - Thursday, 27 June 2013, 13:57 GMT
Redacted. The workaround is has already been well covered. It's up to Lukas if he feels this is going to be worthwhile to implement to make such workarounds required. (/me will stop feeding the troll.)
Comment by (Det) - Thursday, 27 June 2013, 17:54 GMT
Mm, he has already made that decision. I've already told you. See:  FS#15043 .

Also I wonder why does it always magically make the other person troll when you're losing the argument.
Comment by Harley Laue (losinggeneration) - Thursday, 27 June 2013, 18:22 GMT
Except for the fact that  FS#15043  isn't really relevant for this. It was "fixed" by other means (2 was fixed outright, 1&4 can be handled by .AURINFO, and 3 was basically never implemented.) .AURINFO doesn't help here and the issue isn't "fixed" but does have a known workaround to the problem at hand.
Comment by Dave Reisner (falconindy) - Thursday, 27 June 2013, 18:23 GMT
This bug has reached well past the end of it's useful lifespan. I'm closing this, in part, because clearly some people can't figure out how to communicate nicely with other humans.

Det: please seek professional help.
Comment by Harley Laue (losinggeneration) - Thursday, 27 June 2013, 18:32 GMT
Fair enough, it's basically (at this point) a long standing known issue with the dev team. I have no doubt it'll likely be handled in some form or another (though I'd wager, based on how long the issue has been open, it'll be at a time when the workaround no longer works :P.) Who would have thought such a relatively harmless feature request like this would cause so much trouble. :(
Comment by (Det) - Thursday, 27 June 2013, 18:38 GMT
I doubt the "true &&" workaround will ever just 'stop' working unless the actual PKGBUILD language is changed to something else.

Also the reason why I mentioned  FS#15043  was that the general idea is about having a better parser, which in turn would allow split packages, which in turn, after closing, also blocks this from being implemented (see the "This task depends upon" note).
Comment by Harley Laue (losinggeneration) - Thursday, 27 June 2013, 19:08 GMT
The parser is only a single piece to this issue, which you apparently don't understand. The parser already sees that the package is split and (without the workaround) rejects it. So it'd be relatively trivial to detect that and parse the pkgname array.

The real problem here stems from what to do with it. For instance, how should the package(s) be displayed, searched for, etc. A final solution to these questions were never agreed on, thus this issue lived on.  FS#15043  was only /really/ relevant here if the parser was reworked to better support more/all of the features BASH supports (which was eventually decided against as it can depend on things other than just BASH.) The real meat of this issue though wasn't about the parser at all, which is why I keep telling you that related issue isn't really all that relevant here.

That's it. I'm done talking to you on this issue as that should be more than enough to quite anyone who's not trolling *hint hint*
Comment by (Det) - Thursday, 27 June 2013, 20:18 GMT
It's not really the first time you've been 'done' talking to me :).

But the thing is that the discussion here for the last 2-3 years has been about _parsers_ and only the first two posts about what to do, if the problem didn't exist (since it wasn't brought up yet). You don't seem to really understand the connection that the parser/AURINFO issue _is_ the root of this problem - from there, once that's fixed, you could decide how to provide that info to the user in the web interface.

And just a FYI, this is even categorized as 'PKGBUILD Parser'.
Comment by Lukas Fleischer (lfleischer) - Saturday, 05 April 2014, 12:12 GMT
  • Field changed: Category (PKGBUILD Parser → Backend)
  • Field changed: Due in Version (Undecided → 3.0.0)
The next release (3.0.0) will add support for split packages. A patch series has just been submitted to aur-dev [1].

[1] https://mailman.archlinux.org/pipermail/aur-dev/2014-April/002676.html
Comment by (Det) - Sunday, 06 April 2014, 14:57 GMT
You are some busy boys.

Just awesome work.

Loading...