Arch Linux

2.) i think that AUR website have problems with parsing PKGBUILD arrays that are including comments. please take look at this (in most cases there is no real need for using comments, but sometimes it's usefull to make small comment)

2.) I think that has been fixed by now. When I upload the "broken" package to a test-AUR on my computer, it is parsed correctly. Do you still have problem packages? To check if it works well now please upload the package again so it will be parsed once more.

2.) yea, AUR works great with comments in Arrays now.
But there are still the other 3 subissues...

1) Yeah, probably it would be nice, but you can already read those things just by clicking on the PKGBUILD link, and that does not require to do any fancy parsing on AUR's part. Having said that, the optdepends part would indeed be nice. There's some issue that the optdepeneds don't seem to have a standard format, but with a bit of heuristics that can be handled. (E.g. extra/wine only list the dependencies, extra/transmission-gtk has comments for each of them in the form of "optional: the function it would add". I can't remember, but I think I've seen packages where all the optdepends are in one line, not in an array.) But as I said, this is not too bad, I started to check out how to handle it. Other fields, though, I don't really see the point at the moment.

2) I'm glad it's fine now.

3) Started to check that one out too, hopefully will have some time over the weekend.

4) I don't think there should be any AUR specific field in PKGBUILDs. If the pacman maintainers implement it then AUR should definitely support it, but not before that. Not sure how others feel about this....

WRT optdepends, I'm pretty sure the 'official' way is "optional: enables foo support" in an array. Anything else should probably be regarded as an error

Daenyth: yes, but there can be links to those optdepend packages
<a href="http://aur.archlinux.org/packages.php?O=0&K=foo&do_Search=Go">foo</a>: enables foo support

today you can't see optdepends in aur at all...

Hi.

3) Why not link to the package directly insted of going to the search first. A little query on the database, and there you go.

I can go work on the issue, but is there already somebody working on it? (I'm new to this...just subscribe to the mailing list)

oh....nevermind, I just noticed the task status list on the left of the description...as I said, I'm a n00b.

Please take a look at  FS#16394 

I think that BASH itself is the best for parsing BASH scripts (such as PKGBUILDs), so we should use BASH code base to parse PKGBUILDs. Problem is that BASH itself is insecure, because it needs to execute the code and we cannot execute untrusted PKGBUILDs on AUR server. I've suggested to use some scripts like this:
http://aur.pastebin.com/B6x1tfV1
with a little bit modified (crippled & secured) BASH (which will not execute anything and will not access the filesystem - i guess this can be reached by some kind of small "lobotomy" commited on original BASH source).

Maybe something also going on at those projects:
- http://github.com/Daenyth/Bashful
- http://github.com/sebnow/aur2

As I said in the other bug report, it will not work. I've stated this enough times that I'm not going to bother explaining it yet again. Bashful also is not correct for this, as you'd notice if you read the README file (specifically the "It will not [...] be correct (for potentially large numbers of cases)"). There's also a grand total of 0sloc currently. Consider it vaporware.

yes i completely agree with Daenyth; modifying bash/parsing bash is nonsense for several, already touched reasons.

a small, metadata text file is not "bloat"... it's... a small, highly compressible _text file_. in the AUR implementation i am working on [https://bbs.archlinux.org/viewtopic.php?id=99839], SRCINFO will be _required_, as the AUR will run as pure python, as a desktop app, or translated to javascript as a webapp. i am also looking to the future, a distibuted p2p version of AUR, so it has to work on _every_ installed machine. i need a way to get the information i need, in python; text files are a common way to perform this. i'm not even going to attempt trying to parse a pkgbuild.

if anything, SRCINFO could simply be in a common format with high language support such as YAML/etc., this would allow for easy mapping to language constructs.

in my opinion, the entire concept of "parsing bash" is an absolute non-starter, and a complete waste of time; you don't parse code (bash) unless you're a compiler. an easy meta file is the way to go, that can be read by ANY language; this is how we do things in the development world.

that said, i really like the idea of wiki integration; integrate as much as possible.

C Anthony

meta data generated by makepkg --source command would be nice. website can't parse the PKGBUILD and probably it shouldn't. instead it can use a meta-data generated by makepkg(i.e. pre-parsed)

BTW is there possibility to have some simple workaround, so current AUR will be able to accept split-packages? I think it's more important to be able to upload the split-pkgs to AUR than showing the proper informations. We can just take the first package from split-pkg and ignore the others. So users will be able to share their split-pkgs until this get properly fixed...

I don't believe we should try to parse bash PKGBUILDs anymore.
Imagine if pacman had to parse them. Hah. There's a reason packages contain easily parsable metadata.

Let's think of the AUR as a repo, like a pacman repo but for source.
We have done pretty well so far without good metadata, but source packages should no longer be considered second-class.

We should support the idea of including parsable metadata in source packages just as it is in binary
packages. So ultimately I think this is really a makepkg issue. Anyone who is interested in this
problem should open a ticket in the pacman bug tracker and contact their mailing list pacman-dev@archlinux.org

louipc: I still think that if we can't parse PKGBUILD then we should change it's syntax and create PKGBUILDv2 or something like that what will be more parseable (while compatible backwards). I believe that there is way to make new PKGBUILD syntax that will be compatible with bash but easy to parse. makepkg can just force maintainers to keep their PKGBUILDs parseable (eg.: not using advanced bash features/expansions that are not guaranteed to be parseable). We can allow PKGBUILDs to use only limited set of features in the section that we will need to parse.

I have to say that i miss the times when all you needed to upload new package to AUR was single PKGBUILD. it was really KISS architecture and i don't think that we need something more complex for many packages. If we are not able to parse PKGBUILDs then there is problem with parser or with PKGBUILD syntax.

Yeah I do agree that something probably needs to be done about the
PKGBUILD spec to make it more friendly for maintaining a database
of source packages. As far as I understand, some of the more
advanced features of PKGBUILDs make it difficult or impossible
to get metadata out of them without building the actual binary
package - even when parsing with bash. The PKGBUILD was obviously
not designed with the source repo in mind. We're kind of trying
to squeeze a square box into a round hole here.

Uploading plain PKGBUILDs was really just for testing purposes
introduced sometime in the middle of the AUR's life. Don't miss
it so much. The AUR was meant to be a repository of tarballed
build scripts.

I would not say that it's KISS to upload packages in inconsistent
formats. Now we can have a little more consistency. We can still
do better there though.

There's obviously a problem with the parser. It's an incomplete
bash parser written in PHP. PKGBUILDs are fully bash scripts.
Therein the problem lies. We can't just use bash itself because
that opens a security hole. Maybe it's possible with a complicated
server setup and a fair amount of resources to do it, but I want to
keep the implementation of the AUR simple. So while it's KISS for
makepkg to allow PKGBUILDs to be bash scripts and run with it, it
makes life hard (not KISS) if you want to maintain a database of
source packages, like the AUR. So lets just throw that buzz word
in the trash.

Anyways, this is not exactly an AUR issue any more in my eyes.
I would like to help solve it though because it would make the
AUR that much better. Unfortunately I am short on free time.

yeah, if AUR is to become professional, it needs a reliable way to "exhume" these informations... strangling full-blown bash scripts is not a solution.

i recently realized a large piece of what i want out of a "state manager"...

i think we should encourage a move in pacman to use a proven modeling language like puppet:

http://docs.puppetlabs.com/guides/language_tutorial.html

such a move would retain many expressive qualities, and allow an AST object to manipulate freely in the architecture. several things like git/svn/fetch/move/copy/link/etc. are identical between packages...

the format could include range definitions, offer a means to fetch exact numbers on demand (version HEAD for example).

the point is that bash is only going further into nowhere; i want to make the concept of AUR and load/peer sharing fundamental to the the Arch experience. we need to explore richer core data structures, and use high performance persistent storage backends.

C Anthony

I know Loui is against parsing PKGBUILDs, but I thought I'd mention this anyway. I started a parser quite a while ago for exactly this purpose - pkgparse[1]. I don't think I'll continue to develop it as the bash grammar is just too complex and there will be little benefit from actually parsing it like this.

I agree that a better format is the way to go.

[1] https://github.com/sebnow/pkgparse

Well, the problem is that even if we had a full bash parser, it would still fail to do command substituions (like "pkgname=`echo foo | sed 's/foo/bar/'`") properly. To make that work as well, we would have to either implement functionality of every binary in coreutils, sed, awk, ... (basically every userland package from the "base" group) or execute the actual commands, which we already rejected in the discussion of just sourcing PKGBUILDs. Otherwise it wouldn't be a proper solution and if we implement a solution, which again doesn't cover all cases, we'll be at the same point as now. Furthermore, as a full bash parser including functionality of all "base" packages is definitely not KISS, this feature request is likely to never be implemented at all.

There could be guidelines against doing that sort of thing (that particular example could be achieved via pure bash). What I was aiming for was effectively a "secure bash interpreter". The interpreter could execute commands and such, but it could be limited in what was allowed (`sed` is fine, but `rm` is not). These are edge cases anyway so I didn't think too much about it. I think it should be possible to do the same thing in bash restricted mode anyway.

I'm not exactly sure what bash restricted mode does to be honest,.
Callan and Simo investigated it at some point and they
discovered that it didn't actually restrict anything. They could
still run arbitrary commands.

Here's a reference about restricted mode.
http://www.gnu.org/software/bash/manual/html_node/The-Restricted-Shell.html

It was the fact that you can still write malicious shell scripts (something that could bring down the server) in restricted mode, but it does restrict what commands you can run (not sure if there's a better way but you can change PATH to a specific dir with the allowed programs). The point of having a customer interpreter was that we could control exactly what kind of behaviour the interpreter has, and have some limitations (don't allow infinite loops for instance). Like I said, it's too much effort for little gain.

Having a proper plain-text metadata format (such as PKGINFO, or JSON/YAML) would be infinitely better. The issue with that is that either makepkg needs to be modified to support this format, or the file needs to be generated from the PKGBUILD, which is an extra step.

Xilon: Even packages in the official repos use these sort of things and sometimes they're inevitable. There's no way to properly build a perfectly secure environment without implementing most of the functionality of the tools we allow in restricted mode, too. Just some examples to illustrate the problematic nature of allowing a subset of typical commands used in PKGBUILDs (use them at your own risk!):

1. awk(1): Can be used to execute arbitrary commands, e.g. `awk 'BEGIN { system("echo Hello world!") }'`.
2. sed(1): Can be used to overwrite arbitrary files, e.g. `echo '# eval scrapt' | sed 's/a/i/g; 1w /some/other/foobar/script/that/will/be/executed/sometime/later'`.
3. test(1) aka `[`: Can be used to check if a file exists, e.g. `[ -e /some/path/to/a/file/owned/by/some/package ]`.
4. cat(1)/head(1)/tail(1): Can be used to read arbitrary files, e.g. "pkgname=`cat /etc/passwd`", "url=`head -2 /etc/passwd | tail -1`".

Those are just some very simple samples. I can think of much more complex ones that couldn't be detected without implementing full parsers for all commands we allow in restricvted mode as well.

One of the issues related to this bug is sending a pkgbuild with a PKGDESC like this pkgdesc="Foo # blah" # bar

i solved this part in CCR like this:

function strip_comments($text) {
$pass1 = preg_replace('/(".*"|\s*#.*|\'.*\'|\(.*\))/','',$text);
if ( $pass1 != "" ) {
$pass2 = str_replace($pass1,"",$text);
$pass3 = preg_replace('/(".*"|\'.*\'|\(.*\))/','',$pass2);
$final = str_replace($pass3,"",$text);
} else {
$final = '';
}
return $final;

is a bit tricky but allow the use of ### inside pkgdesc="" , '' , =("") or =('') ,
eg: pkgdesc=("this is a c# test") #some text ---> pkgdesc=("this is a c# test")

I reformatted a bit the function:

http://paste.kde.org/83557/

manutortosa: Sorry, but this is crap. It won't work with multi-line strings, doesn't respect escaping ('foo="\"" #nasty"'), backticks ('`echo foo #nasty`'), comments that contain quotes ('foo # bar "nasty"') and much more...

Well after this message i tested a lot and in fact is not a good solution, i reduced the usage to a single line and only for the PKGDESC line and only btw "" and '' but still does not cover all the possible situations.

Is it possible to create a user with a highly limited quota (e.g. 200 bytes), memory limit, process limit, CPU limit, etc., and then take advantage of parse_pkgbuilds.sh from dbtools?

Anyone care to have a look at this?: http://qiaomuf.wordpress.com/category/libbash/
http://git.overlays.gentoo.org/gitweb/?p=proj/libbash.git;a=summary

Alain: Does it support evaluting inline sed(1), awk(1), etc.? Several packages use that.

In my opinion, there are only two viable solutions:

* Execute PKGBUILDs in a sandbox.
* Provide some kind of meta data with source tarballs.

Lukas: AFAICT it does some (limited) emulation of Bash text substitutions e.g. ${var/srch/repl} and such but no sed or awk sadly.
I'm looking into a way of building a super-restricted Qemu -> really-tiny-kernel -> Bash-and-some-tools-in-initrd setup (no drivers or network at all, just an emulated serial port console to communicate with the outside world.
In keeping with the KISS principle I'm thinking about faking, pretending and cheating just enough to trick makepkg into building the barest skeleton of a package, and extracting the .PKGINFO file out of that.
Any help or suggestions on this would be greatly appreciated.

The proper solution to this is a .SRCINFO file. Involving a VM will mean involving makepkg to do the re-parsing of the PKGBUILD (because of split package overrides), which means it could have just been done at 'makepkg --source' time.

Recently, I was sitting in #archlinux-bugs when this bug caught my eye.
Currently, the AUR parses PKGBUILDS minimally, which causes issues for
split packages. There are three paths I can see the AUR taking:

1) Keep doing what it's doing and don't worry about split PKGBUILDs,
but remain safe if there is malicious code in the PKGBUILD.
2) Parse PKGBUILDS more fully but leave the AUR more open to attack
from malicious code
3) Stop parsing PKGBUILDs on the server and use something akin to
the .PKGINFO file in a package file. Have makepkg generate a
.SRCINFO file (as per Dave Reisner's suggestion) and put that in
the src.* file that is uploaded to the server.

The third option seems to be the best option to me, and would help keep
the AUR server safe. Only one thing that I found when trying to patch
makepkg to generate .SRCINFO files for split packages was that the
variables (such as pkgdesc, provides, depends, etc.) are parsed in the
package functions for the specific packages. I was thinking that the
--source flag should generate a .SRCINFO file in two ways depending on
the PKGBUILD:

split package: make sure that at least the package function is run
and generate $pkgname-$full_version$SRCEXT, instead of using the
$pkgbase. Because the .SRCINFO will contain the pkgbase, these can
be grouped together easily. (Or have a single src package with
.SRCINFO_$pkgname all in one file, then split into separate pages on
the AUR with a single location, like the official package
repositories.)

non-split package: allow --source to be run without packaging each
package, because all variables should be able to be written to the
.SRCINFO file easily

Current packages could remain, and the normal way of submitting packages
would be deprecated as pacman 4.1 is released to [core].
While current pacman development builds could still be submitted, as the
same files would be in the src tarball, but would simply include an
extra file.

Thank you for taking the time to read. If anyone wants me to keep
working on this patch or if someone wants to do this way before my
minimal shell scripting skills fail me horribly.

We support .AURINFO since 2.1.0 -- any objections against closing this?

Is it possible to use this ".AURINFO" file before it is officially supported by makepkg (and renamed to .SRCINFO)?

Of course, why not? Just try to add an .AURINFO file to a source tarball and upload it to the AUR.

Sorry, I should have added that I got an error message when doing so: "Invalid name: only lowercase letters are allowed."
This is for a splitted package, with "pkgname=('pkg1' 'pkg2')" in the PKGBUILD and "pkgname = pkg1" in the .AURINFO file.

It seems there is a check done before the PKGBUILD value is overriden. But maybe this is belonging to  FS#16394  instead?

Could you attach your .AURINFO file, please?

This issue is fixed. There was an ending blank space in the AURINFO ("package = value " instead of "package = value").
This won't happen with automatically created .SRCINFO file by makepkg. Thanks, and sorry for the noise.

Seems I've found a parsing issue. For example, in the involved package (etlegacy):

.AURINFO:
pkgname = etlegacy


PKGBUILD:
pkgname=('etlegacy' 'etlegacy-mod')
pkgdesc="Fully compatible client and server for the popular online FPS game Wolfenstein: Enemy Territory
...
source=($pkgname-linux-$pkgver.tar.gz)

The AUR seems to consider $pkgname as being the second element of the pkgname array (etlegacy-mod), despite being overridden in the AURINFO .file.
So the Sources section of the AUR page shows incorrect filename. Have a look at the AUR page (https://aur.archlinux.org/packages/etlegacy/) and the corresponding PKGBUILD.

Makepkg correctly considers $pkgname as being the first element of the array when compiling the package locally.

I found one more parsing issue:

When a (pacman-4.1 compliant) git-pkg has a fragment in it's source url, the source is not listen in the webui:
Without fragment:
https://aur.archlinux.org/packages/gnuradio-git/

With fragment:
https://aur.archlinux.org/packages/libosmocore/

asdil12: So we should either add a way to specify sources in .AURINFO or remove the sources list from the AUR web interface. I will not try to fix our PKGBUILD parser.