Arch Linux

any progress on this? i get really shitty crashes in amarok now and i do not want to compile every program that ever crashes. (i’m currently finishing my master’s thesis, and while i have time to let dr. konqui send crash dumps to the KDE bug tracker, i don’t have time to set up build environments for a program that crashes, build and install it, reproduce the crash there, …)

it would be really nice of you to help users and developers alike by providing debug packages.

i now have time to work on this.

the repo is https://projects.archlinux.org/dbscripts.git, right?

what’s needed to support debug symbol packages? how is this different from split packages? or isn’t it really, but it would be better to special-case debug-packages instead of making everything into split packages?

can someone give me any hint on what is needed to fix this?

or do you not want your users to help file bugs to upstream packages?

Allan explained what is needed in a forum post (https://bbs.archlinux.org/viewtopic.php?pid=1544964#p1544964):
1) devtools: alter to build debug packages by default and sign/upload them to our server with the other packages
2) dbscripts: recognise a debug package and add it to the foo-debug repo when the main package is being added to foo.

Some time ago these patches were posted by Allan McRae as a starting point:

https://lists.archlinux.org/pipermail/arch-projects/2015-August/004300.html
https://lists.archlinux.org/pipermail/arch-projects/2015-August/004301.html

I had a shot at fixing them - didn't get anywhere due to time constraints but I'm pushing what I have and documenting a bit so it's not forgotten.

https://github.com/dequis/arch-devtools/commits/debug
https://github.com/dequis/arch-dbscripts/commits/debug

This is *not* complete, and I'm not even sure it's correct. The devtools patch just had a syntax error, and that works as far as I can see, but dbscripts still needs a lot of work.

The devtools patch can be tested by building something with "extra-x86_64-build" followed by "extrapkg -s localhost" (extrapkg is an alias of commitpkg, and this rsyncs to localhost:~/staging )

dbscripts is a fairly obscure piece of the arch build system, only used by people who have commit access to the svn, but luckily none of that is needed because it includes a completely self-contained test suite which creates tiny svn repositories with a handful of packages, it's pretty cool. That was my main fear when getting in, that this might only be understood by arch devs, but I'm not so worried about that.

One of the problems with debug packages as opposed to split packages is that they are generated implicitly based on the existence of strippable binaries and the right combination of makepkg flags, instead of being announced beforehand as part of the PKGBUILD variables. I'm pretty sure all of this is run when built packages already exist, so in theory every place that reads PKGBUILD can be replaced with reading PKGINFO.

The other main problem is just putting the packages in the right places. If a function receives two package files for 'extra' and one of them is a debug package, should it have additional logic to put the debug package in 'extra-debug', or should it blame the caller for passing it packages for the wrong repos? Plenty of places where that decision should be made, and maybe different answers are more convenient in different places.

Might be a problem, but not sure: there are some packages in AUR that don't strictly fit the definition of "debug package" but their names end in -debug. For example clang-debug, flashplayer-standalone-debug or python-flask-debug. Should this sort of thing be forbidden? Should the code be written so that it never needs to blindly append a -debug to a package? That might not be possible. It would be a problem if you were packaging python-flask (which has no binaries) and it sent python-flask-debug to the debug repo instead of community.

first of all: thank you so much for helping here. i would, too, but i have no idea how to get a testing environment set up (i’d assume that’d be some virtual machine hosting a repo server?)

hmm, the project name for python-flask-debug is “Flask-Debug”. We lowercase it anyway, so s/-/_/ wouldn’t be a prohibitive requirement IMHO. flashplayer-standalone-debug would be more correctly named “flashplayer-standalone-debugger” (see the download links on its webpage). finally i’m not sure about the purpose of the llvm-debug base-package (containing llvm-debug, clang-debug, …) but if it’s just clang/llvm plus debug symbols, it could be easily replaced with {clang,llvm,…}-debug packages containing only the debug symbols and depending on {clang,llvm,…}, respectively

another alternative would be using -dbg as suffix.

Hi, thank you for working on this feature, that I really would like to have in Arch linux. So I can contribute with testing.

A another solution is to have a debug repo. Debug symbols are good for bug reports so it could be an opt in thing. And also testing repo must have debug package for obvious reasons

You also need to add support for dwz (and package it in archlinux).

Actual webkit2gtk-debug size => Around 4gio
On Fedora => 1 gio thanks to dwz support in RPM.

From my understanding, there are 2 possible approaches to packaging debug information:

Method 1 - The one seemingly favored above, with code started, is creating 2 packages:
(1) a package with stripped binaries, as we do now for general distribution from the mirrors
(2) a package in a "*-debug" repo containing unstripped binaries with debug information embedded.

Method 2 - Similarly, creating 2 packages:
(1) a package with splitdebug binaries, for general distribution from the mirrors
(2) a package in a "*-debug" repo containing just the splitdebug symbols.

In method 1 the "*-debug" repo binaries become drop-in replacements for the stripped binaries, and are seemingly simple to generate. We just need to package once before stripping (the debug version), then again after (the stripped version).

But if we used debug repos like that, wouldn't the debug repo packages conflict with the non-debug versions? I see 2 potential issues happening:
(a) conflicting files in the packages - since the debug and non-debug packages contain the same filenames, this may also break dependencies from other packages.
(b) pacman may select packages from the enabled debug repo (or the non-debug repo) against the user's goals. The user may want debug versions of those packages, or not, and pacman may do the opposite. One can, of course, force a particular repo when installing like "pacman -S targetrepo/package", but what about in the default case?

Method 2 has additional benefits that the "debug packages" (containing only the split symbols) do not need to conflict with their non-debug counterparts, and, if anything, the "debug packages" can depend on the standard package via typical Arch package dependency methods. Another benefit to this is that the debug packages are smaller, since they can avoid repackaging the binaries themselves. In my experience, the size of the stripped/split binaries is generally significantly less than the size of the debug symbols, so the size aspect is likely a smaller benefit.

From another perspective, if we had splitdebug packages already made with the stripped binaries and the split out debug files in one package, a user could choose to skip installing debug symbols by simply using the "NoExtract" option in their /etc/pacman.conf. Obviously, this would cause an larger package download size, but it would work.

So building off that "skipping files" approach, we could build the package once with splitdebug enabled, then just package the debug symbols (by the filesystem path to the symbols themselves) independent of the rest of the package. So basically, the packaging tool would skip the path we used for symbols (so /usr/lib64/debug/* or similar) when packaging the standard binary package, then package only the /usr/lib64/debug/* files as the debug package.

Am I missing something as to why Method 1 is preferred?



Regarding the package naming conflicts, we could use a special separator on package names; maybe a double dash separator ("bash--debug") or underscores ("bash__debug") or something like that, but I imagine special cases in package names are not ideal either, so maybe we should use "-debugsymbols" or another non-conflicting suffix instead?



Regarding the "how to build a test environment" comment by Phil (flying-sheep), I think detailing this out would be a good starting point to getting people to provide more patches. If nobody but core Arch developers ever use the tools, it will be very hard to get new people in here with enough experience to help develop the tools.

I haven't tried, but I think dx provided enough detail to test the devtools tool (I have some idea what I would need to do to test it). For dbscripts, on the other hand, I'm still not sure what to do. Looking at the repo, I think it may be as simple as checking it out into some local path, running the test suite, and seeing what the resulting packages look like. Is this correct?



As a bit of background and a slight rant: I'm coming from a Gentoo background, but I run Arch on my smaller/older systems because they don't have the necessary power to rebuild everything from sources. I've ran across the need to debug programs several times on my Arch systems, where debug information is simply not available without doing the exact thing that made me avoid Gentoo and choose Arch to begin with. I realize that even makepkg support for splitdebug is rather new, so I'm not expecting it to manifest tomorrow, but hopefully we can make some progress sooner than 4 years from now. I don't have the time to dig into developing this functionality right now, but I may attempt it in the near future if I get some helpful feedback.

Thanks!

makepkg has builtin support for "foo-debug" packages, as the initial feature request states. They contain only debugging symbols, and you still need the "foo" package for the actual binaries themselves. This is not new, it was implemented in 2013, but it's one of the lesser-known features. As such, there is absolutely zero possibility that we will agree to implement #1.

This feature request is intended to track the progress for implementing #2, adding the necessary tooling to track these packages and add them to [community-debug] style repositories.

Among other things, this requires:

- handling package bases which appear in multiple repositories
- handling the special case of debug packages, which are not in the ${pkgname[@]} array
- handling packages which may or may not exist, and removing the old version of a package that no longer exists

The last case would be made significantly easier by using git, which supports tags, so add in a migration from subversion to git...

Also see some documentation on existing work and brainstorming: https://wiki.archlinux.org/index.php/User:Gbs/dbscripts-status

Chiming in regarding a potential use-case for this and proper debug information packages. At Mozilla we recently improved the analysis of Firefox crashes coming from distro-packaged builds by scraping symbols and debug information directly from the distro packages. This means that starting from two weeks ago crash reports coming from Arch users show up properly and are easier to analyse for us. However since the firefox package is fully stripped the information we get is still somewhat limited which is a shame. If debuginfo packages were available I could easily use them to improve our crash analysis - something I successfully managed for Ubuntu and Debian Firefox builds.

Isn't that what ./mach buildsymbols followed by uploading *crashreporter-symbols-full.zip to https://symbols.mozilla.org/ is for? I was given to understand that Mozilla uses this to make crash reports for their specific use case, upload sensible things regardless of whether the user has debug symbols installed.

Yes, it is. Were you already doing that? If that's the case then I will turn off the symbol scraping for Arch because it's not needed (at least for the Firefox package, we're still scraping all the libraries it depends on).

Yes, this is something our Firefox maintainer currently does, see:

https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/firefox#n113
https://git.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/firefox#n120

And the simple shell script to upload it after the package is built:
https://git.archlinux.org/svntogit/packages.git/tree/trunk/upload-symbol-archive?h=packages/firefox

...

Of course, as you said, it's also useful to still scrape dependencies (and ideally dependencies which contain debug symbols) in order to help resolve crash symbols that invoke code which is not Mozilla-specific.

On that note, thank you for sharing your interesting perspective!

This bug is assigned and it’s been a year. Is there any progress?

Assignment doesn't mean much. But is that a rhetorical question?

https://lists.archlinux.org/pipermail/arch-dev-public/2020-November/030222.html