FS#76686 - [makepkg] --printsrcinfo is slow

Attached to Project: Pacman
Opened by Jelle van der Waa (jelly) - Monday, 28 November 2022, 12:45 GMT
Last edited by Allan McRae (Allan) - Tuesday, 26 September 2023, 02:21 GMT
Task Type Bug Report
Category General
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version 6.0.2
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Summary and Info:

Sourcing PKGBUILD's isn't a nice approach to interact with PKGBUILD's in other languages, so using SRCINFO files is preferred, however generating these files as slow:

Steps to Reproduce:

makepkg --printsrcinfo 0.72s user 0.19s system 108% cpu 0.836 total

The main offender of slowness seems to be `lint_pkgbuild`:

# check the PKGBUILD for some basic requirements
-lint_pkgbuild || exit $E_PKGBUILD_ERROR
+#lint_pkgbuild || exit $E_PKGBUILD_ERROR


/home/jelle/projects/pacman/build/makepkg --printsrcinfo 0.19s user 0.03s system 106% cpu 0.209 total
This task depends upon

Closed by  Allan McRae (Allan)
Tuesday, 26 September 2023, 02:21 GMT
Reason for closing:  Deferred
Additional comments about closing:  https://gitlab.archlinux.org/pacman/pacm an/-/issues/56
Comment by Jelle van der Waa (jelly) - Monday, 28 November 2022, 12:49 GMT
Either we skip linting or lint in parallel as we have multiple cores these days and should use them

for func in ${lint_pkgbuild_functions[@]}; do
$func || ret=1
done
Comment by Jelle van der Waa (jelly) - Monday, 28 November 2022, 13:08 GMT
So with a dumb test, and just $func || ret=1 & and wait, I get 2x speedup, not linting is a 4x speedup.
Comment by Allan McRae (Allan) - Monday, 28 November 2022, 13:28 GMT
Without the linting to confirm the PKGBUILD is well formed, the output from --printsrcinfo can be wrong.

How high throughput do you need this to be?
Comment by Jelle van der Waa (jelly) - Monday, 28 November 2022, 14:23 GMT
I'm not sure how fast it needs to be, linting is one part of the problem, for example thunderbird's PKGBUILD takes 7 seconds here, due to all the bash shenanigans. But for other tools to consume --printsrcinfo waiting almost one second is a bit much.

Note that the linting issue will become exponentially worse, I also found out that lint_{make,check,opt}depends are all almost the same code but in a separate file.
Comment by Allan McRae (Allan) - Tuesday, 27 December 2022, 02:14 GMT
FWIW, there are 64 packages being made in the thunderbird PKGBUILD. That makes 0.1sec/package.

However, I did notice the output was actually wrong for all the i18n packages due to the nesting of functions. I do not consider that a makepkg issue, and it can be fixed in the PKGBUILD.
Comment by Jelle van der Waa (jelly) - Friday, 06 January 2023, 13:05 GMT
It's not just about thunderbird however, a "normal" package takes 0.72 seconds. The issue is that additional linting done by makepkg will make everything exponentially slower.
Comment by Allan McRae (Allan) - Friday, 06 January 2023, 14:13 GMT
Linting is intended to ensure that the bash "parser" used in --printsrcinfo works. This was a compromised made to avoid having to build the package to generate the srcinfo file and get (close to) correct information (mostly for split packages). Keeping this is non-negotiable.

The question remains whether we can do this in parallel. After a quick look, I can't figure out how to do that and keep the output useful in the case of issues being found.

There is no exponential order checks being performed.

Loading...