FS#66438 - [meson] setup.py installation method makes meson start slower

Attached to Project: Arch Linux
Opened by Ran Benita (bluetech) - Monday, 27 April 2020, 22:58 GMT
Last edited by freswa (frederik) - Monday, 27 April 2020, 23:40 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Jan Alexander Steffens (heftig)
Levente Polyak (anthraxx)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
meson can be made to start up faster with a tweak to how it is packaged.

The meson package uses `setup.py install` to install, as recommended by the Python package guidelines:
https://wiki.archlinux.org/index.php/Python_package_guidelines#Installation_methods

This installation method creates an entry point (/usr/bin/meson) which looks like this:

----------
#!/usr/bin/python
# EASY-INSTALL-ENTRY-SCRIPT: 'meson==0.54.1','console_scripts','meson'
__requires__ = 'meson==0.54.1'
import re
import sys
from pkg_resources import load_entry_point

if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0])
sys.exit(
load_entry_point('meson==0.54.1', 'console_scripts', 'meson')()
)
----------

Specifically, it imports the `pkg_resources` package which has a big memory and time overhead.
The issues with `pkg_resources` are discussed here: https://github.com/pypa/setuptools/issues/510
That issue is closed with a recommendation to use the `pip install` installation method instead,
which the package guidelines currently discourage in favor of the setuptools method. The pip
method generates the following entry point:

----------
#!/usr/bin/python
# -*- coding: utf-8 -*-
import re
import sys
from mesonbuild.mesonmain import main
if __name__ == '__main__':
sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0])
sys.exit(main())
----------

Note that this doesn't import `pkg_resources`.

On my computer, with the setuptools method `meson --version` takes 390ms, with the pip method it
takes 247ms, a significant difference for a command which is run often.

BTW, this issue affects all Python executables installed in this way, with the same fixed overhead,
so might be desirable to discuss in the context of the entire Python package guidelines.

Additional info:
* package version(s)
meson 0.54.1-1

* link to upstream bug report, if any
There is some ongoing discussion here: https://github.com/mesonbuild/meson/pull/6829#pullrequestreview-400775558

Steps to reproduce:

For the setuptools method, I used the 0.54.1-1 PKGBUILD.

For the pip method, I changed the PKGBUILD as follows:

--- PKGBUILD-old 2020-04-28 01:50:20.962641090 +0300
+++ PKGBUILD 2020-04-28 01:50:29.029399852 +0300
@@ -10,6 +10,7 @@
arch=('any')
license=('Apache')
depends=('python-setuptools' 'ninja')
+makedepends=('python-pip' 'python-wheel')
checkdepends=('gcc-objc' 'vala' 'rust' 'gcc-fortran' 'mono' 'boost' 'qt5-base' 'git' 'cython'
'gtkmm3' 'gtest' 'gmock' 'protobuf' 'wxgtk' 'python-gobject' 'gobject-introspection'
'itstool' 'gtk3' 'java-environment=8' 'gtk-doc' 'llvm' 'clang' 'sdl2' 'graphviz'
@@ -50,7 +51,9 @@

package() {
cd ${pkgname}-${pkgver}
- python setup.py install --root="${pkgdir}" --optimize=1 --skip-build
+ PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps --no-warn-script-location .
+ local site_packages=$(python -c "import site; print(site.getsitepackages()[0])")
+ python -O -m compileall "${pkgdir}${site_packages}/mesonbuild"

install -d "${pkgdir}/usr/share/vim/vimfiles"
cp -rt "${pkgdir}/usr/share/vim/vimfiles" data/syntax-highlighting/vim/*/
This task depends upon

Closed by  freswa (frederik)
Monday, 27 April 2020, 23:40 GMT
Reason for closing:  Won't fix
Comment by Jan Alexander Steffens (heftig) - Monday, 27 April 2020, 23:06 GMT
I don't see why we should be singling out meson for this, especially since it's not frequently executed.
Comment by Eli Schwartz (eschwartz) - Monday, 27 April 2020, 23:12 GMT
We are not changing the python guidelines to make every single package depend on all of pip before it can be bootstrapped.

The large number of things you need to do to "properly" package with pip, should have been the warning that this is not a good idea.

...

Anyway, the whole point of what setuptools is doing here is checking that all dependencies are installed and correct before starting the program. This is ostensibly a feature, otherwise you could just do "from foo import main; main()"
Comment by Ran Benita (bluetech) - Monday, 27 April 2020, 23:15 GMT
> I don't see why we should be singling out meson for this

I agree -- for me personally this affects the following programs as well, I think it will be better to fix all
of them if the concept is agreeable. What would be a better place to post this?

> We are not changing the python guidelines to make every single package depend on all of pip.

Just to be clear, it would only be a build dependency. But if that's what you meant, then I guess
we can close this as won't-fix.

---

$ grep --files-with-matches 'from pkg_resources import load_entry_point' /bin | sort
borg
borgfs
chardetect
docker-compose
easy_install
easy_install-3.8
f2py
f2py3
f2py3.8
isympy
jsonschema
lit
mako-render
markdown_py
meson
mid3cp
mid3iconv
mid3v2
moggsplit
mutagen-inspect
mutagen-pony
pygmentize
pyrsa-decrypt
pyrsa-encrypt
pyrsa-keygen
pyrsa-priv2pub
pyrsa-sign
pyrsa-verify
tox
tox-quickstart
tqdm
virtualenv
virtualenv3
wheel
youtube-dl
Comment by Eli Schwartz (eschwartz) - Monday, 27 April 2020, 23:17 GMT
> Just to be clear, it would only be a build dependency. But if that's what you meant, then I guess we can close this as won't-fix.

I had already edited my post to specify bootstrapping, and added additional context to why setuptools chooses to do the way it does.
Comment by Ran Benita (bluetech) - Monday, 27 April 2020, 23:39 GMT
> I had already edited my post to specify bootstrapping

I don't see a mention of bootstrapping -- if you mean bootstrapping `pip` and `wheel` themselves, maybe they specifically can keep using `setuptools`.

> Anyway, the whole point of what setuptools is doing here is checking that all dependencies are installed and correct before starting the program.

That's a good point -- the error message setuptools shows is more informative than some import error, or even a hidden error if the dependency exists but is incompatible.
IMO however it is not worth the tradeoff of adding >100ms to every Python program startup time -- it is a broken install after all. But I can see how as a whole-distribution
maintainer your perspective on this would be different, given you you will have to handle the uninformative bug reports.

Thanks for considering!

Loading...