FS#66438 - [meson] setup.py installation method makes meson start slower
Attached to Project:
Arch Linux
Opened by Ran Benita (bluetech) - Monday, 27 April 2020, 22:58 GMT
Last edited by freswa (frederik) - Monday, 27 April 2020, 23:40 GMT
Opened by Ran Benita (bluetech) - Monday, 27 April 2020, 22:58 GMT
Last edited by freswa (frederik) - Monday, 27 April 2020, 23:40 GMT
|
Details
Description:
meson can be made to start up faster with a tweak to how it is packaged. The meson package uses `setup.py install` to install, as recommended by the Python package guidelines: https://wiki.archlinux.org/index.php/Python_package_guidelines#Installation_methods This installation method creates an entry point (/usr/bin/meson) which looks like this: ---------- #!/usr/bin/python # EASY-INSTALL-ENTRY-SCRIPT: 'meson==0.54.1','console_scripts','meson' __requires__ = 'meson==0.54.1' import re import sys from pkg_resources import load_entry_point if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw?|\.exe)?$', '', sys.argv[0]) sys.exit( load_entry_point('meson==0.54.1', 'console_scripts', 'meson')() ) ---------- Specifically, it imports the `pkg_resources` package which has a big memory and time overhead. The issues with `pkg_resources` are discussed here: https://github.com/pypa/setuptools/issues/510 That issue is closed with a recommendation to use the `pip install` installation method instead, which the package guidelines currently discourage in favor of the setuptools method. The pip method generates the following entry point: ---------- #!/usr/bin/python # -*- coding: utf-8 -*- import re import sys from mesonbuild.mesonmain import main if __name__ == '__main__': sys.argv[0] = re.sub(r'(-script\.pyw|\.exe)?$', '', sys.argv[0]) sys.exit(main()) ---------- Note that this doesn't import `pkg_resources`. On my computer, with the setuptools method `meson --version` takes 390ms, with the pip method it takes 247ms, a significant difference for a command which is run often. BTW, this issue affects all Python executables installed in this way, with the same fixed overhead, so might be desirable to discuss in the context of the entire Python package guidelines. Additional info: * package version(s) meson 0.54.1-1 * link to upstream bug report, if any There is some ongoing discussion here: https://github.com/mesonbuild/meson/pull/6829#pullrequestreview-400775558 Steps to reproduce: For the setuptools method, I used the 0.54.1-1 PKGBUILD. For the pip method, I changed the PKGBUILD as follows: --- PKGBUILD-old 2020-04-28 01:50:20.962641090 +0300 +++ PKGBUILD 2020-04-28 01:50:29.029399852 +0300 @@ -10,6 +10,7 @@ arch=('any') license=('Apache') depends=('python-setuptools' 'ninja') +makedepends=('python-pip' 'python-wheel') checkdepends=('gcc-objc' 'vala' 'rust' 'gcc-fortran' 'mono' 'boost' 'qt5-base' 'git' 'cython' 'gtkmm3' 'gtest' 'gmock' 'protobuf' 'wxgtk' 'python-gobject' 'gobject-introspection' 'itstool' 'gtk3' 'java-environment=8' 'gtk-doc' 'llvm' 'clang' 'sdl2' 'graphviz' @@ -50,7 +51,9 @@ package() { cd ${pkgname}-${pkgver} - python setup.py install --root="${pkgdir}" --optimize=1 --skip-build + PIP_CONFIG_FILE=/dev/null pip install --isolated --root="$pkgdir" --ignore-installed --no-deps --no-warn-script-location . + local site_packages=$(python -c "import site; print(site.getsitepackages()[0])") + python -O -m compileall "${pkgdir}${site_packages}/mesonbuild" install -d "${pkgdir}/usr/share/vim/vimfiles" cp -rt "${pkgdir}/usr/share/vim/vimfiles" data/syntax-highlighting/vim/*/ |
This task depends upon
The large number of things you need to do to "properly" package with pip, should have been the warning that this is not a good idea.
...
Anyway, the whole point of what setuptools is doing here is checking that all dependencies are installed and correct before starting the program. This is ostensibly a feature, otherwise you could just do "from foo import main; main()"
I agree -- for me personally this affects the following programs as well, I think it will be better to fix all
of them if the concept is agreeable. What would be a better place to post this?
> We are not changing the python guidelines to make every single package depend on all of pip.
Just to be clear, it would only be a build dependency. But if that's what you meant, then I guess
we can close this as won't-fix.
---
$ grep --files-with-matches 'from pkg_resources import load_entry_point' /bin | sort
borg
borgfs
chardetect
docker-compose
easy_install
easy_install-3.8
f2py
f2py3
f2py3.8
isympy
jsonschema
lit
mako-render
markdown_py
meson
mid3cp
mid3iconv
mid3v2
moggsplit
mutagen-inspect
mutagen-pony
pygmentize
pyrsa-decrypt
pyrsa-encrypt
pyrsa-keygen
pyrsa-priv2pub
pyrsa-sign
pyrsa-verify
tox
tox-quickstart
tqdm
virtualenv
virtualenv3
wheel
youtube-dl
I had already edited my post to specify bootstrapping, and added additional context to why setuptools chooses to do the way it does.
I don't see a mention of bootstrapping -- if you mean bootstrapping `pip` and `wheel` themselves, maybe they specifically can keep using `setuptools`.
> Anyway, the whole point of what setuptools is doing here is checking that all dependencies are installed and correct before starting the program.
That's a good point -- the error message setuptools shows is more informative than some import error, or even a hidden error if the dependency exists but is incompatible.
IMO however it is not worth the tradeoff of adding >100ms to every Python program startup time -- it is a broken install after all. But I can see how as a whole-distribution
maintainer your perspective on this would be different, given you you will have to handle the uninformative bug reports.
Thanks for considering!