FS#34731 - [pypy] Unable to create pypy virtualenv

Attached to Project: Community Packages
Opened by Sam Sneddon (gsnedders) - Thursday, 11 April 2013, 14:38 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Tuesday, 23 April 2013, 01:42 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 2
Private No

Details

This apparently works fine upstream. This clearly doesn't work here. Given virtualenv is the recommended way of using pypy, this is somewhat severe.

gsnedders@vanveen:~$ virtualenv --version
1.9.1
gsnedders@vanveen:~$ pypy --version
Python 2.7.3 (816e4e71734f, Apr 11 2013, 00:32:06)
[PyPy 2.0.0-beta2 with GCC 4.8.0]
gsnedders@vanveen:~$ pacman -Q pypy python-virtualenv
pypy 2.0beta2-1
python-virtualenv 1.9.1-1
gsnedders@vanveen:~$ virtualenv -p pypy foo
Running virtualenv with interpreter /usr/bin/pypy
New pypy executable in foo/bin/pypy
Installing setuptools............................done.
Installing pip..............
Complete output from command /home/gsnedders/foo/bin/pypy /home/gsnedders/foo/bin/easy_install /usr/lib/python3.3/s...ort/pip-1.3.1.tar.gz:
Processing pip-1.3.1.tar.gz
Running pip-1.3.1/setup.py -q bdist_egg --dist-dir /tmp/easy_install-qJdpi6/pip-1.3.1/egg-dist-tmp-FYfWn9
warning: no files found matching '*.html' under directory 'docs'
warning: no previously-included files matching '*.txt' found under directory 'docs/_build'
no previously-included directories found matching 'docs/_build/_sources'
Traceback (most recent call last):
File "app_main.py", line 52, in run_toplevel
File "/home/gsnedders/foo/bin/easy_install", line 8, in <module>
load_entry_point('setuptools==0.6c11', 'console_scripts', 'easy_install')()
File "/home/gsnedders/foo/site-packages/setuptools-0.6c11-py2.7.egg/setuptools/command/easy_install.py", line 1712, in main
MemoryError
----------------------------------------
...Installing pip...done.
Traceback (most recent call last):
File "app_main.py", line 52, in run_toplevel
File "/usr/lib/python3.3/site-packages/virtualenv.py", line 2577, in <module>
main()
File "/usr/lib/python3.3/site-packages/virtualenv.py", line 979, in main
no_pip=options.no_pip)
File "/usr/lib/python3.3/site-packages/virtualenv.py", line 1094, in create_environment
install_pip(py_executable, search_dirs=search_dirs, never_download=never_download)
File "/usr/lib/python3.3/site-packages/virtualenv.py", line 667, in install_pip
filter_stdout=_filter_setup)
File "/usr/lib/python3.3/site-packages/virtualenv.py", line 1057, in call_subprocess
% (cmd_desc, proc.returncode))
OSError: Command /home/gsnedders/foo/bin/pypy /home/gsnedders/foo/bin/easy_install /usr/lib/python3.3/s...ort/pip-1.3.1.tar.gz failed with error code 1
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Tuesday, 23 April 2013, 01:42 GMT
Reason for closing:  Fixed
Comment by Sam Sneddon (gsnedders) - Thursday, 11 April 2013, 20:13 GMT
This build seems to frequently give MemoryError (inc. while trying to translate itself). Can you change the 2.0beta2 package to actually bundle 2.0beta2 and not some random revision from hg?
Comment by Simon Sapin (SimonSapin) - Thursday, 11 April 2013, 21:13 GMT
I can confirm the same issue.

Quoting from the PKGBUILD file:
https://projects.archlinux.org/svntogit/community.git/tree/trunk/PKGBUILD?h=packages/pypy&id=16339f72458549bcff806efb2659c40a23a91274

source=('hg+https://bitbucket.org/pypy/pypy')
md5sums=('SKIP')

What could go wrong?
Seriously though, please use pypy-2.0b2-src.tar.bz2 available here: http://www.pypy.org/download.html#building-from-source
Comment by Sam Sneddon (gsnedders) - Thursday, 11 April 2013, 22:57 GMT
https://bitbucket.org/pypy/pypy/commits/4943ac67b80c39bad62832887d884d0e99f4c9a5 and https://bitbucket.org/pypy/pypy/commits/33015de is the fix for it not building on GCC 4.8. Patching that in to the 2.0b2 release would be better than building random commits from hg.
Comment by Sam Sneddon (gsnedders) - Friday, 12 April 2013, 00:10 GMT
And it turns out the actual 2.0b2 release equally gets MemoryError. Asking around on IRC nobody else has seen this. Odd. More subtle GCC 4.8 b0rkedness?
Comment by Sam Sneddon (gsnedders) - Friday, 12 April 2013, 12:20 GMT
The conclusion upstream is to just use shadowstack with GCC 4.8, so: pypy ../../rpython/bin/rpython -Ojit --gcrootfinder=shadowstack targetpypystandalone
Comment by Sam Sneddon (gsnedders) - Friday, 12 April 2013, 12:39 GMT
And it's still broken… https://bugs.pypy.org/issue1450
Comment by Sven-Hendrik Haase (Svenstaro) - Sunday, 14 April 2013, 03:03 GMT
I'm not merely using the hg tip without reason. If you actually tried building pypy you would see that it will fail with gcc 4.8. I reported some problems to the devs and apparently some of those got fixed in hg. Since this is a beta, why should I package a certain commit? Using the tip will only get us more fixes since pypy is currently in feature freeze anyhow.
Comment by Sven-Hendrik Haase (Svenstaro) - Sunday, 14 April 2013, 03:04 GMT
I'm currently building a new version of pypy for [community]. It might contain fixes for this certain problem. I shall check.
Comment by Sam Sneddon (gsnedders) - Sunday, 14 April 2013, 17:37 GMT
Well, either don't give it the version number 2.0b2 (as it isn't), or use 2.0b2 with patches. (I'd recommend packaging a certain commit because otherwise you assume they never regress anything, and while they *shouldn't*, nobody is perfect.)

Given it still seems broken with 4.8 (see the upstream bug), I'd suggest moving to compiling it with clang (given that's in extra whereas old GCC releases are AUR), as that works and can self-host (more recent PyPy builds; 2.0b2 has a bug such that it cannot self-host) fine.
Comment by Zack Buhman (buhman) - Sunday, 14 April 2013, 19:17 GMT
> I'd suggest moving to compiling it with clang (given that's in extra whereas old GCC releases are AUR), as that works

Upstream says clang is completely incapable of compiling the JIT [1].

> and can self-host (more recent PyPy builds; 2.0b2 has a bug such that it cannot self-host) fine.

You'll need to provide more evidence that supports this claim; upstream's 2.0b2 binary [2] self-hosts just fine.

> Well, either don't give it the version number 2.0b2 (as it isn't), or use 2.0b2 with patches.

{core,testing}/gcc isn't 4.8.0 either [3], it's a relatively "random" snapshot. There seems to be established precedent for doing this sort of thing with package version numbers, so I don't see why it's a problem here.

[1] http://pypy.org/download.html#building-from-source (note #3)
[2] https://bitbucket.org/pypy/pypy/downloads/pypy-2.0-beta2-linux64-libc2.15.tar.bz2
[3] https://projects.archlinux.org/svntogit/packages.git/tree/trunk/PKGBUILD?h=packages/gcc
Comment by Sam Sneddon (gsnedders) - Sunday, 14 April 2013, 20:59 GMT
> Upstream says clang is completely incapable of compiling the JIT [1].

No, they say asmgcc (which may well soon get killed) doesn't support Clang; this is tangential to JIT support. shadowstack works fine with JIT, and works fine when compiled with Clang.

> You'll need to provide more evidence that supports this claim; upstream's 2.0b2 binary [2] self-hosts just fine.

Okay, I'd merely assumed from the upstream bug that 2.0b2 got TyperError always (whereas the current package does with --jit off, and gets MemoryError long before that with it on).

> There seems to be established precedent for doing this sort of thing with package version numbers, so I don't see why it's a problem here.

Well, I wasn't aware of that. :)
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 15 April 2013, 11:27 GMT
Please test release -2 for now.
Comment by Simon Sapin (SimonSapin) - Monday, 15 April 2013, 11:47 GMT
The 2.0beta2-2 seems to work for me. (I could create a virtualenv and run the cairocffi test suite.)
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 15 April 2013, 11:50 GMT
Very nice. The bug is caused by gcc incorrectly optimizing some incorrect c code. I turned it off using O1 for now. "proper" proper fix will come in with upstream fix.
Comment by Simon Sapin (SimonSapin) - Monday, 15 April 2013, 13:54 GMT
  • Field changed: Percent Complete (100% → 0%)
From what I gather on the #pypy IRC channel, -O1 probably makes PyPy horribly slow but -fno-aggressive-loop-optimizations should be enough to work around the bug.

Loading...