Arch Linux

Description:

1. There are non-compiled files in /opt/pypy3/lib* that are required for using some common packages
2. There's some other stuff missing that for some unknown reason is requiring one to run pypy3 as root and with gcc available, else it complains about the lack of ``/build/pypy3/pkg/pypy3/opt/pypy3/lib_pypy/__pycache__/_cffi__ge62e22e3x1211b795.c``


Additional info:

* Testing with a simple ``importlib.import_module("sqlite3")`` in a ``test_empty.py`` file that runs in py.test is enough to fail
* The same procedure that breaks in Arch works in Ubuntu
* The package is community/pypy3 2.4.0-4
* The full configuration is attached as Dockerfiles
* All data in this bug report is stored in the https://bitbucket.org/danilobellini/bug_pypy3_arch

Steps to reproduce 1:

I was trying to use py.test 2.9.2 to test a script that requires ipython 1.2.1 by importing it with importlib.import_module, and::

OSError: [Errno 13] Permission denied: '/opt/pypy3/lib-python/3/logging/__pycache__'

Forcing compilation for the logging stdlib package was not enough, there are a lot of other modules/packages that aren't compiled, e.g. unittest, ctypes_config_cache, etc.::

OSError: [Errno 13] Permission denied: '/opt/pypy3/lib_pypy/ctypes_config_cache/__pycache__'

I've created a list of everything that is missing, but this huge sed-based command I wrote is a "pypy3 -m compileall /opt/pypy3/lib*" that actually works and seem to be simpler than being explicit about each file in the given ``lostfiles.txt``:

.. code-block:: bash

find /opt/pypy3/lib* -type f -name '*.py' | sed 's-^/opt/pypy3/lib[^/]*.3\?/\?-try: import -;s-\(/__init__\)\?\.py$-\nexcept: pass-;s-/-.-g;/-/d;s/import \S*/\n print("&")\n &/;/[. _]test/d'

The last ``/[. _]test/d`` sed command ensures that it will compile doctest and unittest but not the internal pypy3 testing suite. Obviously for a PKGBUILD the ``/opt`` would be replaced by ``${pkgdir}/opt``, but perhaps that's also a problem, and compiling the missing files isn't enough.


Steps to reproduce 2:

Solution for 1 was enough in ubuntu. But, regarding arch, the new problem is::

IOError: [Errno 2] No such file or directory: '/build/pypy3/pkg/pypy3/opt/pypy3/lib_pypy/__pycache__/_cffi__ge62e22e3x1211b795.c'

The same command worked in Ubuntu for the same pypy3 version (``Dockerfile-ubuntu`` and ``Dockerfile-arch``). I don't know why but when Arch is running as root with gcc available, it works! I presume the behavior isn't the same due to some linkage between the pre-compiled packages and Ubuntu/Arch specificities, and due to the lack of some precompiled file in the arch community/pypy3 package. The weird ``/build/pypy3/pkg/pypy3/`` prefix appeared is possibly a residual of ``${pkgdir}/opt``.

When running as root (``Dockerfile-arch-root-nogcc``), it shows the gcc requirement::

cffi.ffiplatform.VerificationError: CompileError: command 'cc' failed with exit status 1
unable to execute cc: No such file or directory

Installing gcc and running as root works (``Dockerfile-arch-root``), but running with a common user afterwards (``Dockerfile-arch-after-root``) or trying to simply recreate the pyc files manually (``Dockerfile-arch-manual``) isn't enough. But the ubuntu environment is passing with a common user.