FS#72979 - [python] When using Py_NewInterpreter in the C API, some modules fail to import in Python 3.10

Attached to Project: Arch Linux
Opened by Jonas Witschel (diabonas) - Saturday, 11 December 2021, 12:24 GMT
Last edited by Toolybird (Toolybird) - Thursday, 14 September 2023, 03:15 GMT
Task Type Bug Report
Category Packages: Testing
Status Closed
Assigned To Felix Yan (felixonmars)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Applications using the C API for Python 3.10 in combination with Py_NewInterpreter() [1] fail to import some modules, e.g. python-jsonschema, producing a "TypeError" instead. This is a regression compared to Python 3.9, and I opened upstream bug report bpo-46047 [2] for it. This affects e.g. the weechat-matrix Python plugin, which tries to load python-jsonschema (through its dependency python-matrix-nio) and fails with the following error:

13:20 python: stdout/stderr (?): Traceback (most recent call last):
13:20 python: stdout/stderr (?): File "/home/diabonas/.local/share/weechat/python/autoload/weechat-matrix.py", line 53, in <module>
13:20 python: stdout/stderr (?): from nio import RemoteProtocolError, RemoteTransportError, TransportType
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/__init__.py", line 2, in <module>
13:20 python: stdout/stderr (?): from .client import *
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/client/__init__.py", line 3, in <module>
13:20 python: stdout/stderr (?): from .base_client import Client, ClientConfig
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/client/base_client.py", line 35, in <module>
13:20 python: stdout/stderr (?): from ..crypto import ENCRYPTION_ENABLED
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/crypto/__init__.py", line 32, in <module>
13:20 python: stdout/stderr (?): from .key_request import OutgoingKeyRequest
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/crypto/key_request.py", line 21, in <module>
13:20 python: stdout/stderr (?): from ..responses import RoomKeyRequestResponse
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/nio/responses.py", line 26, in <module>
13:20 python: stdout/stderr (?): from jsonschema.exceptions import SchemaError, ValidationError
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/jsonschema/__init__.py", line 21, in <module>
13:20 python: stdout/stderr (?): from jsonschema._types import TypeChecker
13:20 python: stdout/stderr (?): File "/usr/lib/python3.10/site-packages/jsonschema/_types.py", line 168, in <module>
13:20 python: stdout/stderr (?): draft3_type_checker = TypeChecker(
13:20 python: stdout/stderr (?): TypeError: TypeChecker() takes no arguments

Additional info:
* python 3.10.0-1
* python-jsonschema 3.2.0-6
* weechat 3.3-3
* weechat-matrix 0.3.0-4
* https://bugs.python.org/issue46047

Steps to reproduce:
1. Upgrade to Python 3.10 from [testing].
2a. Install weechat-matrix, start weechat and run "/script load weechat-matrix.py".
2b. Alternatively, compile and run the minimal example C program given in the upstream bug report [2].
3. Observe the "TypError" failures reported above.

[1] https://docs.python.org/3/c-api/init.html#c.Py_NewInterpreter
[2] https://bugs.python.org/issue46047
This task depends upon

Closed by  Toolybird (Toolybird)
Thursday, 14 September 2023, 03:15 GMT
Reason for closing:  Fixed
Additional comments about closing:  The main issue is apparently fixed in python 3.11

The SSLContext.set_npn_protocols upstream issue is still open, but it's erm, "upstream" and not an Arch packaging issue.
Comment by Jonas Witschel (diabonas) - Saturday, 11 December 2021, 13:07 GMT
I notice the Python issue had already been reported as https://bugs.python.org/issue46006
Comment by Felix Yan (felixonmars) - Saturday, 11 December 2021, 18:20 GMT
Please try again with python 3.10.1-1.
Comment by Jonas Witschel (diabonas) - Saturday, 11 December 2021, 18:45 GMT
Thank you very much for the quick fix! I confirm that this fixes the minimal C example. Attempting to load weechat-matrix still fails, but with a different error:

19:23 python: stdout/stderr (matrix): Traceback (most recent call last):
19:23 python: stdout/stderr (matrix): File "/home/jonas/.local/share/weechat/python/autoload/weechat-matrix.py", line 708, in <module>
19:23 python: stdout/stderr (matrix): create_default_server(G.CONFIG)
19:23 python: stdout/stderr (matrix): File "/usr/lib/python3.10/site-packages/matrix/server.py", line 1989, in create_default_server
19:23 python: stdout/stderr (matrix): server = MatrixServer("matrix_org", config_file._ptr)
19:23 python: stdout/stderr (matrix): File "/usr/lib/python3.10/site-packages/matrix/server.py", line 307, in __init__
19:23 python: stdout/stderr (matrix): self.ssl_context.set_npn_protocols(["h2", "http/1.1"])
19:23 python: stdout/stderr (matrix): File "/usr/lib/python3.10/ssl.py", line 546, in set_npn_protocols
19:23 python: stdout/stderr (matrix): self._set_npn_protocols(protos)
19:23 python: stdout/stderr (matrix): AttributeError: 'SSLContext' object has no attribute '_set_npn_protocols'. Did you mean: 'set_npn_protocols'?

This seems like a Python bug as well: Commit https://github.com/python/cpython/commit/39258d3595300bc7b952854c915f63ae2d4b9c3e removed "_set_npn_protocols", but is is still used here: https://github.com/python/cpython/blob/bfc59ed0a00106f5ba4a32a0c5b3dbe71d12665d/Lib/ssl.py#L551

However set_npn_protocols is deprecated anyway, so I should be able to just patch it in weechat-matrix and everything should be fine :)
Comment by Jonas Witschel (diabonas) - Saturday, 11 December 2021, 19:29 GMT
I just released weechat-matrix 0.3.0-5 with a workaround for the "AttributeError: 'SSLContext' object has no attribute '_set_npn_protocols'" bug. In combination with python 3.10.1-1, everything seems to work fine now.
Comment by Jonas Witschel (diabonas) - Saturday, 11 December 2021, 20:44 GMT
There is still one minor problem with weechat-matrix: when trying to quit WeeChat or just to unload the script (using "/script unload matrix"), WeeChat gets stuck in a deadlock without any further output and needs to be terminated using SIGKILL. My guess is that this is somehow related to the Py_NewInterpreter issue, but I am not sure how to debug this issue. The stack trace looks as follows in case this is helpful:

#0 0x00007f3b17320a8a in __futex_abstimed_wait_common () from /usr/lib/libc.so.6
#1 0x00007f3b17323504 in pthread_cond_timedwait@@GLIBC_2.3.2 () from /usr/lib/libc.so.6
#2 0x00007f3b158f3ea3 in ?? () from /usr/lib/libpython3.10.so.1.0
#3 0x00007f3b159123c2 in PyEval_RestoreThread () from /usr/lib/libpython3.10.so.1.0
#4 0x00007f3b159eaad9 in PyGILState_Ensure () from /usr/lib/libpython3.10.so.1.0
#5 0x00007f3b11e79e90 in ?? () from /usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#6 0x00007f3b10883d7a in sqlite3LeaveMutexAndCloseZombie () from /usr/lib/libsqlite3.so.0
#7 0x00007f3b10884044 in ?? () from /usr/lib/libsqlite3.so.0
#8 0x00007f3b11e75546 in ?? () from /usr/lib/python3.10/lib-dynload/_sqlite3.cpython-310-x86_64-linux-gnu.so
#9 0x00007f3b15909f44 in ?? () from /usr/lib/libpython3.10.so.1.0
#10 0x00007f3b1591e0da in _PyDict_DelItem_KnownHash () from /usr/lib/libpython3.10.so.1.0
#11 0x00007f3b158cafe7 in ?? () from /usr/lib/libpython3.10.so.1.0
#12 0x00007f3b15922ad3 in ?? () from /usr/lib/libpython3.10.so.1.0
#13 0x00007f3b159ca3d8 in ?? () from /usr/lib/libpython3.10.so.1.0
#14 0x00007f3b159cfbcf in ?? () from /usr/lib/libpython3.10.so.1.0
#15 0x00007f3b1593e5ee in PyObject_ClearWeakRefs () from /usr/lib/libpython3.10.so.1.0
#16 0x00007f3b158962d6 in ?? () from /usr/lib/libpython3.10.so.1.0
#17 0x00007f3b15909f44 in ?? () from /usr/lib/libpython3.10.so.1.0
#18 0x00007f3b159eb3a3 in PyThreadState_Clear () from /usr/lib/libpython3.10.so.1.0
#19 0x00007f3b159f188c in ?? () from /usr/lib/libpython3.10.so.1.0
#20 0x00007f3b159f0228 in ?? () from /usr/lib/libpython3.10.so.1.0
#21 0x00007f3b159fb67d in Py_EndInterpreter () from /usr/lib/libpython3.10.so.1.0
#22 0x00007f3b15b90ab6 in weechat_python_unload () from /usr/lib/weechat/plugins/python.so
#23 0x00007f3b15b90b96 in weechat_python_unload_all () from /usr/lib/weechat/plugins/python.so
#24 0x00007f3b1630975c in plugin_script_end () from /usr/lib/weechat/plugins/guile.so
#25 0x00007f3b15b91fa2 in weechat_plugin_end () from /usr/lib/weechat/plugins/python.so
#26 0x000055df26150868 in plugin_unload ()
#27 0x000055df2615091a in plugin_unload_all ()
#28 0x000055df26150bf4 in plugin_end ()
#29 0x000055df260c7004 in weechat_end ()
#30 0x000055df260c608b in main ()
Comment by Jonas Witschel (diabonas) - Monday, 13 December 2021, 16:49 GMT
Apparently the deadlock issue was already a problem in previous Python versions, albeit I have never encountered it with weechat-matrix so far: https://github.com/weechat/weechat/issues/1519 I can reproduce it with the very simple example WeeChat script given in https://github.com/weechat/weechat/issues/1519#issuecomment-646398798 so this is not a problem specific to weechat-matrix.
Comment by Jonas Witschel (diabonas) - Monday, 13 December 2021, 17:20 GMT
I also opened https://bugs.python.org/issue46067 regarding the broken SSLContext.set_npn_protocols in Python 3.10 that I had to patch out in weechat-matrix 0.3.0-5.

Loading...