FS#58472 - [boost] [cmake] Boost Python ImportError

Attached to Project: Arch Linux
Opened by Filip Matzner (floop) - Sunday, 06 May 2018, 10:42 GMT
Last edited by Doug Newgard (Scimmia) - Wednesday, 13 June 2018, 12:44 GMT
Task Type Bug Report
Category Packages: Extra
Status Closed
Assigned To Felix Yan (felixonmars)
Levente Polyak (anthraxx)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

With the combination of Boost 1.66.0-2 and CMake
3.11.1-1 (i.e., the current repository versions), Boost Python modules
are linked to a wrong version of Boost Python when also
linked against Boost Python Numpy.

Maybe the problem is caused by CMake 3.11.1 that supports
Boost 1.67.0 (see https://github.com/Kitware/CMake/commit/1673923c303c6a4184904c4c5849911feddb87e7#diff-555801259d7df67368f7deab1f9deacd)
that changed library naming (see https://www.boost.org/doc/libs/1_67_0/libs/python/doc/html/rn.html#rn.version_1_67).
However, with Boost 1.66.0, not only `libboost_python3`, but also `libboost_python` is linked,
unfortunately, the latter corresponds to Python 2 and causes ImportError when imported:
```
E ImportError: /usr/lib/libboost_python.so.1.66.0: undefined symbol: PyString_InternFromString
```

_Maybe_ just updating boost to the newest 1.67.0 would help. If not, this should be reported
in CMake upstream.

###################
Steps to reproduce:
###################

1. Create file bp.cpp:
```
#include <boost/python.hpp>
BOOST_PYTHON_MODULE(bp)
{
}
```

2. Create file CMakeLists.txt:
```
cmake_minimum_required(VERSION 3.3 FATAL_ERROR)
# Note that everything is fine when numpy3 is removed.
find_package(Boost REQUIRED COMPONENTS python3 numpy3)
find_package(PythonLibs 3 REQUIRED)
add_library(
bp SHARED
bp.cpp
)
set_target_properties(
bp PROPERTIES
PREFIX ""
)
target_include_directories(
bp
PRIVATE ${Boost_INCLUDE_DIRS}
PRIVATE ${PYTHON_INCLUDE_DIRS}
)
target_link_libraries(
bp
PRIVATE ${Boost_LIBRARIES}
PRIVATE ${PYTHON_LIBRARIES}
)
```

3. Run:
```
cmake . && make
python -c "import bp"
```

The import error is thrown. `ldd bp.so` shows that also `libboost_python` is linked.
This task depends upon

Closed by  Doug Newgard (Scimmia)
Wednesday, 13 June 2018, 12:44 GMT
Reason for closing:  Fixed
Comment by Doug Newgard (Scimmia) - Monday, 07 May 2018, 01:28 GMT
There's no way something is getting linked against a version of a library that doesn't exist. What's the exact error you're getting?

Edit: Ok, I see what you're saying now. Does sound like a cmake issue, assuming it's actually linked. Don't use ldd as it's recursive, check objdump or readelf.
Comment by Filip Matzner (floop) - Monday, 07 May 2018, 07:07 GMT
Sorry if the description was not sufficiently clear. To rephrase it simply, when CMake is asked to link Boost Python NumPy 3, it also links Boost Python 2. The generated module therefore cannot be imported from Python 3.

The generated module is really linked to both Boost Python versions, it can be seen when `make` is called with `VERBOSE=1`. The compiler command says `c++ ... -lboost_python3 ... -lboost_python ...`. Readelf confirms:
```
[floop@pear /tmp ]$ readelf -d bp.so

Dynamic section at offset 0x1d88 contains 32 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: [libboost_python3.so.1.66.0]
0x0000000000000001 (NEEDED) Shared library: [libboost_numpy3.so.1.66.0]
0x0000000000000001 (NEEDED) Shared library: [libboost_python.so.1.66.0] # This should not be here
...
```

And yes, it is a CMake issue, but maybe it is caused just by incompatible versions of CMake and Boost. On the other hand, CMake should work even with older versions of Boost, so it should probably be reported upstream anyway.
Comment by Filip Matzner (floop) - Monday, 07 May 2018, 09:11 GMT
So I checked the CMake code and updating Boost to the most recent v1.67.0 should fix the issue.

For older Boost, I made a quick PR: https://gitlab.kitware.com/cmake/cmake/merge_requests/2045.
Comment by Filip Matzner (floop) - Friday, 11 May 2018, 14:05 GMT
The CMake PR was merged to master, so the issue will be fixed with the next release of CMake (3.11.2).
(Alternatively, it can also be fixed by updating Boost to 1.67.0)
Comment by Filip Matzner (floop) - Wednesday, 13 June 2018, 08:58 GMT
This is fixed and ready to be closed. (Now Boost Python NumPy is not working because of  FS#58881  instead. :-))

Loading...