FS#72145 - tensorflow, tensorflow-cuda, and tensorflow-opt-cuda not being detected / include files not found

Attached to Project: Community Packages
Opened by Alexander Farley (Inverarity) - Wednesday, 15 September 2021, 18:22 GMT
Last edited by Konstantin Gizdov (kgizdov) - Saturday, 16 October 2021, 14:13 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Konstantin Gizdov (kgizdov)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
The tensorflow, tensorflow-cuda, and tensorflow-opt-cuda libraries are not found when trying to link libtensorflow library into a program. That is for example: "tensorflow/c/c_api.h: No such file or directory". If however, the folders in "/usr/include/tensorflow/tensorflow" are all moved to "/usr/include/tensorflow" then the program can link to the tensorflow library. The packagekit file looks fine in terms of describing the path to the header files, but clearly something is off locally in how the path is defined going from the .pc file to the .h files. I don't have an easy way to test the actual consequences of moving the include files, but the programs do link in the libraries and compile.

Additional info:
* package version(s)
2.6.0-5
* config and/or log files etc.
* link to upstream bug report, if any

Steps to reproduce:
1.) Try to compile sample Hello_tf program from Tensorflow C website ...or ffmpeg with "--enable-libtensorflow".

2.) Output:
hello_tf.c:2:10: fatal error: tensorflow/c/c_api.h: No such file or directory
2 | #include <tensorflow/c/c_api.h>
| ^~~~~~~~~~~~~~~~~~~~~~
3.) Move folders in /usr/include/tensorflow/tensorflow up to parent directory /usr/include/tensorflow and the program compiles
This task depends upon

Closed by  Konstantin Gizdov (kgizdov)
Saturday, 16 October 2021, 14:13 GMT
Reason for closing:  Won't implement
Additional comments about closing:  Upstream do not implement an easy or transparent way for us to do this without breaking C++ API in the process. Please use the configuration we've supplied under the pkg-config tool as described in the respective issue.
Comment by Konstantin Gizdov (kgizdov) - Friday, 15 October 2021, 08:46 GMT
Are you using the the following flag `-I/usr/include/tensorflow` as indicated by `/usr/lib/pkgconfig/tensorflow.pc` `Cflags: -I/usr/include/tensorflow`. For me, the program located here `https://www.tensorflow.org/install/lang_c#example_program` builds and executes fine using the following command:
`gcc hello_tf.c $(pkg-config tensorflow --cflags) $(pkg-config tensorflow --libs) -o hello_tf`

This resolves to `gcc hello_tf.c -I/usr/include/tensorflow -ltensorflow -ltensorflow_framework -o hello_tf`. It executes as follows:

$ ./hello_tf
Hello from TensorFlow C library version 2.6.0

Notice something specific about TensorFlow itself. The reason we put the headers in `/usr/include/tensorflow` is because their dependency relies on custom versions of headers which would otherwise conflict with system headers. For example, `/usr/include/tensorflow/Eigen` is different from the one shipped in Eigen itself. This is also directly related to how TensorFlow has structured their headers, notice how they explicitly say to include their headers as `#include <tensorflow/c/c_api.h>` realising that in the root include directory their core headers are separated still. They also tell you that sometimes you'd need to include flags such as `-I/usr/local/include`. However, we do not use `/usr/local/include` in Arch packages and each package gets put in `/usr/include`. However, TensorFlow conflicts with other packages and thus goes into `/usr/include/tensorflow` to preserves its internal structure as indicated in pkg-config.

So any program that depends on TensorFlow will have to include the flag specifying the header location during compilation. If we moved `/usr/include/tensorflow/tensorflow` to `/usr/include/tensorflow`, we would get unresolved paths without `-I/usr/include/tensorflow` as some headers refer to headers shipped by Tensorflow. But if we include the flag back during compilation to fix the unresolved paths, then we will end up with conflicts, where the path resolution will be inconsistent - Will GCC look at `/usr/include/Eigen` or `/usr/include/tensorflow/Eigen`? - it's unclear. Therefore, I think moving the headers you suggest is inconsistent at best and unsupportable in the long term at worst.
Comment by Konstantin Gizdov (kgizdov) - Friday, 15 October 2021, 09:43 GMT
So I went and built https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/lib_package/README.md, which is the official way of distributing `libtensorflow.so`. It turns out I was wrong, they only ship `tensorflow/{c,core}` headers. Will fix and release as soon as it builds. Thanks for bringing this up.
Comment by Konstantin Gizdov (kgizdov) - Saturday, 16 October 2021, 14:12 GMT
Unfortunately, we have a blocker. Yes, the TensorFlow C API can be packaged only under `/usr/include`. However, after many attempts and creating a dedicated package for `libtensorflow_cc` using the upstream build system, I've determined that the C++ API cannot be packaged only under `/usr/include`. It requires many headers, and as explained in my first comment, I don't believe it to be possible to ship everything under `/usr/include`; we have to put all the relevant headers into `/usr/include/tensorflow` (or something else like `/usr/include/tf`). That is standard practice and it's also made pretty convenient using `pkg-config`. Please use the abovementioned instruction to make your software compile.

Loading...