FS#67449 - [tensorflow] incompatible with openssl

Attached to Project: Community Packages
Opened by Nicola (drakkan) - Friday, 31 July 2020, 12:48 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Tuesday, 11 August 2020, 12:00 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Konstantin Gizdov (kgizdov)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Tensorflow is incompatible with any library that use openssl. It includes boringssl and if you link your application with Qt and tensorflow and/or with curl and tensorflow it will not work.

Is it possible to build tensorflow against system openssl and/or without boringssl support?


Steps to reproduce:

Please try the attached reproducer, in Qt 5.15 the message is self-explanatory, in previous versions it crashed
   TFQtReproducer.zip (1.4 KiB)
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Tuesday, 11 August 2020, 12:00 GMT
Reason for closing:  Fixed
Comment by Sven-Hendrik Haase (Svenstaro) - Friday, 31 July 2020, 16:10 GMT
I don't see any way to do that. Maybe ask upstream directly how to accomplish that?
Comment by Nicola (drakkan) - Thursday, 06 August 2020, 06:32 GMT
To fix the issue you need to add the following build options

--config=noaws --config=nogcp --config=nohdfs

this way boringssl will not be statically linked
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 08 August 2020, 00:52 GMT
That's fine for me. Check out 2.3.0-2 and see if it's as expected.
Comment by Nicola (drakkan) - Saturday, 08 August 2020, 06:06 GMT
it works fine now, thank you!
Comment by Ilango Rajagopal (ilango100) - Saturday, 08 August 2020, 11:57 GMT
Tensorflow 2.3.0-2 works for linking with OpenSSL but now with the cost of network features like GCS, AWS etc. This means something simple like

$ tensorboard --logdir gs://logbucket/path

will not work. With updated version, it gives the error:

tensorflow.python.framework.errors_impl.UnimplementedError: File system scheme 'gs' not implemented (file: 'gs://logbucket/path')

Loading remote datasets from GCS or S3 also won't work.
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 08 August 2020, 12:37 GMT
Well I don't really see any way to enable both needs. I guess I could manually unbundle boringssl and just kinda hope that it'll be maintainable. Patches most welcome.
Comment by Ilango Rajagopal (ilango100) - Sunday, 09 August 2020, 12:10 GMT
Actually the issue originally mentioned is merely a linking problem. If you add openssl as linking dependency explicitly to Qt project, it works fine with previous version. It's done by simply adding PKGCONFIG += openssl before tensorflow in project file. Most linking problems can be avoided by properly ordering the dependencies. Try the attached reproducer with previous version.
   TFQtReproducer.zip (1.3 KiB)
Comment by Sven-Hendrik Haase (Svenstaro) - Sunday, 09 August 2020, 19:17 GMT
It's not as easy as that. For security reasons, I'd very highly prefer NOT bundling boringssl. I don't really trust tensorflow to update their vendored security-critical stuff quickly enough.
Comment by Ilango Rajagopal (ilango100) - Monday, 10 August 2020, 13:55 GMT
I absolutely agree. Tensorflow's bundling a lot of libraries which should be installed through system. I found this issue https://github.com/tensorflow/tensorflow/issues/25283 which recommends to unbundle BoringSSL and good news is we can link tensorflow with OpenSSL instead of BoringSSL!

As recommended in the same issue, we can (and should) also unbundle all those stuff that are redundant in libtensorflow. The libraries which can be unbundled can be found here: https://github.com/tensorflow/tensorflow/blob/master/third_party/systemlibs/syslibs_configure.bzl
Comment by Sven-Hendrik Haase (Svenstaro) - Tuesday, 11 August 2020, 09:31 GMT
Good finds. Thanks to that, I was able to get rid of a ton of bundled dependencies. Check out 2.3.0-3 and tell me how you like that.
Comment by Ilango Rajagopal (ilango100) - Tuesday, 11 August 2020, 11:49 GMT
That's fantastic! The updated version looks good. Glad I could be of help. Now, the original issue is also solved without sacrificing any features.

$ ldd /usr/lib/libtensorflow.so
now lists more shared libraries linked to it.

Apart from that, I see that the dependencies like protobuf, grpc etc are not unbundled. I tried unbundling those, which caused build errors. I suspect that's the reason you didn't do that.
Comment by Sven-Hendrik Haase (Svenstaro) - Tuesday, 11 August 2020, 12:00 GMT
Yeah, indeed, those others turned out to be kinda hard. Anyway, I think this issue is done. If you figure out a way to use more system libs for the other deps, I'd be super glad if you could open a bug and attach a patch.

Loading...