FS#68772 - [python-tensorflow] requires out-of-date packages

Attached to Project: Community Packages
Opened by Leon Möller (jkhsjdhjs) - Saturday, 28 November 2020, 15:54 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Thursday, 03 December 2020, 07:05 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Konstantin Gizdov (kgizdov)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
Loading tensorflow with setuptools fails, because tensorflow requires
- numpy<1.19.0,>=1.16.0 (repos have 1.19.4)
- gast==0.3.3 (repos have 0.4.0)
- h5py<2.11.0,>=2.10.0 (repos have 3.1.0)

It seems most packages don't load tensorflow with setuptools, thus the error hasn't been noticed yet.
Maybe patching the setupfile of tensorflow is an option to fix this error.
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Thursday, 03 December 2020, 07:05 GMT
Reason for closing:  Fixed
Comment by Sven-Hendrik Haase (Svenstaro) - Saturday, 28 November 2020, 17:50 GMT
Could you create and test a working patch? That would help greatly in this instance.
Comment by Leon Möller (jkhsjdhjs) - Sunday, 29 November 2020, 21:34 GMT
I have created a patch, but I can't test it, because tensorflow takes waayyy too long to compile. I had it compile for eight hours before I aborted.

On their master branch every dependency specification was changed meanwhile, so the patch will most definitely not work with the next release. See https://github.com/tensorflow/tensorflow/blob/r2.4/tensorflow/tools/pip_package/setup.py#L77
They changed the dependency on numpy from `>= 1.16.0, < 1.19.0` to `~= 1.19.2`, so we should be fine for numpy at least with the next release.
However, they're still depending on `h5py ~= 2.10.0` and `gast == 0.3.3`, so these two will still need to be patched.
Comment by Eli Schwartz (eschwartz) - Sunday, 29 November 2020, 21:52 GMT
Maybe just sed everything to remove version pinning. Is any of the version pinning at all valid?
Comment by Sven-Hendrik Haase (Svenstaro) - Sunday, 29 November 2020, 22:05 GMT
I actually like that suggestion. How about we simply get rid of the contents of `REQUIRED_PACKAGES` altogether? Not like it'll ever do us any good. We've always just given tensorflow the very newest packages to work with and if something broke we patched it and I don't see that changing.
Comment by Leon Möller (jkhsjdhjs) - Sunday, 29 November 2020, 22:09 GMT
If by valid you mean they have a valid reason to pin the version for certain packages, then probably yes.
numpy apparently had an ABI breakage, that's why they depended on numpy < 1.19.0 in the first place. Although it seems to work fine with 1.19.0+ (because there aren't any other bug reports here), maybe there were some edge cases that didn't work.
gast apparently needs to be pinned because otherwise ecosystem packages break: https://github.com/tensorflow/tensorflow/commit/710f3c83b4147eb76e748efcc218325c4978726c
Regarding h5py, I don't know why they're still depending on 2.x.

But I agree, just sed-ing everying sounds like a good solution, because we can't care about their pinned versions anyways as downgrading other repository packages is not an option.
Comment by Eli Schwartz (eschwartz) - Sunday, 29 November 2020, 22:18 GMT
I cannot fathom why a numpy ABI breakage would ever be a problem, give we compile tensorflow against the system packages and thus the ABI of whatever is present on the system. If they're doing version pinning, surely it should be due to API incompatibility, not ABI.

If python-gast is a problem on arch, we'd always have the option of reverting our package to an older version and rebuilding it with an epoch bump. So if stuff is actually incompatible then yes, we do care.
Comment by Leon Möller (jkhsjdhjs) - Sunday, 29 November 2020, 22:19 GMT
@Svenstaro: What about just getting rid of the version numbers instead of getting rid of all of `REQUIRED_PACKAGES`? This way newly added dependencies could be spotted easily, as they would result in an error if not installed while packaging.
Comment by Sven-Hendrik Haase (Svenstaro) - Sunday, 29 November 2020, 23:32 GMT
Alright I'll add a sed to get rid of the version specifications and then you'll find the new tensorflow in [community-testing] in an hour or two.

Loading...