FS#65604 - [python-pytorch-cuda] has too many dependencies

Attached to Project: Community Packages
Opened by Jacob Gorm Hansen (jacobgorm) - Sunday, 23 February 2020, 19:51 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Monday, 16 March 2020, 14:15 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Konstantin Gizdov (kgizdov)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Compared to the official releases of pytorch from pytorch.org, this package depends on QT5, FFMPEG, and OpenCV, all of which have large dependency trees by themselves. This precludes the use of arch's pytorch in kind of docker or AWS scenario.

My suggestion would be to mirror the config settings that come default with pytorch.org's PIP packages, instead of enabling every possible option.
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Monday, 16 March 2020, 14:15 GMT
Reason for closing:  Won't fix
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 24 February 2020, 00:48 GMT
As far as I can tell, pytorch actually uses all of those dependencies. Arch is not a minimal distro, despite what people say. We're a practical distro and enabling some common deps is quite reasonable to do. I'm not even sure what technical or practical reason we'd have to build it against fewer deps. I'm not sure how this makes running our pytorch in docker less worthwhile but would like you to explain that part.

Let's discuss.
Comment by Jacob Gorm Hansen (jacobgorm) - Monday, 24 February 2020, 11:12 GMT
I have been using pytorch for a couple of years and never encountered a need for QT or FFMPEG. When running with CUDA you typically don't need any of the competing Intel libs. OpenCV is debatable, the mainline PIP packages manage to do without, and you can install OpenCV for python separately if you need it. The problem with these packages is that they always end up pulling in a full desktop setup, which you don't want or need for a headless machine learning server or cloud image. Using Ubuntu and PIP for the pytorch images allows me to achieve this (but Ubuntu is painful to use in so many other ways, and the PIP images do not always work on Arch due to them lagging a bit with CUDA versions.)

Wrt using in Docker, e.g., for pushing images to the cloud, you will want as few dependencies as possible, to shorten your docker image build times and the amount of data to down- and upload.
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 24 February 2020, 11:34 GMT
Obviously I want as few hard deps as possible but I'll have to investigate whether we can do dynamic loading of deps and then have some opt deps. However, I don't see which of our deps pulls on a desktop environment.
Comment by Sven-Hendrik Haase (Svenstaro) - Monday, 16 March 2020, 14:15 GMT
Turns out pytorch doesn't support dynamic loading of libraries and I'd rather offer more features in the package than fewer.

Loading...