FS#54959 - [python-protobuf] [python2-protobuf] cpp_implementation disabled

Attached to Project: Arch Linux
Opened by Stephen Zhang (zsrkmyn) - Friday, 28 July 2017, 19:15 GMT
Last edited by Sven-Hendrik Haase (Svenstaro) - Friday, 04 January 2019, 07:24 GMT
Task Type Feature Request
Category Packages: Extra
Status Closed
Assigned To Sven-Hendrik Haase (Svenstaro)
Lukas Fleischer (lfleischer)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 5
Private No

Details

Description:
In current python{,2}-protobuf package, `--cpp_implementation` is not enabled, which causes bad performance.
When I try load a 549MiB protobuf record, the current package in the repo takes about 8 minutes. However, after
repacking the package with `--cpp_implementation`, it takes only several seconds.

To run python{,2}-protobuf with `--cpp_implementation` compilation option requires protobuf package as its
depends, which is already satisfied in the current PKGUBILD.

Additional info:
* package version(s)
3.3.2-1

There are some discussions on performance here[1], and a building spec here[2]. And the attachment is a patch
for PKGBUILD.

How to reproduce:
In my case, the protobuf record can be found here[3], and the proto which defines the protocol can be found here[4].
They are really large things, so I list them here just for those who want to compare the performance.
To reproduce it,
1. compile the caffe.proto with `protoc --python_out=. caffe.proto`. This will produce a caffe_pb2.py
2. run `python -c 'import caffe_pb2; net = caffe_pb2.NetParameter(); net.ParseFromString(open("VGG_ILSVRC_19_layers.caffemodel", "rb").read())'`.

[1] https://github.com/google/protobuf/issues/880
[2] https://github.com/google/protobuf/blob/master/python/README.md
[3] http://www.robots.ox.ac.uk/~vgg/software/very_deep/caffe/VGG_ILSVRC_19_layers.caffemodel
[4] https://github.com/BVLC/caffe/blob/master/src/caffe/proto/caffe.proto
   patch (1 KiB)
This task depends upon

Closed by  Sven-Hendrik Haase (Svenstaro)
Friday, 04 January 2019, 07:24 GMT
Reason for closing:  Fixed
Comment by Stephen Zhang (zsrkmyn) - Friday, 28 July 2017, 19:18 GMT
Oh, I am sorry for mischoosing the Task Type :(
Comment by Eli Schwartz (eschwartz) - Friday, 28 July 2017, 19:37 GMT
  • Field changed: Task Type (Bug Report → Feature Request)
  • Field changed: Summary (cpp_implementation disabled in python{,2}-protobuf → [python-protobuf] [python2-protobuf] cpp_implementation disabled)
  • Field changed: Status (Unconfirmed → Assigned)
  • Field changed: Severity (Very Low → Medium)
  • Task assigned to Lukas Fleischer (lfleischer)
On a completely unrelated note, while looking at that PKGBUILD I couldn't help but notice each of the split packages provide and conflict *itself*. :/
Comment by xsmile (xsmile) - Sunday, 25 March 2018, 13:03 GMT
Any news on this? I don't see any downsides to enabling the Python extension.
Comment by bob (getzze) - Monday, 12 November 2018, 11:50 GMT
It is probably related to this bug:
https://bugs.archlinux.org/task/60585
Comment by Leo Mao (leomao) - Wednesday, 14 November 2018, 05:45 GMT
We need the patch https://github.com/protocolbuffers/protobuf/pull/4862 to build protobuf 3.6.1 against python 3.7.
After applying the patch, I tried to build protobuf with `--cpp_implementation` and found that I could not compile it with clang/clang++. Changing to gcc/g++ worked.
Comment by Sven-Hendrik Haase (Svenstaro) - Wednesday, 26 December 2018, 11:45 GMT
Lukas, I've gone ahead and added this flag and updated the package and put it into [testing] in order to try to fix  FS#60585 . I don't think a rebuild is required. Could you sign off on it?

Loading...