Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines
Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.
REPEAT: Do NOT report bugs for outdated packages!
https://wiki.archlinux.org/title/Bug_reporting_guidelines
Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.
REPEAT: Do NOT report bugs for outdated packages!
FS#65239 - [python-boto] Upstream bug in py3 compatibilty has broken duplicity usage
Attached to Project:
Community Packages
Opened by Troy Engel (TE) - Wednesday, 22 January 2020, 16:24 GMT
Last edited by Felix Yan (felixonmars) - Saturday, 07 March 2020, 21:11 GMT
Opened by Troy Engel (TE) - Wednesday, 22 January 2020, 16:24 GMT
Last edited by Felix Yan (felixonmars) - Saturday, 07 March 2020, 21:11 GMT
|
DetailsDescription:
Hi packagers, I recently opened a bug against python2-boto (https://bugs.archlinux.org/task/65036) failing to SNI against Google Cloud Storage while using Duplicity; the decision was made to upgrade duplicity and re-parent on python-boto now (great!). The problem is that the release version of python-boto is _more_ broken than the older python2-boto was. :( The new problem is a py2/p3 compatibility error which has been fixed upstream in Github, but has not made it to a release. The error encountered at runtime: ================= Traceback (innermost last): File "/usr/bin/duplicity", line 101, in <module> with_tempdir(main) File "/usr/bin/duplicity", line 87, in with_tempdir fn() File "/usr/lib/python3.8/site-packages/duplicity/dup_main.py", line 1525, in main action = commandline.ProcessCommandLine(sys.argv[1:]) File "/usr/lib/python3.8/site-packages/duplicity/commandline.py", line 1187, in ProcessCommandLine backup, local_pathname = set_backend(args[0], args[1]) File "/usr/lib/python3.8/site-packages/duplicity/commandline.py", line 1058, in set_backend globals.backend = backend.get_backend(bend) File "/usr/lib/python3.8/site-packages/duplicity/backend.py", line 225, in get_backend obj = get_backend_object(url_string) File "/usr/lib/python3.8/site-packages/duplicity/backend.py", line 211, in get_backend_object return factory(pu) File "/usr/lib/python3.8/site-packages/duplicity/backends/_boto_single.py", line 173, in __init__ self.resetConnection() File "/usr/lib/python3.8/site-packages/duplicity/backends/_boto_single.py", line 198, in resetConnection self.bucket = self.conn.create_bucket(self.bucket_name, File "/usr/lib/python3.8/site-packages/boto/gs/connection.py", line 93, in create_bucket response = self.make_request( File "/usr/lib/python3.8/site-packages/boto/s3/connection.py", line 659, in make_request auth_path = self.calling_format.build_auth_path(bucket, key) File "/usr/lib/python3.8/site-packages/boto/s3/connection.py", line 94, in build_auth_path path = '/' + bucket TypeError: can only concatenate str (not "bytes") to str ================= There is a commit upstream which fixes this, but it's interlaced with a number of other commits fixing other (related?) problems in python3 compatibility: https://github.com/boto/boto/commit/a709f81fd6a8ece8ce74ccf61a8911bf01d5b9fc#diff-9c1010347019c4546ff1604cc7b569bd Specifically in that commit it's the change to s3/connection.py (shown as line 92) which adds a UTF-8 decode wrapper around this broken piece of code. In upgrading duplicity to use py3 + python-boto instead of py2 + python2-boto, my backups are now completely dead due to upstream bugs. I'm not entirely sure if we can cull out just one piece of a commit to try and fix this very specific issue or if that would wreck more than it would solve, hard to say. (I'm not a programmer by trade) Thanks for listening, -te Additional info: * package version(s) python-boto 2.49.0-3 |
This task depends upon
Closed by Felix Yan (felixonmars)
Saturday, 07 March 2020, 21:11 GMT
Reason for closing: Fixed
Additional comments about closing: 2.49.0.20190327-1
Saturday, 07 March 2020, 21:11 GMT
Reason for closing: Fixed
Additional comments about closing: 2.49.0.20190327-1
It cherry-picks the following commits
a709f81fd6a8ece8ce74ccf61a8911bf01d5b9fc Python 2 / 3 compatibility fixes
c046cac2379d5b38e0f495d51cd598c9c923e00d Fixed wget snafu porting from last PR
d4761521a296dd85e0cf06925afcfabd1c3d03aa Fix test failures introduced in #3855 and #3866.
c660c74460c9ca882bf5df60a3085be31a909f04 A sneaky .encode() snuck back in
4a27f95e83ddff93b75ea64a043e215a8706f644 Update six to 1.12
```
File "/usr/lib/python3.8/site-packages/boto/https_connection.py", line 128, in
connect
self.sock = ssl.wrap_socket(sock, keyfile=self.key_file,
File "/usr/lib/python3.8/ssl.py", line 1405, in wrap_socket
return context.wrap_socket(
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate veri
fy failed: self signed certificate (_ssl.c:1108)
06:47:57.418 Task 'BKP' failed with exit code '30'.
--- Finished state FAILED 'code 30' at 06:47:57.418 - Runtime 00:02:35.713 ---
```
I've joined the duplicity-talk mailing list and posted a message asking about how to migrate a GS bucket over to the newer boto3, a few "test the obvious" didn't work out so I fear there may not be code written yet. The boto3+s3:// is pretty new code upstream so it stands to reason, work in progress.
Added 79142a02aa14ba74d66c5a8bf7daf2829892abf9 Send SNI for HTTPS connections
I left the patch for .gitignore in as it applies cleanly with the following PKGBUILD.
Attempt 1 failed. TypeError: a bytes-like object is required, not 'str'
Error processing remote manifest (duplicity-inc.20200125T154957Z.to.20200125T182034Z.manifest.gpg): GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: Unknown system error
===== End GnuPG log =====
This could be due to the previous code not working and it somehow corrupted that very specific manifest file, but each of the two runs complained about a different manifest. This may just be some sort of new duplicity bug and unrelated to boto -- it _appears_ to be connecting to Google Storage now, and it can list and see all my backups and so forth. Maybe we got this part sorted?
Does that also produce the new issue.
Personally I'm OK running the python-boto-git package if we added it to AUR, I realize this old library has been abandoned and there are choices to be made. I'm happy to add this -git to my AUR maintainer stack if you want, lemme know. I'd drop the check() routines, in 5+ years on this system I've never needed the 8 packages I just had to install to test the above PKGBUILD. :) (python-nose python2-nose python-mock python2-mock python-httpretty python2-httpretty python-pbr python2-pbr)
Does this version still work?
Attempt 1 failed. TypeError: a bytes-like object is required, not 'str'
Error processing remote manifest (duplicity-inc.20200126T132821Z.to.20200126T133
455Z.manifest.gpg): GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: Unknown system error
===== End GnuPG log =====
A quick re-upgrade to the tip revision package makes it work again as expected (so it appears it's not corrupting manifest files by accident). As I'm running a layered solution (a bash script calling duply which calls duplicity which is using this library) I'm not getting a classic stack trace to know which actual line of code is causing the problem - one of the layers is eating that output and sanitizing the error on us, I'll need to find time later to go down this rabbit hole.
Can you change the commit in the PKGBUILD to the values below and find the first one that works
9e1cd3bd76e738d80630f1bd9160fd87c8eab865
2f202c95b8021d56466ebd62383664a1cf3992a0
03b2268348ea81d80e3e5ddea0970f4968561010
91ba037e54ef521c379263b0ac769c66182527d7
Upstream duplicity hasn't seen any movement in this area (boto3+gs://) and the moving target aspect of Boto(3)+GCloud supportability made me uneasy, so I've switch my backend to rclone:// and am (successfully) testing it with duplicity/duply. I simply added the same remote bucket as boto was already using and updated my duplicity/duply config from "gs://bucket" to "rclone://remote:bucket". Other than specifying the name of my remote project when prompted, I accepted all defaults during `rclone config` setup: https://rclone.org/googlecloudstorage/