Community Packages

Please read this before reporting a bug:
https://wiki.archlinux.org/title/Bug_reporting_guidelines

Do NOT report bugs when a package is just outdated, or it is in the AUR. Use the 'flag out of date' link on the package page, or the Mailing List.

REPEAT: Do NOT report bugs for outdated packages!
Tasklist

FS#65239 - [python-boto] Upstream bug in py3 compatibilty has broken duplicity usage

Attached to Project: Community Packages
Opened by Troy Engel (TE) - Wednesday, 22 January 2020, 16:24 GMT
Last edited by Felix Yan (felixonmars) - Saturday, 07 March 2020, 21:11 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Felix Yan (felixonmars)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:

Hi packagers,

I recently opened a bug against python2-boto (https://bugs.archlinux.org/task/65036) failing to SNI against Google Cloud Storage while using Duplicity; the decision was made to upgrade duplicity and re-parent on python-boto now (great!). The problem is that the release version of python-boto is _more_ broken than the older python2-boto was. :(

The new problem is a py2/p3 compatibility error which has been fixed upstream in Github, but has not made it to a release. The error encountered at runtime:

=================
Traceback (innermost last):
File "/usr/bin/duplicity", line 101, in <module>
with_tempdir(main)
File "/usr/bin/duplicity", line 87, in with_tempdir
fn()
File "/usr/lib/python3.8/site-packages/duplicity/dup_main.py", line 1525, in main
action = commandline.ProcessCommandLine(sys.argv[1:])
File "/usr/lib/python3.8/site-packages/duplicity/commandline.py", line 1187, in ProcessCommandLine
backup, local_pathname = set_backend(args[0], args[1])
File "/usr/lib/python3.8/site-packages/duplicity/commandline.py", line 1058, in set_backend
globals.backend = backend.get_backend(bend)
File "/usr/lib/python3.8/site-packages/duplicity/backend.py", line 225, in get_backend
obj = get_backend_object(url_string)
File "/usr/lib/python3.8/site-packages/duplicity/backend.py", line 211, in get_backend_object
return factory(pu)
File "/usr/lib/python3.8/site-packages/duplicity/backends/_boto_single.py", line 173, in __init__
self.resetConnection()
File "/usr/lib/python3.8/site-packages/duplicity/backends/_boto_single.py", line 198, in resetConnection
self.bucket = self.conn.create_bucket(self.bucket_name,
File "/usr/lib/python3.8/site-packages/boto/gs/connection.py", line 93, in create_bucket
response = self.make_request(
File "/usr/lib/python3.8/site-packages/boto/s3/connection.py", line 659, in make_request
auth_path = self.calling_format.build_auth_path(bucket, key)
File "/usr/lib/python3.8/site-packages/boto/s3/connection.py", line 94, in build_auth_path
path = '/' + bucket
TypeError: can only concatenate str (not "bytes") to str
=================

There is a commit upstream which fixes this, but it's interlaced with a number of other commits fixing other (related?) problems in python3 compatibility: https://github.com/boto/boto/commit/a709f81fd6a8ece8ce74ccf61a8911bf01d5b9fc#diff-9c1010347019c4546ff1604cc7b569bd Specifically in that commit it's the change to s3/connection.py (shown as line 92) which adds a UTF-8 decode wrapper around this broken piece of code.

In upgrading duplicity to use py3 + python-boto instead of py2 + python2-boto, my backups are now completely dead due to upstream bugs. I'm not entirely sure if we can cull out just one piece of a commit to try and fix this very specific issue or if that would wreck more than it would solve, hard to say. (I'm not a programmer by trade)

Thanks for listening,
-te

Additional info:
* package version(s)

python-boto 2.49.0-3

This task depends upon

Closed by  Felix Yan (felixonmars)
Saturday, 07 March 2020, 21:11 GMT
Reason for closing:  Fixed
Additional comments about closing:  2.49.0.20190327-1
Comment by loqs (loqs) - Wednesday, 22 January 2020, 20:43 GMT
Is using the boto3 backend (python-boto3) with the duplicty 0.8.09-4 less broken?
Comment by Troy Engel (TE) - Thursday, 23 January 2020, 03:57 GMT
Looking through the docs, I'll need a bit of time to look at this later - there appears to be a new duplicity backend created for boto3 in the most recent releases, but all the info is focusing on using S3 with little mention of using it with GS. The conversion isn't just a simple drop-in based on what I'm able to review tonight, it required new coding. (tl;dr - you name your target bucket "s3://" to use boto, but "boto3+s3://" to trigger the new boto3 specific code/class site-packages/duplicity/backends/s3_boto3_backend.py, but GS buckets are named "gs://" with no visible mention of a new "boto3+gs://" to trigger boto3 in the docs)
Comment by loqs (loqs) - Friday, 24 January 2020, 00:25 GMT
Does applying the attached patch work?
It cherry-picks the following commits

a709f81fd6a8ece8ce74ccf61a8911bf01d5b9fc Python 2 / 3 compatibility fixes
c046cac2379d5b38e0f495d51cd598c9c923e00d Fixed wget snafu porting from last PR
d4761521a296dd85e0cf06925afcfabd1c3d03aa Fix test failures introduced in #3855 and #3866.
c660c74460c9ca882bf5df60a3085be31a909f04 A sneaky .encode() snuck back in
4a27f95e83ddff93b75ea64a043e215a8706f644 Update six to 1.12
   test.patch (19.9 KiB)
Comment by Troy Engel (TE) - Friday, 24 January 2020, 15:15 GMT
Apologies on the delay, I did not get a chance to work on this last night after work. I'll test it this weekend.
Comment by Troy Engel (TE) - Saturday, 25 January 2020, 13:13 GMT
I removed the first part of the patch trying to alter .gitignore real quick, the patch applied cleanly. This fixed that specific issue, but I _think_ it's now running into the same SNI problem as before with py2, and interestingly it seems to pause during this failure cycle for about 2.5mins or so. Ignoring the leading part of the stack exception:

```
File "/usr/lib/python3.8/site-packages/boto/https_connection.py", line 128, in
connect
self.sock = ssl.wrap_socket(sock, keyfile=self.key_file,
File "/usr/lib/python3.8/ssl.py", line 1405, in wrap_socket
return context.wrap_socket(
File "/usr/lib/python3.8/ssl.py", line 500, in wrap_socket
return self.sslsocket_class._create(
File "/usr/lib/python3.8/ssl.py", line 1040, in _create
self.do_handshake()
File "/usr/lib/python3.8/ssl.py", line 1309, in do_handshake
self._sslobj.do_handshake()
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate veri
fy failed: self signed certificate (_ssl.c:1108)

06:47:57.418 Task 'BKP' failed with exit code '30'.
--- Finished state FAILED 'code 30' at 06:47:57.418 - Runtime 00:02:35.713 ---
```

I've joined the duplicity-talk mailing list and posted a message asking about how to migrate a GS bucket over to the newer boto3, a few "test the obvious" didn't work out so I fear there may not be code written yet. The boto3+s3:// is pretty new code upstream so it stands to reason, work in progress.
Comment by Troy Engel (TE) - Saturday, 25 January 2020, 14:13 GMT
A reply from upstream, duplicity is not yet capable of using boto3 for the Google buckets. I'm conflicted if we should try continuing to beat on py3 boto by cherry picking patches, or I just downgrade duplicity to the last py2 boto package and lock it on my end until upstream receives new donated code for boto3+gs:// usage.
Comment by loqs (loqs) - Saturday, 25 January 2020, 17:38 GMT
I am willing to continue working on the issue if you are.
Added 79142a02aa14ba74d66c5a8bf7daf2829892abf9 Send SNI for HTTPS connections
I left the patch for .gitignore in as it applies cleanly with the following PKGBUILD.
   test.patch (21.8 KiB)
   PKGBUILD (2.1 KiB)
Comment by Troy Engel (TE) - Saturday, 25 January 2020, 18:26 GMT
I think this helps fix all the boto issues, but I get new error (tried twice) which I _think_ is unrelated, all that the logs are showing:

Attempt 1 failed. TypeError: a bytes-like object is required, not 'str'
Error processing remote manifest (duplicity-inc.20200125T154957Z.to.20200125T182034Z.manifest.gpg): GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: Unknown system error
===== End GnuPG log =====

This could be due to the previous code not working and it somehow corrupted that very specific manifest file, but each of the two runs complained about a different manifest. This may just be some sort of new duplicity bug and unrelated to boto -- it _appears_ to be connecting to Google Storage now, and it can list and see all my backups and so forth. Maybe we got this part sorted?
Comment by loqs (loqs) - Saturday, 25 January 2020, 19:13 GMT
Attached is a PKGBUILD for python-boto-git needs the same boto-python-3.8.patch as python-boto.
Does that also produce the new issue.
   PKGBUILD (1.9 KiB)
Comment by Troy Engel (TE) - Sunday, 26 January 2020, 13:47 GMT
Yes, using the git PKGBUILD (I added a quick conflicts/replaces=('python-boto') in the pkg function, FYI) with the 3.8 patch has gotten rid of my errors and the "death pause" that seemed to be happening. I was able to twice run a full routine (a backup, then a status check all using duply as a meta wrapper) without apparent errors (woo) and normal logs. I think this does tell us there are sadly more outstanding bugfixes to cherry pick which is just no fun. :-/

Personally I'm OK running the python-boto-git package if we added it to AUR, I realize this old library has been abandoned and there are choices to be made. I'm happy to add this -git to my AUR maintainer stack if you want, lemme know. I'd drop the check() routines, in 5+ years on this system I've never needed the 8 packages I just had to install to test the above PKGBUILD. :) (python-nose python2-nose python-mock python2-mock python-httpretty python2-httpretty python-pbr python2-pbr)
Comment by loqs (loqs) - Sunday, 26 January 2020, 23:26 GMT
Updated the PKGBUILD for python-boto-git this pins on commit https://github.com/boto/boto/commit/d4761521a296dd85e0cf06925afcfabd1c3d03aa
Does this version still work?
   PKGBUILD (2 KiB)
Comment by Troy Engel (TE) - Tuesday, 28 January 2020, 00:17 GMT
No, sadly not - it has the same secondary error with the py3 TypeError:

Attempt 1 failed. TypeError: a bytes-like object is required, not 'str'
Error processing remote manifest (duplicity-inc.20200126T132821Z.to.20200126T133
455Z.manifest.gpg): GPG Failed, see log below:
===== Begin GnuPG log =====
gpg: decrypt_message failed: Unknown system error
===== End GnuPG log =====

A quick re-upgrade to the tip revision package makes it work again as expected (so it appears it's not corrupting manifest files by accident). As I'm running a layered solution (a bash script calling duply which calls duplicity which is using this library) I'm not getting a classic stack trace to know which actual line of code is causing the problem - one of the layers is eating that output and sanitizing the error on us, I'll need to find time later to go down this rabbit hole.
Comment by loqs (loqs) - Tuesday, 28 January 2020, 00:50 GMT
There are four merge requests after d4761521a296dd85e0cf06925afcfabd1c3d03aa which look to contain all the commits up to head
Can you change the commit in the PKGBUILD to the values below and find the first one that works
9e1cd3bd76e738d80630f1bd9160fd87c8eab865
2f202c95b8021d56466ebd62383664a1cf3992a0
03b2268348ea81d80e3e5ddea0970f4968561010
91ba037e54ef521c379263b0ac769c66182527d7
Comment by Troy Engel (TE) - Wednesday, 29 January 2020, 00:13 GMT
The first commit 9e1cd3bd76e738d80630f1bd9160fd87c8eab865 worked, I didn't even need to go any further. :) Using this one, the errors are gone and no death pause -backups/logs look normal.
Comment by Troy Engel (TE) - Sunday, 23 February 2020, 14:53 GMT
Minor update and alternate solution - the above commit ID has been working OK for about a month now.

Upstream duplicity hasn't seen any movement in this area (boto3+gs://) and the moving target aspect of Boto(3)+GCloud supportability made me uneasy, so I've switch my backend to rclone:// and am (successfully) testing it with duplicity/duply. I simply added the same remote bucket as boto was already using and updated my duplicity/duply config from "gs://bucket" to "rclone://remote:bucket". Other than specifying the name of my remote project when prompted, I accepted all defaults during `rclone config` setup: https://rclone.org/googlecloudstorage/
Comment by Felix Yan (felixonmars) - Saturday, 07 March 2020, 21:10 GMT
Sorry for leaving it open for so long. Since upstream looks quite inactive and the request to make a new release was unanswered for ~2 months, I'm going to bump boto now. Thanks for all the efforts to fix this!

Loading...