FS#61346 - [borg] "borg mount" doesnt work with python-msgpack 0.6.0-1

Attached to Project: Community Packages
Opened by Kai Hildebrandt (derhil) - Thursday, 10 January 2019, 22:53 GMT
Last edited by Evangelos Foutras (foutrelis) - Saturday, 25 May 2019, 17:01 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Lukas Fleischer (lfleischer)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 36
Private No

Details

Description:

$ borg mount doesn`t work with python-msgpack 0.6.0.
With python-msgpack 0.5.6-2 "borg mount" works fine.

python-msgpack 0.6.0 is not supported for borg 1.1.x, see:

https://github.com/borgbackup/borg/issues/4245#issuecomment-451695443

and

https://github.com/borgbackup/borg/issues/3899

for more info.


Additional info:
* borg 1.1.8-1
* python-msgpack 0.6.0-1


Steps to reproduce:

$ borg mount /path/to/repo /mnt
Enter passphrase for key /path/to/repo

$ ls /mnt/localhost-2019-01-10T22\:52/
ls: cannot open directory '/mnt/localhost-2019-01-10T22:52/': Input/output error

$ ls /mnt/
ls: /mnt: Transport endpoint is not connected
ls: cannot open directory '/mnt': Transport endpoint is not connected
This task depends upon

Closed by  Evangelos Foutras (foutrelis)
Saturday, 25 May 2019, 17:01 GMT
Reason for closing:  Fixed
Additional comments about closing:  borg 1.1.10-1 (upstream now bundles msgpack)
Comment by Stephan Eisvogel (bsdice) - Friday, 11 January 2019, 01:46 GMT
Backup is also broken for me, not just mount.

Workaround:
1) downgrader python-msgpack
Choose "python-msgpack-0.5.6-2 (from ALA)"

2) Add to /etc/pacman.conf
IgnorePkg = python-msgpack

3) Try again

The relationship between borg and python-msgpack is really, really fragile.
IMHO there is now enough precedent for Thomas Waldmann (borg main developer)
to include msgpack within borg. No idea if his mind can be changed on this.

[2019-01-11 01:15:26][DEBUG ][log:267] The files cache is corrupted. [File failed integrity check: /root/.cache/borg/69785bcd5fae78a3f22e43360d0cea61f393fd069cb8155442e769a242aab4f5/files]
[2019-01-11 01:15:26][DEBUG ][log:267] Continuing without files cache - expect lower performance.
[2019-01-11 01:15:28][DEBUG ][log:267] Local Exception
[2019-01-11 01:15:28][DEBUG ][log:267] Traceback (most recent call last):
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4436, in main
[2019-01-11 01:15:28][DEBUG ][log:267] exit_code = archiver.run(args)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4368, in run
[2019-01-11 01:15:28][DEBUG ][log:267] return set_ec(func(args))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 152, in wrapper
[2019-01-11 01:15:28][DEBUG ][log:267] return method(self, args, repository=repository, **kwargs)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 555, in do_create
[2019-01-11 01:15:28][DEBUG ][log:267] create_inner(archive, cache)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 518, in create_inner
[2019-01-11 01:15:28][DEBUG ][log:267] read_special=args.read_special, dry_run=dry_run, st=st)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 632, in _process
[2019-01-11 01:15:28][DEBUG ][log:267] read_special=read_special, dry_run=dry_run)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 606, in _process
[2019-01-11 01:15:28][DEBUG ][log:267] status = archive.process_file(path, st, cache)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 1028, in process_file
[2019-01-11 01:15:28][DEBUG ][log:267] self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 956, in chunk_file
[2019-01-11 01:15:28][DEBUG ][log:267] item.chunks.append(chunk_processor(data))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 944, in chunk_processor
[2019-01-11 01:15:28][DEBUG ][log:267] chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/cache.py", line 898, in add_chunk
[2019-01-11 01:15:28][DEBUG ][log:267] self.repository.put(id, data, wait=wait)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/repository.py", line 1076, in put
[2019-01-11 01:15:28][DEBUG ][log:267] self.prepare_txn(self.get_transaction_id())
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/repository.py", line 504, in prepare_txn
[2019-01-11 01:15:28][DEBUG ][log:267] hints = msgpack.unpack(fd)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/msgpack/__init__.py", line 57, in unpack
[2019-01-11 01:15:28][DEBUG ][log:267] return unpackb(data, **kwargs)
[2019-01-11 01:15:28][DEBUG ][log:267] File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
[2019-01-11 01:15:28][DEBUG ][log:267] ValueError: 35015 exceeds max_map_len(32768)
Comment by Eli Schwartz (eschwartz) - Friday, 11 January 2019, 04:27 GMT
> The relationship between borg and python-msgpack is really, really fragile. IMHO there is now enough precedent for Thomas Waldmann (borg main developer) to include msgpack within borg. No idea if his mind can be changed on this.

Untrue. The borg developer enjoys making mountains out of molehills -- this is the first time that an archlinux msgpack package has ever broken borg, but we have had like a dozen times when overly restrictive dependency specifiers in borg resulted in the package breaking with a pkg_resources.DistributionNotFound() error.

I'll note that borg master apparently has a wonderfully simple, obvious fix that the developer refuses to backport for no explanatory reason whatsoever. The appropriate response for the Arch Linux distribution is *not* to downgrade the entire universe to contort with borg's enjoyment of old software -- the solution is to do his job for him, and backport the fix.

The proper *upstream* solution is to not claim stupidity and nonsense like "we can't fix this in 1.1-maint as we allow msgpack-python 0.4.x there which does not yet have the required .tell() method" but instead acknowledge that the python programming language has useful functionality like try/except to do fallback on AttributeError if the method does not exist and use the old 0.4.x code.

Or even use:
hasattr(unpacker, 'tell') and callable(unpacker.tell)

I would have somewhat more respect for him if he were to practice the policy of honesty and say he does not wish to do so, rather than claiming it is programmatically impossible to do so.
Comment by desbma (desbma) - Sunday, 10 February 2019, 14:56 GMT
@eschwartz:
Since many Arch users are affected by this, and the Borg author seems unlikely to do anything about it, would it be possible to "vendor" the python-msgpack dependency to fix this bug?
That way the python-msgpack package would be unaffected, and borg would use an internal copy of msgpack pinned to the required previous version.
Comment by Eli Schwartz (eschwartz) - Tuesday, 12 February 2019, 03:35 GMT
@desbma,

I'm not the package maintainer. Don't ask me.
Comment by Ng Oon-Ee (ngoonee) - Thursday, 14 February 2019, 03:36 GMT
@desbma the request is much more likely to succeed if you were to include a PKGBUILD which does that. Not that I have any idea whether its acceptable or not (inclined towards thinking that it's not). Arch packages are almost always vanilla.
Comment by desbma (desbma) - Thursday, 14 February 2019, 10:29 GMT
@ngoonee
This was not a request, just an honest question.
I don't know either if there is a precedent in doing that, or how much effort it would require.

Personnally, I have hit this bug while trying to restore a backup, and immediately stopped using Borg as result. I am serious about the reliability of my backup workflow. I have switched back to rdiff-backup which is far less powerful feature wise, but it has never failed me in years. Even if the tool stop working, I can still access the files in the backup directory, unlike Borg.
Comment by Eli Schwartz (eschwartz) - Thursday, 14 February 2019, 15:16 GMT
I'd suggest instead that someone investigate if backporting https://github.com/borgbackup/borg/pull/3900 as referenced in the upstream issue, works.
Comment by loqs (loqs) - Thursday, 14 February 2019, 22:32 GMT
Passes check function.
Comment by Dave (zosodk69) - Saturday, 16 February 2019, 18:22 GMT
I've got a PKGBUILD that bundles all of the dependencies with borg. Not very Arch like but it's working well enough.
Comment by Eli Schwartz (eschwartz) - Sunday, 17 February 2019, 03:54 GMT
We don't package shady virtualenvs. PKGBUILD deleted. If that is really where you're going with this, you can always download a standalone prebuilt executable from https://github.com/borgbackup/borg/releases
Comment by waitnsea (waitnsea) - Sunday, 17 February 2019, 08:20 GMT
I had to make borg working downgrading python-msgpack to 0.5.6-2 (with aur/downgrade)
Comment by François (Frzk) - Monday, 25 February 2019, 13:06 GMT Comment by Xyne (Xyne) - Monday, 11 March 2019, 07:05 GMT
Rather than keep the software in sync with evolving dependencies, the developer has chosen to force unsupported dependencies on all of his users. Such reluctance shows that the developer is either unwilling or unable to support the software and it will eventually stop working for most users. Given that the most important aspect of any backup system is reliability, anyone who is serious about backups now has to drop it. Given the work that this will entail after having switched entirely to borg last year, I'm annoyed.

*sigh*

I've started a discussion on the forum to find an alternative backup system: https://bbs.archlinux.org/viewtopic.php?id=244846
Comment by Philipp Wolfer (phw) - Monday, 11 March 2019, 08:32 GMT
The discussion here in regard to this being a failure of upstream is a bit over the top. The situation is actually quite simple: python-msgpack 0.6 is a newer dependency not yet supported by the release version of borg, but support for this has been added already to the development tree. This is a situation a rolling release like Arch does run into all the time. It's just a matter of fact in software development that newer versions of dependencies might introduce breaking changes and downstream users of these dependencies have to adapt.

It's the responsibility of Arch to deal with this, and in fact this is dealt with all the time in different packages. There are various strategies, e.g. a) withhold a dependency update until other packages depending on it are compatible with it, b) ship two versions of the same dependency (if possible), c) patch the depending packages (ideally using existing upstream patches, or create new patches and submit them upstream), d) use a pre-release version of depending packages. Again, all of these strategies are applied at times in Arch, there might be more. For this specific case IMHO c) seems to be the way to go.
Comment by Stephan Eisvogel (bsdice) - Monday, 11 March 2019, 10:09 GMT
@phw The best solution IMO would be b) that is msgpack upstream should stop the API breakage for "msgpack0" and only break it in "msgpack1". There is enough precedent like gtk2, gtk3, or openssl (1.1+) and openssl-1.0. Arch could roll along and borg would still work nicely.

Option c) would only be my second choice because of maintenance work and potential for disaster (see Debian weak keys). You don't want package maintainers fumble too much in the code base.

Option a) is against Arch's mission statement and would in short order turn it into Ubuntu. :)

Option d) is worst, because here breakage will stem not only from incompatible dependencies but also from bugs in such raw cuts of code. Arch is imho so nice because it delivers -latest but not by default -git or -bleeding-edge.

My own solution here is e), I rolled my own python-msgpack-custom 0.5.6-3 and borgbackup-custom 1.1.9 packages with provides/conflicts/replaces stanzas in place. Got a dozen of those already, tucked into group "custom" that pacman is also specifically instructed to ignore.

Option f) turn everything into one giant statically linked Go binary. (kidding... I think)
Comment by Philipp Wolfer (phw) - Monday, 11 March 2019, 10:52 GMT
@bsdice: Actually fully agree with your post. Just noting that both a) and d) are sometimes used, but only rarely and as a short term fix. I have seen d) specifically as an alternative to pulling in a specific patch, e.g. just use release version + latest 10 git commits (which include the wanted patch plus some other random changes that the maintainer considers ok to also include). But both are not applicable in this case I think.

There are of course also options

g) just wait it out
e) remove the packages that are incompatible with the dependencies.
Comment by Mark Blakeney (bulletmark) - Wednesday, 17 April 2019, 00:53 GMT
Option h) is to simply package up the official standalone binary so I added it to the AUR as https://aur.archlinux.org/packages/borg-bin/ (created at version 1.1.9).
Comment by michael Lojkovic (zerophase) - Wednesday, 01 May 2019, 09:37 GMT
There is also this issue for replacing msgpack completely with custom solution. If anyone is able to implement that we could just avoid any future issues from msgpack updates. Some of the solutions sound like they could lead to performance improvements for borg even. https://github.com/borgbackup/borg/issues/938

From the changelogs it sounds like the new wrapper class might help with some compatibility issues in the future. I still think the alternative solutions probably fit borg more.
Comment by Adam Fontenot (amfontenot) - Wednesday, 15 May 2019, 08:54 GMT
Hi all, I am including a PKGBUILD for borg 1.1.9 that backports the fix for this problem (our version is 3 months old). It also has to bypass a check that Thomas Waldmann put in to basically refuse to run on "unsupported" versions of python-msgpack. Obviously some things may be broken with the new msgpack, but I would argue that we'd be much better off shipping this PKGBUILD than where we are now. After all, things are currently broken because we run with an unsupported msgpack. With this fix, at least the major pain point of "borg mount" being broken would be fixed.

A few new tests were added in 1.1.9 that seem to rely on an older msgpack. I must admit I'm suspicious that ThomasWaldmann added these specifically to be annoying to anyone on a newer msgpack, given their previous behavior. I've responded to this by patching out the tests entirely (though only 9 fail when run - perhaps a better fix could be backported).

Alternatively, we could continue to ship 1.1.8, but still backport the fix. Could @lfleischer take a look at this please?
   PKGBUILD (3.2 KiB)
Comment by Eli Schwartz (eschwartz) - Wednesday, 15 May 2019, 14:42 GMT
@amfontenot,

Several months ago, loqs helpfully reported that backporting that patch (for the old version of borg) resulted in a PKGBUILD that passed the check function. However, no one confirmed whether it fixed their issues. I believe loqs could not do so due to not using borg and therefore being unable to do more than rely on the testsuite.

Can you confirm that you (are the first person in the entire history of this bug report to) have tested the resulting package and confirmed that it runs correctly?
Comment by Stephan Eisvogel (bsdice) - Wednesday, 15 May 2019, 15:49 GMT
> It also has to bypass a check that Thomas Waldmann put in to basically refuse to run on "unsupported" versions of python-msgpack

Adam, bypassing the check will cause more headache down the road than it is worth. Because any new update of python-msgpack will likely again break borg. With certain pieces of software like kernel, openssl, and imho backup software we should go the extra mile to make sure it won't fail easily or preventably.

As consolation, we are not alone because openSUSE Tumbleweed has the same problem: https://github.com/borgbackup/borg/issues/4465 On the bottom the maintainer suggests to create a "msgpack5" package until borg 1.2 is released. Which as of today May 2019 is some months out according to Waldmann.

I think the best solution is to roll with borg 1.1 as stable, as intended in a rolling distribution. In addition create a package "python-msgpack-0.5.6" that will be a dependency for borg. Once borg 1.2 is out, re-evaluate the situation and if possible deprecate it, or update it to a later "python-msgpack". If borg 1.2 allows, then after 6/12 months, nuke it to clean up.

In any case the next borg package should prevent anyone from installing it, if official i.e. tested msgpack is not there and not satisfiable.
Comment by Stephan Eisvogel (bsdice) - Wednesday, 15 May 2019, 16:18 GMT
To add, I just looked at borg-master: https://github.com/borgbackup/borg/blob/master/src/borg/helpers/msgpack.py#L181
It appears for the to-be-released 1.2.0 only msgpack 0.5.6 up to 0.6.1 are known working.

Here is the same for borg 1.1.9: https://github.com/borgbackup/borg/blob/1.1.9/src/borg/helpers.py#L1293
The one common suitable python-msgpack between borg 1.1 and 1.2 appears to be 0.5.6.

Version 1.1.0 was released on 2017-10-07, 1.2.0 is imho bound for release in fall 2019 so 2 years later. With a python-msgpack-0.5.6 or python-msgpack0 or whatever one might call it, we could close this and roll borg well into 2021 until 1.3.0 is released.

Edit: Looked at wrong tree, link and text fixed.
Comment by Adam Fontenot (amfontenot) - Thursday, 16 May 2019, 06:16 GMT
>Adam, bypassing the check will cause more headache down the road than it is worth. Because any new update of python-msgpack will likely again break borg. With certain pieces of software like kernel, openssl, and imho backup software we should go the extra mile to make sure it won't fail easily or preventably. @bsdice

No, I agree with you 100% about that. What I'm offering is a temporary fix to get us out of our current position, which is - we're already shipping borg with a incompatible version of msgpack. So any problems that are going to arise from that are already here. My solution is to at least (a) fix the one known incompatibility with python-msgpack 0.6.1 (borg mount), and (b) update us to the latest stable release. We're *already* bypassing the check that exists in borg 1.1.8, incidentally.

The correct long-term solution is for us to either ship an old version of python-msgpack, or to drop borg from our repositories. But any solution is better than what we have now.

(I would add that if old versions of python-msgpack are *not* supported upstream, I would be more inclined to drop borg. Depending on a library that is unsupported is a dangerous situation that should be remedied. So if borg maintainers take stability and security seriously they will use a supported version of msgpack.)
Comment by Conrad S (conrad784) - Thursday, 16 May 2019, 06:32 GMT
Upstream is bundleling msgpack in the latest release https://github.com/borgbackup/borg/blob/1.1.10/docs/changes.rst#version-1110-2019-05-16.
Then this might the best way to got.
Comment by Jake Kreiger (Magali75) - Thursday, 16 May 2019, 10:31 GMT
Yep, updating to 1.1.10 and dropping python-msgpack would be proper the solution.
Comment by waitnsea (waitnsea) - Thursday, 16 May 2019, 11:13 GMT
My choice: bulletmark's one : "Option h) is to simply package up the official standalone binary so I added it to the AUR as https://aur.archlinux.org/packages/borg-bin/ (created at version 1.1.9)."
Comment by Jake Kreiger (Magali75) - Thursday, 16 May 2019, 19:45 GMT
Tested 1.1.10 without system python-msgpack and mounting repo works ok. Fedora adopted same approach (bundling msgpack is their idea. I attached PKGBUILD.

@lfleischer @eschwartz let's roll!!
   PKGBUILD (1.9 KiB)
Comment by Adam Fontenot (amfontenot) - Friday, 17 May 2019, 20:04 GMT
Excellent, this is great news. Definitely the right decision on their part, though it means the onus is on them (not us) if an old version of msgpack causes security issues. Looking forward to seeing 1.1.10 in the repos!
Comment by Conrad S (conrad784) - Monday, 20 May 2019, 17:45 GMT
the PKGBUILD for the current version 1.1.10 @Magali75 put up here works for me

Loading...