FS#61346 - [borg] "borg mount" doesnt work with python-msgpack 0.6.0-1
Attached to Project:
Community Packages
Opened by Kai Hildebrandt (derhil) - Thursday, 10 January 2019, 22:53 GMT
Last edited by Evangelos Foutras (foutrelis) - Saturday, 25 May 2019, 17:01 GMT
Opened by Kai Hildebrandt (derhil) - Thursday, 10 January 2019, 22:53 GMT
Last edited by Evangelos Foutras (foutrelis) - Saturday, 25 May 2019, 17:01 GMT
|
Details
Description:
$ borg mount doesn`t work with python-msgpack 0.6.0. With python-msgpack 0.5.6-2 "borg mount" works fine. python-msgpack 0.6.0 is not supported for borg 1.1.x, see: https://github.com/borgbackup/borg/issues/4245#issuecomment-451695443 and https://github.com/borgbackup/borg/issues/3899 for more info. Additional info: * borg 1.1.8-1 * python-msgpack 0.6.0-1 Steps to reproduce: $ borg mount /path/to/repo /mnt Enter passphrase for key /path/to/repo $ ls /mnt/localhost-2019-01-10T22\:52/ ls: cannot open directory '/mnt/localhost-2019-01-10T22:52/': Input/output error $ ls /mnt/ ls: /mnt: Transport endpoint is not connected ls: cannot open directory '/mnt': Transport endpoint is not connected |
This task depends upon
Closed by Evangelos Foutras (foutrelis)
Saturday, 25 May 2019, 17:01 GMT
Reason for closing: Fixed
Additional comments about closing: borg 1.1.10-1 (upstream now bundles msgpack)
Saturday, 25 May 2019, 17:01 GMT
Reason for closing: Fixed
Additional comments about closing: borg 1.1.10-1 (upstream now bundles msgpack)
Workaround:
1) downgrader python-msgpack
Choose "python-msgpack-0.5.6-2 (from ALA)"
2) Add to /etc/pacman.conf
IgnorePkg = python-msgpack
3) Try again
The relationship between borg and python-msgpack is really, really fragile.
IMHO there is now enough precedent for Thomas Waldmann (borg main developer)
to include msgpack within borg. No idea if his mind can be changed on this.
[2019-01-11 01:15:26][DEBUG ][log:267] The files cache is corrupted. [File failed integrity check: /root/.cache/borg/69785bcd5fae78a3f22e43360d0cea61f393fd069cb8155442e769a242aab4f5/files]
[2019-01-11 01:15:26][DEBUG ][log:267] Continuing without files cache - expect lower performance.
[2019-01-11 01:15:28][DEBUG ][log:267] Local Exception
[2019-01-11 01:15:28][DEBUG ][log:267] Traceback (most recent call last):
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4436, in main
[2019-01-11 01:15:28][DEBUG ][log:267] exit_code = archiver.run(args)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 4368, in run
[2019-01-11 01:15:28][DEBUG ][log:267] return set_ec(func(args))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 152, in wrapper
[2019-01-11 01:15:28][DEBUG ][log:267] return method(self, args, repository=repository, **kwargs)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 555, in do_create
[2019-01-11 01:15:28][DEBUG ][log:267] create_inner(archive, cache)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 518, in create_inner
[2019-01-11 01:15:28][DEBUG ][log:267] read_special=args.read_special, dry_run=dry_run, st=st)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 632, in _process
[2019-01-11 01:15:28][DEBUG ][log:267] read_special=read_special, dry_run=dry_run)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archiver.py", line 606, in _process
[2019-01-11 01:15:28][DEBUG ][log:267] status = archive.process_file(path, st, cache)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 1028, in process_file
[2019-01-11 01:15:28][DEBUG ][log:267] self.chunk_file(item, cache, self.stats, backup_io_iter(self.chunker.chunkify(fd, fh)))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 956, in chunk_file
[2019-01-11 01:15:28][DEBUG ][log:267] item.chunks.append(chunk_processor(data))
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/archive.py", line 944, in chunk_processor
[2019-01-11 01:15:28][DEBUG ][log:267] chunk_entry = cache.add_chunk(self.key.id_hash(data), data, stats, wait=False)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/cache.py", line 898, in add_chunk
[2019-01-11 01:15:28][DEBUG ][log:267] self.repository.put(id, data, wait=wait)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/repository.py", line 1076, in put
[2019-01-11 01:15:28][DEBUG ][log:267] self.prepare_txn(self.get_transaction_id())
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/borg/repository.py", line 504, in prepare_txn
[2019-01-11 01:15:28][DEBUG ][log:267] hints = msgpack.unpack(fd)
[2019-01-11 01:15:28][DEBUG ][log:267] File "/usr/lib/python3.7/site-packages/msgpack/__init__.py", line 57, in unpack
[2019-01-11 01:15:28][DEBUG ][log:267] return unpackb(data, **kwargs)
[2019-01-11 01:15:28][DEBUG ][log:267] File "msgpack/_unpacker.pyx", line 187, in msgpack._cmsgpack.unpackb
[2019-01-11 01:15:28][DEBUG ][log:267] ValueError: 35015 exceeds max_map_len(32768)
Untrue. The borg developer enjoys making mountains out of molehills -- this is the first time that an archlinux msgpack package has ever broken borg, but we have had like a dozen times when overly restrictive dependency specifiers in borg resulted in the package breaking with a pkg_resources.DistributionNotFound() error.
I'll note that borg master apparently has a wonderfully simple, obvious fix that the developer refuses to backport for no explanatory reason whatsoever. The appropriate response for the Arch Linux distribution is *not* to downgrade the entire universe to contort with borg's enjoyment of old software -- the solution is to do his job for him, and backport the fix.
The proper *upstream* solution is to not claim stupidity and nonsense like "we can't fix this in 1.1-maint as we allow msgpack-python 0.4.x there which does not yet have the required .tell() method" but instead acknowledge that the python programming language has useful functionality like try/except to do fallback on AttributeError if the method does not exist and use the old 0.4.x code.
Or even use:
hasattr(unpacker, 'tell') and callable(unpacker.tell)
I would have somewhat more respect for him if he were to practice the policy of honesty and say he does not wish to do so, rather than claiming it is programmatically impossible to do so.
Since many Arch users are affected by this, and the Borg author seems unlikely to do anything about it, would it be possible to "vendor" the python-msgpack dependency to fix this bug?
That way the python-msgpack package would be unaffected, and borg would use an internal copy of msgpack pinned to the required previous version.
I'm not the package maintainer. Don't ask me.
This was not a request, just an honest question.
I don't know either if there is a precedent in doing that, or how much effort it would require.
Personnally, I have hit this bug while trying to restore a backup, and immediately stopped using Borg as result. I am serious about the reliability of my backup workflow. I have switched back to rdiff-backup which is far less powerful feature wise, but it has never failed me in years. Even if the tool stop working, I can still access the files in the backup directory, unlike Borg.
*sigh*
I've started a discussion on the forum to find an alternative backup system: https://bbs.archlinux.org/viewtopic.php?id=244846
It's the responsibility of Arch to deal with this, and in fact this is dealt with all the time in different packages. There are various strategies, e.g. a) withhold a dependency update until other packages depending on it are compatible with it, b) ship two versions of the same dependency (if possible), c) patch the depending packages (ideally using existing upstream patches, or create new patches and submit them upstream), d) use a pre-release version of depending packages. Again, all of these strategies are applied at times in Arch, there might be more. For this specific case IMHO c) seems to be the way to go.
Option c) would only be my second choice because of maintenance work and potential for disaster (see Debian weak keys). You don't want package maintainers fumble too much in the code base.
Option a) is against Arch's mission statement and would in short order turn it into Ubuntu. :)
Option d) is worst, because here breakage will stem not only from incompatible dependencies but also from bugs in such raw cuts of code. Arch is imho so nice because it delivers -latest but not by default -git or -bleeding-edge.
My own solution here is e), I rolled my own python-msgpack-custom 0.5.6-3 and borgbackup-custom 1.1.9 packages with provides/conflicts/replaces stanzas in place. Got a dozen of those already, tucked into group "custom" that pacman is also specifically instructed to ignore.
Option f) turn everything into one giant statically linked Go binary. (kidding... I think)
There are of course also options
g) just wait it out
e) remove the packages that are incompatible with the dependencies.
From the changelogs it sounds like the new wrapper class might help with some compatibility issues in the future. I still think the alternative solutions probably fit borg more.
A few new tests were added in 1.1.9 that seem to rely on an older msgpack. I must admit I'm suspicious that ThomasWaldmann added these specifically to be annoying to anyone on a newer msgpack, given their previous behavior. I've responded to this by patching out the tests entirely (though only 9 fail when run - perhaps a better fix could be backported).
Alternatively, we could continue to ship 1.1.8, but still backport the fix. Could @lfleischer take a look at this please?
Several months ago, loqs helpfully reported that backporting that patch (for the old version of borg) resulted in a PKGBUILD that passed the check function. However, no one confirmed whether it fixed their issues. I believe loqs could not do so due to not using borg and therefore being unable to do more than rely on the testsuite.
Can you confirm that you (are the first person in the entire history of this bug report to) have tested the resulting package and confirmed that it runs correctly?
Adam, bypassing the check will cause more headache down the road than it is worth. Because any new update of python-msgpack will likely again break borg. With certain pieces of software like kernel, openssl, and imho backup software we should go the extra mile to make sure it won't fail easily or preventably.
As consolation, we are not alone because openSUSE Tumbleweed has the same problem: https://github.com/borgbackup/borg/issues/4465 On the bottom the maintainer suggests to create a "msgpack5" package until borg 1.2 is released. Which as of today May 2019 is some months out according to Waldmann.
I think the best solution is to roll with borg 1.1 as stable, as intended in a rolling distribution. In addition create a package "python-msgpack-0.5.6" that will be a dependency for borg. Once borg 1.2 is out, re-evaluate the situation and if possible deprecate it, or update it to a later "python-msgpack". If borg 1.2 allows, then after 6/12 months, nuke it to clean up.
In any case the next borg package should prevent anyone from installing it, if official i.e. tested msgpack is not there and not satisfiable.
It appears for the to-be-released 1.2.0 only msgpack 0.5.6 up to 0.6.1 are known working.
Here is the same for borg 1.1.9: https://github.com/borgbackup/borg/blob/1.1.9/src/borg/helpers.py#L1293
The one common suitable python-msgpack between borg 1.1 and 1.2 appears to be 0.5.6.
Version 1.1.0 was released on 2017-10-07, 1.2.0 is imho bound for release in fall 2019 so 2 years later. With a python-msgpack-0.5.6 or python-msgpack0 or whatever one might call it, we could close this and roll borg well into 2021 until 1.3.0 is released.
Edit: Looked at wrong tree, link and text fixed.
No, I agree with you 100% about that. What I'm offering is a temporary fix to get us out of our current position, which is - we're already shipping borg with a incompatible version of msgpack. So any problems that are going to arise from that are already here. My solution is to at least (a) fix the one known incompatibility with python-msgpack 0.6.1 (borg mount), and (b) update us to the latest stable release. We're *already* bypassing the check that exists in borg 1.1.8, incidentally.
The correct long-term solution is for us to either ship an old version of python-msgpack, or to drop borg from our repositories. But any solution is better than what we have now.
(I would add that if old versions of python-msgpack are *not* supported upstream, I would be more inclined to drop borg. Depending on a library that is unsupported is a dangerous situation that should be remedied. So if borg maintainers take stability and security seriously they will use a supported version of msgpack.)
Then this might the best way to got.
@lfleischer @eschwartz let's roll!!