FS#75605 - [podman] podman with UID/GID mapping not working anylonger with linux 5.19.1.arch2-1

Attached to Project: Community Packages
Opened by Lichtprotoss (Lichtprotoss) - Sunday, 14 August 2022, 19:17 GMT
Last edited by David Runge (dvzrv) - Friday, 21 October 2022, 12:33 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To David Runge (dvzrv)
Morten Linderud (Foxboron)
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 0
Private No

Details

Description:
Hi all

Hope you are all safe

My containers with UID/GID user namespace mapping do not launch in podman anylonger after updating linux to 5.19.1.arch2-1. It still works with linux 5.18.16.arch1-1.

Any feedback appreciated. Thanks for taking the time.

Additional info:
podman 4.1.1-4
device-mapper 2.03.16-2
gpgme 1.18.0-1
5.19.1.arch2-1

Steps to reproduce:
sudo podman run --rm -it --uidmap 0:10000:1000 --gidmap 0:10000:1000 --name ubuntu ubuntu:22.04

Error message:
Resolved "ubuntu" as an alias (/etc/containers/registries.conf.d/00-shortnames.conf)
Trying to pull docker.io/library/ubuntu:22.04...
Getting image source signatures
Copying blob d19f32bd9e41 skipped: already exists
Copying config df5de72bdb done
Writing manifest to image destination
Storing signatures
ERRO[0002] Unmounting /var/lib/containers/storage/overlay/38063a1cc6ce792f021ff28b298118a8f70ea3fd0979d2f0d7dd269cc5e88306/merged: invalid argument
Error: error mounting storage for container 36bab36c4127a2812c15398ca7fe2f089976bf208a8d924ac59667fa81685d24: creating overlay mount to /var/lib/containers/storage/overlay/38063a1cc6ce792f021ff28b298118a8f70ea3fd0979d2f0d7dd269cc5e88306/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/38063a1cc6ce792f021ff28b298118a8f70ea3fd0979d2f0d7dd269cc5e88306/mapped/0/l/OJUW7KY7BMNMAW3IJPHPCAENY4,upperdir=/var/lib/containers/storage/overlay/38063a1cc6ce792f021ff28b298118a8f70ea3fd0979d2f0d7dd269cc5e88306/diff,workdir=/var/lib/containers/storage/overlay/38063a1cc6ce792f021ff28b298118a8f70ea3fd0979d2f0d7dd269cc5e88306/work,nodev,volatile": permission denied

Works again with downgrading linux:
sudo pacman -U /var/cache/pacman/pkg/linux-5.18.16.arch1-1-x86_64.pkg.tar.zst














   podman_log_level_debug.txt (10.4 KiB)
This task depends upon

Closed by  David Runge (dvzrv)
Friday, 21 October 2022, 12:33 GMT
Reason for closing:  Fixed
Additional comments about closing:  Fixed with podman 4.3.0-1
Comment by Toolybird (Toolybird) - Monday, 15 August 2022, 06:10 GMT
This issue [1] looks very similar. Have you also tried the latest podman in testing? (although in the linked report, it made no difference).

[1] https://github.com/containers/podman/issues/15312
Comment by Lichtprotoss (Lichtprotoss) - Monday, 15 August 2022, 19:58 GMT
Issue [1] is similar. Thanks. With the latest podman (in testing) the above case works, however does not work with a container in a pod.

Additional info:
podman 4.2.0-1
linux 5.19.1.arch2-1

Now works:
sudo podman run --rm -it --uidmap 0:10000:1000 --gidmap 0:10000:1000 --name ubuntu ubuntu:22.04

BUT error with:
sudo podman pod create --uidmap 0:10000:1000 --gidmap 0:10000:1000 --name ub_pod
sudo podman podman run --rm -it --pod ub_pod --name ubuntu ubuntu:22.04

Output:
ERRO[0000] Unmounting /var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/merged: invalid argument
ERRO[0000] Starting some container dependencies
ERRO[0000] "error mounting storage for container 70aba93183dd49ee05c29fb640d86e1e1a7f885f8bcc15ef1ab9d6cc36ae85fd: creating overlay mount to /var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/merged, mount_data="lowerdir=/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/mapped/0/l/JUZCTZH7JSXUH6NMVT5JT4QNDR:/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/mapped/0/l/diff1:/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/mapped/0/l/ULD47QYAH2DLN4BNTZBUHSLUUO:/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/mapped/0/l/diff1,upperdir=/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/diff,workdir=/var/lib/containers/storage/overlay/466860f626179b4993222a843618197136485052a8737a9f20a444949eb5a856/work,nodev": no such file or directory"
Error: error starting some containers: internal libpod error

Note: all working with the previous linux version
podman 4.2.0-1
linux 5.18.16.arch1-1

[1] https://github.com/containers/podman/issues/15312

   podman_log_level_debug_4.2.0-... (11.6 KiB)
Comment by Lichtprotoss (Lichtprotoss) - Monday, 15 August 2022, 20:36 GMT
   podman_log_level_debug_4.2.0-... (11.6 KiB)
Comment by Lichtprotoss (Lichtprotoss) - Friday, 09 September 2022, 21:07 GMT
Update

Additional info:
podman 4.2.1-1
linux 5.19.7.arch1-1

Error with:
sudo podman pod create --uidmap 0:10000:1000 --gidmap 0:10000:1000 --name ub_pod
sudo podman run --rm -it --pod ub_pod --name ubuntu ubuntu:22.04

Output:
ERRO[0005] Unmounting /var/lib/containers/storage/overlay/0e9f173120e7c1b7d5f163f6128eb098763a09be6484ddb4410c42e9da527c28/merged: invalid argument
Error: error mounting storage for container 141f00aac524ae6e930e8d169796ea116dbcdf327b2504236359f5813536f18c: create mapped mount for "/var/lib/containers/storage/overlay" on "/var/lib/containers/storage/overlay/0e9f173120e7c1b7d5f163f6128eb098763a09be6484ddb4410c42e9da527c28/mapped/0": invalid argument

Note: all working with the previous linux version
podman 4.2.1-1
linux 5.18.16.arch1-1





Comment by Toolybird (Toolybird) - Friday, 09 September 2022, 22:51 GMT
IIUC, this is likely fixed upstream in "containers/storage" which is part of "containers-common". See this issue [1] reported by Arch team member @klausenbusk. Not sure if we have to wait for next "containers/storage" release or whether we can cherrypick.

[1] https://github.com/containers/storage/issues/1310
Comment by David Runge (dvzrv) - Saturday, 10 September 2022, 08:58 GMT
@Lichtprotoss Thanks for the ticket!

I think as this is an upstream issue, it would be good to comment on the upstream ticket [1] as well.

@Toolybird The containers-common package only makes use of documentation files from containers/storage. However, containers/storage is also a go library, which is released separately (latest release does not include that fix) and is included in the vendored stack of podman [2].

[1] https://github.com/containers/storage/issues/1308
[2] https://github.com/containers/podman/commits/main/vendor/github.com/containers/storage
Comment by David Runge (dvzrv) - Friday, 21 October 2022, 12:33 GMT
An updated version of containers/storage has been included before the release of podman 4.3.0.

Loading...