FS#63942 - [docker] containerd-shim process left after stopping docker
Attached to Project:
Community Packages
Opened by Lukas (luman) - Saturday, 28 September 2019, 17:39 GMT
Last edited by Morten Linderud (Foxboron) - Friday, 03 January 2020, 16:10 GMT
Opened by Lukas (luman) - Saturday, 28 September 2019, 17:39 GMT
Last edited by Morten Linderud (Foxboron) - Friday, 03 January 2020, 16:10 GMT
|
Details
**Description**
After shutting down docker, some remnant processes seem to be left. This prevents me from proper unmounting and shutting down the server. **Steps to reproduce the issue:** 1. `systemctl stop docker` 2. `ps aux | grep containerd` **Describe the results you received:** `root 6203 0.0 0.0 108988 4760 ? Sl Sep20 0:07 containerd-shim -namespace moby -workdir /mnt/data/docker/containerd ` **Describe the results you expected:** No processes running and all file descriptors closed on the /mnt/data/docker/ volume **Additional information you deem important (e.g. issue happens only occasionally):** This does not always happen. More likely the longer the server was up. **Output of `docker version`:** ``` [le@nx]: ~>$ docker version Client: Version: 19.03.2-ce API version: 1.40 Go version: go1.13 Git commit: 6a30dfca03 Built: Tue Sep 3 23:41:01 2019 OS/Arch: linux/amd64 Experimental: false Server: Engine: Version: 19.03.2-ce API version: 1.40 (minimum version 1.12) Go version: go1.13 Git commit: 6a30dfca03 Built: Tue Sep 3 23:39:37 2019 OS/Arch: linux/amd64 Experimental: false containerd: Version: v1.2.9.m GitCommit: d50db0a42053864a270f648048f9a8b4f24eced3.m runc: Version: 1.0.0-rc8 GitCommit: 425e105d5a03fabd737a126ad93d62a9eeede87f docker-init: Version: 0.18.0 GitCommit: fec3683 ``` **Output of `docker info`:** ``` [le@nx]: ~>$ docker info Client: Debug Mode: false Server: Containers: 20 Running: 20 Paused: 0 Stopped: 0 Images: 89 Server Version: 19.03.2-ce Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: false Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: d50db0a42053864a270f648048f9a8b4f24eced3.m runc version: 425e105d5a03fabd737a126ad93d62a9eeede87f init version: fec3683 Security Options: seccomp Profile: default Kernel Version: 5.3.1-arch1-1-ARCH Operating System: Arch Linux OSType: linux Architecture: x86_64 CPUs: 8 Total Memory: 15.64GiB Name: nx ID: CCE2:SHZM:I6QY:6BG3:3HA2:XHTO:45O3:AJAB:3GPP:UGOO:PETU:YJPB Docker Root Dir: /mnt/data/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` **Additional environment details (AWS, VirtualBox, physical, etc.):** Arch Linux on Proxmox via KVM ***Cross-Post:*** https://github.com/moby/moby/issues/39976 |
This task depends upon
Closed by Morten Linderud (Foxboron)
Friday, 03 January 2020, 16:10 GMT
Reason for closing: Works for me
Additional comments about closing: This is a user issue and can't be reproduced.
Friday, 03 January 2020, 16:10 GMT
Reason for closing: Works for me
Additional comments about closing: This is a user issue and can't be reproduced.
The docker service is in KillMode=process, which mean systemd doesn't cleanup containerd-shim processes when it stops.
I don't known if it's docker fault or an edge case in containerd which make it unreasonable and refuse to die.
I add the containerd guy here to get their opinion.
Do you still have this issue with recent version of containerd?
The actual problem is, that I cannot do a clean reboot on the machine in most of the cases, because the data volume cannot be proper unmounted. I tracked this down to the container-shim process which (I think) keeps handles open on the docker directory.
As for the original bugreport. `containerd-shim` is used to run docker in *deamonless* mode [1]. The process doesn't stop when you stop the service as the service never started it in the first place. You are running docker *somewhere* on your system that keeps the process there, and I suggest you try find it.
[1]: https://groups.google.com/forum/#!topic/docker-dev/zaZFlvIx1_k
This is why I created this report on the first hand.
Edit:
I do not know what 'deamonless' means. I am running docker on arch in a pretty much stock setup and spin up my containers via docker-compose.
If I run
for m in `mount | grep overlay | awk '{ print $3 }'`; do umount $m;done
for m in `mount | grep nsfs | awk '{ print $3 }'`; do umount $m;done
for m in `mount | grep shm | awk '{ print $3 }'`; do umount $m;done
before unmounting, it seems fine.
A containerd-shim process was not left over today.