FS#67755 - Containers with curl based health checks becoming unresponsive with containerd 1.4.0

Attached to Project: Community Packages
Opened by Scott (firecat53) - Saturday, 29 August 2020, 15:02 GMT
Last edited by freswa (frederik) - Saturday, 29 August 2020, 16:24 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To No-one
Architecture All
Severity Low
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

**Description**

After containerd update to 1.4.0, running a Docker container with a health check using curl (specifically bitwardenrs/server and plex images) will result in an 'unhealthy' status after 15 minutes or so. The container is unable to be stopped without killing the applicable `containerd-shim` process.

**Steps to reproduce the issue:**

1. `docker run -d --rm --name bitwarden -v ~/.local/tmp/data:/data/ -e ROCKET_PORT=8080 -p 8080:8080 --init --name bitwarden_run bitwardenrs/server:alpine`
2. Wait
3. `docker ps`

CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS
NAMES
17f4b68347ad bitwardenrs/server:alpine "/start.sh" 1 hours ago Up 1 hours (unhealthy) 80/tcp, 3012/tcp,
0.0.0.0:8080->8080/tcp bitwarden_run

**Describe the results you received:**

1. `docker stop bitwarden_run` -> very long pause (several minutes) before
command completes
2. `docker ps` still shows bitwarden_run up but unhealthy
3. `ps aux|grep 17f4b` -> containerd-shim process is still running

root 672481 4.3 0.2 712696 17812 ? Sl Aug27 56:46 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc``

4. `sudo kill -9 672481` -> container shows as stopped.

**Describe the results you expected:**

1. `docker stop bitwarden_run` -> container shows as stopped and all container processes ended

**Output of `containerd --version`:**


```
containerd --version containerd github.com/containerd/containerd v1.4.0.m 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a.m
```

**Any other relevant information:**

Downgrading containerd to 1.3.4 fixes the issue, as does running bitwarden_rs and Plex with the `--no-healthcheck` option.

OS: Archlinux (other people on the Arch forums have reported this same issue)

uname -a Linux scotty 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux

journalctl -u docker -r |grep warning|head
Aug 28 11:38:07 scotty dockerd[775]: time="2020-08-28T11:38:07.287399050-07:00" level=warning msg="Health check for container 17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 error: context deadline exceeded"`
journalctl -u containerd -r|head

Aug 28 11:38:19 scotty containerd[524]: time="2020-08-28T11:38:19.384986550-07:00" level=warning msg="cleaning up after shim dead" id=17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 namespace=moby
Aug 28 11:38:19 scotty containerd[524]: time="2020-08-28T11:38:19.384862508-07:00" level=info msg="shim reaped" id=17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8

Upstream bug report here: https://github.com/containerd/containerd/issues/4509. So far only Archlinux users have noted this problem in the forums and on the Github report. I haven't found any similar reports yet from non-Arch users.

Thanks!
This task depends upon

This task blocks these from closing
 FS#67773 - [containerd] version 1.4.0-2 break docker 
Closed by  freswa (frederik)
Saturday, 29 August 2020, 16:24 GMT
Reason for closing:  Upstream
Additional comments about closing:  We can't do much about it. You already filled an upstream report here: https://github.com/containerd/containerd /issues/4509
Feel free to fill a re-open request if there is a fix available.
Thank you

Loading...