FS#67755 - Containers with curl based health checks becoming unresponsive with containerd 1.4.0
Attached to Project:
Community Packages
Opened by Scott (firecat53) - Saturday, 29 August 2020, 15:02 GMT
Last edited by freswa (frederik) - Saturday, 29 August 2020, 16:24 GMT
Opened by Scott (firecat53) - Saturday, 29 August 2020, 15:02 GMT
Last edited by freswa (frederik) - Saturday, 29 August 2020, 16:24 GMT
|
Details
**Description**
After containerd update to 1.4.0, running a Docker container with a health check using curl (specifically bitwardenrs/server and plex images) will result in an 'unhealthy' status after 15 minutes or so. The container is unable to be stopped without killing the applicable `containerd-shim` process. **Steps to reproduce the issue:** 1. `docker run -d --rm --name bitwarden -v ~/.local/tmp/data:/data/ -e ROCKET_PORT=8080 -p 8080:8080 --init --name bitwarden_run bitwardenrs/server:alpine` 2. Wait 3. `docker ps` CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES 17f4b68347ad bitwardenrs/server:alpine "/start.sh" 1 hours ago Up 1 hours (unhealthy) 80/tcp, 3012/tcp, 0.0.0.0:8080->8080/tcp bitwarden_run **Describe the results you received:** 1. `docker stop bitwarden_run` -> very long pause (several minutes) before command completes 2. `docker ps` still shows bitwarden_run up but unhealthy 3. `ps aux|grep 17f4b` -> containerd-shim process is still running root 672481 4.3 0.2 712696 17812 ? Sl Aug27 56:46 containerd-shim -namespace moby -workdir /var/lib/containerd/io.containerd.runtime.v1.linux/moby/17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 -address /run/containerd/containerd.sock -containerd-binary /usr/bin/containerd -runtime-root /var/run/docker/runtime-runc`` 4. `sudo kill -9 672481` -> container shows as stopped. **Describe the results you expected:** 1. `docker stop bitwarden_run` -> container shows as stopped and all container processes ended **Output of `containerd --version`:** ``` containerd --version containerd github.com/containerd/containerd v1.4.0.m 09814d48d50816305a8e6c1a4ae3e2bcc4ba725a.m ``` **Any other relevant information:** Downgrading containerd to 1.3.4 fixes the issue, as does running bitwarden_rs and Plex with the `--no-healthcheck` option. OS: Archlinux (other people on the Arch forums have reported this same issue) uname -a Linux scotty 5.8.3-arch1-1 #1 SMP PREEMPT Fri, 21 Aug 2020 16:54:16 +0000 x86_64 GNU/Linux journalctl -u docker -r |grep warning|head Aug 28 11:38:07 scotty dockerd[775]: time="2020-08-28T11:38:07.287399050-07:00" level=warning msg="Health check for container 17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 error: context deadline exceeded"` journalctl -u containerd -r|head Aug 28 11:38:19 scotty containerd[524]: time="2020-08-28T11:38:19.384986550-07:00" level=warning msg="cleaning up after shim dead" id=17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 namespace=moby Aug 28 11:38:19 scotty containerd[524]: time="2020-08-28T11:38:19.384862508-07:00" level=info msg="shim reaped" id=17f4b68347adbe978d8917d575979d72fdd6b9852268506b75014cac7cafafa8 Upstream bug report here: https://github.com/containerd/containerd/issues/4509. So far only Archlinux users have noted this problem in the forums and on the Github report. I haven't found any similar reports yet from non-Arch users. Thanks! |
This task depends upon
Closed by freswa (frederik)
Saturday, 29 August 2020, 16:24 GMT
Reason for closing: Upstream
Additional comments about closing: We can't do much about it. You already filled an upstream report here: https://github.com/containerd/containerd /issues/4509
Feel free to fill a re-open request if there is a fix available.
Thank you
Saturday, 29 August 2020, 16:24 GMT
Reason for closing: Upstream
Additional comments about closing: We can't do much about it. You already filled an upstream report here: https://github.com/containerd/containerd /issues/4509
Feel free to fill a re-open request if there is a fix available.
Thank you