FS#56874 - [linux] Qemu process remains in D state

Attached to Project: Arch Linux
Opened by Giancarlo Razzolini (grazzolini) - Wednesday, 27 December 2017, 15:23 GMT
Last edited by Doug Newgard (Scimmia) - Friday, 19 January 2018, 23:25 GMT
Task Type Bug Report
Category Kernel
Status Closed
Assigned To Tobias Powalowski (tpowa)
Jan Alexander Steffens (heftig)
Architecture All
Severity High
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 1
Private No

Details

Description:

Ever since kernel 4.14 hit the repositories, any qemu guest, regardless of OS, will remain in D state after powering down the guest. This sometimes is not triggered if the guest is not running X. Also, this happens when rebooting the guest.

I have gathered reports from several different people, on #archlinux. They are running different OS'es, but the effect is the same.

When this happens, the host cannot be shut down cleanly, because the qemu process is uninterruptible. The only option is to use Magic SysRq keys to try to power down/reboot cleanly.

I'm using libvirt to drive qemu.

Additional info:

Packages:
linux 4.14.8-1
qemu 2.11.0-3
libvirt 3.10.0-1

After the qemu process enters D state, it remains in the process table, and every 120 seconds you get a message like this one in the journal:

kernel: INFO: task qemu-system-x86:1293 blocked for more than 120 seconds.
kernel: Tainted: G O 4.14.8-1-ARCH #1
kernel: "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
kernel: qemu-system-x86 D 0 1293 1 0x00000004
kernel: Call Trace:
kernel: ? __schedule+0x290/0x890
kernel: ? wake_up_q+0x70/0x70
kernel: schedule+0x2f/0x90
kernel: vhost_net_ubuf_put_and_wait+0x55/0x80 [vhost_net]
kernel: ? wait_woken+0x80/0x80
kernel: vhost_net_ioctl+0x433/0x8b0 [vhost_net]
kernel: ? do_futex+0x429/0xa90
kernel: ? do_send_sig_info+0x65/0x90
kernel: do_vfs_ioctl+0xa1/0x610
kernel: ? SyS_futex+0x12d/0x180
kernel: SyS_ioctl+0x74/0x80
kernel: entry_SYSCALL_64_fastpath+0x1a/0xa5
kernel: RIP: 0033:0x7f8ff9753337
kernel: RSP: 002b:00007ffc18078a58 EFLAGS: 00000246 ORIG_RAX: 0000000000000010
kernel: RAX: ffffffffffffffda RBX: 00007ffc18078b50 RCX: 00007f8ff9753337
kernel: RDX: 00007ffc18078a60 RSI: 000000004008af30 RDI: 000000000000001b
kernel: RBP: 00007f8feea1cd00 R08: 0000559403d91ed0 R09: 00007f8de633c1b0
kernel: R10: 0000000000000000 R11: 0000000000000246 R12: 00007ffc18078bb4
kernel: R13: 00007f8feea1cd00 R14: 00007ffc18079689 R15: 0000559404980ba0


Steps to reproduce:

Start any guest using libvirt, use it for a while and power it down, using any method. The qemu process enters in D state.
This task depends upon

Closed by  Doug Newgard (Scimmia)
Friday, 19 January 2018, 23:25 GMT
Reason for closing:  Fixed
Additional comments about closing:  linux 4.14.11-1
Comment by Jason William Walton (jasonww) - Saturday, 30 December 2017, 12:49 GMT Comment by Giancarlo Razzolini (grazzolini) - Monday, 01 January 2018, 10:07 GMT
Thanks for updating this with the upstream bug. I was not 100% sure if this bug was on qemu or on the kernel itself.
Comment by loqs (loqs) - Monday, 01 January 2018, 20:17 GMT
First patch ( the fix ) should be in 4.14.11 https://git.kernel.org/pub/scm/linux/kernel/git/stable/linux-stable-rc.git/commit/net/core/skbuff.c?id=68a5edb51ee33c0c4612b238e316c943b2e03ef2
Second patch that improves performance of the first http://patchwork.ozlabs.org/patch/852598/ is being held until 4.16.
Comment by Giancarlo Razzolini (grazzolini) - Monday, 01 January 2018, 20:56 GMT
Thanks for the info loqs. I tested the workaround of passing the option to the vhost_net module and it works. So, anyone having this issue can use this:

/etc/modprobe.d/vhost-net.conf:

options vhost_net experimental_zcopytx=0
Comment by Giancarlo Razzolini (grazzolini) - Tuesday, 09 January 2018, 02:09 GMT
I can confirm that kernel 4.14.12 fixes this issue, without the need for the workaround:

systool -vm vhost_net
Module = "vhost_net"

Parameters:
experimental_zcopytx= "1"

Running with default options it is working. I can shutdown the guests properly.

Loading...