FS#48459 - [libvirt] Disconnected from qemu:///session due to I/O error

Attached to Project: Community Packages
Opened by beta990 (beta990) - Friday, 04 March 2016, 20:41 GMT
Last edited by Sergej Pupykin (sergej) - Wednesday, 04 May 2016, 10:09 GMT
Task Type Bug Report
Category Packages
Status Closed
Assigned To Sergej Pupykin (sergej)
Architecture All
Severity Medium
Priority Normal
Reported Version
Due in Version Undecided
Due Date Undecided
Percent Complete 100%
Votes 4
Private No

Details

When using libvrt with QEMU/KVM, frequently the service crashes with 'Disconnected from qemu:///session due to I/O error'.
This can be triggered even more when using $ virt-manager. For some reason the service can't handle the changes/requests and simple crashes.

I have tried multiple configuration changes, like changing authentication (listen options, kvm/creating, adding myself to the libvirt group, alter unix sock permissions, etc.), starting from the beginning (cleaning all configs, etc.), but unfortunately nothing helps.

Don't know if it libvirt issue or a configuration error, but since it happens with 'vanilla' settings, it maybe a libvrt or package issue.

At the moment I'm not using the virt-manager, and so far it seems the I/O errors are not showing. However they do occur sometimes when using $ virsh.

If more (debug) information is needed, please let me know.

Thanks.
This task depends upon

Closed by  Sergej Pupykin (sergej)
Wednesday, 04 May 2016, 10:09 GMT
Reason for closing:  Fixed
Comment by Joshua Marsh (icub3d) - Saturday, 05 March 2016, 07:10 GMT
I am having the same issue as well. Both virsh and virt-manager present the same issue. It makes it nearly impossible to use it.

Virsh does this:

virsh # error: Disconnected from qemu:///system due to I/O error

virt-manager reports:

Error polling connection 'qemu:///system': internal error: client socket is closed

Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/engine.py", line 385, in _handle_tick_queue
conn.tick_from_engine(**kwargs)
File "/usr/share/virt-manager/virtManager/connection.py", line 1319, in tick_from_engine
raise e # pylint: disable=raising-bad-type
libvirtError: internal error: client socket is closed

They both occur at the same time. Here is the log from journalctl:

Mar 05 00:18:51 jmarsh-laptop libvirtd[1807]: libvirt version: 1.3.2
Mar 05 00:18:52 jmarsh-laptop libvirtd[1807]: hostname: jmarsh-laptop
Mar 05 00:18:52 jmarsh-laptop libvirtd[1807]: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Mar 05 00:18:52 jmarsh-laptop kernel: traps: libvirtd[1811] general protection ip:7f2a5c2a14b0 sp:7f2a537d0a88 error:0 in libvirt.so.0.1003.2[7f2a5c1f6000+361000]
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: libvirt version: 1.3.2
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: hostname: jmarsh-laptop
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: End of file while reading data: Input/output error
Mar 05 00:18:52 jmarsh-laptop kernel: virbr0: port 2(vnet0) entered disabled state
Mar 05 00:18:52 jmarsh-laptop kernel: device vnet0 left promiscuous mode
Mar 05 00:18:52 jmarsh-laptop kernel: virbr0: port 2(vnet0) entered disabled state
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (virbr0): link disconnected
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (virbr0): bridge port vnet0 was detached
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Started Process Core Dump (PID 2325/UID 0).
Mar 05 00:18:52 jmarsh-laptop gnome-session[864]: Gjs-Message: JS LOG: Removing a network device that was not added
Mar 05 00:18:52 jmarsh-laptop gnome-session[490]: Gjs-Message: JS LOG: Removing a network device that was not added
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (vnet0): released from master virbr0
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <warn> (vnet0): failed to disable userspace IPv6LL address handling
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Main process exited, code=killed, status=11/SEGV
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Failed to start Virtualization daemon.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Unit entered failed state.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Failed with result 'signal'.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Started Virtual Machine qemu-1-win10.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Stopped Virtualization daemon.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Starting Virtualization daemon...
Mar 05 00:18:52 jmarsh-laptop systemd-coredump[2326]: Process 1807 (libvirtd) of user 0 dumped core.

Stack trace of thread 1811:
#0 0x00007f2a5c2a14b0 virClassIsDerivedFrom (libvirt.so.0)
#1 0x00007f2a5c2a1760 virObjectLock (libvirt.so.0)
#2 0x0000560e0c1d2049 virNetDaemonRemoveShutdownInhibition (libvirtd)
#3 0x00007f2a471cf7d5 qemuProcessStop (libvirt_driver_qemu.so)
#4 0x00007f2a471d0de8 qemuProcessStart (libvirt_driver_qemu.so)
#5 0x00007f2a4721c803 n/a (libvirt_driver_qemu.so)
#6 0x00007f2a5c349703 virDomainCreateXML (libvirt.so.0)
#7 0x0000560e0c1c3a7b n/a (libvirtd)
#8 0x00007f2a5c3bd709 virNetServerProgramDispatch (libvirt.so.0)
#9 0x00007f2a5c3b8c18 n/a (libvirt.so.0)
#10 0x00007f2a5c2bba66 n/a (libvirt.so.0)
#11 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#12 0x00007f2a597cb424 start_thread (libpthread.so.0)
#13 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1818:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1820:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1821:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1819:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1822:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)

Stack trace of thread 1807:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbbc0 virThreadPoolFree (libvirt.so.0)
#3 0x00007f2a5c3b86cc n/a (libvirt.so.0)
#4 0x00007f2a5c2a169b virObjectUnref (libvirt.so.0)
#5 0x0000560e0c19c0de n/a (libvirtd)
#6 0x00007f2a59443710 __libc_start_main (libc.so.6)
#7 0x0000560e0c19dda9 n/a (libvirtd)
Mar 05 00:18:52 jmarsh-laptop dnsmasq[603]: read /etc/hosts - 2 addresses
Mar 05 00:18:52 jmarsh-laptop dnsmasq[603]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 05 00:18:52 jmarsh-laptop dnsmasq-dhcp[603]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: libvirt version: 1.3.2
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: hostname: jmarsh-laptop
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: failed to connect to monitor socket: No such process
Mar 05 00:18:53 jmarsh-laptop virtlogd[1902]: End of file while reading data: Input/output error
Mar 05 00:18:56 jmarsh-laptop dleyna-server-service[2035]: dLeyna: Exit
kkMar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Start operation timed out. Terminating.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Failed to start Virtualization daemon.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Unit entered failed state.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Failed with result 'timeout'.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Stopped Virtualization daemon.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Starting Virtualization daemon...
Mar 05 00:20:23 jmarsh-laptop dnsmasq[603]: read /etc/hosts - 2 addresses
Mar 05 00:20:23 jmarsh-laptop dnsmasq[603]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 05 00:20:23 jmarsh-laptop dnsmasq-dhcp[603]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Comment by beta990 (beta990) - Saturday, 05 March 2016, 12:26 GMT
When not using virt-manager, the service doesn't crash. At least for me, maybe this should be reported upstream?
Comment by Joshua Marsh (icub3d) - Saturday, 05 March 2016, 18:26 GMT
The solution found in the link below fixes the issue for me.

https://www.reddit.com/r/archlinux/comments/490h9d/libvirtqemu_weird_timeout_issue/d0o7lnx

# cp /usr/lib/systemd/system/libvirtd.service /etc/systemd/system/libvirtd.service
# sed -i s/notify/simple/ /etc/systemd/system/libvirtd.service
# systemctl daemon-reload
# systemctl reenable libvirtd # If you enabled it before

Perhaps we can provide some sort of patch for the service until the upstream issue is fixed.
Comment by Oscar Garcia (ogarcia) - Friday, 01 April 2016, 14:45 GMT
The bug still present.

Steps to Reproduce:
1. Connect to User Session (qemu:///session)
2. Wait around 30 seconds, without starting any VM
3. Enjoy :(

Seems that there is a patch in https://github.com/nertpinx/libvirt/commit/78b0ccc71e99f769068974ff56638c99b1c3b4de
Comment by pedrum (pd5rm) - Tuesday, 26 April 2016, 20:21 GMT
This appears to be fixed with following versions (for me at least):

libvirt 1.3.3-1
virt-manager 1.3.2-4

Update: Hmm, it seems I applied the systemd service "fix" listed in previous comment. So perhaps, it was working due to that rather than libvirt updates.
Comment by Oscar Garcia (ogarcia) - Wednesday, 27 April 2016, 06:36 GMT
Yes. I can confirm that libvirt 1.3.3-1 solves the issue.

Loading...