FS#48459 - [libvirt] Disconnected from qemu:///session due to I/O error
Attached to Project:
Community Packages
Opened by beta990 (beta990) - Friday, 04 March 2016, 20:41 GMT
Last edited by Sergej Pupykin (sergej) - Wednesday, 04 May 2016, 10:09 GMT
Opened by beta990 (beta990) - Friday, 04 March 2016, 20:41 GMT
Last edited by Sergej Pupykin (sergej) - Wednesday, 04 May 2016, 10:09 GMT
|
Details
When using libvrt with QEMU/KVM, frequently the service
crashes with 'Disconnected from
qemu:///session
due to I/O error'.
This can be triggered even more when using $ virt-manager. For some reason the service can't handle the changes/requests and simple crashes. I have tried multiple configuration changes, like changing authentication (listen options, kvm/creating, adding myself to the libvirt group, alter unix sock permissions, etc.), starting from the beginning (cleaning all configs, etc.), but unfortunately nothing helps. Don't know if it libvirt issue or a configuration error, but since it happens with 'vanilla' settings, it maybe a libvrt or package issue. At the moment I'm not using the virt-manager, and so far it seems the I/O errors are not showing. However they do occur sometimes when using $ virsh. If more (debug) information is needed, please let me know. Thanks. |
This task depends upon
Virsh does this:
virsh # error: Disconnected from qemu:///system due to I/O error
virt-manager reports:
Error polling connection 'qemu:///system': internal error: client socket is closed
Traceback (most recent call last):
File "/usr/share/virt-manager/virtManager/engine.py", line 385, in _handle_tick_queue
conn.tick_from_engine(**kwargs)
File "/usr/share/virt-manager/virtManager/connection.py", line 1319, in tick_from_engine
raise e # pylint: disable=raising-bad-type
libvirtError: internal error: client socket is closed
They both occur at the same time. Here is the log from journalctl:
Mar 05 00:18:51 jmarsh-laptop libvirtd[1807]: libvirt version: 1.3.2
Mar 05 00:18:52 jmarsh-laptop libvirtd[1807]: hostname: jmarsh-laptop
Mar 05 00:18:52 jmarsh-laptop libvirtd[1807]: Did not receive a reply. Possible causes include: the remote application did not send a reply, the message bus security policy blocked the reply, the reply timeout expired, or the network connection was broken.
Mar 05 00:18:52 jmarsh-laptop kernel: traps: libvirtd[1811] general protection ip:7f2a5c2a14b0 sp:7f2a537d0a88 error:0 in libvirt.so.0.1003.2[7f2a5c1f6000+361000]
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: libvirt version: 1.3.2
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: hostname: jmarsh-laptop
Mar 05 00:18:52 jmarsh-laptop virtlogd[1902]: End of file while reading data: Input/output error
Mar 05 00:18:52 jmarsh-laptop kernel: virbr0: port 2(vnet0) entered disabled state
Mar 05 00:18:52 jmarsh-laptop kernel: device vnet0 left promiscuous mode
Mar 05 00:18:52 jmarsh-laptop kernel: virbr0: port 2(vnet0) entered disabled state
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Created slice system-systemd\x2dcoredump.slice.
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (virbr0): link disconnected
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (virbr0): bridge port vnet0 was detached
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Started Process Core Dump (PID 2325/UID 0).
Mar 05 00:18:52 jmarsh-laptop gnome-session[864]: Gjs-Message: JS LOG: Removing a network device that was not added
Mar 05 00:18:52 jmarsh-laptop gnome-session[490]: Gjs-Message: JS LOG: Removing a network device that was not added
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <info> (vnet0): released from master virbr0
Mar 05 00:18:52 jmarsh-laptop NetworkManager[417]: <warn> (vnet0): failed to disable userspace IPv6LL address handling
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Main process exited, code=killed, status=11/SEGV
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Failed to start Virtualization daemon.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Unit entered failed state.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Failed with result 'signal'.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Started Virtual Machine qemu-1-win10.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Stopped Virtualization daemon.
Mar 05 00:18:52 jmarsh-laptop systemd[1]: Starting Virtualization daemon...
Mar 05 00:18:52 jmarsh-laptop systemd-coredump[2326]: Process 1807 (libvirtd) of user 0 dumped core.
Stack trace of thread 1811:
#0 0x00007f2a5c2a14b0 virClassIsDerivedFrom (libvirt.so.0)
#1 0x00007f2a5c2a1760 virObjectLock (libvirt.so.0)
#2 0x0000560e0c1d2049 virNetDaemonRemoveShutdownInhibition (libvirtd)
#3 0x00007f2a471cf7d5 qemuProcessStop (libvirt_driver_qemu.so)
#4 0x00007f2a471d0de8 qemuProcessStart (libvirt_driver_qemu.so)
#5 0x00007f2a4721c803 n/a (libvirt_driver_qemu.so)
#6 0x00007f2a5c349703 virDomainCreateXML (libvirt.so.0)
#7 0x0000560e0c1c3a7b n/a (libvirtd)
#8 0x00007f2a5c3bd709 virNetServerProgramDispatch (libvirt.so.0)
#9 0x00007f2a5c3b8c18 n/a (libvirt.so.0)
#10 0x00007f2a5c2bba66 n/a (libvirt.so.0)
#11 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#12 0x00007f2a597cb424 start_thread (libpthread.so.0)
#13 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1818:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1820:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1821:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1819:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1822:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbafb n/a (libvirt.so.0)
#3 0x00007f2a5c2bafe8 n/a (libvirt.so.0)
#4 0x00007f2a597cb424 start_thread (libpthread.so.0)
#5 0x00007f2a5950acbd __clone (libc.so.6)
Stack trace of thread 1807:
#0 0x00007f2a597d103f pthread_cond_wait@@GLIBC_2.3.2 (libpthread.so.0)
#1 0x00007f2a5c2bb246 virCondWait (libvirt.so.0)
#2 0x00007f2a5c2bbbc0 virThreadPoolFree (libvirt.so.0)
#3 0x00007f2a5c3b86cc n/a (libvirt.so.0)
#4 0x00007f2a5c2a169b virObjectUnref (libvirt.so.0)
#5 0x0000560e0c19c0de n/a (libvirtd)
#6 0x00007f2a59443710 __libc_start_main (libc.so.6)
#7 0x0000560e0c19dda9 n/a (libvirtd)
Mar 05 00:18:52 jmarsh-laptop dnsmasq[603]: read /etc/hosts - 2 addresses
Mar 05 00:18:52 jmarsh-laptop dnsmasq[603]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 05 00:18:52 jmarsh-laptop dnsmasq-dhcp[603]: read /var/lib/libvirt/dnsmasq/default.hostsfile
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: libvirt version: 1.3.2
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: hostname: jmarsh-laptop
Mar 05 00:18:53 jmarsh-laptop libvirtd[2332]: failed to connect to monitor socket: No such process
Mar 05 00:18:53 jmarsh-laptop virtlogd[1902]: End of file while reading data: Input/output error
Mar 05 00:18:56 jmarsh-laptop dleyna-server-service[2035]: dLeyna: Exit
kkMar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Start operation timed out. Terminating.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Failed to start Virtualization daemon.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Unit entered failed state.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Failed with result 'timeout'.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: libvirtd.service: Service hold-off time over, scheduling restart.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Stopped Virtualization daemon.
Mar 05 00:20:22 jmarsh-laptop systemd[1]: Starting Virtualization daemon...
Mar 05 00:20:23 jmarsh-laptop dnsmasq[603]: read /etc/hosts - 2 addresses
Mar 05 00:20:23 jmarsh-laptop dnsmasq[603]: read /var/lib/libvirt/dnsmasq/default.addnhosts - 0 addresses
Mar 05 00:20:23 jmarsh-laptop dnsmasq-dhcp[603]: read /var/lib/libvirt/dnsmasq/default.hostsfile
https://www.reddit.com/r/archlinux/comments/490h9d/libvirtqemu_weird_timeout_issue/d0o7lnx
# cp /usr/lib/systemd/system/libvirtd.service /etc/systemd/system/libvirtd.service
# sed -i s/notify/simple/ /etc/systemd/system/libvirtd.service
# systemctl daemon-reload
# systemctl reenable libvirtd # If you enabled it before
Perhaps we can provide some sort of patch for the service until the upstream issue is fixed.
Steps to Reproduce:
1. Connect to User Session (qemu:///session)
2. Wait around 30 seconds, without starting any VM
3. Enjoy :(
Seems that there is a patch in https://github.com/nertpinx/libvirt/commit/78b0ccc71e99f769068974ff56638c99b1c3b4de
libvirt 1.3.3-1
virt-manager 1.3.2-4
Update: Hmm, it seems I applied the systemd service "fix" listed in previous comment. So perhaps, it was working due to that rather than libvirt updates.