Description:
VMs using ceph/rbd backed volumes experience a complete I/O stall when attempting to access the ceph/rbd volume

Additional info:
* ceph-10.2.5-2

Steps to reproduce:
* configure libvirt for usage with ceph[1]
* configure at least one VM drive to be backed by ceph/rbd
* boot VM
* attempt to access ceph/rbd backed volume
- dd if=/dev/zero of=${rbd_backed_device} bs=1M oflag=sync status=progress

Background:
I recently found that I could not restart any of my ceph/rbd backed VMs. They all appeared to hang during boot. Initially, I thought this was due to migrating them to a freshly installed host and possibly missing some configuration step. However, attempting to boot them on an existing host revealed similar hangs during boot. Eventually, I attempted to boot one of the VMs on a third host and found that it worked. Looking for differences between the hosts I found that the working host was running my development aur/ceph-git package while the failing hosts were all running the extras/ceph 10.2.5-2 package. To test this theory, I installed my aur/ceph-git package on one of the failing hosts. After installation, VMs were able to start successfully. Reverting to the extras/ceph package resulted in the same VM boot hangs.

Using the extras/ceph PKGBUILD file as a template, I've successfully compiled 10.2.6 and 10.2.7. Both versions exhibit the same I/O hang behavior. In all cases, replacing the extras/ceph package with the aur/ceph-git build resolves the issue.

The specific aur/ceph-git build used is:
ceph-git-1:12.1.0.1018.g171104cb93-1-x86_64.pkg.tar.xz

I'll continue digging into the differences between the two packages and update this report if I'm successful in finding a solution.



[1] - http://docs.ceph.com/docs/master/rbd/libvirt/