kvm: restore backups into the volume they were taken from

[calvix]

Description

restoreBackup of a KVM instance with more than one data disk can write a
backed up disk into a different volume than the one it was taken from. This happens only on a second or further restore of a VM from a backup that was taken. ie: create backup of VM, restore 1, then restore 2 again.

• equal-sized data disks → the volume contents are silently exchanged, and the API returns success: true, but the appearance of the bug is not 100% because the enumeration can randomly be the same as before. I recommend trying instances with 5 or 7 data disks for testing where occurrence is very likely.
• differently sized data disks → the smaller image is written over the larger volume, the restore then
  fails converting the larger image into the smaller volume, and returns an error after the data is already
  destroyed, with no rollback
• the ROOT disk is never affected (always re-attached at device id 0, so it sorts first in both lists)
• createVMFromBackup and restoreVolumeFromBackupAndAttachToVM are not affected

Reproduced on 4.22.1.0 with KVM + Ceph/RBD primary storage and the NAS backup provider.

At first, this can be a very silent error, especially if the VM uses mount by LABEL/UUID, as the VM will mount the data to the right mount paths, but volume A now contains the data of volume B ( including LABELs) and volume B contains data for volume A.

Logs

case 1 - different disk sizes

The logs show output showing this bug on the VM, with 2 data volumes diska, diskb - all stored on Ceph primary storage and backed up via backup and recovery plugin to cephfs storage. diskcheck is a simple bash util that prints useful data from the disks on a remote machine and connects via ssh.

 #:   diskcheck 10.3.77.212
DEVICE SIZE   LABEL   MOUNT        FS-SIZE  FILE
vdb    1G     DISKA   /mnt/diska   974M     DISKA=DISKA
vdc    2G     DISKB   /mnt/diskb   2.0G     DISKB=DISKB

 #: cmk -p stable list volumes virtualmachineid=3d602d8a-c00c-4f49-a432-8c8f82237b3d filter=name,deviceid,size
{
  "count": 3,
  "volume": [
    {
      "deviceid": 0,
      "name": "ROOT-959",
      "size": 630159872
    },
    {
      "deviceid": 1,
      "name": "demo-disk-a",
      "size": 1073741824
    },
    {
      "deviceid": 2,
      "name": "demo-disk-b",
      "size": 2147483648
    }
  ]
}

 #: cmk -p stable stop virtualmachine id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
 ...... redacted
}

 #: cmk -p stable restore backup id=557b755b-6426-4daa-9c96-871281a4112f
{
  "success": true
}
 # Below we can see the deviceID changed for both data disks
 #: cmk -p stable list volumes virtualmachineid=3d602d8a-c00c-4f49-a432-8c8f82237b3d filter=name,deviceid
{
  "count": 3,
  "volume": [
    {
      "deviceid": 0,
      "name": "ROOT-959"
    },
    {
      "deviceid": 4,
      "name": "demo-disk-a"
    },
    {
      "deviceid": 1,
      "name": "demo-disk-b"
    }
  ]
}
 #: cmk -p stable start virtualmachine id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
...
}
 # Here we can see that disk ordering changed - this will basically break any further restore 
 #: diskcheck 10.3.77.212
DEVICE SIZE   LABEL   MOUNT        FS-SIZE  FILE
vdb    2G     DISKB   /mnt/diskb   2.0G     DISKB=DISKB
vdc    1G     DISKA   /mnt/diska   974M     DISKA=DISKA



 #: cmk -p stable stop virtualmachine id=3d602d8a-c00c-4f49-a432-8c8f82237b3d
{
  ...
  
}
 # The second restore fails because it cannot restore 2GB disk into 1 GB volume
 #: cmk -p stable restore backup id=557b755b-6426-4daa-9c96-871281a4112f
{
  "account": "admin",
  "accountid": "65fed4ca-48b6-11f1-a321-bc24119cb31e",
  "cmd": "org.apache.cloudstack.api.command.user.backup.RestoreBackupCmd",
  "completed": "2026-08-19T11:35:55+0200",
  "created": "2026-08-19T11:35:44+0200",
  "domainid": "176895fd-48b6-11f1-a321-bc24119cb31e",
  "domainpath": "ROOT",
  "jobid": "3c7c45dc-68f1-4152-8276-a1f4cf03678f",
  "jobprocstatus": 0,
  "jobresult": {
    "errorcode": 530,
    "errortext": "Error restoring VM from backup [{\"externalId\":\"i-2-959-VM\\/2026.08.19.11.28.52\",\"name\":\"demo\",\"uuid\":\"557b755b-6426-4daa-9c96-871281a4112f\",\"vmId\":959}]."
  },
  "jobresultcode": 530,
  "jobresulttype": "object",
  "jobstatus": 2,
  "userid": "65ff40f2-48b6-11f1-a321-bc24119cb31e"
}

If the disk sizes are the same, the second restore works but disk can be swapped depending on the scenario how disk are renumbered.

Root cause

NASBackupProvider.restoreVMBackup builds two lists the backed up volumes and the instance's current
volumes, each sorted by device id, and LibvirtRestoreBackupCommandWrapper.restoreVolumesOfExistingVM
consumes them index by index:

BackupManagerImpl.restoreBackup - importRestoredVM - KVMGuru.importVirtualMachineFromBackup
re-attaches every data disk with getNextAvailableDeviceId(). The volumes are never detached first, so
their current ids still count as "in use" and the helper can never return the id a volume already holds —
every data disk is shifted by one slot on every restore.

A restore is therefore only correct while the instance's device ids still match the ones recorded in the
backup. The first restore satisfies that and then breaks it for the next one.

Behaviour, restore by restore

The first restore is always correct. It is the restore itself that can plant the fault, so the damage only
appears on second restore.

Restore #1 — succeeds, but renumbers the disks

The device ids recorded in the backup still match the instance, so the two sorted lists line up and every
backup lands in its own volume. Verified on a 2-data-disk instance by reading the volumes directly from
primary storage — each still held its own filesystem.

But on completion KVMGuru.importVirtualMachineFromBackup re-attaches the data disks with
getNextAvailableDeviceId():

t

before restore #1:   ROOT=0   demo-disk-a=1   demo-disk-b=2
after  restore #1:   ROOT=0    ddemo-disk-a=4  emo-disk-b=1         <-- renumbered
after  restore #2:   ROOT=0   demo-disk-a=2   demo-disk-b=4      <-- renumbered again

After the first restore, diskA gets id 4 because ids 1 and 2 are taken; id 3 is cdrom, so the next free id is 4. diskb gets id 1 because the id was just freed.

Restore #2 — the damage, and what it looks like depends on the disk sizes

Sorted by device id the instance is now [ROOT, demo-disk-b, demo-disk-a] while the backup is
[ROOT, demo-disk-a, demo-disk-b], so index 1 and 2 point at each other's volumes.

Equal-sized data disks (1 GB + 1 GB) — silent corruption. Both images fit their new targets, so
qemu-img succeeds and the API returns success: true. The volume contents are simply exchanged:

demo-disk-a  contains ext4 label: DISKB      <-- swapped
demo-disk-b  contains ext4 label: DISKA

A guest that mounts by LABEL=/UUID= still mounts everything at the right paths, because the labels
travel with the filesystems — so nothing looks wrong from inside the instance, while CloudStack's
volume-to-content mapping is now wrong for every per-volume operation (detach, delete, snapshot, resize,
restore single volume).

Differently sized data disks (1 GB + 2 GB), data loss, reported as a failure.
The loop processes the disks in order and only aborts on the second pairing, so the first one has already been written.
The API returns:

"errorcode": 530,
"errortext": "Error restoring VM from backup [{\"externalId\":\"i-2-959-VM/2026.08.19.11.28.52\", ...}]"

and the agent log shows the real cause — the 2 GB backup being written into the 1 GB volume:

qemu-img convert -n -O raw --image-opts
  driver=qcow2,file.filename=.../datadisk.806fa24b-a2da-45b1-9bd4-5fd51d0cc78b.qcow2   <- backup of disk B (2 GB)
  rbd:cloudstack/da2029ad-8517-4f40-af03-bd5f3ec5d959                                   <- volume of disk A (1 GB)
encountered the error: [qemu-img: output file is smaller than input]

State afterwards — the restore is reported as failed, but one volume is already gone, with no rollback:

demo-disk-a (1 GiB)   ext4-label = DISKA     <- untouched, the write into it failed
demo-disk-b (2 GiB)   ext4-label = DISKA     <- OVERWRITTEN, its own data now exists only in the backup

Both volumes still report Ready in CloudStack. An operator seeing error 530 would reasonably assume
nothing happened.

How to reproduce

Instance with two data disks, a NAS backup offering, KVM primary storage. Use two different sizes to get
the loud failure, if you want to reproduce the disk swap, I recommend using more data disks, as there is a chance that reumbering accdientaly get right and the bug does not occur. So trying like 5 or 7 data disks should most likely show the bug.

# 1. baseline: device ids 1 and 2
cmk list volumes virtualmachineid=$VM filter=name,deviceid
#   ROOT deviceid=0 | demo-disk-a deviceid=1 | demo-disk-b deviceid=2

# 2. label the filesystems so they can be told apart
#    mkfs.ext4 -L DISKA /dev/vdb ; mkfs.ext4 -L DISKB /dev/vdc

cmk create backup virtualmachineid=$VM name=demo
cmk stop virtualmachine id=$VM

# 3. FIRST restore - correct, but renumbers the disks
cmk restore backup id=$B
cmk list volumes virtualmachineid=$VM filter=name,deviceid
#   ROOT deviceid=0 | demo-disk-b deviceid=1 | demo-disk-a deviceid=4     <-- misaligned from here on

# 4. SECOND restore - silent swap (equal sizes) or error 530 after destroying a volume (different sizes)
cmk restore backup id=$B

# if you have the same sizes, you could try from UI detach diskA ( after either turning off teh safety config or disabling backups) and see that in OS the different disk vanishes.

Types of changes

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to change)
• Enhancement (improves an existing feature and functionality)
• Cleanup (Code refactoring and cleanup, that may add test cases)
• build/CI

How Has This Been Tested?

Unit tests — three added, all of which fail without the patch:

	with the fix	without the fix
KVMGuruTest	30 tests, 0 failures	30 tests, 1 error
LibvirtRestoreBackupCommandWrapperTest	13 tests, 0 failures	13 tests, 1 failure, 1 error

Every pre-existing test in both classes passes either way.

• LibvirtRestoreBackupCommandWrapperTest.testRestoreOfExistingVmMapsBackupsToVolumesByUuid — the
  instance's volumes arrive in a different order than the backed up volumes; each backup must still be
  written into the volume it was taken from. Also asserts with never() that neither data disk's backup is
  written into the other one.
• LibvirtRestoreBackupCommandWrapperTest.testRestoreOfExistingVmFailsWhenBackedUpVolumeIsNoLongerAttached
• KVMGuruTest.testImportVirtualMachineFromBackupReinstatesRecordedDeviceIds — a data disk recorded on
  device id 5 must be re-attached on device id 5, and getNextAvailableDeviceId() must not be consulted.

Live testing — a patched build was deployed to a KVM + Ceph/RBD zone (management server and KVM
agents). The same 2-data-disk instance was restored three times in place:

• device ids stayed 0,1,2 on every restore (unpatched: 1,2 → 4,1 after the first)
• both volumes kept their own filesystem, verified with rbd export and from inside the guest
• a 5-data-disk instance backed up and restored with all checksums intact
• createVMFromBackup verified unaffected