Advice on how to deal with non-deterministic block device order in automation

Get Help
,
1

Here is a stripped down playbook that illustrates the problem:

- name: Add disk to VM
  community.vmware.vmware_guest:
    ... # auth and other required parameters
    disk:
    - size_gb: "10"  # already exists - sda
    - size_gb: "100" # will be added when playbook is run - should become sdb
  delegate_to: localhost

- name: Create a single partition on second disk
  community.general.parted:
    device: "/dev/sdb"
    number: 1

- name: Create file system on second disk
  community.general.filesystem:
    device: "/dev/sdb1"
    fstype: "ext4"

- name: ansible.posix.mount
  ansible.posix.mount:
    src: "/dev/sdb1"
    path: "/srv"
    fstype: "ext4"

The idea is simple. We add another virtual disk to the VM, create a partition on the disk, create a file system on that partition and mount the file system to a path.

While VM is up and running, this can be done without any problem. Secondary disk becomes sdb and everything is OK.

Unfortunately, after a reboot of a VM, by some random chance, disks can switch places - sdb becomes sda and vice versa and everything gets screwed up, even worse if you run this playbook once again after the fact.

The reason why sda and sdb can sometimes switch places is that as of kernel 5.3, devices are probed asynchronously and thus the order of disk naming is non deterministic. The situation is even worse if there are higher number of disks.

What everyone would say to this kind of problem is that you should use UUIDs but the conceptual problem here is that file system UUIDs don’t exist before file system is created. I could assign a label to the file system at the moment of creation and use it for mounting but not before that. Before that, a partition has to be created and proper device has to be selected.

The question in the end is does anyone know of a way to do this deterministically? Is there any device identifier that can be used that has predictable name in such a way that the first disk always has <some id 1> and second has <some id 2> and which can be determined right after disk is added to the VM?

Thanks in advance.

2

I presume one could use hardware address deterministically.

Here’s example of /dev/disk/by-path from my virtualbox machine with 2 disks:

ls -l /dev/disk/by-path

# lrwxrwxrwx 1 root root  9 Mar  2 10:09 pci-0000:00:0d.0-ata-1 -> ../../sda
# lrwxrwxrwx 1 root root  9 Mar  2 10:29 pci-0000:00:0d.0-ata-2 -> ../../sdb
3

/dev/disk/by-path is the closest thing I got to a solution but it is not universal among virtualization platforms. On VMware in particular it is weird:

lrwxrwxrwx. 1 root root   9 Feb 11 10:40 pci-0000:03:00.0-scsi-0:0:0:0 -> ../../sda
lrwxrwxrwx. 1 root root   9 Feb 14 12:27 pci-0000:03:00.0-scsi-0:0:1:0 -> ../../sdb

What would be the third disk :smiley: ?

4

For god sakes i hope pci-0000:03:00.0-scsi-0:0:2:0

5

VMware supports passing a serial number into the OS via advanced settings (disk.EnableUUID). It’s in the virtual hardware definition by default for Windows VMs but has to be added for Linux.

Alternatively you can try to use by-path determinism as indicated by @kristianheljas @bvitnik

1 Like
6

It sure is… on a particular version of VMware and particular version of VM hardware. It changed in the past and will probably change in the future.

This looks promising, wasn’t aware of it. I will have to look into it. Thanks.

We are also using vCloud Director an Terraform so I’ll have to check if this parameter can be set through vCD or only through vCenter. Is it preserved from VM template etc.

1 Like
7

I would just use LABEL on mount and ensure you label the partitions to match, that gives you the order ‘you want’, vs relying on id/device/etc

8

@bcoca LABELs for mounting sure. But before mounting… it’s a mess.

9

you get a predictable mount on first mount (next sdX), so you can label and proceed from there

10

I’m not really following. Are you saying I should take the advantage of the situation where I add one disk at a time so in that situation, device names are always predictable? Sure, but I described a simple example. In reality, we also deploy VMs with multiple disks from start and tasks loop over the list of disks. Depending on how VM booted the first time, device names could be nicely ordered or a total mess. The underlying issue still stands.