2020-02-01 • RHVH , OVIRT , LINUX

RHVH Ovirt

Red Hat documentation is pretty thorough on the subject of multipath, which is a lovely failover mechanism for connections between remote volumes provided by SANs and the host.^[1]^[2]

There is even this KCS that discusses the matter a little more: Internal disk device is being detected as multipath device

In setting up my bare metal servers for RHVH, after adding all my disks to the system as basically single RAID 0 disks, I found that the base image for RHVH had particular settings for creating a single path multipath entry for these. I guess I could have just used the multipath device names and created filesystems from those, but it seems a bit odd to me. Reading around in the docs suggests that if you have one path you shouldn’t use multipath at all, which kinda seems obvious. I’ll be using a cloud replication strategy, so local disks in RAID 0 seems sufficient for this use case. As you can see, my root filesystem is on another disk as well. If these fail and this host fails, I have other systems available. This is also my home lab and I don’t care about rebuild/down time if one of these disks goes out.

So how do we fix this?

[root@host-a ~]# lsblk
NAME                                       MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINT
sda                                          8:0    0  931G  0 disk
└─36842b2b04ff9d90025ab58e52a9663bc        253:7    0  931G  0 mpath
sdb                                          8:16   0  931G  0 disk
└─36842b2b04ff9d90025ab90d07fd7775b        253:9    0  931G  0 mpath
sdc                                          8:32   0  931G  0 disk
└─36842b2b04ff9d90025ab90f78228beb1        253:10   0  931G  0 mpath
sdd                                          8:48   0  931G  0 disk
└─36842b2b04ff9d90025ab910f8391d6c7        253:5    0  931G  0 mpath
sde                                          8:64   0  931G  0 disk
└─36842b2b04ff9d90025abddda04ad4d15        253:8    0  931G  0 mpath
sdf                                          8:80   0  931G  0 disk
└─36842b2b04ff9d90025abddf306245003        253:6    0  931G  0 mpath
sdg                                          8:96   1 59.6G  0 disk
├─sdg1                                       8:97   1    1G  0 part  /boot
└─sdg2                                       8:98   1 58.6G  0 part
  ├─rhvh-pool00_tmeta                      253:0    0    1G  0 lvm
  │ └─rhvh-pool00-tpool                    253:2    0 40.9G  0 lvm
  │   ├─rhvh-rhvh--4.3.7.1--0.20191211.0+1 253:3    0 13.9G  0 lvm   /
  │   ├─rhvh-pool00                        253:11   0 40.9G  0 lvm
  │   ├─rhvh-var_log_audit                 253:12   0    2G  0 lvm   /var/log/audit
  │   ├─rhvh-var_log                       253:13   0    8G  0 lvm   /var/log
  │   ├─rhvh-var                           253:14   0   15G  0 lvm   /var
  │   ├─rhvh-tmp                           253:15   0    1G  0 lvm   /tmp
  │   ├─rhvh-home                          253:16   0    1G  0 lvm   /home
  │   └─rhvh-var_crash                     253:17   0   10G  0 lvm   /var/crash
  ├─rhvh-pool00_tdata                      253:1    0 40.9G  0 lvm
  │ └─rhvh-pool00-tpool                    253:2    0 40.9G  0 lvm
  │   ├─rhvh-rhvh--4.3.7.1--0.20191211.0+1 253:3    0 13.9G  0 lvm   /
  │   ├─rhvh-pool00                        253:11   0 40.9G  0 lvm
  │   ├─rhvh-var_log_audit                 253:12   0    2G  0 lvm   /var/log/audit
  │   ├─rhvh-var_log                       253:13   0    8G  0 lvm   /var/log
  │   ├─rhvh-var                           253:14   0   15G  0 lvm   /var
  │   ├─rhvh-tmp                           253:15   0    1G  0 lvm   /tmp
  │   ├─rhvh-home                          253:16   0    1G  0 lvm   /home
  │   └─rhvh-var_crash                     253:17   0   10G  0 lvm   /var/crash
  └─rhvh-swap                              253:4    0    6G  0 lvm   [SWAP]

Reading this statement, "If you have previously created a multipath device without using the find_multipaths parameter and then you later set the parameter to yes, you may need to remove the WWIDs of any device you do not want created as a multipath device from the /etc/multipath/wwids file."^[3] we come to understand that since we have only one path to each of these devices, we can set find_multipaths yes in our configuration file while also removing the stored wwid.

Let’s back up the wwid file where these are stored. Then remove the entries corresponding to the drives we want to remove from multipath.

[root@host-a ~]# cp /etc/multipath/wwids{,.bk}

[root@host-a ~]# cat /etc/multipath/wwids.bk
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36842b2b04ff9d90025ab912284b3a47e/
/36842b2b04ff9d90025ab913085928cb2/
/36842b2b04ff9d90025ab90d07fd7775b/
/36842b2b04ff9d90025ab58e52a9663bc/
/36842b2b04ff9d90025ab910f8391d6c7/
/36842b2b04ff9d90025ab90f78228beb1/
/36842b2b04ff9d90025abddda04ad4d15/
/36842b2b04ff9d90025abddf306245003/

[root@host-a ~]# cat /etc/multipath/wwids
# Multipath wwids, Version : 1.0
# NOTE: This file is automatically maintained by multipath and multipathd.
# You should not need to edit this file in normal circumstances.
#
# Valid WWIDs:
/36842b2b04ff9d90025ab912284b3a47e/
/36842b2b04ff9d90025ab913085928cb2/

We can also see these registered by multipath prior to reboot:

[root@host-a ~]# multipath -ll
36842b2b04ff9d90025ab90d07fd7775b dm-9 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:1:0 sdb 8:16 active ready running
36842b2b04ff9d90025ab910f8391d6c7 dm-5 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:3:0 sdd 8:48 active ready running
36842b2b04ff9d90025abddda04ad4d15 dm-8 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:4:0 sde 8:64 active ready running
36842b2b04ff9d90025abddf306245003 dm-6 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:5:0 sdf 8:80 active ready running
36842b2b04ff9d90025ab90f78228beb1 dm-10 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:2:0 sdc 8:32 active ready running
36842b2b04ff9d90025ab58e52a9663bc dm-7 DELL    ,PERC 6/i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='service-time 0' prio=1 status=active
  `- 1:2:0:0 sda 8:0  active ready running

Normally we would edit /etc/multipath.conf but the RHVH image has things configured a little differently, and we see that this file is managed by vdsm.

# head -8 /etc/multipath.conf
# VDSM REVISION 1.8

# This file is managed by vdsm.
#
# The recommended way to add configuration for your storage is to add a
# drop-in configuration file in "/etc/multipath/conf.d/<mydevice>.conf".
# Settings in drop-in configuration files override settings in this
# file.

Instead, we can add our own .conf file to get picked up and parsed after /etc/multipath.conf. Here we will change the settings as needed.

# cat /etc/multipath/conf.d/my.conf
defaults {
        find_multipaths yes
}

Now after a reboot we can see the following:

[root@host-a ~]# lsblk
NAME                                                     MAJ:MIN RM  SIZE RO TYPE MOUNTPOINT
sda                                                        8:0    0  931G  0 disk
sdb                                                        8:16   0  931G  0 disk
sdc                                                        8:32   0  931G  0 disk
sdd                                                        8:48   0  931G  0 disk
sde                                                        8:64   0  931G  0 disk
sdf                                                        8:80   0  931G  0 disk
sdg                                                        8:96   1 59.6G  0 disk
├─sdg1                                                     8:97   1    1G  0 part /boot
└─sdg2                                                     8:98   1 58.6G  0 part
  ├─rhvh-pool00_tmeta                                    253:0    0    1G  0 lvm
  │ └─rhvh-pool00-tpool                                  253:2    0 40.9G  0 lvm
  │   ├─rhvh-rhvh--4.3.7.1--0.20191211.0+1               253:3    0 13.9G  0 lvm  /
  │   ├─rhvh-pool00                                      253:5    0 40.9G  0 lvm
  │   ├─rhvh-var_log_audit                               253:6    0    2G  0 lvm  /var/log/audit
  │   ├─rhvh-var_log                                     253:7    0    8G  0 lvm  /var/log
  │   ├─rhvh-var                                         253:8    0   15G  0 lvm  /var
  │   ├─rhvh-tmp                                         253:9    0    1G  0 lvm  /tmp
  │   ├─rhvh-home                                        253:10   0    1G  0 lvm  /home
  │   └─rhvh-var_crash                                   253:11   0   10G  0 lvm  /var/crash
  ├─rhvh-pool00_tdata                                    253:1    0 40.9G  0 lvm
  │ └─rhvh-pool00-tpool                                  253:2    0 40.9G  0 lvm
  │   ├─rhvh-rhvh--4.3.7.1--0.20191211.0+1               253:3    0 13.9G  0 lvm  /
  │   ├─rhvh-pool00                                      253:5    0 40.9G  0 lvm
  │   ├─rhvh-var_log_audit                               253:6    0    2G  0 lvm  /var/log/audit
  │   ├─rhvh-var_log                                     253:7    0    8G  0 lvm  /var/log
  │   ├─rhvh-var                                         253:8    0   15G  0 lvm  /var
  │   ├─rhvh-tmp                                         253:9    0    1G  0 lvm  /tmp
  │   ├─rhvh-home                                        253:10   0    1G  0 lvm  /home
  │   └─rhvh-var_crash                                   253:11   0   10G  0 lvm  /var/crash
  └─rhvh-swap                                            253:4    0    6G  0 lvm  [SWAP]

[root@host-a ~]# multipath -ll
[root@host-a ~]#

Notes

Location of gluster install logs when installing RHV ovirt engine from the GUI:

-rw-r--r--. 1 root root 26K Jan 18 15:03 /var/log/cockpit/ovirt-dashboard/gluster-deployment.log

1. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/ignore_localdisk_procedure

2. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/config_file_defaults

3. https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/dm_multipath/config_file_blacklist