Setting up SSD Cache on LVM under Proxmox

In the following tutorial you will learn how to set up SSD cache on LVM under Proxmox, a Debian-based an open-source server virtualization environment.

We’re using:

  • four HGST SAS drives (it works just as well on any HDD)
  • 2 Intel SSDs (any other brand will work the same)
  • LSI hardware raid controller, AVAGO 3108 MegaRAID
  • a few Debian 8 (Jessie) tools: sources.list, megacli and megactl which contains megasasctl that you can use for a short overview by entering:
megasasctl -B -t -v

Follow the steps below:

  1. First,  build a raid10 on your HDDs and then go ahead and install Proxmox:
=> lsblk
NAME         MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT
sda            8:0    0   3.7T  0 disk
├─sda1         8:1    0  1007K  0 part
├─sda2         8:2    0   127M  0 part
└─sda3         8:3    0   3.7T  0 part
  ├─pve-root 252:0    0    27G  0 lvm  /
  ├─pve-swap 252:1    0     8G  0 lvm  [SWAP]
  └─pve-data 252:2    0   3.6T  0 lvm  /var/lib/vz

 

  1. Next, install the Intel Enterprise SSDs and build a raid0.
=> parted -l
Model: AVAGO SMC3108 (scsi)
Disk /dev/sda: 4000GB
(Hardware Raid10 (4x2TB SAS))
.....
Disk /dev/sdb: 199GB
(Hardware Raid0 (2x100SSD))

 

  1. Now, you can setup /dev/sdb as a physical volume:
 # pvcreate /dev/sdb
 
 # lvmdiskscan
   ...
   /dev/sdb  [  185.31 GiB] LVM physical volume
   ...
   1 LVM physical volume whole disk

 

When using dm-cache  it is very important to keep in mind that both logical volumes for data and for the cache must in the same volume group (“pve”), Which is why the existing volume group must be extended with the new cache device.

# vgscan
"Found volume group "pve" using metadata type lvm2"

# vgextend pve /dev/sdb
" Volume group "pve" successfully extended"

You can control it with “vgdisplay”

before:
# vgdisplay

VG Name  pve
  Metadata Areas  1
  Metadata Sequence No  4
  VG Access  read/write
  VG Status  resizable
  MAX LV  0
  Cur LV  3
  Open LV  3
  Max PV  0
  Cur PV  1
  Act PV  1
  VG Size  3.64 TiB
  PE Size  4.00 MiB
  Total PE  953567
  Alloc PE / Size  949472 / 3.62 TiB
  Free  PE / Size  4095 / 16.00 GiB
  VG UUID  QIzoZv-EoMX-ZWvR-LRj0-Eofo-o68H-i0vjMz

afterwards:
# vgdisplay
VG Name  pve
...
  Metadata Areas  2
  Metadata Sequence No  5
...
  Cur PV  2
  Act PV  2
  VG Size  3.82 TiB
  PE Size  4.00 MiB
  Total PE  1001006
  Alloc PE / Size  949472 / 3.62 TiB
  Free  PE / Size  51534 / 201.30 GiB
  VG UUID  QIzoZv-EoMX-ZWvR-LRj0-Eofo-o68H-i0vjMz 


 

  1. Next, you will start producing the important cache LV. There are two different cache LVs:
    – data LV
    – cache metadata LV

On a PV of 185GB, you could use about 0,5 GB / 512 MB as CacheMetaLV and 160 GB for CacheDataLV (estimated values).

# lvcreate -n CacheDataLV -L 160G pve /dev/sdb
" Logical volume "CacheDataLV" created."
# lvcreate -n CacheMetaLV -L 0.5G pve /dev/sdb
"Logical volume "CacheMetaLV" created.


 

  1. Now, engage the data cache LV and metadata cache LV in a single logical volume called “cache pool” like so:
# lvconvert --type cache-pool --cachemode writethrough --poolmetadata pve/CacheMetaLV pve/CacheDataLV

You will receive this message:

" WARNING: Converting logical volume pve/CacheDataLV and pve/CacheMetaLV to pool's data and metadata volumes.
  THIS WILL DESTROY CONTENT OF LOGICAL VOLUME (filesystem etc.)
Do you really want to convert pve/CacheDataLV and pve/CacheMetaLV? [y/n]: y
  Converted pve/CacheDataLV to cache pool." 

 

  1. See the final result by entering this command:
# lvs -a -o +devices

Output:
  LV  VG  Attr  LSize  Pool Origin Data%  Meta%  Move Log Cpy%Sync Convert Devices
  CacheDataLV  pve  Cwi---C--- 160.00g  CacheDataLV_cdata(0)
  [CacheDataLV_cdata] pve  Cwi------- 160.00g  /dev/sdb(0)
  [CacheDataLV_cmeta] pve  ewi------- 512.00m  /dev/sdb(40960)
  data  pve  -wi-ao----  3.59t  /dev/sda3(8960)
  [lvol0_pmspare]  pve  ewi------- 512.00m  /dev/sda3(949472)
  root  pve  -wi-ao----  27.00g  /dev/sda3(2048)
  swap  pve  -wi-ao----  8.00g  /dev/sda3(0)

 

  1. Finally, it’s time for the allocation of the cache pool to the meaning data LV (which is called “data” in proxmox). To do that, you must create the cache logical volume by combining the cache pool LV with the “data” LV.
# lvconvert --type cache --cachepool pve/CacheDataLV pve/data
" Logical volume pve/data is now cached."

 

And that’s it. You’re done setting up SSD Cache on LVM under Proxmox!


 

Sources:

Maxprox @ https://forum.proxmox.com/threads/can-we-use-lvm-cache-in-proxmox-4-x.25636/#post-134061

Leave a Reply