Introducing Virtual Data Optimizer to Reduce Cloud and On-premise Storage Costs

New to Red Hat Enterprise Linux 7.5, Virtual Data Optimizer (VDO) is a device mapper module which adds data reduction capabilities to the Linux block storage stack. VDO uses inline compression and data deduplication techniques to transparently shrink data as it is being written to storage media.

VDO combines three techniques — zero-block elimination, data deduplication, and data compression — to reduce data footprint.  The first of these, zero-block elimination, works by eliminating blocks of data consisting entirely of zeros while the second technique, data deduplication, eliminates identical copies of blocks of data that have already been stored. Finally, data compression is applied, which reduces the size of the unique blocks of data stored. By utilizing these techniques, VDO can dramatically increase the efficiency for both storage and network bandwidth utilization.

Why does VDO matter?

VDO can be used to save storage space and reduce costs.  Because it’s a feature of Red Hat Enterprise Linux (RHEL), it can be used anywhere RHEL is deployed; similar savings can be seen in both traditional data center and cloud-based deployments.

In the traditional data center, this means you can repurpose storage resources you already have and make more efficient use of future equipment. Enterprise data replication can also benefit from this efficiency since less data on storage means less data to replicate.

In the cloud, VDO allows you to cut storage costs as well.  Depending on your deployment needs, this can translate to lower costs per compute instance, lower costs for cloud-based external block storage, and reduced costs for long-term retention of data snapshots.  In addition, reduced footprint on premises or in the cloud translates to reduced bandwidth requirements to copy the data to or from the cloud or even between clouds.

The amount of data reduction you will see with VDO will vary depending on the type of data being stored and the workflow that creates and stores the data. Already compressed data types such as video or audio files will not benefit from this technology, but online backups, virtual machine, and container deployments will see substantial benefits. It is not uncommon for users to report 6:1 data reduction rates in mixed container and VM environments using deduplication and compression technologies such as those provided by VDO.  Several good candidates for data reduction with VDO are listed below.

How do I use VDO?

VDO operates at the Linux block layer.  This allows it to deliver benefits to local storage as well as to distributed block, file, or object storage solutions.

VDO Deployment Models

VDO sits on top of an existing block storage device which can be anything from a single local disk, to a RAID device, or even a LUN from an enterprise storage array.  Features like encryption and software RAID live below VDO, while components such as LVM and file systems are layered on top of a VDO device.

Configuring VDO for this case is straightforward and can be done in RHEL 7.5 using the vdo command line utility, or by integrating the provided Ansible module in to your playbooks that perform storage configuration.  A graphical interface for configuring VDO with Cockpit will be available in the coming months.

To configure VDO you need to know the following :

  • The name of the underlying block device (device option)
  • The name of the optimized block device that VDO will present (name option)
  • The logical size you wish to present to storage layers above VDO (vdoLogicalSize
    option)

If you don’t present this last parameter, VDO will create a volume that provides a 1:1 mapping between physical and logical blocks.  You will be able to grow the physical and logical size of the volume later on using the vdo growPhysical and vdo growLogical commands.

As a simple example, we create a VDO volume on the device /dev/sdb with the name vdo1 and logical size of 10TB by running the vdo create command:

# vdo create --device=/dev/sdb --name=vdo1 --vdoLogicalSize=10T

VDO instance 1 volume is ready at /dev/mapper/vdo1

The vdostats command can be used to verify that the volume was configured correctly.  Since VDO provides thin provisioning, this tool should also be used to determine how much free physical space is left on the underlying storage device:

# vdostats

Device               1K-blocks Used    Available   Use%  Space saving%
/dev/mapper/vdo1     117220824 4051080 113169744     3%            N/A

On successful configuration, the output of vdostats shows the device name for your new VDO volume (Device) along with statistics that  show the total number of blocks (1K-blocks), number of blocks in use (Used), number of blocks remaining (Available), percentage of the total blocks in use (Use%) and percentage of space savings (Space saving%).  

You’re now ready to use your new VDO volume just as you would any other block storage for configuring logical volumes and file systems to operate on the device.

Access VDO today in Red Hat Enterprise Linux 7.5, now available in the Red Hat Customer Portal to all customers with an active Red Hat Enterprise Linux subscription. To access and download Red Hat Enterprise Linux 7.5, please visit: https://access.redhat.com/products/red-hat-enterprise-linux.

  1. Any chance for a future blog on increasing the physical and logical space on a VDO device?

    I was able to use https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/#s2-disk-storage-parted-resize-part using fdisk after I doubled the virtual disk size from 16GB to 32GB.

    Following https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html-single/storage_administration_guide/#vdo-ig-increase-logical does not show the increased space when using vdostats –hu command.

    lsblk does show the 32GB disk size but not vdostats:

    sdb 8:16 0 32G 0 disk
    └─sdb1 8:17 0 32G 0 part
    └─vdb1 253:5 0 32G 0 vdo

    Device Size Used Available Use% Space saving%
    /dev/mapper/vdb1 16.0G 4.8G 11.2G 29% 4%

    I do know how to resize LVM volumes. Just can’t figure out VDO.

    1. You want to use the command:

      # vdo growPhysical –name=vdo

      That will tell VDO to use the additional physical space underneath it.

      For details see: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-ig-administering-vdo#vdo-ig-increase-physical

      If you want to grow the thinly provisioned space above vdo, because you’re seeing a good data reduction rate, you’d use:

      # vdo growLogical –name=vdo –vdoLogicalSize=

      For details on that see: https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/7/html/storage_administration_guide/vdo-ig-administering-vdo#vdo-ig-increase-logical

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

w

Connecting to %s