Shaping the Performance of a Linux Distro: Inside Red Hat Enterprise Linux 7

Backstory

Red Hat’s Performance Engineering team is responsible for the performance of many of Red Hat’s products.  We cover existing products such as Red Hat Enterprise Linux, OpenStack Platform, OpenShift and Red Hat Enterprise Virtualization, as well as newer products like Ceph and CloudForms.

Although these days we contribute extensively to Red Hat’s cloud offerings, Red Hat Enterprise Linux remains a core responsibility as the building block for our ecosystem of customers and partners, plus much of Red Hat’s growing product portfolio.

Prior to beginning efforts on Red Hat Enterprise Linux 7 in earnest, Red Hat’s Platform Business Unit continued to deliver minor updates to Red Hat Enterprise Linux 5 and Red Hat Enterprise Linux 6 simultaneously, along with extending support to 10 years. Concurrent delivery of three major streams of Red Hat Enterprise Linux is only possible because of the depth of talent (and frankly, depth of character) present in our engineering teams.

How We Spent 2012 and 2013

As a fairly large team of performance engineers, we are committed to ensuring that Red Hat Enterprise Linux continues to be the trusted, rock-solid foundation that our customers and partners can rely upon.

The years 2012 and 2013 were spent in the lab making nerdy signs…

Notre Dame Sign

…scoping and characterizing the earliest builds of what would become Red Hat Enterprise Linux 7.

Fun stuff like crushing (and crashing!) upstream kernels, and working closely with partners and customers ensured the successful release Red Hat Enterprise Linux 7 in June of 2014.  As a company, this was easily one of our most important releases because we needed to not only impress our existing Red Hat Enterprise Linux customer base, but we also needed to build a solid foundation for our cloud portfolio.  Put simply, we had to get it right.

As some of the initial evaluators of upstream or test code, our most significant (mostly hidden) contribution to the open source community is providing early adopter feedback to our development teams during the design phase, when code is most flexible.  We need to know things like:

  • How it’s designed.
  • Where are the long poles, and can they be mitigated?
  • What makes sense for defaults?
  • Do we need tracepoints — how do I observe the critical sections under load?
  • Finally, a personal goal…how do we avoid surprising the sysadmins (?), who are again further downstream, and the very lifeblood of a infrastructure provider like Red Hat.

One of the key discoveries we made as a team throughout a decade of supporting customers on Red Hat Enterprise Linux, was an obsession with customer experience. We knew we wanted our customers to be absolutely delighted by the security, stability, and most importantly 😉 the performance of Red Hat Enterprise Linux 7.

Workload-specific Tuning…to the “Way-back Machine”!

We actually began the journey towards workload-specific tuning in Red Hat Enterprise Linux 5 with the introduction of ktune.  ktune provides some nascent tuning profiles for a very small set of workloads.  Ultimately, we did not expand much on ktune in the Red Hat Enterprise Linux 5 product.

With the GA of Red Hat Enterprise Linux 6.0, we introduced the tuned package.  I like to describe tuned as a “tuning profile delivery mechanism”, and ends up being our group’s primary feedback loop into Red Hat’s product line.  If you haven’t heard of tuned in Red Hat Enterprise Linux 6, trying it is as simple as:

# yum install tuned

We received lots of great feedback from our partners and customers about performance in Red Hat Enterprise Linux 6.  Coupled with additional R&D, this feedback allowed us to confidently expand the reach of tuned and its profiles beyond Red Hat Enterprise Linux. So in addition to the profiles shipped with Red Hat Enterprise Linux, Red Hat began to ship tuned profiles for a growing list of products such as Red Hat Enterprise Virtualization, OpenStack Platform, OpenShift, Red Hat Enterprise Linux Atomic Host and Red Hat Storage.

Based on extensive testing, these profiles typically boost performance in the double-digit percent range “for free”.  Just some of the benefits of Red Hat’s enterprise-hardened distribution(s) versus free/community software.

Time to Make the donuts RHEL 7

Through 2+ years and 4+ minor releases of Red Hat Enterprise Linux 6, our team gained valuable experience with the tuning profiles that our customers began adopting.  As we began working on Red Hat Enterprise Linux 7, the next logical step to improving out-of-the-box performance for our customers, was to meticulously validate upstream Linux kernel defaults through the lens of enterprise datacenter workloads and cloud environments.

Although it’s (likely) impossible to identify one specific set of tunings that helps all workloads, this effort led us to a set of changes to default kernel tunings that would boost performance of most workloads well past upstream defaults.  In fact, these changes were impactful enough, that we collaborated with our kernel engineering teams to further validate, and propose tuned be enabled by default in Red Hat Enterprise Linux 7.  Note: as a further optimization, tuned automatically customizes itself based on what version of Red Hat Enterprise Linux is installed, and whether it’s running on virtualization/cloud or bare metal.

After many months of discussion and testing, the default-enabled-tuned-proposal was accepted and delivered in Red Hat Enterprise Linux 7.0.  We are very proud of this software and its positive impact on our customers… made even more impactful because of the nature of operating system deployment in the cloud: massive scale-out via hybrid clouds, means we need intelligent operating systems that obviate the need for hand tuning and micro-optmization for all but the most performance-sensitive environments.

Red Hat’s Performance Engineering team hopes that you find Red Hat Enterprise Linux 7 the most enjoyable and performant Linux distribution available, and greatly looks forward to your feedback!

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s