VSPHERE 6.5 DRS AND MEMORY BALANCING IN NON-OVERCOMMITTED CLUSTERS
DRS is over a decade old and is still going strong. DRS is aligned with the premise of virtualization, resource sharing and overcommitment of resources. DRS goal is to provide compute resources to the active workload to improve workload consolidation on a minimal compute footprint. However, virtualization surpassed the original principle of workload consolidation to provide unprecedented workload mobility and availability. With this change of focus, many customers do not overcommit on memory. A lot of customers design their clusters to contain (just) enough memory capacity to ensure all running virtual machines have their memory backed by physical memory. In this scenario, DRS behavior should be adjusted as it traditionally focusses on active memory use. vSphere 6.5 provides this option in the DRS cluster settings. By ticking the box “Memory Metric for Load Balancing” DRS uses the VM consumed memory for load-balancing operations. Please note that DRS is focussed on consumed memory, not configured memory! DRS always keeps a close eye on what is happening rather than accepting static configuration. Let’s take a closer look at DRS input metrics of active and consumed memory. Out-of-the-box DRS Behavior During load balancing operation, DRS calculates the active memory demand of the virtual machines in the cluster. The active memory represents the working set of the virtual machine, which signifies the number of active pages in RAM. By using the working-set estimation, the memory scheduler determines which of the allocated memory pages are actively used by the virtual machine and which allocated pages are idle. To accommodate a sudden rapid increase of the working set, 25% of idle consumed memory is allowed. Memory demand also includes the virtual machine’s memory overhead. Let’s use a 16 GB virtual machine as an example of how DRS calculates the memory demand. The guest OS running in this virtual machine has touched 75% of its memory size since it was booted, but only 35% of its memory size is active. This means that the virtual machine has consumed 12288 MB and 5734 MB of this is used as active memory. As mentioned, DRS accommodate a percentage of the idle consumed memory to be ready for a sudden increase in memory use. To calculate the idle consumed memory, the active memory 5734 MB is subtracted from the consumed memory, 12288 MB, resulting in a total 6554 MB idle consumed memory. By default, DRS includes 25% of the idle consumed memory, i.e. 6554 * 25% = +/- 1639 MB. The virtual machine has a memory overhead of 90 MB. The memory demand DRS uses in its load balancing calculation is as follows: 5734 MB + 1639 MB + 90 MB = 7463 MB. As a result, DRS selects a host that has 7463 MB available for this machine if it needs to move this virtual machine to improve the load balance of the cluster. Memory Metric for Load Balancing Enabled When enabling the option “Memory Metric for Load Balancing” DRS takes into account the consumed memory + the memory overhead for load balancing operations. In essence, DRS uses the metric Active + 100% IdleConsumedMemory. vSphere 6.5 update 1d UI client allows you to get better visibility in the memory usage of the virtual machines in the cluster. The memory utilization view can be toggled between active memory and consumed memory. Recently, Adam Eckerle on Twitter published a great article that outlines all the improves of vSphere 6.5 Update 1d. Go check it out. Animated Gif courtesy of Adam. When reviewing the cluster it shows that the cluster is pretty much balanced. When looking at the default view of the sum of Virtual Machine memory utilization (active memory). It shows that ESXi host ESXi02 is busier than the others. However since the active memory of each host is less than 20% and each virtual machine is receiving the memory they are entitled to, DRS will not move virtual machines around. Remember, DRS is designed to create as little overhead as possible. Moving one virtual machine to another host to make the active usage more balanced, is just a waste of compute cycles and network bandwidth. The virtual machines receive what they want to receive now, so why take the risk of moving VMs? But a different view of the current situation is when you toggle the graph to use consumed memory. Now we see a bigger difference in consumed memory utilization. Much more than 20% between ESXi02 and the other two hosts. By default DRS in vSphere 6.5 tries to clear a utilization difference of 20% between hosts. This is called Pair-Wise Balancing. However, since DRS is focused on Active memory usage, Pair-Wise Balancing won’t be activated with regards to the 20% difference in consumed memory utilization. After enabling the option “Memory Metric for Load Balancing” DRS rebalances the cluster with the optimal number of migrations (as few as possible) to reduce overhead and risk. Active versus Consumed Memory Bias If you design your cluster with no memory overcommitment as guiding principle, I recommend to test out the vSphere 6.5 DRS option “Memory Metric for Load Balancing”. You might want to switch DRS to manual mode, to verify the recommendations first.
EXPLAINER ON #SPECTRE & #MELTDOWN BY GRAHAM SUTHERLAND
Sometimes you stumble across a brilliant Twitter thread, so good, that it should never be lost. Graham Sutherland (@gsuberland) helped the world in understanding the Spectre and Meltdown bugs. I’m publishing his tweet thread in text form as this is just the best explanation of the bugs I’ve seen. Please note that VMware has released its response for Bounds-Check Bypass (CVE-2017-5753), Branch Target Injection (CVE-2017-5715) & Rogue Data Cache Load (CVE-2017-5754) - AKA Meltdown & Spectre.
FREE VSPHERE 6.5 HOST RESOURCES DEEP DIVE E-BOOK
In June of this year, Niels and I published the vSphere 6.5 Host Resources Deep Dive, and the community was buzzing. Twitter exploded, and many community members provided rave reviews. This excitement caught Rubriks attention, and they decided to support the community by giving away 2000 free copies of the printed version at VMworld. The interest was overwhelming, before the end of the second signing session in Barcelona we ran out of books. A lot of people reached out to Rubrik and us to find out if they could get a free book as well. This gave us an idea, and we sat down with Rubrik and the VMUG organization to determine how to cater the community. We are proud to announce that you can download the e-book version (PDF only) for free at rubrik.com. Just sign up and download your full e-book copy here. Spread the word! And if you like, thank @Rubrik and @myVMUG for their efforts to help the VMware community advance. https://www.youtube.com/watch?v=a4spq5B4wtg
WHAT IF THE VM MEMORY CONFIG EXCEEDS THE MEMORY CAPACITY OF THE PHYSICAL NUMA NODE?
This week I had the pleasure to talk to a customer about NUMA use-cases and a very interesting config came up. They have a VM with a particular memory configuration that exceeds the ESXi host NUMA node memory configuration. This scenario is covered in the vSphere 6.5 Host Resources Deep Dive, excerpt below. Memory Configuration The scenario described happens in multi-socket systems that are used to host monster-VMs. Extreme memory footprint VMs are getting more common by the day. The system is equipped with two CPU packages. Each CPU package contains twelve cores. The system has a memory configuration of 128 GB in total. The NUMA nodes are symmetrically configured and contain 64 GB of memory each. However, if the VM requires 96 GB of memory, a maximum of 64 GB can be obtained from a single NUMA node. This means that 32 GB of memory could become remote if the vCPUs of that VM can fit inside one NUMA node. In this case, the VM is configured with 8 vCPUs. The VM fits from a vCPU perspective inside one NUMA node, and therefore the NUMA scheduler configures for this VM a single virtual proximity domain (VPD) and a single a load-balancing group which is internally referred to as a physical proximity domain (PPD).
A VSPHERE FOCUSED GUIDE TO THE INTEL XEON SCALABLE FAMILY - MEMORY SUBSYSTEM
The Intel Xeon Scalable Family introduces a new platform (Purley). The most prominent change regarding system design is the memory subsystem. More Memory Bandwidth and Consistency in Speed The new memory subsystem supports the same number of DIMMs per CPU as the previous models. However, it’s wider and less deep. What I mean by that is that the last platform (Grantley) supported up to three DIMMs per channel (DPC) and made use of four channels. In total, the Grantley platform supported up to twelve DIMMs per CPU. Purley increases the number of channels from four to six but reduces the numbers of supported DIMMs per channel from three to two. Although this sounds like a potato, potato; tomato, tomato discussion it provides a significant increase in bandwidth while ensuring consistency in speed during a scaling up exercise. Let’s take a closer look. DIMMs per Memory Channel Depending on the DIMM slot configuration of the server board, multiple DIMMs are supported per channel. The E5-2600 V-series supports up to 3 DIMMs per channel (3 DPC). Using more DIMMs per channel provides the largest capacity, but unfortunately, it impacts the operational speed of memory. A DIMM groups memory chips into ranks. DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). With the addition of each rank, the electrical load on the channel increases. And as more ranks are used in a memory channel, memory speed drops restricting the use of additional memory. Therefore in certain configurations, DIMMs will run slower than their listed maximum speeds. This reduction in speed occurs when 3 DIMMs per channel is used.
A VSPHERE FOCUSED GUIDE TO THE INTEL XEON SCALABLE FAMILY
Intel released the much-anticipated Skylake Server CPU this year. Moving away from the E5-2600-v moniker, Intel names the new iteration of its server CPU the Intel Xeon Scalable Family. On top of this it uses precious metal categories such as Platinum and Gold to identify different types and abilities. Upholding the tradition, the new Xeon family contains more cores than the previous Xeon version. The new top-of-the-line CPU offers 28 cores on a single processor die, memory speeds are now supported up to 2666 MHz. However, the biggest appeal for vSphere datacenters is the new “Purley” platform and its focus on increasing bandwidth between possibly every component possible. In this series, we are going to look at the new Intel Xeon Scalable family microarchitecture and which functions help to advance vSphere datacenters.
VMWARE CLOUD ON AWS TECHNICAL OVERVIEW
Please note that this information can be outdated due to the ongoing changes of this cloud service. Please consult the https://cloud.vmware.com/vmc-aws/roadmap for recent information about the latest release Yesterday we launched the VMware Cloud on AWS service. VMware Cloud on AWS allows you to run your applications across private, public, and hybrid cloud environments based on VMware vSphere, with optimized access to AWS services. The Cloud SDDC consists of vSphere, NSX and vSAN technology to provide you a familiar environment which can be managed an operated with your current tool and skill set. By leveraging bare-metal AWS infrastructure the Cloud SDDC can scale in an unprecedented way.
GET YOUR FREE BOOK AT VMWORLD
At VMworld, the presenters of the following sessions will be giving away free copies of the Host Deep Dive book to the audience. Saturday Performance Bootcamp Mark Achtemichuk Saturday, Aug 26, 8:00 a.m. - 5:00 p.m. More information about pre-VMworld Performance Bootcamp Sunday An Introduction to VMware Software-Defined Storage [STO2138QU] Lee Dilworth, Principal Systems Engineer, VMware Sunday, Aug 27, 4:00 p.m. - 4:30 p.m. | Oceanside C, Level 2 Monday A Deep Dive into vSphere 6.5 Core Storage Features and Functionality [SER1143BU] Cody Hosterman, Technical Director–VMware Solutions, Pure Storage Cormac Hogan, Director - Chief Technologist, VMware Monday, Aug 28, 11:30 a.m. - 12:30 p.m. | Mandalay Bay Ballroom G, Level 2 Extreme Performance Series: Benchmarking 101 [SER2723BUR] Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Monday, Aug 28, 4:00 p.m. - 5:00 p.m. | Mandalay Bay Ballroom B, Level 2 Maximum Performance with Mark Achtemichuk [VIRT2368GU] Mark Achtemichuk, Staff Engineer, Performance, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Reef E, Level 2 The Top 10 Things to Know About vSAN [STO1264BU] Duncan Epping, Chief Technologist, VMware Cormac Hogan, Director - Chief Technologist, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Mandalay Bay Ballroom H, Level 2 VMware vSAN: From 2 Nodes to 64 Nodes, Architecting and Operating vSAN Like a VCDX for Scalability and Simplicity [STO2114BU] Greg Mulholland, Principal Systems Engineer, VMware Jeff Wong, Customer Success Architect, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Surf E, Level 2 Tuesday Extreme Performance Series: Performance Best Practices [SER2724BU] Reza Taheri, Principal Engineer, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Tuesday, Aug 29, 2:30 p.m. - 3:30 p.m. | Oceanside D, Level 2 Wednesday vSphere 6.5 Host Resources Deep Dive: Part 2 [SER1872BU] Frank Denneman, Senior Staff Architect, VMware Niels Hagoort, Owner, HIC (Hagoort ICT Consultancy) Wednesday, Aug 30, 8:30 a.m. - 9:30 a.m. | Breakers E, Level 2 Extreme Performance Series: Benchmarking 101 [SER2723BUR] Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Wednesday, Aug 30, 8:30 a.m. - 9:30 a.m. | Lagoon L, Level 2 vSAN Networking and Design Best Practices [STO3276GU] John Nicholson, Senior Technical Marketing Manager, VMware Wednesday, Aug 30, 11:30 a.m. - 12:30 p.m. | Reef C, Level 2 vSAN Hardware Deep Dive Panel [STO1540PU] Ed Goggin, Staff Engineer 2, VMware David Edwards, Principal Engineer, Director Solutions, Resurgent Technology Ken Werneburg, Group Manager Technical Marketing, VMware Jeffrey Taylor, Technical Director, VMware Ron Scott-Adams, Hyper-Converged Systems Engineer, VMware Wednesday, Aug 30, 1:00 p.m. - 2:00 p.m. | Mandalay Bay Ballroom D, Level 2 A Closer Look at vSAN Networking Design and Configuration Considerations [STO1193BU] Cormac Hogan, Director - Chief Technologist, VMware Andreas Scherr, Senior Solution Architect, VMware Wednesday, Aug 30, 4:00 p.m. - 5:00 p.m. | Mandalay Bay Ballroom G, Level 2 Thursday Virtual Volumes Technical Deep Dive [STO2446BU] Patrick Dirks, Sr. Manager, VMware Pete Flecha, Sr Technical Marketing Architect, VMware Thursday, Aug 31, 10:30 a.m. - 11:30 a.m. | Oceanside B, Level 2 Book Signing We will be doing two book signing sessions as well. At the Rubrik booth #412 on Monday, Aug 28, 2:00 p.m. - 3:00 p.m. At the VMworld Book store on Tuesday, Aug 29, 11:30 a.m. - 12:00 p.m. Or just feel free to approach us when you see us walking by.
REGISTER NOW FOR VMWARE CLOUD ON AWS TECHNICAL DEEP DIVE SESSION
I noticed that our technical deep dive session on VMware Cloud on AWS was added to the content catalog of VMworld. In this session, Ray Budavari and I will cover the VMware Cloud on AWS infrastructure in detail. For the first time, we are allowed to uncover details about the host configuration, the vSAN infrastructure and of course network topology. We explore advanced features such as Elastic DRS and Autoremediation HA. The last 15 minutes of our session allows you to ask questions about VMC. Please register if you don’t want to miss this session. Both Ray and I have a full schedule, therefore we are unable to schedule a repeat of this session during VMworld US. Session details: VMware Cloud on AWS: A Technical Deep Dive [LHC2384BU] Frank Denneman, Senior Staff Architect, VMware Ray Budavari, Senior Staff Technical Product Manager, VMware Tuesday, Aug 29, 3:30 p.m. - 4:30 p.m. Session Type: Breakout Session Track: Integrate Public Clouds Integrate Public Clouds: Leverage Hybrid Clouds Product and Topics: NSX, vCenter, vSAN, vSphere Technical Level: Technical – Advanced Session Hashtag: #LHC2384BU
VMWARE CLOUD ON AWS - PREDICTABLE CAPACITY PROVISIONING
In preparation for the VMworld Session LHC2971BU - Managing Your Hybrid Cloud with VMware Cloud on AWS which I’m co-presenting with Emad Younis, I asked the following question on Twitter: https://twitter.com/FrankDenneman/status/889841095768776704 And the number of answers were overwhelming. The stories were a bit underwhelming. Funny to see that we strive to automate every single step in the process. Guys like Alan, Luc, and William help the community to create scripted installs and configuration of the ESXi host. Creating a consistent, human-error free, rapid process. Shaving off valuable time of the time-consuming server provisioning process. Some organizations incorporate the vRealize suite to create a consistent user experience for the entire IT services portfolio. Interestingly enough, the overall lead time seems mostly impacted by internal acquisition processes. To give a few examples: https://twitter.com/j0sema/status/889945617312735232 https://twitter.com/tx_drewdad/status/889849858701524992 https://twitter.com/VTsnowboarder42/status/889995588669984772 https://twitter.com/PvdBree/status/889863788983455744 And the list goes on and on. In most organizations, the procurement process is rigid, well-defined process. However, the lead time of the acquisition process is either unpredictable and inconsistent. The overall message is that it cripples the agility of the IT organization. IT organizations need to react fast to the business needs. Resource management of current workload is difficult enough, figuring out what to expect in the upcoming months is challenging. Unfortunately, the introduction of new workload does not follow a linear demand curve. To cater the (possible) future needs of the customer, the order is either doubled in size, or onboarding of new workloads is gated. Either impacting the bottom-line of the company or the ability to facilitate IT services properly. https://twitter.com/BobbyFantast1c/status/889845274805448704 In essence, the CAPEX element of server resource acquisition massively impacts or hinders the execution ability of the IT organization. Strategizing CAPEX\OPEX is not a part of the core focus of many admins and architects, it does affect their means of execution. As demonstrated by the many tweets. With VMware Cloud on AWS, the host resource acquisition process shifts from CAPEX to OPEX. Removing the inconsistent and unpredictable procurement process, allowing for a faster, consistent and predictable method of providing compute, storage and networking resources. VMware Cloud on AWS (VMC) makes my resource-management heart beat faster. By leveraging the AWS operation model, the SDDC cluster running on the AWS infrastructure is resizable by a click of a button. Right-click on the cluster and select resize. Just select the number of hosts you want to add and within moments you will get new dedicated physical hardware added to your cluster. Ready to provide the resources your new workloads require. Resize means you can remove the resources as well, which in result your costs will go down as well. Due to the combined fleet management of AWS and VMC, the new ESXi hosts are fully configured and ready to welcome new workload. All VMkernel and logical networks are automatically configured and made available. The vSAN datastore is automatically expanded with the host-local NVMe flash devices provided by the new hosts. DRS detects the new physical resources and automatically rebalances the cluster, provided the most optimal resource availability. Elastic DRS and Autoremedation HA allows for an automatize method or adding and removing dedicated hardware resources, but these topics will be covered in a different article. From a resource management perspective, a mindset shift will happen. VMC allows you to reduce the time spent on infrastructure configuration and management and allows you to focus more on resource consumption. What cluster configuration is required in the upcoming months? What is my burst strategy? Unfortunately, I can’t go into detail as the service is not released yet. VMworld boasts an exciting line up of VMware Cloud on AWS sessions. I will be hosting a meet the expert on resource management at both VMworlds, sign up if you want to talk more about this exciting new technology