# FrankDenneman.nl Knowledge Corpus Site: https://frankdenneman.nl Author: Frank Denneman This file contains machine-readable versions of articles from frankdenneman.nl. Primary areas of coverage: - AI infrastructure architecture - GPU resource management - vGPU placement behavior - MIG partitioning and placement geometry - NUMA-aware infrastructure design - Virtualization performance - CPU and memory locality For a curated entry point, see: https://frankdenneman.nl/llms.txt For a structured map of the AI Infrastructure series, see: https://frankdenneman.nl/ai-infrastructure/ ================================================================================ Title: Topology-Aware Multi-GPU VM Placement URL: https://frankdenneman.ai/2026-03-31-Topology-Aware-Multi-GPU-VM-Placement/ Date: 2026-03-31 Architecting AI Infrastructure Series - Part 11 A multi-GPU VM isn’t only asking for multiple devices. It’s asking for a specific communication geometry. This distinction matters. When a platform team provisions a VM for LLM inference or fine-tuning, they’re not simply allocating two units of compute. They’re allocating two GPUs that can communicate at hundreds of gigabytes per second over NVLink. Two GPUs on the same server that must communicate via PCIe won’t deliver the same result. The platform must solve three distinct problems: Discovery: Identifying GPU devices and the interconnect relationships between them. Exposure: Presenting interconnect relationships as selectable resource shapes. Placement: Ensuring that when a VM requests a specific communication geometry, the platform assigns hardware that satisfies that requirement. VCF and vSphere addresses these challenges through a layered approach. For PCIe GPUs with direct NVLink connections, the NVIDIA GPU Manager discovers link topology and exposes it through the Device Group abstraction. For NVSwitch-based HGX systems, NVIDIA Fabric Manager provides partition discovery and activation APIs that the hypervisor integrates to manage multi-tenant GPU access. This article focuses on topology-aware resource isolation at the VM level: how vSphere Device Groups, NVIDIA vGPU Manager, and NVIDIA Fabric Manager partitions work together to ensure that multi-GPU VMs receive the communication geometry they require. A later part in this series covers Dynamic Resource Allocation (DRA), which brings similar topology awareness to Kubernetes scheduling for containerized workloads. Discovery of NVLink Domains Before the platform can expose GPU interconnect relationships, it must discover them. For PCIe GPUs with direct NVLink connections, this discovery is handled by the NVIDIA GPU Manager, the ESXi host driver component that runs within the ESXi kernel. When ESXi boots, the GPU Manager enumerates all NVIDIA GPUs and queries each GPU’s NVLink ports to determine connectivity. It identifies which ports are active, what they connect to, and the aggregate bandwidth available. From this information, it constructs a map of NVLink domains: groups of GPUs that share a direct, high-bandwidth interconnect. Consider a server with four A100 80GB PCIe GPUs. The standard nvidia-smi output lists four devices: $ nvidia-smi --query-gpu=index,gpu_bus_id,name --format=csv index, pci.bus_id, name 0, 00000000:4A:00.0, NVIDIA A100 80GB PCIe 1, 00000000:61:00.0, NVIDIA A100 80GB PCIe 2, 00000000:CA:00.0, NVIDIA A100 80GB PCIe 3, 00000000:E1:00.0, NVIDIA A100 80GB PCIe Nothing in that output indicates which GPUs share NVLink connectivity. The vCenter UI presents the same four devices without revealing this relationship. The topology only becomes visible with nvidia-smi topo -m: $ nvidia-smi topo -m GPU0 GPU1 GPU2 GPU3 GPU0 X NV12 SYS SYS GPU1 NV12 X SYS SYS GPU2 SYS SYS X NV12 GPU3 SYS SYS NV12 X GPU0 and GPU1 connect via 12 NVLinks. GPU2 and GPU3 form a second NVLink pair. Cross-domain communication traverses PCIe and the NUMA interconnect. Two NVLink domains exist, but the PCIe bus addresses give no indication of which GPUs belong to which domain. The GPU Manager discovers both domains and registers them separately. In a server with a single A100 NVL pair, the GPU Manager discovers two GPUs and detects the active NVLink connections between them. It registers this as a two-GPU NVLink domain. Both GPUs can still be assigned individually to separate VMs. Fractional vGPU profiles are also an option. In both cases, assigning GPUs individually or fractionally disables NVLink. The interconnect only functions when both GPUs are assigned together as a full-memory pair to a single VM. For PCIe NVLink configurations, discovery is static. The physical bridge connections don’t change at runtime. Once the GPU Manager identifies the NVLink domains at boot, that topology remains constant until hardware changes. Device Groups The GPU Manager discovers NVLink domains. vSphere exposes them through Device Groups. Device Groups shift the operational model for multi-GPU provisioning. Instead of navigating host inventory, identifying PCIe addresses, and manually mapping which GPUs share NVLink connectivity, an administrator selects a geometry shape during VM configuration. A 2-GPU NVLink domain. A 4-GPU partition. The platform handles the translation from abstract shape to physical hardware. A Device Group represents a set of hardware devices that share a specific interconnect relationship. When an administrator assigns a Device Group to a VM, they establish a contract: this VM requires devices with this communication geometry. vSphere enforces that contract through placement decisions and allocation tracking. Device Groups abstract away PCIe topology. Rather than selecting GPU 0 at address 4A:00.0 and GPU 1 at address 61:00.0, an administrator selects “a two-GPU NVLink domain.” The platform resolves that shape to specific hardware at placement time. This decoupling allows the same VM configuration to deploy across different hosts in a cluster, each with their own PCIe enumeration, as long as a matching Device Group is available. The naming convention encodes the shape. A Device Group named Nvidia:2@grid_a100d-80c%NVLink indicates two NVIDIA A100 80GB GPUs assigned as full-memory profiles with NVLink connectivity. The prefix denotes the GPU count, grid_a100d-80c identifies the full GPU profile, and the %NVLink suffix signals that this is a connected pair rather than two independent devices. A single A100 GPU without NVLink would simply be grid_a100d-80c. Instead of adding individual GPUs to a VM one at a time, hoping the combination preserves the desired interconnect, the administrator specifies the entire requirement as a single construct. One selection, one contract, one placement decision. NVLink-connected GPUs are one type of Device Group. A second type pairs a GPU with a network interface card that shares the same PCIe switch. This configuration optimizes GPUDirect RDMA traffic, enabling the GPU to communicate directly with remote systems without staging data through system memory. As multi-node inference becomes more common, assigning optimal GPU and NIC combinations through Device Groups provides a path toward topology-aware distributed deployments. Cluster and Host Placement A Device Group defines what a VM needs. DRS determines where that need can be satisfied. When a VM is configured with a Device Group, the requirement becomes part of its resource contract. At power-on, DRS together with Assignable Hardware evaluates which ESXi hosts can provide the specified Device Group shape. ESXi Hosts without a matching available capacity are filtered out. DRS then applies its Goodness calculation across the remaining eligible ESXi hosts to determine optimal placement. Part 3 covers this two-phase process in detail. Once DRS selects a host, the platform binds specific hardware to the VM. When a VM with a 4-GPU Device Group powers on, one of the available 4-GPU NVLink domains is assigned. The specific GPUs, with their specific PCIe addresses, become bound to that VM for the duration of its lifecycle. For PCIe NVLink systems, this binding is straightforward: fixed physical bridges define which GPUs form valid pairs. HGX systems introduce a different dynamic. Eight GPUs connect through NVSwitch fabric, and valid partition configurations are defined by NVIDIA Fabric Manager at runtime. NVSwitch and Fabric Manager HGX systems replace physical NVLink bridges with NVSwitch fabric. Instead of fixed point-to-point connections between GPU pairs, NVSwitch provides all-to-all connectivity. This changes both the discovery model and the software stack required to manage it. Software Stack Three NVIDIA components work together on NVSwitch-equipped hosts: the ESXi kernel GPU driver, the NVIDIA vGPU Manager, and NVIDIA Fabric Manager. The ESXi kernel GPU driver provides low-level GPU access. vGPU Manager handles device enumeration and vGPU profile management. Fabric Manager discovers the NVSwitch topology, initializes NVLink connections, and defines which GPU groupings form valid partitions. During host initialization, Fabric Manager reports valid partition configurations to hostd. Assignable Hardware uses this information to build a tree of all possible NVSwitch GPU partitions. This tree is constructed at boot, before any VM powers on, giving the platform a complete map of valid multi-GPU configurations. For an HGX H100, the available device groups reflect these partition options: Nvidia:2@nvidia_h100xm-80c%NVLink Nvidia:4@nvidia_h100xm-80c%NVLink Nvidia:8@nvidia_h100xm-80c%NVLink GPU Module IDs HGX systems introduce an additional identifier: the GPU Module ID. This value reflects the GPU’s physical position and determines which partition configurations are valid. The Module ID differs from the GPU index derived from PCIe enumeration. To retrieve the mapping: $ for id in $(nvidia-smi --query-gpu=index --format=csv,noheader); do > mod=$(nvidia-smi -a -i $id | grep "Module ID" | head -n1 | awk '{print $NF}') > echo "GPU $id: Module ID: $mod" > done GPU 0: Module ID: 2 GPU 1: Module ID: 4 GPU 2: Module ID: 1 GPU 3: Module ID: 3 GPU 4: Module ID: 7 GPU 5: Module ID: 5 GPU 6: Module ID: 6 GPU 7: Module ID: 8 The GPU index does not match the physical Module ID. Fabric Manager uses Module IDs when defining partitions. Module IDs 1 through 4 occupy one half of the GPU capacity. Module IDs 5 through 8 occupy the other half. These physical groupings determine valid partition boundaries. The platform maps between Module IDs and PCIe addresses when binding hardware to VMs. Dynamic Partitioning Unlike PCIe NVLink, NVSwitch partition topology is not fixed at boot. Fabric Manager activates and deactivates partitions at runtime based on VM lifecycle events. Partition activation involves more than updating a routing table. Fabric Manager trains the NVLink interfaces for the GPUs in the partition, bringing those links into an operational state. It programs the NVSwitch routing to enable communication among the partition’s GPUs and configures the switches to deny access to GPUs outside the partition. The result is an isolated segment of the NVSwitch fabric dedicated to that VM. Other VMs cannot send traffic into this segment, and the VM cannot reach GPUs assigned to other partitions. When a VM is configured with a fractional vGPU profile or a single full GPU without a device group, Fabric Manager takes the opposite action: it disables NVLink for that GPU. The GPU operates in isolation, communicating only via PCIe. This ensures that individually assigned GPUs cannot interfere with active partitions and that NVLink bandwidth is reserved for workloads that explicitly request multi-GPU communication geometry. When a VM shuts down, Fabric Manager reverses the process. The NVLink interfaces are untrained, the routing entries are removed, and the GPUs become available for reassignment. The fabric is reshaped to match the current set of running workloads. Multiple VMs can share the same HGX system, each with its own GPU partition or individual GPU assignment, each isolated at the switch fabric level. Partition Boundaries and Fragmentation Not every GPU combination forms a valid partition. Fabric Manager defines supported configurations based on physical topology. For an HGX H100, the valid partition sizes are 8, 4, 2, and 1 GPU. These sizes follow a binary pattern. Binary divisions ensure clean resource allocation. Every combination of partitions sums exactly to 8. A 4-GPU partition plus two 2-GPU partitions consumes the complete GPU capacity. A 6-GPU partition would leave 2 GPUs that cannot pair efficiently with future workloads. Fabric Manager enforces these boundaries, returning only valid configurations through its API. Intelligent Partition Selection Valid partition boundaries prevent impossible configurations. But multiple valid options often exist, and the choice shapes what remains available. Consider an HGX H100 ESXi host where DRS places three VMs in sequence. All GPU paritions are available, thus all device group placements are valid and compatible with any incoming workload. The first VM requires a 2-GPU device group. Assignable Hardware, working alongside Fabric Manager, evaluates the available 2-GPU partitions and selects Module IDs 5 and 7. Please note, that the ESXi host is not capable of deploying an 8-GPU device group and one option to place an 4-GPU device group is removed as well. The second VM also requires a 2-GPU device group. Assignable Hardware selects Module IDs 6 and 8. These choices are deliberate. By placing both 2-GPU VMs in the second half of the GPU capacity, the platform preserves Module IDs 1 through 4 as a contiguous block. The third VM requires a 4-GPU device group. Because the first two placements were coordinated, Module IDs 1 through 4 remain available as a valid 4-GPU partition. The VM powers on successfully. This is the platform adding intelligence on top of Fabric Manager’s validity constraints. Fabric Manager defines what combinations are allowed. Assignable Hardware influences which valid combination is chosen, favoring assignments that preserve larger partition options for future workloads. The Alternative Without this intelligence, the first 2-GPU VM might have received Module IDs 2 and 4. The second 2-GPU VM might have received Module IDs 7 and 5. When the 4-GPU request arrives, no valid partition exists. Module IDs 1, 3, 6, and 8 remain available, but they span both halves and cannot form a contiguous 4-GPU partition. The capacity would exist. The connectivity would not. Platform Considerations Fragmentation avoidance is a scheduling optimization, not a Fabric Manager function. Fabric Manager defines what is valid. The platform decides what is optimal. DRS placement policies influence this further. Consolidation mode packs workloads onto fewer hosts, preserving larger partition options on other hosts. Within a single HGX host, Assignable Hardware steers partition assignments toward allocations that maintain flexibility for subsequent requests. The communication geometry a VM requests is only as useful as the platform’s ability to satisfy it without stranding the remaining capacity. ================================================================================ Title: Understanding Multi-GPU Topologies Within a Single Host URL: https://frankdenneman.ai/2026-03-27-Understanding-Multi-GPU-Topologies-Within-a-Single-Host/ Date: 2026-03-27 Architecting AI Infrastructure Series - Part 10 Part 9 covered why it’s important to understand the topology when using multiple GPUs. When a model runs across several GPUs, communication between them becomes part of the process. Not all GPUs in a server communicate at the same speed, and these differences can impact performance. Many AI teams prefer to run their workloads on a single server. This helps reduce network complexity and simplify deployment. Still, there are several ways to set up multiple GPUs in a single server. This article reviews the different ways GPUs can be connected within a single server and explains how these configurations affect distributed models. How distributed models use multiple GPUs When a model uses multiple GPUs, its weights are distributed across the devices. Each GPU stores part of the model in its own memory. During inference, the GPUs share intermediate tensors and KV cache data as they process layers and tokens. These data exchanges occur through coordinated operations in which all GPUs work simultaneously, sharing activations, KV cache pieces, and intermediate results. Communication happens in many small steps, followed by syncing up. Each step has to wait for the slowest connection. This is why the way GPUs are connected affects performance while running models. Understanding GPU interconnect bandwidth Multi-GPU systems use different types of connections between GPUs. The most common are PCIe, NVLink, and NVSwitch. Each one has its own speed and way of handling communication. All these connections can send and receive data at the same time. But vendors often report bandwidth numbers differently, which can be confusing. PCIe Gen5 x16 offers about 64 GB per second in each direction, for a total of 128 GB per second. NVLink and NVSwitch usually list bandwidth per GPU for both directions. For example, the H200 gives 900 GB/s bidirectional bandwidth per GPU. Even though PCIe can send and receive data simultaneously, actual communication speeds are often lower. PCIe traffic goes through shared components and the CPU, which adds delays. This is even more noticeable during collective operations, since everything waits for the slowest link. In practice, PCIe does not scale as well as NVLink or NVSwitch. It’s also important to know the difference between point-to-point bandwidth and aggregate bandwidth. Point-to-point is the speed between two GPUs, while aggregate is the total speed when several GPUs talk at once. With NVLink bridges, the total bandwidth across all GPUs can exceed the bandwidth between just two GPUs. For example, a four-GPU bridge might claim 1.8 TB per second, but each GPU is still limited to 900 GB per second. NVSwitch works differently. Each GPU connects to the switch at its full NVLink speed. The switch does not block traffic, so any GPU can talk to any other GPU without sharing connections. This means each GPU gets 900 GB/s to the network, and communication happens at full speed no matter where the GPUs are. Configuration Form Factor / Topology Interconnect Point to Point BW RTX PRO 6000 PCIe Gen5 PCIe 128 GB/s bidirectional H100 NVL PCIe, 2 card bridge NVLink 600 GB/s bidirectional H200 NVL PCIe, 2 way bridge NVLink 900 GB/s bidirectional H200 NVL PCIe, 4 way bridge NVLink 1.8 TB/s aggregate, 900 GB/s per GPU HGX H100 SXM + NVSwitch NVSwitch 900 GB/s per GPU to fabric HGX H200 SXM + NVSwitch NVSwitch 900 GB/s per GPU to fabric HGX B200 SXM + NVSwitch NVSwitch 1.8 TB/s per GPU to fabric The multi-GPU spectrum There are many types of multi-GPU systems. Performance depends not only on how many GPUs you have, but also on how they are connected inside the server. The diagram below shows some common ways to connect multiple GPUs in a single server. These setups range from PCIe connections to small NVLink groups to full-mesh NVSwitch networks. As you move from left to right in the diagram, the GPUs’ ability to communicate improves. Some setups create small groups with fast connections, while others let all GPUs communicate at the same speed. The next sections go through each setup and explain how they affect the performance of distributed models. PCIe connected GPUs, 4 × RTX PRO 6000 The first setup uses four RTX PRO 6000 GPUs connected by PCIe. There are no direct links between GPUs. Each GPU communicates through the host PCIe fabric. In this setup, the system acts as four independent accelerators that can work together when needed. This makes it flexible, but it also creates topology boundaries. When a model spans multiple GPUs, communication must traverse the PCIe fabric, which becomes the shared point for collective operations. One key thing about this setup is that communication is not only slower than NVLink-based systems, but also less predictable. Collective operations use all GPUs simultaneously, and communication paths may traverse shared PCIe switches, CPU root complexes, or NUMA boundaries. These factors introduce variability, which becomes apparent when models are tightly connected. This is why PCIe-connected GPUs often work well for loosely connected workloads. Running several independent inference services, serving different models, or giving each workload its own GPU fits well with this setup. But when GPUs need to work closely together, communication overhead becomes more noticeable. Another thing to note is that having more GPUs does not always mean better scaling. Going from one GPU to four gives you more memory and compute power, but distributed models might not scale evenly because of communication overhead. This makes PCIe-connected multi-GPU systems a flexible place to start. They let you scale within one host, but they are just the first step in the multi-GPU spectrum, where communication starts to affect how models run. Two NVLink pairs, two NVLink domains The next setup adds direct GPU-to-GPU connections using NVLink. Many systems have GPUs in pairs connected by an NVLink bridge. In a two-GPU setup, both GPUs form a single NVLink domain and communicate directly. Some of these systems can also be configured with four GPUs. With GPUs like the H100, this usually means two separate NVLink pairs. Each pair is connected by NVLink, but communication between pairs goes through PCIe. This introduces the concept of an NVLink domain. An NVLink domain is a group of GPUs directly connected by NVLink, so they can communicate without leaving the group. Within an NVLink domain, GPUs can map and access each other’s memory and use a shared memory model via the appropriate APIs. The hardware still has separate HBM memory pools, not one big VRAM space. Inside the domain, communication is fast and steady. Outside the domain, it has to go through PCIe. In dual-socket servers, which most of these systems are, each NVLink domain maps to one CPU socket. PCIe devices like GPUs, NICs, and NVMe drives connect to a specific CPU package, not to “the server” in general. This is NUMA locality for PCIe. When a GPU in one NVLink domain needs data from a GPU in the other domain, the transfer follows this path: out of the source GPU over PCIe, through a PCIe switch, up to the local CPU’s root complex, across the Ultra Path Interconnect (UPI) links to the remote CPU, down through its PCIe switch, and finally into the destination GPU. That’s two PCIe hops plus a UPI crossing. UPI bandwidth is shared with everything else crossing the socket boundary: memory coherency traffic, remote memory access, and any I/O from devices attached to the other socket. If your fast NVMe storage is attached to one socket and the GPU using it sits on the other, that transfer also competes for UPI bandwidth. A single cross-domain transfer might not saturate anything, but under load, the UPI links become a shared bottleneck that affects system memory access, CPU cache lookups, GPUs, networking, and storage together. So not all GPUs in the same host behave the same anymore. The system becomes two fast GPU islands connected by a slower, shared bridge. Another important point is that this topology is not always obvious to developers. A system with four GPUs might look uniform, but communication speed depends on which GPUs are used. Distributed models that cross domains can perform differently based on placement. This is where topology-aware scheduling matters. Tools like device groups help show these boundaries and let you place workloads within the same NVLink domain. We’ll cover this more in the next article. This is also the first configuration where adding more GPUs does not automatically improve performance. Going from two to four GPUs in a single VM introduces a topology boundary that can reduce scaling efficiency. Four GPUs in a single NVLink domain Setups like H200 NVL let you connect four GPUs in a single NVLink domain. Unlike two NVLink pairs, all GPUs can communicate evenly within the host. This is the first configuration in which four GPUs form a single compute domain, removing internal boundaries and letting distributed models scale more predictably. This evenness only applies to the four-GPU domain, and topology issues come back when you go beyond four GPUs. With a single 4-way NVLink domain, the OEM has two choices. They can place all four GPUs on one socket, keeping everything within a single NUMA node. This gives you clean host I/O, memory, NVMe, and NICs local to that socket all reach the GPUs without crossing UPI. But it also leaves the other socket’s PCIe lanes unused for GPU traffic, and concentrates all the thermal load on one side of the system. Alternatively, they can split the GPUs 2+2 across sockets. This balances PCIe bandwidth and cooling, but now half your GPUs are always remote to any given host resource. GPU-to-GPU is still fast over NVLink, but host memory access, storage, and network I/O become asymmetric. Neither choice is wrong, it depends on whether the workload is more GPU-bound or more I/O-bound. Check the server’s documentation or run nvidia-smi topo -m to see which layout you actually have. Eight GPUs as two four-GPU NVLink domains Some servers can support up to 8 GPUs per host. With H200 available in PCIe form from OEMs, these systems can be configured with 8 H200 GPUs. This setup isn’t common, but it does show up in some OEM server options and is worth mentioning. One reason this setup might be appealing is licensing. PCIe-based GPUs often include NVIDIA AI Enterprise licensing, whereas HGX systems usually do not. This can make an eight-GPU PCIe configuration appear attractive compared to an HGX H200 system. However, the topology is quite different. In an eight-GPU PCIe setup, the GPUs are usually split into two separate four-GPU NVLink domains. Each domain has fast, even communication within it, but communication between domains hits the same bottlenecks we discussed earlier: out through PCIe, across the UPI links between CPU sockets, and back down through PCIe on the other side. The NVLink domains are larger here, four GPUs instead of two, but the boundary between them works the same way. Distributed models that use all eight GPUs must cross that boundary, competing for UPI bandwidth alongside memory coherency, storage, and network traffic. This setup works well if you deliberately split workloads across the two NVLink domains. But if you try to run one distributed model across all eight GPUs, you’re back to the same topology constraint, just with bigger islands. Four H100 SXM GPUs in a single NVLink domain This setup uses four H100 GPUs connected through the SXM socket with direct NVLink and no NVSwitch. All four GPUs form a single NVLink domain, so communication is even across the system. With the H100 generation, this setup had a clear advantage. PCIe-based systems usually have two NVLink pairs, which means separate NVLink domains in one host. The HGX four-GPU setup avoided this by connecting all GPUs in a single domain, making placement easier and scaling better. With H200 NVL, this difference is less noticeable. Four-GPU H200 NVL setups also form a single NVLink domain, giving similar topology and simplicity. This narrows the gap between PCIe-based systems and smaller HGX setups. This setup is still important because it shows the hardware options and another key point. The SXM socket does not always mean there is an NVSwitch. Even in HGX platforms, you can have single NVLink domains without a switching fabric. For infrastructure teams, HGX gives the same GPU fabric with familiar management tools. HGX with NVSwitch The HGX eight-GPU setup adds NVSwitch, creating a GPU fabric within a single host. This is the first setup where GPU communication is managed by a non-blocking switch, not just wiring. Each GPU connects to the NVSwitch at full NVLink speed, and any GPU can communicate with any other GPU without contention. Rather than eight GPUs with fast interconnects, an HGX system behaves more like a symmetric multiprocessor with a unified memory domain. With H200, this means 1.1 TB of aggregate GPU memory accessible across the fabric. This distinction matters because NVSwitch enables memory semantics rather than message passing. A GPU can read from or write to a peer GPU’s memory directly, without CPU involvement and without packet-processing overhead. This makes HGX systems particularly effective for architectures that require frequent, fine-grained data movement, such as Mixture-of-Experts models, where routing decisions occur per token and latency compounds quickly (covered in the AI Memory series). NVSwitch introduces a new software tool: NVIDIA Fabric Manager. It’s part of the driver stack and, at boot, finds the NVSwitch topology, configures routing tables, and exposes GPU partitions to the operating system. While NVSwitch removes physical boundaries, Fabric Manager defines logical partitions based on the platform. By default, it usually creates two four-GPU partitions in an eight-GPU system. These align with the platform design and determine how you can group GPUs for workloads. The HGX platform shares its architectural design as NVIDIA DGX systems. The motherboard layout, GPU placement, and NVSwitch fabric are the same. The main difference is in how they are used. DGX systems come as integrated appliances with NVIDIA’s management tools. HGX platforms are sold by OEM vendors, so they can fit into existing data center setups. This includes out-of-band management with standard BMC interfaces and lifecycle management with vendor tools. For more information about running VMs on HGX, I recommend reviewing the VMware Private AI Foundation with NVIDIA HGX Servers: Reference Design for Inference Please note, that not every workload needs all eight GPUs. Smaller models or multiple jobs can be spread across GPU groups that match the platform’s partitions. Device groups work with Fabric Manager to create topology-aware GPU configurations, enabling admins to efficiently use an HGX system with multiple topology-aligned VMs. The next article covers device groups and HGX topology. ================================================================================ Title: Understanding Unified Memory on DGX Spark Running NemoClaw and Nemotron URL: https://frankdenneman.ai/posts/2026-03-23-understanding-unified-memory-dgx-spark-nemoclaw-nemotron/ Date: 2026-03-23 NemoClaw became the talk of GTC 2026 within hours of its announcement. It wraps OpenClaw in NVIDIA’s OpenShell runtime, adds guardrails, and gives you an always on AI agent with a single install. Jensen Huang called OpenClaw the operating system for personal AI. NemoClaw is what makes that usable. This is part 4 of the AI Memory series and focuses on how memory behaves on real systems. I installed NemoClaw on a DGX Spark and ran Nemotron models locally to understand what actually happens in memory. The most important takeaway is simple. Unified memory breaks the usual GPU mental model. On a traditional system, the GPU has its own memory, tools like nvidia smi show usage, and free memory roughly maps to what you can still use. On DGX Spark, CPU and GPU share one memory pool. The signals you are used to no longer tell the full story. The models used by NemoClaw are Mixture of Experts models. Dense models activate all parameters for every token, so memory and compute scale together. MoE models behave differently. All parameters must be present in memory, but only a subset is used per token. That creates two separate budgets. Total parameters define the memory footprint. Active parameters define the compute cost. The earlier posts in this series explain this in detail: Part 1 - The Dynamic World of LLM Runtime Memory Part 2 - Understanding Activation Memory in Mixture of Experts Models Part 3 - Durable Agentic AI Sessions in GPU Memory On a unified memory system, this separation becomes very visible. The model either fits based on total parameters or it does not. Everything that matters operationally depends on what memory remains after that. Installing NemoClaw and selecting a model After following the DGX Spark NemoClaw playbook, the installer detected Ollama and suggested running locally Inference options: 1) NVIDIA Endpoint API (build.nvidia.com) 2) Local Ollama (localhost:11434) — running (suggested) The default model is Nemotron 3 Nano. The larger Nemotron 3 Super is available but not selected by default. That choice already hints at what matters on this system. Not just whether a model fits, but how much room is left after it does. Loading Nano frankdenneman@spark:~$ ollama ps NAME ID SIZE PROCESSOR CONTEXT UNTIL nemotron-3-nano:30b b725f1117407 27 GB 100% GPU 262144 4 minutes from now Nano downloads as 24 GB and becomes 27 GB in memory. The difference comes from decompression and preallocated context buffers. Ollama reserves space for the full context window up front. frankdenneman@spark:~$ free -h total used free shared buff/cache available Mem: 121Gi 32Gi 66Gi 56Mi 23Gi 89Gi Swap: 15Gi 120Ki 15Gi About 31 GB is in use and 89 GiB remains available. All model parameters are resident, which defines the memory footprint, and the remaining space is what you can use for everything else. Loading Super frankdenneman@spark:~$ ollama ps NAME SIZE PROCESSOR CONTEXT nemotron-3-super:120b 94 GB 100% GPU 262144 Super downloads as 86 GB and becomes 94 GB in memory. frankdenneman@spark:~$ free -h total used free shared buff/cache available Mem: 121Gi 94Gi 861Mi 56Mi 27Gi 27Gi Swap: 15Gi 120Ki 15Gi At first glance this looks like the system is out of memory, but it is not. The available column shows 27 GiB of usable headroom. All parameters are loaded and ready, and what remains is the space available for runtime behavior. Reading memory on DGX Spark On DGX Spark there is no separate VRAM. CPU and GPU share one memory pool, which changes how you read the system. The nvidia smi memory gauge is not useful here, and the real signal comes from Linux. frankdenneman@spark:~$ nvidia-smi Mon Mar 23 15:56:03 2026 +-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 580.142 Driver Version: 580.142 CUDA Version: 13.0 | +-----------------------------------------+------------------------+----------------------+ | GPU Name Persistence-M | Bus-Id Disp.A | Volatile Uncorr. ECC | | Fan Temp Perf Pwr:Usage/Cap | Memory-Usage | GPU-Util Compute M. | | | | MIG M. | |=========================================+========================+======================| | 0 NVIDIA GB10 On | 0000000F:01:00.0 Off | N/A | | N/A 37C P0 10W / N/A | Not Supported | 0% Default | | | | N/A | +-----------------------------------------+------------------------+----------------------+ +-----------------------------------------------------------------------------------------+ | Processes: | | GPU GI CI PID Type Process name GPU Memory | | ID ID Usage | |=========================================================================================| | 0 N/A N/A 2523 G /usr/lib/xorg/Xorg 43MiB | | 0 N/A N/A 2950 G /usr/bin/gnome-shell 16MiB | | 0 N/A N/A 881931 C /usr/local/bin/ollama 89709MiB | +-----------------------------------------------------------------------------------------+ The usual memory bar is not available. The process view still shows allocations, but it does not reflect total system headroom. For that, you need to look at the OS. frankdenneman@spark:~$ free -h total used free shared buff/cache available Mem: 121Gi 94Gi 861Mi 56Mi 27Gi 27Gi Swap: 15Gi 120Ki 15Gi Free memory looks low because Linux uses spare memory as page cache, holding recently read data such as the model file that was just loaded. That memory is not locked and is reclaimed on demand. $ sudo sh -c 'echo 3 > /proc/sys/vm/drop_caches' After dropping cache, free memory jumps to match available. Nothing changed for the model because the weights were already in CUDA managed memory. The key mental shift is that available memory is your real headroom, while free memory is simply what is unused at that moment. frankdenneman@spark:~$ free -h total used free shared buff/cache available Mem: 121Gi 93Gi 28Gi 56Mi 1.0Gi 28Gi Swap: 15Gi 120Ki 15Gi What headroom really means Headroom determines what you can actually run. The context window you see in ollama ps is allocated per parallel request. Each additional slot requires its own memory reservation. For Super at 262K context, that is roughly 7 GB per slot. With about 27 GiB of headroom, running a single agent is straightforward. Running multiple agents is possible, but each additional slot reduces the margin for activation spikes and OS overhead. Nano leaves about 89 GiB of headroom, which allows multiple agents, larger context windows, and more flexibility. This is why the installer defaults to Nano. Not because Super cannot run, but because Nano leaves room for actual usage. Observing behavior under load I ran a sustained agent workload for 27 minutes and monitored memory. Available memory stayed stable at around 27 GiB. Free memory fluctuated as the kernel reclaimed and reused page cache. This is expected behavior. The kernel does not keep large amounts of memory unused. It reclaims what it needs when it needs it. The system never touched swap. Swap on unified memory Swap is disk space, not part of the memory pool. It does not increase headroom. On unified memory systems, swap introduces a different failure mode. If the kernel pages out model data and that data is needed again, inference stalls. With MoE models, where routing is dynamic, this can happen unpredictably. If swap usage increases, performance is already degraded. What to take away Unified memory changes how you read GPU systems. The model fitting into memory is only the starting point. What matters is the headroom that remains. Use free h and focus on available memory. Do not rely on free memory alone. Do not rely on nvidia smi for capacity planning. Treat swap usage as a signal that you are beyond safe operating conditions. Most importantly, think in terms of headroom. That is what determines how many agents you can run, how much context you can support, and how stable the system will be under load. ================================================================================ Title: Why Multi GPU Requires Topology Awareness URL: https://frankdenneman.ai/2026-03-16-why-multi-gpu-requires-topology-awareness/ Date: 2026-03-16 Architecting AI Infrastructure Series - Part 9 The AI Memory series has been showing how AI workloads use GPU memory in different ways. The Dynamic World of LLM Runtime Memory explains how the KV cache grows with each new token and becomes a main user of GPU resources. Understanding Activation Memory in Mixture of Experts Models looks at the hardware pressure that happens when activation memory spikes during the prefill phase. The series also covers how agentic systems keep memory active to stay on track during complex tasks, as discussed in Durable Agentic AI Sessions in GPU Memory. All these behaviors change where system pressure shows up. Inference is a two-stage dance of “Prefill” and “Decode,” and it taxes different parts of the system. In the prefill phase, large prompts need a lot of compute to process the full context. During the decode phase, dynamic (runtime) memory keeps growing. Long sessions make both demands higher. As prompts get bigger, prefill work gets more expensive, and the model’s working memory grows as more tokens are added. At this stage, many deployments switch from using a single GPU to multiple GPUs. The main reason is straightforward: more GPUs mean more High Bandwidth Memory (HBM), giving the model access to a larger memory pool. But once a model runs across multiple GPUs, the system starts behaving differently. Multi-GPU changes the execution model GPUs are usually added during training to increase compute power. For inference, they are often added to provide more memory. However, when a model runs on multiple GPUs, the way it executes changes. When a model spans GPUs, each device holds part of the model state. The Static Memory, which is the weights or the “model’s brain”, is split across the GPUs. The Dynamic Memory, made up of intermediate tensors (the active math) and the KV cache (the short-term memory), is present only when those specific weights are triggered for computation. Because the model’s knowledge is divided, generating a single token needs a coordinated process: the dynamic math from one GPU must be sent to the next before the model can continue. In modern Mixture-of-Experts (MoE) models, this setup is even more specialized. Tensor Parallelism splits weight matrices into smaller pieces across several GPUs, enabling them to work together on a single large calculation, with each GPU handling part of the job. Expert Parallelism then assigns different specialist weights to different GPUs. For every token, the system acts like a high-speed router, sending the active math to the GPU with the right expert. This creates a sparse activation pattern, where only part of the model is used at a time, but it puts a lot of pressure on the connections between GPUs to move data quickly. Modern models keep this distributed math organized using Attention, which acts like the model’s train of thought during inference. When the model generates a new word, it doesn’t just look at the previous word. Instead, it scans the Dynamic Memory (the KV cache) to find the right context. For example, if the model is in the middle of a long paragraph and generates the word “it,” Attention helps it look back through earlier sentences to find what “it” refers to. In a multi-GPU setup, the model must gather pieces of this session state from across all GPUs to keep each new word connected to its original meaning. Key Takeaway: Data has to move between GPUs to complete even a single layer of computation. If this happens quickly, the GPUs work together smoothly. If it’s slow, the GPUs end up waiting for data before they can keep computing. The NUMA analogy For administrators familiar with CPU architectures, the behavior should feel familiar. NUMA systems divide memory across nodes. A processor accessing memory attached to its local node can do so quickly. Accessing memory attached to a remote node takes longer because the request must traverse an interconnect. A GPU accessing its local HBM is nearly instantaneous, but because the model’s state is split, the system is constantly performing remote memory accesses to retrieve context from other devices. If the interconnect is fast, the cluster behaves like a single, massive accelerator. If it is slow, the GPUs spend more time waiting for data than performing math. Even with enough compute capacity available, performance is capped by how fast memory state moves between these nodes. When viewed this way, it is clear that topology is the deciding factor; the real question is whether the platform understands this need for locality. The problem with topology-unaware scheduling In the past, many infrastructure platforms treated all GPUs as equal. If a workload needed four GPUs, it got any four that were available. This approach works for single-GPU jobs, but for distributed models, it can lead to unpredictable performance. If a scheduler doesn’t consider how GPUs are connected, it might assign GPUs that have to communicate over slower links. The model still starts, and the system shows that the workload got the resources it requested. From a capacity standpoint, everything looks fine. The problem appears only when the model is running. The GPUs keep exchanging tensors and KV cache state over slow links, increasing latency and slowing token generation. The real issue isn’t the model or the GPU hardware; it’s that the platform assigns the workload to GPUs that can’t communicate efficiently with each other. Why platforms must become topology-aware When inference workloads use more than one GPU, the scheduler needs to know more than just how many GPUs are available. It also needs to understand how they are connected and group together devices that can communicate quickly. These groupings also need to remain stable over time. If workloads restart or move, the platform must ensure the model continues to run on GPUs that can exchange data efficiently. This makes GPU allocation a problem that requires topology-aware scheduling. The platform should treat groups of GPUs with fast connections as single resource domains. Why agentic workloads amplify the problem Agentic AI systems keep generating tokens as they work through tasks. They gather information, use tools, check results, and keep conversations going over time. The model’s runtime memory stays active throughout this process. Because sessions last longer, deployments often use multiple GPUs to provide enough working memory for the model. The interactive nature of these systems also makes any latency more noticeable. Each reasoning step depends on the tokens from the previous step, so if GPU communication slows down, the entire reasoning process slows down too. Setting the stage for topology-aware infrastructure When a model runs across several GPUs, those devices need to exchange memory state as they generate tokens. How quickly this happens depends on how the GPUs are connected and whether the platform groups them correctly. Multi-GPU inference is similar to a NUMA-sensitive workload. Performance depends on how close the GPUs are and how fast the connections between memory areas are. Modern GPU systems offer different ways to connect devices, each with its own speed. Some setups let GPUs share data almost as if it were a single memory space, while others add extra steps and more delay. Understanding these interconnects is the next step in building multi-GPU systems. In the next article, we’ll look at how modern GPU fabrics like NVLink and NVSwitch affect communication between GPUs and why they matter for designing AI platforms. ================================================================================ Title: Durable Agentic AI Sessions in GPU Memory URL: https://frankdenneman.ai/2026-03-12-durable-agentic-ai-sessions-in-gpu-memory/ Date: 2026-03-12 The durable memory of agentic systems When a user asks a question in a chat interface and the model responds, the interaction is a single prompt completion. A prompt goes in, tokens come out. From an infrastructure perspective this is a predictable transaction. As described in The Dynamic World of LLM Runtime Memory, the KV cache grows with the prompt, peaks during generation, and is released when the session ends. The memory footprint is bounded and relatively easy to plan for. Agentic AI behaves differently. Instead of answering a single prompt, an agent executes a sequence of steps. It reasons about a goal, calls a tool, evaluates the result, and continues until the task is complete. Each step is technically another prompt completion, but the agent does not start fresh each time. It carries forward the entire history of the task, including every prior LLM completion, tool result, and reasoning step, all of which are sent back to the model on every next call. From the perspective of GPU infrastructure, this changes the memory lifecycle of the workload. Architecturally the system still appears stateless. Each request is processed independently and the LLM returns a response. What persists across steps is the accumulated textual context. Agents do not store session memory in GPU memory directly. They store it in the text history that is repeatedly sent back to the model. That history lives in the agent framework’s process memory, container memory in most deployments. It is not persistent. A container restart means the session history is gone and the agent must start over, spending tokens and utilizing GPU memory all over again. For long-running agentic sessions this is not just inconvenient, it is costly. How to protect session state across failures, and where to durably store it without rebuilding from scratch, is a topic for a future article. This difference matters for infrastructure operators. Prompt completion workloads have short and predictable memory lifecycles. Agentic workloads accumulate memory across the lifetime of a task. A GPU that comfortably runs many concurrent chat sessions may struggle with only a few active agents. The model has not changed. The memory lifecycle has. State, memory, and the cost of continuity Every step of an agent session is processed during the prefill stage, where the full session transcript is converted into a KV cache before token generation begins. As the session grows, prefill becomes increasingly expensive, both in compute and in the GPU memory required to hold the resulting cache. This is often the first bottleneck when scaling agentic inference, because prefill cost scales with context length, not with the number of tokens being generated. Some inference systems reduce this cost through KV persistence layers such as LMCache or prefix caching in vLLM, which reuse previously computed KV segments rather than rebuilding from scratch on every step. Without such mechanisms, every step pays the full prefill cost of the entire session history. That these tools exist at all signals that the industry recognizes GPU-bound KV cache as a structural bottleneck worth solving at the infrastructure layer. Prefill and decode also have fundamentally different compute and memory characteristics, which is why disaggregated serving, where the two stages run on separate infrastructure and scale independently, is an increasingly common pattern in production deployments. That topic is covered in a future article in this series. The KV cache of an active agent session is durable. It is neither short-lived like a single prompt completion nor permanently stored like database state. It exists for the lifetime of the task and grows as the session context expands. This introduces a useful distinction when thinking about runtime state. Model behavior Description Example Stateless Each request is independent Standard API call or prompt completion Durable Memory persists for the lifetime of a session KV cache of an agent session Stateful Memory persists indefinitely across sessions Database-backed application state For vSphere administrators, the closest parallel is a VM’s memory swap file, present for the life of the VM, growing under pressure, reclaimed only when the VM powers off. The session is the VM. The KV cache is the working memory it consumes while running. For Kubernetes operators, think of an emptyDir volume, scoped to the pod lifecycle, gone when the pod terminates. KV cache memory and session concurrency Agentic sessions do not grow at a fixed rate. Three distinct mechanisms drive context accumulation, and each compounds the others. Tool outputs inject tokens directly into the session history with no summarization or filtering. When an agent calls an external system, a ServiceNow query, a RAG retrieval, a code execution result, the full response is appended verbatim before the next step begins. A query that returns 500 tokens one day may return 8,000 the next. No configuration parameter at the inference layer caps this. For a deeper look at how agents, tools, and MCP servers fit together, Sam McGeown’s primer Understanding Instructions, Context, Skills and MCP Servers is an excellent starting point. Reasoning-capable models add a second source of growth that is invisible to standard monitoring. Before generating a response, these models produce an internal chain-of-thought trace, real tokens with real KV cache cost, that never appear in logs or user-facing output. A complex reasoning step can add several thousand tokens per iteration. Sizing based on observable token counts alone will systematically underestimate actual GPU memory consumption. The accumulated history of all prior steps is the third. Every completion, tool result, and reasoning trace is carried forward in full. Together these mechanisms push active sessions toward contexts that are larger, less predictable, and longer-lived than anything a prompt-completion sizing model accounts for. Quantifying the problem To make the memory impact concrete, consider a realistic enterprise workflow: an AI agent that analyzes GPU cluster utilization, evaluates upcoming project demand, checks budget constraints, and raises a ServiceNow change request for capacity expansion. Ten steps, three tool integrations, three reasoning phases. Each step adds tokens to the session history and the KV cache grows with it. For GPT-oss 120B the cost is approximately 0.035 MB per token, derived from its hybrid attention architecture of 18 full-attention and 18 sliding-window layers. The full architectural breakdown is covered in Part 2 of this series. The chart shows three distinct growth patterns. Tool calls produce the steepest jumps, their size determined entirely by what the external system returns. Reasoning steps add substantial invisible overhead that never surfaces in logs. The session closes at 854 MB, but the more important infrastructure detail is duration. That 854 MB is held in GPU memory for the full lifetime of the task. Other agents running concurrently may carry heavier contexts or run longer. Each session is a different size, with a different lifetime, and none release memory until their task completes. Fitting an unpredictable mix of sessions into a fixed pool of GPU memory is less a capacity calculation and more a bin packing problem, and unlike compute scheduling, the bins do not empty on a predictable schedule. Designing for durable sessions The bin packing problem has a hard constraint we have not named yet. The bins are not empty, roughly 60 GB of every H100 80GB is already occupied before a single agent session begins. That is the model weights, static, always resident, loaded once and never released, as described in Part 1 and Part 2 of this series. What remains is the headroom that all active sessions must share. Agentic workloads make that headroom increasingly contested, sessions are larger, they run longer, and the dynamic memory consumption compounds with every concurrent agent. There are two architectural responses, and they solve different problems. Multi-GPU deployment splits a single model instance across multiple GPUs using tensor or pipeline parallelism. This addresses size, models too large to fit on a single GPU can be distributed across several. The memory pool grows, but it still serves one model instance. The bin packing problem remains, just with a larger bin. Multi-replica deployment runs multiple complete model copies, each on its own GPU or set of GPUs. This addresses concurrency, each replica maintains its own independent session pool, and a fleet of replicas is capable of handling thousands of simultaneous agent sessions. Here the number of bins grows, not their size. In practice, large-scale agentic deployments require both. A model that requires multiple GPUs to load is deployed with tensor parallelism to handle size, then replicated across multiple such instances to handle concurrency. A long-standing concern with multi-replica deployments was naive load balancing, routing each request to the next available replica regardless of session history, landing on a cold KV cache every time. For stateful chat that was a real liability. For agentic AI it largely disappears. Because the full session transcript is sent with every request, any replica can reconstruct the KV cache from scratch. The property that makes agentic sessions expensive, carrying the full history, is also what makes them portable. Session affinity matters less when the session carries itself. Designing for durable, long-running sessions across multiple GPUs and replicas is the infrastructure challenge that follows from everything covered in this article. That is where the next article in the Architecting AI Infrastrucre begins ================================================================================ Title: MIG Partitioning, Placement Geometry, and Stranded Capacity URL: https://frankdenneman.ai/2026-03-06-mig-partitioning-placement-geometry-and-stranded-capacity/ Date: 2026-03-06 Architecting AI Infrastructure — Part 8 Previous articles in this series explained how time-sliced GPU sharing works in both same-size and mixed-size environments. They showed that choices like profiles and the order in which workloads start can directly affect GPU utilization and whether workloads are placed successfully. In this part, we look at MIG and the design choices that affect placement success and overall resource utilization. MIG takes a different approach to GPU sharing. Instead of multiplexing compute resources between workloads, MIG splits the GPU into hardware instances. Each instance gets its own dedicated compute and memory slices slices. Each instance offers three main features: fault isolation, individual scheduling, and a distinct address space. When strict hardware isolation is required, MIG is the right solution because workloads cannot interfere with one another, and resource consumption becomes predictable. Many admins and operators choose MIG as the technology to provide fractional GPUs without a strict requirement for hard isolation. This article focuses on that use case and identifies the challenges to successful placement and resource utilization, including how profile selection directly determines whether GPU capacity is fully consumed or permanently stranded." MIG Resource Model Earlier articles in this series showed that GPU capacity is not determined solely by free memory. Capacity depends on how resources are divided and placed. MIG adds another layer of placement constraints. All NVIDIA GPU architectures that support MIG, including Ampere, Hopper, and Blackwell, have the same structure. Each GPU provides seven compute slices and eight memory slices. Profiles use both resources simultaneously, so each profile represents a specific combination of compute and memory slices that match the GPU’s physical layout. This article uses an H100 eighty-gigabyte GPU as an example. In this setup, each memory slice represents ten gigabytes of framebuffer memory. Because compute slices and memory slices are allocated together, free memory alone does not determine whether a new instance can start. The required compute slices must also be available and match the correct memory region. The table lists the available MIG profiles for the H100-80GB GPU: H100 80GB MIG Profiles Profile Compute slices Memory slices Memory 1g.10gb 1 1 10 GB 1g.20gb 1 2 20 GB 2g.20gb 2 2 20 GB 3g.40gb 3 4 40 GB 4g.40gb 4 4 40 GB 7g.80gb 7 8 80 GB These profiles show that MIG resource use is asymmetrical in most cases. Some profiles offer the same memory size but differ in compute capacity. For example, both 1g.20 GB and 2g.20gb provide 20 GB of memory but need different numbers of compute slices. The same goes for the 40 GB profiles: 3g.40gb and 4g.40gb both use 40 GB of memory, but need different compute resources. This mismatch between compute and memory can lead to placement results that aren’t obvious at first. Stranded Capacity Because compute and memory slices don’t always match up, some GPU resources can go unused even when the device looks fully used. Take the smallest MIG profile, 1g.10gb. This profile consumes one compute slice and one memory slice. On an eighty-gigabyte GPU, seven instances can be created because the GPU exposes seven compute slices. The GPU still has eight memory slices. After placing seven instances, 10 gigabytes of memory remain unused, or to put it another way, stranded capacity. No compute slices remain, so no other instance can start. This behavior is easy to miss in MIG placement diagrams. These diagrams show memory placement regions, and seven 1g.10gb instances appear to fill the GPU completely. In reality, the limiting factor is compute slices, not memory. Placement Geometry MIG profiles must align with specific memory placement regions inside the GPU. Profiles that consume multiple memory slices require a contiguous region. The 3g.40gb profile consumes four memory slices. On an 80-gigabyte GPU, this creates two valid placement regions: memory slices 0–3 or 4–7. nvidia-smi is NVIDIA’s command-line tool installed with the driver. The mig -lgi flag lists all active MIG instances on the host — list GPU instances — including the profile each instance was created from and where it sits in the GPU’s memory layout. The output includes a placement column formatted as start:size, where start is the index of the first memory slice the instance occupies, and size is the number of slices it consumes. A 3g.40gb instance at 4:4 starts at memory slice 4 and occupies four slices, placing it in the second region. A 4g.40gb instance at 0:4 occupies the first region, the only region where its compute requirement can be met. However, as two 3g.40gb profiles are placed on the GPU, one compute instance is stranded. The important thing to note, and what the 40gb profiles show so well, is that MIG introduces two regions, one with four aligned compute and memory slices, and another with three. MIG placement rules require that compute and memory slices start at the same position, but they don’t have to end together. A great example of this is the 4g.40gb profile. It will only be placed on memory slice 0, and thus directly aligns with compute slice 0. I got the luxury of having (temporarily) access to a Dell PowerEdge XE9680 HGX system, with eight H100 80 GB GPUs, seven empty. When I powered on seven VMs with a 4g.40gb profile, each VM was placed in the first placement region (0-4) of an H100 GPU. The last four memory slices of each GPU were still free, but those regions only have three compute slices, so you can’t place another 4g.40gb VM there. However, you can power on VMs with a 3g.40gb vGPU profile. As shown in the screenshot, I started two VMs with that profile, and they were placed on GPU 1 and 2. Keep in mind that existing instances are never rearranged. The way the GPU is set up determines what can start next. This means the order you start workloads matters, since it affects which profiles can still be deployed, even if there seems to be enough memory available. Placement Behavior As described in part 4, vSphere doesn’t use host-level GPU placement policies when GPUs are in MIG mode. Placement follows the same approach used in mixed-size environments: it fills one GPU before moving to the next, while keeping as many placement options open as possible for future workloads. This behavior has improved a lot in the Hopper architecture, but Ampere sometimes has trouble placing larger profiles because it does not always consider future 4g40gb placements. (Reddit). On hosts with more than one GPU, workloads are placed on one GPU until that device can’t fit the requested profile anymore. The next workload is then placed on another GPU. The same idea applies inside the GPU: instances are placed to keep the largest possible contiguous regions, so larger profiles can still be deployed later. A good example is the 3g.40gb profile. In my test cluster, I cleared out seven GPUs (except GPU 0, which was running a developer’s workload) and started five VMs, each with a 3g.40gb vGPU profile. As shown in the screenshot, the first VM was placed on GPU 0, placement id 4, leaving space for a future 4g.40gb profile. When the next VM was placed with a 3g.40gb profile, the vGPU manager selected GPU 1, leaving the other GPUs open for the possible placement of the largest profile, 7g.80gb. With each new placement, the vGPU manager puts the first vGPU profile on placement 4 before filling up the rest. Please note that I registered all these VMs on this host to keep the test scope limited. In real-world scenarios, DRS, together with Assignable Hardware, distributes VMs across compatible ESXi hosts in the cluster based on cluster balance of CPU and memory and the availability of compatible GPUs. Profile Catalog Design The asymmetric consumption of compute slices forces a deliberate choice when defining the profiles exposed through a self-service portal, because the profiles you include determine what users can request and how efficiently the GPU is used over time. The 40-gigabyte profiles show this tradeoff clearly. A GPU can host two 3g.40gb instances, but only one 4g.40gb, because a second would need eight compute slices and the GPU only has seven. If you offer only 3g.40gb, one compute slice is always stranded on a fully loaded GPU. If you offer 4g.40gb along with smaller profiles, you avoid that waste but risk placement failures: the 4g.40gb profile can only be created in the first memory region, so if another instance is already there, placement is impossible no matter how much memory is left. The 20-gigabyte profiles have the same issue in a different way. Four 2g.20gb instances can’t run on a single GPU—again, eight compute slices are needed, but only seven are available. If you include the 1g.20gb profile as an option, you can fit a fourth 20-gigabyte placement, but this makes stranded capacity more likely as the GPU fills with compute-light instances. There is no configuration that eliminates this tension. Platform teams must decide whether to prioritize placement predictability by offering fewer profile options and more predictable behavior, or to offer the full range of profiles and accept that users may sometimes see failed placements or that some GPUs will have stranded capacity. If you don’t need hard isolation, mixed mode described in part 6 and part 7 avoids these constraints completely. Four 20-gigabyte workloads and two 40-gigabyte workloads can each fully use a GPU in mixed-size environments without leaving compute capacity stranded. ##Looking Ahead The next part covvers the new VCF9 functionality called ‘DirectPath Profiles’ to monitor placed GPU workloads and provide visibility into available GPU resources. ================================================================================ Title: Same Size vs Mixed Size Placement at Cluster Scale URL: https://frankdenneman.ai/2026-03-01-same-size-vs-mixed-size-placement/ Date: 2026-03-01 Architecting AI Infrastructure — Part 7 The Silo Capacity Visualizer from Part 6 shows how profile selection and placement-ID alignment affect memory layout inside a single GPU. While that’s helpful for understanding the basics, real capacity planning happens at the cluster level. This article introduces the Same-size vs Mixed-size Placement simulator, the second tool in the Cluster Profile Strategy Toolset. It lets you simulate vGPU placement across an entire cluster using both same-size and mixed-size policies simultaneously, with the same workload sequence for both. This way, you can directly compare their results. Configuring the Cluster Start by choosing a GPU model from the catalog, which uses the same placement-ID data as the Silo Capacity Visualizer. For this example, I picked the H100 PCIe 80GB. The cluster size and total GPU capacity depend on the number of hosts, GPUs per host, and the GPU model. Here, the cluster has 3 hosts, each with 2 GPUs, so 3 x 2 x 80GB = 480GB. I set up a cluster with four hosts, each with four GPUs. That gives a total of sixteen GPUs and 1,280 GB of GPU memory. In the profile catalog, I enabled the 8GB, 10GB, 20GB, and 40GB profiles, just like in the Silo Capacity Visualizer example. When you click on a profile, it changes color and becomes available in the ‘VM Request Sequence’ section. Building the Workload Sequence The workload sequence is a list of VM requests that the simulation processes in order. You can fill this sequence in three ways: manually, randomly, or by using a Monte Carlo simulation. In manual mode, you click on profiles to add them to the queue. The undo button removes the last step, and clear deletes all VMs from the queue. Below the ‘VM Request Sequence’ header, you’ll see how many VMs you’ve selected and the total capacity needed for the queue. The random option fills the queue close to the cluster’s total capacity. You can keep pressing random until you find a sequence you like. The Randomizer suggests a startup sequence of VMs that could fit the total GPU memory. It follows placement-ID rules and can include requests that can’t be placed, just like in real self-service portals where the order is unpredictable. Once you’ve loaded a manual or random queue, start the simulation by clicking the ‘Run’ button in the Run Simulation section. You can choose from four speeds: slow, fast, super fast, and Ludicrous speed. Speed Delay (ms) Slow 1,200 Normal 800 Fast 350 Super fast 175 Ludicrous speed 0 (instant) At any speed except Ludicrous, you can follow the placement sequence and see the difference between same-size and mixed-mode. With the sequence loaded, after you click run, the simulation steps through each VM request and tries to place it under both policies at the same time. “The tool assigns VMs to hosts in round-robin order, a deliberate simplification. DRS weighs CPU and memory utilization when selecting a host, which can change which GPU receives a given request. The tool intentionally removes that variability to isolate what matters here: how the same-size and mixed-size placement policy, combined with your profile catalog, determines capacity consumption. It is a controlled environment for comparing placement behavior, not a production scheduler simulation.” You can also run a Monte Carlo simulation. A single sequence only shows what happens for one arrival order, but Monte Carlo analysis runs many random sequences on the same cluster and catalog. It tracks the worst case for each mode: most rejections for same-size, most siloed capacity for mixed-size. You can load either worst-case sequence into the simulator for step-by-step review. Choose between a ‘Typical bad day’, a ‘Bad but plausible worst-case scenario’, or a ‘rare but realistic situation’. Tier Runs Label Meaning 1 2,000 Typical bad day ~90th percentile — operationally relevant 2 5,000 Bad but plausible ~95th percentile — design target 3 12,500 Rare but realistic ~99th percentile — stress test For this scenario, I picked the ‘Bad but plausible’ simulation. After running 5,000 simulations, the tool gives you two visualization options: the worst-case for Same Size and the worst-case for Mixed Mode. Click ‘visualize this run’ to see how the VMs are distributed. You can compare both policies side by side, since you’ll also see the distribution for the other policy when using the startup sequence that causes the worst-case scenario for each one. Interestingly, I haven’t seen a case where the startup sequence is the same for both policies. GPU Cards Each of the sixteen GPUs has a card that shows a memory bar. Allocated memory is colored by profile size, siloed memory appears in red at its actual spot in the layout, and free memory is shown in gray. In same-size mode, a GPU locks to the first profile placed on it. Any later requests for a different profile size on that GPU are rejected. As the simulation runs, vGPU profile size badges build up, and rejection counts go up for profiles that arrive after most GPUs are already committed. In mixed-size mode, any profile can be placed on any GPU as long as it follows placement-ID rules. The cards show silo warnings when misalignment creates memory areas that no selected profile can use. Clicking a card opens the GPU Memory Map, which gives a cell-by-cell view of allocated regions by profile, striped red silo regions, and free memory. Placement Log The placement log keeps track of every decision. Successful placements show the VM, which GPU it went to, and any silo notes. For example, VM64 with a 10GB profile is placed on GPU D (Host 2, GPU 2) and creates a 2GB silo, as shown in the screenshot. VM6410G→ D ✓+2GB silo Rejections come with a message specific to the mode. In same-size, it says all GPUs are committed to incompatible profiles. In mixed-size, it says no valid placement slot is available because of alignment. Seeing both logs side by side makes it easy to understand how each policy fails. The table shows, for each placement policy, how many VMs were placed, how many were rejected, how much GPU RAM was used, and the percentage of utilization. The last two numbers can help you decide which policy gives better ROI based on your vGPU Profile catalog. In this example with four GPU profiles, Mixed Mode gives a higher utilization percentage than Same Size. Trying to get a higher utilization percentage might feel odd for admins used to managing CPU and memory in clusters. But since GPUs don’t have overconsolidation features like system compute resources, it’s important to use every bit of GPU you have. Every customer we talk to with GPUs, whether on bare-metal K8s, DGX systems, or VCF platforms, says that fixing low GPU utilization is the top priority for their accelerated cluster operations teams. The placement log also highlights some interesting details. In the mixed mode section, it shows the percentage of siloed capacity. Platform Impact Table After the simulation, the Platform Impact Table sums up the results for each profile: planned requests, VMs placed in each mode, capacity used, and unserviceable VMs. There’s a totals row for the whole cluster and a summary line that highlights siloed capacity in mixed-size mode. GPU Memory Map After a simulation run, you can use the same simulator introduced by the ‘Mixed Mode vGPU Profile Placement Silo Simulator’ by going to the ‘View memory map’ option in the Mixed-Size Mode section. The drop-down menu provides access to each GPU’s memory map. You can also just click on one of the GPU tiles. In this scenario, I’m interested in taking a closer look at GPU C of Host 1. Underneath the placement log, the memory map appears: It shows that 6GB of GPU resources are siloed due to misaligned placement IDs for the chosen vGPU profiles. To see whether a different vGPU Profile Catalog improves GPU utilization or enables more placements, you can use the swap analyzer. Swap Analyzer: 8GB to 10GB The Swap Analyzer opens when you swap profiles in the sequence. I replaced every 8GB request with 10GB and applied the swap. The tool keeps the sequence order the same. The analyzer then shows a before-and-after comparison for both modes: VMs served, VMs rejected, capacity used, and siloed capacity. You can quickly see if your utilization improved or if you gained more placement options. The 8GB profile doesn’t align as well with the 20GB and 40GB profiles as the 10GB profile does on the H100. As explained in Part 6, profiles without clean shared boundaries leave gaps that other profiles can’t use. Swapping to 10GB improves that alignment. In mixed-size mode, this swap reduced siloed capacity and may have increased the total number of VMs placed. In same-size mode, the improvement was smaller because lock-in is the main reason for rejections, no matter how the boundaries align. The difference between 8GB and 10GB is just 2GB per VM, but at the cluster level, what really matters is the compounding effect of better placement-ID alignment across all GPUs and placement decisions. This is what right-sizing means: picking profiles whose boundaries keep the cluster usable for future workloads. To check if you improved the outcome, compare the stats at the bottom of the placement log. Since it’s hard to remember all the numbers from the previous run, a new section called the ‘Swap Analyzer’ appears at the bottom of the screen after you run the profile swap simulation. It shows before (8GB) and after (10GB) results. Here, you can see that Same Size mode improved placement because there was enough room on a GPU locked to the updated profile. This is a powerful way to simulate your cluster setup and decide how many different profiles you want to use. If you choose your vGPU profiles carefully, you can use the minimum number needed while still giving enough resources for your workloads. In both modes, this can lead to better placement options. Try it out and let me know your thoughts on LinkedIn or X. The Cluster Profile Strategy Tool is available at Same-size vs Mixed-size Placement. The full series index is at Architecting AI Infrastructure. ================================================================================ Title: Mixed Size vGPU Mode in Practice URL: https://frankdenneman.ai/2026-02-24-mixed-size-vgpu-mode-in-practice/ Date: 2026-02-24 Architecting AI Infrastructure - Part 6 Last time, I looked at how Same Size vGPU mode works with different assignment policies and how right-sizing profiles can make placement more flexible. The main point was that both profile variety and assignment choices have a big impact on how much GPU capacity you can actually use over time. Understanding Placement IDs and Siloed Capacity This article focuses on Mixed Size mode. Unlike locking a GPU to one profile after the first placement, Mixed Size lets you use different profile sizes on the same device. This might seem like an easy fix for fragmentation, but it brings a new challenge: placement IDs. These are fixed memory spots on the GPU where a profile can begin, so even if memory appears free, you can’t always use it unless it aligns with a valid placement spot. For more details on how placement IDs work, see Part 4. Placement Flow and Local Optimization GPU placement is not a single decision but a sequence of decisions that happen at different layers of the platform. First, Assignable Hardware creates a list of hosts that meet the VM’s GPU needs and labels. Then, DRS evaluates those hosts and selects the best one based on CPU and memory availability, cluster balance, and how well the VM will run there. Only after a host is chosen does the GPU manager pick which physical GPU will handle the workload. To keep things simple, the examples here assume all hosts are used equally. In this setup, DRS appears to be distributing workloads in a round-robin manner. Real environments are usually more complex, but this simplification helps us focus on how GPU placement works. This is where it gets interesting. Inside the chosen host, the GPU manager doesn’t just pick any spot. It examines the memory layout, prefers the least-used GPU, checks possible placements, and picks the one that minimizes fragmentation, all while adhering to placement-ID rules for the profile size. This is a smart process. The scheduler isn’t just finding an open slot—it’s trying to keep future options open by making the best placement it can right now. But even with this smart approach, there’s a basic limit. Each placement is the best choice for the current situation, but the scheduler can’t predict what profile sizes will come next or in what order. Over time, making the best choice each time can still result in a memory layout that future workloads can’t fully utilize. The platform focuses on making the best placement right now, not for every possible future placement. Bridging the gap between these local decisions and the overall outcome is where good profile catalog design comes in. Fragmentation Depends on the Profile Fragmentation depends on the profile you want to place. A GPU might look like it has enough free memory, but you still might not be able to use it if the placement IDs don’t line up. Here’s an example. At first, the layout looks fine. There’s a 10GB gap between the two placements, so it seems like an 8GB profile should fit. But placement IDs decide where profiles can start. In Mixed Size mode, 8GB profiles on an 80GB GPU can’t start just anywhere. In this case, placement ID 8 is already inside the 10GB profile. The next valid placement ID is 16, but there isn’t enough space there because the 20GB profile starts at placement ID 20. So, even though the memory gap is large enough, there’s no valid placement ID for the 8GB profile in that spot. So, the vGPU manager places the profile at the next available placement ID, further along in memory. The gap persists, and fragmentation increases, even though the scheduler picked the best possible spot. This shows an important point: fragmentation depends on the profile you’re trying to place. A GPU might look fragmented for one profile size but work fine for another. It’s not just about free memory; placement alignment matters most. Profiles that share the same placement-ID boundaries are more likely to fill gaps left by earlier placements. Over time, using aligned profile catalogs helps reduce fragmentation and maintain flexible placement. Slim-Fit versus Right-Size Before we get into tools, it’s important to look at the main design choice. Slim-fitting profiles try to match workload memory use as closely as possible. Right-sizing considers both the workload’s needs and the platform’s efficiency. Siloed capacity is memory that no workload can use because there’s no compatible placement ID left. Picking profiles that align well with placement boundaries reduces the risk of siloed capacity and gives workloads more room to grow. This is important because AI workloads rarely use the same amount of memory at all times. Things like KV cache growth, changes in batch size, or more users can make memory needs go up and down, so it’s hard to size exactly. That’s why right-sizing is really a platform decision, not just a workload one. Equal Size versus Mixed Size Reality People often say the Mixed Size mode lowers density compared to the Equal Size mode. While that’s technically true, it’s often misunderstood because it doesn’t affect all profiles equally. Only certain profile sizes see a drop in density. For many common profiles, there’s no loss at all when switching from Same Size to Mixed Size mode. The Density Delta Chart below shows the real difference. Profile Equal Mixed Practical Impact 10GB same same none 20GB same same none 40GB same same none 16GB -1 slot reduced moderate 8GB -2 slots reduced situational 4GB larger drop reduced mostly dev The profiles most often used for production AI workloads, such as 10GB, 20GB, and 40GB, don’t lose density in Mixed Size mode. The reductions mostly affect smaller profiles, which are usually used for development, testing, or very limited edge cases. For more details on the maximum number of vGPU profiles per GPU, check the NVIDIA AI Enterprise documentation. So, the real trade-off isn’t just about density versus flexibility. Equal Size mode avoids placement-ID issues by locking GPUs early, while Mixed Size mode skips the locking but makes placement-ID alignment the key to efficiency. Your choice should be based on how you design your profile catalog, not just on theoretical slot numbers. Introducing the Silo Capacity Visualizer To help explain how placement-IDs work, I created the Silo Capacity Visualizer. This tool is just for learning and isn’t connected to Broadcom or NVIDIA. It doesn’t predict how a whole cluster will behave or simulate every scheduling decision. Instead, it answers one question: how does your choice of profile catalog affect the memory layout inside a GPU? By running random placement sequences, the tool shows how different profile mixes affect the amount of capacity siloed due to placement-ID misalignment. It’s not meant to give exact predictions. The goal is to help architects see why some profile combinations work better than others before using them at scale. The first step is to select the GPU model you use in your datacenter. The dropdown menu lists all GPUs, showing their placement IDs as found in the official NVIDIA vGPU User Guide. These are the ones used in VCF environments. I selected the popular A100 PCIe 80GB device, and the UI displays the related vGPU time-sliced profiles. To demonstrate the tool, I picked three profiles: A100-8C (8GB), A100-10C (10GB), and A100-20C (20GB). In the field below, the three profiles appear, and the tool lists the placement IDs for those vGPU profiles. The tool provides three options. In the profile, the + button lets you manually place the profile in GPU memory. When clicking the + button for the A100D-8C profile, it appears in the GPU memory map, and the allocated memory counter is incremented by the profile’s memory capacity. The tool shows how much memory is free in GB (72GB) and as a percentage (90%). There is also a counter that shows how many profiles are currently on that device. The memory map shows the placement IDs consumed by the profile in the color that matches the GPU profile shown in the allocate instances element. Beneath the GPU memory map, the allocated instances show the placed GPU profiles in order and where they are placed on the GPU. When manually adding a 10GB profile, the following happens: the 10GB profile is placed, and as it its placement ID doesn’t align with the last placement ID of the 8GB profile, the memory map shows the siloed memory. Directly below the GPU memory map, a warning appears that 2GB (3%) is siloed. At the end of every profile, an option (x) is provided to remove that profile. Besides manual placement, there’s a randomizer that places the selected profiles in random order. It keeps going until it can’t place any more profiles. This helps you see what might happen in practice. In most environments, VMs with vGPU profiles are set up through a self-service portal, so random startup and placement orders are common. The next feature lets you randomize 10,000 times. This uses a Monte Carlo simulation, which models the chances of different outcomes by running thousands of random scenarios. Instead of checking every possible combination, this approach provides useful insights into the worst-case, median, and average siloed capacity. The tool provides a report with these results. If you want to see more, you can display the worst-case scenario on the memory map. At the bottom of the screen, the profile recommender helps you pick the profile that works best with your selected profiles. It aims to avoid fragmentation and extra memory use for your workloads. In this example, I chose 7GB to see which profile would give enough resources without causing siloed memory on the GPU or across the cluster: Here, the recommender suggests giving a 7GB workload an A100D-10C profile. This gives the workload an additional 3GB of space and avoids siloing issues. When used with the 20GB profile, all profiles line up perfectly, and the Placement density tile shows 12 slots. Let’s break down what that means. Placement density is the total number of single-profile slots on the GPU, summed across all the profile types you’re using. For each profile size, you calculate: slots = floor(GPU total GB / profile GB). Then you add them together. For example, with an A100 80GB and profiles 10C and 20C: 10C (10 GB): floor(80 / 10) = 8 slots 20C (20 GB): floor(80 / 20) = 4 slots Placement density is 8 + 4 = 12. So, ‘placement density 12’ means you can fit 12 single-VM placements in total across the 10C and 20C profiles (8 with just 10C, 4 with just 20C)(not all at the same time). The recommender uses this to measure scheduling flexibility: more slots mean more ways to fit VMs on the GPU, which usually leads to better utilization. Higher density is better when the silo impact is similar. This matters when we look at probability calculations for the whole cluster, which I’ll cover in part 7. The ‘active profile types’ tile shows how many profiles are used in the calculations, how their placement boundaries line up, and gives a short explanation of why a profile is the best fit, acceptable, or one to avoid for this workload. The last tool you’ll see with the profile recommender is the ‘Profile not in your mix’ analysis. It suggests up to three profiles you haven’t picked but might want to consider. These are chosen based on headroom and good boundary alignment. There’s no best fit or avoid label, but you can check the ‘silo if added’ tile to see if it makes sense to add them to your catalog. Practical Example on an A100 80GB Using the tool walkthrough examples, I took an A100 80GB PCIe GPU and compared two catalogs. Slim-Fit Approach Catalog A This catalog seems efficient for workloads because it gives you lots of sizing options: A100-8C (8GB) A100-10C (10GB) A100-20C (20GB) Running Monte Carlo simulations with random placement order shows the downside. In the worst case, you can end up with 24GB of siloed capacity on one GPU. Even though the memory is technically free, placement-ID boundaries stop more workloads from using it. The visualizer’s worst-case layout makes it clear how misalignment between 8GB and 20GB profiles creates unusable gaps. Right-Sized Approach Catalog A With the Profile Recommender, I looked at a workload that needs about 7GB of GPU memory. Instead of picking an 8GB profile, the tool suggests a 10GB profile because it lines up better for placement. This results in a simpler catalog: A100-10C (10GB) A100-20C (20GB) Running the same simulation produces a striking result. Across 10,000 randomized runs, the worst-case siloed capacity observed is 0GB. The median and average siloed capacities are also zero. The reason is simple: these profiles align perfectly with placement IDs, allowing the GPU manager to continuously fill gaps without creating unusable space. What This Actually Shows The lesson is not that smaller profiles are bad. The lesson is that placement alignment matters more than theoretical density. Catalog A appears flexible but creates layouts that are sensitive to startup order. Catalog B reduces profile variety slightly but produces predictable outcomes regardless of placement sequence. In other words, right-sizing is not about giving workloads more memory than they need. It is about choosing profiles that allow the platform to keep reusing capacity. Key Insight Mixed-size mode doesn’t automatically stop fragmentation. It just changes where the problem shows up. Instead of profile locking, efficiency now depends on alignment between placement-ID and profile selection. The platform will always try to minimize fragmentation locally, but profile catalogs that align well allow those local decisions to produce better global outcomes. Looking Ahead So far, I’ve focused on what happens inside a single GPU. The next big question is: how do these choices affect capacity across a whole cluster? That’s where I’ll introduce a new idea in the next part: Continuous Placeable Capacity (CPC). GPU capacity isn’t just about what’s free, but about what can still be placed. Part 7 will build on these ideas and move from looking at single GPUs to making decisions for the whole cluster. ================================================================================ Title: How Same Size vGPU Mode and Right-sizing Shape GPU Placement Efficiency URL: https://frankdenneman.ai/2026-02-19-How-Same-Size-vGPU-Mode-and-Right-sizing-Shape-GPU-Placement-Efficiency/ Date: 2026-02-19 Architecting AI Infrastructure - Part 5 In the previous article, we looked at how GPUs are placed within an ESXi host and how GPU modes and assignment policies determine which physical GPU a workload uses. These decisions impact more than just the initial placement of workloads. They also shape how GPU capacity changes over time, affecting fragmentation, consolidation, and how easily new workloads can be scheduled. In this article, we will look at workloads that use fractional GPU profiles and how their sizing choices impact overall platform efficiency. AI platforms typically run a wide range of workloads. Along with large language models that use full GPUs, span several GPUs, or span across multiple hosts, most setups also include smaller but important services. Embedding models support retrieval pipelines, small language models handle tasks like moderation and security analysis, and data science notebooks support experimentation and model development. As mentioned in the earlier article about same-size and mixed-size deployments, these workloads often use fractional vGPU profiles. To get the most from the platform, you need to choose the right vGPU profile and vGPU mode. It can be tempting to size profiles as closely as possible to actual memory use, but sizing too tightly can isolate GPU capacity and lower overall efficiency. Slim-fitting vGPU Profiles to Workload Memory Footprint The following examples show a representative selection of common AI workloads, their typical GPU memory consumption, and the vGPU profile each would receive when sized as closely as possible to its observed memory footprint. Instead of right-sizing (you’ll see later), let’s call this slim-fitting. Workload Type Model Static Memory Dynamic Memory Total GPU Memory Slim-Fit Embedding BGE-M3 (560M) ~1.1GB (FP16) 2–3GB (activations + batch) 3–4GB 4GB Embedding Qwen3-Embedding-8B ~16GB (FP16) 4–6GB (larger batches + activations) 20–22GB 20GB (lean) or 40GB (production) Small LLM GLM-4.7-Flash (30B MoE) 12–15GB (Q3–INT4) 3–5GB (KV cache, low concurrency) 18–20GB 20GB Small LLM GPT-OSS-20B (MXFP4) ~10GB static (MXFP4 weights) 3–5GB (KV cache + activations) 13–15GB 16GB Full-slot LLM GPT-OSS-120B (heuristic/security scanning) Full model footprint Minimal concurrency, low context ~80GB 80GB Notebook / Data Science Jupyter workloads Variable Variable Typically <20GB 10GB Theoretical Slot Capacity Before we look at placement behavior, it’s useful to understand the cluster’s theoretical capacity. With six 80GB GPUs, the number of possible slots depends on the vGPU profile size. The numbers below show a scenario where every GPU uses the same profile type. These are the maximum slot counts if only one vGPU profile is used. In real setups with different profile sizes, it’s much harder to predict available slots because it depends on how profiles are distributed across GPUs. Profile Size Slots per 80GB GPU Total Slots (6 GPUs) 4GB 20 120 10GB 8 48 16GB 5 30 20GB 4 24 40GB 2 12 80GB 1 6 These numbers show the highest possible density. In reality, factors like vGPU mode, assignment policy, profile variety, and how workloads start will decide how much of this capacity you can actually use over time. Choosing the vGPU Mode This points to a catalog of six profiles: 4GB, 10GB, 16GB, 20GB, 40GB, and 80GB. But defining the catalog is just the first step. Next, you need to decide how these profiles can share physical GPUs, which depends on the chosen vGPU mode and assignment policy. To see how these choices affect real-world behavior, let’s look at practical examples of how GPU modes work when workloads start up. Before we go through the distribution example, it’s important to know the default behavior. By default, DRS uses Best Performance mode, which evenly distributes workloads across hosts rather than consolidating them. There is an advanced setting to favor consolidation, but it mainly affects full-GPU and multi-GPU setups; fractional vGPU profiles are not considered during initial placement. The example below uses an empty cluster to make the behavior clear. In real environments with running workloads, host selection can change based on cluster balance and VM scoring, which can affect where new workloads are placed. Same Size vGPU Mode – Best Performance Assignment Policy With Same Size mode and the Best Performance assignment policy, vSphere spreads workloads across available GPUs to balance load. In a self-service environment where startup order is unpredictable, smaller workloads often land first, causing GPUs to lock to different profile sizes early in the process. In this example, embedding models, notebooks, and small LLMs start before the larger 40GB and 80GB workloads. Since identical profiles are spread across hosts, GPUs quickly lock into 4GB, 10GB, 16GB, and 20GB profiles throughout the cluster. When the larger workloads try to start, there are no GPUs left that can take a 40GB or 80GB profile, even though there is still enough total memory in the environment. Once the first workloads start, each GPU becomes locked to a specific profile size. At this stage, the cluster stops acting as a shared pool of GPU memory. Now, capacity is only available within the profile domains set during startup. Profile Size Available Slots in Cluster 4GB 38 10GB 14 16GB 4 20GB 3 40GB 0 80GB 0 Same Size vGPU Mode – Consolidation Assignment Policy Consolidation does not affect which host is chosen; it only changes how GPUs within the selected host are used. If you use the same workload mix with Same Size mode but switch to the Consolidation assignment policy, the outcome changes a lot. Host selection still uses round-robin, but within each host, workloads are grouped onto compatible GPUs instead of being spread out. This means fewer GPUs get locked early, leaving some GPUs free for larger profiles later on. As a result, the 40GB workload can use an unlocked GPU, and a full-slot 80GB workload can still be placed because at least one GPU is left completely free. The total resources in the cluster stay the same, but the assignment strategy decides whether capacity stays usable or becomes fragmented. The failure is not caused by a lack of GPU memory, but by profile fragmentation introduced during random startup. Available Slot Capacity After Consolidation The consolidation example shows that placement policy does not increase total GPU capacity, but it does affect how much of that capacity you can actually use. Profile Size Spread (Best Performance) Group (Consolidation) 4GB 38 18 10GB 14 6 16GB 4 3 20GB 3 2 40GB 0 1 80GB 0 0 In Group mode, an empty GPU is available for any profile because it is not locked yet. The main difference from the spread example is not total memory, but flexibility. Consolidation leaves fewer profile domains active, so larger profiles still have valid placement options. Re-evaluating Right-sizing Right-sizing is therefore not only a workload decision but also a platform decision. The goal is not simply to match a workload’s minimum memory requirement (slim-fit), but to choose profile sizes that improve opportunities for cluster-wide capacity allocation (right-size). Profile Size Spread – Placed Spread – Available Group – Placed Group – Available 4GB 2 38 2 18 10GB 3 13 3 5 16GB 1 4 1 4 20GB 2 2 2 2 40GB 0 0 1 1 80GB 0 0 1 0 Right-sizing GPUs = workload fit + cluster efficiency Same Size vGPU Mode – Best Performance (Right-sized Profiles) In the right-sized spread scenario, the cluster converges toward only two active profile sizes: 10GB and 20GB. While this reduces profile diversity compared to the slim-fit experiment, the spreading behavior still distributes workloads across hosts and GPUs, leading to early profile locking. As a result, only a subset of the workload types can be placed, and larger profiles never become active. The diagram below illustrates how the cluster ends up operating with just two profile domains despite having sufficient raw GPU capacity. Profile Size Placed Available 10GB 5 27 20GB 3 5 40GB 0 0 80GB 0 0 Same Size vGPU Mode – Consolidation (Right-sized Profiles) With consolidation enabled, the same workload mix produces a very different outcome. All four profile sizes become active, meaning every workload type in the catalog is successfully deployed. Because compatible workloads are grouped onto existing GPUs, one GPU remains completely uncommitted. This fully open device represents flexible capacity that can still accept any profile size, including large or full-slot workloads. The diagram below highlights how consolidation preserves both workload coverage and future placement flexibility. Profile Size Spread – Placed Spread – Available Group – Placed Group – Available 10GB 5 27 5 19 20GB 3 5 3 5 40GB 0 0 1 3 80GB 0 0 1 1 Consolidation increases the benefits of right-sizing because having fewer profiles means there are fewer lock boundaries. Observation Right-sizing profiles does not add more total GPU capacity. Instead, it helps make better use of what you have. With fewer profile sizes, consolidation works better, fragmentation goes down, and the cluster acts more like a shared resource pool instead of a set of isolated profile domains. Looking Ahead: Mixed Size Mode This article focused on Same Size mode, where GPUs lock to a profile after the first workload is placed. In this setup, right-sizing helps efficiency by reducing the number of profile domains and keeping placement flexible. Mixed Size mode changes things completely. Profile locking goes away, but a new challenge appears: placement IDs and how profile sizes fit into GPU memory ranges. In the next part of this series, we will repeat the experiment using Mixed Size mode to show how placement changes, why fragmentation still happens, and why right-sizing still matters even without GPU locking. Same Size mode fragments GPUs across different profiles, while Mixed Size mode can cause fragmentation within placement ranges. ================================================================================ Title: How vSphere GPU Modes and Assignment Policies Determine Host Level Placement URL: https://frankdenneman.ai/2026-02-17-How-vSphere-GPU-Modes-and-Assignment-Policies-Determine-Host-Level-Placement/ Date: 2026-02-17 Architecting AI Infrastructure - Part 4 In the last article, we tracked a GPU-backed VM from resource configuration to host selection. DRS evaluated the cluster, Assignable Hardware filtered hosts for GPU compatibility, DRS ran its Goodness calculation, and picked a destination host. Now, the host is selected. But the placement is not finished. Inside the host, another set of decisions decides which physical GPU gets the workload and what types of workloads that GPU will handle from then on. These host-level choices are less visible than DRS decisions. They do not show up in dashboards or trigger alerts. However, their effects add up over time, and they play a key role in keeping a shared AI platform healthy or letting it decline. What AI Workloads Actually Ask of a GPU Most AI workloads fall into four main GPU usage patterns. Passthrough lets one virtual machine use one physical GPU directly as a PCI device. This setup is used when software requires bare-metal-level GPU access or when a workload is a specialized appliance with a fixed GPU configuration. The rule is simple: one VM, one GPU, no sharing. The trade-offs are no vMotion, no DRS load balancing, and no automatic HA recovery if static direct path IO is used. Fractional GPU splits a physical GPU so that several virtual machines can share it simultaneously, each with its own memory and limited compute power. Embedding models, reranker services, and Jupyter notebooks often use this setup. For example, an embedding model usually needs 4 to 20GB of GPU memory, and a notebook uses the GPU in short bursts between long waits for data. Neither needs a full 80GB H100. A fractional profile gives each workload what it needs: dedicated memory, limited compute, and a resource shape the platform can manage for many users at once. This article focuses on this shape, its setup options, and the fragmentation it can cause. Full GPU gives one virtual machine full access to a physical GPU using a full-size vGPU profile. Unlike passthrough, this setup still allows vMotion, DRS, and HA recovery. It is used for production inference services for large models, and fine-tuning runs that use most of the GPU’s memory. Full GPU use has stricter placement limits than fractional GPU use, since each use takes up an entire device. Multi-GPU gives a single virtual machine access to several physical GPUs, connected via NVLink or NVSwitch for fast communication. This is not just a bigger version of single-GPU use; it is a different shape in which the interconnect topology is part of the resource contract. This setup is used for fine-tuning and inference on the largest production models. Full GPU and multi-GPU placement will be discussed in Part 6. The Handoff from DRS to the Host After DRS picks a host, it hands off the placement to the Assignable Hardware framework. The platform now knows which host will get the virtual machine, but it does not yet know which physical GPU on that host will be used, or whether any are available given their current state. At this stage, two host-level settings come into play. The first is the GPU Assignment Policy, which controls how virtual machines are spread across the host’s physical GPUs. The second is the vGPU Mode, which specifies which profile combination a GPU accepts at any given time. These settings work together to decide where workloads go and how flexible the platform will be for future jobs. For Kubernetes operators, this is similar to how the scheduler picks a node and the device plugin chooses which GPU to use. In vSphere, the device plugin role is managed by administrator-defined policies. This is not just a first-available match, but a planned way to distribute work across the platform. The diagram below shows all host and GPU policies and their relationships. Each part will be explained in this series. GPU Assignment Policy The GPU Assignment Policy answers a key question: when a VM lands on a host with several eligible GPUs, which one should it use? Spread VMs across GPUs does what it promises: spreads VMs across GPUs, placing each new VM on the device with the fewest vGPUs. This keeps the load balanced and prevents any one device from becoming a bottleneck. In the UI, this is also referred to as Best Performance. Group VMs on GPU until full, does the opposite, filling up one GPU before moving to the next. This keeps as many GPUs as possible in a neutral, unbound state, ready to accept any compatible profile when needed. In the UI, this is also referred to as GPU consolidation. Please note: the GPU Assignment Policy only applies when the GPU is in Same Size mode. In Mixed Size mode, the host selects the GPU based on availability and profile compatibility, so this setting is ignored. The diagram shows a startup order of three VMs: the first two have a 10GB vGPU profile configured, while the third has a 20GB vGPU profile. In the left scenario, the ESXi host is configured with a ‘Spread VMs across GPUs’ assignment policy, thus VM1 goes to GPU1 and VM2 goes to GPU2. The available GPU memory is still 140 GB, yet VM3 cannot find a compatible GPU in this scenario. Hopefully, another ESXi host in the cluster is either empty or has a 20GB profile active. On the right, the ESXi host is now configured with a ‘Group VMs on GPU policy’ and VMs 1 and 2 are both placed on GPU 1, VM3 can be powered on, as GPU2 is in a neutral, unbound state. Both scenarios use the ‘Same Size’ vGPU mode. vGPU Mode vGPU Mode is set for each physical GPU and determines which profiles that GPU accepts at any time. There are two modes in the UI, but one can consider MIG mode as the silent third one. Choosing between these modes is one of the most important decisions when setting up a shared AI platform. Same Size: Predictability Through Uniformity In Same Size mode, a physical GPU starts in a neutral state and can accept any supported vGPU profile. The first virtual machine placed on that GPU sets the profile. After that, the GPU is locked, and every VM must use the same profile. Any request for a different size is rejected, no matter how much capacity is left, until all VMs are powered off and the GPU returns to neutral. In a controlled environment with a limited and clear set of profiles, this locking is helpful. It enforces a GPU contract and makes usage predictable. Problems appear in self-service setups where placement order is random. For example, if a data scientist starts a workload with a 10C profile on Monday morning, that GPU is locked to 10C until the workload ends. Later, an engineer who needs an 8C slot finds a GPU with 70GB free and six open slots, but none can accept their request. The capacity is there, but the lock prevents its use. Mixed Size: Flexibility With Alignment Constraints Mixed Size mode, added in vSphere 8 Update 3 for Ampere and newer GPUs, removes the locking behavior. In this mode, a GPU can accept profiles of different sizes at the same time. For example, a notebook with a 10C profile and an embedding service with an 8C profile can share the same GPU without blocking each other. This flexibility comes with one main constraint: placement IDs. A GPU’s memory is divided into fixed ranges, and profiles must fit into ranges that match their size. On an 80GB H100, profiles of 10GB, 20GB, and 40GB fit together without leaving unused gaps. Profiles like 8GB or 16GB do not line up with 20GB, so mixing them can leave small gaps that no other profile can use. For example, the H100 supports 4GB, 5GB, 8GB, 10GB, 16GB, 20GB, 40GB, and 80GB profiles. If VM1, VM2, and VM4 use a 16GB profile and VM3 uses a 20GB profile, all VMs can be placed, but VM3 must fit within its placement ID, leaving 12GB unused. The platform does not stop these gaps from happening. The key is to choose profiles that both support your workload and align with placement IDs. The NVIDIA AI Enterprise User Guide is a helpful resource. Sometimes, it is better to use a slightly larger profile, like 20GB instead of 16GB, if it helps support multiple vGPU profiles. If done right, Mixed Size mode gives the most placement flexibility with no locking and no wasted capacity. MIG: Hardware Isolation With a Fixed Resource Budget Multi-Instance GPU mode splits the physical GPU at the hardware level before any workloads start. Each partition gets its own compute and memory slices that no other workload can use. This is not just isolation by software or a scheduler. It is a true hardware partition, so two workloads on the same GPU cannot see or affect each other’s resources. The trade-off is less flexibility. An H100 80GB has seven compute slices and eight memory slices. MIG profiles use these in fixed combinations, and some choices leave compute slices unused. The 3g.40gb profile uses three compute, and four memory slices, so you can fit two per GPU and only one compute slice is left unused. The 4g.40gb profile also gives 40GB, but only one fits per GPU and three compute slices are left unused. For 40GB workloads, the 3g.40gb profile is usually the better choice. MIG mode also changes how the platform works in two key ways. First, all GPU Assignment Policy settings are ignored; the host assigns MIG instances based only on availability, without using Best Performance or Consolidation logic. Second, GPUs in MIG mode cannot be used in full-GPU or multi-GPU setups which require time-slice mode. On Hopper-generation GPUs, dynamic MIG switching lets an idle GPU switch modes automatically, but this only happens when the GPU is completely empty. In a busy cluster, you need to plan for this when designing capacity. What Comes Next These three modes affect not only how a GPU handles today’s workload, but also how the host fits into the cluster’s capacity over time. The next article will help you decide which mode, policy, and combination best fit your environment, and will look at what is possible when GPU isolation means you no longer need to separate production and testing workloads. ================================================================================ Title: How vSphere DRS Makes GPU Placement Decisions URL: https://frankdenneman.ai/2026-02-13-how-vsphere-drs-makes-gpu-placement-decisions/ Date: 2026-02-13 Architecting AI Infrastructure - Part 3. In the first two articles, I looked at GPU consumption models and how AI workloads state their accelerator needs. In vSphere, these models take shape through virtual machine settings. CPU reservations, memory guarantees, and GPU profile choices together create a clear resource contract. This article moves from discussing resource consumption to explaining placement. After a workload states its requirements, how does the platform decide where to run it? GPU placement in vSphere is not random or based on guesswork. Instead, it is a structured orchestration process that combines declarative intent, hardware awareness, and cluster-wide optimization. VM Configuration Defines the Resource Contract In vSphere, a GPU-backed workload is defined by its virtual machine configuration. That configuration expresses intent: how much GPU memory is required, which profile to use, how many devices are needed, and what CPU and memory guarantees accompany them. These needs are evaluated when the virtual machine starts. The platform ensures the cluster meets the resource contract before allowing the workload to run. Once resources are assigned, the accelerator setup remains the same throughout the workload’s lifecycle. This stability helps keep performance predictable, which is important for model behavior and service reliability. The same principle applies in Kubernetes environments such as VMware Kubernetes Service. Worker nodes are virtual machines, and their GPU configuration determines the accelerator capacity exposed to the container runtime. Pods request GPUs through Kubernetes abstractions, but those requests are handled at the infrastructure level based on the VM configuration. Kubernetes expresses workload-level demand. vSphere enforces infrastructure-level guarantees. GPU Mode Defines the Device Participation Model At the host level, GPU mode determines how devices participate in the cluster resource model. With passthrough mode, the GPU appears as a full physical device. The workload uses the whole accelerator for as long as it runs. This setup gives clear ownership and full access to the device. With vGPU mode, the GPU uses profiles to set clear memory and compute limits. One physical device can support several workloads at once, while keeping them separate. Whether using time-sliced setups or hardware-partitioned MIG mode, allocations always stay within the device’s physical limits. GPU memory is never oversubscribed and shared beyond its limits. In time-sliced mode, compute cores are shared, but memory stays dedicated. In the image below, two VMs with different vGPU profiles use the same physical device. VM4 is scheduled first, and feeds all the cores with the data stored in its allocated memory space. VM8 awaits its time-slice, when it’s scheduled it feeds all the cores with the data stored in its memory space. In MIG mode, hardware enforces isolation with fixed memory and compute partitions, referred to by NVIDIA as instances. VM4 and VM8 are scheduled continuously, each having access to a subset of cores and memory space that are shaped by the MIG vGPU profile (MIG instance). The selected GPU mode (MIG or time-slicing) determines how a device participates in the cluster’s allocation model. Allocation Shapes Device Topology In vGPU mode, a device begins in a neutral state. The first admitted workload selects a profile, shaping how the device is partitioned or scheduled. After that, the device’s setup matches the chosen allocation. Future workloads follow this setup. There is flexibility when assigning resources initially, but the mode (Time-slice or MIG) is maintained throughout use until the GPU is empty again. This approach ensures the device setup matches the workload’s needs and keeps things predictable across the cluster. From Static Device Binding to Cluster-Aware Allocation With passthrough enabled, the GPU is presented to the virtual machine as a physical PCI device. How the device is referenced in the VM configuration directly affects placement and recovery. With traditional static DirectPath I/O, the virtual machine configuration references a specific physical device using its Segment/Bus/Device/Function (SBDF) address. That address corresponds to a particular ESXi host and a specific PCIe slot on that host. The relationship between workload and hardware is explicit and immutable. This model gives each device a fixed identity. The workload is bound to a specific accelerator. However, it can only run on that host-device pair. If the host goes into maintenance or the device fails, the virtual machine cannot restart elsewhere without manual changes. If the GPU is replaced or moved, the SBDF reference must be updated before the VM can start again. Placement and recovery depend on a single hardware instance. Dynamic DirectPath I/O modernizes this approach using the Assignable Hardware framework of the vSphere cluster. Instead of using a fixed SBDF address, the virtual machine declares device characteristics such as vendor IDs, device features, and hardware labels. In this example below, a label ‘Inference’ was assigned to the H100 GPU. When powered on, the platform evaluates all hosts in the cluster and finds the GPUs that meet the declared requirements stated in the VM configuration (Inference). A compatible device is then assigned automatically. The allocation remains exclusive and predictable during runtime. What changes is the range of eligible hosts and recovery options. Placement is now cluster-aware. If a host fails, vSphere High Availability can try to restart the virtual machine on another host that meets the device requirements, as long as there is enough capacity. Dynamic DirectPath I/O, therefore, expands failover options from a single node to the entire compatible cluster. Hardware ownership remains explicit. Runtime allocation remains stable. Recovery behavior becomes infrastructure-driven rather than manually reconfigured. This model balances predictability with cluster-level resiliency. The workload keeps direct access to hardware, while the platform stays aware of other compatible options. From Resource Declaration to Cluster-Level Orchestration When a GPU-backed virtual machine powers on, its resource contract is already set. The placement process starts by checking that contract against the cluster’s hardware setup. The Assignable Hardware framework does the first evaluation. For passthrough configurations using Dynamic DirectPath I/O, host eligibility is determined by matching device characteristics to available hardware. For vGPU configuration, the system checks vGPU profile compatibility, device mode, available memory, and topology limits. Hosts that do not meet the GPU requirements are removed from the list. Only hosts that can provide the needed accelerator setup are left. Once the list of compatible hosts is defined, DRS evaluates placement using its Goodness calculation. This process looks at everything together: CPU headroom, memory reservations, NUMA alignment, datastore access, network reachability, and overall cluster balance. Consider a cluster of six ESXi hosts. Two hosts operate in passthrough mode. Three hosts operate in vGPU time-slice mode. One host operates in MIG mode. Now imagine a virtual machine configured with a 40GB time-sliced vGPU profile, 16 vCPUs, and 128GB of fully reserved memory. The placement process begins with compatibility filtering. The two passthrough hosts are excluded because they cannot provide a time-sliced profile. The MIG host is excluded because the requested configuration requires time-slice mode. That leaves three compatible hosts. Within the eligible host pool, DRS applies its Goodness calculation. It evaluates which host can satisfy the VM’s CPU and memory guarantees while keeping the cluster load balanced. The goal is to maximize VM Happiness while keeping the cluster healthy. VM Happiness shows how closely a virtual machine gets the resources it needs. Accelerator capacity alone does not guarantee good performance. CPU scheduling, memory location, storage access, and network placement all affect how well things run. DRS looks at all these factors together to find the best placement. GPU compatibility decides where a workload can run. The Goodness calculation determines where it should run. Heterogeneous Accelerators and Topology Awareness This orchestration model supports heterogeneity at the device level. A cluster can simultaneously support passthrough GPUs, time-sliced vGPU devices, and MIG-partitioned accelerators. Assignable Hardware filters hosts based on declared device characteristics, allowing each virtual machine to express the accelerator model that best fits its workload. Please note that time-sliced vGPU mode spans a broad spectrum of configurations. It can expose fractional profiles for shared inference, full-device profiles for exclusive consumption, or be captured in a topology-aware multi-GPU allocation with vSphere device groups for larger workloads. When combined with Device Groups and integration with NVIDIA Fabric Manager, vSphere understands GPU interconnect topology. Multi-GPU virtual machines can be placed on devices that share NVLink domains, preserving bandwidth and latency characteristics required by distributed inference. GPUs are evaluated not as abstract counters, but as structured, topology-aware resources. Placement Policy: Balancing Utilization and Fragmentation Besides compatibility and Goodness evaluation, DRS placement can also be guided by policy. By default, DRS operates in a Best Performance assignment model. GPU-backed workloads are distributed across compatible hosts to balance load and maximize aggregate utilization. This keeps the workload distribution balanced and avoids concentrating demand on a single host. In fast-changing AI environments, fragmentation can become a big problem. As single-GPU and multi-GPU workloads come and go, isolated devices may be left scattered across hosts, making it harder to meet future high-demand placements. For these situations, DRS can use Consolidation mode. In this mode, workloads are packed onto fewer hosts when possible. Reducing the spread of allocations makes it easier to place future workloads. Best Performance focuses on balanced use of resources right now. Consolidation focuses on keeping placement flexible for the future. Compatibility is always enforced. The Goodness calculation still looks at everything together. Policy guides which options are preferred within the eligible group. Complementary to Kubernetes Scheduling This layered orchestration model aligns naturally with Kubernetes scheduling principles. Kubernetes DRA evaluates node labels, resource availability, and topology constraints before binding a Pod. vSphere works with similar principles at the infrastructure layer. Device characteristics are declared explicitly. Hardware capabilities are surfaced through structured identifiers and topology awareness. Placement decisions respect real interconnect domains and resource boundaries. Kubernetes decides which node a workload should run on. vSphere decides how hardware resources are allocated and balanced across the infrastructure. Together, they create a layered control system that matches workload intent with reliable hardware enforcement. Looking Ahead: GPU Policy and Long-Term Cluster Efficiency Placement rules set correctness and best use at the time of admission. Long-term efficiency, though, depends on GPU policy decisions that guide allocation over time. Mixed mode versus Same Size mode, profile mixing rules, device grouping, and assignment strategies all affect how flexible an accelerated cluster can be. These policies decide how GPUs are divided, how fragmentation builds up, and how likely it is that future placements will work as workloads grow. The next article will look at these GPU policy controls in detail. Knowing how policy affects topology and allocation is key to building AI clusters that stay resilient, efficient, and scalable under heavy use. ================================================================================ Title: GPU Consumption Models as the First Architectural Choice in Production AI URL: https://frankdenneman.ai/2026-02-11-gpu-consumption-models-as-the-first-architectural-choice-in-production-ai/ Date: 2026-02-11 Architecting AI Infrastructure - Part 2 The previous article covered GPU placement as part of the platform’s lifecycle, not just a scheduling step. These choices affect what the platform can handle as workloads evolve. Before making placement decisions, it’s worth asking: how do AI workloads use GPUs? This question is important because not every GPU workload requires the same resources. Two services might both need accelerators, but can be very different in memory use, how they run, and how much they depend on other GPUs. These differences set the platform’s limits well before the scheduler gets involved. So, GPU consumption models are not just an optimization detail. They are the first architectural choice for any AI platform. GPU consumption follows request patterns, not scarcity In most cases, users ask for the biggest configuration they can get. More GPU memory, more compute, and exclusive access are seen as the safest ways to ensure stable and consistent performance. This behavior follows old patterns in how people use infrastructure. When performance is important, asking for more resources seems like the easiest way to lower risk. Bigger allocations are thought to prevent interference, make troubleshooting easier, and reduce uncertainty. What has changed is not user behavior, but the nature of GPU contention. Why GPU contention is different from CPU and memory contention CPU and system memory have been built to handle oversubscription for a long time. Advanced schedulers, preemption, and memory management help manage contention smoothly. When CPUs are overloaded, workloads just slow down. When memory is tight, systems use reservations, limits, and reclamation to stay stable. GPU contention is different. Depending on the setup, a GPU either shares execution over time or splits it into separate parts. In time-sliced setups, compute tasks are shared, but each workload keeps its own memory. A virtual GPU keeps its full frame buffer whether it’s busy or not. In partitioned setups like Multi-Instance GPU, the device is split into hardware-isolated units, each with its own compute and memory. So, sharing a GPU isn’t like traditional oversubscription. It’s more like structured isolation, with clear resource boundaries. This difference changes how we should think about GPU sharing. Fractional GPUs are not a compromise People often describe fractional GPU use as a way to increase density or save costs. In reality, it matches the needs of many production AI workloads. Embedding models, rerankers, and inference services usually have small, fixed memory needs and focus on throughput. They need reliable access to acceleration, not full control of a device. What’s important is memory isolation and controlled compute access, not owning all the resources. Modern quantization techniques support this change. Large models in formats like FP4 use much less static memory. Even models with tens or hundreds of billions of parameters might not fill a high-end accelerator’s memory. While dynamic memory use still matters, especially with many users, a static footprint alone doesn’t justify exclusive allocation anymore. Here, fractional GPUs aren’t about oversubscription. They set realistic resource limits that match how workloads actually behave once we understand them. By clearly limiting GPU memory and compute, fractional GPUs define a workload shape the platform can manage reliably. This model works best when both static needs and dynamic behavior are predictable. If memory use, concurrency, and execution patterns are well understood, you can set clear limits. Then, multiple workloads can run together without interference, and placement stays flexible. Stability and consistency come from clear, enforceable resource boundaries, not exclusivity. Passthrough and exclusivity Passthrough remains an important consumption model in production environments that require predictability, consistent performance, and clear ownership of accelerator resources. When a GPU is set up for passthrough, it is consumed as a whole device. Assigning that GPU to a single workload removes contention and makes performance easier to measure. At runtime, the workload has full control over the device, with no interference from other users. The key feature of passthrough is not how it works at runtime, but where rigidity is introduced. Enabling passthrough sets up the GPU to be used only as a full device. This is true whether or not the GPU is currently assigned to a virtual machine. When a virtual machine uses a passthrough GPU, the workload keeps this fixed consumption shape for as long as it runs. Changing that shape is not something you can do at the workload level. It requires stopping the workload and reconfiguring how the GPU is set up and assigned at the host and device levels. Unlike vGPU-based consumption, where an administrator can pick a different profile or VM class and just reboot the virtual machine. This difference matters in environments where AI workloads change quickly. Data scientists often try new model versions, use new quantization techniques, and adjust serving parameters, all of which can change a workload’s resource profile. Hardware-level allocation assumes the right consumption shape is known ahead of time. When that is not true, the cost of rigidity becomes clear. Tools like Dynamic DirectPath I/O can make placement more flexible and help with failure recovery by hiding the physical device identity at power-on. However, they do not change the main feature of passthrough: while the workload is running, the GPU stays a single, indivisible allocation unit. Passthrough is not the wrong choice. It is the right option when workload requirements are stable and well understood. In environments with frequent change, it becomes a deliberate architectural decision that trades adaptability for certainty. Full GPUs as a necessity, not a preference Assigning a full GPU to a workload is often not about comfort or simplicity. In many cases, it is a direct result of the workload’s resource needs. Some workloads require the full capacity of a GPU to run properly. Their static memory footprint may already use most of the device, leaving little room for sharing. Others have dynamic behavior where memory use grows with sequence length, batch size, or concurrency, making it hard to set safe fractional boundaries. User behavior shows this pattern. When data scientists are sure a workload fits within clear memory and compute limits, they often use fractional GPUs. When they are not confident, they tend to use full GPUs to regain predictability. Full GPUs are often chosen because the workload shape is uncertain, not because sharing is undesirable. In many cases, the static requirements of a model are well understood and clearly fit within GPU memory. What remains unknown is how the application will behave under real traffic. Consumption rate, concurrency, input variability, and batching strategies all influence dynamic memory usage, and these characteristics are often only discovered after deployment. Allocating a full GPU becomes a way to absorb that uncertainty while the workload’s true behavior reveals itself. It is important to distinguish this consumption model from passthrough. A full GPU may be delivered through passthrough or through a full-size virtual GPU profile. In both cases, the workload consumes the entire device. What differs is how the platform manages placement and lifecycle. What remains consistent is that the unit of allocation is the full GPU. From a platform perspective, full-GPU consumption introduces stronger constraints than fractional sharing. Each placement fixes the workload shape at the largest possible granularity. This does not make full GPUs inefficient or undesirable. It makes them precise. They are the correct choice when the workload requires it. When a workload spans GPUs When a workload needs more than one GPU, GPU consumption changes in a fundamental way. The GPU is no longer the unit of placement. Multi-GPU workloads are not just larger versions of single-GPU workloads. They depend on communication between devices, often at bandwidths higher than what PCIe alone can provide. At that point, topology becomes part of the contract between the workload and the platform. Distributed model serving, large-scale inference, and training workloads rely on fast interconnects to synchronize parameters, exchange activations, or shard model state. Technologies such as NVIDIA NVLink and NVSwitch enable this communication at scale. From a placement perspective, this creates a hard boundary. GPUs can no longer be treated as independent units. They must be allocated as connected groups, and the topology of those connections directly affects which hosts can satisfy the request and which future placements are possible. GPUs are not isolated from the system GPU consumption never exists in isolation. Every GPU-accelerated workload depends on system memory and CPU resources to feed data, manage execution, and handle results. The way a workload consumes GPU resources directly shapes what it requires from the rest of the system. Static GPU memory footprint sets a baseline requirement. Model parameters must stay in GPU memory at all times, no matter the workload activity. Dynamic behavior, such as activations and key-value caches, determines how the workload scales with concurrency. These GPU characteristics do not exist in isolation. They directly dictate how the workload must be provisioned at the system level. The amount of GPU memory consumed and how it is accessed determine how much CPU is needed to feed the device, how much system memory must be reserved to support the working set, and how closely the virtual machine must align with NUMA boundaries. In other words, choosing a GPU consumption model also defines the CPU and memory shape of the virtual machine that can support it. Once that VM configuration is set, placement decisions go beyond the GPU. Host selection, CPU scheduling flexibility, memory locality, and even cluster-level behaviors like vSphere HA admission control are all affected by what first seems like a GPU-only choice. Consumption models shape everything that follows Traditionally, GPU allocation was handled near the infrastructure layer. Now, that decision is happening at higher levels. New Kubernetes features, such as Dynamic Resource Allocation (DRA), let workloads specify their hardware needs directly in their scheduling requirements. With DRA, workloads can now make structured ResourceClaims, which drivers use to pick compatible devices. While the full range of features is still developing, the main trend is clear: allocation decisions are becoming more declarative and focused on workload needs. In the future, this approach could let workloads set ordered hardware preferences (cascading ordered GPU list). Instead of just asking for one device, a workload could list preferred GPU types, memory sizes, or features. This way, a pod could share a flexible wish list, and the scheduler and driver would pick the best available device at runtime. Even as these abstractions mature, one principle remains constant. A GPU, once allocated, is still consumed as a discrete unit. What changes is not the granularity of the device itself, but the layer at which intent is declared, and binding happens. Within that limit, different GPU consumption models create different constraints for the platform. Each GPU consumption model brings different constraints to the platform. Passthrough favors predictable performance. Fractional GPUs keep flexibility once behavior is understood. Full GPUs absorb uncertainty. Multi-GPU workloads make topology a first-class requirement, pulling in more system resources for placement decisions. The architectural challenge is not picking a single GPU consumption model, but making sure different workload needs can be met at the same time. Workloads express their needs through different consumption models, while the platform enforces isolation, makes correct placement decisions, and keeps global awareness across hosts and clusters. Solving this challenge lets GPU resources be used efficiently over time without fragmenting the platform. Looking ahead The next articles in this series will move from consumption models to placement mechanics. It covers how host-level GPU policies, CPU and memory reservations, and cluster-level behavior interact with the workload shapes described here. ================================================================================ Title: Why GPU Placement Becomes the Defining Problem URL: https://frankdenneman.ai/2026-02-09-why-gpu-placement-becomes-the-defining-problem/ Date: 2026-02-09 Architecting AI Infrastructure Series - Part 1 In earlier articles, I looked at how modern AI models use GPU resources. I covered dynamic memory consumption, activation patterns, and how designs like mixture-of-experts change resource needs over time. Those pieces focused on what models require from accelerators. This new series shifts the focus. Instead of starting with the model, we will look at the platform itself. The goal of Architecting AI Infrastructure is to understand what changes when AI workloads move from ad-hoc experiments to long-running production services. At this stage, models need to be deployed, scaled, upgraded, and retired in a predictable way. GPUs are no longer tied to a single project, but become shared resources that support many teams, models, and use cases over time. This article sets the scene. It explains why GPU placement is challenging, why early AI platforms often struggle as they grow, and why solving placement issues requires a new way of thinking about architecture rather than just introducing another scheduler. From experiments to services Early AI platforms usually grow in a natural, unplanned way. A team gets GPU servers, installs Kubernetes on them, and begins experimenting. They train, fine-tune, or deploy models for inference. At this point, keeping things simple is more important than being efficient. If a GPU is idle, no one worries too much. This approach works well at first. Workloads are similar, usage patterns are easy to predict, and overall utilization is low. Direct hardware access feels efficient, and the trade-offs are manageable because the platform is small and changes do not happen often. Things change when AI shifts from being a project to becoming a service. At this point, the platform needs to handle different types of models, each with its own resource needs. It must deliver predictable performance and adapt to constant change. GPUs are no longer just for experiments; they become shared infrastructure. When this happens, placement decisions start to have hidden impacts. The hidden cost of placement decisions Every AI workload placed on a GPU makes a choice, often implicitly, about how that GPU can be used in the future. A placement decision may consume a specific amount of memory, lock a GPU into a particular partitioning mode, or occupy a topology boundary, which can impact future workload placement. The key thing about these decisions is that they usually work at first. They seem fine in the moment, but they quietly limit future choices. Over time, these choices add up, and the platform ends up in a common situation: there looks like there is enough GPU capacity, but new workloads cannot be placed. This is not mainly a problem of utilization. It is a problem of fragmentation. As time goes on, earlier placement choices reduce the number of ways the cluster can be used. GPUs are not fully used up, but they cannot be used for the new requests the platform needs to handle. So, capacity exists on paper, but not in reality. Why most schedulers struggle with GPUs This issue is not limited to one platform or orchestration system. It happens because of how most schedulers are built. Traditional schedulers work one request at a time. They check each workload separately, see if the needed resources are available, and place the workload if possible. After that, they move on to the next request. This approach works well for resources like CPU and memory, which are easy to swap. It does not work as well for GPUs. GPUs have fixed memory sizes, specific hardware layouts, interconnect domains, partitioning modes, and rules about which workloads can run together. Treating GPUs as if they are all the same might let you place a workload today, but it can quietly remove options you will need later. The system ends up focusing on short-term wins instead of long-term stability. Signals from the ecosystem As more organizations use GPUs at scale, the community has become more aware of these limits. In the Kubernetes ecosystem, projects such as HAMI and platforms like Run:AI have emerged to introduce additional coordination, smarter scheduling, and a broader view of GPU usage beyond the placement of individual workloads. The key point is not whether these methods improve utilization in certain cases, or which one is best. What matters is what their existence tells us. These methods exist because GPU placement cannot be solved just by looking at individual workloads. Any system that wants to use GPUs efficiently over time needs to see beyond the current request and understand constraints, compatibility, and future needs. Put simply, GPU scheduling is not a pod-level problem. It is a platform-level problem. GPU placement is a lifecycle problem This requires a new way of thinking. AI infrastructure is not just about placing one workload correctly. It is about placing many workloads over time and making sure the platform can handle future needs. Placement decisions should consider not only if a workload can run now, but also if running it a certain way will limit future options. In practice, this means clearly separating what a workload wants from what is possible. Workloads state their needs, and the platform decides how and where to meet those needs without harming the long-term health of the system. At this point, GPU placement is no longer just a scheduling feature. It becomes a core part of the platform’s architecture. Why virtualization changes the equation Virtualization changes how placement decisions are made and reviewed. Instead of tying workloads directly to certain devices, the system keeps a global view of GPU resources, their compatibility, their layout, and how they are used now and in the future. This approach lets the platform match what workloads want with what is possible, apply policies at different levels, and prevent fragmentation before users notice it. The benefit is not just abstraction, but careful control over how valuable and limited resources are used over time. This is why virtualization becomes increasingly valuable as AI workloads move from experimentation to production. Not because it abstracts hardware away, but because it understands hardware deeply enough to manage it deliberately. What this series will cover As this series continues, it will connect these architectural ideas to practical tools that vSphere administrators already know. We will look at how GPU usage models like passthrough, fractional vGPU, and multi-GPU setups affect placement; how host-level GPU policies and vGPU modes impact consolidation and fragmentation; and how cluster-level decisions are shaped by DRS, device groups, and hardware layout. Building AI infrastructure means treating GPU placement as a core issue from the start. It is not something to adjust later, but something to plan for deliberately. Next in the series: GPU Consumption Models as the First Architectural Choice in Production AI ================================================================================ Title: Understanding Activation Memory in Mixture of Experts Models URL: https://frankdenneman.ai/2026-02-05-understanding-activation-in-mixture-of-experts-models/ Date: 2026-02-05 In my previous article, The Dynamic World of LLM Runtime Memory, I focused on KV-cache as the primary driver of runtime memory pressure. Today, as inference workloads move toward long-context and agentic execution, activation memory has emerged as an equally important and often overlooked constraint. Long-context inference, once niche, is now expected as models handle tens of thousands of tokens in lengthy prefill phases. Agentic inference introduces variable execution, including reasoning, tool calls, pauses, and uneven token generation. These patterns put sustained pressure on both KV-cache and intermediate activations. This article moves the focus from the KV cache to the activation memory. Before diving into its behavior in Mixture of Experts (MoE) models, which route inputs through specialized experts, it’s important to examine the evolving phases of inference that highlight the importance of activation memory. Prefill versus decode To see why activation memory matters more now, it’s useful to look at how inference has changed. Inference has two main phases. The prefill phase processes the entire input: prompts, history, documents, tool outputs, and any prior reasoning. The model runs a full forward pass over the complete sequence here. All layers are active, activations are created for every token, and KV cache entries are built but not yet reused. The decode phase follows, generating tokens one by one while reusing KV cache entries. Activations are short-lived, so memory consumption stays steady and grows slowly. Recently, decode-dominated workloads prevailed: prompts were short, prefill was brief, and memory pressure increased mainly with response length. Under these conditions, KV cache dominates memory use while activation memory remains mostly transient and remains unnoticed operationally. Today, long context inputs cause models to process extended sequences in single prefill passes before producing output. In these, all layers are active across the sequence. Activation memory grows with context length, leading to peak memory use before the first output. Agentic workflows amplify this pattern. Each pause, new context injection, or execution resume can trigger another prefill phase, making activation-heavy execution a recurring, not just initial, memory cost. Mixture of Experts In dense language models like Llama 3.1, each transformer layer has a single feed-forward network. Every token passes through it, with weights used each time. As computation moves through the layer, intermediate, short-lived activations are stored until layer completion, contributing to runtime memory consumption. MoE models replace each feed-forward network within each layer with multiple independent experts. Please note that experts increase model capacity without requiring all parameters for every token. Dense models use one feed-forward network for all patterns, while MoE distributes capacity via routing across multiple experts per token. Think of the model as a building: each layer is a floor, and every floor contains a room of specialists. In gpt-oss-120b, each room holds 128 experts. As a token moves through the model, it must pass through every floor in order. At each floor, it enters the expert room and is routed to a small number of specialists. In gpt-oss-120b, four experts are selected per token at every MoE layer. Those four experts together perform the feed-forward computation for that layer before the token continues upward to the next floor. Unlike human education, there is no predefined meaning assigned to an expert. Specialization emerges during training. If an expert happens to perform slightly better on a certain class of tokens early on, the routing mechanism will tend to send more similar tokens to that expert. Over time, this creates a feedback loop in which different experts become optimized for different patterns in the data. From a memory perspective, when an expert is selected, all its weights in the layer are involved in computation, creating intermediate activations that must remain in memory until the layer completes. Because multiple experts are selected per token, each one contributes its own set of activations. The cumulative effect of having several experts active per token increases the total activation memory footprint in each layer. During long prefill phases, many tokens are processed simultaneously, each passing through all layers while multiple experts may execute in parallel. Activation memory thus grows with the number of tokens, layers, and experts per token, making MoE models show less predictable activation memory behavior than dense models. Quantifying the active payload per layer A common assumption is that MoE models must be heavier at runtime because multiple experts execute per token. When measured at the single-layer level, that assumption does not always hold. The table below compares the number of parameters that actively participate in computation for a single token in a single layer. Model LLaMA 3.1 70B (Dense) gpt-oss-120b (MoE) Model Type Dense Mixture of Expert Feed-forward structure per layer 1 dense FFN 4 selected experts Active parameters per layer ~882 million ~141 million Activation precision FP16 / BF16 FP16 / BF16 Approx. activation memory per layer ~1.7 GB ~280 MB How these numbers were derived? LLaMA 3.1 70B uses ~882M active parameters per layer, generating ~1.76 GB of activation data (at 2 bytes/param). gpt-oss-120b activates 4 experts totaling ~141M params per layer generating ~282 MB of activations. Each expert is far smaller than Llama’s dense FFN; even combined, MoE uses ~6x less memory per layer despite selecting multiple experts per token. These figures capture the dominant feed-forward activation costs (held until layer completion), excluding KV-cache or other buffers, highlighting MoE’s clear inference memory edge. Why does MoE still stress activation memory? Dense models generate a single large activation set per token per layer, predictable and concentrated, which GPUs handle efficiently with minimal allocation overhead and smooth memory usage. MoE models create multiple smaller sets per token per layer, triggered conditionally by routing decisions that vary across tokens and layers. During long prefill phases (when processing many tokens simultaneously), each token hits every layer with parallel expert activations. For an 8-token batch, this means 32 tiny memory allocations (4 experts × 8 tokens) versus dense’s clean 8 chunks, thrashing the memory allocator, creating external fragmentation and peak spikes, even though total bytes (~2.25GB) are lower than dense (~7GB). Dense stays smooth; MoE gets jagged. The shift in failure mode What ultimately changes with MoE models is not just how much memory is used, but when and how memory pressure appears. In dense models, activation memory is dominated by a small number of computationally intensive operations. Memory usage rises and falls in a relatively smooth and predictable way, and failures tend to correlate clearly with context length or batch size. In MoE models, activation memory is created in smaller pieces but repeated many times across tokens, layers, and routing decisions. Peak memory pressure emerges during prefill phases and can vary significantly depending on input composition and concurrency. The result is a shift in failure mode. Dense models typically exhaust memory gradually and predictably. MoE models are more prone to sudden activation spikes that occur before the first token is generated, even when average memory usage appears safe, making it more difficult to monitor their resource consumption trends. The practical takeaway MoE models are not necessarily more computationally expensive than dense models, but they are more sensitive to context length, batching, and concurrency. Activation memory becomes harder to predict, even when the number of active parameters per layer is lower than in a dense model. The key shift is that inference stability is no longer governed by steady-state behavior. It is governed by short-lived activation peaks during prefill and agent resumes. Infrastructure must therefore be sized and operated for these peaks, not for average utilization. This changes how hardware is selected, how workloads are placed, and how concurrency is controlled. For on-prem enterprise platforms, this favors conservative, locality-focused designs during activation-heavy phases. Prefill and agent resumes create short, intense activation spikes that are both memory-bound and latency-sensitive. Keeping these phases within a single host or an NVLink or NVSwitch domain avoids network hops when memory pressure is highest. Expert routing is also frequent and fine-grained. Every layer and token involves routing decisions, and crossing hosts turns these into synchronization points, increasing tail latency and reducing operational stability. This does not eliminate scale-out. Replication remains an effective way to increase capacity and isolation. However, distributing activation-heavy execution across hosts provides little benefit and increases fragility. For enterprise deployments, predictability matters more than peak throughput. Shared environments require additional care. Dense models typically increase memory usage gradually, making overcommitment and time-slicing relatively safe when well tuned. MoE workloads exhibit sharper and less predictable activation spikes, especially during long prefill phases. This reduces the margin for error in multi-tenant deployments. GPU virtualization strategies must account for worst-case activation behavior, not just averages, and overcommitment techniques that work for dense models can lead to abrupt failures when applied to MoE workloads. The core trade-off is simple. Dense models reward aggressive batching and tight packing. MoE and agentic models reward headroom, locality, and control. The infrastructure that succeeds in this next phase is not the one that maximizes average utilization, but the one that remains stable under bursty, activation-heavy execution. ================================================================================ Title: The Dynamic World of LLM Runtime Memory URL: https://frankdenneman.ai/2026-01-12-the-dynamic-world-of-llm-runtime-memory/ Date: 2026-01-12 When meeting with customers and architectural teams, we often perform a specific exercise to separate a model’s static consumption (its weights) from its dynamic runtime consumption. In the unpredictable world of production AI, where concurrent users, complex system prompts, and varying RAG content create constant flux, it is easy to view memory as an elusive target. This article is designed to move your service level from probabilistic to deterministic concurrency. To make this accessible to those managing the hardware, I have intentionally used language common to system administrators rather than data scientists. Instead of focusing on the mathematical constructs of vectors and matrices, we will use the term representations to highlight the actual memory consumption of these data structures. While these calculations are not a mandatory step for every deployment, they provide the essential north star for architects who are curious or need to establish a deterministic floor for their capacity planning. Dynamic Memory Consumption of User Prompts Or better said: KV cache memory calculation. Let’s use Llama 3.1 70B in FP16 with a maximum 32K token prompt as an example. How many bytes per token does it consume? The calculation is: bytes/token = 2 (Key+Value) x Layers x kv_heads x head_dim x 2 (FP16 bytes) = 2 x 80 x 8 x 128 x 2 = 327,680 = 0.33MB / token x 131072 = 42.95GB * * All memory sizes in this document are expressed in decimal gigabytes (GB), consistent with GPU vendor specifications. These specifications of the model can be found in its config.json file. Below is a screenshot of Llama 3.1 70B config.json: Let’s look closely at each step of this calculation. 2 (Key+Value) **What is stored per token? **For each token, the key and value vectors are stored in the KV cache. What is a key and value in the context of an LLM? When a model reads a prompt, it needs a short-term working memory of the prompt and a mechanism to decide what parts of that memory matter for the next token. The attention mechanism of the LLM answers the question: ‘Given what I’m trying to generate now, which parts of the prompt should I look at, and how strongly should each of those tokens influence the answer?’ The important thing to understand about attention[b][c] is that it’s about understanding the question and identifying relevant information from the LLM in the prompt to answer it. It’s not a database query looking for an answer in the model weights. Tokenization As a result, the prompt’s words are first tokenized. LLMs do not operate on words directly; they operate on tokens, which are then converted into numerical vectors. Tokens offer a finite, reusable set of symbols. Since not every possible word appears during training, the model must be able to compose meaning from smaller pieces. Common words are often represented as a single token, while rarer words may be split into multiple tokens. Tokens can represent whole words, parts of words, and punctuation. If we use the prompt ‘What is the capital of Texas?’ may be tokenized in the following way: [What] [is] [the] [capital] [of] [Texas] [?] From tokens to token representations At this point, the model is no longer operating on words, but on token representations (vectors). A token representation can be seen as a fixed-size data structure that summarizes a token’s meaning and role in the sentence. (For example, in a prompt about Apple it is distinguishing Apple the tech company from apple the fruit). Once the prompt has been tokenized and converted into token representations, the model begins processing the prompt step by step. Key and value information per token Token Key information Value information [What] Question indicator Signals that an answer is required [capital] Capital-city relationship Information about capitals [Texas] U.S. state entity Information about Texas [Is] / [the] / [of] Grammatical structure Minimal semantic contribution As the model processes the prompt, attention is used to determine which tokens in the prompt are relevant to the current step, and how strongly each should influence the outcome. Queries, keys, and values in action To do this, the model generates a query representation. The query does not describe the task in words; it is simply a signal that reflects what the model is currently working on. Each token in the prompt has associated key information that describes what the token represents. The model uses the query to determine which tokens are relevant at the moment. Tokens that are more relevant have a stronger influence on the next step, while others contribute very little. In our prompt example, tokens such as [Texas] and [capital] are more relevant than grammatical tokens like [is], [the], or [of] because they carry the information needed to answer the question. Why do queries not appear in the memory calculation? Although query representations play an important role in determining the relevance of each tokenin the prompt, they do not contribute to KV cache memory consumption. Query representations are generated on the fly, used once, and discarded immediately. From a system perspective, query representations reside only in temporary GPU memory, such as registers or short-lived activation buffers. Their memory footprint is small and short-lived, and does not grow with prompt length.[f][g] Keys and values, on the other hand, are generated once per token, must remain available for all future steps, and are therefore stored persistently in GPU memory as a part of KV Cache. This is why the memory calculation includes keys and values, but not queries. Layers, KV heads, and Grouped-Query Attention So far, we’ve explained why keys and values dominate memory. The next question is: how many keys and values do we store per token? That is where layers, KV heads, and Grouped-Query Attention come in Layers multiple memory consumption A transformer model is built from many stacked layers. Each layer performs its own attention operation, so each layer needs its own key and value for every token. This means that KV cache memory scales linearly with the number of layers. For example, in LLama 3.1 70B, the architecture has 80 layers, so each token stores 80 key-value pairs. How many Key/value pairs per layer? Within each layer, attention is divided into multiple heads. Each head represents a separate way of interpreting the token information. Keys and values are stored per head, so the KV cache memory also scales with the number of KV heads. Grouped-Query Attention In traditional multi-head attention, every query head has its own key and value head. Llama 3.1 uses Grouped-Query Attention (GQA), where many query heads share a smaller number of key and value heads. Llama 3.1 70B has 64 query heads, but only 8 KV heads. This reduces the KV cache memory by a factor of 8x compared to traditional multi-head attention. Head_dim determines how big each key/value is Each KV head stores a fixed-size block of numerical data for every token. The size of the block is referred to as the head dimension. The head dimension is determined by how the model’s internal representation of a token is split across attention heads. To understand this, it helps to introduce the concept of hidden size. The term hidden size comes from earlier neural network designs, in which models were described by input layers, output layers, and internal (or hidden) layers. In this context, hidden simply means internal to the model, i.e., not directly visible to the user. In modern transformer models, hidden size refers to the total size of the model’s internal representation of a token. In practical terms, it defines how much numerical information the model maintains for each token as it processes a prompt. Llama 3.1 has a hidden size of 8192, which means that for every token, the model maintains 8192 numerical values as its internal valuation. During attention, this internal representation is not used as a single block. Instead, it is divided evenly among the model’s attention heads, allowing it to process different aspects of the token’s information in parallel. This design also explains why GPUs are so effective for the transformer models. Once the internal representation is split across attention heads, each head can be processed largely independently. GPUs are optimized for executing many similar operations in parallel across independent data blocks. Attention heads naturally fit this execution model, allowing the GPU to process multiple heads simultaneously with high throughput. From a system perspective, this means that increasing the number of heads increases parallel work, not sequential complexity, a pattern that maps well to GPU architectures. Because attention heads operate independently, they form natural units of parallel execution. As models scale beyond a single GPU, the number and organization of heads directly influence how work can be distributed across multiple GPUs. A topic later discussed in another article. The size of the portion assigned to each head is the head dimension, which is calculated by dividing the hidden size by the number of attention heads. For Llama 3.1 70B, the hidden size is 8192, it has 64 attention heads. This means that the internal representation of each token is split into 64 equal parts: 8192 / 64 = 128. As a result, each KV head stores 128 numerical values per token, per layer, for keys and for values. Why does the calculation use KV Heads with GQA involved So far, we have explained how keys and values are stored per token, per layer, and per head. The remaining question is why the calculation uses the number of KV heads, rather than the total number of attention heads. In GQA, Llama 3.1 70B, the multiple query heads share the same key-value data. The 64 query heads independently determine relevance, but those queries are mapped onto only 8 shared sets of keys and values. Because the key and value data is shared in this way, only 8 KV heads need to be stored per token, which is why the KV cache calculation uses ‘Kv_heads = 8’ rather than the total number of attention heads. Tying the architecture back to the formula With these components in place, we can now return to the KV cache memory calculation and see how each part of the formula directly maps to the model architecture. For each token, the model stores key and value data: Across 80 layers, because each layer maintains its own attention state Across 8 KV heads, because GQA maps 64 query heads onto 8 shared sets of key and value data With 128 numerical values per head, determined by how the model’s internal representation is split across attention heads At 2 bytes per value, because keys and values are stored in FP16 This leads directly to the per-token KV cache size: 2 (Key + Value) x 80 (Layers) x 8 (KV heads) x 128 (head dimension) x 2 bytes = 327,680 bytes x 131,072 tokens = 42.95GB KV cache memory, excluding the model’s weights and temporary activations. The concurrency trade-off While the math gives us a clear number, such as the 42.95GB required for a full 128K context on this model, its true value lies in its power to transform how we plan services. In production, we aren’t just deploying a model; we are deploying a memory budget. The fundamental architectural goal of this exercise is to move a service level from probabilistic to deterministic concurrency. Establishing the baseline floor In a live environment, prompts are unpredictable. One user might send a 500-token prompt, while another triggers a RAG retrieval that generates 15.000 tokens. If an architect plans only for average usage, the system remains in a probabilistic state, in which performance guarantees are impossible. By calculating the memory required for the desired maximum context length, we establish a deterministic floor on the number of concurrent users. This is the guaranteed number of sessions the GPU/model can support simultaneously, even in the worst-case scenario in which every user hits their maximum limit at once. This floor becomes the baseline for future monitoring. By comparing this deterministic baseline against real-world telemetry, teams can see exactly how many additional ‘practical’ concurrent users a GPU actually serves, given that the average prompt is shorter than the maximum. The data scientist lever This is why data scientists and architects often actively test and reduce the maximum content length. By constraining the context window, they can significantly increase the total number of users a single system can serve simultaneously. Reducing the maximum context length from 128K to 8K essentially reduces the ‘reserved’ memory per user. This immediately increases the deterministic number of users we can guarantee. Reducing the maximum context length allows the business to decide the trade-off. Do we need a few users with massive context windows (i.e., is that required by the workload?), or is it more valuable to have a higher deterministic floor for many users with shorter (8K) context windows? By grounding the deployment in these calculations,, we move away from reacting to memory issues/ lower response times and towards a strategy of guaranteed capacity planning. The infrastructure parallel For vAdmins, this mirrors the strategy of maintaining a 1-to-1 mapping of physical to virtual system memory during normal operations to eliminate overcommitment risks. However, we still enjoy the benefits of over-consolidation during critical infrastructure events, such as maintenance mode or HA failover events. Because real-world prompts are probabilistic and often stay well below the maximum token limit, the deterministic floor we’ve calculated can actually serve as a massive safety buffer. It allows you to safely over-consolidate user sessions onto fewer GPUs during a failover, relying on average usage patterns to keep the system standing while the physical infrastructure is being serviced. It is important to acknowledge that maintaining this safety buffer comes at a high cost. Because GPUs are extremely high-cost assets, planning solely for deterministic ‘worst-case’ can lead to under-utilized hardware and inflated budgets. This is why the dialogue between infrastructure architects and data science teams is so critical. Together, the teams must determine the right cost factor, finding a middle ground where the context window is large enough to be useful, but constrained enough to ensure service resilience and hardware efficiency. By aligning the context window with actual organizational needs, we ensure that every byte of RAM drives value rather than just sits in reserve. Quantization Impact Quantizing a model (for example, to FP8) primarily affects model weight and compute precision. Keys and Values stored in the KV cache are still FP16 by default, unless the inference stack explicitly supports and enables FP8 KV cache. In other words, the KV cache precision is a runtime choice, not a model property. From a memory perspective, this means that quantizing weights alone does not reduce KV cache memory consumption Temporary activation memory In addition to the KV cache, inference also requires temporary activation memory. These activations hold intermediate results while the model processes tokens, such as query representations and intermediate attention outputs. Unlike the KV cache, activation memory is short-lived. It is allocated during computation and released immediately after use. As a result, activation memory does not grow with context length. Activation memory is most significant during the prefill phase, when the entire prompt is processed at once. During token-by-token decoding (decode phase), activation memory usage is relatively small and stable. Applying the formula to your model of interest Once the calculation is understood for this specific model, applying the same approach to your model of interest is relatively straightforward. All modern transformer models expose the required parameters in their model configuration, typically through a config.json file. Please only refer to that file or an equivalent runtime object, as it is the single source of truth. Many online GPT services easily hallucinate. To determine the KV cache memory footprint, we need to identify the following: What we need Common config field names Layers Num_hidden_layers, n_layers Hidden size Hidden_size, d_model Attention heads Num_attention_heads, n_heads KV heads num_key_value_heads GQA Num_key_value_heads < num_attention_heads KV precision Runtime / inference setting Maximum context length Max_position_embeddings If a model uses GQA, there is typically a separate key/value head count listed in the file. The head dimension can then be derived by dividing the hidden size by the number of attention heads. If a model does not explicitly define a separate number of key/value heads, it can be assumed that each attention head has its own keys and values, resulting in higher KV cache memory usage. Conclusion: Balancing Resilience and ROI Ultimately, this sizing exercise is about transforming the way we plan and monitor services. By calculating your “worst-case” KV cache, we effectively establish a 1-to-1 mapping of physical to virtual memory during normal operations. This ensures that our ‘steady-state’ performance is guaranteed and predictable. However, because GPUs are extremely high-cost assets, this safety buffer must be balanced against hardware efficiency. Planning solely for the absolute “worst-case” can lead to under-utilized hardware and inflated budgets. This is why the dialogue between infrastructure architects and data science teams is so critical. Together, you must determine the right cost factor, finding a middle ground where the context window is large enough to be useful but constrained enough to ensure that every byte of VRAM drives actual organizational value. ================================================================================ Title: Talking VCF 9 and Private AI Foundation on the Unexplored Territory Podcast URL: https://frankdenneman.ai/2025-09-02-talking-vcf-9-and-private-ai-foundation-on-the-unexplored-territory-podcast/ Date: 2025-09-02 Just before VMware Explore, I joined the Unexplored Territory Podcast to talk about the enhancements in VMware Cloud Foundation 9 and the Private AI Foundation with NVIDIA. We covered new functionality, such as Agent Builder, and walked through the broader enhancements for AI workloads. We also highlighted a few must-attend sessions at Explore. You can listen to the full episode here: Apple Podcasts Spotify During Explore, many people told me this episode was a great starting point to wrap their heads around VMware Private AI Foundation. If you’re looking for a concise way to catch up, this is a good place to begin. If you prefer shorter clips, here are a few highlights: Because Great AI Starts With Great Infrastructure! https://www.youtube.com/embed/rh4JFicg8vY What is Dynamic DirectPath I/O and why is it useful for AI workloads? https://www.youtube.com/embed/1DBoMKDK1FU What are Deep Learning VM templates and how do they help developers? https://www.youtube.com/embed/OFNW4aB8cmY What about HA and DRS when it comes to GPUs and AI workloads? https://www.youtube.com/embed/u7QkRoPx45k If you’ve had a chance to listen or watch, I’d love to hear what stood out most about the changes in VCF 9 or the Private AI Foundation. ================================================================================ Title: Which Multi-GPU Configurations Are You Planning to Deploy? URL: https://frankdenneman.ai/2025-09-02-which-multi-gpu-configurations-are-you-planning-to-deploy/ Date: 2025-09-02 During VMware Explore, numerous conversations highlighted that most customers plan to deploy systems with two or more GPUs. The next challenge is deciding which type of multi-GPU configuration to adopt — a choice that depends on intra-node communication, inter-node interconnects, and cooling strategies. To better understand where organizations are heading, I’ve created a short survey. The diagram below illustrates the options available in the NVIDIA-certified systems portfolio, which I use as a reference point in the questions. Your feedback will help map out how different configurations are being considered and provide valuable input as we align our product strategy with customer needs. ** How to Read the Diagram** The diagram is intended to illustrate the spectrum of multi-GPU configurations, ranging from PCIe-based systems to NVLink bridge topologies and NVSwitch-enabled SXM platforms. To make this more tangible, I’ve used Dell’s AI server portfolio as an example, allowing you to check out the exact systems on Dell’s website to view their specifications. As an additional detail, the boxes are color-coded: White boxes represent air-cooled servers Blue boxes represent liquid-cooled servers PCIe Connected GPUs PCIe-connected systems support 1–16 GPUs as standard add-in cards. Communication between GPUs relies on PCIe lanes or, in some cases, NVLink bridges. Some NVIDIA datacenter GPUs, such as the L4 and the highly anticipated RTX PRO 6000, do not offer NVLink capabilities. H100 and H200 GPUs can be deployed without an NVLink Bridge, resulting in a theoretical GPU-to-GPU bandwidth of between 64 GB/s and 128 GB/s, depending on the PCIe specification of the device. NVLink An NVLink bridge is a direct, high-bandwidth connection between GPUs, designed to bypass the limitations of PCIe and enable GPUs to share memory and exchange data at significantly higher speeds. Unlike PCIe, which is a general-purpose bus, NVLink is purpose-built for GPU-to-GPU communication. 2-way NVLink Bridge (H100 PCIe): In a 2-way setup, two GPUs are directly linked, creating a fast point-to-point connection. On the NVIDIA H100 PCIe, each GPU can communicate with its peer at up to 600 GB/s of bandwidth (NVLink 4.0), compared to just 128 GB/s over PCIe Gen5. This is typically used in dual-GPU or paired configurations. 4-way NVLink Bridge (H200 PCIe): In a 4-way setup, four GPUs are interconnected in a mesh topology. Each GPU can communicate with multiple peers simultaneously, thereby improving bandwidth across the group. On the NVIDIA H200 PCIe, each GPU supports up to 900 GB/s of GPU-to-GPU bandwidth (NVLink 5.0). This enables stronger scaling for 4-GPU systems, though it doesn’t provide the full all-to-all fabric that NVSwitch delivers. NVSwitch with SXM GPUs SXM is NVIDIA’s server GPU module format; instead of a PCIe add-in card, the GPU is mounted directly onto the motherboard using an SXM socket. This allows for higher power delivery, denser designs, and direct integration with high-bandwidth GPU interconnects. SXM GPUs can be deployed in two main HGX configurations: 4-GPU HGX (NVLink): In 4-GPU systems, SXM GPUs are interconnected using NVLink without an NVSwitch fabric. On the H100 SXM, each GPU provides up to 900 GB/s of GPU-to-GPU bandwidth (NVLink 4.0), enabling strong scaling within the node while avoiding PCIe bottlenecks. 8-GPU HGX (NVSwitch): In larger systems with eight SXM GPUs, NVSwitch components are integrated on the motherboard to create a fully connected all-to-all fabric. Every GPU can communicate with every other GPU at NVLink speeds, eliminating peer-to-peer bottlenecks. On both the H100 SXM and H200 SXM, each GPU supports up to 900 GB/s of GPU-to-GPU bandwidth when connected through NVSwitch. The B200 HGX server supports 1.8 TB/s of GPU-to-GPU bandwidth. This design enables GPUs to operate effectively as a single, unified pool of compute and memory. Call to Action I’d like to ask you to take a few minutes to answer the six questions in this survey. The goal is to gain a better understanding of how organizations plan their future GPU server configurations — from retrofitting versus new systems, to GPU scale, interconnects, and cooling. Your input will help me build a clearer picture of the distribution of server choices across the industry. I’ll use this insight to take back into our discussions and help align VMware’s product strategy with the directions customers are heading. Loading… ================================================================================ Title: Enhanced vMotion for vGPU VMs in VCF 9.0 URL: https://frankdenneman.ai/2025-06-24-enhanced-vmotion-for-vgpu-vms-in-vmware-cloud-foundation-9-0/ Date: 2025-06-24 VMware’s latest release of Cloud Foundation 9.0 introduces an important new feature for managing AI infrastructure: Enhanced vMotion for vGPU VMs. This new feature substantially improves the management of large language models (LLMs) in virtualized environments. For an in-depth technical overview, please read Justin Murray’s detailed article on the subject. The Power of vMotion Traditionally, vMotion has been a cornerstone of VMware’s value proposition, enabling two critical benefits: Infrastructure maintenance without workload disruption Maintenance without coordination with workload owners These capabilities have allowed vAdmins to perform updates and maintenance with minimal impact on running services, a crucial advantage in today’s always-on digital landscape. The Reality of Maintenance However, the process of bringing down a model for maintenance is more complex and time-consuming than often assumed. Let’s consider the steps involved in taking down and bringing back a 70B parameter model in a three-replica deployment: Initiate graceful shutdown of one replica (30-60 seconds) Redirect traffic to remaining replicas (immediate) Wait for in-flight requests to complete (variable) Unload model from GPU memory (10-20 seconds) Perform maintenance or updates (variable) Reload 70B model into GPU (2-5 minutes) Warm up the model (1-3 minutes) Re-enable the replica for traffic (30-60 seconds) Rebalance load across all replicas (immediate) This process could take 4-10 minutes, excluding maintenance time coordination, potentially impacting service levels and requiring careful coordination between the infrastructure and AI teams. Enhanced vMotion for vGPU VMs: A New Approach The new Enhanced vMotion feature in VCF 9.0 addresses these challenges by dramatically reducing the impact on LLM services. For a 70B model with a massive KV cache, the service interruption is reduced to approximately 21 seconds. This improvement is achieved through intelligent handling of different types of GPU memory: Static Memory (Model Weights): vMotion identifies the immutable collection of weights that form the core of the LLM. These are pre-copied to the destination machine while keeping the LLM active and available for the application frontend. Dynamic Memory (KV-Cache and Activations): The memory used for processing prompts and generating answers is continuously changing. vMotion manages this by briefly stunning the VM to control the GPU I/O flow of these dynamic bits. The stun time depends on the amount of dynamic memory on the GPU and the bandwidth between the source and destination hosts. Crucially, this process preserves the KV cache intact, minimizing the impact on service-level behavior immediately after maintenance. Key Benefits This advancement in vMotion technology offers several significant benefits for AI workload management: Minimal service disruption during maintenance Preserved model state, including the valuable KV cache Improved operational efficiency for infrastructure maintenance Reduced the need for coordination between infrastructure and AI teams Consistent service level behavior post-maintenance Conclusion Enhanced vMotion for vGPU VMs in VMware Cloud Foundation 9.0 represents a significant step forward in bridging the gap between the flexibility of virtualization and the demanding requirements of AI workloads. By addressing the specific needs of LLMs while leveraging the strengths of virtualization, it’s setting a new standard for managing AI deployments in enterprise environments. As organizations continue to integrate AI into their core operations, solutions like this will be crucial in maintaining the delicate balance between operational efficiency and service reliability. The impact of this technology on AI operations in various organizations will be an interesting development to watch in the coming months. ================================================================================ Title: Building an Efficient AI Ingestion Pipeline: Data Ingestion Strategies URL: https://frankdenneman.ai/2024-10-30-building-an-efficient-ai-ingestion-pipeline-data-ingestion-strategies/ Date: 2024-10-30 Traditionally, deploying applications is a straightforward process that moves from development to production. For instance, enterprise apps usually work with databases to perform standard tasks, which makes resource management and maintenance predictable. Generative AI (Gen-AI) applications, however, are more flexible and complex. They need to adapt quickly, since they work with constantly changing data and must handle a wide range of demands. Gen-AI apps, especially those using Large Language Models (LLMs) and Retrieval Augmented Generation (RAG), don’t follow the same linear path as traditional workloads. Instead, they move through a circular, adaptive lifecycle with two main stages: research and production. Understanding these two phases is crucial for infrastructure administrators who manage and scale AI-driven applications. This requires an approach that supports continuous change and adaptation rather than a single, linear deployment. Research phase The research phase in Gen-AI workloads is like a continuous lab experiment. Here, data scientists keep testing models, adjusting algorithms, and trying out different setups to find the best balance of accuracy, efficiency, and flexibility. This phase moves quickly and values flexibility over stability. For example, a data scientist might change the chunk size or try new indexing methods in a vector database to speed up retrieval. Often, it takes several rounds of testing to see what works best for each dataset. To work flexibly, data scientists need to manage their own resources. They should have easy access to set up things like Deep Learning VMs, AI Kubernetes clusters, and Vector Databases. When they can do this themselves, they can quickly adjust to new discoveries and keep experimenting without waiting for help from infrastructure teams. Production phase The production phase is about getting the Gen-AI application ready for real-world use. The main goals here are scalability, reliability, and steady performance, so the system can meet user needs. At this stage, the infrastructure must support stable and efficient operations as the app moves from testing to serving many users. In this phase, the lessons learned during research help build production-ready systems. While research focuses on flexibility, production is all about reliability and predictability. Workloads are set up for speed and stability, so the app can handle lots of user queries, manage new data smoothly, and give accurate answers every time. Containerized platforms are key in this phase because they let you package the app in a consistent environment that’s easy to scale. Platforms like Kubernetes allow for horizontal scaling, either automatically or with help from DevOps teams, so workloads can grow as demand increases. For example, if user traffic spikes, new containers can be started to handle the extra load and keep the user experience smooth. Data ingestion strategies Data ingestion is a key part of Gen-AI workflows, providing the RAG system with the data it needs to generate useful responses. Depending on whether you’re in the research or production phase, different ingestion strategies are used. Here are the three main types: Automated batch ingestion is good for handling large amounts of data on a set schedule, without needing manual work. It’s ideal for processing lots of information during off-peak hours, so updates to the knowledge base are regular and predictable. For example, a company with a big, unchanging dataset like product catalogs or research papers can use batch ingestion to update its system every night. This method helps infrastructure admins plan CPU, memory, and storage use, especially during scheduled low-traffic times. Streaming ingestion works nonstop, bringing in real-time changes so the system always has the latest information. This is especially useful for time-sensitive apps where it’s important to keep delays low. However, streaming can lead to unpredictable resource needs. ML Ops teams need to watch and manage resources closely to avoid slowdowns, especially when there’s a lot of data coming in. Using cluster services like DRS and choosing the right VM types helps keep resources available. Manual ad-hoc ingestion lets you process specific data on demand, giving you a lot of control and flexibility. It’s useful when you need to add important information right away, like new legal rules or emergency alerts. While this method gives precise control, it doesn’t scale well for large or frequent updates and needs human input. Infrastructure admins should know that ad hoc ingestion can cause sudden spikes in resource use, which might affect other processes if the system isn’t flexible enough. Production data ingestion strategies Automated batch and streaming ingestion are mainly used in production, where a steady and reliable flow of data is needed to keep the model current and accurate. When looking at how ingestion affects infrastructure and resource use, the data window and peak throughput are key. They determine how well resources are used and how fresh the data stays. Data Window The data window is the period when data is collected, processed, and brought into the system. Having a clear data window helps manage batch jobs efficiently. How long this window is can have a big impact on infrastructure needs: Short Data Window: With a short data window, ingestion jobs happen more often, so smaller batches are processed each time. This lowers latency and allows for near-real-time updates, but it also means more frequent jobs to manage. The infrastructure must be flexible enough to handle bursts of activity, making sure compute, memory, and network resources are available when needed. Long Data Window: A long data window means bigger batches are processed less often, which cuts down on the overhead of frequent jobs. But processing more data at once can put a lot of pressure on storage, compute, and I/O resources during those times. The infrastructure needs to handle these peak demands efficiently. Another important factor is the data volume multiplication that happens during ingestion, as data is extracted, transformed, and loaded into the vector database. The intermediate data is usually kept in memory or storage. When working with large datasets and long data windows, it’s important to clean up properly. The next article explains data volume multiplication in more detail. Peak Throughput Peak throughput is the highest amount of data the system needs to handle during ingestion. Knowing and managing this is key to keeping the system running smoothly and avoiding slowdowns: Compute Resources: During peak times, CPU use can be high but often goes up and down, mostly because of how data is transferred and processed. As data moves from different sources into storage or memory, CPU load can spike, depending on the tools used for ingestion and transformation. These tools might not be fully optimized, so CPU use can vary a lot. For example, a performance graph might show that even if a machine has 16 cores, only 4 are used at a time, even when the process is set to use all 16. There are different ways to handle batch ingestion, and each affects CPU and resource use differently. One method is to collect all new data from every source before moving to the next step. This approach gathers data in a set time frame, causing high CPU and I/O activity at first, then moves on to processing. This way, the GPU-based embedding model can use parallel processing fully. Resource use spikes during each phase, but it stays manageable. Another way is to process each data source one at a time, finishing the whole ingestion pipeline for one dataset before starting the next. This serial method means each source is fully ingested, transformed, and stored before moving on. It often leads to short bursts of heavy resource use, followed by quieter periods. CPU and GPU use can change a lot during this, depending on the data and the tools. In both cases, resource use is spiky, with short bursts of high CPU and memory use followed by idle times. How long these idle periods last depends on the method, data size, and processing speed. For infrastructure admins, knowing about these spikes is important for planning resources, whether by reserving CPU for peak times or relying on DRS and cluster idleness to adjust as needed. To support these workflows well, infrastructure planning should consider the uneven peaks in CPU and memory use during ingestion. This means providing enough burst capacity, timing workloads to avoid clashing with other important tasks, and thinking about using dynamic scaling or resource quotas to reduce waste while staying flexible. By noticing the uneven, high-demand resource patterns during the data window, infrastructure admins can make better decisions to keep the data pipeline running smoothly. Storage I/O: Peak throughput affects storage needs, especially I/O operations per second (IOPS). High-throughput ingestion creates lots of read and write actions, especially when embeddings are made and saved in real time. To prevent I/O slowdowns, it’s important to have high IOPS, like with an all-flash storage platform that can handle many users at once. Network Bandwidth: Ingestion often means moving data between storage, processing nodes, and sometimes outside sources. High peak throughput puts pressure on network bandwidth, especially if lots of data comes from different places. Making sure the network has enough capacity, or timing ingestion to avoid busy periods, helps prevent slowdowns and keeps performance steady. Memory Utilization: Memory usage can spike during peak ingestion, particularly if data transformations, such as chunking or embedding generation, require large amounts of CPU and GPU memory. The system needs enough RAM to handle this without causing too much swapping or memory thrashing, which would hurt performance. Data freshness means how fast updates from data sources show up in the system, which affects how relevant and accurate Gen-AI apps are in production. When up-to-date info is crucial, like in customer support, the delay between updates and ingestion should be as short as possible. Frequent updates may need shorter ingestion times, while less important updates can wait for off-peak hours to ease the load. The timing of data ingestion should also fit the work patterns of global teams. Planning ingestion for after-office hours, like updating Confluence pages at night, means the Gen-AI has all new data by the next workday. But in global companies, work happens in all time zones, so there’s no single quiet period. To solve this, you can stagger ingestion times based on regional work hours or use micro-batching—running small, frequent ingestions all day—to spread out the load and keep data fresh. It’s important to manage infrastructure load carefully, making sure there’s enough capacity for ingestion without affecting other services. Scheduling ingestion during low-usage times, when computing and storage needs are lowest, helps avoid resource conflicts and keeps things running smoothly. In a global setup, using cloud infrastructure to ingest data closer to its source can also boost throughput and cut down on delays. It’s also important to think about how often different data sources change and how important they are. Internal documents might need frequent updates, while external datasets or archives may only need occasional ingestion. Setting ingestion schedules based on how often each source updates helps keep things efficient and fresh. Prioritizing key content, like customer-facing docs, makes sure the most important info is always current. In the end, successful data ingestion for an RAG pipeline means balancing data freshness, infrastructure efficiency, and user needs. Scheduling should be flexible for global use and able to adapt as things change. By keeping an eye on activity and adjusting batch schedules as needed, infrastructure teams can make sure data is ingested efficiently and the knowledge base stays current in a fast-changing environment. Research data ingestion strategies During the research phase, data ingestion is more flexible and experimental, often using manual ad hoc methods. Data scientists try different sources, formats, and methods for bringing in data to see how they affect model performance. However, some elements of automated batch processing can be mimicked in the research phase by using scripts on (Jupyter) notebooks to perform bulk data loads on a trial basis. These ad-hoc batch loads provide insights into resource needs, data processing times, and scalability considerations, helping data scientists refine ingestion methods before implementing them in production. In this way, the research phase serves as a proving ground, where different ingestion strategies are tested and optimized for optimal performance when transitioned to production. Conclusion In summary, managing resources and peak throughput during the data window means understanding that resource demands can be spiky and unpredictable. Infrastructure admins should use smart scheduling, dynamic scaling, and effective capacity planning to address these challenges effectively. ================================================================================ Title: How to Build an Efficient AI Ingestion Pipeline URL: https://frankdenneman.ai/2024-10-25-how-to-build-an-efficient-ai-ingestion-pipeline/ Date: 2024-10-25 On-premises AI deployments are becoming increasingly important, but infrastructure administrators and architects often face a steep learning curve due to unfamiliar terminology. While much AI information is tailored to data scientists, there’s a growing need for resources to clarify how these workloads impact infrastructure. Understanding the RAG Ingestion Pipeline for AI Workloads When planning for AI, you’ll often hear terms like “embedding models,” “vector embeddings,” and “vector databases,” especially in Retrieval-Augmented Generation (RAG) frameworks that support scalable, responsive AI. But what actually happens when you run a RAG pipeline? How much computing power do you need? How much I/O is involved when processing a 60 MB dataset? These questions, along with key scaling factors, are important for sizing your infrastructure and planning for changes. For infrastructure teams, understanding key AI concepts and their real-world impact is essential to supporting on-premises AI workloads. The RAG process has two main parts: the ingestion pipeline, which prepares data for retrieval, and the retrieval pipeline, which handles real-time queries and responses. This is the first article in a series that explains each part of the RAG workflow. Here, we’ll focus on the ingestion pipeline, what it does, its main parts, and how they work together. Later articles will look at resource needs in more detail and include a Jupyter notebook so infrastructure teams can test system impacts themselves. Understanding the Ingestion Pipeline for AI Workloads The ingestion pipeline prepares data so large language models (LLMs) can use it efficiently in AI applications. Each step turns raw data into a format AI models can understand, which helps them respond faster and more accurately to user questions. Let’s look at each stage, from loading the data to storing it in a vector database. Data Ingestion: Loading & Extracting Data LLMs need data in a format they can understand, but they can’t read raw documents, text files, PDFs, or other file types directly. Instead, an extra step turns the data into “embeddings”—numerical representations that capture the meaning and context of the text. An embedding model does this by converting the prepared text into structured vectors. To get data ready for the embedding model, you first load, extract, and clean it. This means pulling data from places like document folders, databases, or online sources, then putting it into a standard, machine-readable format. Only after preprocessing can you send the data to the embedding model, which creates the vectors LLMs use to retrieve and understand information. Why this matters for infrastructure: Preparing data for embedding uses a lot of computing power and storage. Infrastructure administrators need to make sure there are enough resources to handle each step efficiently. Extracting and cleaning large files takes CPU and memory, and organizing data in a machine-readable format needs fast, easy-to-access storage. The data extraction tools are typically written in Python with little optimization for resource management. In reality, that means these libraries and tools run as single-threaded processes by default. The data scientist must be aware of this and how to augment the code to enable multi-processing or multi-threading. The difference between those two techniques is a topic we will save for later. As a result, loading copious amounts of data single-threaded will take a long time, and a parallel processing ingestion process will consume all the CPUs for a short but high-intense duration. Data transfer rates and I/O performance are key when loading and processing files. High I/O throughput helps prevent bottlenecks, especially with large datasets. Depending on the data’s size and complexity, this step can use a lot of memory and temporary storage, since intermediate and cleaned data are stored before embedding. There are three main ways to ingest data: automated batch ingestion, streaming ingestion, and manual attachment. Each method has its own purpose, benefits, and resource needs, depending on the situation. We’ll cover these strategies in another article. Data Preparation: Preprocessing and Metadata Handling Preprocessing makes sure the data is clean, standardized, and free of unnecessary or sensitive information. This step includes several processes that help make the data consistent and ready for embedding. Addressing regional data formats: Dates, times, currency, and measurement units can vary by region. Removing personally identifiable information (PII): Protects privacy and ensures compliance with regulations such as GDPR and HIPAA. Correcting spelling mistakes: Improves embedding accuracy, especially for domain-specific terms. Metadata handling: Adds context such as source, date, and topic to improve retrieval accuracy. Why this matters for infrastructure: Preprocessing can be CPU- and memory-intensive, especially for large datasets or complex transformations. Repeated read/write operations also increase I/O demands. Data Transformation: Chunking and Embedding Data transformation converts prepared text into a format AI models can directly use by breaking it into smaller pieces and generating vector embeddings. Chunking: Splits large documents into smaller, manageable segments. Embedding: Converts each chunk into a vector that captures semantic meaning. The embedding model choice affects infrastructure design, especially vector dimension size and database layout. Infrastructure teams must align model configuration with vector database schema design. Vector Database Ingestion: Storing and Indexing Embeddings In the final ingestion stage, embeddings and metadata are stored in a vector database optimized for similarity search. Using PostgreSQL with the Pgvector extension enables scalable, high-performance vector search on a familiar platform. Indexing methods such as HNSW and IVFFlat offer different trade-offs between build time and query performance. Why this matters for infrastructure: Storage performance, index choice, and embedding dimensions directly impact query latency, I/O load, and scalability. Looking Ahead: Resource Consumption in the Ingestion Pipeline This article introduced the ingestion pipeline and its core components. The next article will examine resource consumption across the pipeline in more detail and include a Jupyter notebook for hands-on testing and validation. ================================================================================ Title: VMware Private AI Foundation - Privacy and Security Best Practices white paper URL: https://frankdenneman.ai/2024-07-01-vmware-private-ai-foundation-privacy-and-security-best-practices-white-paper/ Date: 2024-07-01 I’m excited to announce the release of my latest white paper, “VMware Private AI Foundation - Privacy and Security Best Practices.” As many of you know, the world of artificial intelligence is rapidly evolving, and with that comes a new set of challenges, particularly around privacy and security. This white paper is not just about theory. It’s a practical guide introducing the foundational concepts, frameworks, and models underpinning private AI security. It’s a deep dive into the critical aspects of privacy and security in the context of AI, providing you with the tools to implement these principles in your own work. You’ll learn about the principle of shared responsibility, threat modeling for Gen-AI applications, and the CIA triad – confidentiality, integrity, and availability – as a guiding model for information security. And that’s not all. In the near future, we’ll be taking these concepts to the next level with a follow-up white paper focused on VMware Cloud Foundation (VCF) settings. This paper will be your go-to resource for detailed guidance on configuring and optimizing VCF to establish a robust and secure private AI environment. Stay tuned for this next installment, where we’ll bridge the gap between theory and practice, empowering you to build and deploy private AI solutions confidently. Thank you for your continued support, and I look forward to hearing your feedback! ================================================================================ Title: RAG Architecture Deep Dive URL: https://frankdenneman.ai/2024-03-19-rag-architecture-deep-dive/ Date: 2024-03-19 Retrieval Augmented Generation (RAG) is a way to enhance Large Language Models (LLMs) by giving them access to extra data. In a typical Gen-AI setup, the LLM answers questions using only what it learned during training. It does not look up new information beyond its training data. RAG changes this by combining retrieval and generation. It uses a retriever to find relevant information from a large text collection, called a corpus, which is stored in a vector database. The generative part, powered by the LLM, then uses this information to create responses. Most discussions about RAG architecture focus on the retrieval process, the vector database, and the LLM. But to design the system and allocate resources properly, it’s also important to understand the indexing workflow and shared components. The tokenizer and embedding model are especially important for making sure retrieval and generation work smoothly. This is the first part of a series where I will move step by step toward Private AI foundation components for running this application in your on-premises virtual datacenter. Let’s look more closely at the indexing and retrieval process and see how each component works in the main stages of retrieval augmented generation. This diagram shows the components, their inputs, and their outputs (elements with dotted lines). Most data scientists design such a system using frameworks such as LangChain or Llama Index. These frameworks provide functionality, as depicted below, but sometimes without distinct, separate components for each task. This diagram shows the conceptual tasks and components used in the indexing and retrieval processes. Its primary goal is to highlight the different phases data go through and the shared components used by both processes. Building the Foundation for Retrieval: The Indexing Process in RAG Architectures The indexing stage in a RAG architecture sets up efficient information retrieval. It transforms many types of data, unstructured documents like PDFs, semi-structured data like JSON, or structured data from databases, into a format that LLMs can use. This process follows a Load-Transform-Embed-Store workflow. Loading Diverse Data Sources The indexing process begins with data loaders, which act as information gatherers. They retrieve data from various sources, including unstructured documents (e.g., PDFs, docs), semi-structured data (e.g., XML, JSON, CSV), and even structured data residing in SQL databases. These loaders then convert the retrieved data into a standardized document format for further processing. Transforming Data for Efficient Processing Document splitters take the stage next. Their role is crucial in organizing the data and preparing it for efficient processing by the embedding model. They achieve this by segmenting the documents into logical units – sentences or paragraphs – based on predefined rules. This segmentation ensures that information remains semantically intact while preparing it for further processing. Tokenization: The Building Blocks of Meaning After splitting, the tokenizer processes each unit, like a paragraph, and breaks it into tokens. Tokens can be words, parts of words, or even characters, depending on the embedding model and how detailed you want to be. Accurate tokenization is important because it shapes how the LLM understands the text. Using one shared tokenizer for all parts of the system helps keep everything consistent. Embedding: Capturing Semantic Meaning After tokenization, the embedding model converts each token into a numerical vector that captures its meaning in context. Pre-trained embedding models map tokens to vector representations. Finally, an indexing component packages the generated embedding vectors along with associated metadata (such as document source information) and sends them to a vector database for efficient storage. This database becomes the foundation for the retrieval stage. The Stored Foundation The vector database plays a crucial role in efficient retrieval. It stores embedding vectors in a multi-dimensional space, enabling fast, effective search based on vector similarity. Retrieval: Efficiently Finding Relevant Information The retrieval stage finds relevant information from indexed data to help the LLM generate answers. The user’s query is processed using the same tokenizer and embedding model as during indexing. Understanding User Queries The process starts when a user submits a query. For private AI deployments, an API gateway validates users, enforces rate limiting, and logs requests. Guardrails ensure prompts meet safety and quality standards before processing. The tokenizer breaks the prompt into tokens, and the embedding model converts it into a vector that captures semantic meaning. Matching Queries with Encoded Information The system searches the vector database for embeddings that closely match the query vector. These matches represent relevant passages from the indexed corpus. Prioritizing Relevant Passages A ranking service scores and sorts retrieved passages based on relevance, ensuring the most useful information is selected. Preparing Information for the LLM The integration module formats the top-ranked passages for LLM consumption. It may summarize, extract key points, or combine passages to fit the model’s input constraints. Feeding the LLM The prepared passages and embedded prompt are sent to the LLM, which generates a response. Post-processing and guardrails ensure quality, safety, and clarity. Polishing the Response: Post-Processing Post-processing may include text normalization, spell checking, grammar correction, and redundancy removal. These steps improve clarity and readability. Tailoring the Response for Presentation Responses are formatted for the target interface, such as a web app or chat UI, using headings, lists, or other visual elements. Seamless Integration: User Interface and Presentation The response is integrated into the application interface, ensuring smooth delivery to the user. User Presentation and Interaction Users receive the final response and may provide feedback or submit follow-up queries. Maintaining a Positive User Experience Accuracy, relevance, and consistency are essential. Error handling and feedback loops help continuously improve system performance. The Unsung Heroes of RAG: Shared Components and Resource Management The tokenizer, embedding model, and vector database form the shared foundation of indexing and retrieval. Proper resource sizing for these components is essential to avoid bottlenecks. Batch-based scraping and ingestion can cause unpredictable resource usage. Flexible infrastructure and scalable resource allocation help manage growth as data volume and sources increase. ================================================================================ Title: The misconception of self-learning capabilities of Large Language Models during Production URL: https://frankdenneman.ai/2024-02-22-the-misconception-of-self-learning-capabilities-of-large-language-models-during-production/ Date: 2024-02-22 I enjoyed engaging with many customers about bringing Gen-AI to the on-prem data center at VMware Explore. Many customers want to keep their data and IP between the four walls of their organization, and rightly so. With VMware Private AI Foundation, we aim to utilize foundation models and build upon the great work of many smart data scientists. Foundation models like Llama 2, StarCoder, and Mistral 7b. Instead of building and training a large language model (LLM) from the ground up, which can be time-consuming and computationally expensive, organizations can leverage foundation models pre-trained on a massive dataset of text and code. If necessary, organizations can further fine-tune a foundation model on specific tasks and data in a short period of time. At VMware, we believe in using Vector DBs with Retrieval Augmented Generation (RAG) to decouple the IP and data from the foundation model. Using RAG, you offload the knowledge updates to another system so the Gen-AI application is always up to date. The vector DB is used for memorizing facts, while the foundation model is used for reasoning functionality. If necessary, the foundation model can be replaced by a newer version. Typically, this doesn’t happen, but if a data science team thinks it will improve the Gen-AI application’s reasoning or generative capability, they can do that without losing any IP. And this particular fact, not losing any IP by replacing the model, got me some pushback from the people I spoke to. By digging into this topic a bit more, I discovered misconceptions among many about the learning ability of a neural network (LLM) model. When you use an LLM, i.e., asking it a question (prompt), the model does NOT learn from your question. LLMs have no self-learning mechanisms during the deployment phase (inference). Let’s dive a bit deeper into the difference between inference and training. Inference: When asking a model a question, you ask it to make a prediction, and the model feeds the input data to the network and its weights (parameters) to compute the output. This sequence is also called a forward pass. Data scientists freeze the parameters when the neural network is accurate enough, and inference uses the same parameters for every question during inference. Training: When training a neural network, the first step is called the forward pass, which is the same as inference. The forward pass calculates the prediction of the model for the input data. After the forward pass, the loss is calculated by comparing the prediction to the expected result. The loss is used to calculate a gradient. The gradient guides the framework to increase or decrease the values of each parameter. The Backpropagation pass adjusts the parameters layer by layer to minimize the loss. The training process repeats the forward pass and backpropagation until the model converges, meaning the loss no longer decreases. Once the model converges, a checkpoint is created, and the parameters are frozen. The Gen-AI application uses this version of the model to generate answers. I believe the misconception of self-learning capabilities occurs when thinking about either recommender systems (Netflix proposing which series to look at next or Amazon telling you other customers bought item X along with item Y), but a recommender system uses a converged model (frozen weights), with an online feature store. An online feature store provides real-time access to essential features for generating accurate and personalized recommendations. Amazon uses an online feature store to store features about its products and users. Product features: These describe the products themselves, such as their price, category, popularity, brand, color, size, rating, and reviews. User features: These describe the users, such as their past purchase history, demographics (age, gender, location), interests, and browsing behavior. Suppose an Amazon customer has purchased many books in the past. In that case, the recommender system might recommend other books to the user based on collaborative or content-based filtering. With collaborative filtering, the algorithm identifies users with similar tastes to the target user and recommends items that those users have liked. With content-based filtering**,** the algorithm recommends items similar to items the target user has liked in the past. It seems like the model is learning when using it. In reality, the model really calculates predictions by using its parameters and data (features) from the online feature store (a database). The context window is another example of why a Gen-AI application like ChatGPT seems to be learning while using it. Models can appear to “learn” during inference in the same context, but it is because the same information is still part of its context window. The best analogy for an LLM context window is the working memory of a human. For example, if I ask Llama 2, “Who is the CEO of VMware?” and the answer it returns is “Pat Gelsinger,” and I respond with, “No, it’s Raghu,” and it responds with, “Yes, that is correct, the CEO of VMware is Raghu Raghuram.” Then, if I ask it, “Who is the CEO of VMware?” it will respond with “Raghu Raghuram,” but that is only because the same context has the answer. Google’s Bard supports up to 2048 tokens, and ChatGPT supports up to 4096 tokens. By default, Llama 2 has a context window of 4096 tokens, but Huge Llama 2 supports up to 32K tokens. A token is a unit of text that the model uses to process and generate text. Tokens can be words, subwords, or characters. Everything comes at a price. The larger the context window is, the more memory and compute inference are consumed. Outside of the same conversational context, models are completely stateless and cannot permanently “learn” anything in inference mode. What most LLMaaS do is capture the user prompts, which might contain sensitive data, and use it to form training data sets. On top of that, they use a technique called Reinforcement Learning with Human Feedback (RHLF), which allows it to learn more frequently. Chip Huyen published an excellent article describing RLHF in detail while being understandable for non-data-scientists. Why does this distinction matter of self-learning matter for non-data scientists? By understanding that default foundation models do not alter state during use, VI-admins, architects, and application developers can design an infrastructure and application that offers high availability while offering a proper life cycle management strategy for the Gen-AI application. For example, the checkpoint of a model can be loaded and exposed by an inference framework running in separate VMs on separate accelerated ESXi hosts with an anti-affinity rule that ensures that each model API endpoint will not share the same physical infrastructure to reduce the blast radius. A load-balancer in front of the inference frameworks offers the flexibility to take one inference framework offline during maintenance jobs without seeing some behavior change. As the model is frozen, no model version would have learned more than the other during their service time. We are currently building and developing VMware Private AI Foundation to allow VMware customers to deploy LLMs on-prem securely and keep the data and Gen-AI application access secure. Private data should be kept private, and together with using state-of-the-art foundation models, organizations can safely build their Gen-AI applications to support their business goals. A special thanks goes out to @Steve Liang for making this article more informative. ================================================================================ Title: Retrieval-Augmented Generation Basics for the Data Center Admin URL: https://frankdenneman.ai/2024-01-16-retrieval-augmented-generation-basics-for-the-data-center-admin/ Date: 2024-01-16 Thanks to ChatGPT, Large Language Models (LLMs) have caught the attention of people everywhere. When built into products and services, LLMs can make most interactions with systems much faster. Current LLM-enabled apps mostly use open-source LLMs such as Llama 2, Mistral, Vicuna, and sometimes even Falcon 170B. These models are trained on publicly available data, allowing them to react appropriately to most prompts. Yet organizations often want LLMs to respond using domain-specific or private data. In that case, a data scientist can fine-tune the model by providing additional examples. Fine-tuning builds on the model’s existing capabilities. Techniques such as LoRA freeze the model’s weights and add small, trainable adapter layers focused on domain-specific needs. Hugging Face recently published an article showing that LoRA introduces only 0.12% additional parameters for Llama 2 7B—about 8.4 million parameters. This means fine-tuning can often be done using just a pair of data center GPUs. This is why, at VMware at Broadcom, we believe combining open-source LLMs with fine-tuning is a practical path toward building strategic business applications. However, each time a new product or service is launched, a data scientist must gather information, organize the data, and fine-tune the model again so it can answer new questions accurately. Introducing Retrieval-Augmented Generation Retrieval-Augmented Generation (RAG) offers a faster, more flexible alternative. RAG adds database capabilities to an LLM, allowing it to retrieve relevant information dynamically while generating an answer. Instead of retraining or fine-tuning the LLM every time new data appears, the model can access up-to-date information directly. Adding RAG to an LLM-enabled application requires more than just connecting a database. Additional steps are involved, but thanks to ongoing work in the data science community, building RAG systems is becoming more accessible. At VMware, this approach is central to the development of VMware Private AI Foundation. Let’s first look at how a basic (non-RAG) LLM workflow operates. A user submits a prompt in an application (1). The app sends the prompt to the LLM (2). The LLM generates a response (3), which the app then displays to the user (4). Vector Databases: The Core of RAG Before diving deeper into RAG, it’s important to understand the vector database. Unlike traditional databases with rows and columns, a vector database stores numerical representations of data, known as vector embeddings. These vectors are grouped based on similarity. Neural network models such as LLMs can only process numbers. In the Natural Language Processing (NLP) pipeline, words are converted into tokens, which are then represented as vectors. These vectors capture the meaning of words and their relationships to other words. For a deeper explanation, Sasha Metzger’s article A Beginner’s Guide to Tokens, Vectors, and Embeddings in NLP is highly recommended. With RAG, the vector database effectively becomes the LLM’s long-term memory. To populate this memory, data is converted into tokens and then into vector embeddings. Tools such as Word2Vec, fastText, and GloVe can be used for this purpose, while frameworks like LlamaIndex help manage ingestion, indexing, and retrieval. Data can be vectorized asynchronously, allowing new information to be added without retraining or fine-tuning the LLM. How RAG Works in Practice From a user’s perspective, the workflow changes slightly. The user submits a prompt in the application (1). Instead of sending the prompt directly to the LLM, the app queries the vector database (2). The vector database performs a similarity search and retrieves relevant data (3). This data is sent back to the application, which augments the user’s prompt with the retrieved context (4). The application then instructs the LLM to generate a response using both the original prompt and the retrieved data (5). Finally, the app presents the response to the user (6). RAG behaves much like a theater prompter, or souffleur. Just as a prompter provides cues to actors during a play, RAG provides contextual cues to the LLM. The vector database acts as the system of record, while the LLM and application layer provide the intelligence to generate accurate, context-aware responses. ================================================================================ Title: Gen AI Sessions at Explore Barcelona 2023 URL: https://frankdenneman.ai/2023-11-01-gen-ai-sessions-at-explore-barcelona-2023/ Date: 2023-11-01 I’m looking forward to next week’s VMware Explore conference in Barcelona. It’s going to be a busy week. Hopefully, I will meet many old friends, make new friends, and talk about Gen AI all week. I’m presenting a few sessions, listed below, and meeting with customers to talk about the VMware Private AI foundation. If you are interested and you see me, walk by, come, and have a talk with me. Hopefully, I will see you at one of the following sessions: Monday, Nov 6: Executive Summit For invite only Time: 11:00 AM - 1:00 PM CET For VMware Certified Instructors only: VCI Forum Keynote. Time: 2:15 PM - 3:00 PM CET Location: Las Arenas II (2) Hotel Porta Fira, Pl. d’Europa, 45, 08908 L’Hospitalet de Llobregat, Barcelona, Spain 15 min walk from conference, 3 min taxi) Meet the Experts Sessions Machine Learning Accelerator Deep Dive [CEIM1199BCN] Time: 4:30 PM - 5:00 PM CET Location: Hall 8.0, Meet the Experts, Table 4 Tuesday, Nov 7: Meet the Experts Sessions Machine Learning Accelerator Deep Dive [CEIM1199BCN] Time: 11:30 AM - 12:00 PM CET Location: Hall 8.0, Meet the Experts, Table 1 Wednesday, Nov 8: AI and ML Accelerator Deep Dive [CEIB1197BCN] Time: 9:00 AM - 9:45 AM CET Location: Hall 8.0, Room 31 CTEX: Building a LLM Deployment Architecture: Five Lessons Learned Speakers: Shawn Kelly & Frank Denneman Time: 4:00 PM - 5:00 PM CET Location: Room CC5 501 Register here: https://lnkd.in/eiTbZusU Recommended AI Sessions: Monday, Nov 6: ‘Til the Last Drop of GPU: Run Large Language Models Efficiently [VIT2101BCN] (Tutorial) Speakers: Agustin Malanco - Triple VCDX Time: 11:00 AM - 12:30 PM CET Location: Hall 8.0, Room 18 Tuesday, Nov 7: Using VMware Private AI for Large Language Models and Generative AI on VMware Cloud Foundation and VMware vSphere [CEIB2050BCN] Speakers: Justin Murray, Shawn Kelly Time: 10:30 AM - 11:15 AM CET Location: Hall 8.0, Room 12 Empowering Business Growth with Generative AI [VIB2368BCN] Speakers: Robbie Jerrom, Serge Palaric, Shobhit Bhutani Time: 2:15 PM - 3:00 PM CET Location: Hall 8.0, Room 14 Why do I need AI in my Data Center? Does this help me become a differentiator? Speaker: Gareth Edwards. Time: Nov 7, 3:30 - 4:40 PM Location: CTEX: Room CC5 501 Meet the Experts: ML/AI and Large Language Models – Implications for VMware Infrastructure [CEIM2282BCN] Expert: Justin Murray Time: 12:30 PM - 1:00 PM CET Location: Hall 8.0, Meet the Experts, Table 2 Wednesday, Nov 8 AI Without GPUs: Run AI/ML Workloads on Intel AMX CPUs with vSphere 8 and Tanzu [MAPB2367BCN] Speakers: Chris J Gully, Earl Ruby Time: 12:45 PM - 1:30 PM CET Location: Hall 8.0, Room 33 Meet the Experts: ML/AI and Large Language Models – Implications for VMware Infrastructure [CEIM2282BCN] Expert: Justin Murray Time: 4:30 PM - 5:00 PM CET Location: Hall 8.0, Meet the Experts, Table 4 ================================================================================ Title: Basic Terminologies Large Language Models URL: https://frankdenneman.ai/2023-08-18-basic-terminologies-large-language-models/ Date: 2023-08-18 Many organizations are in the process of deploying large language models to apply to their use cases. Publically available Large Language Models (LLMs), such as ChatGPT, are trained on publicly available data through September 2021. However, they are unaware of proprietary private data. Such information is critical to the majority of enterprise processes. To help an LLM to become a useful tool in the enterprise space, an LLM is further trained of finetuned on proprietary data to adapt to organization-specific concepts. This process introduces terminology that is used more often outside the data science community. Having a better understanding of the following concepts should enhance your ability to navigate the data science team’s requirements when building an LLM deployment architecture. Neural Network: A Large Language Model (LLM) is a sophisticated neural network architecture designed for natural language processing tasks. LLMs use multiple layers of interconnected nodes to learn language patterns from vast amounts of text data. Parameters, specifically weights, and biases, are crucial components that define how the model processes information. Weights govern the strength of connections between nodes, influencing the LLM’s ability to capture linguistic nuances. Biases adjust the activation levels of nodes, allowing the model to adapt its responses. During forward propagation, the input text is transformed into tokens and flows through the network, undergoing contextual analysis and predicting subsequent words or phrases. Backward propagation, an integral part of training, calculates gradients to adjust weights and biases. This process refines the model’s parameters, aligning its language generation with human-written text. Through continuous learning, LLMs become adept at tasks like completing text, translating, and generating new content. Large Language Model: A Large Language Model is a specific Natural Language Processing (NLP) model that predicts the next word or token. Compared to other NLP models (which are based on language models LM), they are characterized by their size (parameter count of >1B) and are typically trained on vast amounts of text data from the internet. This enables them to learn a broad range of language features and general knowledge. LLMs can handle a number of tasks, from summarization to content generation to question and answer. NLP models are typically focused on sentiment analysis and text classification. LLMs can be further fine-tuned for various tasks with minimal additional training, while NLP is less versatile and requires extensive task-specific training. Foundation model: A foundation model is an LLM, sometimes referred to as a Pretrained Language Model (PLM), that is robust enough to act as a “foundation” that can be used as is or fine-tuned/adapted for newer domains. In general, it is trained on diverse data, capable of being customized and fine-tuned for various applications and tasks. For example, the open-source LLaMA 2 models are trained on 2 trillion tokens primarily sourced from publicly available online data sources. Meta required 368640 GPU (A100-80 GBs) hours to train the 13B LLaMa-2 model. Token: A token is a fundamental unit of text that an LLM uses to process and understand language. In English, a token can be as short as a single character or as long as a word. For example, in the sentence “I love vSphere,” there are three tokens: “I,” “love,” and “vSphere.” However, the word “vSphere” might be broken down into two tokens depending on the model’s tokenizer. Token length is important because it can impact the overall size of the input data, computational resources required for processing, and the model’s ability to perform linguistically. Tokens help break down the text into manageable pieces for analysis and are essential for language generation. They serve as the building blocks that allow LLMs to comprehend human language. Embeddings: are numerical representations of tokens. Each token is transformed into a vector of numbers. These vectors encode the semantic meaning of the text and facilitate computer-based processing and analysis of human language. This language-to-number translation equips machine-learning processes to interact with language data efficiently. Vector: Vectors are commonly used to represent data points in a multi-dimensional space. Each element of a vector corresponds to a specific dimension, and the combination of these elements defines the position of the vector in that space. Vectors are essential for measuring similarities and transforming data. The article “Explaining Vector Databases in 3 Levels of Difficulty” provides a great primer on vectors, embeddings, and Vector databases. Image by Leonie Monigatti Parameters: Parameters are the learned values that a model acquires through training to facilitate predictions or classifications on new data. In neural networks, these parameters are commonly denoted as weights and biases, dictating how input data undergoes transformation into output predictions. An LLM model size is typically expressed in parameter count in billions, such as the LLaMA-2 13B. This model contains roughly 13 Billion parameters. The memory consumption depends on the floating-point format (precision). Typically for LLM, a BF16 or FP16 is used. A parameter using BF16 consumes 2 bytes. 1 Billion bytes equal a gigabyte. Thus, we can easily calculate the “static” model memory consumption. LLaMA-2 13B consumes 26 GB of GPU memory when loaded into the GPU. Streaming data into the LLM or fine-tuning the model increases memory consumption. Transformer Architecture: The transformer architecture is a foundational framework for training and utilizing large neural networks in natural language processing (NLP). It revolutionized the field by introducing the attention mechanism, which allows the model to weigh the importance of different words in a sentence. It introduced a self-attention mechanism to process input data in parallel, allowing it to truly understand the context of words depending on other words that are placed further away within the sentence (long-range dependencies). The transformer architecture has an encoder and decoder functionality. The encoder creates a “contextualized representation” of a prompt, capturing the meaning and significance of the words in the prompt by considering how they relate to each other within the given context. The decoder receives and processes the contextualized representation, using its understanding of the context to guide output generation. Typically, it generates one word of the output at a time. The first word generated is based on the input context, and the next word is generated based on the input context and the previously generated word. This is called autoregressive context. (Downstream) Task: An LLM is typically refined further to achieve a specific goal or perform a particular downstream task. Many foundation models today can perform a wide range of NLP tasks without refinement for a particular downstream task. A “task” refers to a specific job or activity that the model is designed to perform using its language understanding and generation capabilities. Tasks can include a wide range of natural language processing objectives. The most common ones are text generation (predicting the next word/token), summarization (given an input, return a short output), question answering (predicting the next word/token based on search results in the prompt), sentiment analysis, translation, and instruction. When a new downstream task is needed, we still need to fine-tune the foundation LLM to respond to domain-specific prompts (eg; ensure it favors a particular term, “Tanzu,” over others). Prompt: A prompt is a specific input given to the model to guide its behavior and generate desired text output. The prompt serves as an instruction or query that helps the LLM understand the task or context it needs to respond to. It can be a sentence, a paragraph, or even just a few keywords, depending on the task at hand. For example, if technical marketing wants an LLM to generate a technical specification overview, they can use the prompt “Write a detailed description of our latest update release, highlighting its unique features and capabilities.” Prompt engineering: The quality and specificity of the prompt can greatly influence the LLM’s output. A well-crafted prompt provides clear guidance to the model, leading to more relevant and accurate generated text. Prompt engineering involves designing prompts effectively to achieve the desired results for various tasks such as translation, summarization, question answering, and more. Prompt engineering can also be used to induce LLMs to say undesirable things, for example, with a prompt that tells an LLM to forget its existing rules before responding. Zero-shot: refers to the model’s ability to perform tasks or generate text about new topics without additional training or examples in the prompt. It relies on its existing knowledge to tackle new challenges without requiring specific learning for each one. P-tuning: This technique involves fine-tuning a small trainable model prior to engaging the LLM. The small model encodes the text prompt and crafts task-specific virtual tokens, which are then added to the prompt and fed into the LLM. Once the fine-tuning is done, these virtual tokens are stored in a lookup table and used during inference, replacing the smaller model. P-tuning is far more resource efficient compared to other forms of fine-tuning of an LLM. The time required to tune a smaller model can often be measured in minutes instead of days with fine-tuning the LLM. This is a great talk about P-tuning and what exactly virtual tokens are. Parameter Efficient Fine-Tuning: PEFT aims to make the fine-tuning process more efficient by focusing on updating only a subset of the model’s parameters (e.g., 100M parameters for a 15B model), rather than retraining the entire model from the ground up. The idea behind PEFT is to balance retaining the knowledge captured by the pre-trained model and tailoring it to perform well on a specific task. By carefully selecting and adjusting certain parameters, you can achieve good performance on the task while reducing the computational cost and time required for fine-tuning. PEFT also overcomes the issues of catastrophic forgetting, a behavior observed during the full fine-tuning of LLMs. There are multiple PEFT methods: : Adapter-based PEFT: Adapters are new modules added to the pre-trained network, and only the new parameters are trained, while the original LLM-trained parameters are left untouched. As a result, a small proportion of parameters of the original LLM is trained. This means that the model keeps remembering the previous tasks and uses a small number of new parameters to learn the new task. However, the downside of adding these new layers is the inference latency increase. This issue appears unavoidable because the adapter layers are added sequentially to an LLM. They must be processed sequentially, and there is no way to parallel process them. LoRA: Low-Rank Adaptation of Large Language Models: LoRA also freezes the pre-trained parameters, but instead of adding additional layers to the neural network, it adds values to the parameters. As a result, the model can be executed fully in parallel, avoiding additional inferencing latency. In addition, LoRA applies a very intelligent method to reduce the number of trainable parameters, reducing fine-tuning time and memory consumption. A further reduction of memory consumption can be achieved by quantizing the majority (non-outliers) trainable parameters, which results in using an integer data type (INT8) instead of a floating point. A parameter stored in INT8 consumes 8 bits, reducing the memory footprint in half compared to BF16. This is the most popular method of PEFT. Quantized Low-Ranking Adaptation (QLoRA): QLoRA takes it one step further and compresses the weights and activations to 4-bit precision. QLoRA uses a specific data type for storing the base model weights and data type to perform computations. During the computations, QLoRA “dequantizes” the weights from a 4-bit precision (4-bit NormalFloat) into a 16-bit bfloat. The weights are only decompressed when needed; therefore, QLoRA allows large models to run on GPUs with smaller memory capacities. IA3 (Infused Adapter by Inhibiting and Amplifying Inner Activations): IA3 is a newer PEFT technique intended to improve over LoRA. It offers the same benefits as the LoRA. However, it’s only tested on really “small” LLM models (3B) currently, and most backends do not support this yet. The last update on the GitHub IA3 repo was done in September 2022, which might hint at a lack of popularity of this PEFT method in the data science community. Fine Tuning: Fine-tuning is the mechanism to improve LLM models for a specific task (e.g., summarizing legal documents) or domain (e.g., more knowledge about virtualization). During the training of an LLM, the neural network is exposed to unlabeled data and learns through a form of self-supervision (e.g., predicting the next word or sentence entailment). That means that the algorithm explores the patterns, structures, and relationships within the dataset on its own without being provided with specific labeled output for every task, but the robustness of its dataset may mean that it will already be good at some tasks (eg; generating coherent sounding sentences). Fine-tuning always involves supervised learning, where human labelers validate and curate data for a specific task (e.g., to fine-tune the model’s question-answering capabilities, we would have question and answer pairs as the input). Reinforcement Learning with Human Feedback (RLHF): When validating the accuracy of a model designed for an image recognition task, we can quickly determine its accuracy. It either classifies the cat correctly or not. With LLMs, it’s a bit more challenging, as it is more difficult to define what makes a “good” text as it is subjective and context-dependent. The RHLF method uses feedback generated by humans for generated text to measure the model performance. The method involves deploying multiple models during the training process. A pretrained language model and typically a smaller reward model. The PLM generates multiple responses based on a prompt and a reward model numerical scores each response on how well humans perceive this text. It ranks the response according to human preference and then applies reinforcement learning to train the PLM further to prioritize the response with the higher numerical scores. RHLF systems are complex and challenging as gathering human preference data is expensive. RLHF performance is only as good as the quality of the human annotations. People tend to disagree. Therefore, ground truth is often lacking due to variance in opinions by the annotating team. There are multiple methods of human feedback besides the preference order, such as: Corrections: Upvoting or downvoting the model output (also known as prompt completion) Demonstrations: Humans write the preferred answer to a prompt Natural Language Input: Humans are asked to provide feedback on the model output in natural language Supervised fine-tuning (SFT): involves adapting a PLM to a specific downstream task using validated training examples. Validated training examples are pairs of input data and output labels that have been carefully checked and confirmed to be accurate and reliable. SFT tunes the model to a specific task, such as responding to customer support questions. The model can be trained to adapt to specific knowledge-based or demonstrate a particular persona or empathy by using validated training examples. Is PEFT considered supervised learning?: PEFT is not strictly categorized as supervised learning; rather, it is a technique that can be applied within the context of supervised learning. Supervised learning involves training a model on labeled data pairs, where the labels serve as the ground truth for training. PEFT, on the other hand, is a method that aims to fine-tune a pre-trained model using a limited amount of new data specific to a task to adapt the model’s parameters while leveraging its existing knowledge. PEFT is characterized by its emphasis on efficiency and parameter reuse. It does not require extensive retraining on the entire dataset, as it focuses on updating only a subset of the model’s parameters to adapt it to a new task. In-Context Learning Retrieval Augmented Generation (RAG): Once an LLM is trained, it is ignorant of new data. When an organization launches a new product or service, the customer service-focused LLM needs to be retrained to incorporate these new data points. RAG allows the LLM to act as a conversational interface, while RAG “grounds” the LLM with information that aligns with its use case while reducing hallucinations. RAG allows LLMs to access knowledge sources outside the trained model and augment the completion of the prompt with relevant information found in external sources. These sources can be the organizations’ proprietary data sources, like knowledge bases, Bugzilla, Confluence, internal documents, etc. RAG typically works together with a Vector Database or a search engine. When the user provides the prompt to the LLM, the RAG framework performs a “contextual search” on the Vector database and generates the context. The RAG framework augments the original prompt by injecting the context into the prompt. The LLM receives the enriched prompt and can generate a better response as it has access to factual data. The LLM sends the generated response back to the user. RAG frameworks and Vector DBs can be critical differentiators for organizations with rapidly changing knowledge bases or any other source of information. These organizations cannot change the LLM and redeploy at the same velocity as the demand for up-to-date information. Keeping the models’ answers in lock-step with the velocity of their service offering is challenging. A good example is a FAQ for a chatbot function at a call center for a new product or service. The data science team can add up-to-date data to the Vector DB answer and question domains and ensure that the RAG framework prioritizes the Vector DB over the LLM model for information retrieval. Grounding: refers to the process of connecting the model’s prompt completion (responses) to real-world knowledge. Grounding is about ensuring that the model’s outputs are coherent, accurate, and relevant to the information available in the world. Pretraining, prompt-engineering, fine-tuning, and in-context learning all help to ground the LLM. Hallucination: a hallucination is a confident response by the LLM that appears coherent and contextually relevant but is not based on accurate information from the input data or is not logically grounded in reality. An example is URLs that are generated by an LLM that does not exist. Guardrails: Guardrails restrict LLMs to respond in a particular (safe) manner. These guardrails guide the LLM to stay on topic, avoid hallucinations or toxic responses, or execute malicious code. Chatbots can become an attack surface, and a security guardrail can protect LLM platforms. Guardrails are programmable constraints and are placed between the chatbot and the LLM. The NVIDIA NeMo framework offers a guardrail workflow to apply constraints to the LLM easily. ================================================================================ Title: My Sessions at VMware Explore 2023 Las Vegas URL: https://frankdenneman.ai/2023-08-17-my-sessions-at-vmware-explore-2023-las-vegas/ Date: 2023-08-17 Next week we are back in Las Vegas. Busy times ahead with meeting customers, old friends, making new friends, and presenting a few sessions. Next week I will present at Customer Technical Exchange (CTEX), {code}, and host two meet-the-expert sessions. I will also participate as a part-time judge at the {code} hackathon. Breakout Sessions 45 Minutes of NUMA (A CPU is not a CPU Anymore [CODEB2761LV] Tuesday, Aug 22, 2:45 PM - 3:30 PM PDT Level 4, Delfino 4003 Yu Wang and I will dive deep into the new Multi-Chip CPU architecture, On-board Accelerators, and Sub-NUMA clustering and highlight cool new vSphere 8 features that will make your life as a Vi-admin easier. Building an LLM Deployment Architecture - 5 lessons learned [CTEX] Tuesday, Aug 22, 4:00 PM - 5:00 PM PDT, Zeno 4708 Shawn Kelly and I will go over the details on how to create a deployment architecture for fine-tuning and deploying Large Language Models on your vSphere environment. Shawn and I have been deploying and developing a chatbot application with a data science team within VMware, and we want to share our lessons learned. What’s new with VMware+NVIDIA AI-Ready Enterprise Platform [CEIB3051LV] Thursday, Aug 24, 10:00 AM - 10:45 AM PDT Level 2, Titian 2305 Watch Raghu’s keynote, and then come to this session later next week. Due to NDA rules, we cannot disclose any content in the description of the current Explore Content Catalog. If you are planning to run Machine Learning workloads in your organization, join this session to hear about our latest offering that NVIDIA and VMware have created together. Meet The Expert Sessions Machine Learning Accelerator Deep Dive [CEIM1849LV] Monday, Aug 211:00 PM - 1:30 PM PDTMeet the Experts, Level 2, Ballroom G, Table 7 Wednesday, Aug 2311:00 AM - 11:30 AM PDTMeet the Experts, Level 2, Ballroom G, Table 7 Have questions about your ML workload or are not sure whether vSphere is the platform for ML workload? Sign up for the MtE session, and let’s discuss your challenges. If you see me walk by, say hi! ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive – Installing the NVAIE vGPU Driver URL: https://frankdenneman.ai/2023-07-06-vsphere-ml-accelerator-spectrum-deep-dive-installing-the-nvaie-vgpu-driver/ Date: 2023-07-06 After setting up the Cloud License Service Instance, the NVIDIA AI Enterprise vGPU driver must be installed on the ESXi host. A single version driver amongst all the ESXi hosts in the cluster containing NVIDIA GPU devices is recommended. The most common error during the GPU install process is using the wrong driver. And it’s an easy mistake to make. In vGPU version 13 (the current NVAIE version is 15.2), NVIDIA split its ESXi host vGPU driver into two kinds. A standard vGPU driver component supports graphics, and an AI Enterprise (AIE) vGPU component supports compute. The Ampere generation devices, such as the A30 and A100 device, support compute only, so it requires an AIE vGPU component. There are AIE components available for all NVIDIA drivers since vGPU 13. This article series focuses on building a vSphere infrastructure for ML platforms, and thus this article lists the steps to install the NVD-AIE driver. To download the NVD-AIE driver, ensure you have a user id that has access to the NVIDIA License Portal. The next step is to ensure the platform meets the requirements before installing the driver component. The installation process can be done using vSphere Lifecycle Manager or manually installing the driver component on each ESXi host in the cluster. Both methods are covered in this article. Requirements Make sure to configure the ESXi Host settings as follows before installing the NVIDIA vGPU Driver: Component Requirements Notes Physical ESXi host Must have Intel VT-d or AMD I/O VT enabled in the BIOS Must have SR-IOV enabled in the BIOS Enable on Ampere & Hopper GPUs Must have Memory Mapping Above 4G enabled in the BIOS Not applicable for NVIDIA T4 (32-bit BAR1) vCenter Must be configured with Advanced Setting vgpu.hotmigrate.enabled If you want to live-migrate VMs vSphere GPU Device Settings Graphics Type: Basic Graphics Device Settings: Shared Direct The article vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings provides detailed information about every setting listed. Preparing the GPU Device for the vGPU Driver The GPU device must be in “Basic Graphics Type” mode to successfully install the vGPU driver. It is the default setting for the device. That means that the GPU should not be configured as a Passthrough device. In vCenter, go to the Inventory view, select the ESXi host containing the GPU device, go to the Configure menu option, Hardware, PCI Devices. The GUI should present the following settings: Go to the Graphics menu options Select the GPU and choose EDIT... The default graphics type is Shared, which provides vSGA functionality. To enable vGPU support for VMs, you must change the default graphics type to Shared Direct. You can verify these settings via the CLI using the following command: % esxcli graphics device list 0000:af:00.0 Vendor Name: NVIDIA Corporation Device Name: GA100 [A100 PCIe 40GB] Module Name: None Graphics Type: Basic Memory Size in KB: 0 Number of VMs: 0 Verify if the default graphics settings is set to Shared Direct using the command: % esxcli graphics host get Default Graphics Type: SharedPassthru Shared Passthru Assignment Policy: Performance If the UI is having an off-day and won’t want to listen to your soothing mouse clicks, use the following commands to place GPU in the correct mode, and restart the X.Org server. % esxcli graphics host set --default-type SharedPassthru % /etc/init.d/xorg restart Getting Exclusive access, please wait… Exclusive access granted. % esxcli graphics host get Default Graphics Type: SharedPassthru Shared Passthru Assignment Policy: Performance NVAIE Driver Download Selecting the correct vGPU version The vGPU driver is available at the NVIDIA Licensing Portal, part of the NVIDIA APPLICATION HUB. Go to the SOFTWARE DOWNLOADS option in the left menu, or go directly to https://ui.licensing.nvidia.com/software if logged in. Two options look applicable: the vGPU product family and the NVAIE product family. For ML workloads, choose the NVAIE product family. The NVAIE product family provides vGPU capability for compute (ML/AI) workload. The vGPU family is for vGPU functionality for the VDI workload. You can recognize the correct vGPU Driver components with the NVD-AIE (NVIDIA AI Enterprise) prefix. To reduce the results shown on the screen, click PLATFORM and select VMware vSphere. Select the latest release. At the time of writing this article, the latest NVAIE version was 3.1 for both vSphere 7 and vSphere 8. Since I’m running vSphere 8.0 update 1, I’m selecting NVAIE 3.1 for vSphere 8. The file is downloaded in a zip-file format. Extract the file to review the contents. As described in the articles “vSphere ML Accelerator Deep Dive – Fractional and Full GPUs” and “vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings”, two components drive the GPU. The guest OS driver controls the MMIO space and the communication of the device, and the GPU manager (the NVIDIA name for the ESXi Driver) controls the time-sharing mechanism to multiplex control across multiple workloads. The zip file contains the ESXi Host Drivers and the Guest Drivers. It does not contain the GPU operator for TKGs, which installs and manages the LCM of the TKGs worker node driver. The GPU operator is installed via a helm chart. Before being able to install the GPU operator, the ESXi host driver must be installed to allow the VM Class to present a vGPU device to the TKGs worker node. Installing the Driver vSphere offers two possibilities to install the NVAIE vGPU driver onto all the cluster hosts, and which method is best for you depends on the cluster configuration. If every ESXi host in the cluster is equipped with at least one GPU device. Installing and managing the NVAIE vGPU driver across all hosts in the cluster using a vSphere Lifecycle Manager (vLCM) desired state image is recommended. The desired state functionality of Lifecycle Manager ensures the standardization of drivers across every ESXi host in the cluster. Visual confirmation in vCenter allows VI-admins to verify driver consistency across ESXi hosts. If an ESXi host configuration is not compliant with the base image vCenter lifecycle manager reports this violation. If a few ESXi hosts in the cluster contain a GPU card, installing the driver using the manual process might be better. You can always opt for using a desired state image in a heterogeneous cluster, but some ESXi hosts will have a driver installed without use. In general, using the desired state image throughout the cluster, even in heterogeneous clusters, is recommended to manage a consistent version of the NVAIE vGPU Driver at scale. vSphere Lifecycle Manager vSphere Lifecycle Manager (vCLM) integrates the NVAIE GPU driver with the vSphere base image to enforce consistency across the ESXi hosts in the cluster. To extend the base image, import the NVAIE GPU driver component by opening up the vSphere client menu (click on the three lines in the top left corner next to vSphere Client) select Lifecycle Manager, select ACTIONS, Import Updates. Browse to the download location of the NVAIE vGPU driver and select the Host Driver Zip file to IMPORT. Verify if the import of the driver component is successful by clicking on Image Depot. Search the Components list. To extend the base image with the newly added component, go to the vCenter Inventory view, right-click the vSphere cluster, choose Settings, and click the update tab. In the Image view, Click on EDIT, review the confirmation, and choose RESUME EDITING. Click on ADD COMPONENTS, add the NVIDIA AI Enterprise vGPU driver for VMware ESX-version number, and choose SELECT. The driver is added to the base Image. Click on Save. The ESXi hosts in the cluster should be listed as non-compliant. Select the ESXi hosts to validate and remediate to apply the new base image. Once the base image is installed on all ESXi hosts, the Image Compliance screen indicates that all ESXi hosts in the cluster are compliant. The cluster is ready to deploy vGPU-enabled virtual machines. Manual NVAIE vGPU Driver Install If multiple ESXi hosts in the cluster contain GPUs, it is efficient to store the ESXi host driver on a shared datastore accessible by all ESXi hosts. Uploading the Driver to a Shared Repository In this example, I upload the ESXi host driver to the vSAN datastore. SFTP requires SSH to be active. Enable it via the Host configuration of the Inventory view of vCenter, go to the Configure Menu option of the host, System, Services, click on SSH, and select Start. Remember, this service keeps on running until you boot the ESXi host. A folder is created iso/nvaie. Use the put command to transfer the file to the vSAN datastore % sftp root@esxi host (root@esxi host) Password: Connected to esxi host sftp> lpwd Local working directory: /home/vadmin/Downloads/NVIDIA-AI-Enterprise-vSphere-8.0-525.105.14-525.105.17-528.89/Host_Drivers sftp> cd /vmfs/volumes/vsanDatastore/iso/nvidia/ sftp> lls NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_21506612.zip nvd-gpu-mgmt-daemon_525.105.14-0.0.0000_21514245.zip NVD-AIE_ESXi_8.0.0_Driver_525.105.14-1OEM.800.1.0.20613240.vib sftp> put NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_21506612.zip Uploading NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_21506612.zip to /vmfs/volumes/vsan:5255fe0bd2a28e17-a7cdc87427ad0c55/869a9363-2ca2-c917-f0f6-bc97e169cdf0/nvidia/NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_21506612.zip NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_21506612.zip 100% 108MB 76.6MB/s 00:01 Install the NVAIE vGPU Driver Before installing any software components on an ESXi host, always ensure that no workloads are running by putting the ESXi host into maintenance mode. Right-click the ESXi host in the cluster view of vCenter, select the submenu Maintenance mode, and select the option Enter Maintenance mode. Install the NVIDIA vGPU hypervisor host driver and the NVIDIA GPU Management daemon using the esxcli command: esxcli software component apply -d /path_to_component/NVD-AIE-%.zip % esxcli software component apply -d /vmfs/volumes/vsanDatastore/iso/nvidia/NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240_ 21506612.zip Installation Result Message: Operation finished successfully. Components Installed: NVD-AIE-800_525.105.14-1OEM.800.1.0.20613240 Components Removed: Components Skipped: Reboot Required: false DPU Results: Please note that the full path is required, even if you are running the esxcli command from the directory where the file is located. And although the output indicates that no reboot is required, reboot the ESXi host to load the driver. Verify the NVAIE vGPU Driver Verify if the driver is operational by executing the command nvidia-smi in an SSH session. Unfortunately, NVIDIA SMI doesn’t show whether an NVD-AIE or a regular vGPU driver is installed. If you are using an A30, A100, or H100, NVIDIA-SMI only works if an NVD-AIE driver is installed. You can check the current loaded VIB in vSphere with the following command: % esxcli software component list | grep NVD NVD-AIE-800 NVIDIA AI Enterprise vGPU driver for VMWare ESX-8.0.0 525.105.14-1OEM.800.1.0.20613240 525.105.14 NVIDIA 03-27-2023 VMwareAccepted host The UI shows that the active type configuration is now Shared Direct. The cluster is ready to deploy vGPU-enabled virtual machines. The next article focusses on installing the TKGs NVAIE GPU Operator. Previous articles in the vSphere ML Accelerator Spectrum Deep Dive Series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup URL: https://frankdenneman.ai/2023-07-05-vsphere-ml-accelerator-spectrum-deep-dive-nvaie-cloud-license-service-setup/ Date: 2023-07-05 Next in this series is installing the NVAIE GPU operator on a TKGs guest cluster. However, we must satisfy a few requirements before we can get to that step. NVIDIA NVAIE Licence activated Access to NVIDIA NGC NVIDIA Enterprise Catalog and Licensing Portal The license Server Instance activated NVIDIA vGPU Manager installed on ESXi Host with NVIDIA GPU Installed VM Class with GPU specification configured Ubuntu image available in the content library for TKGs Worker Node vCenter and Supervisor Access The following diagram provides an overview of all the components, settings, and commands involved. Although I do not shy away from publishing long articles, these steps combined are too much for a single article. I’ve split the process up in three separate articles: NVAIE Cloud License Service Setup NVIDIA vGPU Manager Install TKGs GPU operator install In this article, I follow a greenfield scenario where no NVIDIA license service instance is set up. As I cannot describe your internal licensing processes, I list the requirements of access rights and permissions to set up the NVIDIA license. NVIDIA NVAIE Licence activated All commercial-supported software installations require a license key, and the NVAIE suite is no different. In NVIDIA terms, this license key is called the Product Activation Key ID (PAK ID) and it’s necessary to set up a License Service Instance. The component which distributes and tracks client license allocation. Before you start your journey, ensure you have this PAK ID or have access to a fully configured License Service Instance. Access to NVIDIA NGC NVIDIA Enterprise Catalog and Licensing Portal The NVIDIA AI Enterprise software is available via the NGC NVIDIA Enterprise Catalog (GPU operator repository) and the NVIDIA License Portal (ESXi VIBs). This software is only available for users linked to an NVAIE-licensed organization. The pre-configured GPU Operator differs from the open-source GPU Operator in the public NGC catalog. The differences are: It is configured to use a prebuilt vGPU driver image (Only available to NVIDIA AI Enterprise customers) It is configured to use containerd as the container runtime It is configured to use the NVIDIA License System (NLS) In larger organizations, a liaison manages NVIDIA licensing, who can get your NGC account listed as a part of the company’s NGC organization and provide you access to the NVIDIA Enterprise Application Hub licensing portal. These steps are out-of-scope for this article. The starting point for this article is that you have: an NGC account connected to an NGC org that has access to NGC NVIDIA Enterprise Catalog for downloading the drivers and helm charts an NVIDIA Enterprise Application Hub account that has access to the NVIDIA licensing portal for creating a License Service Instance or download a client configuration token An NVIDIA Entitlement Certificate with a valid Product Activation Key (PAK) ID if you plan to install a License Service Instance. Please note that I’m an employee of VMware, I do not own the licenses, nor did I participate in the process of obtaining the licenses. I requested the licenses via a VMware internal process. I have no knowledge of internal NVIDIA processes which provides access to the NGC NVIDIA Enterprise Catalog or the NVIDIA licensing portal. I cannot share my product activation key ID. Please connect with your NVIDIA contact for these questions. Selecting a License Service Instance Type The NVIDIA License system supports two license server instances: Cloud License Service (CLS) instance Delegated License Service (DLS) instance For AI/ML workloads, most customers prefer the CLS instance hosted by the NVIDIA license portal. Since NVIDIA maintains the CLS instance, the on-prem platform operators do not have to worry about the license service instance’s availability, scalability, and lifecycle management. The only requirement is that workloads can connect to the CLS instance. To establish communication between the workload clients and the CLS instance, the following firewall or proxy server ports must be open: Port Protocol Eg-/Ingress Service Source Destination 80 TLS, TCP Egress License release Client CLS 443 TLS, TCP Egress License acquisition License renewal Client CLS A DLS instance is necessary if you are running an ML cluster in an air-gapped data center. The DLS instance is fully disconnected from the NVIDIA licensing portal. The VI-admin must manually download the licenses from the NVIDIA license portal and upload them to the instance. A highly available DLS setup is recommended to provide licensed clients with continued access to NVAIE licenses if one DLS instance fails. A DLS instance can either run as a virtual appliance or as a containerized software image. The minimum footprint of the DLS virtual appliance is 4 vCPUs, 8GB RAM, and 10GB disk space. A fixed or reserved DHCP address and an FQDN must be registered before installing the DLS virtual appliance. It is recommended to synchronize the DLS virtual appliance with an NTP server. Please review the NVIDIA License System User Guide for a detailed installation guide and an overview of all the required firewall rules. For this example, A CLS instance is created and configured. Setting up a Cloud License Service Instance Log in to the NVIDIA Enterprise Application Hub anc click on NVIDIA Licensing Portal to go to the “NVIDIA licensing portal”. Creating a License Server Expand “License Server” in the menu on the left of your screen and select “create server” Ensure “Create legacy server” is disabled (slide to the left) and provide a name and description for your CLS. Click on “Select features”. The node-locked functionality allows air-gapped client systems to obtain a node-locked vGPU software license from a file installed locally on the client system. The express CLS installation. Find your licensed product using the PAK ID that is listed in your NVIDIA Entitlement Certificate. The PAK ID should contain 30 alphanumeric characters). In the textbox in the ADDED column, enter the number of licenses for the product that you want to add. Click Next: Preview server creation. On the Preview server creation page, click CREATE SERVER. Once the server is created, the portal shows the license server is in an “unbound state”. Creating a CLS Instance In the left navigation pane of the NVIDIA Licensing Portal dashboard, click SERVICE INSTANCES. Provide a name and description for the CLS Service Instance. Binding a License Server to a Service Instance Binding a license server to a service instance ensures that licenses on the server are available only from that service instance. As a result, the licenses are available only to the licensed clients that are served by the service instance to which the license server is bound. In the left menu, select License Servers, and click LIST SERVERS. Find your license server using its name, click the Actions button on the right side of your screen, and select Bind. In the Bind Service Instance pop-up window that opens select the CLS service instance to which you want to bind the license server and click BIND. The Bind Service Instance pop-up window confirms that the license server has been bound to the service instance. The event viewer lists the successful bind action. Installing a License Server on a CLS Instance This step is necessary if you are using multiple CLS instances in your organization that is registered at NVIDIA and you are using the new CLS instance, not the default CLS instance for the organization. In the left navigation pane, expand LICENSE SERVER and click LIST SERVERS. Select your License Server, click on Actions, and choose Install. In the Install License Server pop-up window that opens, click INSTALL SERVER. The event viewer lists a successful deployment of the license service on the service instance. The next step is to install the NVIDIA NVAIE vGPU Driver on the ESXi host. This step is covered in the next article. Previous articles in the vSphere ML Accelerator Spectrum Deep Dive Series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs URL: https://frankdenneman.ai/2023-06-06-vsphere-ml-accelerator-spectrum-deep-dive-using-dynamic-directpath-io-passthrough-with-vms/ Date: 2023-06-06 vSphere 7 and 8 offer two passthrough options, DirectPath IO and Dynamic DirectPath IO. Dynamic DirectPath IO is the vSphere brand name of the passthrough functionality of PCI devices to virtual machines. It allows assigning a dedicated GPU to a VM with the lowest overhead possible. DirectPath I/O assigns a PCI Passthrough device by identifying a specific physical device located on a specific ESXi host at a specific bus location on that ESXi host using the Segment/Bus/Device/Function format. This configuration path restricts that VM to that specific ESXi host. In contrast, Dynamic DirectPath I/O utilizes the assignable hardware framework with vSphere that provides a key-value method using custom or vendor-device-generated labels. It allows vSphere to decouple the static relationship between VM and device and provides a flexible mechanism for assigning PCI devices exclusively to VMs. In other words, it makes passthrough devices work with DRS initial placement and, subsequently vSphere HA. The assignable hardware framework allows a device to describe itself with key-value attributes. The framework allows the VM to specify the attributes of the device. It relies on the framework to match these two for design assignment before DRS handles the virtual machine placement decision. It allows operation teams to specify custom labels that help to indicate hardware or site-specific functionality. For example, Labels in a heterogeneous ML cluster can designate which GPUs serve for training and inference workloads. DirectPath IO Dynamic DirectPath IO VMX device configuration notation pciPassthru%d.id = pciPassthru%d.allowedDevices = vendorId:deviceId pciPassthru%d.customLabel = Example pciPassthru0.id= 0000:AF:00.0 pciPassthru0.allowedDevices = “0x10de:0x20f1” pciPassthru0.customLabel = “Training” Example explanation The VM is configured with a passthrough device at SBDF address of 0000:AF:00.0 The VM is configured with a passthrough device that has vendor ID as “0x10de” and device model id as “0x20f1” The custom label indicates this device is designated as a Training device by the organization. Impact The device is assigned statically at VM configuration time. The VM is not migratable across ESXi hosts because it is bound to that specific device on that specific ESXi host. The VM is configured with a passthrough device with vendor ID as “0x10de” and device model id as “0x20f1” The custom label indicates this device is designated as a Training device by the organization. The operations team can describe what kind of device a VM needs. Dynamic Directpath IO, with the help of the assignable hardware framework, assigns a device that satisfies the description. As DRS has a global view of all the GPU devices in the cluster, it is DRS that coordinates the matching of the VM, the ESXi host, and the GPU device. DRS selects and GPU device and moves the VM to the corresponding ESXi host. During power on, the host services follow up on DRS’s decision and assign the GPU device to the VM. The combination of vSphere clustering services and Dynamic DirectPath IO is a significant differentiator between running ML workloads on a virtualized platform and bare-metal hosting. Dynamic DirectPath IO allows DRS to automate the initial placement of accelerated workloads. With Dynamic DirectPath IO and vSphere HA, workloads can frictionlessly return to operation on other available hardware if the current accelerated ESXi host fails. Initial Placement of Accelerated Workload Dynamic DirectPath I/O solves scalability problems within accelerated clusters. If we dive deeper into this process, the system must take many steps to assign a device to a VM. In a cluster, you must find a compatible device and match the physical device to the device listed in the VM(X) configuration. The VMkernel must perform some accounting to determine that the physical device is assigned to the VM. With DirectPath IO, the matching and accounting process uses the host, bus, and other PCIe device locator identifiers. With Dynamic DirectPath IO, the Assignable Hardware framework (AH) is responsible for the finding, matching, and accounting. AH does not expose any API functionality to user-facing systems. It is solely an internal framework that provides a search and accounting service for vCenter and DRS. NVIDIA vGPU utilizes AH as well. The ESXi host implements a performance or consolidation allocation policy using fractional GPUs. AH helps assign weights to a device instance to satisfy the allocation policy while multiple available devices in the vSphere cluster match the description. But more on that in the vGPU article. For Dynamic DirectPath IO, DRS uses the internal AH search engine to find a suitable host within the cluster and selects an available GPU device if multiple GPUs exist in the ESXi host. Every ESXi host reports device assignments and the GPU availability to AH. During the host selection and VM initial placement process, DRS provides GPU device selection as a hint to the VMkernel processes running within the ESXi. During the power-on process, the actual device assignment happens. If an ESXi host fails, vSphere High Availability restarts the VMs on the remaining ESXi hosts within the cluster. If vCenter runs, HA relies on DRS to find an appropriate ESXi host. With Dynamic DirectPath IO, AH assists DRS in finding a new host based on the device assignment availability. Workloads automatically restart on the remaining available GPUs without any human intervention. With DirectPath IO, the VM is powered down by HA during an isolation event or has crashed due to an ESXi host failure. However, it will remain powered off, as the VM is confined to running on that specific host due to its static SBDF configuration. Default ESXi GPU Setting Although DirectPath IO and Dynamic DirectPath IO are the brand names we at VMware like to use in most public-facing collateral, most of the UI uses the term passthrough as the name of the overarching technology. (If we poke around with the esxcli, we also see Passthru.) But the two DirectPath IO types are distinguished at VM creation time as “Access Type”. A freshly installed ESXi OS does not automatically configure the GPU as a passthrough device. If you select the accelerated ESXi host in the inventory view and click on the configure menu, and click on PCIe Devices. Graphics shows that the GPU device is set to Basics Graphics Type and in a Shared configuration. It is the state the device should be in before configuring either any DirectPath IO type or NVIDIA vGPU functionality. You can verify these settings via the CLI: esxcli graphics device list esxcli graphics host get Enable Passthrough Once we know the GPU device is in its default state, we can go back to the PCI Devices overview, select GPU Device, and click “Toggle Passthrough.” The UI reports that the GPU device has enabled Passthrough. Now, the UI list the active type of the GPU device as Direct. Keep the Graphics Device Settings set to shared (NVIDIA vGPU devices use Shared Direct). You can verify these settings via the CLI. esxcli graphics device list esxcli graphics host get Add Hardware Label to PCI Passthrough Device Select the accelerated ESXi host in the inventory view, click on the configure menu, click on PCIe Devices, select Passthrough-enabled Devices, and select the GPU device. Click on the “Hardware Label” menu option and provide a custom label for the device. For example, “Training.” Click on OK when finished. You can verify the label via the CLI with the following command: esxcli hardware pci list | grep NVIDIA -B 6 -A 32 Create VM with GPU Passthrough using Dynamic DirectPath IO To create a VM with a GPU assigned using Dynamic DirectPath IO, VM level 17 is required. The following table lists the available vSphere functionality for VMs that have a Directpath IO and Dynamic DirectPath IO device associated with them. Functionality DirectPath IO Dynamic DirectPath IO Failover HA No Yes Initial Placement DRS No Yes Load Balance DRS No No vMotion No No Host Maintenance Mode Shutdown VM and Reconfigure Cold Migration Snapshot No No Suspend and Resume No No Fractional GPUs No No TKGs VMClass Support No Yes To associate a GPU device using Dynamic DirectPath IO, open the VM configuration, click “Add New Device,” and select “PCI Device.” Select the appropriate GPU Device and click on Select. The UI shows the new PCI device using Dynamic DirectPath IO in the VM Settings. Requirement Notes Supported 64-bits Operating System Reserve all guest memory (automatically set in vSphere 8) EFI firmware Boot option Advanced Settings pciPassthru.set.usebitMMIO = true pciPassthru.64bitMMIOSizeGB = (size in GB) The Summary page of the VM lists the ESXi host and the PCIe Device. However, the UI shows no associated VMs connected to the GPU at the Host view. Two commands are available in the CLI. You can verify if a VM is associated with the GPU device with the following command: esxcli graphics device list The following command list the associated VM: esxcli graphics vm list The following articles will cover NVAIE vGPU driver installation on ESXi and TKGs Other articles in this series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: #47 - How VMware accelerates customers achieving their net zero carbon emissions goal URL: https://frankdenneman.ai/2023-05-30-47-how-vmware-accelerates-customers-achieving-their-net-zero-carbon-emissions-goal/ Date: 2023-05-30 In episode 047, we spoke with Varghese Philipose about VMware’s sustainability efforts and how they help our customers meet their sustainability goals. Features like the green score help many of our customers understand how they can lower their carbon emissions and hopefully reach net zero. Topics discussed: Creating sustainability dashboards - https://blogs.vmware.com/management/2019/06/sustainability-dashboards-in-vrealize-operations-find-how-much-did-you-contribute-to-a-greener-planet.html Sustainability dashboards in VROps 8.6 - https://blogs.vmware.com/management/2021/10/sustainability-dashboards-in-vrealize-operations-8-6.html VMware Green Score - https://blogs.vmware.com/management/2022/11/vmware-green-score-in-aria-operations-formerly-vrealize-operations.html Intrinsically green - https://news.vmware.com/esg/intrinsically-evergreen-vmware-earth-day-2023 Customer success story - https://blogs.vmware.com/customer-experience-and-success/2023/04/tam-partnerships-make-customers-the-hero.html Follow the podcast on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings URL: https://frankdenneman.ai/2023-05-30-vsphere-ml-accelerator-spectrum-deep-dive-esxi-host-bios-vm-and-vcenter-settings/ Date: 2023-05-30 To deploy a virtual machine with a vGPU, whether a TKG worker node or a regular VM, you must enable some ESXi host-level and VM-level settings. All these settings are related to the isolation of GPU resources and memory-mapped I/O (MMIO) and the ability of the (v)CPU to engage with the GPU using native CPU instructions. MMIO provides the most consistent high performance possible. By default, vSphere assigns a MMIO region (an address range, not actual memory pages) of 32GB to each VM. However, modern GPUs are ever more demanding and introduce new technologies requiring the ESXi Host, VM, and GPU settings to be in sync. This article shows why you need to configure these settings, but let’s start with an overview of the required settings. Component Requirements vSphere Functionality Notes Physical ESXi host Must have Intel VT-d or AMD I/O VT enabled in the BIOS Passthrough & vGPU Must have SR-IOV enabled in the BIOS vGPU MIG Enable on Ampere & Hopper GPUs Must have Memory Mapping Above 4G enabled in the BIOS Passthrough & vGPU Not applicable for NVIDIA T4 Must use a supported 64-bits OS Passthrough & vGPU Must be configured with EFI firmware Boot option Passthrough & vGPU Must reserve all guest memory Passthrough & vGPU pciPassthru.set.usebitMMIO = true pciPassthru.64bitMMIOSizeGB = xxx * Passthrough & vGPU Not applicable for NVIDIA T4 Set automatically for TKG worker nodes Must be configured with Advanced Setting vgpu.hotmigrate.enabled vGPU * Calculation follows in the article Memory Management Basics Before diving into each requirement’s details, we should revisit some of the memory management basics. In an ESXi host, there are three layers of memory. The guest virtual memory (the memory available at the application level of a VM) The guest physical memory (the memory available to operating systems running on VMs) The host physical memory (the memory available to the ESXi hypervisor from the physical hosts) The CPU uses the memory management unit (MMU) to translate virtual addresses to physical addresses. A GPU exposes device addresses to control and use the resources on the device. The IOMMU is used to translate IO virtual addresses to physical addresses. From the view of the application running inside the virtual machine, the ESXi hypervisor adds an extra level of address translation that maps the guest physical address to the host physical address. With direct assigning a device to a VM, the native driver running in the guest OS controls the GPU and only “sees” the guest’s physical memory. If an application would directly perform a direct memory access (DMA) to the memory address of a GPU device, it would fail as the VMkernel remaps the virtual machine memory addresses. The Input-Output Memory Management Unit (IOMMU) handles this remapping, allowing native GPU device drivers to be used in a virtual machine by the guest operating system. Let’s review the requirements in more detail. Physical Host Settings Intel VT-D and AMD I/O It is required to enable VT-D in the ESXi host BIOS for both passthrough-enabled GPUs as well as NVIDIA GPUs. In 2006 Intel introduced Intel Virtualization Technology for Directed I/O (Intel VT-d) architecture, an I/O memory management unit (IOMMU). One of the key features of the IOMMU is providing DMA isolation, allowing the VMkernel to assign devices to specific virtual machines directly. Complete isolation of hardware resources while providing a direct path and reducing overhead typically associated with software emulation. The left part of the diagram is outdated technology, which succeeded in vSphere by VT-D. In AMD systems, this feature is called AMD-IO Virtualization Technology (previously called AMD IOMMU). Please note that VT-D is a sub-feature of the Intel Virtualization Technology (Intel VT) and AMD Virtualization (AMD-V). Enabling Virtualization Technology in the BIOS should enable all Intel VT sub-features, such as VT-D. You can verify if Intel VT-d or AMD-V is enabled in the BIOS by running the following command in the shell of ESXi (requires root access to an SSH session) esxcfg-info|grep "\----\HV Support" If the command returns the value 3, it indicates that VT or AMD-V is enabled in the BIOS and can be used. If it returns the value of 2, it indicates that the CPU is VT/D or AMD-V is supported by the CPU but is currently not enabled in the BIOS. If it returns 0 or 1, it’s time to ask someone to acquire some budget for new server hardware. :) For more info about status 0 or 1, visit VMware KB article 1011712. Single Root I/O Virtualization It is required to enable Single Root I/O Virtualization (SR-IOV) in the ESXi host BIOS for only NVIDIA Multi-Instance GPUs (vGPU MIG). Single Root I/O Virtualization (SR-IOV) is sometimes called Global SR-IOV in the BIOS. SR-IOV permits a physical GPU to partition and isolates its resources, allowing it to appear as multiple separate physical devices to the ESXi host. SR-IOV uses physical functions (PFs) and virtual functions (VFs) to manage global functions for the SR-IOV devices. The PF handles the functions that control the physical card. The PF is not tied to any virtual machine. Global functions are responsible for initializing and configuring the physical GPU, moving data in and out of the device, and managing resources such as memory allocation and Quality of Service (QoS) policies. VFs are associated with the virtual machine. They have their own PCI configuration space and complete IOMMU protection for the VM, I/O queues, interrupts, and memory resources. The number of virtual functions provided to the VMkernel depends on the device. The VMkernel manages the allocation and configuration of the vGPU device, while the PF handles the initialization and management of the underlying hardware resources. Unlike NICs, GPUs cannot directly be exposed to VMs using SR-IOV alone. NVIDIA vGPU MIG technology uses SR-IOV as an underlying technology to partition its physical GPU devices and present them as individual smaller vGPU devices. Additionally, ESXi requires VT-d to be enabled to properly configure and manage virtual functions associated with a physical NIC. Without VT-d enabled, SR-IOV could not provide the necessary isolation and security between virtual functions and could potentially cause conflicts or other issues with the physical GPU. NVIDIA requires enablement of SR-IOV in the BIOS to have the NVIDIA T4 to work properly. T4 GPUs offer only time-sliced GPU functionality. Memory Mapped I/O in Detail CPU cores execute instructions. Two main instruction categories are reading and writing to system memory (RAM) and reading and writing to I/O devices such as network cards or GPUs. Modern systems apply a memory-mapped I/O (MMIO) method; in this system, the processor does not know the difference between its system memory and memory from I/O devices. If the processor needs to read into a particular location in RAM, it can just figure out its address from the memory map and read and write from it. But what about the memory from an I/O device? If the CPU core executes an instruction that requires reading memory from the GPU, then the CPU will send a transaction to its system agent. The system agent identifies the I/O transaction and routes it to an address range designated for I/O instructions in the memory system range called the MMIO space. The MMIO contains memory mappings of the GPU registers. The CPU uses these mappings to access the memory of the GPU directly. The processor does not know whether it reads its internal memory or generates an I/O instruction to a PCIe device. The processor only accesses a single memory map. So this is why it’s called memory-mapped I/O. Let’s dig deeper into this statement to understand the fundamental role of the MMIO space. It’s important to know that the MMIO region is not used to store data but for accessing, configuring, and controlling GPU operations. To interact with the GPU, the CPU can read from and write to the GPU’s registers, mapped into the system’s memory address space through MMIO. The MMIO space points towards the MMIO hardware registers on the GPU device. These memory-mapped I/O hardware registers on the GPU are also known as BARs, Base Address Registers. Mapping the GPU BARs into the system’s physical address space provides two significant benefits. One, the CPU can access them through the same kind of instructions used for memory, not having to deal with a different method of interaction; two, the CPU can directly interact with the GPU without going through any virtual memory management layers. Both provide tremendous performance benefits. The CPU can control the GPU via the BARs, such as setting up input and output buffers, launching computation kernels on the GPU, initiating data transfers, monitoring the device status, regulating power management, and performing error handling. The GPU maintains page tables to translate a GPU virtual address to a GPU physical address and a host physical memory address. Let’s use a Host-to-Device memory transfer operation as an example, the NVIDIA technical term for loading a data set into the GPU. The system relies on direct memory access (DMA) to move large amounts of data between the system and GPU memory. The native driver in the guest OS controls the GPU and issues a DMA request. DMA is very useful, as the CPU cannot keep up with the data transfer rate of modern GPUs. Without DMA, a CPU uses programmed I/O, occupying the CPU core for the entire duration of the read or write operation, and is thus unavailable to perform other work. With DMA, the CPU first initiates the transfer. It does other operations while the transfer is in progress, and it finally receives an interrupt from the DMA controller when the operation is done. The MMIO space for a VM is outside its VM memory configuration (guest physical memory mapped into host physical memory) as it is “device memory”. It is exclusively used for communication between the CPU and GPU and only for that configuration - VM and passthrough device. When the application in the user space is issuing a data transfer, the communication library, or the native GPU driver, determines the virtual memory address and size of the data set and issues a data request to the GPU. The GPU initiates a DMA request, and the CPU uses the MMIO space to set up the transfer by writing the necessary configuration data to the GPU MMIO registers to specify the source and destination addresses of the data transfer. The GPU has page tables which contain page tables of the host system memory and the frame buffer capacity. “Frame buffer” is a GPU terminology for onboard GPU DRAM, a remnant of the times when GPUs were actually used to generate graphical images on a screen ;) As we use reserved memory on the host side, these page addresses do not change, allowing the GPU to cache the host’s physical memory addresses. When the GPU is all set up and configured to receive the data, the GPU kicks off a DMA transfer and copies the data between the host’s physical memory and GPU memory without involving the CPU in the transfer. Please note that MMIO space is a separate entity in the host physical memory. Assigning an MMIO space does not consume any memory resources from the VM memory pool. Let’s look at how the MMIO space is configured in an X86 system. Memory Mapping Above 4G It is required to enable the setting “Memory mapping above 4G”, often called “above 4G decoding”, “PCI Express 64-Bit BAR Support,” or “64-Bit IOMMU Mapping.” This requirement is because storing the MMIO space above 4 GB can be accessed by 64-bit operating systems without conflicts. And to understand the 4GB threshold, we have to look at the default behavior of x86 systems. At boot time, the BIOS assigns an MMIO space for PCIe devices. It discovers the GPU memory size and its matching MMIO space request and assigns a memory address range from the MMIO space. By default, the system carves out a part for the I/O address space in the first 32 bits of the address space. Because it’s in the first 4 gigabytes of the system memory address range, it is why this region is called MMIO-low or “MMIO below 4G”. The BAR size of the GPU impacts the MMIO space at the CPU side, and the size of a BAR determines the amount of allocated memory available for communication purposes. Suppose the GPU requires more than 256 MB to function. In that case, it has to incorporate multiple bars during its operations, which typically increases complexity, resulting in additional overhead and impacting performance negatively. Sometimes a GPU requires contiguous memory space, and a BAR size limit of 256 MB can prevent the device from being used. X86 64-bit architectures can address much larger address spaces. However, by default, most server hardware is still configured to work correctly with X86 32-bit systems. By enabling the system BIOS setting “Memory mapping above 4G”, the system can create an MMIO space beyond the 4G threshold and has the following benefits: It allows the system to generate a larger MMIO space to map, for example, the entire BAR1 in the MMIO space. BAR1 maps the GPU device memory so the CPU can access it directly. Enabling “Above 4G Mapping” can help reduce memory fragmentation by providing a larger contiguous address space, which can help improve system stability and performance. Virtual Machine Settings 64-Bit Guest Operating System To enjoy your GPU’s memory capacity, you require a guest operating system with a physical address limit that can contain that memory capacity. A 32-bit OS can maximally address 4 GB of memory, and 64-bit has a theoretical limit of 16 million terabytes (16,777,216TB). In summary, a 64-bit operating system is necessary for modern GPUs because it allows for more significant amounts of memory to be addressed, which is critical for their performance. This is why the NVIDIA Driver installed in the guest OS only supports Windows X86_64 operating systems and Linux 64-bit distributions. Unified Extensible Firmware Interface Unified Extensible Firmware Interface (UEFI), or as it’s called in the vSphere UI, the “EFI Firmware Boot option,” is the replacement for the older BIOS firmware used to boot a computer operating system. Besides many advantages, like faster boot times, improved security (secure boot), and better compatibility with modern hardware, it supports MMIO. The VMware recommendation is to enable EFI for GPUs with 16GB and more. The reason is because of their BAR size. NVIDIA GPUs present three BARs to the system. BAR0, BAR1, and BAR3. Let’s compare an NVIDIA T4 with 16GB to an NVIDIA A100 with 40GB. BAR address (Physical Function) T4 A100 (40GB) BAR0 16 MB 16 MB BAR1 256 MB 64 GB BAR2 32 MB 32 MB BAR0 is the card’s main control space, allowing control of all the engines and spaces of the GPU. NVIDIA uses a standard size for BAR0 throughout its GPU lineup. The T4, A2, V100, A30, A100 40GB, A100 80GB, and the new H100 all have a BAR0 size of 16 MB. The BAR uses 32-bit addressing for compatibility reasons, as it contains the GPU id information and the master interrupt control. Now this is where it becomes interesting. BAR1 maps the frame buffer. Whether to use a BIOS or an EFI firmware depends on the size of BAR1, not on the total amount of frame buffer the GPU has. In short, if the GPU has a BAR1 size exceeding 256 MB, you must configure the VM with an EFI firmware. That means that if you use an NVIDIA T4, you could use the classic BIOS, but if you just got that shiny new A2, you must use an EFI firmware for the VM, even though both GPU devices have a total memory capacity of 16 GB. Device T4 A2 Memory capacity 16 GB 16 GB BAR1 size 256 MB 16 GB As the memory-mapped I/O part mentions, every system has an MMIO below the 4 GB region. The system maps BARs with a size of 256 MB in this region, and the BIOS firmware supports this. Anything larger than 256 MB and you want to switch over to EFI. Please remember that EFI is the better choice of the two regardless of BAR sizes and that you cannot change the firmware once the guest OS is installed. Changing it from BIOS to EFI requires a reinstallation of the guest OS. I recommend saving yourself a lot of time by configuring your templates with the EFI firmware. Please note that the BAR (1) sizes are independent of the actual frame buffer size of the GPU. The best method to determine this is by reading out the BAR size and comparing it to the device’s memory capacity. By default, most modern GPUs use a 64-bit decoder for addressing. You can request the size of the BAR1 in vSphere via VSI Shell (not supported, so don’t generate any support tickets based on your findings). In that case, you will notice that the A100 BAR1 has an address range of 64 GB, while the physically available memory capacity is 40 GB. However, ultimately it’s a combination of the device and driver that determines what the guest OS detects. Many drivers use a 256 MB BAR1 aperture for backward compatibility reasons. This aperture acts as a window into the much larger device memory. This removes the requirement of contiguous access to the device memory. However, if SR-IOV is used, a VF has contiguous access to its own isolated VF memory space (typically smaller than device memory). If I load the datacenter driver in the VMkernel and run the nvidia-smi -q command, it shows a BAR1 aperture size of 4 GB. BAR3 is another control space primarily used by kernel processes Reserve all guest memory (All locked) To use a passthrough GPU or vGPU, vSphere requires a VM memory to be protected by a reservation. Memory reservations protect virtual machine memory pages from being swapped out or ballooned. The reservation is needed to fix all the virtual machine memory at power on, and the ESXi memory scheduler cannot move or reclaim it during memory pressure moments. As mentioned in the “Memory Mapped I/O in detail,” data is copied using DMA and is performed by the GPU device. It uses the host’s physical addresses to access these pages to get the data from the system memory into the GPU device. If, during the data transfer, the ESXi host is pushed into an overcommitted state, it might select those data set pages to swap out or balloon. That situation would cause a page fault at the ESXi host level, but due to IOMMU requirements, we cannot service those requests in flight. In other words, we cannot restart an IO operation from a passthrough device and must ensure the host’s physical page is at the position the GPU expects. A memory reservation “pins” that page to that physical memory address to ensure no page faults happen during DMA operations. As the VM MMIO space is considered device memory, it falls in the virtual machine overhead memory category and is automatically protected by a memory reservation. As mentioned, VT-D records that host physical memory regions are mapped to which GPUs, allowing it to control access to those memory locations based on which I/O device requests access. VT-d creates DMA isolation by restricting access to these MMIO regions or, as they are called in DMA terminology, protection domains. This mechanism works both ways, it isolates the device and restricts other VMs from accessing the assigned GPU, but due to its address-translation tables, it keeps it from accessing other VMs’ memory as well. In vSphere 8, a GPU VM is automatically configured with the option “Reserve all guest memory (All locked)”. Advanced Configuration Parameters If the default 32GB MMIO space is not sufficient, set the following two advanced configuration parameters: pciPassthru.set.usebitMMIO = true pciPassthru.64bitMMIOSizeGB = xxx The setting pciPassthru.set.usebitMMIO = true enables 64-bit MMIO. The setting “pciPassthru.64bitMMIOSizeGB =” specifies the size of the MMIO region for the entire VM. That means if you assign multiple GPUs to a single VM, you must calculate to total required MMIO space for that virtual machine to operate correctly. A popular method is to use the frame buffer size (GPU memory capacity), round it up to a power of two, use the next power of two values, and use that value as the MMIO size. Let’s use an A100 40 GB as an example. The frame buffer capacity is 40 GB. Rounding it up would result in 64 GB, then using the next power of two values would result in a 128 GB MMIO space. Until the 15.0 GRID documentation, NVIDIA used to list the recommended MMIO Size. It aligns with this calculation method. If you assign two A100 40 GBs to one VM, you should assign a value of 256 GB as the MMIO Size. But why is this necessary? If you have a 40 GB card, why do you need more than 40 GB of MMIO? If you need more, why isn’t 64GB enough? Why is 128 GB required? Let’s look deeper into the PCIe BAR structure in the configuration space of the GPU. A GPU config space contains six BARs with a 32-bit addressable space. Each base register is 32-bits wide and can be mapped anywhere in the 32-bit memory space. Two BARs are combined to provide a 64-bit memory space. Modern GPUs expose multiple 64-bit BARs. The BIOS determines the size. How this works exceeds the depth of this deep dive, Sarayhy Jayakumar explains it very well in the video “System Architecture 10 - PCIe MMIO Resource Assignment.” What is essential to know is that the MMIO space for a BAR has to be naturally aligned. The concept of a “naturally aligned MMIO space” refers to the idea that these memory addresses should be allocated in a way that is efficient for the device’s data access patterns. That means for a 32-bit BAR, the data is stored in four consecutive bytes, and the first byte lies on a 4-byte boundary, while a 64-bit BAR uses an 8-byte boundary, and the first byte lies on an 8-byte boundary. If we take a closer look at an a100 40 GB, it exposes three memory-mapped BARs to the system. BAR0 acts as the config space for the GPU is a 32-bit addressable BAR, and is 16 MB. BAR1 is mapped to the frame buffer. It is a 64-bit addressable BAR and consumes two base address registers in the PCIe configuration space of the GPU. That is why the next detectable BAR is listed as BAR3, as BAR1 consumes BAR1 and BAR2. The combined BAR1 typically requires the largest address space. In the case of the A100 40 GB, it is 64 GB. The role of BAR3 is device-specific. It is a 64-bit addressable BAR and is 32 MB in the case of the A100 40 GB. Most of the time, it’s debug or IO space. As a result, we need to combine these 32-bit and 64-bit BARs into the MMIO space available for a virtual machine and naturally align them. If we add up the address space requirement, it’s 16MB + 64 GB + 32 MB = 64 GB and a little more. To ensure the system can align them perfectly, you round it up to the next power of two, 128 GB. But I think most admins and architects will wonder, how much overhead does the MMIO space generate? Luckily, the MMIO space of an A100 40 GB is not consuming 128 GB after setting the “pciPassthru.64bitMMIOSizeGB =128” advanced parameter. As it lives outside the VM memory capacity, you can quickly check its overhead by monitoring the VM overhead reservation. Let’s use an A100 40 GB in this MMIO size overhead experiment. If we check the NVIDIA recommendation chart, it shows an MMIO size of 128 GB. Model Memory Size BAR1 Size (PF) MMIO Size - Single GPU MMIO Size - Two GPUs V100 16 GB / 32 GB 16 GB / 32 GB 64 GB, all variants 128 GB A30 24 GB 32 GB 64 GB 128 GB A100 40 GB 64 GB 128 GB 256 GB A100 80 GB 128 GB 256 GB 512 GB H100 80 GB 128 GB 256 GB 512 GB The VM is configured with 128 GB of memory. This memory configuration should be enough to keep a data set in system memory that can fill up the entire frame buffer of the GPU. Before setting the MMIO space and assigning the GPU as a passthrough device, the overhead memory consumption of the virtual machine is 773.91 MB. You can check that by selecting the VM in vCenter, going to the Monitor tab, and selecting utilization or monitoring the memory consumption using ESXTOP. The VM is configured with an MMIO space of 128 GB. If you only assign the MMIO space but don’t assign a GPU, the VM overhead does not change as there is no communication happening via the MMIO space. It will only become active once a GPU is assigned to the VM. The GPU device is assigned, and if you monitor the VM memory consumption, you notice that the memory overhead of the VM is increased to 856.82 MB. The 128GB MMIO space consumes 82.91 MB. Let’s go crazy and increase the MMIO space to 512GB. Going from an MMIO space of 128GB to 512GB increases the VM overhead to 870.94MB, which results in an increment of ~14MB. An adequate-sized MMIO space is vital to performance. Looking at the minimal overhead an MMIO space introduces, I recommend not to size the MMIO space too conservatively. TKGS Worker Nodes We have to do two things because we cannot predict how many GPUs and which GPU types are attached to TKGS GPU-enabled worker nodes. Enable the MMIO space automatically to continue a seamless developer experience and set an adequate MMIO space for a worker node. By default, an 512 GB MMIO space is automatically configured, or to state it differently, it provides enough space for four A100 40 GB GPUs per TKGS worker node. If this is not enough space for your configuration, we have a way to change that, but this is not a developer-facing option. Let me know in the comments below if you foresee any challenges by not exposing this option. Enable vGPU Hot Migration at vCenter Level One of the primary benefits of vGPU over (Dynamic) Direct Path I/O is its capability of live migration of vGPU-enabled workload. Before you can vMotion a VM with a vGPU attached to it, you need to tick the checkbox of the vgpu.hotmigrate.enabled setting in the Advanced vCenter Server Settings section of your vCenter. In vSphere 7 and 8, the setting is already present and only needs to be ticked to get enabled. Other articles in this series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive –NVIDIA AI Enterprise Suite URL: https://frankdenneman.ai/2023-05-23-vsphere-ml-accelerator-spectrum-deep-dive-nvidia-ai-enterprise-suite/ Date: 2023-05-23 vSphere allows assigning GPU devices to a VM using VMware’s (Dynamic) Direct Path I/O technology (Passthru) or NVIDIA’s vGPU technology. The NVIDIA vGPU technology is a core part of the NVIDIA AI Enterprise suite (NVAIE). NVAIE is more than just the vGPU driver. It’s a complete technology stack that allows data scientists to run an end-to-end workflow on certified accelerated infrastructure. Let’s look at what NVAIE offers and how it works under the cover. The operators, VI admins, and architects facilitate the technology stack while the data science team and developers consume it. Most elements can be consumed via self-service. However, there is one place in the technology stack, NVIDIA Magnum IO, where the expertise of both roles (facilitators and consumers) come together, and their joint effort produces an efficient and optimized distributed training solution. Accelerated Infrastructure As mentioned before, NVAIE is more than just the vGPU driver. It offers an engineered solution that provides an end-to-end solution in a building block fashion. It allows for a repeatable certified infrastructure deployable at the edge or in your on-prem data center. Server vendors like HPE and Dell offer accelerated servers with various GPU devices. NVIDIA qualifies and certifies specific enterprise-class servers to ensure the server can accelerate the application properly. There are three types of validation: Validation Type Description Qualified Servers A server that has been qualified for a particular NVIDIA GPU has undergone thermal, mechanical, power, and signal integrity qualification to ensure that the GPU is fully functional in that server design. Servers in qualified configurations are supported for production use. NGC-Ready Servers NGC-Ready servers consist of NVIDIA GPUs installed in qualified enterprise-class servers that have passed extensive tests that validate their ability to deliver high performance for NGC containers. NVIDIA-Certified Systems NVIDIA-Certified Systems consist of NVIDIA GPUs and networking installed in qualified enterprise-class servers that have passed a set of certification tests that validate the best system configurations for a wide range of workloads and for manageability, scalability, and security. NVIDIA has expanded the NVIDIA-Certified Systems program beyond servers designed for the data center, including GPU-powered workstations, high-density VDI systems, and Edge devices. During the certification process, NVIDIA completes a series of functional and performance tests on systems for their intended use case. With edge systems, NVIDIA runs the following tests: Single and multi-GPU Deep Learning training performance using TensorFlow and PyTorch High volume, low latency inference using NVIDIA TensorRT and TRITON GPU-Accelerated Data Analytics & Machine Learning using RAPIDS Application development using the NVIDIA CUDA Toolkit and the NVIDIA HPC SDK Certified systems for the data center are tested both as single nodes and in a 2-node configuration. NVIDIA executes the following tests: Multi-Node Deep Learning training performance High bandwidth, low latency networking, and accelerated packet processing System-level security and hardware-based key management The NVIDIA Qualified Server Catalog provides an easy overview of all the server models and their specific configuration and NVIDIA validation types. It offers the ability to export the table in a PDF and Excel format at the bottom of the page. Still, I like the available filter system to drill down to the exact specification that suits your workload needs. The GPU device differentiators article can help you select the GPUs that fit your workloads and deploy location. The only distinction the qualified server catalog doesn’t appear to make is whether the system is classified as a data center or an edge system and thus receives a different functional and performance test pattern. The NVIDIA-Certified Systems web page lists the recent data center server and edge servers. A PDF is also available for download. Existing active servers in the data center can be expanded. Ensure your server vendor lists your selected server type as a GPU-ready node. And don’t forget to order the power cables along with the GPU device. Enterprise Platform Multiple variations of vSphere implementations support NVAIE. The data science team often needs virtual machines to run particular platforms, like Ray, or just native docker images without any need for orchestration. However, if container orchestration is needed, the operation team can opt for VMware Tanzu Kubernetes Grid Services (TKGS) or Red Hat Open Shift Container Platform (OCP). TKGs offer vSphere integrated namespaces and VMclasses, to further abstract, pool, and isolate accelerated infrastructure, while providing self-service provisioning functionality to the data science team. Additionally, the VM service allows data scientists to deploy VMs in their assigned namespace while using a Kubectl API framework. It allows the data science team to engage with the platform using its native user interface. VCF workload domains allow organizations to further pool and abstract accelerated infrastructure at the SDDC level. If your organization has standardized on Red Hat OpenShift, vSphere is more than happy to run that as well. NVIDIA supports NVAIE with Red Hat OCP on vSphere and provides all the vGPU functionality. Ensure you download the correct vGPU operator. Infrastructure Optimization vGPU Driver The GPU device requires a driver to interact with the rest of the system. If (Dynamic) Direct Path I/O (passthru) is used, the driver is installed inside the guest operating system. For this configuration, NVIDIA releases a guest OS driver. No NVIDIA driver is required at the vSphere VMkernel layer. When using NVAIE, you need a vGPU software license, and the drivers used by the NVAIE suite are available at the NVIDIA enterprise software download portal. This source is only available to registered enterprise customers. The NVIDIA Application Hub portal offers all the software packages available. What is important to note, and the cause of many troubleshooting hours, is that NVIDIA provides two different kinds of vSphere Installation Bundles (VIBs). The “Complete vGPU Package for vSphere 8.0 including supported guest drivers” contains the regular graphics host VIB. You should NOT download this package if you want to run GPU-accelerated applications. We are interested in the “NVIDIA AI Enterprise 3.1 Software Package for VMware vSphere 8.0” package. This package contains both the NVD-AIE Host VIB and the compatible guest os driver. The easiest and failsafe method of downloading the correct package is to select the NVAIE option in the Product Family. You can finetune the results by only selecting the vSphere platform version you are running in your environment. What’s in the package? The extracted AIE zip file screenshot shows that a vGPU Software release contains the Host VIB, the NVIDIA Windows driver, and the NVIDIA Linux driver. NVIDIA uses the term NVIDIA Virtual GPU Manager, what we like to call the ESXi host VIB. There are some compatibility requirements between the host and guest driver, hence the reason to package them together. The best experience is to keep both drivers in lockstep when updating the ESXi host with a new vGPU driver release. But I can imagine that it’s simply not doable for some workloads, and the operating team would prefer to delay the outage required for the guest OS upgrade until later. Luckily, NVIDIA has relaxed this requirement and now supports host and guest drivers from major release branches (15. x) and previous branches. Suppose the combination is used where the guest VM drivers are from the previous branches. In that case, the combination supports only the features, hardware, and software (including guest OSes) supported on both releases. According to the vGPU software documentation, the host driver 15.0 through 15.2 is compatible with the guest drivers of the 14.x release. A future article in this series shows how to correctly configure a VM in passthrough mode or with a vGPU profile. NVIDIA Magnum IO The name Magnum IO is derived from multi-GPU, multi-node input/output. NVIDIA likes to explain Magnum IO as a collection of technologies related to data at rest, data on the move, and data at work for the IO subsystem of the data center. It divides into four categories: Network IO, In-network compute, Storage IO, and IO management. I won’t cover IO management as they focus on bare-metal implementations. All these acceleration technologies focus on optimizing distributed training. The data science team deploys most of these components in their preferred runtime (VM, container). However, it’s necessary to understand the infrastructure topology the data science team wants to leverage technologies like GPUDirect RDMA, NCCL, SHARP, and GPUDirect Storage. Typically this requires the involvement of the virtual infrastructure team. The previous parts in this series help the infrastructure team to have a basic understanding of distributed training. In-Network Compute Technology Description MPI Tag Matching MPI Tag Matching reduces MPI communication time on NVIDIA Mellanox Infiniband adapters. SHARP Scalable Hierarchical Aggregation and Reduction Protocol (SHARP) offloads collective communication operations from the CPU to the network and eliminates the need to send data multiple times between nodes. SHARP support was added to NCCL to offload all-reduce collective operations into the network fabric. Additionally, SHARP accelerators are present in NVSwitch v3. Instead of distributing the data to each GPU and having the GPUs perform the calculations, they send their data to SHARP accelerators inside the NVSwitch. The accelerators then perform the calculations and then send the results back. This results in 2_N_+2 operations, or approximately halving the number of read/write operations needed to perform the all-reduce calculation. Network IO The Network IO stack contains IO acceleration technologies that bypass the kernel and CPU to reduce overhead and enable and optimizes direct data transfers between GPUs and the NVLink, Infiniband, RDMA-based, or Ethernet-connected network device. The components included in the Network IO stack are: Technology Description GPUDirect RDMA GPUDirect RDMA enables a direct path for data exchange between the GPU and a NIC. It allows for direct communication between NVIDIA GPUs in different ESXi hosts. NCCL NVIDIA Collective Communications Library (NCCL) is a library that contains inter-GPU communication primitives optimizing distributed training on multi-GPU multi-node systems. NVSHMEM NVIDIA Symmetrical Hierarchical Memory (NVSHMEM) creates a global address space for data that spans the memory of multiple GPUs. NVSHMEM-enabled CUDA uses asynchronous, GPU-initiated data transfers, thereby reducing critical-path latencies and eliminating synchronization overheads between the CPU and the GPU while scaling. HPC-X for MPI NVIDIA HPC-X for MPI offloads collective communication from Message Passing Interface (MPI) onto NVIDIA Quantum InfiniBand networking hardware. UCX Unified Communication X (UCX) is an open-source communication framework that provides GPU-accelerated point-to-point communications, supporting NVLink, PCIe, Ethernet, or Infiniband connections between GPUs. ASAP2 NVIDIA Accelerated Switch and Packet Processing (ASAP2) technology allows SmartNICs and data processing units (DPUs) to offload and accelerate software-defined network operations. DPDK The Data Plane Deployment Kit (DPDK) contains the poll mode driver (PMD) for ConnectX Ethernet adapters, NVIDIA Bluefield-2 SmartNICs (DPUs). Kernel bypass optimizations allow the system to reach 200 GbE throughput on a single NIC port. The article “vSphere ML Accelerator Spectrum Deep Dive for Distributed Training – Multi-GPU” has more info about GPUDirect RDMA, NCCL, and distributed training. Storage IO The Storage IO technologies aim to improve performance in the critical path of data access from local or remote storage devices. Like network IO technologies, improvements are obtained by bypassing the host’s computing resources. In the case of Storage IO, this means CPU and system memory. Technology Description GPUDirect Storage GPUDirect Storage enables a direct data path for direct memory access (DMA) transfers between GPU memory and storage, avoiding ESXi host system CPU and memory involvement. (Local NVMe storage or Remote RDMA storage). NVMe SNAP NVMe SNAP (Software-defined, Network Accelerated Processing) allows Bluefield-2 SmartNICs (DPUs) to present networked flash storage as local NVMe storage. (Not currently supported by vSphere). NVIDIA CUDA-X AI NVIDIA CUDA-X is a CUDA (Compute Unified Device Architecture) platform extension. It includes specialized libraries for domains such as image and video, deep learning, math, and computational lithography. It also contains a collection of partner libraries for various application areas. CUDA-X “feeds” other technology stacks, such as Magnum IO. For example, NCCL and NVSHMEM are developed and maintained by NVIDIA as part of the CUDA-X library. Besides the math libraries, CUDA-X allows for deep learning training and inference. These are: Technology Description cuDNN CUDA Deep Neural Network library (cuDNN) is a GPU-accelerated library of neural network primitives, such as convolutional layers, pool operations, and forward and backward propagation. TensorRT Tensor RunTime (TensorRT) is an inference optimization library and SDK for deep learning inference. It provides optimization techniques that minimize model memory footprint and improve inference speeds. Riva Riva is a GPU-accelerated SDK for developing real-time speech-ai applications, such as automatic speech recognition (ASR) and text-to-speech (TTS). The ASR pipeline converts raw audio to text, and the TTS pipeline converts text to audio. Deepstream SDK DeepStream SDK provides plugins, APIs, and tools to develop and deploy real-time vision AI applications and services incorporating object detection, image processing, and instance segmentation AI models. DALI The Data Loading Library (DALI) accelerates input data preprocessing for deep learning applications. It allows the GPU to accelerate images, videos, and speech decoding and augmenting. NVIDIA Operators NVIDIA uses the GPU and Network operator to automate the drivers, container runtimes, and relevant libraries configuration for GPU and network devices on Kubernetes nodes. The Network Operator automates the management of all the NVIDIA software components needed to provision fast networking, such as RDMA and GPUDirect. The Network operator works together with the GPU operator to enable GPU-Direct RDMA. The GPU operator is open-source and packaged as a helm chart. The GPU operator automates the management of all software components needed to provision GPU. The components are: Technology Description GPU Driver Container The GPU driver container provisions the driver using a container, allowing portability and reproducibility within any environment. A container runtime favors the driver containers over the host drivers. GPU Feature Discovery The GPU Feature Discovery component automatically generates labels for the GPUs available on a worker node. It leverages the Node Feature Discover inside the Kubernetes layer to perform this labeling. It’s automatically enabled in TKGS. During OCP installations, you need to install the NFD Operator. Kubernetes Device Plugin The Kubernetes Device plugin Daemonset automatically exposes the number of GPUs on each node of the Kubernetes cluster, keeps track of the GPU health, and allows running GPU-enabled containers in the Kubernetes cluster. MIG Manager The MIG Manager controller is available on worker nodes that contain MIG-capable GPUs (Ampere, Hopper) NVIDIA Container ToolKit The NVIDIA Container ToolKit allows users to build and run GPU-accelerated containers. The toolkit includes a container runtime library and utilities to automatically configure containers to leverage NVIDIA GPUs. DCGM Monitoring The Data Center GPU Manager toolset manages and monitors GPUs within the Kubernetes cluster. The GPU Operator components deploy within a Kubernetes Guest Cluster via a helm chart. On vSphere, this can be a Red Hat OpenShift Container Platform Cluster or a Tanzu Kubernetes Grid Service Kubernetes guest cluster. A specific vGPU operator is available for each platform. The correct helm repo for TKGS is https://helm.ngc.nvidia.com/nvaie, and the helm repo for OCP is https://helm.ngc.nvidia.com/nvidia. Data Science Development and Deployment Tools The NVIDIA GPU Cloud (NGC) provides AI and data science tools and framework container images to the NVAIE suite. TensorRT is always depicted in this suite by NVIDIA. Since it’s already mentioned and included in the CUDA-X-AI section, I left it out to avoid redundancies within the stack overview. Technology Description RAPIDS The Rapid Acceleration of Data Science (RAPIDS) framework provides and accelerates end-to-end data science and analytics pipelines. The core component of Rapids is the cuDF library, which provides a GPU-accelerated DataFrame data structure similar to Pandas, a popular data manipulation library in Python. TAO Toolkit The Train Adapt Optimize (TAO) Toolkit is a python based AI toolkit for taking purpose-built pre-trained AI models and customizing them with your data. Pytorch\Tensorflow Container Image NGC has done the heavy lifting and provides a prebuilt container with all the necessary libraries validated for compatibility, a heavily underestimated task. It contains CUDA, cuBLAS, cuDNN, NCCL, RAPIDS, DALI, TensorRT, TensorFlow-TensorRT (TF-TRT), or Torch-TensorRT. Triton Inference Server The Triton platform enables deploying and scaling ML models for inference. The Triton Inference Server serves models from one or more model repositories. Compatible file paths are Google Cloud Storage, S3 compatible (local and Amazon), and Azure Storage. NeMo Framework The NeMo (Neural Modules) framework is an end-to-end GPU-accelerated framework for training and deploying transformer-based Large Language Models (LLMs) up to a trillion parameters. In the NeMo framework, you can train different variants of GPT, Bert, and T5 style models. A future article explores the NVIDIA NeMo offering. Other articles in the vSphere ML Accelerator Spectrum Deep Dive vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive - GPU Device Differentiators URL: https://frankdenneman.ai/2023-05-16-vsphere-ml-accelerator-spectrum-deep-dive-gpu-device-differentiators/ Date: 2023-05-16 The two last parts reviewed the capabilities of the platform. vSphere can offer fractional GPUs to Multi-GPU setups, catering to the workload’s needs in every stage of its development life cycle. Let’s look at the features and functionality of each supported GPU device. Currently, the range of supported GPU devices is quite broad. In total, 29 GPU devices are supported, dating back from 2016 to the last release in 2023. A table at the end of the article includes links to each GPUs product brief and their datasheet. Although NVIDIA and VMware form a close partnership, the listed support of devices is not a complete match. This can lead to some interesting questions typically answered with; it should work. But as always, if you want bulletproof support, follow the guides to ensure more leisure time on weekends and nights. VMware HCL and NVAIE Support The first overview shows the GPU device spectrum and if NVAIE supports them. The VMware HCL supports every device listed in this overview, but NVIDIA decided not to put some of the older devices through their NVAIE certification program. As this is a series about Machine Learning, the diagram shows the support of the device and a C-series vGPU type. The VMware compatibility guide has an AI/ML column, listed as Compute, for a specific certification program that tests these capabilities. If the driver offers a C-series type, the device can run GPU-assisted applications; therefore, I’m listing some older GPU devices that customers still use. With some other devices, VMware hasn’t tested the compute capabilities, but NVIDIA has, and therefore there might be some discrepancies between the VMware HCL and NVAIE supportability matrix. For the newer models, the supportability matrix is aligned. Review the table and follow the GPU device HCL page link to view the supported NVIDIA driver version for your vSphere release. The Y axis shows the device Interface type and possible slot consumption. This allows for easy analysis of whether a device is the “right fit” for edge locations. Due to space constraints, single-slot PCIe cards allow for denser or smaller configurations. Although every NVIDIA device supported by NVAIE can provide time-shared fractional GPUs, not all provide spatial MIG functionality. A subdivision is made on the Y-axis to show that distinction. The X-axis represents the GPU memory available per device. It allows for easier selection if you know the workload’s technical requirements. The Ampere A16 is the only device that is listed twice in these overviews. The A16 device uses a dual-slot PCIe interface to offer four distinct GPUs on a single PCB card. The card contains 64GB GPU memory, but vSphere shall report four devices offering 16G of GPU memory. I thought this was the best solution to avoid confusion or remarks that the A16 was omitted, as some architects like to calculate the overall available GPU memory capacity per PCIe slot. NVLink Support If you plan to create a platform that supports distributed training using multi-GPU technology, this overview shows the available and supported NVLinks bandwidth capabilities. Not all GPU devices include NVLink support, and the ones with support can wildly differ. The MIG capability is omitted as MIG technology does not support NVLink. NVIDIA Encoder Support The GPU decodes the video file before running it through an ML model. But it depends on the process following the outcome of the model prediction, whether to encode the video again and replay it to a display. With some models, the action required after, for example, an anomaly detection, is to generate a warning event. But if a human needs to look at the video for verification, a hardware encoder must be available on the GPU. The Q-series vGPU type is required to utilize the encoders. What may surprise most readers is that most high-end datacenter does not have encoders. This can affect the GPU selection process if you want to create isolated media streams at the edge using MIG technology. Other GPU devices might be a better choice or investigate the performance impact of CPU encoding. NVIDIA Decoder Support Every GPU has at least one decoder, but many have more. With MIG, you can assign and isolate decoders to a specific workload. When a GPU is time-sliced, the active workload utilizes all GPU decoders available. Please note that the A16 list has eight decoders, but each distinct GPU on the A16 exposes two decoders to the workload. GPUDirect RDMA Support GPUDirect RDMA is supported on all time-sliced and MIG-backed C-series vGPUs on GPU devices that support single root I/O virtualization (SR-IOV). Please note that Linux is the only supported Guest OS for GPUDirect technology. Unfortunately, MS Windows isn’t supported. Power Consumption When deploying at an edge location, power consumption can be a constraint. This table list the specified power consumption of each GPU device. Supported GPUs Overview The table contains all the GPUs depicted in the diagrams above. Instead of repeating non-descriptive labels like webpage or PDFs, the table shows the GPU release date while linking to its product brief. The label for the datasheet indicates the amount of GPU memory, allowing for easy GPU selection if you want to compare specific GPU devices. Please note that VMware has not conducted C-series vGPU type tests on the device if the HCL Column indicates No. However, the NVIDIA driver does support the C-series vGPU type. Architecture GPU Device HCL/ML Support NVAIE 3.0 Support Product Brief Datasheet Pascal Tesla P100 No No October 2016 16GB Pascal Tesla P6 No No March 2017 16GB Volta Tesla V100 No Yes September 2017 16GB Turing T4 No Yes October 2018 16GB Ampere A2 Yes No November 2021 16GB Pascal P40 No Yes November 2016 24GB Turing RTX 6000 passive No Yes December 2019 24GB Ampere RTX A5000 No Yes April 2021 24GB Ampere RTX A5500 N/A Yes March 2022 24GB Ampere A30 Yes Yes March 2021 24GB Ampere A30X Yes Yes March 2021 24GB Ampere A 10 Yes Yes March 2021 24GB Ada Lovelace L4 Yes Yes March 2023 24GB Volta Tesla V100(S) No Yes March 2018 32GB Ampere A100 (HGX) N/A Yes September 2020 40GB Turing RTX 8000 passive No Yes December 2019 48GB Ampere A40 Yes Yes May 2020 48GB Ampere RTX A6000 No Yes December 2022 48GB Ada Lovelace RTX 6000 Ada N/A Yes December 2022 48GB Ada Lovelace L40 Yes Yes October 2020 48GB Ampere A 16 Yes Yes June 2021 64GB Ampere A100 Yes Yes June 2021 80GB Ampere A100X Yes Yes June 2021 80GB Ampere A100 HGX N/A Yes November 2020 80GB Ada Lovelace H100 Yes Yes September 2022 80GB Other articles in the vSphere ML Accelerator Spectrum Deep Dive vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: #46 - VMware Cloud Flex Compute Tech Preview URL: https://frankdenneman.ai/2023-05-15-46-vmware-cloud-flex-compute-tech-preview/ Date: 2023-05-15 We’re extending the VMware Cloud Services overview series with a tech preview of the VMware Cloud Flex Compute service. Frances Wong shares a lot of interesting use cases and details with us in this episode! In short, VMware Cloud Flex Compute is a new approach to the Enterprise-grade VMware Cloud, but instead of obtaining a full SDDC, it is sliced, diced, sold, and deployed by fractional SDDC increments in the global cloud. Make sure to follow Frances on Twitter (https://twitter.com/frances_wong) to keep up to date with her adventures, and check out the VMware website for more details on the Cloud Flex Compute offering! Additional resources can be found here: Announcement - https://blogs.vmware.com/cloud/2022-08-30-announcing-vmware-cloud-flex-compute/ Deep Dive - https://blogs.vmware.com/cloud/2022-08-30-vmware-cloud-flex-compute-deep-dive/ Early Access Demo - https://vmc.techzone.vmware.com/?share=video2847&title=vmware-cloud-flex-compute-early-access-demo Follow us on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. Last but not least, make sure to hit that subscribe button, rate where ever possible, and share the episode with your friends and colleagues! ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive for Distributed Training - Multi-GPU URL: https://frankdenneman.ai/2023-05-12-vsphere-ml-accelerator-spectrum-deep-dive-for-distributed-training-multi-gpu/ Date: 2023-05-12 The first part of the series reviewed the capabilities of the vSphere platform to assign fractional and full GPU to workloads. This part zooms in on the multi-GPU capabilities of the platform. Let’s review the full spectrum of ML accelerators that vSphere offers today. In vSphere 8.0 Update 1, an ESXi host can assign up to 64 (dynamic) direct path I/O (passthru) full GPU devices to a single VM. In the case of NVIDIA vGPU technology, vSphere supports up to 8 full vGPU devices per ESXi host. All of these GPU devices can be assigned to a single VM. Multi-GPU technology allows the data science team to present as many GPU resources to the training job as possible. When do you need multi-GPU? Let’s look at the user requirements. A data science team’s goal is to create a neural network model that provides the highest level of accuracy (Performance in data science terminology). There are multiple ways to achieve accuracy. One is by processing vast amounts of data. You can push monstrous amounts of data through a (smaller) model, and at one point, the model reaches a certain level of acceptable accuracy (convergence). Another method is to increase the sample (data) efficiency. Do more with less, but if you want to use data more efficiently, you must increase the model size. A larger model can use more complex functions to “describe” the data. In either scenario, you need to increase the compute resources if you push extreme amounts of data or push your datasets through larger models. In essence, Machine Learning scale is a triangle of three factors: data size, model size, and the available compute size. The most popular method of training a neural network is stochastic gradient descent (SGD). Oversimplified, it feeds examples into the network and starts with an initial guess. It trains the network by adjusting its “guesses” gradually. The neural network measures how “wrong” or “right” the guess is and, based on this, calculates a loss. Based on this loss, it adjusts the network’s parameters (weights and biases) and feeds a new set of examples. It repeats this cycle and refines the network until it’s accurate enough. During the training cycle, the neural network processes all the examples in a dataset. This cycle is called an epoch. Typically a complete dataset cannot fit onto the GPU memory. Therefore data scientist splits up the entire dataset into smaller batch sets. The number of training examples in a single batch defines a batch size. An iteration is a complete pass of a batch, sometimes called a step. The number of iterations is how many batches are needed to complete a single epoch. For example, the Imagenet-1K dataset contains 1.28 million images. Well-recommended batch size is 32 images. It will take 1.280.000 / 32 = 40.000 iterations to complete a single epoch of the dataset. Now how fast an epoch completes depends on multiple factors. One crucial factor is data loading, transferring the data from storage into the ESXi host and GPU memory. The other significant latency factor is the communication of gradients to update the parameters after each iteration in distributed training. A training run typically invokes multiple epochs. The model size, typically expressed in the parameter count, is interesting, especially today, where everyone is captivated by Large Language Models (LLMs). Where the AI/ML story mainly revolved around vision AI until a year ago, many organizations are keen to start with LLMs. The chart below shows the growth of parameters of image classification (orange line) and Natural Language Processing (blue line) in state-of-the-art (SOTA) neural network architectures. Although GPT-4 has been released, Microsoft hasn’t announced its parameter count yet, although many indicate that it’s six times larger than GPT-3. (1 Trillion parameters). Why is parameter count so important? We have to look more closely at the training sequence. The article “Training vs. Inference - memory consumption by neural network” explores the memory consumption of parameters, network architecture, and data sets in detail. In short, a GPU has a finite amount of memory capacity. If I loaded a GPT-3 model with 175 Billion parameters using single-precision floating-point (FP32), it would need 700 GB of memory. And that’s just a static model consumption before pushing a single dataset example through. Quoting the paper “Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LM,” “Training GPT-3 with 175 billion parameters would require approximately 288 years with a single V100 NVIDIA GPU.” With huge models, data scientists need to distribute the model across multiple GPUs. Data scientists sometimes prefer pushing more data through a smaller model than using a large model and dealing with model distribution. Regardless of model size, data distribution is the most common method of distributed learning. With this method, the entire model is replicated across multiple GPUs, and the dataset is split up and distributed across the pool of GPUs. Native data distribution modules are available in PyTorch and TensorFlow. With data distribution, the model is intact, but the dataset is split up. But to train the model coherently, the models must receive the result of each GPU’s training iteration. The models need to be trained in a certain lockstep; thus, the communication rate between the GPUs impacts the overall progression of the training job. The faster the GPUs communicate their learnings, the faster the model converges. It is why NVIDIA invests heavily in NVLINK and NVSwitch technology, and vSphere supports these technologies. Let’s look at the training process to understand the benefit of fast interconnects. To make sense of the behavior of distributed training, we need to look at how deep learning training on a single GPU works first. The data set is processed in batches to train a neural network, and we pass the data across the neural network. This process is called the forward pass, and it computes the error. The error indicates how wrong the neural network is as it compares the predicted label to the annotation (the gold-truth label). The next step for the ML framework is to run the backpropagation (backward pass), which runs the error back through the network, producing gradients for each parameter in the neural network. These gradients tell us how to learn from our errors, and the optimizer updates the parameters. And the neural network is ready for the next batch. It’s up to the data scientist to find the correct batch size to utilize as much GPU memory as possible while leaving enough room for the activations of the backward pass. For more detail: Training vs. Inference - memory consumption by a neural network." Now let’s look at the most popular form of distributed training, distributed data parallelism with Multi-GPU architecture utilizing a ring-AllReduce to share gradients optimally. In this scenario, the framework copies a replica of the neural network model to each GPU and splits the dataset across the multiple GPUs. Each GPU runs the forward and backward pass to compute the gradient for the processed batch subset. Now comes the interesting part, the gradients have to be shared across the GPUs as if all the GPUs have processed the complete batch. The most commonly used operation that shares the gradients between GPUs is called Gradient Ring-AllReduce. PyTorch DistributedDataParallel, Horovod, and TensorFlow Mirrored Strategy use this operation to compute the mean of the local gradients on all the GPUs and then update the model with the averaged global gradient. The optimizer updates the models’ parameters and processes the next batch of the data set. The memory consumption of a model gradient mostly depends on the model architecture. It’s challenging to provide an average size of a typical model gradient. Still, a reasonable indication of a gradient size can be inferred from the number of parameters. The more parameters per model to update, the more data must be sent. Bandwidth between GPUs impacts how long it will take to send all this data. As models get larger and larger, so does the gradient size required to update the parameters during training. Larger batches generate larger gradients to update the model parameters in each training step. Let’s use the Bert-Large model as an example. It has 340 million parameters. Gradients use FP32 regardless of the forward pass numerical precision (BFLOAT16, FP16, FP32). As a result, each parameter requires 4 bytes (32 bits) of memory. The total memory required to store the gradient for all the parameters would be 320 million x 4 bytes = 1.36GB of data per iteration per GPU. The Ring-All Reduce method manages that each GPU receives an identical copy of the averaged gradients at the end of the backward pass to ensure that the updates to model parameters are identical. With Ring AllReduce, the GPUs are arranged in a logical ring, and each GPU receives data from its left neighbor and sends data to its right neighbor. The beauty of this ring structure is that each (N) GPU will send and receive values N-1. There are two steps involved, the scatter-reduce and the all-gather step. It would lengthen this article significantly if I covered the finer details of these steps, but what matters is that data is roughly transferred twice. So using the Ring AllReduce, each GPU training the Bert-Large must send and receive about 2.72GB of data per iteration. Using 25Gb Ethernet (providing 3.125 GB/s) 2.72GB *8 = 21.72Gb /25 Gbps = 870 milliseconds per iteration. This delay ramps up quite quickly if you run 30.000 iterations per epoch, and it takes 100 epochs to get the model accurate (convergence). That’s 725 hours or 30 days of latency. Bringing HPC Techniques to Deep Learning and Distributed data-parallel training using Pytorch on AWS are fantastic resources if you want to understand Ring AllReduce better. Different configurations allow ML frameworks to consume multiple GPU devices. Multiple GPUs from a single ESXi host can be assigned to a VM for a single node-multi GPU setup. In a multi-node setup, multiple VMs are active and can consume GPUs from their local ESXi host. With different setups, there are different bandwidth bottlenecks. Coming back to the data-load process, it makes sense to review the bandwidth within the ESXi host to recognize the added benefit of specialized GPU interconnects. Internal host bandwidth ranges from high bandwidth areas to low bandwidth areas. High bandwidth areas are located on the GPU itself, where GPU cores can access High Bandwidth Memory (HBM) between 2 TB/s or 3.35 TB/s, depending on the form factor of the H100. The GPU device connects to the system with a PCIe Gen 5 interconnect, offering 126 GB/s of bandwidth, allowing the GPU to access ESXi host memory to read the data set or write the results of the training job. And suppose the distributed training method uses a multi-node configuration. In that case, the PCIe bus connects to the NIC, and data, such as gradients, are sent across (hopefully) a 25 Gbps connection equal to 3 GB/sec. More complex models require more floating point operations per second (FLOPS) per byte of data. Thus, the combination of GPU processor speed and data loading times introduces an upper bound of the algorithm’s performance. Infra-tech savvy data scientists compute the limitations of the GPU hardware in terms of algorithm performance and visually plot this in a Roofline Model. Helping the data scientist understand which GPU models vSphere supports and how they can be connected to enable distributed training helps you build a successful ML platform. Selecting the correct setup and utilizing dedicated interconnects isolates this noisy neighbor, allowing the ESXi host to run complementary workloads. Let’s look at the different optimized interconnect technologies supported by vSphere for Multi-GPU distributed training. NVIDIA GPUDirect RDMA NVIDIA GPUDirect RDMA (Remote Direct Memory Access) improves the performance of Multi-Node distributed training and is a technology that optimizes the complete path between GPUs in separate ESXi hosts. It provides a direct peer-to-peer data path between the GPU memory directly to and from the Mellanox NIC. It decreases GPU-to-GPU communication latency speeding up the workload. It alleviates the overall overhead of this workload on the ESXi Host as it avoids unnecessary system memory copies (and CPU overhead) by copying data to and from GPU memory. With GPUDirect RDMA, distributed training can now write gradients directly to each GPU input buffer without having the systems copy the gradients to the system memory first before moving it onto the sending NIC or into the receiving GPU. The HPC OCTO team ran performance tests comparing the data path between no-GPUDirect RDMA vs. GPUDirect RDMA setups. This test used a GPU as a passthrough device. GPUDirect RDMA supports both passthrough GPU and vGPU in 7.0U2. One essential requirement is that the Mellanox NIC and the NVIDIA GPU must share the same PCIe switch or PCIe root complex. A modern CPU, like the Intel Scalable Xeon, has multiple PCIe controllers. Each PCIe controller is considered to be a PCIe root complex. Each PCIe root complex provides a dedicated connection to the connected PCIe devices, allowing for simultaneous data transfers between multiple devices. However, finding documentation about the PCIe slot to specific PCIe root complex mapping is challenging with most systems. Most server documentation only exposes PCIe slots to CPU mapping. Forget about discovering which PCIe slot is connected to which of one of the four PCIe root complexes a dual-socket Intel Scalable Xeon 4th generation server has. An easy way out is to place both PCIe cards on a PCIe riser card. When a PCIe device is installed on a PCIe riser card, it generally connects to the PCIe root complex associated with the slot where the riser card is installed. Please note CPUs are not optimized to work as PCIe switches, and if you are designing your server platform to incorporate RDMA fabrics, I recommend looking for server hardware that includes PCIe switches. Most servers dedicated to machine learning or HPC workload have PCIe switchboards, such as the Dell DS8440. vSphere 7.0 u2 supports Address Translation Service (ATS) with Intel CPUs. ATS, part of the PCIe standard, allows efficient addressing by bypassing the IO Memory Management Unit of the CPU. If a PCIe device needs to access ESXi host memory, it must request the CPU translate the device memory address into a physical one. With ATS, the PCIe device, with the help of a translation agent, can directly perform the translation itself, bypassing the CPU and improving performance. Device groups allow the VI-admin or operator to easily assign a combination of NVIDIA GPU and Mellanox NICs to a VM. vSphere performs a topology detection and exposes which devices share the same PCIe root complex or PCIe Switch in the UI. The device group in the screenshots shows two groups. The group listed at the top is a collection of two A100s connected via NVLink. The device group listed at the bottom combines an A100 GPU, using a 40c vGPU profile (a complete assignment of the card) and a Mellanox ConnectX-6 NIC connected to the same Switch. I must admit that the automatically generated device group names can be a bit more polished. Communication backends such as NCCL, MPI (v1.7.4), and Horovod support GPUDirect RDMA. NVIDIA NVLink Bridge NVLink is designed to offer a low-latency, high-speed interconnect between two adjacent GPU devices to improve GPU-to-GPU communications. NVLINK Bridge is a hardware interconnection plug that connects two PCIe GPUs. The photo shows two PCIe A100 GPUs connected by three NVlink bridges. Using an NVLink setup requires some planning ahead, as the server hardware should be able to allocate two double PCI slot cards directly above each other. It rules out almost every 2U server configuration. For all peer-to-peer access, data flows across the NVLink connections. The beauty is that the CUDA API enables peer access if both GPUs can access each other over NVLINK, even if they don’t belong to the same PCIe domain managed by a single PCI root complex. The P100 introduces the first generation of NVlink, and the H100 has the latest generation incorporated in its design. Each generation increases its links per GPU and, subsequently, the total bandwidth between the GPUs. NVLink Specifications 2nd Gen 3rd Gen 4th Gen Maximum Number of Links per GPU 6 12 18 NVLink bandwidth per GPU 300 GB/s 600 GB/s 900 GB/s Supported GPU Architectures Volta GPUs Ampere GPUs Hopper GPUs The fourth generation offers up to 900 GB/s of bandwidth between GPUs, creating an interesting bandwidth landscape within the system. The PCIe connection is used when the dataset is loaded into GPU memory. In CUDA terminology, this is referred to as a host-to-device copy. Each GPU has its memory address space, so the data set flows to each GPU separately across its PCIe connection. The GPU initiates direct memory access for this process. When models need to synchronize, such as sharing or updating gradients, they use the NVLink connection. In addition to the bandwidth increase, latency is about 1/10th of the PCie connection (1.3 ms vs. 13 ms). An upcoming article covers DMA and memory-mapped I/O extensively. But what about if you want to integrate four PCIe GPUs in a single ESXi host system? vSphere 7 and 8 support the number of GPUs but do not expect scalable linear performance when assigning all four GPUs to a single VM, as NVLink works per bridged card pair. Synchronization data of machine learning models between the pairs traverse across the PCIe bus, creating a congestion point. Going back to Ring-AllReduce, all the transfers happen synchronously. Thus the speed of the allreduce operation is limited by the connection with the lowest bandwidth between adjacent GPUs in the ring. For these configurations, it makes sense to look at HGX systems with 4 GPUs connected to NVLink integrated into the motherboard and using SXM-type GPUs or 8-GPU systems with an integrated NVSwitch. NVSwitch vSphere 8.0 Update 1 supports up to 8 vGPU devices connected via an NVSwitch fabric. An NVSwitch connects multiple NVLinks providing an all-to-all communication and single memory fabric. NVSwitch fabrics are available in NVIDIA HGX-type systems and use GPUs with the SXM interface. The Dell PowerEdge XE8545 (AMD) (4 x A100), XE9680 (8 x A100\H100) (Intel), and HPE Apollo 6500 Gen10 Plus (AMD) are such systems. If we open up an HGX machine, the first thing that sticks out is SXM from factor GPU. It moves away from the PCIe physical interface. The SXM socket handles power delivery, eliminating the need for external power cables, but more importantly, it results in a better (horizontal) mounting position, allowing for better cooling options. A H100 SXM5 also runs more cores (132 streaming multi-processors (SMs)) vs. H100 PCIe (113 SMs). When the data arrives at the onboard GPU memory, after a host-to-device copy, communication remains between GPUs. All communication flows across the NVLinks and NVswitch fabrics, essentially keeping GPU-related traffic of the CPU interconnect (AMD Infinity fabric, Intel UPI ~40 GB/s theoretical bandwidth). With the help of vSphere device groups, the vi-admin or operator can configure the virtual machines with various vGPU configurations. They can be assigned in groups of 2, 4, and 8. Suppose a device group selects a subset of GPU devices of the HGX system. In that case, vSphere isolates these GPUs and disables the NVlink connections to the other GPUs, offering complete isolation between the device groups. No virtualization tax One of the counterarguments I face when discussing these technologies with tech-savvy data scientists is the perception of overhead. Virtualization impacts performance. Why inject a virtualization layer if I can run it on bare metal? Purely focusing on performance, I can safely say this is a thing of the past. MLCommons (an open engineering consortium that aims to accelerate machine learning innovation and its impact on society) has published the MLPerf v3.0 results. The performance team ran MLPerf Inference v3.0 benchmarks on Dell XE8545 with 4x virtualized NVIDIA SXM A100-80GB and Dell R750xa with 2x virtualized NVIDIA H100-PCIE-80GB, both with only 16 vCPUs out of 128. The ESXi host runs the ML workload while providing ample room for other workloads. For the full write-up and more results, please visit the VROOM! Performance Blog. What is interesting is that NVIDIA released a GPU designed to accelerate inference workloads for generative AI applications. The H100 NVL for Large Language Model Deployment contains 188GB of memory and features a “transformer engine” that can deliver up to 12x faster inference performance for GPT-3 compared to the prior generation A100 at data center scale. It is interesting that NVIDIA now sells H100 directly connected with NVLinks as a single device. It promotes the NVLink as a first-class building block instead of an article that should be ordered alongside the devices. With that in mind, the number of available devices is incredibly high. Each with its unique selling points. The following article overviews all the available and supported GPU devices. Other articles in this series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Deep Dive - Fractional and Full GPUs URL: https://frankdenneman.ai/2023-05-10-vsphere-ml-accelerator-deep-dive-fractional-and-full-gpus/ Date: 2023-05-10 Many organizations are building a sovereign ML platform that aids their data scientist, software developers, and operator teams. Although plenty of great ML platform services are available, many practitioners have discovered that a one-size-fits-all platform doesn’t suit their needs. There are plenty of reasons why an organization chooses to build its own ML platform; it can be as simple as control over maintenance windows, being able to curate their own toolchain, relying on a non-opinionated tech stack, or governance/regulations reasons. The first step is to determine the primary workload. Will this be only inference, training, or a mix of both? Getting some servers and a few GPU resources might sound like a good start, but understanding the workload in more detail allows you to get the right resources and create a relevant and valuable platform. If your organization plans to purchase ML-assisted products and services, the focus shifts towards deploying an “inference” workload. Inference workloads are production-ready machine models infused in services or applications that process unseen data and generate an action for the subsequent business process or a recommendation. These workloads require the appropriate hardware and orchestration services. Only a monitoring suite focusing on service availability could suffice if the models are vendor-proprietary. If your organization builds models, the ML platform should focus on two distinct disciplines: Model development and model deployment. A term often heard in this scenario is MLOPs, DevOPs for the Machine Learning ecosystem. The ML platform should provide an infrastructure and software platform that helps data scientists develop their models. Data Scientists are highly skilled in calculus, linear algebra, and statistics. They are typically not hardcore developers, nor are they infrastructure-tech savvy. The unicorns are the ones that know enough to help themselves with creating their own world and developing their model. This blog series and the training vs. inference series intend to bring you closer to the data science team and help you understand some of the nuances of machine learning without going through a full-fledged linear algebra course. ML Development Lifecycle A machine learning model that is fully trained and deemed production ready must be deployed. It cannot run in thin air. It needs to be incorporated into a service or an application. A model is never a standalone feature; thus, developers are needed after the data scientist is done with this model version. And therefore, you need developers ready to incorporate the model into a software system, deploy it, and scale it to serve the inference requests. Software tools are needed to build, test, release, deploy, and monitor these ML-assisted services. The model development lifecycle or ML project work-flow is typically categorized into three broad areas: Build process Training process Deployment process In the build process, the data science team determines what framework and algorithm to use during the concept phase. They explore what data is available, where the data lives, and how they can access it. They study the idea’s feasibility by running some tests using small data sets. In the training process, the data science team limited the possible algorithms and trained the models to learn from the data. Based on the training process results, the model is tuned and retrained. The training process can be cyclical and include various steps from the built process, such as finding more data, as the current dataset might not be satisfactory. The deployment process is where the model is moved into production. It’s now available to the user and processes unseen data. Models facing human behavior tend to deteriorate over time at a much faster rate than models built to augment or support closed-mechanical looped systems. Simply as human nature changes over time and thus the model will slowly detect fewer patterns, it’s trained to recognize. For these models, a recurring training loop must be created where production data is captured, prepared as new datasets to train the model and replace the old model with a freshly trained one. To successfully integrate, deploy, operate, monitor and retrain and re-release, you must create a platform that allows DevOps and Machine Learning teams to develop together. This is where MLOPs platforms add tremendous value to the parties involved. Mix this with an ML-savvy VI-admin and operator team, and this group can help the organization achieve its goals. This series covers the features and functionalities of the ML accelerators available in vSphere and how to set them up in vSphere and Tanzu Kubernetes Grid Services. Articles about MLOps platforms are planned for later this year. Understanding the three ML processes better is essential for the infrastructure focussed operator, as this translates to hardware requirements. Let’s look at what’s supported by vSphere first and then map these features and functionalities to the ML development lifecycle processes. vSphere and Tanzu Kubernetes Grid Services can assign ML accelerators (GPUs) to workloads. Three configurations are possible: a full GPU, a fractional GPU, and multiple GPUs assigned to a single VM. Fractional GPU functionality allows vSphere to split up a full GPU and assign smaller GPUs to multiple VMs. With Multi-GPU, the ESXi hosts can assign multiple GPUs to VMs. NVIDIA GPUDirect RDMA technology significantly improves communication between GPU-enabled VMs on different ESXi hosts. Throughout this series, we will continuously dive deeper into each technology. Full GPUs vSphere allows assigning a full GPU to a VM. Either by using VMware’s (Dynamic) Direct Path I/O technology (Passthru) or NVIDIA’s vGPU technology. This full GPU is exclusively available for this workload. No sharing between VMs is possible. (Dynamic) Direct Path I/O provides VMs access to the physical functions of the GPU device with the help of Memory Mapped I/O. One of the articles in this series covers this topic in detail. The difference between Dynamic Direct Path I/O and Direct Path I/O is the method of assigning the device to the VM. Direct Path I/O assigns the GPU Device to the VM based on the PCIe address of the device. In contrast, Dynamic Direct Path I/O uses a key-value method using either custom or vendor-device generated labels. This allows vSphere to decouple the static relationship between VM and device and provides more flexibility for initial placement processes used by DRS and HA. By default, vSphere 8 uses Dynamic Direct Path I/O with vendor-generated labels. NVIDIA vGPU builds on Dynamic Direct Path I/O and installs the NVIDIA vGPU Manager in the kernel. It allows for creating fractional GPUs and makes vMotion possible. And this is where the choice between both technologies becomes interesting when assigning a full GPU. Direct Path I/O Dynamic Direct Path I/O NVIDIA vGPU Failover HA No Yes Yes Initial Placement DRS No Yes Yes Load Balance DRS No No No vMotion No No Yes Host Maintenance Mode Shutdown VM Cold Migration Manual vMotion Snapshot No No Yes Suspend and Resume No No Yes Fractional GPUs No No Yes TKGS VMClass Support No Yes Yes In both scenarios, dynamic direct path I/O and vGPU allow assigning a dedicated GPU to a VM, which can help the data science team achieve their goals in the build, train or deploy process. But often, more elegant, more efficient technologies are available that create a suitable environment for the workloads but increase overall resource availability within the platform, ready for the data science teams to utilize. Fractional GPUs Fractional GPUs enable multiple VMs to have simultaneous, direct access to a single physical GPU by partitioning a physical GPU device into multiple smaller GPU instances. This functionality is provided by NVIDIA Virtual GPU technology and is available on data center class GPUs and a subset of NVIDIA RTX GPU devices. A vGPU device supports multiple vGPU types that are optimized for specific workloads. The vGPU types applicable for machine learning workloads are C-series and Q-series. The C-series is optimized for compute-intensive workloads. These are pretty much the classical ML workloads. The Q-series type can do the same, but the key difference is that the C-type can only decode video streams, and the Q-type can also (hardware) encode video streams. This difference is essential to know if the data science team plans to deploy a vision AI model. If the model only generates an action or a warning after object\anomaly detection in a video stream, the video is not encoded, and thus only decoders are necessary. A C-series vGPU type is sufficient. However, if the video stream is encoded after being processed by the model because human intervention or a second opinion is required, then a Q-type series is required. NVIDIA vGPU offers two modes, the default Time-sliced mode or the vGPU Multi-Instance GPU (MIG) mode, available from the Ampere architecture onwards. A vGPU is assigned to a VM by selecting a vGPU type (C or Q-series) and a frame buffer size (GPU memory). When using MIG mode, the vGPU type also provides the ability to specify compute elements. The GPU device runs either in time-sliced mode or in MIG mode. There is no possibility of creating a heterogenous vGPU environment where MIG and time-sliced profiles share the same physical GPU device. You can deploy multiple GPU devices in one ESXi host and configure one GPU in time-sliced mode and one in MIG mode. The number of C and Q-series vGPU types are GPU type dependent. For example, an A100 40GB allows ten time-sliced C-Series with a 4GB frame buffer per instance type. In comparison, the A100 80GB allows twenty instances of the same configuration. The A30 and A100 only have video encoders onboard, not video decoders. There is no Q-series vGPU Type available for the A100. A time-sliced vGPU type provides exclusive use of the configured frame buffer until the VM is destroyed. Interesting to note is that the frame buffer cannot be over-allocated. Thus a 40 GB GPU will only accept five VMs with an 8GB frame buffer. Attempting to power on the sixth VM with an 8GB frame buffer fails. Even if all the VMs are idle. The GPU best effort scheduler coordinates access to the GPU device, allowing active workloads to utilize all the compute architecture, such as decoders (NVDEC), encoders (NVENC), and copy engines (CE) on the GPU device. If multiple VM access the GPU, the scheduler schedules these workloads serially. A time slice determines the time window a vGPU can generate workload on the GPU before it is preempted and access is granted to another VM. It is based on the maximum number of vGPUs allowed for the vGPU type on that physical GPU. This is based on the total GPU memory and the assigned frame buffer per vGPU type. If the maximum number of vGPUs on that device is less than or equal to eight, then the time slice is 2 ms. The time-slice window is reduced to 1 ms if it’s more than eight. The scheduler round-robins the active workload. Thus if only one workload is active, the scheduler constantly assigns a time slice. The moment another workload activates, the scheduler adjusts. Most ML applications appreciate a more prolonged time slice as they require maximum throughput. NVIDIA allows for policy and time-slice adjustments. The following articles in this blog series cover the elements in the diagram (GPU Processing Clusters, Streaming Multiprocessors, MMIO space, BAR, etc.). Time-shared Fractional GPU use case - The Build Process If we return to the ML model development cycle, a time-sliced vGPU during the build process might be an excellent fit for most teams. They study the idea’s feasibility by running some tests using small data sets. As a result, the team will run and test some code, with lots of idle time in between. The typical run time is seconds to minutes for these code tests. In many cases, the CPU provides enough power to run these tests. Still, if the data science team wants to research the effect and behavior of the combination of the ML model and the GPU architecture, a vGPU be beneficial. When looking at the situation from a platform operator perspective, this moment is where pooling and abstraction, two core tenets of VMware’s DNA, come into play. We can consolidate the efforts of different data science teams in a centralized environment and offer fractional GPUs. Sometimes a full GPU makes sense in these situations. But that is up to the discretion of the teams and organization. Fractional GPU provides tremendous benefits when used in the proper context. Multi-Instance GPU vGPU Multi-instance GPU functionality is also great for the build process. It can create up to seven separate GPU partitions called instances by isolating the frame buffer, GPU cores, compute engines, and decoders. Predictable and consistent performance is the outcome of this strict isolation, and therefore MIG vGPUs are typically deployed to accelerate inference production workloads. Profile Name Memory SMs Decoders Copy Engines Instances MIG 1g.10gb 1/8 1/7 0 NVDECs/0 JPEG/0 OFA 1 7 MIG 1g.10gb+m 1/8 1/7 1 NVDEC/1 JPEG/1 OFA 1 1 MIG 1g.20gb 1/8 1/7 1 NVDECs/0 JPEG/0 OFA 1 4 MIG 2g.20gb 2/8 2/7 1 NVDECs/0 JPEG/0 OFA 2 3 MIG 3g.40gb 4/8 3/7 2 NVDECs/0 JPEG/0 OFA 3 2 MIG 4g.40gb 4/8 4/7 2 NVDECs/0 JPEG/0 OFA 4 1 MIG 7g.80gb Full Full 5 NVDECs/1 JPEG/1 OFA 7 1 MIG provides a composable configuration of GPU resources. Although the profiles are pre-configured and cannot be changed, users can isolate the correct elements for the job. An A100 80GB GPU device contains seven GPU processing clusters (GPCs). Each GPC contains 16 streaming Multiprocessors (SMs). An SM contains L0 and L1 cache and four tensor cores that perform the needed FP16/FP32 mixed-precision fused multiply-add (FMA) operations and acceleration for all the data types (FP16, BF16, TF32, FP64, INT8, INT4). An A100 GPU contains ten memory controllers that offer access to five HBM2 stacks, which will be logically grouped into eight GPU memory slices. There are seven NVDECs (Video decoders), 7 NVJPGs (Image decoders), and one Optical Flow Accelerator (OFA). In total, there are seven copy engines. They are responsible for transferring data in and out of the GPU. For example, the MIG vGPU profile MIG4g.40gb constructs a GPU instance from four “GPU slices.” A GPU slice includes a “Sys Pipe,” a GPC, an L2 cache slice, and a GPU memory slice. A GPU memory slice includes the L2 cache slices and the associated frame buffer. These are dedicated memory resources. An application consuming a GPU instance does not consume an L2 slice from another GPU instance. This partitioning ensures fault isolation, error containment, recovery, and QoS. The sys pipe communicates with the CPU and is responsible for GPC task scheduling. MIG creates a separate and isolated data path through the entire system, from the crossbar parts all the way to the memory controllers and its DRAM address buses. It’s expected that if more GPU memory is assigned to a GPU instance, more data is copied between the ESXi host system and GPU memory. Thus, dedicated copy engines are assigned to the GPU instance. Additionally, a dedicated number of decoders are assigned per GPU instance. Returning to the MIG Instance example, the MIG vGPU profile MIG4g.40gb isolates four of the eight available memory slices, four GPCs, four copy engines, and two decoders. MIG provides a defined Quality of Service (QoS) and enhanced security due to isolation. Consistent performance throughout the vSphere cluster as consistent performance is maintained even if the VM is migrated to another ESXi host in the cluster. The vGPU is never impacted if another workload is saturating their GPU instance. In contrast, time-sliced provides a more dynamic environment, sometimes leading to better performance if the other GPU tenants are idling. However, there is no prioritization mechanism to indicate if a particular workload requires priority over others. Performance can be inconsistent. It depends on the activity of other tenants. Although MIG instances are hard-coded swimming lanes, and no other workload will dip in this pool of resources and internal pathways, the workload cannot go beyond its own swimming lane if the other MIG slices are idle. So there is no clear winner. Peak performance depends on the other GPU tenants, but if consistent performance is required, look no further than MIG technology. The Ampere and Hopper architecture provides MIG technology in specific data center GPUs. One of the following articles in the series depicts the availability of all the features in the supported vSphere and NVIDIA AI Enterprise (NVAIE) range. MIG vGPU Fractional GPU use case - The Deployment Process If we return to the ML model development cycle, the deployment phase requires consistent performance. Of course, there is no problem in assigning Full GPUs to the workload, but not every inference workload needs that many resources. MIG can offer the right amount of high-performing yet efficient technology. The training vs. inference series dove deep into both workload characteristics. For the typical inference workload, we notice a pattern of lightweight, latency-sensitive streaming data with lower computational needs than the training workload. Training Inference Data Flow Batch data Streaming data Storage Characteristics Throughput based Latency-based, occasionally throughput Batch Size Many recommendations between 1-32 Smaller batch size reduces the memory footprint Smaller batch size increases algorithm performance (generalization) Larger batch size increases compute efficiency Larger batch size increases parallelization (Multi-gpu) 1-4 Data Access Random Access on a large data set Multiple batches are prefetches to keep the pipeline full Fast storage medium recommended Fast storage and network recommended for distributed training Streaming data Memory Footprint Large memory footprint Forward propagation pass – backpropagation pass – model parameters Long time duration of the memory footprint of activations (large bulk of memory footprint) Smaller memory footprint Forward propagation pass – model parameters Activations are short-lived (Total memory footprint = est. 2 largest consecutive layers) Numerical Precision Higher Precision Required Lower Precision Required Data Type FP32 BF16 Mixed Precision (FP16+FP32) BF16 INT8 INT4 (Not seen Often) Training Process For training workloads, it’s typically relatively straightforward present as many GPU resources to the training job as possible. The table shows that training is throughput based, requiring a large memory footprint, which often exceeds the memory capacity of a single GPU. Many data science teams explore distributed training methods to speed up training jobs to reduce training time duration. With today’s large models and large datasets, it’s common to see training jobs of 150+ hours (a whole week of continuous training). For these workloads, vSphere supports the latest and greatest technology available. VSphere 7 and 8 support assigning multiple physical GPUs to a single VM. NVIDIA technology provides high-speed interconnect technology to speed up inter-GPU communication during training jobs. Part 2 dives into the ML accelerator spectrum for distributed training - Multi-GPU technology. Other articles in this series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere ML Accelerator Spectrum Deep Dive Series URL: https://frankdenneman.ai/2023-05-03-vsphere-ml-accelerator-spectrum-deep-dive-series/ Date: 2023-05-03 The number of machine learning workloads is increasing in on-prem data centers rapidly. It arrives in different ways, either within the application itself or data science teams build solutions that incorporate machine learning models to generate predictions or influence actions when needed. Another significant influx of ML workloads is the previously prototyped ML solutions in the cloud that are now moved into the on-prem environment, either for data gravity, governance, economics, or infrastructure (maintenance) control reasons. Techcrunch recently published an interesting article on this phenomenon. But as an operator stuck between the data scientist, developer, and infra, you can be overwhelmed with the requirements that need to be met, the new software stack, and new terminology. You’ll soon realize that a machine-learning model does not run in a vacuum. It’s either integrated into an application or runs as a service. Training and running a model are just steps in applying machine learning to an organizational process. A software stack is required to develop the model, a software stack is required to train it, and a software stack is to integrate it into a service or application, and monitor its accuracy. Models aimed at human behavior tend to deteriorate over time. Our world changes, and the model need to adjust to that behavior. As a result, a continuous development cycle is introduced to retrain the model regularly. It’s essential to understand the data science teams’ world to be successful as an operator. Building the hardware and software technology stack, together with a data science team, helps you to get early traction with other data science teams in the organization. As machine learning can be a shadow IT monster, it is vital to discover the needs of the data science teams. Build the infrastructure from the ground up, starting with the proper hardware ready to satisfy the requirements for training and inference jobs, and provide the right self-serving platform that allows data science teams to curate their own toolset that helps them achieve their goals. To create the proper fundament, you need to understand the workload. However, most machine-learning content is geared toward data scientists. These articles primarily focus on solving an algorithmic challenge while using domain-specific terminology. I’ve written several articles about the training and inference workloads to overcome this gap. Part 1: focuses on the ML Model development lifecycle Part 2: Gives a brief overview of the pipeline structure Part 3: Zooms into Training versus Inference Data Flow and Access Patterns Part 4: Provides a deep dive into memory consumption by Neural Networks Part 5: Provides a deep dive into Numerical Precision Part 6: Explores network compression technology in detail, such as pruning and sparsity. Parts 3 to 6 offer detailed insights into the technical requirements of the neural networks during training jobs and the inference process. It helps to interpret GPU functionality and gauge the expected load of the platform. To successfully accelerate the workload, I want to dive deeper into the available vSphere and Tanzu options in the upcoming series. It focuses on the available spectrum of machine learning accelerators the NVIDIA AI Enterprise suite offers. What hardware capabilities are available, and how do you configure the platform? Although this series focuses on GPUs, I want to note that CPUs are an excellent resource for light training and inference. And with the latest release of the Intel Sapphire Rapids CPU with its Advanced Matrix Extensions (AMX), the future of CPUs in the ML ecosystem looks bright. But I’ll save that topic for another blog post (series). Articles in this series: vSphere ML Accelerator Spectrum Deep Dive Series vSphere ML Accelerator Spectrum Deep Dive – Fractional and Full GPUs vSphere ML Accelerator Spectrum Deep Dive – Multi-GPU for Distributed Training vSphere ML Accelerator Spectrum Deep Dive – GPU Device Differentiators vSphere ML Accelerator Spectrum Deep Dive – NVIDIA AI Enterprise Suite vSphere ML Accelerator Spectrum Deep Dive – ESXi Host BIOS, VM, and vCenter Settings vSphere ML Accelerator Spectrum Deep Dive – Using Dynamic DirectPath IO (Passthrough) with VMs vSphere ML Accelerator Spectrum Deep Dive – NVAIE Cloud License Service Setup ================================================================================ Title: vSphere 8.0 Update 1 Enhancements for Accelerating Machine Learning Workloads URL: https://frankdenneman.ai/2023-04-26-vsphere-8-0-update-1-enhancements-for-accelerating-machine-learning-workloads/ Date: 2023-04-26 Recently vSphere 8 Update 1 was released, introducing excellent enhancements, ranging from VM-level power consumption metrics to Okta Identity Federation for vCenter. In this article, I want to investigate the enhancements to accelerate Machine Learning workloads. If you want to listen to all the goodness provided by update 1, I recommend listening to episode 40 of the Unexplored Territory Podcast with Féidhlim O’Leary (Spotify | Apple). Machine learning is rapidly becoming an essential tool for organizations and businesses worldwide. The desire for accurate models is overwhelming; in many cases, the value of a model comes from accuracy. The machine learning community strives to build more intelligent algorithms, but we still live in a world where processing more training data generates a more accurate model. A prime example is the large language models (LLM) such as ChatGPT. The more data you add, the more accurate they get. Source: ChatGPT Statistics (2023) — The Key Facts and Figures To train ChatGPT, they used textual data from 5 sources. 60% of the dataset was based on a filtered version of data from 8 years of web crawling. I was surprised that 22% of that dataset came from Reddit posts with three or more upvotes (WebText2). But I digress. Large datasets need computation power, and our customers are increasing their machine learning accelerator footprint in their data centers. vSphere 8 update 1 caters to that need. vSphere 8 Update 1 provides the following enhancements focusing on Machine Learning workloads. Increase of PCI Passthrough devices per VM Support for NVIDIA NVSwitch vGPU VMotion Improvements Heterogeneous GPU Profile Support The spectrum of ML Accelerators in vSphere 8 Update 1 Update 1 again increases the maximum number of PCI passthrough devices for a VM. In 7.0 with hardware version 19, 16 passthrough devices are supported. In 8.0, with hardware version 20, a VM can contain up to 32 passthrough devices. With 8.0 update 1, hardware version 20, vSphere supports up to 64 PCIe passthrough devices per VM. vSphere 8 Update 1 extends the spectrum of ML accelerator by supporting NVIDIA NVSwitch Architecture. NVIDIA NVSwitch is a technology that bolts onto the system’s motherboard and connects four to sixteen SXM form factor GPUs. Such systems are known as NVIDIA HGX systems. The Dell PowerEdge XE8545 (AMD) (4 x A100), XE9680 (8 x A100\H100) (Intel), and HPE Apollo 6500 Gen10 Plus (AMD) are such systems. The HGX lineup consists of two platforms, the “Redstone” platform, which contains 4 x SXM4 A100 GPUs, and the “Delta” platform, which contains 8 x SXM4 A100 SXMe GPUs. With the introduction of the NVIDIA Hopper architecture, the HGX platforms are now called Redstone-Next and Delta-Next, containing SXM5 H100 GPUs. There is the possibility of connecting two baseboards of a Delta (-Next) platform via the NVSwitch together in a single server, providing the ability to connect sixteen A100/H100 GPUs directly, but I haven’t seen a server SKU of the major server vendors offering that configuration. If we open up an HGX machine, the first thing that sticks out is SXM from factor GPU. It moves away from the PCIe physical interface. The SXM socket handles power delivery, eliminating the need for external power cables, but more importantly, it results in a better (horizontal) mounting position, allowing for better cooling options. As the GPUs are better cooled, the H100 SXM5 can run more cores (132 streaming multi-processors (SMs)) vs. H100 PCIe (113 SMs). What is the benefit of SXM, NVLINK, and NVSwitch? Training machine Learning models require a lot of data, which the system has to move between components such as CPUs and GPUs and between GPUs and GPUs. Distributed training uses multiple GPUs to provide enough onboard GPU memory capacity to either process and execute the model parameters or to process the data set. If we dissect the data flow, this process has three major steps. Load the data from system memory on the GPUs Run the process (distributed training), which can initiate communication between GPUs Retrieve results from GPU to system memory. Rinse and repeat Internal data buses move data between components, significantly affecting the system’s overall throughput. The most common expansion bus standard is PCI Express (PCIe). Its latest iteration (PCIe5) offers a theoretical bandwidth of 64 GB/s. That is fast, but nothing compared to the onboard GPU RAM speed of an A100 (600 GB/s) or an H100 (900 GB/s). To benefit the most from that memory speed is to build a non-blocking interconnect between the GPUs. If you go one level deeper, by creating a proprietary interconnect system, NVIDIA does not have to wait for the industry to develop and accept standards such as PCIe 6 or 7. It can develop and iterate much faster, attempting to match the interconnect speed to the high bandwidth memory speed of the onboard GPU RAM. However, NVIDIA has to play well with others in the industry to connect the SXM socket to the CPU, and therefore the SXM4 (A100) connects to the CPU via a PCIe 4.0 x16 bus interface (source), and SXM5 (H100) connects to the CPU via a PCIe 5.0 x16 interface (source). That means that during a host-to-device memory copy, the data flows from the system memory across the PCIe controller to the SXM Socket with the matching PCIe bandwidth. Suppose you are a regular ready of my content. In that case, you might expect me to start deep diving into PCIe NUMA locality and the challenges of having multiple GPUs connected in a dual-socket system. However, our engineers and NVIDIA engineers helped the NVIDIA library be aware of the home NUMA configuration. It uses CPU and PCIe information to guide the data traffic between the CPU and PCIe interface. When the data arrives at the onboard GPU memory, communication remains between GPUs. All communication flows across the NVLinks and NVswitch fabrics, essentially keeping GPU-related traffic of the CPU interconnect (AMD Infinity fabric, Intel UPI ~40 GB/s theoretical bandwidth). Please note, on the left side of the diagram, the NVLinks are greyed out of three GPUs to provide a better view of the NVLink connection of an individual GPU in an A100 HGX system. GPU device-to-device communications occur across NVLinks and NVSwitches. An A100 GPU includes 12 3rd-generation NVLinks to provide up to 600 GB/sec bandwidth. The H100 increases the NVlinks to 18, providing 900 GB/sec, seven times the bandwidth of PCIe 5. With the help of vSphere device groups, the vi-admin can configure the virtual machines with various vGPU configurations. They can be assigned in groups of 2, 4, and 8. Suppose a device group selects a subset of GPU devices of the HGX system. In that case, vSphere isolates these GPUs and disables the NVlink connections to the other GPUs, offering complete isolation between the device groups. At this moment, the UI displays quite a cryptic name. If we look at the image, we see Nvidia:2@grid_a100x-40c%NVLink. This name means that this is a group of two A100S with a 40C type profile (the entire card) connected via NVLink. Although the system contains eight GPUs, that doesn’t mean that vSphere only allows assigning multiple GPUs to virtual machines and TKGS worker nodes. Fractional GPU technologies, such as time-sliced or Multi-Instance GPU (MIG), are available. A later article provides a deep dive into NVIDIA Switch functionality. The beauty of this solution is that it uses vGPU technology, and thus we can live-migrate workload between different ESXi hosts if necessary. With each vSphere update, we introduce new enhancements to vGPU vMotion. vSphere 8 Update 1 offers two improvements to improve the utilization of high bandwidth vMotion networks. vGPU vMotion Improvements This new update introduces improvements to the internals of the vMotion process. Update 1 does not present any new buttons or functionalities to the user, but the vMotion internals are more aligned now with the high data load and high-speed transports. A vGPU vMotion is a lot more complex than a regular vMotion, which by itself is still a magical thing in itself. With vGPU workloads, we have to deal with memory-mapped I/O and the situation that 100s of GPU stream processors access vGPU memory regions and can completely change multiple times within a second. An article about MMIO and GPUs will be published soon. To cope with this behavior, we stun the VM so we can drain the memory as quickly as possible. The vMotion team significantly improved by moving checkpoint data to a more efficient vMotion data channel that can leverage multiple threads and sockets. In the previous configuration, the channel for transferring checkpoint data was fixed at two connections, while the new setup can consume as many TCP connections as the network infrastructure permits. Additional optimizations are made to the communication process between the source and destination host to reduce “CPU Driven copies .“A more innovative method of sharing memory is applied, reducing the processes involved in getting the data over from the source host to the destination. With the help of vMotions stream multi-threaded architecture, vGPU vMotion can now saturate high-speed networks up to 80 Gbps. Heterogeneous GPU Profile Support Not necessarily a Machine Learning Workload enhancement, but it allows for a different method of GPU resource consumption, so there is some relationship worth mentioning. Before vSphere 8 Update 1, the first active vGPU workload determines the vGPU profile compatibility of the GPU device. For example, if a VM starts with a C-type vGPU profile with 12G on an NVIDIA A40, the GPU will not accept any other virtual machine with a 12A or 12Q profile. Although each of these profiles consumes the same amount of onboard GPU memory (frame buffer), the GPU rejects these virtual machines. With update 1, this is no longer the case. The GPU accepts different GPU types as long as they have identical frame buffer size configurations. And this makes one of the compelling use cases, “VDI by day, Compute by Night,” even more attainable. This flexibility does offer the ability to mix and match Q, C, and A workloads. The frame buffer size gap between B and the other profile types is too large to expect these profiles to run together on the same physical GPU. The largest B profile contains a 2 GB frame buffer. vGPU Profile Type Optimal Workload Q-Type Virtual workstations for creative and technical professionals who require the performance and features of Quadro technology C-Type Compute-intensive server workloads, such as artificial intelligence (AI), deep learning, or high-performance computing (HPC) B-Type Virtual desktops for business professionals and knowledge workers A-Type App streaming or session-based solutions for virtual applications users Source: Virtual GPU Software Documentation vSphere 8 introduces a tremendous step forwards in accelerator resource scalability, from the ideation phase to big dataset training to securely isolating production streams of unseen data to tailored-sized GPUs. The spectrum of machine learning accelerators available in vSphere 8 update 1 allows organizations to cater to the needs of any data science team regardless of where they are within the life-cycle of their machine learning model development. ================================================================================ Title: VMware Cloud Services Overview Podcast Series URL: https://frankdenneman.ai/2023-04-17-vmware-cloud-services-overview-podcast-series/ Date: 2023-04-17 Over the last year, we’ve interviewed many guests, and throughout the Unexplored Territory Podcast show, we wanted to provide a mini overview series of the VMware Cloud Services. Today we released the latest episode featuring Jeremiah Megie discussing the Azure VMware Solution. Azure VMware Solution Listen on Spotify or Apple. VMware Cloud on AWS In episode 013, we talk to Adrian Roberts, Head of EMEA Solution Architecture for VMware Cloud on AWS at AWS. Adrian discusses the various reasons customers are looking to utilize VMware Cloud on AWS, some of the challenges, and the opportunities that arise when you have your VMware workloads close to native AWS services. Listen on Spotify or Apple. Google Cloud VMware Engine In episode 016, we talk to Dr. Wade Holmes, Security Solutions Global Lead at Google. Wade introduces Google Cloud VMware Engine, discusses various use cases with us, and highlights some operational differences between on-prem only and multi-cloud. Listen on Spotify or Apple. Oracle Cloud VMware Solution In episode 023, we talk to Richard Garsthagen, Oracle’s Director of Cloud Business Development. Our discussion was all about Oracle Cloud VMware Solution. What is unique about Oracle Cloud VMware Solution compared to other solutions? Why does Richard believe this is a platform everyone should consider when you are exploring public cloud offerings? Listen on Spotify or Apple. Cloud Flex Storage In episode 037, we talk to Kristopher Groh, Direct Product Management at VMware, responsible for various storage projects. Kris introduces us to Cloud Flex Storage and discusses the implementation in-depth. Kris also explains the different use cases for Cloud Flex Storage versus vSAN within VMware Cloud on AWS. Listen on Spotify or Apple. Cloud Migration In episode 039, we have a conversation with Niels Hagoort, Technical Marketing Architect at VMware. Niels guides us through the concept of Cloud Migration and dives into the solutions that VMware offers to make the migration as smooth as possible. Listen on Spotify or Apple. Follow us on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. ================================================================================ Title: Research and Innovation at VMware with Chris Wolf URL: https://frankdenneman.ai/2023-03-27-research-and-innovation-at-vmware-with-chris-wolf/ Date: 2023-03-27 In episode 042 of the Unexplored Territory podcast, we talk to Chris Wolf, Chief Research and Innovation Officer of VMware, about innovation at VMware and exciting new research projects. Make sure to follow Chris on Twitter. Also, check the following resources Chris mentioned in the episode. Cloud Native Security Inspector (Project Narrows) FATE OpenFL WASM Follow us on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. Last but not least, hit that subscribe button, rate where ever possible, and share the episode with your friends and colleagues! ================================================================================ Title: My Picks for NVIDIA GTC Spring 2023 URL: https://frankdenneman.ai/2023-03-21-my-picks-for-nvidia-gtc-spring-2023/ Date: 2023-03-21 This week GTC Spring 2023 kicks off again. These are the sessions I look forward to next week. Please leave a comment if you want to share a must-see session. MLOps Title: Enterprise MLOps 101 [S51616] The boom in AI has seen a rising demand for better AI infrastructure — both in the compute hardware layer and AI framework optimizations that make optimal use of accelerated compute. Unfortunately, organizations often overlook the critical importance of a middle tier: infrastructure software that standardizes the machine learning (ML) life cycle, adding a common platform for teams of data scientists and researchers to standardize their approach and eliminate distracting DevOps work. This process of building the ML life cycle is known as MLOps, with end-to-end platforms being built to automate and standardize repeatable manual processes. Although dozens of MLOps solutions exist, adopting them can be confusing and cumbersome. What should you consider when employing MLOps? How can you build a robust MLOps practice? Join us as we dive into this emerging, exciting, and critically important space. Michael Balint, Senior Manager, Product Architecture, NVIDIA William Benton, Principal Product Architect, NVIDIA Title: Solving MLOps: A First-Principles Approach to Machine Learning Production [S51116] We love talking about deploying our machine learning models. One famous (but probably wrong) statement says that “87% of data science projects never make it to production.” But how can we get to the promised land of “Production” if we’re not even sure what “Production” even means? If we could define it, we could more easily build a framework to choose the tools and methods to support our journey. Learn a first-principles approach to thinking about deploying models to production and MLOps. I’ll present a mental framework to guide you through the process of solving the MLOps challenges and selecting the tools associated with machine learning deployments. Dean Lewis Pleban, Co-Founder and CEO, DagsHub Title: Deploying Hugging Face Models to Production at Scale with GPUs [S51553] Seems like everyone’s using Hugging Face to simplify and reuse advanced models and work collectively as a community. But how do you deploy these models into real business environments, along with the required data and application logic? How do you serve them continuously, efficiently, and at scale? How do you manage their life cycle in production (deploy, monitor, retrain)? How do you leverage GPUs efficiently for your Hugging Face deep learning models? We’ll share MLOps orchestration best practices that’ll enable you to automate the continuous integration and deployment of your Hugging Face models, along with the application logic in production. Learn how to manage and monitor the application pipelines, at scale. We’ll show how to enable GPU sharing to maximize application performance while protecting your investment in AI infrastructure and share how to make the whole process efficient, effective, and collaborative. Yaron Haviv, Co-Founder and CTO, Iguazio Title: Democratizing ML Inference for the Metaverse [S51948] In this talk, I will drive you through the Roblox ML Platform inference service. You will learn how we integrate Triton inference service with Kubeflow and Kserve. I will describe how we simplify the deployment for our end users to serve models on both CPU and GPUs. Finally, I will highlight few of our current cases like game recommendation and other computer vision models. Denis Goupil, Principal ML Engineer, Roblox Data Center / Cloud Title: Using NVIDIA GPUs in Financial Applications: Not Just for Machine Learning Applications [S52211] Deploying GPUs to accelerate applications in the financial service industry has been widely accepted and the trend is growing rapidly, driven in large part by the increasing uptake of machine learning techniques. However, banks have been using NVIDIA GPUs for traditional risk calculations for much longer, and these workloads present some challenges due to their multi-tenancy requirements. We’ll explore the use of multiple GPUs on virtualized servers leveraging NVIDIA AI Enterprise to accelerate an application that uses Monte Carlo techniques for risk/pricing application in a large international bank. We’ll explore various combinations of the virtualized application on VMware to show how NVIDIA AI Enterprise software runs this application faster. We’ll also discuss process scheduling on the GPUs and explain interesting performance comparisons using different VM configs. We’ll also detail best practices for application deployments. Manvender Rawat, Senior Manager, Product Management, NVIDIA Justin Murray, Technical Marketing Architect, VMware Richard Hayden, Executive Director and Head of the QR Analytics Team, JP Morgan Chase Title: AI in the Clouds: Navigating the Hybrid Sky with Ease (Presented by Run:ai) [S52352] We’ll focus on the different use cases of running AI workloads in hybrid cloud and multi-cloud environments, and the challenges that come along with that. NVIDIA’s Michael Balint Run:ai’s and Gijsbert Janssen van Doorn will discuss how organizations can successfully implement a hybrid cloud strategy for their AI workloads. Examples of use cases include leveraging the power of on-premises resources for sensitive data while utilizing the scalability of the cloud for compute-intensive tasks. We’ll also discuss potential challenges, such as data security and compliance, and how to navigate them. You’ll gain a deeper understanding of the various use cases of hybrid cloud for AI workloads, the challenges that may arise, and how to effectively implement them in your organization. Michael Balint, Senior Manager, Product Architecture, NVIDIA Gijsbert Janssen van Doorn, Director Technical Product Marketing, Run:ai Title: vSphere on DPUs Behind the Scenes: A Technical Deep Dive (Presented by VMware Inc.) [S52382] We’ll explore how vSphere on DPUs offloads traffic to the data processing unit (DPU), allowing for additional workload resources, zero-trust security, and enhanced performance. But what goes on behind the scenes that makes vSphere on DPUs so good at enhancing performance? Is it just adding a DPU? Join this session to find the answer and more technical nuggets to help you see the power of DPUs with vSphere on DPUs. Dave Morera, Senior Technical Marketing Architect, VMware Meghana Badrinath, Technical Product Manager, VMware Title: Developer Breakout: What’s New in NVAIE 3.0 and vSphere 8 [SE52148] NVIDIA and VMware have collaborated to unlock the power of AI for all enterprises by delivering an end-to-end enterprise platform optimized for AI workloads. This integrated platform delivers NVIDIA AI Enterprise, the best-in-class, end-to-end, secure, cloud-native suite of AI software running on VMware vSphere. With the recent launches of vSphere 8 and NVIDIA AI Enterprise 3.0, this platform’s ability to deliver AI solutions is greatly expanded. Let’s look at some of these state-of-the-art capabilities. Jia Dai, Senior MLOps Solution Architect, NVIDIA Veer Mehta, Solutions Architect, NVIDIA Dan Skwara, Senior Solutions Architect, NVIDIA Autonomous Vehicles Title: From Tortoise to Hare: How AI Can Turn Any Driver into a Race Car Driver [S51328] Performance driving on a racetrack is exciting, but it’s not widely accessible as it requires advanced driving skills honed over many years. Rimac’s Driver Coach enables any driver to learn from the onboard AI system, and enjoy performance driving on racetracks using full autonomous driving at very high speeds (over 350km/h). We’ll discuss how AI can be used to accelerate driver education and safely provide racing experiences at incredibly high speeds. We’ll dive deep into the overall development pipeline, from collecting data to training models to simulation testing using NVIDIA DRIVE Sim, and finally, implementing software on the NVIDIA DRIVE platform. Discover how AI technology can beat human professional race drivers. Sacha Vrazic, Director - Autonomous Driving R&D, Rimac Technology Deep Learning Title: Scaling Deep Learning Training: Fast Inter-GPU Communication with NCCL [S51111] Learn why fast inter-GPU communication is critical to accelerate deep learning training, and how to make sure your system has the right level of performance for your model. Discover NCCL, the inter-GPU communication library used by all deep learning frameworks for inter-GPU communication, and how it combines NVLink with high-speed networks like Infiniband to accelerate communication by an order of magnitude, allowing training to be run on hundreds, or even thousands, of GPUs. See how new technologies in Hopper GPUs and ConnectX-7 allow for NCCL performance to reach new highs on the latest generation of DGX and HGX systems. Finally, get updates on the latest improvements in NCCL, and what should come in the near future. Sylvain Jeaugey, Principal Engineer, NVIDIA Title: FP8 Mixed-Precision Training with Hugging Face Accelerate [S51370] Accelerate is a library that allows you to run your raw PyTorch training loop on any kind of distributed setup with multiple speedup techniques. One of these techniques is mixed precision training, which can speed up training by a factor between 2 and 4. Accelerate recently integrated Nvidia Transformers FP8 mixed-precision training which can be even faster. In this session, we’ll dive into what mixed precision training exactly is, how to implement it in various floating point precisions and how Accelerate provides a unified API to use all of them. Sylvain Gugger, Senior ML Open Source Engineer, Hugging Face HPC Title: Accelerating MPI and DNN Training Applications with BlueField DPUs [S51745] Learn how NVIDIA Bluefield DPUs can accelerate the performance of HPC applications using message passing interface (MPI) libraries and deep neural network (DNN) training applications. Under the first direction, we highlight the features and performance of the MVAPICH2-DPU library in offloading non-blocking collective communication operations to the DPUs. Under the second direction, we demonstrate how some parts of computation in DNN training can be offloaded to the DPUs. We’ll present sample performance numbers of these designs on various computing platforms (x86 and AMD) and Bluefield adapters (HDR-100Gbps and HDR-200 Gbps), along with some initial results using the newly proposed cross-GVMI support with DPU. Dhabaleswar K. (DK) Panda, Professor and University Distinguished Scholar, The Ohio State University Title: Tuning Machine Learning and HPC Workloads Performance in Virtualized Environments using GPUs [S51670] Today’s machine learning (ML) and HPC applications run in containers. VMware vSphere runs containers in virtual machines (VMs) with VMware Tanzu for container orchestration and Kubernetes cluster management. This allows servers in the hybrid cloud to simultaneously host multi-tenant workloads like ML inference, virtual desktop infrastructure/graphics, and telco workloads that benefit from NVIDIA AI and VMware virtualization technologies. NVIDIA AI Enterprise software in VMware vSphere combines the outstanding virtualization benefits of vSphere with near-bare metal, or in HPC applications, better than bare-metal performance. NVIDIA AI Enterprise on vSphere supports NVLink and NVSwitch, which allows ML training, and HPC applications to maximize multi-GPU performance. We’ll describe these technologies in detail, and you’ll learn how to leverage and tune performance to achieve significant savings in total cost of ownership for your preferred cloud environment. We’ll highlight the performance of the latest NVIDIA GPUs in virtual environments. Uday Kurkure, Staff Engineer, VMware Lan Vu, Senior Member of the Technical Staff, VMware Manvender Rawat, Senior Manager, Product Management, NVIDIA ================================================================================ Title: Discover what's new in vSphere 8.0 U1 and vSAN 8.0 U1 URL: https://frankdenneman.ai/2023-03-16-discover-whats-new-in-vsphere-8-0-u1-and-vsan-8-0-u1/ Date: 2023-03-16 We (the Unexplored Territory team) work with the vSphere release team to get you the latest information about the new releases as quickly as possible. This week we published two new episodes discussing what’s new with vSphere 8.0 U1 and vSAN 8.0 U1. To enjoy the content, you can listen to them using your favorite podcast apps, such as Apple or Spotify, or the embedded players below. ================================================================================ Title: Simulating NUMA Nodes for Nested ESXi Virtual Appliances URL: https://frankdenneman.ai/2023-03-02-simulating-numa-nodes-for-nested-esxi-virtual-appliances/ Date: 2023-03-02 To troubleshoot a particular NUMA client behavior in a heterogeneous multi-cloud environment, I needed to set up an ESXi 7.0 environment. Currently, my lab is running ESXi 8.0, so I’ve turned to William Lams’ excellent repository of nested ESXi virtual appliances and downloaded a copy of the 7.0 u3k version. My physical ESXi hosts are equipped with Intel Xeon Gold 5218R CPUs, containing 20 cores per socket. The smallest ESXi host contains ten cores per socket in the environment I need to simulate. Therefore, I created a virtual ESXi host with 20 vCPUs and ensured that there were two virtual sockets (10 cores per socket) Once everything was set up and the ESXi host was operational, I checked to see if I could deploy a 16 vCPU VM to simulate particular NUMA client configuration behavior and verify the CPU environment. The first command I use is to check the “physical” NUMA node configuration “sched-stats -t numa-node”. But this command does not give me any output, which should not happen. Let’s investigate, let’s start off by querying the CPUinfo of the VMkernel Sys Info Shell (vsish): vsish -e get /hardware/cpu/cpuInfo The ESXi host contains two CPU packages. The VM configuration Cores per Socket has provided the correct information to the ESXi kernel. The same info can be seen in the UI at Host Configuration, Hardware, Overview, and Processor. However, it doesn’t indicate the number of NUMA nodes supported by the ESXi kernel. You would expect that two CPU packages would correspond to at least two NUMA nodes. The command vsish -e dir /hardware/cpuTopology/numa/nodes shows the number of NUMA nodes that the ESXi kernel detects It only detects 1 NUMA node as the virtual NUMA client configuration has been decoupled from the Cores Per Socket configuration since ESXi 6.5. As a result, the VM is presented by the physical ESXi host as a single virtual NUMA node, and the virtual ESXi host picks this up. Logging in to the physical host, we can validate the nested ESXi VM configuration and run the following command. vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi "DICT._(displayname._|numa._|cores._|vcpu._|memsize._|affinity._)= ._|numa:._|numaHost:._" "/$path/vmware.log"; echo -e; done The screen dump shows that the VM is configured with one Virtual Proximity Domain (VPD) and one Physical Proximity Domain (PPD). The VPD is the NUMA client element that is exposed to the VM as the virtual NUMA topology, and the screenshot shows that all the vCPUs (0-19) are part of a single NUMA client. The NUMA scheduler uses the PPD to group and place the vCPUs on a specific NUMA domain (CPU package). By default, the NUMA scheduler consolidates vCPUs of a single VM into a single NUMA client up to the same number of physical cores in a CPU package. In this example, that is 20. As my physical ESXi host contains 20 CPU cores per CPU package, all the vCPUs in my nested ESXi virtual appliance are placed in a single NUMA client and scheduled on a single physical NUMA node as this will provide the best possible performance for the VM, regardless of the Cores per Socket setting. The VM advanced configuration parameter numa.consolidate = "false" forces the NUMA scheduler to evenly distribute the vCPU across the available physical NUMA nodes. After running the vmdumper instruction once more, you see that the NUMA configuration has changed. The vCPUs are now evenly distributed across two PPDs, but only one VPD exists. This is done on purpose, as we typically do not want to change the CPU configuration for the guest OS and application, as that can interfere with previously made optimizations. You can do two things to change the configuration of the VPD, use the VM advanced configuration parameter numa.vcpu.maxPerVirtualNode and set it to 10. Or remove the numa.autosize.vcpu.maxPerVirtualNode = “20” from the VMX file. I prefer removing the numa.autosize.vcpu.maxPerVirtualNode setting, as this automatically follows the PPD configuration, it avoids mismatches between numa.vcpu.maxPerVirtualNode and the automatic numa.consolidate = "false" configuration. Plus, it’s one less advanced setting in the VMX, but that’s just splitting hairs. After powering up the nested ESXi virtual appliance, you can verify the NUMA configuration once more in the physical ESXi host: The vsish command vsish -e dir /hardware/cpuTopology/numa/nodes shows ESXi detects two NUMA nodes and sched-stats -t numa-pnode now returns the information you expect to see Please note that if the vCPU count of the nested ESXi virtual appliance exceeds the CPU core count of the CPU package, the NUMA scheduler automatically creates multiple NUMA clients. ================================================================================ Title: Sapphire Rapids Memory Configuration URL: https://frankdenneman.ai/2023-02-28-sapphire-rapids-memory-configuration/ Date: 2023-02-28 The 4th generation of the Intel Xeon Scalable Processors (codenamed Sapphire Rapids) was released early this year, and I’ve been trying to wrap my head around what’s new, what’s good, and what’s challenging. Besides those new hardware native accelerators, which a later blog post covers, I noticed the return of different memory speeds when using multiple DIMMs per channel. Before the Scalable Processor Architecture in 2017, you faced the devil’s triangle when configuring memory capacity, cheap, high capacity, fast, pick two. The Xeons offered four memory channels per CPU package, and each memory channel could support up to three DIMMs. The memory speed decreased when equipped with three DIMMs per channel (3 DPC). Skylake, the first Scalable Processor generation, introduced six channels; each supporting a maximum of two DIMMs, with no performance degradation between 1 DPC or 2 DPC configurations. However, most server vendors introduced a new challenge by only selling servers with 8 DIMM slots instead of 12, and thus unbalanced memory configurations were introduced when all DIMM slots were populated. Unbalanced memory configuration negatively impacts performance. Dell and others have reported drops in memory bandwidth between 35% to 65%. The 3rd generation of the Scalable Processor Architecture introduced eight channels of DDR4 per CPU. To solve the server vendors’ unbalanced memory configuration problem and provide parity to the AMD EPYC memory configuration. It also meant we were back to the “natural” order of base 10 memory capacity configuration, 256, 512, 1024,2048. Many servers weren’t following the optimal 6-channel configuration of 384, 768, and 1536; for some admins, it felt unnatural. And this brings me to the 4th generation, the Sapphire Rapids. It provides eight channels of DDR5 per CPU with a maximum memory speed of 4800 MHz. Compared to the 3rd generation, it results in up to 50% more aggregated bandwidth as the Ice Lake generation supports eight channels using DDR4 3200 MHz. But the behavior between the 3rd and the 4th generation differ when pushing them to their max capacity. With Sapphire Rapids, each CPU has eight memory controllers, providing high-speed throughput and allowing advanced sub-NUMA clustering configurations similar to the AMD EPYC of four clusters within a single CPU (An upcoming blog post covers this topic in-depth). These features sound very promising. However, Intel reintroduced different memory speeds when loading the memory channel with multiple DIMMs. Sapphire Roads supports multiple memory speeds. The bronze and silver families support a maximum memory speed of 4000 MHz. The Gold family is all over the place, supporting a maximum of 4000, 4400, and 4800 MHz. The Platinum family supports up to 4800 MHz. However, this is in a 1 DPC configuration. 4400 MHz in a 2 DPC configuration. The massive step up from 3200 MHz to 4800 MHz is slightly reduced when loading the server with more than eight DIMMs per CPU. When comparing theoretical bandwidth speeds, 1 DPC and 2 DPC, bandwidth performance looks as follows: DDR4 3200 MHZ provides a theoretical bandwidth of 25.6 GB/s. DDR5 4800 MHz provides a theoretical bandwidth of 38.2 GB/s, while 4400 MHz DDR5 memory provides a theoretical bandwidth of 35.2 GB/s. Xeon Architecture 1 DPC GB/s 8 Ch +% 2 DPC GB/s 16 Ch +% 3rd Gen Xeon 3200 MHz 23.4 204.8 GB/s 3200 MHz 25.6 409.6 GB/s 4th Gen Xeon 4800 MHz 38.4 307.2 GB/s 50% 4400 MHz 35.2 563.2 GB/s 37.5% Dell published a report of performance study measuring memory bandwidth using the STREAM Triad benchmark. The study compared the performance of the 3rd and 4th generation Xeons and shows “real” bandwidth numbers. Sapphire Rapids improves memory bandwidth by 46% in a 1 DPC configuration, “but only” 26% in a 2 DPC configuration. Although STREAM is a synthetic benchmark, it does give us a better idea of what bandwidth to expect in real life. I hope this information helps to guide you when configuring the memory configuration of your next vSphere ESXi host platform. Selecting the right DIMM capacity can quickly get 20% better memory performance. ================================================================================ Title: How to Create a Windows 11 Bootable USB on Mac OS Monterey URL: https://frankdenneman.ai/2022-12-21-how-to-create-a-windows-11-bootable-usb-on-mac-os-monterey/ Date: 2022-12-21 I need to install Windows 11 on a gaming PC, but I only have a MacBook in my house, as this is my primary machine for work. To make things worse, trying to do this on macOS Monterey is extra difficult due to the heightened security levels that withhold you from running unsigned software. I.e., most free tooling software. However, most tooling is provided by macOS itself. You have to remember the correct steps. And because this is not a process I often do, I decided to document it for easy retrieval, which might help others facing the same challenge. As I mentioned, most of the tools are installed on macOS. The only missing one is the open-source Windows Imaging Library (wimlib). This tool helps you to split a particular file (install.wim) as it is too large for the filesystem we use on the USB drive. To install wimlib, you need to have Homebrew installed. Homebrew is a package manager for macOS. Some already have it installed, and some don’t, so I will include the install command for Homebrew. Install Homebrew Open a terminal window and run the following command: /bin/bash -c "$(curl -fsSL https://raw.githubusercontent.com/Homebrew/install/HEAD/install.sh)" This can be a lengthy process. My recent model Macbook Pro took about 6 minutes to complete. To Install wimlib, run the following command: brew install wimlib Insert a USB key that is large enough to contain Windows 11. I use a 64GB USB drive that can also store extra drivers. Newer motherboards are typically equipped with these Intel 2.5 GbE NICs, and Windows 11 do not have the driver built-in. If you want to store these drivers on the USB drive as well to be able to continue the Windows install process. To discover which disk identifier macOS assigned to the USB driver, run the following command: diskutil list external The option “external” only displays mounted volumes. This helps you to spot your USB drive easily. In my case, macOS assigned the identifier “disk5” to it. Note the disk name. We need that for our erasedisk command. Erase the USB Drive The next command is going to erase the USB drive using MS-DOS format. Use a Master Boot Record scheme (MBR) as this is necessary to find all the files during the installation of Windows 11. We need to retain the disk’s name and use the identifier. The name is 64GBUSB, and the identifier is disk5 diskutil eraseDisk MS-DOS "64GBUSB" MBR disk5 Mount Windows 11 ISO The next step is to mount the Windows 11 ISO, you can use finder for that and click on the file, but as we are doing everything in the terminal, the command you can use to mount the iso is: hdiutil mount Win11_22H2_English_x64v1.iso I’ve executed this command in the directory in which the ISO file was stored. You can execute this command anywhere but ensure to include the full path to the Windows 11 ISO file. The main challenge of creating the Windows 11 bootable USB drive is the install size.wim file in combination with the MS-DOS format of the drive itself. The install.wim file is larger than 4GB and thus incompatible with the file system. To solve that, you can compress or split the file using wimlib. The Windows installation process knows how to deal with split files; thus, this is the preferred method, as compressing files impacts the duration of the installation process. Copy files Windows ISO to USB Drive The first step is to copy over all the files of the Windows 11 ISO file that we just mounted EXCEPT the install.wim. The easiest way is using the following rsync command: rsync -avh --progress --exclude=sources/install.wim /Volumes/CCCOMA_X64FRE_EN-US_DV9/ /Volumes/64GBUSB The progress option shows each file’s copy progress; the first volume is the source (the windows ISO), and the second volume directory is the destination (the USB drive). The exclude option tells rsync to ignore install.vim during the copy process. Split Install.wim The last step is to split the install.wim into two parts and place it into the sources folder onto the USB drive. To do so, execute the following command: wimlib-imagex split /Volumes/CCCOMA_X64FRE_EN-US_DV9/sources/install.wim /Volumes/64GBUSB/sources/install.swm 4000 The key element of this command is the option “4000” this tells the command to split the file into chunks with a maximum size of 4000 MB. The MS-DOS (fat32) maximum file size is 4096MB. You can decide to lower the number if you’re comfortable, but keep it a little bit under the max. Once this process is complete, you’re done. You can safely unmount the USB drive and use it to install Windows 11. diskutil unmount /dev/disk5 ================================================================================ Title: Unexplored Territory Ep 34 - William Lam Talks Home Labs - Christmas Special URL: https://frankdenneman.ai/2022-12-21-unexplored-territory-ep-34-william-lam-talks-home-labs-christmas-special/ Date: 2022-12-21 It’s the end of the year, and everybody is winding down from a hectic year, so we wanted to give you some light stuff to listen to in our last episode. But William had other plans. He is on fire in this episode, dropping one gem after another, sharing a decade-long of home lab wisdom. We asked William what his top 10 home lab gifts would be, and he got gift ideas from stocking stuffers to full-blown systems. Listen via Spotify (https://spoti.fi/3jdOmUp), Apple (https://apple.co/3WpMB50), or online (https://unexploredterritory.tech) Williams’ Christmas Top 10 Wishlist Number 10: Velcro cable management Number 09: Smart Power meter /UPS/ APC Surge Arrest Number 08: Kubernetes for Administrators, VDI Design Guide Part 2, vSAN 7.0 U3 Deep Dive Number 07: VMUG Advantage Membership Number 06: Memory Upgrade (64GB memory on the Intel NUC) Number 05: Thunderbolt Storage Number 04: USB/Thunderbolt Networking Number 03: 10GbE Switch (Netgear/Ubiquiti) Number 02: Intel NUC Serpent Canyon / Supermicro E302-12D Number 01: Raspberry Pi / Dell Precision 7770 Articles and solutions discussed during the show: VMware Fling: USB Network Native Driver for ESXi How to build a customizable Raspberry Pi OS Virtual Appliance (OVA)? Stateless ESXi-Arm with Raspberry Pi Follow us on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. Last but not least, make sure to hit that subscribe button, rate where ever possible, and share the episode with your friends and colleagues! ================================================================================ Title: Unexplored Territory Podcast 32 - IT giving McLaren Racing the edge URL: https://frankdenneman.ai/2022-11-28-unexplored-territory-podcast-32-it-giving-mclaren-racing-the-edge/ Date: 2022-11-28 Edward Green, Head of Commercial Technology at McLaren Racing, keynoted at the VMware Explore tech conference in Barcelona. I had the honor of sitting down with him for a few minutes to talk about the role of IT in F1. Most of us watch the races with multiple screens, the main TV for the race and additional screens to look at the various telemetry feeds. And so you know how much data flows between cars and the teams. But sitting down with Edward and hearing him explain how data transfer feeds and disk sizes impact real training time for the driver is just candy to the ears of every tech-savvy Formula 1 fan. The episode starts with a short interview with Joe Baguley, the racing CTO of VMware, discussing his involvement with the McLaren Racing partnership and his passion for racing! Listen to Edward and Joe at Spotify Apple Website Lando Norris, McLaren MCL36, prepares to head to the grid ================================================================================ Title: ML Session at CTEX VMware Explore URL: https://frankdenneman.ai/2022-11-04-ml-session-at-ctex-vmware-explore/ Date: 2022-11-04 Next week during VMware Explore, VMware is also organizing the Customer Technical Exchange. I’m presenting the session “vSphere Infrastructure for Machine Learning workloads”. I will discuss how vSphere act as a self-service platform for data science teams to easily and quickly deploy ML platforms with acceleration resources. I CTEX is happening at the Fira Barcelona Gran Via in room CC4 4.2. This is an NDA event. Therefore, you will need to register vi Next week during VMware Explore, the VMware Office of the CTO is organizing the Customer Technical Exchange. I’m presenting the session “vSphere Infrastructure for Machine Learning workloads”. I will discuss how vSphere act as a self-service platform for data science teams to easily and quickly deploy ML platforms with acceleration resources. I CTEX is happening on the 8th and 9th of November at the Fira Barcelona Gran Via in room CC4 4.2. This is an NDA event. Therefore, you will need to register via https://via.vmw.com/CTEXExploreEurope2022-Register. ================================================================================ Title: vSphere 8 CPU Topology for Large Memory Footprint VMs Exceeding NUMA Boundaries URL: https://frankdenneman.ai/2022-11-03-vsphere-8-cpu-topology-for-large-memory-footprint-vms-exceeding-numa-boundaries/ Date: 2022-11-03 By default, vSphere manages the vCPU configuration and vNUMA topology automatically. vSphere attempts to keep the VM within a NUMA node until the vCPU count of that VM exceeds the number of physical cores inside a single CPU socket of that particular host. For example, my lab has dual-socket ESXi host configurations, and each host has 20 processor cores per socket. As a result, vSphere creates a VM with a vCPU topology with a unified memory address (UMA) up to the vCPU count of 20. Once I assign 21 vCPU, it creates a vNUMA topology with two virtual NUMA nodes and exposes this to the guest OS for further memory optimization. You might have noticed that vNUMA topology sizing is structured around vCPU and physical core count. But what happens when the virtual machine configuration fits inside the NUMA node with its vCPU configuration, i.e., has less vCPU than the CPU has physical cores? But the VM requires more memory than the NUMA node can provide. I.e., the VM configuration exceeds the local memory configuration of the NUMA node. As a test, I’ve created a VM with 12 vCPUs and 384GB. The vCPU configuration fits a single NUMA node (12<20), but the memory configuration of 384GB exceeds the 256GB of each NUMA node. By default, vSphere creates the vCPU topology and exposes a unified memory address to the guest OS. For scheduling purposes, it creates two separate scheduling constructs to allocate the physical memory from both NUMA nodes but just doesn’t exposes that information to the guest OS. Inconsistent performance results from this situation, as the guest OS just starts to allocate the memory from the beginning of the memory range to the end without knowing anything about the physical origins. As a test, an application is running that allocates 380GB. As all the vCPU run in a single NUMA node, the memory scheduler will do its best and allocate the memory as close to the vCPUs as possible. As a result, the memory scheduler can allocate 226 GB locally (237081912 KB by vm.8823431) while having to allocate the rest from the remote NUMA node (155 GB). Latency on an Intel for local NUMA nodes hovers around 73ns for a Xeon v4 and 89ns for a Skylake generation. At the same time, remote memory is about 130ns for a v4 and 139ns for a Skylake, AMD Epyc local is 129ns, and remote memory is 205ns. On Intel, that is a performance impact of 73% v4 and 56% on Skylake. On AMD, having to fetch remote memory will slow it down by 56%. That means, in this case, the application fetches 31% of its memory with a 73% latency penalty. Another problem with this setup is that this VM is now quite an unbalanced noisy neighbor. There is little to do about monster VMs in your system, but this VM has an unbalanced memory footprint. It eats up most of the memory of NUMA node 0. I would rather see a more balanced footprint to utilize other cores on the NUMA node and enjoy local memory access. In this scenario, new virtual machines are forced to retrieve their memory from the other NUMA node if they are scheduled alongside this VM from a vCPU perspective. To solve this problem, we used to set the advanced setting “numa.consolidate = FALSE” in vSphere 7 and older versions. vSphere 8 provides the option to configure the vCPU topology and, specifically, the vNUMA topology from the UI that solves the aforementioned problem. In the VM options of the VM configuration vSphere 8 includes the option CPU Topology. By default, the Cores per Socket and NUMA Nodes settings are “assigned at power on,” which is the recommended setting for most workloads. In our case, we want to change it, and first, we have to change the Cores per Socket settings before we can adjust the NUMA Nodes setting. It’s best to distribute the vCPUs equally across the physical NUMA nodes so that each group of vCPU can allocate an equal amount of memory capacity. The VM configuration is set to 6 cores per socket in our test case, creating two vSockets. The next step is to configure the virtual NUMA nodes. Aligning this with the underlying physical configuration helps to get the best predictable and consistent performance behavior for the virtual machine. In our case, the VM is configured with two NUMA nodes. After the virtual machine is powered-on, I opened up a SSH session to the ESXi host and ran the following command: “vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi “DICT.(displayname.|numa.|cores.|vcpu.|memsize.|affinity.)= .|numa:.|numaHost:.” “/$path/vmware.log”; echo -e; done” That provided me with the following output; notice the following lines: cpuid.coresPerSocket = “6” numa.vcpu.coresPerNode = “6” numaHost 12 VCPUs 2 VPDs 2 PPDs It shows that the VM is configured with 12 vCPUs, 6 cores per socket, 6 vCPUs per NUMA node. At the ESXi scheduling level, the NUMA scheduler creates two scheduling constructs. A virtual proximity domain (VPD) is a construct used to expose a CPU topology to the guest OS. You can see that it has created two VPDs for this VM. VPD 0 contains VCPU 0 to VCPU 5, and VPD 1 contains VCPU 6 to VCPU 11. A PPD is a physical proximity domain and is used by the NUMA scheduler to assign a PPD to a physical NUMA node. To check if everything has worked, I looked at the task manager of windows and enabled the NUMA view, which now shows two NUMA nodes. Running the memory test again on the virtual machine shows a different behavior at the ESXi level. Memory consumption is far more balanced. The VMs NUMA client on NUMA node 0 is consuming 192 GB (201326592 KB), while the NUMA client of the VM on NUMA node 1 is consuming 189 GB (198604800 KB). The vSphere 8 vCPU topology is a very nice method of helping manage VM configurations that are special cases. Instead of having to set advanced settings, a straightforward UI that can be easily understood by any team member that wasn’t involved with configuring the VM is a big step forward. But I like to stress once more that the default setting is the best for most virtual machines. Do not use this setting in your standard configuration for your virtual machine real estate. Keep it set to default and enjoy our 20 years of NUMA engineering. We got your back most of the time. And for some of the outliers, we now have this brilliant UI. ================================================================================ Title: Unexplored Territory Podcast EP30 - Project Keswick with Alan Renouf URL: https://frankdenneman.ai/2022-10-31-unexplored-territory-podcast-ep30-project-keswick-with-alan-renouf/ Date: 2022-10-31 While preparing the podcast, I knew this episode would be good. Edge technology immensely excites me, and the way the project team strays away from the proverbial hammer and looks at ways to incorporate different principles like Gitops management concepts is inspiring. To top it off, you have Alan Renouf to talk about it, a long-time colleague and friend, but unfortunately, Covid prohibited me from partaking in this discussion. But, of course, Duncan and Johan had an excellent conversation with Alan. Please check it out on Spotify, Apple, or via our website. Enjoy! ================================================================================ Title: vSphere 8 CPU Topology Device Assignment URL: https://frankdenneman.ai/2022-10-25-vsphere-8-cpu-topology-device-assignment/ Date: 2022-10-25 There seems to be some misunderstanding about the new vSphere 8 CPU Topology Device Assignment feature, and I hope this article will help you understand (when to use) this feature. This feature defines the mapping of the virtual PCIe device to the vNUMA topology. The main purpose is to optimize guest OS and application optimization. This setting does not impact NUMA affinity and scheduling of vCPU and memory locality at the physical resource layer. This is based on the VM placement policy (best effort). Let’s explore the settings and their effect on the virtual machine. Let’s go over the basics first. The feature is located in the VM Options menu of the virtual machine. Click on CPU Topology By default, the Cores per Socket and NUMA Nodes settings are “assigned at power on” and this prevents you from assigning any PCI device to any NUMA node. To be able to assign PCI devices to NUMA nodes, you need to change the Cores per Socket setting. Immediately, a warning indicates that you need to know what you are doing, as incorrectly configuring the Cores per Socket can lead to performance degradation. Typically we recommend aligning the cores per socket to the physical layout of the server. In my case, my ESXi host system is a dual-socket server, and each CPU package contains 20 cores. By default, the NUMA scheduler maps vCPU to cores for NUMA client sizing; thus, this VM configuration cannot fit inside a single physical NUMA node. The NUMA scheduler will distribute the vCPUs across two NUMA clients equally; thus, 12 vCPUs will be placed per NUMA node (socket). As a result, the Cores per Socket configuration should be 12 Cores per Socket, which will inform ESXi to create two virtual sockets for that particular VM. For completeness’ sake, I specified two NUMA nodes as well. This setting is a PER-VM setting, it is not NUMA Nodes per Socket. You can easily leave this to the default setting, as ESXi will create a vNUMA topology based on the Cores per Socket settings. Unless you want to create some funky topology that your application absolutely requires. My recommendation, keep this one set to default as much as possible unless your application developer begs you otherwise. This allows us to configure the PCIe devices. As you might have noticed, I’ve added a PCIe device. This device is an NVIDIA A30 GPU in Dynamic Direct Path I/O (Passthrough) mode. But before we dive into the details of this device, let’s look at the virtual machine’s configuration from within the guest OS. I’ve installed Ubuntu 22.04 LTS and used the command lstopo. (install using: sudo apt install hwloc) You see the two NUMA nodes, with each twelve vCPUs (Cores) and a separate PCI structure. This is the way a virtual motherboard is structured. Compare this to a physical machine, and you notice that each PCI device is attached to the PCI controller that is located within the NUMA node. And that is exactly what we can do with the device assignment feature in vSphere 8. We can provide more insights to the guest OS and the applications if they need this information. Typically, this optimization is not necessary, but for some specific network load-balancing algorithms or machine learning use cases, you want the application to understand the NUMA PCI locality of the PCIe devices. In the case of the A30, we need to understand its PCIe-NUMA locality. The easiest way to do this is to log on to the ESXi server through an SSH session and search for the device via the esxcli hardware pci list command. As I’m searching for an NVIDIA device, I can restrict the output of this command by using the following command “esxcli hardware pci list | grep “NVIDIA -A 32 -B 6”. This instructs the grep command to output 32 lines (A)after and 6 lines (B)before the NVIDIA line. The output shows us that the A30 card is managed by the PCI controller located in NUMA node 1 (Third line from the bottom). We can now adjust the device assignment accordingly and assign it to NUMA node 1. Please note the feature allows you to also assign it to NUMA node 0. You are on your own here. You can do silly things. But just because you can, doesn’t mean you should. Please understand that most PCIe slots on a server motherboard are directly connected to the CPU socket, and thus a direct physical connection exists between the NIC or the GPU and the CPU. You cannot logically change this within the ESXi schedulers. The only thing you can do is to map the virtual world as close to the physical world as possible to keep everything as clear and transparent as possible. I mapped PCI device 0 (the A30) to NUMA node 1. Running lstopo in the virtual machine provided me this result: Now the GPU is a part of NUMA node 1. How we can confirm that is true is by taking the PCI device at address 04:00:00 given in the small green box that is inside Package 1 and seeing that is the same address as that given in the “esxcli hardware pci list” for the GPU - that is seen at the line titled “Device Layer Bus Address” in that esxcli output. Because the virtual GPU device is now a part of NUMA node 1 the guest OS memory optimization can allocate memory within NUMA node 1 to store the dataset there so that it is as close to the device as possible. The NUMA scheduler and the CPU and memory scheduler within the ESXi layer attempt to follow these instructions to the best of their ability. If you want to be absolutely sure, you can assign NUMA affinity and CPU affinity at the lowest layers, but we recommend starting at this layer and testing this first before impacting the lowest scheduling algorithms. ================================================================================ Title: Could not initialize plugin ‘libnvidia-vgx.so - Check SR-IOV in the BIOS URL: https://frankdenneman.ai/2022-10-18-could-not-initialize-plugin-libnvidia-vgx-so-check-sr-iov-in-the-bios/ Date: 2022-10-18 I was building a new lab with some NVIDIA A30 GPUs in a few hosts, and after installing the NVIDIA driver onto the ESXi host, I got the following error when powering up a VM with a vGPU profile: Typically that means three things: Shared Direct passthrough is not enabled on the GPU ECC memory is enabled VM Memory reservation was not set to protect its full memory range/ But shared direct passthrough was enabled, and because I was using a C-type profile and an NVIDIA A30 GPU, I did not have to disable ECC memory. According to the NVIDIA Virtual GPU software documentation: 3.4 Disabling and Enabling ECC Memory Reserve all guest memory (all locked) was enabled, and this setting is recommended. If someone changes the memory setting of the VM at a later stage, the memory reservation is automatically updated, and no errors will emerge. I discovered that my systems did not have SR-IOV enabled in the BIOS. By enabling “SR-IOV Global Enable” I could finally boot a VM SR-IOV is also required if you want to use vGPU Multi-Instance GPU, so please check for this setting when setting up your ESXi hosts. But for completeness’ sake, let’s go over shared direct passthrough and GPU ECC memory configurations and see how to check both settings: Shared Direct Passthrough Step 1: Select the ESXi host with the GPU in the inventory view in vCenter Step 2: Select Configure in the menu shown on the right side of the screen Step 3: Select Graphics in the Hardware section Select the GPU and click on Edit - the Edit Graphics Device Settings window opens If you are going to change a setting, ensure that the ESXi host is in maintenance mode. Select Shared Direct and click on OK. Disabling ECC Memory on the GPU Device To disable ECC memory on the GPU Device, you must use the NVIDIA-SMI command, which you need to operate from the ESXi host shell. Ensure you have SSH enabled on the host. (Select ESXi host, go to configure, System, Services, select SSH and click on Start) Open an ssh session to the host and enter the following command: nvidia-smi --query-gpu=ecc.mode.current --format=csv If you want to disable ECC on your GPU (you do not need to if you use C-type vGPU Profiles for ML workloads), run the following command. Please ensure your ESXi host is in maintenance mode if you change a setting on the ESXi host. nvidia-smi -e 0 you can now reboot your host, or if you want to verify whether the setting has been changed, enter the following command: nvidia-smi --query-gpu=ecc.mode.pending --format=csv Now reboot your host, and ECC will be disabled once its powered-on ================================================================================ Title: Sub-NUMA Clustering URL: https://frankdenneman.ai/2022-09-21-sub-numa-clustering/ Date: 2022-09-21 I’m noticing a trend that more ESXi hosts have Sub-NUMA Clustering enabled. Typically this setting is used in the High-Performance Computing space or Telco world, where they need to reduce every last millisecond of latency and squeeze out every bit of bandwidth the system can offer. Such workloads are mostly highly tuned and operate in a controlled environment, whereas an ESXi server generally runs a collection of every type of workload in the organization imaginable. Let’s explore what Sub-NUMA clustering does and see whether it makes sense if you should enable it in your environment. NUMA Most data center server systems are NUMA systems. In a NUMA system, each CPU contains its own memory controllers that provide access to locally connected memory. The overwhelming majority of systems in data centers worldwide are dual-socket systems. Each CPU has access to local memory capacity via its own memory controller, but it can also access the memory connected and controlled by the remote CPU. There is a difference in latency and bandwidth between reading and writing local and remote memory, hence the term Non-Uniform Memory Access (NUMA). AMD EPYC architecture is a Multi-Chip-Module (MCM) architecture and wildly differs from the monolithic architecture and its I/O path behavior. Sub-NUMA clustering is the functionality of partitioning Intel CPU packages. AMD provides similar functionality called NUMA per Socket (NPS). This article only focuses on Intel Sub-NUMA clustering technology as I have not seen server vendors’ default enablement of the NPS setting at this moment. Logical Partitions The Intel E5-2600 processor family had a ring architecture, allowing Intel to optimize CPU core-to-memory access further. With the Haswell release (v3), Intel introduced Cluster-On-Die (COD) functionality. COD logically splits the CPU into two NUMA nodes. The COD feature reduces the search domain of the memory hierarchy. NUMA systems are cache-coherent. When a core generates a memory request, it checks its local L1 and L2 cache, the shared L3 cache (LLC), and the remote CPU cache. By splitting the CPU along its “natural” ring structure barrier, you end up with a smaller memory domain to search if there is a cache miss. And on top of that, you should have less local memory traffic if the applications and operating systems are NUMA optimized. Please look at the NUMA deep dive or read this research paper about COD caching structures for more info. The Skylake architecture (Intel Xeon Scalable Processors) moved away from the ring architecture and introduced a mesh architecture. The logical partition functionality remained and was introduced under a new name, Sub-NUMA Clustering (SNC). With SNC enabled, that previously shown dual CPU socket ESXi host system is now a four NUMA node system. Comparing SNC to NUMA Performance Is SNC that Turbo feature that is hidden in your BIOS? If you look at the description some vendors use, you want to enable it immediately. Dell and Lenovo describe SNC: “…It improves average latency to the LLC”. I’m using the performance numbers that Hadar Greinsmark published in his research “Effective Task Scheduling of In-Memory Databases on a Sub-NUMA Processor Topology.” In a default NUMA configuration (SNC disabled), let’s look at the latency difference between Socket 0 (S0) and Socket 1 (S1). Latency (ns) NUMA Node 0 (S0) NUMA Node 1 (S1) NUMA Node 0 80.8 138.9 NUMA Node 1 139.7 79.9 With SNC enabled, Socket 0 contains two NUMA nodes, 0 and 1. Socket 1 contains NUMA Nodes 2 and 3. These logical partitions are genuine NUMA nodes. Although the different memory controllers and cache domains are on the same die, the caching mechanisms and non-interleaving of memory controllers create a non-uniform memory access pattern between the domains. Consequently, there is an increase in latency when fetching memory from the other “remote” NUMA node located within the same socket. Latency (ns) NUMA Node 0 (S0) NUMA Node 1 (S0) NUMA Node 2 (S1) NUMA Node 3 (S1) NUMA Node 0 (S0) 74.2 (-7.5%) 81.5 (+0.8%) 132.0 (-5%) 142.1 (+2.3%) NUMA Node 1 (S0) 82.0 (+1.4%) 76.4 (-5.4%) 135.6 (-2.4%) 144.5 (+4%) NUMA Node 2 (S1) 132.4 (-5.2%) 142.0 (+1.7%) 73.6 (-7.9%) 81.5 (+2%) NUMA Node 3 (S1) 136.0 (-2.6%) 144.4 (+3.4%) 81.5 (+2%) 76.6 (-4.1%) The SNC mapping method of memory addresses from the local memory controller to the closest LLC certainly works as the local NUMA latency with SNC drops on average between 6 to 7% compared to the default NUMA configuration. Take NUMA node 0 as an example. With SNC enabled, it experiences a memory latency of 74.2 ns; compared to SNC disabled, the access latency is 80.8 ns. As a result, SNC specifically reduces memory latency by 7.5% for NUMA node 0. The performance numbers show that remote connections handled by node 0 and node 2 are performing better than in the SNC disabled state, whereas NUMA node 1 and node 3 are performing less than in the SNC disabled state. The latency numbers reported to the remote node on the same socket are very interesting. It possibly shows the fascinating behavior of the interconnect architecture. However, Intel does not share detailed information about the Ultra Path Interconnect (UPI) framework. If we look at the architecture diagram, we notice that above NUMA node 0, a controller exists with two UPI connections. Above NUMA node 1, a controller is located with a single UPI connection. Possibly the single-UPI experiences more blocking I/O traffic on the mesh, whereas the UPI controller with two connections has more methods to manage the flow better. But this is just pure speculation on my end. The latency shoots up if we look at the absolute remote I/O numbers. What matters is that workloads execute remote I/O operations if they cannot read or write memory locally. If it can find the memory in the NUMA node located on the same socket, it sees a latency increase of 8.6%. When it travels across the interconnect to a NUMA node in the other socket, the latency increases to 78.2%. When it needs to travel to the farthest NUMA node, latency almost doubles (90%). A default NUMA system has an average remote latency hit of 73%. But SNC has a more extensive performance spread as it improves up to 7% on average locally but also degrades remote memory access up to 5%. Let’s compare. In the default situation, local access is 80 ns, and remote access is 138.9 ns. With SNC, it has to deal with a worst-case scenario of 73.6 ns vs. 142.0. Which is the reason why the performance gap extends to 92.9%. And what decides what workload becomes local and remote? That is the key of this article. But before we dive into that, let’s look at bandwidth performance first. Bandwidth Your Skylake CPU model has either two or three UPI links. Each UPI link is a point-to-point full duplex connection with separate lanes for each direction. A UPI link has a theoretical transfer speed of 10.4 Gigatransfers per second (GT/s), which translates to 20.8 gigabytes per second (GB/s). The Intel Xeon Platinum 8180 processor used in the report contains three UPI links, possibly providing an aggregated theoretical bandwidth of 62.4 GB/s. One controller has two UPI links, and the other controller has one UPI link. The research paper shows that when communicating with a remote node, the average bandwidth is roughly 34.4 GB/s. Speculation As limited information about UPI communication patterns is available, I assume the system uses only two UPI links in the default NUMA node. With default NUMA, memory interleaves across memory controllers; thus, memory has to be retrieved from both memory controllers, and therefore, the systems use both UPI controllers. Why it doesn’t use all three links, I think the overhead of syncing the I/O operation across three links and allowing other operations to use the interconnect outweighs the potential benefit of additional uplink. /speculation But let’s stick to the facts. Here you see the impact of remote memory access. Bandwidth performance drops 69% when doing remote I/O on the default NUMA system. And for this exact reason, you want to have NUMA optimized workloads or right-sized virtual machines. Why doesn’t the progress bar move linearly on your screen? Possibly some non-NUMA optimized code fetching memory from the remote NUMA node. Bandwidth (MB/s) NUMA Node 0 (S0) NUMA Node 1 (S1) NUMA Node 0 111 083 34 451 NUMA Node 1 34 455 111 619 With SNC enabled, the system stops interleaving the whole memory range across both memory controllers within the CPU package and assigns each memory controller a subset of the memory range. Each memory controller has three channels, splitting the NUMA node’s bandwidth in half. The test system uses DDR4 2666 MHz (21.3 GB/s) memory modules, theoretically providing up to 63.9 GB/s per SNC NUMA node. When reviewing the research findings, the default NUMA node provided 111 GB/s (6 channels) by enabling SNC, which should result in approximately 55.5 GB/s per NUMA node. Yet the test results report 58 GB/s. SNC improves local bandwidth by an average of 4.5% due to the isolation of workload and, therefore, fewer blocking moments of other I/O operations on the mesh. Similar improvements occur for the NUMA node on the same socket. Bandwidth (MB/s) NUMA Node 0 (S0) NUMA Node 1 (S0) NUMA Node 2 (S1) NUMA Node 3 (S1) NUMA Node 0 (S0) 58 087 58 123 34 254 34 239 NUMA Node 1 (S0) 58 145 58 013 34 266 34 235 NUMA Node 2 (S1) 34 288 34 248 58 064 58 147 NUMA Node 3 (S1) 34 288 34 254 58 145 58 007 Therefore, SNC is a great way to squeeze out that last bit of performance for a highly tuned workload. If the workload fits inside the smaller NUMA node from a core count and memory capacity, it can expect a 7% improvement in latency and a 4% memory bandwidth. But, and there is a big but, but not the way Sir Mix-a-Lot likes it. But only if you deploy a scale-out workload. If you can deploy two workers in each worker node that run separate workloads, they both can benefit from extra obtainable bandwidth. If you only deploy a single workload, you’ve just robbed that workload of half its obtainable bandwidth. The workload can access remote memory capacity and possibly obtain more bandwidth. Still, it’s up to the NUMA scheduler or application’s discretion to make the smart move and choose the right NUMA node. And this is the point where we arrive at the fork in the road, the difference between a dedicated workload on bare metal and dealing with a NUMA scheduler that must take entitlement and workload patterns into account of multiple workloads. SNC effectively reduces the per-NUMA node capacity and, thus, decreases the boundary of the single NUMA node virtual machine size. I have a question: Do 4% bandwidth improvement for scale-out the workload and 7% latency improvement sound like something you want to enable on a virtualization platform? How is a 10-vCPU VM placed on a dual socket with 18 cores per CPU package system? Single NUMA node VM sizing Not every application is NUMA aware. Most platform operators and admin teams attempt to “right-size” the virtual machine to circumvent this problem. Right-sizing means the VM contains fewer vCPUs and memory capacity than the CPU socket contains CPU cores and memory capacity, yet still can function correctly. With SNC, the NUMA node is split in half, resulting in smaller VMs if they need to fit inside a single NUMA node. vNUMA Topology If a VM contains more vCPUs than a NUMA node contains CPU cores, the NUMA scheduler in ESXi creates a vNUMA topology for this VM and exposes this to the guest OS for NUMA optimizations. The NUMA scheduler creates multiple NUMA clients for this VM and places these accordingly, which is the key to understanding why SNC should or shouldn’t be enabled in your environment. Initial placement The NUMA scheduler gets an initial placement request if a VM is powered on or a virtual machine is migrated into the host via DRS. During the initial placement operation, the NUMA scheduler is aware of the distance between the NUMA nodes. And it will attempt to optimize the placement of the NUMA clients of the VMs. In other words, it attempts to place the NUMA clients as close to each other as possible. As a result, most of the time, if a VM consists of two NUMA clients, both NUMA clients are placed on NUMA nodes sharing the same socket with SNC enabled. Typically this happens during every synthetic test where this VM is the only VM running on the host; thus, the NUMA scheduler does not have to deal with contention or a complex puzzle or to fit these new clients amongst the other busy 44 other NUMA clients. NUMA load-balancing The hypervisor is a very dynamic environment. The CPU scheduler has to deal with a variety of workload patterns, and there are workload patterns such as load correlation and load synchronicity. With load correlation, the schedulers must deal with the load spikes generated by the relationship between workloads running on different machines. The NUMA scheduler reviews the CPU load every 2 seconds to catch these patterns. For example, an application with a front-end VM communicates with a database. With load synchronicity workloads trend together, a VDI environment that spins up several desktops each morning will cause a persistent load spike. And so, the NUMA load-balancer might decide that it’s better to move some NUMA clients around the system. Getting into the NUMA load balancing algorithm’s details is too deep for this article. I’ve covered most in the NUMA deep dive. But the crucial thing to understand is that if it’s necessary to move the NUMA client, it will move the NUMA client, but it won’t take distance into account. The attempt will be the smartest thing to do for the system, but it might not always be the best for the VM. In many cases, If you would not enable SNC, that VM would have fit inside a single NUMA node, and no remote access occurred as it would fit a single NUMA node. With SNC, a large VM might be larger than the SNC-NUMA node size and thus is split up. It’s even worse if this VM connects to a PCIe device such as a GPU. Batch data transfers can occur across the interconnect, creating inconsistent data loading behavior during the host-to-device memory transfer operations. Learn more about NUMA PCI-e locality here. SNC Enabled by Default Why am I making this statement? I discovered that HP enabled SNC with the workload profiles “Virtualization - Max Performance” and “General Throughput Compute.” ESXi does not have a setting in the UI that shows whether SNC is enabled or not, but we can apply the beautiful art form of deduction. By running the following (unsupported) command via SSH on the ESXi host: echo "CPU Packages";vsish -e dir /hardware/cpu/packageList;echo "NUMA nodes";vsish -e dir /hardware/cpuTopology/numa/nodes You get a list of the number of CPU packages (a fancy name for the device you can hold in your hand that contains the CPU cores, memory controllers, and PCI controllers) and the number of NUMA nodes in the system. If SNC is disabled, the number of NUMA nodes should equal the number of CPU packages. In this scenario, SNC is enabled. In most systems, you can enable and disable the setting individually, but if it’s part of a profile such as on the HP systems, you need to customize this. Dan tweeted the method to do this. https://twitter.com/Casper042/status/1567680625423024129 Disclaimer! Please note that this tweet and this article are not official VMware recommendations to turn off SNC. This article will help you understand the implication of SNC on the overall behavior of SNC on the hardware layer and the way ESXi NUMA scheduler works. Always test a setting in your environment and your workload in a normal operating condition before changing a production environment. My Personal Thoughts on SNC I believe that SNC has very little to offer to a virtualization platform that runs all types of workloads. Workloads that range from tiny to monster VMs. If you have a dedicated vSphere platform running a specific low-latency workload that needs to run, for example, VOIP workload or High-Frequency Trading workload, then SNC makes sense. For the average vSphere environment that runs Oracle, SQL, or any high-performing database that needs lots of vCPUs, along with some front-end applications and a whole bunch of other frameworks, SNC will impact your performance. Most admins want the VM to fit inside a single NUMA node. SNC reduces the VM footprint. SNC taxes memory access more severely. Due to the increased gap between local and remote I/O, the user will detect an even more inconsistent feel of workload performance. The NUMA scheduler now needs to balance four smaller NUMA domains instead of two larger ones; thus, more decisions will be made that might not be optimal. Take Short-Term Migration, for example. The NUMA scheduler moves a VM to solve an imbalance between NUMA nodes. In that scenario, the scheduler migrates the vCPU immediately, but the memory follows more slowly. Since memory relocation is not immediate, remote memory access will temporarily increase while the pages migrate to the new NUMA node. With four smaller NUMA nodes to deal with, this can impact the overall user experience, especially if the gap between local and remote memory is enlarged from 69% to 90%. VMs that could be Uniform Memory Access VM (fit inside a single NUMA node) now span multiple NUMA nodes. And as a result, we hope the guest OS and the application are NUMA optimized. Recent Linux optimization to their NUMA scheduler makes me hopeful, but keeping the host to default NUMA would avoid so many performance inconsistencies. In essence, It is my opinion that SNC is for the highly optimized, well-curated environment that has exploited every single trick in the book. It should not be the starting point for every virtualization platform. Want to learn more about NUMA? We spoke with the driving force behind NUMA optimizations at VMware in episode 19 of the Unexplored Territory Podcast. You can listen to the conversation with Richard Lu via the Unexplored Territory website, Apple Podcasts or Spotify ================================================================================ Title: VMware Sessions at NVIDIA GTC URL: https://frankdenneman.ai/2022-09-20-vmware-sessions-at-nvidia-gtc/ Date: 2022-09-20 Overcome Your AI/ML Challenges with VMware + NVIDIA AI-Ready Enterprise Platform (Presented by VMware, Inc.) [A41422] Tuesday, Sep 20, 8:00 PM - 9:00 PM CEST / 11:00 Pacific Time (PT) Shobhit Bhutani, Justin Murray, and I are honored to present at NVIDIA GTC. In the [session](http://Overcome Your AI/ML Challenges with VMware + NVIDIA AI-Ready Enterprise Platform (Presented by VMware, Inc.) [A41422]), we provide a deep-level overview of the VMware and NVIDIA AI-Ready Enterprise platform and the new ML-focused features of the vSphere 8 release. You can join online and register for free at nvidia.com/gtc/ Scale, Secure, and Boost Infrastructure Performance with DPUs (Presented by VMware Inc.) [A41448] Ragav Gopalan and Dave Morera will discuss the vSphere Distributed Services Engine at GTC. DSE, formally known as Project Monterey, is VMware’s story around SmartNICs or Data Processing Units (DPUs). Unfortunately, the scheduler tool does not reveal the time of this session yet, so please check often. Hope to see you on our sessions! ================================================================================ Title: vSphere 8 and vSan 8 Unexplored Territory Podcast Double Header URL: https://frankdenneman.ai/2022-09-05-vsphere-8-and-vsan-8-unexplored-territory-podcast-double-header/ Date: 2022-09-05 This week we released two episodes covering the vSphere 8 and vSan 8 releases. Together with Feidhlim O’Leary, we discover all the new functions and features of the vSphere 8 platform. You can listen to this episode on Spotify, Apple, or on our website: unexploredterritory.tech Pete Koehler repeats his stellar performance of last time and helps us understand the completely new architecture of vSAN 8. You can listen to this episode on Spotify, Apple, or on our website: unexploredterritory.tech or anywhere else you get your podcasts! ================================================================================ Title: Unexplored Territory - VMware Explore USA Special URL: https://frankdenneman.ai/2022-09-02-unexplored-territory-vmware-explore-usa-special/ Date: 2022-09-02 This week Duncan and I attended VMware Explore to co-present the session “60 Minutes of Virtually Speaking Live: Accelerating Cloud Transformation.” with William Lam and our buddies of the Virtually Speaking Podcast, Pete Flecha and John Nicholson. The recordings should be made available soon or sign up for the session in Barcelona. During the week, we caught up with many people and captured soundbites of people such as Kit Colbert, Chris Wolf, Stephen Foskett, Sazalla Reddy, and a few more. You can listen to this special VMware Explore episode on Spotify (spoti.fi/3cITI7p), Apple (apple.co/3q35dJJ) or our website: unexploredterritory.tech/episodes/ ================================================================================ Title: New vSphere 8 Features for Consistent ML Workload Performance URL: https://frankdenneman.ai/2022-08-30-new-vsphere-8-features-for-consistent-ml-workload-performance/ Date: 2022-08-30 vSphere 8 is full of enhancements. Go to blogs.vmware.com or yellow-bricks.com for more extensive overviews of the vSphere 8 release. In this article, I want to highlight two features of the new vSphere 8 version that will help machine learning (ML) workloads perform consistently and possibly faster than manually configured workload constructs. The two features which make this possible are UI enhancements for the vNUMA Topology and the Device Groups. Hardware 20 Scalability Enhancements Before we dive into the features, vSphere 8 introduces a new virtual hardware feature that allows us to introduce new wonderful things and push the boundaries again of the platform. With vSphere 8, the virtual hardware level advances to version 20, and it offers new capabilities for ML accelerators. The support for DirectPath I/O devices went up from 16 to 32. We also worked with NVIDIA to increase the support of the vGPU devices, and now, with vSphere 8, each ESXi host can support up to 8 vGPU devices. These enhancements improve the spectrum of the ML accelerator range tremendously. With vGPU, the platform team, or in some cases, the MLOPs team, can create workload constructs (VMs, containers) and utilize fractional GPU resources that allow the data scientists to run some light testing or compartmentalize GPUs for inference workloads. At the other end of the spectrum, we have the work beast for training workloads, the multi-GPU configurations. We offer these technologies host-local and remote with VMware Bitfusion technology for allowing fast attach and detach of workloads and hardware resources. In the diagram, the orange dots indicate vSphere 7 maximum supported devices. The blue dots indicate vSphere 8. Simplified Virtual NUMA Configuration The device assignment functionality in the new vSphere 8 UI of the vNUMA Topology helps VI-admins and MLOPs teams assign the vCPU and GPU of a VM to the same NUMA node. This feature improves the possibility that the memory of the VM remains the same NUMA node as the GPU. I wrote an extensive article about this in January 2020, “Machine Learning Workload and GPGPU NUMA Node Locality.” The idea of the script is now codified correctly in the official product, a personal highlight to see for me. Device Groups Device Groups is a brilliant new feature. And before we dive into device groups, we have to look at Dynamic DirectPath I/O. Before Dynamic Direct Path, a VI-admin specified a GPU device by PCI location. That meant that the VI-admin must track what ESXi hosts have which devices and what VMs are using those devices. The VI-admin selects that particular PCI address and constraints that VM to run only on that particular device. With the introduction of hardware labels, Dynamic DirectPath I/O (DDIO) allows a VM-admin to specify the kind of device to add to a VM. Niels Hagoort wrote a very informative article about Dynamic DirectPath I/O with its initial product name title: “vSphere 7 – Assignable Hardware.” The problem is that DDPIO is only for one device, but as I have shown at the beginning of the article, we support the full spectrum of ML accelerator configurations. What if a data science team requires a multi-GPU configuration? Multi-GPU configuration is an infrastructure way of looking at this. The data science teams call it distributed training or distributed deep learning. The workload distribution happens between GPUs within an ESXi host or across multiple ESXi hosts. That’s where device groups come into play. With Device Groups, vSphere 8 allows the VI-admin or MLOps team to create a configuration for workloads requiring multiple GPUs connected by a high-speed link or devices that must be on the same PCI switch. Distributed workloads running across GPUs located on multiple ESXi hosts want the lowest possible latency. The interconnect between the ESXi hosts receives the most attention, but the path from the GPU to the external interconnect is also essential. To minimize latency, we have to take the NUMA locality of both the GPU and the NIC into account. Modern CPUs have PCI controllers baked into the CPU package; thus, NUMA PCI-Locality exists. To provide consistent performance, you must select devices connected to the same PCI controller or PCI switch (available in large systems). A high-speed interconnect between GPU accelerators allows for a stable, consistent high bandwidth to ensure the most performance from the available local hardware. NVIDIA offers NVLINK, a direct GPU to GPU interconnect. An A30 card offers one link per card. An a100 is equipped with three links, offering 150 GB/s GPU to GPU bandwidth. Each link provides 50 GB/s of theoretical bandwidth per link. Device Groups allow VI-admins of MLOPs team to add these multiple devices as a single unit to a virtual machine. More in-depth articles about these features will follow in the upcoming weeks. ================================================================================ Title: Training vs Inference - Network Compression URL: https://frankdenneman.ai/2022-08-26-training-vs-inference-network-compression/ Date: 2022-08-26 This training versus inference workload series provides platform architects and owners insights about ML workload characteristics. Instead of treating deep neural networks as black box workloads, ML architectures and techniques are covered with infrastructure experts in mind. A better comprehension of the workload opens up the dialog between infrastructure and data science teams, hopefully resulting in better matching workload requirements and platform capabilities. Part 3 of the series focussed on the memory consumption of deep learning neural network architectures. It introduced the different types of operands (weights, activations, gradients) and how each consumes memory and requires computational power throughout the different stages and layers of the neural network. Part 4 showed that a floating point data type impacts a neural network’s memory consumption and computational power requirement. I want to cover neural network compression in this part of the training versus inference workload deep dive. The goal of neural network compression is inference optimization to either help fit and run a model at a constrained endpoint or to reduce inference infrastructure running costs. A data science team’s goal is to create a neural network model that provides the highest level of accuracy (Performance in data science terminology). To achieve high levels of accuracy, data science teams feed high-quality data sets to the ML platform and execute multiple training runs (epochs). The ML community builds newer, more complex, and more extensive neural networks to improve precision. The chart below shows the growth of parameters of image classification (orange line) and Natural Language Processing (blue line) in state-of-the-art (SOTA) neural network architectures. If we deconstruct any neural network architecture, we can see that each neural network has different layers and operands, i.e., weights, activations, and gradients. These layers and operands impact a model’s performance and inference time. Data scientists select an appropriate floating-point data type to reduce the neural network model’s memory utilization and increase the processing speed. Sometimes, the neural network size (the memory footprint) prohibits successful deployment to the target production infrastructure. For example, it can be an edge deployment onto a particular device, a physical space with a restricted-energy envelope. As a result, the data science team can optimize the network even further by performing quantization. Post-training quantization converts floating point data points into integers. If done smartly, it can reduce the neural network memory footprint tremendously while retaining accuracy. An additional technique to improve the efficiency of the algorithm is pruning. Pruning and quantization go hand in hand. The CERN Large Hadron Collider team is exploring a Quantization-Aware Pruning technique. Pruning Pruning helps to identify the important connections within the neural network and uses methods to remove either the connection to individual weights (unstructured pruning) or remove the connection of groups of weights by disconnecting an entire channel or filter (structured pruning). The most popular frameworks, like TensorFlow (Keras) and Pytorch, contain standard modules to perform unstructured pruning on neural networks. An interesting thing about pruning is that many online literature and research papers use the terms remove or delete weights. It does not change the neural network layout. See the screenshot of the Optimal Brain Damage research paper, or check out the PyTorch Pruning tutorial. When replacing the trained parameter with a zero, sparsity is introduced into the tensor or dense matrix data structure. Sparsity is the proportion of zero to non-zero weights. Algorithms can use sparsity to speed up or compress the footprint of the neural network. Pruning is possible as many weights in a trained neural network end up close to the value of zero. Many researchers believe that most neural networks are over-parameterized. As a result, pruning has been a hot topic since the 90s. There have been some influential papers that are still referenced today and used as the starting point for research on new pruning techniques: In the paper “Optimal Brain Damage,” LeCun et al. discover that “reducing the size of a learning network**”** improved generalization (the neural network’s ability to adapt correctly to new, previously unseen data) and inference speed. Fast forward to 2015, Han et al. published “deep compression,” combining pruning, trained quantization, and Huffman coding to reduce the neural network footprint for mobile and other low-power applications. The pruning mechanism is (unstructured) magnitude-based, the most common today. Magnitude-based pruning assumes that the weight with the smallest value has the most negligible contribution to the neural network’s performance and removes those weights. Interestingly, if you train a network and prune the network connections, you end up with a neural network that retains its accuracy but has more than 50% of fewer parameters. However, if you start training with that neural network architecture, it will not achieve that high accuracy level. In the paper “The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networks” (2019), Jonathan Frankle and Michael Carbin asked why training a network with the topology of the pruned network yields worse performance. They conclude that within an extensive neural network, a smaller neural network exists that would match the performance of the larger one. This “winning lottery ticket” subnetwork exists, but you can only discover it if the same weight initializations are used as the original networks. And therefore, it’s almost impossible to discover that smaller network before you train the larger one to completion. The paper hypothesis automatically makes you wonder as a platform operator/architect about the approach data scientists use to optimize their models. Why not start training with a small neural network and slowly build more extensive networks? (constructive approach). Getting a neural network to react to new data correctly (generalize) with a few parameters is complicated. There seems to be no traction in the research space to investigate “Constructive neural network learning” thoroughly. Pruning Scheduling Generally, the data science team takes a destructive approach to model development. Multiple epochs are used to train the complete neural network with all its parameters to achieve the highest accuracy possible. The next step is to apply a pruning method in which learned parameters are set to zero, and the connections are stripped away. The most popular pruning method is “train, prune and fine-tune.” Pruning takes place after training. The data science team determines a pruning percentage. The pruning percentage indicates the number of learned parameters that will be set to zero across the neural network. Once the pruning is complete, the neural network retrains to “recover from the loss of parameters " these two steps are one iteration. Please keep in mind that one iteration can contain multiple epochs of training. The data science team can choose to execute the pruning method in different ways that affect the time the accelerators are in use and when they are idling. There are two mainstream methods that I want to highlight: One-shot Pruning Iterative Pruning One-shot pruning prunes the neural network to a target sparsity level in one iteration. The deep compression paper showed that pruning and retraining are repeated iteratively until the target sparsity threshold is met, typically resulting in higher accuracy than one-shot pruning. It’s common to prune 20% of the weights with the lowest magnitudes during one iteration. Generally, iterative pruning is computationally intensive and time-consuming, especially if the global target sparsity is high and the number of parameters removed at each iteration is low. The pruning extends the use of accelerator time, but downtime occurs between iteration sessions. It can be a very “jerky” process from an accelerator utilization perspective. The iterative process introduces additional criteria besides the pruning strength: Pruning Strength: How many weights should be removed? Saliency: Which weights should be pruned? Pruning Stop Condition: When should the pruning process end? Commonly, the data science team evaluates the progress between iterations. During this time, the accelerator is typically assigned to a workload construct (VM, container), yet it is not productive. When reviewing accelerator use for pruning purposes, it’s not uncommon to see the same number of epochs used for training. The paper “Retrain or Not Retrain? - Efficient Pruning Methods of Deep CNN Networks” shows a great example: Retraining of the pre-trained Resnet50 with global sparsity of 20%. Retraining starts at epoch 104. Top5 is marked in blue, and Top1 in red. Sparsity Introducing sparsity allows for efficient compression. There is the machine learning definition of compression (i.e., sparsity) and the definition we have used since Robert Jung blessed us with ARJ in MS-DOS and when Winzip burst onto the scene in 1991. Let’s use the old-school definition for now. A pruned neural network is highly susceptible to efficient compression, as you have millions of zeroes floating around. Compression is perfect if you want to reduce the model file size for the model’s distribution to mobile devices. Mobile devices have limited memory, and as the data compression paper points out, there is a power consumption difference between data retrieval from cache or SRAM. But for large retail organizations or Telco companies dealing with countless edge locations, reducing the model size can significantly speed up the distribution of the model. But pruning is not easy, and pruning will not automatically guarantee success. Results vary widely, depending on the neural network type and task. Some architecture responds better to pruning than others. And then, of course, there is the hardware. Replacing a trained weight for a zero doesn’t make it easier for the hardware. We, as humans, know that when we multiply by zero, the answer is … zero, and any number added to 0 is equal to itself. So we take shortcuts, but this calculation needs to be executed for a computer. It cannot be ignored unless we include this logic in an algorithm. The focus for pruning is to retain similar accuracy while increasing sparsity. However, feeding a dense matrix data structure riddled with zeroes can cause irregular memory access patterns for accelerators. And so, a sparse, dense matrix data structure isn’t always faster. The data science team has to apply more optimizations or choose structured pruning to speed up successfully on particular accelerator devices. But removing entire filters or layers dramatically changes the neural network structure’s layout and reduces the neural network accuracy. Hardware vendors have also been researching this field for years, especially NVIDIA. The papers “Learning both weights and connections for efficient neural networks” and “Exploring the granularity of sparsity in convolutional neural networks” by Jeff Pool et al. (Senior Architect NVIDIA) are interesting reads. With the Ampere Architecture, NVIDIA introduced sparse tensor cores and Automatic Sparsity (ASP), a concept to generate the correct sparsity level for the hardware to accelerate. NVIDIA Sparsity Support Part 5 - Numerical Precision showed the spec sheet of an NVIDIA A100 (Ampere architecture), listed 624 TOPS for INT8 operations, and listed 1248 TOPS with an asterisk. Those 1248 TOPS are sparse tensor core operations following the 2:4 structured-sparse matrix pattern prescribed by NVIDIA. This predefined pattern means any pruned neural network can get the best performance from an Ampere accelerator (A2, A30, A40, A100). It has to follow the 2:4 structured-sparse matrix pattern. And that means that two values out of each contiguous block of four must be zeroed out. In the end, 50% of the trained values across the network are replaced by a zero. This method’s beauty is that it uses metadata to track where those zeroes are stored. To some, this sounds like a sparse matrix data structure format. But the problem is that many machine learning libraries do not offer support for sparse matrices. So NVIDIA uses a compressed format of a dense matrix data structure which contains the trained weights and a metadata structure that is necessary for sparse tensor cores to exploit the sparsity. This format is fed into the sparse tensor cores to get that speed up. Thus from an infrastructure perspective, you need to have your development/inference infrastructure in lockstep. They have to run both the Ampere architecture. Possibly A100 for training and pruning with ASP operations, A2 accelerator cards for inference operations. A 2:4 structured sparse matrix W and its compressed representation (source NVIDIA) The smart part is in the metadata. The metadata stores the positions of the non-zero weights in the compressed matrix. We must look at a standard General Matrix Multiplication operation (GEMM). In a forward propagation operation, you have a matrix (A) with weights, A matrix with activations (B), and you capture the output in Matrix C. Matrix A and B are identical in dimensions for mapping the weights and activations. What happens with the sparse operation of the Ampere architecture? The non-zero weights are in matrix A. This one is now in a compressed state, so the dimensions do not match matrix B anymore. It only needs to pull the values from the other matrix to perform the sparse operation. It needs to perform the multiplication with the trained weight. The metadata allows the algorithm to do that. As matrix A only contains non-zero values, the metadata helps the algorithm to know precisely which activation to pull from B to match up with the train values within the compressed matrix A. What’s next for Pruning? NVIDIA is looking into incorporating ASP into training operations, and I can imagine this will be a significant unique selling point. As of this point, data scientists use a huge budget to train the model before starting the activities to optimize the neural network for edge deployment. These activities are not always successful. Pruning requires long fine-tuning times that could exceed the original training time by a factor of 3 or sometimes even more. Pruning consumes tremendous amounts of resources, whether on-prem or OPEX, without proper guarantees. NVIDIA has proven that sparsity can be leveraged, it is just a matter of time before other solutions pop up. ================================================================================ Title: How to Write a Book - Show Up Daily URL: https://frankdenneman.ai/2022-08-01-how-to-write-a-book-show-up-daily/ Date: 2022-08-01 During the Belgium VMUG, I talked with Jeffrey Kusters and the VMUG leadership team about the challenges of writing a book. Interestingly enough, since that VMUG, the question of how to start writing a book kept appearing in my inbox, dm, and Linkedin Messaging regularly. This morning Michael Rebmann’s question convinced me that it’s book writing season again, so maybe it’s better to put my response in a central place. https://twitter.com/_michaelrebmann/status/1553498293736538116 There are millions of books and millions of authors, and that makes me believe a million ways to write a book. Here is what works best for me. Hopefully, there is something in it that will work for you too. Show up daily The biggest thing you can do to help you succeed is to show up daily. I suppose this can be applied to anything in life, but it applies to writing books especially. The key to understanding is the level of showing up, i.e., your output. You simply cannot write thousands and thousands of words every day. You will have high energy days and low energy days. Days spent on research. Days spent questioning simple things that lead to rabbit holes such as this: “How much MB is there in a GB? What about Gibibyte? And which one will I be using in the book? How many tables do I need to convert now?” You will lose much time and energy on things that will not show up in the book. And that can have a demoralizing effect and give you a feeling that you are not making any progress. Especially when you are still operating under the impression that you must write a couple of thousands of words daily. And this is not true, at least not in my experience. Showing up means doing something. This cartoon, full credits to @saraharnoldhall says it all. Add some words to your draft each day. It doesn’t have to be a lot. Sometimes you have good days. Sometimes you have bad days. People who have known me a bit longer and listened to my podcasts might know I suffer from migraines. Writing books while being a migraine patient is not an ideal combination. You cannot “send it” every day. So you have to work around the bad days. On my bad days, I try to do light work. I reorganize my system. Clean out the whiteboard, or if possible, I will remove the interrupts. Because when I have a good day, I don’t want to get interrupted. Getting rid of interrupts In 2019, I wrote an article about the three books that helped me focus, Getting things done, Essentialism, and KonMari. Getting rid of stuff, ensuring everything is in the right place and only getting the stuff you need helps you eliminate interruptions. Maybe you can’t do that for your entire household, but try to do it in your office. When you want to write, you do not get interrupted by things that need your attention. You can focus on the things you want to focus on. What also helps me to retain focus is music. Preferably music without any lyrics as that seems to distract me from writing, I’ve created a 33-hour Spotify list that helps me to zone in, but I know it’s not everyone’s taste. Tools I use Evernote to store links to interesting articles and research papers to organize my thoughts. Grammarly heavily corrects my English, and I use Omnigraffle for my diagrams. Keeping organized properly is a timesaver long term. You will go back to your notes often, and I lost countless hours finding that one paragraph with that one datapoint that verified my thought. Safe everything, and label everything correctly. The last thing I want to say is just do it. Write that book that you wanted to write. Cover that topic from your perspective. Make sure it is factually correct, but add some personal flavor to it. To quote Rick Rubin, “Make what you love, whatever it is, be your own audience. So make the thing you love for you, the audience.” ================================================================================ Title: Training vs Inference - Numerical Precision URL: https://frankdenneman.ai/2022-07-26-training-vs-inference-numerical-precision/ Date: 2022-07-26 Part 4 focused on the memory consumption of a CNN and revealed that neural networks require parameter data (weights) and input data (activations) to generate the computations. Most machine learning is linear algebra at its core; therefore, training and inference rely heavily on the arithmetic capabilities of the platform. By default, neural network architectures use the single-precision floating-point data type for numerical representation. However, modern CPUs and GPUs support various floating-point data types, which can significantly impact memory consumption or arithmetic bandwidth requirements, leading to a smaller footprint for inference (production placement) and reduced training time. Let’s look at a spec sheet of a modern data center GPU. Let’s use the NVIDIA A100 as an example. I’m aware that NVIDIA announced the Hopper architecture, but as they are not out in the wild, let’s stick with what we can use in our systems today. This overview shows six floating-point data types and one integer data type (INT8). Integer data types are another helpful data type to optimize inference workloads, and this topic is covered later. Some floating-point data types have values listed with an asterisk. Sparsity functionality support allows the A100 to obtain these performance numbers, and sparsity is a topic saved for a future article. What do these numbers mean? We have to look at the anatomy of the different floating-point data types to understand the performance benefit of each one better. Anatomy of a Floating-Point Data Type A fantastic 11-minute video on YouTube describes how floating-point works very well. I’ll stick to the basics that help frame the difference between the floating-point data types. The floating-point format is the standard way to represent real numbers on a computer. However, the binary system cannot represent some values accurately. Due to the limited number of bits used, it cannot store numbers with infinite precision; thus, there will always be a trade-off between range and precision. A wide range of numbers is necessary for neural network training for the weights, activations (forward pass), and gradients (backpropagation). Weights typically have values hovering around one, activations are magnitudes larger than one, and gradients are again smaller than one. Precision provides the same level of accuracy across the different magnitudes of values. Different floating standards exist, each with different configurations to provide range and precision. The floating-point format uses several bits to specify the decimal point placement. The floating-point bit range consists of three parts, the sign, the exponent, and the significand precision (sometimes called the mantissa). The sign bit tells us whether the value is positive or negative. The exponent part in the floating-point number tells the system where to place the decimal point. The significand precision part represents the actual digits of the number. Modern GPU specs list the following three IEEE 754-2008 standards; Double precision (FP64) consumes 64 bits. 1 bit for the sign value, 11 bits for the exponent, and 52 for the significand precision. Single precision (FP32) consumes 32 bits. 1 bit for the sign value, 8 bits for the exponent, and 23 bits for the significand precision. Half precision (FP16) consumes 16 bits. 1 bit for the sign value, 5 bits for the exponent, and 10 for the significand precision. Click on the image to enlarge Let’s place these floating points data types in the context of deep learning. FP64 is typically not used in neural network computations as these do not require that high precision. High-Performance Computing (HPC) simulations use FP64. When reviewing GPU specs, FP64 performance shouldn’t be your first concern if you build an ML-only platform. Single precision is the gold standard for training. Weights, activations, and gradients in neural networks are represented in FP32 by default. But much research showed that for deep learning use cases, you don’t need all that precision FP32 offers, and you rarely need all that much magnitude either. When using FP16 for training, memory requirements are reduced by fifty percent. Fewer bits to process means fewer computations are required, so the training time should be significantly faster. But unfortunately, there are some drawbacks. First of all, you cannot easily use FP16. It’s not a drop-in replacement code. The data scientist has to make many changes to the model to use FP16. The range offered by FP16 is significantly smaller than FP32 and can introduce two conditions during training. An underflow condition where the number moves toward zero, and as a result, the neural network does not learn anything or the overflow condition where the number becomes so large that it learns nothing meaningful. As you can imagine, underflow and overflow conditions are something data scientists always want to avoid. But the concept of reducing memory consumption is alluring. The industry started to work on alternatives. One alternative that is now widely supported by CPU and GPU vendors is BFLOAT16. BFLOAT16: Google Brain developed Brain Floating Point (BF16) specifically to reduce the memory consumption requirements of neural networks and increase the computation speeds of the ML algorithms. It consumes 16 bits of memory, 8 bits for the exponent, and 7 for the precision. For more information about BFLOAT16, see A Study of BFLOAT16 for Deep Learning Training. BF16 provides the same range of values as FP32, so the conversion to and from FP32 is simple. FP32 is the default data type for deep learning frameworks such as Pytorch and TensorFlow. Click on the image to enlarge Google explains the advantages of BFLOAT16 in one of their blogs: The physical size of a hardware multiplier scales with the square of the mantissa width. With fewer mantissa bits than FP16, the bfloat16 multipliers are about half the size in silicon of a typical FP16 multiplier, and they are eight times smaller than an FP32 multiplier! The quote tells us that the BF16 workload with seven precision bits takes half the silicon area compared to the FP16 workload that uses ten precision bits. If you compare BF16 to FP32, you can do with a system that has an eight times smaller silicon area. For Google, which designs its own ML accelerator hardware (TPU) and runs its own ML services on top of it, reducing its workload footprint is a tremendous cost saver and a service enabler. BF16 is more or less a truncated version of FP32, and with minimal code conversion, it can replace FP32 code. It does not require techniques such as loss scaling, which attempts to solve the underflow problem occurring with FP16, reducing boat-loads of the data scientists’ headaches. On top of that, BF16 allows the data scientist to train deeper and wider neural network models. Fewer bits to move means fewer throughput requirements, and fewer bits to compute means less arithmetic complexity, meaning less silicon area required per experiment. As a result, BF16 allows data scientist to increase their batch size or create more extensive neural networks. BF16 is becoming a prevalent floating point data type within the data science community. Look for hardware that supports the BF16 data type, such as the NVIDIA Ampere generation (A100/A30/A40/A2), AMD Instinct MI200 Accelerator GPU series, Intel Xeon Scalable Processor Third Gen supports it (Intel Deep Learning Boost AVX-512_BF16 Extension), and ARMv8-A. From a platform operator perspective, BF16 allows more teams to use the same hardware when developing models and running experiments. As 8 GB of memory suddenly feels more like 16 GB by using lower precision, data science teams can use fractional GPUs without experiencing performance impact. That additional headroom works in favor of workload consolidation for ML workloads. Part 1 described the ML model development lifecycle, and if data science teams are currently developing models (concept phase), they can share a single GPU without drastically impacting their performance. Combine fractional GPU functionality such as NVIDIA vGPU (MIG) with a platform such as Kubernetes. You can easily create a platform that quickly attaches and detaches accelerator resources to data science teams developing new neural network models or ML-infused services. Justin Murray and Catherine Xu wrote an extensive article on deploying an AI-ready platform with vSphere and Kubernetes. Another article in this series will dive into the spectrum of ML accelerators and when to deploy fractional GPUs regarding the ML model development lifecycle. TensorFloat32: NVIDIA developed TensorFloat32 (TF32). TF32 is internal to CUDA, meaning only NVIDIA devices support it. This one is interesting as it’s not explicitly called in frameworks like TensorFlow or PyTorch like all the other floating point data types. Well, it’s a Tensor core mode, not a data type. For example, if you want to use the data type BF16, you use tf.bfloat16 in Tensorflow or torch.bfloat16 in Pytorch. With TF32, you keep using the default (FP32), tf.float32, and torch.cuda.FloatTensor (default PyTorch GPU float) and the CUDA compiler handles the conversion. Click on the image to enlarge You quickly spot the similarities when comparing TF32 to the other data types. TF32 uses the same 8-bit exponent as FP32, thus supporting the same extensive numeric range. It uses the same number of bits as FP16 for precision. As research has proved, not all 23-bits are required for ML workloads. I’ve seen some worrisome threads on hacker news and StackOverflow where people are destroying each other why it’s not called TF19, as it’s using 19-bits, so I’m not going near this topic. Let’s understand the marketing aspects of things here and that it can be a drop-in replacement of FP32. Please do not start a war in the comments section on this. Let’s compare the performance between FP32, BF16, and TF32 of the A100 GPU listed above, and of course, these are peak performances. If the model uses FP32, the device can provide a theoretical performance of 19.5 teraFLOPS. 19.5 trillion floating-point operations per second! If the data scientists call some additional CUDA libraries, it can exploit Tensor Cores to drive up the theoretical speed to 156 teraFLOPS. To put this into perspective, this device could have run Skynet as it processed information at ninety teraflops. They just needed to use TF32. ;) If the data scientist adjusts the framework code and uses BF16, the GPU produces 312 teraFLOPS, more speeds, but more work for the data scientist. TF32 is the default math mode for single precision for A100 accelerators using the NVIDIA optimized deep learning framework containers for TensorFlow, Pytorch, and MXNet. TF32 is enabled by default for A100 in framework repositories starting with PyTorch 1.7, TensorFlow 2.4, and MXNet 1.8. As a result, the data scientist must make an extra effort to avoid using TF32 when running up-to-date frameworks on an A100. That means FP32 performance specs are not necessarily the primary performance spec to look at when reviewing the accelerator’s performance. What sets TF32 math mode apart from the FP data types is that it converts computation operations, but all the storage of the bits remains in FP32. As a result, TF32 is only increasing math throughput but not decreasing memory bandwidth pressure like FP16 and BF16 do. And this can be hard to wrap your head around. In another article in this series, I will cover this and the concept of arithmetic intensity. I will look at a CNN’s specific operations to understand whether memory bandwidth or computational capabilities limit their performance. Mixed Precision: Not a floating point data type but a method. Why not combine the best of both worlds? Mixed precision training uses a combination of FP16 and FP32 to reduce the memory and math bandwidth. Mixed precision starts by keeping a copy of all the network weights in FP32. Forward pass and backpropagation pass parameters are stored in the FP16 data type. Therefore, most operations require less memory bandwidth, speeding up data transfers and increasing the math operation speeds due to lower precision. As mixed precision leans heavily on FP16, underflow and overflow can occur. Frameworks such as Tensorflow dynamically determine the loss scale if the mixed precision policy is active. If your data science teams are talking about (automatic) Mixed Precision training, pay attention to the FP16 performance claims of a GPU spec sheet, as most of the training is done with that data type. Quantization So far, I have mainly focused on floating-point data types in the training context. For inference, optimizing the neural network footprint may be even more critical. If the model runs in the cloud, you want to minimize infrastructure costs. If you run the model near or at the edge, hardware limitations are your primary constraint. Data scientists, or in some organizations, MLOps teams spend much time reducing memory footprint and the computational complexity of models before they deploy them in production. They do this by quantizing the model. Model quantization replaces the floating points inside the neural network with integers. This process approximates the values within the network, and due to this, accuracy loss occurs (performance). The most popular integer used is the 8-bit signed integer (INT8). You can imagine that going from capturing values in 32-bit data types to now using 8-bit values might require work to keep the network performing accurately. Song Han, Huizi Mao, and William J Dally used quantization and other optimization techniques to reduce the storage requirement of their neural networks by 35× to 49× without affecting their accuracy. The AVX-512 instruction set includes the INT8 data type. And with each new CPU generation, they introduce improvements to the Intel Deep Learning Boost kit. In the 2nd generation scalable Xeon family, they reduced INT8 operations to a single instruction. There are rumors that Intel is removing it from the desktop CPU. I guess they want to drive the ML-related workload towards the data center CPU, forgetting that most data scientist do their concept work on laptops and workstations that don’t have Xeons. The Intel Sapphire Rapids generation will introduce a new ML suite called the Advanced Matrix Extension (AMX). If you want to dive in deep, Intel published its Intel Architecture. Instruction Set Extensions and Future Features Programming Reference online. Chapter 3 contains all the details. Have fun! To some, it may surprise that Intel focuses on ML extensions in their CPUs, but much inference at the edge runs on them. As Part 4 shows, the Inference workload is, on average, a streaming workload. We now have to deal with a tiny workload that we must quickly process. GPUs are throughput and parallel beasts, and CPUs are latency-focused sprinters. We now have a choice, should we allow the CPU to process this data directly, or should we get the data through the system, from the CPU and memory, across the PCIe bus, to a GPU core that runs on a lower clock cycle than a CPU. Because there isn’t much data, we are losing the advantage of parallelism. Letting the CPU take care of that workload with the proper optimization sometimes makes more sense. A great example of the power of quantization is the story of Roblox, which uses CPUs to run its inference workload. They serve over 1 billion requests a day using a fine-tuned Bert model. But not every inference workload can just run on a CPU. Plenty of inference workloads generate a data stream that overwhelms a CPU. The data scientist can use a roofline analysis to determine the CPU and GPU performance headroom. Another article in this series will cover the roofline analysis. The Tesla P4 started the support for the INT8 data type, and you can imagine that the ML community hasn’t stopped looking for finding ways to optimize. Turing Architecture introduced support for INT4 precision. CPUs do not have native INT4 support. Hopefully, the spec sheets of GPUs will make more sense now. During conversations with the data science teams within your organization, you can translate their functional requirements to technical impact. As always, leave feedback and comments below on which topics you want to see covered in future articles. Training Inference Numerical Precision Higher Precision Required Lower Precision Required Data Type FP32 BF16 BF16 INT8 Mixed Precision (FP16+FP32) INT4 (Not seen Often) Previous parts in the Machine Learning on the VMware Platform series Part 1 - covering ML development lifecycle and the data science team Part 2 - covering Resource Utilization Efficiency Part 3 - Training vs Inference - Data flow, Data sets & Batches, Dataset Random Read Access Part 4 - Training vs Inference - Memory Consumption by Neural Networks ================================================================================ Title: Training vs Inference - Memory Consumption by Neural Networks URL: https://frankdenneman.ai/2022-07-15-training-vs-inference-memory-consumption-by-neural-networks/ Date: 2022-07-15 This article dives deeper into the memory consumption of deep learning neural network architectures. What exactly happens when an input is presented to a neural network, and why do data scientists mainly struggle with out-of-memory errors? Besides Natural Language Processing (NLP), computer vision is one of the most popular applications of deep learning networks. Most of us use a form of computer vision daily. For example, we use it to unlock our phones using facial recognition or exit parking structures smoothly using license plate recognition. It’s used to assist with your medical diagnosis. Or, to end this paragraph with a happy note, find all the pictures of your dog on your phone. Plenty of content discusses using image classification to distinguish cats from dogs in a picture, but let’s look beyond the scope of pet projects. Many organizations are looking for ways to increase revenue or decrease costs by applying image classification, object identification, edge perception, or pattern discovery to their business processes. You can expect an application on your platform that incorporates such functionality. Part 3 of this series covered batch sizes and mentioned a batch size of 32, which seems a small number nowadays. An uncompressed 8K image (7680 x 4320) consumes 265 MB. The memory capacity of a modern data center GPU ranges from 16 GB to 80 GB. You would argue that it could easily fit more than 32 uncompressed (8 GB) 8K images, let alone 32 8K jpegs (896 MB). Why do we see so many questions about memory consumption on data science forums? Why are the most commonly used datasets and neural networks focused on images with dimensions hovering around the 224 x 224 image size? Memory consumption of neural networks depends on many factors. Such as which network architecture is used and its depth. The image size and the batch size. And whether it’s performing a training operation or an inference operation. This article is by no means an in-depth course on neural networks. I recommend you follow Stanfords’ CS231n or sign up for the free online courses at fast.ai. Let’s dig into neural networks a little bit, explore the constructs of a neural network and its components and figure out why an image eats up a hefty chunk of memory. Understanding the workload characteristics helps with resource management, troubleshooting, and capacity planning. When I cover fractional vGPU and multi-GPU in a later part of this series, you can map these functional requirements easier to the technical capabilities of your platform. So let’s start slowly by peeling off the first layer of the onion and look at a commonly used neural network architecture for image classification, the convolutional neural network. Convolutional Neural Networks Convolutional neural networks (CNN) efficiently recognize and capture patterns and objects in images and are the key components in computer vision tasks. A CNN is a multilayer neural network and consists of three different types of layers. The convolution layer, the pooling layer, and the fully connected layer. The first part of the neural network is responsible for feature extraction and consists of convolution and pooling layers. I’ll cover what that exactly is in the convolution layer paragraph. The second part of the network consists of the fully connected layers and a softmax layer. This part is responsible for the classification of the image. Convolution layer Convolution layers are the backbone of the CNN as they perform the feature extraction. A feature can be an edge of a (license plate) number or an outline of a supermarket item. Feature extraction is deconstructing the image into details, and the deeper you go into the network, the more detailed the feature becomes. CNNs process an image, but how does a computer see an image? To a computer, images are just numbers. A color image, i.e., an RGB image, has a value for each red, green, and blue channel, and this pixel representation becomes the foundation for the classification pipeline. For (us) non-native English speakers, convolution (layer) is not to be confused with convolute (make an argument complex). In the convolution layer, there is a convolve action, which means “to combine” or how something is modified by another element. In this case, the convolution layer performs a dot product between two matrices to generate a feature map containing activations. Don’t be afraid. It’s not going to be a linear algebra lesson. (I wouldn’t be able to, even if I tried). But we need to look at the convolution process and its components at a high level to better understand the memory consumption throughout the pipeline within the neural network. A neural network is a pipeline, meaning that the process’s output is the input of the following process. There are three components in a convolution layer, an array of input, a filter, and an array of output. The initial input of a CNN is an image and is the input array(a matrix of values). The convolution layer applies a feature detector known as a kernel or filter, and the most straightforward way of describing this is a sliding window. This sliding window, also in a matrix shape, includes the neural network’s weights. Weights are learnable parameters in the neural network that are adjusted during training. The weight starts with a random value, and as training continues, it, alongside the bias (another parameter), is adjusted towards a value that provides the accurate output. The weights and biases need to be stored in memory during training and are the core IP of a trained network. Typically a CNN uses a standard kernel (filter) height and width size that determines the number of weights applied per filter. The typical kernel size is 3 x 3. This filter is applied to the input array in the case of the first convolutional layer, the image. As mentioned, the convolution layer performs a dot product between two matrices to generate a feature map containing activations. Let’s look at the image to get a better understanding. For this example, a 3x3 filter is applied to a 6 x 6 image (normally, the image dimensions would be 224 x 224). This kernel or filter size is sometimes called the receptive field. During this process, the filter calculates a single value, called activation, by multiplying the value in the kernel with every value in the highlight input array field and then adding up the “products” to get the final output value, the activation. Output = (1*1)+(2*4)+(1*1)+(4*1)+(2*3)+(1*5)+(0*9)+(1*1)+(1*3)=1+8+1+4+6+5+0+1+3=29. Once the activation is calculated, the filter moves over a number of pixels, determined by the stride setting. Typically this is 1 or 2 pixels. And repeats the process. Once the entire input array is processed, the output array is completed. And a new filter is applied to the input array. This output array is known as a feature map or sometimes an activation map. Each convolution layer has a predefined number of filters, each with a different configuration of weights. Each filter creates its own feature map, which turns into the input for the following convolutional or pooling layer. One bias parameter is applied per filter. The parameter paragraph clarifies the impact of these relationships on memory consumption. Pooling Layer A pooling layer follows multiple convolution layers. It summarizes essential parts of the previous layers without losing critical information. An example is using a filter to detect the outlines of a ketchup bottle and then using the pooling layer to obscure the exact location of the ketchup bottle. Knowing the location of the bottle in a particular stage of the network is unnecessary. Therefore, a pooling layer filters out unnecessary details and keeps the network focused on the most prominent features. One of the reasons to introduce the pooling layer is to reduce as many parameters throughout the network to reduce the complexity and the computational load. To consolidate the previous feature map, it either uses an average of the numbers in a specific region (average pooling) or the maximum value detected in a specific region (max pooling). Similar to the filter applied to the input, the size is much smaller (2 x 2), and the movement (stride) is much larger (2 pixels). As a result, the size of each feature map is reduced by a factor of two., i.e., each dimension halves. The number of feature maps remains the same as in the previous layer. An important detail is that there are no weights or biases present in this layer, and as a result, it’s a non-trainable layer. It’s an operation rather than a learning function of the network. It impacts the layer’s overall memory consumption, which we shall discover in a later paragraph. Fully Connected Layer The fully connected layer is the poster child of neural networks. Look up any image or icon of a neural network, and you will get an artist’s impression of a fully connected layer. The fully connected layer contains a set of neurons (placeholder for a mathematical function) connected to each neuron in the following layer. It’s the task of the fully connected layer to perform image classification. Throughout the pipeline of convolutional layers, the filters detect specific features of the image and do not “see” the total picture. They detect certain features, and it’s the task of the fully connected layers to tie it all together. The first fully connected layer takes the feature maps of the last pooling layer and flattens the matrix into a single vector. It feeds the inputs into the neurons in its layer and applies weights to predict the correct label. Parameters Memory Consumption What is fascinating for us is the number of parameters involved as they consume memory. Each network architecture differs in layout and its number of parameters. There are several different CNN architectures, AlexNet (2012), GoogLeNet (2014), VGG (2014), and ResNet (2017). Today ResNet and VGG-16 are the most popular CNN architectures, often pitted against each other to find the most accurate architecture comparing training from scratch versus transfer learning. ResNet-50 vs VGG-19 vs training from scratch: A comparative analysis of the segmentation and classification of Pneumonia from chest X-ray images is a fascinating read. Let’s use the VGG-16 neural network architecture as our example CNN to understand memory consumption better. VGG-16 is a well-documented network, so if you doubt my calculations, you can easily verify them elsewhere. VGG-16 has thirteen convolutional layers, five Max Pooling layers, and three fully-connected layers. If you count all the layers, you will see it sums up to 21, but the 16 in VGG-16 refers to the 16 layers with learnable parameters. The picture below shows the commonly used diagram illustrating the neural network configuration. It’s important to note that memory consumption predominantly spits into two significant categories memory used to store parameters (weights & biases) and memory stored for the activations in the feature maps. The feature map memory consumption depends on the image’s height, the image’s width, and the batch’s size. The parameters’ memory remains constant regardless of the image or batch size. Let’s look at the parameters of memory consumption of the neural network first. The VGG-16 network accepts images with a dimension of 224 x 224 in RGB. That means that there are three channels for input. The dimension of the image is not relevant in this stage for memory calculation. The first convolutional layer applies 64 filters (stated on the architectural diagram as 224 x 224 x 64). It applies a filter with a kernel size of 3 x 3, and thus we calculate 64 distinct filters applying a kernel of 3 x 3 (9) weights on three input arrays (Red channel, Blue channel, Green channel). The number of weights applied in this layer is 1,728. One bias is applied per filter, increasing the total to 1,792 parameters for this convolutional layer. Each weight is stored in memory as a float (floating point numbers), and each single-precision floating-point (FP32) occupies 4 bytes, resulting in a memory footprint of 7KB. The following article of this series covers the impact of floating point types on memory consumption. The second layer uses the 64 feature maps produced by the first convolutional layer (CL) as input. It maintains using a 3 x 3 kernel and using 64 filters. The calculation turns into 64 inputs x 64 distinct filters using nine weights; each equals 36.864 weights + 64 biases = 36928 parameters x 4 bytes = 147 kb. The pooling layer applies a max pooling operation with a kernel size of 2 x 2 and a stride of 2. In essence, it is reducing the matrix size of the last feature map in half. The exact number of feature maps remain, and they act as input for the next convolutional layer as no weight is involved. No, there is no memory footprint consumed from a parameter perspective. 7kb and 147kb are certainly not earth-shattering numbers, but now let’s see the rest of the network. As you can see, the memory parameters slowly grow throughout the convolutional layers of the network and then dramatically explode at the fully connected layers. What’s interesting to note is that there is a flattening operation after the last pool layer of the feature extraction part of the network. This operation will flatten the pooled feature map into a single column that produces a long vector of input data that can pass through the fully connected layers. The 512 matrices of 7 x 7 turn into a single vector containing 25,088 activations. (Click on the image to enlarge). In total, the network requires 540 MB to store the weights and the biases. That’s quite a footprint if you consider deploying this to edge devices. But there are always bigger fish. The state-of-the-art (SOTA) neural network for generating text GPT-3, or the third generation Generative Pre-trained Transformer, has 175 billion parameters. If we use a single-precision floating-point (FP32), it needs 700 GB of memory. Most companies don’t use a GPT-3 model to enhance their business processes, but it illustrates the range of memory footprint some neural networks can have. Network Architecture # of Convolutional Layers # of Fully Connected Layers # of Parameters AlexNet 5 3 61 Million GoogLeNet 21 1 40 Million ResNet 49 1 50 Million VGG-16 13 3 138 Million Feature Map Memory Consumption The memory consumption of the feature maps is a relatively straightforward calculation, i.e., the dimensions of the image x number of the feature maps. The feature map contains the activations from the filter moving across the input array. Each convolution layer receives the feature maps of the previous layer, and each pooling reduces the dimensions of the feature maps in half. Feature map memory consumption depends on the image’s size as the kernel with weights moves across the image. The larger the image, the more activations there are. The batch size impacts the memory consumption as well. More images mean more activations to store in the memory. The network executes a batch of images in parallel. With a batch size of 32 and a default image size of 224 x 224, the calculation of memory consumption becomes as follows: 224 x 224 x 64 channels x 32 images = 102.760.448 x 4 bytes (as it is stored as a float) = 401.40 MB. Let’s take a step back. On average, a 224 x 224 image takes up 19kb of space on your hard drive. Some quick math tells us that 32 images consume 602 KB. That can easily fit on a double-density 3.5" floppy disk drive, not even a fancy high-density one. And now, after the first convolution, it occupies a little over 401 MB. Oh yeah, and 7 kb for the parameters! Interestingly, we noticed the parameters’ memory footprint go up while moving towards the network’s end. We see the memory footprint per feature map go simply as the pooling layers reduce the dimensions of each feature map. This is important to note for inference requirements for your GPU device! But I’ll cover that in detail later. During the batch iteration, 32 images with a 244 x 244 dimension consume roughly 1.88 GBs of memory. (Click on the image to enlarge). If you applied the same math to a 4K image (ignoring whether it’s possible with a VGG-16 network), the memory consumption of a 3840 x 2160 image would be roughly 9.6 GB for one image and 307,2 GB for 32 images. This means that the data scientist needs to choose between reducing the batch size and thus agree with the increase in training time. Or spend more time pre-processing and reducing the image size or distributing the model across multiple GPUs to increase the available GPU memory. Training versus Inference When the batch of images reaches the softmax layer, the output is generated. And from this point on, we must distinguish whether it’s a training or inference operation to understand the subsequent memory consumption. The process I described in the paragraphs above is a forward propagation or typically referred to as the forward pass. This forward pass exists both in the training and inference operations. For training, an extra process is required, the backward pass or backpropagation. And to fully understand this, we have to dig deep into linear algebra and calculus, and you are already 2600 words deep into this article. It all comes down to that you train image classification via the supervised learning method, which means the set of images is trained along with their corresponding labels. When the image or batch training completes, the network determines the total error by calculating the difference between the expected value (image label) and the observed value (the value generated by the forward pass). The network needs to figure out which weight contributed the most to the error and which weight to change to get the “loss” to a minimum. If the loss is zero, the label is correct. It does this by calculating a partial derivative of the error concerning each weight. What does that mean? Essentially, each weight contributes to the loss as they are one way to the other connected to the other. A derivative in mathematics is the rate of change of a function with respect to a variable, and in the case of a neural network, how fast can we move the error rate up or down. With this generic description, I’m losing the finer details of this art form, but it helps to get an idea of what’s going on. The differentials are multiplied by the learning rate, and the calculation result is subtracted from the respective weights. As a result, backpropagation requires space to store each weight’s gradients and learning rates. Roughly the memory consumption of the parameters is doubled during training. If the data scientist uses an optimizer, such as ADAM, it’s normal to expect the memory consumption to triple. What’s important to note is that the duration of the memory consumption of the activations (the feature maps) remains as long as the neural network needs to calculate the derivates. With Inference, the memory consumption is quite different. The neural network has optimized weights; thus, only a forward pass is necessary, and only the parameters need to be active in the memory. There is no backpropagation pass. Better yet, the activations are short-lived. The activations are discarded once the forward pass moves to a new layer. As a result, you only need to consider the model parameters and the two most “expensive” consecutive layers for memory consumption calculation. Typically those will be the first two layers. The layer that is active in memory and the layer that gets calculated. And this means that the GPU for Inference does not have to be a massive device. It only needs to continuously hold the network parameters and temporarily hold two feature maps. Knowing this, it makes sense to look for different solutions for your edge/inference deployments. Training Inference Memory Footprint Large memory footprint Forward propagation pass - backpropagation pass - model parameters Long time duration of the memory footprint of activations (large bulk of memory footprint) Smaller memory footprint Forward propagation pass - model parameters Activations are short-lived (Total memory footprint = est. 2 largest consecutive layers) Previous parts in the Machine Learning on the VMware Platform series Part 1 - covering ML development lifecycle and the data science team Part 2 - covering Resource Utilization Efficiency Part 3 - Training vs Inference - Data flow, Data sets & Batches, Dataset Random Read Access ================================================================================ Title: Unexplored Territory Podcast Episode 19 - Discussing NUMA and Cores per Sockets with the main CPU engineer of vSphere URL: https://frankdenneman.ai/2022-07-01-unexplored-territory-podcast-episode-19-discussing-numa-and-cores-per-sockets-with-the-main-cpu-engineer-of-vsphere/ Date: 2022-07-01 Richard Lu joined us to talk basics of NUMA, Cores per Socket, why modern windows and mac systems have a default 2 cores per socket setting, how cores per socket help the guest OS interpret the cache topology better, the impact of incorrectly configured NUMA and Cores per Socket systems and many other interesting CPU related topics. Enjoy another deep dive episode, you can listen to and download the episode on the following platforms: Unexplored Territory website Apple Podcasts Spotify Topics discussed in the episode L1TF Speculative-Execution vulnerability 60 minutes of NUMA - VMworld session 2022 Extreme Performance Series: vSphere Compute and Memory Schedulers [HCP2583] NUMA counters and command line tools - part 1 NUMA command lines tools - part 2 ================================================================================ Title: Machine Learning on VMware Platform – Part 3 - Training versus Inference URL: https://frankdenneman.ai/2022-06-30-machine-learning-on-vmware-platform-part-3-training-versus-inference/ Date: 2022-06-30 Machine Learning on VMware Cloud Platform – Part 1 covered the three distinct phases: concept, training, and deployment, part 2 explored the data streams, the infrastructure components needed and vSphere can help with increasing resource utilization efficiency of ML platforms. In this part, I want to go a little bit deeper into the territory of training and inference workloads. It would be best to consider the platform’s purpose when building an ML infrastructure. Are you building it for serving inference workloads, or are you building a training platform? Are there data science teams inside the organization that create and train the models themselves? Or will pre-trained models be acquired? Where will the trained (converged) model be deployed? Will it be in the data center, industrial sites, or retail locations? From an IT architecture resource requirement perspective, these training and inference workloads differ in computational power and data stream requirements. One of the platform architect’s tasks is to create a platform that reduces the time to train. It’s the data scientist’s skill and knowledge to use the platform’s technology to reduce the time even more without sacrificing accuracy. This part will dive into the key differences between training and inference workloads and their requirements. It helps you get acquainted with terminology and concepts used by data scientists and apply that knowledge to your domain of expertise. Ultimately, this overview helps set the stage for presenting an overview of the technical solutions of the vSphere platform that accelerate machine learning workloads. Types of machine learning algorithms When reviewing popular machine learning outlets and podcasts, you typically only hear about training large models. For many, machine learning equals deep learning with giant models and massive networks that require endless training days. But in reality, that is not the case. We do not all work at the most prominent US bank. We do not all need to do real-time fleet management and route management of worldwide shipping companies or calculate all possible trajectories of five simultaneously incoming tornadoes. The reality is that most companies work on simple models with simple algorithms. Simple models are easier to train. Simple models are easier to test. Simple models are not resource-hogs, and above all, simple models are simpler to update and keep aligned with the rapidly changing world. As a result, not every company is deploying a deep-learning GPT-3 model or massive ResNet to solve their business needs. They are looking at “simpler” machine learning algorithms or neural networks that can help increase the revenue or decrease the business cost without running it on 400 GPUs. In the following articles, I will cover neural networks, but if you are interested in understanding the basics of machine learning algorithms I recommend looking at the following popular ones: Support Vector Machines (SVM) Decision trees Logistic Regression Random forest k-means (not listed in the google search result) Data Flow Training produces a neural network model that generates a classification, detection, recommendation, or any other service with the highest level of accuracy. The golden rule for training is that the more data you can use, the higher accuracy you achieve. That means the data scientist will unleash copious amounts of data on the system. Understanding the data flow and the components involved helps you design a platform that can significantly reduce training time. Most neural networks are trained via the (offline) batch learning method, but the online training method is also used. In this method, the model is trained by feeding it smaller batches of data. The model is active and learns on the fly. Whether it is less resource-intensive than batch learning, or often referred to as offline training, is debatable as the model trains itself continuously. It needs to be monitored very carefully as it can be sensitive to new data that can quickly influence the model. Specific stock price systems deploy ML models that use online training to respond to market trends quickly. The inference service is about latency, for example, pedestrian identification in autonomous vehicles, packages flying across high-speed conveyor belts, product recommendations, or speech-to-text translations. You simply cannot afford to wait on a response from the system in some cases. Most of these workloads are a single data sample or, at best, a small number of instructions batched up. The data flow of inference is considered to be streaming of nature. As a result, the overall compute load of inference is much lower than that of the training workload. Training Inference Data Flow Batch Data Streaming Data Data sets and Batches During model training, models train with various data sets: training sets, validation sets, and testing sets. The training set helps the model recognize what it should be supposed to learn. The validation dataset is helpful for the data scientist to understand the effect of tuning particular hyperparameters, such as the number of hidden layers or the network layer size. The third dataset is the testing set and proves how well the trained neural network performs on unseen data before being put into production. A dataset provides the samples used for training. These data sets can be created from company data or acquired from third parties. Or a combination of both, sometimes businesses acquire extra data on top of their own to get better insights into their customers. These datasets can be quite large. An example is a Resnet50 model with Imagenet-1K dataset. Resnet50 is an image classification training network, and the Imagenet-1K dataset contains 1.28 million images (155.84 GiB). Even the latest NVIDIA GPU generations (Ampere and Hopper) offer GPU devices with up to 80GB of memory and cannot fit that dataset entirely in memory. As a result, the dataset is split into smaller batches. Batch size plays a significant role in the training of neural network models. This training technique is called mini-batch gradient descent. Besides circumventing the practical memory limitation, It impacts the accuracy of models, as well as the performance of the training process. If you’re curious about batch sizing, read the research paper “Revisiting small batch training for deep neural networks”. Let’s cover some more nomenclature while we are at it. During the training cycle, the neural network processes the dataset’s examples. This cycle is called an epoch. A data scientist splits up the entire dataset into smaller batch sets. The number of training examples used is called a batch size. An iteration is a complete pass of a batch. The number of iterations is how many batches are needed to complete a single epoch. For example, the Imagenet-1K dataset contains 1.28 million images. Well-recommended batch size is 32 images. It will take 1.280.000 / 32 = 40.000 iterations to complete a single epoch of the dataset. Now how fast an epoch completes depends on multiple factors. A training run typically invokes multiple epochs. Both training and inference use batch sizes. In most use cases, inference focuses on responding as quickly as possible. Many inference use-cases ingest and transform data in real-time and generate a prediction, classification, or recommendation. Translating this into real-life use cases, we are talking about predicting stock prices to counting cars in a drive-through. The request needs to be processed the moment it comes in. As a result, no batching to limited batching occurs. It depends on how much workload the system receives when it is operational. By batching 1-4 examples, Inference classes as streaming workload. Determining the correct batch size for training is a science by itself. Many research papers and Medium articles exist about the sweet spot for batch sizes. There are benefits and disadvantages to be found at any point in the spectrum of batch sizes. Smaller batch sizes can lead to lower memory footprint and improvement of throughput, while larger batch sizes can increase parallelism and decrease the computational cost. This last factor might not be relevant for a data scientist when training in an on-premises environment, but it’s good to understand. When batches are moved from storage or host memory to GPU device memory, CPU cycles are needed. If you are using larger batches, you reduce the number of computing calls to move data, ultimately reducing your CPU footprint. Off course, there is a downside to this as well, primarily on the performance side of the algorithm, something the data scientist needs to figure out how to solve. Therefore you notice that depending on the use case, you see different batch sizes per model. Two excellent papers that highlight both ends of the spectrum: “Friends don’t let friends use mini-batches larger than 32” and “Scaling TensorFlow to 300 million predictions per second” The takeaway for the platform architect is that inference is primarily latency-focused. If the inference workload is a video-streaming-based workload for image classification or object detection, the system should be able to provide a particular level of throughput. Training is predominantly throughput based. Batch sizing is a domain-specific (hyper)parameter for the data scientist. Still, it can ultimately affect the overall CPU footprint and whether efficient distributed training is used. Depending on the dataset size, the data scientist can opt for distributed training, dispatching the batches across multiple GPUs. Training Inference Storage Characteristics Throughput based Latency-based, occasionally throughput Batch Size Many recommendations between 1-32 Smaller batch size reduces the memory footprint Smaller batch size increases algorithm performance (generalization) Larger batch size increases compute efficiency Larger batch size increases parallelization (Multi-gpu) 1-4 Data Pipeline and Access Patterns Data loading is essential to building a deep Learning pipeline and training a model. Remember that everything you do with data takes up memory. Let’s go over the architecture and look at all the “moving parts” before diving into each one. The dataset is stored on a storage device. It can be a vSAN datastore or any supported network-attached storage platform (NFS, VMFS, vVOLs). The batch is retrieved from the datastore and stored in host memory before it loads into GPU device memory (Host to Device (HtoD)). Once the model algorithm completes the batch, the algorithm copies the output back to host memory (Device to Host - DtoH). Please note that I made a simple diagram and showed the most simple data flow. Typically, we have a dual-socket system, meaning there are interconnects involved, multiple PCI controllers involved, and we have to deal with VM placement regarding the NUMA locality of the GPU. But these complex topics are discussed later in another article. one step at a time. We immediately notice the length of the path without going into the details of NUMA madness. Data scientists prefer that the dataset is stored as close to the accelerator as possible on a fast storage device. Why? Data loading can reduce the training time tremendously. Quoting Gorkem Polat, who did some research on his test environment: One iteration of the ResNet18 Model on ImageNet data with 32 batch size takes 0.44 seconds. For 100 epochs, it takes 20 days! When we measure the timing of the functions, data loading+preprocessing takes 0.38 seconds (where 90% of this time belongs to the data loading part) while the optimization (forward+backward pass) time takes only 0.055 seconds. If the data loading time is reduced to a reasonable time, full training can be easily reduced to 2.5 days! Source Most datasets are too large to fit into the GPU memory. Most of the time, it does not make sense to preload the entire dataset into host memory. The best practice is to prefetch multiple batches and thereby mask the latency of the network. Most ML frameworks provide built-in solutions for data loading. The data pipeline can run asynchronously with training as long as the pipeline prefetches several batches to keep it full. The trick is to keep multiple pipelines full, where fast storage and low-latency and high throughput networks come into play. According to the paper “ImageNet training in Minutes,” it takes an Nvidia M40 GPU 14 days to finish just one 90-epoch Resnet-50 training execution on the ImageNet-1k dataset. The M40 was released in 2015 and had 24GB of memory space. As a result, data scientists are looking at parallelization, distributing the workload across multiple GPUs. These multiple GPUs need to access that dataset as fast as possible, and they need to communicate with each other as well. There are multiple methods to achieve multi-GPU accelerator setups. This is a topic I happily reserve for the next part. Dataset Random Read Access To add injury after insult, training batch reads are entirely random. The API lets the data scientist specify the number of samples, and that’s it. Using a Pytorch example: train_loader = torch.utils.data.DataLoader(train_set, batch_size=32, shuffle=True, num_workers=4) The process extracts 32 random examples from the dataset and sends them over as a batch. The command Shuffle=true is for what happens after the Epoch completes. This way, the next Epoch won’t see the same images in the same order. Extracting 32 random examples from a large dataset on a slow medium won’t help reduce the training time. Placing the dataset on a bunch of spindles would drive (pun intended) your data science team crazy. Keeping the dataset on a fast medium and possibly as close to the GPU device as possible is recommended. Training Inference Data Access Random Access on large data set Multiple batches are prefetches to keep the pipeline full Fast storage medium recommended Fast storage and network recommended for distributed training Streaming Data The next part will cover the memory footprint of the model and numerical precision. ================================================================================ Title: Unexplored Territory Podcast Episode 18 - Not just artificially intelligent featuring Mazhar Memon URL: https://frankdenneman.ai/2022-06-13-unexplored-territory-podcast-episode-18-not-just-artificially-intelligent-featuring-mazhar-memon/ Date: 2022-06-13 In this week’s Unexplored Territory Podcast, we have Mazhar Memon as our guest. Mazhar is one of the founders of VMware Bitfusion and the principal inventor of Project Radium. In this episode, we talk to him about the start of Bitfusion, what challenges Project Radium solves, and what role the CPU has in an ML world. If you like deep-dive podcast episodes, grab a nice cup of coffee or any other beverage of your liking, open your favorite podcast app, strap in and press play. Listen to the full Unexplored Territory Podcast episode via Spotify - https://spoti.fi/3QdnXlX Apple - https://apple.co/3O7TsMj, or with your favorite podcast app. Links and topics discussed during the episode: Techcrunch demo – https://www.youtube.com/watch?v=p3cAzt1PLBA Intro to Radium – https://octo.vmware.com/introducing-project-radium/ IPUs and Radium – https://octo.vmware.com/vmware-and-graphcore-collaborate-to-bring-virtualized-ipus-to-enterprise-environments/ You can follow us on Twitter for updates and news about upcoming episodes: https://twitter.com/UnexploredPod. Also, make sure to hit that subscribe button, rate where ever possible, and share the episode with your friends and colleagues! And for those who hadn’t seen it, we made the Top 15 Podcast list on feedspot, the first non-corp branded podcast on the list! ================================================================================ Title: MACHINE LEARNING ON VMWARE PLATFORM – PART 2 URL: https://frankdenneman.ai/2022-06-08-machine-learning-on-vmware-cloud-platform-part-2/ Date: 2022-06-08 Resource Utilization Efficiency Machine learning, especially deep learning, is notorious for consuming large amounts of GPU resources during training. However, as the last part already highlighted, machine learning is more than just training a model. And these components within the machine learning workflow require large amounts of CPU, memory, storage, and network resources. Machine Learning on VMware Cloud Platform – Part 1 covered the three distinct phases: concept, training, and deployment. Existing “known data” is required to explore and train the model in both the concept and training phases. During the development of the model, it is common to use three different data sets: the training set, the validation set, and the testing set. Creating data sets is not only about getting as much data as possible. It is even more critical getting meaningful data and high levels of quality because the accuracy of the recommendation produced by the model is highly dependent on the quality of the dataset used for training and validation. The data science team needs to “wrangle” existing raw data into shape to get such a high-quality dataset. Data wrangling transforms the raw data into more valuable data that can be used as a dataset “downstream” to train a model. And all this wrangling requires a lot of collateral infrastructure and services besides just a bunch of GPUs. Massive amounts of datasets are not new to any enterprise IT organization. “System of records” has always been the backbone of business processes. Think back about mainframes in the ’70s and the rise of on-premises ERP systems like SAP and Oracle in the ’80s and ’90s. And today, databases, data warehouses, and data lakes contain petabytes of data. A 100-gigabyte dataset used for training purposes might sound unimpressive. But there is a difference. This dataset is constantly on the move. Bits and bytes do not slowly accrue in a database and sit there waiting to be cherry-picked by a SQL statement. No, these datasets are pulled and pushed through the infrastructure. It will be extracted from multiple sources, transformed by different platforms, and stored and versioned many times over. For example, many ML projects capture and store massive amounts of unstructured data (video, audio, log files) in its native format in systems that need to be structured, transformed, and analyzed. These processes generate a lot of data movement, which, in virtualized platforms, burns (a lot of) x86 cycles and exposes data on the network. Below is a diagram highlighting some of the structural components of such an environment, omitting a lot of essential data science teams tools such as the various collaborative Juypter notebook solutions, artifacts stores, or complete data science or MLops platforms such as Databricks, Domino, H20.ai, DataRobot, or Dataiku. If we look at the processes after training, they belong to the deployment phase. In this phase, the data science team, or the MLops team, takes a converged model and integrates it into a system or platform that engages with the customer or an end system, like a robot arm or factory installation. A converged model is a model that is trained up to a state where additional training will not improve the model. Why not say, finished model? As the world changes, the model might not be trained to reflect the current state of the world. Think about what happened during Covid and the ML models deployed in the hotel, airliner, and entertainment industry. They needed some readjustments to reflect the current situation in the world. And because of this, some models need retraining. This retraining does not happen in real-time. It depends on the use case and the data flow. Think about self-scanning registers, holiday season packing changes for supermarkets, or additional items in a warehouse. In general, most teams start with manual retraining. Still, they want to move to automation and look at CI/CD pipelines, create a cyclical deployment process, retrain new models, and capture un-seen data as new training sets. As you can imagine, running such a platform and its (pipeline) components requires a lot of processing power. On top of that, in many organizations, AI\ML projects are initiated by individual business units attempting to solve their pressing business challenge. Each team runs its solution tech stack and develops its models along its development lifecycle with its resource utilization. What we see today is the start of ML-projects sprawl. Many ML projects start small, with small data sets, starting on laptops, some small datasets, and slowly grow and become essential to the business. And we attach the same organization and process models to this phenomenon. Someone will detect this, start to ask for more consolidation, start a Machine Learning Center of Excellence and start to think of centralizing resource utilization. And this is the right way of thinking. You do not want each team to erect its own isolated platform. We’ve seen this happen in the ’90s when organizations moved away from mainframes to decentralized X86 servers. A lot of these machines were underutilized. We see now that a lot of data scientists are simply not aware of the power of virtualization. It’s cloud or bare-metal. Cloud platforms are great to start but are too opinionated, and thus they turn to bare-metal. They leave out the greatest thing since sliced bread (now, I might be a bit biased here). Let’s use two examples from the concept and training phase. When the data scientist team performs a “Hyperparameter search,” they use multiple smaller GPU-equipped machines and smaller data sets to find the correct ML model architecture and model. Or, when they are using their Spark cluster for intensive data processing tasks, they typically only concentrate on the pre-processing task. The key here is that an ML platform mainly consists of many parts, but only one part is heavily utilized. From a resource utilization perspective, we have to deal with load-correlated utilization spikes per model development since many platform parts are originally designed as a distributed architecture. I can imagine that all of the above can sound extremely complex, but in many cases, it isn’t. If organizations centralize their efforts onto a single platform, we have to deal with this workload’s noisy-neighbor aspect. But, we all have dealt with noisy neighbors before, and the best part is that a lot of the core parts of vSphere are updated to handle new distributed workloads. For example, DRS 2.0. Runs every 60 seconds instead of every 5 minutes to handle containerized workloads and focuses on workload happiness instead of dealing at a high level of balancing host utilization across a vSphere cluster. The partnership with NVIDIA that brought us NVIDIA AI Enterprise allows us to spatially partition the GPU to isolate compute resources and allow full multi-tenancy. See the recent blogpost of Lan Vu about how the different vGPU technology can best suit the ML use case ) But we are also working on newer technologies with our partners to think about the heavy IO streams that ML model development will introduce to the vSphere platform. Project Monterey introduces the Data Processing Unit (DPU) into our cluster architecture. There are many use-cases, but the one that excites me is the sheer amount of innovative network IO offload you can generate with DPUs and potentially more intelligent things with NVIDIAs Bluefield architecture. Nowadays, every network IO within a virtualization platform consumes an X86 of the ESXi host—more than one X86 cycle. And so, pulling and pushing datasets and datasets for many different models through the platform will impact the X86 cycles left over to run the rest of the virtual machines and containers used for the toolchain for the data scientists and other employees and services consuming the virtualization platform. By introducing DPUs, you isolate these network IO streams from the compute layer. Another project with immense potential for the ML platform is project Capitola, or as the vSphere team calls it, the Software-Defined Memory structure. It would be best to have copious amounts of storage space during the pre-processing data phase, but you also want it fast. But you might not want to break the bank and spend your entire annual IT budget on RAM modules. Project Capitola allows you to use different memory technologies and offer them different tiers of memory capacity to your workload without rewriting your applications. The critical challenge is to have a platform that can provide the right resources and attach and detach the resources efficiently and economically so that the data science team and the organization benefit from this. The platform needs to provide the workload environment as quickly as possible (self-service) and be resilient. I’ll dive into risk mitigation in the next part. If you want to learn more about project Monterey, Capitola, or the self-service part, please join Cormac Hogan, Duncan Epping, and me at our (Virtual) Roadshow “The ever-evolving VMware Infrastructure” . Ask your VMUG leader about the possibilities. ================================================================================ Title: Machine Learning on VMware Platform - Part 1 URL: https://frankdenneman.ai/2022-05-25-machine-learning-on-vmware-cloud-platform-part-1/ Date: 2022-05-25 Machine Learning is reshaping modern business. Most VMware customers look at machine learning to increase revenue or decrease cost. When talking to customers, we mainly discuss the (vertical) training and inference stack details. The stack runs a machine learning model inside a container or a VM, preferably onto an accelerator device like a general-purpose GPU. And I think that is mostly due to our company DNA letting us relate machine learning workload directly to a hardware resource. But Machine Learning (ML) allows us to think much broader than that. One of the most cited machine learning papers, “Machine Learning: The High-Interest Credit Card of Technical Debt,” shows us that running ML code is just a tiny part of the extensive and complex hardware and software engineering system. Machine Learning Infrastructure Platform As of today, VMware’s platform can offer much functionality to ML practitioners already. We can break down a machine learning platform into horizontal fabrics connecting the vertical training and inference stack. An example of a horizontal fabric can be a data pipeline that runs from ingesting data all the way to feeding the training stack a training data set. Or it can be a pipeline that streams data from a sensor to an ISV pre-trained model and goes all the way to control the actions of a robot arm. All the functions, applications, and services are backed by containers and VMs that could run on a VMware platform. Looking beyond the vertical training stack, we immediately see familiar constructs we have spent years caring for as platform architects. The majority of databases run on vSphere platforms. Many of these contain the data; data scientists want to use to train their models. Your vSphere platform can efficiently and economically run all the services needed to extract and transform the data. The storage services safely and securely store the data close to the computational power. The networking and security services, combined with the core services, can offer intrinsic security to the data stream from the point of data ingestion all the way to the edge location where the model is served. This service allows each data science team to pull software to their development environment whenever they need it. Using self-service marketplace services, such as “VMware Application Catalog” (formerly known as Bitnami ), allows IT organizations to work together with the head of data science to curate their ML infrastructure toolchains. Improving ROI of low utilized resources If we focus on the vertical training and inference stack first, we can apply the same story (I at least) discussed for a long time. Instead of talking about X86 cycles and memory utilization, we now focus on accelerator resources. Twenty years ago, we revolutionized the world by detaching the workload from the metal by introducing the virtual machine. The compelling story was that the server hardware was vastly underutilized, typically around 4%. This convinced customers to start virtualizing. By consolidating workload and using different peak times of these workloads, the customer could enjoy a better return on investment of the metal. This story applies to today’s accelerators. During GTC fall, it was quoted that today’s utilization of accelerators is below 15%. Accelerators are expensive, and with the current supply chain problems, very challenging to obtain these resources. Let alone scale-out. The availability and distribution of accelerator resources within the organizations might become a focal point once the organization realizes that there are multiple active projects and the current approach of dedicated resources is not economically sustainable. Or the IT organization is looking to provide a machine learning platform service. The pooling of accelerator resources is beneficial to the IT organization from an economic perspective, plus it has a positive effect on the data science teams. The key to convincing the data science teams is understanding the functional requirements of the phases of the model development lifecycle and deploying an infrastructure that can facilitate those needs. A machine learning model follows a particular development lifecycle, Concept - Training - Deployment (inference). Concept phase: The data science team determines what framework and algorithm to use during the concept phase. During this stage, the DS team explores what data is available, where the data lives and how they can access it. They study the idea’s feasibility by running some tests using small data sets. As a result, the team will run and test some code, with lots of idle time in between. The run time will be seconds to minutes when the code is tested. As you can imagine, a collection of bare metal machines assigned to individual data scientists or teams with dedicated expensive GPUs might be overkill for this scenario. In many cases, the CPU provides enough power to run these tests. Still, if the data science team wants to research the effect and behavior of the combination of the model and the GPU architecture, virtualization can be beneficial. This moment is where pooling and abstraction, two core tenets of VMware’s DNA, come into play. We can consolidate the efforts of different data science teams in a centralized environment and offer fractional GPUs (NVIDIA vGPU) or remoting with VMware Bitfusion (intercept and remoting of CUDA API). Or use multi-instance GPU (MIG) functionality that allows for stricter resource isolation and thus predictable performance. These functionalities allow the organization to efficiently and economically provide an infrastructure for the data science teams to run their workload when they are in the testing phase of developing a particular model. Training phase: A virtualized platform is also advantageous for the training phase of the model. During the training phase of the model development lifecycle, the data science team pushes workload for longer times, which is throughput-oriented. Typically they opt for larger pools of parallel processing power. In this phase, CPU resources aren’t cutting anymore, and most of the time, a part of a single physical GPU isn’t enough either. Depending on their performance needs, the customer can use remoting technology (Bitfusion) to pools of GPU cards or use PCI passthrough device technology for running the workload directly on one or more physical devices. But the hungry nature of the workload still should not warrant a dedicated infrastructure as training jobs are transient in nature. Sometimes a training job is 20 hours, and sometimes 200 hours. The key point is that idle time of GPU resources occurs after the training job is finished. The data science team reviews the results of the training job and discusses which hyperparameter to tune. This situation is very inefficient. The costly accelerators are idling while other data science teams struggle for resources. Virtualization platforms can democratize the resources and help scale-out resources for the data science teams. Convince data science teams to “give up” their dedicated infrastructure is based on helping them understand the nature of virtualization and how pooling and abstracting can benefit their needs. Instead of having isolated pools of GPUs scattered throughout the organizations, virtualization allows data science teams to have a centralized pool of resources at their disposal. When they need to run a training job, they can allocate far more (spare and unused) resources from that pool than the number of resources they could have allocated in their dedicated workstation. In return for giving up their dedicated resources, they have a larger pool of resources at their disposal, reducing the duration of training time. Deployment phase: The deployment phase is typically performance-focused in nature. This is where we meet the age-old discussion of hypervisor overhead. The hypervisor introduces some overhead, but the performance teams are working on this to get this gap reduced. We are now at the single-digit mark (+-6%), and in some cases near bare-metal performance, and thus other benefits should be highlighted. Such as the ability to deal with heterogeneous architecture due to using virtual machines for workloads or running Kubernetes in VMs. The VMware ecosystem allows for deploying and managing at scale and focuses on delivering lifecycle management at scale. Portability and mobility of VMware’s workload constructs, all these unique capabilities matter to mature organizations and offset a single-digit performance loss. The Data Science Team In general data scientist team ultimately focuses on delivering machine learning models, their MBOs state to deliver these models as fast as possible, with the highest level of accuracy. Although light coding (Python, R, Scala) is a part of their daily activities, most data scientists do not have a software engineering background and thus can be seen as a step above developers. As a result, our services should not treat them as developers but align more with the low-code, no-code trend seen in the AI world today. Data Scientists do not and should not have a deep understanding of infrastructure architectures. Their lowest level of abstraction is predominantly a Docker container. Kubernetes is, in most cases, a “vendor” requirement. Any infrastructure layer underneath the container should be of no concern to the data scientist, and our platform should align with that thought. The data scientist should not learn how to optimally configure their data and machine learning platform to operate on the VMware platform. The VMware Cloud platform should abstract these decisions away from the data scientist and provide (automatic) constructs for ML Ops teams, IT architects, and central operation teams to run these applications and services optimally. Data scientists are primarily unaware of IT policies. They have machines and software often not included in the corporate IT service catalog. This situation can be a massive security liability as they consistently work with the crown jewels of the company, highly sensitive company data. Data is now exposed, usually not well secured, a prime target for ransomware or other malicious practices. One of the examples that our field teams repeatedly see is that this data is often stored on very expensive shiny Alienware laptops. This data needs to be protected and should not be stored on high-risk target devices. These devices are bought as the current IT platforms lack the hardware accelerators or any collaborative containerized service platform. Embezzlement of these devices leaks out company data and halts the model development progress. Besides both of these problems, an organization does not want to have its highly paid data science team twiddling its thumbs. Having a platform ready to cater to the needs of the data science teams, both in software and hardware requirements, eliminates these threads and helps the organizations benefit from the many benefits of a centralized platform. The current method of providing a data scientist team with highly specialized, portable equipment is an extreme liability from a security perspective. Many data science teams don’t see the centralized IT team as an enabler due to the lack of service alignment. Providing a centralized platform with the right services and equipment can help the organization severely reduce the security risk while improving the data science teams’ capability to deliver a model by leveraging all the advantages of a centralized platform. Centralizing ML Infrastructure Platforms If we take it one step further, you can obtain a higher level of economics by running the ML infrastructure platforms on the shared infrastructure instead of dedicated physical machines. As described in the previous paragraph, data science teams operate with full autonomy in many organizations and have dedicated resources at their disposal. Individual business groups within the organizations are forming their data science teams, and due to this decentralized effort, a lot of scattered high-powered machines around the organization create pools of fragmented compute and storage resources. By centralizing the different ML infrastructure services workload of different teams on one platform, you benefit from the many advantages the VMware ecosystem provides: Resource Utilization Efficiency Risk Mitigation Workload Construct Standardization and Life Cycle Management Data adjacency Intrinsic security for data streams Open platform But I’ll leave these topics for part 2 of this series ================================================================================ Title: Solving vNUMA Topology Mismatch When Migrating between Dual Socket Servers and Quad Socket Servers URL: https://frankdenneman.ai/2022-03-11-solving-vnuma-topology-mismatch-when-migrating-between-dual-socket-servers-and-quad-socket-servers/ Date: 2022-03-11 I recently received a few questions from customers migrating between clusters with different CPU socket footprints. The challenge is not necessarily migrating live workloads between clusters because we have Enhanced vMotion Compatibility (EVC) to solve this problem. For VMware users just learning about this technology, EVC masks certain unique features of newer CPU generations and creates a generic baseline of CPU features throughout the cluster. If workloads move between two clusters, vMotion still checks whether the same CPU features are presented to the virtual machine. If you are planning to move workloads, ensure the EVC modes of the clusters are matching to get the smoothest experience. The challenge when moving live workloads between ESXi hosts with different socket configurations is that vNUMA topology of the virtual machine does not match the physical topology. A virtual NUMA topology exists out of two components, the component that presents the CPU topology to the virtual machine, called the VPD. The VPD exists to help the guest OS and the applications optimize their CPU scheduling decisions. This VPD construct is principally the virtual NUMA topology. The other component, the PPD, groups the vCPUs and helps the NUMA scheduler for placement decisions across the physical NUMA nodes. The fascinating part of this story is that the VPD and PPD are closely linked, yet they can differ if needed. The scheduler attempts to mirror the configuration between the two elements; the PPD configuration is dynamic, but the VPD configuration always remains the same. From the moment the VM is powered on, the VPD configuration does not change. And that is a good thing because operating systems generally do not like to see whole CPU layouts change. Adding a core with CPU hot add is all right. But drastically rearranging caches and socket configurations it’s pretty much a bridge too far. As mentioned before, the VPD remains the same. Still, the NUMA scheduler can reconfigure the PPD to optimize the vCPU grouping for the CPU scheduler. When will this happen? When you move a VM to a host with a different physical CPU configuration, i.e. Socket Count, or physical cores per socket count. This way, ESXi still squeezes out the best performance it can in this situation. The drawback of this situation is the mismatch between presentation and scheduling. This functionality is great as it allows workloads to enjoy mobility between different CPU topologies without any downtime. However, we might want to squeeze out all the performance possible. Some vCPUs might not share the same cache, although the application thinks they do. Or, some vCPU might not even be scheduled together in the same physical NUMA node, experiencing latency and bandwidth reduction. To be more precise, this mismatch can impact memory locality and the action-affinity load-balancing operations of the scheduler. Thus, it can impact the VM performance and create more inter CPU traffic. This impact might be minor on a per-VM basis, but you have to think in scale, the combined performance loss of all the VMs, so for larger environments, it might be worthwhile to get it fixed. I’ve created a 36 vCPU VM on a dual-socket system with twenty physical CPU cores per socket. The power-on process of the virtual machine creates the vNUMA topology and enters all kinds of entries in the VMX file. Once the VM powers on, the VMX file receives the following entries. numa.autosize.cookie = "360022" numa.autosize.vcpu.maxPerVirtualNode = "18" The key entry for this example is the “numa.autosize.vcpu.maxPerVirtualNode = “18”, as the NUMA scheduler likes to distribute as many vCPUs across many cores as possible and evenly across sockets. But what happens if this virtual machine moves to a quad-socket system with 14 physical cores per socket? The NUMA scheduler will create three scheduling constructs to distribute those vCPUs across the NUMA nodes but keep the presentation layer the same not to confuse the guest OS and the applications. Since the NUMA topologies are created during a VM’s power-on, we have to shut down the virtual machine and power it back to realign the VPD and PPD topology again. Well, since 2019, we don’t need to power down the VM anymore! And I have to admit. I only found out about it just recently. Bob Plankers (not this Bob) writes about the vmx.reboot.PowerCycle advanced parameter here. This setting does not require a complete power cycle anymore. That means that if you are in the process of migrating your VM estate from dual-socket systems to quad-socket systems, you can add the following adjustments in the VMX file while the VM is running. (for example via PowerCLI / New-AdvancedSetting) vmx.reboot.PowerCycle = true numa.autosize.once = false The setting vmx.reboot.PowerCycle will remove itself from the VMX file, but it’s best to remove the numa.autosize.once = false from the VMX file. So you might want to track this. Same as adding the setting, you can remove the setting while the VM is up and running. When you have applied these settings to the VMX, the next time the VM reboots, the vNUMA topology will be changed. As always, keep in mind that older systems might react more dramatically than newer systems. After all, you are changing the hardware topology of the system. It might upset an older windows system or optimizations of an older application. Some older operating systems do not like this and will need to do reconfiguration themselves or need some help from the IT ops team. In the worst-case scenario, it will treat the customer to a BSOD with that in mind. It’s recommended to work with the customer with old OSes and figure out a test and migration plan. Special thanks to Gilles Le Ridou for helping me confirm my suspicion and helping me test scenarios on his environment. #vCommunity! ================================================================================ Title: Stop designing your server platform with solely the CPU roadmap in mind URL: https://frankdenneman.ai/2021-12-20-stop-designing-your-server-platform-with-solely-the-cpu-roadmap-in-mind/ Date: 2021-12-20 Over the last 20 years, we designed our core data center platform following the CPU roadmap. But in today’s world, the devices attached to the processor make radical and revolutionary improvements, catering to the needs of the new workloads. I’m talking about devices like the GPU, the network adapter, and its natural offspring, the data processing unit (DPU). In the article “Project Monterey and the need for network cycles offload for ML workloads” I zoom into what’s in store for us data center architects in the upcoming years. To service the request of these new workloads, we need to move away from designing a platform solely based on a CPU roadmap and plug these devices in a server as an afterthought. When designing a platform for these new workloads, we have to start holistically designing data center systems. Together with Luke Wignall from NVIDIA, we (Duncan Epping, Johan van Amersfoort, and I) discuss DPU technology and other efforts to run and manage modern workloads in episode 5 of the Unexplored Territory podcast. Apple: apple.co/3lYZGCF Google: bit.ly/3oQVarH Spotify: spoti.fi/3INgN3R Or anywhere else where you get your podcasts! ================================================================================ Title: Exciting Sessions from NVIDIA GTC Fall 2021 URL: https://frankdenneman.ai/2021-12-09-exciting-sessions-from-nvidia-gtc-fall-2021/ Date: 2021-12-09 Over the last few weeks, I watched many sessions of the NVIDIA Fall version of GTC. I created a list of interesting sessions for a group of people internally at VMware, but I thought the list might interest some outside VMware. It’s primarily focused on understanding NVIDIA’s product and services suite and not necessarily deep diving into technology or geeking out on core counts and speeds and feeds. If you found exciting sessions that I haven’t listed, please leave them in the comments below. Data Science Accelerating Data Science: State of RAPIDS [A31490] Reason to watch: A 55-minute overview of the state of RAPIDS (the OS framework for data science), upcoming features, and improvements Inference Please note: Triton is part of the NVIDIA AI Enterprise stack (NVAIE) How Hugging Face Delivers 1 Millisecond Inference Latency for Transformers in Infinity [A31172] Reason to watch: A 50-minute session. Hugging Face is the dominant play for NLP Transformers. Hugging Face is pushing the open platform \ democratizing ML message. Scalable, Accelerated Hardware-agnostic ML Inference with NVIDIA Triton and Arm NN [A31177] Reason to watch: A 50-minute session covering ARM NN architecture and deploying models on far edge technology (Jetson\Pi’s) using NVIDIA Triton Inference Server Deploy AI Models at Scale Using the Triton Inference Server and ONNX Runtime and Maximize Performance with TensorRT [A31405] Reason to watch: 50 minutes overview of Triton architecture, features, customer case studies, and onyx runtime integration. It covers ONNX RT, which provides optimization for target platforms (inference). NVIDIA Triton Inference Server on AWS: Customer success stories and AWS deployment methods to optimize inference throughput, reduce latency, and lower GPU or CPU inference costs. [SE31488] Reason to watch: 45 minutes session covering Triton on AWS SageMaker and two customers sharing their deployment overview and their lessons learned. No-Code Approach NVIDIA TAO: Create Custom Production-Ready AI Models without AI Expertise [D31030] Reason to watch: A 3-minute overview of TAO, an model adaptation framework that can fine-tune pre-trained models by feeding proprietary (smaller) datasets and optimizing it for the inference hardware architecture. AI Life-cycle Management for the Intelligent Edge [A31160] Reason to watch: A 50-minute session that covers NVIDIAs approach of Transfer Learning. NVIDIA provides a Pre-trained model, while customers optimize the model, NVIDIA TAO assists with future optimization for inference deployment. Fleet command to orchestrate the deployment of the model at the edge. A Zero-code Approach to Creating Production-ready AI Models [A31176] Reason to watch: A 35-minute session that explores TAO in more detail and provides a demo of the TAO (GUI-based) solution. NVIDIA LaunchPad Simplifying Enterprise AI from Develop to Deploy with NVIDIA LaunchPad [A31455] Reason to watch: A 30-minute overview of the NVIDIA Cloud AI platform delivered through their partnership with Equinix. (Rapid testing and prototyping AI) How to Quickly Pilot and Scale Smart Infrastructure Solutions with LaunchPad and Metropolis [A31622] Reason to watch: A 30-minute overview of how to use Metropolis (Computer Vision AI Application Framework) and LaunchPad to accelerate POCs. (Zero Touch Testing and System Sizing). The session covers Metropolis Certification for system design (TS: 11:30) and the bare metal access. NVIDIA and Cloud Integration From NGC to MLOps with NVIDIA GPUs and Vertex AI Workbench on Google Cloud (Presented by Google Cloud) [A31680] Reason to watch: A 40-minutes overview of NCG integration with Google Vertex AI (Google End-to-End ML AI Platform) Automate Your Operations with Edge AI (Presented by Microsoft Azure) [A31707] Reason to watch: A 15-minutes overview of Azure Percept. Azure Percept is an edge AI solution for IoT devices, now available on the Azure HCI stack. Fast Provisioning of Kubernetes Clusters for the AI/ML Developer on VMware: Technical Details (Presented by VMware) [A31659] Reason to watch: A 45-minute technical overview of the NVIDIA and VMware partnership demonstrating the key elements of the NVIDIA AI-Ready Enterprise Platform in detail. NVIDIA EGX Exploring Cloud-native Edge AI [A31166] Reason to watch: A 50-minute overview of NVIDIA’s cloud-native platform (Kubernetes based, edge AI platform) Retail The One Retail AI Use Case That Stands Out Above the Rest [A31548] Reason to watch: A 55-minute session providing great insights into real use-case of Everseen technology deployed at Kroger One of the World’s Top Retailers is Bringing AI-Powered Convenience to a Store Near You [A31359] Reason to watch: A 25-minute session providing insights into the challenges of deploying and using autonomous store technology. DPU Real-time AI Processing at the Edge [A31164] Reason to watch: A 40-minute session covering the GPU+DPU converged accelerators (A40X and A100X), their architecture and their DOCA (Data Center on a Chip Architecture) and CUDA architecture and programming environment. Programming the Data Center of the Future Today with the New NVIDIA DOCA Release [A31069] Reason to watch: A 40-minute session covering DOCA architecture in detail. NVIDIA AI Enterprise with VMware vSphere: Combining NVIDIA GPU’s Superior Performance, NVIDIA AI Software, and Virtualization Benefits for AI Workflows (Presented by VMware, Inc.) [A31694] Reason to watch: A 50-minute session offering an excellent explanation of the different vGPU modes (native vs MIG mode). Starting from the 37 minute time stamp the presenters dive into the use of Network Function virtualization on smartNICs. Developer / Engineer Type Sessions CUDA New Features and Beyond [A31399] Reason to watch: A 50-minutes overview of what’s new in the CUDA toolkit. Developing Versatile and Efficient Cloud-native Services with Deepstream and Triton Inference Server [A31202] Reason to watch: A 50-minutes deep dive session on the end-to-end pipelines for vision-based AI Accelerating the Development of Next-Generation AI Applications with DeepStream 6.0 [A31185] Reason to Watch: A 50-minutes overview of DeepStream solution. DeepStream helps to develop and deploy vision-based AI End-to-end Extremely Parallelized Multi-agent Reinforcement Learning on a GPU [A31051] Reason to watch: A 40-minute deep dive session on how Salesforce worked on a framework to drastically reduce CPU-GPU communication ================================================================================ Title: vSphere 7 Cores per Socket and Virtual NUMA URL: https://frankdenneman.ai/2021-12-02-vsphere-7-cores-per-socket-and-virtual-numa/ Date: 2021-12-02 Regularly I meet with customers to discuss NUMA technology, and one of the topics that are always on the list is the Cores per Socket setting and its potential impact. In vSphere 6.5, we made some significant adjustments to the scheduler that allowed us to decouple the NUMA client creation from the Cores per Socket setting. Before 6.5, if the vAdmin configured the VM with a non-default Cores per Socket setting, the NUMA scheduler automatically aligned the NUMA client configured to that Cores per Socket settings. Regardless of whether this configuration was optimal for performing in its physical surroundings. To help understand the impact of this behavior, we need to look at the components of the NUMA client and how this impacts the NUMA scheduler options for initial placement and load-balancing options. A NUMA client consists of two elements, a virtual component and a physical component. The virtual component is called the virtual proximity domain (VPD) and is used to expose the virtual NUMA topology to the guest OS in the virtual machine, so the Operating System and its applications can apply optimizations for the NUMA topology they detect. The physical component is called the physical proximity domain (PPD). It is used by the NUMA scheduler in the VMkernel as a grouping construct for vCPUs to place a group of vCPUs on a NUMA node (physical CPU+memory) and to move that group of vCPUs between NUMA nodes for load-balancing purposes. You can see this PPD as some form of an affinity group. These vCPUs always stick together. Please note that the CPU scheduler determines which vCPU will be scheduled on a CPU core. The NUMA scheduler only selects the NUMA node. As depicted in the diagram, this VM runs on a dual-socket ESXi host. Each CPU socket contains a CPU package with 10 CPU cores. If this VM gets configured with a vCPU range between 11 and 20 vCPUs, the NUMA scheduler creates two NUMA clients and distributes these vCPUs evenly across the two NUMA nodes. The guest OS is presented with a virtual NUMA topology by the VPDs that aligns with the physical layout. In other words, there will be two NUMA nodes, with an x number of CPUs inside that NUMA node. Previous to ESXi 6.5, if a virtual machine is created with 16 CPUs and 2 Cores per Socket, the NUMA scheduler will create 8 NUMA Clients, and thus the VM exposes eight NUMA nodes to the guest OS. As a result, the guest OS and application could make the wrong process placement decisions or miss out on ideal optimizations as you are reducing resource domains as memory ranges and cache domains. Luckily we solved this behavior, and now the Cores Per Socket setting does not influence NUMA client configuration anymore. But! But, a Cores per Socket setting other than Cores per Socket =1 will impact the distribution of vCPUs of that virtual machine across the physical NUMA nodes. As the Cores per Socket act as a mini affinity rule-set. That means that the NUMA schedule has to distribute the vCPUs of the virtual machine across the NUMA nodes with Cores per Settings still in mind, not as drastically as before, where it also exposed it to the guest OS. Still, it will impact overall balance if you start to do weird things. Let me show you some examples. Let’s start by using a typical example of Windows 2019. In my lab, I have a dual-socket Xeon system with each 20 cores, so I’m forced to equip the example VMs with more than 20 vCPUs to show you the effect of Cores per Socket on vNUMA. In the first example, I deploy a Windows 2019 VM with the default settings and 30 vCPUs. That means that it is configured with 2 Cores per Socket. Resulting in a virtual machine configuration with 15 sockets. The effect is seen by using the command “sched-stats -t numa-clients” if I log into the ESXi host via an SSH session. The groupID of the VM is 1405619, which I got from ESXTOP, and this command shows that 16 vCPUs are running on homeNode 0 and 14 vCPUs on homeNode 1. Maybe, this should not be a problem, but I’ve seen things! Horrible things! This time I’ve selected 6 Cores per socket. And now the NUMA scheduler distributes the vCPUs as follows: At this point, the NUMA nodes are imbalanced more severely. This will impact the guest OS and the application. And If you do this at scale, for every VM in your DC, you will create a scenario where you impact the guest OS and the underlying fabric of connected devices. More NUMA rebalancing is expected, which occurs across the processor interconnects (QPI-Infinity Fabric). More remote memory is fetched, all the operations impact PCIe traffic, networking traffic, HCI storage traffic, GPU VDI or Machine Learning application performance. The Cores per Socket setting is invented to solve a licensing problem. There are small performance gains to be made if the application is sensitive to cache performance and can keep the data in the physical cache. If you have that application, please align your Cores per Socket setting to the actual physical layout. Using the last example, for the 30 vCPU VM on a dual-socket 40 cores host, the Cores per Socket should be 15, resulting in 2 virtual sockets. But my recommendation is to avoid the management overhead for 99% of all your workload and keep the default Cores per Socket Settings. Overall you have a higher chance not to impact any underlying fabric or influence scheduler decisions. Update Looking at the comments, I notice that there is some confusion. Keeping the default means that you will use 2 Cores per Socket for the new Windows systems. The minor imbalance of vCPU distribution within the VM should not be noticeable. The Windows OS is smart enough to distribute processes efficiently between these NUMA nodes setup like this. The compounded effect of many VMs within the hypervisor and the load-balancing mechanism of the NUMA scheduler and the overall load frequencies (not every vCPU is active all of the time) will not make this a problem. For AMD EPYC users, please keep in mind, that you are not working with a single chip surface anymore. EPYC is a multi-chip module architecture and hence, you should take notice of the cache boundaries of the CCX’s within the EPYCs. The new Milan (v3) has 8 cores per CCX, the Rome (v2) and Naples (v1) have 4 cores. Please test the cache boundary sensitivity of your workload if you are running your workload on AMD EPYC systems. Please note that not every workload is cache sensitive and not every combination of workloads responds the same way. This is due to the effect that load correlation and load synchronicity patterns can have on scheduling behavior and cache evictions patterns. ================================================================================ Title: DRS threshold 1 does not initiate Load balancing vMotions URL: https://frankdenneman.ai/2021-10-22-drs-threshold-1-does-not-initiate-load-balancing-vmotions/ Date: 2021-10-22 vSphere 7.0 introduces DRS 2.0 and its new load balancing algorithm. In essence, the new DRS is completely focused on taking care of the needs of the VMs and does this at a more aggressive pace than the old DRS. As a result, DRS will resort to vMotioning a virtual machine faster than the previous DRS. And this is something that a lot of customers are noticing. In highly consolidated clusters, you might see a lot of vMotions occur. I perceive this as an infrastructural service. However, some customers might see this as a turbulent or nervous environment and rather see fewer vMotions. As a result, these customers like to dial down the DRS threshold, which is the right thing to do. But please be aware, if you still want DRS to have load-balancing functionality, do not slide the threshold all the way to the left. Using the setting “Conservative (1)”, DRS only triggers migrations for solving cluster constraints and violations. Meaning that if you put an ESXi host into maintenance mode, DRS moves the VMs out of that host. Or if due to a maintenance mode migration, or HA event, an anti-affinity rule is violated, DRS moves that particular VM to solve that problem. But that’s it. No moves to solve any VM happiness or host imbalance. If you want to reduce the number of vMotions and still like DRS to get the best resources for the virtual machines, do not set DRS to the utmost left setting but set it to setting 2. ================================================================================ Title: Project Monterey and the need for Network Cycles Offload for ML Workloads. URL: https://frankdenneman.ai/2021-10-06-project-monterey-and-the-need-for-network-cycles-offload-for-ml-workloads/ Date: 2021-10-06 VMworld has started, and that means a lot of new announcements. One of the most significant projects VMware is working on is project Monterey. Project Monterey allows the use of SmartNICS, also known as Data Processing Units, of various VMware partners within the vSphere platform. Today we use the CPU inside the ESXi host to run workloads and to process network operations. With the shift towards distributed applications, the CPUs inside the ESXi hosts need to spend more time processing network IO instead of application operations. This extra utilization impacts data center economics like consolidation ratios and availability calculations. On top of this shift from monolith application to distributed application is the advent of machine learning supported services in the enterprise data center. As we all know, the enterprise data center is the goldmine of data. Business units within organizations are looking at the data. If combined with machine learning, it can solve their business challenges. And so, they use the data to train machine learning models. However, data stored in databases or coming from modern systems such as sensors or video systems cannot be directly fed into the vertical application stack used for machine learning model training. This data needs to be “wrangled” into shape. The higher quality of the data, the better the model can generate a prediction or recommendation. As a result, the data flows from its source through multiple systems. Now you might say machine learning datasets are only a few 100 gigabytes in size. I’ve got databases that are a few terabytes. The problem is that a database sits nice and quietly on a datastore on an array somewhere and doesn’t go anywhere. This dataset moves from one system to another in an ML infrastructure depicted below and gets transformed, copied, and versioned many times over. You need a lot of CPU horsepower to transform the data and continuously move the data around! One of the most frequent questions I get is why vSphere is such an excellent platform for machine learning, simply data adjacency. We run an incredible amount of database systems in the world. That holds the data of the organizations, which in turn holds the key to solve those challenges. Our platform provides the tools and capabilities to use modern technologies such as GPU accelerators and data processing units to help process the data. And data is the fuel of machine learning. The problem is that we (the industry) haven’t gotten around to make a Tesla version of machine learning model training yet. We are in the age of gas-guzzling giants. People in the machine learning space are looking into improving techniques for datasets for model training. Instead of using copious amounts of data, use more focused data points. But that’s work in progress. In the meantime, we need to deal with this massive data stream flowing through many different systems and platforms that typically run within virtual machines and containers on top of the core platform vSphere. Instead of overwhelming the X86 CPUs in the ESXi host to deal with all the network traffic generated by sending those datasets between the individual components of the ML infrastructure, we need to offload it to another device in an intelligent way. And that’s where project Monterey can come into play. There are many presentations about project Monterey and all it’s capabilities. I would suggest you start with “10 Things You Need to Know About Project Monterey” by Niels Hagoort #MCL1833 ================================================================================ Title: Machine Learning Infrastructure from a vSphere Infrastructure Perspective URL: https://frankdenneman.ai/2021-08-12-machine-learning-infrastructure-from-a-vsphere-infrastructure-perspective/ Date: 2021-08-12 For the last 18 months, I’ve been focusing on machine learning, especially how customers can successfully deploy machine learning infrastructure on a vSphere infrastructure. This space is exciting as it has so many great angles to explore. Besides the model training, a lot of stuff happens with the data. Data is transformed, data is moved. Data sets are often hundreds of gigabytes in size. Although that doesn’t sound that much compared to modern databases, these data sets are transformed and versioned. Where massive databases nest on an array, these data sets travel through pipelines that connect multiple systems with different architectures, from data lakes to in-memory key-value stores. As a data center architect, you need to think about the various components involved, where the compute horsepower is needed? How do you deal with an explosion of data? Where do you place particular storage platforms, what kind of bandwidth is needed, and do you always need the extreme low-latency systems in your ML infrastructure landscape? The different ML model engineering life cycle phases generate different functional and technical requirements, and the persona involved is not data center architectural-minded. Sure they talk about ML infrastructure, but their concept of infrastructure is different from “our” concept of infrastructure. Typically, the lowest level of abstraction a data science team deals with is a container or a VM. Concepts of availability zones, hypervisors, or storage areas are foreign to them. When investigating ML pipelines and other toolsets, technical requirements are usually omitted. This isn’t weird, as containers are more or less system-like processes, and you typically do not specify system resource requirements for system processes. But for an architect or a VI team that wants to shape a platform capable of dealing with ML workload, you need to get a sense of what’s required. I intend to publish a set of articles that helps to describe where the two worlds of data center infrastructure and ML infrastructure interact, where the rubber meets the road. The series covers the different phases of the ML model engineering lifecycle and what kind of requirements they produce. What is MLOps, and how does this differ from DevOps? Why is data lineage so crucial in today’s world, and how does this impact your infrastructure services? What type of persona is involved with machine learning, their tasks and role in the process, and what type of infrastructure service can benefit them. And how we can map actual data and ML pipeline components to an abstract process diagram full of beautiful terms such as data processing, feature engineering, model validation, and model serving. I’m sure that I will introduce more topics along the way, but if you have any topic in mind that you want to see covered, please leave a comment! ================================================================================ Title: CPU pinning is not an exclusive right to a CPU core! URL: https://frankdenneman.ai/2021-06-11-cpu-pinning-is-not-an-exclusive-right-to-a-cpu-core/ Date: 2021-06-11 https://twitter.com/MrsBrookfield/status/1402955235497287685 Katarina tweeted a very expressive tweet about her love/hate (mostly hate) relation with CPU pinning, and lately I have been in conversations with customers contemplating whether they should use CPU pinning. The analogy that I typically use to describe CPU pinning is the story of the favorite parking space at your office parking lot. CPU pinning limits the compliant CPU “slots” for that vCPU to be scheduled on. So think about that CPU slot as the parking spot closest to the entrance of your office. You have decided that you only want to park in that spot. Every day of the year, that’s your spot and no other place else. The problem is, this is not a company-wide directive. Anyone can park in that spot, but you just limited yourself to that spot only. So it can happen that Bob arrives at the office first and lazy as he is, he will park to the office entrance as close as he can. Right in your spot. Now the problem with your self-imposed rule is that you cannot and will not park anywhere else. So when you show up (late to the party), you notice that Bob’s car is in YOUR parking spot, and the only thing you can do is to drive circles in some holding pattern until Bob leaves the office again. The stupidest thing. It’s Sunday, and you and Bob are the only ones doing some work. You’re out there on the parking lot, driving circles waiting until Bob leaves again, while Bob is inside in the empty building waiting on you to get started. CPU pinning is not an exclusive right for that vCPU to use that particular CPU slot (Core or HT). It’s just a self-imposed limitation. If you want exclusive rights to a full core, check out the setting Latency Sensitivity ================================================================================ Title: VM Service - Help Developers by Aligning Their Kubernetes Nodes to the Physical Infrastructure URL: https://frankdenneman.ai/2021-05-03-vm-service-help-developers-by-aligning-their-kubernetes-nodes-to-the-physical-infrastructure/ Date: 2021-05-03 The vSphere 7.0 U2a update released on April 27th introduces the new VM service and VM operator. Hidden away in what seems to be a trivial update is a collection of all-new functionalities. Myles Gray has written an extensive article about the new features. I want to highlight the administrative controls of VM classes of the VM service. VM Classes What are VM classes, and how are they used? With Tanzu Kubernetes Grid Service running in the Supervisor cluster, developers can deploy Kubernetes clusters without the help of the Infra Ops team. Using their native tooling, they specify the size of the cluster control plane and worker nodes by using a specific VM class. The VM class configuration acts a template used to define CPU, memory resources and possibly reservations for these resources. These templates allow the InfraOps team to set guardrails for the consumption of cluster resources by these TKG clusters. The supervisor cluster provides twelve predefined VM classes. They are derived from popular VM sizes used in the Kubernetes space. Two types of VM classes are provided, a best-effort class and a guaranteed class. The guaranteed class edition fully reserves its configured resources. That is, for a cluster, the spec.policies.resources.requests match the spec.hardware settings. A best-effort class edition does not, that is, it allows resources to be overcommitted. Let’s take a closer look at the default VM classes. VM Class Type CPU Reservation Memory Reservation Best-Effort-‘size’ 0 MHz 0 GB Guaranteed-‘size’ Equal to CPU config Equal to memory config There are eight default sizes available for both VM class types. All VM classes are configured with a 16GB disk. VM Class Size CPU Resources Configuration Memory Resources Configuration XSmall 2 2 Gi Small 2 4 Gi Medium 2 8 Gi Large 4 16 Gi XLarge 4 32 Gi 2 XLarge 16 128 Gi 4 XLarge 16 128 Gi 8 XLarge 32 128 Gi Burstable Class One of the first things you might notice if you are familiar with Kubernetes is that the default setup is missing a QoS class, the Burstable kind. Guaranteed and Best-Effort classes are located at both ends of the spectrum of reserved resources (all or nothing). The burstable class can be anywhere in the middle. I.e., the VM class applies a reservation for memory and or CPU. Typically, the burstable class is portrayed to be a lower-cost option for workloads that do not have a sustained high resource usage. Still, I think the class can play an essential role in no-chargeback cloud deployments. To add burstable classes to the Supervisor Cluster, go to the Workload Management view, select the Services tab, and click on the manage option of the VM Service. Click on the “Create VM Class” option and enter the appropriate settings. In the example below, I entered 60% reservations for both CPU and memory resources, but you can set independent values for those resources. Interestingly enough, no disk size configuration is possible. Although the VM Class is created, you have to add it to a namespace to be made available for self-service deployments. Click on “Add VM Class” in the VM Service tile. I modified the view by clicking on the vCPU column, to find the different “small” VM classes and selected the three available classes. After selecting the appropriate classes, click ok. The Namespace Summary overview shows that the namespace offers three VM classes. The developer can view the VM classes assigned to the namespace by using the following command: kubectl get virtualmachineclassbindings.vmoperator.vmware.com -n namespacename I logged into the API server of the supervisor cluster, changed the context to the namespace “onlinebankapp” and executed the command: kubectl get virtualmachineclassbindings.vmoperator.vmware.com -n onlinebankapp If you would have used the command “kubectl get virtualmachineclass -n onlinebankapp”, you get presented with the list of virtualmachineclasses available within the cluster. Help Developers by Aligning Their Kubernetes Nodes to the Physical Infrastructure With the new VM service and the customizable VM classes, you can help the developer align their nodes to the infrastructure. Infrastructure details are not always visible at the Kubernetes layers, and maybe not all developers are keen to learn about the intricacies of your environment. The VM service allows you to publish only the VM classes you see fit for that particular application project. One of the reasons could be the avoidance of monster-VM deployment. Before this update, developers could have deployed a six worker node Kubernetes cluster using the guaranteed 8XLarge class (each worker node equipped with 32 vCPUs, 128Gi all reserved), granted if your hosts config is sufficient. But the restriction is only one angle to this situation. Long-lived relationships are typically symbiotic of nature, and powerplays typically don’t help build relationships between developers and the InfraOps team. What would be better is to align it with the NUMA configuration of the ESXi hosts within the cluster. NUMA Alignment I’ve published many articles on NUMA, but here is a short overview of the various NUMA configuration of VMs. If a virtual machine (VM) powers on, the NUMA scheduler creates one or more NUMA clients based on the VM CPU count and the physical NUMA topology of the ESXi host. For example, a VM with ten vCPUs powered on an ESXi host with ten cores per NUMA node (CPN2) is configured with a single NUMA client to maximize resource locality. This configuration is a narrow-VM configuration. Because all vCPU have access to the same localized memory pool, this can be considered an Unified Memory Architecture (UMA). Take the example of a VM with twelve vCPUs powered-on on the same host. The NUMA scheduler assigns two NUMA clients to this VM. The NUMA scheduler places both NUMA clients on different NUMA nodes, and each NUMA client contains six vCPUs to distribute the workload equally. This configuration is a wide VM configuration. If simultaneous multithreading (SMT) is enabled, a VM can have as many vCPUs equal to the number of logical CPUs within a system. The NUMA scheduler distributes the vCPUs across the available NUMA nodes and trusts the CPU scheduler to allocate the required resources. A 24 vCPU VM would be configured with two NUMA clients, each containing 12 vCPUs if deployed on a 10 CPN2 host. This configuration is a high-density wide VM. A great use of VM service is to create a new set of VM classes aligned with the various NUMA configurations. Using the dual ten core system as an example, I would create the following VM classes and the associated CPU and memory resource reservations : CPU Memory Best Effort Burstable Burstable Mem Optimized Guaranteed UMA-Small 2 16GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% UMA-Medium 4 32GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% UMA-Large 6 48GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% UMA-XLarge 8 64GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% NUMA-Small 12 96GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% NUMA-Medium 14 128GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% NUMA-Large 16 160GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% NUMA-XLarge 18 196GB 0% | 0% 50% | 50% 50% | 75% 100% | 100% The advantage of curating VM classes is that you can align the Kubernetes nodes with a physical NUMA node’s boundaries at CPU level AND memory level. In the table above, I create four classes that remain within a NUMA node’s boundaries and allow the system to breathe. Instead of maxing out the vCPU count to what’s possible, I allowed for some headroom, avoiding a noisy neighbor with a single NUMA node and system-wide. Similar to memory capacity configuration, the UMA-sized (narrow-VM) classes have a memory configuration that does not exceed the physical NUMA boundary of 128GB, increasing the chance that the ESXi system can allocate memory from the local address range. The developer can now query the available VM classes and select the appropriate VM class with his or her knowledge about the application resource access patterns. Are you deploying a low-latency memory application with a moderate CPU footprint? Maybe a UMA-Medium or UMA-large VM class helps to get the best performance. The custom VM class can transition the selection process from just a numbers game (how many vCPUs do I want?) to a more functional requirement exploration (How does it behave?) Of course, these are just examples, and these are not official VMware endorsements. In addition, I created a new class, “Burstable mem optimized”, A class that reserves 25% more memory capacity than its sibling VM class “Burstable”. This could be useful for memory-bound applications that require the majority of memory to be reserved to provide consistent performance but do not require all of it. The beauty of custom VM classes is that you can design them as they fit your environment and your workload. With your skillset and knowledge about the infrastructure, you can help the developer to become more successful. ================================================================================ Title: vSphere with Tanzu vCenter Server Network Configuration Overview URL: https://frankdenneman.ai/2020-11-06-vsphere-with-tanzu-vcenter-server-network-configuration-overview/ Date: 2020-11-06 I noticed quite a few network-related queries about the install of vSphere with Tanzu together with vCenter Server Networks (Distributed Switch and HA-Proxy). The whole install process can be a little overwhelming when it’s your first time dealing with Kubernetes and load-balancing services. Before installing Workload Management (the user interface designation for running Kubernetes inside vSphere), you have to setup HA-Proxy, a virtual appliance for provisioning load-balancers. Cormac did a tremendous job describing the configuration steps of all components. Still, when installing vSphere with Tanzu, I found myself mapping out the network topology to make sense of it all since there seems to be a mismatch of terminology between HA-proxy and Workload Management configuration workflow. To help you navigate this diagram, I’ve provided annotations of the actual UI install steps. In total, three networks can be used for this platform; Network Main Purpose Management Communicating with vCenter, HA Proxy Workload IP addresses for Kubernetes Nodes Frontend Virtual IP range for Kubernetes clusters You can use the same network for the workload and frontend but be sure you are using IP-ranges that do not overlap. Also, do not use a DHCP range on that network. I wasted two days figuring out that this was not a smart thing to do. The supervisor VMs are dual-homed and connect to the management network and the workload network. The frontend network contains the virtual IP range assigned to the kubernetes clusters, which can be the supervisor cluster and the TKG clusters. To make it simple, I created three distinct VLANs with each their own IP-range: Network IP Range VLAN Management network 192.168.115/24 115 Workload network 192.168.116/24 116 Frontend network 192.168.117/24 117 This overview can help you with mapping the ip-addresses used in the screenshot to the network designations in the diagram. For example, during the HA-Proxy install, you have to configure the network. This is step 2 of this process and step 2.3 requests the management IP of the HA Proxy virtual appliance. HA-Proxy requires you to provide a load balancer IP range, that is used to provide a Kubernetes cluster a virtual IP. The next stop is workload management. In step 5, the first IP-address that you need to supply is the management IP address which you provided in step 2.3 of the HA-Proxy config process. Staying on same config page, you need to provide the IP ranges for virtual servers, those are the ip-address defined in the frontend network. It’s the exact same range you used when configuring step 3.1 of the HA-proxy configuration process, but this time you have to write it out instead of using a CIDR format (Let’s keep you sharp! ;) ) Step 6 of the workload management config process requires you to specify the IP addresses of the supervisor control plane VMs on the management network. And the last network related configuration option, is step 7 in which you define the Kubernetes node IP range, this applies to both the supervisor cluster as well as the guest TKG clusters. This range is defined in the workload network portion in the top part the screen: Click on add to open workload network config screen. Tip, the network name you provide is presented to you when configuring a namespace for a workload within the vSphere\supervisor cluster. Please provide a meaningful name other than network-1 I hope this article helps you to wrap your head around these network requirements a little bit better. Please follow the instructions laid out in Cormacs Blog series and refer to these sets of diagram to get some visual aid in the process. ================================================================================ Title: What's Your Favorite Tech Novel? URL: https://frankdenneman.ai/2020-07-30-whats-your-favorite-tech-novel/ Date: 2020-07-30 Today I was discussing with Duncan some great books to read. Disconnecting fully from work is difficult for me, so typically, the books I read are tech-related. I have read some brilliant books that I want to share with you, but mostly I want to hear your recommendations for other excellent tech novels. Cyberwarfare Cyberwarfare intrigues me, so any book covering Operation Olympic Games -or- Stuxnet interests me. One of the best books on this topic “Countdown to Zero day” by Kim Zetter. The book is filled with footnotes and references to corresponding research. David Sanger, the NYT reporter who broke the Olympic Games story, wrote another brilliant piece on the future of Cyberwarfare, “The Perfect Weapon: War, Sabotage, and Fear in the Cyber Age”. This book is the basis of an upcoming HBO documentary. “No Place to Hide,” tells the story of Glenn Greenwald (The Guardian Journalist) of the encounters with Snowden right after he walked away with highly classified material. Greenwald explores some of the technology used by the NSA uncovered by the Snowden leak. Tech History If you are interested in the inception of the internet, then “Where Wizards stay up late” should be on your bookshelf or a part of your digital library. Moving the computer from being a giant calculator to a communication device. “Command and Control” explores the systems used to manage the American nuclear arsenal. It tells stories about near misses. If you think you’re behind on patching and updating your systems, do yourself a favor and read this book ;) Technothriller Old-timers know Mark Russinovich from the advanced system utility toolset called Sysinternals, the new generation knows him as CTO of Azure Webservices. It turns out, that Mark is a gifted author as well. He published three tech novels that are highly entertaining to read: “Zero day”, “Rogue Code” and “Trojan Horse". Little Brother by Cory Doctorow (Thanks to Mark Brookfield for recommending this) tells an entertaining story of how a young hacker takes on the department of homeland security. Hacking “Ghost in the Wires” reads like a technothriller, but tells the story of the hunt on Kevin Mitnick. A must-have. “Kingpin: How One Hacker Took Over the Billion-Dollar Cybercrime Underground” tells the story of Kevin Poulsen, ofter referred to in Ghost in the Wires, as one of the most notorious hackers focusing on credit card fraud. An exciting and quick read. What’s in your top 5? ================================================================================ Title: New whitepaper available on vSphere 7 DRS Load Balancing URL: https://frankdenneman.ai/2020-07-22-new-whitepaper-available-on-vsphere-7-drs-load-balancing/ Date: 2020-07-22 vSphere 7 contains the new DRS algorithm that differs tremendously from the old one. The performance team has put the new algorithm through the test and have published a whitepaper presenting their findings. Read the white paper: Load Balancing Performance of DRS in vSphere 7.0. ================================================================================ Title: vSphere 7 DRS Scalable Shares Deep Dive URL: https://frankdenneman.ai/2020-05-27-vsphere-7-drs-scalable-shares-deep-dive/ Date: 2020-05-27 You are one tickbox away from completely overhauling the way you look at resource pools. Yes you can still use them as folders (sigh), but with the newly introduced Scalable Shares option in vSphere 7 you can turn resource pools into more or less Quality of Service classes. Sounds interesting right? Let’s first take a look at the traditional working of a resource pool, the challenges they introduced, and how this new delivery of resource distribution works. To understand that we have to take a look at the basics of how DRS distributes unreserved resources first. Compute Resource Distribution A cluster is the root of the resource pool. The cluster embodies the collection of the consumable resources of all the ESXi hosts in the cluster. Let’s use an example of a small cluster of two hosts. After overhead reduction, each host provides 50GHz and 50GB of memory. As a result, the cluster offers 100 GHz and 100 GB of memory for consumption. A resource pool provides an additional level of abstraction, allowing the admin to manage pools of resources instead of micro-managing each VM or vSphere pod individually. A resource pool is a child object of a cluster. In this scenario, two resource pools exist; a resource pool with the HighShares, and a resource pool (RP) with the name NormalShares. The HighShares RP is configured with a high CPU shares level and a high memory shares level, the NormalShares RP is configured with normal CPU shares level, and normal memory shares level. As a result, HighShares RP receives 8000 CPU shares and 327680 shares of memory, while the NormalShares RP receives 4000 CPU shares and 163840 shares of memory. A ratio is created between the two RPs of 2:1. In this example, eight VM with each two vCPUs and 32 GBs are placed in the cluster. Six in the HighShares RP and two VMs in the NormalShares RP. If contention occurs, the cluster awards 2/3 of the cluster resources to HighShares RP and 1/3 of cluster resources to the NormalShares RP. The next step for the RP is to divide the awarded resources to its child-objects, those can be another level of resource pools or workload objects such as VMs and vSphere Pods. If all VMs are 100% active, HighShares RP is entitled to 66 GHz and 66 GBs of memory, the NormalShares RP gets 33 GHz and 33 GBs of memory. And this is perfect because the distribution of resources follows the desired ratio “described” by the number of shares. However, it doesn’t capture the actual intent of the user. Many customers use resource pools to declare the relative priority of workload compared to the workload in the other RPs, which means that every VM in the resource pool HighShares is twice as important as the VMs in the NormalShare RP. The normal behavior does not work that way, as it just simply passes along the awarded resources. In our example, each of the six VMs in HighShares RP gets 1/6 of 2/3s of the cluster resources. In other words, 16% of 66Ghz & 66GB = ~11 GHz & ~ 11 GBs, while the two VMs in the NormalShares RP get 1/2 of 1/3 of the cluster resources. 50% of 33 GHz & 33 GB = ~16 GHz and ~16 GBs. In essence, the lower priority group VMs can provide more resources per individual workload. This phenomenon is called the priority pie paradox. Scalable Shares To solve this problem and align resource pool sizing more with the intent of many of our customers, we need to create a new method. A technique that auto-scales the shares of RP to reflect the workloads deployed inside it. Nice for VMs, necessary for high-churn containerized workloads. (See vSphere Supervisor Namespace for more information about vSphere Pods and vSphere namespaces. And this new functionality is included in vSphere 7 and is called Scalable Shares. (Nice backstory, the initial idea was developed by Duncan Epping and me, not on the back of a napkin, but on some in-flight magazine found on the plane on our way to Palo Alto back in 2012. It felt like a tremendous honor to receive a patent award on it. It’s even more rewarding to see people rave about the new functionality). Enable Scalable Shares Scalable shares functionality can be enabled at the cluster level and the individual resource pool level. It’s easier to enable it at the cluster level as each child-RP automatically inherits the scalable shares functionality. You can also leave it “unticked” at the cluster level, and enable the scalable shares on each individual resource pool. The share value of each RP in that specific resource pool is automatically adjusted. Setting it at this level is pretty much intended for service providers as they want to carve up the cluster at top-level and assign static portions to customers while providing a self-service IAAS layer beneath it. When enabling shares at the cluster-level, nothing really visible happens. The UI shows that the functionality is enabled, but it does not automatically change the depicted share values. They are now turned into static values, depended on the share value setting (High/Normal/Low). We have to trust the system to do its thing. And typically, that’s what you want anyway. We don’t expect you to keep on staring at dynamically changing share values. But to prove it works, it would be nice if we can see what happens under the cover. And you can, but of course, this is not something that we expect you to do during normal operations. To get the share values, you can use the vSphere Managed Object Browser. William (of course, who else) has written extensively about the MOB. Please remember that it’s disabled by default, so follow William’s guidance on how to enable it. To make the scenario easy to follow, I grouped the VMs of each RP on a separate host. The six VMs deployed in the HighShares RP run on host ESXi01. The two VMs deployed in the NormalShares RP run on host ESXi02. I did this because when you create a resource pool tree on a cluster, the RP-tree is copied to the individual hosts inside the cluster. But only the RPs that are associated with the VMs that run on that particular host. Therefore when reviewing the resource pool tree on ESXi01, we will only see the HighShares RP, and when we look at the resource pool tree of ESXi02, it will only show the NormalShares RP. To view the RP tree of a host, open up a browser, ensure the MOB is enabled and go to https:///mob/?moid=ha%2droot%2dpool&doPath=childConfiguration Thanks to William for tracking this path for me. When reviewing ESXi01 before enabling scalable shares, we see the following: ManagedObjectReference:ResourcePool: pool0 (HighShares) CpuAllocation: Share value 8000 MemoryAllocation: Share value: 327680 I cropped the image for ESXi02, but here we can see that the NormalShare RP defaults are: ManagedObjectReference:ResourcePool: pool1 (NormalShares) CpuAllocation: Share value 4000 MemoryAllocation: Share value: 163840 Resource Pool Default Shares Value If you wonder about how these numbers are chosen, an RP is internally sized as a 4vCPU 16GB virtual machine. With a normal setting (default), you get 1000 shares of CPU for each vCPU and ten shares of memory for each MB (16384x10). High Share setting award 2000 shares for each vCPU and twenty shares of memory for each MB. Using a low share setting leaves you with 500 shares per CPU and five shares of memory for each MB. When enabled, we can see that scalable shares have done its magic. The shares value of HighShares is now 24000 for CPU and 392160 shares of memory. How is this calculation made: Each VM is set to normal share value. Each VM has 2 vCPUs ( 2 x 1000 shares = 2000 CPU shares) Each VM has 32 GB of memory = 327680 shares. There are six VMs inside the RP, and they all run on ESXi01: Sum of CPU shares active in RP: 2000 + 2000 + 2000 + 2000 + 2000 + 2000 = 12000 Sum of Memory shares active in RP: 327680 + 327680 + 327680 + 327680 + 327680 + 327680 = 1966080 The result is multiplied by the ratio defined by the share level of the resource pools. The ratio between the three values (High:Normal:Low) is 4:2:1. That means that the ratio between high and normal is 2:1, and thus, HighShares RP is awarded 12000 x 2 = 24000 shares of CPU and 1966080 x 2 = 3932160 shares of memory. To verify, the MOB shows the adjusted values of NormalShares RP, which is 2 x 2000 CPU shares = 4000 CPU shares and 2 x 163840 = 655360 shares of memory. If we are going to look at the worst-case-scenario allocation of each VM (if every VM in the cluster is 100% active), then we notice that the VMs allocation is increased in the HighShares RP, and decreased in the NormalShares RP. VM7 and VM8 now get a max of 7 GB instead of 16 GB, VMs 1 to 6 allocation increases 3 GHz and 3 GB each. Easily spotted, but the worst-case-scenario allocation is modeled after the RP share level ratio. What if I adjust the share level at the RP-level? The NormalShares RP is downgraded to a low memory share level. The CPU shares remain the same. The RP receives 81920 of shares and now establishes a ratio of 4:1 compared to the HighShares RP (327680 vs. 81920). The interesting thing is that the MOB shows the same values as before, 655360 shares of memory. Why? Because it just sums the shares of the entities in the RP. As a test, I’ve reduced the memory shares of VM7 from 327680 to 163840. The MOB indicates a drop of shares from 655360 to 491520 (327680+163840), proofing that the share value is a total of shares of child-objects. Please note that this is a fundamental change in behavior. With non-scalable shares RP, share values are only relative at the sibling level. That means that a VM inside a resource pool competes for resources with other VMs on the same level inside that resource pool. Now a VM with an absurd high number (custom-set or monster-VM) impacts the whole resource distribution in the cluster. The resource pool share value is a summation of its child-object. Inserting a monster-VM in a resource pool automatically increases the share value of the resource pool; therefore, the entire group of workloads benefits from this. I corrected the share value of VM7 to the default of 327680 to verify the ratio of the increase occurring on HighShares RP. The ratio between low and high is 4:1, and therefore the adjusted memory shares at HighShares should be 1966080 x 4 = 7864320. What if we return NormalShares to the normal share value similar to the beginning of this test, but add another High Share value RP to the environment? For this test, we add VM9 and VM10, both equipped with two vCPUs and 32GBs of memory. For test purposes, they are affined with ESXi01, similar to the HighShare RP VMs. The MOB on ESXi01 shows the following values for the new RP HighShares-II: 8000 shares of CPU, 1310720 shares of memory, following the ratio of 2:1. If we are going to look at the worst-case-scenario allocation of each VM, then we notice that the VMs allocation is decreased for all the VMs in the HighShares and NormalShares RP. VMs 1 to 6 get 16% (11 GHz & 11 GBs), while VM 7 and 8 get 50% of 11% of the cluster resources, i.e. 5.5 GHz and 5.5 GBs each. The new VMs 9 and 10 each can allocate up to 11 GHz and 11 GB, same as the VMs in Highshares RP, following the RP share level ratio. What happens if we remove the HighShares-II RP and move VM9 and VM10 into a new LowShares RP? This creates a situation where there are three RPs with a different share level assigned to it, providing us with a ratio of 4:2:1. The MOB view of ESXi01 shows that the LowShares RP shares value is not modified, and the HighShares RP shares quadrupled. The MOB view of ESXi01 shows that the share value of the NormalShares RP shares is now doubled, following the 4:2:1 ratio exactly. This RP design results in the following worst-case-scenario allocation distribution: VMs as Siblings The last scenario I want to highlight is a VM deployed at the same level at the RP level. A common occurrence. Without scalable shares, this could be catastrophic as a Monster-VM could cast a shadow over a resource pool. A (normal share value) VM with 16 vCPUs and 128 GB would receive 16000 shares of CPU and 1310720 shares of memory. In the pre-scalable shares, it would dwarf a normal share value RP with 4000 shares and 163840 shares of memory. Now with scalable shares bubbling up the number of shares of its child-objects, it evens out the playing field. It doesn’t completely solve it, but it reduces the damage. As always, the recommendation is to commit to a single object per level. Once you use resource pools, provision only resource pools at that level. Do not mix VMs and RPs on the same level, especially when you are in the habit of deploying monster VMs. As an example, I’ve deployed the VM “High-VM11” at the same level as the resource pool, and DRS placed it on ESXi02, where the NormalShares RP lives in this scenario. The share value level is set to high, thus receiving 4000 shares for its two vCPUs and 655360 shares for its memory configuration, matching the RP config, which needs to feed the need of two VMs inside. I hope this write-up helps to understand how outstanding Scalable Shares is, turning Share levels more or less into QoS levels. Is it perfect? Not yet, as it is not bulletproof against VMs being provisioned out of place. My recommendation is to explore VEBA (4) for this and generate a function to automatically move root-deployed VMs into a General RP, avoiding mismatch. Closing Notes Please note that I constrained the placement of VMs of an entire RP to a single host in the scenarios I used. In everyday environments, this situation will not exist, and RPs will not be tied to a single host. The settings I used are to demonstrate the inner workings of scalable shares and must not be seen as endorsements or any kind of description of normal vSphere behavior. The platform was heavily tuned to provide an uncluttered view to make it more comprehensible. Worst-case-scenario numbers are something that shows a situation that is highly unlikely to occur. This is the situation where each VM is simultaneously 100% active. It helps to highlight resource distribution while explaining a mechanism, typically resource demand ebbs and flows between different workloads, thus the examples used in these scenarios are not indicative of expected resource allocation when using resource pools and shares only. ================================================================================ Title: vSphere 7 vMotion with Attached Remote Device URL: https://frankdenneman.ai/2020-05-12-vsphere-7-vmotion-with-attached-remote-device/ Date: 2020-05-12 A lot of cool new features fly under the radar during a major release announcement. Even the new DRS algorithm didn’t get much air time. One thing that I discovered this week, is that vSphere 7 allows for vMotion with an attached remote device attached. When using the VM Remote Console, you can attach ISOs stored on your local machine to the VM. An incredibly useful tool that allows you to quickly configure a VM. The feature avoids the hassle of uploading an ISO to a shared datastore, but unfortunately, it does disable vMotion for that particular machine. Even worse, this prohibits DRS to migrate it for load-balancing reasons, but maybe even more annoying, it will fail maintenance mode. We’ve all been there, putting a host into maintenance mode, to notice ten minutes later that the host isn’t placed in MM-mode. Once you exit MM to figure out what’s going on DRS seems to be on steroids and piles on every moveable VM it can find onto that host again. With vSphere 7, this enhancement makes that problem a thing of the past. vMotions will work, and that means so does DRS and Maintenance Mode. When a VM is vMotioned with a remote device attached, the session ends as the connection is closed. In vSphere 7, when a vMotion is initiated, the VMX process sends a disconnect command with a session cookie to the source ESXi host. The device is marked as remote and once the vMotion process is complete, the remote device is connected through VMRC again. Any buffered accesses to the device are completed. Please note that the feature “vSphere vMotion with attached remote devices” is a vSphere 7 feature only and that means that it only works when migrating between vSphere 7 hosts. It has the look and feel of a small function upgrade, but I’m sure it will reduce a lot of frustration during maintenance windows. ================================================================================ Title: DRS Migration Threshold in vSphere 7 URL: https://frankdenneman.ai/2020-05-08-drs-migration-threshold-in-vsphere-7/ Date: 2020-05-08 DRS in vSphere 7 is equipped with a new algorithm. The old algorithm measured the load of each host in the cluster and tried to keep the difference in workload within a specific range. And that range, the target host load standard deviation, was tuned via the migration threshold. The new algorithm is designed to find an efficient host for the VM that can provide the resources the workload demand while considering the potential behavior of other workloads. Throughout a series of articles, I will explain the actions of the algorithm in more detail. Due to the changes, the behavior of the migration threshold changed as well. In general, the migration threshold still acts as the metaphorical gas pedal. By sliding it to the right, you press the gas pedal to the floor. You tell DRS to become more aggressive, to reduce or relax certain thresholds. Underneath the covers, things changed a lot. There is no host load standard deviation to compare, but DRS needs to understand workload demand change and how much host inefficiency to tolerate. Let’s take a look closer: Migration Threshold Other than greatly improving the text describing each migration threshold, the appearance and behavior of the Migration Threshold (MT) have not changed much in vSphere 7. By default, the slider is set to the moderate “3” setting. There are two more aggressive load-balance settings, 4, and 5. 5 Being the most aggressive. To the left of the default, there are two settings, 1 and 2. However, only setting 2 generates load-balancing recommendations. If you set the slider to 1, the most conservative option, DRS only produces migration recommendations that are considered mandatory. Mandatory moves are generated for a few events, the three common reasons are when the host is placed into maintenance mode, when a proactive HA evacuation occurs, or when a VM violates an (anti-) affinity rule. The remainder of the article describes the setting of the sliders by referring to the numbers. Selecting the Appropriate Migration Threshold Setting MT setting 2 is intended for a cluster with mostly stable workloads. Stable in the sense of workload variation, not failure rates ;). When workload generates a continuous workload, it makes less sense to move workload aggressively around. MT 3 is a healthy mix of stable and bursty workloads, while MT 4 and MT5 are designed to react to workloads spikes and bursts. Headroom threshold To influence balancing moves, DRS uses a threshold that identifies a particular level of headroom (used capacity) within a host. To point out, this is not a strict admission control function. The threshold identifies a point where overall host utilization starts to introduce (some) performance loss to the associated workloads on the host. As the new DRS algorithm is designed to consider VM demand, it is focused on finding the best host possible. DRS compares the overall host utilization of each host and migrates VMs to help workloads have enough room to burst. By default, DRS starts to consider the host less efficient when the host load exceeds 50%. Once the threshold is surpassed, DRS examines possible migrations to help find workloads a more efficient host. Similar to the old algorithm, the cost of the migration needs to exceed the improvement of efficiency. When you select a more aggressive migration threshold setting (MT4, MT5), the tolerance for host load is lowered to 30%. From that point on, DRS will take a particular level of inefficiency into account and starts to analyze other hosts to understand whether specific workloads will benefit from placement on another host. Another way to put it is that DRS attempts to provide 70% headroom in this situation. As a result, you will notice more workload migrations when selecting MT4 or MT5. Demand Estimation To find an efficient host, DRS needs to understand the demand of the vSphere Pod or the VM. DRS uses a number of stats to get a proper understanding of the workload demand. These stats are provided by the ESXi hosts. DRS is designed to be conservative as it doesn’t want to move a virtual machine based on an isolated event. Taking care of a sudden increase in demand is the role of the ESXi host. When this becomes structural behavior, then it’s DRS task to find the right spot for this VM. Please note that a vSphere pod will not be migrated. Although DRS does not load-balance vSphere pods to get a better overall capacity usage, it will keep track of the vSphere Pods demand since it could affect the performance of other workloads (VMs and pods) running on the host. For more info about vSphere pods, please read the article “Initial Placement of a vSphere Pod”. By default, DRS calculates an average demand of over 40 minutes for each workload. This period depends on the VM memory configuration and the migration threshold. We learned that DRS needs to use a shorter history for smaller VMs to better catch the behavior of vSphere pods or VMs with a smaller memory footprint. Below is an overview of the Migration Threshold settings and the number of minutes used to determine the demand for each workload. Cost-Benefit DRS needs to consider the state of the cluster, the workload demand of all the vSphere pods and the VMs, and it needs to ensure that whatever it does, does not interfere with the primary goal of the virtual infrastructure, and that is providing resources to workloads. Every move made consumes CPU resources. It absorbs bandwidth, and in some cases, can affect memory sharing benefits. DRS must weigh all the possibilities. With DRS in vSphere 7 that functionality is called cost-benefit filtering. Moving the migration threshold to the right reduces certain cost filtering aspects while relaxing some benefit requirements. This allows DRS to become more responsive to bursts. As a result, you will notice more workload migrations when selecting MT4 or MT5. DRS Responsibility Please remember that vSphere consists of many layers of schedulers all working closely together. DRS is responsible for placement, finding the best host for that particular vSphere pod or VM. However, it is the responsibility of the actual ESXi host schedulers to ensure the workload gets scheduled on the physical resources. Consider DRS as the host or hostess of the restaurant, escorting you to a suitable table, while the ESXi schedulers are the cooks and the waiters. Individual workload behavior can change quite suddenly, or there is an abrupt change in resource availability. DRS needs to coordinate and balance all the workloads and their behavior in any possible scenario. The new DRS algorithm is completely redesigned, and I think it’s an incredible step forward. But a new algorithm, with new tweakable parameters, also means that we can expect different behavior. It’s expected that you will see more vMotions compared to the old algorithm, regardless of MT selection. A future article will explain the selection process of the algorithm. As always, it’s recommended to test the new software in a controlled environment. Get to understand its behavior and test out which migration threshold fits your workload best. ================================================================================ Title: vSphere Supervisor Namespace URL: https://frankdenneman.ai/2020-04-01-vsphere-supervisor-namespace/ Date: 2020-04-01 vSphere 7 with Kubernetes enables the vSphere cluster to run and manage containers as native constructs (vSphere Pods). The previous two articles in this series cover the initial placement of a vSphere pod and compute resource management of individual vSphere pods. This article covers the compute resource management aspect of the vSphere Supervisor namespace construct. Cormac Hogan will dive into the storage aspects of the Supervisor namespace in his (excellent) blog series. Supervisor Cluster A vSphere cluster turns into a Supervisor cluster once Kubernetes is enabled. Three control plane nodes are spun up and placed in the vSphere cluster, and the ESXi nodes within the cluster act as worker node (resource providers) for the Kubernetes cluster. A vSpherelet runs on each ESXi node to control and communicate with the vSphere pod. More information about the container runtime for ESXi is available in the article “Initial Placement of a vSphere pod.” Supervisor Namespace Once up and running, a Supervisor cluster is a contiguous range of compute resources. The chances are that you want to carve up the cluster into smaller pools of resources. Using namespaces turns the supervisor cluster into a multi-tenancy platform. Proper multi-tenancy requires a security model, and the namespace allows the vAdmin to control which users (developers) have access to that namespace. Storage policies that are connected to the namespace provide different types and classes for (persistent) storage for the workload. Not only vSphere Pods can consume the resources exposed by a Supervisor namespace. Both vSphere pods and virtual machines can be placed inside the namespace. Typically the virtual machine placed inside the namespace could be running a Tanzu Kubernetes Grid Cluster (TKG). Still, you are entirely free to deploy any other virtual machine in a namespace as well. Namespaces allow you to manage application landscapes at a higher level. If you have an application that consists of virtual machines running a traditional setup and adding new services to this application that run in containers, you can group these constructs in a single namespace. Assign the appropriate storage and compute resources to the namespace and monitor the application as a whole. We want to move from managing hundreds or thousands of virtual machines individually to managing a small group of namespaces. (i.e., up-leveling workload management). Default Namespace Compute resources are provided to the namespace by a vSphere DRS Resource Pool. This resource pool is not directly exposed, a vAdmin interfaces with the resource pool via the namespace UI. In the screenshot below, you can see a Workload Domain Cluster (WLD) with vSphere with Kubernetes enabled. Inside the Supervisor cluster, a top resource pool “Namespace” is created automatically, and the three Control Plane VMs of the Supervisor cluster are directly deployed in the Namespaces resource pool. (I will address this later). Cormac and I have created a couple of namespaces, and the namespace “frank-ns” is highlighted. As you can see, this new construct is treated to a new icon. The summary page on the right side of the screen shows the status, the permission (not configured yet), the configured storage policy attached, and the capacity and usage of compute resources. The bottom part of the screen shows whether you have deployed pods or TKG clusters. In this example, three pods are currently running. Compute Resource Management With a traditional Resource Pool, the vAdmin can set CPU and memory reservations, shares, and limits to guarantee and restrict the consumption of compute resources. A supervisor namespace does not expose the same settings. A namespace allows the vAdmin to set limits or requests (reservations) and limits on a per-container basis. Limits A vAdmin can set a limit on CPU or memory resources for the entire namespace. This way, the developer can deploy the workload in the namespace, and not risk consuming the full compute capacity of the supervisor cluster. Beyond the resource pool limits, a vAdmin can also set a per container default limit. The namespace will automatically apply a limit to each incoming workload, regardless of the resource configuration specified in the YAML file of the containerized workload. On top of this, the vAdmin can also specify object limits. A maximum number of pods can be specified for the namespace, ultimately limiting the total consumed resources by the workload constructs deployed in the namespace. Reservations A Supervisor namespace does not provide the option to set a reservation at the namespace level. However, the resource pool is configured with an expandable reservation and that allows the resource pool to request for unreserved resources from its parent. These unreserved resources are necessary to satisfy the request for reservable resources for a workload. The resource pool “Namespaces” is the parent resource pool where all namespaces are deployed in. The resource pool “Namespaces” is not configured with reserved resources and as a result, it will request unreserved resources from its parent, which is the root resource pool, better known as the cluster object. A reservation of resources is needed to protect a workload from contention. This can be done via two methods. A vAdmin can set a default reservation per container, or the resource requests must be specified in the resource configuration of the YAML file. If the vAdmin sets a default reservation per container, every container that is deployed in that namespace will be configured with that setting. The developer can specify a request or a limit for each container individually in the workload YAML file. Based on the combination of requests and limits used, Kubernetes automatically assigns a QoS class to the containers inside the pod. And based on Qos classes, reclamation occurs. There are three Quality of Service (QoS) classes in Kubernetes, BestEffort, Burstable, and Guaranteed. Both the Burstable and Guaranteed classes consist of a request configuration. With Burstable QoS class, the limit exceeds the number specified by the request. The Guaranteed QoS class requires that the limit and request are set to an identical value. That means that the relative priority of the namespace determines whether BestEffort or the part of the resources of the Burstable workload that is not protected by a request setting will get the resources they require during resource contention. The relative priority is specified by the level of shares assigned to the namespace. Shares DRS assigns shares to objects within the cluster to determine the relative priority when there is resource contention. The more shares you own, the higher priority you have on obtaining the resources you desire. It’s an incredibly dynamic (and elegant) method of catering to the needs of the active objects. A VM or resource pool can have all the shares in the world, but if that object is not pushing an active workload, these shares are not directly in play. Typically, to determine the value of shares awarded to an object, we use the worst-case scenario calculation. In such an exercise, we calculate the value of the shares, if each object is 100% active. I.e., the perfect storm. DRS assigns each object shares. DRS awards shares based on the configured resources of the object. The number of vCPUs and the amount of memory and then multiplying it with a number of shares. The priority level (low, normal, high) of the object determines the factor of shares. The priority levels have a 1:2:4 ratio. The normal priority is the default priority, and each vCPU gets 1000 CPU shares awarded. For every MB of memory, 10 shares are allocated. For instance, a VM with a 2 vCPU configuration, assigned the normal priority level, receives 2000 shares (2 vCPU x 1000). If the VM is configured with a high priority level, it will receive 4000 shares (2 vCPU x 2000). Although a resource pool cannot run workload by itself, DRS needs to assign shares to this construct to determine relative priority. As such, the internal definition of a resource pool for DRS equals that of a 4 vCPU, 16GB VM. As a result, a normal priority resource pool, regardless of the number of objects it contains, is awarded 4000 CPU shares and 163840 shares of memory. A namespace is equipped with a resource pool configured with a normal priority level. Any object that is deployed inside the namespace receives a normal priority as well, and this cannot be changed. As described in the “Scheduling vSphere Pods” article, a container is a set of processes and does not contain any hardware-specific sizing configuration. It just lists the number of CPU cycles and the amount of memory it wants to have and that the upper limit of resource consumption should be. vSphere interprets the requests and limits as CPU and memory sizing for the vSphere pod (CRX), and DRS can assign shares based on that configuration. As more containers can be deployed inside a pod, a combination of limits and requests of the containers is used to assign virtual hardware to the vSphere Pod. BestEffort workloads do not have any requests and limit set, and as such, a default sizing is used of 1 vCPU and 512MB. From a shares perspective, this means that a vSphere pod running a single container receives 1000 CPU shares and 5120 shares of memory. A Burstable QoS class has a request set or both a request and a limit. If either setting is larger than the default size, that metric is used to determine the size of the container (see image below). If the pod manifest contains multiple containers, the largest parameter of each container is added, and the result is used as a vSphere pod size. For example, a pod includes two containers, each with a request and limit that are greater than the default size of the container. The CPU limit exceeds the quantity of the CPU request. As a result, vSphere uses the sum of both CPU limits and adds a little padding for the components that are responsible for the pod lifecycle, pod configuration, and vSpherelet interaction. A similar calculation is done for memory. Relative Priority During Sibling Rivalry Why are these vSphere pod sizes so interesting? DRS in vSphere 7 is equipped with a new feature called Scalable shares and it uses the CPU and memory configurations of the child objects to correctly translate the relative priority of the resource pools with regards of its siblings. The resource pool is the parent of the objects deployed inside. That means that during resource contention, the resource pool will request resources from its parent, the “Namespaces” resource pool, and it will, in turn, request resources from its parent the root resource pool (Supervisor Cluster). At each level, other objects exist doing the same thing during a perfect storm. That means we have to deal with sibling rivalry. Within the “Namespaces” RP, a few objects are present. Two namespaces and three control plane VMs. A reservation protects none of the objects, and thus each object has to battle it out with their share value if they want some of the 126.14 GB. Each control plane VM is configured with 24 GBs, owning 245,760 shares. Both RPs own 163,840 of CPU shares. A total of 1,064,960 shares are issued within the “Namespaces” RP, as shown in the UI, each control plane owns 23.08% of the total shares, whereas both resource pools own 15.38%. In a worst-case scenario, that means that the “Namespaces” RP will divide the 126.14 GB between the five objects (siblings). Each control plane node is entitled to consume 23.08% of 126.14 GB = 29.11 GB. Since it cannot allocate more than its configured hardware, it will be able to consume up to 24GB (and its VM overhead) in this situation. The remaining 5 GB will flow back to the resource pool and will be distributed amongst the objects that require it. In this case, all three control planes consume 72 GB (3 x 24 GB), and the 54.14 GB will be distributed amongst the “frank-ns” namespace and “vmware-system-reg…” (which is the harbor) namespace. The resource requirements of the objects within each namespace can quickly exceed the relative priority of the namespace amongst its siblings. And it is expected that more namespaces will be deployed, further diluting the relative priority amongst its siblings. This behavior is highlighted in the next screenshot. In the meantime, Cormac has been deploying new workloads. He created a namespace for his own vSphere pods. He deployed a TKG cluster and a Cassandra cluster. All deployed in their own namespace. As you can see, my namespace “frank-ns” is experiencing relative priority dilution. The percentage of shares has been diluted from 15.38% to 10.53%. I can expect that my BestEffort and Burstable deployments will not get the same amount of resources they got before if resource contention occurs. The same applies to the control plane nodes. They are now entitled to 15.79% of the total amount of memory resources. That means that each control plane node can access 19.92 GB (15.79% of 126.14GB). Design Decision I would consider applying either a reservation on the control plane nodes or create a resource pool and set a reservation at the RP level. If a reservation is set at the VM object-level it has an impact on admission control and HA restart operations (Are there enough unreserved host resources left after one or multiple host failures in the cluster? Reserved Resources The available amount of unreserved resources in the “Namespaces” RP are diluted when a Guaranteed or Burstable workload is deployed in one of the namespaces. The RPs backing the namespaces are configured as “Expandable” and therefore request these resources from their parent. If the parent has these resources available, it will provide them to the child resource and immediately mark it as reserved. The namespace will own these resources as long as the namespace exists. Once the Guaranteed or Burstable workload is destroyed, the reserved resources flow back to the parent. Reserved resources, when in use, cannot be allocated by other workloads based on their share value. The interesting to note here is that in this situation, multiple Burstable workloads are deployed inside the namespaces. The Used Reservation of the “Namespaces” RP shows that 36.75GB of resources are reserved. Yet when looking at the table, none of the namespaces or VMs are confirming any reservation. That is because that column shows the reservation that is directly configured on the object itself. And no resource pool that backs a namespace will be configured directly with a reservation. Please note that it will not sum the vSphere pod or VM reservations that are running inside the RP! The summary view of the namespace shows the capacity and usage of the namespace. In this example, the summary page is shown of the “Cormac-ns”. It shows that the namespace is “consuming” 3.3 GHz and 4.46 GB. These numbers are a combination of reservation (request) and actually usage. This can be been seen when each individual pod is inspected. The summary page of the “cassandra-0” pod shows that 1 CPU is allocated and 1 GB is allocated, the pod consumes some memory and some CPU cycles. The metadata of the pod shows that this pod has a QoS class of Guaranteed. When viewing the YAML file, we can see that the request and limit of both CPU and Memory resources are identical. Interestingly enough, the CPU resource settings show 500m. The m stands for millicpu. A 1000 millicpu is equal to 1 vCPU, so this YAML file states that this container is fine with consuming half a core. However, vSphere does not have a configuration spec for a virtual CPU of half a core. vSphere can schedule per MHz, but this setting is used to define the CRX (vSphere pod) configuration. And therefore the vSphere pod is configured with the minimum of 1 vCPU and this is listed in the Capacity and Usage view. Scalable Shares The reason why this is interesting is that Scalable shares can calculate a new share value based on the number of vCPU of total memory configuration of all the objects inside the resource pool. How this new functionality behaves in an extensive resource pool structure is the topic of the next article. Previous Articles in this Series Initial Placement of a vSphere Pod Scheduling vSphere Pods ================================================================================ Title: Scheduling vSphere Pods URL: https://frankdenneman.ai/2020-03-20-scheduling-vsphere-pods/ Date: 2020-03-20 The previous article “Initial Placement of a vSphere Pod,” covered the internal construction of a vSphere Pod to show that it is a combination of a tailor-made VM construct and a group of containers. Both the Kubernetes and vSphere platforms contain a rich set of resource management controls, policies, and features that guide, control, or restrict scheduling of workloads. Both control planes use similar types of expressions, creating a (false) sense of unification in the context of managing a vSphere pod. This series of articles examines the overlap of the different control plane dialects, and how the new vSphere 7 platform allows the developer to use Kubernetes native expressions, while the vSphere Admin continues to use the familiar vSphere native resource management functionalities. Workload Deployment Process Both control planes implement a similar process to deploy a workload, the workload needs to be scheduled on a resource producer (worker node, or ESXi host), the control plane selects a resource producer based on the criteria presented by the resource consumer (pod manifest / VM configuration). The control plane verifies which resource producer is equipped with enough resource capacity, if there are enough resources available and if it meets the criteria listed in the pod manifest/VM configuration. An instruction is sent over to the workload producer to initiate a power-up process of the workload. The difference between the deployment processes of containers and virtual machines is the size aspect. A virtual machine is defined by its virtual hardware and this configuration acts as a boundary for the guest OS and its processes (hence the strong isolation aspect of a virtual machine). A container is a controlled process and uses an abstract OS to control the resource allocation, there is no direct hardware configuration assigned to a process. And this difference introduces a few interesting challenges when you want to run a container natively on a hypervisor. The hypervisor requires workload constructs to define their hardware configuration so it can schedule the workloads and manage resource allocation between active workloads. How do you size a construct that might provide you with absolutely no hints on expected resource usage? You can prescribe an arbitrary hardware configuration, but then you miss out on capturing the intent of the developer if he or she wants the application to be able to burst and temporarily use more resources than it structurally needs. You do not want to create a new resource management paradigm, where developers need to change their current methods of deploying workloads, you want to be able to accept new workloads with the least amount of manual effort. But having these control planes work together is not only a process of solving challenges, it provides the ability to enrich the user experience as well. This article explores the difference in resource scheduler behavior and how Kubernetes resource requests, limits, and QoS policies affect vSphere pod sizing. It starts off by introducing Kubernetes constructs and Kubernetes scheduling to provide enough background information to understand how it can impact vSphere pod sizing and eventually placement of a vSphere pod. Managing Compute Resources for Containers In Kubernetes, you can specify how much resources a container can consume (limits) and how many resources the worker node must allocate to a container (request). These are similar to vSphere VM reservations and limits. Similar to vSphere, Kubernetes selects a worker node (Kubernetes term for a host that runs workload), based on the request (reservation) of a container. In vSphere, the atomic unit to assign reservation, shares, and limits is the virtual machine, in Kubernetes, it’s the container. It sounds straight-forward, but there is a bit of a catch. A container is deployed not directly onto a Kubernetes Worker Node, but it is encapsulated in a higher-level construct called a pod. In short, the reason why a pod exists is that it’s expected to run a single process in a container. If an app exists out of multiple processes, a group of containers should exist, and you do not want to manage a group of processes independently, but the app itself, hence the existence of a pod. What’s the catch? Although you deploy a pod, you have to specify the resource allocation settings per container and not per pod. But since a pod is an atomic unit for scheduling, the requests of all the containers inside the pod are summed, and the result is used for worker node selection. Once the pod is placed, the worker node resource scheduler has to take care of each container request and limits individually. But that is a topic for a future article. Let’s take a closer look at a pod manifest. Container Size Thinking versus VM Size Thinking The pod manifest list two containers, each equipped with a request and a limit for both CPU and memory. A CPU can be expressed in a few different ways. 1 equals to 1 CPU, which is the same as a hyperthread of an Intel processor. If that seems lavishly outlandish, you can actually use a smaller unit of expression by using millicpu or decimals. That means that 0.5 means half of a hyperthread or 500 millicpu. For a seasoned vSphere admin, you are now exploring the other end of the spectrum, instead of dealing with users who are demanding 64 Cores, we are now trying to split atoms here. With memory, you can express memory requirements in plain integers (126754378954) or fixed-point integers using suffixes 64MiB (226 bytes). Kubernetes.Io documentation can help you with which fixed-point integer exists. In this example, the pod request is 128 Mi of memory and 500m of CPU resources. Kubernetes Scheduling During the process of initial placement of the container, the scheduler needs to check the “compatibility” first. After the Kubernetes scheduler has considered the taints and tolerations, the pod affinity and anti-affinity, and the node affinity, it looks at the node capacity to understand if it can satisfy the pod requests (128Mi and 500m CPU). To be more precise, it inspects the “node allocatable.” This is the number of resources that is available for pods to consume. Kubernetes reserves some resources of the node to make sure system daemons and itself can run without risking resource starvation. The node allocatable resources are divided into two parts, allocated and unallocated. The total of allocated resources is the sum of all request configurations of all active containers on the worker node. As a result, Kubernetes matches the request stated in the pod manifest and the unallocated resources listed by each worker node in the cluster. The node with the most unallocated resources is selected to run the pod. To be clear, Kubernetes’ initial placement does not consider actual resource usage. As depicted in the diagram, the workload needs to be scheduled. The Kubernetes control plane reviews the individual worker nodes, filters the nodes out which node can fit the pods, and then selects the node based on the configured prioritization function. The most used function is the “LeastRequestedPriority” option, which favors worker nodes with fewer requested resources. As node B has the least amount of reserved resources, the scheduler deems this node to be the best candidate to run the workload. vSphere Scheduling DRS has a more extensive resource scheduling model. The method used by Kubernetes is more or less in-line with the vSphere admission control model. Kubernetes scheduling contains more nuances and features than I just described in the paragraphs above. It checks if a node is reporting memory pressure, allowing it to exclude from the node selection list (CheckNodeMemoryPressure), and a priority functionality is in beta, but overall looking at reserved and unreserved memory can be considered to be a little bit coarse. vSphere has three admission controls that all work together to ensure continuity and resource availability. DRS resource scheduling aligns the host resource availability, with the resource entitlement of the workload. Reservations, shares, limits, and actual resource usage of a workload is used to determine the appropriate host. Now you might want to argue that a workload that needs to be placed does not use any resources yet, so how does this work? During initial placement, DRS considers the configured size as resource entitlement, in this case the resource entitlement is a worst-case scenario. So a VM with a hardware configuration of 4 vCPUs and 16 GB has a resource entitlement before power-up of 4 vCPUs and 16GB plus some overhead for running the VM (VM overhead). However, if a reservation is set of 8GB, then the resource entitlement is now switched to a minimum resource entitlement of 4 vCPU, 8GB+VM overhead. A host must have at least have 8GB(+VM overhead ) of unreserved resources available to be considered. How is this different from Kubernetes? Well, this part isn’t. The key is taking the future state of a workload into consideration. DRS understands the actual resource usage (and the entitlement) of all other active workloads running on the different hosts. And thus, it has an accurate view of the ability (and possibility) of the workload to perform on each host. Entitlement indicates the number of resources the workload has the right to consume; as such, it also functions as a form of prediction, an estimation on workload pressure. Would you rather place a new workload on a crowded host or on one that is less busy? In this example, there are three hosts, each with 100 GB of memory capacity. Host A has workload active that has reserved 60 GBs of memory. 40 GB of memory is unreserved. Host B and host C have workload active that have reserved 40 GBs of memory. 60GBs of memory is unreserved. A new workload with a 35 GB reservation comes in. The Kubernetes scheduler would have considered both hosts to be equally good. However, DRS is aware of active use. Host B has an active resource consumption of 70 GB, while host C has an active use of 45 GBs. As host B resource usage is closer to its capacity, DRS selects host C as the destination host for initial placement. Considering active resource usage of other active resource consumers, whether they are VM constructs or containers in vSphere pods, creates a platform that is more capable of satisfying intent. If a pod is configured with a burstable Quality of Service class (limit exceeds request), the developer declares the intent that the workload should be able to consume more resources if available. With initial placement enriched with active host resource usage, the probability of having that capability is highly increased. Managing Compute Resources for vSphere Pods Seeing that a vSphere pod is a combination of containers and a VM construct, both control planes interact with the different components of the vSphere pod. But some of the qualities of the different constructs impact the other constructs’ behavior, for example, sizing. A VM is a construct defined by its hardware configuration. In essence, a VM is virtualized hardware, and the scheduler needs to understand the boundaries of the VM to place and schedule it properly. This configuration acts as a boundary of the world where the guest OS lives in. A guest OS cannot see beyond the borders of a VM, and therefore it acts and optimizes for the world it lives in. It has no concept of “outer-space”. A container is the opposite of this. A container is, contrary to its definition, not a rigid or a solid structure. It’s a group of processes that are contained by features to isolate or “contain” resource usage (control groups). Compare this to a process or an application on a Windows machine. When you start or configure an application, you are not defining how many CPUs or memory that particular app can consume. You hope it doesn’t behave like Galactus (better known as Google Chrome) and that it just won’t eat up all your resources. That means that a process in Windows can see all the resources the host (i.e., laptop or virtual machine) contains. The same applies to Linux containers. A container can see all the resources that are available to its host; the limit setting restricts it to consume above this boundary. And that means that if no limit is set, the container should be able to consume as much as the worker node can provide. I.e., in VM-sizing terms, the size of the container is equal to the size worker node. If this container was to run inside a vSphere pod, the vSphere pod should have the size of the ESXi host. Although we all love our monster-VMs, we shouldn’t be doing this. Especially when most expressions of container resource management borders on splitting atoms, and are not intended to introduce planet-sized container entities in the data center. Kubernetes QoS Classes and the impact of vSphere Pod Sizing A very interesting behavior of Kubernetes is the implicit definition of Quality of Service (QoS) classes due to combinations of limits and requests definition in the pod manifest. As seen in the introduction of this article, a pod manifest contains limits and requests definitions of each container. However, these specifications are entirely optional. Based on the combination used, Kubernetes automatically assigns a QoS class to the containers inside the pod. And based on Qos classes, reclamation occurs. A developer well versed in the Kubernetes dialect understands this behavior and configures the pod manifest accordingly. Let’s take a look at the three QoS classes to understand a developers’ intent better. Three Qos Classes exist BestEffort class, Burstable class, and Guaranteed class. If no requests and limits are set on all containers in the pod manifest, the BestEffort class is assigned to that pod. That means all containers in that pod can allocate as many resources as they want, but they are also the first containers to be evicted if resource pressure occurs. If all containers in the pod manifest contain both a memory and CPU requests and the request equals the limit, then Kubernetes assigns the Guaranteed QoS class to the Pod. Guaranteed pods are the last candidates to be hit if resource contention occurs. Every other thinkable combination of CPU and memory requests and limits ensures that Kubernetes assigns the Burstable class. It is important not to disrupt the expected behavior of resource allocation and reclamation and as a result, the requests and limits used in the pod manifest are used as guidance for vSphere Pod sizing while keeping the expected reclamation behavior of the various combinations. If there is no limit set on a container, how must vSphere interpret this when sizing the vSphere pod? To prevent host-sized vSphere pods, a default container size is introduced. It’s on a per-container basis. To be exact, if a simplest pod with one container and no request/limit settings is created, that vSphere Pod will get 1 vCPU and 512 MB. It actually gets 0.5 cores by default, but if there is only one container we will round the vCPU up to 1. Why not on a pod basis? Simply because of scalability reasons, the size of the Pod scales up with the number of BestEffort containers inside. If a request or a limit is set, that is larger than the default size, than this metric is used to determine the size of the container. If the pod manifest contains multiple containers, the largest metric of each container is added and the result is used as a vSphere pod size. For example, a pod contains two containers, each with a request and limit that are greater than the default size of the container. The CPU limit exceeds the size of the CPU request, as a result, vSphere uses the sum of both CPU limits, and adds a little padding for the components that are responsible for the pod lifecycle, pod configuration, and vSpherelet interaction. A similar calculation is done for memory. Initial Placement of Containers Inside a vSphere Pod on vSphere with Kubernetes When a developer pushes a pod manifest to the Kubernetes control plane, the Kube-Scheduler is required to find an appropriate worker node. In the Kubernetes dialect, the worker-node that meets the resource allocation requirements of the pod is called a feasible node. In order to determine which nodes are feasible, Kube-Scheduler will filter the nodes that do not have enough unallocated resources that are required to satisfy the listed requests in the pod manifest. The second step in the process done by the Kube-Scheduler is to score each feasible node in the list based on additional requirements, such as affinity and labels. The ranked list is sent over to the Pacific Scheduler Extension which in turn sends it over to the vSphere API server, who forwards it to the vSphere DRS service. DRS determines which host aligns best with the resource requirements and is the most suitable candidate to ensure that the vSphere pod reaches the highest happiness score (getting the resources the vSphere pod is entitled to). The vSphere Pod LifeCycle Controller ensures that the Spherelet on the selected host creates the pod and injects the Photon Linux Kernel into the vSphere pod. The Spherelet starts the container. (See Initial Placement of a vSphere Pod for a more detailed diagram of the creation process of a vSphere Pod). Please note that if the developer specifies a limit that exceeds the host capabilities than the configuration is created, however, the vSphere pod fails to deploy. Resource Reclamation In addition to sizing the vSphere pod, vSphere uses the resources requests listed in the pod manifest to apply vSphere resource allocation settings to guarantee the requested resources are available. There can be a gap between the set reservation and the size of the vSphere pod. Similar to VM behavior, these resources are available as long as there is no resource contention. When resource contention occurs, the reclamation of resources is initiated. In the case of a vSphere pod, vSphere broad spectrum of consolidation techniques are used, but when it comes to the eviction of a pod, vSphere lets Kubernetes do the dirty work. In all seriousness, this is due to internal Kubernetes event management and a more granular view of resource usage. Namespaces In Kubernetes, a higher-level construct is available to guide and control its member pods, this construct is called the namespace. vSphere 7 with Kubernetes provides a similar construct at the vSphere level, the supervisor namespace. A vSphere resource pool is used to manage the compute resources of the supervisor namespace. The namespace can be configured with an optional limit range that defines a default request or limit on containers, influencing vSphere pod sizes and reclamation behavior. The supervisor namespaces is a vast topic and therefore more info about Namespace will appear in the next article in this series. Previous articles in this series Part 1: Initial Placement of a vSphere Pod ================================================================================ Title: Deep Learning Technology Stack Overview for the vAdmin - Part 1 URL: https://frankdenneman.ai/2020-03-12-deep-learning-technology-stack-overview-for-the-vadmin-part-1/ Date: 2020-03-12 Introduction We are amid the AI “gold rush.” More organizations are looking to incorporate any form of machine learning (ML) or deep learning in their services to enhance customer experience, drive efficiencies in their processes or improve quality of life (healthcare, transportation, smart cities). Train Where Data is Generated One of the key elements that drive on-premise ML focused infrastructure growth is the reality of data gravity. Mentioned in the article “Multi-GPU and Distributed Deep Learning,” deep learning (DL) gets better with data. Consequently, data sets used for ML and DL training purposes are growing at a tremendous rate. These vast data sets need to be processed. Data transit, hosting, and the necessary compute cycles impact the overall OPEX budget. Additionally, data protection regulations such as data residency, data sovereignty, and data locality impact where data can be stored outside the place it is created. As a result, a lot of forward-leaning organizations are repatriating their AI platforms to run ML and DL workloads close to the systems that generate the data. And what better platform to run ML and DL workloads than vSphere? Machine learning comes with its own set of lingo, and different personas interacting with the machine learning stack. To be able to have a meaningful conversation with data scientists and ML engineers, you need to have a basic understanding of how each component interacts with each other. You don’t have to learn the ins and out of the different neural networks, but having an idea of what a particular component does help you understand how it might impact your service levels and your selection of components of the vSphere platform used for ML workloads. To give an example, OpenCL and Vulkan are frameworks that allow for the execution of code on GPU (General Purpose GPU). Using this framework allows you to theoretically expose any GPUs to a machine learning framework such as Tensorflow or Pytorch. As it’s open-source, you can use it on all kinds of GPUs from different vendors. However, all popular actively-developed frameworks do not support OpenCL or Vulkan and only use the NVIDIA CUDA API framework, thus impacting your hardware selection for the vSphere host design. I created an overview of the different layers of the deep learning technology stack, attempting to make sense of the relationships between the components of each different layer. Deep Learning Technology Stack Let’s use a bottom-up approach for reviewing the deep learning technology stack. vSphere Constructs and Accelerators Hardware over the last 20 years looked uniform, other than the vendor and some minor vendor-specific functionality; the devices appeared relatively similar to a guest OS. Most code can run on an AMD as well as an Intel without changing. Hardware competed on scale and speed, not on different ways how it can interact with software. Today’s acceleration devices are very diverse and expose their explicit architecture to the application. The hardware specifics determine the code and algorithm used in the application. And therefore, we need to expose these devices to the application in its most raw and unique form. As a result, the overview primarily covers acceleration devices such as GPUs and FPGAs. However, recently Rice University released a new research paper called: “SLIDE: In Defense of Smart Algorithms over Hardware Acceleration for Large-Scale Deep Learning Systems”. They are demonstrating that by using a fundamentally different approach, it is possible to accelerate deep learning without using hardware accelerators like GPUs. Our evaluations on industry-scale recommendation datasets, with large fully connected architectures, show that training with SLIDE on a 44 core CPU is more than 3.5 times (1-hour vs. 3.5 hours) faster than the same network trained using TF on Tesla V100 at any given accuracy level. On the same CPU hardware, SLIDE is over 10x faster than TF I assume the authors meant a dual-socket system with two 22-cores Xeon CPUs, but an exciting development that certainly needs to be closely followed. For now, let’s concentrate on the components used by the majority of the deep-learning community. vSphere can expose the accelerator devices via two constructs right now, DirectPath I/O (Passthrough) and NVIDIA vGPU. In 2020 two additional constructs will be available; Dynamic DirectPath I/O (information will follow soon) and Bitfusion. Bitfusion pools and shares accelerators across VMs inside the cluster. It provides virtual remote attached GPUs that can be shared between VMs fully or fractionally. Bitfusion can even assign multiple GPUs to a single VM to support any form of distributed deep learning strategy. DirectPath I/O DirectPath I/O, often called Passthrough, provides similar functionality as bare-metal GPU. DirectPath I/O is used for maximum hardware device performance inside the VM and to use native vendor driver and app stack support in the guest OS and the application. Perfect for running specialized software libraries provided by the CUDA stack, which is covered in a later paragraph. DirectPath I/O allows for maximum performance because the I/O mapped from the application and guest OS directly to the hardware device; the VMkernel (Hypervisor) is not involved. A complete device is assigned to a single VM and cannot be shared with other active VMs. This prohibits fractional use of the device (i.e., assigning half of the device resources to a VM). DirectPath I/O allows assigning multiple GPUs to a VM. When a DirectPath I/O device is assigned to a virtual machine, it uses the physical location of the device for the assignment, i.e., Host:Bus: Device-Physical-Function. (The article “Machine Learning Workload and GPGPU NUMA Node Locality” describes locality assignment in detail). Due to this, DirectPath I/O is not compatible with certain core virtualization features, such as vMotion (and thus DRS). Design Impact Currently, vSphere does not support the AMD Radeon Server Accelerators for Deep learning (Radeon Instinct). vSphere supports only NVIDIA GPUs at the moment. From vSphere 6.7 update 1, FPGAs can also be directly exposed to the guest OS by using DirectPath I/O. At the time of writing this article, vSphere 6.7 update 1 supports the Intel Arria 10 GX FPGA. More details on the vSphere blog. NVIDIA vGPU NVIDIA virtual GPU (vGPU) provides advanced GPU virtualization functionality that enables the sharing of GPU devices across VMs. An NVIDIA GPU can be logically partitioned (fractional GPU) to multiple virtual GPUs. A VM can use multiple vGPUs that are located in the same host. Both the hypervisor and the VM need to run NVIDIA software to provide fractional, full, and multiple vGPU functionalities. Bitfusion In 2019, VMware acquired Bitfusion, and I’m looking forward to having this functionality available to our customers. Bitfusion FlexDirect software allows for pooling GPU resources and providing a dynamic remote attach service. That means that workload can run on vSphere hosts that do not have GPU hardware installed. The beauty of this solution is that it does not require any changes to the application. It uses native CUDA (see acceleration libraries paragraph) to intercept the application calls, the FlexDirect software sends it to the FlexDirect server across the network. The Flexdirect server has a DirectPath I/O connection to all the GPUs in that host and manages the placement and scheduling of the workloads. This model corresponds heavily to the early days of virtualization. We used to have 1000’s x86 servers in the data center, each having an average utilization of less than 10% while costing a lot of money. We consolidated compute resources and managed the workload in such a manner than peak utilization did not overlap. With the rise of general-purpose computing on GPU, we see the same patterns. The GPUs are not cheap, sometimes the cost of a GPU server is an order of magnitude more expensive than a “traditional” server. However, when we look at the utilization, we see an average usage of 5 to 20%. **Deep Learning Development Cycle **With deep-learning, you cannot just pump some data into a deep-learning model and expect a result. The data scientist has to gather training data, asses the data quality. The next step is to choose an algorithm and a framework. Often the data is not formatted correctly for the used model. The data set needs to be improved; typically, data scientists need to deal with outliers and extreme values, missing or inaccurate data. Data for supervised learning needs to be labeled, and the data set needs to be split up into training data sets and evaluation data sets. Now the deep-learning can begin, and the GPU is fed the data. The deep learning framework executes the model. After a single epoch, the data scientist reviews the effectiveness of the model and possibly adjust the model to improve prediction. The model is trained again to verify if the adjustments are correct. Once the model behaves appropriately, it deployed to production where it can run inference tasks that generate predictions based on new data. Each step consuming a lot of time, however only two moments (marked in red) utilize the expensive GPU hardware. Creating the problem, interestingly called “dark silicon”. It doesn’t make sense to keep those resource isolated and assigned to a VM that can only be used by a specific data scientist. By introducing remote virtualization, a GPU can be shared between many different virtual machines. A Bitfusion server can contain multiple GPUs, and many Bitfusion servers can be active on the network. Abstracting the hardware and allow for remote execution of API calls, creates a solution that is easily scaled out. Orchestrating workload placement ensures that a pool of GPUs can be made available to the data scientist when the model is ready for training. During the Tech Field Day, Mazhar Memom (CTO Bitfusion) covered the Bitfusion architecture, showing the use of CUDA libraries by the application to interact with the GPU device. In Bitfusion’s case, it is sending these remote API calls to a server that controls the hardware. But this brings us to the statement made earlier in the article. We have arrived at a time in which the software depends heavily on abstraction layers. In the AI space, it is no different. A deep learning model is going to use a framework that uses a set of libraries that are provided by the hardware vendor. This model allows the application developer to quickly (and correctly) to consume the hardware functionality to drive application performance. The defacto toolkit for the deep learning ecosystem is NVIDIA’s Compute Unified Device Architecture (CUDA). The next article covers the subsequent layers in the DL framework stack in more depth. ================================================================================ Title: Initial Placement of a vSphere Pod URL: https://frankdenneman.ai/2020-03-06-initial-placement-of-a-vsphere-native-pod/ Date: 2020-03-06 Project Pacific transforms vSphere into a unified application platform. This new platform runs both virtual machine and Linux containers as native workload constructs. Just introducing Linux containers as a new workload object is not enough. To manage containers properly, you need a legitimate orchestrator. And on top of that, you need to make sure that existing services, such as DRS, can handle the different lifecycles of these different objects. Containers typically have a shorter lifecycle than virtual machines, where VMs “live” for years, containers have a shorter life expectancy. And this massively different churn impacts initial placement and load-balancing operations of resource management services. Being able to run containers as first-class citizens in the VMkernel generates a couple of fascinating challenges by itself. As Michael Gasch highlighted in our VMworld 2018 session, “Running Kubernetes on vSphere Deep Dive: The Value of Running Kubernetes on vSphere (CNA1553BU)” a container is not a separate entity but a collection of Linux processes and objects. Container Runtime for ESXi The ESXi VMkernel is not a Linux operating system. The VMkernel hardware and process abstractions were built with the intent of servicing virtual machines, not to support Linux processes directly. To do so, project Pacific introduces a container runtime for ESXi (CRX). The CRX provides a Linux Application Binary Interface (ABI) that allows you to execute a Linux application (container) if it was running in the VMkernel directly. The beauty of the CRX is that it is completely isolated from any other process or UserWorld running on the ESXi host. How is that possible? By using our old friend, the VM-construct. The virtual machine aspect of a CRX instance is the use of the virtual machine monitor (VMM) and the configuration of the virtual hardware (VMX). The VMM provides the exception and interrupt handling for the VM. Inside the CRX instance, a CRX init process is active to provide communication between CRX instance and the VMkernel services. But we need to have a Linux kernel to provide a Linux ABI for the container to run. What better Linux kernel than to use our own? VMware Photon was chosen as it’s a VMware supported and maintained LTS Linux kernel. Photon is used as the base for VCSA and other VMware products and has an extremely light footprint. Now the interesting part is that this kernel is not stored onto and loaded from a separate disk. The bare Linux kernel is directly loaded into the memory space of the CRX instance when it is instantiated. Additionally, the CRX instance is stripped down. Only the necessary devices and functionalities are enabled to make the CRX and kernel as lightweight and fast as possible. For example, the CRX only exposes paravirtualized devices to the Photon kernel. On top of this base, a container runtime is active that allows us to spin up OCI compatible containers inside the CRX instance. Inside the VMkernel, we introduced our implementation of the Kubelet, called the Spherelet. In-short, the Spherelet turns the ESXi host into a Kubernetes worker node, and the Spherelet acts as an extension to the Kubernetes control plane. The container runtime inside the CRX instance contains a Spherelet agent that allows the communication between the Spherelet and the container runtime. The Spherelet Agent provides the functionality that Kubernetes expects from a pod. Actions like; health checks, mounting storage, setting up networking, controlling the state of the containers inside the pod, and it provides an interactive endpoint to the Kubernetes command-line tool Kubectl. The Spherelet agent is linked with libcontainer and understands how to launch containers using that method. Once containers are running inside the CRX instance we refer to this group of objects as a native pod. Two Captains on one Ship Now that we understand how ESXi can run containers, we need to think about who controls what from a resource management perspective. Not only does project Pacific introduces a way to run containers natively inside the VMkernel, but it also introduces Kubernetes as an orchestrator of containers. In essence, that means it is adding a control plane to a platform that already has a control plane in place for VM workloads (vCenter+HostD) plus additional services such as DRS to simplify resource management. At first sight, this should not be difficult as Kubernetes does not manage and control VMs, so there is no overlap. However, you must have noticed that there is a duality at play. A vSphere pod is a combination of a VM and a group of containers. And to make it even more interesting, both control platforms have similar constructs to control behavior and placement. In Kubernetes, you control the placement of containers on worker nodes by using labels and deployment policies. In DRS, you use affinity rules. In Kubernetes, you use requests and limits to specify resource entitlement, while vSphere uses vSphere Reservation, Shares, and Limits (RLS) settings. Also, we have to think about how individual requests and limits of multiple containers running inside a single CRX instance will translate to the RLS settings of the VM. I will address the resource entitlement considerations in an upcoming article, what I want to explore in this article is the placement of containers and VMs when these two control planes are active. Initial Placement of a CRX Instance When a developer deploys an application, he or she interacts with the Kubernetes API server, and the API server will trigger all sorts of events to various components that are present in the Kubernetes architecture. Project pacific extended the API of Kubernetes and introduced multiple controllers to interact with the vSphere platform. Therefore it seems Kubernetes is in charge. However, we cannot ignore the pure brilliance of the resource management capabilities of DRS and HostD. Whereas Kubernetes uses a very brusque and coarse method of simpy using request (the equivalent of reservations) to mix and match resource consumers (containers) and resource providers (worker nodes). vSphere is far more elegant with its ability to understand resource activity, its ability to translate idleness into a temporary priority adjustment, and the alignment of resource entitlement beyond a single host. And to make it even better, Project Pacific is using the scalable shares functionality that allows for instant readjustment of priority of resource pools if new workload (i.e., containers) are added to the vSphere namespace. An invention Duncan Epping and I so proudly created with the DRS engineering team back in 2013. Yet the Kubernetes architecture has a very elegant way to express business logic, to easily dictate the placement of containers based on labels, taints, and tolerations. Therefore, it makes sense to integrate or create a mesh of functionality of both control planes. Initial Placement Order The developer engages with the Kubernetes API server to deploy an application. Typically these deployments are submitted to the API server with the use of a YAML file that contains pod specifications. The deployment and pod specification is stored in the etcd server and the API server publishes this event to a watch-list, to which the kube-scheduler is subscribed to. Read this article for more information about event-based architectures. The kube-scheduler initiates the selection process of an adequate host. It filters the available worker node (ESXi hosts) list based on affinity, pod and node labels and other nodeSelector constraints. It sends the curated list to DRS to pick a node. DRS selects the node based on its decision tree (VM resource entitlement, host state, host compatibility). Once the host is selected, the information is returned via the vCenter API server to the Pacific Scheduler. The scheduler has it stored as an event in the etcd database. While the event is stored in the etcd database, vCenter issues a command to the HostD process on the selected ESXi host to power-on the virtual machine. HostD powers on the VM (VMX, VMM) and loads the Photon kernel into the memory address space of this virtual machine. HostD returns the VM ID of the newly created VM to the Pacific Scheduler Extension. The VM ID is stored in the etcd database and now the control plane node has enough data for the Spherelet to configure the pod. The vSphere Pod Lifecycle Controller is updated on the event and issues the vSpherelet to configure the pod. The Spherelet connects with HostD to configure the personality of the pod and configure networking and storage elements. The CRX container runtime initiates the start of the containers based on the pod specification. The Spherelet returns the state of the containers back to the Kubernetes control plane node to have it stored as an event in the etcd database. With this architecture, you have the best of both worlds, the expressiveness of the Kubernetes control plane while enjoying the elegancy of vSphere resource management capabilities. The next article on this topic dives into resource allocation based on container resource configuration settings. Please be aware that project Pacific is still in beta phase and is not (yet) available as a finalized product. Stay tuned for more. ================================================================================ Title: Multi-GPU and Distributed Deep Learning URL: https://frankdenneman.ai/2020-02-19-multi-gpu-and-distributed-deep-learning/ Date: 2020-02-19 More enterprises are incorporating machine learning (ML) into their operations, products, and services. Similar to other workloads, a hybrid-cloud model strategy is used for ML development and deployment. A common strategy is using the excellent toolset and training data offered by public cloud ML services for generic ML capabilities. These ML activities typically improve an organization’s quality of service and increase in productivity. But the real differentiation lies within using the organization’s unique data and know-how to create what’s called differentiated machine learning. The data used is primarily generated by own processes or through interaction with its customers. As a result, specific rules and regulations come into play when handling and storing that data. Another strong aspect of determining where to deploy ML activities is data gravity. Placing compute close to where the data is generated provides a consistent (often high-performing) service. As a result, many organizations invest in the infrastructure needed to deploy ML and deep learning (DL) solutions. Deep Learning Deep learning is a subset of the more extensive collection of machine learning techniques. The critical difference between ML and DL is the way the data is presented to the solution. ML uses mathematical techniques and data to build predictive models. It uses labeled (structured) data to train the model, and once the model is trained accurately enough, the model keeps on learning by feeding new data. Deep learning does not necessarily need structured or labeled data to create an accurate model to provide a predictive answer. It uses larger neural networks (layers of algorithms, imitating the brain’s neural network), and it needs to be fed vast amounts of data to provide an accurate prediction. Interestingly, at one point, ML experiences a performance plateau regardless of the amount of incoming new data, while deep learning keeps on improving. For more information, about this phenomenon review the notes from Andrew Ng Coursera Deep Learning course or watch his 5-minute clip on youtube: How Scale is Enabling Deep Learning. In essence, the magic of deep-learning is that it gets better with data, and thus, how do we create an infrastructure that is capable of feeding, transporting, and processing these vast amounts of data, while still being able to run non-ML/DL workload? Parallelism The best way of dealing with massive amounts of data is to process it in a parallel way. And that’s where general-purpose computing on GPU (GPGPU) comes into play. A simple TensorFlow test compared the performance between a dual AMD Opteron 6168 (2x12 cores) vs. a system with a (consumer-grade NVIDIA Geforce 1070. The AMD system recorded 440 examples per second, while the Geforce processed 6500 examples per second. There are many performance tests available, but this showed the power of a consumer-grade GPU versus a data center grade CPU system. Today data center focused GPUs have more than 5000 cores all optimized to operate in parallel. These cores have access to 32 GB of high bandwidth memory (HBM2) with speeds up to 900 GB/s (theoretical bandwidth). According to the paper “Analysis of Relationship between SIMD-Processing Features Used in NVIDIA GPUs and NEC SX-Aurora TSUBASA Vector Processors” by Ilya.V. Afanasyev et al. the achievable bandwidth on the tested NVIDIA Volta V100 was 809 GB/s. Getting all the data loaded in memory with consistent performance is one element that impacts virtual machine design. See “Machine Learning Workload and GPGPU NUMA Node Locality” for more information. Although the improvement of processing speed is enormous, up to 10x over a CPU according to this performance study, sometimes this speed-up is not enough. After processing all the training examples in a dataset (called an epoch), a data scientist might make some adjustments as well and start another epoch to improve the prediction model. It’s common to run multiple epochs before getting an adequate trained model (and in the process pushing lots of data through the system). Reducing training time, allows the organization to deploy the trained model faster, and start benefiting from their ML and DL initiatives. A “simple” way to reduce training time is to use multiple GPU devices to increase parallelism. Distributed Deep Learning Strategies How do you scale out your training model across the multiple GPUs in your system? You add another layer of parallelism on top of GPUs. Parallelism is a common strategy is distributed deep learning. There are two popular methods of parallelizing DL models: model parallelism and data parallelism. Model parallelism With model parallelism, a single model (Neural Network A) is split and distributed across different GPUs (GPU0 and GPU1). The same (full) training data will be processed by the different GPUs depending on which layer is active. Models with a very large number of parameters, that are too big to fit inside a single device’s memory, benefit from this type of strategy. Neural networks have data dependency. The output of the previous layer is the input of the next layer. Asynchronous processing of data can be used to reduce training time, however, model parallelism is more about having the ability to run large models. Maybe model sequentiality would be a better name for this mode as it primarily is using devices in sequential order. More than often a device is idling, waiting to receive the data from another device. Once the model part is trained on one device, it has to synchronize the outcome with the next layer possibly handled by another device. This synchronization is interesting when designing your ML platform as specific data to help run the model has to traverse the interconnect either between devices within the ESXi system or between VMs (or containers) running on the platform. More about this in a later paragraph. Data parallelism Data parallelism is the most common strategy deployed. As covered in the previous article: “Machine Learning Workload and GPGPU NUMA node locality” it is common to split up the entire training dataset into batches (batch 0 and batch1). With data parallelism, these batches are sent to the multiple GPUs (GPU 0 and GPU1). Each GPU will load a full replica of the model (Neural Network A) and run their batch of training examples through the model. The models running on the GPUs must communicate with each other to share the results. Communication timing and patterns between the GPUs depend on the DL model ( Convolutional Neural Networks (CNN) or Recurrent Neural Networks (RNN)) and on the framework used (TensorFlow, Pytorch, MXNet). Currently, there are a few projects active that are exploring the possibility of hybrid parallelization. This strategy uses both model and data parallelization strategies to minimize end-to-end training time. Parallelism introduces communication between GPUs. Understanding the data-flow is essential to build a system that can provide consistent high-performance while ensuring the DL workloads are isolated enough and do not impact other workloads that are using the system. Various distributions of GPU resources are possible, such as a cluster of single GPU systems or multi-GPUs hosts. The next article focusses only on a single node with a multi-GPU configuration, to highlight the different in-system (on-node) interconnects On-Node Interconnect vSphere allows for different multi-GPU configurations. A VM can be equipped with multiple GPU configured as a passthrough device, or configured with vGPUs with the help of NVIDIA drivers, or by using a Bitfusion solution. Details about the different solutions will be covered in a future article. But regardless of the chosen configuration, the application will be able to use multiple GPUs in a single VM. When deploying deep learning models across multiple GPUs in a single VM, the ESXi host PCIe bus becomes an inter-GPU network that is used for loading the data from system memory into the device memory. Once the model is active, the PCIe bus is used for GPU to GPU communication for synchronization between models or communication between layers. If two PCIe devices communicate with each other, then the CPU is involved. Data coming from the source device is stored in system memory before transferring it to the destination device. The new Skylake architecture with it’s updated IIO structure, and additional mesh stops improved the CPU to PCIe communication over the previous ring-based architecture featured on the Xeon v1 through v4. (Each mesh stop has a dedicated cache and traffic controller). CPU to GPU to CPU communication within a single NUMA node (Skylake Architecture) For this purpose, NVIDIA introduced GPUDirect in CUDA 4.0, allowing direct memory access between two devices. However, this requires a full topology view of the system, and this is something currently vSphere is not exposing. As such, no direct PCI to PCI communication is available (yet). Discovering this seems like this lack of topology view is an enormous bottleneck, but this doesn’t necessarily mean an application performance slowdown. Modern frameworks optimize their GPU code to minimize communication. As a result, communication between devices is just a portion of total time. Depending on the framework used and the parallelism strategy, the performance can still be close to the bare-metal performance. NVIDIA NVLink In 2016, NVIDIA introduced the NVLink interconnect, a high-speed mesh network that allows GPUs to communicate directly with each other. NVLink is designed to replace the inter GPU-GPU communication across the PCIe lanes, and as a result, NVLINK uses a separate interconnect. A new custom form factor SXM2 (supported by vSphere) allows the GPU to interface with the NVIDIA High-Speed Signalling interconnect (NVHS). The NVHS allows the GPU to communicate with the other GPUs as well as direct system memory access. Currently, NVLink 2.0 (available on NVIDIA Tesla v100 GPUs) provides an aggregate maximum theoretical bidirectional bandwidth of 300 GBps. (AMD does not have any equivalent to NVlink) NVIDIA V100 SXM2 Design Decisions Data movement within an ML system (VM) can be substantial. Fetching the data from storage, storing it into system memory before dispatching it to multiple vGPUs can produce a significant load on the platform. Depending on the neural network, framework, and parallelism strategy, communication between GPUs can add additional load to the system. It’s key to understand this behavior before considering retrofitting your current platform with GPU devices or while designing your new vSphere clusters. Depending on the purpose of the platform it might be interesting to research the value of having a separate interconnect mesh for ML/DL workload. It allows for incredible isolation that will enable you to run other workloads on the ESXi host as well. Couple this the ability to share multiple GPUs with the Bitfusion solution, and you can create a platform that provides consistent high-performance for ML workload to numerous data scientists. ================================================================================ Title: Machine Learning Workload and GPGPU NUMA Node Locality URL: https://frankdenneman.ai/2020-01-30-machine-learning-workload-and-gpgpu-numa-node-locality/ Date: 2020-01-30 In the previous article “PCIe Device NUMA Node Locality” I covered the physical connection between the processor and the PCIe device briefly touched upon machine learning workloads with regards to PCIe NUMA locality. This article zooms in on why it is important to consider PCIe NUMA locality. General-Purpose Computing on Graphics Processing Units New compute-intensive workloads take advantage of the new programming model called general-purpose computing on GPU (GPGPU). With GPGPU, the many cores integrated on modern GPUs are used to offload a vast number of (parallel) compute threads from the CPU. By adding another computational device with different characteristics, a heterogeneous compute architecture is born. GPUs are optimized for streaming sequential (or easily predictable) access patterns, while CPUs are designed for general access patterns and concurrency of threads. Combined, they form a GPGPU pipeline, that is exceptionally well-suited to analyze data. The vSphere platform is well-suited to create GPGPU pipelines and optimizations are provided to VMs, such as DirectPath I/O Access (also known as Passthrough). Passthrough allows the application to interface with the accelerator device directly; however, data must be transferred from disk/network through the system (RAM) to the GPU. And controlling the data transfer is of interest to the overall performance of the platform for both GPGPU workload and non-GPGPU workload. A very popular GPGPU workload is Machine Learning (ML). Many ML workloads process gigabytes of data, sometimes even terabytes, this data flows from the storage device up to the PCIe device. Finetuning the configuration and placement of the virtual machine running the ML workload can benefit the data-scientist and other consumers of the platform. Not every ML workload is latency-sensitive, but most data scientists prefer to get the training done as quickly as possible. This allows them to perform more training iterations to fine-tune the model (also known as the neural network). Due to the movement of data through the system, a ML workload can quickly become the noisiest neighbor you ever saw in your system. But with the right guard-rails in place, data-scientists take advantage of running their workload on a consistent performing platform, while the rest of the organization can consume resources from this platform as well. Machine Learning Concepts Oversimplified ML is “using data to answer questions.” With traditional programming models, you create “rules” by using the programming language and apply these rules to the input to get output (results) (output). With ML training, you provide input and the output to train the program to create rules. This creates a predictive model that can be used to analyze previously unseen data to provide accurate answers. The key component of the entire ML process is data. This data is stored on a storage device and fetched to be used as input for the model to be trained on, or to use the trained model to provide results. Training a machine learning model is primarily done by a neural network of nodes that are executed by thousands of cores on GPUs. The nature of the cores (SIMT - Single Instruction, Multiple Data) allows for extremely fast parallel processing, ideal for this sort of workload, hence you want to use GPUs for this task and not the serial-workload optimized CPUs. The heavy lifting of the compute part is done by the GPU, but the challenge is getting the data to the costly GPU cores as fast and consistent as possible. If you do not keep the GPU cores fed with all the data it needs, then a large part of the GPU cores sit idle until new data shows up. And this is the challenge to overcome, handling large quantities of training data that flows from storage, through the host memory, into the VM memory before flowing into the memory of the GPU. High-speed storage systems with fast caching and fast paths between the storage, CPU, server memory and PCIe device are necessary. Anatomy of an ML Training Workload The collection of training examples is called a dataset, and the golden rule is, the more data you can use during the training, the better the predictive model becomes. That means that the data scientist will unleash copious amounts of data on the system, data so large that it cannot fit inside the memory device of the GPU. Perhaps not even the memory assigned to the virtual machine, as a result, the data is stored on disk and is retrieved in batches. The data scientist typically finetunes the size of the batch set, finetuning a batch set size is considered an art form in the world of ML. You, the virtual admin, slowly graduating into an ML infrastructure engineer (managing and help to design the ML platform), can help inform the data scientist by sizing the virtual machine correctly. Look at CPU consumption and determine the correct number of vCPUs necessary to push the workload. Once the GPU receives a batch, the workload is contained within the GPU. Rightsizing the VM can help to improve performance further as the VM might fit a single NUMA node. To understand the dataflow of an ML workload through the system, let’s get familiar with some neural network terminology. Most of the ML workload use the Compute Unified Device Architecture (CUDA) for GPU programming, and when using a batch of the training data, the CUDA program takes the following steps: 1: Allocate space on the GPU device memory 2: Copy (batch set) input data to the device (aka Host to Device (HtoD)) 3: Run the algorithm on the GPU cores 4: Copy output (results) back to host memory (aka Device to Host (DtoH)) During training, the program processes all the training examples in the dataset. This cycle is called an epoch. As mentioned before, a data scientist can decide to split up the entire dataset into smaller batch sets. The number of training examples used is called a batch size. An iteration is the number of passes the program needs to use to go through the entire dataset to complete a single epoch. For example, a dataset contains 100.000 samples, and each batch size contains 1000 training examples, then it takes 100 iterations to complete a single epoch. Each iteration uses the previously described CUDA loop. To get a better result, multiple epochs are pushed to get a better convergence of the training model. Within each epoch, the neural network self-tweaks its own parameters (called weights and is done for each node) in the neural network, this finetuning provides a more accurate prediction result when it’s used during the inference operation. The interesting part is that the data scientist can also make some adjustments to the (hyper)parameters of the ML model. Simply put, a hyperparameter is a parameter whose value is set before the training process begins. Such as the number of weights or the batch size set. To verify if this tuning was helpful, a new sequence of epochs is kicked off. A great series of videos about neural networks can be found here. Josh Simons and Justin Murray gave a 4-hour workshop on ML workload on vSphere at VMworld last year. In this workshop, they stated that the typical values they saw were gigabytes of data (D), 10 to 100s of epochs (E), and 10 or more tuning cycles (T), which can be substantially more (in the 1000s) when researching new models. You can imagine that such data volumes can become a challenge in a shared system such as the hypervisor. Let’s take a look at why isolation can benefit both ML workload and the other resident workload on the system. CPU Scheduler and NUMA optimizations When the data is fetched from the storage device it is stored in memory. The compute schedulers of the VMkernel are optimized to store the memory as close to the CPUs as possible. Let’s use the most popular server configuration in today’s data center, the dual-socket system. Each socket contains a processor and within the processor, memory controllers exist. Memory modules (DIMMs) attached to these memory controllers are considered local memory capacity. Both processors are connected to each other to allow each processor to access the memory connected to the other processor. Due to the difference in latency and bandwidth, this is considered to be non-uniform memory access (NUMA). For more information about NUMA, check out this series. Let use the example of a 4 vCPU VM with 32 GB, running on a host with 512GB memory with 2 processors containing 10 cores each. The dataset used is 160GB of data and it cannot be stored in the VM memory and in the GPU device memory, thus the data scientist sets the batch size to 16GB. The program fetches 16GB of training data from the datastore and the NUMA scheduler ensures the data is stored within the local memory of the processor the four vCPU run on. In this case, the vCPUs of the VM are scheduled on the cores of CPU 1 (NUMA node 1) and thus the NUMA scheduler requests the VMkernel memory scheduler to store it in the memory pages belonging to the memory address space managed by the memory controllers of CPU 1. The VM is configured with a passthrough GPU and the training data is pushed to the GPU. The problem is that the GPU is manually selected by the admin and no direct relation is visible in the UI or command-line, it just shows the type name and a PCI address. GPUs are PCIe devices and they are hardwired and controlled by a CPU. The admin selected the first GPU in the list and now the dataset is pushed directly from the VM memory to the GPU Device memory to be used by the cores of the GPU. Data now flows through the interconnect to the PCIe controller of CPU 0 and to the GPU device. Each dataset that is retrieved from storage, is stored in NUMA node 1 and then moved through the interconnect to the device, this is done for each iteration, for each epoch and this can be done 1000’s of time. The problem is that the interconnect is used by the entire system. When the CPU needs to rebalance, it can reschedule the vCPU on cores belonging to a different CPU if this improves the overall resource availability for the active virtual machines. Memory can be transferred over to the new NUMA home node of that recently migrated virtual machine, or memory is just accessed across the interconnect. Same with Wide-VMs, VMs that span multiple NUMA nodes, it can happen that these Wide-VMs access a lot of “remote” memory. Also do not forget the data being handled by other PCIe devices. All network traffic has to flow from the NIC to a particular VM, for optimized performance, the kernel prefers to store that data in memory that is local to the vCPUs of that VM. The same goes for data coming from external storage devices, if the HBA or NIC is “hanging” off the other CPU, data has to flow through the interconnect. The interconnect is a highway shared by a lot of components and workloads. These operations can impact the performance of the ML workload but the opposite is also true, pushing 1000 epochs of gigabytes of data to a GPU ensures other workloads will notice the presence of that workload, even if it has a small CPU and memory footprint. Remember, ML is “using data to answer questions.” PTNumaTopology PowerCLI Module To make sense of it all, I created a simple PowerCLI module with two functions that show the VMs that have a passthrough device configured. The output shows the VM name and the PCI address of the device so that you can relate that to what you see in the UI. The next column shows the NUMA node to which the PCIe device is connected. The next column indicates whether the advanced setting numa.affinity is set for that particular VM and its value. The last column shows the power state of the VM. To set the NUMA affinity, the VM has to be powered off. To run the script, import the module (available at GitHub) and execute the Get-PTNumaTopology command. Specify the FQDN of the ESXi host. For example: Get-PTNumaTopology -esxhost sc2esx27.vslab.local. As the script needs to execute a command on the ESXi host locally an SSH session is initiated. This results in a prompt for a (root) username and password in a separate login screen. (The Github page has a thorough walk-through of all the steps involved and a list of requirements.) NUMA Affinity Advanced Setting In most situations, it is not recommended to set any affinity setting as it simply restricts the scheduler to generate an optimal balance between resource providers (CPUs) and consumers (vCPUs). At the host level and cluster level. However since the VM is configured with a passthrough (PT) GPU, it cannot move to another host and chances are a lot of data will flow to this device. Another assumption is that the host contains a small number of GPUs and thus a small number of VMs are active. If no other restrictions are configured, the CPU and NUMA scheduler can try to work “around” the affined VM and attempt to optimize the placement and resource consumptions of the other active VMs. Hopefully, the isolation of these particular passthrough-enabled VMs are reducing overall system load and thus evening out the possible enforced restrictions. Testing this first before using it on the production workload is always recommended! For more information about the NUMA affinity setting, please consult the VMware Docs for your specific vSphere version, linked is the VMware Docs page for vSphere 6.7. Why set numa.affinity and not use CPU pinning? First of all, CPU pinning is something that should not be done ever. And even when you think you have a valid use case, chances are that CPU pinning will still reduce performance significantly. This topic is rearing its ugly head again and I will soon post another article why CPU pinning is just a bad idea. NUMA affinity creates a rule for the CPU scheduler to find a CPU core or HT within the boundaries of the CPU itself. In the example of the 4 vCPU running on the 10 core CPU. Let’s say hyperthreading is enabled, it allows the CPU scheduler to schedule one of these four CPUs on the 20 available logical processors. If the system is not over-utilized, it can use a complete core for a vCPU, it can find the optimal placement for that workload and for the others using the same CPU. With pinning you restrict the vCPUs to only run on that particular logical processor. If chosen incorrectly you might have just selected HTs only. If you decide to set a NUMA affinity on a particular VM, the Get-PTNumaTopology function can help you to set it correctly. As a failsafe, the script proceeds to ask if you would like to set the NUMA node affinity of a powered off VM. Answer “N” to end the script and return to the command-line. If you answer “Y” for yes, it will then ask you to provide the name of the VM. Please note that this setting can only be applied on a powered-off VM. Setting an advanced setting means that the system is writing to this to the VMX file and the VMX file is in a locked state during the power-on state of a VM. The next step is to provide the NUMA Node you want the vCPUs to set the affinity for. Use the same number listed in the PCI NUMA Node column behind the attached passthrough device. Once the advanced setting is configurated it shows the configured value. To verify whether the setting matches the NUMA node of the passthrough device, run the command Get-PTNumaTopology again. As it has closed the SSH connection after the last run, you are required to log in again with the root user account to retrieve the current settings. Setting the NUMA node advanced option for a VM is something that should be done for specific reasons, do not use the script for all your virtual machines. The NUMA affinity setting applies to the placement of vCPU only. The NUMA scheduler provides recommendations to the memory scheduler, but it is up to the memory scheduler discretion to store the data. The kernel is optimized to keep the memory close to the vCPUs as possible, but sometimes it cannot fit that memory into that node. Either because the VM configuration exceeds the total capacity of that node, or that other active VMs are already using large amounts of memory of that node. Setting the affinity is not a 100% guarantee that all the resources are local, but in the majority of use-cases, it will. Isolating the workload within a specific NUMA node will help to provide you consistent performance and will reduce a lot of interconnect bandwidth consumption. Enjoy using the script! Font used in PowerShell environment: JetBrains Mono - available at - https://www.jetbrains.com/lp/mono/#intro ================================================================================ Title: PCIe Device NUMA Node Locality URL: https://frankdenneman.ai/2020-01-10-pcie-device-numa-node-locality/ Date: 2020-01-10 During this Christmas break, I wanted to learn PowerCLI properly. As I’m researching the use-cases of new hardware types and workloads in the data center, I managed to produce a script to identify the PCIe Device to NUMA Node Locality within a VMware ESXi Host. The script set contains a script for the most popular PCIe Device types for data centers that can be assigned as a passthrough device. The current script set is available on Github and contains scripts for GPUs, NICs and (Intel) FPGAs. PCIe Devices Becoming the Primary Units of Data Processing Due to the character of new workloads, the PCIe device is quickly moving up from “just” being a peripheral device to become the primary unit for data processing. Two great examples of this development are the rise of General Purpose GPU (GPGPU), often referred to as GPU Compute, and the virtualization of the telecommunication space. The concept of GPU computing implies using GPUs and CPUs together. In many new workloads, the processes of an application are executed on a few CPU cores, while the GPU, with its many cores, handles the computational intensive data-processing part. Another workload, or better said, a whole industry that leans heavily on the performance of PCIe devices, is the telecommunication industry. Virtual Network Functions (VNF) require platforms using SR-IOV capable NICs or SmartNICs to provide ultra-fast packet processing performance. In both scenarios having insight into PCIe Device to processor locality is a must to provide the best performance to the application or avoid introducing menacing noisy neighbors that can influence the performance of other workloads active in the system. PCIe Device NUMA Node Locality The majority of servers used in VMware virtualized environments are two CPU socket systems. Each CPU socket accommodates a processor containing several CPU cores. A processor contains multiple memory controllers offering a connection to directly connected memory. An interconnect (Intel: QuickPath Interconnect (QPI) & UltraPath Interconnect (UPI), AMD: Infinity Fabric (IF)) connects the two processors and allows the cores within each processor to access the memory connected to the other processor. When accessing memory connected directly to the processor, it is called local memory access. When accessing memory connected to the other processor, it is called remote memory access. This architecture provides Non-Uniform Memory Access (NUMA) as access latency, and bandwidth differs between local memory access or remote memory access. Henceforth these systems are referred to as NUMA systems. It was big news when the AMD Opteron and Intel Nehalem Processor integrated the memory controller within the processor. But what about PCIe devices in such a system? Since the Sandy Bridge Architecture (2009), Intel reorganized the functions critical to the core and grouped them in the Uncore, which is a “construct” that is integrated into the processor as well. And it is this Uncore that handles the PCIe bus functions. It provides access to NVMe devices, GPUs, and NICs. Below is a schematic overview of a 28 core Intel Sky lake processor showing the PCIe ports and their own PCIe root stack. Intel Skylake Mesh Architecture In essence, a PCIe device is hardwired to a particular port on a processor. And that means that we can introduce another concept to NUMA locality, which is PCIe locality. Considering PCIe locality when scheduling low-latency or GPU compute workload can be beneficial not only to the performance of the application itself but also to the other workloads active on the system. NUMA Locality Venn Diagram For example, Machine Learning involves processing a lot of data, and this data flows within the system from the CPU and memory subsystem to the GPU to be processed. Properly written Machine Learning application routines minimize communication between the GPU and CPU once the dataset is loaded on the GPU, but getting the data onto the GPU typically turns the application into a noisy neighbor to the rest of the system. Imagine if the GPU card is connected to NUMA node 0, and the application is running on cores located in NUMA node 1. All that data has to go through the interconnect to the GPU card. The interconnect provides more theoretical bandwidth than a single PCIe 3.0 device can operate at, ~40 GB/s vs. 15 GB/s. But we have to understand that interconnect is used for all PCIe connectivity and memory transfers by the CPU scheduler. If you want to explore this topic more, I recommend reviewing Amdahl’s Law - Validity of the single processor approach to achieving large scale computing capabilities - published in 1967. (Still very relevant) And the strongly related Little’s Law. Keeping the application processes and data-processing software components on the same NUMA node keeps the workloads from flooding the QPI/UPI/ AMD IF interconnect. For VNF workloads, it is essential to avoid any latency introduced by the system. Concepts like VT-d (Virtualization Technology for Directed I/O) reduces the time spent in a system for IOs and isolate the path so that no other workload can affect its operation. Ensuring the vCPU operates within the same NUMA domain ensures that no additional penalties are introduced by traffic on the interconnect and ensures the shortest path is provided from the CPU to the PCIe device. Constraining CPU Placement The PCIe Device NUMA Node Locality script assists in obtaining the best possible performance by identifying the PCIe locality of GPU, NIC of FPGA PCIe devices within VMware ESXi hosts. Typically VMs running NFV or GPGPU workloads are configured with a PCI passthrough enabled device. As a result, these VMware PowerCLI scripts inform the user which VMs are attached directly to the particular PCIe devices. Currently, the VMkernel schedulers do no provide any automatic placement based on PCIe locality. CPU placement can be controlled by associating the listed virtual machines with a specific NUMA node using an advanced setting. Please note that applying this setting can interfere with the ability of the ESXi NUMA scheduler to rebalance virtual machines across NUMA nodes for fairness. Specify NUMA node affinity only after you consider the rebalancing issues. The Script Set The purpose of these scripts is to identify the PCIe Device to NUMA Node locality within a VMware ESXi Host. The script set contains a script for the most popular PCIe Device types for Datacenters that can be assigned as a passthrough device. The current script set contains scripts for GPUs, NICs, and (Intel) FPGAs. Please note that these scripts only collect information and do not alter any configuration in any way possible. Requirements VMware PowerCLI Connection to VMware vCenter Unrestricted Script Execution Policy Posh-SSH Root Access to ESXi hosts Please note that Posh-SSH only works on Windows version of PowerShell. The VMware PowerCLI script primarily interfaces with the virtual infrastructure via a connection to the VMware vCenter Server. A connection (Connect-VIServer) with the proper level of certificates must be in place before executing these scripts. The script does not initiate any connect session itself. It assumes this is already in-place. As the script extracts information from the VMkernel Sys Info Shell (VSI Shell) the script uses Posh-SSH to log into ESXi host of choice and extracts the data from the VSI Shell for further processing. The Posh-SSH module needs to be installed before running the PCIe-NUMA-Locality scripts, the script does not install Posh-SSH itself. This module can be installed by running the following command Install-Module -Name Posh-SSH (Admin rights required). More information can be found at https://github.com/darkoperator/Posh-SSH Root access is required to execute a vanish command via the SSH session. It might be possible to use SUDO, but this has functionality has not been included in the script (yet). The script uses Posh-SSH keyboard-interactive authentication method and presents a screen that allows you to enter your root credentials securely. Script Content Each script consists of three stages, Host selection & logon, data collection, and data modeling. The script uses the module Posh-SSH to create an SSH connection and runs a vsish command directly on the node itself. Due to this behavior, the script creates an output per server and cannot invoke at the cluster level. Host Selection & Logon The script requires you to enter the FQDN of the ESXi Host, and since you are already providing input via the keyboard, the script initiates the SSH session to the host, requiring you to login with the root user account of the host. When using the GPU script, the input of the GPU vendor name is requested. The input can be, for example, NVIDIA, AMD, Intel, or any other vendor providing supported GPU devices. This input is not case-sensitive. Data Collection The script initiates an esxcli command that collects the PCIe address of the chosen PCIe device type. It stores the PCIe addresses in a simple array. Data Modeling The NUMA node information of the PCIe device is available in the VSI Shell. However, it is listed under the decimal value of the Bus ID of the PCIe address of the device. The part that follows is a collection of instructions converting the full address space into a double-digit decimal value. Once this address is available, it’s inserted in a VSISH command and execute on the ESXi host via the already opened SSH connection. The NUMA node, plus some other information, is returned by the host, and this data is trimmed to get the core value and store it in a PSobject. Throughout all the steps of the data modeling phase, each output of the used filter functions is stored in a PSObject. This object can be retrieved to verify if the translation process was executed correctly. Call $bdfOutput to retrieve the most recent conversion. (as the data of each GPU flows serially through the function pipeline, only the last device conversion can be retrieved by calling $bdfOutput. The next step is to identify if any virtual machines registered on the selected host are configured with PCIe passthrough devices corresponding with the discovered PCIe addresses. Output A selection of data points is generated as output by the script: PCIe Device Output Values GPU PCI ID, NUMA Node, Passthrough Attached VMs NIC VMNIC name, PCI ID, NUMA Node, Passthrough Attached VMs FPGA PCI ID, NUMA Node, Passthrough Attached VMs The reason why the PCI ID address is displayed is that when you create a VM, the vCenter UI displays the (unique) PCI-ID first to identify the correct card. An FPGA and GPU do not have a VMkernel label, such as the VMNIC label of a network card. No additional information about the VMs is provided, such as CPU scheduling locations or vNUMA topology, as these are expensive calls to make and can change every CPU Quorum (50 ms). It’s recommended to review the CPU topology of the virtual machine and if possible to set the NUMA Node affinity following the instructions listed in VMware Resource Management Guide. Please note that using this advanced setting can impact the ability of the CPU and NUMA schedulers to achieve an optimal balance. Using the Script Set Step 1. Download the script by clicking the “Download” button on the Github repository Step 2. Unlock scripts (Properties .ps1 file, General tab, select Unlock.) Step 3. Open PowerCLI session. Step 4. Connect to VIServer Step 5. Execute script for example, the GPU script: .\PCIE-NUMA-Locality-GPU.ps1 Step 6. Enter ESXi Host Name Step 7. Enter GPU Vendor Name Step 8. Enter Root credentials to establish SSH session Step 8. Consume output and possibly set NUMA Node affinity for VMs Acknowledgments This script set would not have been created without the guidance of @kmruddy and @lucdekens. Thanks, Valentin Bondzio, for verification of NUMA details and Niels Hagoort and the vSphere TM team for making their lab available to me. ================================================================================ Title: vSphere 6.5+ DRS Pairwise Balancing URL: https://frankdenneman.ai/2019-10-30-vsphere-6-5-drs-pairwise-balancing/ Date: 2019-10-30 Or maybe I should have called this blog post, “I’m seeing an excessive number of DRS initiated vMotions on my newly upgraded 6.5 environment”. Recently I was part of a few conversations about the nature of DRS load balancing in systems running vSphere 6.5 and newer. It was noticed that more vMotion operations where occurring since running 6.5 and it’s highly likely that these operations occur due to the new DRS pairwise balancing functionality. Pairwise balancing was introduced by vSphere 6.5 and is focused on keeping the host resource utilization disparity within a certain threshold. As a result, DRS performs load-balancing operations if the difference between the lowest-utilized host and the highest-utilized host is a certain percentage. That percentage depends on your migration threshold. The default migration threshold uses a 20% tolerable difference in utilization. Migration Threshold Level Tolerable CPU/Memory usage difference between any two hosts in the cluster 1 Not Available (only Affinity violations and MM migrations allowed) 2 30% 3 (Default migration threshold) 20% 4 10% 5 5% This new feature is needed as clusters keep on growing larger and larger. To determine if load-balancing operations are necessary, DRS calculates two metrics, the current host load standard deviation (CHLSTD) and the target host load standard deviation (THLSTD). Each host reports its load and DRS calculates the standard deviation of the host load metric across all the hosts in the cluster. DRS calculates a target host load balance for the cluster and as long as the current host load standard deviation is less than or equal to the target host load value, DRS will consider the cluster balanced. The migration threshold allows how far apart the CHLSTD and THLSTD before it triggers load balancing operations. The higher the aggressiveness of the migration threshold, the lower the difference between the CHLSTD and THLSTD is tolerated. A situation can occur that a few hosts in a large cluster can experience a high resource utilization, while the majority of hosts are not. Due to the size of the cluster, the few high host load become just some statistical outliers than simply disappear as noise due to the vast number of hosts that experience (far) lower utilization. As a result, these outliers are missed as the calculate CHLSTD is below the threshold required to trigger load balancing. By adding the functionality of pairwise balancing, and “simply” comparing the highest reported utilization with the lowest utilization, these outliers might be a thing of the past. That means that in certain cases, the DRS UI might report that the cluster is in a balanced state, yet load-balance operations still occur. This behavior can be attributed to pairwise balancing. Please keep in mind that if you are using a migration threshold that is more aggressive than the default setting, the tolerable difference between hosts is reduced, more migrations are likely to occur. So what happens when the tolerable difference is detected in the cluster? Does this mean that VMs are migrated from the highest utilized host to the lowest utilized host? Not necessarily. VMs can be migrated to any other host in the cluster. DRS still takes many different requirements into account when selecting a virtual machine migration for load-balancing purposes. Anti-affinity and affinity rules cannot be violated to obtain a better cluster load-balance, so these moves are not considered. Compatibility of hosts and VM configuration also impact migration options (typically a missing datastore or network portgroup are common reasons why particular hosts are overloaded and why other hosts are lower utilized), but also the “cost-benefit” of a VM migration is still taken into account. It still needs to make sense for the cluster balance to incur infrastructure costs and risk to move a particular VM. If you recently updated your vCenter to 6.5/6.7 and are curious to see whether the vMotions are triggered due to Pairwise imbalance operations, you can use the online version of the DRS Dump Insight tool available at https://www.drsdumpinsight.vmware.com/. You can also run the DRS dump insight tool on-prem by installing one of the flings available here: https://flings.vmware.com/?utf8=%E2%9C%93&q=DRS+Dump+Insight&button=. Grep for “Pairwise Imbalance”. If this behavior is not appreciated, and you do not want to alter the migration threshold, you can switch back to the old behavior by turning off pair-wise balancing by setting the cluster advanced option “CheckPairWiseImbalance to 0. (case-sensitive). Although this functionality was introduced by vSphere 6.5 and is active by default in all newer releases, we have backported this functionality to vSphere 6.0 u3. One thing I would like to ask if you want to disable it, what are the reasons? I expect “too much vMotions”, but I would like to understand why a vMotion or a collection of vMotions is considered not desirable? The main goal is to get the VMs to a place where they have access to enough resources, why is that still a bad thing? ================================================================================ Title: AMD EPYC Naples vs Rome and vSphere CPU Scheduler Updates URL: https://frankdenneman.ai/2019-10-14-amd-epyc-naples-vs-rome-and-vsphere-cpu-scheduler-updates/ Date: 2019-10-14 Recently AMD announced the 2nd generation of the AMD EPYC CPU architecture, the EPYC 7002 series. Most refer to the new CPU architecture using its internal codename Rome. When AMD introduced the 1st generation EPYC (Naples), they succeeded in setting a new record of core count and memory capacity per socket. However, due to the CPU multi-chip-module (MCM) architecture, it is not an apples-to-apples comparison when compared to an Intel Xeon architecture. As each chip module contains a memory controller, each module presents a standalone NUMA domain. This impacts OS scheduling decisions and, thus, virtual machine sizing. A detailed look can be found here in English or here translated by Grigory Pryalukhin in Russian. Rome is different, the new CPU architecture is more aligned with the single NUMA per Socket paradigm, and this helps with obtaining workload performance consistency. There are some differences between Xeons and Rome. In addition, we made some adjustments to the CPU scheduler to deal with this new architecture. Let’s take a closer look at the difference between Naples and Rome. 7 nanometer (7 nm) lithography process forcing a new architecture Rome is using the new 7nm Zen 2 microarchitecture. A smaller lithography process (7nm vs. 14nm) allows CPU manufacturers to cram more CPU cores in a CPU package. However, there are more elements on a CPU chip than CPU cores alone, such as I/O and memory controllers. The scalability of I/O interfaces is limited, and therefore, AMD decided to use a separated and more massive 14nm die that contains the memory and I/O controllers. This die is typically revered to as the server I/O Die (sIOD). In the picture below, you see a side by side comparison of an unlidded Naples (left) and an unlidded Rome, exposing the core chiplet dies and the SIOD. AMD EPYC Naples vs. EPYC Rome Naples Zeppelin vs. Rome Chiplet The photo above provides a clear overview of the structure of the CPU package. The Naples CPU package contains four Zeppelin dies (black rectangles). A Zeppelin die provides a maximum of eight Zen cores. The cores are divided across two compute complexes (CCX). A Zeppelin of a 32 core EPYC contains 4 cores per CCX. When Simultaneous Multi-Threading (SMT) is enabled, a CCX offers eight threads. Each CCX is connected to the Scalable Data Fabric (SDF) through the Cache-Coherent Master (CCM) that is responsible for sending traffic cross CCXes. The SDF contains two Unified Memory Controllers (UMC) connecting the DRAM memory modules. Each UMC provides a memory channel to two DIMMs. Providing the memory capacity of 4 DIMMs in total. Due to the combination of Cores, cache, and memory controller, a Zeppelin is a NUMA domain. To access a “remote” on-package memory controller, the Infinity Fabric On Package Controller (IFOP) sets up and coordinates the data communication. Naples Zeppelin The Rome CPU package contains a 14nm I/O Die (the center black rectangle), and 8 chiplet dies (the smaller black rectangles). A Rome chiplet contains two CCX’es with each containing four cores and L3 cache, but no I/O components or the memory controllers. There is a small Infinity Fabric “controller” on each CCX that connects the CCX to the sIOD. As a result, every memory read beyond the local CCX L3 cache has to go to the sIOD. Even for a cache line (data from memory stored in the cache) that is stored in the LL3 cache of the CCX sharing the same Rome chiplet. A Chiplet is a part of the NUMA Domain. Rome Chiplet NUMA Domain per Socket As mentioned before, a NUMA Domain, typically called NUMA node, is a combination of CPU cores, cache, and memory capacity connected to a local memory controller. Intel architecture design uses a single NUMA Domain per Socket (NPS), AMD Naples offered four NPS, while Rome is back to a single NPS. Single NPS simplifies VM and application sizing while providing the best and consistent performance. NUMA per Socket Overview The bandwidth to local memory differs between each CPU architecture. The Intel Xeon Scalable Family provides a maximum of six channels of memory supporting a DDR4-2933 memory type. The Naples provides two memory channels to its locally connected memory, supporting a DDR4-2666 memory type. The Rome architecture provides eight memory channels to its locally connected memory, supporting a DDR4-3200 memory type. Please note that the memory controllers in the Rome architecture are located on the centralized die, handling all types of I/O and memory traffic, the Intel memory controllers are constructs isolated from any other traffic. Real-life application testing must be used to determine whether this architecture impacts memory bandwidth performance. CPU Architecture Local Channels Mem Types Peak transfer Intel Xeon Scalable 6 DDR4-2666 127.8 GB/s AMD EPYC v1 (Naples) 2 DDR4-2933 46.92 GB/s AMD EPYC v2 (Rome) 8 DDR4-3200 204.8 GB/s With a dual-socket system, there are typically two different distances with regards to memory access. Accessing memory connected to the local memory controller and accessing memory connected to the memory controller located on the other socket. With Naples, there are three different distances. The IFOP is used for intra-socket communication, while the Infinity Fabric Inter Socket (IFIS) controller takes care of routing traffic across sockets. As there are eight Zeppelins in a dual-socket system, not every Zeppelin is connected directly to each other and thus sometimes the memory access is routed through the IFIS first before hitting an IFOP to get to the appropriate Zeppelin. Naples Memory Access Hops Local memory access within a Zeppelin 0 Intra-socket memory access between Zeppelins 1 Inter-socket memory access between Zeppelins with direct IFIS connection 1 Inter-socket memory access between Zeppelins with indirect connection (IFIS+Remote IFOP) 2 AMD Rome provides equidistant memory access within the die and a single hop connection between sockets. Every memory access within the socket, every cache line load within the socket has to go to the I/O die. Every remote memory and cache access goes across the Infinity Fabric between sockets. This is somewhat similar to the Intel architecture that we have been familiar with since Nehalem, which launched in 2008. Why somewhat? Because there is a difference in cache domain design. The Importance of Cache in CPU Scheduling Getting memory capacity as close to the CPU improves performance tremendously. That’s the reason why each CPU package contains multiple levels of cache. Each core has a small but extremely fast cache capacity for instructions and data (L1), a slightly larger but relatively slower (L2) cache. A third and larger cache (L3) capacity is shared amongst the cores in the socket (Intel paradigm). Every time when a core request data to be loaded, it makes sense to retrieve this from the closest source possible, typically this is cache. To get an idea of how fast cache is relative to local and remote memory, look at the following table: System Event Actual Latency Human Scaled Latency One CPU cycle (2.3 GHz) 0.4 ns 1 second Level 1 cache access 1.6 ns 4 seconds Level 2 cache access 4.8 ns 12 seconds Level 3 cache access 15.2 ns 38 seconds Remote level 3 cache access 63 ns 157 seconds Local memory access 75 ns 188 seconds (3min) Remote memory access 130 ns 325 seconds (5min) Optane PMEM Access 350 ns 875 seconds (15min) Optane SSD I/O 10 us 7 hours NVMe SSD I/O 25 us 17 hours Back in the day when you could disable the cache of the CPU, someone tested the effect of cache on loading Windows 95. With cache it took almost five minutes, without the use of the cache, it took over an hour. Cache performance is crucial to get the best performance. And because of this, the vSphere NUMA scheduler and the CPU scheduler work together to optimize workloads that communicate with each other often. As they are communicating, they typically use the same data sources. Therefore, if vSphere can run the workload on the same cores that share the cache, then this could improve performance tremendously. The challenge is that AMD uses a different cache domain design than Intel. Last Level Cache Domains As depicted in the diagram above, Intel uses a 1:1:1 relationship model. One socket equals one NUMA domain and contains one Last Level Cache domain. As Intel is used in more than 98% of the dual-socket systems (info based on internal telemetry reports), our scheduling team obviously focused most of their efforts on this model. EPYC Naples introduced a 1:4:2 model, one socket, that contains four NUMA domains, and each NUMA domain contains two LLC domains. Rome provides a NUMA model similar to the XEON, with a single socket and single NUMA domain. However, each chiplet contains two separate LLC domains. A Rome CPU package contains eight chiplets, and thus, 16 different LLC domains exist within a socket & NUMA domain. Relational Scheduling vSphere uses this LLC domain as a target for its relational scheduling functionality. Relational scheduling is better known as Action-Affinity. Its actions have made most customers think that the NUMA scheduler was broken. As the scheduler is optimized for cache sharing, it can happen that a majority of vCPU is running on a single socket, while the cores of the other sockets are idling. When reviewing ESXTOP you might see an unbalanced number of VMs running on the same NUMA Host Node (NHN). As a result, the VMs running in this NUMA domain (or in ESX terminology NHN) might compete with CPU resources and thus experience increased %Ready time. Side note: It is my opinion to test the difference of relational scheduling on the performance of the application. Do not test this with synthetic test software. Although %Ready time is something to avoid, some applications benefit more from low-latency and highly consistent memory access than being impacted by an increase of CPU scheduling latency. Action-Affinity can lead to ready time on an Intel CPU architecture where more than eight cores share the same cache domain, imagine what impact it can have on AMD EPYC systems where the maximum number of cores per cache domain is four. In lower-core count AMD EPYC systems, the cores are disabled per CCX, reducing the scheduling domain any further. As the majority of the data centers are running on Intel, vSphere is optimized for a CPU topology where the NUMA and LLC domain are of consistent scope, i.e. the same size. With AMD the scopes are different and thus the current CPU scheduler can make “sub-optimal” decisions, impact performance. What happens is that the NUMA scheduler dictates the client size, the number of vCPUs to run on a NUMA Home Node, but it’s up to the CPU scheduler discretion to decide which vCPU to run on which physical core. As there are multiple Cache domains within a NUMA client, it can happen that there is an extraordinary amount of vCPU migrations between the cache domains within the NUMA domain. And that means cold cache access and a very crowded group of cores. Therefore, the CPU team worked very hard to introduce optimizations for the AMD architecture and these optimizations are released in the updates ESXi 6.5 Update 3 and ESXi 6.7 Update 2. The fix informs the CPU scheduler about the presence of the multiple cache domains within the NUMA node, allowing it to schedule the vCPU more intelligently. The fix also introduces a automatic virtual NUMA client sizer. By default, a virtual NUMA architecture is exposed to the guest OS when the vCPU count exceeds the physical core count of the physical NUMA domain and if the vCPU count is no less than the numa.vcpu.min setting, which defaults to 9. A physical NUMA domain in Naples counts eight cores, and thus no virtual NUMA topology is exposed. With the patch, this is solved. What is crucial to note is that the virtual NUMA topology is determined at first boot by default. Therefore, existing VMs need to have its virtual NUMA topology reset to leverage this new functionality. This involves a power-down to remove the NUMA settings in the VMX. When introducing Naples/Rome based systems in your virtual data center, it’s strongly recommended to deploy the latest update of your preferred vSphere platform version. This allows you to extract as much performance from your recent investment. ================================================================================ Title: 60 Minutes of NUMA VMworld Session Commands URL: https://frankdenneman.ai/2019-08-27-60-minutes-of-numa-vmworld-session-commands/ Date: 2019-08-27 Verify Distribution of Memory Modules with PowerCLI Get-CimInstance -CimSession $Session CIM_PhysicalMemory | select BankLabel, Description, @{n=‘Capacity in GB';e={$_.Capacity/1GB}} PowerCLI Script to Detect Node Interleaving Get-VMhost | select @{Name="Host Name";Expression={$_.Name}}, ​@{Name="CPU Sockets";Expression={$_.ExtensionData.Hardware.CpuInfo.NumCpuPackages}}, ​@{Name="NUMA Nodes";Expression={$_.ExtensionData.Hardware.NumaInfo.NumNodes}} Action-Affinity Monitoring Sched-Stats -t numa-migration Disable Action Affinity numa.LocalityWeightActionAffinity = 0 numa.PreferHT For more information on how to enable PreferHT: KB article 2003582 Host Setting: numa.PreferHT=1 VM Setting: numa.vcpu.PreferHT = TRUE ================================================================================ Title: 5 Things to Know About Project Pacific URL: https://frankdenneman.ai/2019-08-26-5-things-to-know-about-project-pacific/ Date: 2019-08-26 During the keynote of the first day of VMworld 2019, Pat unveiled Project Pacific. In short, project Pacific transforms vSphere into a unified application platform. By deeply integrating Kubernetes into the vSphere platform, developers can deploy and operate their applications through a well-known control plane. Additionally, containers are now first-class citizens enjoying all the operations generally available to virtual machines. Although it might seem that the acquisition of Heptio and Pivotal kickstarted project Pacific, VMware has been working on project Pacific for nearly three years! Jared Rosoff, the initiator or the project and overall product manager, told me that over 200 engineers are involved as it affects almost every component of the vSphere platform. Lengthy technical articles are going to be published in the following days. With this article, I want to highlight the five key takeaways from project Pacific. 1: One Control Plane to Rule Them All By integrating Kubernetes into the vSphere platform, we can expose the Kubernetes control plane to allow both developers and operation teams to interact with the platform. Instead of going through the hassle of installing, configuring, and maintaining Kubernetes clusters, each ESXi host acts as a Kubernetes worker node. Every cluster runs a Kubernetes control plane that is lifecycle managed by vCenter. We call this Kubernetes cluster the supervisor cluster, and it runs natively inside the cluster. This means that Kubernetes functionality, just like DRS and HA, is just a toggle switch away. 2: Unified Platform = Simplified Operational Effort As containers are first-class citizens, multiple teams can now interact with them. By being able to run them natively on vSphere means they are visible to all your monitoring, log analytics, change management operations as well. This allows IT teams to move away from the dual-stack environments. Many IT teams that have been investing in Kubernetes over the last few years started to create a full operational stack beside the stack to manage, monitor, and operate the virtualization environment. Running independent and separate stacks next to each other is a challenge by itself. However, most modern application landscapes are not silo’ed in either one of these stacks. They are a mix of containers, virtual machines, and sometimes even functions. Getting the same view across multiple operational stacks is near impossible. Project Pacific provides a unified platform where developers and operations share the same concepts. Each team can see all the objects across the compute, storage, and network layers of the SDDC. The platform provides a universal view with common naming and organization methods while offering a unified view of the complete application landscape. 3: Namespaces Providing Developer Self-service and Simplifying Management Historically, vSphere is designed with the administrator group in mind as the sole operator. By exposing the Kubernetes API, developers can now deploy and manage their applications directly. As mentioned earlier, modern applications are a collection of containers and VMs, and therefore the vSphere Kubernetes API has been extended to support virtual machines, allowing the developer to use the Kubernetes API to deploy and manage both containers as well as virtual machines. To guide the deployments of applications by the developers, project Pacific uses namespaces. Within Kubernetes, namespaces allow for resource allocation requirements and restrictions, and grouping of objects such as containers and disks. Withing project Pacific it’s way more than that. In addition, these namespaces allow the IT ops team to apply policies to it as well. For example, in combination with Cloud-Native Storage (CNS), a storage policy can be attached to the namespace, providing a persistent volume with the appropriate service levels. For more info on CNS, check out Myles Gray’s session: HCI2763BU Technical Deep Dive on Cloud Native Storage for vSphere Besides the benefits for the developers, as the supervisor cluster is subdivided into namespaces, they become a unit of tenancy and isolation. In essence, they become a unit of management within vCenter, allowing IT ops to resource allocation, policy management, and diagnostic and troubleshooting at namespace and workload level. As the namespace is now a native component within vCenter, it is intended to group every workload, both VMs, containers, and guest clusters and allow operators to manage it as a whole. 4: Guest Clusters The supervisor cluster is meant to enrich vSphere, providing integrations with cloud-native storage and networking. However, the supervisor cluster is not an upstream conformant Kubernetes cluster. Guest clusters use the Kubernetes upstream cluster API for lifecycle management. It is an open system that’s going to work with the whole Kubernetes ecosystem. 5: vSphere Native Pods providing lightweight containers with the isolation of VMs As we almost squashed the incorrect belief that ESXi is a Linux OS, we are now stating that containers are first-class citizens. Is ESXi after all a Linux OS, since you need to run Linux to operate containers? No ESXi is still not Linux, to run containers project Pacific is using a new container runtime called CRX. Extremely simplified, a vSphere Native Pod is a virtual machine. We took out all the unnecessary components and run a lightweight Linux kernel and a small container runtime (CRX). To utilize our years of experience of paravirtualization, we optimized this CRX in such a way that it outperforms containers running on the traditional platforms. As Pat mentioned in the keynote, 30% faster than a traditional Linux VM and 8% faster than bare-metal Linux. The beauty of using a VM construct is that these vSphere Native Pods are isolated at the hypervisor layer. Unlike pods that run on the same Linux host which share the same Linux kernel and virtual hardware (CPU and memory). vSphere Native Pods have completely separate Linux Kernel and virtual hardware, hence much stronger isolation from security and resource consumption perspective. Simplifying security and ensuring proper isolation models for multi-tenancy. Modern IT Centers Around Flexibility It’s all about using the right tool for the job. The current focus of the industry is to reach cloud-native nirvana. However, cloud-native can be great for some products, while other applications benefit from a more monolith perspective. Most applications are a hybrid form of microservices mixed with stateful data collections. Project Pacific allows the customer to use the correct tool for the job; all managed and operated from a single platform. VMware Breakouts to Attend or Watch HBI4937BU - The future of vSphere: What you need to know now by Kit Colbert. Monday, August 26, 01:00 PM - 02:00 PM | Moscone West, Level 3, Room 3022 More to follow Where Can I Sign Up for a Beta? We called this initiative a project as it is not tied to a particular release of vSphere. Because it’s in tech preview, we do not have a beta program going on at the moment. As this project is a significant overhaul of the vSphere platform, we want to collect as much direct feedback from customers as we can. You can expect we will make much noise when the beta program of Project Pacific starts. Stay tuned! ================================================================================ Title: VMworld US 2019 - Know Before You Go Podcast URL: https://frankdenneman.ai/2019-07-22-vmworld-us-2019-know-before-you-go-podcast/ Date: 2019-07-22 Last week I had the pleasure of connecting again with my friends and colleagues Pete Flecha, Duncan Epping and amateur back up dancer to Pat Benatar, Mr. Ken Werneburg. During the podcast, we discussed the upcoming VMworld. As it is returning to San Francisco, it might be interesting to revisit your conference strategy. Although Moscone Center has been rebuilt and expanded, I believe we are still using all three buildings; North, South, and West (Located at Howard and 3rd). So take at least a jacket with you, SF Summers can be treacherous For more tips about what to wear, what to bring, and which sessions to attend, listen to the episode below or search for it on Spotify. I hope you enjoy the show as much as I did. ================================================================================ Title: Allen, McKeown, and Kondo URL: https://frankdenneman.ai/2019-04-24-allen-mckeown-and-kondo/ Date: 2019-04-24 The title is a reference to one of the most interesting books I have ever read, Escher, Godel, and Bach. Someone described it as, “Read this book if you like to think about thinking, as well as to think about thinking about thinking”. The three books I want to share my thoughts on are in a sense feeding and shaping the behavior that allows you to clear your mind and focus more on the task at hand. The three books that I’m referring to are Getting Things Done (Allen), Essentialism (McKeown), and the KonMari (Kondo) method. They are written by three different authors, from three different continents, impacted by three different cultures in different years. Seemingly they have nothing to do with each other, but they complement each other so perfectly it’s downright amazing. After reading all three and re-reading them again, you start to discover hooks where these individual books mesh together The three books are instrumental in the way on how I live my life. I can imagine other people in IT with a similar travel lifestyle can benefit from reading these books as well. When you travel a lot, you need to get everything in order, as you have to ensure you have the essentials with you. You sometimes have little time to decompress from your last trip and prepare for the next one. You have to keep track of meetings and obligations both in your personal life as well as your professional one. And above all, you want to avoid wasting time on mundane or trivial tasks while being at home and spend your precious time most optimally. Three books have changed my mindset, and they help me guide my decisions. It helps to provide clarity and streamline day-to-day tasks. When this topic comes up during a conversation, many of the people I talk to end up buying one (or more) of these books, and it seems they catch the same bug, optimizing life, streamlining their behavior. I thought maybe it’s interesting to more people, let’s write an article about something else than hardcore CPU and Memory resource management. Let’s focus on how to manage time and to some extent energy. Getting Things Done The overall theme of Getting Things Done (GTD) is helping you manage to focus and therefore time. The main premise of the book in order, any time, everywhere. In your mind but also your surroundings. Instead of getting distracted by things that you need to do, the main rule is to do it immediately or write it down so you can do when appropriate. Writing it down and classifying the tasks helps to clear your mind, it helps you to focus on the task at hand. The author stresses to get rid of context switching. A perfect example is the junk drawer. Every time you walk past the junk drawer, it reminds you that you need to sort it out. You need to sift through the junk and see what you can use or what can be tossed away. That’s the context switch. Here you are walking around your house thinking about your big project, and there’s that junk drawer again, providing you with the annoying feeling that you really need to sort that out. You don’t want that, you want your mind focused on bigger things, no guilt trips when walking around the house. That’s where the other two books come into play. Same applies with the GTD method of categorizing tasks. To have oversight of the tasks at hand, you need to have a clear and tidy surrounding. You can’t keep efficient track of things if you have to go through a lot of junk to find the relevant to-do list. This scene in the movie Limitless is a perfect example. The protagonist is a writer who happens to be excellent in procrastinating. This results in no goals finished and an untidy house. When taking the cognitive expanding drugs, he wants to finish his life long goal of writing a book, but before he begins, he realizes that he cannot deal with any distraction and want order around him, resulting in a big cleanup of the house. That’s what GTD wants you to do as well, sans drugs of course. https://www.youtube.com/watch?v=zeI7tP1YEcQ Essentialism When cleaning the house, you typically end up throwing things away. A time-consuming job that never seems to finish. Sometimes you come across that you can’t let go, but you also don’t know what to do with it. In the end, it generates a conflicting feeling, introducing a context switch every time you see it. Hooking back into GTD. Essentialism allows you to prevent this by restructuring your behavior when buying new things and helps you to understand the role of your current belongings. Essentialism is not a lot different than minimalism. However, there is one significant difference, and that is the factor of happiness. With essentialism, you get to rate your belongings on the scale of happiness and usefulness. Does it make you happy or is it useful in day to day activities? If answered yes, then keep it. The interesting thing is that the book starts to reshape your decision making - or better said, the selecting criteria when buying something new. After reading it, I began to buy less of the things that I was eyeing because they just didn’t meet both criteria completely. The time of the acquisition process of an item is extended as you start to look for the object that provides the most happiness while delivering the required functionality. You begin to research the available options more thoroughly, it’s not uncommon to come to the conclusion that it’s better to approach the “problem” differently. You start to drive towards the essence of the problem, what am I solving here? Is there a better way. This ties in with a mindset that has been introduced by the book of Michael Hammer, reengineering the corporation. A fantastic book about redesigning processes, but I’ll cover that book another time. Another benefit of the elaborate purchase process is the occurrence of (re)buying a similar product, or actually the lack off. We’ve all bought a similar object after the first one because the current one wasn’t living up to its expectation or isn’t functioning properly. As you do your due diligence, you analyze the problem and research the best “tools” available. This can go as far as understand your preference of tactile feel of your cutlery. Trust me, you can go very far with applying this pattern of behavior. As a result, you surround yourself with a minimal set of objects that satisfy your needs perfectly. The stuff you have makes you very happy while decluttering your home as much as possible. Another example is my collection of Air Jordan shoes. Completely unnecessary, but they bring me joy. I collected these from the period when I played basketball myself. In the beginning, it was almost like a free-for-all, get the next version that is released. Buying it because you can (almost must). After reading essentialism, I reviewed my collection. Yes, collecting specific models makes me happy, but most ones that I have are not meeting the criteria of some of the special ones. In result, I reduced my collection by 70%, sold them so others can have them while reducing “footprint” of the collection in the house. I applied focus to the collection. To this day, with everything that I buy I ask myself: Do I need it? And is this the best I can obtain? What I learned is that the majority of objects acquired after reading essentialism have a longer lifespan than buying the first thing you come across when discovering the need for it. It improves the sustainability of your household tremendously. In short, you end up with a lot less stuff in your house, making it easier to get it organized and clean, increasing or maintaining your focus to the choirs at hand. The Life-changing Magic of Tidying Up (KonMari Method) This book took the world by storm, I discovered that the author, Marie Kondo, now has a show on Netflix. Before you wonder, I do not talk to my socks and thank them for the days’ work. ;) The key takeaway I had from reading this book is that junk is stuff that does not has a permanent place in your home. Everything that keeps moving through the house is junk. It generates context switching. To reduce junk, you have to learn some techniques about how to efficiently store things. Some things have exceeded their purpose and can be let go off. This ties back to the essentialism part. Does it make me happy or is it useful? These are excellent criteria to review all your belongings while cleaning up the house. By ending up with less stuff, it frees up room in your home to find permanent places for things that matter. And with a permanent location, it means less time spent on searching for things. Fewer context switches as the junk drawer is now the drawer that houses x, y, and z. I store my phone, wallet, keys in one particular place. When leaving the house, I do not waste time finding the stuff. I can maintain my focus while grabbing the necessities. The time to pack for a trip is significantly reduced, I just have to understand the weather and the purpose of the trip, I know exactly where everything is stored. These three books helped me tremendously, maybe they can be of help to you as well, give them a try. Please leave a comment about the books that structurally changed your perception on how to deal with these type of things, hopefully, it expands the must-read book list of others and me. ================================================================================ Title: VMware Cloud on AWS on Virtually Speaking Podcast URL: https://frankdenneman.ai/2019-04-09-vmware-cloud-on-aws-on-virtually-speaking-podcast/ Date: 2019-04-09 Last week I had the pleasure of connecting again with my friends and colleagues Pete Flecha a.k.a PedroArrow and eternal sunshine John Nicholson. During the podcast, we discussed the road to Hybrid cloud, cloud mobility, multi-cloud operations, and the necessity of replatforming apps or not. It’s always fun hanging out with these guys especially when talking about cool things. Hope you enjoy the show as much as I did. ================================================================================ Title: AMD EPYC and vSphere vNUMA URL: https://frankdenneman.ai/2019-02-19-amd-epyc-and-vsphere-vnuma/ Date: 2019-02-19 AMD is gaining popularity in the server market with the EPYC CPU platform. The EPYC CPU platform provides a high core count and a large memory capacity. If you are familiar with previous AMD generations, you know AMD’s method of operation is different than Intel’s. For reference, take a look at the article I wrote in 2011 about the 12-core 6100 Opteron code name Magny-Cours. EPYC provides an increase of scale but builds on the previously introduced principles. Let’s review the EPYC architecture and see how it can impact your VM sizing and ESXi configuration. (Please note that this article is NOT intended as a good/bad comparison between AMD and Intel, I’m just describing the architectural differences). EPYC Architecture The EPYC processor architecture is what AMD refers to as a Multi-Chip-Module (MCM). EPYC is designed to provide a high core count platform by combining multiple silicon dies within a CPU Package. A silicon die (named Zeppelin) is a wafer that contains the circuitry. In simple terms, it’s the component that contains CPU cores, memory cache, and various controllers. Regardless of the core-count, an EPYC CPU package always contains four Zeppelin dies. Comparing this to Intel Xeon, a Xeon CPU package is a single-chip-design which consist of a single silicon die containing all components. The reason why the difference in chip design is interesting is that impacts the logical grouping of compute resources. The size of the logical group, better known as a NUMA node, impacts scheduling decisions made by the CPU scheduler of the operating system (both the hypervisor kernel and possibly the guest operating system). It might be necessary to change some of the default settings of the ESXi host to alter scheduling behavior, these settings are covered in the last part of the article. Let’s continue to explore the architecture of the EPYC CPU. AMD EPYC - image courtesy of wccftech.com Compute Complex The photo above provides a clear overview of the structure of the CPU package. The CPU package houses four Zeppelin dies. In the current generation, a Zeppelin die provides a maximum of eight Zen cores. The cores are divided across two compute complexes (CCX). A Zeppelin of a 32 core EPYC contains 4 cores per CCX. When Simultaneous Multi-Threading (SMT) is enabled within the BIOS, a CCX offers eight threads. Zeppelin CCX Layout of 32 Core EPYC Each core has its own L1 (instruction (64KB) and data (32KB)) and L2 caches (4 MB total L2 cache). A Zeppelin has 16 MB L3 cache. Interestingly enough, each CCX has it’s own L3 Cache of 8MB, in turn, split up into four slices of 2 MB. The two CCXes within a Zeppelin die are connected to each other through an interconnect (Infinity Fabric). Adding hops to memory access is not beneficial to bandwidth and latency. Multiple tech-sites have performed in-depth testing on cache performance, and to quote Anandtech.com: “The local “inside the CCX” 8 MB L3-cache is accessed with very little latency. But once the core needs to access another L3-cache chunk – even on the same die – unloaded latency is pretty bad: it’s only slightly better than the DRAM access latency." In essence, this means that you cannot think of the 64MB L3 cache as one single pool of cache capacity. Better is to approach it as eight 8MB capacity pools. This is important to realize if multiple workloads share the same data, the NUMA scheduler of ESXi attempts to place both workloads in the same NUMA node to optimize cache and memory performance for these workloads. It might happen that the L3 cache size is not sufficient enough. The option that impacts this behavior is called Action Affinity, more details about this setting can be found in the last part of the article. Zeppelin Core Count EPYC is offered in multiple SKUs. Next, to the 32 core count model, there are lower-core count models. Since the EPYC architecture always includes four Zeppelins, the difference in core count is created by disabling cores per CCX in a symmetrical way. For example, in a 24 core count EPYC, a single Zeppelin die would look like this. Zeppelin design of 24 Core EPYC The table shows the core count per Zeppelin of the three largest EPYC CPUs. The total cores per Zeppelin count can be used as a guideline for the vNUMA setting described later in this article Cores Cores per CCX Total Cores per Zeppelin Zeppelin Count 32 4 8 4 24 3 6 4 16 2 4 4 Infinity Fabric The cores within a CCX communicate with memory (DIMMs) via an on-die memory controller through the infinity Fabric. The Infinity fabric is AMD’s proprietary system interconnect architecture that facilitates data and control transmission across all linked components. The Infinity Fabric consists of two communication planes; the Scalable Data Fabric (SDF) and the Infinity Scalable Control Fabric (SCF). The SCF is responsible for processing system control signals, such as thermal and power management. Although very important, we are more interested in the SDF which is responsible for transmitting data within the system. The rest of the article zooms into SDF design and its impact on scheduling decisions. Each CCX is connected to the SDF through the Cache-Coherent Master (CCM) that is responsible for sending coherent data traffic cross CCXes. The SDF uses a Unified Memory Controller (UMC) to connect to DRAM memory modules. Each UMC provides a memory channel to two DIMMs. Providing the memory capacity of 4 DIMMs in total. Zeppelin CCX and SDF Architecture How does this design impact VM sizing? A Zeppelin is a NUMA node that contains a maximum of 8 cores (16 threads) with the memory capacity of four DIMMs. This design results in a single EPYC CPU package presents four NUMA nodes to the operating system. **Server Memory Capacity and NUMA **Intel moved from a 3 DIMMs per channel configuration (DPC) with 4 channels to a model with 6 channels and 2 DIMMs deep. This new model broke the capacity model cadence. For example, using 16 GB DIMMs, you had either 64 GB, 128GB or 192GB available per socket. Now with the scalable architecture, it’s either 96GB or 192GB. That is if you follow the high- performance best practice of populating all channels for maximum bandwidth availability. However, with the current DIMM pricing, a lot of customers cannot afford such a configuration. With the EPYC, every Zeppelin has two memory channels. Each memory channel can drive two DIMMs. For good performance, each Zeppelin should be equipped with at least 1 DPC. That means that a proper performing dual socket EPYC system should be configured with 16 DIMMs. This configuration allows for a theoretical bandwidth of 42.6 GB/s while providing a (shallow) memory capacity of just the two DIMMs combined. This design results in a single EPYC CPU package presents four NUMA nodes to the operating system. If the minimum of 1DPC is used, the NUMA node size can be too small and thus the overall performance if the VM memory size exceeds the physical memory configuration of each Zeppelin. Servethehome published some benchmark tests about the performance difference between the different memory configurations of EPYC. 1 EPYC CPU Package = 4 NUMA Nodes With NUMA, it’s important to understand the boundaries of your local memory domain and your remote memory domain. Traditionally the domains were easily demarcated by the CPU package core count and attached memory capacity. With EPYC, a new distinction has to be made between the different remote memory access types. It can be remote on-package memory access or remote socket memory access. The reason why this distinction has to be made is the impact on performance and consistency of application memory access. Having your VM and application span multiple NUMA nodes can introduce a very inconsistent response time. Local Memory Access Let’s start with the best and most consistent performance. When a core within the Zeppelin access local memory the path is as follows: Local Memory Access The presentation “Zeppelin an SOC for Multi-Chip Architectures” by AMD list the latency of local memory access within the Zeppelin at 90 nanoseconds. Remote Memory Access On Package A core can access memory attached to a different Zeppelin within the same CPU package. This is called remote on-package memory access or “on-package Die-to-Die” memory access. This means we are still using memory controllers within the same socket. In total the EPYC CPU has eight memory channels, but two are local to the Zeppelin. To access a “remote” on-package memory controller the Infinity Fabric On Package Controller (IFOP) sets up and coordinates the data communication. In total each Zeppelin has 4 IFOPs, but actually, only three are needed since there are 3 other Zeppelins within the same CPU package. To be more precise, the IO traverses an additional component before hitting the IFOP. This component is called the Coherent AMD socKet Extender (CAKE). It facilitates die-to-die or socket-to-socket memory transactions. This module translates the request and response formats used by the SDF transport layer to and from the serialized format used by the IFOP. What that means is that a few extra hops and CPU cycles are introduced when fetching data stored within DIMMs attached to other Zeppelins on the same die. AMD reports a latency of ~145ns. Remote Memory Access within EPYC CPU Inter Package Remote Access And then we have the chance that memory needs to be fetched from DIMMs attached to UMCs from a Zeppelin that is a part of another EPYC CPU package within the system (dual socket system). Instead of routing the traffic across the IFOP, the traffic is routed across Infinity Fabric Inter Socket (IFIS) controller. Package-to-package traffic has 8/9 of the bandwidth of IFOP traffic, resulting in a theoretical bandwidth of 37.9 GB/s. The reduction in bandwidth increases the chance of experiencing inconsistent performance. The increased length of the path, increments latency. AMD reports a latency of ~200ns. Remote Access Across EPYC CPUs Because there are two IFIS controllers per Zeppelin, not every Zeppelin within a dual socket system is directly connected to each other. In the worst case scenario, there are two hops. One hop from one package to the other package and an extra hop to go from one Zeppelin to the Zeppelin that is connected to the DIMM holding the data. Unfortunately, AMD as not shared latency data. Remote Access Inter-package, die-to-die communication VM Sizing The key is to keep memory access as much local as possible. ESXi and most modern guest operating systems are optimized to deal with NUMA. However as with most things in life, for the most optimal performance, reduce distance and reduce any form of variation. Apply this to VM sizing and try to keep the vCPU count of a VM within the core count of NUMA domain. Same applies to VM memory capacity, try to fit this with the capacity of the NUMA node. If the VM cannot fit inside a NUMA node, there is no need to stress, ESXi has got the best NUMA scheduler in the business. To help ESXi to optimize for the EPYC architecture, some advanced settings might be necessary to adjust. As always, tests these settings in a non-revenue critical environment before applying them to production systems. Virtual NUMA Virtual NUMA (vNUMA) allows the operating system to understand the “physical” layout of the virtual machine. vNUMA presents the mapping of the VM vCPU to the physical NUMA nodes of the ESXi host. For example, if a VM has 12 vCPUs and the physical core count within a single NUMA node was 10 cores, ESXi would present the guest OS a topology of 2 NUMA nodes with each counting 6 cores. ESXi would group 6 vCPUs into a NUMA client and schedule these across the 10 CPU cores within a NUMA node. When vNUMA was introduced, the highest core count of a CPU was 8 CPUs, thus the VMware engineers introduced a vNUMA threshold of 9 (numa.vcpu.min=9). Meaning that the VM needs to contain at least 9 vCPUs in order to generate the virtual NUMA topology.Considering the highest core-count of an EPYC system is eight cores per Zeppelin, you might want to adjust the vNUMA default threshold to resemble the physical layout of the used EPYC model. For example, the EPYC 7401 contains 24 cores, 6 cores per Zeppelin and thus 6 cores per NUMA node. When using the default setting of numa.vcpu.min=9, an 8 vCPU VM is automatically configured like this. Screenshot by @AartKenens A VPD is the virtual NUMA client that is exposed to the guest OS system, while a PPD is the NUMA client used by the VMkernel CPU scheduler. In this situation, the ESXi scheduler uses two physical NUMA nodes to satisfy CPU and memory requests while the guest OS perceives the layout as a Uniform Memory Access (UMA) system. In a UMA system, the access time to a memory location is independent of which processor makes the request, or which memory chip contains the transferred data). I.e., pretty much the same latency and bandwidth throughout the system. However, this is not the case as reported in this article above. Reading and writing remote CCX cache and remote memory (on-die) is slower than local memory even within the same Zeppelin. By setting the numa.vcpu.min=6, two VPDs are created, and thus the guest OS is made aware of the physical layout by the ESXi scheduler. The guest OS and the applications can optimize memory operations to attain consistent performance. Action Affinity When the ESXi scheduler detects multiple VMs communicating with each other, it can decide of placing them together on the same NUMA node to increase intra-NUMA node communication. This behavior is called action affinity, and it can increase performance by up to 30%. However, with the small NUMA nodes of max 8 CPUs, it can also lead to a lot of cache thrashing and remote memory access if the configured memory of the VMs cannot fit inside a single NUMA node. If this is the case, it might be helpful to test disabling the action affinity on the ESXi host. This is done by configuring the /Numa/LocalityWeightActionAffinity to 0 (KB 2097369). What if the VM Memory Config Exceeds the Memory Capacity of the Physical NUMA Node? I wrote an article about this situation back in 2017, and it’s featured in the vSphere 6.5 Host deep dive book. However, what happens if your VM memory configuration exceeds the physical capacity of a NUMA node. By default, the ESXi scheduler optimizes for local memory access and attempts to place as much memory along with the vCPU in the same NUMA node. Sometimes it can improve local memory access to creating multiple smaller NUMA clients. For example, on an EPYC 7601 (32 core), the NUMA node contains 8 cores, and this server is equipped with 256 GB by using 16 x 16 GB DIMMs. A NUMA node has 4 DIMMs attached to it. Thus the NUMA node provides 8 cores and 64 GB. What happens if a VM is configured with 6 vCPUs and 96 GB? By default the NUMA scheduler attempts to store 64GB of VM memory inside the NUMA node, leaving 32 GB in a remote NUMA node. By enabling the VM advanced setting numa_.consolidate = FALSE_. It instructs the NUMA scheduler to distribute the VM configuration across the optimal number of NUMA nodes greater than 1. In this case, 2 NUMA clients are created, and this will schedule 3 vCPUs in each NUMA node. Now the performance and the behavior of the application depends on its design. If you have a single-threaded application, this setting might not be helpful at all. However, if it’s a multi-threaded application, you might see some benefit. The only thing to do is to set the numa.vcpu.min equal to the number of vCPUs per virtual NUMA client to expose the vNUMA architecture to the guest OS and the application. The following command helps you to retrieve the NUMA configuration of the VM: vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi “DICT.(displayname.|numa.|cores.|vcpu.|memsize.|affinity.)= .|numa:.|numaHost:.” “/$path/vmware.log”; echo -e; done Please bear in mind that the ESXi CPU and NUMA scheduler do not use an SRAT (System Resource Allocation Table) to determine the distance of the individual NUMA nodes between each other. ESXi uses its own method to determine latency between the different NUMA nodes within the system. It uses these latency numbers for initial placement and attempts to schedule the NUMA clients of a VM as close to each other as possible. However, the ESXi scheduler does not leverage this information during load-balancing operations. This is work in progress. Adding a new first class metric to a heuristic is not a simple task and knowing the CPU engineers, they want to provide a system that is thoroughly improved by augmenting new code. Increase NUMA Node Compute Sizing For workloads that are memory latency sensitive with a low processor utilization, you can alter the way the NUMA scheduler sizes the NUMA client of that particular VM. The VM advanced setting numa.vcpu.preferHT=TRUE allows the NUMA scheduler to count threads instead of cores for NUMA node size configuration. For example, an 8 vCPU VM that uses this advanced setting and runs on an EPYC 7401 system (6 cores, 12 threads), is scheduled within a single Zeppelin. If all workloads follow the same utilization pattern, you can alter the ESXi host setting by adding numa_.PreferHT=1_ to the ESXi host advanced configuration. Channel-Pair Interleaving (1 NUMA node per socket) The EPYC architecture can interleave the memory channels and thus present the cores of the four zeppelins as a single NUMA node. This setting requires that every channel is populated with equal memory size. Some vendors use a different name for it. For example, Dell calls this setting “Memory Die Interleaving”. Little to no data can be found about the performance impact of this setting, but keep in mind, software settings do not change the physical layout (and thus physics). Typically abstraction filters out the outliers and presents an average performance behavior. For NUMA benchmarking, please take a look at the article “AMD EPYC – STREAM, HPL, InfiniBand, and WRF Performance Study” located on the Dell website. Research Your Workload Requirements ESXi can handle complex NUMA architectures as the best. However, it’s always best to avoid complexity as possible. Determine if your workload can fit in a minimum number of small NUMA nodes when using the EPYC architecture? Can the workload handle inconsistent memory performance if it does exceed the NUMA node size of 8? The EPYC architecture is an excellent way of adding scale to the server platform but do remember that for real-life workload optimal performance is achieved when you take the NUMA configuration boundaries into account. On Twitter some asked what my thoughts are about the EPYC CPU architecture? For every tech challenge, there is a solution. When looking at the architecture, I think EPYC is an excellent solution for small and medium-sized workloads. I expect that larger monolithic apps, that require consistent performance, are better off looking at different architectures. (My opinion, not VMware’s!) ================================================================================ Title: Kubernetes, Swap and the VMware Balloon Driver URL: https://frankdenneman.ai/2018-11-15-kubernetes-swap-and-the-vmware-balloon-driver/ Date: 2018-11-15 Kubernetes requires to disable the swap file at the OS level. As stated in the 1.8 release changelog: The kubelet now fails if swap is enabled on a node. Why disable swap? Turning off swap doesn’t mean you are unable to create memory pressure. Why disable such a benevolent tool? Disable swap doesn’t make any sense if you look at it from a single workload, single system perspective. However, Kubernetes is a distributed system that is designed to operate at scale. When running a large number of containers on a vast fleet of machines, you want predictability and consistency. Disabling swap is the right approach. It’s better to kill a single container than to have multiple containers run on a machine at unpredictable, probably slow, rate. Therefore the kubelet is not designed to handle swap situations. It’s expected that workload demand should fit within the memory of the host. On top of that, it is recommended to apply quality of service (QoS) settings to workloads that matter. Kubernetes provides three QoS classes to pods; Guaranteed, Burstable, and BestEffort . Kubernetes provides the construct request to ensure the availability of resources. Similar to reservations at the vSphere level. Guaranteed pods have a request configuration that’s equal to the CPU and memory limit. All memory the container can consume is guaranteed, and therefore it should never need swap. With Burstable a portion of the CPU and memory is protected by a request setting, while a BestEffort pod does not have a CPU and memory request and limit setting specified. Multi-level Resource Management Resource management is difficult, mainly when you deal with virtualized infrastructure. You have to ensure the workloads receive the resources they require. Furthermore, you want to drive the utilization of the infrastructure in an economically sound manner. Sometimes resources are limited, and not all workloads are equal, thus adding another level of complexity of prioritization. Once you solved that problem, you need to think about availability and serviceability. Now the good news is that this is relatively easy with boundaries introduced by virtual machine configuration. I.e., you specify the size of the VM by assigning it CPU and memory resources. And this becomes a bin packing problem. Given n items of different weights and bins each of capacity c, assign each item to a bin such that the number of total used bins is minimized. A virtual machine is, in essence, a virtual hardware representation. You define the size of the box, with the number of CPUs and the amount of memory. This is a mandatory step in the virtual machine creation process. With containers it’s a little bit different. In its default state, the most minimal configuration, a container inherits the attributes of the system it runs on. It is possible to consume the entire system, depending on a workload. (a single threaded application, might detect all CPU cores available in the system, but its nature won’t allow it to run on more than a single core. In essence, a container is a process running in the Linux OS. For a detailed explanation, please (re)view our VMworld session, CNA1553BE. This means that if you do not specify any limit, the container has no restriction of how much resources such a pod can use. Similar to vSphere admission control, you cannot overcommit reserved resources. Thus, if you commit to an IT policy that only allows configuration of Guaranteed pods, you leverage Kubernetes admission control to avoid overcommitment of resources. One of the questions to solve either on a technical level or organization level is, how you are going to control pod configuration? From a technical level, you can solve this by using Kubernetes admission control, but that is out of scope for this article. Pod utilization is ultimately limited by the resources provided by the virtual machine, but you still want to provide predictability and consistency service to all workloads deployed in containers. Guarantees are only as good as the underlying foundation they are built upon. So how do you make sure behavior remains consistent for pods? Leveraging vSphere Resource Management Constructs When running Kubernetes within virtual machines (like the majority of the global cloud providers), you have to control the allocation of resources on multiple levels. From the top-down, the container is scheduled by Kubernetes on a worker node, predominantly Linux is used in the Kubernetes world, so let’s use that as an example. The guest OS allocates the resources and schedules the container. Please remember that a container is just a set of processes that are isolated from the rest of the system. Containers share the same operating system kernel and thus it’s the OS responsibility to manage and maintain resources. Lastly, the virtual machine runs on the hypervisor and the VMkernel manages resource allocation. VM-Level Reservation To ensure resources to the virtual machine, two constructs can be used. VM-level reservations or Resource Pool reservations. With VM-level reservations, the (ESXi host) physical resources are dedicated to the virtual machine, once allocated by the guest OS, it’s not shared with other virtual machines running on that ESXi host. This is the most deterministic way to allocate physical resources. However, this method impacts the virtual machine consolidation ratio. When using the vSphere HA admission control policy of Slot Policy it can impact the VM consolidation ratio at cluster level as well. Resource Pool Reservation A resource pool reservation is dynamic in nature. The resources backed by a reservation are distributed amongst the child-objects of the resource pool by usage and priority. If a Kubernetes worker node is inactive or running at a lower utilization-rate, these resources are allocated to other (active) Kubernetes worker nodes within the same resource pool. Resource Pools and Kubernetes are a great fit together, however, resource pool sizing must be adjusted when the Kubernetes cluster is scaled out with new workers. If the resource pool reservation is not adjusted, resources are allocated in an opportunistic manner from the cluster, possibly impacting predictability and consistency of resource behavior. Non-overcommitted Physical Resources Some vSphere customers design and size their vSphere clusters to fully back virtual machine memory with physical memory. This can be quite costly, but it does reduce operational overhead tremendously. The challenge is the keep the growth of the physical cluster aligned with the deployment of workload. Overcommited Resources But what if this strategy does not go the way as planned? What if for some reason resources are constrained within a host and the VMkernel applies one of its resource reclamation techniques? One of the feature that is in the first line of defense is the balloon driver. Designed to be as non-intrusive as possible to the applications running inside the VMs. Balloon Driver The balloon driver is installed within the guest VM as part of the VMware-Tools package. When memory is over-committed the ESXi server reclaims memory by instructing the balloon driver to inflate by allocating pinned physical pages inside the guest OS. This causes memory pressure within the guest OS which invokes its own native memory management techniques to reclaim memory. Balloon driver then communicates these physical pages to the VMkernel which can then reclaim the corresponding machine page. Deflating the balloon driver releases the pinned pages and frees up memory for general use by the guest OS. The interesting part is the dependencies of guest OS native memory management techniques. As a requirement, the swap file inside the guest OS needs to be set to disabled when you install Kubernetes. Otherwise, the kubelet won’t start. The swap file is the main reason why the balloon driver is so non-intrusive. It allows the guest OS to select memory page it deems fit. Typically these are idle pages and thus the working set of the application is not affected. What happens if the swap file is disabled. Is the balloon driver disabled? The answer is no. Let’s verify if the swap file is disabled, by using the command cat /proc/swaps. Just to be sure I used another command swapon -s. Both outputs shows no swap file. The command vmware-toolbox-cmd stat balloon shows the balloon driver size. Just to be sure I used another command lsmod | grep -E ‘vmmemctl|vmware_balloon to show if the balloon driver is loaded I created an overcommit scenario on the host and soon enough the balloon driver kicked into action. The command vmware-toolbox-cmd stat balloon confirmed the output of the stats showed by vCenter. The balloon driver pinned 4GB of memory within the guest. 4GB memory pinnned, but top showed nothing in swap. dmesg shows the kernel messages, one of them is the activity of the OOM Killer. OOM stands for out of memory. According to online description: _The Out-Of-Memory Killer process that It is the task of the OOM Killer to continue killing processes until enough memory is freed for the smooth functioning of the rest of the process that the Kernel is attempting to run. _ _The OOM Killer has to select the best process(es) to kill. Best here refers to that process which will free up the maximum memory upon killing and is also the least important to the system. _ The primary goal is to kill the least number of processes that minimizes the damage done and at the same time maximizing the amount of memory freed. Beauty is in the eye of the beholder, but I wouldn’t call killing CoreDNS the best process to kill in a Kubernetes system. Guaranteed Scheduling For Critical Add-On Pods In the (must-watch) presentation at Kubecon 2018, Michael Gasch provided some best practices from the field. One of them is to protect critical system pods, like DaemonSets, Controllers and Master Components. In addition to Kubernetes core components like api-server, scheduler, controller-manager running on a control plane (master) nodes there are a number of add-ons which run on a worker node . Some of these add-ons are critical to a fully functional cluster, such as CoreDNS. A cluster may stop working properly if a critical add-on is evicted. Please take a look at the settings and recommendations listed in “Reserve Compute Resources for System Daemons”. Please keep in mind that the guest OS, the Linux kernel, is a shared resource. Kubernetes runs a lot of its services as containers, however, not everything is managed by Kubernetes. For these services, it is best to monitor these important Linux resources in order that you don’t run out of them if you are using the QoS classes other than guaranteed. Exploring the Kubernetes Landscape For the vSphere admin who is just beginning to explore Kubernetes, we recommend keeping the resource management constructs aligned. Use reservations at the vSphere level and use guaranteed QoS class for your pods at the Kubernetes level. Solely using Guaranteed QoS class won’t allow for overcommitment, possibly impacting cluster utilization, but it gives you a nice safety net to learn Kubernetes without chasing weird behavior due to processes such as the OOM killer. Thanks to Michael Gasch for the invaluable feedback ================================================================================ Title: Free vSphere Clustering Deep Dive Book at VMworld Europe URL: https://frankdenneman.ai/2018-11-02-free-vsphere-clustering-deep-dive-book-at-vmworld-europe/ Date: 2018-11-02 Last year Rubrik gave away hard copies of the vSphere Host Deep Dive book, this year they are doing it again with the vSphere 6.7 Clustering Deep Dive Book. Come by the Rubrik Booth #P305 on Tuesday from 4:00 PM - 5:00 PM to get a signed, complimentary copy of vSphere 6.7 Clustering Deep Dive and meet the authors. Last year we gave away a thousand copies and were gone within an hour. As most of you can remember, the line was insane. This year we have a similar amount, so make sure you’re on time. https://www.youtube.com/watch?v=v0j3tJr0lUQ ================================================================================ Title: Kubernetes at VMworld Europe URL: https://frankdenneman.ai/2018-10-30-kubernetes-at-vmworld-europe/ Date: 2018-10-30 With only a few days left until VMworld Europe 2018 kicks off in Barcelona, I would like to highlight some of the many Kubernetes focussed sessions. I’ve selected a bunch of breakout sessions and meet the expert sessions based on my exposure to them at VMworld US or the quality of the speaker. The content catalog has marked some sessions as “at capacity”, but experience thought us that there are always a couple of no-shows. Plans change during VMworld. People register for a session they would like to attend but get pulled in an interesting conversation along the way. Or sometimes you suffer from information overload and want to catch a breather. In many cases, spots open at sold-out sessions and therefore it’s always recommended to walk up to sold out sessions and try your luck. Tuesday 06 November 11:00 - 12:00 [NET1285BE] (Breakout Session) The Future of Networking and Security with VMware NSX This talk provides detailed insights into the architecture and capabilities of NSX-T. We’ll show how NSX-T addresses container workloads and integrates with frameworks like Kubernetes. We’ll also cover the multi-cloud networking and security capabilities that allow consistent networking policies across any cloud, public or private. Finally, we’ll look at how SD-WAN has become part of the NSX portfolio, enabling networking and security to be deployed from cloud to data center to edge. More info. By Bruce Davie, CTO, APJ, VMware 12:15 - 13:00 [MTE5044E] (Expert Roundtable) Selecting the Right Container Platform for Your Use Case with Patrick Daigle There are a variety of containers and Kubernetes platforms out there in market today. Ever got confused and wanted some expert insight into what types of container or Kubernetes platforms are best suited to your use case? By Patrick Daigle, Sr. Technical Marketing Architect, VMware 13:15 - 14:00 [MTE5209E] (Expert Roundtable) Cloud Native Applications and vSAN with Myles Gray Learn how vSAN can provide storage for next generation applications autonomously, including Kubernetes, PKS or any K8S distribution and moves the provisioning of storage from the admin into the hands of the developer. By Myles Gray, Sr. Technical Marketing Architect, VMware 14:00 - 15:00 [CNA1553BE] (Breakout Session) Deep Dive: The Value of Running Kubernetes on vSphere In this technical session, you will find out how VMware vSphere provides a lot of value, especially in large-scale Kubernetes deployments. With 20 years of engineering experience in kernel and distributed computing, VMware solved many challenges Kubernetes currently faces. Building on work done with enterprises running Kubernetes at scale, you will see a hypothetical customer scenario to illustrate the benefits of running Kubernetes on top of VMware vSphere and avoid the common pitfalls associated with running on bare metal. More info. By Frank Denneman, Chief Technologist, VMware Michael Gasch, Customer Success Architect - Application Platforms, VMware 15:30 - 16:30 [HCI1338BE] (Breakout Session) vSAN: An Ideal Storage Platform for Kubernetes-controlled Cloud-Native Apps The session discusses how VMware’s HCI offering (vSphere and vSAN) is becoming a platform of choice for deploying, running and managing the data needs of Cloud-Native Applications (CNA). We will use real world examples to highlight the benefits of an HCI control plane for Kubernetes environments. More info. By Christos Karamanolis, Fellow and CTO Storage & Availability, VMware Cormac Hogan, Director and Chief Technologist, VMware Wednesday 07 November 11:15 - 12:00 [MTE5057E] (Expert Roundtable) Next-Gen Apps on vSAN by expert Chen Wei Are you planning to migrate your next-gen workload to the vSAN cluster? Attend this roundtable to talk to our vSAN Solutions Architect about different aspects regarding putting Next-gen applications on vSAN. Those aspects include the next-gen application deployment best practices, performance tuning, availability/performance trade-off. Bring the questions and let’s talk. Chen Wei, Sr. Solutions Architect, VMware 12:30 - 13:30 [CNA1493BE] (Breakout Session) Run Docker on Existing Infrastructure with vSphere Integrated Containers In this session, you will find out how to run Docker on vSphere with VMware vSphere Integrated Containers. See a live demo on how vSphere Integrated Containers leverage vSphere for isolation and scheduling. Find out how vSphere Integrated Containers are the ideal way to host containers on vSphere, providing a Docker-native experience for end users and a vSphere-native experience for IT. More info. By Patrick Daigle, Sr. Technical Marketing Architect, VMware Martijn Baecke, Cloud Evangelist, VMware 13:15 - 14:00 [MTE5116E] (Expert Roundtable) Function as a Service with Mark Peek During this roundtable, we will discuss Dispatch, the VMware framework for deploying and managing serverless style applications. By Mark Peek, Principal Engineer, VMware 15:30 - 16:30 [DC3845KE] (Keynote) Cloud and Developer Keynote: Public Clouds and Kubernetes at Scale This session will cover VMware’s strategy to deliver an enterprise-grade Kubernetes platform while supporting the needs of DevOps and CloudOps teams. VMware’s Cloud and Developer keynote will outline how to deliver developers a consistent experience across native clouds while enabling operators with more flexibility and control for how they support next generation workloads. More info. By Guido Appenzeller, CTO, VMware Joseph Kinsella, Vice President and CTO, Products, CloudHealth, VMware 15:30 - 16:30 [CNA2755BE] (Breakout Session) Architecting PKS for Production: Lessons Learned from PKS Deployments In this session, you will get a deep dive into PKS within the context of real-world customer deployment scenarios. The speakers will share the lessons learned from their successful PKS and NSX-T deployments, and show you how to architect PKS for a production environment. Come and learn about the do’s, don’ts, and best practices. After this session, you will be better equipped to deploy and manage enterprise-grade Kubernetes in your infrastructure and use NSX-T to bridge the gap in network and security for container workloads. By Romain Decker, Senior Solutions Architect, VMware Dominic Foley, Senior Solutions Architect, VMware Thursday 08 November 15:00 - 16:00 [NET1677BE] (Breakout Session) Kubernetes Container Networking with NSX-T Data Center Deep Dive In this session, you will get technical details of how the NSX-T Data Center integration with Kubernetes in Pivotal Container Service (PKS), OpenShift, and upstream Kubernetes is implemented. Get a deep dive into each identified problem statement, find out how the solution was implemented with NSX-T Data Center, and see a demo of each of the solutions live on stage using PKS with NSX-T Data Center. More info. By Dennis Breithaupt, Sr. Systems Engineer (NSX), VMware Yasen Simeonov, Technical Product Manager, VMware Product Preview This year the UX team organizes design studios that allows you to provide feedback on a future product. The product you will see will blow your mind. But since it’s NDA, I can’t tell ;) Just show up and see for yourself! Every day - multiple sessions available [UX8011E] (Design Studio) Kubernetes on vSphere Do you want to offer Kubernetes? Explore user interface concepts for managing containerized cloud native applications using vSphere together with other products such as PKS. This session is part of the VMware Design Studio where you have the opportunity to participate in interactive sessions exploring technical previews and early design ideas. Because of the early nature of these designs, participants will be asked to sign a Non-Disclosure Agreement (NDA) to participate. By Boaz Gurdin, User Experience Researcher, VMware Pamel Shinh, Product Designer, VMware Hope to see you there. Enjoy your VMworld! ================================================================================ Title: Repeat Session vSphere Clustering Deep Dive at VMworld Europe URL: https://frankdenneman.ai/2018-10-22-repeat-session-vsphere-clustering-deep-dive-at-vmworld-europe/ Date: 2018-10-22 Good news for the VMworld attendees who couldn’t sign up anymore for the vSphere Clustering Deep Dive session on Tuesday. I’m happy to announce that the VMworld team scheduled a repeat session for the vSphere Clustering Deep Dive session on Thursday 08 November at 10:30 to 11:30. Session Outline In this session, Duncan and Frank will take you through the trenches of VMware vSphere Distributed Resource Scheduler (DRS) and vSphere High Availability (HA). Find out about options to optimize your DRS settings for your specific requirements and goals, such as if you should be load balancing on active or consumed memory, as well as what has recently changed in the DRS algorithm and if it will impact DRS behavior. And for vSphere HA, you will learn about when it restarts virtual machines (VMs), what kind of restart times to expect, and where you can find evidence that a VM (or multiple) have been restarted. You will find out about all of these items and more. Prepare to dive deep, as the basics will not be covered. Don’t wait too long with registering, VMworld Europe room sizes max out at 400 people. We hope to see you there! ================================================================================ Title: Compute Policy in VMware Cloud on AWS URL: https://frankdenneman.ai/2018-10-19-compute-policy-in-vmware-cloud-on-aws/ Date: 2018-10-19 The latest update of VMware Cloud on AWS introduced a new feature called compute policies. In its initial release, the compute policies provide the ability to configure affinity rules and mobility control based of declarative policies and vSphere tags. Management of affinity rules Historically, affinity rules are a part of the cluster configuration. Within VMware Cloud on AWS, cluster configuration is controlled by VMware and thus customers cannot set affinity rules for virtual machines running within the SDDC. Instead of merely pulling the affinity rules configuration outside the cluster configuration, we decided to improve the affinity functionality and work towards a more uniform and consistent experience across multiple clouds. The road to declarative policies Within a declarative system, you describe what you want to happen. This is the opposite of imperative operations where you specify actions. Declarative commands define state and to some extent affinity rules are declarative statements. Let’s take VM anti-affinity rules as an example. You want to keep VM1 and VM2 separated and keep them in different fault domains. Instead of providing imperative actions of pinning VM1 to host A and pinning VM2 to host B, you create an anti-affinity rule with VM1 and VM2 as members. You state that these two VMs should not run on the same ESXi host. vCenter (DRS) controls placement and takes the necessary actions to solve any violations of this intent. We want to apply this model to other features. Instead of logging into vCenter to deal with configuration issues, and manually correct the situation, we want vCenter to manage the functions of your behalf. The way you interact with vCenter, in this more declarative way, is with policies. Instead of specifying more detailed imperative actions, you would declare your intent and the only thing you want to monitor after that is whether the policy is compliant or not. We have to start somewhere, thus we concentrated on affinity rules (VM-VM and VM-host) and anti-mobility (vMotion disabled) policies. Once we have this more abstract way of interacting with vCenter Server, it provides more advantages. One of them is an additional level of abstraction. And abstraction allows for a more uniform and consistent experience across multiple clouds. With today’s ability on-prem setup, you configure your cluster for a particular workload and this could inhibit the ability to move your workload to another cluster, on-prem or even to the cloud. To make sure you can easily burst out to VMware Cloud environments, you want this to be seamless. The directions where we are going to is that you do not need to have configurations that are specific to on-prem clusters and in-cloud or at-edge clusters. But ideally you express what you want and it should be the job of the cloud control plane, such as vCenter, to push this configuration to the environment the workload is presently in. So that could be to an on-prem cluster or an in-cloud cluster. Compute policies are active at vCenter level Due to this model, the rules are decoupled from cluster level and are now managed at vCenter level. If you would configure a VM-VM anti-affinity rule and you would move the VMs to another cluster, the policy remains active. At the time of writing, VMware Cloud on AWS allows the customer to create 10 clusters per SDDC. Clusters can span multiple AWS availability zones (AZs). The VM-Host affinity ruleset allows customers to tag the hosts per AZ and tag the VMs that needs to remain in that availability zone. You can move the VMs to hosts between clusters within the same AZ, the compute policy remains active while vCenter ensures the compliance of the rule. Introduction of firm rules An interesting fact is that the VM-Host rules are firm rules, these firm rules differ from the traditional soft (should run on) and hard (must run on). They sit in between these two rules. DRS cannot violate these rules, only if the host is placed in maintenance mode. This ensures that during normal operations the rules are never broken while providing VMware the ability to service the SDDC. The only time a host is placed into maintenance mode in VMware Cloud on AWS is during upgrades which are handled by VMware and well communicated before the service window. This allows the customer to generate a strategy for these virtual machines well ahead before the service window. In the next article, I will go through the steps on how to create a compute policy. ================================================================================ Title: My New Role URL: https://frankdenneman.ai/2018-10-08-my-new-role/ Date: 2018-10-08 A couple of months ago I joined the Office of the CTO of the Cloud Platform Business Unit and started reporting directly to the CTO, Kit Colbert. Kit asked me to select a few areas to focus on. One of these areas is running Kubernetes on vSphere. I’ve increased my focus on Kubernetes, as this architecture becomes increasingly important in the datacenter. When talking to customers, two questions I ask is, what is the current ratio of VMs to containers in your data center and what is the most popular format of deployment today? The common response is respectively 90% VMs and net-new is 90% containers. Today’s trend moves away from installing shrink-wrapped software and more towards custom building revenue-critical applications by their development teams. The standard tool for developers is container-based infrastructure. Kubernetes is the defacto choice of orchestration of containers and consists of many infrastructure-focused options. The operations that interest me are the high availability and resource management operations. It appears these operations replace HA and DRS processes when glancing over them, but when looking more closely they strongly augment each other. At VMworld in Las Vegas, Michael Gasch and I presented the session “Deep Dive: The Value of Running Kubernetes on vSphere” (CNA1553BU). If you are not going to VMworld Europe, I recommend watching the video recording, if you are going to VMworld Europe I recommend you to sign up. One thing you can expect from me is more Kubernetes focused articles. One of the things that I noticed is that many articles are written by cloud-native natives for cloud-native natives. I.e. they rely on extensive previous exposure to this ecosystem. I’m trying to cover some of the challenges I have faced and the quirks I notice as a “newcomer”. ================================================================================ Title: Help Us Make vMotion Even Better URL: https://frankdenneman.ai/2018-10-02-help-us-make-vmotion-even-better/ Date: 2018-10-02 The vMotion product team is looking for input on how to improve vMotion. vMotion has proven to be a paradigm shift of datacenter management. Workload mobility is a must-have requirement in today’s datacenter operational model. vMotion handles the majority of workload flawlessly. However, there are some corner cases that introduce some challenges. The vMotion product team is interested in these corner cases, to improve the vMotion architecture bringing workload mobility to all workloads everywhere. It would be very helpful if you can provide us with some more information to make vMotion even better. Thanks! Take the survey here ================================================================================ Title: Terminal Affinity Poll URL: https://frankdenneman.ai/2018-09-12-terminal-affinity-poll/ Date: 2018-09-12 We are looking into the combination of licensed workload and hard-affinity rules (Must run on rule). If you deploy this in your environment right now, how do you deal with this during maintenance hours? Your input helps in shaping future features. (Scroll down in the survey window to access the done button to submit your response) ================================================================================ Title: Six Interesting Kubernetes Sessions at VMworld 2018 URL: https://frankdenneman.ai/2018-09-07-six-interesting-kubernetes-sessions-at-vmworld-2018/ Date: 2018-09-07 This year VMworld provided a broad selection of talks focusing on various forms of Kubernetes. Which is not surprising at all. Many organizations move away from buying and installing shrink-wrapped software and move towards in-house built custom applications. And what is the modern developer tool of choice? For many, it is the container. It’s expected to have 1.5 Billion containers shipped by the end of 2021. Containers are nothing more than a new format of virtualized workload. Michael Gasch explains it very well in our session Deep Dive: The Value of Running Kubernetes on vSphere (CNA1553BU), where containers are task structs in the Linux kernel, not very different than executing an LS command. Well, a bit more than that as containers require CPU, memory, network, storage, and security. Containers satisfy the developers’ need for speed, and they remove dependencies on underlying operating systems. When deploying massive amounts of containers, you need a container management platform, and Kubernetes is clearly the defacto standard in the industry. Source: [Cloud Native Computing Foundation](source: https://www.cncf.io/blog/2017-06-28-survey-shows-kubernetes-leading-orchestration-platform/) For the infrastructure team, running Kubernetes can provide a way to create an infrastructure agnostic platform. That is, it can run on any cloud. VMware is fully vested in making this happen; you can run containers natively (VIC), containers and Kubernetes in Linux VMs on vSphere. Pivotal Container Service (PKS) on-prem or in-cloud that helps customer deploy and operationalize day 1 and day 2 kubernetes solution and VMware Kubernetes Engine (VKE) (Kubernetes as a Service) for organizations who want to consume Kubernetes without owning, building or operationalizing any infrastructure. I’ve selected a few VMworld sessions that cover these container consumption models. There are many more, and please check them out at the VMworld On-Demand Video Library. Container and Kubernetes 101 for vSphere Admins (CNA1564BU) A very popular session at VMworld was the 101 session for vSphere Admins. Nathan Ness and Sachin Thatte go over the basics of Container, Kubernetes and Pivotal Container Services. A very helpful primer for the rest of the listed videos. (Watch here) Running Kubernetes on vSphere Deep Dive: The Value of Running Kubernetes on vSphere (CNA1553BU) Michael Gasch (Resident Kubernetes Expert at VMware) and I go over the reasons why vSphere and Kubernetes are better together. We provide guidelines on how to successfully run your Kubernetes environment. (Watch Here) A Deep Dive on Why Storage Matters in a Cloud-Native World (HCI1813BU) 7 out of 10 applications that run in containers are stateful applications (source: Datadog), you want to provide persistent storage. Myles and Tushar talk about project Hatchway and provide a preview of the upcoming Cloud Native Storage (CNS) Control plane. (Watch Here) Operating and Managing Kubernetes on Day 2 with PKS (CNA1075BU) If you are planning to run large-scale kubernetes deployments on-prem, you should consider Pivotal Container Service (PKS). PKS allows you to deploy multiple kubernetes clusters quite easily. Thomas Kraus and Merlin Glynn show how to tackle day 2 operations and review SDDC products, such as vRealize and Wavefront, that integrates with PKS. (Watch Here) VMware Kubernetes Engine VMware Kubernetes Engine (VKE) offers a turn-key solution of managed Kubernetes clusters that run natively on AWS. Not in VMware Cloud on AWS, not on vSphere, pure native EC2! Plans are to run VKE at multiple cloud providers, allowing you to create environments that no-other cloud provider themselves can provide. Think about an HA cluster spanning both AWS and Azure. However, we are not that far right now, but it is interesting to take a look at what VKE is and how Smart Clusters will change the way you will operate Kubernetes. Intro to VMware Kubernetes Engine-Managed K8s Service on Public Cloud (CNA2084BU) Tom and Valentina go over the concepts and customer value of VKE, including a nice demo. (Watch Here) Deep Dive: VMware Kubernetes Engine-K8s as a Service on Public Cloud (CNA3124BU) After getting familiar with VKE, I recommend to watch the session of Tom and Alain. They dive deeper into the concept of Smart Clusters. (Watch Here) I hope you enjoy watching these sessions, please leave a comment about sessions you think are worth watching. ================================================================================ Title: Tech Paper DRS Enhancements in vSphere 6.7 URL: https://frankdenneman.ai/2018-09-06-tech-paper-drs-enhancements-in-vsphere-6-7/ Date: 2018-09-06 During VMworld, the DRS performance team released a new tech paper covering the DRS Enhancements in vSphere 6.7. It’s a short white paper uncovering the interesting improvements made to DRS. Download it here. ================================================================================ Title: Catch me at VMworld 2018 URL: https://frankdenneman.ai/2018-08-10-catch-me-at-vmworld-2018/ Date: 2018-08-10 Two weeks left before the biggest VMware show is happening again, and I can’t wait for it to start. The last eight years I’ve been going to both the US and European show, and both have their own charm. But there is one thing that every VMware community member should experience, and that is the US welcome reception in the solution exchange on Sunday night. Almost every attendee in one big room, the buzz is just phenomenal. I recently joined Kit Colbert’s team, the CTO of Cloud Platform business unit. In my new role, I work on upcoming products and influence their strategy. One project I focus on is how VMware can help customers to run Kubernetes successfully on vSphere. Please reach out to me at VMworld if you have ideas or feedback. Luckily I will be presenting a few sessions this year as well, and I hope to see you there: VIN1249BU vSphere Clustering Deep Dive, Part 1: vSphere HA and DRS 2018-08-27 12:30 PM The legendary session is back, Duncan and I talking about vSphere 6.7 HA and DRS. There is so much to tell, but we are hoping to keep some time open for some questions. CNA1553BU Deep Dive: The Value of Running Kubernetes on vSphere 2018-08-27 3:30 PM I’m so much looking forward to this session, together with Michael Gasch, our resident Kubernetes expert, and popular Kubecon speaker. In this session, we will go over the reasons why vSphere and Kubernetes are better together and provide you with some guidelines on how to successfully run your kubernetes environment. VIN2256BU Tech Preview: The Road to a Declarative Compute Control Plane 2018-08-28 12:30 PM I tweeted about every session on this list except this one. The reason why I had to keep quiet about this session is that we are showing some NDA stuff. In this session, Maarten Wiggers and I look at the changes that are happening in the industry. Most companies develop their strategic apps in-house, impacting the role of the VI-admin. We will go over the transformation from VI-admin to Site Reliability Engineering. With new technologies and different Life Cycle Management strategies, different ways of managing applications and infrastructure are necessary. We go over the changes from an infrastructure that responds to Imperative statements to an environment that is controlled by declarative statements. Within the software-defined data center (SDDC), VMware vSphere offers two declarative control planes: one for networking and one for storage. However, there is no declarative control plane for compute yet. We will tech preview the capabilities introduced in the VMware Cloud SDDC as a path to achieve that goal. VIN1738BU vSphere Host Resources Deep Dive: Part 3 2018-08-29 2:00 PM The third edition of the vSphere Host Resources Deep Dive. The vSphere platform is designed to run most workloads at near bare-metal performance. More than enough for more than 95% of the workload. But what if you need to squeeze out that last bit of performance? How can you do it and how will it impact the rest of the system? Please join Niels and me on Wednesday at 2:00 PM. ================================================================================ Title: vSphere 6.x Deep Dive Resource Kit Completed URL: https://frankdenneman.ai/2018-07-30-vsphere-6-x-deep-dive-resource-kit-completed/ Date: 2018-07-30 The new version of the vSphere clustering deep dive is available on Amazon. The vSphere 6.7 Clustering Deep Dive is the fourth edition of the best selling series. Over 50.000 clustering deep dive books have been distributed, and I hope this version will find its way on your desk. The new version of the clustering deep dive covers HA, DRS, Storage DRS, Storage I/O Control and Network I/O Control. In the last part of the book, we bring all the theory together and apply it to create and describe a stretched cluster configuration. Now, why am I using the title vSphere 6.x Deep Dive Resource Kit? Well, it’s because we believe that when you pair this with the vSphere 6.5 Host Resource Deep Dive book, you get this bundle that allows you to understand the core of your virtual infrastructure. Changing the Game When Duncan and I set out to write the 4.1 HA and DRS deep dive, we wanted to change the content of technical books. Instead of having a collection of screenshots paired with the text, next, next finish, we wanted to provide a thorough explanation of what happens under the cover. When you push this button, this happens in the code. By uncovering the inside, we arm the administrator and architect with the knowledge to create or troubleshoot any architecture anywhere. When combining these books together, it creates a real end-to-end guide for your architecture. For example, in the DRS section, we explain how the cluster determines the resource entitlement of the VMs in a resource pool. In the vSphere 6.5 Host resource deep dive, we describe the inner workings of the memory and CPU scheduler and how they allocate the physical resources based on the resource entitlement of the VM. Back Side of the Book When releasing the host resource deep dive, we came up with a cool little logo of a divers helmet. If you want to get deep, you need more than a snorkel. One divers helmet to explore the host, but in the cluster deep dive, we cover multiple hosts, grouped in a cluster. What do you need when you need a lot of people to explore the deep? You need a submarine! ;) It might even end up on some T-shirt. New Name on the Cover As you might have noticed, a new name appears on the cover. We asked Niels Hagoort to help us to cover the quality of service aspect of the book. Niels dove into the deeps of Storage I/O Control and Network I/O Control and created an excellent addition to the book. Foreword And last but not least, the foreword. In the previous books, industry luminaries generously provided us with amazing forewords. This time we looked at the community. We asked Chris Wahl to write the introduction. Chris has been an early supporter of the book series, and he has helped the community in many ways. We asked him to provide us with his point of view. I hope you enjoy the book as much as we enjoyed writing it. ================================================================================ Title: Hotdog-Not Hotdog: The SDDC of VMware Cloud on AWS URL: https://frankdenneman.ai/2018-07-20-hotdog-not-hotdog-the-sddc-of-vmware-cloud-on-aws/ Date: 2018-07-20 Yesterday, Kenneth Hui was on stage at the VTUG providing his personal opinion about VMware Cloud on AWS. The reason I say personal is that he forgot to remove the Rubrik Logo’s from his slide (I checked with Rubrik). On one slide he mentions that the SDDC, that is the Software Defined Data Center provided by VMware Cloud on AWS (VMC) is not an SDDC out of the box. And to me, that sounds a bit weird. Let’s go over the process of spinning up an SDDC. First, you log onto vmc.vmware.com and you sign up for the service. In the console you define the number of hosts for deployment, click apply. If you select a multi-host deployment, by default an SDDC cluster contains 4 hosts, that means that VMC deploys four physical hosts (for more info: Dedicated Hardware in a Public Cloud World) on the AWS infrastructure. It installs and configures vSphere, vSAN, and NSX for you automatically. After roughly two hours you are the sole-owner of dedicated hardware with a fully software-defined data center running on top of that. Just log into your in-cloud vCenter and start to deploy your workload. So to reiterate, you just clicked a button on a website and a fully functional data center is deployed for you. https://twitter.com/kenhuiny/status/1019995175735758848 Ok so what about day 2 operations, let’s define this a bit clearer because there are multiple definitions available. Dzone provides the following definition: Once “something” goes into operations, “day 2 operations” is the remaining time period until this “something” isn’t killed or replaced with “something else.” We build a cloud management platform in AWS in order to deal with day-2 operations. VMware provides the service, we will keep the lights on for you, troubleshoot and maintain your environment. This CMP plaform allows us to provide services like automated hardware remediation. If a component inside the ESXi hosts fails, such as a NIC, or an NVMe device, the backend will detect this and it will initiate a process to replace the faulty host with a fully operational one. The customer won’t have to do a thing. Elastic DRS allows the cluster to respond to workload utilization automatically. It allows for automatic scale-out and scale-in, without the need for human intervention. Stretched Clusters protects the workload in the Cloud SDDC from AZ outages. If something happens, HA detects the failed VMs and restarts them on different physical servers in the remaining AZ without manual human involvement. Content library, allows the customer to subscribe the in-cloud SDDC to a template repository that automatically provides VM templates to the in-cloud SDDC. Read Williams post for more info Disaster Recovery as a Service, just go to the console, enable the add-on and the in-cloud components for SRM and vSphere Replication are automatically deployed and configured. Connect it to your on-prem components and you can build your DR runbooks. And there are many more functions that cover the lights-on, maintenance, housekeeping and optimize tasks of day 2. Now with that explained, the stories continue and a debate broke out on twitter. Some said it needs a form of CMP (eg. vRealize) for operating the SDDC. https://twitter.com/KenNalbone/status/1020106642287886337 This is an interesting observation, for which operation? Not for life-cycle or infrastructure management. We will take care of that for you. VMC is a fully managed service by VMware. It is responsible for the uptime and the lifecycle of the SDDC. we have built a CMP platform on the AWS infrastructure that allows us to deal with VMC. In a presentation of Chris Wegner (one of the principal engineers of VMC) the architecture is explained. The blue box is the actual SDDC, The green box is a custom-built CMP that allows VMware to identify customers, billing customers, providing support for customers (as a VMC customer, you only deal with VMware) but most importantly for this story, it allows VMware to deploy hardware and software (Fleet management). The next image provides a more detailed view of the green box. This is what you need to support hundreds of SDDCs across multiple regions (Oregon, N. Virginia, London, Frankfurt). Here you can see the bits for provisioning management, dealing with AWS services, acquiring hardware, configuring all the software and of course the ability to troubleshoot. You as a customer, do not need to worry again about ripping and replacing hardware because it failed, or because it’s nearing the end of support. You only need to care about deploying your workload. And because we took the conscious decision of using vCenter as the management structure, you can use your on-prem vRealize suite and deploy your workload on-prem or in-cloud. Using vRealize to deploy workload is the way to go forward because 80% of our customers have a hybrid cloud strategy a on-prem deployment is expected. It makes sense to run your tooling on-premises. With VMware Cloud on AWS, your responsibility shift from managing hardware to managing the consumption of resources. ================================================================================ Title: Kubernetes and vSphere Compute and Storage Doubleheader VMworld Session URL: https://frankdenneman.ai/2018-07-19-kubernetes-vsphere-compute-storage-doubleheader-vmworld-session/ Date: 2018-07-19 Kubernetes is hot! It is one of the most talked about technologies of this year. For some, it’s the next platform, for others it’s just another tech making its way in the datacenter. Will it replace virtual machines, will it get displace vSphere? Some ask, why run Kubernetes on top of vSphere when you can run it on bare metal? We rather not go back to 2005 and deal with a sprawl of bare-metal servers, we believe Kubernetes and vSphere are better together! In the session “CNA1553BU - Deep Dive: The value of Running Kubernetes on vSphere” Michael Gasch and I review the behavior of Kubernetes resource management, optimization, and availability for container orchestration. Kubernetes is a system optimized for cloud-native workloads where failure and disruption is anticipated, but how about the infrastructure that is required to run these cloud-native apps? How about Kubernetes ability to economically and optimally consume the available resources? We will answer these questions and reveal why vSphere is such a good match with its extensive features such as high availability, NUMA optimization, and distributed resource scheduler. In this session, we explore the critical elements of a container and demonstrate that Kubernetes does not run in thin air. Running Linux on bare-metal or inside a VM determines your scalability, your recoverability, and your portability. If you spin up a Kubernetes cluster at Amazon or Google, they will deploy it for you in virtual machines, if these cloud-native giants use VMs, why would you use bare-metal? Adding vSphere to the picture, Kubernetes gains several advantages for both, cloud-native and traditional workloads. vSphere also plays a critical role in keeping the Kubernetes control plane components highly-available in case of planned and unplanned downtime. We are going to detail recommended DRS and HA settings and many other best practices for Kubernetes on vSphere based on real-world customer scenarios. Of course, an outlook on upcoming improvements for the Kubernetes on vSphere integration should not be missing in a deep dive session! Last but not least, you’ll definitely learn how to respond to common objections to win back your end-user. Still not convinced? Let’s dive into the behavior of Linux CPU scheduling versus ESXi CPU and NUMA scheduling and help you understand how to size and deploy your Kubernetes cluster on vSphere correctly. Developers shouldn’t need to worry about all these settings and the underlying layers. They just want to deploy the application, but it’s our job to cater to the needs of the application and make sure the application runs consistently and constantly. This applies to compute, but also to storage.7 out of 10 applications that run on kubernetes are stateful, so it makes sense to incorporate persistent storage in your kubernetes design. Some applications are able to provide certain services like replication themselves, thus it makes no sense to “replicate” that service at the infrastructure layer. vSAN and its storage policies allow the admin to provide storage services that are tailor-made to the application stack. Cormac Hogan and Christos Karamanolis talk about why vSAN is the ultimate choice for running next-gen apps. Visit their session “HCI1338BU-HCI: The Ideal Operational Environment for Cloud-Native Applications” to hear about real-world use-cases and learn what you need to do when dealing with these next-gen apps. Please note that if you attempt adding these sessions to your schedule, you might get a warning that you are on the waiting list. As we understood it, all sessions are booked in small rooms and depending on the waiting list they are moved to bigger rooms. Thus sign up for these sessions even if they state waiting list only. It will be sorted out during the upcoming weeks. Hope to see you in our session! ================================================================================ Title: Introduction to Elastic DRS URL: https://frankdenneman.ai/2018-07-17-introduction-elastic-drs/ Date: 2018-07-17 VMware Cloud on AWS allows you to deploy physical ESXi hosts on demand. You can scale in and scale out your cluster by logging into the console. This elasticity allows you to right-size your SDDC environment for the current workload demand. No more long procurement process, no more waiting for the vendor to ship the goods. No more racking, stacking in a cold dark datacenter. Just with a few clicks, you get new physical resources added to your cluster, ESXi and vSAN fully installed, configured, patched and ready to go! Having physical resources available on demand is fantastic, but it still requires manual monitoring and manual operations to scale out or scale in the vSphere cluster. Wouldn’t it be more comfortable if the cluster automatically responds to the dynamic nature of the workloads? As of today, you can enable Elastic DRS. Introducing Elastic DRS Elastic Distributed Resources Scheduler (EDRS) is a policy-based solution that automatically scales a vSphere Cluster in VMware Cloud on AWS based on utilization. EDRS monitors CPU, memory, and storage resources for scaling operations. EDRS monitors the vSphere cluster continuously, and each 5 minutes EDRS runs the algorithm to determine if scale-out or scale-in operations is necessary. Algorithm Behavior EDRS is configured with thresholds for each resource and generates scaling recommendations if utilization consistently remains above or below their respective thresholds. EDRS algorithm takes spikes and randomness of utilization into consideration when generating these scaling recommendations. Scaling Operations Thresholds are defined for scale up operations and scale down operations. To avoid generating recommendations by spikes, EDRS generates a scale operation if the resource utilization shows consistent progress towards a threshold. To generate a scale out operation, a single threshold must be exceeded. That means that if CPU utilization shows consistent progress towards the threshold and at one point exceeds the threshold, EDRS triggers an event and adds an ESXi host to the vSphere cluster. Similar to adding an ESXi host manually, the ESXi host is installed with the same ESXi version, patch level and is configured with the appropriate logical networks and adds the capacity to the vSAN datastore. To automatically scale down the cluster, utilization across ALL three resources must be consistently below the specified scale-in thresholds. Minimum and Maximum Number of ESXi hosts You can restrict the bounds of a minimum and a maximum number of ESXi hosts. EDRS can be enabled if the cluster consists of four ESXi hosts, EDRS does not scale in beyond the four ESXi host minimum. When setting a maximum number of ESXi hosts, all ESXi hosts in the vSphere cluster, including those in maintenance mode are included in the count. Only active ESXi hosts are counted towards the minimum. As a result, the VMware cloud on AWS SDDC ignores EDRS recommendations during maintenance and hardware remediation operations. Currently, the maximum number of host in an Elastic-DRS enabled cluster is 16. Scaling Policies EDRS provides policies to adjust the behavior of scaling operations. EDRS provides two scaling policy that optimizes for cost or performance. Both policies have the same scale-out threshold. They only differ on scale-in thresholds. Scale Out Threshold Performance Optimized Cost Optimized CPU 90% 90% Memory 80% 80% Storage 70% 70% As a result, if the cluster consistently utilizes memory over 80%, EDRS triggers a scale out operation that adds a new host to the vSphere cluster. Please note that the load is tracked at the host level and then aggregated. EDRS aggregates the CPU and memory load per fault-domain, for storage it is aggregated at the vSAN datastore level. Scale In Threshold Performance Optimized Cost Optimized CPU 50% 60% Memory 50% 60% Storage 20% 20% In essence, the performance policy is more eager to keep the resources than the cost-optimized policy. If you set the EDRS cluster to cost-optimized, an ESXi host is removed from the vSphere cluster if CPU and memory utilization is consistently below 60% and storage utilization is consistently below 20%. How to Configure Elastic DRS Log into your VMware Cloud on AWS console and select the cluster. A message box will show Elastic DRS is enabled on the cluster. To further fine-tune Elastic DRS, you have the choice of clicking on the green message box at the top right of your screen or select the option “Edit EDRS settings” in the bottom of your screen. The next step is to select the Scaling Policy. As mentioned, EDRS provides two scaling policy that optimizes for cost or performance. In this screen, you can fine-tune the behavior of Elastic DRS or disable it if you want to keep the host count of your cluster at a static level. Please note that if you would select the default settings, you give EDRS the permission to scale up to 16 physical ESXi nodes. If this number of ESXi hosts is too high for you, please adjust the maximum cluster size. EDRS scales up per single node and evaluates the current workload, it uses a time window of 1 hour for evaluation. It does not add multiple hosts at once. Elastic DRS is designed to adjust to your workload dynamically, it responds to the current demand and scales in and out in a more fluid way. If you are aware of a high volume of incoming workload, you can add multiple hosts to the cluster swiftly by logging into the console, select the cluster to scale out and select add hosts. You can finetune EDRS by using PowerCLI. Kyle Ruddy will publish an article containing the PowerCLI commands shortly. ================================================================================ Title: Resource Pools and Sibling Rivalry URL: https://frankdenneman.ai/2018-07-16-resource-pools-sibling-rivalry/ Date: 2018-07-16 One of the most powerful constructs in the Software Defined Data Center is the resource pool. The resource pool allows you to abstract and isolate cluster compute resources. Unfortunately, it’s mostly misunderstood and it received a bad rep in the past that it cannot get rid off. One of the challenges of resource pools is to fully commit to resource pools. Placing virtual machines next to resource pools can have an impact of resource distribution. This article zooms in on sibling rivalry. But before this adventure begins, I would like to stress that the examples provided in the article are a worst-case scenario. In this scenario, all VMs are 100% active. An uncommon situation, but it helps to easily explain the resource distribution. Later in the article, I use a few examples, in which some VMs are active and some are idle. And as you will see, resource pools aren’t that bad after all. Resource Pool Size Because resource pool shares are relative to other resource pools or virtual machines with the same parent resource pool, it is important to understand how vCenter sizes resource pools. The values of CPU and memory shares applied to resource pools are similar to virtual machines. By default, a resource pool is sized like a virtual machine with 4 vCPUs and 16GB of RAM. Depending on the selected share level, a predefined number of shares are issued. Similar to VMs, four share levels can be selected. There are three predefined settings: High, Normal or Low, which specify share values with a 4:2:1 ratio, and the Custom setting, which can be used to specify a different relative relationship. Share Level Shares of CPU Shares of Memory Low 2000 81920 Normal 4000 163 840 High 8000 327 680 Caution must be taken when placing VMs at the same hierarchical level as resource pools, as VMs can end up with a higher priority than intended. For example, in vSphere 6.7, the largest virtual machine can be equipped with 128 vCPUs and 6 TB of memory. A 128vCPU and 6TB VM owns 256 000 (128 x 2000) CPU shares and 122 560 000 (6 128 000 x 20) memory shares. Comparing these two results in a CPU ratio is 32:1 and memory 374:1. The previous is an extreme example, but the reality is that 16GB and 4 vCPU VM is not uncommon anymore. Placing such a VM next to a resource pool results in unfair sibling rivalry. The Family Tree of Resource Consumers As shares determine the priority of the resource pool or virtual machine relative to its siblings, it is important to determine which objects compete for priority. In the scenario depicted above, multiple sibling levels are present. VM01 and Resource Pool-1 are child objects of the cluster and therefore are on the same sibling level. VM02 and VM03 are child objects of Resource Pool-1. VM02 and VM03 are siblings, and both compete for resources provided by Resource Pool-1. DRS compares their share values to each other. The share values of VM01 and the other two VMs cannot be compared with each other because they each have different parents and thus do not experience sibling rivalry. Shares indicate the priority at that particular hierarchical level, but the relative priority of the parent at its level determines the availability of the total amount of resources. VM01 is a 2-vCPU 8GB virtual machine. The share value of Resource Pool-1 is set to high. As a result, the resource pool owns 8000 shares of CPU. The share value of VM01 is set to Normal and thus it owns 2000 CPU shares. Contention occurs, and the cluster distributes its resources between Resource Pool-1 and VM01. If both VM02 and VM03 are 100% utilized, Resource Pool-1 receives 80% of the cluster resources based on its share value. Resource Pool-1 divides its resources between VM02 and VM03. Both child-objects own an equal number of shares and therefore receive each 50% of the resources of Resource Pool-1 This 50% of Resource Pool-1 resources equals to 40% of the cluster resources. As for now, both VM02 and VM03 are able to receive more resources than VM-1. However, three additional VMs are placed inside Resource Pool-1. The new VMs own each 2000 CPU shares, increasing the total number of outstanding shares to 10.000. The distribution at the first level remains the same during contention. The cluster distributes its resources amongst its child-object, VM01 and Resource Pool-1; 20% to VM01 and 80% to Resource Pool-1. Please note this only occurs when all objects are generating 100% utilized. If VM01 was generating 50% of its load and the VMs in Resource Pool-1 are 100% utilized, the cluster would flow the unused resources to the resource pool to satisfy the demand of its child objects. The dynamic entitlement is adjusted to the actual demand. The VMs inside RP-1 are equally active, as a result of the reduced activity of VM01, they each receive 2% more resources. VM02, VM03, and VM04 start to idle. The Resource Pool shifts the entitlement and allocates the cluster resources to the VMs that are active, VM05 and VM06. They each get 50% of 80% of the cluster resources due to their sibling rivalry. Share Levels are Pre-sets, not Classes A VM that is placed inside the resource pool, or created in a resource pool, does not inherit the share level of the resource pool. When creating a VM or a resource pool, vCenter assigns the Normal share level by default, independent of the share level of its parent. Think of share levels as presets of share values. Configure a resource pool or virtual machine with the share-level set to high, and it gets 2000 CPU shares per vCPU. A VM configured with the share level set to low gets 500 CPU shares. If the VM has 4 vCPUs, the VM owns the same number of shares than the 1 vCPU with a share value set to high. Both compete with each other based on share amounts, not based on share level values. Next Article This article is a primer for a question about which direction we should take with Resource Pools. This week I shall post a follow up article that zooms in on the possible changes of shares behavior of resource pool in the near future. Stay tuned. ================================================================================ Title: Virtually Speaking Podcast about Technical Writing URL: https://frankdenneman.ai/2018-07-11-virtually-speaking-podcast-technical-writing/ Date: 2018-07-11 Last week Duncan and I were guests on the ever popular Virtually Speaking Podcast. In this show we discussed the difference in technical writing, i.e., writing a blog post versus writing a book. We spoke a lot about the challenges of writing a book and the importance of a supporting cast. We received a lot of great feedback on social media, and Pete told me the episode was downloaded more than a 1000 times in the first 24 hours. I think this is especially impressive as he published the podcast on a Saturday Afternoon. Due to this popularity, I thought it might be cool to share the episode in case you missed the announcement. Pete and John shared the links to our VMworld sessions on this page. During the show, I mentioned the VMworld session of Katarina Wagnerova and Mark Brookfield. If you go to VMworld, I would recommend attending this session. It’s always interesting to hear people talk about how they designed an environment and dealt with problems in a very isolated place on earth. Enjoy listening to the show. ================================================================================ Title: Resource Consumption of Encrypted vMotion URL: https://frankdenneman.ai/2018-06-26-resource-consumption-encrypted-vmotion/ Date: 2018-06-26 vSphere 6.5 introduced encrypted vMotion and encrypts vMotion traffic if the destination and source host are capable of supporting encrypted vMotion. If true, vMotion traffic consumes more CPU cycles on both the source and destination host. This article zooms in on the impact of CPU consumption of encrypted vMotion on the vSphere cluster and how DRS leverages this new(ish) technology. CPU Consumption of vMotion Process ESXi reserves CPU resources on both the destination and the source host to ensure vMotion can consume the available bandwidth. ESXi only takes the number of vMotion NICs, and their respective speed into account, the number of vMotion operations does not affect the total of CPU resources reserved! 10% of a CPU core for a 1 GbE NIC, 100% of a CPU core for a 10 GbE NIC. vMotion is configured with a minimum reservation of 30%. Therefore, if you have 1 GbE NIC configured for vMotion, it reserves at least 30% of a single core. Encrypted vMotion As mentioned, vSphere 6.5 introduced encrypted vMotion and by doing so it also introduced a new stream channel architecture. When an encrypted vMotion process is started, 3 stream channels are created. Prepare, Encrypt and Transmit. The encryption and decryption process consumes CPU cycles and to reduce the overhead as much as possible, the encrypted vMotion process uses the AES-NI Instruction set of the physical CPU. AES-NI stands for Advanced Encryption Standard- New Instruction and was introduced in the Intel Westmere-EP generation (2010) and AMD Bulldozer (2011). It’s safe to say that most data centers run on AES-NI equipped CPUs. However, if the source or destination host is not equipped with AES-NI, vMotion automatically reverts to unencrypted if the default setting is selected. Although the encrypted vMotion leverages special CPU hardware instructions set to offload overhead, it does increase the CPU utilization. The technical paper “VMware vSphere Encrypted vMotion Architecture, Performance, and Best Practices” published by VMware list the overhead on the source and destination host. [caption id=“attachment_7302” align=“aligncenter” width=“676”] Encrypted vMotion CPU Overhead on the Source Host[/caption] [caption id=“attachment_7303” align=“aligncenter” width=“676”] Encrypted vMotion CPU Overhead on the Destination Host[/caption] Encrypted vMotion is a per-VM setting, by default, every VM is configured with Encrypted vMotion set to Opportunistic. The three settings are: [caption id=“attachment_7304” align=“aligncenter” width=“676”] VM Options Encrypted vMotion[/caption] Setting Behavior Disabled Does not use encrypted vMotion Opportunistic Use encrypted vMotion if the source and destination host supports it. Only vSphere 6.5 and later use encrypted vMotion Required Only allow encrypted vMotion. If the source and destination host does not support encrypted vMotion, migration with vMotion is not allowed. Please be aware that encrypted vMotion settings are transparent to DRS. DRS generates a load balancing migration, and when the vMotion process starts, the vMotion process verifies the requirements. Due to the transparency, DRS does not take encrypted vMotion settings and host compatibility into account when generating a recommendation. If you select required, because of security standards, it is important to understand if you are running a heterogeneous cluster with various vSphere versions. Is every host in your cluster 6.5 otherwise you are impacting the ability of DRS to load-balance optimally. Or are different types of CPU generations inside the cluster, do they support AES-NI? Please make sure the BIOS version supports AES-NI and make sure AES-NI is enabled in the BIOS! Also, verify if the applied Enhanced vMotion Compatibility (EVC) baseline exposes AES-NI. CPU Headroom It is important to keep some unreserved and unallocated CPU resources available for the vMotion process, to avoid creating gridlock. DRS needs some resources to run its threads, and vMotion requires resources to move VMs to lesser utilized ESXi host. Know that encrypted vMotion taxes the system more, in oversaturated clusters, it might be interesting to understand whether your security officer state encrypted vMotion as a requirement. ================================================================================ Title: New Fling: DRS Entitlement URL: https://frankdenneman.ai/2018-05-30-new-fling-drs-entitlement/ Date: 2018-05-30 I’m proud to announce the latest fling; DRS entitlement. This fling is built by the performance team and it provides insight to the demand and entitlement of the virtual machines and resource pools within a vSphere cluster. By default, it shows the active CPU and memory consumption, which by itself helps to understand the dynamics within the cluster. Especially when you are using resource pools with different levels of share values. In this example, I have two resource pools, one containing the high-value workloads for the organization, and one resource pool containing virtual machines that are used for test and dev operations. The high-value workloads should receive the resources they require all the time. The What-If functionality allows you to simulate a few different scenarios. A 100% demand option and a simulation of resource allocation settings. The screenshot below shows the what-if entitlement. What if these workloads generate 100% of activity, what resources do these workloads require if they go to the max? This allows you to set the appropriate resource allocations settings such as reservations and limits on the resource pools or maybe even on particular virtual machines. Another option is to specify particular Reservation, Limits, and Shares (RLS) settings to an object. Select the RLS option and select the object you want to use in the simulation. In this example, I selected the Low Value Workload resource pool and changed the share value setting of the resource pool. You can verify the new setting before running the analysis. Please note, that this is an analysis, it does not affect the resource allocation of active workload whatsoever. You can simulate different settings and understand the outcome. Once the correct setting is determined you can apply the setting on the object manually, or you can use the PowerCLI setting and export the PowerCLI one-liner to programmatically change the RLS settings. Follow the instruction on the flings website to install it on your vCenter. I would like to thank Sai Inabattini and Adarsh Jagadeeshwaran for creating this fling and for listening to my input! RUN DRS! ================================================================================ Title: Stretched Clusters on VMware Cloud on AWS, a Really Big Thing URL: https://frankdenneman.ai/2018-05-16-stretched-clusters-vmware-cloud-aws-really-big-thing/ Date: 2018-05-16 This week Emad published an excellent article about the stretched cluster functionality of VMware Cloud on AWS. To sum up, you can now deploy a single vSphere cluster across two AWS availability zones. A trip to Memory Lane I think the ability to stretch a vSphere cluster across two availability zones is a really big thing. Go back to the days where we had to refactor the application to make it highly available. To reduce application downtime, you typically used clustering software such as Microsoft cluster or Veritas clustering services. But not all applications were fit for this solution. When we introduced VMware High Availability back in 2006, we brought a big change to the industry. From that point on you could provide crash-consistent failover ability to all your workloads. No need to refactor any application, no need to build outlandish hardware solutions. Just enable a few tickboxes at the infrastructure layer, and every workload running inside a VM is protected. And to this day, HA remains the most popular functionality of vSphere. Amazon Web Services Resiliency Strategy Amazon urges you to design your application to be resilient to infrastructure outages. Amazon AWS is hosted in multiple locations worldwide. These locations are composed of regions and Availability Zones. Each region is a separate geographic area that has multiple, isolated locations known as Availability Zones. AWS provides the ability to place instances and data in multiple locations. And you can take advantage of the safety and reliability of geographic redundancy by spanning your Auto Scaling group across multiple Availability Zones within a region and then attach a load balancer to distribute incoming traffic across those Availability Zones. Incoming traffic is distributed equally across all Availability Zones enabled for your load balancer. And this works very well if you are refactoring your application or if you are building a complete new cloud-native stack. The challenge we face today is that not all applications lend to getting refactored, or some applications do not require the journey from monolithic to full-FAAS. Hybrid-Cloud Experience With stretched clusters in VMware Cloud on AWS, we introduce the same ease of infrastructure resiliency to workloads that run on AWS infrastructure. Merely expand you vSphere cluster to 6 hosts and select multi-az deployment. After that, the workload in the Cloud SDDC is protected for AZ outages. If something happens, HA detects the failed VMs and restarts them on different physical servers in the remaining AZ without manual human involvement. The ability to stretch your vSphere cluster across AZs allows you to easily provide resiliency to your workload within the AWS infrastructure without the Herculean effort of refactoring all your applications. ================================================================================ Title: Dying Home Lab - Feedback Welcome URL: https://frankdenneman.ai/2018-05-15-dying-home-lab-feedback-welcome/ Date: 2018-05-15 The servers in my home lab are dying on a daily basis. After four years of active duty, I think they have the right to retire. So I need something else. But what? I can’t rent lab space as I work with unreleased ESXi code. I’ve been waiting for the Intel Xeon D 21xx Supermicro systems, but I have the feeling that Elon will reach Mars before we see these systems widely available. The system that I have in mind is the following: Intel Xeon Silver 4108 - 8 Core at 1.8 GHz (85TDP) Supermicro X11SPM-TF (6 DIMMs, 2 x 10 GbE) 4 x Kingston Premier 16GB 2133 Intel Optane M.2 2280 32 GB CPU Intel Xeon Silver 4108 8 Core. I need to have a healthy number of cores in my system to run some test workload. Primarily to understand host and cluster scheduling. I do not need to run performance tests, thus no need for screaming fast CPU cores. TDP value of 85W. I know there is a 4109T with a TDP value of 70W, but they are very hard to get in the Netherlands. Motherboard Supermicro X11SPM-TF.Rocksolid Supermicro, 2 x Intel X722 10GbE NICs onboard and IPMI. Memory Kingston Premier 4 x 16 GB 2133 MHz. DDR4 money is nearing HP Printer Ink prices, 2133 MHz is fast enough for my testing, and I don’t need to test 6 channels of RAM at the moment. The motherboard is equipped with 6 DIMM slots, so if memory prices are reducing, I can expand my system. Boot Device Intel Optane M.2 32 GB. ESXi still needs to have a boot device, no need to put in 256 GB SSD. This is the config I’m considering. What do you think? Any recommendations or alternate views? ================================================================================ Title: Dedicated Hardware in a Public Cloud World URL: https://frankdenneman.ai/2018-05-03-dedicated-hardware-public-cloud-world/ Date: 2018-05-03 One of the more persistent misconceptions is that the components of VMware’s Software Defined Data Center (SDDC) on VMware Cloud on AWS are virtualized or that the deployed VMs run natively on Amazon. And to be honest, it’s not even weird that most people think this way. After all, Amazon Web Services launched in March 2006, 12 years ago. AWS and Elastic Compute Cloud (EC2) and Amazon Simple Storage Service (S3) are synonymous with each other. All of a sudden, you can know “run vSphere on AWS”. To be short and sweet, VMware Cloud on AWS runs on physical hardware, it is not virtualized and running inside EC2 instances! VMware Cloud is consuming the AWS infrastructure and using a bare-metal service offered by AWS. Of course, it is not as simple as installing vSphere on a bare-metal server and you got yourself a fully elastic cloud service. More than that needs to happen. VMware Cloud on AWS is a partnership between the two companies and both have done some extensive R&D work to make this happen. If you want to know more, Chris Wagner - Principle Architect of the service presented an excellent session (LHC3174BU) at VMworld on how we built it. Back to the service offering, when deploying an SDDC, by default a four node cluster is erected. Four physical hosts are assigned to a single customer account, and the service installs, patches and rolls out the full SDDC stack of vSphere, vSAN, and NSX. You just have to log on to vCenter and start deploying workloads. Each ESXi host provides 36 CPU cores of 2.3 GHz (72 threads), 512 GB of RAM and 10.7 TB of raw storage capacity for the virtual machines to consume. As a result, a default vSphere cluster provides 144 CPU cores (288 threads), 2 TB of RAM and 42.8 TB of raw storage capacity. All physical resources! Due to leveraging the scale of AWS data centers and its operational framework, the VMware Cloud on AWS fleet management service can deploy physical resources on demand! By logging into the console (vmc.vmware.com) you can add and remove physical host to the cluster. This allows you add physical hardware to the cluster, whenever you need it. No more long procurement process, no more waiting for the vendor to ship the goods. No more racking, stacking in a cold dark datacenter. Just with a few clicks, you get fresh new hardware added to your cluster, fully installed, configured, patched and ready to go. Typically this takes about 10 minutes for VMware Cloud on AWS to add a single physical host to your vSphere cluster. I’ve been to data centers that it took me more than 10 minutes to arrive at the correct cabinet. If one is not enough, you can add up to 28 ESXi hosts in the cluster. In the example above, I added 10 additional hosts. The console list the host type, the extra capacity added by this action (10 ESXi hosts = 360 Cores, 5 TB RAM and 107 TB of Storage and sums the new cluster capacity. If you want to isolate specific workloads and add a separate cluster, just go right ahead and select the add cluster option in the console. In total, a VMware Cloud on AWS customer can deploy up to 10 clusters of each 32 ESXi hosts in a single SDDC. In total two SDDCs can be erected. That means that a customer can have 23040 of physical CPU cores, 327 TB of memory and 6.8 Petabyte of storage. All physical hardware. You can imagine all this is done by firing off a collection of API-calls to get this process orchestrated. The beauty of having this functionality capacity-by-code is that you can incorporate it into software features, such as vSphere HA and DRS. An upcoming new feature is Elastic DRS. In short, the ability to scale out and scale the cluster with physical hardware whenever workload demand requires it. I will provide a more in-depth view once we release this new feature. ================================================================================ Title: vBrownBag Techtalks VMworld Call for Papers now open URL: https://frankdenneman.ai/2018-04-26-vbrownbag-techtalks-vmworld-call-papers-now-open/ Date: 2018-04-26 Although the selection process of the submitted VMworld 2018 sessions is still ongoing, vBrownbag announced their call for papers. As Duncan mentioned in his Call for paper article ‘Good luck, and remember: if you don’t end up getting selected, submit the proposal to a VMUG near you instead. They are always begging for community sessions.’ Think about signing up for the vBrownbag as well. Since last year all the vBrownbag sessions are published in the content catalog. Thus your session is visible for all 23.000+ attendees. Go right ahead and fill out this form. ================================================================================ Title: The public Shaming of Resource Pool-as-a-Folder User URL: https://frankdenneman.ai/2018-04-25-public-shaming-resource-pool-folder-user/ Date: 2018-04-25 Yesterday there was some public shaming done of Antony Spiteri. He was outed that he was using vSphere resource pool as folders. https://twitter.com/davidhill_co/status/988797652346245126 A funny thread and he truly deserved all the public shaming by the community members ;). All fun aside, using resource pools as folders are not recommended by VMware. As I described in the new vSphere 6.5 DRS white paper available at vSphere central: Correct use: Resource pools are an excellent construct to isolate a particular amount of resources for a group of virtual machines without having to micro-manage resource setting for each individual virtual machine. A reservation set at the resource pool level guarantees each virtual machine inside the resource pool access to these resources. Depending on the activity of these virtual machines these virtual machines can operate without any contention. Incorrect use: Resource pools should not be used as a form of folders within the inventory view of the cluster. Resource pools consume resources from the cluster and distribute these amongst its child objects within the resource pool; this can be additional resource pools and virtual machines. Due to the isolation of resources, using resource pools as folders in a heavily utilized vSphere cluster can lead to an unintended level of performance degradation for some virtual machines inside or outside the resource pool. Understanding this behavior allows you to design a correct resource pool structure. Currently, I’m working on a new vSphere DRS Resource Pool white paper which sheds some new light on the distribution of resources under normal conditions and under load (the Resource Pool Pie Paradox). I will keep you posted! ================================================================================ Title: Public Speaking Schedule URL: https://frankdenneman.ai/2018-02-21-public-speaking-schedule/ Date: 2018-02-21 The VMUG season has started, and I have a few speaking sessions at various events. I thought it might be convenient to list the events and topics: Date: February, 22 Organization: North East UK VMUG Location: Newcastle Topic: VMware Cloud on AWS from a resource management perspective Date: March, 7 Organization: Swiss-French VMUG Location: Lausanne Switzerland Topic: VMware Cloud on AWS from a resource management perspective Date: March, 8 Organization: Swiss-German VMUG Location: Zurich Switzerland Topic: VMware Cloud on AWS from a resource management perspective Date: March, 20 Organization: Dutch VMUG Location: Den Bosch Netherlands Topic: vSphere Resource Kit Double-Hour Session 1: vSphere 6.5 Host Resource Deep Dive with Niels Hagoort Session 2: vSphere 6.5 Clustering Deep Dive with Duncan Epping Date: March 29, Organization: Virtual VMUG Location: Online Topic: VMware Cloud on AWS from a resource management perspective Date: April 10, Organization: Turkey VMUG Location: Istanbul, Turkey Topic: VMware Cloud on AWS from a resource management perspective Date: May 24 Organization: Czech Republic VMUG Location: Prague Topic: vSphere 6.5 Host Resource Deep Dive with Niels Hagoort Hope to see you there ================================================================================ Title: Virtually Speaking Podcast #67 Resource Management URL: https://frankdenneman.ai/2018-01-29-virtually-speaking-podcast-67-resource-management/ Date: 2018-01-29 Two weeks ago Pete Flecha (a.k.a. Pedro Arrow) and John Nicholson invited me to their always awesome podcast to talk about resource management. During our conversation, we covered both on-prem and the features of VMware Cloud on AWS that help cater the needs of your workload. Being a guest on this podcast is an honour and times flies talking to these two guys. Hope you enjoy it as much as I did. ================================================================================ Title: vSphere 6.5 DRS and Memory Balancing in Non-Overcommitted Clusters URL: https://frankdenneman.ai/2018-01-15-vsphere-6-5-drs-memory-balancing-non-overcommitted-clusters/ Date: 2018-01-15 DRS is over a decade old and is still going strong. DRS is aligned with the premise of virtualization, resource sharing and overcommitment of resources. DRS goal is to provide compute resources to the active workload to improve workload consolidation on a minimal compute footprint. However, virtualization surpassed the original principle of workload consolidation to provide unprecedented workload mobility and availability. With this change of focus, many customers do not overcommit on memory. A lot of customers design their clusters to contain (just) enough memory capacity to ensure all running virtual machines have their memory backed by physical memory. In this scenario, DRS behavior should be adjusted as it traditionally focusses on active memory use. vSphere 6.5 provides this option in the DRS cluster settings. By ticking the box “Memory Metric for Load Balancing” DRS uses the VM consumed memory for load-balancing operations. Please note that DRS is focussed on consumed memory, not configured memory! DRS always keeps a close eye on what is happening rather than accepting static configuration. Let’s take a closer look at DRS input metrics of active and consumed memory. Out-of-the-box DRS Behavior During load balancing operation, DRS calculates the active memory demand of the virtual machines in the cluster. The active memory represents the working set of the virtual machine, which signifies the number of active pages in RAM. By using the working-set estimation, the memory scheduler determines which of the allocated memory pages are actively used by the virtual machine and which allocated pages are idle. To accommodate a sudden rapid increase of the working set, 25% of idle consumed memory is allowed. Memory demand also includes the virtual machine’s memory overhead. Let’s use a 16 GB virtual machine as an example of how DRS calculates the memory demand. The guest OS running in this virtual machine has touched 75% of its memory size since it was booted, but only 35% of its memory size is active. This means that the virtual machine has consumed 12288 MB and 5734 MB of this is used as active memory. As mentioned, DRS accommodate a percentage of the idle consumed memory to be ready for a sudden increase in memory use. To calculate the idle consumed memory, the active memory 5734 MB is subtracted from the consumed memory, 12288 MB, resulting in a total 6554 MB idle consumed memory. By default, DRS includes 25% of the idle consumed memory, i.e. 6554 * 25% = +/- 1639 MB. The virtual machine has a memory overhead of 90 MB. The memory demand DRS uses in its load balancing calculation is as follows: 5734 MB + 1639 MB + 90 MB = 7463 MB. As a result, DRS selects a host that has 7463 MB available for this machine if it needs to move this virtual machine to improve the load balance of the cluster. Memory Metric for Load Balancing Enabled When enabling the option “Memory Metric for Load Balancing” DRS takes into account the consumed memory + the memory overhead for load balancing operations. In essence, DRS uses the metric Active + 100% IdleConsumedMemory. vSphere 6.5 update 1d UI client allows you to get better visibility in the memory usage of the virtual machines in the cluster. The memory utilization view can be toggled between active memory and consumed memory. Recently, Adam Eckerle on Twitter published a great article that outlines all the improves of vSphere 6.5 Update 1d. Go check it out. Animated Gif courtesy of Adam. When reviewing the cluster it shows that the cluster is pretty much balanced. When looking at the default view of the sum of Virtual Machine memory utilization (active memory). It shows that ESXi host ESXi02 is busier than the others. However since the active memory of each host is less than 20% and each virtual machine is receiving the memory they are entitled to, DRS will not move virtual machines around. Remember, DRS is designed to create as little overhead as possible. Moving one virtual machine to another host to make the active usage more balanced, is just a waste of compute cycles and network bandwidth. The virtual machines receive what they want to receive now, so why take the risk of moving VMs? But a different view of the current situation is when you toggle the graph to use consumed memory. Now we see a bigger difference in consumed memory utilization. Much more than 20% between ESXi02 and the other two hosts. By default DRS in vSphere 6.5 tries to clear a utilization difference of 20% between hosts. This is called Pair-Wise Balancing. However, since DRS is focused on Active memory usage, Pair-Wise Balancing won’t be activated with regards to the 20% difference in consumed memory utilization. After enabling the option “Memory Metric for Load Balancing” DRS rebalances the cluster with the optimal number of migrations (as few as possible) to reduce overhead and risk. Active versus Consumed Memory Bias If you design your cluster with no memory overcommitment as guiding principle, I recommend to test out the vSphere 6.5 DRS option “Memory Metric for Load Balancing”. You might want to switch DRS to manual mode, to verify the recommendations first. ================================================================================ Title: Explainer on #Spectre & #Meltdown by Graham Sutherland URL: https://frankdenneman.ai/2018-01-05-explainer-spectre-meltdown-graham-sutherland/ Date: 2018-01-05 Sometimes you stumble across a brilliant Twitter thread, so good, that it should never be lost. Graham Sutherland (@gsuberland) helped the world in understanding the Spectre and Meltdown bugs. I’m publishing his tweet thread in text form as this is just the best explanation of the bugs I’ve seen. Please note that VMware has released its response for Bounds-Check Bypass (CVE-2017-5753), Branch Target Injection (CVE-2017-5715) & Rogue Data Cache Load (CVE-2017-5754) - AKA Meltdown & Spectre. https://blogs.vmware.com/security/2018/01/vmsa-2018-0002.html Disclaimer: All text below is produced by Graham Sutherland, I’m not taking any credits for this work https://twitter.com/gsuberland/status/948907452786933762 Explainer on #Spectre & #Meltdown: When a processor reaches a conditional branch in code (e.g. an ‘if’ clause), it tries to predict which branch will be taken before it actually knows the result. It executes that branch ahead of time - a feature called “speculative execution”. The idea is that if it gets the prediction right (which modern processors are quite good at) it’ll already have executed the next bit of code by the time the actually-selected branch is known. If it gets it wrong, execution unwinds back and the correct branch is executed instead. What makes the processor so good at branch prediction is that it stores details about previous branch operations, in what’s called the Branch History Buffer (BHB). If a particular branch instruction took path A before, it’ll probably take path A again, rather than path B. What makes this interesting is that code is executed *speculatively*, before the result of a conditional statement has completed. That conditional statement could be security-critical. Thankfully the processor is (mostly) smart enough to roll back any side-effects of execution. There are two important exclusions to the rollback of side-effects: cache and branch prediction history. These generally aren’t rolled back because speculative execution is a performance feature, and rolling back cache and BHB contents would generally hurt performance. There are three ways to exploit this behaviour. The Spectre paper describes the first two exploits, with the following results: 1. Kernel memory disclosure from userspace on bare metal. 2. Kernel memory disclosure of the VM host/hypervisor from kernelspace in a VM. The first exploit works by getting the kernel to execute some carefully written attacker-specified code which contains an array bounds check followed by an array read, where the read index is controlled by an attacker. This sounds like a big ask, but it’s not thanks to JIT. On Linux, Extended Berkley Packet Filter (eBPF) allows users to write socket filters from usermode which get JIT compiled by the kernel in order to efficiently filter packets on a socket. The details aren’t important, but it means an attacker can get the kernel to execute code. The exploit involves writing eBPF code which compiles to the following steps: 1. Allocate two fixed-size arrays 2. Bounds-check the user-provided index 3. If ok, read from the array1 at that index 4. Compute another index from 1 bit of the result 5. Read from array2 at that index There’s actually a step before 5, which is “bounds check the read to array2”, but we never intend to do an out-of-bounds read here, so it’s irrelevant. I omitted it because I ran out of characters. In terms of “real” execution, this code always terminates at step 2 when the user passes an out-of-bounds index for array1. But if the processor’s branch predictor assumes that check will succeed, it’ll speculatively execute the out-of-bounds read in step 3, and continue to 5. Here’s the clever bit. In step 4 we take the value we got from the out-of-bounds read (which we wouldn’t normally have access to) and use one bit from it to select a particular memory address (array index) to read. If b=0 it reads index 0x200; if b=1 it reads index 0x300. This ensures that the memory at either index 0x200 or index 0x300 is now cached. The CPU then realises that the bounds check in step 2 failed, so it unwinds back to that branch. However, the data from step 5 is still cached! We can then go in and read the data at 0x200 and 0x300 and see which is cached by measuring how quick the read is. Once we know which index was cached we can directly infer one bit of kernel memory, based on the index selection from step 4. There are some details as to how the cache needs to be primed before this attack, but it is possible to do this whole process in a loop and dump kernel memory from unprivileged userspace. The second attack described in the Spectre paper involves poisoning the branch prediction history to trick the processor into speculatively executing code at an attacker-specified address, leading to further cache attacks as described above. By performing a carefully selected sequence of indirect jumps, an attacker can fill up the branch prediction history in a way that allows the attacker to select which branch will be speculatively executed when performing an indirect jump. This can be very powerful. If I know there’s a piece of code in kernel space that exhibits similar behaviour to our eBPF example from before, and I know what the address of that code is, I can indirectly jump to that code and the CPU will speculatively execute it. If you’ve done exploitation before, you’ll probably recognise this as being similar to a ROP gadget. We’re looking for a sequence of code in kernel space that happens to have the right sequence of instructions to leak information via cache. Keep in mind that the execution is speculative only - the processor will later realise that I didn’t have the privilege to jump to that code and throw an exception. So the target code has to leak kernel data via cache side-channels like before. You’ll also notice that we need to know address of the target kernel code. With KASLR this isn’t so easy. Project Zero’s writeup explains how KASLR can be defeated using branch prediction and caching as side-channels, so I won’t go into the details here. https://googleprojectzero.blogspot.co.uk/2018/01/reading-privileged-memory-with-side.html What makes this extra powerful is that it works across VM boundaries too. Instead of a traditional indirect jump (e.g. jmp eax), we can use the vmcall instruction to speculatively execute code within the VM host’s kernel in the same way we would our VM’s kernel. Finally, there’s the third approach. This involves a flush+reload cache attack against kernel memory, similar to the first variant of the attack but without requiring kernel code execution - it can all be done from usermode. The idea is that we try to read kernelspace memory using a mov instruction, then perform a secondary memory read with an address based on the value that was read. If you’re thinking the first mov will fail because we’re in usermode and can’t read kernel addresses, you’re right. The trick is that the microarchitectural implementation of mov contains the memory page privilege level check, which itself is a branch instruction. The processor may speculatively execute that branch like any other. So, if you can outrun the interrupt, you can speculatively execute some other instruction that loads data into cache based on the value read from kernelspace. This then becomes a cache attack like the previous tricks. And that’s just about it. For full details I recommend checking out the two papers, as well as the Project Zero writeup I linked above. https://spectreattack.com/spectre.pdf https://meltdownattack.com/meltdown.pdf Thanks Graham for this excellent explanation! ================================================================================ Title: Free vSphere 6.5 Host Resources Deep Dive E-Book URL: https://frankdenneman.ai/2017-11-07-free-vsphere-6-5-host-resources-deep-dive-ebook/ Date: 2017-11-07 In June of this year, Niels and I published the vSphere 6.5 Host Resources Deep Dive, and the community was buzzing. Twitter exploded, and many community members provided rave reviews. This excitement caught Rubriks attention, and they decided to support the community by giving away 2000 free copies of the printed version at VMworld. The interest was overwhelming, before the end of the second signing session in Barcelona we ran out of books. A lot of people reached out to Rubrik and us to find out if they could get a free book as well. This gave us an idea, and we sat down with Rubrik and the VMUG organization to determine how to cater the community. We are proud to announce that you can download the e-book version (PDF only) for free at rubrik.com. Just sign up and download your full e-book copy here. Spread the word! And if you like, thank @Rubrik and @myVMUG for their efforts to help the VMware community advance. https://www.youtube.com/watch?v=a4spq5B4wtg ================================================================================ Title: What if the VM Memory Config Exceeds the Memory Capacity of the Physical NUMA Node? URL: https://frankdenneman.ai/2017-10-05-vm-memory-config-exceeds-memory-capacity-physical-numa-node/ Date: 2017-10-05 This week I had the pleasure to talk to a customer about NUMA use-cases and a very interesting config came up. They have a VM with a particular memory configuration that exceeds the ESXi host NUMA node memory configuration. This scenario is covered in the vSphere 6.5 Host Resources Deep Dive, excerpt below. Memory Configuration The scenario described happens in multi-socket systems that are used to host monster-VMs. Extreme memory footprint VMs are getting more common by the day. The system is equipped with two CPU packages. Each CPU package contains twelve cores. The system has a memory configuration of 128 GB in total. The NUMA nodes are symmetrically configured and contain 64 GB of memory each. However, if the VM requires 96 GB of memory, a maximum of 64 GB can be obtained from a single NUMA node. This means that 32 GB of memory could become remote if the vCPUs of that VM can fit inside one NUMA node. In this case, the VM is configured with 8 vCPUs. The VM fits from a vCPU perspective inside one NUMA node, and therefore the NUMA scheduler configures for this VM a single virtual proximity domain (VPD) and a single a load-balancing group which is internally referred to as a physical proximity domain (PPD). Example Workload Running a SQL DB on this machine resulted in the following local and remote memory consumption. The VM consumes nearly 64 GB on its local NUMA node (clientID shows the location of the vCPUs) while it consumes 31 GB of remote memory. In this scenario, it could be beneficial to the performance of the VM to rely on the NUMA optimizations that exist in the guest OS and application. The VM advanced setting numa.consolidate = FALSE instructs the NUMA scheduler to distribute the VM configuration across as many NUMA nodes as possible. In this scenario, the NUMA scheduler creates 2 load-balancing domains (PPDs) and allows for a more symmetrical configuration of 4 vCPUs per node. Please note that a single VPD (VPD0) is created and as a result, the guest OS and the application only detect a single NUMA node. Local and remote memory optimizations are (only) applied by the NUMA scheduler in the hypervisor. Whether or not the application can benefit from this configuration depends on its design. If it’s a multi-threaded application, the NUMA scheduler can allocate memory closes to the CPU operation. However, if the VM is running a single-threaded application, you still might end up with a lot of remote memory access, as the physical NUMA node hosting the vCPU is unable to provide the memory demand by itself. Test the behavior of your application before making the change to create a baseline. As always, use advanced settings only if necessary! ================================================================================ Title: A vSphere Focused Guide to the Intel Xeon Scalable Family - Memory Subsystem URL: https://frankdenneman.ai/2017-10-03-vsphere-focused-guide-intel-xeon-scalable-family-memory-subsystem/ Date: 2017-10-03 The Intel Xeon Scalable Family introduces a new platform (Purley). The most prominent change regarding system design is the memory subsystem. More Memory Bandwidth and Consistency in Speed The new memory subsystem supports the same number of DIMMs per CPU as the previous models. However, it’s wider and less deep. What I mean by that is that the last platform (Grantley) supported up to three DIMMs per channel (DPC) and made use of four channels. In total, the Grantley platform supported up to twelve DIMMs per CPU. Purley increases the number of channels from four to six but reduces the numbers of supported DIMMs per channel from three to two. Although this sounds like a potato, potato; tomato, tomato discussion it provides a significant increase in bandwidth while ensuring consistency in speed during a scaling up exercise. Let’s take a closer look. DIMMs per Memory Channel Depending on the DIMM slot configuration of the server board, multiple DIMMs are supported per channel. The E5-2600 V-series supports up to 3 DIMMs per channel (3 DPC). Using more DIMMs per channel provides the largest capacity, but unfortunately, it impacts the operational speed of memory. A DIMM groups memory chips into ranks. DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). With the addition of each rank, the electrical load on the channel increases. And as more ranks are used in a memory channel, memory speed drops restricting the use of additional memory. Therefore in certain configurations, DIMMs will run slower than their listed maximum speeds. This reduction in speed occurs when 3 DIMMs per channel is used. RDIMM 1 DPC 2 DPC 3 DPC LRDIMM 1 DPC 2 DPC 3 DPC Source Cisco 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Cisco PDF Dell 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Dell.com Fujitsu 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 1866 MHz Fujitsu PDF HP 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2400 MHz* HP PDF Performance Drop 0 0 28% 0 0 12%/28% * HP claims no reduction of speed due to proprietary memory technology. I have not tested this. Moving to 2 DIMMs per Channel Configuration The Purley platform avoids this pitfall by reducing the supported number of DIMMs per channel. It supports up to 2 DIMMs per channel, maintaining the same performance regardless the number of DIMMs per channel. However, reducing the number of DIMMs supported per channel severely impacts the total supported memory capacity per CPU. Intel solves this by adding more channels to the memory controller. For some organizations, this change can result in dropping the requirement of obtaining LRDIMMs. Some organizations avoid the steep performance reduction by purchasing the (more) expensive LRDIMMs, 2 DPC configurations will not affect the performance characteristics of the memory modules. Six Memory Channel Support The Purley platform supports up to six channels per CPU. As a result, the bandwidth increases and the support for high capacity memory systems remains. By default, the new Xeon CPU supports up to 768 GB. Please review part 1 of this series in which covers the high memory capacity optimization option (M-suffix). If all six channels are populated with DIMMs, the CPU interleaves memory access across the multiple memory channels. When creating a 1 DIMM per channel (1 DPC) configuration, the CPU forms one region (Region 0) and interleaves the memory access. Theoretically, this multiplies the data rate by exactly the number of channels present. A 2666 MT/s DIMM has a theoretical peak transfer rate of 21,300 MB/s. If populating all six DIMM slots, the memory controller accesses each module sequentially. Instead of writing all the data to a single DIMM, the data is written across the modules in one region in an alternating pattern, leveraging each channel bandwidth separately. That means that the memory controllers of a single Xeon CPU have access to a combined bandwidth of 127,800 MB/s. In theory, that means that a dual Xeon system has access to 256 GB per second (21,300 MB/s x 6 channels x 2 sockets). In theory! This all depends on the type of workload and the compute power that drives the workload. The Xeon’s cores have direct access to the six channels in the CPU package. One thread can never obtain 256 GB due to the interconnect and the raw power it can produce to feed the channels. Anandtech has an excellent write-up about this behavior. Memory Configuration As a result of an increase of channels and the design consideration of populating every DIMM slot to create a 1 DPC or 2 DPC configuration, a new vSphere system will likely have a different memory capacity configuration than your previous systems (Inform your standards commission). Please note that the table lists the memory configuration of a single NUMA node. 6 x DIMM 16 GB 32 GB 64 GB 128 GB 1 DPC 96 GB 192 GB 384 GB 768 GB 2 DPC 192 GB 384 GB 768 GB 1536 GB * * M-suffix Xeon CPU required Please note that the table lists the memory configuration of a single NUMA node. Dual CPU systems is the most common configuration for vSphere servers. That means that you can expect the major system integrators such as DELL and HP to offer the following configurations: Dual CPU Socket 16 GB 32 GB 64 GB 128 GB 1 DPC 192 GB 384 GB 768 GB 1536 GB 2 DPC 384 GB 768 GB 1536 GB 3072 GB * For completeness sake, the next table shows the configuration of a v4 system with a maximum of 2-DPC. Very familiar configuration numbers, I guess we just need to get used to the new configuration standards such as 384, 768 and 1536 GB per system. Dual CPU Socket v4 16 GB 32 GB 64 GB 128 GB 1 DPC 128 GB 256 GB 512 GB 1024 GB 2 DPC 256 GB 512 GB 1024 GB 2048 GB * M-suffix Xeon CPU required. vSphere 6.5 supports up to 12 TB per host. As a result, the entire range of Intel Xeon Scalable CPU with the extended memory feature is fully supported (8 CPUs x 1536 GB). Interesting data point, the vSphere 6.5 Configurations Maximum Guide started to list a maximum number of NUMA nodes per system. This limit is set to 16. The Intel Scalable Xeon supports sub-NUMA clustering (similar to Cluster-on-Die functionality), splitting up the CPU package into two NUMA nodes. As a result, vSphere 6.5 would support a system equipped with 8 Intel Xeon Platinum 8176M Processors each fully loaded with 1.5 TB of memory and configured with sub-NUMA clustering. This setup would create one system, offering 16 NUMA nodes each fitted with 768 GB of local memory. Take Caution of 8 DIMM System Board Designs The introduction of Purley forces system integrators to redesign the system boards to support the new functionality. To support the full possibilities of the memory subsystem, system boards should be equipped with either 6 or 12 DIMM sockets. Some entry-level systems are designed with 8 DIMM slots. The Intel Xeon is designed to use the six channels when creating a region, this results in an unbalanced region design of 6+2. Region 0 consists of 6 DIMM slots, offering a theoretical peak transfer rate of 127,800 MB/s (when using 2666 GT/s), while region 1 offers 42,600 MB/s. This will result in inconsistent performance, something to definitely to avoid. Thus it’s recommended to equip these systems with the six-channel configuration in mind, order these systems and only populate the first six DIMM slots per CPU. Interconnect The performance of a dual CPU system can be impacted by the interconnect between the CPU packages if you span VMs across the NUMA nodes (Wide-VMs). Purley introduces a new interconnect called the UltraPath Interconnect (UPI) and replaces the QuickPath Interconnect. The next article in this series provides an in-depth look at the UPI. ================================================================================ Title: A vSphere Focused Guide to the Intel Xeon Scalable Family URL: https://frankdenneman.ai/2017-09-26-vsphere-focused-guide-intel-xeon-scalable-family/ Date: 2017-09-26 Intel released the much-anticipated Skylake Server CPU this year. Moving away from the E5-2600-v moniker, Intel names the new iteration of its server CPU the Intel Xeon Scalable Family. On top of this it uses precious metal categories such as Platinum and Gold to identify different types and abilities. Upholding the tradition, the new Xeon family contains more cores than the previous Xeon version. The new top-of-the-line CPU offers 28 cores on a single processor die, memory speeds are now supported up to 2666 MHz. However, the biggest appeal for vSphere datacenters is the new “Purley” platform and its focus on increasing bandwidth between possibly every component possible. In this series, we are going to look at the new Intel Xeon Scalable family microarchitecture and which functions help to advance vSphere datacenters. NUMA and vNUMA Focus Instead of solely listing the speeds and feeds of the new architecture, I will be reviewing the new functionality with considerations of today’s vSphere VM configuration landscape. In modern vSphere datacenters, small VMs and large VMs co-exists with a single server. Many VMs consume the interconnect between physical CPUs. Some VMs span multiple NUMA nodes (Wide-VMs) while others fit inside a single physical NUMA node. The VMkernel NUMA scheduler attempts to optimize local memory consumption as much as possible. Sometimes remote memory is unavoidable. Consolidation ratios increase each year, hence the focus on the interconnect. Yet, single-threaded applications are still prevalent in many DC’s. Therefore single-core improvements will not be ignored in this series. Designing a system that is bound to run a high consolidation with a mix of small and large VMs is not an easy task. Rebranding The Xeon Scalable family introduces a new naming scheme. Gone are the names such as the E5-2630 v4, E5-4660 v4 or E7-8894 v4. Now Bronze, Silver, Gold, and Platinum class indicate the range of overall performance, Bronze representing the entry-level class CPU comparable to the previous E3 series, while Platinum class CPUs provide you the highest levels of scalability and most cores possible. As of today, Intel offers 58 different CPU types within the Xeon Scalable family, i.e., 2 Bronze CPUs, 8 Silver, 6 Gold 51xx, 26 Gold 61xx and 16 Platinum CPUs. Bronze Silver Gold 51xx Gold 61xx Platinum Scalability 2 2 2 2-4 2-8 Max Cores 8 12 14 22 28 Max Base Frequency (GHz) 1.7 2.6 3.6 3.5 3.6 Max Memory Speed (MHz) 2133 2400 2400 2666 2666 UPI* Links 2 2 2 3 3 UPI Speed (GT/s) 9.6 9.6 10.4 10.4 10.4 * The Xeon Scalable family introduces a new processor interconnect called the UltraPath Interconnect (UPI) and replaces the QuickPath Interconnect. The next article in this series provides an in-depth look at the UPI. Integrations and Optimizations Intel uses suffixes to indicate particular integrations or optimizations. Suffix Function Integration | Optimization Availability F Fabric Integrated Intel® Omni-Path Architecture Gold 61xx, Platinum M Memory Capacity 1.5 TB Support per Socket Gold 61xx, Platinum T High Tcase Extended Reliability (10-Year Use) Silver,Gold, Platinum Omni-Path Architecture The new Xeon family offers on-die Omni-Path Architecture that allows for 100 Gbps connectivity. In-line with the industry effort to remove as much “moving parts or components as possible the new architecture the signal is not being routed through the socket and motherboard but provides a direct connection to the processor. [caption id=“attachment_7085” align=“aligncenter” width=“676”] Image by ServeTheHome.com[/caption] The always excellent Serve The Home has published a nice article about the F-type Xeons. Unfortunately, the current vSphere version does not support the Omni-Path Architecture. Total Addressable Memory Intel hard-coded the addressable memory capacity on the CPU. As a result, non-M CPUs will not function if more than 768 GB of RAM is present in the DIMM sockets connected to its memory controllers. If you tend to scale-up your servers during their lifecycle, consider this limitation. If you are planning to run monster VMs that require more than 768 GB of RAM and want to avoid spanning it across NUMA nodes, consider obtaining “M” designation CPUs. Why wouldn’t you just buy M designated CPUs in the first place, you might wonder? Well, the M badge comes with a near-3K USD price hike. Comparing, 6142 (16 cores at 2.6 GHz) and 6140 (18 cores at 2.3 GHz) the list price for the 6142 is $2946, while the 6142M is $5949. The similar price difference for the 6140, vanilla style 6140 costs $2445, M-badge $5448. But with current RAM prices, we are talking about a minimum of $100.000 price tag for 1.5 TB of memory PER socket! Extended Reliability For specific use-cases, Intel provides CPUs with an extended reliability of up to 10 years. As you can imagine, these CPUs do not operate at top speeds. The fastest T enabled CPU runs at 2.6 GHz base frequency. Similar, but not identical CPU packaging When reviewing the spectrum of available CPUs, one noticeable thing is the availability of identical core count CPUs across the precious metals. For example, the 12 core CPU package. It’s available in Silver, Gold 51xx, Gold 61xx and Platinum. It’s available with an extended Tcase (Intel Xeon Silver 4116), and the Intel Xeon Gold 6126 is also available as 6126T and 6126F. One has to dig a little bit further to determine the added benefits of selecting a Gold version over a Silver version. Processor Silver 4116 Gold 5118 Gold 6126 Gold 6136 Gold 6146 Platinum 8158 Cores 12 12 12 12 12 12 Base Frequency (GHz) 2.10 2.30 2.60 3.00 3.20 3.00 Max Turbo Frequency (GHz) 3.00 3.20 3.70 3.70 4.20 3.70 TDP (W) 85 105 125 150 165 150 L3 Cache (MB) 16.5 16.5 19.25 24.75 24.75 24.75 # of UPI Links 2 2 3 3 3 3 Scalability 2S 4S 4S 4S 4S 8S # of AVX-512 FMA Units 1 1 2 2 2 2 Max Memory Size (GB) 768 768 768 768 768 768 Max Memory Speed (MHz) 2400 2400 2666 2666 2666 2666 Memory Channels 6 6 6 6 6 6 Part 2: Memory Subsystem available ================================================================================ Title: VMware Cloud on AWS Technical Overview URL: https://frankdenneman.ai/2017-08-29-vmware-cloud-aws-technical-overview/ Date: 2017-08-29 Please note that this information can be outdated due to the ongoing changes of this cloud service. Please consult the https://cloud.vmware.com/vmc-aws/roadmap for recent information about the latest release Yesterday we launched the VMware Cloud on AWS service. VMware Cloud on AWS allows you to run your applications across private, public, and hybrid cloud environments based on VMware vSphere, with optimized access to AWS services. The Cloud SDDC consists of vSphere, NSX and vSAN technology to provide you a familiar environment which can be managed an operated with your current tool and skill set. By leveraging bare-metal AWS infrastructure the Cloud SDDC can scale in an unprecedented way. VMware Cloud on AWS is a service and that means that we will not using product versions when we refer to the service. Instead we will be calling the first release the initial availability of the service. Any release after is referred to as future release. VMware Cloud on AWS is operated by VMware. In short that means that VMware is responsible for providing infrastructure resources, the customer is responsible for consuming the resources. This article explores the resource capacity of the Cloud SDDC at initial availability. Compute in VMware Cloud on AWS At initial availability, the VMware Cloud on AWS base cluster configuration contains four hosts. Each host is configured with 512GB of memory and contains dual CPUs. These CPUs are custom-built Intel Xeon Processor E5-2686 v4 CPU. Each CPU contains 18 cores running at 2.3GHz, resulting in a physical cluster core count of 144. Please note, that VMware Cloud on AWS uses a single, fixed host configuration; the option to add components to the host configuration is not offered at this time. However, the scale-out model enables expansion to up to 16 hosts, resulting in 576 CPU cores and 8TB of memory. vSphere DRS and vSphere HA are enabled and are configured to provide the best availability and resource utilization. vSphere DRS is full automated and the migration threshold is set to the default vSphere DRS level to avoid excessive vSphere vMotion operations. High availability of cluster resources is provided by vSphere HA and Auto remediation hardware. vSphere High Availability is used to guarantee enough resources for restarting VMs during an ESXi host failure. The ESXi hosts are monitored and in the event of a failure, the VMs on a failed host are restarted on alternative ESXi hosts in the cluster. To maximize productivity while minimizing overhead, the vSphere HA settings of the cluster is configured to tolerate the equivalent of one ESXi host failure (25% percentage-based admission control policy). The host isolation response is set to power off and restart the VMs. Host failures remediation is the responsibility of VMware. If a host fails permanently, VMware replaces this ESXi host without user intervention. Automatic remediation of failed hardware eliminates the impact of long-term resource reduction of a permanent host failure. The Cloud SDDC is configured with two DRS resource pools. One resource pool contains the management VMs to operate the Cloud SDDC, while the other top-level resource pool is created to manage customer workloads. Customers have the option to create child resource pools. Storage in VMware Cloud on AWS The SDDC cluster includes a vSAN all-flash array and each host provides a total of 10TB of raw capacity for VMs to consume. A default Cloud SDDC cluster provides 40TB of raw capacity. The capacity consumption of the VM depends on the configured storage policy. By default, a RAID-1 Fault Tolerance Method is applied, but customers can create storage profiles that provide less overhead, such as RAID-5 or RAID-6 Failure Tolerance Method. Please note that for using RAID-6 Failure Tolerance Method a minimum of 6 hosts are required inside the Cloud SDDC cluster. Each ESXi host contains 8 NVMe devices. These 8 devices are distributed across two vSAN disk groups. Within a disk group, the write-caching tier leverages one NVMe device with 1.7TB of storage; the storage capacity tier leverages the other three NVMe devices with a combined 5.1TB of storage. Storage Encryption Datastore-level encryption with vSAN encryption, or VM-level encryption with vSphere VM encryption, is not available at initial availability of VMware Cloud on AWS. To provide data security, all local storage NVMe devices are encrypted at the firmware level by AWS. The encryption keys are managed by AWS and are not exposed to or controlled by VMware or VMware Cloud on AWS customers. Cloud SDDC Configuration At initial availability, the Cloud SDDC is restricted to a single AWS region and availability zone (AZ). Failed hardware can be automatically detected, and automated remediation enables failed host to be automatically replaced by other ESXi hosts. If necessary the VSAN datastore is automatically rebuilt without user intervention. In future VMware Cloud on AWS releases, through the partnership of VMware and AWS, multi-AZ availability will be possible for the first time ever, by stretching the cluster across two AZs in the same region. With this groundbreaking offering, refactoring of traditional applications will no longer be required to obtain high availability on the AWS infrastructure. Instead, synchronous write replication will be leveraged across AZs, resulting in a recovery point objective (RPO) of zero and a recovery time objective (RTO) that depends on the vSphere HA restart. Networking in VMware Cloud on AWS VMware Cloud on AWS is built around NSX. It’s optimized to provide VM networking in the Cloud SDDC, while abstracting the Amazon Virtual Private Cloud (VPC) networks. It enables ease of management by providing logical networks to VMs and automatically connecting new hosts to logical and VMkernel networks as clusters are scaled out. At initial availability, users connect to VMware Cloud on AWS via a layer 3 VPN connection. Future releases of VMware Cloud on AWS, however, will support AWS Direct Connect and allow cross-cloud vSphere vMotion operations. An IPsec layer 3 VPN is set up to securely connect the on-premises vCenter Server instance with the management components running on the in-cloud SDDC cluster. A separate IPsec layer 3 VPN is set up to create connectivity between the on-premises workloads and the VMs running inside the in-cloud SDDC cluster. NSX is used for all networking and security and is decoupled from Amazon VPC networking. The compute gateway and DLR are pre-configured as part of the prescriptive network topology and cannot be changed by the customer. Customers provide only their own subnets and IP ranges. VMware Cloud on AWS ready for your workload VMware Cloud on AWS provides you cloud resources that can be consumed by using your current skill set and tool set. Each cloud SDDC provides state-of-the-art resources that can run the most demanding applications of today. The best enterprise software combined with the best cloud operator in the world allows you to run and scale your data center in an unprecedented way. For more information, go to https://cloud.vmware.com/vmc-aws/resources ================================================================================ Title: Get your Free Book at VMworld URL: https://frankdenneman.ai/2017-08-25-get-free-book-vmworld/ Date: 2017-08-25 At VMworld, the presenters of the following sessions will be giving away free copies of the Host Deep Dive book to the audience. Saturday Performance Bootcamp Mark Achtemichuk Saturday, Aug 26, 8:00 a.m. - 5:00 p.m. More information about pre-VMworld Performance Bootcamp Sunday An Introduction to VMware Software-Defined Storage [STO2138QU] Lee Dilworth, Principal Systems Engineer, VMware Sunday, Aug 27, 4:00 p.m. - 4:30 p.m. | Oceanside C, Level 2 Monday A Deep Dive into vSphere 6.5 Core Storage Features and Functionality [SER1143BU] Cody Hosterman, Technical Director–VMware Solutions, Pure Storage Cormac Hogan, Director - Chief Technologist, VMware Monday, Aug 28, 11:30 a.m. - 12:30 p.m. | Mandalay Bay Ballroom G, Level 2 Extreme Performance Series: Benchmarking 101 [SER2723BUR] Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Monday, Aug 28, 4:00 p.m. - 5:00 p.m. | Mandalay Bay Ballroom B, Level 2 Maximum Performance with Mark Achtemichuk [VIRT2368GU] Mark Achtemichuk, Staff Engineer, Performance, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Reef E, Level 2 The Top 10 Things to Know About vSAN [STO1264BU] Duncan Epping, Chief Technologist, VMware Cormac Hogan, Director - Chief Technologist, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Mandalay Bay Ballroom H, Level 2 VMware vSAN: From 2 Nodes to 64 Nodes, Architecting and Operating vSAN Like a VCDX for Scalability and Simplicity [STO2114BU] Greg Mulholland, Principal Systems Engineer, VMware Jeff Wong, Customer Success Architect, VMware Monday, Aug 28, 5:30 p.m. - 6:30 p.m. | Surf E, Level 2 Tuesday Extreme Performance Series: Performance Best Practices [SER2724BU] Reza Taheri, Principal Engineer, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Tuesday, Aug 29, 2:30 p.m. - 3:30 p.m. | Oceanside D, Level 2 Wednesday vSphere 6.5 Host Resources Deep Dive: Part 2 [SER1872BU] Frank Denneman, Senior Staff Architect, VMware Niels Hagoort, Owner, HIC (Hagoort ICT Consultancy) Wednesday, Aug 30, 8:30 a.m. - 9:30 a.m. | Breakers E, Level 2 Extreme Performance Series: Benchmarking 101 [SER2723BUR] Joshua Schnee, Senior Staff Engineer @ VMware Performance, VMware Mark Achtemichuk, Staff Engineer, Performance, VMware Wednesday, Aug 30, 8:30 a.m. - 9:30 a.m. | Lagoon L, Level 2 vSAN Networking and Design Best Practices [STO3276GU] John Nicholson, Senior Technical Marketing Manager, VMware Wednesday, Aug 30, 11:30 a.m. - 12:30 p.m. | Reef C, Level 2 vSAN Hardware Deep Dive Panel [STO1540PU] Ed Goggin, Staff Engineer 2, VMware David Edwards, Principal Engineer, Director Solutions, Resurgent Technology Ken Werneburg, Group Manager Technical Marketing, VMware Jeffrey Taylor, Technical Director, VMware Ron Scott-Adams, Hyper-Converged Systems Engineer, VMware Wednesday, Aug 30, 1:00 p.m. - 2:00 p.m. | Mandalay Bay Ballroom D, Level 2 A Closer Look at vSAN Networking Design and Configuration Considerations [STO1193BU] Cormac Hogan, Director - Chief Technologist, VMware Andreas Scherr, Senior Solution Architect, VMware Wednesday, Aug 30, 4:00 p.m. - 5:00 p.m. | Mandalay Bay Ballroom G, Level 2 Thursday Virtual Volumes Technical Deep Dive [STO2446BU] Patrick Dirks, Sr. Manager, VMware Pete Flecha, Sr Technical Marketing Architect, VMware Thursday, Aug 31, 10:30 a.m. - 11:30 a.m. | Oceanside B, Level 2 Book Signing We will be doing two book signing sessions as well. At the Rubrik booth #412 on Monday, Aug 28, 2:00 p.m. - 3:00 p.m. At the VMworld Book store on Tuesday, Aug 29, 11:30 a.m. - 12:00 p.m. Or just feel free to approach us when you see us walking by. ================================================================================ Title: Register now for VMware Cloud on AWS Technical Deep Dive Session URL: https://frankdenneman.ai/2017-08-15-register-now-vmware-cloud-aws-technical-deep-dive-session/ Date: 2017-08-15 I noticed that our technical deep dive session on VMware Cloud on AWS was added to the content catalog of VMworld. In this session, Ray Budavari and I will cover the VMware Cloud on AWS infrastructure in detail. For the first time, we are allowed to uncover details about the host configuration, the vSAN infrastructure and of course network topology. We explore advanced features such as Elastic DRS and Autoremediation HA. The last 15 minutes of our session allows you to ask questions about VMC. Please register if you don’t want to miss this session. Both Ray and I have a full schedule, therefore we are unable to schedule a repeat of this session during VMworld US. Session details: VMware Cloud on AWS: A Technical Deep Dive [LHC2384BU] Frank Denneman, Senior Staff Architect, VMware Ray Budavari, Senior Staff Technical Product Manager, VMware Tuesday, Aug 29, 3:30 p.m. - 4:30 p.m. Session Type: Breakout Session Track: Integrate Public Clouds Integrate Public Clouds: Leverage Hybrid Clouds Product and Topics: NSX, vCenter, vSAN, vSphere Technical Level: Technical – Advanced Session Hashtag: #LHC2384BU ================================================================================ Title: VMware Cloud on AWS - Predictable Capacity Provisioning URL: https://frankdenneman.ai/2017-07-26-vmware-cloud-aws-predictable-capacity-provisioning/ Date: 2017-07-26 In preparation for the VMworld Session LHC2971BU - Managing Your Hybrid Cloud with VMware Cloud on AWS which I’m co-presenting with Emad Younis, I asked the following question on Twitter: https://twitter.com/FrankDenneman/status/889841095768776704 And the number of answers were overwhelming. The stories were a bit underwhelming. Funny to see that we strive to automate every single step in the process. Guys like Alan, Luc, and William help the community to create scripted installs and configuration of the ESXi host. Creating a consistent, human-error free, rapid process. Shaving off valuable time of the time-consuming server provisioning process. Some organizations incorporate the vRealize suite to create a consistent user experience for the entire IT services portfolio. Interestingly enough, the overall lead time seems mostly impacted by internal acquisition processes. To give a few examples: https://twitter.com/j0sema/status/889945617312735232 https://twitter.com/tx_drewdad/status/889849858701524992 https://twitter.com/VTsnowboarder42/status/889995588669984772 https://twitter.com/PvdBree/status/889863788983455744 And the list goes on and on. In most organizations, the procurement process is rigid, well-defined process. However, the lead time of the acquisition process is either unpredictable and inconsistent. The overall message is that it cripples the agility of the IT organization. IT organizations need to react fast to the business needs. Resource management of current workload is difficult enough, figuring out what to expect in the upcoming months is challenging. Unfortunately, the introduction of new workload does not follow a linear demand curve. To cater the (possible) future needs of the customer, the order is either doubled in size, or onboarding of new workloads is gated. Either impacting the bottom-line of the company or the ability to facilitate IT services properly. https://twitter.com/BobbyFantast1c/status/889845274805448704 In essence, the CAPEX element of server resource acquisition massively impacts or hinders the execution ability of the IT organization. Strategizing CAPEX\OPEX is not a part of the core focus of many admins and architects, it does affect their means of execution. As demonstrated by the many tweets. With VMware Cloud on AWS, the host resource acquisition process shifts from CAPEX to OPEX. Removing the inconsistent and unpredictable procurement process, allowing for a faster, consistent and predictable method of providing compute, storage and networking resources. VMware Cloud on AWS (VMC) makes my resource-management heart beat faster. By leveraging the AWS operation model, the SDDC cluster running on the AWS infrastructure is resizable by a click of a button. Right-click on the cluster and select resize. Just select the number of hosts you want to add and within moments you will get new dedicated physical hardware added to your cluster. Ready to provide the resources your new workloads require. Resize means you can remove the resources as well, which in result your costs will go down as well. Due to the combined fleet management of AWS and VMC, the new ESXi hosts are fully configured and ready to welcome new workload. All VMkernel and logical networks are automatically configured and made available. The vSAN datastore is automatically expanded with the host-local NVMe flash devices provided by the new hosts. DRS detects the new physical resources and automatically rebalances the cluster, provided the most optimal resource availability. Elastic DRS and Autoremedation HA allows for an automatize method or adding and removing dedicated hardware resources, but these topics will be covered in a different article. From a resource management perspective, a mindset shift will happen. VMC allows you to reduce the time spent on infrastructure configuration and management and allows you to focus more on resource consumption. What cluster configuration is required in the upcoming months? What is my burst strategy? Unfortunately, I can’t go into detail as the service is not released yet. VMworld boasts an exciting line up of VMware Cloud on AWS sessions. I will be hosting a meet the expert on resource management at both VMworlds, sign up if you want to talk more about this exciting new technology ================================================================================ Title: Host Deep Dive First Major Milestone URL: https://frankdenneman.ai/2017-07-24-host-deep-dive-first-major-milestone/ Date: 2017-07-24 Exactly one month ago Niels and I published the VMware vSphere 6.5 Host Deep Dive and it is a major success. Within 30 days we sold over 4000 copies of the book. Building consistent high-performing ESXi hosts remains a strong focus point for the virtual community. The attention for the book is overwhelming. The hashtag #HostDeepDive felt like it was trending. Tweets from around the world letting us know the book arrived, from Brasil to New Zealand. https://twitter.com/DemitasseNZ/status/884599021980991488 The book seems to be a beloved companion during the summer holiday. Christian Mohn and Erik Bussink engaged in a competition to provide us the best vacation shot possible. Mohn: https://twitter.com/h0bbel/status/886158966018977793 Bussink https://twitter.com/ErikBussink/status/886501115201818624 Brad Tompkins of the VMUG organization joined the party by giving away twenty copies to the audience of the Indy VMUG last week. https://twitter.com/kmruddy/status/887293261714522112 Amazon awarded the book with many accolades, being the number one book in the network section and at one point it was in the top 25 of computer books overall. Quite an achievement! The absolute fantastic reviews help a lot. Thank you all for submitting a review! Due to the popularity of the book, Amazon offered us help with creating an ebook version of the book. They put their professional team to work two weeks ago, and we expect to have it online soon. Stay tuned! ================================================================================ Title: Kindle Ebook Host Deep Dive Available URL: https://frankdenneman.ai/2017-07-24-kindle-ebook-host-deep-dive-available/ Date: 2017-07-24 Funny enough I’ve just published an article announcing the major milestone of 4000 hard copies sold within the first month of release. I just received confirmation that the Kindle e-book version is available at the Amazon Kindle Store. It is scheduled to appear on various Amazon sites. Please check out your local Amazon for the best offer. Amazon US Amazon DE Amazon NL Amazon UK Amazon India The most popular e-book version of the cluster deep dive was Kindle, and therefore we focused on getting the Kindle e-book out as fast as possible. The professionals of Amazon Kindle Direct Publishing created an awesome e-book version. Go check it out. ================================================================================ Title: Virtually Speaking Podcast: Host Deep Dive URL: https://frankdenneman.ai/2017-07-15-virtually-speaking-podcast-host-deep-dive/ Date: 2017-07-15 Last Friday I had the honor to join Pete Flecha a.k.a. Pedro Arrow and John Nicholson on their always fantastic podcast Virtually Speaking. Together with Niels we talked about what it takes to write a book such as the VMware vSphere 6.5 Host Resources Deep Dive. Thanks John and Pete for having me on again. Check it out. https://soundcloud.com/virtuallyspeakingpodcast/episode-49-host-resources-deep-dive ================================================================================ Title: Exploring the Core Motivation of Writing a Book URL: https://frankdenneman.ai/2017-07-11-exploring-core-motivation-writing-book/ Date: 2017-07-11 More than a week ago Niels and I released the VMware vSphere 6.5 Host Resources Deep Dive and the community has welcomed it with open arms. The book is finding its way across the globe, from Argentina to New Zealand. To see the massive amounts of tweets praising the books brings us pride and joy. Over the last couple of days, I have received many inquiries what it takes to write a book and if I could provide some hints and tips. I thought it might be an interesting blog post. Three questions you need to ask yourself What is my core motivation for wanting to write a book? What are you willing to give up for pursuing this goal? Do I have the platform to launch the book and capture and maintain the attention of the book/brand? Core motivation The first question you need to ask yourself is why exactly you want to write a book? An answer often heard is not money or fame. It’s typically a charitable cause, such as to educate the community or a stepping-stone for one’s career. The last reason is by far the most likely one that will provide you the return on investment. People respond differently to you once they found out that you wrote a book. It shows dedication, it hints at mastery of a subject, it differentiates you from the rest. Nay-sayers will automatically point to the option of self-publishing, but they forget that the leading word in self-publishing is SELF. You have to do it all by yourself. If money is your answer, then I need to burst that bubble. The chances are that the same amount of time spent working at the local supermarket will be more profitable. What I’ve learned with publishing five books, is that the creation of a single page takes approximately 90 minutes. The sole exercise of writing 300 to 500 words does not take 90 minutes. It’s the second-guessing, the formatting and the phrasing that takes a lot of time. Once you write something down, thoughts will start to flow; they will lead to more questions, they will lead to second-guessing your initial idea. This leads you back to vendor collateral, academic papers, or testing in your lab. And you will hit writer’s block. 90 minutes is a good number to work with when you are in the planning phase of the book. We wrote 569 pages. 569 pages times 90 minutes equals 51210 minutes. That is 853,5 hours. Money This number leads to additional questions. But first, let’s answer the money question, this intertwines with the platform question. How many books does one sell? Duncan and I sold over 75.000 copies of the vSphere clustering deep dive series. I know that the early Bible of virtualization; VMware ESX Server, Advanced Technical Design Guide by Scott Herold and Ron Oglesby sold approximately 30.000 copies. Both are exceptions to the rule. Most successful self-published books sell between 500 and 1500 copies. Let’s say you earn 5 dollars per book, and you sell 1500 copies, you receive 7500 dollars before tax. If you spend 700 hours on the book, you will make a little over 10 dollars an hour. Self-publishing books provide more revenue to the author than using a publishing house. Published books by VMware Press or any other publisher house will get you into bookstores. Unfortunately, this will eat into your royalties. Deadline Those 700 hours need to come from somewhere, and because you are writing a tech book, you are bound to the time limit of the software version. It doesn’t make sense to publish a book about the previous version of the software, so you typically have one year of writing. If you have a day-time job, you need to spend times in the weekend and evening hours. Let’s say you keep the weekends for your friends and family and household chores. That leaves you with five evenings to write. If you write 3 hours a night, that means you are spending 233 consecutive workdays to write your book. Question yourself whether you or your loved ones will have this stamina? What are you willing to give up So that brings me to the second question, what are you willing to give up. I’m not saying divorce your spouse, but if you want to keep your family happy, you need to get the hours from something else. Gym time, drinking time, game time or sleep time. Typically all of the above, because sometimes you will get sick, or other responsibilities will get in the way. Going back to question one, is this worth the ˜10K you probably make? When you want to use it as a stepping stone for your career, the money is just a nice bonus. Platform If you want to educate the community, or you want to get more exposure, then you need to have a platform already in place. This platform can be a successful blog, a popular Twitter account or you are a regular on a podcast. Due to the maturity of this particular industry (virtualization) you need to be a regular in the community to have people accept your wisdom. There are a lot of people sharing their knowledge, some not always as correct as they believe. So ask yourself, why would anyone want to buy your book? Why should he believe you? Are you seen as an authority on a particular subject by the community? Just getting a book out and expect people will buy it because of the subject is, unfortunately, a thing of the past. There are a lot of books about virtualization on Amazon and people need an extra level of confirmation before they spend their money. Find your niche and share your knowledge! But, before spending a lot of time on writing a book and getting disappointed by the sales result ask yourself. Am I an authority on a certain topic and does the community share the same perception? A book can certainly help built this image. But in general, people already need to understand that you know what you are talking about. How can you become one? Publishing articles on your blog or LinkedIn will help. Appear on podcasts such as Virtually Speaking Podcast or vBrownbag. And speak! Speak a lot at local and neighboring VMUGs. Hone your skill, so that you can shine at VMworld. You need to harvest that popularity and keep riding that wave. That’s why you need to have that platform. Otherwise, it will be a short 15 minutes of fame. Popularity has a momentum. People will forget, and the attention to your new book will soon be pointed at another new thing. You need to provide a platform that can maintain that momentum. Write about the book, publish sections of the book on your blog, speak about it in podcasts. This allows you to create better and bigger things after your first book, maybe a second one. As with everything in life, nothing is self-contained. It’s always intertwined with other elements. This blog post is not to discourage you from writing a book, but hopefully, helps you prepare to launch a successful book. There is a lot of work that needs to be done before releasing your knowledge in the form of a book. Don’t let my words discourage you. To paraphrase Nike: Just don’t quit! ================================================================================ Title: Host Deep Dive Stickers and More URL: https://frankdenneman.ai/2017-07-03-host-deep-dive-stickers/ Date: 2017-07-03 Last week we released the VMware vSphere 6.5 Host Resources Deep Dive book and Twitter and Facebook exploded. We’ve seen some pretty bad-ass pictures on our Twitter feeds such as this one by Jamie Girdwood (@creamcookie) It’s always nice to hear some praise after spending more than 800 hours on something. (When writing and self-publish a book, expect to spend over 90 minutes on one page). Thanks! The top three most often heard questions were: When will you release an ebook version? Do you have any stickers? When is Niels joining VMware? When will you release an ebook version? We hope to get the ebook finalized after VMworld. Vacation time is coming up, and we also need to prep for VMworld (vSphere 6.5 Host Resources Deep Dive: Part 2 [SER1872BU]). It might happen sooner, but that depends on the process of creating an eBook itself. Unfortunately, it’s not as easy as sharing a PDF online. Please stay tuned. Do you have any stickers? We got you covered. We met up with our designer over at digitalmaterial.nl and explained our wishes. We received a lot of comments on the depth of the book. Such as the one from Duncan’s article Must have book: Host Resources Deep Dive: As most of you know, I wrote the Clustering Deepdive series together with Frank, which means I kinda knew what to expect in terms of level of depth. Kinda, as this is a whole new level of depth. I don’t think I have ever seen (for example) topics like NUMA or NIC drivers explained at this level of depth. If you ask me, it is fair to say that Frank and Niels redefined the term “deep dive”. So instead of snorkeling and hovering a bit below sea-level, we help you get into the depths of the material. What better way to express this than a divers helmet. We will bring 250 stickers to VMworld. First come first serve. If you can’t wait, download the 800 DPI PNG here and create one for yourself. White Background Transparent Background I think the design rocks, so much that Niels and I decided to put it on some t-shirts as well. We are not backed by a vendor, so we can’t give away shirts. Similar to the book, we kept the price low. We created two campaigns, one for the US and one for EU.This allows you to get the order as fast as possible. The shirts and hoodies come in various colors. When is Niels joining VMware? I don’t know, he should though! ================================================================================ Title: Why the Recent Reported Intel HT Bug is Not in Your Data Center URL: https://frankdenneman.ai/2017-06-26-recent-reported-intel-ht-bug-not-data-center/ Date: 2017-06-26 Yesterday I tweeted out the warning message about the HT bug of Skylake and Kaby Lake processors posted on debian.org. https://lists.debian.org/debian-devel/2017/06/msg00308.html My tweet got a LOT of retweets. A lot replied with concerns about their systems. I believe most Data Centers will not suffer from this bug as it is present on Skylake and Kaby Lake processors. What is the Bug? According to the warning: Unfixed Skylake and Kaby Lake processors could, in some situations, dangerously misbehave when hyper-threading is enabled. Disable hyper-threading immediately in BIOS/UEFI to work around the problem. Read this advisory for instructions about an Intel-provided fix. https://www.intel.com/content/www/us/en/processors/xeon/xeon-e3-1200v5-spec-update.html Unlikely Present in Your Data Center The reason why I believe most systems in data centers are not hit by this bug is that it solely applies to E3 Xeons from the Skylake microarchitecture. E3 CPUs are designed to operate in a single socket system, they have no QuickPath Interconnect. Therefore unable to create a symmetric multiprocessing system. The current E5 (dual-socket) system is based on the Broadwell microarchitecture. The Skylake microarchitecture is expected to appear within the next couple of months. According to the report, they will have the fix included when the product launched. If you are running a NUC in your lab, you might want to check to see whether your system might hit that bug http://ark.intel.com/products/codename/82879/Kaby-Lake http://ark.intel.com/products/codename/37572/Skylake The link will forward you to a perl script that can help detect if your system is affected or not. Many thanks to Uwe Kleine-König for suggesting, and writing this script. https://lists.debian.org/debian-devel/2017/06/msg00309.html - ================================================================================ Title: Keynoting Deutsche VMUG and London VMUG URL: https://frankdenneman.ai/2017-06-06-keynoting-deutsche-vmug-london-vmug/ Date: 2017-06-06 Later on this month, I will be attending the Deutsche VMUG and the London VMUG. As part of the events, I have the opportunity to deliver the keynote on the upcoming service VMware Cloud on AWS. Many of you will already be aware that Niels and I are releasing the VMware vSphere 6.5 Host Resource Deep Dive. Together we will provide a session at both events zooming into ESXi hosts designs, highlighting some interesting behavior from a component and VMkernel perspective. DEUTSCHE VMUG USERCON 2017 14 June 2017 KAP Europa, Kongresshaus der Messe Frankfurt Osloerstrasse 5 Frankfurt am Main, 60327 DE LONDON VMUG 22 June 2017 10:00 AM - 5:15 PM (UTC) TechUK 10 St Bride Street London, EC4A 4AD If you can’t make it to the LONDON VMUG, join us at vBeers that night. We will be heading over to the Fourpure brewing company at 22 Bermondsey Trading Estate, Rotherhithe New Road, London Hope to see you at one of these events! ================================================================================ Title: Memory-Like Storage Means File Systems Must Change - My Take URL: https://frankdenneman.ai/2017-05-25-memory-like-storage-means-file-systems-must-change-take/ Date: 2017-05-25 I’m an avid reader of thenextplatform.com. They always provide great insights into new technology. This week they published the article “Memory-Like Storage Means File Systems Must Change” and as usually full of good stuff. The focus of this article is about the upcoming non-volatile memory technologies that leverage the memory channel to provide incredible amounts of bandwidth to the storage medium. I can’t wait to see this happen and we can start to build systems with performance characteristics that weren’t conceivable a half a decade ago. The article mentions 3D XPoint and Intel Apache Pass is the codename for 3D XPoint in DIMM format. It could be NVDIMM it could be something else. We don’t know yet. This article argues that storage systems need to change and I fully agree. If you consider the current performance overhead on recently released PCIe NVMe 3D XPoint devices, it is clear that the system and the software have the largest impact on latency. The solved the device characteristics pretty much; it’s now the PCIe bus and the software stack that delays the I/O. Moving to the memory bus makes sense. Less overhead and almost five times the bandwidth. For example, four-lane PCIe 3.0 provides a theoretical bandwidth of close to 4 GB/s while 2400 MHz memory has a peak transfer rate of close to 19 GB/s. This sounds great and very promising, but I do wonder how will it impact memory operations. The key is to deliver an additional level of memory hierarchy, increasing capacity while abstracting the behavior of the new media. It’s key to understand that memory is accessed after an L3 miss. It can spend a lot of time waiting on DRAM. A number often heard is that it can spend 19 out of every 20 instruction slots waiting on data from memory. This figure seems accurate as the latency of an instruction inside a CPU register is one ns while memory latency is close to 15 ns. Each core requires memory bandwidth, and this impacts the average memory bandwidth per core. Introducing a media that is magnitudes slower than DRAM can negatively affect the overall system performance. More cycles are wasted on waiting on memory media. Please remember that not every workload is storage I/O bound. Great system design is not only about making I/O faster; it’s about removing bottlenecks in a balanced matter. It’s essential that the storage I/O should not interrupt DRAM traffic. An analogy would be a car that can go 65MPH. The car in front of him drives 55 MPH. By selecting another lane, the slower car does not interfere anymore, and he can drive the speed he wants. The problem is in this lane cars typically drive 200 MPHs. The key point for both NVDIMM as Intel Apache Pass is that adding storage on the memory bus to improve I/O latency should not interfere with DRAM operations. This content is an excerpt of the upcoming vSphere 6.5 Host Resources Deep Dive book. ================================================================================ Title: Virtually Speaking Podcast: VMware Cloud on AWS & HostDeepDive URL: https://frankdenneman.ai/2017-05-08-virtually-speaking-podcast-vmware-cloud-aws-hostdeepdive/ Date: 2017-05-08 Last Friday I had the honor to join Pete Fletcha a.k.a. Pedro Arrow and John Nicholson on their always fantastic podcast Virtually Speaking. Unfortunately, John was ill that morning, but Duncan helped us out by taking a break from his vacation. We spoke about the upcoming service VMware Cloud on AWS (#VMWonAWS). Why it bring such a tremendous value for customers who are in the process of building a hybrid cloud, and how it can help organizations who are already a customer of both VMware and AWS. Closing off we touched upon the progress of the upcoming book ‘vSphere 6.5 Host Resources Deep Dive’. I had a blast being a guest again, enjoy the show! https://soundcloud.com/virtuallyspeakingpodcast/episode-43-vmware-cloud-on-aws?utm_source=soundcloud&utm_campaign=wtshare&utm_medium=Twitter&utm_content=https%3A//soundcloud.com/virtuallyspeakingpodcast/episode-43-vmware-cloud-on-aws ================================================================================ Title: Impact of CPU Hot Add on NUMA scheduling URL: https://frankdenneman.ai/2017-04-14-impact-cpu-hot-add-numa-scheduling/ Date: 2017-04-14 On a regular basis, I receive the question if CPU Hot-add impacts CPU performance of the VM. It depends on the vCPU configuration of the VM. CPU Hot-Add is not compatible with vNUMA, if hot-add is enabled the virtual NUMA topology is not exposed to the guest OS and this may impact application performance. Please note that vNUMA topology is only exposed when the vCPU count of the VM exceeds the core count, thus if the ESXi host contains two CPU packages with 10 cores, the vNUMA topology is presented to the VM if the vCPU count equals 11 or more. vNUMA in a Nutshell The benefit of a wide-VM is that the guest OS is informed about the physical grouping of the vCPUs. In the example of a 12 vCPU VM on a dual-10 core system, the NUMA scheduler creates 2 virtual proximity domains (VPD) better know as NUMA-clients and distributes the 12 vCPUs equally across them. As a result, a load-balancing group is created containing 6 vCPUs that are scheduled on a physical CPU package. A load-balancing group is internally referred to as a physical proximity domain (PPD). Please note that the PPD does not determine the scheduling of vCPU on a specific HT or full core, a PPD can be seen as a vCPU to CPU affinity group From a memory perspective, the guest OS is presented with a vNUMA node sized, separated address space. These address spaces are local to the subset of the vCPUs. As a result, a 12 vCPU 32 GB VM gets to detect a system with two NUMA nodes. Each NUMA node contains 6 CPUs and has a local address space of 16 GB. Contrary to popular belief vNUMA does not expose the full CPU and memory architecture, a better way to describe it that vNUMA shows a tailor-made world to the VM. But what happens when the VM is configured with less vCPUs than the core count of the physical CPU package and CPU Hot-Add is enabled? Will there be performance impact? And the answer is no. The VPD configured for the VM fits inside a NUMA node, and thus the CPU scheduler and the NUMA scheduler optimizes memory operations. It’s all about memory locality. Let’s make use of some application workload test to determine the behavior of the VMkernel CPU scheduling. For this test, I’ve installed DVD Store 3.0 and ran some test loads on the MS-SQL server. To determine the baseline, I’ve logged in the ESXi host via an SSH session and executed the command: sched-stats -t numa-pnode. This command shows the CPU and memory configuration of each NUMA node in the system. This screenshot shows that the system is only running the ESXi operating system. Hardly any memory is consumed. TotalMem indicates the total amount of physical memory in the NUMA node in kb. FreeMem indicates the amount of free physical memory in the NUMA node in kb. An 8 vCPU 32 GB VM is created with CPU hot add disabled. NUMA scheduler has selected NUMA node 1 for initial placement and the system consumes ~13759 MB (67108864-53019184=14089680/1024). The command memstats -r vm-stats -s name:memSize:allocTgt:mapped:consumed:touched -u mb allows us to verify the VM memory consumption of the VM. The numbers are a close match, please note that VM-stats does not include overhead memory and that the VMkernel can consume some additional overhead in the same NUMA node for other processes. When hot-add is enabled (power down VM is necessary to enable this feature), nothing really changes. The memory for this VM is still allocated from a single NUMA node. To get a better understanding of the CPU scheduling constructs at play here, the following command provides detailed insight of all the NUMA related settings of the VM. (Command courtesy of Valentin Bondzio) vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi "DICT.*(displayname.*|numa.*|cores.*|vcpu.*|memsize.*|affinity.*)= .*|numa:.*|numaHost:.*" "/$path/vmware.log"; echo -e; done It shows hot-add is enabled and the VM is configured with a single VPD that is scheduled on a single PPD. In normal language, the vCPUs of the VM are contained with a single physical NUMA node. It’s the responsibility of the NUMA scheduler that physical local memory is consumed. To verify if the VM is consuming local memory, Esxtop can be used (memory, f, NUMA stats). However sched-stats -t numa-clients provides me also a lot of insight As a result, you can conclude that enabling hot-add on a NUMA system does not lead to performance degradation as long as the vCPU count does not exceed the core count of the CPU package. That means that hot-add can be enabled on VMs, but the instruction must be clear that adding vCPUs can happen up and to the threshold of the physical core count. After that point, the VM becomes a wide-VM and vNUMA comes into play. And in the case of CPU hot-add, its sidelined. What’s the impact of disregarding the physical NUMA topology? The key lies within the message that’s entered in the VMware.log of the VM after boot. The VMkernel is forced into using UMA, Unified Memory Access on a NUMA architecture. As a result, memory is interleaved between the two physical NUMA nodes. In essence, it’s load-balancing memory across two nodes, while ignoring the vCPU location. Let’s explore this behavior a bit more. Christmas is coming early for this VM and it gets another 4 vCPUs. Hot-add is disabled again and thus vNUMA is full in play. The Vmdumper command reveals the following: The vCPUs are split up in two virtual nodes (VPD0 & VPD1), each containing 6 vCPUs. After running the DVD Store query the following memory allocation happened: The guest OS (Windows 2012 R2) consumed some memory from node 1, SQL consumed all of its memory from node 0. For people intimate with SQL resource management this might be strange behavior and this is true. To display memory management at the VMkernel layer I had to restrict SQL to only run on a subset of CPUs. I’ve allowed SQL to run on the first 4 vCPUs. All these were mapped to CPUs located in NUMA node 0. The NUMA scheduler ensured these CPUs consumed local memory. After powering down and enabling Hot-add the same test was run again. No NUMA architecture is exposed to the guest OS and therefore a single memory address space is used by Windows. The memory scheduler follows the rules of UMA and interleaves memory between the two physical nodes. And as the output shows, memory is consumed from both NUMA nodes in a very balanced manner. The problem is, the executing vCPUs are all located in NUMA node 0, therefore they have to fetch a lot of memory from remote, creating an inconsistent – less – performing application. Conclusion Hot-add great feature for when you stay within the confines of the CPU package but expect performance degradation, or at least inconsistent performance when going beyond the CPU core count. This content will appear in the upcoming vSphere 6.5 Host Resources Deep Dive book I’m writing with Niels Hagoort (expected May time-frame). For updates about the book, please follow us on twitter @hostdeepdive or like our page on Facebook ================================================================================ Title: Performance Study: DRS Cluster Management with Reservation and Shares URL: https://frankdenneman.ai/2017-04-10-performance-study-drs-cluster-management-reservation-shares/ Date: 2017-04-10 Last Friday a new performance study was published about DRS Cluster Management. This paper covers the behavior of reservation and shares within a DRS cluster in-depth. It’s a great read! And being honest, it’s always awesome to see a reference to the vSphere Clustering Deep Dive in official documentation. Download it here: http://www.vmware.com/content/dam/digitalmarketing/vmware/en/pdf/techpaper/performance/drs-cluster-mgmt-perf.pdf ================================================================================ Title: vSphere 6.5 Host Deep Dive Update URL: https://frankdenneman.ai/2017-02-24-vsphere-6-5-host-deep-dive-update/ Date: 2017-02-24 Maybe you have noticed that no new content has appeared on the site for a while. And the upcoming book “vSphere 6.5 Host Resources Deep Dive” is to blame for this situation. Last year, Niels Hagoort and I started working on the companion book of the highly successful vSphere Clustering Deep Dive book. We set out writing this book to refocus on the fundament component of the virtual data center, the ESXi host. Today’s focal point is on upper levels/overlay’s (SDDC stack, NSX, Cloud). These topics are exciting and take IT services to the next level, but we also understand that proper host design and management fabricates the foundation for success. As a result, this book explores the host resources, CPU, memory, storage, and network in depth. Our goal is to provide you with an in-depth view of the four major host resources. Instead of showing you where to click to achieve a certain configuration, we explain the inner-workings of these components and how various physical and virtual constructs interact with each other. We believe that this method provides a basis – a foundation on its own - that helps you to design and build the best possible architecture that aligns with the customer requirements each and every time. As you can imagine, trying to write a fitting companion to the cluster deep dive is no small feat. Research, reverse engineering and reading through a lot of academic papers consume most of our time besides our day-time job, hence the progress is not as fast as we would like. Expect the book to be released between April and May this year. Working on this book reminds me of the African Proverb “If you want to go quickly, go alone. If you want to go far, go together”. Although Niels and I generate the content, a lot of people are involved ensuring the quality is up to par. Both Niels and I would like to acknowledge the following persons: Jane Rimmer (has the challenging task of restructure our content into proper English). Chris Gianos (Lead Engineer of Intel Xeon microarchitecture), Haoqiang Zheng (Principal Engineer CPU Scheduler VMkernel) Valentin Bondzio (All-Star Badass GSS VMware) Duncan Epping (Chief Technologist Storage BU VMware) Marco van Baggum (Architect ITQ) Myles Gray (Infrastructure Engineer Novosco) Rutger Kosters (Solution Architect Rubrik) Anthony Spiteri (Technical Evanglist Veeam Software) Joop Carels (Sr. Solution Integrator Ericsson) We expect to publish the book in print in the April/May timeframe. An ebook version will be scheduled to appear at the end of this year. Throughout the writing process, we update the books’ twitter account (@HostDeepDive) and Facebook page with sneak peeks and interesting reference material such as academic papers. Please subscribe to these channels to receive updates. ================================================================================ Title: Decoupling of Cores per Socket from Virtual NUMA Topology in vSphere 6.5 URL: https://frankdenneman.ai/2016-12-12-decoupling-cores-per-socket-virtual-numa-topology-vsphere-6-5/ Date: 2016-12-12 Some changes are made in ESXi 6.5 with regards to sizing and configuration of the virtual NUMA topology of a VM. A big step forward in improving performance is the decoupling of Cores per Socket setting from the virtual NUMA topology sizing. Understanding elemental behavior is crucial for building a stable, consistent and proper performing infrastructure. If you are using VMs with a non-default Cores per Socket setting and planning to upgrade to ESXi 6.5, please read this article, as you might want to set a host advanced settings before migrating VMs between ESXi hosts. More details about this setting is located at the end of the article, but let’s start by understanding how the CPU setting Cores per Socket impacts the sizing of the virtual NUMA topology. Cores per Socket By default, a vCPU is a virtual CPU package that contains a single core and occupies a single socket. The setting Cores per Socket controls this behavior; by default, this setting is set to 1. Every time you add another vCPU to the VM another virtual CPU package is added, and as follows the socket count increases. Virtual NUMA Toplogy Since vSphere 5.0 the VMkernel exposes a virtual NUMA topology, improving performance by facilitating NUMA information to guest operating systems and applications. By default the virtual NUMA topology is exposed to the VM if two conditions are met: The VM contains 9 or more vCPUs The vCPU count exceeds the core count* of the physical NUMA node. * When using the advanced setting “numa.vcpu.preferHT=TRUE”, SMT threads are counted instead of cores to determine scheduling options. Using the following one-liner we can explore the NUMA configuration of active virtual machines on an ESXi host: vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi "DICT.*(displayname.*|numa.*|cores.*|vcpu.*|memsize.*|affinity.*)= .*|numa:.*|numaHost:.*" "/$path/vmware.log"; echo -e; done When using this one-liner after powering-on a 10-vCPU virtual machine on the dual E5-2630 v4 (10 cores per socket) ESXi 6.0 host the following NUMA configuration is shown: The VM is configured with 10 vCPUs (numvcpus). The cpuid.coresPerSocket = 1 indicates that it’s configured with one core per socket. The last entry summarizes the virtual NUMA topology of the virtual machine. The constructs virtual nodes and physical domains will be covered later in detail. All 10 virtual sockets are grouped into a single physical domain, which means that the vCPUs will be scheduled in a single physical CPU package that typically is similar to a single NUMA node. To match the physical placement, a single virtual NUMA node is exposed to the virtual machine. Microsoft Sysinternals tool CoreInfo exposes the CPU architecture in the virtual machine with great detail. (Linux machines contain the command numactl - - hardware and use lstopo -s to determine cache configuration). Each socket is listed and with each logical processor a cache map is displayed. Keep the cache map in mind when we start to consolidate multiple vCPUs into sockets. Virtual NUMA topology The vCPU count of the VM is increased to 16 vCPUs; as a consequence, this configuration exceeds the physical core count. MS Coreinfo provides the following insight: Coreinfo uses an asterisk to represent the mapping of the logical processor to socket map and NUMA Node Map. In this configuration the logical processors in socket 0 to socket 7 belong to NUMA Node 0, the CPUs in socket 8 to socket 15 belong to NUMA 1. Please note that this screenshot does not contain the entire logical processor cache map overview. Running the vmdumper one-liner on the ESXi host, the following is displayed: In the previous example in which the VM was configured with 10 vCPUs, the numa.autosize.vcpu.maxPerVirtualNode = “10”, in this scenario, the 16 vCPU VM, has a numa.autosize.vcpu.maxPerVirtualNode = “8”. The VMkernel symmetrically distributes the 16 vCPUs across multiple virtual NUMA Nodes. It attempts to fit as much vCPUs into the minimum number of virtual NUMA nodes, hence the distribution of 8 vCPU per virtual node. It actually states this “Exposing multicore topology with cpuid.coresPerSocket = 8 is suggested for best performance”. Virtual Proximity Domains and Physical Proximity Domains A virtual NUMA topology consists of two elements, the Virtual Proximity Domains (VPD) and the Physical Proximity Domains (PPD). The VPD is the construct what is exposed to the VM, the PPD is the construct used by NUMA for placement (Initial placement and load-balancing). The PPD auto sizes to the optimal number of vCPUs per physical CPU Package based on the core count of the CPU package. Unless the setting Cores per Socket within a VM configuration is used. In ESXi 6.0 the configuration of Cores per Socket dictates the size of the PPD, up to the point where the vCPU count is equal to the number of cores in the physical CPU package. In other words, a PPD can never span multiple physical CPU packages. The best way to perceive a proximity domain is to compare it to a VM to host affinity group, but in this context, it is there to group vCPU to CPU Package resources. The PPD acts like an affinity of a group of vCPUs to all the CPUs of a CPU package. A proximity group is not a construct that is scheduled by itself. It does not determine whether a vCPU gets scheduled on a physical resource. It just makes sure that this particular group of vCPUs consumes the available resources on that particular CPU package. A VPD is the construct that exposes the virtual NUMA topology to the virtual machine. The number of VPDs depends on the number of vCPUs and the physical core count or the use of Cores per Socket setting. By default, the VPD aligns with the PPD. If a VM is created with 16 vCPUs on the test server two PPD’s are created. These PPD allow the VPDs and its vCPUs to map and consume physical 8 cores of the CPU package. If the default vCPU settings are used, each vCPU is placed in its own CPU socket (Cores per Socket = 1). In the diagram, the dark blue boxes on top of the VPD represent the virtual sockets, while the light blue boxes represent vCPUs. The VPD to PPD alignment can be overruled if a non-default Cores per Socket setting is used. A VPD spans multiple PPDs if the number of the vCPUs and the Cores per Socket configuration exceeds the physical core count of a CPU package. For example, a virtual machine with 40 vCPUs and 20 Cores per Socket configuration on a host with four CPU packages containing each 10 cores, creates a topology of 2 VPD’s that each contains 20 vCPUs, but spans 4 PPDs. The Cores per Socket configuration overwrites the default VPD configuration and this can lead to suboptimal configurations if the physical layout is not taken into account correctly. Specifically, spanning VPDs across PPDs is something that should be avoided at all times. This configuration can render most CPU optimizations inside the guest OS and application completely useless. For example, OS and applications potentially encounter remote memory access latencies while expecting local memory latencies after optimizing thread placements. It’s recommended to configure the VMs Cores per Socket to align with the physical boundaries of the CPU package. ESXi 6.5 Cores per Socket behavior In ESXi 6.5 the CPU scheduler has got some interesting changes. One of the changes in ESXi 6.5 is the decoupling of Cores per Socket configuration and VPD creation to further optimize virtual NUMA topology. Up to ESXi 6.0, if a virtual machine is created with 16 CPUs and 2 Cores per Socket, 8 PPDs are created and 8 VPDs are exposed to the virtual machine. The problem with this configuration is that it the virtual NUMA topology does not represent the physical NUMA topology correctly. The guest OS is presented with 16 CPUs distributed across 8 sockets. Each pair of CPUs has its own cache and its own local memory. The operating system considers the memory addresses from the other CPU pairs to be remote. The OS has to deal with 8 small chunks of memory spaces and optimize its cache management and memory placement based on the NUMA scheduling optimizations. Where in truth, the 16 vCPUs are distributed across 2 physical nodes, thus 8 vCPUs share the same L3 cache and have access to the physical memory pool. From a cache and memory-centric perspective it looks more like this: CoreInfo output of the CPU configuration of the virtual machine: To avoid “fragmentation” of local memory, the behavior of VPDs and it’s relation to the Cores per Socket setting has changed. In ESXi 6.5 the size of the VPD is dependent on the number of cores in the CPU package. This results in a virtual NUMA topology of VPDs and PPDs that attempt to resemble the physical NUMA topology as much as possible. Using the same example of 16 vCPU, 2 Cores per Socket, on a dual Intel Xeon E5-2630 v4 (20 cores in total), the vmdumper one-liner shows the following output in ESXi 6.5: As a result of having only two physical NUMA nodes, only two PPDs and VPDs are created. Please note that the Cores per Socket setting has not changed, thus multiple sockets are created in a single VPD. A new line appears in ESXi 6.5; “NUMA config: consolidation =1”, indicating that the vCPUs will be consolidated into the least amount of proximity domains as possible. In this example, the 16 vCPUs can be distributed across 2 NUMA nodes, thus 2 PPDs and VPDs are created. Each VPD exposes a single memory address space that correlates with the characteristics of the physical machine. The Windows 2012 guest operating system running inside the virtual machine detects two NUMA nodes. The CPU view of the task managers shows the following configuration: The NUMA node view is selected and at the bottom right of the screen, it shows that virtual machine contains 8 sockets and 16 virtual CPUs. CoreInfo provides the following information: With this new optimization, the virtual NUMA topology corresponds more to the actual physical NUMA topology, allowing the operating system to correctly optimize its processes for correct local and remote memory access. Guest OS NUMA optimization Modern applications and operating systems manage memory access based on NUMA nodes (memory access latency) and cache structures (sharing of data). Unfortunately most applications, even the ones that are highly optimized for SMP, do not balance the workload perfectly across NUMA nodes. Modern operating systems apply a first-touch-allocation policy, which means that when an application requests memory, the virtual address is not mapped to any physical memory. When the application accesses the memory, the OS typically attempts to allocate it on the local or specified NUMA if possible. In an ideal world, the thread that accessed or created the memory first is the thread that processes it. Unfortunately, many applications use single threads to create something, but multiple threads distributed across multiple sockets access the data intensively in the future. Please take this into account when configuring the virtual machine and especially when configuring Cores per Socket. The new optimization will help to overcome some of these inefficiencies created in the operating system. However, sometimes it’s required to configure the VM with a non-default Cores per Socket setting, due to licensing constraints for example. If you are required to set Cores per Socket and you want to optimize guest operating system memory behavior any further, then configure the Cores per Socket to align with the physical characteristics of the CPU package. As demonstrated the new virtual NUMA topology optimizes the memory address space, providing a bigger more uniform memory slice that aligns better with the physical characteristics of the system. One element has not been thoroughly addressed and that is cache address space created by a virtual socket. As presented by CoreInfo, each virtual socket advertises its own L3 cache. In the scenario of the 16 vCPU VM on the test system, configuring it with 8 Cores per socket, this configuration resembles both the memory and the cache address space of the physical CPU package the most. Coreinfo shows the 16 vCPUs distributed symmetrically across two NUMA nodes and two sockets. Each socket contains 8 CPUs that share L3 cache, similar to the physical world. Word of caution! Migrating VMs configured with Cores per Socket from older ESXi versions to ESXi 6.5 hosts can create PSODs and/or VM Panic Unfortunately, there are some virtual NUMA topology configurations that cannot be consolidated properly by the GA release of ESXi 6.5 when vMotioning a VM from an older ESXi version. If you have VMs configured with a non-default Cores per Socket setting or you have set the advanced parameter numa.autosize.once to False, enable the following advanced host configuration on the ESXi 6.5 host: Numa.FollowCoresPerSocket = 1 A reboot of the host is not necessary! This setting makes ESXi 6.5 behave as ESXi 6.0 when creating the virtual NUMA topology. That means that the Cores per Socket setting determines the VPD sizing. There have been some cases reported where the ESXi 6.5 crashes (PSOD). Test it in your lab and if your VM configuration triggers the error set the FollowCoresPerSocket setting as an advanced configuration. Knowledge base article 2147958 has more information. I’ve been told that the CPU team is working on a permanent fix, I do not have insights when this fix will be released! ================================================================================ Title: VMware Cloud on AWS at re:Invent URL: https://frankdenneman.ai/2016-11-22-vmware-cloud-aws-reinvent/ Date: 2016-11-22 Thank you for all the great feedback since we announced our partnership with Amazon Web Services (AWS) on October 13! We have seen a lot of interest for VMware Cloud on AWS (VMC) from customers, partners, industry analysts, and social media. Following on from the announcement in San Francisco, we went on to Barcelona for VMworld Europe, and had multiple sold out sessions with our customers and partners in attendance. The #VMWonAWS hashtag on Twitter was pretty active as well, and we had our hands full answering all your questions! Next stop is AWS re:Invent in Las Vegas, unfortunately I won’t be at re:Invent, but the core group of VMware on AWS cloud product team is. VMware is a Platinum Sponsor at re:Invent and the team to eager to talk about the service offering, use cases, architecture, Tech Preview demos and a lot more. Here are the top three ways to get the most out of re:Invent: ENT317 - VMware and AWS Together - VMware Cloud on AWS Thursday, Dec 1, 2:00 PM - 3:00 PM Location: Venetian Level 3, Murano 3205 (please check exact location on the portal) Speakers: Matt Dreyer, VMware Product Management, Paul Bockelman - AWS Sr. Solutions Architect Description: VMware CloudTM on AWS brings VMware’s enterprise class Software-Defined Data Center software to Amazon’s public cloud, delivered as an on-demand, elastically scalable, cloud-based VMware sold, operated and supported service for any application and optimized for next-generation, elastic, bare metal AWS infrastructure. This solution enables customers to use a common set of software and tools to manage both their AWS-based and on-premises vSphere resources consistently. Further virtual machines in this environment have seamless access to the broad range of AWS services as well. This session will introduce this exciting new service and examine some of the use cases and benefits of the service. The session will also include a VMware Tech Preview that demonstrates standing up a complete SDDC cluster on AWS and various operations using standard tools like vCenter. PTS205 - VMware Cloud on AWS Wednesday, Nov 30, 1:30 PM - 1:45 PM Location: Partner Theater - Expo Hall Speaker: Marc Umeno, VMware Product Management Description: Learn about how VMware and AWS are joining hands to deliver a new vSphere-based service running on next-generation, elastic, bare-metal AWS infrastructure with seamless integration with AWS services. VMware Booth 2525 at Sands Expo, Hall D. Full exhibitor list and map is here Tue Nov 29th 5-7 pm ; Wed Nov 30th 10:30am-6pm ; Thu Dec 1st 10:30am-6pm Description: We have three demo pods: a) VMware Cloud on AWS, b) Networking & Security, and c) Cloud Management Beyond the sessions and booth, you can also engage with us using the following means: Sign up for [Beta, news updates, or both](http://Sign up for Beta, news updates) Follow us on Twitter @vmwarecloud Ask a question, share a use case, or just give us a shout out using #VMWonAWS hashtag Pick a 30 minute slot to talk to a member of the VMware Cloud on AWS product team 1:1 Thank you and we hope to see you there! https://www.youtube.com/watch?v=wBai31IywlI&t=49s ================================================================================ Title: My thought on 800 page VCDX designs URL: https://frankdenneman.ai/2016-11-15-thought-800-page-vcdx-designs/ Date: 2016-11-15 Although I’m not participating in the VCDX program any more, I still hold it dear to my heart. Many aspiring VCDX’es approach me and seek guidance on how to successfully pass the last part of the VCDX process, the defense. Typically this starts with the discussion on the design itself and particularly how many pages the design should be comprised off. I heard stories about people advocating 800 page designs. And that makes me laugh, but mostly cry. Let’s go back to the essence of the program and understand that the VCDX program has been erected with the idea to validate that someone is a skilled architect. That they can assist IT-organizations into building a successful vSphere architecture. In short, it’s just a stamp of approval of your skill as an architect. Now with that in mind, how many skilled architects hand in an 800 page vSphere design document to a customer? How many customers would accept that? We are not in the business of writing the next Lord of the Rings novel. I worked on complex and massive architectures and most designs didn’t touch 150 pages. When reviewing such 800 page designs, I noticed it’s more a cut and paste of official documentation on how a certain features work. It’s imperative that you know the inner workings of the pillars and foundation of your architecture. But your design should not be a thesis or a showcase of your knowledge of the products. A design should highlight the requirements, the constraints and the chosen direction and technology. It should explain the workings of the used technology in a short and concise manner. Explain how this technology meet the customer requirements and if certain constraints require you to deviate from the default settings. Document thoroughly the effect the chose design on the service levels of the applications and architecture. I feel that some people try to portray the defense as this herculean feat. And to be honest, if you haven’t operated as an architect for multiple customers, it might feel that way. But if you are the architect that has worked on multiple designs, that recognizes the risk-awareness culture differences between companies and how to cater to this need. That can drill down to the essence and explain why a certain requirement impacts a design decision and what effect this has on service levels or other requirements you should be fine! Try to not to see it as the Mount Everest of your career, see it passing the defense as ceremony that validates your upward path of being a great architect. Do what you’ve always have been doing. If you provided your customers with 100 to 200 page designs, keep on doing that and submit such a design for your VCDX defense. ================================================================================ Title: Host not Ready Error When Installing NSX Agents URL: https://frankdenneman.ai/2016-11-04-host-not-ready/ Date: 2016-11-04 Management summary: Make sure your NSX Controller is connected a distributed vSwitch instead of the standard vSwitch During the install process of NSX, my environment refused to install the NSX agents on the host. When you prepare the host clusters for network virtualization a collection of VIBs are installed on each ESXi Node of the selected cluster. This process installs the functionality such as Distributed Routing, Distributed Firewalls and the user world agent that allows the distributed vSwitch to evolve into a NSX Virtual Switch. Unfortunately, this process didn’t go as smooth as the other processes such as installing the NSX Manager and deploying the NSX Controller. Each time I selected Install at Host Preparation, (Within vCenter, select Networking & Security > Installation > Host Preparation. Select the cluster and click the Install link) the process returned an error “Host Not Ready”.The recent task view showed that the task cannot be completed Events shows the following entry: Not very helpful in order to troubleshoot the error. I followed the KB article 2075600 (Installation Status appears as Not Ready in NSX (2075600), and made sure time and DNS were set up correctly. But unfortunately, it didn’t solve the problem. Until I started to dissect the process of what Install at the Host Preparation actually does and how the components connect to each other. This made me review the settings of the NSX Manager and discovered I selected the port group designated for my management VMs on the standard switch instead of the distributed switch. It makes sense to connect it to a Distributed Switch, maybe this is the reason why many write-ups on how to install NSX assume this is basically knowledge and fail to list it as a requirement. The UI allows you to select a standard vSwitch Port Group or a Distributed Port Group. Don’t make the same mistake I made and make sure you select the appropriate Distributed Port Group. ================================================================================ Title: VMware Cloud on AWS - Elastic DRS preview URL: https://frankdenneman.ai/2016-10-18-vmware-cloud-aws-elastic-drs/ Date: 2016-10-18 The VMworld Europe keynote featured the future VMware Cloud on AWS services. In short this services gives VMware customers instant scale and global reach delivered by AWS while continuing to use their own skill set driving and operating VMware SDDC environments on-prem and in-cloud. Avoid the risk that comes with re-platforming, re-architecting current application landscape to run on a different platform while providing the same service. In turn it allows the IT organization to connect the current applications with AWS vast service catalog and use services like RDS, Red Shift, Glacier and many more. One of the interesting features that is under tech preview is Elastic DRS. Elastic DRS helps to solve one of the toughest challenges an IT architect can face: capacity planning. Major key points of capacity planning are current and future resource demand, failure recovery capacity and maintenance capacity. Finding the right balance between maintaining workload performance versus the downside of CAPEX and OPEX of reserved failover capacity is difficult. By leveraging the IT-at-scale operations of AWS, Elastic DRS transforms vSphere clusters into an agility powerhouse. Rapid scaling ability allows to add additional hosts to the cluster. No more ordering new hardware, racking and stacking, just add the new host to the cluster with a right-click of the mouse. By using native metrics, DRS can detect that the cluster is running out of host resources and presents a recommendation of adding another host. Like regular DRS, you can also put Elastic DRS into automatic mode and allow it to add or remove hosts based on observed load on the cluster. Sometimes we forget how extremely complex running IT at super scale is. Automating the install, configuration and operaing one host is interesting, doing this by the dozen is already pushing the limits for a lot of IT organizations. Now think about this doing it in more than a dozen datacenters around the world at the same time while being required to do it instantly when a customer wants this. Undeniably impressive. When joining the team, learning about Elastic DRS was exciting, understanding how this works for all the customers on all the AWS datacenters around the world is just mind-blowing! IT-at-Scale to its finest. When you have ready-to-go ESXi hosts at your fingertips it allows you to do so many cool things , for example allow DRS to aid and assist vSphere HA. Since ESXi 3.0, vSphere HA has ensured that workloads are restarted on the surviving hosts in the cluster. However, when a host outage is not temporary, but permanently, application performance can be impacted due to the reduction of available host resources on a longer term. Auto remediation helps to address this challenge. Auto remediation builds upon Elastic DRS and ensures that the available host resources remain consistent during an ESXi host outage. When a host failure is detected, auto remediation adds another hosts to the cluster, ensuring that the workload performance will not be impacted in the long run by a host failure. If partial (hardware) failure occurs, auto remediation ensures that VSAN operations complete before ejecting the degraded host. Another benefit of this framework is the ability to retain similar levels of resources during maintenance. Typically during maintenance operations, hosts are patched and temporarily unavailable to run and service applications. Many IT organizations deal with this situation, by either “oversizing” cluster or by offering SLA’s that provides a reduced service during maintenance hours. With Elastic DRS, the cluster size is not reduced during maintenance operations. This way workloads are not impacted by a loss of resources and continue to perform similarly as to normal operation hours. To emphasize this is a a technical preview of a service that is not operational yet. For more info about VMware Cloud on AWS, take a closer look. ================================================================================ Title: VMware Cloud™ on AWS – A Closer Look URL: https://frankdenneman.ai/2016-10-13-vmware-cloud-aws-closer-look/ Date: 2016-10-13 After a long time of keeping this silent, I can finally share a little bit what I’ve been focussing on at VMware. (This is a repost of content on blogs.vmware.com) Today, VMware and Amazon Web Services (AWS) are announcing a strategic partnership providing the ability to run a full VMware Software Defined Data Center (SDDC) as a cloud service on AWS. This service will include all the enterprise tools you’re familiar with including vSphere, ESXi, VSAN and NSX. This article provides a technical preview of the new service VMware Cloud on AWS (VMC), allowing me to give you a sneak peak of the incredibly cool stuff that is coming. This architecture is a match made in heaven if you ask me. It allows administrators and architects that are used to vSphere to leverage the agility of AWS without re-architecting applications and reconstructing operational procedures. One great advantage is that vCenter will be the main platform of operations, therefore all tools that you currently run against vCenter in your on-premises vSphere deployment will work with the in-cloud SDDC environment. All these tools and functionalities that have been developed over the years are now coming together and provide an environment that allows workload mobility between clouds while pushing data center agility to new levels. In short, once signed up, select a cluster size and a SDDC environment is created for you in a very short time. To emphasize (and to avoid any misconception), the VMware cloud will run on native ESX on next-generation, bare metal AWS Infrastructure. The VMware cloud will be deployed as a private cloud containing vSphere ESXi hosts, VSAN and NSX on AWS infrastructure. This will allow you to run enterprise workloads with the same performance, reliability and availability levels as your on-premises vSphere deployments but now on an AWS architecture. The main difference between the on-prem and in-cloud deployment is that VMware manages and operates the infrastructure of the VMware Cloud on AWS. It is important to note here that this is a fully managed service. That is to say, VMware will install, manage and maintain the underlying ESXi, VSAN, vCenter and NSX infrastructure. Routine operations like patching or hardware failure remediation will be taken care of by VMware as part of the service. Customers will have delegated permissions to things like vCenter and will be able to use vCenter to perform administrative tasks but there will be some actions like patching which VMware will provide to you as part of the service. This means that VMware takes care of the core infrastructure in partnership with AWS. VMware Cloud on AWS will be available as a stand-alone deployment, as a Hybrid cloud deployment or as a cloud-to-cloud deployment. With hybrid and cloud-to-cloud deployments, vCenter enhanced linked-mode provides a single pane of glass that assists IT operation teams to manage the SDDC deployments from a centralized console. NSX extends this single pane of glass by providing consistent network and security services between the various deployments. However, NSX is not a requirement! If you are not running NSX on premise right now, you will still be able to run VMware Cloud on AWS but you won’t be able to utilize the hybrid cloud features of NSX until you do. With the ability to span networks and clouds, vMotion provides workload mobility, allowing the movement of workloads in and out the various cloud deployments. Yes, you read that correctly, you can vMotion from your existing on-premises vSphere environment to AWS! One of the interesting concepts is elastic scaling. Elastic scaling would help to solve one of the toughest challenges an IT architect can face: capacity planning. Major key points of capacity planning are current and future resource demand, failure recovery capacity and maintenance capacity. Finding the right balance between maintaining workload performance versus the downside of CAPEX and OPEX of reserved failover capacity is difficult. Think about how elastic scaling would transform vSphere clusters into agile powerhouses. Instead of going through the tedious procuring and installing process yourself, benefit from the IT-at-scale mindset and services delivered by AWS. Since ESXi 4.0, vSphere HA has enabled workloads to restart the surviving hosts in the cluster. However, when a host outage is not temporary, host resources can become constrained due to the reduction of the available hosts. Auto-remediation can builds upon DR solutions ensuring available host resources remain consistent during an ESXi host outage. When a host failure is detected, auto-remediation adds other hosts to the cluster, ensuring that the workload performance will not be impacted in the long run by a host failure. If partial (hardware) failure occurs, auto-remediation ensures that VSAN operations complete before ejecting the degraded host. Another benefit of this framework is the ability to retain similar levels of resources during maintenance. During maintenance operations, the cluster size is not reduced, workloads are not impacted by a loss of resources and continue to perform similarly as to normal operation hours. I believe one of the strengths of VMware Cloud on AWS service is that it allows administrators, operation teams and architects to use their existing skill set and tools to consume AWS infrastructure. You can move workloads to the cloud without having to replatform them in any way, no conversion of virtual machines, no repackaging and very important no extensive testing, you just migrate the VM. Another strength it the ability to pair current workloads with the advanced feature set of AWS. As a result, IT teams will be able to extend their skill set discovering the vast catalog of services AWS has to offer. This creates an environment that works seamlessly with both on-premises private clouds and advanced AWS Public Cloud Services. There are so many other great features that I want to cover, but let’s save that for future articles. VMworld If you want to learn more about the upcoming service VMware Cloud on AWS, come join us at VMworld Europe, breakout session INF7849: VMware Cloud on AWS – a closer look. In this session Alex Jauch and I dive a little deeper into the details of this service. For a more generic view, please register for the breakout session INF7711 – VMware Cloud Foundation on Public Clouds. At VMworld Europe, I have a limited set of Meet the expert slots available to me, please register if you would like to have a more focused conversation about the service. If you are interested in applying for the beta, please click here: http://learn.vmware.com/37941_REG ================================================================================ Title: This blog has been hacked by vS0ciety URL: https://frankdenneman.ai/2016-09-27-blog-hacked-vs0ciety/ Date: 2016-09-27 Follow us @vS0ciety ================================================================================ Title: VMworld Geek Whisperers Podcast - Choosing Titles You Want To Have URL: https://frankdenneman.ai/2016-09-21-vmworld-geek-whisperers-podcast-choosing-titles-want/ Date: 2016-09-21 Amy Lewis asked me to appear on the Geek Whisperers Live podcast at VMworld 2016 in Las Vegas. And as always I had a blast discussing various topics with Amy, Matt, and John. In this talk, we spoke about becoming an evangelist, what the challenges are as an evangelist and why you won’t want to pick the title of evangelist yourself. Of course, while interacting with this magnificent group of people you tend to talk about a lot more things. So go on and check it out, I had a blast doing it. http://geek-whisperers.com/2016/09/choosing-titles-you-want-to-have-wfrank-denneman-at-vmworld-2016-episode-120/ ================================================================================ Title: I'm Coming Home URL: https://frankdenneman.ai/2016-08-30-im-coming-home/ Date: 2016-08-30 I’m excited to announce that I’ve accepted a position at VMware as Senior Staff Architect. I can’t share the details of this next-level product that I will be working on right now. But I look forward to sharing more information when the time is right. I cannot wait to get started. #GameOn! ================================================================================ Title: NUMA Deep Dive Part 5: ESXi VMkernel NUMA Constructs URL: https://frankdenneman.ai/2016-08-22-numa-deep-dive-part-5-esxi-vmkernel-numa-constructs/ Date: 2016-08-22 ESXi Server is optimized for NUMA systems and contains a NUMA scheduler and a CPU scheduler. When ESXi runs on a NUMA platform, the VMkernel activates the NUMA scheduler. The primary role of the NUMA scheduler is to optimize the CPU and memory allocation of virtual machines by managing the initial placement and load balance virtual machine workloads dynamically across the NUMA nodes. Allocation of physical CPU resources to virtual machines is carried out by the CPU scheduler. It is crucial to understand that the NUMA scheduler is responsible for the placement of the virtual machine, but it’s the CPU scheduler that is ultimately responsible for allocating physical CPU resources and scheduling of vCPUs of the virtual machine. The main reason to emphasize this is to understand how hyper-threading fits into CPU and NUMA scheduling. Before diving into the specifics of NUMA optimizations, let’s calibrate the understanding of the various components used at the physical layer, the ESXi kernel layer, and the virtual machine layer. A host consist of a CPU Package, that is the physical CPU piece with the pins, this is inserted in a socket (pSocket). Together with the local memory, they form a NUMA node. Within the CPU package, cores exist. In this example, the CPU package contains four cores and each core has hyper-threading (HT) enabled. All cores (and thus HT) share the same cache architecture. At the ESXi layer, the PCPU exist. A PCPU is an abstraction layer inside the ESXi kernel and can consume a full core or it can leverage HT. At the VM layer, a virtual socket, and a vCPU exists. A virtual socket can map to a single PCPU or span multiple PCPUs. This depends on the number of vCPUs and the settings cores per socket inside the UI (cpuid.CoresPerSocket). The vCPU is the logical representation of the PCPU inside the virtual machine. The configuration vCPU and cores per socket impact the ability of applications (and operating systems) to optimize for cache usage. ESXi VMkernel NUMA Constructs In order to apply initial placement and load balancing operations, the NUMA scheduler creates two logical constructs, the NUMA home node (NHN) and the NUMA client. NUMA Home Node The NUMA home node is a logical representation of a physical CPU package and its local memory. In this example, the NUMA home node consists of 4 cores and its local memory. By default the NUMA Home Node allows the NUMA client to count the physical cores in the CPU package. This count impacts the default NUMA client size. This NUMA home node size is important to understand for virtual machine sizing. If the number of VCPUs of a VM exceeds the physical core count of one CPU package it is distributed across multiple nodes. If necessary, due to workload characteristics, distribution can be avoided by reducing the number of the vCPUs, or have the NUMA scheduler consider HTs. By default NUMA optimization does not count the HTs when determining if the virtual machine could fit inside the NUMA home node. For particular workload that benefits from sharing cache and memory, it might be preferable to have the NUMA scheduler count the available HTs during the power-on operation. This setting, preferHT, is expanded upon in a paragraph below. Similar consideration should be applied when sizing memory for the virtual machine. If the virtual memory configuration exceeds the NUMA home node configuration, then the memory scheduler is forced to consume memory from that is attached to another NUMA node. Please note that the NUMA scheduler is focused on consuming as much local memory as possible, it tries to avoid consuming remote memory. Typically a CPU Package and its local memory are synonymous with a NUMA home node, exceptions are Intel Cluster-on-Die technology and AMD Opteron (Magny Cours and newer). When Cluster-on-Die is enabled on an Intel Xeon CPU, the CPU package is split up into two NUMA nodes optimizing the local cache structures. If Cluster-on-Die is enabled on a dual Intel Xeon system , there are two CPU packages but four NUMA nodes. Marc Lang (@marcandreaslang) demonstrated COD on a 512GB system. Before COD, the system created two NUMA nodes, each addressing 256 GB per NUMA node. 3rd line from above NUMA/MB, two nodes are listed both containing ~262000 MB. After enabling COD the system created four NUMA nodes, each addressing 128 GB per NUMA node. Transparent Page Sharing and NUMA Home Node Traditionally, the NUMA home node is the boundary for Transparent Page Sharing (TPS). That means that only memory is shared between VMs within a NUMA node and not across NUMA nodes. However, due to multiple modifications to memory management, benefits of TPS during normal operations have been reduced increasingly. First, large pages sharing index small pages inside the large page, but won’t allow to share and collapse until memory pressure occurs. (Duncan wrote an must read in-depth article about the thresholds of breaking large pages in 6.0) With the introduction of a security patch, described in KB 2080735, salting was introduced. I described salting in detail here, but in short, salting restricts TPS to share only memory within the VM itself. Inter-VM TPS is no longer enabled by default. Please remember that salting did not increase the memory footprint directly, it just impacts savings when memory pressure occurs and large pages are collapsed. Instead of mapping many VMs to the same memory page, each VM will still have its own memory page. Although it makes sense to consider TPS, to reduce memory footprint and get more cache hits by referring to memory that is already local, but the overall benefit of large pages is overwhelming due to fewer TLB misses and faster page table look-up time. Up to 30% performance improvements are claimed by VMware. If you want to use TPS as much as possible during memory pressure, please follow the instructions listed in KB 2080735. Verify is you operating system is using ASLR (Address Space Layout Randomization) for security purposes or SuperFetch (proactive caching), if you run a Windows VDI environment, as both technologies can prevents sharing of memory pages. NUMA Client A NUMA client is the collection of vCPU and memory configuration of a virtual machine. The NUMA client is the atomic unit of the NUMA scheduler that is subject to initial placement and load balancing operations. By default, the maximum number of vCPUs grouped with a NUMA client cannot exceed the physical core count of a CPU package. During power-on operations, the number of vCPUs are counted and are compared to the number of physical cores available inside the CPU Package. If the vCPU count does not exceed the physical core count a single NUMA client is created. These VCPUs will consume PCPUs from a single CPU package. If the number of vCPUs exceeds the number of physical cores inside a single CPU package, multiple NUMA clients are created. For example, if a VM is configured with 12 vCPUs and the CPU package contains 10 cores, two NUMA clients are created for that virtual machine and the vCPUs are equally distributed across the two NUMA clients. Please note that there is no affinity set between a PCPU and a NUMA client. The CPU scheduler can migrate vCPUs between any PCPU provided by the CPU package! This allows the CPU scheduler to balance the workload optimally. vNUMA Node If multiple NUMA clients are created for a single virtual machine, then this configuration is considered to be a Wide-VM. The NUMA scheduler provides an extra optimization called vNUMA. vNUMA exposes the NUMA structure of the virtual machine, not the entire NUMA topology of the host, to the Guest OS running in the virtual machine. This means in the case of the 12 vCPU VM, vNUMA exposes two NUMA nodes with each 6 CPUs to the guest operating system. This allows the operating system itself to apply NUMA optimizations. NUMA client in-depth Now that the basics are covered, let’s dive into the NUMA client construct a little deeper and determine why proper sizing and sockets per core count can be beneficial to virtual machine performance. During power-on, the NUMA scheduler creates a NUMA client, the internal name for a NUMA client is a Physical Proximity Domain (PPD). The vCPUs grouped into a single NUMA client are placed in its entirety on a NUMA node. During load-balancing operations, the group of vCPUs is migrated together. vCPUs remain inside a NUMA client and cannot be migrated between NUMA nodes or NUMA clients individually. Memory load balancing operations is determined by reviewing the NUMA client configuration and the current overall activity within the system. The NUMA scheduler has different load-balancing types to solve imbalance or improve performance. For example, if a virtual machine has local and remote memory, NUMA determines whether it makes sense to migrate the group of vCPUs or to migrate the memory to the NUMA home node if possible. Initial placement and load balancing operations are covered in more detail in the next article of this series. A Virtual Proximity Domain (VPD) is presented to the guest as the NUMA node. The size of the VPD is determined by the number of vCPUs and the cpuid.CoresPerSocket configuration or the number of vCPUs and the preferHT setting (PCPU count / Logical CPU count). By default, the VPD aligns with the PPD, unless the vCPU count exceeds the physical core count and cpuid.CoresPerSocket is more than 1. For example, a virtual machine with 40 vCPUs and cpuid.CoresPerSocket of 20, creates a topology of 2 VPD’s containing 20 vCPUs spanning 4 PPDs containing each 10 PCPUs. Spanning VPDs across PPDs is something that should be avoided at all times. This configuration can create cache pollution and render most CPU optimizations inside the guest OS and application completely useless. It’s recommended to configure the VMs Cores Per Socket to align with the physical boundaries of the CPU package. Auto sizing vNUMA clients If multiple vNUMA clients are created, the NUMA scheduler auto-sizes the vNUMA clients. By default, it equally balances the number of vCPUs across the least amount of NUMA clients. Autosizing is done on the first boot of the virtual machine. It sizes the NUMA client as optimally as possible regarding the host it boots. During the initial boot, the VMkernel adds two advanced settings to the virtual machine: numa.autosize.vcpu.maxPerVirtualNode=X numa.autosize.cookie = “XXXXXX” The autosize setting reflects the number of vCPUs inside the NUMA node. This setting is not changed, unless the number of vCPUs of the VM changes. This is particularly of interest for clusters that contain heterogeneous host configurations. If your cluster contains hosts with different core counts, you could end up with a NUMA misalignment. In this scenario, the following advanced settings can be used: [numa.autosize.once = FALSE](https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.resm gmt.oc%2FGUID-3E956FB5-8ACB-42C3-B068-664989C3FF44.html) [numa.autosize = TRUE](https://pubs.vmware.com/vsphere-60/index.jsp?topic=%2Fcom.vmware.vsphere.resm gmt.oc%2FGUID-3E956FB5-8ACB-42C3-B068-664989C3FF44.html) This forces the NUMA scheduler to reconfigure the NUMA clients on every power-cycle. Be aware that some workloads that can be negatively impacted when NUMA topology changes. Be careful using this setting. Determining the vNUMA layout VMware.log of the virtual machine contains information about the VPD and PPD configuration. Instead of downloading the VMware.log file you can use the command-line tool vmdumper to display the information: vmdumper -l | cut -d \/ -f 2-5 | while read path; do egrep -oi "DICT.*(displayname.*|numa.*|cores.*|vcpu.*|memsize.*|affinity.*)= .*|numa:.*|numaHost:.*" "/$path/vmware.log"; echo -e; done Courtesy of Valentin Bondzio of VMware. Let’s use the scenario of a 12 vCPUs VM on the 10 core system. The VCPU count exceeds the physical core count, therefore two NUMA clients are expected: The output shows that the virtual machine is backed by two Physical Proximity Domain (PPD0 and PPD1) and that two Virtual Proximity Domain exists (VPD0 and VPD1). Both VPDs are backed by a single PPD. The vCPUs are equally distributed across the proximity domains, vCPU0 - vCPU5 are running on PPD0, vCP6-vCPU11 are running on PPD1. ESXTOP shows that the VM is running on two NUMA home nodes (ESXTOP, press M for memory, F to adjust fields, G to enable NUMA stats, SHIFT-V to display VMs only). NHM stands for NUMA home node and in this case, the VM has two NUMA home nodes, NHN0 and NHN1. When running Windows 2012 R2 inside the virtual machine, the CPU Performance Monitor displays NUMA nodes and displays the NUMA node the CPU belongs to. Another great tool to use to expose the NUMA topology witnessed by the Windows guest OS is the Sysinternals tools CoreInfo. Linux machines contain the command [numactl](http://linux.die.net/man/8/numactl) But what if the virtual machine contains 10 vCPUs instead of 12? The VM is backed by a single vNUMA client (VPD0) running on a single NUMA home node, NHN0. Although there is one vNUMA node present, it is not exposed to Windows. Thus windows only detect 10 CPUS. Any reference to NUMA is lacking inside the CPU performance monitor. Increasing NUMA client size, by counting threads, not cores (preferHT) The advanced parameter numa.vcpu.preferHT=TRUE is an interesting one as it is the source of confusion whether a NUMA system utilizes HT or not. In essence, it impacts the sizing of the NUMA client and therefore subsequent scheduling and load balancing behavior. By default the NUMA scheduler places the virtual machines into as few NUMA nodes as possible, trying spread the workload over the fewest cache structures it can. During placement, it only considers full physical cores for scheduling opportunity, as it wants to live up to the true potential of the core performance. Therefore, the NUMA client size is limited to the number of physical cores per CPU package. Some applications share lots of memory between its threads (cache intensive footprint) and would benefit from having as much as memory local as possible. And usually benefitting from using a single local cache structure as well. For these workloads, it could make sense to prefer using HTs with local memory, instead of spreading the vCPUs across full cores of multiple NUMA home nodes. The preferHT setting allows the NUMA scheduler to create a NUMA client that goes beyond the physical core count, by counting the present threads. For example, when running a 12 vCPU virtual machine on a 10 core system, the vCPUs are distributed equally across two NUMA clients (6-6)C. When using numa.vcpu.preferHT=TRUE the NUMA scheduler counts 20 scheduling possibilities and thus a single VPD is created of 12, which allows the NUMA scheduler to place all the vCPU’s into a single CPU package. Please note that this setting does not force the CPU scheduler to only run vCPUs on HTs. It can still (and possible attempt to) schedule a vCPU on a full physical core. The scheduling decisions are up to the CPU scheduler discretion and typically depends on the over-commitment ratio and utilization of the system. For more information about this behavior please review the article Reservations and CPU scheduling. Because logical processors share resources within a physical core, it results in lower CPU progression than running a vCPU on a dedicated physical core. Therefore, it is imperative to understand whether your application has a cache intensive footprint or whether it relies more on CPU cycles. When using the numa.vcpu.preferHT=TRUE setting, it instructs the CPU scheduler to prioritize on memory access over CPU resources. As always, test thoroughly and make a data-driven decision before moving away from the default! I’m maybe overstating the obvious, but in this scenario, make absolutely sure that the memory sizing of the VM fits within a NUMA home node. The NUMA scheduler attempts to keep the memory local, but if the amount of memory does not fit a single NUMA node it has to place it in a remote node, reducing the optimization of preferHT. numa.vcpu.preferHT=TRUE is a per-vm setting, if necessary this setting can be applied at host level. KB article 2003582 contains the instructions to apply the setting at VM and host level. Keep in mind that when you set preferHT on a virtual machine that has already been powered-on once the NUMA client auto size is still active. Adjust the auto size setting in the advanced configuration of the virtual machine or adjust the Cores Per Socket. More about this combination of settings are covered in a paragraph below. Reducing NUMA client size Sometimes it’s necessary to reduce the NUMA client size for application memory bandwidth requirements or for smaller systems. These advanced parameters can help you change the default behavior. As always make a data-driven-decision before you apply advanced parameters in your environment. Advanced parameter numa.vcpu.min Interesting to note is the size of 10 vCPUs in relationship to the vNUMA setting. One of the most documented settings is the advanced setting numa.vcpu.min. Many sites and articles will tell you that vNUMA is enabled by default on VMs with 8 vCPUs or more. This is not entirely true. vNUMA is enabled by default once the vCPU count is 9 or more AND the vCPU count exceeds the number of physical core count. You can use the numa.vcpu.min setting when your NUMA nodes and VM vCPU configurations are smaller than 8 and you want to expose vNUMA topology to the guest OS. Advanced parameter numa.vcpu.maxPerMachineNode Some workloads are bandwidth intensive rather than memory latency sensitive. In this scenario, you want to achieve the opposite of what numa.vcpu.preferHT achieves and use the setting numa.vcpu.maxPerMachineNode. This setting allows you to reduce the number of vCPU that is grouped within a NUMA client. It forces the NUMA scheduler to create multiple NUMA clients for a virtual machine which would have fit inside a single NUMA home node if the default settings were used. Cores per Socket The UI setting Cores per Socket (Advanced parameter: cpuid.coresPerSocket) directly creates a vNUMA node if a value is used that is higher than 1 (and the number of total vCPUs exceeds the numa.vcpu.min count). Using the 10 vCPU VM example again, when selecting 5 Cores per Socket, the ESXi kernel exposes two vSockets and groups 5 virtual CPUs per vSocket. When reviewing the VPD and PPD info, the VMware.log shows two virtual nodes are created, running on 2 virtual sockets deployed on 2 physical domains. If you change cpuid.coresPerSocket you also change numa.vcpu.maxPerVirtualNode and the log files confirms this: Setting.vcpu.maxPerVirtualNode=5 to match cpuid.coresPerSocket. CoreInfo ran inside the guest os shows the topology of having 5 cores in a single socket (Logical Processor to Socket Map). Combine preferHT and Cores Per Socket to leverage application cache optimizations Now compare the previous output with the Coreinfo output of a virtual machine that has 10 cores split across 2 NUMA nodes but using the default setting cores per socket = 1. It’s the “Logical Processor to Cache Map” that is interesting! This shows that the virtual socket topology is exposed to the guest operating system, along with its cache topology. Many applications that are designed to leverage multi-CPU systems, run optimizations to leverage the shared caching.Therefore it makes sense that when the option preferHT is used, to retain the vCPUs in a single socket, the Cores Per Socket reflect the physical cache topology. This allows the application to make full use of the shared cache structure. Take the following steps to align the Cores Per Socket to 12, creating a single vNUMA node to match the physical topology: Set numa.vcpu.preferHT=TRUE (Edit settings VM, VM Options, Advanced, Edit Configuration, Add Row) Verify with the vmdumper command that numa.vcpu.preferHT is accepted and that the guest OS will see 1 NUMA node with all vCPUs grouped on a single socket. When running CoreInfo the following output is shown; Please note that applications and operating systems can now apply their cache optimizations as they have determined all CPUs share the same last level cache. However, not all applications are this advanced. Contact your software vendor to learn if your application can benefit from such a configuration. NUMA and CPU Hot Add If CPU Hot Add is enabled, NUMA client cannot be sized deterministically. Remember that NUMA client sizing only happens during power-on operations and the Hot Add option is the complete opposite by avoiding any power operation. Due to this, NUMA optimizations are disabled and memory is interleaved between the NUMA Home Nodes for the virtual machine. This typically results in performance degradation as memory access has to traverse the interconnect. The problem with enabling Hot Add is that this is not directly visible when reviewing the virtual machines with ESXTOP. If the vCPU count exceeds the physical core count of a CPU package, a single VPD and PPD are created while spanning across two physical domains. CoreInfo also shows that there are no NUMA nodes. However, ESXTOP shows something different.The two physical domains is the one that throws people off when reviewing the virtual machine in ESXTOP. As the virtual machine spans across two physical NUMA nodes, ESXTOP correctly reports it’s using the resources of NHN1 and NHN2. However, memory is spanned across the Nodes. The 100% locality is presented from a CPU perspective, i.e. whether the NUMA clients memory is on the same physical NUMA node its vCPUs are on.In this scenario, where memory is interleaved, you cannot determine whether the virtual machine is accessing local or remote memory. Size your VM correct For most workloads, the best performance occurs when memory is accessed locally. The VM vCPU and memory configuration should reflect the workload requirements to extract the performance from the system. Typically VMs should be sized to fit in a single NUMA node. NUMA optimizations are a great help when VM configuration span multiple NUMA nodes, but if it can be avoided, aim for a single CPU package design. If a wide VM configuration is non-avoidable, I recommend researching the CPU consumption of the application. Often HTs provide enough performance to have VM still fit into a single CPU package and leverage 100% memory locality. This is achieved by setting the preferHT setting. If preferHT is used, align the cores per socket to the physical CPU package layout. This to leverage the operating system and application last level cache optimizations. The 2016 NUMA Deep Dive Series: Part 0: Introduction NUMA Deep Dive Series Part 1: From UMA to NUMA Part 2: System Architecture Part 3: Cache Coherency Part 4: Local Memory Optimization Part 5: ESXi VMkernel NUMA Constructs Part 6: NUMA Initial Placement and Load Balancing Operations Part 7: From NUMA to UMA ================================================================================ Title: NUMA Deep Dive Part 4: Local Memory Optimization URL: https://frankdenneman.ai/2016-07-13-numa-deep-dive-4-local-memory-optimization/ Date: 2016-07-13 If a cache miss occurs, the memory controller responsible for that memory line retrieves the data from RAM. Fetching data from local memory could take 190 cycles, while it could take the CPU a whopping 310 cycles to load the data from remote memory. Creating a NUMA architecture that provides enough capacity per CPU is a challenge considering the impact memory configuration has on bandwidth and latency. Part 2 of the NUMA Deep Dive covered QPI bandwidth configurations, with the QPI bandwidth ‘restrictions’ in mind, optimizing the memory configuration contributes local access performance the most. Similar to CPU, memory is a very complex subject and I cannot cover all the intricate details in one post. Last year I published the memory deep dive series and I recommend to review that series as well to get a better understanding of the characteristics of memory. Memory Channel The Intel Xeon microarchitecture contains one or two integrated memory controllers. The memory controller connects through a channel to the DIMMs. Sandy Bridge (v1) introduced quadruple memory channels. These multiple independent channels increase data transfer rates due to concurrent access of multiple DIMMs. When operating in quad-channel mode, latency is reduced due to interleaving. The memory controller distributes the data amongst the DIMM in an alternating pattern, allowing the memory controller to access each DIMM for smaller bits of data instead of accessing a single DIMM for the entire chunk of data. This provides the memory controller more bandwidth for accessing the same amount of data across channels instead of traversing a single channel storing all data into a single DIMM. In total, there are four memory channels per processor, each channel connect up to three DIMM slots. Within a 2 CPU system, eight channels are present, connecting the CPUs to a maximum of 24 DIMMs. Quad-channel mode is activated when four identical DIMMs are put in quad-channel slots. When three identical DIMMs are used in Quad-channel CPU architectures, triple-channel is activated, when two identical DIMMs are used, the system will operate in dual-channel mode. Please note that interleaving memory across channels is not the same as the Node Interleaving setting of the BIOS. When enabling Node Interleaving the system breaks down the entire memory range of both CPUs into a single memory address space consisting 4KB addressable regions and maps them in a round robin fashion from each node (more info can be found in part 1). Channel interleaving is done within a NUMA node itself. Regions When all four channels are populated the CPU interleaves memory access across the multiple memory channels. This configuration has the largest impact on performance, especially on throughput. To leverage interleaving optimally, the CPU creates regions. The memory controller groups memory across the channels as much as possible. When creating a 1 DIMM per Channel (DPC) configuration, the CPU creates one region (Region 0) and interleaves the memory access. In this example, one DIMM is placed in a DIMM slot of each channel . Both NUMA nodes are configured identically. The total amount of memory is 128 GB, each NUMA node contains 64 GB. Each NUMA node benefits from the quad-channel mode and has its region 1 filled. Each NUMA node controls its own regions. Unbalanced NUMA Configuration In this example, an additional 64GB is installed in NUMA node 0. CPU 0 will create two regions and will interleave the memory across the four channels and benefit from the extra capacity. NUMA Node 0 contains 128 GB, NUMA node 1 contains 64 GB. However, this level of optimization of local bandwidth would not help the virtual machines who are scheduled to run on NUMA node 1, less memory available means it could require fetching memory remotely. Remote memory access experiences the extra latency of multi-hops and the bandwidth constraint of the QPI compared to local memory access. Unbalanced Channel Configuration The memory capacity is equally distributed across the NUMA Nodes, both nodes contain 96 GB of RAM. The CPUs create two regions, region 0 (64GB) interleaves across four channels, region 1 (32GB) interleaves across 2 channels. Native DIMM speed remains the same (MHz). However, some performance loss occurs due to control management overhead. With local and remote memory access in mind, this configuration does not provide a consistent memory performance. Data access is done across four channels for region 0 and two channels for region 1. Data access across the QPI to the other memory controller might fetch the data across two or four channels. LCC configurations contain a single memory controller whereas MCC and HCC contain two memory controllers. This memory layout creates an unbalance in load on memory controllers with MCC and HCC configuration as well. When adding more DIMMs to the channel, the memory controller consumes more bandwidth for control commands. By adding more DIMMs, more management overhead is created which reduces the available bandwidth for read and write data. The question arises, do you solve the capacity requirement by using higher capacity DIMMS or take the throughput hit by moving to a 3 DIMMs per Channel (DPC) configuration. DIMMS per Channel When designing a system that provides memory capacity while maintaining performance requires combining memory ranking configuration , DIMMs per Channel and CPU SKU knowledge. Adding more DIMMS to the channel increases capacity, unfortunately, there is a downside when aiming for high memory capacity configurations and that is the loss of bandwidth. This has to do with the number of ranks per channel. Ranks DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). Together the DRAM chips grouped into a rank contain 64-bit of data. If a DIMM contains DRAM chips on just one side of the printed circuit board (PCB), containing a single 64-bit chunk of data, it is referred to as a single-rank (1R) module. A dual rank (2R) module contains at least two 64-bit chunks of data, one chunk on each side of the PCB. Quad ranked DIMMs (4R) contains four 64-bit chunks, two chunks on each side. To increase capacity, combine the ranks with the largest DRAM chips. A quad-ranked DIMM with 4Gb chips equals 32GB DIMM (4Gb x 8bits x 4 ranks). As server boards have a finite amount of DIMM slots, quad-ranked DIMMs are the most effective way to achieve the highest memory capacity. However, a channel supports a limited amount of ranks due to maximal capacitance. Ivy Bridge (v2) contained a generation 2 DDR3 memory controller that is aware of the physical ranks behind the data buffer. Allowing the memory controller to adjust the timings and providing better back-to-back reads and writes. Gen 2 DDR3 systems reduce the latency gap between Registered DIMMs (RDIMMS) and Load Reduced DIMMs but most importantly it reduces the bandwidth gap. Memory rank impacts the number of DIMMS supported per channel. Modern CPUs can support up to 8 physical ranks per channel. This means that if a large amount of capacity is required quad ranked RDIMMs or LRDIMMs should be used. When using quad ranked RDIMMs, only 2 DPC configurations are possible as 3 DPC equals 12 ranks, which exceeds the 8 ranks per memory rank limit of currents systems. The memory deep dive article Memory Subsystem Organisation covers ranks more in-depth. Load Reduced DIMMs Load Reduced DIMMs (LRDIMMs) buffer both the control and data lines from the DRAM chips. This decreases the electrical load on the memory controller allowing for denser memory configurations. DDR3 LRDIMMS experienced added latency due to the use of a buffer, DDR4 changed the design of the DIMM structure and placed the buffer closer to the DRAM chips removing the extra latency (For more info: Memory Deep Dive: DDR4 Memory). DPC Bandwidth Impact The CPU SKU determines the maximum memory frequency. Broadwell (v4) LLC support up to 2133 MHz, MCC and HCC configurations support up to 2400 MHz (source: ark.intel.com). Choosing between a 10 core E5-2630 v4 and a 12 core E5-2650 v4, does not only provide you 2 extra cores, it provides an additional memory bandwidth. 2133 MHz equals 17064 MB/s, whereas 2400 MHz has a theoretical bandwidth of 19200 MB/s. By moving to an MCC configuration, your not only increasing the core count, but you will increase the memory subsystem with 13%, each core will benefit from this. The DIMM type and the DPC value of the memory configuration restrict the frequency. As mentioned, using more physical ranks per channel lowers the clock frequency of the memory banks. When more ranks per DIMM are used the electrical loading of the memory module increases. And as more ranks are used in a memory channel, memory speed drops restricting the use of additional memory. Therefore in certain configurations, DIMMs will run slower than their listed maximum speeds. RDIMM 1 DPC 2 DPC 3 DPC LRDIMM 1 DPC 2 DPC 3 DPC Source Cisco 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Cisco PDF Dell 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2133 MHz Dell.com Fujitsu 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 1866 MHz Fujitsu PDF HP 2400 MHz 2400 MHz 1866 MHz 2400 MHz 2400 MHz 2400 MHz* HP PDF Performance Drop 0 0 28% 0 0 12%/28% * I believe this is a documentation error from HP side, DDR4 standards support 2133 MHz with 3 DPC configurations. When creating a system containing 384 GB, using 16GB and populat every DIMM slot results in a memory frequency of 1866 MHz, while using (and let’s not forget, paying for) 2400 MHZ RDIMMs. Using previous examples of unbalanced NUMA or unbalanced channel configuration, you simply cannot create a 384GB configuration with 32GB DIMMs alone. The next correct configuration, leveraging quadruple channels, is a mix of 32 GB and 16 GB DIMMs. Mixed configurations are supported however there are some requirements and limitations server vendors state when using mixed configurations: RDIMMS and LRDIMMS must not be mixed. RDIMMs of type x4 and x8 must not be mixed. The configuration is incrementing from bank 1 to 3 with decreasing DIMM sizes. The larger modules should be installed first. To get the best performance, select the memory module with the highest rank configuration. Due to the limitation of 8 ranks per channel, Rx4 RDIMMs and LRDIMMS allow for the largest capacity configuration, while maintaining bandwidth. Looking at today’s memory prices, DDR4 32GB memory modules are the sweet spot. Bandwidth and CAS Timings The memory area of a memory bank inside a DRAM chip is made up of rows and columns. To access the data, the chip needs to be selected, then the row is selected, and after activating the row the column can be accessed. At this time the actual read command is issued. From that moment onwards to the moment the data is ready at the pin of the module, that is the CAS latency. It’s not the same as load-to-use as that is the round trip time measured from a CPU perspective. CAS latencies (CL) increase with each new generation of memory, but as mentioned before latency is a factor of clock speed as well as the CAS latency. Generally, a lower CL will be better, however, they are only better when using the same base clock. If you have faster memory, higher CL could end up better. When DDR3 was released it offered two speeds, 1066MHz CL7 and 1333 MHz CL8. Today servers are equipped with 1600 MHz CL9 memory.DDR4 was released with 2133 MHz CL13. However, 2133 MHz CL15 is available at the major server vendors. To work out the unloaded latency is: (CL/Frequency) * 2000. This means that 1600 MHz CL9 provides an unloaded latency of 11.25ns, while 2133 MHz CL15 provides an unloaded latency of 14.06ns. A drop of 24.9%. However, there is an interesting correlation with DDR4 bandwidth and CAS latency. Many memory vendors offer DDR4 2800 MHz CL14 to CL 16. When using the same calculation, 2800 MHz CL16 provides an unloaded latency of (16/2800) * 2000 = 11.42ns. Almost the same latency at DDR3 1600 MHz CL9! 2800 MHZ CL14 provides an unloaded latency of 10ns, resulting in similarly loaded latencies while providing more than 75% bandwidth. Energy optimized DDR3 memory runs at 1.5 V, low voltage DDR3 memory runs at 1.3V. There is currently no low-voltage extension for DDR4 (yet), however, DDR4 runs standard at 1.2 V. This reduction in power consumption is a big advantage of DDR4 as it provides energy savings of approximately 30% with the same data rate. Most BIOSes contain energy optimized settings reducing power consumption of memory. Due to the already low voltage, little power savings are gained. However bandwidth drops, Fujitsu states memory frequency drops down to 1866 MHz regardless of DPC configuration when using 2400 MHz memory modules. Be aware when configuring the BIOS and verify memory frequency is not modified when using particular energy settings. NUMA system architecture configuration In order to allow your virtual machines to get the best performance, especially consistent performance, care must be taken when designing and configuring an ESXi host. The selection of CPU die design, low core count, medium core count or high core count, impacts local memory bandwidth (2133 MHz vs 2400 MHz, Interconnect bandwidth (QPI 6.4 GT/s, 8.0 GT/s or 9.6 GT/s) and thus remote memory performance as well well as the cache snoop modes, HS with DR + OSB or Cluster-on-Die. These elements all have an important role in overall performance of your virtual datacenter. This concludes the physical configuration portion of the NUMA Deep Dive series, up next, VMkernel CPU and Memory Scheduling. The 2016 NUMA Deep Dive Series: Part 0: Introduction NUMA Deep Dive Series Part 1: From UMA to NUMA Part 2: System Architecture Part 3: Cache Coherency Part 4: Local Memory Optimization Part 5: ESXi VMkernel NUMA Constructs Part 6: NUMA Initial Placement and Load Balancing Operations Part 7: From NUMA to UMA ================================================================================ Title: NUMA Deep Dive Part 3: Cache Coherency URL: https://frankdenneman.ai/2016-07-11-numa-deep-dive-part-3-cache-coherency/ Date: 2016-07-11 When people talk about NUMA, most talk about the RAM and the core count of the physical CPU. Unfortunately, the importance of cache coherency in this architecture is mostly ignored. Locating memory close to CPUs increases scalability and reduces latency if data locality occurs. However, a great deal of the efficiency of a NUMA system depends on the scalability and efficiency of the cache coherence protocol! When researching the older material of NUMA, today’s architecture is primarily labeled as ccNUMA, Cache Coherent NUMA. hpcresearch.nl: The term “Cache Coherent” refers to the fact that for all CPUs any variable that is to be used must have a consistent value. Therefore, it must be assured that the caches that provide these variables are also consistent in this respect. This means that a memory system of a multi-CPU system is coherent if CPU 1 writes to a memory address (X) and later on CPU2 reads X, and no other writes happened to X in between, CPU 2 read operation returns the value written by CPU 1 write operation. To ensure that the local cache is up to date, the snoopy bus protocol was invented, which allowed caches to listen in on the transport of these “variables” to any of the CPU and update their own copies of these variables if they have them. The interesting thing is that with today’s multicore CPU architecture, cache coherency manifest itself within the CPU package as well as cache coherency between CPU packages. A great deal of memory performance (bandwidth and latency) depends on the snoop protocol. Caching Architecture Sandy Bridge (v1) introduced a new cache architecture. The hierarchy exists of an L1, L2 and a distributed LLC accessed via the on-die scalable ring architecture. Although they are all located on the CPU die, there are differences in latency between L1, L2 and LLC. L1 is the fastest cache and it typically takes the CPU 4 cycles to load data from the L1 cache, 12 cycles to load data from the L2 cache and between 26 and 31 cycles to load the data from L3 cache. In comparison, it takes roughly 190 cycles to get the data from local memory while it could take the CPU a whopping 310 cycles to load the data from remote memory. Each core has a dedicated L1 and L2 cache, this is referred to as private cache as no other core can overwrite the cache lines, the LLC is shared between cores. The L1 cache is split into two separate elements, the Instruction cache (32KB) and the data cache (L1D) (32KB). The L2 cache (256KB) is shared by instructions and data (Unified) and is considered to be an exclusive cache. That means that it does not have to contain all the bits that is present in the L1 cache (instructions and data). However, it’s likely to have the same data and instructions as it’s bigger (less evictions). When data is fetched from memory it fills all cache levels on the way to the core (LCC->L2->L1). The reason why it’s also in the LCC is because the LCC is designed as an inclusive cache. It must have all the data contained in the L2 or L1 cache. More about LLC in a later paragraph. Data Prefetching In order to improve performance, data can be speculatively loaded into the L1 and L2 cache, this is called prefetching. It’s the job of the prefetcher to load data into the cache before the core needs it. Performance improvements up to 30% have been quoted by Intel. The Xeon microarchitecture can make use of both hardware and software prefetching. A well-known software prefetching technology is SSE (Streaming SIMD Extension; SIMD: Single Instruction Multiple Data) SSE provides hints to the CPU which data to prefetch for an instruction. The hardware prefetchers are split between L1 and L2 cache. The component that actual stores the data in the L1D is called the data cache unit (DCU) and is 32KB in size. The L1D manages all the loads and stores of the data. The DCU prefetcher fetches next cache lines from the memory hierarchy when a particular access pattern is detected. The DCU IP prefetcher attempts to load the next instruction before the core actually request it. L2 prefetchers also interact with the LLC. When the L2 contains too many outstanding requests, the L2 prefetchers stores the data in the LLC to avoid eviction of useful cache lines. Note In storage, the unit of transportation is a block, in memory its called a line. The Intel Xeon microarchitecture uses a cache line size of 64 bytes. Two L2 prefetchers exists Spatial Prefetcher and Streamer. The spatial prefetcher attempts to complete every cache line fetched to the L2 cache with another cache line in order to fill a 128-byte aligned chunk. The streamer monitors read requests from the L1D cache and fetch the appropriate data and instructions. Server vendors might use their own designation for L1 and L2 prefetchers Intel Server Vendors DCU Prefetcher DCU Streamer Prefetcher DCU IP-based stride prefetcher DCU IP Prefetcher Spatial prefetcher Adjacent Cache Line Prefetch Streamer Hardware Prefetcher All four prefetchers are extremely important for performance. There are some use cases known where prefetchers consume more bandwidth and CPU cycles than actually benefit performance, but these cases are extremely rare. Testing prefetcher effectiveness is extremely difficult as synthetic test are usually focused on measuring best case scenario bandwidth and latency using sequential access patterns. And you guess it, they workload pattern where prefetchers shine. My recommendation is to have the prefetchers set to enabled. Last Level Cache The L1 and L2 cache are private to the core and stores data it reads, writes or modifies. The LLC cache is shared amongst the cores. Sandy Bridge (v1) moved away from a single unified cache entity in the Uncore to a distributed and partitioned cache structure. The total LLC is carved up into 2.5 MB slices and can be fully accessed and utilized by all cores in the system. It’s mentioned in many articles that a core is associated with a core, but the association is just a physical construct. A core cannot control the placement of data in the LCC and has no ability to access anything but the entire LLC. The LCC is accessed through the scalable on-die ring and latency depends on the number of hops to access the data. Scalable on-die ring The cache is an inclusive cache, meaning that it includes all of the data that is stored in the lower level caches. The memory addresses are hashed and distributed amongst the slices. This approach leverages the incredible bandwidth of the scalable on-die interconnect while reducing hot spots and contention for the cache addresses. It also helps coherency. The L3 slices are connected to the scalable on-die interconnect, that connects the cores and the Uncore containing the R3QPI (Ring to QPI interconnect) and the home agent servicing the Integrated Memory Controller. There are two tag arrays, one for data accesses and one for coherency requests and prefetching. The rings run in a clockwise direction as well as a counter-clockwise direction in order to provide the shortest path between core and cache slice. Intel stated the bandwidth of the ring was ~ 844GB/s in the Sandy Bridge Architecture. Since Haswell (v3) the rings are connected by buffered interconnects to allow the rings to operate independently, coinciding with the introduction of Cluster-on-Die cache snoop mode. The core scalability of the Xeon family results in different die designs. There are three core-count configurations, Low core count (LCC), medium core count (MCC) and high core count (HCC). With every new generation Xeon, the classification of the various configurations change. For example in Haswell (v3) 8 core CPUs were labeled as LCC, in the Broadwell (v4) architecture, 10 core CPUs are labeled as LCC. Max core count Die Design Core Columns Memory Controllers 10 Low Core Count 2 1 16* Medium Core Count 3 2 22 High Core Count 4 2 The Xeon E5-2690 v4 is considered to be a medium core count configuration, while the E5-2683 and E5-2697A are classified as a high core count configuration. Both MCC and HCC configuration have two integrated memory controllers and no performance difference should occur. This configuration outlier should be treated as an academic curiosity. The availability of multiple rings and home agents allows for a specific NUMA optimized cache snoop algorithm. This will become evident in a later section. Cache Snooping Data from the LCC slice can be read by any core in the system, once the data is in the private cache it can be modified. When a cache operation occurs that can affect coherence the cache broadcast this to all other caches. Each cache listens (Snoops) for these messages and react accordingly. Cache coherency protocols keep track of these changes and the most popular invalidation-based-protocol is MESI. Within the MESI Protocol data in cache can be in four states, Modified (M), Exclusive (E), Shared (S), Invalid (I). L2 Cache State Definition State Definition Cache line exists in M Modified The cache line is updated relative to memory Single core E Exclusive The cache line is consistent with memory Single cores S Shared The cache line is shared with other cores, the cache line is consistent with other cores, but may not be consistent with memory Multiple cores I Invalid The cache line is not present in this core L1 or L2 Multiple cores A simple example, a 2 vCPU VM consuming on core 1 and 2 runs SQL server. The VM runs on a 4 core ESXi host. 1: A SQL query requests memory at address X. The query runs on vCPU1 and core 1 detects it does not have this data in it’s L1 and L2 cache. A snoop request is made to the caching agents. Both the L1 and L2 cache of core 1 do not contain this data and a request is made to the caching agent, this could be the caching agent of the LCC slice or the home agent depending on the snoop algorithm. The agent will send out a snoop request to all the cache agents (or the home agent) to determine if they have the cache line. At this point, no cache has this data and MESI protocol states that data is in an invalid state for all four cores. 2: The data is retrieved from memory and stores it into the LLC and the private cache of the core 1. The MESI state of this cache line changes and is Exclusive for core 1 and invalid for the remaining cores. 3; Core1 updates the data which transitions the state of the cache line from Exclusive to Modified. 4: At this point, another query that runs on core 2 wants X as well. The core checks L1 and L2 and both miss, the request is forwarded to the LCC and determines X is present. It might not be consistent anymore, therefore a snoop is sent to core 1 to determine whether the data is modified. It is and retrieves the data and sends it over to core 2, the MESI state of the cache line is changed and now it’s in an shared state. The example provided was based on the traditional MESI protocol, however, Intel applies the MESIF protocol. With the introduction of forwarding, it changed the role of the S state. With MESI when data is in a shared state, each cache owning that cache line can respond to the inquiry. In a 20 core count system this can create a lot of traffic, and as a NUMA system shares its memory address space, it can produce many redundant responses between the CPU, often with varying (high) latency. To solve this problem, one cache line is promoted to the F state. This cache line is the only one that can respond and forward data, all the other cache lines containing the data are placed in the shared mode, which now is silent. The F state transitions to the newest version of the data, solving temporal locality problems of the cache, as this cache is the least likely to evict the cache line. The forwarding state reduces interconnect traffic, as in MESI, all caches in S states responds. Although I would love to go in-depth on this theory, a detailed explanation of the MESIF protocol is out of the scope of this article. I tried to keep it as simple as possible, losing some interesting details, such as Cache Valid Bits (CVB) For more information see the manuscript of J.R. Goodman and H.H.J HUM - MESIF: A Two-Hop Cache Coherency Protocol for Point-to-Point Interconnects Snoop Modes A Snoop mode determine which agent will management snoop operations issues by the cores. Snoops can be sent by the caching agent (Cbox) of each LLC slice or by the home agent. Until now, with every new generation micro-architecture, a new Snoop Mode is introduced. These modes are configurable through BIOS settings and have an effect on cache latency and bandwidth consumption, impact overall performance. Although Intel recommends a default Snoop Mode to the server vendors, not every BIOS conforms to that recommendation. My recommendation is to include QPI Snoop Modes in your documentation as a configuration item. If the system is configured with the default option recommended by Intel do not change this without any data-driven reason. Today four snoop modes are available, one snoop mode (Cluster-on-Die) is available only if two home nodes are available in the package (MCC and HCC die designs). Early Snoop This snoop mode was introduced by Sandy Bridge (v1) and is available on all newer generations. Within early snoop the caching agent generates the snoop probe or the snoop request, using the scalable on-die ring it can directly send that snoop to other cache agents or broadcast it to all the other agents in the system. This model provides a low latency response time, although the amounts broadcast (especially in HCC die designs) can eat up the bandwidth between the NUMA nodes. Typically this snoop mode is not recommended when using NUMA optimized workload. Some vendors did not optimize the BIOS defaults and use this snoop mode even for their newest models. Please check your BIOS. Home Snoop This snoop mode was introduced by Ivy Bridge (v2) and is available on all newer generations. Instead of each caching agent generating snoop messages, it’s the home agent tied to the memory controller that generates the snoop request. Since the snoop request has to go to the home agent and travel the on-die scalable ring, it has a higher latency than early snoop. By leveraging a more centralized entity such as the home agent, it does reduce the bandwidth consumption. Home Snoop mode is geared towards workloads which are bandwidth sensitive. Home Snoop with Directory and Opportunistic Snoop Broadcast (OSB) This mode uses the home agent but it can also speculatively snoop the remote CPU in parallel with the directory read on the home agent. The home agent contains an “in-memory snoop directory” to determine the state of the various cache lines, this reduces snoop traffic primarily on reads. The home agent snoops in parallel with the directory lookup when it thinks there is available system bandwidth to support the snoop traffic…When the system gets more heavily loaded, the snoops are delayed and only sent to the agents the directory information indicates need to be snoop. That way the snoop overhead is kept low in heavily loaded systems and it will focus the available bandwidth on the data instead. This snoop mode was introduced by Ivy Bridge (v2) and was removed in Haswell (v3). It has been reintroduced by Broadwell (v4) and is the recommended default snoop mode by Intel for the Broadwell generation. Please check your BIOS settings as not every vendor follows Intel recommendations. Cluster-on-Die Although the Home Snoop with DIR + OSB has the overall best performance, when running a highly optimized NUMA workload you might want to consider the Cluster-on-Die snoop mode. This mode provides the best performance for local operations. It provides the lowest LLC hit latency and a low local memory latency. Remote memory performance depends on the write activity of the workloads. If you have your workload correctly sized and are able to fit workloads within NUMA nodes, Cluster-on-Die can improve performance. If the virtual data center is designed to run a high consolidation ratio, forcing the ESXi CPU scheduler to span small footprint VMs across NUMA nodes, Home Snoop with Directory and OSB might be a better fit. Cluster-on-Die architecture Cluster-on-Die (COD) is only available on MCC and HCC die design packages. When enabling COD, it logically divides the CPU into two equal NUMA node, incorporating a part of the scalable ring on-die Interconnect that services the home agent & integrated memory controller. In the MCC and HCC die design, there are two active memory controllers, each servicing two channels. The NUMA nodes are associated with the respective controllers. Please note that there will be two NUMA nodes in one CPU package! That means there will be four NUMA nodes in a dual socket system. Marc Lang (@marcandreaslang) demonstrated COD on a 512GB system. Before COD, the system created two NUMA nodes, each addressing 256 GB per NUMA node. After enabling COD the system created four NUMA nodes, each addressing 128 GB per NUMA node. COD segments the LLC and the RAM. By segmenting the LLC, it decreases the latency by reducing the number of slices in the NUMA node. For example, the E5-2699 v4 contains 22 cores, with COD enabled, it creates two affinity domains of 11 slices. Data will be distributed in only 11 LLC slices inside each affinity domain instead of 22 slices, thereby decreasing hop count. In addition, the COD in Broadwell (v4) microarchitecture eliminates cross buffered interconnect traffic, reducing ring collisions and other overhead that reduces the available bandwidth. If there is a cache miss in the LLC within the affinity domain, it will contact the home agent responsible for the memory directly. Each home agent tracks the memory lines it is responsible for. Therefore the LLC can contain cache lines of “remote memory” and traffic will occur across the buffered interconnect if the NUMA scheduler cannot “affinitize” the process and the memory properly. ESXi 5.5 update 3 and ESXi 6.0 supports COD, check https://kb.vmware.com/kb/2142499. As mentioned in part 2, ESXi does not use SLIT information to understand topological distance between the physical CPUs. Instead, ESXi determines the inter-domain latencies by probing the CPUs at boot-time and use this information for initial placement and migration decisions. Since COD is a boot-time configuration, ESXi has a good view of the latencies of the NUMA domains. Having multiple NUMA nodes presented by a single CPU package is not a new thing. In 2011 AMD released the Magny-Cours architecture, which combined 2 6 core Bulldozer CPUs in one package. Unfortunately, a lot of negative performance results were reported by VMware community members due to the ESXi NUMA round-robin scheduling decisions. The cache architecture of the AMD didn’t help as well. Snoop mode recommendation If the VMs are right-sized to fit into a single NUMA node, COD could deliver a stellar performance, when operating a large collection of Wide VMs I would recommend to select the snoop mode “Home Snoop with Directory and Opportunistic Snoop Broadcast (OSB)” as COD is all about reducing latency through affinity Up next, Part 4: Local Memory Optimization The 2016 NUMA Deep Dive Series: Part 0: Introduction NUMA Deep Dive Series Part 1: From UMA to NUMA Part 2: System Architecture Part 3: Cache Coherency Part 4: Local Memory Optimization Part 5: ESXi VMkernel NUMA Constructs Part 6: NUMA Initial Placement and Load Balancing Operations Part 7: From NUMA to UMA ================================================================================ Title: NUMA Deep Dive Part 2: System Architecture URL: https://frankdenneman.ai/2016-07-08-numa-deep-dive-part-2-system-architecture/ Date: 2016-07-08 Reviewing the physical layers helps to understand the behavior of the CPU scheduler of the VMkernel. This helps to select a physical configuration that is optimized for performance. This part covers the Intel Xeon microarchitecture and zooms in on the Uncore. Primarily focusing on Uncore frequency management and QPI design decisions. Terminology There a are a lot of different names used for something that is apparently the same thing. Let’s review the terminology of the Physical CPU and the NUMA architecture. The CPU package is the device you hold in your hand, it contains the CPU die and is installed in the CPU socket on the motherboard. The CPU die contains the CPU cores and the system agent. A core is an independent execution unit and can present two virtual cores to run simultaneous multithreading (SMT). Intel proprietary SMT implementation is called Hyper-Threading (HT). Both SMT threads share the components such as cache layers and access to the scalable ring on-die Interconnect for I/O operations. Interesting entomology; The word “die” is the singular of dice. Elements such as processing units are produced on a large round silicon wafer. The wafer is cut “diced” into many pieces. Each of these pieces is called a die. NUMA Architecture In the following scenario, the system contains two CPUs, Intel 2630 v4, each containing 10 cores (20 HT threads). The Intel 2630 v4 is based on the Broadwell microarchitecture and contains 4 memory channels, with a maximum of 3 DIMMS per channel. Each channel is filled with a single 16 GB DDR4 RAM DIMM. 64 GB memory is available per CPU with a total of 128 GB in the system. The system reports two NUMA Nodes, each NUMA nodes, sometimes called NUMA domain, contains 10 cores and 64 GB. Consuming NUMA The CPU can access both its local memory and the memory controlled by the other CPUs in the system. Memory capacity managed by other CPUs are considered remote memory and is accessed through the QPI (Part 1). The allocation of memory to a virtual machine is handled by the CPU and NUMA schedulers of the ESXi kernel. The goal of the NUMA scheduler is to maximize local memory access and attempts to distribute the workload as efficient as possible. This depends on the virtual machine CPU and memory configuration and the physical core count and memory configuration. A more detailed look into the behavior of the ESXi CPU and NUMA scheduler is done in part 5, how to size and configure your virtual machines is discussed in part 6. This part focusses on the low-level configuration of a modern dual-CPU socket system. ESXtop reports 130961 MB (PMEM /MB) and displays the NUMA nodes with its local memory count. Each core can address up to 128 GB of memory, as described earlier the NUMA scheduler of the ESXI kernel attempts to place and distribute vCPU as optimal as possible, allocating as much local memory to the CPU workload that is available. When the number of VCPUs of a virtual machine exceeds the core count of a physical CPU, the ESXi server distributes the vCPU even across the minimal number of physical CPUs.It also exposes the physical NUMA layout to the virtual machine operating system, allowing the NUMA-aware operating system and / or application to schedule their processes as optimal as possible. To ensure this all occurs, verify if the BIOS is configured correctly and that the setting NUMA = enabled or Node Interleaving is disabled. In this example a 12 vCPU VM is running on the dual Intel 2630 v4 system, each containing 10 cores. CoreInfo informs us that 6 vCPUs are running on NUMA node 0 and 6 vCPUs are running on NUMA node 1. BIOS Setting: Node Interleaving There seems to be a lot of confusion about this BIOS setting, I receive lots of questions on whether to enable or disable Node interleaving. I guess the term “enable” make people think it some sort of performance enhancement. Unfortunately, the opposite is true and it is strongly recommended to keep the default setting and keep Node Interleaving disabled. Node Interleaving Disabled: NUMA By using the default setting of Node Interleaving (disabled), the ACPI “BIOS” will build a System Resource Allocation Table (SRAT). Within this SRAT, the physical configuration and CPU memory architecture are described, i.e. which CPU and memory ranges belong to a single NUMA node. It proceeds to map the memory of each node into a single sequential block of memory address space. ESXi uses the SRAT to understand which memory bank is local to a physical CPU and attempts to allocate local memory to each vCPU of the virtual machine. Node Interleaving Enabled: SUMA One question that is asked a lot is how do you turn off NUMA? You can turn off NUMA, but remember your system is not a transformer, changing your CPUs and memory layout from a point-to-point-connection architecture to a bus system. Therefore, when enabling Node Interleaving the system will not become a traditional UMA system. Part 1 contains a more info on SUMA. BIOS setting: ACPI SLIT Preferences The ACPI System Locality Information Table (SLIT) provides a matrix that describes the relative distance (i.e. memory latency) between the proximity domains. In the past, a large NUMA system the latency from Node 0 to Node 7 can be much greater than the latency from Node 0 to Node 1, and this kind of information is provided by the SLIT table. Modern point-to-point architectures moved from a ring topology to a full mesh topology reducing hop counts, reducing the importance of SLIT. Many server vendor whitepapers describing best practices for VMware ESXi recommend enabling ACPI SLIT. Do not worry if you forgot to enable this setting as ESXi does not use the SLIT. Instead, the ESXi kernel determines the inter-node latencies by probing the nodes at boot-time and use this information for initial placement of wide virtual machines. A wide virtual machine contains more vCPUs than the Core count of a physical CPU, more about wide virtual machines and virtual NUMA can be found in the next article. CPU System Architecture Since Sandy Bridge (v1) the CPU system architecture applied by Intel can be described as a System-on-Chip (SoC) architecture, integrating the CPU, GPU, system IO and last level cache into a single package. The QPI and the Uncore are critical components of the memory system and its performance can be impacted by BIOS settings. Available QPI bandwidth depends on the CPU model, therefore it’s of interest to have a proper understanding of the CPU system architecture to design a high performing system. Uncore As mentioned in part 1, the Nehalem microarchitecture introduced a flexible architecture that could be optimized for different segments. In order to facilitate scalability, Intel separated the core processing functionality (ALU, FPU, L1 and L2 cache) from the ‘uncore’ functionality. A nice way to put it is that the Uncore is a collection of components of a CPU that do not carry out core computational functions but are essential for core performance. This architectural system change brought the Northbridge functionality closer to the processing unit, reducing latency while being able to increase the speed due to the removal of serial bus controllers. The Uncore featured the following elements: Uncore element Description Responsible for: QPI Agent QuickPath Interconnect QPI caching agent , manages R3QPI and QPI Link Interface PCU Power Controller Core/Uncore power unit and thermal manager, governs P-state of the CPU, C-state of the Core and package. It enables Turbo Mode and can throttle cores when a thermal violation occurs Ubox System Config controller Intermediary for interrupt traffic between system and core IIO Integrated IO Provides the interface to PCIe Devices R2PCI Ring to PCI Interface Provides interface to the ring for PCIe access IMC Integrated Memory Controller Provides the interface to RAM and communicates with Uncore through home agent HA Integrated Memory Controller Provides the interface to RAM and communicates with Uncore through home agent SMI Scalable Memory Interface Provides IMC access to DIMMs Intel provides a schematic overview of a CPU to understand the relationship between the Uncore and the cores, I’ve recreated this overview to help emphasise certain components. Please note that the following diagram depicts a High Core Count architecture of the Intel Xeon v4 (Broadwell). This is a single CPU package. The cores are spread out in a “chop-able” design, allowing Intel to offer three different core counts, Low, Medium and High. The red line is depicting the scalable on-die ring connecting the cores with the rest of the Uncore components. More in-depth information can be found in part 4 of this series. If a CPU core wants to access data it has to communicate with the Uncore. Data can be in the last-level cache (LLC), thus interfacing with the Cbox, it might require memory from local memory, interfacing with the home agent and integrated memory controller (IMC). Or it needs to fetch memory from a remote NUMA node, as a consequence, the QPI comes into play. Due to the many components located in the Uncore, it plays a significant part in the overall power consumption of the system. With today’s focus on power reduction, the Uncore is equipped with frequency scaling functionality (UFS). Haswell (v4) introduces Per Core Power States (PCPS) that allows each core to run at its own frequency. UFS allows the Uncore components to scale their frequency up and down independently of the cores. This allows Turbo Boost 2.0 to turbo up and owns the two elements independently, allowing cores to scale up the frequency of their LLC and ring on-ramp modules, without having to enforce all Uncore elements to turbo boost up and waste power. The feature that regulates boosting of the two elements is called Energy Efficient Turbo, some vendors provide the ability to manage power consumption with the settings Uncore Frequency Override or Uncore Frequency. These settings are geared towards applying performance savings in a more holistic way. The Uncore provides access to all interfaces, plus it regulates the power states of the cores, therefore it has to be functional even when there is a minimal load on the CPU. To reduce overall CPU power consumption, the power control mechanism attempts to reduce the CPU frequency to a minimum by using C1E states on separate cores. If a C1E state occurs, the frequency of the Uncore is likely to be lowered as well. This could have a negative effect on the I/O throughput of the overall throughput of the CPU. To avoid this from happening some server vendors provide the BIOS option; Uncore Frequency Override. By default this option is set to Disabled, allowing the system to reduce the Uncore frequency to obtain power consumption savings. By selecting Enabled it prevents frequency scaling of the Uncore, ensuring high performance. To secure high levels of throughput of the QPI links, select the option enabled, keep in mind that this can have a negative (increased) effect on the power consumption of the system. Some vendors provide the Uncore Frequency option of Dynamic and Maximum. When set to Dynamic, the Uncore frequency matches the frequency of the fastest core. With most server vendors, when selecting the dynamic option, the optimization of the Uncore frequency is to save power or to optimize the performance. The bias towards power saving and optimize performance is influenced by the setting of power-management policies. When the Uncore frequency option is set to maximum the frequency remains fixed. Generally, this modularity should make it more power efficient, however, some IT teams don’t want their system to swing up and down but provide a consistent performance. Especially when the workload is active across multiple nodes in a cluster, running the workload consistently is more important that having a specific node to go as fast as it can. Quick Path Interconnect Link Virtual machine configuration can impact memory allocation, for example when the memory configuration consumption exceeds the available amount of local memory, ESXi allocates remote memory to this virtual machine. An imbalance of VM activity and VM resource consumption can trigger the ESXi host to rebalance the virtual machines across the NUMA nodes which lead to data migration between the two NUMA nodes. These two examples occur quite frequently, as such the performance of remote memory access, memory migration, and low-level CPU processes such as cache snooping and validation traffic depends on the QPI architecture. It is imperative when designing and configuring a system that attention must be given to the QuickPath Interconnect configuration. Xeon CPUs designated for dual CPU setup (E5-26xx) is equipped with two QPI bi-directional links. Depending on the CPU model selected, the QPI links operates at high frequencies measured in giga-transfers per second (GT/s). Today the majority of E5 Xeons (v4) operate at 9.6 GT/s, while some run at 6.4 GT/sec or 8.6 GT/sec. Giga-transfer per second refers to the number of operations transferring data that occur in each second in a data-transfer channel. It’s an interesting metric, however, it does not specify the bit rate. In order to calculate the data-transmission rate, the transfer rate must be multiplied by the channel width. The QPI link has the ability to transfer 16 bits of data-payload. The calculation is as follows: GT/s x channel width /bits-to-bytes. 9.6 GT/sec x 16 bits = 153.6 Bits per second / 8 = 19.2 GB/s. The purist will argue that this is not a comprehensive calculation, as this neglects the clock rate of the QPI. The complete calculation is: QPI clock rate x bits per Hz x channel width × duplex = bits ÷ byte. 4.8 Ghz x 2 bits/Hz x 16 x 2 / 8 = 38.4 GB/s. Haswell (v3) and Broadwell (v4) offer three QPI clock rates, 3.2 GHz, 4.0 GHz, and 4.8 GHz. Intel does not provide clock rate details, it just provide GT/s. Therefore to simplify this calculations, just multiple GT/s by two (16 bits / 8 bits to bytes = 2). Listed as 9.6 GT/s a QPI link can transmit up to 19.2 GB/s from one CPU to another CPU. As it is bidirectional, it can receive the same amount from the other side. In total, the two 9.6 GT/s links provide a theoretical peak data bandwidth of 38.4 GB/sec in one direction. QPI link speed Unidirectional peak bandwidth Total peak bandwidth 6.4 GT/s 12.8 GB/s 25.6 GB/s 8.0 GT/s 16.0 GB/s 32 GB/s 9.6 GT/s 19.2 GB/s 38.4 GB/s There is no direct relationship with core-count and QPI link speeds. For example the v4 product family features 3 8-core count CPUs, each with a different QPI link speed, but there are also 10 core CPUs with a bandwidth of 8.0 GT/s. To understand the logic, you need to know that Intel categorizes their CPU product family into segments. Six segments exist; Basic, Standard, Advanced, Segment Optimized, Low Power and Workstation. The Segment Optimized features a sub segment of Frequency Optimized, these CPU’s push the gigabit boundaries. And then off course there is the custom-build segment, which is off the list, but if you have enough money, Intel can look into your problems. The most popular CPUs used in the virtual datacenter come from the advanced and segment optimized segments. These CPUs provide enough cores and cache to drive a healthy consolidation ratio. Primarily the high core count CPUs from the Segment Optimized category are used. All CPU’s from these segments are equipped with a QPI link speed of 9.6 GT/s. Segment Model Number Core count Clock cycle TDP QPI speed Advanced E5-2650 v4 12 2.2 GHz 105W 9.6 GT/s Advanced E5-2660 v4 14 2.0 GHz 105W 9.6 GT/s Advanced E5-2680 v4 14 2.4 GHz 120W 9.6 GT/s Advanced E5-2690 v4 14 2.6 GHz 135W 9.6 GT/s Optimized E5-2683 v4 16 2.1 GHz 120W 9.6 GT/s Optimized E5-2695 v4 18 2.1 GHz 120W 9.6 GT/s Optimized E5-2697 v4 18 2.3 GHz 145W 9.6 GT/s Optimized E5-2697A v4 16 2.6 GHz 145W 9.6 GT/s Optimized E5-2698 v4 20 2.2 GHz 135W 9.6 GT/s Optimized E5-2699 v4 22 2.2 GHz 145W 9.6 GT/s QPI Link Speed Impact on Performance When opting for a CPU with a lower QPI link speeds, remote memory access will be impacted. During the tests of QPI bandwidth using the Intel Memory Latency Checker v3.1. it reported an average of ˜75% of the theoretical bandwidth when fetching memory from the remote NUMA node. The peak bandwidth is more a theoretical maximum number as transfer data comes with protocol overhead. Additionally tracking resources are needed when using multiple links to track each data request and maintain coherency. The maximum QPI bandwidth that is available at the time of writing is lower than the minimum supported memory frequency of 1600 MHz (Intel Xeon v3 & v4). The peak bandwidth of DDR4 1600 MHz is 51 GB/s, which exceeds the theoretical bandwidth of the QPI by 32%. As such, QPI bandwidth can impact remote memory access performance. In order to obtain the most performance, it’s recommended to select a CPU with a QPI configuration of 9.6 GT/s to reduce the bandwidth loss to a minimum, the difference between 9.6 GT/s and 8.0 GT/s configuration is a 29% performance drop. AS QPI bandwidth impacts remote memory access, it’s the DIMM configuration and memory frequency that impacts local memory access. Local memory optimization is covered in Part 4. Note! The reason why I’m exploring nuances of power settings is that high-performance power consumption settings are not always the most optimal setting for today’s CPU microarchitecture. Turbo mode allows cores to burst to a higher clock rate if the power budget allows it. The finer details of Power management and Turbo mode are beyond the scope of this NUMA deep dive, but will be covered in the upcoming CPU Power Management Deep Dive. Intel QPI Link Power Management Some servers allow you to configure the QPI Link Power Management in the BIOS. When enabled, the buffers in the QPI links are allowed to enter a sleep state when the links are not being used. When there is relatively little traffic, the QPI link shuts down some of its data transmissions lanes, this to achieve power consumption reduction. Within a higher state, it only reduces bandwidth, when entering a deeper state memory access will occur latency impact. A QPI link consists of a transmit circuit (TX), 20 data lanes, 1 clock lane and a receive circuit (RX). Every element can be progressively switched off. When the QPI is under heavy load it will use all 20 lanes, however when experiencing a workload of 40% or less it can decide to modulate to half width. Half width mode, called L0p state saves power by shutting down at least 10 lanes. The QPI power management spec allows to reduce the lanes to a quarter width, but research has shown that power savings are too small compared to modulating to 10 links. Typically when the 10 links are utilized for 80% to 90% the state shifts from L0p back to the full-width L0 state. L0p allows the system to continue to transmit data without any significant latency penalty. When no data transmit occurs, the system can invoke the L0s state. This state only operates the clock lane and its part of the physical TX and RX circuits, due to the sleep mode of the majority of circuits (lane drivers) within the transceivers no data can be sent. The last state, L1, allows the system to shut down the complete link, benefitting from the highest level of power consumption. L0s and L1 states are costly from a performance perspective, Intel’s’ patent US 8935578 B2 indicates that exiting L1 state will cost multiple microseconds and L0s tens of nanoseconds. Idle remote memory access latency measured on 2133 MHz memory is on average 130 nanoseconds, adding 20 nanoseconds will add roughly 15% latency and that’s quite a latency penalty. A low power state with longer latency and lower power than L0s and is activated in conjunction with package C-states below C00 State Description Properties Lanes L0 Link Normal Operational State All lanes and Forward Clock active 20 L0p Link power saving state A lower power state from L0 that reduces the link from full width to half width 10 L0s Low Power Link State Turns odd most lane drivers, rapid recovery to the L0 state 1 L0s Deeper Low Power State Lane drivers and Fwd clock turned off, greater power savings than L0s, Longer time to return to L0 state If the focus is on architecting a consistent high performing platform, I recommend to disable QPI Power Management in the BIOS. Many vendors have switched their default setting from enabled to disabled, nevertheless its wise to verify this setting. The memory subsystem and the QPI architecture lay the foundation of the NUMA architecture. Last level cache is a large part of the memory subsystem, the QPI architecture provides the interface and bandwidth between NUMA nodes. It’s the cache coherency mechanisms that play a great part in providing the ability to span virtual machines across nodes, but in turn, will impact overall performance and bandwidth consumption. Up next, Part 3: Cache Coherency The 2016 NUMA Deep Dive Series: Part 0: Introduction NUMA Deep Dive Series Part 1: From UMA to NUMA Part 2: System Architecture Part 3: Cache Coherency Part 4: Local Memory Optimization Part 5: ESXi VMkernel NUMA Constructs Part 6: NUMA Initial Placement and Load Balancing Operations Part 7: From NUMA to UMA ================================================================================ Title: NUMA Deep Dive Part 1: From UMA to NUMA URL: https://frankdenneman.ai/2016-07-07-numa-deep-dive-part-1-uma-numa/ Date: 2016-07-07 Non-uniform memory access (NUMA) is a shared memory architecture used in today’s multiprocessing systems. Each CPU is assigned its own local memory and can access memory from other CPUs in the system. Local memory access provides a low latency - high bandwidth performance. While accessing memory owned by the other CPU has higher latency and lower bandwidth performance. Modern applications and operating systems such as ESXi support NUMA by default, yet to provide the best performance, virtual machine configuration should be done with the NUMA architecture in mind. If incorrect designed, inconsequent behavior or overall performance degradation occurs for that particular virtual machine or in worst case scenario for all VMs running on that ESXi host. This series aims to provide insights of the CPU architecture, the memory subsystem and the ESXi CPU and memory scheduler. Allowing you in creating a high performing platform that lays the foundation for the higher services and increased consolidating ratios. Before we arrive at modern compute architectures, it’s helpful to review the history of shared-memory multiprocessor architectures to understand why we are using NUMA systems today. The evolution of shared-memory multiprocessors architecture in the last decades It seems that an architecture called Uniform Memory Access would be a better fit when designing a consistent low latency, high bandwidth platform. Yet modern system architectures will restrict it from being truly uniform. To understand the reason behind this we need to go back in history to identify the key drivers of parallel computing. With the introduction of relational databases in the early seventies the need for systems that could service multiple concurrent user operations and excessive data generation became mainstream. Despite the impressive rate of uniprocessor performance, multiprocessor systems were better equipped to handle this workload. In order to provide a cost-effective system, shared memory address space became the focus of research. Early on, systems using a crossbar switch were advocated, however with this design complexity scaled along with the increase of processors, which made the bus-based system more attractive. Processors in a bus system are allowed to access the entire memory space by sending requests on the bus, a very cost effective way to use the available memory as optimally as possible. However, bus-based systems have their own scalability problems. The main issue is the limited amount of bandwidth, this restrains the number of processors the bus can accommodate. Adding CPUs to the system introduces two major areas of concern: The available bandwidth per node decreases as each CPU added. The bus length increases when adding more processors, thereby increasing latency. The performance growth of CPU and specifically the speed gap between the processor and the memory performance was, and actually still is, devastating for multiprocessors. Since the memory gap between processor and memory was expected to increase, a lot of effort went into developing effective strategies to manage the memory systems. One of these strategies was adding memory cache, which introduced a multitude of challenges. Solving these challenges is still the main focus of today for CPU design teams, a lot of research is done on caching structures and sophisticated algorithms to avoid cache misses. Introduction of caching snoop protocols Attaching a cache to each CPU increases performance in many ways. Bringing memory closer to the CPU reduces the average memory access time and at the same time reducing the bandwidth load on the memory bus. The challenge with adding cache to each CPU in a shared memory architecture is that it allows multiple copies of a memory block to exist. This is called the cache-coherency problem. To solve this, caching snoop protocols were invented attempting to create a model that provided the correct data while not trying to eat up all the bandwidth on the bus. The most popular protocol, write invalidate, erases all other copies of data before writing the local cache. Any subsequent read of this data by other processors will detect a cache miss in their local cache and will be serviced from the cache of another CPU containing the most recently modified data. This model saved a lot of bus bandwidth and allowed for Uniform Memory Access systems to emerge in the early 1990s. Modern cache coherency protocols are covered in more detail by part 3. Uniform Memory Access Architecture Processors of Bus-based multiprocessors that experience the same - uniform - access time to any memory module in the system are often referred to as Uniform Memory Access (UMA) systems or Symmetric Multi-Processors (SMPs). With UMA systems, the CPUs are connected via a system bus (Front-Side Bus) to the Northbridge. The Northbridge contains the memory controller and all communication to and from memory must pass through the Northbridge. The I/O controller, responsible for managing I/O to all devices, is connected to the Northbridge. Therefore, every I/O has to go through the Northbridge to reach the CPU. Multiple buses and memory channels are used to double the available bandwidth and reduce the bottleneck of the Northbridge. To increase the memory bandwidth even further some systems connected external memory controllers to the Northbridge, improving bandwidth and support of more memory. However due the internal bandwidth of the Northbridge and the broadcasting nature of early snoopy cache protocols, UMA was considered to have a limited scalability. With today’s use of high-speed flash devices, pushing hundreds of thousands of IO’s per second, they were absolutely right that this architecture would not scale for future workloads. Non-Uniform Memory Access Architecture To improve scalability and performance three critical changes are made to the shared-memory multiprocessors architecture; Non-Uniform Memory Access organization Point-to-Point interconnect topology Scalable cache coherence solutions 1: Non-Uniform Memory Access organization NUMA moves away from a centralized pool of memory and introduces topological properties. By classifying memory location bases on signal path length from the processor to the memory, latency and bandwidth bottlenecks can be avoided. This is done by redesigning the whole system of processor and chipset. NUMA architectures gained popularity at the end of the 90’s when it was used on SGI supercomputers such as the Cray Origin 2000. NUMA helped to identify the location of the memory, in this case of these systems, they had to wonder which memory region in which chassis was holding the memory bits. In the first half of the millennium decade, AMD brought NUMA to the enterprise landscape where UMA systems reigned supreme. In 2003 the AMD Opteron family was introduced, featuring integrated memory controllers with each CPU owning designated memory banks. Each CPU has now its own memory address space. A NUMA optimized operating system such as ESXi allows workload to consume memory from both memory addresses spaces while optimizing for local memory access. Let’s use an example of a two CPU system to clarify the distinction between local and remote memory access within a single system. The memory connected to the memory controller of the CPU1 is considered to be local memory. Memory connected to another CPU socket (CPU2)is considered to be foreign or remote for CPU1. Remote memory access has additional latency overhead to local memory access, as it has to traverse an interconnect (point-to-point link) and connect to the remote memory controller. As a result of the different memory locations, this system experiences “non-uniform” memory access time. 2: Point-to-Point interconnect AMD introduced their point-to-point connection HyperTransport with the AMD Opteron microarchitecture. Intel moved away from their dual independent bus architecture in 2007 by introducing the QuickPath Architecture in their Nehalem Processor family design. The Nehalem architecture was a significant design change within the Intel microarchitecture and is considered the first true generation of the Intel Core series. The current Broadwell architecture is the 4th generation of the Intel Core brand (Intel Xeon E5 v4), the last paragraph contains more information on the microarchitecture generations. Within the QuickPath architecture, the memory controllers moved to the CPU and introduced the QuickPath point-to-point Interconnect (QPI) as data-links between CPUs in the system. The Nehalem microarchitecture not only replaced the legacy front-side bus but reorganized the entire sub-system into a modular design for server CPU. This modular design was introduced as the “Uncore” and creates a building block library for caching and interconnect speeds. Removing the front-side bus improves bandwidth scalability issues, yet intra- and inter-processor communication have to be solved when dealing with enormous amounts of memory capacity and bandwidth. Both the integrated memory controller and the QuickPath Interconnects are a part of the Uncore and are Model Specific Registers (MSR) ). They connect to a MSR that provides the intra- and inter-processor communication. The modularity of the Uncore also allows Intel to offer different QPI speeds, at the time of writing the Intel Broadwell-EP microarchitecture (2016) offers 6.4 Giga-transfers per second (GT/s), 8.0 GT/s and 9.6 GT/s. Respectively providing a theoretical maximum bandwidth of 25.6 GB/s, 32 GB/s and 38.4 GB/s between the CPUs. To put this in perspective, the last used front-side bus provided 1.6 GT/s or 12.8 GB/s of platform bandwidth. When introducing Sandy Bridge Intel rebranded Uncore into System Agent, yet the term Uncore is still used in current documentation. You can find more about QuickPath and the Uncore in part 2. 3: Scalable Cache Coherence Each core had a private path to the L3 cache. Each path consisted of a thousand wires and you can imagine this doesn’t scale well if you want to decrease the nanometer manufacturing process while also increasing the cores that want to access the cache. In order to be able to scale, the Sandy Bridge Architecture moved the L3 cache out of the Uncore and introduced the scalable ring on-die Interconnect. This allowed Intel to partition and distribute the L3 cache in equal slices. This provides higher bandwidth and associativity. Each slice is 2.5 MB and one slice is associated with each core. The ring allows each core to access every other slice as well. Pictured below is the die configuration of a Low Core Count (LCC) Xeon CPU of the Broadwell Microarchitecture (v4) (2016). This caching architecture requires a snooping protocol that incorporates both distributed local cache as well as the other processors in the system to ensure cache coherency. With the addition of more cores in the system, the amount of snoop traffic grows, since each core has its own steady stream of cache misses. This affects the consumption of the QPI links and last level caches, requiring ongoing development in snoop coherency protocols. An in-depth view of the Uncore, scalable ring on-Die Interconnect and the importance of caching snoop protocols on NUMA performance will be included in part 3. Non-interleaved enabled NUMA = SUMA Physical memory is distributed across the motherboard, however, the system can provide a single memory address space by interleaving the memory between the two NUMA nodes. This is called Node-interleaving (setting is covered in part 2). When node interleaving is enabled, the system becomes a Sufficiently Uniform Memory Architecture (SUMA). Instead of relaying the topology info and nature of the processors and memory in the system to the operating system, the system breaks down the entire memory range into 4KB addressable regions and maps them in a round robin fashion from each node. This provides an ‘interleaved’ memory structure where the memory address space is distributed across the nodes. When ESXi assigns memory to virtual machine it allocates physical memory located from two different nodes when the physical CPU located in Node 0 needs to fetch the memory from Node 1, the memory will traverse the QPI links. The interesting thing is that the SUMA system provides a uniform memory access time. Only not the most optimal one and heavily depends on contention levels in the QPI architecture. Intel Memory Latency Checker was used to demonstrate the differences between NUMA and SUMA configuration on the same system. This test measures the idle latencies (in nanoseconds) from each socket to the other socket in the system. The latency reported of Memory Node 0 by Socket 0 is local memory access, memory access from socket 0 of memory node 1 is remote memory access in the system configured as NUMA. NUMA Memory Node 0 Memory Node 1 - SUMA Memory Node 0 Memory Node 1 Socket 0 75.7 132.0 - Socket 0 105.5 106.4 Socket 1 131.9 75.8 - Socket 1 106.0 104.6 As expected interleaving is impacted by constant traversing the QPI links. The idle memory test is the best case scenario, a more interesting test is measuring loaded latencies. It would have been a bad investment if your ESXi servers are idling, therefor you can assume that an ESXi system is processing data. Measuring loaded latencies provides a better insight on how the system will perform under normal load. During the test the load injection delays are automatically changed every 2 seconds and both the bandwidth and the corresponding latency is measured at that level. This test uses 100% read traffic.NUMA test results on the left, SUMA test results on the right. The reported bandwidth for the SUMA system is lower while maintaining a higher latency than the system configured as NUMA. Therefore, the focus should be on optimizing the VM size to leverage the NUMA characteristics of the system. Nehalem & Core microarchitecture overview With the introduction of the Nehalem microarchitecture in 2008, Intel moved away from the Netburst architecture. The Nehalem microarchitecture introduced Intel customers to NUMA. Along the years Intel introduced new microarchitectures and optimizations, according to its famous Tick-Tock model. With every Tick, optimization takes place, shrinking the process technology and with every Tock a new microarchitecture is introduced. Even though Intel provides a consistent branding model since 2012, people tend to Intel architecture codenames to discuss the CPU tick and tock generations. Even the EVC baselines lists these internal Intel codenames, both branding names and architecture codenames will be used throughout this series: Microarchitecture DP servers Branding Year Cores LLC (MB) QPI Speed (GT/s) Memory Frequency Architectural change Fabrication Process Nehalem x55xx 10-2008 4 8 6.4 3xDDR3-1333 Tock 45nm Westmere x56xx 01-2010 6 12 6.4 3xDDR3-1333 Tick 32nm Sandy Bridge E5-26xx v1 03-2012 8 20 8.0 4xDDR3-1600 Tock 32nm Ivy Bridge E5-26xx v2 09-2013 12 30 8.0 4xDDR3-1866 Tick 22 nm Haswell E5-26xx v3 09-2014 18 45 9.6 4xDDR3-2133 Tock 22nm Broadwell E5-26xx v4 03-2016 22 55 9.6 4xDDR3-2400 Tick 14 nm Up next, Part 2: System Architecture The 2016 NUMA Deep Dive Series: Part 0: Introduction NUMA Deep Dive Series Part 1: From UMA to NUMA Part 2: System Architecture Part 3: Cache Coherency Part 4: Local Memory Optimization Part 5: ESXi VMkernel NUMA Constructs ================================================================================ Title: Introduction 2016 NUMA Deep Dive Series URL: https://frankdenneman.ai/2016-07-06-introduction-2016-numa-deep-dive-series/ Date: 2016-07-06 Recently I’ve been analyzing traffic to my site and it appears that a lot CPU and memory articles are still very popular. Even my first article about NUMA published in february 2010 is still in high demand. And although you see a lot of talk about the upper levels and overlay technology today, the focus on proper host design and management remains. After all, it’s the correct selection and configuration of these physical components that produces a consistent high performing platform. And it’s this platform that lays the foundation for the higher services and increased consolidating ratios. Most of my NUMA content published throughout the years is still applicable to the modern datacenter, yet I believe the content should be refreshed and expanded with the advancements that are made in the software and hardware layers since 2009. To avoid ambiguity, this deep dive is geared towards configuring and deploying dual socket systems using recent Intel Xeon server processors. After analyzing the dataset of more than 25.000 ESXi host configurations collected from virtual datacenters worldwide, we discovered that more than 80% of ESXi host configuration are dual socket systems. Today, according to IDC, Intel controls 99 percent of the server chip market. Despite the strong focus of this series on the Xeon E5 processor in a dual socket setup, the VMkernel, and VM content is applicable to systems running AMD processors or multiprocessor systems. No additional research was done on AMD hardware configurations or performance impact when using high-density CPU configurations. The 2016 NUMA Deep Dive Series The 2016 NUMA Deep Dive Series consists of 7 parts.The 2016 NUMA deep dive series is split into three main categories; Physical, VMkernel, and Virtual Machine. Part 1: From UMA to NUMA Part 1 covers the history of multi-processor system design and clarifies why modern NUMA systems cannot behave as UMA systems anymore. Part 2: System Architecture The system architecture part covers the Intel Xeon microarchitecture and zooms in on the Uncore. Primarily focusing on Uncore frequency management and QPI design decisions. Part 3: Cache Coherency The unsung hero of today’s NUMA architecture. Part 3 zooms in to cache coherency protocols and the importance of selection the proper snoop mode. Part 4: Local Memory Optimization Memory density impacts the overall performance of the NUMA system, part 4 dives into the intricacy of channel balance and DIMM per Channel configuration. Part 5: ESXi VMkernel NUMA Constructs The VMkernel has to distribute the virtual machines to provide the best performance. This part explores the NUMA constructs that are subject to initial placement and load-balancing operations. Part 6: NUMA Initial Placement and Load Balancing Operations The VMkernel has to distribute the virtual machines to provide the best performance. This part explores the NUMA initial placement and load-balancing operations. (not yet released) Part 7: From NUMA to UMA The world of IT moves in loops of iteration, the last 15 years we moved from UMA to NUMA systems, which today’s focus on latency and the looming licensing pressure, some forward-thinking architects are looking into creating high performing UMA systems. (not yet released) The articles will be published on a daily basis to avoid saturation. Similar to other deep dives, the articles are lengthy and contain lots of detail. Up next, Part 1: From UMA to NUMA ================================================================================ Title: Top 5 vBlog Again, Thanks!!!! URL: https://frankdenneman.ai/2016-07-01-top-5-vblog/ Date: 2016-07-01 Yesterday the top 25 vBlogs were announced and once again I’m in the top 5. I would like to thank all who have voted for me! It’s great to see that the content is appreciated. The broadcast: https://youtu.be/-Kb6gAys0jc Looking forward, there is a lot of content getting ready to be published and I hope to release my 5th book this year, the vSphere 6.x host resource deep dive. I’m excited about the content I’m working on and I’ll hope you guys will too! Thanks! Frank ================================================================================ Title: New Home Lab Hardware - Dual Socket Xeon v4 URL: https://frankdenneman.ai/2016-06-22-new-home-lab-hardware-dual-socket-xeon-v4/ Date: 2016-06-22  A new challenge - a new system for the home lab About one year ago my home lab was expanded with a third server and a fresh coat of networking. During this upgrade, which you can read at “When your Home Lab turns into a Home DC” I faced the dilemma of adding new a new generation of CPU (Haswell) or expanding the hosts with another Ivy Bridge system. This year I’ve proceeded to expand my home lab with a dual Xeon system, and decided to invest in the latest and greatest hardware available. Like most good tools, you buy them for the next upcoming job, but in the end, you will use it for countless other projects. I expect the same thing with this year’s home lab ‘augmentation’. Initially, the dual socket system will be used to test and verify the theory published in the upcoming book “vSphere 6 Host resource deep dive” and the accompanying VMworld presentation (session id 8430), but I have a feeling that it’s going to become my default test platform. Besides the dual socket system, the Intel Xeon 1650 v2 servers are expanded with more memory and a Micron 9100 PCIe NVMe SSD 1.2 TB flash device. Listed below is the bill of materials of the dual socket system: Amount Component Type Cost in Euro 2 CPU Intel Xeon E5 2630 v4 1484 (742) 2 CPU Cooler Noctua NH-U12DX i4 CPU Cooler 118 (59) 1 Motherboard Supermicro X10DRi-T 623 8 Memory Kingston KVR24R17D8/16MA - 16 GB 2400 MHz CL17 760 (95) 1 Flash Device Micron 9100 PCIe NVMe SSD 1.2 TB Sample 1 Flash Device Intel PC P3700 PCIe NVMe SSD 400 GB Sample 1 Flash Device Intel SSD PC S3700 100 GB 170 1 Ethernet Controller HP NC365T 4-port Ethernet Server Adapter 182 1 Case Fan Noctua NF-A14 FLX FAN 140 MM 25 1 Case Fractal Design Define XL R2 - Titanium 135 Total Cost 3497 EUR Update: I received a lot of questions about the cost of this system, I’ve listed the price in EURO’s. With todays exchange rate Euro to USD 1.1369 it’s about 3976 U.S. Dollar. Dual socket configuration Building a dual socket system is still interesting, for me it brought back the feeling of the times when I built a dual Celeron system. Some might remember the fantastic Abit BP6 motherboard with the Intel BX440 chip. Do you know today’s virtual machines still use a virtualized BX440 chipset? But I digress. Building a dual socket system is less difficult than in those days when you had to drill a hole in CPU to attach it to a specific voltage, but it still has some minor challenges. Form Factor A dual-socket design requires more motherboard real estate than a single socket system. However there are some dual socket motherboards that are offered in the popular ATX form factor format, however, concessions are made by omitting some components. Typically this results in a reduced number of PCIe slots or the absolute minimum number of DIMM slots supported by the CPUs. Typically, you end up with selecting an E-ATX motherboard or if you really feel adventures EE-ATX or a proprietary format of the motherboard manufacturer. I wanted to have a decent amount of PCI-E slots as it needs to fit both the Intel and the Micron NVMe devices as well as the Quad 1 GB NIC. On the plus side, there seems to be a decent amount of PC cases available that support this E-ATX format. One of them is the Fractal Design Define R4, but there are many others. I selected the Fractal R4 as it’s the same case design all the other servers use, only slightly bigger to fit the motherboard. The build quality of this case is some of the best I’ve seen. Due to all the noise-reducing material its quite heavy. Although it states it supports E-ATX on their spec sheet I assume they’ve only focused on the size of the motherboard. Unfortunately, the chassis does not contain the necessary mounting holes to secure the motherboard properly. In total 4 mounting points cannot be used, as the E-ATX uses in total 10 mounting points it should not be a big problem, however, it is missing a crucial one, the one in the top left corner. This might lead to problems when installing the DIMMs or the CPUs. I solved this by drilling a hole in the chassis to mount a brass grommet, but next time I would rather go for a different case. The red circles indicate the missing grommets. Power Supply Dual Socket systems are considered heavy load configurations and require the dual 8 Pin EPS 12V connectors to be populated. Make sure that your power supply kit contains these connectors. Pictured below are the 12v 8-pin power connectors located on the motherboard. When researching power supplies, I noticed that other people prefer to use 700 or 1000 watt power supplies. I don’t believe you need to go that extreme. The amount of watts required all depends on what configuration you want to run. In my design I’m not going to run dual SLI video card, it will ‘only’ contain the Intel Xeon 2640 v4 CPUs, 3 PCIe devices and 8 DDR4 2400 MHz Modules. Although it sounds like an extreme configuration already, it’s actually not that bad. Let’s do the math. With a TDP value of 85 watts each, the CPU’s consume a maximum of 170 W. The PCIe devices increases the power requirement to 207 W. According to Intel the DC P3700 NVMe device consumes 12 W on write, 9 of read. The quad NIC Ethernet controller is reported to consume 5 W. Micron states that the active power consumption of the P9100 1.2TB PCIe device is between 7 and 21 W. DDR4 DIMM voltage is set to 1.2V, compared to the 1.5V DDR3 requires, the reduce voltage which likely translate in a lower power consumption than similar DDR3 configurations. Unfortunately, Memory vendors do not provide exact power consumption specs. Toms Hardware measured the power consumption of 4 DDR4 modules and discovered the consumption ranged from 6 to 12 W depending on the manufacturer. Worst case scenario, my 8 DIMMS consume 24 W. As the motherboard is quite large I assume it consumes a great deal of power as well, build computer states motherboard power consumption of high-end motherboards is between 45 to 80 Watts. As the board features two X540 10 G NICS, I will add the 12.5 W of power consumption stated by Intel to the overall motherboard consumption. In my calculation I assume a consumption of a 100 W . The Intel SSD DC3700 100 GB acting as base OS disk is rated to consume 2.9. This totals to a power consumption of roughly 330 W. There are some assumptions made therefore I’m playing it safe by using the Corsair RM 550 power supply which provides an output of 550W at 12 Volt. The cooling solutions don’t move the needle that much, but for completeness sake, I’ve included them in the table. Component Estimated Active Power Consumption Vendor Spec Intel Xeon E5 2630 v4 85 * 2 = 170 W ark.intel.com Micron P9100 1.2 TB 21 W Micron Product Brief (PDF) Kingston KVR24R17D8/16MA ~24 W Toms Hardware review Intel SSD PC P3700 400 GB 12 W ark.intel.com HP NC365T 4-port Ethernet Server Adapter 5 W HP.com Intel SSD PC S3700 100 GB 2.9 W ark.intel.com Noctua NF-A14 FLX FAN 140 MM 0.96 W Noctua.at Noctua NH-U12DX i4 CPU Cooler 2 * 0.6 = 1.2 W Noctua.at Supermicro X10DRi-T ~80 W Buildcomputers.net Intel X540-AT2 Dual 10 Gb Ethernet Controller 12.5 W ark.intel.com Total Power Consumption ˜330 W One thing you might want to consider is the fan noise when buying “Zero RPM Fan Mode” power supply. Typically these power supplies can operate without using the fan up to an X percentage of system load. It will increase the use of the fan when the system loads go up. With my calculation I operate in the 60% system load range, above the threshold of the Zero RPM Fan Mode nut still in the Low Noise mode, while benefiting from the maximum efficiency of the power supply. Cooling With more power consumptions comes the great need of cooling. And as we all know power consumption is the path to the dark side, power consumption leads to heat, heat leads to active cooling. Active cooling leads to noise. Noise leads to suffering. Or something like that. To reduce the noise generated by the home lab, I use Noctua cooling. High-quality low noise, it cost a pretty penny, but as always, quality products cost more. In the UP (Uni-Processor) servers, I use the Noctua NH-U9DX-I4, which is an absolute behemoth. Due to the dual CPU setup, I selected the Noctua NH-U12DX i4 CPU Cooler which is specified as a slim design. In retrospect, I could have gone with the U9DX-I4 as well. Detailed information on both CPU coolers: https://www.quietpc.com/nh-udxi4 Please ensure that your choice of cooler supports the 2011-3 socket configuration. According to Wikipedia: The 2011-3 socket use the so-called Independent Loading Mechanism (ILM) retention device that holds the CPU in place on the motherboard. 2011-3. Two types of ILM exist, with different shapes and heatsink mounting hole patterns. The square ILM (80×80 mm mounting pattern), and narrow ILM (56×94 mm mounting pattern). Square ILM is the standard type, while the narrow one is alternatively available for space-constrained applications. It’s not a surprise to see the Supermicro X10DRi-T features the arrow ILM configuration. Noctua ships the coolers with mounting kits for both ILM configuration, check your motherboard specs and the supported ILM configuration of your cooling solution before ordering. Micron 9100 PCIe NVMe SSD 1.2 TB Recently Micron supplied me with three engineering samples of their soon to be released PCIe NVMe SSD device. According to the product brief the 1.2 TB version provides 2.8/1.3 GB/s sequential write speed at a steady state when using a 128KB transfer size, impressive to say the least. The random read/write performance of the device of 4 KB blocks are 700.000 IOPS/180.000 IOPS. Remember the times where you were figuring out if you had to 150 IOPS or 180 IOPS when using a 15K spindle disk. :) I’m planning to use the devices to create a EMC ScaleIO 2.0 architecture. Paired together with DFTM and a 10Gb network, this will be a very interesting setup. Mark Brookfield published an extensive write up of the ScaleIO 2.0 installation. Expect a couple of blog posts about the performance insights on Scale IO soon. RAM for Intel Xeon 1650 V2 servers I’ve purchased some additional RAM to further test the in-memory I/O acceleration of FVP (DFTM) and the impact of various DIMM Per Channel configuration on memory bandwidth. One server will have a configuration of 2 DPC, containing 128 GB of DDR3 1600MHz RAM, the second UP server is also 2 DPC configuration, equipped with 128GB of DDR3 1866 MHz and the Dual Xeon system runs a 1 DPC configuration with DDR4 2400 MHz RAM. The attentive reader will notice that I’ve over-specced the memory for the Intel Xeon v4 as this CPU supports memory up to 2133 MHz. Apparently 2400 MHz memory is produced more than the 2133 MHz equivalent, resulting in cheaper 2400 MHz memory. The mainboard adjusts the memory to the supported frequency accordingly. The various memory configurations will also aid in the development of the FVP and Architect coverage. We recently released FVP 3.5 and Architect 1.1 and this release provided the long awaited management virtual appliance. Couple that with the FVP freedom edition (RAM acceleration) and you can speed up your home lab with just a couple of clicks. I will publish an article on this soon. ================================================================================ Title: vSphere 6.x host resource deep dive session (8430) accepted for VMworld US and Europe URL: https://frankdenneman.ai/2016-06-16-vsphere-6-x-host-resource-deep-dive-session-8430-accepted-for-vmworld-us-and-europe/ Date: 2016-06-16 Yesterday both Niels and I received the congratulatory message from the VMworld team, informing us that our session is accepted for both VMworld US and Europe. We are both very excited that our session was selected and we are looking forward at presenting to the VMworld audience. Our session is called the vSphere 6.x host resource deep dive (session ID 8430) and is an abstract of our similar titled book (publish date will be disclosed soon). Session Outline Today’s focus is on upper levels/overlay’s (SDDC stack, NSX, Cloud) but proper host design and management still remains the foundation of success. With the introduction of these new ‘overlay’ services, we are presented with a new consumer of host resources. Ironically it’s the attention to these abstraction layers that returns us to focusing on individual host components. Correct selection and configuration of these physical components leads to creating a stable high performing platform, that lays the foundation for the higher services and increased consolidating ratios. Topics we will address in this presentation are: The introduction of NUMA (Non-Uniform Memory Access) required changes in memory management. Host physical memory is now split into local and remote memory structures for CPUs that can impact virtual machine performance. We will discuss how to right size your VMs CPU and memory configuration in regards to NUMA and vNUMA VMkernel CPU scheduler characteristics. Processor speed and core counts are important factors when designing a new server platform. However with virtualization platforms the memory subsystem can have equal or sometimes even have a greater impact on application performance than the processor speed. In this talk we focus on physical memory configurations. Providing consistent performance is key to predictable application behavior. It benefits day-to-day customer satisfaction and helps reduce application performance troubleshooting. This talk covers flash architecture and highlights the differences between the predominant types of local storage technologies. We look closer into recurring questions about virtual networking. For example, how many resources does the VMkernel claim for networking, what impact does a vNIC type has on resource consumption. Such info allows you to get better grips on sizing your virtual datacenter for NFV workloads. Key Takeaway 1: Identifying how proper NUMA and physical memory configuration allows for increased VM performance Key Takeaway 2: What is the impact of virtual network services on consumption of host compute resources? Key Takeaway 3: How next-gen storage components lead to low latency, higher bandwidth and increased scalability. Key dates: VMworld US takes place at Mandalay Bay Hotel & Convention Center in Las Vegas, NV from August 28 - September 1, 2016 VMworld Europe takes place at Fira Barcelona Gran Via in Barcelona, Spain from 17 - 20 October, 2016 Repeat the feat Five years ago Duncan and I got this room completely full with our vSphere Clustering Deepdive Q&A, I would love to repeat that feat doing a Host deep dive session. I hope to see you all in our session! ================================================================================ Title: Home Lab Fundamentals: DNS Reverse Lookup Zones URL: https://frankdenneman.ai/2016-06-13-home-lab-fundamentals-dns-reverse-lookup-zones/ Date: 2016-06-13 When starting your home lab, all hints and tips are welcome. The community is full of wisdom, yet sometimes certain topics are taken for granted or are perceived as common knowledge. The Home Lab fundamentals series focusses on these subjects, helping you how to avoid common pitfalls that provide headaches and waste incredible amounts of time. One thing we always keep learning about vSphere is that both time and DNS needs to be correct. DNS resolution is important to many vSphere components. You can go a long way without DNS and use IP-addresses within your lab, but at one point you will experience weird behavior or installs just stop without any clear explanation.In reality vSphere is build for professional environments where it’s expected that proper networking structure is in place, physical and logical. When reviewing a lot of community questions, blog posts and tweets, it appears that DNS is partially setup, i.e. only forward lookup zones are configured. And although it appears to be ‘‘just enough DNS to get things going, many have experienced that their labs start to behave differently when no reverse lookup zones are present. Time-outs or delays are more frequent, the whole environment isn’t snappy anymore. Ill-configured DNS might give you the idea that the software is crap but in reality, it’s the environment that is just configured crappy. When using DNS, use the four golden rules; forward, reverse, short and full. DNS in a lab environment isn’t difficult to set up and if you want to simulate a proper working vSphere environment then invest time in setting up a DNS structure. It’s worth it! Besides expanding your knowledge, your systems will feel more robust and believe me, you will wait a lot less on systems to respond. vCenter and DNS vCenter inventory and search rely heavy on DNS. And since the introduction of vCenter Single Sign-On service (SSO) as a part of the vCenter Server management infrastructure DNS has become a crucial element. SSO is an authentication broker and security token exchange infrastructure. As described in the KB article Upgrading to vCenter Server 5.5 best practices (2053132); With vCenter Single Sign-On, local operating system users become far less important than the users in a directory service such as Active Directory. As a result, it is not always possible, or even desirable, to keep local operating system users as authenticated users. This means that you are somewhat pressured into using an ’external’ identity source for user authentication, even for your lab environment . One of the most popular configurations is the use of Active Directory as an identity source. Active Directory itself uses DNS as the location mechanism for domain controllers and services. If you have configured SSO to use Microsoft Active Directory for authentication, you might have seen some weird behavior when you haven’t created a reverse DNS lookup zone. Installation of vCenter Server (Appliance) fails if the FQDN and IP addresses used are not resolvable by the DNS server specified during the deployment process. The vSphere 6.0 Documentation Center vSphere DNS requirements state the following: Ensure that DNS reverse lookup returns a Fully Qualified Domain Name (FQDN) when queried with the IP address of the host machine on which vCenter Server is installed. When you install or upgrade vCenter Server, the installation or upgrade of the Web server component that supports the vSphere Web Client fails if the installer cannot look up the fully qualified domain name of the vCenter Server host machine from its IP address. Reverse lookup is implemented using PTR records. Before deploying vCenter I recommend to deploy a virtual machine on the first host running a DNS server. The ESXi Embedded Host Client allows you to deploy a virtual machine on an ESXi host without the need of having an operational vCenter first. As I use active Directory as identity source for authentication, I deploy a Windows AD server with DNS before deploying the vCenter Server Appliance (VCSA). Toms IT pro has a great article on how to configure DNS on a Windows 2012 server, but if you want to configure a lightweight DNS server running on Linux, follow the steps Brandon Lee has documented. If you want to explore the interesting world of DNS, you can also opt to use Dynamic DNS to automatically register both the VCSA and ESXi hosts in the DNS server. Dynamic DNS registration is the process by which a DHCP client register its DNS with a name server. For more information please check out William article “Does ESXi Support DDNS (Dynamic DNS)?” . Although he published it in 2013. it’s still a valid configuration in ESXi 6.0. Flexibility of using DNS Interestingly enough, having a proper DNS structure in place before deploying the virtual infrastructure provides future flexibility. One of the more annoying time wasters is the result of using an IP address instead of an FQDN during setup of the VCSA. When you use only an IP-address instead of a Fully Qualified Domain Name (FQDN) during setup, changing the hostname or IP-address will produce this error: IPv4 configuration for nic0 of this node cannot be edited post deployment. Kb article 2124422 states the following: Attempting to change the IP address of the VMware vCenter Server Appliance 6.0 fails with the error: IPv4 configuration for nic0 of this node cannot be edited post deployment. (2124422) This occurs when the VMware vCenter Server Appliance 6.0 is deployed using an IP address. During the initial configuration of the VMware vCenter Server Appliance, the system name is used as the Primary Network Identifier. If the Primary Network Identifier is an IP address, it cannot be changed after deployment. This is an expected behavior of the VMware vCenter Server Appliance 6.0. To change the IP address for the VMware vCenter Server Appliance 6.0 that was deployed using an IP address, not a Fully Qualified Domain Name, you must redeploy the appliance with the new IP address information. Changing the hostname will result in the Platform Service Controller (responsible for SSO) to fail. According to Kb article: Changing the IP address or host name of the vCenter Server or Platform Service controller cause services to fail (2130599) Changing the Primary Network Identifier (PNID) of the vCenter Server or PSC is currently not supported and will cause the vSphere services to fail to start. If the vCenter Server or PSC has been deployed with an FQDN or IP as the PNID, you will not be able to change this configuration. To resolve this issue, use one of these options: Revert to a snapshot or backup prior to the IP address or hostname change. Redeploy the vSphere environment. This means that you cannot change the IP-address or the host name of the vCenter Appliance. Yet another reason to deploy a proper DNS structure before deploying your VCSA in your lab. FQDN and vCenter permissions Even when you have managed to install vCenter without a reverse lookup zone, the absence of DNS pointer records can obstruct proper permission configuration according to (KB article 2127213) Unable to add Active Directory users or groups to vCenter Server Appliance or vRealize Automation permissions Attempting to browse and add users to the vCenter Server permissions (Local Permission: Hosts and Clusters > vCenter >Manage >Permissions)(Global Permissions: Administration > Global Permissions) fails with the error: Cannot load the users for the selected domain A workaround for this issue is to ensure that all DNS servers have the Reverse Lookup Zone configured as well as Active Directory Domain Controller (AD DC) Pointer (PTR) records present. Please note that allowing domain authentication (assuming AD) on the ESXi host does not automatically add it to an AD managed DNS zone. You’ll need to manually create the forward lookup (which will give the option for the reverse lookup creation too). SSH session password delay When running multiple hosts most of you will recognize the waste of time when (quickly) wanting to log into ESXi via an SSH session. Typically this happens when you start a test and you want to monitor ESXTOP output. You start your ssh session, to save time you type on the command line ssh root@esxi.homelab.com and then you have to wait more than 30 seconds to get a password prompt back. Especially funny when you are chasing a VM and DRS decided to move it to another server when you weren’t paying attention. To get rid of this annoying time waster forever: DNS name resolution using nslookup takes up to 40 seconds on an ESXi host(KB article 2070192) When you do not have a reverse lookup zone configured, you may experience a delay of several seconds when logging in to hosts via SSH. When you’re management machine is not using the same DNS structure, you can apply the quick hack of adding “useDNS no” to the /etc/ssh/sshd.config file on the ESXi host to avoid the 30-second password delay. Troubleshoot DNS BuildVirtual.net published an excellent article on how to troubleshoot ESXi Host DNS and Routing related issues. For more information about setting the DNS configuration from the command line, review this section of the VMware vSphere 6.0 Documentation Center vSphere components moving away from DNS As DNS is an extra dependency, a lot of newer technologies try to avoid incorporate DNS dependencies. One of those is VMware HA. HA has been redesigned and the new FDM architecture avoided DNS dependencies. Unfortunately not all VMware official documentation has been updated with this notion: https://kb.vmware.com/kb/1003735 states that ESX 5.x also has this problem but that is not true. Simply put, VMware HA in vSphere 5.x and above does not depend on DNS for operations or configurations. Home Lab Fundamentals Series: Time Sync DNS Reverse Lookup Zones Up next in this series: vSwitch0 routing ================================================================================ Title: Home Lab Fundamentals: Time Sync URL: https://frankdenneman.ai/2016-06-03-home-lab-fundamentals-time-sync/ Date: 2016-06-03 First rule of Home Lab club, don’t talk about time sync! Or so it seems. When starting your home lab, all hints and tips are welcome. The community is full of wisdom, however sometimes certain topics are taken for granted or are perceived as common knowledge. The Home Lab fundamentals series focusses on these subjects, helping you how to avoid the most common pitfalls that provide headaches and waste incredible amounts of time. A ’time-consuming’ pitfall is dealing with improper time synchronization between the various components in your lab environment. Most often, the need for time synchronization is seen as an Enterprise requirement but not really necessary for lab environments. Maybe because most think time synchronization is solely necessary for troubleshooting purposes. In some cases, this is true as ensuring correct time notation allows for proper correlation of events. Interestingly enough, this alone should be enough reason to maintain synchronized clocks throughout your lab, but most home labs are just rebuilt when troubleshooting becomes too time-consuming. However time sync is much more expedite troubleshooting and ignoring time drift is a straight path into the rabbit hole. Time synchronization utilities such as NTP are necessary to correct time drift introduced by hardware time drift and guest operating system timekeeping imprecision. When time differs between systems to much it can lead to installation and authentication errors. Unfortunately, time issues are not always easily identifiable, to provide a great example; “[400] An error occurred while sending an authentication request to the vCenter Single Sign-On server – An error occurred when processing the metadata during vCenter Single Sign-On setup – null.” This particular issue occurs due to a time skew between the vCenter Server Appliance 6.0 and the external Platform Service Controller. Here are just a few other examples of what can go wrong in your lab due to time skew issues; Adding a host in vCenter Server fails with the error: Failed to configure the VIM account on the host (1029863) Time skew between ESXi host hardware clock and vCenter Server system time. https://kb.vmware.com/kb/1029863 After joining the Virtual Center Server Appliance to a domain you cannot see domain when adding user permissions (2011965): This issue occurs when the time skew between the Virtual Center Server Appliance(VCSA) and a related Domain Controller is greater than 5 minutes. https://kb.vmware.com/kb/2011965 Cluster level performance graphs show the most recent value as 0: This metric is susceptible to clock skew between the vSphere Client, vCenter Server, and ESX hosts. If any of the hosts have a skewed clock, the entire cluster shows as 0. https://kb.vmware.com/kb/2009550 The vCenter Server Appliance installation fails when connecting to an External Platform Services Controller: This issue occurs when the system time on the system hosting the PSC does not match the time of the system where vCenter Server is installed. https://kb.vmware.com/kb/2128811 Configuring the NSX SSO Lookup Service fails (2102041): Connectivity issues between the NSX Manager to vCenter Server due to time skew between NSX Manager and vCenter Server. https://kb.vmware.com/kb/2102041 Authentication Errors are Caused by Unsynchronized Clocks: If there is too great a time difference between the KDC and a client requesting tickets, the KDC cannot determine whether the request is legitimate or a replay. Therefore, it is vital that the time on all of the computers on a network be synchronized in order for Kerberos authentication to function properly. https://technet.microsoft.com/en-us/library/cc780011(v=ws.10).aspx Timekeeping best practices by VMware Simply put, when weird behavior during setup or authentication occurs, check the time between the various components first. VMware released multiple knowledge Base articles and technical documents that contain detailed information and instructions on timekeeping within the various components of the virtual datacenter: Timekeeping in Vmware virtual Machines: http://www.vmware.com/files/pdf/Timekeeping-In-VirtualMachines.pdf Timekeeping best practices for Windows, including NTP (1318): https://kb.vmware.com/kb/1318 Timekeeping best practices for Linux guests (1006427): https://kb.vmware.com/kb/1006427 ESX and ESXi host time keeping Best Practices: https://kb.vmware.com/kb/2004453 VMware doesn’t provide a separate time keeping best practices document for vCenter, but provides multiple guidelines in the vCenter Server Appliance configuration guide. When installing vCenter on a Windows machine it’s recommended to sync to the PDC emulator within the Active Directory domain. In general, VMware recommends to use native time synchronization software, such as Network Time Protocol (NTP) with the various vSphere components. NTP is typically more accurate than VMware Tools periodic time synchronization and is therefore preferred. Time synchronization design There are multiple schools of thoughts when it comes to time sync in a virtual data center. One of the most common ones is to synchronize the virtual datacenter infrastructure components such as ESXi hosts and the VCSA to a collection of external NTP server. Typically provided by http://www.pool.ntp.org/en/ or the US Naval observatory: http://tycho.usno.navy.mil/NTP/. Windows virtual machines sync their time to the Active Directory domain controller running the PDC emulator FSMO role: Time Synchronization in Active Directory Forests http://social.technet.microsoft.com/wiki/contents/articles/18573.time-synchronization-in-active-directory-forests.aspx It’s recommended to point the ESXi hosts to the same time source as the PDC emulator of the active directory. When running Linux best practice is to sync these systems with an NTP server. Another widely adopted design is to sync ESX servers to the to the Active Directory domain controller running the PDC emulator FSMO role. VCSA time keeping configuration provide two valid options; NTP and hosts. In this scenario, select the host option to ensure time between the host and VCSA is in sync. If the VCSA is using a different time source other than the ESX host, a race condition can occur between time sync operations and can lead to failing the vpxd. Source: VMware vCenter Server 6.0 Update 1b Release Notes: http://pubs.vmware.com/Release_Notes/en/vsphere/60/vsphere-vcenter-server-60u1b-release-notes.html But the most interesting thing I witnessed that can easily become a wild-goose chase is the VM tools time synchronization when time on an ESXi host is incorrect. As described earlier, enabling VMware tools time sync on virtual machines was a best practice for a long time. Shifting towards native time synchronization software led VMware to disable the periodic time synchronization option by default. The keyword in the last sentence is PERIODIC. By default VMware tools synchronizes time with the host during the following options: Resuming a suspended virtual machine Migrating a virtual machine using vMotion Taking a snapshot Restoring a snapshot Shrinking the virtual disk Restarting the VMware tools service inside the VM Rebooting the virtual machine The time synchronization checkbox controls only whether time is periodically resynchronized while the virtual machine is running. Even if this box is unselected, by default VMware Tools synchronizes the virtual machine’s time to the ESXi host clock after the listed events. If the ESXi host time is incorrect it is likely that “unexplainable” errors will occur. I experienced this behavior after migrating a VM with vMotion. I couldn’t log on to a windows server as the time skew prevented me from authenticating. You can either disable these options by adding rules to the VMX file of each VM or just ensure that the ESXi host is syncing the time with a proper external time source. For more information: Disabling Time Synchronization (1189) https://kb.vmware.com/kb/1189. No time zone for ESXi Be aware that as of vSphere 4.1 ESXi hosts are set to Universal Time Coordinated (UTC) time. UTC is interesting as its the successor of Greenwich Mean Time (GMT) but UTC itself is not a time zone, but a time standard. There are plenty of articles about UTC, but the key thing to understand is that it never observes Daylight Saving Time. As UTC is not a time zone, you cannot change the time notation in ESXi itself. The vSphere client, web client and HTML5 client automatically display the time in your local time zone and will take into account the UTC setting on the host. This isn’t bad behavior, just be aware of this so you don’t freak out when you check the time via the command line. CMOS clock ESXi synchronizes its system time with the hardware clock (CMOS/BIOS/ACPI)of the server if the NTP service is not running on the ESXi host. SuperMicro boards allow for NTP synchronization, but most home lab motherboards just provide the time as being configured in the BIOS. When the NTP daemon is started on the ESXi host it synchronizes its system time to the external time source AND it updates the hardware clock as well. I ran a test to verify this behavior. At the time of testing it was 12:37 (GMT+1 | UTC 10:37), NTP turned off and set the time in the BIOS to 6:37 UTC time. After booting the machine the command esxcli system time get confirmed ESX system time retrieved the time from the hardware clock. After starting the NTP Services, the system time was set to the correct time: 10:37.The command esxcli hardware clock get demonstrated that NTP also corrected the BIOS time. A quick BIOS check confirmed esxcli hardware clock get was displaying the BIOS configuration. If your lab is not connected to the internet, confirm the BIOS time with the command esxcli hardware clock get and if necessary use the command esxcli hardware clock set -d (Day) -H (Hour) -m (Minute) -M (Month) -s (Second) -y (Year) to set the correct time. Please note that ESXCLI reports time with the Z (Zulu) notation, this is the military name for UTC. Raspberry Pi as a Stratum-1 NTP Server When having a home lab, you usually face the age old dilemma common sense vs ’exciting new stuff that you might not need but you would like to have’. You can update your CMOS clock manually or scripted, you can connect to an array of external NTP servers or you can build your own Stratum-1 NTP server using a Raspberry PI with a GPS add-on board . Up next in this series:Home Lab fundamentals: Reverse DNS ================================================================================ Title: ntpq -p connection refused error message URL: https://frankdenneman.ai/2016-05-30-ntpq-p-connection-refused-error-message/ Date: 2016-05-30 Sometimes a small misconfiguration can cause havoc in a complex distributed system. It becomes really annoying when no proper output is provided by log files and status report. While investigating time issues in my lab I ran into the following error message while executing the ntpq -p command: TL;DR NTP client is disabled, enable it via the GUI The standard NTP query program (ntpq) is one of the quickest way to verify that the Network Time Protocol Daemon (ntpd) is up and running. The command ntpq -p prints a list of peers known to the ESXi host as well as a summary of their state. Running the command on another ESXi host provided the following output. Requesting the status of the NTPD status on the host with weird time issues, shows it’s not running. No proper feedback is provided by the command line other than it’s starting, no failure code is returned. Management service initialisation, such as ntpd starts are logged in the file /var/log/syslog.log in ESXi 5.1 and up. Unfortunately, nothing useful is logged in this logfile as well. I couldn’t find a command that provides accurate output whether the NTP client was enabled or not. Time to open up the web client. Host time configuration can be found when selecting the ESXi host, Manage, Time Configuration. Apparently NTP was not enabled. Simple problem to fix, unfortunately there is no simple command line function that allows to verify while NTP client is enabled (sans PowerCli) ================================================================================ Title: No network connection after re-registering VCSA using the I've copied it answer URL: https://frankdenneman.ai/2016-05-24-no-network-connection-after-re-registering-vcsa-using-the-ive-copied-it-answer/ Date: 2016-05-24 Paulo Coelho once stated “Life moves very fast. It rushes from Heaven to Hell in a matter of seconds” Well I think he perfectly described a day working in the lab and rushing through a migration. I’m upgrading the lab and I moved the vCenter Server Appliance (VCSA) to its new home. While trying to do a million things all at once, I didn’t pay attention to the question whether I moved the virtual machine or whether I copied it. I selected the option “I copied it”. And that’s when the fun started, vCenter down. TL;DR: Selecting “I copied it” implies that this machine is a duplicate and that a new identity should be generated. This means that the VM is getting a new UUID and a new MAC address. SUSE Linux Enterprise Server 11 detects this new MAC address and views this as a new Ethernet Device. The VCSA does not allow the creation of a new ethernet controller. Rename 70-persistent-net.rules file and reboot to have SUSE auto-generate a new 70-persistent-net.rules file with the correct MAC Address that allows you restore network connectivity via the console. Troubleshooting the problem Both the web client and the VCSA config web page are unreachable, time to open up the VM console (Alt-F1). When logging in and pinging the gateway the error, the system returns the error message “Network is unreachable” Before tinkering with the configuration files, I like to restart the services and see if the status report exposes interesting information. “No configuration found for eth1”. The VCSA is configured with a single NIC and SUSE Linux Enterprise Server 11, which is the OS for the appliance, assigns the label eth0 to the first Ethernet adapter. VCSA networking is configured through the Virtual Appliance Management Interface (VAMI). Executing the command “/opt/vmware/share/vami/vami_config_net allows you to retrieve the current network configuration When selecting option 6 “IP Address Allocation for eth1” VAMI reveals that it cannot read the interface files for ’eth1’ The networking interface files are stored in the directory /etc/sysconfig/networking/devices. When listing the files (ls) only ifcfg-eth0 shows up. Reviewing the ifcfg-eth0 file with cat shows that the correct networking configuration is still applied to eth0. It looks like the problem occurs due to the way SUSE handles devices. The following text is copied directly from the SUSE documentation: When the Kernel detects a network card and creates a corresponding network interface, it assigns the device a name depending on the order of device discovery, or order of the loading of the Kernel modules. The default Kernel device names are only predictable in very simple or tightly controlled hardware environments. Systems which allow adding or removing hardware during runtime or support automatic configuration of devices cannot expect stable network device names assigned by the Kernel across reboots. However, all system configuration tools rely on persistent interface names. This problem is solved by udev. The udev persistent net generator (/lib/udev/rules.d/75-persistent-net-generator.rules) generates a rule matching the hardware (using its hardware address by default) and assigns a persistently unique interface for the hardware. The udev database of network interfaces is stored in the file/etc/udev/rules.d/70-persistent-net.rules. Every line in the file describes one network interface and specifies its persistent name Source: https://www.suse.com/documentation/sled11/book_sle_admin/data/sec_basicnet_manconf.html When the ESXi host assigns the VM a new MAC Address, SUSE assigns a new unique interface to this MAC address and stores this in the file etc/udev/rules.d/70-persistent-net.rules. It shows two Ethernet adapters, eth1 is using the MAC address currently assigned to the VM. We are now entering a twilight zone, where there is one ethernet interface configured with an IP-address (ifcfg-eth0) while SUSE is applying all rules to a device it created and using the MAC Address assigned to the only NIC attached to the VM (Network Adapter 1). Time to clean up. Luckily udev rules are automatically generated during boot. To solve the mac address assignment fast, rename the file 70-persistent-net.rules After rebooting the VCSA, review the 70-persistent-net.rules file to verify that SUSE assigned the MAC address to eth0. You can now safely customize the system (Press F2 in the console) and configure the management network A reboot of the VCSA is necessary as it appears that a restart of the management services is not enough to restore all services. Funny how times change, nowadays you get really happy seeing a blue screen. ================================================================================ Title: Monitoring power consumption of your home lab with a smart plug URL: https://frankdenneman.ai/2016-05-19-monitoring-power-consumption-of-your-home-lab-with-a-smart-plug/ Date: 2016-05-19 Home labs are interesting beasts, at one hand you would love to have all the compute, storage and network power available, on the other hand you do not want to have a power bill similar to a Google data center. I have a decent setup, with 4 Xeon servers, two cisco 1GB switches, a 10Gb switch and 3 Synology’s, but I don’t keep everything on all the time. One server acts as the management server, running a Windows DC, vCenter appliance, the PernixMS server and some other stuff. These machines are always on, not only to save time when I want to use my lab but increased stability as well. Due to this, my network gear and storage systems are also on. Which made me wonder how much the need for availability and stability will cost me on a yearly basis. The big Xeon rigs equipped with multiple PCIe devices are usually shut down after tests because I expect them to consume lots of power. Time to stop guessing and start monitoring. As always Home Lab Sensei Erik Bussink pointed me out to a simple solution the Smart Plug Edimax SP-2101W Smart Plug Switch. Please leave a comment if you are using a different solution that is a better alternative to this device. The device Nothing much to add about the device itself, it is sleek enough so it will not eat up multiple power outlets. The device is managed via an apple or android app, the following screenshots are taken from an Apple device, you can monitor it with both your iPhone or iPad. You can manage multiple smart plugs from one device. As I’ve spread my lab over two power-groups I’ve installed two power-plugs to monitor my home lab. Unfortunately, the app doesn’t allow displaying two smart plugs simultaneously, you have to open each individually. The monitor page shows the real time power consumption registered by the plug. It displays Amps and Watts. Quite cool to see what happens when you power-on devices or even a virtual machine, this monitored server generates a spike of 30 watts when powering on a VM, it quickly returns to a steady state though. Fun to see that ESXi hosts do not consume a steady high state of power. The Now button shows the real-time power consumption and the total power consumption registered of today, this week and this month. By providing the price of energy, it calculates the total cost additionally. Unfortunately I haven’t found the option to change the currency sign, so you are stuck with the dollar sign. By selecting the Usage button provides you a chart to view the power consumption of that day. The app allows you to analyze power consumption trends of your home lab by provides an overview based on 24 hours of data, a week, a month and a full year. Conclusion The smart plugs are a great addition to my home lab, it provides me insights in the consumption and it for me personally have removed the reluctancy of leaving my full lab on. The answer to the question whether you need a smart plug if you run a home lab is in my opinion a straight and simple no. You can estimate cost or you can just ignore it and pay the bill when it arrives. I’m just curious about these things and it helps to clear my conscious. ================================================================================ Title: Tracking down noisy neighbors URL: https://frankdenneman.ai/2016-05-03-tracking-down-noisy-neighbors/ Date: 2016-05-03 A big part of resource management is sizing of the virtual machines. Right-sizing the virtual machines allows IT teams to optimize the resource utilization of the virtual machines. Right sizing has become a tactical tool for enterprise IT-teams to ensure maximum workload performance and efficient use of the physical infrastructure. Another big part of resource management is keeping track of resource utilization, some of these processes are a part of the daily operation tasks performed by specialized monitoring teams or the administrators themselves. Service Providers usually cannot influence the right sizing element, therefor they focus more on the monitoring part. What is almost universal across virtual infrastructure owners is the incidental nature of tracking down ’noisy-neighbors’ VMs . Noisy neighbor VMs generate workload in such a way that it monopolizes resources and have negative impact on the performance of other virtual machines. Service Providers and enterprise IT teams have to deal with these consumer outliers in order to meet the SLAs of existing workloads and being able to satisfy the SLA requirements of new workloads. It’s interesting that noisy neighbor tracking is an incidental activity as it can be so detrimental to the performance of the virtual datacenter. Tools such as vSphere Storage IO Control (short term focus) and vSphere Storage DRS (long term focus) assist to alleviate the infrastructure from the burden of noisy neighbors, but attacking this problem structurally is necessary to ensure consistent and predictable performance from your infrastructure. At long term, noisy neighbor VMs impact the projected consolidation ratio, which in turn influences the growth rate of the infrastructure. I’ve seen plenty of knee jerk reactions, creating a server and storage infrastructure sprawl due to introduction of these outlier workloads. Identifying noisy neighbors can become a valuable tool in both strategic and tactical playbooks of the IT organization. Having insight of which VMs are monopolizing the resources allow IT teams to act appropriately. Similar to real life the behavior of noisy neighbor can be changed often, but sometimes that’s the nature of the beast and you just have to live with it. In that situation noisy neighbors become outliers of conduct and one ha to make external adjustments. This insight allows IT teams to respond along the entire vertical axis of the virtual datacenter, from application to infrastructure choice. By having the correct analysis, the IT team can provide insights to the application owner, allowing them to adjust accordingly. It helps the IT team to understand whether the environment can handle the workload and make adjustment to the infrastructure necessary. Sometimes the intensity of the workload is just what it is and hosting that workload is necessary to support the business. In that case the IT team has to understand whether the infrastructure is suitable to support the application. As most IT organization have access to multiple platforms, the accurate insight of characteristics (and requirements) of the workload allows them to identify the correct platform. Virtual Datacenters are difficult to monitor. They are comprised of a disparate stack of components. Every component logs and presents data differently. Different granularity of information, different time frames, and different output formats make it extremely difficult to correlate data. In addition you need to be able to correctly identify the workload characteristics and interpret the impact it has on the shared environment. We do not live in a world anymore where we have to deal with isolated technology stacks. Applications typically do not run anymore on a single box, connected to a single and isolated raid array. Today everything within the infrastructure is shared, the level of hardware resource distribution is diluting with each introduction of new hardware. Where we used to run a single application in a VM on top of server with ten other VMs, sharing a couple of NICs and HBA’s, we slowly moved towards converged network platforms. In the last 10 years, we shared and shared more, the only monolith remaining is the application in the VM and that is rapidly changing as well with the popularity of containers and micro services. Yet most of our testing mechanisms and monitoring efforts are still based on the architecture we left behind 10 years ago. Virtual Datacenters require continuous analytics that fully comprehends the context of the environment, with the ability to zoom in and focus on outliers if necessary. In the upcoming series I’m going to focus on how to explore cluster level workloads and progressively zooming into specific workloads based on IOPS, block size, throughput and unaligned IOs. ================================================================================ Title: Managing your virtual datacenter and home lab with a MAC URL: https://frankdenneman.ai/2016-04-21-managing-your-virtual-datacenter-and-home-lab-with-a-mac/ Date: 2016-04-21 The majority of virtual datacenters are managed from Windows systems. When I started with virtualization I also used a windows system, however when I joined VMware I received a MacBook and this was the beginning of the end. Soon ever window device was replaced with an Apple device in my home. The problem was that I still needed to manage by home lab. To circumvent this, I created a Windows admin VM and installed all my trusted Windows apps, such as Putty, vSphere client and WinSCP. Works great! Until you want to rebuild your lab or restructure the environment. It always felt as a burden and on top of that I didn’t want to spend CPU cycles and waste memory of my home lab on admin VM. Throughout the years I discovered tools for Mac OS that replaced their trusted Windows equivalent and with the new release of the HMTL 5 Web client fling it removed dependency on the Client Integration Plugin (CIP). Here is the list of program and tips and tricks I use on my Mac to manage my Home lab. Putty > iTerm2 PuTTY is an SSH and telnet client for the Windows Platform allowing you to have access to the command line of the ESXi server. For the Mac platform I recommend iTerm2. Although Mac OS has a native terminal application, iTerm2 has a couple of cool features that I absolutely love. It can run multiple sessions, each in its own tab. With profiles you can configure the connection settings to your ESX host and with a simple shortcut key combination (for example, Control-command-1, you open a tab to the ESXi host. Download iTerm2 here. Remote Desktop > Royal TSX MS Remote Desktop is available for Mac OS, but the one remote desktop application you want to get is Royal TSX. The free version allows up for ten remote desktop connections, typically more than enough for the majority of home labs. I bought a licensed version as I’m using more than ten profiles and like to separate the workload part of the lab in a separate configuration document from the management part of the lab. One of the cool features is the tabbed layout, allowing you to switch between remote desktops quite easily. The screens at home have a minimum resolution 2560 x 1440 resolution, Royal TSX allows for any resolution, even native Retina resolution. I like to use the smart zoom and the resolution set by the virtual machine allowing you to have a proper environment to work in without hitting the time-consuming scroll bars. If security isn’t a big concern for you, you can specify the user and password for the connection at multiple levels. Either on the remote desktop connection profile itself or specify it on the ‘connection document’ for the entire environment. A nice time saver! If you are the complete opposite and you need higher levels of security, such as Network Level Authentication (NLA) Royal TSX is the application to get. NLA is enabled by default and you can configure to use Transport Layer Security (TLS) as well. Download Royal TSX here. WinSCP > CyberDuck WinSCP and Veeam Backup Free Edition (previously Veeam FastSCP) are the most popular Secure FTP applications that allows you to copy files directly onto the ESXi host. Unfortunately the once announced port to MacOS of WinSCP never came into fruition and therefor I looked for alternatives. There are plenty SFTP clients, the one I use and like is Cyberduck It allows for creating connection profiles called bookmarks, allowing you to connect to the correct folder directly. It also supports various encryption ciphers and authentication algorithms if you operate in a secure environment. Cyberduck is like all the other listed tools free but the occasionally ask for a donation. Download Cyberduck here. VMware ESXi Embedded Host Client fling The ESXi embedded host client fling allows you to manage the ESXi host directly through a web client. Its fast, it’s easy to install and it provides most of the functionality you need when you are building your lab before deploying the vCenter Appliance. One of the great assets to this tool is the integrated VM console. It’s directly accessible within the browser and does not require any addiotnal plugins or installers. Solving the annoying Client Integration Plugin problem most Mac users faced when connecting to the vCenter via the web client. The Fling currently only supports ESXi 6.0, however William published a workaround for ESXi 5.x. found here: http://www.virtuallyghetto.com/2015/08/new-html5-embedded-host-client-for-esxi.html Download the VMware ESXi Embedded Host Client fling here. vSphere HTML5 Web Client Fling v1.2 (h5client) This fling got released this week and it allows you to connect with an HTML5 based web client to the vCenter server. Be aware that this client is designed for managing vCenter only! This release focuses on removing the dependency of the client integration plugin allowing administrators to connect with the VM console via the web client and do the basic stuff. Combine that with the normal web client and you execute the majority of operations to setup and deploy your home lab / virtual datacenter. The client is deployed as a vib on one of the ESXi host. For detailed install instructions visit the VMware vSphere blog. Download the HTML5 Web Client Fling v1.2 here. Function keys Not a tool, but sometimes you are required to press a function key, such as F11 when installing ESXi. No problem when installing physical boxes, a challenge when installing a nested ESXi system using a remote (VM) console. In order to send the correct key, press FN-CMD-F11. This works on most function keys and other non-alphanumeric keys Please leave a comment if you want to share your favorite tool or handy tips and tricks to save time. ================================================================================ Title: Adjust timeout value ESXi Embedded Host Client URL: https://frankdenneman.ai/2016-04-12-adjust-timeout-value-esxi-embedded-host-client/ Date: 2016-04-12 I love to use the ESXi Embedded Host Client next to vCenter in my lab. It’s quick, it provide most of the functionality and best of it all it has a functioning VM console when accessing it from a MAC. The ESXi Embedded Host Client time-out default is set to 15 minutes, but you can adjust this setting. On the right side of the menu bar there is a drop down menu next to the IP-address or DNS name of your ESXi server. Open it and go to: Settings Application timeout Select the appropriate timeout value As I use it in my lab, I select the option off, but if you use this in other environments I can expect you use a different value. ================================================================================ Title: DVD Store, the perfect homelab workload tool URL: https://frankdenneman.ai/2016-03-31-dvd-store-the-perfect-homelab-workload-tool/ Date: 2016-03-31 DVD Store 2.1, a magnificent tool for all aspiring VCP/VCAP candidates. A great tool for home lab enthousiasts to understand performance metrics, a fantastic tool to understand the behavior of an application stack in a virtual datacenter. WHAT IS DVD STORE? According to the official site the DVD Store Version 2.1 (DS2) is a complete open source online e-commerce test application, with a backend database component, a web application layer, and driver programs. The goal in designing the database component as well as the midtier application was to utilize many advanced database features (transactions, stored procedures, triggers, referential integrity) while keeping the database easy to install and understand. The DS2 workload may be used to test databases or as a stress tool for any purpose. Thanks Todd Muirhead and Dave Jaffe for creating this! However there is a slight challenge in installing it properly. You can install it on windows or on Linux and use many different database programs. I like to use windows for this. Unfortunately I tried to follow the instruction video on youtube and it was lacking some crucial details to get it deployed successfully. Therefor I started to document the steps involved to get it deployed on a Windows 2012 system using SQL 2014 SP1. Please note that you can run DVD store on Linux as well, and it might be even better (more lean and mean than a windows install) for homelabs. If you have a detailed write-up (100% reproducible) of a working deployment DVD store on Linux, please share the link to your article in the comments. DVDSTORE ARCHITECTURE As described above the DVD Store is an application stack that can run on a single or multiple virtual machines. By using multiple virtual machines, you can test various components and layers in your virtual datacenter. As this is my goal I’m creating a VM that will run the database and another VM that generates the workload. Requirements I’m listing the software I’ve used in order to create a working environment. Many variations are possible. If you can create a lightweight version of this build, or a complete community edition (license free) please share URL of your article in the comments. Two virtual machines Windows 2008 R2 and Windows 2012 Windows 2008 R2 SP1 DVD Store 2.1 ds21.tar.gz DVD Store 2.1 ds21_sqlserver.tar.gz Winzip SQL 2014 SP1* ActiveState ActivePerl Community Edition DATABASE VM In this exercise I’m going to install and configure a 20GB database on a Windows 2012 VM. If you are using templates, check if you have enough space for the DVD store on your C-drive. During the first stage the temporary files will be stored on the C: drive, provide enough space which is at least equal to the DB size. The database hard disk needs to be twice the size of the DB in order to successfully import the data. Post configuration optimizations can reduce the consumed space of the database, but don’t be too frugal when configuring the hard disks. Play around with the compute settings depending on your lab equipment. I noticed that Windows 2012 uses 5.4 GB of memory to run its OS and SQL Express when idling, but during installation it consumed close to 11GB. Windows 2012 configuration Update Windows 2012 with all the latest patches and update VMtools, enable remote desktop if you don’t want to use the VM console. Disable the firewall, as this I run an air-gapped lab I don’t want to spend too much time on firewall rules. SQL requires to Enable Microsoft .Net Framework 3.5 SP1. and Download and install Microsoft .Net Framework 4.0. .Net Framework 4.0 is already a part of the Windows 2012 OS, therefore you only have to enable 3.5. by executing the following steps: Go to Server Manager Add roles and features Next Role-based or feature-based installation Click Next until you reach Features Select .Net Framework 3.5 Features Click Install Extracting DVD Store The DVD Store kit is available at linux.dell.com/dvdstore. Download the file ds21.tar.gz and ds21_sqlserver.tar.gz. Both include scripts that are made on a unix based machine, missing the proper CR/LF format for a windows system. Winzip converts files to proper windows format while extracting, therefor I recommend using Winzip. Alternatively you can use a tool such as Unix2Dos to convert the files if you don’t want to use Winzip. Extract both files to the C:\ Drive creating a directory structure as follows: Install ActivePerl The installation of DVD Store is done via a Perl script, Windows 2012 doesn’t contain a Perl utility. One of the recommended Perl Utility is ActiveState ActivePerl Community Edition. You can download it here. As I’m using Windows 2012, I need to download the x64 MSI version. The install is straightforward, no specific options need to be selected, basically a next next finish install. SQL 2014 DVD store can leverage both the full version or the Express version of SQL. Microsoft allows you to evaluate their products 180 days. If you do maintain a VM configuration for more than 180 days you can use the free version of SQL 2014 express. Please be aware that you need SQL Server Express with Advanced Services as it includes the full version of SQL Server 2014 Management Studio and Full Text Search and Reporting Service. Both features are required to run DVD Store. For more info on SQL 2014 versions go here: https://www.microsoft.com/en-us/download/details.aspx?id=42299. Download SQL 2014 Express ADV SP1 here: https://www.microsoft.com/en-us/download/details.aspx?id=46697 If you are going to use the express version, adjust your VM configuration. Unfortunately SQL Express has some CPU limitations for the database engine (Limited to lesser of 1 Socket or 4 cores) and a 10 GB DB limitation. Therefore a 4 vCPU configuration would be 1 virtual sockets: 1 and 4 cores per socket. For more info about virtual sockets and cores please read this article: /2013-09-18-vcpu-configuration-performance-impact-between-virtual-sockets-and-virtual-cores/ Install SQL 2014 Express ADV SP1 Run Install and select the following options: New SQL Server stand-alone instalation Accept the license terms Check “Use MS Update to check for updates” Database Engine Configuration: Mixed Mode (SQL Server Authentication and Windows Authentication) (provide password) Reporting Services Native Mode: Install and Configure Install SQL 2014 SP1 Download the eval version of SQL 2014 SP1 here: http://technet.microsoft.com/evalcenter/dn205290.aspx Run Install and select the following options: New SQL Server stand-alone instalation Select Evaluation Accept the license terms Setup Role: Select All Features using default values for service accounts* Database Engine Configuration: Mixed Mode (SQL Server Authentication and Windows Authentication) (provide password) Analysis Services Configuration: Add current User Reporting Services Configuration: Install and Configure Distributed Replay Controller: Add Current User Install During the install it can happen that the install process freezes when on a step called “Install_WatsonX86_Cpu32_Action”. To solve this state, open up task manager and end all “extra” processes called “Windows Installer (32 bit) ” leaving only a single Windows Installer process. I’m sure you can improve and optimise the SQL installation, but I haven’t really looked into this. For more information I recommend David Klee’s blog (http://www.davidklee.net/) and the book of Michael Webster “Virtualising SQL Server with VMware” (http://longwhiteclouds.com/) INSTALLING DVD STORE Once SQL is installed you can begin installing DVD Store.The process of installing DVD store consists of executing two scripts, the Install_DVDStore.pl script and the SQL script. Install_DVDStore.pl script The Install_DVDStore.pl script generates the database content (such as users and products) by creating CSV files and it generates a SQL script that allows MSSQL to create the DB2user, the databases and importing the CSV content files. In order to correctly generate these files, you must create the directories where the MSSQL Database files will be stored. I’m using a single drive for all databases, therefore I create a directory SQL\DBfiles on the E: drive (E:\SQL\DBfiles). Please note that the workload CSV files are generated in the C:\DS2 folder! That means that if you are going to generate a 20GB database, you need at least 20GB of free space on your C:\ drive as well to temporarily store the CSV files. Once installed SQL you can run the Install_DVDStore script in the C:\DS2 folder. I prefer to open up a command prompt to run the script. The window remains open after the script has completed successfully, allowing me to do other stuff in the mean time. If you have more trust in scripts than me, go right ahead and click on the perl script from the windows explorer. C:\ds2\Install_DVDStore.pl. In order to create a 20GB DB in the directory E:\SQL\DBfiles, I’m going to answer the questions as follows: Database size: 20 Database size is in MB or GB: GB Database type: MSSQL System type: WIN Path where Database files will be stored: E:\SQL\DBFiles\ * * Please note the trailing \ in E:\SQL\DBFiles\, this is required otherwise the script will fail. Creating the custom CSV and the sql script files took my system roughly 20 minutes. The CSV files are stored in the directory structure of the C:\DS2\Data_files. The SQL script is stored in the directory C:\DS2\sqlserverds2\. The Install_DVDStore script generated the following script: sqlserverds2_create_all_20GB. Thats the script we want to run in order to get the DB loaded with the records. Edit the SQL script David Klee (@kleegeek), the SQL MVP, discovered there was a slight error in the script. In order to fix this, edit the script in notepad or SQL management studio. Go to line 91 (or use find) and change (1) of GENDER VARCHAR(1) into (2) resulting in GENDER VARCHAR(2). Save and exit. It seems the DS2 scripts use the SA account with an blank password. You can do two thing, go through all the scripts or change the SA password on your SQL server. If someone knows the location of the SA user in the scripts, please leave a comment. In order to change the SA password, open up the SQL 2014 management studio. (Go to start, apps, SQL Server 2014 Management Studio). Select “SQL Server Authentication” and use the SA user with the password you entered during the installation process of SQL. Go to Security \ Logins and select the SA account, go to properties and deselect the option “Enforce password policy”. Now remove the password and click on OK. Yes you are sure you want to continue so click on Yes ;) Exit the management studio. Execute the SQL script Go to the C:\DS2\sqlserverds2 directory and click on the sqlserverds2_create_all_20GB script. This opens SQL2014 management Studio and you need to authenticate again. A good time to check to see if the SA account is using a blank password, use the SA user account and click on connect. Management Studio shows the script, press F5 to execute or go to the Query menu and click on Execute. In the bottom left corner, it will show executing query. Select the Message tab to monitor the progress of the script. It took my system 1 hour and 5 minutes to complete the script, it might be a good time to start working on the “workload” VM that’s going to generate the queries in the mean time. After the script finishes, it’s time to run a SQL maintenance task. Although the script creates a 20GB database, 37GBs of space is consumed on the hard disk. SQL2014 Maintenance Plan In the DVD Store documentation it’s recommended to run the maintenance plan to optimize performance. The SQL Agent service is turned off by default in SQL 2014. Start this service by opening a command prompt and type in the command: net start sqlserveragent otherwise the follow error will be presented when attempting to create a maintenance plan in SQL Management Studio: Open the SQL Server 2014 Management Studio(GUI), follow following steps: Go to Object Explorer and click and expand database server tree. Under server tree, expand management and right click on maintenance plans. Left Click on “Maintenance Plan Wizard Option”. In the wizard opened, click next and enter name of plan as “ds2”. Click next and check “Update Statistics” checkbox and again click next. Click next and then choose database as DS2 and click OK. Ensure “All existing statistics” and “Sample By” checkbox are set along with value “18” “percent”. Once above step is done click next twice to create a task under “Maintenance Plans” under “Management” object under SQL Server tree. Now right click on this task “ds2” created from above steps and it will show a menu option for right click. Click execute to update statistics on all tables in DS2 database using task created due to above steps. Visit the sites of the SQL experts to learn more about optimizing SQL DB’s if you want to get more performance out of this database. At this point, the Database VM configuration is complete and we can start generating some workload by running the ds2sqlserverdriver program on the worload VMs. DS2 WORKLOAD VM Unfortunately the DS2webdriver kept on crashing on a Windows 2012 system, complaining about invalid registry settings. Therefor I’m using a Windows 2008 system. The configuration of the VM is straightforward. Ensure that the workload VM can connect to the database VM across the network and run the ds2sqlserverdriver program. Database VM configuration OS: Windows 2008 CPU config: Number of virtual sockets: 2 Number of cores per socket: 1 Memory 12GB Harddisk 1: 40 GB SCSI controller 0: LSI Logic SAS Network Adapter: VMXNET 3 Windows 2008 configuration Update Windows 2008 with all the latest patches, service packs and update VMtools. Download SP1 here: https://www.microsoft.com/en-us/download/details.aspx?id=5842 Disable the firewall. * Enable remote desktop if you don’t want to use the VM console Enable .Net 3.5 if you want to install SQL management studio * As this I run an air gapped lab I don’t want to spend too much time on firewall rules) DS2SQLSERVERDRIVER Extract the ds2.tar.gz and ds2_sqlserver.tar.gz on the C:\. Open command prompt and go to c:\ds2\sqlserverds2\ and run ds2sqlserverdriver.exe. This will show the options: An example script (by David Klee): c:\ds2\sqlserverds2\ds2sqlserverdriver.exe --target=192.168.0.132 --run_time=60 --db_size=20GB --n_threads=4 --ramp_rate=10 --pct_newcustomers=0 --warmup_time=0 --think_time=0.085 This program allows you to customize every workload possible. The command that I like the most is the think time. This is the amount of time that a simulated user would ‘think’ before clicking again. This command allows you to create a more realistic workload that differs from any synthetic benchmark tool out there. You can run spawn multiple virtual machines running different configured workloads against a single database. Adjust the think time, adjust the average number of search order per customer. The application stack allows you to investigate the complete stack. You can run multiple workload VMs and the DB VM on a single host, allowing to understand CPU or memory contention. It allows to distribute the workload across multiple hosts, allowing you do dive into the impact of networking and possibly DRS. Moving VMs onto a single datastore and monitor the storage path and the impact of SIOC. The possibilities are endless. Genuinely a tool that can help anyone at any level understand virtualization and IT infrastructures better. ================================================================================ Title: You do not have permissions to view this object error after updating VCSA to 6.0 Update 1b URL: https://frankdenneman.ai/2016-03-01-you-do-not-have-permissions-to-view-this-object-error-after-updating-vcsa-to-6-0-update-1b/ Date: 2016-03-01 Today I’ve updated my vCenter Server Appliance with the VC-6.0.0U1b-Appliance.ISO in my lab. After rebooting I was surprised to see the error “You do not have permissions to view this object” on almost every object in the inventory screen. Unfortunately a reboot of the DC (home lab, I do not run an elaborate AD here)Time to google and it seems that a lot of other people have hit this bug. After googling some more I found the the VMware KB article: KB 2125229. Problem is, this is solely focused on the windows version of vCenter and not focussed on solving the problem occurring on the VCSA. Although I can log in and see the inventory when using my admin account (Lab\vAdmin) I can’t access the objects. Maybe a permission problem? When checking the global permissions the (vAdmin) user is still listed as an administrator. However administrators should be able to access all objects, as I found out a refresh is required. Here is how I solved it: 1. Log out of vC and login with the default admin account “administrator@vsphere.local” 2. In the Home view, select “Administration” from the menu 3. Go to Global Permissions, remove the user (In my case vAdmin) 4. Click on “Add Permission” 5. Select your AD domain and select the correct user 6. Click on Ok 7. Check the list to see whether your user is added with the correct role (administrator). 8. Logout and login with the correct AD user. 9. Back to work. Time for me to power on these servers again. Follow Frank on twitter @frankdenneman ================================================================================ Title: Insights into VM density URL: https://frankdenneman.ai/2016-02-15-insights-into-vm-density/ Date: 2016-02-15 For the last 3 months my main focus within PernixData has been (and still is) the PernixCloud program. In short PernixData Cloud is the next logical progression of PernixData Architect and provides visibility into and analytics of virtual datacenters, it’s infrastructure, and it’s applications. By providing facts on the various elements of the virtual infrastructure, architects and administrators can design their environment in a data-driven way. The previous article “Insights into CPU and Memory configurations of ESXi hosts” zoomed in to the compute configuration of 8000 ESXi hosts and helped us understand which is the most popular system in today’s datacenter running VMware vSphere. The obvious next step was to determine the average number of virtual machines on these systems. Since that time the dataset has expanded and it now contains data of more than 25.000 ESXi hosts. An incredible dataset to explore I can tell you and it’s growing each day. Learning how to deal with these vast quantities of data is incredibly interesting. Extracting various metrics from a dataset this big is challenging. Most commercial tools are not designed to cope with this amount of data, thus you have to custom build everything. And with the dataset growing at a rapid pace, you are constantly exploring the boundaries of what’s possible with software and hardware. VM Density After learning what system configurations are popular, you immediately wonder how many virtual machines are running on that system. But what level of detail do you want to know? Will this info be useable for architects and administrators to compare their systems and practical to help them design their new datacenter? One of the most sought after question is the virtual CPU to physical CPU ratio. A very interesting one, but unfortunately to get a result that is actually meaningful you have to take multiple metrics into account. Sure you can map out the vCPU to pCPU ratio, but how do you deal with the fact of oversizing of virtual machines that has been happing since the birth of virtualization? What about all these countless discussions whether the system only needs a single or double CPU because it’s running a single threaded program? How many times have you heard the remark that the vendor explicitly states that the software requires at least 8 CPU’s? Therefor you need to add utilization of CPU to get an accurate view, which in turn leads to the question what timeframe you need to use to understand whether the VM is accurately sized or whether the vCPUs are just idling most of the time? You are now mixing static data (inventory) and transient data (utilization). Same story applies for memory. In consequence I focused just on the density of virtual machines per host. The whole premise of virtualization is to exploit the variation of activity of applications, combined with distribution mechanisms as DRS and VMturbo you can argue that virtual and physical compute configurations will be matched properly. Therefor it’s interesting to see how far datacenters stretch their systems and understand the consolidation ratio of virtual machines. Can we determine a sweet spot of the number of virtual machines per host? The numbers Discovered earlier, dual socket systems are the most popular system configuration in the virtual datacenters, therefor I focused on these systems only. With the dataset now containing more than 25.000 ESXi hosts, it’s interesting to see what the popular CPU types are. The popular systems contained in total 12, 16, 20 and 24 cores. Therefor the popular CPU’s of today are 6, 8, 10 and 12 cores. But since we typically see a host as a “closed” system and trust on the host local CPU scheduler to distribute the vCPUs amongst the available pCPUs, all charts use the total cores per system instead of on a per-CPU basis. For example a 16 cores system is ESXi host containing two 8 cores CPUs. Before selecting a subset of CPU configurations let’s determine the overall distribution of VM density. Interesting to see that it’s all across the board, VM density ranging from 0-10 VM’s per host up to more than 250. There were some outliers, but I haven’t included them. One system runs over 580 VM’s, this system contains 16 cores and 192 GB. Let’s dissect the VM density per CPU config. Dissecting it per CPU configuration Instead of focusing on all dual socket CPU configurations, I narrowed it down to three popular configurations. The 16 core config as it’s the most popular today, and the 20 to 24 core as I expect this to be the configuration as the default choice for new systems this year. This allows us to compare the current systems in today’s datacenter to the average number and help you to understand what VM density possible future systems run. Memory Since host memory is an integral part of providing performance to virtual machines, it’s only logical to determine VM density based on CPU and Memory configurations. What is the distribution of memory configuration of dual socket systems in today’s virtual datacenters? Memory config and VM density of 16 cores systems 30% of all 16 cores ESXi hosts is equipped with 384GB of memory. Within this configuration, 21 to 30 VMs is the most popular VM density. Memory config and VM density of 20 cores systems 50% of all 20 cores ESXi hosts is equipped with 256GB of memory. Within this configuration, 31 to 40 VMs is the most popular VM density. Interesting to see that these systems, on average, have to cope with less memory per core than the 16 cores system (24GB per core versus 12,8GB per core) Memory config and VM density of 24 cores systems 39% of all 24 cores ESXi hosts is equipped with 384GB of memory. Within this configuration, 101 to 150 VMs is the most popular VM density. 101 to 150 VM’s sound like a VDI platform usage. Are these systems the sweetspot for virtual desktop environments? Conclusion Not only do we have an actual data on VM density now, other interesting facts were discovered as well. When I was crunching these numbers one thing that stood out to me was memory configurations used. Most architects I speak with tend to configure the hosts with as much memory as possible and swap out the systems when their financial lifespan has ended. However I seen some interesting facts, for example memory configurations such as 104 GB or 136 GB per system. How do you even get 104GB of memory in such a system, did someone actually found a 4GB DIMM laying around and decided to stick it in th system? More memory = better performance right? Please ready my memory deepdive series on how this hurts your overall performance. But I digress. Another interesting fact is that 4% of all 24 cores systems in our database are equipped with 128GB of memory. That is an average of 5,3 GB per core, 64GB per NUMA node. Which immediately raises questions such as average host memory per VM or VM density per NUMA node. The more we look at data, the more questions arise. Please let me know what questions you have! ================================================================================ Title: Insights into CPU and Memory configuration of ESXi Hosts URL: https://frankdenneman.ai/2015-12-23-insights-into-cpu-and-memory-configuration-of-esxi-hosts/ Date: 2015-12-23 Recently Satyam Vaghani wrote about PernixData cloud. In short PernixData Cloud is the next logical progression of PernixData Architect and provide visibility and analytics around virtual datacenters, it’s infrastructure and applications. As a former architect I love it. The most common question asked by customers around the world was how other companies are running and designing their virtual datacenters. Which systems do they use and how do these system perform with similar workload? Many architects struggle with justifying their bill of materials list when designing their virtual infrastructure. Or even worse getting the budget. Who hasn’t heard the reply when suggesting their hardware configuration: “you want to build a Ferrari, a Mercedes is good enough”. With PernixData Cloud you will be able to show trends in the datacenter, popularity of particular hardware and application details. It let you start ahead of the curve, aligned with the current datacenter trends instead of trailing. Of course I can’t go into detail as we are still developing the solution, but I can occasionally provide a glimpse of what we are seeing so far. For the last couple of days I’ve been using a part of the dataset and queried 8000 hosts on their CPU, memory and ESXi build configuration to get insight in popularity of particular host configurations. CPU socket configuration I was curious about the distribution of CPU socket configurations. After analyzing the dataset it is clear that dual socket CPU configurations are the most popular setup. Although single CPU socket configuration are more common than quad CPU socket in the dataset, quad core are more geared towards running real world workload while single CPU configurations are typically test/dev/lab servers. Therefor the focus will primarily on dual CPU socket systems and partially quad CPU sockets systems. The outlier of this dataset is the 8 socket servers. Interesting enough some of these are chuck-full with options. Some of them were equipped with 15 core CPU’s. 120 CPU cores per host, talk about CPU power! CPU core distribution What about the CPU core popularity? The most popular configuration is 16 cores per ESXi host, but without the context of CPU sockets one can only guess which CPU configuration is the most popular. Core distribution of dual CPU socket ESXi hosts When zooming in to the dataset of dual CPU socket ESXi host, it becomes clear that 8 Core CPU’s are the most popular. I compared it with an earlier dataset and quad and six core systems are slowly reducing popularity. Six core CPU’s were introduced in 2010, assumable most will be up for a refresh in 2016. I intend to track the CPU configurations to provide trend analysis on popular CPU configurations in 2016. Core count quad socket CPU systems What about quad socket CPU systems? Which CPU configuration is the most populair? It turns out that CPU’s containing 10 cores are the sweetspot when it comes to configuring a Quad core CPU system. Memory configuration Getting insights into memory configuration of the servers provides us a clear picture of the compute power of these systems. What is the most popular memory configuration of dual socket server? As it turns out 256 and 384 GB are the most memory popular configuration. Today’s servers are getting beefy! Zooming into the dataset quering the memory configuration of dual socket 8 core servers, the memory configuration distribution is as follows: What about the memory configuration of quad CPU servers? NUMA 512 GB is the most popular memory configuration for quad CPU socket ESXi host. Assuming the servers are configured properly, this configuration is providing the same amount of memory to each NUMA node of the systems. The most popular NUMA node is configured with 128 GB in both dual and quad CPU socket systems. ESXi version distribution I was also curious about the distribution of ESXi versions amongst the dual and quad CPU socket systems. It turns out that 5.1.0 is the most popular ESXi version for dual CPU systems, while most Quad CPU socket machines have ESXi version 5.5 installed More To Come Satyam and I hope to publish more results from our dataset in the coming months. The dataset is expanding rapidly, increasing the insights of the datacenters around the globe. And we hope to cover other dimensions like applications and the virtualization layer itself. Please feel free to send me interesting questions you might have for the planet’s datacenters and we’ll see what we can do. Follow me on twitter @frankdenneman ================================================================================ Title: When your Home Lab turns into a Home DC URL: https://frankdenneman.ai/2015-06-04-when-your-home-lab-turns-into-a-home-dc/ Date: 2015-06-04 A little bit over a year ago I decide to update my lab and build two servers. My old lab had plenty of compute power, however they were lacking bandwidth, 3 Gbit/s SATA and 1 Gb network bandwidth. I turned to one of the masters of building a home lab, Erik Bussink, and we thought that the following configuration was sufficient to handle my needs. Overview Component Type Cost CPU Intel Xeon E5 1650 v2 540 EUR CPU Cooler Noctua NH-U9DX i4 67 EUR Motherboard SuperMicro X9SRH-7TF 482 EUR Memory Kingston ValueRAM KVR16R11D4/16HA 569 EUR SSD Intel DC 3700 100GB 203 EUR Power Supply Corsair RM550 90 EUR Case Fractal Design Define R4 95 EUR Price per Server (without disks) 1843 EUR The systems are great, but really quickly I started to hit some limitations. Limitations that I have addressed in the last year, and that are interesting enough to share. Adding a third host As FVP is a scale out clustered platform, having two hosts to test with simply just don’t cut it. For big scale out testing I use nested ESXi but to do simple tests I just needed one more host. The challenged I faced was the dilemma of investing in “old” tech or going with new hardware. Intel updated their Xeon line to version 3, the Intel Xeon v3 has more cache (from 12MB to 15MB) more memory bandwidth, increased max memory support up to 768GB and uses DDR4 memory. (Intel ark comparison) New shiny hardware might be better, but the main goal is to expand my cluster and one of the things I believe in is a uniform host configuration within a cluster. Time is a precious resource and the last thing I want to spend time on is to troubleshoot behavior that is caused by using non-uniform hardware. You might win some time by having a little bit more cache, more memory bandwidth but once you need to troubleshoot weird behavior you lose a lot more. A dilemma is not a dilemma if you go back and forth between the options, thus I researched if it was possible. The most predominant one is the change in CPU microarchitecture. The v3 is part of the new Intel Haswell microarchitecture. The Xeon v2 is build upon the Ivy Bridge. That means that the cluster has to run in EVC mode Ivy Bridge. The EVC dialog box of the cluster indicates that Haswell chips are supported in this EVC mode, thus DRS functionality remains available if I go for the Haswell chip The Haswell chip uses DDR4 memory and that means different memory timings and different memory bandwidth. FVP can use memory as a storage I/O acceleration resource and a lot of testing will be done with memory. That means that applications can behave differently when FVP decides to replicate fault tolerant writes to the DDR4 host or vice versa. In itself it’s a very interesting test, thus again another dilemma is faced. However, these tests are quite unique and I rather have uniform performance across the cluster and avoid any troubleshooting behavior due to hardware disparity. Due to the difference in memory type, a new Motherboard is required too. That meant that I have to find a motherboard that contains the same chipsets and network configuration. The SuperMicro X9SRH-7TF rocks. Onboard 10 GbE is excellent. Some other users in the community have reported overheating problems, Erik Bussink was hit hard by the overheat problem and bought another board just to get rid of weird errors caused by the overheating. That by itself made me wonder if I would buy another X9SRH-7TF or go for a new Supermicro board and buy a separate Intel X540-T2 dual port 10GbE NIC to get the same connectivity levels. After weighing the pros and cons I decided to go for the uniform cluster configuration. Primarily because testing and understanding software behavior is hard enough. Second-guessing whether behavior is caused by the hardware disparity is a time sucking beast and even worse, it typically kills a lot of joy in your work. Contrary to popular belief, prices of older hardware does not decline forever, due to availability of newer hardware and remaining stock, prices go up. The third host was almost 500 Euro’s, more expensive than the previous price I had to pay. Networking Networking is interesting as I changed a lot during the last year. The hosts are now equipped with an Intel PRO 1000 PT dual ports with the 82571 chip. Contrary to my initial post these are supported by vSphere 5.5. However network behavior is a large part of understanding scale out architectures, thus more NIC ports are needed. An additional HP NC365T Quad-port Ethernet Server Adapter was placed in each server. The HP NIC is based on the Intel 82580 chipset but is a lot cheaper than buying Intel branded cards. Each host has one NIC dedicated to IPMI, 2 10GbE ports and 6 1GbE. In hindsight, I would rather go for two Quad NIC cards as it allows me to setup different network configurations without having to tear them down each time. With the introduction of the third host I had to buy a 10GbE switch. The two host were directly connected to each other, however this configuration is not possible with three hosts. Thus I had to look for a nice cheap 10GbE switch that doesn’t break the bank and is quiet. Most 10GbE switches are made for the data center where noise isn’t really a big issue. My home lab is located in my home office, spending most of my day with something that sounds like a jet plane is not my idea of fun. The NETGEAR ProSafe Plus XS708E 8-port 10-Gigabit fit most of my needs. 8 ports for less than 900 euro’s, it’s kind of a steal compared to the alternatives. However I wasn’t really impressed by the noise levels (and spending 900 euro’s but that’s a different story). Again my main go-to-guy for all hardware related questions Erik Bussink provided the solution, the Noctua NF-A4-x10 FLX coolers. Designed to fit into 1U boxes they were perfect. But as you can see the design of the Netgear is a bit weird. The coolers are positioned at the far end of the PCB with all the heatsinks. When the switch is properly loaded, the thing emits a lot of heat. Regardless of what type of internal fan is used. To avoid heat buildup in the switch I used simple physics, but I will come to that later. Now having three hosts with seven 1 GbE connections, two storage systems eating up 3 ports and an uplink to the rest of the network I needed a proper switch. Lessons learned in that area, research thoroughly before pressing the buy button. I started of with buying an HP 1810-24G v2 switch. Silent, 24 ports, VLAN support. Awesome! No not awesome because it couldn’t route VLANs. And to the observant reader, 25 ports required, 24 ports offered. A VCDX’esque-like constraint. To work around the 24 ports limitation I changed my network design and wrote some scripts to build and tear down different network configurations. Not optimal, but dealing with home labs is almost like the real world. While testing network behavior and hitting the VMkernel network stack routing problem I decided it was time to upgrade my network with some proper equipment. I asked around in the community and a lot where using the Cisco SG300 series switch. Craig Kilborn on twitter blogged about his HP v1910 24G and told me that it was quite noisy. A Noctua hack might do the trick, but I actually wanted some more ports than 24. Erik pointed out the Cisco SG500-28-K9-G5 switches that are stackable and fanless. Perfect! I could finally use all the NICs in my servers and have room for some expansion. Time for a new rack So from this point on I have three 19” sized switches, the IKEA lack hack table was nice, but these babies deserved better. The third host didn’t fit the table therefor new furniture had to be bought anyways. After spending countless of hours looking at 19” racks I came across a 6U Patch case. This case had a lockable glass door (kids) and removable side panels, perfect for my little physics experiment. Just place the case in an upright position, remove the side panels and let the heat escape from the top. The fans will suck in “cold” air from the bottom. The dimensions of the patch case were perfect as it fitted exactly in my setup. The case is an Alfaco 19-6406. But with this networking equipment I’m feeling that my home lab is slowly turning into a #HomeDC. With all this compute and network power I wanted to see what you can do when you have enterprise grade flash devices. I’m already using the Intel DC S3700 SSD’s and I’m very impressed by their consistent high performance. However Intel has released the Intel SSD DC p3700 PCIe card that use NVMe. I turned to Intel and they were so generous of loaning me three of these beasts for a couple of months. The results are extremely impressive, soon I will post some cool test results, but imagine seeing more than 500.000 IOPS in your homeDC on a daily basis. Management server To keep the power bill as low as possible, all three hosts are shutdown after testing, but I would like to have the basic management VMs running. In order to do this, I used a Mac Mini. William wrote extensively about how to install ESXi on a Mac, if you are interested I would recommend to check out his work: http://www.virtuallyghetto.com/apple. Unfortunately 16Gb is quite limited when you are running three windows VMs with SQL DB’s, therefor I might expand my management cluster by adding another Mac Mini. Time to find me some additional sponsors. :) ================================================================================ Title: Ballooning, Queue Depths and other back pressure features revisited URL: https://frankdenneman.ai/2015-05-20-ballooning-queue-depths-and-other-back-pressure-features-revisited/ Date: 2015-05-20 Recently I’ve been involved in a couple conversations about ballooning, QoS and queue depths. Remarks like ballooning is bad, increase the queue depths and use QoS are just the sound bits that spark the conversation. What I learn from these conversations is that it seems we might have lost track of the original intention of these features. Hypervisor resource management Features such as ballooning and queue depths are invented to solve the gap between resource demand and resource availability. When a system experiences a state where resource demand exceeds the resources it controls the system has a problem. This is especially true in systems such as a hypervisor where you cannot control the demand directly. A guest operating system or an application in a virtual machine can demand a lot of resources. The resource management schedulers inside the hypervisor are tasked to fulfilling the demand of that particular machine while at the same time satisfy the resource demand of other virtual machines. Typically the guest OS resource schedulers are not integrated with the hypervisor resource schedulers, this can lead to a situation in which the administrator typically resorts to taking draconian measures. Measures such as disabling ballooning, increasing the queue-depth to become the digital equivalent of the Mariana trench. Sometimes it is taken for granted, but resource management inside the hypervisor is actually a though challenge to solve. Much research is done on solving this problem; a lot of research papers trying to find an answer to this challenge are published on a monthly basis. Let’s step back and take a look from an engineer perspective (or should I say developer?) and see what the problem is and how to solve it in the most elegant way. It’s an architect job to understand that this functionality is not a replacement for a proper design. Let’s start by understanding the problem. Load-shedding or back pressure When dealing with the situation where resource demand exceeds resource availability you can do two things. Well if you don’t do anything, it’s likely to encounter a system failure that can affect a lot more than only that particular resource or virtual machines. Overall you don’t design for system failure, you want to avoid it and to do so you can either drop the load or apply some form of back pressure. I think we all agree that dropping load, sometimes referred to as load-shedding is not the most elegant way of dealing with temporary overload, that’s why a lot of effort is going into back pressure features. A back pressure feature that everyone is familiar with is the memory balloon driver. Guest OS memory schedulers deal with used and free memory in such a way that this is transparent to the hypervisor. When the hypervisor is running out of physical machine memory it needs to figure out a way to retrieve memory. By using the balloon driver, the hypervisor asks the guest OS memory scheduler to provide a list of pages that it doesn’t use or doesn’t deem as important. After getting the info, the hypervisor proceeds to free up the physical memory pages to be able to satisfy incoming memory requests. Instead of dropping the new incoming workload it applies a back pressure system in the most elegant way. I don’t know why people are still talking about ballooning as bad. The feature is awesome, it’s the architect / sys admin job to come up with a plan to avoid back pressure in the system. Again the back pressure feature is not substitute for proper design and management. But the most misunderstood back pressure feature could be queue-depths. Sometimes I hear people refer to queue depths as a performance enhancement. And this is not true. It just allows you to temporarily deal with I/O overload. The best way to have a clear understanding of queue depths is to use the bathroom sink analogy. The drain is the equivalent of the data path leading to the storage array, the sink itself is the queue sitting on top of the data path / drain. The faucet represents the virtual machine workloads. Typically you open up the faucet to a level that allows the drain to cope with the flow of water. Same applies to virtual machine workloads and the underlying storage system. You run an x amount of workload that is suitable for your storage system. The moment you open up the faucet more your sink will fill up and at one point your sink will overflow. Thus you have to do some back pressure mechanism. In the bathroom sink world this typically is done by flowing the water back into the second sink. In the I/O scheduler world this typically resolves in a queue full statement. This typically bogs down the performance of the virtual machine so much that many admins/architect resolve by increasing the queue depth. Because this allows them to avoid the queue full state (temporarily) But in essence you just replaced your bathroom sink by a bigger sink, or something when people go overboard the increase the queue depth to the digital equivalent of a full size bathtub. This bathtub impacts a lot of other workloads as many workload now end up at the top of the queue instead of the deeper part, waiting their turn to go through the drain to the storage system. Result: latency increases in all applications due to improper designed systems. And remember when the bathtub overflows you typically have a bigger mess to deal with. Back pressure features are not a substitute for proper design, therefor think about implementing a bigger drain of even better multiple drains. More bandwidth or just more data paths to the same storage system lead to a short delay of seeing the same back pressure problem again, it just occurs on a different level. Typically when a storage controller fills up its cache, it sends a queue full to all the connected systems, so the problem has now evolved from a system wide problem to a cluster wide problem. This is one of the big reasons why scale out storage systems are a great fit in the virtual datacenter. You create drains to different sewer system typically in a more plan-able manner. If you are looking for more information about this topic, I published a short series on the challenge of traditional storage architectures in virtual datacenters. Quality of Service faces the same predicament. QoS is a great feature dealing with temporary overload, but again, it is not a substitute for a proper design. Back pressure features are great, it allows the system to deal with resource contentions while avoiding system failures. These features are unmissable in the dynamic virtual datacenter of today. When detecting that these features are activated on a frequent basis, one must review the virtual datacenter architecture, the current workloads and future workloads. I think overall it all boils down to understand the workload in your system and have an accurate view of the capabilities of your systems. Proper monitoring and analytics tools are therefore indispensable. Not only for daily operations but also for architects dealing with maintaining a proper service level for their current workloads while architecting an environment that can deal with unknown future workloads. ================================================================================ Title: VMware Tools is out of date on this virtual machine while summary states Current URL: https://frankdenneman.ai/2015-05-05-vmware-tools-is-out-of-data-on-this-virtual-machine-while-summary-states-current/ Date: 2015-05-05 For some apparent reason all my virtual machines show an alert that the VMware tools is out of date. While the summary states that its running the current version When trying to upgrade the VMware tools, all options are grayed out: It appears to be a cosmetic error but I stil wanted to know why it shows the alert on my virtual machines. As it turns out I created my templates on my management cluster host (lab) and that host runs a newer version of vSphere 5.5 (2068190) My workload cluster host run a slightly older version of vSphere 5.5 and it uses a different version of VMtools. The VMtools version list helps you to identify which version of VMtools installed in your virtual machine belongs to which ESXi version: https://packages.vmware.com/tools/versions Hope this clarifies this weird behavior of the UI for some. ================================================================================ Title: Hammer, MagicBands and challenging status quo URL: https://frankdenneman.ai/2015-04-23-hammer-magicbands-and-challenging-status-quo/ Date: 2015-04-23 One of my favorite business books is the famous book of Michael Hammer “Reengineering the Corporation”. The theme is the book is to take a hard look at business processes and radically change these old and existing processes. Hammer states that typically companies sped up their processes by implementing a newer iteration of existing technology. Many processes are dated before the advent of the computers and just by automating the process it can only optimizes performance marginally. Embedding computers in the archaic processes cannot address their fundamental performance challenges. By understanding what the process is trying to achieve one can break away from the existing design principles of the process. One great example is one of my favorite technologies is the Disney’s MagicBand and the way it’s used to radically change processes. [caption id=“attachment_5292” align=“aligncenter” width=“480”] Photo by Adam Voorhes / wired.com[/caption] Disney’s MagicBand The MagicBand is what is commonly referred to as a wearable. Inside the bracelet are a RFID chip and a radio. The parks have long range and short-range scanners along with sensors to interact with the MagicBand bracelet. One of the perks of my job is to speak to people who deliver cutting edge technology and as you can imagine I was very thrilled to speak to some of the team members who worked on the MagicBand platform. In essence the Disney MagicBand replaces every transaction between the customer and cast members or Disney parks and resorts. The MagicBand becomes your key to anything. It allows access to the park, access to the resorts and automatic payment. Its goal is to create a frictionless experience for the customer, increasing the satisfaction, which of course will increase spending. Instead of tinkering with existing processes Disney overhauled a lot of processes and many more are to follow. A great example is the overhaul of the check-in process and how it’s completely inline with the Hammer doctrine. Instead of buying newer faster desktop computers to speed up the check-in process, customers can go directly to their hotel room, bypassing the check in process all together. Disney’s sends the MagicBand to your home and prior to your visit the resort informs you via email which room is yours for your stay. Just walk up to your room and unlock the door by taping your MagicBand against the sensor on the door. MagicBands allows Disney to get rid of the ancient turnstiles that is the first port of anxiety of new parents and their strollers, instead of being funnelled into narrow cramped isles the entrance is now in the shape of a inviting V shape form with incredible process speeds. Just hold your MagicBand to the access point and wait to be greeted by an welcoming green glow, from then on its off to your favorite experience. The MagicBand platform allows for new experiences as well. What about having a more personalized interaction with cast members? What if Cinderella greets your daughter by name and tells her that she knows she is her favorite princess or that she wishes her a happy birthday? Just mind-blowing and an experience she will never forget. Range scanners, sensors, WIFI, smart phones apps, user profiles and an insane amount of data crunching make this happen. Customers use the smart phone app to access their schedules and their user profiles. Countless short and long-range sensors scattered across the park pick up the signals of the bracelets. These systems are connected to each other, they collect data, and they use the captured data to optimize the experience of the customers. Data on visitors traffic flow, food orders and waiting times can be used to realign internal resources. Ever heard about Internet of things? This is the poster child of Internet of things. Stop! Hammer Time Circling back to the opening statement, using newer iteration of existing technology only provide marginally performance increases. Advances like the MagicBand require new technologies and new ways to operationalize these technologies in datacenters. Not every company is of the same size as Disney, but one thing is certain, most companies face the same challenge Disney has. How to reduce cost, increase efficiency and provide a new experience that makes them unique in a highly competitive market? A lot of brilliant people are trying to solve these problems by creating technical solutions and it’s up to the IT team to understand if these suit their operation models. How can you reengineer your corporation and create a new service offering while your IT is stuck in the past? Stuck using systems that are designed to work in infrastructures dating back to the early ‘70’s? Where they just found out that the market was bigger than 5 computers? Everything has to align, when the business changes it models, the IT team should not take anything for granted, they too need to aim for quantum leaps of performance. Markets shift rapidly; IT needs to be able to respond almost in a way that anticipates their needs. Maybe even in a way that it doesn’t feel remarkable at all, high service standards are the norm! Within the realm of virtualized datacenters two technology advancements can create an experience that might provide a ubiquitous experience to the business but provide the magic to the IT team; Scale out storage and object based storage. Scale out storage and object based storage Proper Scale out storage systems allows you to operationalize new advancement in storage technology as soon as they are available. They allow virtual datacenters to cater to any performance requirement possible any time. While, and this is very important, without impact current workloads that are using the same platform. Many application vendors move away from a monolithic application architecture, why keep holding on to the relics of the past by using a monolithic storage architecture for performance requirements? Object based storage, such as VMware VVOLs, allows you to fundamentally change how to provide data services to systems (virtual machines). Instead of creating management constructs and aligning data services to these logical layers (LUNs and datastores), data services can be directly applied to the specific machine. Virtual machines become first class citizens on storage systems, allowing IT teams to cater to requirements that both affect the business as well as the IT team requirements. Challenge status quo Hammer stated don’t ask, “How can we do what we do faster?” But ask; Why do we do it the way we do?” In essence, challenge status quo if you want to keep on moving forward in a time that introduces new technologies and application landscapes on a daily basis! ================================================================================ Title: DB Deepdive part 6: Query Plans, Intermediate Results, tempdb and Storage Performance URL: https://frankdenneman.ai/2015-04-09-db-deepdive-part-6-query-plans-intermediate-results-tempdb-and-storage-performance/ Date: 2015-04-09 Welcome to part 6 of the Database workload characteristics series. Databases are considered to be one of the biggest I/O consumers in the virtual infrastructure. Database operations and database design are a study upon themselves, but I thought it might be interested to take a small peak underneath the surface of database design land. I turned to our resident Database expert Bala Narasimhan, PernixData’s VP of products to provide some insights about the database designs and their I/O preferences. Previous instalments of the series: Part 1 - Database Structures Part 2 – Data pipelines Part 3 - Ancillary structures for tuning databases Part 4 - NoSQL platforms Part 5 - Query Execution Plans In a previous article I introduced the database query optimizer and described how it works. I then used a TPC-H like query and the SQL Server database to explain how to understand the storage requirements of a query via the query optimizer. In today’s article we will deep dive into a specific aspect of query execution that severely impacts storage performance; namely intermediate results processing. For today’s discussion I will use the query optimizer within the PostgreSQL database. The reason I do this is because I want to show you that these problems are not database specific. Instead, they are storage performance problems that all databases run into. In the process I hope to make the point that these storage performance problems are best solved at the infrastructure level as opposed to doing proprietary infrastructure tweaks or rewrites within the database. After a tour of the PostgreSQL optimizer we will go back to SQL Server and talk about a persistent problem regarding intermediate results processing in SQL Server; namely tempdb. We’ll discuss how users have tried to overcome tempdb performance problems to date and introduce a better way. What are intermediate results? Databases perform many different operations such as sorts, aggregations and joins. To the extent possible a database will perform these operations in RAM. Many times the data sets are large enough and the amount of RAM available is limited enough that these operations won’t fully fit in RAM. When this happens these operations will be forced to spill to disk. The data sets that are written to and subsequently read from disk as part of executing operations such as sorts, joins and aggregations are called intermediate results. In today’s article we will use sorting as an example to drive home the point that storage performance is a key requirement for managing intermediate results processing. The use case For today’s example we will use a table called BANK that has two columns ACCTNUM and BALANCE. This table tracks the account numbers in a bank and the balance within each account. The table is created as shown below: Create Table BANK (AcctNum int, Balance int); The query we are going to analyze is one that computes the number of accounts that have a given balance and then provides this information in ascending order by balance. This query is written in SQL as follows: Select count(AcctNum), Balance from BANK GROUP BY Balance ORDER BY Balance; The ORDER BY clause is what will force a sort operation in this query. Specifically we will be sorting on the Balance column. I used the PostgreSQL database to run this query. I loaded approximately 230 million rows into the BANK table. I made sure that the cardinality of the Balance column is very high. Below I have a screenshot from the PostgreSQL optimizer for this query. Note that the query will do a disk based merge sort and will consume approximately 4 GB of disk space to do this sort. A good chunk of the query execution time was spent in the sort operation. A disk-based sort, and other database operations that generate intermediate results, is characterized by large writes of intermediate results followed by reads of those results for further processing. IOPS is therefore a key requirement. What is especially excruciating about the sort operation is that it is a materialization point. What this means is that the query cannot make progress until the sort is finished. You’ve essentially bottlenecked the entire query on the sort and the intermediate results it is processing. There is no better validation of the fact that storage performance is a huge impediment for good query times. What is tempdb? tempdb is a system database within SQL Server that is used for a number of reasons including the processing of intermediate results. This means that if we run the query above against SQL Server the sorting operation will spill intermediate results into tempdb as part of processing. It is no surprise then that tempdb performance is a serious consideration in SQL Server environments. You can read more about tempdb here. How do users manage storage performance for intermediate results including tempdb? Over the last couple of years I’ve talked to a number of SQL Server users about tempdb performance. This is a sore point as far as SQL Server performance goes. One thing I’ve seen customers do to remediate the tempdb performance problem is to host tempdb alone in arrays that have fast media, such as flash, in them in the form of either hybrid arrays or All Flash Arrays (AFA). The thought process is that while the ‘fast array’ is too expensive to standardize on, it makes sense to carve out tempdb alone from it. In this manner, customers look at the ‘fast array’ as a performance band aid for tempdb issues. On the surface this makes sense since an AFA or a hybrid array can provide a performance boost for tempdb. Yet it comes with several challenges. Here are a few: You now have to manage tempdb separately from all the other datastores for your SQL Server. You procure the array for tempdb yet you do not leverage any of its data services. You use it as a performance band aid alone. This makes the purchase a lot more expensive than it seems on paper. For queries that don’t leverage tempdb the array is not useful. Performance problems in databases are not limited to tempdb. For example, you may be doing full table scans and these don’t benefit from the array. You cannot leverage innovations in media. You cannot, for example, leverage RAM or PCM or anything else that will come in the future for tempdb. How can PernixData FVP help? In my mind PernixData FVP is the ideal solution for storage performance problems related to intermediate results in general and tempdb in particular. Intermediate result processing, including tempdb, shows very good temporal locality and is therefore an ideal candidate for FVP. Below are some other reasons why FVP is ideal for this scenario: PernixData FVP allows you to use all server side media, flash or RAM, for accelerating tempdb and intermediate results processing. You don’t need to configure anything separately for this. Instead you operate at the VM level and accelerate your database VM as a result of which every I/O operation is enhanced including intermediate results processing. As your tempdb requirements change – lets say you need for space for handling it – it’s simply a matter of replacing one flash card with bigger one as far as FVP is concerned. There is no down time and the application is not impacted. This allows you to ride the price/performance curve of flash seamlessly. ================================================================================ Title: Database workload characteristics and their impact on storage architecture design – part 5 - Query Execution Plans URL: https://frankdenneman.ai/2015-04-07-database-workload-characteristics-and-their-impact-on-storage-architecture-design-part-5-query-execution-plans/ Date: 2015-04-07 Welcome to part 5 of the Database workload characteristics series. Databases are considered to be one of the biggest I/O consumers in the virtual infrastructure. Database operations and database design are a study upon themselves, but I thought it might be interested to take a small peak underneath the surface of database design land. I turned to our resident Database expert Bala Narasimhan, PernixData’s VP of products to provide some insights about the database designs and their I/O preferences. Previous instalments of the series: Part 1 - Database Structures Part 2 – Data pipelines Part 3 - Ancillary structures for tuning databases Part 4 - NoSQL platforms Databases are a critical application for the enterprise and usually have demanding storage performance requirements. In this blog post I will describe how to understand the storage performance requirements of a database at the query level using database tools. I’ll then explain why PernixData FVP helps not only to solve the database storage performance problem but also the database manageability problem that manifests itself when storage performance becomes a bottleneck. Throughout the discussion I will use SQL Server as an example database although the principles apply across the board. Query Execution Plans When writing code in a language such as C++ one describes the algorithm one wants to execute. For example, implementing a sorting algorithm in C++ means describing the control flow involved in that particular implementation of sorting. This will be different in a bubble sort implementation versus a merge sort implementation and the onus is on the programmer to implement the control flow for each sort algorithm correctly. In contrast, SQL is a declarative language. SQL statements simply describe what the end user wants to do. The control flow is something the database decides. For example, when joining two tables the database decides whether to execute a hash join, a merge join or a nested loop join. The user doesn’t decide this. The user simply executes a SQL statement that performs a join of two tables without any mention of the actual join algorithm to use. The component within the database that comes up with the plan on how to execute the SQL statement is usually called the query optimizer. The query optimizer searches the entire space of possible execution plans for a given SQL statement and tries to pick the optimal one. As you can imagine this problem of picking the most optimal plan out of all possible plans can be computationally intensive. SQL’s declarative nature can be sub-optimal for query performance because the query optimizer might not always pick the best possible query plan. This is usually because it doesn’t have full information regarding a number of critical components such as the kind of infrastructure in place, the load on the system when the SQL statement is run or the properties of the data. . One example of where this can manifest is called Join Ordering. Suppose you run a SQL query that joins three tables T1, T2, and T3. What order will you join these tables in? Will you join T1 and T2 first or will you join T1 and T3 first? Maybe you should join T2 and T3 first instead. Picking the wrong order can be hugely detrimental for query performance. This means that database users and DBAs usually end up tuning databases extensively. In turn this adds both an operational and a cost overhead. Query Optimization in Action Let’s take a concrete example to better understand query optimization. Below is a SQL statement from a TPC-H like benchmark. select top 20 c_custkey, c_name, sum(l_extendedprice * (1 - l_discount)) as revenue, c_acctbal, n_name, c_address, c_phone, c_comment from customer, orders, lineitem, nation where c_custkey = o_custkey and l_orderkey = o_orderkey and o_orderdate >= ':1' and o_orderdate < dateadd(mm,3,cast(':1'as datetime)) and l_returnflag = 'R' and c_nationkey = n_nationkey group by c_custkey, c_name, c_acctbal, c_phone, n_name, c_address, c_comment order by revenue; The SQL statement finds the top 20 customers, in terms of their effect on lost revenue for a given quarter, who have returned parts they bought. Before you run this query against your database you can find out what query plan the optimizer is going to choose and how much it is going to cost you. Figure 1 depicts the query plan for this SQL statement from SQL Server 2014 [You can learn how to generate a query plan for any SQL statement on SQL Server at https://msdn.microsoft.com/en-us/library/ms191194.aspx. You should read the query plan from right to left. The direction of the arrow depicts the flow of control as the query executes. Each node in the plan is an operation that the database will perform in order to execute the query. You’ll notice how this query starts off with two Scans. These are I/O operations (scans) from the tables involved in the query. These scans are I/O intensive and are usually throughput bound. In data warehousing environments block sizes could be pretty large as well. A SAN will have serious performance problems with these scans. If the data is not laid out properly on disk, you may end up with a large number of random I/O. You will also get inconsistent performance depending on what else is going on in the SAN when these scans are happening. The controller will also limit overall performance. The query begins by performing scans on the lineitem table and the orders table. Note that the database is telling what percentage of time it thinks it will spend in each operation within the statement. In our example, the database thinks that it will spend about 84% of the total execution time on the Clustered Index Scan on lineitem and 5% on the other. In other words, 89% of the execution time of this SQL statement is spent in I/O operations! It is no wonder then that users are wary of virtualizing databases such as these. You can get even more granular information from the query optimizer. In SQL Server Management Studio, if you hover your mouse over a particular operation a yellow pop up box will appear showing very interesting statistics. Below is an example of data I got from SQL Server 2014 when I hovered over the Clustered Index Scan on the lineitem able that is highlighted in Figure 1. Notice how Estimated I/O cost dominates over Estimated CPU cost. This again is an indication of how I/O bound this SQL statement is. You can learn more about the fields in the figure above here. An Operational Overhead There is a lot one can learn about one’s infrastructure needs by understanding the query execution plans that a database generates. A typical next step after understanding the query execution plans is to tune the query or database for better performance. For example, one may build new indexes or completely rewrite a query for better performance. One may decide that certain tables are frequently hit and should be stored on faster storage or pinned in RAM. Or, one may decide to simply do a complete infrastructure rehaul. All of these result in operational overheads for the enterprise. For starters, this model assumes someone is constantly evaluating queries, tuning the database and making sure performance isn’t impacted. Secondly, this model assumes a static environment. It assumes that the database schema is fixed, it assumes that all the queries that will be run are known before hand and that someone is always at hand to study the query and tune the database. That’s a lot of rigidity in this day and age where flexibility and agility are key requirements for the business to stay ahead. A solution to database performance needs without the operational overhead What if we could build out a storage performance platform that satisfies the performance requirements of the database irrespective of whether query plans are optimal, whether the schema design is appropriate or whether queries are ad-hoc or not? One imagines such a storage performance platform will completely take away the sometimes excessive tuning required to achieve acceptable query performance. The platform results in an environment where SQL is executed as needed by the business and the storage performance platform provides the required performance to meet the business SLA irrespective of query plans. This is exactly what PernixData FVP is designed to do. PernixData FVP decouples storage performance from storage capacity by building a server side performance tier using server side flash or RAM. What this means is that all the active I/O coming from the database, both reads and writes, whether sequential or random, and irrespective of block size is satisfied at the server layer by FVP right next to the database. You are longer limited by how data is laid out on the SAN, or the controller within the SAN or what else is running on the SAN when the SQL is executed. This means that even if the query optimizer generates a sub optimal query plan resulting in excessive I/O we are still okay because all of that I/O will be served from server side RAM or flash instead of network attached storage. In a future blog post we will look at a query that generates large intermediate results and explain why a server side performance platform such as FVP can make a huge difference. Post originally appeared on ToddMace.io ================================================================================ Title: Don’t backup. Go forward with Rubrik URL: https://frankdenneman.ai/2015-03-24-dont-backup-go-forward-with-rubrik/ Date: 2015-03-24 Rubrik has set out to build a time machine for cloud infrastructures. I like the message as it shows that they are focused on bringing simplicity to the enterprise backup world. Last week I had the opportunity to catch up with them and they had some great news to share as they were planning to come out of stealth this week. And that day is today. Rubrik platform This time machine is delivered on a 2U commodity appliance that runs the Rubrik software. By installing this appliance you greatly reduce the number of machines that are necessary to provide backup and restore services today. Reducing the number of machines simplifies the infrastructure for architects and support, while the UI of Rubrik simplifies the day-to-day operations of the administrators. User Interface No agents are needed in the virtual datacenters to discover the workload and the user interface is centered on policy driven SLAs. Unfortunately I can’t show the user-interface, but trust me this is something you longed for a long time. Due to the pedigree of the co-founders it comes as no surprise that the Rubrik platform is fully programmable with REST API’s. Typically moving to a new backup system introduces risk and cost. Learning curves are high, misconfigured backup configurations possibly risking data loss. Policy driven and the ability to use REST APIs ensure that the platform easily integrates in every environment. The policies are so easy to use that no training is necessary; this reduces the impact of transition to a new backup system. The low learning curve means that no countless hours are lost by figuring out how to safely backup your data, while the REST APIs allow advanced tech crews to integrate Rubrik in their highly automated service offerings. Architecture One thing that made me very happy to see is the Rubriks’ ability to “cloud-out” your data. Rubrik provides a gateway to AWS allowing you to send “aged data” to the cloud in a very secure way. This feature benefits the complexity reduction of local architecture. Instead of having to incorporate a tape library, you now only need an Internet connection. Having worked with a big tape libraries myself for years I know this will not only bring a lot of datacenter space back and reduce your energy bill, you WILL have way less heat to cool. As the team understand the concept of distributed architectures thoroughly (more about that in the next paragraph) it doesn’t come as a surprise that it scales very well. The architecture can scale to 1000s of nodes. What’s interesting is that it can mount the snapshots directly on the Rubrik platform allowing virtual machines to run directly on the appliance. Think about the possibilities for development. Snapshot your current production workload and test your new code instantly without any impact on active services. Rubrik starts off by supporting VMware vSphere and it makes sense to focus on the biggest market out there as a startup. But support for other hypervisors and cloud infrastructures (to cloud-out data) will follow. I expect Rubrik to become a success, the product aligns with the todays enterprise datacenter requirements and the pedigree of the team is amazing. As mentioned before the co-founders have a very rich background in distributed systems, Arvind Jain was the founding Engineer of Riverbed and was a Distinguished Engineer at Google before co-founding the other three members. Interesting enough (for me at least) there are stong ties with PernixData. Prior to Rubrik, Bipul Sinha was a partner at LightSpeed before founding Rubrik. I had the great pleasure of talking to Bipul often, as he is the initial investor of PernixData. Funny enough I can recall a conversation where Bipul asked me on my view of the backup world. I believe boring and totally not sexy was my initial reply. Guess he is setting out to change that fast! The CTO of Rubrik, Arvind Nithrakashyap, worked at Oracle where he co-founded Oracle Exadata. The other Co-founder of Exadata is PernixData CEO Poojan Kumar. Last but certainly not least Soham Mazumdar, who worked at Google on the search engine and founded Tagtile. As of today you can sign-up for the early access program. Go visit the website to read more and follow them on twitter. Exciting times ahead for Rubrik! Don’t Backup. Go Forward! ================================================================================ Title: Part 3 - Data path is not managed as a clustered resource URL: https://frankdenneman.ai/2015-03-16-part-3-data-path-is-not-managed-as-a-clustered-resource/ Date: 2015-03-16 Welcome to part 3 of the Virtual Datacenter scaling problems with traditional shared storage series. Last week I published an article about the FAST presentation “A Practical Implementation of Clustered Fault Tolerant Write Acceleration in a Virtualized Environment”. Ian Forbes followed up with the question about the advantages of throughput and latency of host-to-host network versus a traditional SAN when both have similar network speeds. Part 1: Intro Part 2: Storage Area Network topology IOPS distribution amongst ESXi hosts In the previous part of this series the IOPS provided by the array were equally divided amongst the ESXi hosts. In reality given the nature of applications and their variance it’s the application demand that drives the I/O demand. And due to this I/O demand will not be equally balanced across all the host in the cluster. The virtual datacenter is comprised of different resource layers each with components that introduce their own set of load balancing algorithms. Back in 2009 Chad published nice diagram depicting all the queues and buffers of a typical storage environment. Go read the excellent article “VMware I/O queues, micro bursting and multipathing”. How can you ensure that the available paths to the array are load-balanced based on virtual machine demand and importance? Unfortunately for us today’s virtual datacenter lacks load balance functionality that clusters these different layers, reducing hotspots and optimally distributes workloads. Let’s focus on the existing algorithms currently available, and possibly present, in virtual datacenters around the world. Clustered load balancers The only cluster-wide load balancing tools are DRS and Storage DRS. Both cluster resources into seamless pools and distribute workload according to their demand and their priority. When the current host cannot provide the entitled resources a virtual machine demands, the virtual machine is migrated to another host or datastore. DRS aggregates CPU and memory resources, Storage DRS tries to mix and match the VM I/O & capacity demand with the datastore I/O and capacity availability. The layers between compute and datastores are equally important yet network bandwidth and data paths are not managed as a clustered resource. Load balancing occurs within the boundaries of the host; specifically they focus on outgoing data streams. Data Path load balancing With IP-based storage networks, multiple options exist to balance the workload across the outgoing ports. With iSCSI, binding of multiple VMkernel NICs can be used to distribute workload, some storage vendors prefer a configuration using multiple VLANS to load balance across storage ports. When using NFS Load Based Teaming (LBT) can be used to load balance data across multiple NICs. Unfortunately all these solutions don’t take the path behind the first switch port in consideration. Although the existing workload is distributed across the available uplinks as efficiently as possible, no solution exists that pools the connected paths of the hosts in the cluster and distribute the workloads across the hosts accordingly. A solution that distributes virtual machines across host with less congested data paths in a well-informed and automatic manner simply does not exist. Distributed I/O control Storage I/O Control (SIOC) is a datastore-wide scheduler, allowing distributing of queue priority amongst the virtual machines located on various hosts that are connected to that datastore. SIOC is designed to deal with situations where contention occurs. If necessary it divides the available queue slots across the hosts to satisfy the I/O requirements based on the virtual machine priority. SIOC measures the latency from the (datastore) device inside the kernel to disk inside the array. It is not designed to migrate virtual machines to other hosts into the cluster to reduce latency or bandwidth limitations incurred by the data path. Network IO Control (NetIOC) is based on similar framework. It allocates and distributes bandwidth across the virtual machines that are using the NICs of that particular host. It has no ability to migrate virtual machines by taking lower utilized links of other hosts in the cluster into account. Multipathing software VMware Pluggable Storage Architecture (PSA) is interesting. The PSA allows third party vendors to provide their own native multipathing software (NMP). Within the PSA, Path Selection Plugins (PSPs) are active that are responsible for choosing a physical path for I/O requests. The VMware Native Multipathing Plugin framework supports three types of PSPs; Most Recently Used (MRU), Fixed and Round Robin. The Storage Array Type Plugins (SATP) run in conjunction with NMP and manages array specific operations. SATPs are aware of storage array specifics, such as whether it’s an active/active array or active/passive array. For example, when the array uses ALUA (Asymmetric LUN Unit Access) it determines which paths lead to the ports of the managing controllers. The Round Robin PSP distributes I/O for a datastore down all active paths to the managing controller and uses a single path for a given number of I/O operations. Although it distributes workload across all (optimized) paths, it does not guarantee that throughput will be constant. There is no optimization on the I/O profile of the host. Its load balance algorithm is based purely on equal numbers of I/O down a given path, without regard to what the block size of I/O type is, it will not be balanced on application workload characteristics or the current bandwidth utilization of the particular path. Similar to SIOC and NetIOC, NMP is not designed to treat data paths as clustered resource and has no ability to distribute workloads across all available uplinks in the cluster. EMC PowerPath is a third party NMP and has multiple algorithms that consider current bandwidth consumption of the paths and the pending types of I/O. It also integrates certain storage controller statistics to avoid negative affects by continuously switching paths. PowerPath squeezes as much performance (and resilience) out of their storage paths as possible because it can probe the link from host all the way to the back-end of a supported array and make decisions about active links accordingly. However PowerPath hosts do not communicate with each other and balances the I/O load on a host-by-host basis. This paragraph focuses on EMC solution; other storage vendors are releasing their NMP software with similar functionality. However not all vendors are providing their own software, and EMC PowerPath is only supported on a short list of storage vendors other than EMC own products. Quality of Services on data paths Quality of Services on data paths (QoS) is an interesting solution if it provides end-to-end QoS; from virtual machine to datastore. The hypervisor is context-rich environment, allowing kernel services to understand which I/O belongs to which virtual machine. However when the I/O exits the host and hits the network, the remaining identification is the address of the transmitting device of the host. There is no differentiation of priority possible other then at host level. Not all applications are equally important to the business, therefor end-to-end QoS is necessary to guarantee that business critical application get the resources they deserve. Scalability limitation on storage controller ports influences the overall impact of QoS. Similar to most algorithms, it does not aim to provide a balanced utilization of all available data paths; it deals with priority control during resource contention. Storage Array layer Storage DRS is able to migrate virtual machine files based on their resource demand. Storage DRS monitors the VMobserved latency, this includes the kernel and data path latency. Storage DRS incorporates the latency to calculate the benefit a migration has on the overall change in latency at source and destination datastore. It does not use the different latencies of kernel and data path to initiate a migration at compute level. Storage DRS initiates a self-vMotion to load the new VMX file as it has a different location after the storage vMotion, the virtual machine remains on the same hosts. In other words Storage DRS is not designed to migrate virtual machine at the compute layer or datastore layer to solve bandwidth imbalance. Most popular arrays provide asymmetric LUN unit access. All ports on the Storage controllers accept incoming read and write operations, however the controller owning the LUN always manages read operations. Distributing LUNs across controllers are crucial as imbalance of CPU utilization or port utilization of the storage controllers can be easily introduced. LUNs can be manually transferred to improve CPU utilization, however this is not done dynamically unfortunately. Manual detection and management is as good as people watching it and not many organizations watch the environment at that scrutiny level all the time. Some might argue that arrays transfer the LUNs automatically, but that’s when a certain amount of “proxy reads” are detected. This means that the I/O’s are transmitted across the non-optimized paths, and likely either the PSP is not doing a great job, or all your active optimized paths are dead. Both not hallmarks of an healthy – or –properly architected environment. Is oversizing a solution? Oversizing bandwidth can help you so far, as its difficult to predict workload increase and intensity variation. Introduction of radically new application landscape impact current designs tremendously. When looking at the industry developments, its almost certain that most datacenters will be forced to absorb these new application landscapes, can current solutions applied in a traditional storage stack provide and guarantee the services they require and are they able to scale to provide the resources necessary? Non-holistic load balancer available In essence, the data-path between the compute layer and datastore layer is not treated as a clustered resource. Virtual machine host placement is based on compute resource availability and entitlement, disregarding the data path towards storage layer. This can potentially lead to hotspots inside the cluster, where some hosts saturate their data paths, while data paths of other hosts are underutilized. Data path saturation impacts application performance. Unfortunately there is no mechanism available today that takes the various resource demands of a virtual machine into account. No solution at this time has the ability to intelligently manage these resources without creating other bottlenecks or impracticalities. This article is not a stab at the current solutions. It is a very difficult problem to solve, especially for an industry that relies on various components of various vendors, expecting everything to integrate and perform optimally aligning. And think about moving forward and attempt to incorporate new technology advancements from all different vendors when they are available. And don’t forget about backward compatibility, world peace might be easier to solve. With this problem in mind, the existence of uncontrollable data paths, oversubscribed inter-switch links and its inability to be application aware, many solutions nowadays move away from the traditional storage architecture paradigm. Deterministic performance delivery and Policy-driven management are the future and when no centralized control plane is available that stitching these disparate components together a different architecture arises. PernixData FVP, VMware VSAN and Hyper-converged systems rely on point-to-point network architectures of the crossbar switch architecture to provide consistent and non-blocking network performance to cater their storage performance and storage service resiliency needs. Leveraging point-to-point connections and being able to leverage the context-aware hypervisor allows you not only to scale easily, it allows you to create environments that provide consistent and deterministic performance levels. The future datacenter is one step closer! Part 4 focusses on the storage controller architecture and why leveraging host-to-host communication and host resource availability remove scalability issues. ================================================================================ Title: Virtual Datacenter scaling problems with traditional shared storage - part 2 URL: https://frankdenneman.ai/2015-03-12-virtual-datacenter-scaling-problems-with-traditional-shared-storage-part-2/ Date: 2015-03-12 Oversubscription Ratios The most common virtual datacenter architecture consists of a group of ESXi hosts connected via a network to a centralized storage array. The storage area network design typically follows the core-to-edge topology. In this topology a few high-capacity core switches are placed in the middle this topology. ESXi hosts are sometimes connected directly to this core switch but usually to a switch at the edge. An inter-switch link connects the edge switch to the core switch. Same design applies to the storage array; it’s either connected directly to the core switch or to an edge switch. This network topology allows scaling out easily from a network perspective. Edge switches (sometimes called distribution switches) are used to extend the port count. Although core switches are beefy, there is however a limitation on the number of ports. And each layer in the network topology has its inherent roles in the design. One of the drawbacks is that Edge-to-Core topologies introduce oversubscription ratios, where a number of edge ports use a single connection to connect to a core port. Placement of systems begins to matter in this design as multiple hops increase latency. To reduce latency the number of hops should be reduced, but this impacts port count (and thus number of connected hosts) which impacts bandwidth availability, as there are a finite amounts of ports per switch. Adding switches to increase port count brings you back to device placement problems again. As latency play a key role in application performance, most storage area network aim to be as flat as possible. Some use a single switch layer to connect the host layer to the storage layer. Let’s take a closer look on how scale out compute and this network topology impacts storage performance: Storage Area Network topology This architecture starts of with two hosts, connected with 2 x 10GB to a storage array that can deliver 33K IOPS. The storage area network is 10GB and each storage controller has two 10GB ports. To reduce latency as much as possible, a single redundant switch layer is used that connects the ESXi hosts to the storage controller ports. It looks like this: In this scenario the oversubscription ratio of links between the switch and the storage controller is 1:1. The ratio of consumer network connectivity to resource network connectivity is equal. New workload is introduced which requires more compute resources. The storage team increases the spindle count to increase more capacity and performance at the storage array level. Although both the compute resources and the storage resources are increased, no additional links between the storage controllers are added. The oversubscription ratio is increased and now a single 10Gb link of an ESXi host has to share this link with potentially 5 other hosts. The answer is obviously increasing the number of links between the switch and the storage controllers. However most storage controllers don’t allow scaling the network ports. This design stems from the era where storage arrays where connected to a small number of hosts running a single application. On top of that, it took the non-concurrent activity into account. Not every application is active at the same time and with the same intensity. The premise of grouping intermitted workloads led to virtualization, allowing multiple applications using powerful server hardware. Consolidation ratios are ever expanding, normalizing intermittent workload into a steady stream of I/O operations. Workloads have changed, more and more data is processed every day, pushing these IO’s of all these applications through a single pipe. Bandwidth requirements are shooting through the roof, however many storage area network designs are based of best practices prior to the virtualization era. And although many vendors stress to aim for a low oversubscription ratio, the limitation of storage controller ports prevents removing this constraint. In the scenario above I only used 6 ESXi hosts, typically you will see a lot more ESXi hosts connected to the same-shared storage array, stressing the oversubscription ratio. In essence you have too squeeze more IO through a smaller funnel, this will impact latency and bandwidth performance. Frequently scale-out problems with traditional storage architecture are explained by calculating the average number of IOPS per host by dividing the number of host by the total number of IOPS provided by the array. In my scenario, the average number of IOPS is 16.5K IOPS remained the same due to the expansion of storage resources at the same time the compute resources were added (33/2 or 100/6). Due to the way storage is procured (mentioned in part 1) storage arrays are configured for expected peak performance at the end of its life cycle. When the first hosts are connected, bandwidth and performance are (hopefully) not a problem. New workloads lead to higher consolidation ratio’s, which typically result in expansion of compute cycles to keep the consolidation ratio at a certain level to satisfy performance and availability requirements. This generally leads to reduction of bandwidth and IOPS per hosts. Arguably this should not pose a problem if sizing was done correctly. Problem is, workload increase and workload behavior typically do not align with expectations, catching the architects off-guard or simply new application landscape turn up unexpectedly when business needs change. A lack of proper analytics impacts consumption of the storage resources and avoiding hitting limits. It’s not unusually for organizations to experience performance problems due to lack of proper visibility in workload behavior. To counteract this, more capacity is added to the storage array to satisfy capacity and performance requirements. However this does not solve the problem that exists right in the middle of these two layers. It ignores the funnel created by the oversubscription ratio of the links connected to the storage controller ports. The storage controller port count impact the ability to solve this problem, another problem is that the way total bandwidth is consumed. The activity of the applications and the distribution of the virtual machines across the compute layer affect the storage performance, as workload might not be distributed equally across the links to the storage controllers. Part 3 of this series will focus on this problem. ================================================================================ Title: Memory Deep Dive Summary URL: https://frankdenneman.ai/2015-03-02-memory-deep-dive-summary/ Date: 2015-03-02 This is last part of the memory deep dive. As the total series count 7667 words, I thought it would be a good idea to create a summary of the previous 6 parts. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary The reason why I started this deep dive is to understand the scalability of server memory configurations and constrains certain memory type introduce. Having unpopulated DIMM slot does not always translate into future expandability of memory capacity. A great example is the DIMM layout of today’s most popular server hardware. The server boards of the Cisco UCS B200 M4, HP Proliant DL380 Gen 9, and Dell PowerEdge 730 Gen 13 come equipped with 2 CPU’s and 24 DIMM slots. Processors used in the aforementioned systems are part of the new Intel Xeon 26xx v3 micro-architecture. It uses multiple onboard memory controllers to provide a multi-channel memory architecture. Multi-channel configurations, DIMM ranking and DIMM types must be considered when designing your new server platform. If these are not taken into account, future scalability might not be possible or the memory will not perform as advertised. Multi-channel memory architecture Modern CPU microarchitectures support triple or quadruple memory channels. This allows the memory controller to access the multiple DIMMs simultaneously. The key to high bandwidth and low latency is interleaving. Data is distributed in small chunks across multiple DIMMs. Smaller bits of data are retrieved from each DIMM across independent channels instead of accessing a single DIMM for the entire chunk of data across one channel. For in-depth information, please go to part 2. The Intel Xeon 26xx v3 micro-architecture offers a quad-channel memory architecture. To leverage all the available bandwidth each channel should be populated with at least one DIMM. This configuration has the largest impact on performance and especially on throughput. Part 4 dives into multi-channel configuration in depth. The configuration depicted above leverages all four channels and allows the CPU to interleave memory operations across all four channels. The memory controller groups memory across the channels in a region. When creating a 1 DIMM per Channel configuration, the CPU creates one region (Region 0) and interleaves the memory access. The CPU will use the available bandwidth if less than 4 DIMMs are used, for example when 3 DIMMs are used, the memory controller uses three channels to interleave memory. 2 DIMMs result in two usable channels, and one DIMM will use a single channel, disabling interleaving across channels. Populating four channels provide the best performance, however sometimes extra capacity is required, but less than DIMMS populating four channels provide. For example if 384 GB is required and 32 GB DIMMs are used, 6 DIMMS are used. The CPU will create two Regions. Region 0 will run in quad channel mode, while region 1 runs in dual channel mode: This creates an unbalanced memory channel configuration, resulting in inconsistent performance. With Quad Channel configurations its recommended to add memory in groups of 4 DIMMS. Therefor use 4, 8 or 12 DIMMs per CPU to achieve the required memory capacity. Memory Ranking A DIMM groups the chips together in ranks. The memory controller can access ranks simultaneously and that allows interleaving to continue from channel to rank interleaving. Rank interleaving provides performance benefits as it provides the memory controller to parallelize the memory request. Typically it results in a better improvement of latency. DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). To increase capacity, combine the ranks with the largest DRAM chips. A quad-ranked DIMM with 4Gb chips equals 32GB DIMM (4Gb x 8bits x 4 ranks). As server boards have a finite amount of DIMM slots, quad-ranked DIMMs are the most effective way to achieve the highest memory capacity. Unfortunately current systems allow up to 8 ranks per channel. Therefor limiting the total capacity and future expandability of the system. 3 DIMMs of QR provide the most capacity however it is not a supported configuration as 12 ranks exceeds the allowable 8 ranks. Ranking impacts the maximum number of DIMMs used per channel. If current memory capacity of your servers needs to be increased, verify the ranking configuration of the current memory modules. Although there might be enough unpopulated DIMM slots, quad rank memory modules might prevent you from utilizing these empty DIMM slots. LRDIMMs allow large capacity configurations by using a memory buffer to obscure the number of ranks on the memory module. Although LRDIMMs are quad ranked, the memory controller only communicates to the memory buffer reducing the electrical load on the memory controller. DIMMs per Channel A maximum of 3 DIMMs per channel are allowed. If one DIMM is used per channel, this configuration is commonly referred to as 1 DIMM Per Channel (1 DPC). 2 DIMMs per channel (2 DPC) and if 3 DIMMs are used per channel, this configuration is referred to as 3 DPC. When multiple DIMMs are used per channel they operate at a slower frequency. DIMM Type 1 DPC 2 DPC 3 DPC SR RDIMM 2133 MHz 1866 MHz 1600 MHz DR RDIMM 2133 MHz 1866 MHz 1600 MHz QR RDIMM 2133 MHz 1866 MHz N/A QR LRDIMM 2133 MHz 2133 MHz 1600 MHz The frequency of DDR4 LRDIMMs remains the same whether it is used in 1 DPC or 2 DPC configurations. It drops to RDIMM frequency levels when using it in a 3-DPC configuration. Multiple tests published online show that LRDIMM frequency drop-off is less than the proposed standard. Most tests witnessed a drop from 2133 MHz to 1866 MHz, retaining high levels of performance. Memory frequency impact both available bandwidth and latency. Performance As mentioned in part 5, the two primary measurements for performance in storage and memory are latency and throughput. Interestingly enough, memory bandwidth increases with every generation, however latency does not always improve immediately. Actually every generation of memory moves the performance dial backwards when comparing latency with its predecessor. The interesting part is that memory bandwidth is a factor of latency. Latency is a generic term, when reviewing the latency and bandwidth relationship, one has to review unloaded and loaded latencies. Memory latency is measured from the moment the CPU issues a read request to the moment the memory supplies it to the core. This is referred to as load to use. However, the load to use latencies differ when the memory system is idle or when it’s saturated. Unloaded latency is a measurement of an idle system and it represents the lowest latency that the system can achieve. A well-known indicator of memory latency is the CAS timings (Column Address Strobe) and it represents the unloaded latency. Basically it demonstrates the most optimal scenario. CAS timings is a good way to understand the relative latency between two memory DIMMS, however, it does not always indicate the real world performance of a system. Loaded latency is the latency when the memory subsystem is saturated with memory request and that’s where bandwidth has a positive impact on real world latency. Under loaded conditions memory requests spend time in the queue, the more bandwidth speed the memory has, the more quickly the memory controller can process the queued commands. For example, memory running at 1600 MHz has about 20% lower loaded latency than memory running at 1333 MHz. Loaded latency is the real world measurement of performance applications will experience, having bandwidth speed to reduce loaded latency is important when reviewing the DPC configuration (part 4) of your server configuration. Ranks will also have a positive impact on the loaded latency (lower latency). Having more ranks allows the memory controller to empty out its queue’s by parallelizing the process of memory requests. Parallelization is covered in part 4 of this series. Please visit the individual parts of the series for more in-depth information. I hope you enjoyed reading this series as much as I have been writing it. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive: NUMA and Data Locality URL: https://frankdenneman.ai/2015-02-27-memory-deep-dive-numa-data-locality/ Date: 2015-02-27 This is part 6 of the memory deep dive. This is a series of articles that I wrote to share what I learned while documenting memory internals for large memory server configurations. This topic amongst others will be covered in the upcoming FVP book. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary Background of multiprocessor architectures Moore’s law is often misquoted, linking the increase of transistor count to the increase of speed. According to Wikipedia Moore’s law is: An observation made by Intel co-founder Gordon Moore in 1965. He noticed that the number of transistors per square inch on integrated circuits had doubled every year since their invention. Moore’s law predicts that this trend will continue into the foreseeable future. The interesting thing is that Moore’s law is still applicable to this day. The transistor count has been increasing every year, however the speed increase was barely doubled in the last decade. From 2000 to 2009, the CPU speed went from 1.3 GHz to 2.8 GHz. Transistor-count on the other hand increased from 37.5 million in 2000 to 904 million in 2009. This means that translator count does not automatically translate in raw CPU speed increase. For that we have to get back to 2004 where the heat build-up in the chips cause Intel to abandon the consistent speed improvement and move towards a design with multiple processor (cores) on the same CPU chip. The industry followed soon after. [caption id=“attachment_5100” align=“aligncenter” width=“844”] Source: Computer Architecture, A quantitative Approach by Hennessy and Patterson[/caption] This fundamental shift in hardware design had a profound impact on software performance. Previously, faster CPU speeds translated directly into faster application performance. When switching to multi-core design, only software that could take advantage of multiple processors would get this benefit. Importance of memory architecture The importance of memory architecture has increased with the advances in performance and architecture in CPU. The shows the gap in performance, measured as the difference in the time between processor memory requests (for a single processor or core) and the latency of a DRAM access, is plotted over time. [caption id=“attachment_5099” align=“aligncenter” width=“563”] Source: Computer Architecture, A quantitative Approach by Hennessy and Patterson[/caption] The interesting thing is the plateauing performance of CPU from 2004 and onwards. This due to the increase of core count instead of CPU clock speed. This development increases the bandwidth requirements even more. The aggregate peak bandwidth essentially grows as the number of cores grows. Simply put, we can process data faster than ever, but we can’t get that data faster. To battle this, CPU design has been focusing on parallel memory architectures, and specifically the attempt to keep the data as close to the executing core as possible. Parallel Memory Architecture Two major parallel memory architectures exist Distributed Memory Architecture and Shared Memory Architecture. Shared Memory Architecture is split up in two types: Uniform Memory Access (UMA), and Non-Uniform Memory Access (NUMA). Distributed Memory Architecture is an architecture used in clusters, with different hosts connected over the network typically without cache coherency. Shared Memory Architecture is a layout of processors and memory inside a server. With UMA memory is shared across all CPUs. To access memory, the CPUs have to access a Memory Controller Hub (MCH). This type of architecture is limited to scalability, bandwidth, and latency. The MCH is connected to an I/O controller across a bus, this bus has finite speed and for any communication, the CPUs need to take control of the bus, which leads to contention problems. UMA does not scale after a certain number of processors. To solve latency, bandwidth, and scalability NUMA was introduced. NUMA moves away from a centralized pool of memory and introduces topological properties. By classifying location bases on signal path length from the processor to the memory, latency and bandwidth bottlenecks can be avoided. NUMA memory is directly attached to the CPU and this is considered to be local. Memory connected to another CPU socket is considered to be remote. Remote memory access has additional latency overhead to local memory access, as it has to traverse the interconnect and connect to the remote memory controller. As a result of the different locations memory can exist, this system experiences “non-uniform” memory access time. Keeping the memory access local or maximizing memory locality provides the best performance. However, due to CPU load balancing in the hypervisor layer it can happen that local memory becomes a remote memory. Designing the server that retains the highest bandwidth while offering the highest capacity is key with NUMA configurations. NUMA scheduling ESXi is NUMA aware and has its own NUMA scheduling. When you think about it, how would you characterize NUMA? You want to keep the memory as close to the CPU instruction as possible, so it would make sense to consider it a memory scheduler. Within ESXi, it’s a part of the CPU scheduler and this explains the focus of the NUMA scheduler. As CPU load balancing across the NUMA nodes is crucial to performance, the emphasis of the NUMA scheduler is to achieve a more balanced CPU load. When a virtual machine is powered on, the NUMA schedule assigns a home node. A home node, typically referred to as NUMA node, is the set of CPU and its local memory. In an ideal situation, the NUMA node provides the CPU and memory resources the virtual machine requires. The NUMA scheduler tries to load balance all virtual machines across the NUMA nodes in the server. The administrator can help the NUMA scheduler by right-sizing the virtual machine, attempt to keep the memory footprint within the capacity of a single NUMA node. However, when multiple virtual machines run on a server, it can happen that no optimal distribution of virtual machines can be obtained where each virtual machine working set can fit into their local NUMA node. When a virtual machine has a certain amount of memory located remote, the NUMA scheduler migrates it to another NUMA node to improve locality. It’s not documented what threshold must be exceeded to trigger the migration, but its considered poor memory locality when a virtual machine has less than 80% mapped locally. My “educated” guess is that it will be migrated when it’s below 80%. ESXTOP memory NUMA statistics show the memory location of each virtual machine. Start ESXTOP, press m for memory view, press f for customizing ESXTOP and press f to select the NUMA Statistics. Memory locality rate can be input for sizing the memory capacity of the server. DIMM type, capacity and DPC configuration need to be taken into account to maintain high throughput values. Importance of Interconnect Bandwidth CPU load varies dynamically and can cause CPU load imbalance between the NUMA nodes. This can trigger the NUMA scheduler to rebalance every couple of seconds (I believe the NUMA scheduler checks every 4 seconds). However, unlike vCPUs, memory migrates very slowly because of the cost involved of bandwidth consumption and address mappings updates. VCPUs are nimble and can bounce around the NUMA nodes quite frequently, the last thing you want to do is to migrate memory after every vCPU migration. To solve this, the NUMA scheduler initiates memory migration at a slow pace until the vCPU stops migration and stays on a NUMA node for a long period of time. Once determined the vCPU is settled, the memory migration is accelerated to achieve locality again. This behavior increases the importance of the interconnect bandwidth of a CPU when designing the server platform. With Intel the interconnect is called Intel QuickPath Interconnect, HyperTransport is the name of AMD interconnect technology (HTX 3.1 spec) Intel categorizes their CPU into roughly five segments: Basic, Standard, Advanced, Segmented Optimized and Low Power. Processor Model QPI Bandwidth Max Memory Frequency Max Memory Bandwidth E5-2603 v3 6.4 GT/s 1600 MHz 51 GB/s E5-2620 v3 8 GT/s 1866 MHz 59 GB/s E5-2637 v3 9.6 GT/s 2133 MHz 68 GB/s The QPI is configured as a link pair, two unidirectional paths exists between the CPU’s. As the links can be simultaneously active, it doubles the bandwidth spec. A clock rate of 4.0 GHz yields a data rate of 32 GB/s. QPI Transfer Speed Clock Rate Unidirectional Bandwidth Total Bandwidth 6.4 GT/s 3.20 GHz 12.8 GB/s 25.6 GB/s 8 GT/s 4.0 GHz 16.0 GB/s 32.0 GB/s 9.6 GT/s 4.80 GHz 19.2 GB/s 38.4 GB/s Although QPI bandwidth is increased with every new CPU generation, it lags in comparison with local memory bandwidth. Increasing memory capacity and CPU count should help to reduce remote memory access. This of course with right-sizing the virtual machine to the application work set however as experience taught me, sometimes the political forces are stronger than financial constraints. Local memory optimization Optimization is done across different axes. Care must be taken when configuring capacity, previous parts of this memory deep dive covered bandwidth reduction of DIMMs Per Channel Configuration. Intel Xeon v3 follows the following spec: DIMM Type 1 DPC 2 DPC 3 DPC 1R RDIMM 2133 MHz 1866 MHz 1600 MHz 2R RDIMM 2133 MHz 1866 MHz 1600 MHz 4R RDIMM 2133 MHz 1866 MHz N/A 4R LRDIMM 2133 MHz 2133 MHz 1600 MHz Sustaining high levels of bandwidth while providing large amounts of capacity is the key benefit of DDR4 LRDIMMs. Part 5 covered the advancements made with LRDIMM technology of DDR4 and especially the reduction of latency compared to its predecessors. Importance of balanced memory population When designing a server system with NUMA architecture, memory population is very important. Design for an even population of DIMMs across sockets and channels. NUMA configurations are extremely sensitive to unbalanced configurations. After the required total capacity is calculated ensure that this capacity can be distributed equally across the sockets (NUMA nodes) and the channels. A balanced memory configuration for a 2 socket Intel Xeon v3 system means that 8, 16 or 24 DIMMs are populated, and where the DIMMs are evenly distributed across all 8-memory channels (4 memory channels per CPU). Resulting in a 1 DPC, 2 DPC or 3DPC across all channels. Capacity and Bandwidth requirements impact whether RDIMMs or LRDIMMs are required. Scenario: In this scenario, the server contains 24 DIMM slots and contains two Intel E5-2637 v3 CPUs. The QPI bandwidth is 38.4 GB/s while the total amount of local memory bandwidth per CPU is 68 GB/s. DDR4-2133 provides 17 GB/s allowing quad-channel use. Tests demonstrated that using 2 DPC has a minimal impact on bandwidth when using LRDIMMs. When using RDIMM configuration in 2DPC a 16% drop was recorded. Unfortunately, some capacities are not ideal that results in NUMA balance, Channel usage and optimized bandwidth. Take for example the popular memory configuration of 384GB. 12 x 32 GB LRDIMMs can be used or 24 RDIMMs. When using 12 x 32 GB DIMMs problems occur as there are 12 DIMM slot per server managed by 4 channels. Problem 1: Unbalanced NUMA configuration: NUMA Node 0 has 256 GB, while NUMA node 1 contains 128 GB. In this configuration DPC is used to its full extent, leveraging parallelism and memory interleaving. However, the optimization of local bandwidth would not help the virtual machines who are scheduled to run in NUMA node 1, less memory available means it is required to fetch it remotely, experiencing the extra latency of multi-hops and the bandwidth constraint of the QPI compared to local memory. P****roblem 2: Unbalanced DPC Regions: The memory capacity is equally distributed across the NUMA Nodes, however, a mixed DPC configuration exists of 1 DPC and 2 DPC. This configuration does not provide a consistent memory performance. Data access is done across 4 channels, 2 channels, and across the QPI to the other memory controller than might fetch the data across two or four channels. As described in part 4 - Optimizing Performance, unbalanced DPC configurations can lead to a 30% performance decrease and this is without the impact of access data remotely. Problem 3: RDIMM 3 DPC Bandwidth reduction: Using 16GB RDIMMs results in a balanced NUMA configuration while leveraging 4 channels to its full potential. 3 DIMMs per channel per processor results in a decrease of memory bandwidth, dropping from 2133 MHz to 1600 MHz. While 2133 MHz provides up to 17 GB/s per DIMM, 1600 MHz provides a maximum bandwidth up to 12.8 GB/s per DIMM. A drop of 25%. Unfortunately, latency will also be impacted by running at a lower clock cycle. Future scalability is ruled out by using all DIMM slots. Requirements and the constraints will impact memory configurations, depending on the budget or future requirements it makes sense to increase or decrease the memory configuration or accept that future scale-out capabilities are unavailable after populating all slots. Up next, part 7: Memory Deep Dive Summary The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive: DDR4 Memory URL: https://frankdenneman.ai/2015-02-25-memory-deep-dive-ddr4/ Date: 2015-02-25 This is part 5 of the memory deep dive. This is a series of articles that I wrote to share what I learned while documenting memory internals for large memory server configurations. This topic amongst others will be covered in the upcoming FVP book. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary DDR4 Released mid-2014, DDR4 is the latest variant of DDR memory. JEDEC is the semiconductor standardization body and published the DDR4 specs (JESD79-4) in September 2012. The standard describes that the per-pin data rate ranges from 1.6 gigatransfers per second to an initial maximum objective of 3.2 gigatransfers per second. However it states that it’s likely that higher performance speed grades will be added in the future. DIMM Type Data Rate Module Name Peak Transfer Rate DDR3-800 800 MT/s PC-6400 6400 MB/s DDR3-1066 1066 MT/s PC-8500 8533 MB/s DDR3-1333 1333 MT/s PC-10600 10600 MB/s DDR3-1600 1600 MT/s PC-12800 12800 MB/s DDR3-1866 1866 MT/s PC-14900 14933 MB/s DDR3-2133 2133 MT/s PC-17000 17064 MB/s DDR4-2133 2133 MT/s PC-17000 17064 MB/s DDR4-2400 2400 MT/s PC-19200 19200 MB/s DDR4-2666 2600 MT/s PC-20800 20800 MB/s DDR4-2800 2800 MT/s PC-22400 22400 MB/s DDR4-3000 3000 MT/s PC-24000 17066 MB/s DDR4-3200 3200 MT/s PC-25600 25600 MB/s We are now two years further and DDR4 is the quickly becoming the standard of generic server hardware due to the use of the new Intel Xeon E5 v3 processor with the Haswell-E micro-architecture. This micro-architecture uses DDR4 exclusively. AMD’s upcoming Zen micro-architecture is expected to support DDR4. Zen is expected to appear on the market late 2016. Note that the pin-layout of DDR4 is different from DDR3. Not only is the key notch at a different location on the PCB, the pin size and arrangement is different as well. Towards the middle of the PCB, there is a change gradient, making some pins longer: [caption id=“attachment_5066” align=“aligncenter” width=“680”] Source: Anandtech.com[/caption] Besides the higher data rate transfer speeds, DDR4 offers higher module density. While DDR3 DRAM chip contains 8 internal banks, DDR4 can contain up to 16 internal banks. The DDR4 standard allows up to 128GB per DIMM, allowing extreme high-density memory configurations. Another major improvements is memory power consumption. DDR4 memory operates at a lower voltage than DDR3. DDR4 modules typically require 1.2 volts with a frequency between 2133MHz and 4266MHz. As a comparison; DDR3 operates between 800 and 2400 MHz with a voltage requirement between 1.5 and 1.65V. DDR3 Low voltage can operates at 800 MHz while requiring 1.35V. DDR4 performance better at a lower voltage, low voltage DDR4 is not yet announced but it’s estimated to operate at 1.05 volts. Latency As mentioned in part 4, the two primary measurements for performance in storage and memory are latency and throughput. Interestingly enough, memory bandwidth increases with every generation, however latency does not always improve immediately. Actually every generation of memory moves the performance dial backwards when comparing latency with its predecessor. Why does latency lag behind bandwidth? Moore’s law is a big factor. Moore’s law helps bandwidth more than latency as transistors become faster and smaller. This allows the memory vendor to place more transistors on the board. More transistors means more pins, means more bandwidth, but it also means that the communication traverses longer lines. The size of the DRAM chip increases as well, resulting in longer bit and word lines. It basically comes down to distance and as you are all well aware of, distance is a big factor when it comes to latency. The interesting part is that memory bandwidth is a factor of latency. Latency is a generic term, when reviewing the latency and bandwidth relationship, one has to review unloaded and loaded latencies. Memory latency is measured from the moment the CPU issues a read request to the moment the memory supplies it to the core. This is referred to as load to use. However the load to use latencies differ when the memory system is idle or when it’s saturated. Unloaded latency is a measurement of an idle system and it represents the lowest latency that the system can achieve. A well-known indicator of memory latency is the CAS timings (Column Address Strobe) and it represents the unloaded latency. Basically it demonstrates the most optimal scenario. CAS timings is a good way to understand the relative latency between two memory DIMMS, however it does not always indicate the real world performance of a system. Loaded latency is the latency when the memory subsystem is saturated with memory request and that’s where bandwidth has a positive impact on real world latency. Under loaded conditions memory requests spend time in the queue, the more bandwidth speed the memory has, the more quickly the memory controller can process the queued commands. For example, memory running at 1600 MHz has about 20% lower loaded latency than memory running at 1333 MHz. Loaded latency is the real world measurement of performance applications will experience, having bandwidth speed to reduce loaded latency is important when reviewing the DPC configuration (part 4) of your server configuration. Ranks will also have a positive impact on the loaded latency (lower latency). Having more ranks allows the memory controller to empty out its queue’s by parallelize the process of memory requests. Parallelization is covered in part 4 of this series. Bandwidth and CAS Timings The memory area of a memory bank inside a DRAM chip is made up of rows and columns. To access the data, the chip needs to be selected, then the row is selected, and after activating the row the column can be accessed. At this time the actual read command is issued. From that moment onwards to the moment the data is ready at the pin of the module, that is the CAS latency. Its not the same as load to use as that is the round trip time measured from a CPU perspective. CAS latencies (CL) increase with each new generation of memory, but as mentioned before latency is a factor of clock speed as well as the CAS latency. Generally a lower CL will be better, however they are only better when using the same base clock. If you have faster memory, higher CL could end up better. When DDR3 was released it offered two speeds, 1066MHz CL7 and 1333 MHz CL8. Today servers are equipped with 1600 MHz CL9 memory.DDR4 was released with 2133 MHz CL13. However 2133 MHz CL15 is available at the major server vendors. To work out the unloaded latency is: (CL/Frequency) * 2000. This means that 1600 MHz CL9 provides an unloaded latency of 11.25ns, while 2133 MHz CL15 provides an unloaded latency of 14.06ns. A drop of 24.9%. However DDR4 latency will drop when bandwidth increases faster then the increase of CAS latency. At the time of writing “prosumer” DDR4 memory is available at higher speeds than the server vendors offer, but it’s just a matter of time before those modules become available for server hardware. Many memory vendors offer DDR4 2800 MHz CL14 to CL 16. When using the same calculation, 2800 MHz CL16 provides an unloaded latency of (16/2800) * 2000 = 11.42ns. Almost the same latency at DDR3 1600 MHz CL9! 2800 MHZ CL14 provides an unloaded latency of 10ns, meaning that CL14 beating CL9 with a lower latency while providing more than 75% bandwidth. LRDIMMs When LRDIMMs were introduced, they delivered higher capacity at the expense of lower bandwidth and higher latency. With the introduction of the new memory controller of Intel Xeon v2, the bandwidth drop was reduced and latency was slightly improved. However still not up to the standards of registered DIMMs. The screenshot below, originating from the IDT DDR4 LRDIMM white paper, shows the bandwidth drops of LRDIMM and RDIMMs in the three Intel Xeon architectures. On the far right the Intel Xeon v1 (Sandy Bridge) is shown, the middle graph shows the Intel Xeon v2 (Ivy Bridge) and the left is the Intel Xeon v3 (Haswell-E microachitecture). [caption id=“attachment_5076” align=“aligncenter” width=“680”] Source: IDT DDR4 LRDIMM whitepaper[/caption] The reason why latency is higher in LRDIMM architecture is due to the use of the memory buffer. The fastest way is always a direct line, with unbuffered DIMMs, the memory controller communicates directly with the DRAM chip. The drawback of this architecture is described in part 2. With registered DIMMs, the memory controller sends control and management messaging to the register but still fetches data straight from the DRAM chip. DDR3 Load Reduced DIMMs, use a memory buffer for all communication, including data. That means that the distance of data travel, typically referred to as trace lengths is much longer. Distance is the enemy of latency; therefore DDR4 LRDIMMs leverage data buffers close to the DRAM chips to reduce the I/O trace lengths. With DDR3 LRDIMMs, a trace length is up to 77 millimeter. DDR4 LRDIMM trace lengths are claimed to be between 2 and 8 millimeter. This reduces the added latency tremendously. DDR4 LRDIMMs trace lengths are comparable to DDR4 RDIMM trace lengths. Where in DDR3 the added component latency of the memory buffer is approximately 2.5 ns when compared to an RDIMM, the component delay of DDR4 LRDIMM is approximately 1.2 ns. DDR4 uses smaller memory buffers with an improved buffering scheme to decrease the latency even further. The reduction of trace lengths decreases the impact on signal integrity, which in turn allows DDR4 DIMMs to operate at a higher bandwidth when using 3DPC configuration. Anandtech.com compared the memory bandwidth of RDIMMs against LRDIMM in various DPC configurations Anandtech.com noted the following: Registered DIMMs are slightly faster at 1 DPC, but LRDIMMs are clearly faster when you insert more than one DIMM per channel. We measured a 16% to 18% difference in performance. It’s interesting to note that LRDIMMs are supposed to run at 1600 at 3DPC according to Intel’s documentation, but our bandwidth measurement points to 1866. The command “dmidecode -type 17” that reads out the BIOS confirmed this. Article: Intel Xeon E5 Version 3: Up to 18 Haswell EP Cores by Johan De Gelas By using a buffer for each DRAM chip the latency overhead is thus significantly lower on DDR4 LRDIMMs. Compared to RDIIMMs at the same speed with 1 DPC the latency overhead will be small, but as soon as more DIMMS per channel are used, the LRDIMMs actually offer lower latency as they run at higher bus speeds. DDR4 in the real world Although DDR4 has to fight the perception of much slower memory, real world use need to prove this otherwise. I hope this article proved that just looking as CAS Latency numbers is a futile exercise. Memory speed is a factor in determining latency, and loaded latency (saturated system) is something enteprise applications will experience more than an idle fully unloaded system. High density, high bandwidth configurations that sustain their bandwidth is the real benefit of DDR4. DDR4 memory in general can cope with multi-DPC much better than their counterparts of the previous generation. As more DIMMs are populated the speed typically falls due to previous described electrical load on the memory controller. Multi-DPC frequency fall-off is less steep. Unfortunately at the time of writing DDR4 is still in a premium price range, however its expected to see a decline in price at the end of 2015 making it an interesting choice as a resource for both virtual machine memory and storage acceleration. Up next, part 6: NUMA Architecture and Data Locality The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive: Optimizing for Performance URL: https://frankdenneman.ai/2015-02-20-memory-deep-dive/ Date: 2015-02-20 This is part 4 of the memory deep dive. This is a series of articles that I wrote to share what I learned while documenting memory internals for large memory server configurations. This topic amongst others will be covered in the upcoming FVP book. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary Optimizing for Performance The two primary measurements for performance in storage and memory are latency and throughput. Part 2 covered the relation between bandwidth and frequency. It is interesting to see how the memory components and the how the DIMMs are populated on the server board impact performance. Let’s use the same type of processor used in the previous example’s the Intel Xeon E5 2600 v2. The Haswell edition (v3) uses DDR4, which is covered in part 5. Processor Memory Architecture The Intel E5 2600 family contains 18 different processors. They differ in number of cores, core frequency, amount of cache memory and CPU instruction features. Besides the obvious CPU metrics, system bus speed, memory types, and maximum throughput can differ as well. Instead of listing all 18, I selected three CPUs to show the difference. To compare all 18 processors of the 2600 family, please go to ark.intel.com Processor Model System Bus speed Max Memory Frequency Max Memory Bandwidth E5-2603 v2 6.4 GT/s 1333 MHz 42.6 GB/s E5-2620 v2 7.2 GT/s 1600 MHz 51.2 GB/s E5-2637 v2 8 GT/s 1866 MHz 59.7 GB/s Source: Ark.Intel.com The system bus speed is important when communicating over the quick path interconnect (QPI) to the other CPU local memory resources. This is a crucial part of the performance of systems with a Non-Uniform Memory Access (NUMA). NUMA will be covered in part 6. Maximum memory frequency and maximum memory bandwidth are closely connected to each other (Review Part 3, Table 2 for Peak Transfer rate calculation). Max Memory Frequency Peak Transfer Rate Channels Max Memory Bandwidth 1333 MHz 10.6 GB/s 4 42.6 GB/s 1600 MHz 12.8 GB/s 4 51.2 GB/s 1866 MHz 14.9 GB/s 4 59.7 GB/s Interleaving across channels Populating the memory channels equally allows the CPU to leverage its multiple memory controllers. When all four channels are populated the CPU interleaves memory access across the multiple memory channels. This configuration has the largest impact on performance and especially on throughput. To leverage interleaving optimally, the CPU creates regions. The memory controller groups memory across the channels as much as possible. When creating a 1 DIMM per Channel configuration, the CPU creates one region (Region 0) and interleaves the memory access. Populating four channels provide the best performance, however sometimes extra capacity is required, but not as much as four channels can provide. Populate the DIMMs in groups of two. For example if 384 GB is required and 32 GB DIMMs are used, populate each CPU with 6 DIMMS. The CPU will create two Regions. The CPU will interleave access to Region 2 across 2 channels. This will decrease the throughput of region 2. Interleaving across Ranks Interleaving is continued from interleaving across the channels to interleaving across the ranks in a channel. This only occurs when using dual or quad rank DIMMs. If a channel is populated with mixed ranking DIMMS and a single rank DIMM is present, rank interleaving will revert back to 1-way interleaving. 1-way rank interleaving results in storing bits in a single DRAM chip until it’s at capacity before moving to another DRAM chip. Rank interleaving provides performance benefits as it provides the memory controller to parallelize the memory request. Typically it results in a better improvement of latency. However the performance difference between dual ranking and quad ranking is minute and comes only into play when squeezing out the very last ounce of performance. Try to avoid single rank DIMMs. Number of DIMMs per channel When adding a single DIMM to each channel in the system, performance (read throughput) scales almost linearly until all eight channels are populated (4 channels per CPU x 2 CPU). [caption id=“attachment_5020” align=“aligncenter” width=“438”] Source: HP[/caption] However when adding more DIMMs per channel, due to capacity requirement, throughput per DIMM decreases. When adding more DIMMs to the channel, the memory controller consumes more bandwidth for control commands. Basically you are increasing management overhead by adding more DIMM, reducing available bandwidth for read and write data. The creates a challenge whether the capacity can be solved by using higher capacity DIMMs or taking the throughput hit as more capacity is only obtainable by populating all slots. Populating the channels with 2 DIMMs (2 DPC) does not drastically impact throughput. The system allows to DIMMs to run in native speed. However it becomes interesting when choosing between 2 DPC and 3 DPC configurations. Vendor DIMM Type 1 DPC 2 DPC 3 DPC HP 1R RDIMM 1866 MHz 1866 MHz 1333 MHz HP 2R RDIMM 1866 MHz 1866 MHz 1333 MHz Dell 4R RDIMM 1333 MHz 1066 MHz N/A HP 4R LRDIMM 1866 MHz 1866 MHz 1333 MHz When creating a system with 384 GB of memory, each CPU has 12 slots, divided between 4 channels. Option 1 is to use 32GB LRDIMMs, populating 6 DIMM slots with a 32GB DIMM per CPU. The CPU will create two regions, region 0 interleaves across four channels, region 1 interleaves across 2 channels. Native speed remains the same. However some performance loss occurs due to some control management overhead and asymmetrical configuration. This configuration allows for future upgrade. If quad ranked RDIMMs were used, a total 128 GB ram could be added to the system, 32GB RDIMMs are quad ranked DIMMS, limiting the system to a 2 DPC configuration due to the maximum number or ranks. Quad Rank RDIMMs run at a lower clock cycle speed than LRDIMMs and reduce overal scale-up abilities due to the maximum rank limitation. Option 2 is to use 16GB RDIMMs, all channels are populated achieving maximum interleaving, however the DIMMs are not able to run in native speed anymore and will be toggled to run at a lower speed. Instead of 1866 MHz they will run at 1333 MHz. 1866 MHz provides a maximum DIMM throughput of 14.9 GB/s, 1333 MHz a maximum throughput of 10.6 GB/s a drop of nearly 30%. This drop of performance is significant, therefor it should be taken into consideration when configuring the server memory. Be cognisant of the scalability issues with ranking and the native speed drop off when moving towards a multi DPC configuration. The number of DIMM slots does not always mean that you can scale up to a certain capacity configuration. DDR3 LRDIMMs provide a great way to maximise capacity while retaining bandwidth. Begin 2014 DDR4 was released, providing higher density, better performance and decreased drop off rates when using multi DPC configurations. Up next, part 5: DDR4 Memory The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive: Memory Subsystem Bandwidth URL: https://frankdenneman.ai/2015-02-19-memory-deep-dive-memory-subsystem-bandwidth/ Date: 2015-02-19 This is part 3 of the memory deep dive. This is a series of articles that I wrote to share what I learned while documenting memory internals for large memory server configurations. This topic amongst others will be covered in the upcoming FVP book. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary Memory Subsystem Bandwidth Unfortunately, there is a downside when aiming for high memory capacity configurations and that is the loss of bandwidth. As shown in Table 1, using more physical ranks per channel lowers the clock frequency of the memory banks. As more ranks per DIMM are used the electrical loading of the memory module increases. And as more ranks are used in a memory channel, memory speed drops restricting the use of additional memory. Therefore in certain configurations, DIMMs will run slower than their listed maximum speeds. Let’s use an Intel Xeon E5 v2 (Ivy Bridge) as an example. The Intel Xeon E5 is one of the most popular CPUs used in server platforms. Although the E5 CPU supports 3 DIMMs per channel, most servers are equipped with a maximum of two DIMMs per channel as the memory speed decreases with the use of the third bank. Vendor DIMM Type 1 DPC 2 DPC 3 DPC HP 1R RDIMM 1866 MHz 1866 MHz 1333 MHz HP 2R RDIMM 1866 MHz 1866 MHz 1333 MHz Dell 4R RDIMM 1333 MHz 1066 MHz N/A HP 4R LRDIMM 1866 MHz 1866 MHz 1333 MHz Table 1: DDR3 Memory channel pairing impact on memory bandwidth Source: HP Smart Memory Dell R720 12G Memory Performance Guide Relation of bandwidth and frequency As is often the case in competitive markets in and out of technology, memory vendors use a lot of different terminology. Sometimes we see MHz indicate bandwidth, other times transfer rate per second (MT/s). Typically, the metric that resonates the most is the bandwidth per second in Megabytes. Some examples of popular designations of DDR modules: DIMM Type Memory Clock I/O Bus Clock Data Rate Module Name Peak Transfer Rate DDR3-800 100 MHz 400 MHz 800 MT/s PC-6400 6400 MB/s DDR3-1066 133 MHz 533 MHz 1066 MT/s PC-8500 8533 MB/s DDR3-1333 166 MHz 666 MHz 1333 MT/s PC-10600 10600 MB/s DDR3-1600 200 MHz 800 MHz 1600 MT/s PC-12800 12800 MB/s DDR3-1866 233 MHz 933 MHz 1866 MT/s PC-14900 14933 MB/s DDR3-2133 266 MHz 1066 MHz 2133 MT/s PC-17000 17066 MB/s DDR stands for double data rate which means data is transferred on both the rising and falling edges of the clock signal. Meaning that the transfer rate is roughly twice the speed of the I/O bus clock. For example, if the I/O bus clock runs at 800 MHz per second, then the effective rate is 1600 mega transfers per second (MT/s) because there are 800 million rising edges per second and 800 million falling edges per second of a clock signal running at 800 MHz. The transfer rate refers to the number of operations transferring data that occur in each second in the data-transfer channel. Transfer rates are generally indicated by MT/s or gigatransfers per second (GT/s). 1 MT/s is 106 or one million transfers per second; similarly, 1 GT/s means 109, or one billion transfers per second. [caption id=“attachment_4986” align=“aligncenter” width=“680”] DDR signal rate per clock cycle[/caption] Please be aware that sometimes MT/s and MHz are used interchangeably. This is not correct! As mentioned above, the MT/s is normally twice of the I/O clock rate (MHz) due to the sampling, one transfer on the rising clock edge, and one transfer on the falling. Therefore it’s more interesting to calculate the theoretical bandwidth. The transfer rate itself does not specify the bit rate at which data is being transferred. To calculate the data transmission rate, one must multiply the transfer rate by the information channel width. The formula for a data transfer rate is: Channel width (bits/transfer) × transfers/second = bits transferred/second This means that a 64-bit wide DDR3-1600 DIMM can achieve a maximum transfer rate of 12800 MB/s. To arrive at 12800 MB/s multiply the memory clock rate (200) by the bus clock multiplier (4) x data rate (2) = 1600 x number of bits transferred (64) = 102400 bits / 8 = 12800 MB/s Design considerations The most popular DDR3 frequencies are DIMMs operating 1600 MHz, 1333 MHz, and 1066 MHz. Many tests published on the net show on average 13% decline in memory bandwidth when dropping down from 1600 MHz to 1333 MHz. When using 3 DPC configuration, bandwidth drops down 29% when comparing 1066 MHz with 1600 MHz. It’s recommended to leverage LRDIMMS when spec’ing servers with large-capacity memory configurations. If you want to measure the memory bandwidth on your system, Intel released the tool Intel® VTune™ Performance Analyzer. Low Voltage Low Voltage RAM is gaining more popularity recently. DDR3 RDIMMs require 1.5 volts to operate, low voltage RDIMMS require 1.35 volts. While this doesn’t sound much, dealing with hundreds of servers each equipped with 20 or more DIMM modules this can become a tremendous power saver. Unfortunately using less power results in a lower memory clock cycle of the memory bus. This leads to reduced memory bandwidth. Table xyx shows the memory bandwidth of low voltage DDR3 DIMMs compared to 1.5V DIMM rated voltage. DIMM Type Ranking Speed 1 DPC 1.35V 1 DPC 1.5V 2 DPC 1.35V 2 DPC 1.5V 3 DPC 1.35V 3 DPC 1.5V RDIMM SR/DR 1600 MHz N/A 1600 N/A 1600 N/A 1066 RDIMM SR/DR 1333 MHz 1333 1333 1333 1333 N/A 1066 RDIMM QR 1333 MHz 800 1066 800 800 N/A N/A LRDIMM QR 1333 MHz 1333 1333 1333 1333 1066 1066 Table 2: Rated voltage and impact on memory bandwidth Low voltage RDIMMs cannot operate at the highest achievable speed as their 1.5V counterparts. Frequency fall-off is dramatic with Quad-ranked Low voltage operating at 800 MHz. ECC Memory Error Checking and Correction (ECC) memory is essential in enterprise architectures. With the increased capacity and the speed at which memory operates, memory reliability is an utmost concern. DIMM Modules equipped with ECC contain an additional DRAM chip for every eight DRAM chips storing data. The memory controller to exploits is an extra DRAM chip to record parity or use it for error-correcting code. The error-correcting code provides single-bit error correction and double-bit error detection (SEC-DED). When a single bit goes bad, ECC can correct this by using the parity to reconstruct the data. When multiple bits are generating errors, ECC memory detects this but is not capable to correct this. The trade-off for the protection of data loss is cost and performance reduction. ECC may lower memory performance by around 2–3 percent on some systems, depending on application and implementation, due to the additional time needed for ECC memory controllers to perform error checking. Please note that ECC memory cannot be used in a system containing non-ECC memory. Up next, part 4: Optimizing for Performance The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive Series URL: https://frankdenneman.ai/2015-02-18-memory-configuration-scalability-blog-series/ Date: 2015-02-18 Processor speed and core counts are important factors when designing a new server platform. However with virtualization platforms, the memory subsystem can have equal or sometimes even have a greater impact on application performance than the processor speed. During my last trip I spend a lot talking about server configurations with customers. vSphere 5.5 update 2 supports up to 6 TB and vSphere 6.0 will support up to 12TB per server. All this memory can be leveraged for Virtual Machine memory and if you run FVP, Distributed Fault Tolerant Memory. With the possibility of creating high-density memory configurations, care must be taken when spec’ing the server. The availability of DIMM slots does not automatically mean expandability. Especially when you want to expand the current memory configuration. The CPU type and generation impacts the memory configuration and when deciding on a new server spec you get a wide variety of options presented. Memory Channels, Memory bus frequency, ranking, DIMM type are just a selection of options you encounter. DIMM type, the number of DIMMs used and how the DIMMs are populated on the server board impact performance and supported maximal memory capacity. In this short series of blog posts, I attempt to provide a primer on memory tech and how it impacts scalability. Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: Memory Deep Dive: Memory Subsystem Organisation URL: https://frankdenneman.ai/2015-02-18-memory-tech-primer-memory-subsystem-organization/ Date: 2015-02-18 This is part 2 of the memory deep dive. This is a series of articles that I wrote to share what I learned while documenting memory internals for large memory server configurations. This topic amongst others will be covered in the upcoming FVP book. The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary Today’s CPU micro-architectures contain integrated memory controllers. The memory controller connects through a channel to the DIMMs. DIMM stands for Dual Inline Memory Module and contains the memory modules (DRAM chips) that provide 4 or 8 bits of data. Dual Inline refers to pins on both side of the module. Chips on the DIMM are arranged in groups called ranks that can be accessed simultaneously by the memory controller. Within a single memory cycle 64 bits of data will be accessed. These 64 bits may come from the 8 or 16 DRAM chips depending on how the DIMM is organized. An Overview of Server DIMM types There are different types of DIMMs, registered and unregistered. Unregistered DIMM (UDIMM) type is targeted towards the consumer market and systems that don’t require supporting very large amounts of memory. An UDIMM allows the memory controller address each memory chip individually and in parallel. Each memory chip places a certain amount of capacitance on the memory channel and weakens the signal. As a result, a limited number of memory chips can be used while maintaining stable and consistent performance. Servers running virtualized enterprise applications require a high concentration of memory. However with these high concentrations, the connection between the memory controller and the DRAM chips can overload, causing errors and delays in the flow of data. CPU speeds increase and therefor memory speeds have to increase as well. Consequently higher speeds of the memory bus leads to data flooding the channel faster, resulting in more errors occurring. To increase scale and robustness, a register is placed between the DRAM chips and the memory controller. This register, sometimes referred to as a buffer, isolates the control lines between the memory controller and each DRAM chip. This reduced the electrical load, allowing the memory controller to address more DRAM chips while maintaining stability. Registered DIMMs are referred to as RDIMMs. Load Reduced DIMMs (LRDIMMs) were introduced in the third generation of DDR memory (DDR3) and buffers both the control and data lines from the DRAM chips. This decreases the electrical load on the memory controller allowing for denser memory configurations. The increased memory capacity leads to increased power consumption, however by implementing the buffer structure differently it provides substantially higher operating data rates than RDIMMs in the same configuration. The key to increased capacity and performance of LRDIMMs is the abstraction of DRAM chips and especially the rank count by the buffer. RDIMMs register only buffers the command and address while leaving the more important data bus unbuffered. This leaves the group of DRAM chips (ranks) exposed to the memory controller. A memory controller accesses the grouped DRAM chips simultaneously. A Quad rank DIMM configuration presents four separate electrical loads on the data bus per DIMM. The memory controller can handle up to a certain amount of load and therefor there is a limitation on the number of exposed ranks. LRDIMMs scale to higher speeds by using rank multiplication, where multiple ranks appear to the memory controller as a logical rank of a larger size. DIMM Ranking DIMMs come in three rank configurations; single-rank, dual-rank or quad-rank configuration, ranks are denoted as (xR). Together the DRAM chips grouped into a rank contain 64-bit of data. If a DIMM contains DRAM chips on just one side of the printed circuit board (PCB), containing a single 64-bit chunk of data, it is referred to as a single-rank (1R) module. A dual rank (2R) module contains at least two 64-bit chunks of data, one chunk on each side of the PCB. Quad ranked DIMMs (4R) contains four 64-bit chunks, two chunks on each side. To increase capacity, combine the ranks with the largest DRAM chips. A quad-ranked DIMM with 4Gb chips equals 32GB DIMM (4Gb x 8bits x 4 ranks). As server boards have a finite amount of DIMM slots, quad-ranked DIMMs are the most effective way to achieve the highest memory capacity. As mentioned before there are some limitations when it comes to the amount of ranks used in a system. Memory controllers use channels to communicate with DIMM slots and each channel supports a limited amount of ranks due to maximal capacitance. Memory Channel Modern CPU microarchitectures support triple or quadruple memory channels. These multiple independent channels increases data transfer rates due to concurrent access of multiple DIMMs. When operating in triple-channel or in quad-channel mode, latency is reduced due to interleaving. The memory controller distributes the data amongst the DIMM in an alternating pattern, allowing the memory controller to access each DIMM for smaller bits of data instead of accessing a single DIMM for the entire chunk of data. This provides the memory controller more bandwidth for accessing the same amount of data across channels instead of traversing a single channel when it stores all data in one DIMM. If the CPU supports triple-channel mode, it is enabled when three identical memory modules are installed in the separate channel DIMM slots. If two of the three-channel slots are populated with identical DIMMs, then the CPU activates dual-channel mode. Quad-channel mode is activated when four identical DIMMs are put in quad-channel slots. When three matched DIMMs are used in Quad-channel CPU architectures, triple-channel is activated, when two identical DIMMs are used, the system will operate in dual-channel mode. LRDIMM rank aware controllers With the introduction of LRDIMMs, memory controllers have been enhanced to improve the utilization of the LRDIMMs memory capacity. Rank multiplication is of of these enhancements and improved latency and bandwidth tremendously. Generally memory controllers of systems prior to 2012 were “rank unaware” when operating in rank multiplication mode. Due to the onboard register on the DIMM it was unaware whether the rank was on the same DIMM it had to account for time to switch between DRAMS on the same bus. This resulted in lower back-to-back read transactions performance, sometimes up to 25% performance penalty. Many tests have been done between RDIMMs and LRDIMMs operating at the same speed. In systems with rank unaware memory controllers you can see a performance loss of 30% when comparing LRDIMMs and RDIMMS. Systems after 2012 are referred to generation 2 DDR3 platforms and contain controllers that are aware of the physical ranks behind the data buffer. Allowing the memory controller to adjust the timings and providing better back-to-back reads and writes. Gen 2 DDR3 systems reduce the latency gap between RDIMMs and LRDIMMs but most importantly it reduces the bandwidth gap. Please be aware of this difference when reading memory reviews posted on the net by independent hardware review sites. Verify the date of the publication to understand if they tested a configuration that was rank aware or rank unaware systems. DDR4 LRDIMMs improves lantencies even further due to use of distributed data buffers. DDR4 memory is covered in the third article in this series. Pairing DIMMs per Memory Channel Depending on the DIMM slot configuration of the server board, multiple DIMMs can be used per channel. If one DIMM is used per channel, this configuration is commonly referred to as 1 DIMM Per Channel (1 DPC). 2 DIMMs per channel (2 DPC) and if 3 DIMMs are used per channel, this configuration is referred to as 3 DPC. [caption id=“attachment_4949” align=“aligncenter” width=“511”] Figure 1: DPC configurations and channels[/caption] The diagram illustrates different DPC configurations; please note that balanced DIMM population (same number and type of DIMMs in each channel) is generally recommended for the best overall memory performance. The configuration displayed above is non-functional do not try to repeat. However there are some limitations to channels and ranking. To achieve more memory density, higher capacity DIMMs are required. As you move up in the size of gigabytes of memory, you are forced to move up in the ranks of memory. For example single rank and dual rank RDIMMs have a maximum capacity per DIMM of 16GB. DDR3 32GB RDIMMs are available in quad rank (QR). Recently 64GB DIMS are made available, but only in LRDIMM format. Memory rank impacts the number of DIMMS supported per channel. Modern CPUs can support up to 8 physical ranks per channel. This means that if a large amount of capacity is required quad ranked RDIMMs or LRDIMMs should be used. When using quad ranked RDIMMs, only 2 DPC configurations are possible as 3 DPC equals 12 ranks, which exceeds the 8 ranks per memory rank limit of currents systems. [caption id=“attachment_4952” align=“aligncenter” width=“556”] Maximum RDIMM configuration (256 GB per CPU)[/caption] When comparing 32GB LRDIMMs and 32GB Quad Rank RDIMMs it becomes apparent that LRDIMMS allow for higher capacity while retaining the bandwidth. For example, a Gen 12 Dell R720 contains two Intel Xeon E5 2600 CPU, allowing up to 1.5TB of RAM. The system contains 24 DIMM slots and allows up to 64GB DDR3 DIMMs up to 1866 Mhz. Dells memory configuration samples only contain configurations up to 1600 MHz. Table 1: Total capacity configuration of RDIMMs and LRDIMMs DIMM Type Capacity Number of DIMMs Ranking Total capacity Bandwidth RDIMM 16GB 16 2R 256GB 1600MHz RDIMM 32GB 16 4R 512GB 1333 MHz LRDIMM 32GB 24 4R 768GB 1333MHz LRDIMM 64GB 24 4R 1536GB 1333MHz Source: Online R720 manual Source: Sample Memory configurations Design consideration DIMM types impact future expandability of the server. Due to the maximum of ranks support per channel, care must be taken when initially designing the server spec of the server. Unfortunately, there is a downside when aiming for high memory capacity configurations and that is the loss of bandwidth. The interesting thing is when you increase DIMM count the bandwidth typically decreases. This impacts memory performance. The relationship between frequency and bandwidth is the topic of the next article in this series; Up next, part 3: Memory Subsystem Bandwidth The memory deep dive series: Part 1: Memory Deep Dive Intro Part 2: Memory subsystem Organisation Part 3: Memory Subsystem Bandwidth Part 4: Optimizing for Performance Part 5: DDR4 Memory Part 6: NUMA Architecture and Data Locality Part 7: Memory Deep Dive Summary ================================================================================ Title: New TPS management capabilities URL: https://frankdenneman.ai/2015-02-02-new-tps-management-capabilities/ Date: 2015-02-02 Recently VMware decided that it’s best to change Transparent Page Sharing (TPS) behavior. In KB 2080735 they state the following: Although VMware believes the risk of TPS being used to gather sensitive information is low, we strive to ensure that products ship with default settings that are as secure as possible. For this reason new TPS management options are being introduced and inter-Virtual Machine TPS will no longer be enabled by default in ESXi 5.5, 5.1, 5.0 Updates and the next major ESXi release. Administrators may revert to the previous behavior if they so wish. VMware reworked the TPS code and the new code is included in version: ESXi 5.5 Update 2d (Q1, 2015), ESXi 5.1 Update 3 (12/4, 2014) and ESXi 5.0 Update 3d (Q1, 2015). In the previous released patches*, new TPS management capabilities where introduced but not enabled by default. The new TPS management capabilities introduce the concept of salting has been introduced to control Intra-VM TPS. What is salting? This whole exercise of protecting TPS started when researchers found a way to determine the AES encryption key in use of virtual machines on a physical processor (grossly simplified explanation). To counter act this, VMware added salting options to harden TPS. In encryption salting is the act of adding random data to make a common password uncommon. By concatenating random data to a common password, the password now becomes uncommon, making it unlikely to show up in any common password list. This slows down the attack. Martin Suecia provided a more elaborate, but easy to understand, explanation about salting on crypto.stackexchange.com. VMware adopted this concept to group virtual machines. If they contain the same random number they are perceived to be trustworthy and can share pages. If the random number doesn’t match, no memory page sharing occurs between the virtual machines. By default the vc.uuid of the virtual machine is used as random number. And because the vc.uuid is unique randomly generated string for a virtual machine in a Virtual Center, it will never be able to share pages with other virtual machines. Lets rehash TPS, as there seems to be some misconception on how TPS works. TPS by itself is a two-tier process. Two tier process There is an act of identifying identical pages and there is an act of sharing (collapsing) identical pages. TPS cannot collapse pages immediately when starting a virtual machine. TPS is a process in the VMkernel; it runs in the background and searches for redundant pages. Default TPS will have a cycle of 60 minutes (Mem.ShareScanTime) to scan a VM for page sharing opportunities. The speed of TPS mostly depends on the load and specs of the Server. Default TPS will scan 4MB/sec per 1 GHz. (Mem.ShareScanGHz). Slow CPU equals slow TPS process. (But it’s not a secret that a slow CPU will offer less performance that a fast CPU.) TPS defaults can be altered, but it is advised to keep to the default. VMware optimized memory management in ESX 4 that allow pages which Windows initially zeroes will be page-shared by TPS immediately. Please not that this is based on best effort basis this to avoid creating massive overhead on trying to scan in-line. TPS and large pages One caveat, TPS will not collapse large pages when the ESX server is not under memory pressure. ESX will back large pages with machine memory, but installs page sharing hints. When memory pressure occurs, the large page will be broken down and TPS can do it’s magic. For more info: Future direction of disabling TPS by default and its impact on capacity planning. TPS and CPU NUMA structures Another impact on the memory sharing potential is the NUMA processor architecture. NUMA allows the best memory performance by storing memory pages as close to a CPU as possible. TPS memory sharing could reduce the performance while pages are shared between two separate CPU systems. For more info about NUMA and TPS please read the article: “Sizing VMS and NUMA nodes” Intra-VM and Inter-VM When TPS identifies a common page it will collapse it, common pages occur within the memory footprint of a virtual machine itself (Intra-VM) and between virtual machines (Inter-VM). The new setting allows for TPS to collapse page within the memory footprint of the virtual machine itself, but not between virtual machines! Be aware that Intra-VM sharing only occurs today within a NUMA node, with small pages or when large pages are torn down. TPS salting In order to Salt pages, two settings must be activated, one at the host (VMkernel) level and one at the virtual machine level. The VMkernel setting is Mem.ShareForceSalting and in the upcoming update releases it is set to “2”. Why not use the setting “1” you might ask? By reviewing the various KB articles, it seems that VMware extending the current salting options introduced in update releases: ({5.5,5.1}201410401 and 5.0 201412401) (KB: 2091682) KB article 2097593 provides us with the following table: Re-enable Intra-VM TPS That means that if you want to re-enable Intra-VM TPS you have two options. In-line with security guidelines or reverting back to traditional TPS behavior. 1: To be in-line with the security guidelines you have to set Mem.ShareForceSalting to 1 or 2 and for the virtual machines you wish to share, set sched.mem.pshare.salt to a common value. (Bottom row in the table) 2: To revert back to the traditional TPS behavior you have to set Mem.ShareForceSalting to 0. For the changes to take effect do either of the two: 1. Migrate all the virtual machines to another host in cluster and then back to original host. 2. Shutdown and power-on the virtual machines. Since its normal to place host in maintenance mode before changing its configuration, option 1 seems like the most common operation. Put a host into maintenance mode, let DRS migrate all the virtual machines to another host, change the setting and exit maintenance mode. Rinse and repeat for all hosts in the cluster. Recommendations whether to use salting? Honestly I don’t have any. Security is something that shouldn’t be taken lightly. VMware implies that this security measure is somewhat excessive. Therefor it depends on your security guidelines and your service offering (Public cloud versus own infrastructure) whether you should go through the extra length of securing TPS or not. Would I recommend enabling TPS? Of course! It’s one of the most intelligent features of the vSphere stack. Allowing you to use the available resources as efficiently as possible. By default salting is disabled (Mem.ShareForceSalting=0). This means TPS happens as it used to before this patch, that is, all the Virtual Machines on an ESXi box participate in TPS. * Previous released patches VMware ESXi 5.5, Patch ESXi550-201410401 VMware ESXi 5.1, Patch ESXi510-201410401 VMware ESXi 5.0, Patch ESXi500-201412401 ================================================================================ Title: KB 2104983 explained: Default behavior of DRS has been changed to make the feature less aggressive URL: https://frankdenneman.ai/2015-01-28-kb-2104983-explained-default-behavior-drs-changed-make-feature-less-aggressive/ Date: 2015-01-28 Yesterday a couple of tweets were in my timeline discussing DRS behavior mentioned in KB article 2104983. The article is terse at best, therefor I thought lets discuss this a little bit more in-depth. During normal behavior DRS uses an upper limit of 100% utilization in its load-balancing algorithm. It will never migrate a virtual machine to a host if that migration results in a host utilization of 100% or more. However this behavior can prolong the time to upgrade all the hosts in the cluster when using the cluster maintenance mode feature in vCenter update manager (parallel remediation). To reduce the overall remediation time, vSphere 5.5 contains an increased limit for cluster maintenance mode and uses a default setting of 150%. This can impact the performance of the virtual machine during the cluster upgrade. vCenter Server 5.5 Update 2d includes a fix that allows users to override the default and can specify the range between 40% and 200%. If no change is made to the setting, the default of 150% is used during cluster maintenance mode. Please note that normal load balancing behavior in vSphere 5.5 still uses a 100% upper limit for utilization calculation. ================================================================================ Title: My wish for 2015 - better tooling to provide better insight URL: https://frankdenneman.ai/2014-12-30-wish-2015-better-tooling-provide-better-insight/ Date: 2014-12-30 I’ve seen this image pop up quite a bit in my twitter timeline this week and it’s a very recognizable situation. Most of us have been in such conversation; I know I was when I was an Enterprise architect. And most tweets are wishing this situation changes in 2015, and I totally agree with it. However in my opinion it’s not the “app owner” who gives the wrong answer, it’s a wrong question to begin with. When I order a bread at the bakery, the baker ask what kind of bread I want, not how he needs to operate and fine tune his machinery in order to give me the product I want. Why do we think our industry, our service offering is different? In reality, can you expect from app owners to truly understand the I/O characteristics of his application? Maybe they read all the documentation of the vendor, maybe they followed a couple of courses on how to configure and operate the application, or maybe they might even got a few certifications under their belt. But in reality there are no classes and courses in the dynamics of the workload you are running. The application stack is merely a framework in order to delivery a service to the business you are servicing. The dynamics of workload is very complex, especially in a virtualized datacenter. Typically enterprise applications do not generate a consistent workload pattern. These patterns are different when servicing users or when interacting with infrastructure services. During their life cycle, applications are updated, code/query improves impacting application behaviour. Pete Koehler wrote down his experience in his article “Using a new tool to discover old problems”. Besides generating a variety of different workload patterns, applications are subject to change during its lifecycle. Change in interaction and demand, impacting the underlying infrastructure differently throughout time. Typically an application experiences a lot of interaction during test/dev/acceptance process before going into production. After the introduction period, demand is low but increasing. During the maturity of the application, demand will peak. At one point application will be replaced and is phased out. During this phase workload demand will taper off, but the service still demands a particular level of service. During all these phases, the infrastructure needs to provide the service the organization demands. And this is just an isolated case of one particular application. Typically the virtual datacenter infrastructure is shared. A virtual machine containing an application lands on a storage array, typically storing multiple virtual machines on that datastore. The datastore is backed by a LUN, backed by multiple physical devices. Access to the devices is done via shared controllers and the list continues all the way up the stack to hypervisor. Maybe the application owner understands what type of I/O the application is generally producing, but the underlying stack will impact the performance. Can you ensure the application gets the performance it requires? Do you know if the infrastructure is capable of delivering the service the application requires? And what about the impact of the application on the infrastructure. How will introducing this application impact the current active applications? Will it impact their service levels? Therefor I believe that two things need to change, behavior and tooling. IT needs to switch from asking technical questions to asking functional questions. It’s better to understand the role and place in the business process. Typically this provides insight on the availability, concurrency and response requirements of the application. The second thing that needs to change, and this is what I hope 2015 will bring, is better tooling that provides insight on workload characteristics. Tooling that provides better analysis of application demand and it’s impact on infrastructure. At this stage, most tooling is ineffective in proving proper information. Virtualized Datacenters need tooling that provides a better view into the theatre of consumers and producers. Tooling that provides a more holistic view of the application workload characteristics while being able to monitor the resource usage. Having such tools allows IT departments to operationalize and manage their environments much better, ensuring proper service levels while being able to understand the capability of the environment. Looking at the current developments in the IT industry, it is incredible difficult to predict what type of workloads (and especially in what form/platform) will hit the enterprise IT landscape in the next two years. Understanding what your environment truly delivers is a necessity when discussing future workloads. I think this is a necessary step for datacenter advancements. Once you know what’s going on, once you got proper tooling to provide better insights you can feed this data into advanced algorithms to distribute the load across the infrastructure to provide the performance it requires while optimizing resource utilization. All of this providing the correct priority aligning it with business needs. This goes beyond todays offering such as DRS, Storage DRS, SIOC in vSphere datacenters and Mesos in container landscapes. ================================================================================ Title: Interesting IT related documentaries URL: https://frankdenneman.ai/2014-12-24-interesting-related-documentaries/ Date: 2014-12-24 The holidays are upon us and for most its time to wind down. Maybe time for some nice though-provoking documentaries before the food-coma sets in. ;) Most of the documentaries listed here are created by Tegenlicht (Backlight). Backlight provides some of the best documentaries on Dutch television and luckily they made most of them available in English. The following list is a set of documentaries that impressed me. If you found some awesome documentaries yourself, please leave a comment. Tech revolution on Wall Street Backlight created a trilogy on the tech revolution on Wall Street over a period of three years. The most famous one is the second one, “Money and Speed, inside the black box”. It received multiple awards and although it’s the second documentary in the series of three, I recommend starting with that one. If you are intrigued about how the impact of these algorithms, continue with the other two episodes, “Quants. The alchemists of Wall Street” and “Wall Street Code”. They almost make you feel like you are watching a thriller, highly recommended! 08.02.2010: Quants, The alchemists of Wall Street. English | Dutch 20.03.2013: Money & Speed: Inside the Black Box. English | Dutch 04.11.2013: Wall Street Code. English | Dutch Extra video 08.02.2010: Quants, George Dyson. English 01.07.2011: Kevin Slavin: How algorithms shape our world. English (Ted Talk) Unfortunately these two documentaries are Dutch only. 21.10.2013: Big Data, the Shell Search. Dutch Tegenlicht onderzoekt hoe je met behulp van big data kunt doordringen in gesloten bolwerken. Wat geven deze enorme informatiestromen prijs over een multinational als Shell? 1.10.2014: Zero Days veiligheidslekken te koop Dutch Tegenlicht neemt je mee in de handel van ‘zero days’, onbekende lekken in software of op het internet. Een strijd tussen ‘white hat’ en ‘black hat’ hackers bepaalt onze online veiligheid. Although the voice over is Dutch, most of this documentary is in English, you might want to give it a try. It focuses on legal trade of unknown security vulnerabilities, so called zero days. Yes your government is also acquiring these from hackers, all perfectly legal! Enjoy! And of course I wish you all happy holidays! ================================================================================ Title: Playing tonight: DRS and the IO controllers URL: https://frankdenneman.ai/2014-12-01-playing-tonight-drs-io-controllers/ Date: 2014-12-01 Ever wondered why the band is always mentioned second, is the band replaceable? Is the sound of the instruments so ambiguous that you can swap out any musician with another? Apparently the front man is the headliner of the show and if he does he job well he will never be forgotten. The people who truly recognize talent are the ones that care about the musicians. They understand that the artist backing the singer create the true sound of the song. And I think this is also the case when it comes to DRS and his supporting act the Storage controllers. Namely SIOC and NETIOC. If you do it right, the combination creates the music in your virtual datacenter, well at least from a resource management perspective. ;) Last week Chris Wahl started a discussion about DRS and its inability to not load-balance perfectly the VMs amongst host. Chris knows about the fact that DRS is not a VM distribution mechanism, his argument is more focused on the distribution of load on the backend; the north-south and east-west uplinks. And for this I would recommend SIOC and NETIOC. Let’s do a 10.000 foot flyby over the different mechanisms. Distributed Resource Scheduler (DRS) DRS distributes the virtual machines – the consumers – across the ESXi hosts, the producers. Whenever the virtual machine wants to consume more resources, DRS attempts to provide these resources to this virtual machine. It can do this by moving other virtual machines to different hosts, or move the virtual machine to another host. Trying to create an environment where the consumers can consume as much as possible. As workload patterns differ from time to time, from day to day, an equal number of VMs per host does not provide a balanced resource offering. It’s best to create a combination of idle and active virtual machines per host. And now think about the size of virtual machines, most environments do not have a virtual machine configuration landscape to utilizes a identical hardware configuration. And if that was the case, think about the applications, Some are memory bound, some applications are CPU bound. And to make it worse, think load correlation and load synchronicity. Load correlation defines the relationship between loads running in different machines. If an event initiates multiple loads, for example, a search query on front-end webserver resulting in commands in the supporting stack and backend. Load synchronicity is often caused by load correlation but can also exist due to user activity. It’s very common to see spikes in workload at specific hours, for example think about log-on activity in the morning. And for every action, there is an equal and opposite re-action, quite often load correlation and load synchronicity will introduce periods of collective non-or low utilization, which reduce the displayed resource utilization. All these things, all this coordination is done by DRS, fixating an identical number of VMs per host is in my opinion lobotomizing DRS. But DRS is only focused on CPU and Memory. Arguably you can treat network and storage somewhat CPU consumption as well, but lets not go that deep. Some applications are storage bound some applications are network bound. For this other components are available in your vSphere infrastructure. The forgotten heroes, SIOC and NETIOC. Storage IO Control (SIOC) Storage I/O Control (SIOC) provides a method to fairly distribute storage I/O resources during times of contention. SIOC provides a datastore-wide scheduling using virtual disk shares to calculate priority. In a healthy and properly designed environment, every host that is part of the cluster should have a connection to the datastore and all host should have an equal amount of paths to the datastore. SIOC monitors the consumption and if the latency experienced by the virtual machine exceeds the user-defined threshold, SIOC distributes priority amongst the virtual machines hitting that datastore. By default every virtual machine receives the same priority per VMDK per datastore, but this can be modified if the application requires this from a service level perspective. Network I/O Control (NETIOC) The east-west equivalent of its north-south brother SIOC. NETIOC provides control for predictable networking performance while different network traffic streams are contending for the same bandwidth. Similar controls are offered, but are now done on traffic patterns instead of a per virtual machine basis. Similar architecture design hygiene applies here as well. All hosts across the cluster should have the same connection configuration and amount of bandwidth available to them. The article “A primer on Network I/O Control” provides more info on how NETIOC works, VMware published a NETIOC Best Practice white paper a while ago, but most of it is still accurate. And the bass guitar player of the virtual datacenter, Storage DRS. Storage DRS provides virtual machine disk placement and load balancing mechanisms based on both space and I/O capacity. Where SIOC reactively throttles hosts and virtual machines to ensure fairness, SDRS proactively generates recommendations to prevent imbalances from both space utilization and latency perspectives. More simply, Storage DRS does for storage what DRS does for compute resources. These mechanism combined with a healthy – well architected – environment will help you distribute the consumers across the producers with the proper context in mind. Which virtual machines are hot and which are not? Much better than playing the numbers game! Now, one might argue but what about failure scenarios? If a have an equal number of VMs running on my host, my failover time decreases as well. Well it depends. HA distributes virtual machines across the cluster and if DRS is up and running, it moves virtual machines around if it cannot satisfy the resource entitlement of the virtual machines (VM level reservations). Duncan wrote about DRS and HA behavior a while ago, and of course we touched upon this in our book the 5.1 clustering deepdive. (still fully applicable for 5.5 environments) In my opinion, trying to outsmart advanced and adaptive computer algorithms with basic math reasoning is really weird. Especially when most people are talking about Software defined datacenters and whether you are managing pets versus cattle. When your environment is healthy and layed-out in a homogenous way , you cannot beat computer algorithms. The thing you should focus on is the alignment of resource priority to business service levels. And that’s what you achieve by applying the correct share levels at DRS, SIOC and NETIOC levels. Maybe you can devops your way into leveraging various scripting languages. ;) ================================================================================ Title: VCDX- You cannot abstract your way out of things indefinitely URL: https://frankdenneman.ai/2014-11-11-vcdx-abstract-way-things-indefinitely/ Date: 2014-11-11 The amount of abstraction in IT is amazing. Every level in the software and hardware stack attempts to abstract operations and details. And the industry is craving for more. Look at the impact “All Things Software Defined” has on todays datacenter. It touches almost every aspect, from design to operations. The user provides the bare minimum of inputs and the underlying structure automagically tunes itself to a working solution. Brilliant! However sometimes I get the feeling that this level of abstraction becomes an excuse to not understand the underlying technology. As an architect you need to do your due diligence. You need to understand the wheels and cogs that are turning when dialing a specific knob at the abstracted layer. But sometimes it seems that the abstraction level becomes the right to refuse to answer questions. This was always an interesting discussion during a VCDX defense session. When candidates argued that they weren’t aware of the details because other groups were responsible for that design. I tend to disagree What level of abstraction is sufficient? I am in the lucky position to work with PernixData R&D engineers and before that VMware R&D engineers. They tend to go deep, right down to the core of things. Discussing every little step of a process. Is this the necessary level of understanding the applied technology and solutions for an architect? I don’t think so. It’s interesting to know, but on a day-to-day basis you don’t have to understand the function of ceiling when DRS calculates priority levels of recommendations. What is interesting is to understand what happens if you place a virtual machine at the same hierarchical level as a resource pool filled with virtual machines. What is the impact on the service levels of these various entities? Something in the middle might be the NFS series of Josh Odgers. Josh goes in-depth about the technology involved using NFS datastores. Virtual SCSI Hard Drives are presented to virtual machines, even when ESXi is connected to an NFS datastore. How does this impact the integrity of I/O’s? How does the SCSI protocol emulation process affect write ordering and of I/O’s of business critical applications. You as the virtual datacenter architect should be able to discuss the impact of using this technology with application owners. You should understand the potential impact a selected technology has on the various levels throughout the stack and what impact it has on the service it provides. Recently I published a series on databases and what impact their workload characteristics have on storage architecture design. Understanding the position of a solution in the business process allows an architect to design a suitable solution. Lets use the OLTP example. Typically OLTP databases are at the front of the process, customer-facing process, dramatically put they are in the line of fire. When the OLTP database is performing slow or is unavailable it will typically impact revenue-generating processes. This means that latency is a priority but also concurrency and availability. You can then tailor your design to provide the best services to this application. This is just a simplified example, but it shows that you have to understand multiple aspects of the technology. Not just the behavior of a single component. The idea is to get a holistic view and then design your environment to cater the needs of the business, cause that’s why we get hired. Circling back to the abstraction and the power of software defined, I though the post from Bart Heungens was interesting. Bart argues that Software Defined Storage is not the panacea for all storage related challenges. Which is true. Bart illustrates an architecture that is comprised of heterogeneous components. In his example, he illustrates what happens when you combine two servers HP DL380, but from different generations. Different generations primarily noticeable from a storage controller perspective and especially the way software behave. This is interesting on so many levels, and it would be a very interesting discussion if this were a VCDX defense session. SDS abstracts many things, but it still relies on the underlying structure to provide the services. From a VCDX defense perspective, Bart has a constraint. And that is the already available hardware and the requirement to use these different generation hardware in his design. VCDX is not about providing the ideal design, but showing how you deal with constrains, requirements and demonstrating your expertise on technology how it impacts the requested solution. He didn’t solve the problem entirely, but by digging in deeper he managed to squeeze out performance to provide a better architecture to service the customers applications. He states the following: Conclusion: the design and the components of the solution is very important to make this SDS based SAN a success. I hear often companies and people telling that hardware is more and more commodity and so not important in the Software Defined Datacenter, well I am not convinced at all. I like the idea of VMware that states that, to enable VSAN, you need and SAS and SSD storage (HCL is quite restricted), just to be sure that they can guarantee performance. The HP VSA however is much more open and has lower requirements, however do not start complaining that your SAN is slow. Because you should understand this is not the fault of the VSA but from your hardware. So be cognizant about the fact that while you are not responsible for every decision being made when creating an architecture for a virtual datacenter, you should be able to understand the impact various components, software settings and business requirements have on your part of the design. We are moving faster and faster towards abstracting everything. However this abstraction process does not exonerate you from understanding the potential impact it has on your area of responsibility ================================================================================ Title: MS Word style formatting shortcut keys for Mac URL: https://frankdenneman.ai/2014-10-27-ms-word-style-formatting-shortcut-keys-mac/ Date: 2014-10-27 Recently I started to spend a lot of time in MS word again, and as a stickler for details I dislike a mishmash of font types throughout my document. I spend a lot of time on configuring the styles of the document, yet when I paste something from other documents, MS word tend to ignore these. Correcting the format burns a lot of time and it simply annoys the crap out of me. To avoid this further, I started to dig around to find some font and style related shortcut keys. Yesterday I tweeted the shortcut key to apply the normal style and by the looks of retweets many of you are facing the same challenge. Below is a short list of shortcut keys that I use. There are many more, share the common ones you like to use. As I use Mac I listed the Mac shortcut combination. Replace CTRL for CMD if you are using MS Word on a windows machine. Select text: Select all: CTRL+A Select sentence: CMD + click Select word: Double click Select paragraph: Triple click Formatting: Clear formatting: CTRL+spacebar Apply Normal Style: Shift+CMD+N Header 1: CMD+ALT+1 Header 2: CMD+ALT+2 Header 3: CMD+ALT+3 Change Case: CMD+Option+C (repeat combination to cycle through options) Indent paragraph: CTRL+Shift+M Remove indent: CMD+Shift+M Find and replace: F5 ================================================================================ Title: 99 cents Promo to celebrate a major milestone of the vSphere Clustering Deepdive series URL: https://frankdenneman.ai/2014-10-09-upcoming-vmware-vsphere-5-1-clustering-deepdive-promo/ Date: 2014-10-09 This week Duncan was looking at the sales numbers of the vSphere Clustering Deep Dive series and he noticed that we hit a major milestone in September. In September 2014 we passed the 45000 copies distributed of the vSphere Clustering Deep Dive. Duncan and I never ever expected this or even dared to dream to hit this milestone. When we first started writing the 4.1 book we had discussions around what to expect from a sales point of view and we placed a bet, I was happy if we sold 100 books, Duncan was more ambitious with 400 books. Needless to say we reset our expectations many times since then… We didn’t really follow it closely in the last 12-18 months, and as today we were discussing a potentially update of the book we figured it was time to look at the numbers again just to get an idea. 45000 copies distributed (ebook + printed) is just remarkable. We’ve noticed that the ebook is still very popular, and decided to do a promo. As of Monday the 13th of October the 5.1 e-book will be available for only $ 0.99 for 72 hours, then after 72 hours the price will go up to $ 3.99 and then after 72 hours it will be back to the normal price. So make sure to get it while it is low priced! Pick it up here on Amazon.com! The only other kindle store we could open the promotion up for was amazon.co.uk, so that is also an option! ================================================================================ Title: Database workload characteristics and their impact on storage architecture design – part 3 - Ancillary structures for tuning databases URL: https://frankdenneman.ai/2014-09-29-database-workload-characteristics-impact-storage-architecture-design-part-3-ancillary-structures-tuning-databases/ Date: 2014-09-29 Welcome to part 3 of the Database workload characteristics series. Databases are considered to be one of the biggest I/O consumers in the virtual infrastructure. Database operations and database design are a study upon themselves, but I thought it might be interested to take a small peak underneath the surface of database design land. I turned to our resident Database expert Bala Narasimhan, PernixData’s director of products to provide some insights about the database designs and their I/O preferences. Previous instalments of the series: Part 1 - Database Structures Part 2 – Data pipelines Question 3: You’ve talked about ancillary structures for tuning databases, what are they and what role does FVP play here? It goes without saying that database performance, whether OLTP or data warehousing, is critical for the business. As a result, DBA use ancillary structures to enhance database performance. Examples of such ancillary structures include indexes and Materialized Views (MV). MV are called Indexed Views on SQL Server. Index An index is an ancillary structure that allows a table to be sorted in multiple ways. This helps with quick lookups and operations, such as Merge Joins, that require that the data be sorted. Imagine a table with many columns in it. This table can be sorted in only one way on disk. For example, consider the Customer table shown below CREATE TABLE Customer ( CustID int, Name Char(20), Zipcode int, PRIMARY KEY (CustID)); The customer ID column, CustID, is the primary key in this table. This means that all customers can be uniquely identified by their CustID value. The table will most probably be sorted on this column. Now imagine you ran a query that wanted to find out the number of customers in ZIP code 95051. Since the table is not sorted on ZIP code you will need to scan every single row in the table, see whether its ZIP code value is 95051 and add up the number of such rows. This can be extremely expensive. Instead what you can do is build an index on the ZIP code column. This index will be sorted on ZIP code and you can do a potentially faster lookup because of this. Materialized View A Materialized View (MV) is different from an index because an MV is a database object that contains the results of a query. If you know that a query will be run repeatedly then you can simply cache the results of that query in an MV and return the results as opposed to running the query itself each time. Example syntax to create an MV is as follows: CREATE MATERIALIZED VIEW FOO AS SELECT * FROM BAZ WHERE BAZ.id = ‘11’; In the SQL statement above the materialized view FOO stores the results of the query ‘SELECT * FROM BAZ WHERE BAZ.id = 11’. So, when someone executes the query ‘SELECT * FROM BAZ WHERE BAZ.id = 11’ you can simply return the rows in FOO instead because the results of the query are already saved in FOO. Now, this example is very simplistic but you can imagine that the query can be arbitrarily complex and storing its results in an MV can therefore be hugely beneficial. Based on this explanation, one thing is apparent about both indexes and MV. Both indexes and MV are not ephemeral structures. This means that both of them need to be persisted on disk just like the tables in a database are. This means they consume disk space but more importantly it means that accessing them requires one to potentially do a lot of I/O. Good storage performance is therefore key to making these ancillary structures do their job. These ancillary structures also come with a number of limitations. Firstly, they consume a lot of disk space. Sometimes they consume as much space as the underlying table and so it becomes more of an overhead than a benefit. Secondly, especially in the case of the MV, refresh rates make a huge difference. What do I mean by this? Consider my example MV above. Let’s say that everyday I load new rows into the table BAZ and some of these rows have the value ‘11’ for the column id. In other words, there are new rows being added to BAZ every day where BAZ.id = 11. Once these new rows are added the MV foo has become stale because it is no longer storing the correct rows anymore. So, each time a new row is inserted into BAZ where BAZ.id = 11 not only must we do a write into the base table BAZ but we must also refresh the MV foo that sits on it. One I/O therefore ends up becoming multiple I/O! And, now if someone tries to query the MV foo when it is being refreshed you have all sorts of storage performance problems. Note that both of these ancillary structures are great if you know what queries you are going to run. If so, you can both create the required indexes and MV. If, however, you run a query that cannot leverage these structures you get severe performance problems. And the truth of the matter is that it is seldom that case that you know all the queries you will run up front. So, ancillary structures can take you only so far sometimes. How can FVP help? When using server-side resources such as flash and RAM not only will writes to the underlying base table go faster in Write Back mode, but, refreshes on the MV sitting on top of those base tables will go much faster too. This means better query performance, higher concurrency and better scalability. FVP will allow you to run ad-hoc queries at high speed. Even if you cannot leverage the existing indexes or MV for your query, accesses to the underlying base tables will be much faster owing to the fact that FVP will use server side flash and RAM for those base tables. The above point means you do not need to create as many indexes or MV as you used to. This results in both tremendous savings from a storage capacity perspective and from an operational perspective of managing and running the database. Part 4 coming soon! ================================================================================ Title: Database workload characteristics and their impact on storage architecture design – part 2 - Data pipelines URL: https://frankdenneman.ai/2014-09-24-database-workload-characteristics-impact-storage-architecture-design-part-2-data-pipelines/ Date: 2014-09-24 Welcome to part 2 of the Database workload characteristics series. Databases are considered to be one of the biggest I/O consumers in the virtual infrastructure. Database operations and database design are a study upon themselves, but I thought it might be interested to take a small peak underneath the surface of database design land. I turned to our resident Database expert Bala Narasimhan, PernixData’s director of products to provide some insights about the database designs and their I/O preferences. Question 2: You mentioned data pipelines in your previous podcast, what do you mean by this? What I meant by data pipeline is the process by which this data flows in the enterprise. Data is not a static entity in the enterprise; it flows through the enterprise continuously and at various points is used for different things. As mentioned in part 1 of this series, data usually enters the pipeline via OLTP databases and this can be from numerous sources. For example, retailers may have Point of Sale (POS) databases that record all transactions (purchases, returns etc.). Similarly, manufacturers may have sensors that are continuously sending data about health of the machines to an OLTP database. It is very important that this data enter the system as fast as possible. In addition, these databases must be highly available, have support for high concurrency and have consistent performance. Low latency transactions are the name of the game in this part of the pipeline. At some point, the business may be interested in analyzing this data to make better decisions. For example, a product manager at the retailer may want to analyze the Point of Sale data to better understand what products are selling at each store and why. In order to do this, he will need to run reports and analytics on the data. But as we discussed earlier, these reports and analytics are usually throughput bound and ad-hoc in nature. If we run these reports and analytics on the same OLTP database that is ingesting the low latency Point of Sale transactions then this will impact the performance of the OLTP database. Since OLTP databases are usually customer facing and interactive, a performance impact can have severe negative outcomes for the business. As a result what enterprises usually do is Extract the data from the OLTP database, Transform the data into a new shape and Load it into another database, usually a data warehouse. This is known as the ETL process. In order to do the ETL, customers use a solution such as Informatica (ETL) (3) or Hadoop (4) between your OLTP database and data warehouse. Some times customers will simply suck in all the data of the OLTP database (Read intensive, larger block size, throughput sensitive query) and than do the ETL inside the data warehouse itself. Transforming the data into a different shape requires reading the data, modifying it, and writing the data into new tables. You’ve most probably heard of nightly loads that happen into the data warehouse. This process is what is being referring to! As we discussed before, OLTP databases may have a normalized schema and the data warehouse may have a more denormalized schema such as a Star schema. As a result, you can’t simply do a nightly load of the data directly from the OLTP database into the data warehouse as is. Instead you have to Extract the data from the OLTP database, Transform it from a normalized schema to a Star schema and then Load it into the data warehouse. This is the data pipeline. Here is an image that explains this: In addition, there can also be continuous small feeds of data into the data warehouse by trickle loading small subsets of data, such as most recent or freshest data. By using the freshest data in your data warehouse you make sure that the reports you run or the analytics you do is not stale and is up to date and therefore enables the most accurate decisions. As mentioned earlier, the ETL process and the data warehouse are typically throughput bound. Server side flash and RAM can play a huge role here because the ETL process and the data warehouse can now leverage the throughput capabilities of these server side resources. Using PernixData FVP Some specific, key benefits of using FVP with the data pipeline include: OLTP databases can leverage the low latency characteristics of server side flash & RAM. This means more transactions per second and higher levels of concurrency all while providing protection against data loss via FVP’s write back replication capabilities. Trickle loads of data into the data warehouse will get tremendously faster in Write Back mode because the new rows will be added to the table as soon as it touches the server side flash or RAM. The reports and analytics may execute joins, aggregations, sorts etc. These require rapid access to large volumes of data and can also generate large intermediate results. High read and write throughput are therefore beneficial and having this done on the server right next to the database will help performance tremendously. Again, Write Back is a huge win. Analytics can be ad-hoc and any tuning that the DBA have done may not help. Having the base tables on flash via FVP can help performance tremendously for ad-hoc queries. Analytics workloads tend to create and leverage temporary tables within the database. Using server side resources for read enhances performance on these temporary tables and write accesses to them. In addition, there is also a huge operational benefit. We can now virtualize the entire data pipeline (OLTP databases, ETL, data warehouse, data marts etc.) because we are able to provide high performance and consistent performance via server side resources and FVP. This brings together the best of both workloads. Leverage the operational benefits of a virtualization platform, such as vSphere HA, DRS and vMotion, and standardize the entire data pipeline on it without sacrificing performance at all. Other parts of this series: Part 1 - Database Structures Part 3 – Ancillary structures for tuning databases ================================================================================ Title: Database workload characteristics and their impact on storage architecture design - part 1 URL: https://frankdenneman.ai/2014-09-23-database-workload-characteristics-impact-storage-architecture-design-part-1/ Date: 2014-09-23 Frequently PernixData FVP is used to accelerate databases. Databases are for many a black box solution. Sure we all know they consume resources like there is no tomorrow, but can we make some general statements about database resource consumption from a storage technology perspective? I asked Bala Narasimhan, our director of Products, a couple of questions to get a better understanding about the database operations and how FVP can help to provide the performance the business needs. The reason why I asked Bala about databases is because of his rich background in database technology. After spending some time at HP writing kernel memory management software, he moved to Oracle and was responsible for memory SGA and PGA. One of his proudest achievements was to build the automatic memory management in 10G. He then went on and worked at a startup where he rewrote the open source database, Postgres, to be a scale out, columnar relational databases for data warehousing and analytics. Bala recently recorded a webinar eliminate performance bottlenecks in virtualized Databases. Bala’s twitter account can be found here. As the topic databases is an extensive one, the article is split up into a series of smaller articles, making it more digestible. Question 1: What are the various databases use cases one typically sees? There is a spectrum of use cases, with OLTP, Reporting, OLAP and analytics being the common ones. Reporting, OLAP (online analytical processing) and Analytics can be seen as a part of the data warehousing family. OLTP (online transaction processing) databases are typically aligned with a single application and acts as an input source for data warehouses. Therefore a data warehouse can be seen as a layer on top of the OLTP database optimized for reporting and analytics. When you deal with setting up architectures for databases you have to ask yourself, what do you try to solve? What is technical requirement of the workload? Is it latency intensive, do you retrieve or do you want to read a lot of data as fast as possible? Is the application latency sensitive or throughput bound? Meaning that if you go from left to right in the table on average the block size grows. Hint: the larger the block size means that on average you are dealing with a more throughput bound workload instead of a latency sensitive block size. From left to right the database design go from normalized to denormalized. OLTP Reporting OLAP Analytics Database Schema Design OLTP is an excellent example of a normalized schema. A database schema can be seen as a container objects and allows to logically group objects such as tables, views and stored procedures. When using a normalized schema you start to split a table into smaller tables. For example, lets assume a bank database has only one table that logs all activities by all its customers. This means that there are multiple rows in this table for each customer. Now if a customer updates her address you need to update many rows in the database for the database to be consistent. This can have a impact on the performance and concurrency of the database. Instead of this, you could build out a schema for the database such that there are multiple tables and there is only one table that has customer details in it. This way when the customer changes her address you only need to update one row in this table and this improves concurrency and performance If you normalize your database enough every insert, delete and update statement will only hit a single table, very small updates that require fast responds, therefor small blocks, very latency sensitive. While OLTP databases tend to be normalized, data warehouses tend to be denormalized and therefore have lesser number of tables. For example, when querying the DB to find out who owns account 1234, it needs to join two tables, the Account-table with the Customer-table. In this example it is a two way join but it is possible for data warehousing systems to do many way joins (that is, joining multiple tables at once) and these are generally throughput bound. Business Processes An interesting way to look at the databases is its place in a business process. This provides you insight about the availability, concurrency and response requirements of the database. Typically OLTP databases are at the front of the process, customer-facing process, dramatically put they are in the line of fire. You want to have fast response, you want to read, insert and update data as fast as possible therefore the database are heavily normalized for reasons described above. When the OLTP database is performing slow or is unavailable it will typically impact revenue-generating processes. Data warehousing operations generally occur away from customer facing operations. Data is typically loaded into the data warehouse from multiple sources to provide the business insights into its day-to-day operations. For example, a business may want to understand from its data how it can drive quality and cost improvements. While we talk about a data warehouse as a single entity this is seldom the case. Many times you will find that a business has one large data warehouse and many so called ‘data marts’ that hang from it. Database proliferation is a real problem in the enterprise and managing all these databases and providing them the storage performance they need can be challenging. Let’s dive into the four database types to understand their requirements and the impact on architecture design: OLTP OLTP workloads have a good mix of read and write operations. It is latency sensitive, and it requires the support for high levels of concurrency. When talking about concurrency a good example are ATM machines. Each customer at an ATM machine is generating a connection doing a few simple instructions, however a bank typically has a lot of ATM machines servicing its many customers concurrently. If a customer wants to withdraw money, the process needs to read the records of the customer in the database. It needs to confirm that he or she is allowed to withdraw the money, and then it needs to record (write) the transaction. In DBA jargon that is a SQL SELECT statement followed by an UPDATE statement. A proper OLTP database should be able to handle a lot of users at the same time preferably with a low latency. It’s interactive in nature, meaning that latency impacts user experience. You cannot keep the customer waiting for a long time at the ATM machine or a bank teller. From an availability perspective you cannot afford to have the database go down, the connections cannot be lost, it just needs to be up and running all the time (24x7). OLTP Reporting OLAP Analytics Availability +++ Concurrency +++ Latency sensitivity +++ Throughput oriented + Ad hoc + I/O Operations Mix R/W Reporting Reporting databases experience predominately read intensive operations and requires more throughput than anything else. Concurrency and availability are not as important for reporting databases as they are for OLTP. Characteristically workload is repeated read of data. Reporting is usually done when the users want to understand the performance of the business, for example how many accounts were opened this week, how many accounts were closed, is the private banking account team hitting it’s quota of acquiring new customers? Think of reporting as predictable requests, the user knows what data he wants to see and has a specific report design that structures the data in order needs to understand these numbers. This means, this report is repetitive which allow the DBA to design and optimize database and schema so that this query gets executed predictable and efficiently. Database design can be optimized for this report. Typical database schema designs for reporting include the Star Schema and the Snow Flake Schema. As it serves the back office processes, availability and concurrency are not a strict requirement of this kind of database. As long as the database is available when the report is required. Enhanced throughput helps tremendously. OLTP Reporting OLAP Analytics Availability +++ + Concurrency +++ + Latency sensitivity +++ + Throughput oriented + +++ Ad hoc + + I/O Operations Mix R/W Read Intensive OLAP OLAP can be seen as the analytical counterpart of OLTP. Where OLTP is the original source of data, OLAP is the consolidation of data, typically originating from various OLTP databases. A common remark made in database world is that OLAP provides a multi-dimension view, meaning that you drill down the data coming from various sources and then analyze the data amongst different attributes. This workload is more ad-hoc in nature then reporting as you slice and dice the data in different ways depending on the nature of the query. The workload is primarily read intensive and can run complex queries involving aggregations of multiple databases, therefore its throughput oriented. An example of an OLAP query would be the amount of additional insurance services gold credit card customers were signing up for during the summer months. OLTP Reporting OLAP Analytics Availability +++ + + Concurrency +++ + + Latency sensitivity +++ + ++ Throughput oriented + +++ +++ Ad hoc + + ++ I/O Operations Mix R/W Read Intensive Read Intensive Analytics Analytical workload is truly ad-hoc in nature. Whereas reporting aims to provide perspective of the numbers that are being presented, analytics provide insights in why the numbers are what they are. Reporting provides the how many new accounts where acquired by the private banking account team, analytics aims to provide insights why the private banking account team did not hit their quota in the last quarter. Analytics can query multiple databases and can be multi-step processes. Typically analytic queries write out large temporary results. Potentially it generates large intermediate results before slicing and dicing the temp data again. This means this data needs to be stored as fast as possible, the data is read again for the next query therefor read performance is crucial as well. Output is the input of the next query and this can happen multiple times, requiring both fast read and write performance otherwise your query will slow down dramatically. Another problem is the sort process, for example you are retrieving data that needs to be sorted however the dataset is so large that you can’t hold everything in memory during the sort process resulting in spilling data to disk. Because analytics queries can be truly ad-hoc in nature it is difficult to design an effecient schema for it upfront. This makes analytics an especially difficult use case from a performance perspective. OLTP Reporting OLAP Analytics Availability +++ + + + Concurrency +++ + + + Latency sensitivity +++ + ++ +++ Throughput oriented + +++ +++ +++ Ad hoc + + ++ +++ I/O Operations Mix R/W Read Intensive Read Intensive Mix R/W Designing and testing your storage architecture in line with DB-workload By having a better grasp of the storage performance requirements of each specific database you can now design your environment to suits its need. Understanding these requirements helps you to test the infrastructure more focused on the expected workload. Instead of running “your average db workload" in Iometer this allows you to test more towards latency or throughput oriented workloads when understanding what type of database will be used. The next article of this series dives into understanding whether tuning databases or storage architectures can solve performance. Other parts of this series Part 2 – Data pipelines Part 3 – Ancillary structures for tuning databases ================================================================================ Title: Improve public speaking by reading a book? URL: https://frankdenneman.ai/2014-09-08-improve-public-speaking-reading-book/ Date: 2014-09-08 Although it sounds like an oxymoron I do have the feeling that books about this topic can help you become a better public speaker, or in a matter of fact more skillful in driving home your message. After our talk at VMworld a lot of friends complimented not only on the talk itself but also on the improvements I’ve made when it comes to public speaking. My first public speaking engagement was VMworld 2010 at Vegas, 8 o’clock Monday morning for 1200 people. Talk about a challenge! Since then I have been slowly improving my skills. Last year I’ve done more talks than the previous 3 years before combined. Although Malcolm Gladwell’s 10.000 –hour rule is heavily debated nowadays, I do believe that practice is by far the best way to improve your skill. By itself getting 10.000 hours of public speaking time is rather a challenge and just going through the motions alone will be very inefficient. To maximize efficiency I started to dive into the theory behind public speaking or even more broadly theory about communicating. Over the year I read a decent stack of books but these four stood out the most. 1: Confessions of a public speaker by Scott Berkun Funny and highly practical. If you want to buy only one book, this one should be it. The book helps you with the act of public speaking; How to deal with stage fright, how to work a tough room, what are the things I need to take care of to make my talk go smoothly. 2: Made to Stick by Chip and Dan Heath This book helps you structure the message you want to convey. It helps you to dive into the core of your message and communicate them in a memorable way. It’s a great book to read, lots of interesting stories and it’s one of those books that you should read multiple times to keep on refining your skillset. 3: Talk like Ted by Carmine Gallo To some extent a combination of the two first books. The interesting part is the focus on the listener experience and its capability to focus for 18 minutes. In addition, it gives you insights into some of the greatest TED talks. 4 Pitch Perfect by Bill and Alisa Bowman This book helps you to enhance your communication skills. It dives deeper into the act’s verbal and non-verbal language. It helps you to become cognizant of some of the mistakes everyone makes, yet can be avoided quite easily. The book helps you to drive your point in a more confident, persuasive, and certain manner. The beauty of these books is that you can use them, learn from them even if you are not a public speaker. In everyday life we all need to communicate, we all want our idea to be heard and possibly get a buy-in from others. I believe these books will help you achieve this. If you have found other books useful and interesting please leave a comment. ================================================================================ Title: Virtual machines versus Containers who will win? URL: https://frankdenneman.ai/2014-08-21-virtual-machines-versus-containers-will-win/ Date: 2014-08-21 Ah round X in the battle between who will win, which technology will prevail and when will the displacement of technology happen. Can we stop with this nonsense, with this everlasting tug-of-war mimicking the characteristics of a schoolyard battle. And I can’t wait to hear these conversations at VMworld. In reality there aren’t that many technologies that completely displaced a prevailing technology. We all remember the birth of the CD and the message of revolutionising music carriers. And in a large way it did, yet still there are many people who prefer to listen to vinyl. Experience the subtle sounds of the medium, giving it more warmth and character. The only solution I can think of that displaced the dominant technology was video disc (DVD & Blue Ray) rendering video tape completely obsolete (VHS/Betamax). There isn’t anybody (well let’s only use the subset Sane people) that prefers a good old VHS tape above a Blue ray tape. The dialog of “Nah let’s leave the blue-ray for what it is, and pop in the VHS tape, cause I like to have that blocky grainy experience" will not happen very often I expect. So in reality most technologies coexist in life. Fast forward to today. Dockers’ popularity put Linux Containers on the map for the majority of the IT population. A lot of people are talking about it and see the merits of leveraging a container instead of using a virtual machine. To me the choice seems to stem from the layer you present and manage your services. If your application is designed to provide high availability and scalability, then a container may be the best fit. If your application doesn’t than place it in a virtual machine and leverage the services provided by the virtual infrastructure. Sure there are many other requirements and constraints to incorporate in your decision tree, but I believe the service availability argument should be one of the first steps. Now the next step is, where do you want to run your container environment? If you are a VMware shop, are you going to invest time and money to expand your IT services with containers or are you going to leverage an online PAAS provider? Introducing an APPS centric solution into an organization that has years of experience in managing Infrastructure centric platforms might require a shift of perspective Just my two cents. ================================================================================ Title: Disable vMotion for a single VM URL: https://frankdenneman.ai/2014-08-18-disable-vmotion-single-vm/ Date: 2014-08-18 This question pops up regularly on the VMTN forums and reddit. It’s a viable question but the admins who request this feature usually don’t want Maintenance mode to break or any other feature that helps them to manage large scale environments. When you drill down, you discover that they only want to limit the option of a manual vMotion triggered by an administrator. Instead of configuring complex DRS rules, connect the VM to an unique portgroup or use bus sharing configurations, you just have to add an extra permission to the VM. The key is all about context and permission structures. When executing Maintenance mode the move of a virtual machine is done under a different context (System) then when the VM is manually migrated by the administrator. As vCenter honors the most restrictive rule you can still execute a Maintenance mode operation of a host, while being unable to migrate a specific VM. Here is how you disable vMotion for a single VM via the Webclient: Step 1: Add another Role let’s call it No-vMotion Log in as a vCenter administrator Go to the home screen Select Roles in the Administration screen Select Create Role Action (Green plus icon) Add Role name (No-vMotion) Select All Priveleges Scroll down to Resource Deselect the following Privileges: Migrate powered off virtual machine Migrate powered on virtual machine Query vMotion Step 2: Restrict User privilege on VM. Select “Host and Clusters” or “VMs and Templates” view, the one you feel comfortable with. Select the VM and click on the Manage tab Select Permissions Click on “Add Permissions” (Green plus icon) Click on Add and select the User or Group who you want to restrict. In my example I selected the user FrankD and clicks on Add on OK On the right side of the screen in the pulldown menu select the role “No-vMotion" and click on OK. Ensure that the role is applied to This object. FrankD is a member of the vCenterAdmins group which has Administrator privileges propagated through the virtual datacenter and all its children. However FrankD has an additional role on this object “No-vMotion”. Let’s check if it works. Log in with the user id you restricted and right-click the VM. As shown, the option Migrate is greyed out. The VM is running on Host ESX01 The option Mainentance Mode is still valid for Host ESX01. Click on the option “More Tasks” in the Recent Task window, here you can verify that FrankD is the initiator of the operation Maintenance mode, and System migrated the virtual machine. ================================================================================ Title: Platform 9 - transform your virtual infrastructure into a private cloud within seconds URL: https://frankdenneman.ai/2014-08-12-platform-9-transform-virtual-infrastructure-private-cloud-within-seconds/ Date: 2014-08-12 Recently I had the joy of reconnecting with some of my old VMware colleagues to learn that their new startup was coming out of stealth. Today Platform 9 announced their SaaS platform. In short, Platform 9 allows IT organisations to transform their local IT infrastructure into a self-service private cloud. The beauty of this product is that it can be implemented on existing infrastructures. No need to create a new infrastructure to introduce the private cloud within your organisation. Just install the agent on your hypervisor layer, connect with the Platform 9 cloud management platform and you are off into the world of private clouds. The ease of integration is amazing and I believe that Platform 9 will be the accelerator of private cloud adoption. No need to go to AWS, no migration to Azure. You manage your own resources while allowing the customer to provision their own virtual machines or containers. Today Platform 9 supports KVM, but they will support both VMware and docker environments soon. I can dive into the details of Platform 9 but Eric Wright has done a tremendous job of publishing an extensive write-up and I recommend reading his article to learn more about Platform 9 private cloud offering. If you want to meet the team of Platform 9 and hear their vision, visit booth #324 at the solution exchange of VMworld 2014. ================================================================================ Title: Life in the Data Center - a story of love, betrayal and virtualization URL: https://frankdenneman.ai/2014-07-31-life-data-center-story-love-betrayal-virtualization/ Date: 2014-07-31 I’m excited to announce the first ever “collective novel”, in which members of the virtualization community collaborated to create a book with intrigue, mystery, romance, and a whole lot of geeky data center references. The concept of the project is that one person writes a section and then passes it along. The writers don’t know their fellow contributors. They get an unfinished story in their mailbox and are allowed to take the story in whatever direction it needs to go. The only limitation is the author imagination. For me it was a fun and interesting project. Writing a chapter for a novel is a whole different ballgame than writing technical focused content. As I rarely read novels it’s a challenge how to properly describe the situation the protagonist is getting himself into. On top of that I needed to figure out how to extend and expand the story line set by the previous authors but also get the story into a direction I prefer. And to make it more challenging, you do not know what the next author will be writing, therefor your intention for the direction of the storyline may be ignored. All in all a great experience and I hope we can do a second collective novel. I’m already collecting ideas ☺ I would like to thank Jeff Aaron. He came up with the idea and guided the project perfectly. Once again Jon Atterbury did a tremendous job on the formatting and artwork of the book. And of course I would like to thank the authors of taking time out of their busy schedules to contribute to the book. The authors: [caption id=“attachment_4495” align=“alignleft” width=“125”] Jeff Aaron (@jeffreysaaron)[/caption] [caption id=“attachment_4491” align=“alignleft” width=“125”] Josh Atwell (@Josh_Atwell)[/caption] [caption id=“attachment_4490” align=“alignleft” width=“125”] Kendrick Coleman (@KendrickColeman)[/caption] [caption id=“attachment_4488” align=“alignleft” width=“125”] Amy Lewis (@commsNinja)[/caption] [caption id=“attachment_4489” align=“alignleft” width=“125”] Lauren Malhoit (@malhoit)[/caption] [caption id=“attachment_4492” align=“alignleft” width=“125”] Bob Planker (@plankers)[/caption] [caption id=“attachment_4494” align=“alignleft” width=“125”] Satyam Vaghani (@SatyamVaghani)[/caption] [caption id=“attachment_4493” align=“alignleft” width=“125”] Chris Wahl (@ChrisWahl)[/caption] To make it more interesting for the readers, we deliberately hid which author wrote which chapter you can have some fun guessing via a short quiz. Prizes will be given to those people with the best scores. I’m not entirely sure that this book will be nominated for a Pulitzer, but it is worth a read to see what is in the authors’ crazy heads – and to witness how well they work together when collaborating on a project like this. Go download the book and take the quiz ================================================================================ Title: Let Cloudphysics help rid yourself of Heartbleed URL: https://frankdenneman.ai/2014-07-28-let-cloudphysics-help-rid-heartbleed/ Date: 2014-07-28 Unfortunately the Open SSL Heartbleed bug (CVE-2014-0224) is present in the ESXi and vCenter 5.5 builds. VMware responded by incorporating a patch to solve the OpenSSL vulnerability in the OpenSSL 1.0.1 library. For more info about the ESXI 5.5 patch read KB 2076665, VMware issued two releases for vCenter 5.5, read KB 2076692. Unfortunately some NFS environments experienced connection loss after applying the ESXi 5.5 patch, VMware responded by releasing patch 2077360 and more recently vCenter update 1b. The coverage on the NFS problems and the amount of ESX and vCenter update releases to fix a bunch of problems may left organizations in the dark whether they patched the Heartbleed vulnerability. Cloudphysics released a free Heartbleed analytic card in their card store that helps identify which hosts in your environment are unprotected. Check out the recent article of Cloudphysics CTO, Irfan Ahmad about their recently released Heartbleed analytic package. I would recommend to run the card and rid yourself of this nasty bug. ================================================================================ Title: Homelab - Power-on your Supermicro system by SSH'ing into IPMI URL: https://frankdenneman.ai/2014-04-29-homelab-power-supermicro-system-sshing-ipmi/ Date: 2014-04-29 Just a short article, recently I discovered you can access Supermicro IPMI via SSH and power on the system by using the command: start /system1/pwrmgtsvc1 A nice short command that saves you a lot of time by eliminating the need to log in the webUI and wait until the app responds. ================================================================================ Title: Which HA admission control policy do you use? URL: https://frankdenneman.ai/2014-04-04-ha-admission-control-policy-use/ Date: 2014-04-04 Yesterday Duncan and I where discussing the 5.5 update of the vSphere clustering deepdive book and we were debating which HA admission control policy is the most popular. Last week I asked around on twitter, but hopefully a short poll will give us better insights. Please cast your vote. [socialpoll id=“2195435”] ================================================================================ Title: Gotcha - Disable reserve all guest memory setting does not remove the reservation URL: https://frankdenneman.ai/2014-04-03-gotcha-disable-reserve-guest-memory-setting-remove-reservation/ Date: 2014-04-03 A while ago I wrote about the nice feature Reserve all guest memory available in vSphere 5.1 and 5.5. The feature automatically adjusts the memory reservation when the memory configuration changes. Increase the memory size and the memory reservation is automatically increased as well. Reduce the memory size of a virtual machine, and the reservation is immediately reduced. This week I received an email from someone who used the settings temporarily and when disabling this setting he was surprised that the reservation was not set to 0, reverting back to the default. [caption id=“attachment_4377” align=“aligncenter” width=“603”] Expected behavior[/caption] [caption id=“attachment_4379” align=“aligncenter” width=“604”] Real product behavior[/caption] Although I understand his point of view, the reality is that when you enabled the feature your intent was to apply a memory reservation to the virtual machine. The primary function of this setting is to take away the responsibility of adjusting the reservation when you change the memory reservation. If your goal is to remove the memory reservation, disable the setting Reserve all guest memory and then change the memory reservation to 0. ================================================================================ Title: vSphere 5.5 Home lab URL: https://frankdenneman.ai/2014-03-27-vsphere-5-5-home-lab/ Date: 2014-03-27 For a while I’ve been using three Dell R610 servers in my home lab. The machines specs are quite decent, each server equipped with two Intel Xeon 5530 CPUs, 48GB of memory and four 1GB NICs. With a total of 24 cores (48 HT Threads) and 144GB of memory the cluster has more than enough compute power. However from a bandwidth perspective they are quite limited, 3 Gbit/s SATA and 1GbE network bandwidth is not really pushing the envelope. These limitations do not allow me to properly understand what a customer can expect when running FVP software. In addition I don’t have proper cooling to keep the machines cool and their power consumption is something troubling. Time for something new, but where to begin? CPU Looking at the current lineup of CPUs doesn’t make it easier. Within the same CPU vendor product line multiple types of CPU socket exist, multiple different processor series exist with comparable performance levels. I think I spent most of my time on figuring out which processor to select. Some selection criteria were quite straightforward. I want a single CPU system and at least 6 cores with Hyper-Threading technology. The CPU must have a high clock speed, preferably above 3GHz. Intel ARK (Automated Relational Knowledge base) provided me the answer. Two candidates stood out; the Intel Core i7 4930 and the Intel Xeon E5 1650 v2. Both 6 core, both HT-enabled, both supporting the advanced technologies such as VT-x, VT-d and EPT. http://ark.intel.com/compare/77780,75780 The main difference between the two CPU that matters the most to me is the higher number of supported memory of the Intel Xeon E5. However the i7-4930 supports 64GB, which should be enough for a long time. But the motherboard provided me the answer Motherboard Contrary to the variety of choices at CPU level, there is currently one Motherboard that stands out for me. It looks it almost too good to be true and I’m talking about the SuperMicro X9SRH-7TF. This thing got it all and for a price that is unbelievable. The most remarkable features are the on-board Intel X540 Dual Port 10GbE NIC and the LSI 2308 SAS controller. 8 DIMM slots, Intel C602J chipset and a dedicated IPMI LAN port complete the story. And the best part is that its price is similar of a PCI version of the Intel X540 Dual Port 10GbE NIC. The motherboard only supports Intel E5 Xeons, therefor the CPU selection is narrowed down to one choice, the Intel Xeon E5 1650 v2. CPU Cooler The SuperMicro X9SRH-7TF contains an Intel LGA2011 socket with Narrow ILM (Independent Loading Mechanism) mounting. This requires a cooler designed to fit this narrow socket. The goal is to create silent machines and the listed maximum acoustical noise of 17.6 dB(A) of the Noctua NH-U9DX i4 “sounds” promising. Memory The server will be equipped with 64GB. Four 16GB DDR3-1600 modules allowing for a future upgrade of memory. The full product name: Kingston ValueRAM KVR16R11D4/16HA Modules. Network Although two 10 GbE NICs provide more than enough bandwidth, I need to test scenarios where 1GbE is used. Unfortunately vSphere 5.5 does not support the 82571 chipset used by the Intel PRO/1000 Pt Dual Port Server Adapter currently inserted in my Dell servers. I need to find an alternative 1 GbE NIC recommendations are welcome. Power supply I prefer a power supply that is low noise and fully modular. Therefore I selected the Corsair RM550. Besides a noise-reducing fan the PSU has a Zero RPM Fan Mode, which does not spin the fan until it is under heavy load, reducing the overall noise level of my lab when I’m not stressing the environment. Case The case of choice is the Fractal Design Define R4. Simple but elegant design, enough space inside and has some sound reducing features. Instead of the standard black color, I decided to order the titanium grey. SSD Due to the PernixDrive program I have access to many different SSD devices. Currently my lab contains Intel DC 3700 100GB and Kingston SSDNOW enterprise e100 200GB drives. Fusion I/O currently not (yet) in the PernixDrive program was so kind to lend me a Fusion I/O IODrive of 3.2 TB, unfortunately I need to return this to Fusion someday. Overview Component Type Cost CPU Intel Xeon E5 1650 v2 540 EUR CPU Cooler Noctua NH-U9DX i4 67 EUR Motherboard SuperMicro X9SRH-7TF 482 EUR Memory Kingston ValueRAM KVR16R11D4/16HA 569 EUR SSD Intel DC 3700 100GB 203 EUR Kingston SSDNOW enterprise e100 200GB 579 EUR Power Supply Corsair RM550 90 EUR Case Fractal Design Define R4 95 EUR Price per Server (without disks) 1843 EUR In total two of these machines are build as a start of my new lab. Later this year more of these machines will be added. I would like to thank Erik Bussink for providing me recommendations and feedback on the component selection of my new vSphere 5.5 Home Lab. I’m sure he will post a new article of his new lab soon. ================================================================================ Title: Help my DRS cluster is not load balancing! URL: https://frankdenneman.ai/2014-03-18-help-drs-cluster-load-balancing/ Date: 2014-03-18 Unfortunately I still see this cry for help appearing on the VMTN forums and on twitter. And they usually are accompanied by screenshots like this: This screen doesn’t really show you if your DRS cluster is balanced or not. It just shows if the virtual machine receives the resources they are entitled to. The reason why I don’t use the word demand is that DRS calculates priority based on virtual machine and resource pool resource settings and resource availability. To understand if the virtual machine received the resources it requires, hover over the bar and find the virtual machine. A new window is displayed with the metric “Entitled Resources Delivered” DRS attempts providing the resources requested by the virtual machine. If the current host is not able to provide the resources, DRS move it to another host that is able to provide the resources. If the virtual machine is receiving the resources it requires then there is no need to move the virtual machine to another hosts. Moves by DRS consume resources as well and you don’t want to waste resources on unnecessary migrations. To avoid wasting resources, DRS calculates two metrics, the current host load standard deviation and the target host load standard deviation. These metrics indicate how far the current load of the host is removed from the ideal load. The migration threshold determines how far these two metrics can lie apart before indicating that the distribution of virtual machines needs to be reviewed. The web client contains this cool water level image that indicates the overall cluster balance. It can be found at the cluster summary page and should be used as a default indicator of the cluster resource status. One of main arguments is that a host contain more than CPU and memory resources alone. Multiple virtual machines located on one host, can stress or saturate the network and storage paths extensively, whereas a better distribution of virtual machine across the hosts would also result in a better distribution of resources at the storage and network path layer. And this is a very valid argument, however DRS is designed to take care of CPU and Memory resource distribution and is therefor unable to take these other resource consumption constraints into account. In reality DRS takes a lot of metrics into account during its load balance task. For more in-depth information I would recommend to read the article: “DRS and memory balancing in non-overcommitted clusters” and “Disabling mingoodness and costbenefit”. ================================================================================ Title: Consumer grade SSD versus Enterprise grade SSD, which one to pick? URL: https://frankdenneman.ai/2014-03-10-consumer-grade-ssd-versus-enterprise-grade-ssd-one-pick/ Date: 2014-03-10 Should I use consumer grade SSD drives or should I use enterprise grade SSD drives? This a very popular question and I receive it almost on a daily basis. Lab or production environment, my answer is always the same: Enterprise grade without a doubt! Why? Enterprise Grade drives have a higher endurance level, they contain power loss data protection features and they consistently provide high level of performance. All align with a strategy ensuring reliable and consistent performance. Lets expand on these three key features; Endurance Recently a lot of information is released about the endurance levels of consumer grade SSDs and tests show that they operate well beyond the claimed endurance levels. Exciting news as it shows how much progression is made during the last few years. But be aware that vendors test their consumer grade SSDs with client workloads while enterprise grade SSDs are tested with worst-case data center workload. The interesting question is whether the SSD vendor is list the rate a drive in DWPD or drive-writes per-day in a conservative manner or an aggressive manner? As I don’t want to gamble with customers’ data, I’m not planning to find out whether the consumer SSD wasn’t able to sustain high levels of continuous data center load. I believe vSphere architectures have high endurance requirements; therefore use enterprise drives as they are specifically designed and tested for this use. Power loss data protection features Not often highlighted but most enterprise SSDs contain power loss data protection features. These SSDs typically contains a small buffer or cache in which the data is stored before it’s written to disk. Enterprise SSD leverages various on-board capacitance solutions to provide enough energy for the SSD to move the data from the cache to the drive itself. Protecting the drive and the data. It protects the drive because if a sector is partially written it becomes unreadable. This can lead to performance problems, as the drive will perform time-consuming error recovery on that sector. Select Enterprise drives with power loss data protection features, it avoids erratic performance levels or even drive failure after a power-loss. Consistent performance Last but certainly not least is the fact that enterprise SSDs are designed to provide a consistent level of performance. SSD vendors expect their enterprise disks to be used intensively for an extended period of time. This means that possibility of a full disk increases dramatically when comparing it to a consumer grade SSD. As data can only be written to a cell that is in an erased state, high levels of write amplification is expected. Please read this article to learn more about write amplification (write amp). Write amp impacts the ratio of drive writes to host writes, that means that when write amp occurs the number of writes a drive needs to make increases considerably in order to execute those host writes. One way to reduce this strain is to “over-provision” the drive. Vendors, such as Intel, allocate a large amount of flash resource to allow the drive to absorb these write amp operations. This results in a more consistent rate of IOPS and predictable IOPS. Impact on IOPS and Latency I’ve done some testing in my lab, and used two enterprise flash drives, a Intel DC S3700 and a Kingston E-100. I also used two different consumer grade flash devices. I refrain from listing the type and vendor name of these disks. I ran the first test from 11:30 to 11:50 I ran the test an enterprise grade SSD drive, the rate of IOPS was consistent and predictable. The VM was migrated to the host with the consumer grade SSD and the same test was run again, not a single moment did the disk provide a steady rate of IOs. Anandtech.com performed similar tests and witnessed similar behaviour, the publish their results in the article “Exploring the Relationship Between Spare Area and Performance Consistency in Modern SSDs” An excellent read, highly recommended. [caption id=“attachment_4262” align=“aligncenter” width=“646”] Picture courtesy of Anandtech.com[/caption] Click on the different drive sizes to view their default performance and the impact of spare flash resources on the ability to provide consistent performance. Next step was to determine latency behaviour. Both Enterprise grade SSD provided an extreme case of predictable latency. To try to create an even playing field I ran read tests instead of write centric tests. The first graph was a read test on the Kingston e100. Latency was consistent providing predictable and consistent application response time. The consumer grade drive performance charts were not as pretty. The virtual machine running the read test was the only workload hitting the drive and yet the drive had trouble providing steady response times. Please note that the test were ran multiple times and the graphs shown are the most positive ones for the consumer grades. Multiple (enterprise-level) controllers were used to avoid any impact from that layer. As more and more SSD drive hit the market we decided to help to determine which drives fit in a strategy ensuring reliable and consistent performance. Therefor PernixData started the PernixDrive initiative, in which we test and approve flash devices. Conclusion Providing consistent performance is key for predictable application behaviour. This applies to many levels of operation. First of all it benefits day-to-day customer satisfaction and helps you to reduce troubleshooting application performance. Power-loss data protection features help you to cope with short-term service loss, and avoid continuous performance loss as the drive can survive power-loss situations. Reverting applications to a non-accelerated state, due to complete loss of SSD drive can result in customer dissatisfaction or neglecting your SLA. Higher levels of drive-writes per-day help you to create and ensure high levels of consistent performance for longer terms. In short, use the correct tool for the job and go for enterprise SSD drives. ================================================================================ Title: Who to vote for? URL: https://frankdenneman.ai/2014-02-25-vote/ Date: 2014-02-25 This week Eric Siebert opened up the 2014 edition of top virtualization blog contest. For the industry this is one of the highlights and applaud the effort Eric and his team of volunteers put in to make this work. I cannot wait to the watch the show in which they unveil this years top 25 winners. A big thank you to Eric and the team! Most of the time you will see blog articles that highlight this years effort and I think they are great. As there are so many great bloggers writing and sharing their thoughts and ideas, it’s very easy to miss out on some brilliant post. A quick scan of these posts helps to (re)discover the wealth of information that is out there. Last year I was voted number 2, however this year the frequency (hopefully not the quality) of my blog articles went down. This was due to my career change and the new responsibilities my job role encompasses. Plus creating the vSphere design book took a lot of time and effort. For this years VMworld we have planned something even better, so please stay tuned for this years VMworld book! But this post is not about me as a blogger and my material, but to highlight some of the bloggers that help the community understand the product better, comprehend the behavior of the complex systems we work with every day and the insights they provide by spending a lot of their (spare) time writing and creating these great articles. Voting for them you will help them understand that their time and effort is well spend! First of all, guys like Duncan Epping, Cormac Hogan, William Lam and Eric Sloof relentlessly churn out great collateral, whether it is a written article, podcast or video. It keeps the community well fed when it comes to quality information. Writing a great article is a challenge, doing this on a continuous basis is even more impressive! But I would like to highlight some of the guys that are considered “new” guys. They are all industry veterans, but they decided to pick up blogging recently. I would like to highlight these guys, but there are many more of course. Pete Koehler - vmpete.com Pete writes a lot about PernixData, but that’s not the reason I want to highlight him. His articles are quite in-depth and I love reading those articles as I learn from them every time Pete decides to post his most recent insights. For example in the article “Observations of PernixData in a Production environment” he covers the IOPS, Throughput & Latency relationship in great detail. In this exercise he discovers that applications do not use a static block size, something you don’t read that often. He correlates specific output and explains how each metric interacts which each other, educating you along the way and helping you to do a better and more effective job in your own environment. Josh Odgers - joshodgers.com Josh is listed both on the general blogging list as well as a newcomer and I think he deserves to be “rookie of the year” Josh’s insight are very valuable and its always a joy to read his articles. His VCDX articles are top notch and are a must read for every aspiring VCDX candidate. Just too bad he decided to join Nutanix ;). Luca Dell’Oca – virtualtothecore.com Dropping knowledge both in English and Italian, Luca is covering new technologies as well as insight full tips and tricks on a frequent basis. Ranging from reclaiming space on a Windows 2012 installation to a complete write up on how to create a valuable I/O test virtual machine. A blog that should be visited regularly. Willem ter Harmsel - willemterharmsel.nl Not your average virtualization blog, Willem covers the startup world by interviewing CEO’s and CTOs of the hottest and newest startups this world currently has to offer. Willem provides insights of upcoming technology and allows its readers to place and compare different technologies. A welcome change of pace after spending a day knee-deep into the bits and bytes Consuming those stories and articles on a daily basis, are they helpful in your daily work? Please show your appreciation and vote today on your favorite blogs! Thanks! Please vote now! ================================================================================ Title: VCDX Defence: Are you planning to use a fictitious design? URL: https://frankdenneman.ai/2014-02-10-vcdx-defence-planning-use-fictitious-design/ Date: 2014-02-10 This week the following tweet caught my eye: https://twitter.com/VirtualSnook/status/432274972992487424 Apparently Marc Brunstad (VCDX program manager) stated this fact during the PEX VCDX workshop. But what does this stat mean and to what level do you need to take this into regard when submitting your own design? During my days as a panel member, I’ve seen only a handful of fictitious designs and although they were technically sound, the reasoning and defense were usually not that strong. Be aware that the VCDX program isn’t born into existence to find the best design ever. It determines if the candidate has aligned the technical functionality with the customers’ requirements, the constraints provided by the environment and the assumptions the team made about for example future workloads or organizational growth. But does that mean that you shouldn’t use any fictitious element in your design? Are fictiticous elements inherently bad? I don’t think so. Speaking from own experience I made some adjustments to my design I submitted. My submitted design was largely based on the environment that I worked on for a couple of years. At that time the customer used rack-based systems, my design contained a blade architecture. The reason why I changed this, as it allowed me to demonstrate my knowledge of the HA stack featured in vSphere 4.1. Some might argue that I deliberately made my design more complex, but I was comfortable enough to defend my choices and explain High Availability Primary and Secondary node interaction and how to mitigate risk. More over it allowed me to demonstrate the pros and cons of such a design on various levels, such as the impact it had on operational processes, the influence on scalability and the alignment of availability policies to org-defined failure domains. Did I have these discussions in real life? Yes, with many other customers but just not with that specific customer that this design was based on. And that’s why complete fictitious designs fail and why most reasoning is incomplete. The candidate only focused on the alignment of technical specs and workload. Not the “softer” side of things. Arguing that this design element was just the wish of a customer just doesn’t cut it. Sure we all met customers that were strung on having that particular setting configured in the way they saw fit, but its your responsibility to explain to the panel which steps you took to inform the customer about the risk and potential impact that setting had. Try to explain which setting you would have used and why. Demonstrate your knowledge about feasible alternatives. My recommendation to future candidates; when incorporating a specific fictitious design element in your design, make sure you had a conversation with a customer about that element once. You can easily align this with the main design and it helps to recollect the specifics during your defense. ================================================================================ Title: Installing Exchange Jetstress without full installation media. URL: https://frankdenneman.ai/2014-02-05-installing-exchange-jetstress-without-full-installation-media/ Date: 2014-02-05 I believe in testing environments with applications that will be used in the infrastructure itself. Pure synthetic workloads, such as IOmeter, are useful to push hardware to their theoretical limit but that’s about it. Using a real life workload, common to your infrastructure, will give you a better understanding of the performance and behavior of the environment you are testing. However, it can be cumbersome to setup the full application stack to simulate that workload and it might be difficult to simulate future workload. Simulators made by the application vendor, such as SQLIO Disk Subsystem Benchmark Tool or Exchange Server Jetstress, provide an easy way to test system behaviour and simulate workloads that might be present in the future. One of my favourite workload simulators is MS Exchange server Jetstress however its not a turn-key solution. After installing Exchange Jetstress you are required to install the ESE binary files from an Exchange server. It can happen that you don’t have the MS exchange installation media available or a live MS exchange system installed. Microsoft recommends downloading the trail version of Exchange, install the software and then copy the files from its directory. Fortunately you can save a lot of time by skipping these steps and extract the ESE files straight from an Exchange Service Pack. Added bonus, you immediately know you have the latest versions of the files. I want use Jetstress 2010 and therefor I downloaded Microsoft Exchange Server Jetstress 2010 (64 bit) and Microsoft Exchange Server 2010 Service Pack 3 (SP3). To extract the files direct from the .exe file, I use 7zip file archiver. () The ESE files are located in the following directory: File Path ese.dll \setup\serverroles\common eseperf.dll \setup\serverroles\common\perf\amd64 eseperf.hxx \setup\serverroles\common\perf\amd64 eseperf.ini \setup\serverroles\common\perf\amd64 eseperf.xml \setup\serverroles\common\perf\amd64 Copy the ESE files into the Exchange Jetstress installation folder. By default, this folder is “C:\Program Files\Exchange Jetstress”. Be aware that you need to run Jetstress as an administrator. Although you might login your system using you local and domain admin account, Jetstress will be kind enough to throw the following error: The MSExchange Database or MSExchange Database ==> Instrances performance counter category isn’t registered Just right-click the Jetstress shortcut and select “run as administrator” and you are ready for action. Happy testing! ================================================================================ Title: vSphere 5.5 vCenter server inventory 0 URL: https://frankdenneman.ai/2014-01-16-vsphere-5-5-vcenter-server-0-inventory/ Date: 2014-01-16 After logging into my brand spanking new vCenter 5.5 server I was treated with a vCenter server inventory count of 0. Interesting to say the least as I installed vCenter on a new windows 2008 R2 machine, connected to a fresh MS active directory domain. I installed vCenter with a user account that is domain admin, local admin and has all the appropriate local rights (M_ember of the Administrators group, Act as part of the operating system and Log on as a Service_). The install process went like a breeze, no error messages whatsoever and yet the vCenter server object was mysteriously missing after I logged in. A mindbender! Being able to log into the vCenter server and finding no trace of this object whatsoever, it felt like someone answering the door and saying he’s not home. I believed I did my due diligence, I read the topic “Prerequisites for Installing vCenter Single Sign-On, Inventory Service, and vCenter Server” and followed every step, however it appeared I did not RTFM enough. administrator@vsphere.local only Apparently vSphere will only attach the permissions and assign the role of administrator to the default account administrator@vsphere.local and you have to logon with this account after the installation is complete. See “How vCenter Single Sign-On Affects Log In Behavior” for the following quote: After installation on a Windows system, the user administrator@vsphere.local has administrator privileges to both the vCenter Single Sign-On server and to the vCenter Server system. It threw my off balance by allowing me to log in with the account that I used to install vCenter, this made me assume the account automatically received the appropriate rights to manage the vCenter server. To gain access to the vCenter database you must manually assign the administrator role to the AD group or user account of your liking. As an improvement over 5.1 vCenter 5.5 adds the active directory as an identity source, but will not assign any administrator rights, ignoring the user account used for installing the product. Follow these steps to use your AD accounts to manage vCenter. 1. Verify AD domain is listed as an Identity Source Log in with administrator@vsphere.local and select Configuration in the home menu tree. Only when you are logged in with an SSO administrator vCenter will show the Single Sign-on menu option. Select Single Sign-on | Configuration and verify if AD domain is listed. 2. Add Permissions to top object vCenter Go back to home, select menu option vCenter, vCenter Servers and then the vCenter server object. Select the menu option Manage, Permissions 3. Add User or Group to vCenter Click on the green + icon to open the add permission screen. Click on the Add button located at the bottom. 4. Select the AD domain Select the AD domain and then the user or group. In my example I selected the AD group “vSphere-admins”. I’m using groups to keep the vCenter configuration as low-touch as possible. When I need grant additional users administrator rights I can simple do this in my AD Users and Computers tool. Traditionally auditing is of a higher level in AD then in vCenter. 5. Assign Administrator Role In order to manage the vCenter server all privileges need to be assigned to that user, by selecting the administrator role all privileges are assigned and propagated to all the child objects in the database. 6. Log in with your AD account Log out the user administrator@vsphere.local and enter your AD account. Click on vCenter to view the vCenter Inventory list. vCenter Servers should list the new vCenter server. ================================================================================ Title: VCDX defend clinic: Choosing between Multi-NIC vMotion and LBT URL: https://frankdenneman.ai/2014-01-07-vcdx-defend-clinic-choosing-multi-nic-vmotion-lbt/ Date: 2014-01-07 A new round of VCDX defenses will kickoff soon and I want to wish everyone that participates in the panel session good luck. Usually when VCDX panels are near, I receive questions on how to prepare for a panel. And one recommendation I usually provide is “Know why you used a specific configuration of a feature and especially know why you haven’t used the available alternatives”. Let’s have some fun with this and go through a “defend clinic”. The point of this clinic is to provide you an exercise model than you can use for any configuration, not only for a vMotion configuration. It helps you to understand the relationship of information you provide throughout your documentation set and helps you explain how you derived through every decision to come to this design. To give you some background, when a panel member is provided the participants documentation set, he enters a game of connecting the dots. This set of documents are his only view into the your world while creating the design and dealing with your customer. He needs to take your design and compare it to the requirements of the customer, the uncertainties you dealt with in the form of assumptions and the constraints that were given. Reviewing the design on technical accuracy is only a small portion of the process. That’s just basically checking to see if you are using your tools and material correctly, the remaining part is to understand if you build the house to the specification of the customer while dealing with regional laws and the available space and layout of the land. Building a 90.000 square feet single floor villa might provide you the most amount of easily accessible space, but if you want to build that thing in downtown Manhattan you’re gonna have a bad time. ;) Structure of the article This exercise lists the design goals and its influencers, requirements, constraints and assumptions. The normal printed text is architects (technical) argument while the paragraphs are displayed in Italic can be seen as questions or thoughts of a reviewer/panel member. Is this a blue print on how to beat the panel? No! It’s just an exhibition on how to connect and correlate certain statement made in various documents. Now let’s have some fun exploring and connecting the dots in this exercise. Design goal and influencers Your design needs to contain a vMotion network as the customer wants to leverage DRS load balancing, maintenance mode and overall enjoy the fantastic ability of VM mobility. How will you design your vMotion network? In your application form you have stated that the customer want to see a design that reduces complexity, increases scalability, prefers to have the best performance available as possible. Financial budget and the amount of IP-addresses are constraints and the level of expertise of the virtualization management team is an assumption. Listing the technical requirements Since you are planning to use vSphere 5.x you have the choice to create a traditional single vMotion-enabled VMKnic, Multi-NIC vMotion setup or use vMotion configuration that uses “Route based on physical NIC load” load balance algorithm (commonly known as LBT) to distribute vMotion traffic amongst multiple active NICs. As the customer does not prefer to use link aggregation, IP-hash based / EtherChannel configurations are not valid. First let’s review the newer vMotion configurations and how they differentiate from the traditional vMotion configuration, where you have one single VMKnic, a single IP address, connected to a single Portgroup which is configured to use an active and standby NIC? Multi-NIC vMotion • Multiple VMKnics required • Multiple IP-addresses required • Consistent configuration of NIC failover order required • Multiple physical NICs required Route based on physical NIC load • Distributed vSwitch required • Multiple physical NICs required It goes without saying that you want to provide the best performance possible that leads you into considering using multiple NICs to increase bandwidth. But which one will be better? A simple performance test will determine that. VCDX application form: Requirements In your application document you stated that one of the customer requirements was “Reducing complexity”. Which of the two configurations do you choose now, what are your arguments? How do you balance or prioritize performance over complexity reduction? If Multi-NIC vMotion beats LBT configuration in performance, leading to faster maintenance mode operations, better DRS load balance operations and overall reduction in lead time of a manual vMotion process, would you still choose the simpler configuration over the complex one? Simplicity is LBTs forte, just enable vMotion on a VMKnic, add multiple uplinks, set them to active and your good to go. Multi-NIC vMotion exists of more intricate steps to get a proper configuration up and running. Multiple vMotion-enabled VMKnics are necessary, each with their own IP-range configuration, secondly vMotion requires deterministic path control, meaning that it wants to know which path is selects to send traffic across. As the vMotion load balancing process is higher up in the stack, NIC failover orders are transparent for vMotion. It selects a VMKnic and assumes it resembles a different physical path then the other available VMKnics. That means its up to the administrator to provide these unique and deterministic paths. Are they capable of doing this? You mentioned the level of expertise of the admin team as an assumption, how do you guarantee that they can execute this design, properly manage it for a long period and expand the design without the use of external resources? Automation to the rescue Complexity of technology by itself should not pose a problem, its how you (are required to) interact with it that can lead to challenges. As mentioned before Multi-NIC vMotion requires multiple IP-addresses to function. On a side note this could put pressure on the IP-ranges as all vMotion enabled VMKnics inside the cluster requires being a part of the same network. Unfortunately routed vMotion is not supported yet. Every vMotion VMKnic needs to be configured properly, Pair this with availability requirements and the active and standby NIC configuration of each VMKnic can cause headaches if you want to have a consistent and identical network configuration across the cluster. Power-CLI and Host Profiles can help tremendously in this area. Supporting documents Now have you included these scripts in your documentation? Have you covered the installation steps on how to configure vMotion on a distributed switch? Make sure that these elements are included in your supporting documents! What about the constraints and limitations? Licensing Unfortunately LBT is only available in distributed vSwitches, resulting in a top-tier licensing requirement if LBT is selected. The LBT configuration might be preferred over Multi-NIC vMotion configuration because it provides the least amount of complexity increase over the traditional configuration. How does this intersect with the listed budget constraint and the customer is not able –or willing – to invest in enterprise licenses? IP4 pressure One of the listed constraints in the application form is the limited amount of IP addresses in the available IP range destined for the virtual infrastructure. This could impact your decision on which configuration to select. Would you “sacrifice” the amount of IP-s to get a better vMotion performance and all the related improvements on the remaining dependent features or is scalability and future expansion of your cluster more important? Remember scalability is also listed in the application form as a requirement. Try this at home! These are just an example of questions that can be asked during a defense. Try to find these answers when preparing for you VCDX panel. When finalizing the document set, try to do this exercise. Even better to find a group of your peers and try to review each others design while reviewing the application form and the supporting set of documents. At the Nordic VMUG Duncan and I spoke with a group of people that are setting up a VCDX study group, I think this is a great way of not only preparing for a VCDX panel but to learn and improve your skill set you can use in your daily profession. ================================================================================ Title: My lab and the birth of the portable Ikea lack 19” datacenter rack URL: https://frankdenneman.ai/2013-12-16-my-lab-and-the-birth-of-the-portable-ikea-lack-19-datacenter-rack/ Date: 2013-12-16 Currently, topics about labs are hot, and when meeting people at the VMUGs or other tech conferences, I get asked a lot about my lab configuration. I’m a big fan of labs, and I think everybody who works in IT needs a lab, whether it’s at home or in a centralized location. At PernixData, we have two major labs. One on the east coast and one on the west coast of the U.S. Both these labs are shared, so you cannot do everything you like. However, sometimes you want to break stuff. You want to pull cables and disks and kill an entire server or array. To see what happens. For these reasons having a lab that is 4000 miles away doesn’t work. Enough reasons to build a small lab at home. Currently topic about labs are hot and when meeting people at the VMUGs or other tech conferences, I get asked a lot about my lab configuration. I’m a big fan of labs and I think everybody who works in IT needs a lab, whether it’s at home or in a centralised location. At PernixData we have two major labs. One at the east coast and one at the west coast of the U.S of A. Both these labs are shared, that means you cannot do everything you like. However sometimes you want to break stuff, you want to pull cables, disks, kill an entire server or array. Just to see what happens. For these reasons having a lab that is 4000 miles away doesn’t really work, enough reasons to build a small lab at home. Nested or physical hardware? To nest or not to nest, that’s not even the question. Nesting is amazing, and VMware spends a lot of energy and time on nested environments (think HOL). Recently the fling VMware tools for Nested ESXi was released, and I assume more nested ESXi flings will follow after seeing the attention it received from the community. But to run nested ESXi, you need to have physical hardware. Thanks to a generous donation, I received 6 Dell r610s, which covered my compute level requirements. But sometimes, you only want to test the software, and in those cases, you do not need to fire up an incredibly loud semi-datacenter rig. For those situations, I created an ESXi host that is near silent when running full speed. This ESXi server also hosts a nested ESXi environment and is just a white box with a simple ASUS mobo, 24GB, and the Intel 1GB Ethernet port. Once this machine is due for renewal, a white box following the baby dragon design will replace it. To test the software at the enterprise level, you require multiple levels of bandwidth, sometimes the bare minimum and sometimes copious amounts of it. The R610 sports 4 x 1GB Ethernet connections, allowing me to test scenarios that can happen in a bandwidth-constrained environment. Usually, compelling cases happen when you have a lot of restrictions to deal with, and these 1GB NICs are perfect for this. 10GB connections are on my wish list, but to have a nice setup, you still need to invest more than 1000 bucks in testing it adequately. A little bit over the top for my home lab, but the community came to the rescue and provided me with a solution; the Infiniband hack. A special thanks go out to Raphael Schitz and Eric Bussink for providing me the software and the information to run my lab at 10Gbps and being able to provide incredibly low latencies to my virtual machines. With the InfiniBand setup, I can test scenarios where bandwidth is not a restriction and investigate specific setups and configurations. For more info, listen to the vBrownbag tech talk where Erik Bussink dives into the topic “InfiniBand in the Lab” The storage layer is provided by some virtual storage appliances, each backed by a collection of different SSD disks and WD Black Caviar 750GB disks. Multiple solutions allow me to test various scenarios such as all-flash arrays, hybrid, and all magnetic disk arrays. If I need to understand the specific dynamics of an array, I log in to one of the two US-based labs. Home office My home office is designed to be an office and not a data center. So where do you place 19" rack servers without ruining the esthetics of your minimalistic designed home office ;). Well, you create a 19" rack on wheels so you can roll it out of sight and place it wherever you want it. Introducing the portable Ikea lack 19" datacenter rack. Regular readers of my blog or Twitter followers know I’m a big fan of hacking IKEA furniture. I created a whiteboard desk that got the attention of multiple sites and ikeahackers.net provided me with a lot of ideas on how to hack the famous lack table side table. I bought two lack tables, a couple of L-shaped brackets, four wheels, nuts, and bolts. The first lack table provides the base platform. Only the tabletop is used. The legs are discarded and act as a backup if I make a mistake during the drilling. I didn’t test the center of the tabletop, but the corners of the tabletop are solid and can be used to install wheels. I used heavy-duty ball-bearing wheels with an offset swivel caster design that permits ease of directional movement. Simple 5mm nuts and bots keep the L shape brackets in place, but beware, the table legs are not made of solid wood. They are hollow! Only a few centimeters of the top of the leg is solid. This to hold the screw that connects the table and leg. To avoid having the server pull the screw through the leg due to its weight, I used washers to keep them in place What’s next? From a hardware perspective, 10GbE is still high on my wishlist. When looking at the software layer, I want to create a more automated way of deploying and testing PernixData FVP software. One of the things I’m looking into is using and incorporating Auto Deploy in the lab. But that’s another blog post.t. ================================================================================ Title: Send F11 key to nested ESXi on Mac URL: https://frankdenneman.ai/2013-11-05-send-f11-key-to-nested-esxi-on-mac/ Date: 2013-11-05 I only use Mac at home, most of the time it’s great sometimes it’s not. For example when installing or configuring your remote lab. I have a windows server installed on a virtual machine that runs vCenter and the vSphere client. When I’m installing a new nested ESXi server, I connect with a remote desktop session to the Windows machine and use the VMware vSphere client. During the ESXi install process, it requires to press the F11 key to continue with the install process. However, F11 isn’t mapped by the vSphere client automatically and there isn’t a menu option in the vSphere client to send it to the client. Fortunately, I found the combination, so I’m writing it down here as I’m bound to forget. Press FN-CMD-F11 to send the key to the install screen of ESXi. Happy installing! ================================================================================ Title: vCPU configuration. Performance impact between virtual sockets and virtual cores? URL: https://frankdenneman.ai/2013-09-18-vcpu-configuration-performance-impact-between-virtual-sockets-and-virtual-cores/ Date: 2013-09-18 A question that I frequently receive is if there is a difference in virtual machine performance if the virtual machine is created with multiple cores instead of selecting multiple sockets? Single core CPU VMware introduced multi core virtual CPU in vSphere 4.1 to avoid socket restrictions used by operating systems. In vSphere a vCPU is presented to the operating system as a single core cpu in a single socket, this limits the number of vCPUs that can be operating system. Typically the OS-vendor only restricts the number of physical CPU and not the number of logical CPU (better know as cores). For example, Windows 2008 standard is limited to 4 physical CPUs, and it will not utilize any additional vCPUs if you configure the virtual machine with more than 4 vCPUs. To solve the limitation of physical, VMware introduced the vCPU configuration options “virtual sockets” and “cores per socket”. With this change you can for example configure the virtual machine with 1 virtual sockets and 8 cores per socket allowing the operating system to use 8 vCPUs. Just to show it works, I initially equipped the VM running Windows 2008 standard with 8 vCPU each presented as a single core. When reviewing the cpu configuration inside the Guest OS, the task manager shows 4 CPUs: A final check by opening windows task manager verified it only uses 4 vCPUs. I reconfigured the virtual machine to present 8 vCPU using a single socket and 8 number of cores per socket. I proceeded to power-on the virtual machine: Performance impact Ok so it worked, now the big question, will it make a difference to use multiple sockets or one socket? How will the Vmkernel utilize the physical cores? Might it impact any NUMA configuration. And it can be a very short answer. No! There is no performance impact between using virtual cores or virtual sockets. (Other than the number of usuable vCPU of course). Abstraction layer And its because of the power of the abstraction layer. Virtual socket and virtual socket are “constructs” presented upstream to the tightly isolated software container which we call a virtual machine. When you run a operating system it detects the hardware (layout) within the virtual machine. The VMkernel schedules a Virtual Machine Monitor (VMM) for every vCPU. The virtual machine vCPU configuration is the sum of number of cores x number of sockets. Lets use the example of 2 virtual socket 2 virtual core configuration. The light blue box shows the configuration the virtual machine presents to the guest OS. When a CPU instruction leaves the virtual machine it get picked up the Vmkernel. For each vCPU the VMkernel schedules a VMM world. When a CPU instruction leaves the virtual machine it gets picked up by a vCPU VMM world. Socket configurations are transparent for the VMkernel NUMA When a virtual machine powers on in a NUMA system, it is assigned a home node where memory is preferentially allocated. The vCPUs of a virtual machine are grouped in a NUMA client and this NUMA client is scheduled on a physical NUMA node. For more information about NUMA please read the article: “Sizing VMs and NUMA nodes” Although it’s a not covering the most current vSphere release, the basics remain the same. To verify that the sockets have no impact on the NUMA scheduler I powered up a new virtual machine and configured it with two sockets with each 2 cores. The host running the virtual machine is a dual socket quad core machine with HT enabled. Providing 4 vCPUs to the virtual machine ensures me that it will fit inside a single NUMA node. When reviewing the memory configuration of the virtual machine in ESXTOP we can deduct that its running on a single physical CPU using 4 cores on that die. Open the console, run ESXTOP, press M for memory view. Use V (capital v) to display on VM worlds only. Press F and select G for NUMA stats. You might want to disable other fields to reduce the amount of information on your screen. The column, NHN identifies the current Numa Home Node, which in Machine2 case is Numa node 0. N%L indicates how much memory is accessed by the NUMA client and it shows 100%, indicating that all vCPUs access local memory. The column GST_ND0 indicates how much memory is provided by Node0 to the Guest. This number is equal to the NLMEM counter, which indicated the current amount of local memory being accessed by VM on that home node. vNUMA What if you have a virtual machine with more than 8 CPU (for clarity, life of a Wide NUMA starts at a vCPU count of 9). Then the VMkernel presents the NUMA client home nodes to the Guest OS. Similar to the normal scheduling, the socket configuration are also transparent in this case. Why differentiate between sockets and cores? Well there is a difference and it has to do with the Hot-Add CPU feature. When enabling the option CPU Hot Plug you can only increase the virtual socket count. In short using virtual sockets or virtual cores does not impact the performance of the virtual machine. It only effects the initial configuration and the ability to assign more vCPU when your Operating System restricts the maximum number of physical CPUs. Always check if your VM configuration is in compliance with the vendor licensing rules before increasing the vCPU count! ================================================================================ Title: Only two days left to sign up for your VMworld speaker shirt! URL: https://frankdenneman.ai/2013-08-12-only-two-days-left-to-sign-up-for-your-vmworld-speaker-shirt/ Date: 2013-08-12 During our advisory call with CloudPhysics, a great idea was born. Why not provide all the speakers at VMworld a cool speaker shirt? Unfortunately in 2011 VMworld made the decision to stop providing speaker shirts to the people on stage, so most people started wearing older speaker shirt or even RUN DRS shirts ☺. For most speakers this move by VMworld was disappointing as the speaker shirts made them more recognizable but it also served as a cool badge of honor. I think CloudPhysics stepped up big time and gave us back that cool badge of honor. If you are a speaker this year, go register here before tomorrow evening as the deadline is Tuesday August 13th end of day(pst). ================================================================================ Title: Have you seen the new video introduction to PernixData? URL: https://frankdenneman.ai/2013-08-06-have-you-seen-the-new-video-introduction-to-pernixdata/ Date: 2013-08-06 I know PernixData FVP is cool, you know it’s cool, now how can you tell you all your friends and colleagues about it? Going around and telling them one by one doesn’t scale, just show them this video! http://www.youtube.com/watch?v=_t5b61xtT4Q I think Jeff sums it up nicely: ================================================================================ Title: New Book Project: Tweet sized vSphere Design Considerations – Selection process URL: https://frankdenneman.ai/2013-06-21-new-book-project-tweet-sized-vsphere-design-considerations-selection-process/ Date: 2013-06-21 Just a quick update - this week we passed the deadline for Call of entries and have closed the form. Well over 400 design considerations were submitted and it’s now up to the judges to go through the design considerations. Thanks all for submitting your entries! Stay tuned for more updates. ================================================================================ Title: Download the vSphere 4.1 and 5.0 clustering deepdive for free URL: https://frankdenneman.ai/2013-06-05-download-the-vsphere-4-1-and-5-0-clustering-deepdive-for-free/ Date: 2013-06-05 Do you want a free Kindle copy of the vSphere 4.1 HA and DRS Deepdive or the vSphere 5.0 Clustering Deepdive? Today and tommorow, Thursday June the 6th, you can download the Kindle (US Kindle Store) copy of both these books for free! So make sure you pick it up either today or tomorrow, it might be the only time this year it is on promo. ================================================================================ Title: Want to have my former job? URL: https://frankdenneman.ai/2013-05-20-want-to-have-my-former-job/ Date: 2013-05-20 My old technical marketing team of VMware is looking for someone to cover resource management. If you have a passion for resource management of virtual infrastructures and like to help VMware’s field personnel, partners and customers understand the technology then this job might be something for you. A large part of my role was bridging between engineering, product management, product marketing and the field / customers. Provide information to the R&D side of VMware how the products are used and what features customers are requesting. You create collateral in every way or form to help the customer and field personnel understand and adopt the features. I always enjoyed working with the different teams at VMware. The cloud resource management and vMotion team are an awesome group to work with. Be prepared to deep dive with these guys Marianas trench style. Having a customer facing background helps you provide the team valuable information to align the features to the customer wishes. In this role you assist product marketing and product management in achieving their tactical and strategic plans. Besides working with the responsible engineering and product marketing teams you collaborate with your technical marketing colleagues. You have the ability to interact with guys such as Ken Werneberg, Cormac Hogan, Mike Foley, William Lam, Alan Renouf or Rawlinson Rivera on a daily basis. If you have thorough understanding of the vMotion features, DRS, Storage DRS, SIOC and DPM and love to help customers adopt these features, then apply now! http://jobs.vmware.com/job/Palo-Alto-Sr_-Technical-Marketing-Manager-Resource-Management-Job-CA-94301/2593496/ Please note Be aware that this is my former role and that I no longer work for VMware. Therefor I cannot answer any further inquiries. Please contact the VMware career team. ================================================================================ Title: New Storage DRS whitepaper available at VMware.com URL: https://frankdenneman.ai/2013-05-13-new-storage-drs-whitepaper-available-at-vmware-com/ Date: 2013-05-13 Last Friday my last and latest whitepaper about Storage DRS was published on VMware.com. Go to http://www.vmware.com/resources/techresources/10363 and download the whitepaper: “Understanding vSphere 5.1 Storage DRS”. Download and read this whitepaper if you want to learn more about the five key elements of Storage DRS. Here’s a little snippet from the whitepaper: Step 1. Determine Whether Datastores Are Violating the Space-Utilization Threshold If the space utilization of a datastore exceeds 80 percent, the datastore violates the threshold and the vSphere Storage DRS load-balancing algorithm is invoked. vSphere Storage DRS attempts to avoid an out-of-space situation and therefore runs a load-balancing operation as soon as the datastore exceeds its space-utilization threshold. This operation can be outside of the normal load-balancing interval of every 8 hours. The space-utilization threshold is a soft limit, enabling vSphere Storage DRS to place virtual machines in the datastore cluster even if all datastores exceed the space-utilization threshold. vSphere Storage DRS attempts to generate prerequisite migrations before virtual machine placement. If this fails, the virtual machine is placed on the datastore that provides the best overall cluster balance. This performance applies to space-utilization load-balancing operations as well, even if all datastores violate the space-utilization threshold. vSphere Storage DRS tries to keep space utilization near the threshold across all datastores. Download: http://www.vmware.com/files/pdf/vmw-vsphr-5-1-stor-drs-uslet-101-web.pdf ================================================================================ Title: Cloudphysics webinar: Expert Tips for Managing Datastore Space in a vSphere Environment URL: https://frankdenneman.ai/2013-05-09-cloudphysics-webinar-expert-tips-for-managing-datastore-space-in-a-vsphere-environment/ Date: 2013-05-09 Tonight I will join Erik Haus and Krishna Raj Raja of CloudPhysics to talk about Datastore space management in a virtual infrastructure. During the webinar Krishna will show you how the new CloudPhysics “Datastore Space” card and “Snapshots Gone Wild” card help you to identify and resolve space problems. Join us! The event will start at 6:00 pm Amsterdam Time on May 9, 2013. (9:00 AM PDT) Go to the CloudPhysics site to register! ================================================================================ Title: Embarking on a new adventure - Joining PernixData as Tech Evangelist URL: https://frankdenneman.ai/2013-04-29-embarking-on-a-new-adventure-joining-pernixdata-as-tech-evangelist/ Date: 2013-04-29 Sometimes something comes along that makes you feel you need to get involved with. Something that makes you want to leave the comfortable position you have now and take up the challenge of starting all over again. Help turn that something into something big. Well that something is in my case PernixData and its Flash Virtualization Platform. Joining PernixData means I’m leaving the great company of VMware and an awful lot of great colleague behind. Some of them I consider to be good friends. During my years at VMware I learned a lot and words cannot describe how awesome those years were. Designing the vCloud environment for the European launching partner, consulting a lot of the Fortune 500 firms, participating in VCDX panels around the world and co-authoring three books are some of the highlights during my time at VMware but I’m sure I’m forgetting a lot of other great moments. Being a part of the technical marketing team was amazing! Besides working alongside the best bloggers in the world I had the privilege to work with the engineers on a daily basis. Having a job that allows you to think, talk and write about technology you absolutely love is great and difficult to let go. But opportunities do come along and as I mentioned in the beginning some of these opportunities spark the desire to become a part of that story. When I attended a technical preview of the Flash Virtualization Platform at PernixData I got excited. I think just as excited as when I saw my first vMotion. Meeting the founders and the team made me realize that this company and product was more than just a single product, this platform is a game changer in the world of virtual infrastructure and datacenter design. Which drove me to the decision to accept a position with PernixData as Technology Evangelist. As the Technology Evangelist I’m responsible for helping the virtualization community understand PernixData’s Flash Virtualization Platform (FVP). And as the first international employee I also will be focusing on expanding the European organization. I will be starting at PernixData soon, can’t wait to start ================================================================================ Title: vSphere 5.1 update 1 release fixes Storage vMotion rename "bug" URL: https://frankdenneman.ai/2013-04-26-vsphere-5-1-update-1-release-fixes-storage-vmotion-rename-bug/ Date: 2013-04-26 vSphere 5.1 update 1 is released today which contains several updates and bug fixes for both ESXi and vCenter Server 5.1. This release contains the return of the much requested functionality of renaming VM files by using Storage vMotion. Renaming a virtual machine within vCenter did not automatically rename the files, but in previous versions Storage vMotion renamed the files and folder to match the virtual machine name. A nice trick to keep the file structure aligned with the vCenter inventory. However engineers considered it a bug and “fixed” the problem. Duncan and I pushed hard for this fix, but the strong voice of the community lead (thanks for all who submitted a feature request) helped the engineers and product managers understand that this bug was actually considered to be a very useful feature. The engineers introduced the “bugfix” in 5.0 update 2 end of last year and now the fix is included in this update for vSphere 5.1 Here’s the details of the bugfix: vSphere 5 Storage vMotion is unable to rename virtual machine files on completing migration In vCenter Server , when you rename a virtual machine in the vSphere Client, the VMDK disks are not renamed following a successful Storage vMotion task. When you perform a Storage vMotion task for the virtual machine to have its folder and associated files renamed to match the new name, the virtual machine folder name changes, but the virtual machine file names do not change. This issue is resolved in this release. To enable this renaming feature, you need to configure the advanced settings in vCenter Server and set the value of the provisioning.relocate.enableRename parameter to true. Read the rest of the vCenter 5.1 update 1release notes and ESXi 5.1 update 1 release notes to discover other bugfixes ================================================================================ Title: Awesome read: Storage Performance And Testing Best Practices URL: https://frankdenneman.ai/2013-04-24-awesome-read-storage-performance-and-testing-best-practices/ Date: 2013-04-24 The last couple of days I’ve been reading up on EMC VPLEX technology as I’m testing VPLEX metro with SIOC and Storage DRS. Yesterday I discovered a technical paper called “EMC VPLEX: Elements Of Performance And Testing Best Practices Defined” and I think this paper should be read by anyone who is interested in testing storage or even wanting to understand the difference between workloads. Even if you do not plan to use EMC VPLEX the paper delivers some great insights about IOPS versus MB/s. What to expect when testing for transactional-based workloads and throughput-based workload? Here’s a little snippet: “Let’s begin our discussion of VPLEX performance by considering performance in general terms. What is good performance anyway? Performance can be considered to be a measure of the amount of work that is being accomplished in a specific time period. Storage resource performance is frequently quoted in terms of IOPS (IO per second) and/or throughput (MB/s). While IOPS and throughput are both measures of performance, they are not synonymous and are actually inversely related – meaning if you want high IOPS, you typically get low MB/s. This is driven in large part by the size of the IO buffers used by each storage product and the time it takes to load and unload each of them. This produces a relationship between IOPS and throughput as shown in Figure 1 below.” Although it’s primarily focused on VPLEX, the paper helps you understand the different layers of a storage solution and how each layer affects performance. Another useful section is the overview of good benchmark software which describes the basic operation of each listed benchmark program. The paper is very well written and I bet even a joy to read for both the beginner as well as the the most hardened storage geek. Download the paper here. ================================================================================ Title: Migrating VMs between DRS clusters in an elastic vDC URL: https://frankdenneman.ai/2013-04-17-migrating-vapps-between-drs-clusters-in-an-elastic-vdc/ Date: 2013-04-17 In the article “Migrating datastore clusters by changing storage profiles in a vCloud“ I closed with the remark that vCD is not providing an option to migrate virtual machines between compute clusters that are part of an elastic vDC. Fortunately my statement was not correct. Tomas Fojta pointed out that vCD does provide this functionality. Unfortunately this feature is not exposed in the vCloud organization portal but in the system portal of the vCloud infrastructure itself. In other words, to be able to use this functionality you need to have system administrator privileges. In the previous article, I created the scenario where you want to move virtual machines between two sites. Site 1 contains compute cluster “vCloud-Cluster1” and datastore cluster “ DSC-Site-1”. Site 2 contains “vCloud-Cluster2” and datastore cluster “DSC-Site-2” . By changing the VM storage profile from Site-1 to Site-2, we have vCD instruct vSphere to storage vMotion the virtual machine disk files from one datastore cluster to another. Now at this point we need to migrate the compute state of the virtual machine. Migrate virtual machine between clusters Please note that vCD refers to clusters as resource pools. To migrate the virtual machine between clusters, log into the vCloud director and select the system tab. Go to the vSphere resources and select Resource Pools menu option. The UI displays the clusters that are a part of the Provide vDC. Select the cluster a.k.a. resource pool in which the virtual machine resides. Select the virtual machine to migrate, right click the virtual machine to have vCD display the submenu and select the option “Migrate to…” The user interface allows you to choose how you want to select the destination resource pool for the virtual machine: Either automatic and let vCD select the resource pool for you, or select the appropriate resource pool manually. When selecting automatic vCD selects the cluster with the most unreserved resources available. If the virtual machine happens to be in the cluster with the most unreserved resources available vCD might not move the virtual machine. In this case we want to place the virtual machine in site 2 so that means we need to select the appropriate cluster. We select vCloud-Cluster2 and click on OK to start the migration process. vCD instructs vSphere to migrate the virtual machine between clusters with the use of vMotion. In order to use vMotion, both clusters need to have access to the datastore on which the virtual machine files reside. vCD does not use “enhanced’ vMotion where it can live migrate between host without being connected shared storage. Hopefully we see this enhancement in the future. When we log into vSphere we can verify if the life migration of the virtual machine was completed. Select the destination cluster, in this case that would be vCloud-Cluster2, go to menu option Monitor, select tasks and click on the entry “Migrate virtual machine” In the lower part of the screen, you get more detailed information of the Migrate-virtua-machine entry. As you can seem the virtual machine W2K8_RS_SP1 is migrated between servers 10.27.51.155 and 10.27.51.152. As we do not change anything to the storage configuration, the virtual machine files remains untouched and stay on the same datastore. To determine if vCD has updated the current location of the virtual machine, log into vCD again, go to the menu option “Resource Pools” and select the cluster chosen as destination as the previously org cluster. ================================================================================ Title: 3 common questions about DRS preferential VM-Host affinity rules URL: https://frankdenneman.ai/2013-04-15-3-common-questions-about-drs-preferential-vm-host-affinity-rules/ Date: 2013-04-15 On a regular basis I receive questions about the behavior of DRS when dealing with preferential VM to Host affinity rules. The rules configured with the rule set “should run on / should not run on” are considered preferential. Meaning that DRS prefers to satisfy the requirements of the rules, but is somewhat flexible to run a VM outside the designated hosts. It is this flexibility that raises questions; lets see how “loosely” DRS can operate within the terms of conditions of a preferential rule: Question 1: If the cluster is imbalanced does DRS migrate the virtual machines out of the DRS host group? DRS only considers migrating the virtual machines to hosts external to the DRS host group if each host inside the group is 100% utilized. And if the hosts are 100% utilized, then DRS will consider virtual machines that are not part of a VM-Host affinity rule first. DRS will always avoid violating an affinity rule Question 2: When a virtual machine is powered on, will DRS start the virtual machine on a host external to the DRS host group? By default DRS will start the virtual machine on hosts listed in the associated Host DRS group. If all hosts are 100% utilized – or – if they do not meet the virtual machine hardware requirements such as datastore or network connectivity, then DRS will start the virtual machine on a host external to the Host DRS group. Question 3: If a virtual machine is running on a host external to the associated host DRS group, shall DRS try to migrate the virtual machine to a host listed in the DRS host group? The first action DRS triggers during an invocation is to determine if an affinity rules is violated. If a virtual machine is running on a host external to the associated Host DRS group then DRS will try to correct this violation. This move will have the highest priority ensuring that this move is carried out during this invocation. ================================================================================ Title: Migrating datastore clusters by changing storage profiles in a vCloud URL: https://frankdenneman.ai/2013-04-12-migrating-datastore-clusters-by-changing-storage-profiles-in-a-vcloud/ Date: 2013-04-12 vCloud director 5.1 supports the use of both storage profiles and Storage DRS. One of the coolest features and unfortunately relatively unknown is the ability to live migrate virtual machines between datastore clusters by changing the storage profile in the vCloud director portal. In my lab I’ve set up a provider vDC that contains two compute clusters. Each compute cluster connects to two datastore clusters. Datastore Cluster “vCloud-SDC-Gold” is compatible with the VM storage profile “vCloud-Gold-Storage”, while Datastore Cluster “vCloud-SDC-Silver” is compatible with the VM storage profile “vCloud-Silver-Storage”. When creating a vApp the default storage profile of the organization vDC is applied to the vApp and all its virtual machines. In this case, the VM storage profile Gold is applied to all the virtual machines in the vApp. You can determine which VM Storage Profile is associated with the virtual machine by selecting the properties of the virtual machine in the “My Cloud” tab. Please note that vCloud Director does not show the VM Storage Profile at the vApp level! By selecting the drop-down box, all storage profiles that are associated with the organization vCD are displayed. By selecting the Storage Profile “vCloud-Silver-Storage” vCloud Director determines that the virtual machine is stored on a datastore that is not compatible with the associated storage profile. In other words the current configuration is violating the storage level policy. To correct this violation, vCloud director instructs vSphere to migrate the virtual machine via Storage vMotion to a datastore that is compatible with the VM storage Profile. In this case the datastore cluster “vCloud-DSC-Silver” is selected as the destination. Storage DRS determines the most suitable datastore by using its initial placement algorithm and selects the datastore that has the most amount of free space and the lowest I/O load. To demonstrate the feature, I selected the virtual machine “W2K8_R2-SP1”. The VM storage profile “vCloud-Gold-Storage” is applied and Storage DRS determined that the datastore “nfs-f-vcloud03” of the datastore cluster “vCloud-DSC-Gold” was the most suitable location. By changing the Storage Profile to “vCloud-Silver-Storage” vCloud director instructed vSphere to migrate it to the datastore cluster that is compatible with the newly associated VM storage profile. When logging into the vCenter server managing the ESXi hosts the following task is running: After the task is complete, vCenter shows that the virtual machine is now stored on datastore “nfs-f-vcloud06” in the datastore cluster “vCloud-DSC-Silver”. The power of abstraction The abstraction layer of vCloud Director makes this possible. When changing the storage profile directly on the vSphere layer, nothing happens. vSphere will not migrate the virtual machine to the appropriate datastore cluster that is compatible with the selected VM storage profile. Useful for stretched clusters? The reason why I was looking into this feature in my lab is due to an conversation with my esteemed colleagues Lee Dilworth and Aidan Dalgleish. We were looking to an alternative scenario for a stretched cluster. By leveraging the elastic vDC feature of vCloud director, a seperate DRS cluster is created in each site. Due to the automatic initial placement engine on the compute level, we needed to find a construct that can provide us a more deterministic method of virtual machine placement. We immediately thought of the VM profile storage feature. Create two datastore clusters, one per site and associate a profile storage based on site name to the respective datastore clusters. When creating the vApp, just select the site-related Storage Profile to place the virtual machine in a specific site. Due to the compatibility check, vCloud Director determines that in order to be compliant with the storage profile it places the virtual machine on the compute cluster in the same site. For example, if you want to place a virtual machine in site 1, select the VM storage Profile “site 1”. vCloud director determines that the virtual machine needs to be stored in datastore cluster “DSC-Site-1”. The compute cluster Site-1 is the only compute cluster connected to the datastore cluster, therefor both the compute and storage configuration of the virtual machine is stored in Site 1. This configuration works perfect if you want to simplify initial placement if you have multiple sites/locations and you always want to keep the virtual machine in the same site. However this solution might not be optimal for a Stretched cluster configuration where failover to another site is necessary. Connectivity to all datastores necessary As this feature uses storage vMotion instead of cross-host/datastore vMotion, means that the cluster needs to be connected to both datastore clusters. When selecting the different storage profile, the storage state is migrated to another datastore cluster. However it doesn’t move the compute state of the virtual machine. This means that storage is moved to site B, while the compute state is still in Site A. vCloud director does not provide an option to migrate the virtual machine to a different compute cluster within the provider vDC. You can either solve it by logging into the vCenter server that manages the ESXi hosts and manually vMotion the virtual machines to cluster in Site B, or power-off the virtual machine in vCloud Director, then change the storage profile and power-on the virtual machine. Both “solutions” are not very enterprise-level scenario’s therefor I think this is not yet suitable as a stretched cluster configuration ================================================================================ Title: VMworld 2013 - Call of Papers deadline ends today URL: https://frankdenneman.ai/2013-04-12-vmworld-2013-call-of-papers-deadline-ends-today/ Date: 2013-04-12 Just a reminder here on submitting VMworld sessions. The deadline is coming up quickly. If you haven’t submitted yet, you have still some hours left to submit a Session Proposal for VMworld 2013. Submit your session today! ================================================================================ Title: vMotion over layer 3? URL: https://frankdenneman.ai/2013-04-09-vmotion-over-layer-3/ Date: 2013-04-09 This question regularly pops up on twitter and the community forums. And yes it works but VMware does not support vMotion interfaces in different subnets. The reason is that this can break functionality in higher-level features that rely on vMotion to work. If you think Routed vMotion (vMotion interfaces in different subnets) is something that should be available in the modern datacenter, please fill out a feature request. The more feature requests we receive; the more priority can be applied to the development process of the feature. ================================================================================ Title: Saving a Resource Pool Structure web client feature not suitable for vCD environments URL: https://frankdenneman.ai/2013-04-08-saving-a-resource-pool-structure-web-client-feature-not-suitable-for-vcd-environments/ Date: 2013-04-08 Last week I published the article “Saving a Resource Pool Structure” describing the RP-tree backup and restore feature of vSphere 5.1 web client. Multiple people immediately asked if the feature keeps the Managed Object Reference ID (MoRef) of the resource pools identical when it restores the resource pool tree? This is important for vCloud Director as it creates a relationship between vCloud Director objects organization vCD and the vSphere level resource pool. vCloud Director ties the org vCD UUID with the vSphere resource pool Moref id within vCD database. For more information read Chris his post: “Gotcha: Disabling VMware DRS with vCloud Director”. Unfortunately the feature just captures the old tree structure and rebuilds a new tree structure. I tested it by using William Lam’s custom Perl script called moRefFinder.pl. Please visit Williams site to download his script. Then I proceeded to backup and restore the resource pool tree. vCenter showed the follow commands being processed. Then I checked if the MoRef ID was the same as prior to disabling DRS. As shown, the current MoRef ID of the “00-Infra-mgmt” resource pool is 137 contrary to MoRef ID of 129 before disabling DRS. Therefor you should not use this feature when planning to backup and restore the resource pool used by VCD for its organization vCD structures. ================================================================================ Title: Saving a Resource Pool Structure URL: https://frankdenneman.ai/2013-04-05-saving-a-resource-pool-structure/ Date: 2013-04-05 During a troubleshooting exercise of a problem with vCenter I needed to disable DRS to make sure DRS was not the culprit. However a resource pool tree exisited in the infrastructure and I was not looking forward reconfiguring all the resource allocation settings again and documenting which VM belonged to which resource pool. The web client of vSphere 5.1 has a cool feature that helps in these cases. When deactivating DRS (Select cluster, Manage, Settings, Edit, deselect “Turn ON vSphere DRS”) the user interface displays the following question: Backup resource pool tree Click “Yes” to backup the tree and select an appropriate destination for the resource pool tree snapshot file. This file uses the name structure clustername.snapshot and should the file size be not bigger than 1 or 2 KB. Restore resource pool tree When enabling DRS on the cluster, the User interface does not ask the question to restore the tree. In order to restore the tree, enable DRS first and select the cluster in the tree view. Open the submenu by performing a right-click on the cluster, expand the “All vCenter Actions” and select the option “Restore Resource Pool Tree…” A window appears and click browse in order to select the saved resource pool tree snapshot and click on OK vCenter restores the tree, the resource pool settings (shares, reservations limits) and moves the virtual machines back to the resource pool they were placed in before disabling DRS. If you want to save the complete vCenter inventory configuration I suggest you download the fling “InventorySnapshot”. Update: If you want to use this tool to backup and restore resource pool trees used by vCloud Director, please read this article: Saving a Resource Pool Structure web client feature not suitable for vCD environments ================================================================================ Title: Elastic vDC and how to span a provider vDC across multiple DRS clusters URL: https://frankdenneman.ai/2013-03-29-elastic-vdc-and-how-to-span-a-provider-vdc-across-multiple-drs-clusters/ Date: 2013-03-29 vCloud director 5.1 provides the ability to create elastic vDC which allows an organization vDC to consume resources from multiple DRS clusters. By having the provider vDC abstract the resources from multiple DRS clusters, its simpler to grow capacity when needed. Before elastic vDC, a new provider vDC and Org vDCs needed to be created when an org vDC wanted to grow beyond the capacity of the provider vDC. With Elastic vDC you just add new clusters when needed and allow the Provider vDC to manage initial placement of vApps. During research of elastic vDCs I discovered that the way to span a provider vDC isn’t that intuitive. In order to save you some time, here are the steps to create a provider vDC that spans multiple DRS clusters. Create a Provider vDC, give it a name and select the highest supported hardware version. If you run a homogenous environment with solely 5.1 ESX hosts I highly recommend changing it to Hardware Version 9. If the clusters run different ESX versions, lower the hardware version to the appropriate supported level. Please note that the provider vDC is responsible for initial placement of the vApp. It will place the vApp on the cluster that contains the most available “unreserved” compute resources and storage resources. It is possible that vApps of the same organization run on different ESX versions. Select Resource pool. This screen is a little bit ambiguous. The user interface “talks” about resource pools, but that doesn’t mean you cannot select a complete DRS cluster for consumption by the provider vDC. A DRS cluster is in essence a resource pool, the root resource pool for all its child resource pools. So don’t worry if you want to select an entire cluster, in matter of fact, when you select the vCenter it shows the DRS clusters as well as the resource pools. In this example, the vCenter contains two DRS clusters; vCloud-Cluster1 and vCloud-Cluster2. The DRS cluster vCloud-Cluster2 contains a resource pool called RP1. Unfortunately the user interface does not use any icons to differentiate between clusters and resource pools, but shows a vCenter path notation. As RP1 is the child resource pool of vCloud-Cluster2, the vCenter path is as follows: vCloud-Cluster2/RP1. Unfortunately the interface only allows to select a single resource pool or cluster, therefor I select the vCloud-Cluster1 and select next. Select an appropriate Storage profile and click on next. The ready to complete screen displays an overview of your selected configuration. Click on Finish to create the Provider vDC. At this point in time, the provider vDC maps to only one DRS cluster. To add additional clusters, go to the Manage and Monitor tab and select Provider vDCs. Click on the provider vDC and select the resource pools tab Click on the green plus icon to add another DRS cluster. The attach resource pool window is displayed and you can select another cluster from the same vCenter as the primary cluster. Please note that a provider vDC can only span clusters managed by the same vCenter server. Click on Finish to add the DRS cluster to the provider vDC. The Provider vDC is now able to provider resources from multiple DRS clusters. In vCloud Director 5.1 both the Pay-as-You-Go and Allocation Pool model org vCD are able to consume resources from an elastic vDC. In order to allow the Allocation Pool model to leverage an Elastic vDC changes needed to be made. Massimo Re Ferrè wrote an extensive post about the changes of the different allocation models in vCloud director 5.1. ================================================================================ Title: Would you be interested in Storage-level reservations? URL: https://frankdenneman.ai/2013-03-26-would-you-be-interested-in-storage-level-reservations/ Date: 2013-03-26 In todays world it’s quite common to virtualize higher priority / tier-1 applications and services. These applications and services are usually subject to service level agreements that typically include requirements for strong performance guarantees. For the compute resources (CPU and Memory) we are relying on the virtualization layer to give us that resource allocation solution by setting reservation, shares and limits. You might want to ensure that the storage requirements of these virtual machines are met and when contention for storage resources occurs these workloads are not impacted. Today vSphere offers Storage I/O Control (SIOC) to allocates I/O resources based on the virtual machine priority if datastore latency is exceeded. Shares identify priority while limits restrict the amount of IOPS for a virtual machine. Although these are useful controls it does not provide a method to define a minimum amount of IOPS that is available all the time to the application. Providing lots of shares to these virtual machines can solve help to meet the SLA, however continuously calculating the correct share value in a highly dynamic virtual datacenter is cumbersome and complex job. Storage level reservations Therefore we are working on Storage level reservations. A storage reservation allows you to specify a minimum number of IOPS that should be available to the virtual machine at all times. This allows the virtual machine to make minimum progress in order to comply with the service level agreement. In a relative closed environment such as the compute layer its fairly easy to guarantee a minimum level of resource availability, but when it comes to a shared storage platform new challenges arise. The hypervisor owns the computes resource and distributes it to the virtual machine it’s hosting. In a shared storage environment we are dealing with multiple layers of infrastructure, each susceptible to congestion and contention. And then there is the possibility of multiple external storage resource consumers such as non-virtualized workloads using the same array impacting the availability of resources and the control of distributing the resources. These challenges must be taken into account when developing storage reservations and we must understand how stringent you want the guarantee to be. One of the questions we are dealing with is whether you would like a strict admission control or a relaxed admission control. With strict admission control, a virtual machine power-on operation is denied when vSphere cannot guarantee the storage reservation (similar to compute reservations). Relaxed admission control turns storage reservations into a share-like construct, defining relative priority at times where not enough IOPS are available at power-on. For example: Storage reservation on VM1 = 800 and VM2 = 200. At boot 600 IOPS are available; therefore VM1 gets 80% of 600 = 480, while VM2 gets 20%, i.e. 120 IOPS. When the array is able to provide more IOPS the correct number of IOPS are distributed to the virtual machines in order to to satisfy the storage reservation. In order to decide which features to include and define the behavior of storage reservation we are very interested in your opinion. We have created a short list of questions and by answering you can help us define our priorities during the development process. I intentionally kept the question to a minimum so that it would not take more than 5 minutes of your time to complete the survey. Disclaimer As always, this article provides information about a feature that is currently under development. This means this feature is subject to change and nor VMware nor I in no way promises to deliver on any features mentioned in this article or survey. Any other ideas about storage reservations? Please leave a comment below. The survey is closed, thanks for your interest in participating ================================================================================ Title: Hello world! Again URL: https://frankdenneman.ai/2013-03-25-hello-world-2/ Date: 2013-03-25 During my holiday, frankdenneman.nl got some unwanted attention. I’m currently in the process of rebuilding the site. Stay tuned for new updates! ================================================================================ Title: WOW, voted number 2 of top virtualization blogs! URL: https://frankdenneman.ai/2013-03-12-wow-voted-number-2-of-top-virtualization-blogs/ Date: 2013-03-12 Voted number 2 of top virtualization blogs As many other IT-addicts, the first thing I do is pick up my phone to see what’s new on twitter, google+ and facebook and to my surprise I received a lot of direct messages and mentions congratulating on taking the second spot on the top 25 virtualization blog list. WOW talk about excitement! From being drowsy to uber-hyped in under a millisecond. Thanks for voting me! I really appreciate the recognition. I love to blog and write articles and when I’m not researching I’m thinking of topics I can cover. Reaching the number 2 spot proves I’m doing something you all like. But actually I want to thank you for taking the time to vote on any of the top 25 blogs. Everybody spends a great deal of time researching and writing articles, getting votes is a great way to receive acknowledgement for your hard work. A big thank you goes out to Eric for organizing this competition again. Awesome work and thanks for putting in all the effort. Viewing the stats it shows that this event is becoming more and more an industry event, organized by community members for community members. Great stuff. John, David, Simon similar to last year, great vChat. A delight to watch! BTW, thank you for the compliments! It’s always cool to hear some background details of the top 25 bloggers. I encourage you to watch the special vChat it’s great entertainment! Congrats to Duncan for taking the number 1 spot. Well deserved! I know how much effort you put into the blog. Outstanding stuff. Congrats to the rest of the top 25 and a special congrats goes out to Cormac. Well deserved to enter in the top 10. If you are on twitter make sure you follow each and everyone of the top 25. These guys are a special bunch, all passionately about virtualization and great bunch of people in general. Here is the list of the top 25 on twitter: Rank Name Twitter 01 Duncan Epping @DuncanYB 02 Frank Denneman @FrankDenneman 03 Scott Lowe @scott_lowe 04 Eric Sloof @ESloof 05 Chad Sakac @SakacC 06 William Lam @LamW 07 Mike Laverick @Mike_Laverick 08 Alan Renouf @AlanRenouf 09 Cormac Hogan @VMwareStorage 10 Eric Siebert @EricSiebert 11 Jason Boche @JasonBoche 12 Chris Wahl @Wahlnetwork 13 Vaugh Stewart @vStewed 14 Andre Leibovici @AndreLeibovici 15 Luc Dekens @LucD 16 Vladan Seget @vladan 17 Nick Howell @that1guynick 18 Stephen Foskett @SFoskett 19 Gabrie van Zanten @gabvirtualworld 20 Tommy Trogden @vtexan 21 Michael Webster @vcdxnz001 22 Kendrick Coleman @KendrickColeman 23 Simon Seagrave @kiwi_si 24 Derek Seaman @vDerekS 25 Brian Madden @BrianMadden ================================================================================ Title: Distribution of resources based on shares in a Resource pool environment URL: https://frankdenneman.ai/2013-02-28-distribution-of-resources-based-on-shares-in-a-resource-pool-environment/ Date: 2013-02-28 Unfortunately Resource pools seem to have a bad rep, pair them with the word shares and we might as well call death and destruction to our virtual infrastructure while we’re at it. Now in reality shares and resource pools are an excellent way of maintaining a free flow of resource distribution to the virtual machine who require these resources. Some articles, and the examples I use in the book are meant to illustrate the worst-case scenario, but unfortunately those examples are perceived to be the default method of operation. Let me use an example: In a cluster two resource pool exist, resource pool gold is used for production and is configured with a high share level. Resource pool bronze is used for development and test and is configured with a low share level. Meaning that the ratio of shares is 4:1. Now this environment contains a 8:1 ratio when it comes to virtual machines. The gold resource pool contains 320 virtual machines and the bronze resource pool contains 40 virtual machines. The cluster contains 200 GB of memory and 200 GHz of CPU, this means that the each virtual machine in the gold resource pool has access to 0.5 MHz and 0.5 GB right? Well yes BUT…. (take a deep breath because this will be one long sentence)… Only in the scenario where all the virtual machines in the environment are 100% utilized (CPU and memory), where the ESXi hosts can provide enough network bandwidth and storage bandwidth to back the activity of the virtual machines, no other operations are active in the environment and where all virtual machines are configured identically in size and operating system than yes that happens. In all other scenarios a more dynamic distribution of resources is happening. The distribution process Now let’s deconstruct the distribution process. First of all let’s refresh some basic resource management behavior and determine the distinction between shares and reservations. A share is a relative weight, identifying the priority of the virtual machine during contention. It is only relative to its peers and only relative to other active shares. This means that using the previous scenario, the resource pool shares compete against each other and the virtual machine shares inside a single resource pool compete against each other. It’s important to note that only active shares are used when determining distribution. This is to prevent resource hoarding based on shares, if you do not exercise you shares, you lose the rights to compete in the bidding of resources. Reservations are the complete opposite, the resource is protected by a reservation the moment you used it. Basically the virtual machine “owns” that resources and cannot be pressured to relinquish it. Therefor reservations can be seen as the complete opposite of shares, a basic mechanism to hoard resources. Back to the scenario, what happens in most environments? First of all the demand is driven from bottom to top, that means that virtual machines ask their parent if they can have the resources they demand. The resource pool will ask the cluster for resources. The distribution is going in the opposite direction; top to bottom and that’s where activity and shares come in to play. If both resource pools are asking for more resources than the cluster can supply, then the cluster needs to decide which resource pool gets the resources. As resource pool (RP) Gold contains a lot more virtual machines its safe to assume that RP Gold is demanding more resources than RP Bronze. The total demand of the virtual machines in RP Gold is 180 GB while the virtual machines in RP Bronze demand a total of 25GB. In total the two RP’s demand 205GB while the cluster can only provide 200GB. Notice that I split up demand request into two levels, VMs to RP, RP to cluster. The cluster will take multiple passes to distribute the resources. In the first pass the resources are distributed according to the relative share value, in this case 4:1 that means that RP Gold is entitled to 160GB of memory (4/5 of 200) and RP Bronze 40GB (1/5 of 200). While RP Bronze gets awarded 40GB, it is only requesting 25GB, returning the excessive 15GB of memory to the cluster. (Remember if you don’t use it, you lose it) As the cluster has a “spare” 15GB to distribute it executes a second distribution pass and since there are no other resource consumers in the cluster it awards these 15GB of memory resources to the claim of RP Gold. This leads to a distribution of 175GB to Resource Pool Gold and 25GB of memory of Resource Pool Bronze. Please note that in this scenario I broke down the sequence into multiple passes, in reality these multiple passes are contained within a (extremely fast) single operation. The moment resource demand changes, a new distribution of resources will occur. Allowing the cluster resources to satisfy the demand in the most dynamic way. The same sequence is happening in the resource pool itself; virtual machines receive their resources based on their activity and their share value. Hereby distributing the resources “owned” by the resource pool to the most important and active virtual machines within the pool If no custom share values are configured on the virtual machine itself, the virtual machine CPU and memory configuration along with the configured share level will determine the amount of shares the virtual machine posses. For example a virtual machine configured with a normal share value and a configuration of 2vCPU and 2GB will posses 2000 shares of CPU and 20480 shares of memory. For more info about share calculation please consult the VMware vSphere 5.1 resource management guide, table 2-1 page 12. (share values have not been changed since the introduction, therefor it’s applicable to ESX and all vSphere versions) Key takeaway I hope that by using this scenario it’s clear that shares do not hoard resources. The most important thing to understand that it all comes down to activity. Supply is to meet its demand, whenever demand changes new distribution of resources are executed. And although the number of the virtual machines might not be comparable to the share ratio of the resource pools, it’s the activity that drives the dynamic distribution. Mixing multiple resource allocation settings In theory an unequal distribution of resources is possible, in reality the presences of more virtual machines equal more demand. Now architecting an environment can be done in many ways, a popular method is to design for worst-case scenario. Great designs usually do not rely on a single element and therefor a configuration with the use of multiple resource allocation settings (reservations, shares and limits) might provide the level of performance throughout the cluster. If you are using a cluster design as described in the scenario and you want to ensure that load and smoke testing do not interfere with the performance levels of the virtual machines in RP Gold, than a mix of resource pool reservations and shares might be a solution. Determine the amount of resources that need to be permanently available to your production environment and configure a reservation on RP Gold. Hereby creating a pool of guaranteed resources and a pool for burstability. Allowing the remaining resources to be allocated by both resource pools on a dynamic and opportunistic basis. You can even further restrict the use of physical resources to the RP bronze by setting a limit on the resource pool. Longing for SDDC? Start with resource pools! Its too bad resource pools got a bad rep and maybe I have been a part of it by only describing worst-case scenarios. When understanding resource pool one recongnizes that resource pools are a crucial element in the Software Defined Datacenter. By using the correct mix of resource allocation settings you can provide an abstraction layer that is able to isolate resources for specific workloads or customers. Resources can be flexibly added, removed, or reorganized in resource pools as per changing business needs and priorities. All this is available to you without the need for tinkering with low-level settings on virtual machines or using power-cli scripts to adjust the shares on resource pools. ================================================================================ Title: There is a new fling in town: DRMdiagnose URL: https://frankdenneman.ai/2013-02-28-there-is-a-new-fling-in-town-drmdiagnose/ Date: 2013-02-28 This week the DRMdiagnose fling is published. Produced by the resource management team and just in case you are wondering, DRM stands for Distributed Resource Manager; the internal code for DRS. Download DRMdiagnose at the VMware fling site. Please note that this fling only works on vSphere 5.1 environments Purpose of DRMdiagnose This tool is created to understand the impact on the virtual machines own performance and the impact on other virtual machines in the cluster if the resource allocation settings of a virtual machine are changed. DRMdiagnose compares the current resource demand of the virtual machine and suggest changes to the resource allocation settings to achieve the appropriate performance. This tool can assist you to meet service level agreements by providing feedback on desired resource entitlement. Although you might know what performance you want for a virtual machine, you might not be aware of the impact or consequences an adjustments might have on other parts of the resource environment or cluster policies. DRMdiagnose provides recommendations that provides the meets the resource allocation requirement of the virtual machines with the least amount of impact. A DRMdiagnose recommendation could look like this: Increase CPU size of VM Webserver by 1 Increase CPU shares of VM Webserver by 4000 Increase memory size of VM Database01 by 800 MB Increase memory shares of VM Database01 by 2000 Decrease CPU reservation of RP Silver by 340 MHz Decrease CPU reservation of VM AD01 by 214 MHz Increase CPU reservation of VM Database01 by 1000 MHz How does it work DRMdiagnose reviews the DRS cluster snapshot. This snapshot contains the current cluster state and the resource demand of the virtual machines. The cluster snapshot is stored on the vCenter server. These snapshot files can be found: vCenter server appliance: /var/log/vmware/vpx/drmdump/cluster_X_/ vCenter server Windows 2003: %ALLUSERSPROFILE%\Application Data\VMware\VMware VirtualCenter\Logs\drmdump\cluster_X_\ vCenter server Windows 2008: %ALLUSERSPROFILE%\VMware\VMware VirtualCenter\Logs\drmdump\cluster_X_\ The fling can be run in three modes: Default: Given a link to a drmdump, it lists all the VMs in the cluster, and their current demands and entitlements. Guided: Given a link to a drmdump, and a target allocation for the VM, generates a set of recommendations to achieve it. Auto: Given a link to a drmdump, generates a recommendation to satisfy the demand of the most distressed VM (the VM for which the gap between demand and entitlement is the highest). Two things to note: One: The fling does not have run on the vCenter server itself. Just install the fling on your local windows or linux system, copy over the latest drmdump file and run the fling. And second the drmdump file is zipped (GZ), unzip the file first to and run DRMdiagnose against the .dump file. A “normal” dumpfile should look like this: How to run: Open a command prompt in windows: This command will provide the default output and provide you a list with CPU and Memory demand as well as entitlement. Instead of showing it on screen I chose to port it to a file as the output contains a lot of data. A next article will expand on auto-mode and guided-mode use of DRMdiagnose. In the mean time, I would suggest to download DRMdiagnose and review your current environment. ================================================================================ Title: Do you use vApps? URL: https://frankdenneman.ai/2013-02-26-do-you-use-vapps/ Date: 2013-02-26 We’re interested in learning more about how you use vApps for workload provisioning today and how you envision it evolving in the future. If you have a couple of spare minutes, please fill out these 15 questions: http://www.surveymethods.com/EndUser.aspx?FFDBB7AEFDB5AAAAFB Thanks! ================================================================================ Title: Reserve all guest memory (all locked) URL: https://frankdenneman.ai/2013-02-21-reserve-all-guest-memory-all-locked/ Date: 2013-02-21 Some applications do not perform well when memory is reclaimed from the virtual machine. Most users set a virtual machine memory reservation to prevent memory reclamation and to ensure stable performance levels. Memory reservation settings are static, meaning that when you change the memory configuration of the virtual machine itself the memory reservation remains the same. If you want to keep the reservation equal to the virtual machine memory reservation, the UI (included in both the vSphere client and the web client) offers the setting: “Reserve all guest memory (all locked)”. This setting is linked to the virtual machine memory configuration. The memory reservation is immediately readjusted when the memory configuration changes. Increase the memory size and the memory reservation is automatically increased as well. Reduce the memory size of a virtual machine, and the reservation is immediately reduced. The behavior is extremely useful when using the vSphere client as management tool. Within the vSphere client the memory configuration and the memory reservation settings do not share the same screen. While changing the memory configuration one can easily forget to adjust the memory reservation. The web client is redesigned and shows the memory configuration and reservation in a single screen. Yet having a setting that automates and controls alignment of memory configuration and reservation reduce the change for human error. ================================================================================ Title: PernixData Flash Virtualization Platform will revolutionize virtual infrastructure ecosystem design URL: https://frankdenneman.ai/2013-02-20-pernixdata-flash-virtualization-platform-will-revolutionize-virtual-infrastructure-ecosystem-design-2/ Date: 2013-02-20 A couple of weeks ago I was fortunate enough to attend a tech preview of PernixData Flash Virtualization Platform (FVP). Today PernixData exited stealth mode so we can finally talk about FVP. Duncan already posted a lengthy article about PernixData and FVP and I recommend you to read it. At this moment a lot of companies are focusing on flash based solutions. PernixData distinguishes itself in today’s flash focused world by providing a new flash based technology but that is not a storage array based solution or a server-bound service. I’ll expand on what FVP does in a bit, let’s take a look at the aforementioned solutions. The solutions have drawbacks. A Storage array based flash solution is plagued by common physics. Distance between the workload and the fast medium (flash) generates a higher latency than when the flash disk is placed near the workload. Placing the flash inside a server provides the best performance but it must be shared between the hosts in the cluster to become a true enterprise solution. If the solution breaks important functions such as DRS and vMotion than the use case of this technology remains limited. FVP solves these problems by providing a flash based data tier that becomes a cluster-based resource. FVP virtualizes server side flash devices such as SSD drives or PCIe flash devices (or both) and pools these resources into a data tier that is accessible to all the hosts in the cluster. One feature that stands out is remote access. By allowing access to remote devices, FVP allows the cluster to migrate virtual machines around while still offering performance acceleration. Therefor cluster features such as HA, DRS and Storage DRS are fully supported when using FVP. Unlike other server based flash solutions, FVP accelerates both read and write operations. Turning the flash pool in to a “data-in-motion-tier”. All hot data exists in this tier, thus turning the compute layer into an all-IOPS-providing platform. Data that is at rest is moved to the storage array level, turning this layer into the capacity platform. By keeping the I/O operations as close to the source (virtual machines) as possible, performance is increased while reducing the traffic load to the storage platform as well. By filtering out read I/Os the traffic pattern to the array is changed as well, allow the array to focus more on the writes Another great option is the ability to configure multiple protection levels when using write-back. Data is synchronously replicated to remote devices. During the tech preview Satyam and Poojan provided some insights on the available protection levels, however I’m not sure if I’m allowed to share these publically. For more information about FVP visit Pernixdata.com The beauty of FVP is that its not a virtual appliance and that it does not require any agents installed in the guest OS. FVP is embedded inside the hypervisor. Now this for me is the key to believe that this ”data-in-motion-tier” is only the beginning of PernixData. By having insights in the hypervisor and understanding the dataflow of the virtual machines, FVP can become a true platform that accelerates all types of IOPS. I do not see any reasons why FVP is not able to replicate/encrypt/duplicate any type of input and output of a virtual machine. :) As you can see I’m quite excited by this technology. I believe FVP is as revolutionary/disruptive as vMotion. It might not be as “flashy” (forgive the pun) as vMotion but it sure is exciting to know that the limitation of use-cases is actually the limitation of your imagination. I truly believe this technology will revolutionize virtual infrastructure ecosystem design. ================================================================================ Title: Voting for the 2013 top virtualization blogs - A year in review URL: https://frankdenneman.ai/2013-02-20-voting-for-the-2013-top-virtualization-blogs-a-year-in-review/ Date: 2013-02-20 When Eric Siebert opens up the voting for the top VMware & virtualization blogs you know another (blogging) year has passed. First of all I want to thank Eric for organizing this year in year out. I know he spends an awful lot of time on this. Thanks Eric! Its amazing to see that there are more than 200 blogs dedicated to virtualization and that each month new blogs appear. Unfortunately I don’t have the time to read them all but I do want to show my appreciation for the blog sites that I usually visit. Best newcomer is an easy one, Cormac Hogan. The content is absolutely great and he should be in the top 10. Then we have the usually suspects, my technical marketing colleagues and buddies: Alan Renouf, Rawlinson Rivera and William Lam. I start of the day by making coffee, checking my email and logging into yellow-bricks.com. It’s the de facto standard of the virtualization blogs. Duncan’s blog provide not only technical in-depth articles, but also insights in the industry. Who else? Eric Sloof of course! Always nice to read to find out that your white paper is published before you get the official word through company channels. ;) Two relative unknown blog sites but quality content: Erik Bussink and Rickard Nobel. These guys create awesome material. One blog that I’m missing in the list is the one from Josh Odgers. Great content. Hope to be able to vote for him next year. When reviewing content from others you end up reviewing the stuff you did yourself and 2012 was a very busy year for me. During the year I published and co-authored a couple of white papers such as the vSphere Metro Cluster Case Study, Storage DRS interoperability guide and vCloud Director Resource Allocation Models. I presented at a couple of VMUGS and at both VMword San Francisco and Europe. The resource pool best practice session was voted as one of the top 10 presentations of VMworld. And of course Duncan and I released the vSphere 5.1 Clustering Deepdive, also know as 50 shades of Orange. ☺ I believe it’s the best one of the series. In the mean time I ended up writing for the vSphere blog, appearing on a couple of podcast and writing a little over a 100 blog articles on frankdenneman.nl. I then to focus on DRS, Storage DRS, SIOC and vMotion but once in a while I like to write about something that gives a little insight peek of my life such as the whiteboard desk or the documentaries I like to watch. It seems you like these articles also as they are frequently visited. In my articles I try to give insights in the behavior of the features of vSphere, this to help you understand the impact of these features. Understanding the behavior allows you to match your design to the requirements and constrains of the project/virtual infrastructure your working on. During my years in the field I was always looking for this type of information, by providing this material I hope to help out my fellow architects. When publishing more than over 100 articles you tend to like some more than others. While it’s very difficult to choose individual articles, I enjoyed spending time on writing a series of articles on the same topic, such as the series Architecture and design of datastore clusters (5 posts) and Designing your (Multi-NIC) vMotion network (5 posts). But I also like the individual post: • vSphere 5.1 vMotion Deepdive • A primer on Network I/O Control • vSphere 5.1 Storage DRS load balancing and SIOC threshold enhancements • HA admission control is not a capacity management tool • Limiting the number of storage vMotions I hope you can spare a couple of minutes to cast your vote and show your appreciation for the effort these bloggers put into their work. Instead of picking the customary names please look back and review last year, think about the cool articles you read that helped you or sparked your interest to dive into the technology yourself. Thanks I can’t wait to watch the Top 25 countdown show Eric, John and Simon did in the previous years. ================================================================================ Title: Implicit anti-affinity rules and DRS placement behavior URL: https://frankdenneman.ai/2013-02-19-implicit-anti-affinity-rules-and-drs-placement-behavior/ Date: 2013-02-19 Yesterday I had an interesting conversation with a colleague about affinity rules and if DRS reviews the complete state of the cluster and affinity rules when placing a virtual machine. The following scenario was used to illustrate to question: The following affinity rules are defined: 1. VM1 and VM2 must stay on the same host 2. VM3 and VM4 must stay on the same host 3. VM1 and VM3 must NOT stay on the same host If VM1 and VM3 is deployed first, everything will be fine. Because VM1 and VM3 will be placed on 2 different hosts, and VM2 and VM 4 will also be placed accordingly However, if VM1 is deployed first, and then VM4, there isn’t an explicit rule to say these two need to be on separate hosts, this is implied by looking into dependencies of the 3 rules created above. Would DRS be intelligent enough to recognize this? Or will it place VM1 and VM4 on the same host, but by the time VM3 needs to be placed, there is a clear deadlock. The situation where its not logical to place VM4 and VM1 on the same host can be deemed as a implicit anti-affinity rule. It’s not a real rule, but if all virtual machines are operational, VM4 should not be on the same host as VM1. DRS doesn’t react to these implicit rules. Here’s why: When provisioning a virtual machine, DRS sorts the available hosts on utilization first. Then it goes through a series of checks such as the compatibility between the virtual machine and the host. Does the host have a connection to the datastore? Is the vNetwork available at the host? And then it will check to see if placing the virtual machine violates any constraints. A constraint could be a VM-VM affinity/anti-affinity rule or a VM-Host affinity/anti-affinity rule. In the scenario where VM1 is running, DRS is safe to place VM4 on the same host as it does not violate any affinity rule. When DRS wants to place VM3, it determines that placing VM3 on the same host VM4 is running violates the anti-affinity rule VM1 and VM3. Therefor it will migrate VM4 the moment VM3 is deployed. During placement DRS only checks the current affinity rules and determines if placement violates any affinity rules. If not, then the host with the most connections and the lowest utilization is selected. DRS cannot be aware of any future power-on operations, there is no vCrystal bowl. The next power-on operation might be 1 minute away or might be 4 days away. By allowing DRS to select the best possible placement, the virtual machine is provided an operating environment that has the most resources available at that time. If DRS took al the possible placement configurations into account, it could either end up in gridlock or place the virtual machine on a higher utilized host for a long time in order to prevent a vMotion operation of another virtual machine to satisfy the affinity rule. All that time that virtual machine could be performing beter if it was placed on a lower utilized host. On the long run, dealing with constraints the moment they occur is far more economical. Similar behavior occurs when creating a rule. DRS will not display a warning when creating a collections of rules that create a conflict when all virtual machines are turned on. As DRS is unaware of the intentions of the user, it cannot throw a warning. Maybe the virtual machines will not be powered on in the current cluster state. Or maybe this ruleset is in preparation for the new hosts that will be added to the cluster shortly. Also understands that if a host is in maintenance mode, this host is considered to be external to the cluster. It does not count as an valid destination and the resources are not used in the equation. However we as users still see the host part of the cluster. If those rule sets are created while a host is in maintenance mode, than according to the previous logic DRS must throw an error, while the user assumes the rules are correct as the cluster provides enough placement options. As clusters can grow and shrink dynamically, DRS deals only with violations when the rules become active and that is during power-on operations (DRS placement). ================================================================================ Title: HA Percentage based admission control from a resource management perspective – Part 1 URL: https://frankdenneman.ai/2013-02-15-ha-percentage-based-admission-control-from-a-resource-management-perspective-part-1/ Date: 2013-02-15 Disclaimer: This article contains references to the words master and slave. I recognize these as exclusionary words. The words are used in this article for consistency because it’s currently the words that appear in the software, in the UI, and in the log files. When the software is updated to remove the words, this article will be updated to be in alignment. HA admission control is quite challenging to understand as it interacts with multiple layers of resource management. In the upcoming series of articles I want to focus on the HA Percentage based admission control and how it interacts with vCenter and Host management. Let’s cover the basis first before diving into percentage based admission control. Virtual machine service level agreements HA provides the service to start up a virtual machine when the host it’s running on fails. That’s what HA is designed to do, however, HA admission control is tightly interlinked with virtual machine reservations and that because virtual machines is a hard SLA for the entire virtual infrastructure. Let’s focus on the two different “service level agreements” you can define for your virtual machine. Share-based priority: A shared based priority allows the virtual machine to get all the resources it demands until the demand exceeds supply on the ESXi host. Depending on the number of shares and activity, the resource manager will determine the relative priority to resource access. Resources will be reclaimed from inactive virtual machines and distributed to high priority active virtual machines. If all virtual machines are active, then the share values determine the distribution of resources. Let’s coin the term “Soft SLA” for shared based priority as the resource manager allows a virtual machine to be powered on even if there are not enough resources available to provide an adequate user experience/performance of the application running inside the virtual machine. The resource manager just distributes resources based on the shares value set by the administrator, it expects that correct shares were set to provide an adequate performance level at all times. Reservation based priority: Reservations can be defined as a “Hard SLA”. Under all circumstances, the resources protected by a reservation must be available to that particular virtual machine. Even if every other virtual machine is in need of resources, the resource manager cannot reclaim these resources as it is restricted by this Hard SLA. It must provide. In order to meet the SLA, the host checks to see if it has enough free unreserved resources available during the power-on operation of the virtual machine. If it doesn’t have the necessary unreserved resources available, the host cannot meet the Hard SLA and therefore the host rejects the virtual machine. Go look somewhere else buddy ;) Percentage based admission control The percentage-based admission control is my favorite HA admission control policy because it gets rid of the over-conservative slot mechanism used by the “host failure tolerates” policy. With percentage-based admission control, you set a percentage of the cluster resources that will be reserved for failover capacity. For example, when you set a 25% percent of reserved failover memory capacity, 25% of the cluster resources are reserved. In a 4-host cluster, this makes sense as the 25% embodies the available resources of a single host, thus 25% equals a failover tolerance of one host. If one host fails, the remaining three hosts can restart the virtual machines that were potentially using 75% of resources of the failed host. For the sake of simplicity, this diagrams show an equal load distribution, however, due to the VM reservations and other factors, the distribution of virtual machines might differ. Let’s take a closer look at that 25% and that 75 %. The 25% of reserved failover memory capacity is not done on a per-host basis; therefore the previous diagram is not completely accurate. It’s because this failover capacity is tracked and enforced by HA at the vCenter layer, to be more precise it’s on the HA cluster level. This is crucial information to understand the difference between admission control during normal provisioning/ power-on operations and admission control during restart operations done by HA. 25% reserved failover memory capacity The resource allocation tab of the cluster shows what happening after enabling HA. The first screenshot is the resource allocation of the cluster before HA is enabled. Notice the 1 GB reservation. When setting the reserved failover memory capacity to 25% the following thing happens: 25% of the cluster capacity (363.19*0.25=90.79) is added to the reserved capacity, plus the existing 1.03GB, totaling the reserved capacity to 91.83GB. This means that this cluster has 271.37 of available capacity left. Exactly what is this available capacity? This is capacity what’s often revered as “unreserved capacity”. What will happen with this capacity when we power-on a 16GB virtual machine without a reservation? Will it reduce the available capacity to 255.37? No, it will not. This graph shows only how much of the total capacity is assigned to an object with a hard SLA (reservations). Thus when a virtual machine is powered on or provisioned into the cluster via vCenter by the user it goes through HA admission controls first: After HA accepts the virtual machine, DRS admissions control and Host admission control review the virtual machine first before powering it on. The article admission control family describes the admission control workflow in-depth. 75% unreserved capacity What happened to that 90 GB? Is it gone? Is a part of this capacity reserved on each host in the cluster and unavailable for virtual machines to use? No luckily the 90GBs are not gone, HA just reduced the available capacity so that during a placement operation (deployment or power-on of an existing VM) vCenter knows if one of the clusters can meet the hard SLA of a reservation. To illustrate this behavior I took a screenshot of ESXtop output of a host: In this capture, you can see that the host is serving 5 virtual machines (W2K8-00 to W2K8-04). Each virtual machine is configured with 16GB (Memsz) and the resource manager has assigned a size target of memory above the 16GB (SZTGT). This size target is the number of resources the resource manager has allocated. The reason why it’s higher than the memsize is because of the overhead memory reservation. The memory needed by the VMkernel to run the virtual machine. As you can see these 5 virtual machines use up 82GB, which is more than the 67.5 GB is supposed to have if 25% was reserved as failover capacity on each host. Failover process and the role of host admission control This is the key to understand why HA “ignores” reserved failover capacity during a failover process. As HA consists of FDM agents running on each host, it is the master FDM agent who reviews the protected list and initiates a power-on operation of a virtual machine that is listed as protected but is not running. The FDM agent ensures that the virtual machines are powered on. As you can see this all happens on a host-level basis, vCenter is not included in this party. Therefore the virtual machine start-up operation is reviewed by the host admission control. If the virtual machine is configured with a soft SLA, host admission control only checks if it can satisfy the VM overhead reservation. If the VM is protected by a VM reservation, host admission control checks if it can satisfy both the VM reservation as well as the VM overhead reservation. If it cannot if will fail the startup and FDM has to find another host that can run this virtual machine. If all host fail to power-on the virtual machine, HA will request DRS to “defragment” the cluster by moving virtual machines around to make room on a host and free up some unreserved capacity. But remember, if a virtual machine has a soft SLA, HA will restart the virtual machine regardless of the amount of capacity to run the virtual machines providing adequate performance to the users. This behavior is covered in-depth in the article: “HA admission control is not a capacity management tool”. To ensure virtual machine performance during a host failure, one must focus on capacity planning and/or configuration of resource reservations. Part 2 of this series will take a closer look at how to configure a proper percentage value that avoids memory overcommitment. ================================================================================ Title: Have you signed up for the Benelux Software Defined Datacenter Roadshow yet? URL: https://frankdenneman.ai/2013-02-14-have-you-signed-up-for-the-benelux-software-defined-datacenter-roadshow-yet/ Date: 2013-02-14 In less than 3 weeks time, the Benelux Software Defined Datacenter Roadshow starts. Industry-recognized experts from both IBM and VMware share their vision and insights on how to build a unified datacenter platform that provides automation, flexibility and efficiency to transform the way you deliver IT. Not only can you attend their sessions and learn how to abstract, pool and automate your IT services, the SDDC roadshow provides you to meet the expert, sit down and discus technology. The speakers and their field of expertise: VMware Frank Denneman – Resource Management Expert Cormac Hogan – Storage Expert Kamau Wanguhu – Software Defined Networking Expert Mike Laverick – Cloud Infrastructure Expert Ton Hermes – End User Computing Expert IBM Tikiri Wanduragala – IBM PureSystems Expert Dennis Lauwers – Converged Systems Expert Geordy Korte – Software Defined Networking Expert Andreas Groth – End User Computing Expert The roadshow is held in three different countries: Netherlands – IBM forum in Amsterdam - March 5th 2013 Belgium – IBM forum in Brussels - March 7th 2013 Luxembourg – March 8th 2013 The Software Defined Datacenter Roadshow is a full day event and best of all it is free! Sign up now! ================================================================================ Title: Using Remote desktop connection on a Mac? Switch to CoRD URL: https://frankdenneman.ai/2013-02-13-using-remote-desktop-connection-on-a-mac-switch-to-cord/ Date: 2013-02-13 One of the benefits of working for VMware technical marketing, is that you have your own lab. Luckily my lab is hosted by an external datacenter, which helps me avoid a costly power-bill at home each month :) However, that means I need to connect to my lab remotely. As a MAC user I used Remote Desktop Connection for MAC from Microsoft. One of the limiting factors of this RDP for MAC is the limited resolution of 1400 x 1050 px. The screens at home have a minimum resolution 2560 x 1440 px. This first world problem bugged me until today! Today I found CoRD - http://cord.sourceforge.net/. CoRD allows me to connect to my servers with a resolution 2500 x 1600, using the full potential of my displays at home. Another create option is the hotkey function, using a key combination I spin up a remote desktop connection. I love these kinds of shortcuts that help me reduce time spend navigating throughout the UI. If you are using a MAC and often RDP into your lab, I highly recommend to download CoRD. Btw, it’s free ;) ================================================================================ Title: VCD and initial placement of virtual disks in a Storage DRS datastore cluster URL: https://frankdenneman.ai/2013-02-13-vcd-and-initial-placement-of-virtual-disks-in-a-storage-drs-datastore-cluster/ Date: 2013-02-13 Recently a couple of consultants brought some unexpected behavior of vCloud Director to my attention. If the provider vDC is connected to a datastore cluster and a virtual disk or vApp is placed in the datastore, vCD displays an error when the datastores do not have enough free space available. Last year I wrote an article (Storage DRS initial placement and datastore cluster defragmentation) describing storage DRS initial placement engine and it’s ability to move virtual machines around the datastore cluster if individual datastores did not have enough free space to store the virtual disk. I couldn’t figure it out why Storage DRS did not defragment the datastores in order to place the vApp, thus I asked the engineers about this behavior. It turns out that this behavior is by design. When creating vCloud director the engineers optimized the initial placement engine of vCD for speed. When deploying a virtual machine, defragmenting a datastore cluster can take some time. To avoid waiting, vCD reports an error of not enough free space and relies on the vCloud administrator to manage and correct the storage layer. In other words, Storage DRS initial placement datastore cluster defragmentation is disabled in vCloud Director. I can understand the choice the vCD engineers made, but I also believe in the benefit of having datastore cluster defragmentation. I’m interested in your opinion? Would you trade initial placement speed over reduced storage management? ================================================================================ Title: Expandable reservation on resource pools, how does it work? URL: https://frankdenneman.ai/2013-02-12-expandable-reservation-on-resource-pools-how-does-it-work/ Date: 2013-02-12 It seems that the expandable reservation setting of a resource pool appears to be shrouded in mystery. How does it work, what is it for, and what does it really expand? The expandable reservation allows the resource pool to allocate physical resource (CPU/memory) protected by a reservation from a parent source to satisfy its child object reservation. Let’s dig a little deeper into this. Parent-child relation A resource pool provides resources to its child objects. A child object can either be a virtual machine or a resource pool. This is what called the parent-child relationship. If a resource pool (A), contains a resource pool (B), which contains a resource pool (C), then C is the child of B. B is the parent of C, but is the child of A, A is the parent of B. There is no terminology for the relation A-C as A only provides resource to B, it does not care if B provide any resource to C. As a virtual machine is placed in to a resource pool, the virtual machine becomes a child-object of the resource pool. It is the responsibility of the resource pool to provide the resources the virtual machine requires. If a virtual machine is configured with a reservation, than it will request the physical resources from its parent resource pool. Remember that a reservation guarantees that the resources protected by the reservation will and cannot be reclaimed by the VMkernel, even during memory pressure. Therefor the reservation of the virtual machine is directed to its parent and the parent must exclusively provide this to the virtual machine. It can only provide these resources from its own pool of protected resources. The resource pool can only distribute the resources it has obtained itself. Protected or reserved resources? I’m deliberately calling a resource claimed by a reservation a protected resource, as the VMkernel cannot reclaim it. However when a resource pool is configured with a reservation, it immediately claims this memory from its parent. This goes on all the way up to the cluster level. The cluster is the root resource pool and all the resources provided by the ESXi hosts are owned by the resource pool and protected by a reservation. Therefor the cluster – root resource pool – contains and manages the protected pool of resources. For example, the cluster has 100GB of resources, meaning that the root resource pool consists of 100GB of protected memory. Resource pool A is configured with a 50GB reservation, consuming this 50Gb from the root resource pool. However resource pool B is configured with a 30GB reservation, immediately claiming 30 GB of resources protected by the reservation of resource pool A. Leaving resource pool A with only 20 GB of protected resources for itself. Resource Pool C is configured with a 20GB memory reservation. Resource pool C claims this from its parent, resource pool B which is left with 10GB of protected resources for itself. But what happens if the resource pool runs out of protected resources? Or is not configured with a reservation at all? In other words, If the child objects in the resource pool are configured with reservations that exceeds the reservation set on the resource pool, the resource pool needs to request protected resources from its parent. This can only be done if expandable reservation is enabled. Please note that the resource pool request protected resources, it will not accept resources that are not protected by a reservation. Now in this scenario, the five virtual machines in the resource pool are each configured with 5GB memory reservation, totaling it to 25GB. Resource pool C is configured with a 20GB memory reservation. Therefor resource pool is required to make a request for 5GB of protected memory resources on behalf of the virtual machines to its parent resource pool B. If resource pool B does not have the protected resources itself, it can request these protected resources from its parent. This can only occur when the resource pool is configured with expandable reservation enabled. The last stop in the cluster it the cluster itself. What can stop this river of requests? Two things, the request for protected resources is stopped by a resource limit or by a disabled expandable reservation. If a resource pool has expandable reservation disabled, it will try to satisfy the reservation itself if it’s unable to do so, it will deny the reservation request. If a resource pool is set with a limit, the resource pool is limited to that amount of physical resources. For example if the parent resource pool has a reservation and a limit of 20GB, the reservation on behalf of its child need to be satisfied by the protected pool itself otherwise it will deny the resource request. Now lets use a more complex scenario, resource pool B is configured with expandable reservation enabled and a 30 GB reservation. A limit is set to 35GB. Resource pool C is requesting an additional 10GB on top of the 20GB it is already granted. Resource pool B is running 2 VM with a total reservation of 10GB. This means the protected pool of Resource pool B is servicing 20GB resource request from resource pool C and 10 GB for its own virtual machines. Its protected pool is depleted, the additional 10GB request of resource pool C is denied, as this would raise the protected pool of resource pool B to a total of 40GB memory, which exceeds the 35GB limit. Virtual machine memory overhead Please remember that each virtual machine is configured with a memory reservation. To run the virtual machine a small amount of memory resources are required by the VMkernel. This is called the virtual machine memory overhead. To be able to run a virtual machine inside a resource pool, either the expandable reservation should be enabled or a memory reservation is configured on the resource pool. ================================================================================ Title: vSphere Storage Area Network Traffic system network resource pool -NetIOC URL: https://frankdenneman.ai/2013-02-12-vsphere-storage-area-network-traffic-system-network-resource-pool-netioc/ Date: 2013-02-12 After posting the Network I/O Control primer I received a couple of questions about the vSAN traffic system network resource pool, such as: What’s the “vSphere Storage Area Network Traffic” system network resource pool for? I tried to further investigate by searching practically everywhere, but I didn’t manage to find any detailed description… The vSphere Storage Area Network Traffic is a system network pool designed for a future vSphere storage feature that is not released yet. Unfortunately Network I/O Control exposes this system network resource pool in vSphere 5.1 already. Although it is defined as system network resource pool, the vSphere client lists the network pool as user-defined, providing the impression that this pool can be assigned to other streams of traffic. Unfortunately this is not possible. The pool is a system network resource pool and therefor only available to traffic that is specifically tagged by the VMkernel. I received the question if this network pool could be assigned to a third party NIC or an FCoE card. As mentioned, network pools only manage traffic that is assigned with the appropriate tag. Tagging of traffic is only done by the VMkernel and this functionality is not exposed to the user. Although its exposed in the user-interface, this system network pool has no function and it will not have any affect on other network streams. It can be happily ignored. ================================================================================ Title: Error -1 in opening & reading the slot file error in storageRM.log (SIOC) URL: https://frankdenneman.ai/2013-02-11-error-1-in-opening-reading-the-slot-file-error-in-storagerm-log-sioc-2/ Date: 2013-02-11 The problem Recently I noticed that my datastore cluster was not providing Latency statistics during initial placement. The datastore recommendation during initial placement displayed space utilization statistics, but displayed 0 in the I/O Latency Before column The performance statistics of my datastores showed that there was I/O activity on the datastores. However the SIOC statistics all showed no I/O activity on the datastore The SIOC log file (storagerm.log) showed the following error: Open /vmfs/volumes/ /.iorm.sf/slotsfile (0x10000042, 0x0) failed: permission denied Giving UP Permission denied Error -1 opening SLOT file /vmfs/volumes/datastore/.iorm.sf/slotsfile Error -1 in opening & reading the slot file Couldn’t get a slot Successfully closed file 6 Error in opening stat file for device: datastore. Ignoring this device The following permissions were applied on the slotfile: The Solution Engineering explained to me that these permissions were not the default standards and default permissions are read and execute access for everyone and write access for the owner of the file. The following command sets the correct permissions on the slotsfile: Chmod 755 /vmfs/volumes/datastore/.iorm.sf/slotsfile Checking the permission shows that the permissions are applied: The SIOC statistics started to show the I/O activity on the datastore Before changing the permissions on the slotsfile I stopped the SIOC service on the host by entering the command: /etc/init.d/storageRM stop However I believe this isn’t necessary. Changing the permissions without stopping SIOC on the host should work. Cause We are not sure what causes this problem and support and engineering are troubleshooting this error. In my case I believe it has to do with the frequent restructuring of my lab. vCenter and ESXi servers are reinstalled regularly, but I have never reformatted my datastores. I do not expect to see this error appear in stable production environments. Please check the current permissions on the slotsfile if Storage DRS does not show I/O utilization on the datastore. (VMs must be running and I/O metric on the Datastore cluster must be enabled of course) I expect the knowledge base article to be available soon. ================================================================================ Title: Error -1 in opening & reading the slot file error in storageRM.log (SIOC) URL: https://frankdenneman.ai/2013-02-11-error-1-in-opening-reading-the-slot-file-error-in-storagerm-log-sioc/ Date: 2013-02-11 The problem Recently I noticed that my datastore cluster was not providing Latency statistics during initial placement. The datastore recommendation during initial placement displayed space utilization statistics, but displayed 0 in the I/O Latency Before column The performance statistics of my datastores showed that there was I/O activity on the datastores. However the SIOC statistics all showed no I/O activity on the datastore The SIOC log file (storagerm.log) showed the following error: Open /vmfs/volumes/ /.iorm.sf/slotsfile (0x10000042, 0x0) failed: permission denied Giving UP Permission denied Error -1 opening SLOT file /vmfs/volumes/datastore/.iorm.sf/slotsfile Error -1 in opening & reading the slot file Couldn’t get a slot Successfully closed file 6 Error in opening stat file for device: datastore. Ignoring this device The following permissions were applied on the slotfile: The Solution Engineering explained to me that these permissions were not the default standards and default permissions are read and execute access for everyone and write access for the owner of the file. The following command sets the correct permissions on the slotsfile: Chmod 755 /vmfs/volumes/datastore/.iorm.sf/slotsfile Checking the permission shows that the permissions are applied: The SIOC statistics started to show the I/O activity on the datastore Before changing the permissions on the slotsfile I stopped the SIOC service on the host by entering the command: /etc/init.d/storageRM stop However I believe this isn’t necessary. Changing the permissions without stopping SIOC on the host should work. Cause We are not sure what causes this problem and support and engineering are troubleshooting this error. In my case I believe it has to do with the frequent restructuring of my lab. vCenter and ESXi servers are reinstalled regularly, but I have never reformatted my datastores. I do not expect to see this error appear in stable production environments. Please check the current permissions on the slotsfile if Storage DRS does not show I/O utilization on the datastore. (VMs must be running and I/O metric on the Datastore cluster must be enabled of course) I expect the knowledge base article to be available soon. ================================================================================ Title: How to enable SIOC stats only mode? URL: https://frankdenneman.ai/2013-02-11-how-to-enable-sioc-stats-only-mode/ Date: 2013-02-11 Today on twitter, David Chadwick, Cormac Hogan and I were discussing SIOC stats only mode. SIOC stats only mode gathers statistics to provide you insights on the I/O utilization of the datastore. Please note that Stats only mode does not enable the datastore-wide scheduler and will not enforce throttling. Stats only mode is disabled due to the (significant) increase of log data into the vCenter database. SIOC stats only mode is available from vSphere 5.1 and can be enabled via the web client. To enable SIOC stats only mode go to: Storage view Select the datastore Select Manage Select Settings By default both SIOC and SIOC stats only mode is disabled. Click on the edit button at the right side of the screen. Un-tick the check box “Disable Storage I/O statistics collection (applicable only if Storage I/O Control is disabled)”. Click on OK To test to see if there is any difference, I used a datastore that SIOC had enabled. I disabled SIOC and un-ticked the “Disable Storage I/O statistics collection (applicable only if Storage I/O Control is disabled)” option. I opened up the performance view and selected the “Realtime” Time Range. Storage view Select the datastore Select Monitor Select Performance Select “Realtime” Time range At 15:35 I disabled SIOC, which explains the dip, at 15:36 SIOC stats only mode was enabled and it took vCenter roughly a minute to start displaying the stats again. As all new vSphere 5.1 features, SIOC stats only mode can only be enabled via the vSphere web client. ================================================================================ Title: Why is vMotion using the management network instead of the vMotion network? URL: https://frankdenneman.ai/2013-02-07-why-is-vmotion-using-the-management-network-instead-of-the-vmotion-network/ Date: 2013-02-07 On the community forums, I’ve seen some questions about the use of the management network by vMotion operations. The two most common scenarios are explained, please let me know if you notice this behavior in other scenarios. Scenario 1: Cross host and non-shared datastore migration vSphere 5.1 provides the ability to migrate a virtual machine between hosts and non-shared datastores simultaneously. If the virtual machine is stored on a local or non-shared datastore vMotion is using the vMotion network to transfer the data to the destination datastore. When monitoring the VMkernel NICs, some traffic can be seen following over the management NIC instead of the VMkernel NIC enabled for vMotion. When migrating a virtual machine, vMotion determines hot data and cold data. Virtual disks or snapshots that are actively used are considered hot data, while the cold data are the underlying snapshots and base disk. Let’s use a virtual machine with 5 snapshots as an example. The active data is the recent snapshot, this is sent over across the vMotion network while the base disk and the 4 older snapshots are migrated via a network file copy operation across the first VMkernel NIC (vmk0). The reason why vMotion uses separate networks is that the vMotion network is reserved for data migration of performance-related content. If the vMotion network is used for network file copies of cold data, it could saturate the network with non-performance related content and thereby starving traffic that is dependent on bandwidth. Please remember that everything sent over the vMotion network directly affects the performance of the migrating virtual machine. During a vMotion the VMkernel mirrors the active I/O between the source and the destination host. If vMotion would pump the entire disk hierarchy across the vMotion network it would steal bandwidth from the I/O mirror process and this will hurt the performance of the virtual machine. If the virtual machine does not contain any snapshots, the VMDK is considered active and it is migrated across the vMotion network. The files in the VMDK directory are copied across the network of the first VMkernel NIC. Scenario 2: Management network and vMotion network sharing same IP-range/subnet If the management network (actually the first VMkernel NIC) and the vMotion network share the same subnet (same IP-range) vMotion sends traffic across the network attached to first VMkernel NIC. It does not matter if you create a vMotion network on a different standard switch or distributed switch or assign different NICs to it, vMotion will default to the first VMkernel NIC if same IP-range/subnet is detected. Please be aware that this behavior is only applicable to traffic that is sent by the source host. The destination host receives incoming vMotion traffic on the vMotion network! I’ve been conducting an online-poll and more than 95% of the respondents are using a dedicated IP-range for the vMotion traffic. Nevertheless, I would like to remind you that it’s recommended to use a separate network for vMotion. The management network is considered to be an unsecured network and therefore vMotion traffic should not be using this network. You might see this behavior in POC environments where you use a single IP-range for virtual infrastructure management traffic. If the host is configured with a Multi-NIC vMotion configuration using the same subnet as the management network/1st VMkernel NIC, then vMotion respects the vMotion configuration and only sends traffic through the vMotion-enabled VMkernel NICs. If you have an environment that is using a single IP-range for the management network and the vMotion network, I would recommend creating a Multi-NIC vMotion configuration. If you have a limited amount of NICs, you can assign the same NIC to both VMkernel NICs, although you do not leverage the load balancing functionality, you force the VMkernel to use the vMotion-enabled networks exclusively. ================================================================================ Title: Please help VMware bring project NEE down to its (k)nees URL: https://frankdenneman.ai/2013-02-05-please-help-vmware-bring-project-nee-down-to-its-knees/ Date: 2013-02-05 Folks, We have been testing the HOL platform for a few weeks using automated scripts and thought it would be great if we could do a real time stress test of our environment. The goal of this test is to put a massive load on our infrastructure and see how fast we can get the service to crawl to its knees. We understand that this is not a very good scientific approach but think collecting real user data will help us prepare for massive loads like Partner Exchange and VMworld. Currently we have close to 10,000 users in the Beta so we expect the application / infrastructure to keel over right after we start. We want to use this test as a way to learn what happens and where the smoke is coming from. If you registered for the Beta and you do not have an account please check your inbox from email from admin projectnee.com to verify your account. If you have not registered its time to do so,…REGISTER FOR BETA Here is what we need you to do: Take any lab on Thursday Feb 7th from 2:00 – 4:00 PM PST. Send us feedback (on this thread) on your experience. Include Lab Name, Description of Problem, Screen Shot. Follow Project NEE on Twitter for latest Updates http://twitter.com/vmwarehol Thanks for your support! ================================================================================ Title: Storage DRS Initial placement workflow URL: https://frankdenneman.ai/2013-02-05-storage-drs-initial-placement-workflow/ Date: 2013-02-05 Last week I received the question how exactly Storage DRS picks a datastore. On a SDRS the initial placement of a vm is done on the weight calculated based on the storage free and IO. My question is: when I have a similar weight between all the datastore in the cluster, which datastore is choose for the initial placement? Storage DRS takes the virtual machine configuration into account, the platform & user-defined constraints and the resource utilization of the datastores within the cluster. Let’s take a closer look at the Storage DRS initial placement workflow. User-defined constraint When selecting the datastore cluster as a storage destination, the default datastore cluster affinity rule is applied to the virtual machine configuration. The datastore cluster can be configured with a VMDK affinity rule (Keep files together) or a VMDK anti-affinity rule (Keep files separated). Storage DRS obeys the affinity rule and is forced to find a datastore that is big enough to store the entire virtual machine or the individual VMDK files. The affinity rule is considered to be a user-defined constraint. Platform constraint The next step in the process is to present a list of valid datastores to the Storage DRS initial placement algorithm. The Storage DRS placement engine checks for platform constraints. The first platform constraint is the check of the connectivity state of the datastores. Fully connected datastores (datastores connected to all host in the compute cluster) are preferred over partially connected datastores (datastores that are not connected to all host in the cluster) due to the impact of mobility of the virtual machine in the compute cluster. The second platform constraint is applicable to thin-provisioned LUNs. If the datastore exceeds the thin-provisioning threshold of 75 percent, the VASA provider (if installed) triggers the thin-provisioning alarm. In response to this alarm Storage DRS removes the datastores from the list of valid destination datastores, in order to prevent virtual machine placement on low-capacity datastores. Resource utilization After the constraint handling, Storage DRS sorts the valid datastores in order of combined resource utilization rate. The combined resource utilization rate consists of the space utilization and the I/O utilization of a datastore. The best-combined resource utilization rate is a datastore that has a high level of free capacity and Low I/O utilization. Storage DRS selects the datastore that has the best-combined utilization rate and attempts to place the virtual machine. If the virtual machine is configured with a VMDK anti-affinity rule, Storage DRS starts with placing the biggest VMDK first. ================================================================================ Title: 10 guidelines for creating good looking diagrams URL: https://frankdenneman.ai/2013-02-01-10-guidelines-for-creating-good-looking-diagrams/ Date: 2013-02-01 Frequently I receive the question which application I use to create my diagrams. I used to use Microsoft Visio but starting to use Omnigraffle a year ago. However I feel it’s not the program that makes these diagrams. Although it’s true that some functionality help me to create the diagrams more easily, it’s more about following some basic guidelines. I’ve picked up these guidelines along the way, they work for me and hopefully they can help you too. 1: Find a suitable color scheme A color scheme plays a very important role in a diagram. Colors have various functions within a diagram. I like to use various tints of a color to indicate a relation between objects, whether it has to indicate a relation within the same structure layer or the same consumer or provider. For example all storage related functions or objects have different shades of blue or resource pool structure of customer A have different shades of green. Picking the correct color for a diagram is very difficult and trying to select the perfect collection of colors wasted (I should say invested) many hours of my life. During that time I learned a lot, here are a few tips: Use a color scheme that provides contrast between different objects. Use the wheel of color to easily select complimentary colors (Colors on opposite sides of the color wheels). I prefer using multiple triad (3-point) complement color schemes. When using multiple triads, uses colors of similar saturation levels. Saturation refers how a “color” appears under a particular lighting condition. Mixing primary and secondary colors with similar saturation levels provides a more cohesive looking design. For example, a bright red color mixed with a blue-ish green color can give some strange effects, sometimes giving the illusion of vibrating when not looking directly at them (very annoying). Use a limited set of colors, don’t allow your diagram to become the poster-child for circus publication guidelines. Resources: Smashing magazine have published an excellent series on color theory colorschemedesiginer.com shows the Color wheel, use the triad function 2: Fonts Besides legibility and readability a proper font (typeface) makes the diagram “look right”. Objects and fonts are interrelated when it comes to conveying a subject. Both the font type and the objects in the diagram translate and visualize an idea or concept. Colors evoke feelings and moods, while the font determine the tone of voice in which the message is broadcasted. Two major categories of typefaces can be identified in the world of fonts. Serif and sans-serif. Serif fonts can be recognized by having small lines at the end of the strokes of a letter. Times new roman is a good example of a serif font. Serif fonts mimic handwriting and can provide a outdated but also formal feeling. Sans-serif fonts lack the small lines and provide a much cleaner and modern look. I have seen diagrams illustrating technologies and features that weren’t released yet, but still gave me an outdated, well-worn feeling. Try use sans-serif fonts when creating computer technology related diagrams. Use a sans-serif font (PT sans or corbel are excellent choices) Try to use a font that compliments the font used in the articles Use a single font in a diagram; use different font weights (light, medium, bold) to emphasize. Color contrast; use dark colors on white background, white on black background. Tip: dark blue on white gives a rich feeling to the labels. Use handwritten fonts only if you use hand drawn objects. Match font style with objects. Swooping lines as connectors allow for the use of a more elegant font, however keep it in style with your overall blog theme and used fonts. Do not use Comic sans, unless you are diagramming your lemonade stand! Resources: Google fonts and myfonts.com provide an enormous font collection. Most of them can be downloaded for free. Use whatthefont service to identify a specific font used in an image. 3: Lines Lines come in all shapes and forms. Try to be consistent with the types of lines you use. If you use a dashed line for indicating standby functionality, do not use the same line pattern for an active connection. Think about the thickness of the lines used. If you selected a very clean lightweight font, don’t use thick lines for the framework of boxes and other objects. Mix and match line weight with font weight. Strive for balance across the entire diagram. 4: Whitespace Whitespace or often revered to as negative space is the portion of the diagram left unused. It’s the space between the objects and this is what I believe actually the most important thing to get right. The balance between the positive (objects) and the negative (whitespace) is fundamental to create an aesthetic pleasing diagram. Whitespace can help to emphasize particular elements but also help to balance the objects in the diagram. Using too much whitespace and a relationship between two objects may get lost. 5: Align! Always align objects horizontally and if applicable vertically. These details matter. It might not be easily identified by eye, but your subconscious picks it up and alerts you “something is not right”. Most people tend to shy away and that’s conflicts the first reason why you made the diagram. To help people better understand the subject by creating additional visual aids. Omnigraffle is far more advanced than Visio when it comes to auto alignment. Omnigraffle provides automatic guides displaying the white space between objects in the same line. That feature saved me many hours 6: Go minimal Try to reduce the number of objects as much as possible. Get to the essence of the subject as much as possible. IT people like to put in as much as detail as possible. If these objects are not relevant to the subject you are trying to depict, leave them out. This increases the focus point of the diagram. Going minimal is harder than it sounds. By using as little objects as possible you spend a lot time focusing on spacing, positioning, typography and contrast. 7: Shadows The novelty of shadows beneath lines and boxes wear off quickly. After viewing the diagram a couple of times, the shadows give the diagram an unclean and grimy feeling. It doesn’t look clean, fresh and rapidly feels outdated after seeing the diagram a couple of times. My advice: Try to avoid it as much as possible. 8: Real men make block diagrams Sometimes I jokily reply this when somebody is asking for vendor stencils and icon-packs on twitter. Vendor stencils can be very useful for some types of diagrams, for example wiring diagram of a core Ethernet switch. I prefer to stay away from using pre-made icons in diagrams indicating architecture or relationships. Pre made icons come in their own color scheme and are usually in an isometric perspective to give that 3d feel. Forcing you to design the whole diagram in an isometric perspective. Certain Icon designs distract the viewer, reducing the ability of the diagram to convey the message. By creating your own objects, you can choose your own color scheme, your own level of detail, and your own direction of perspective. 9: Commit to a single perspective Already mentioned in the number 8, when using an icon in isometric perspective commit to drawing in an isometric perspective. The viewpoint of an isometric diagram is slightly rotated to reveal other surfaces than those visible from a top-down perspective. Isometric diagrams are a great way for illustrating all the physical components of (virtual) architectures. A while ago I stumbled upon an old isometric diagram I created for a client of mine. Mixing isometric icons with top-down icons provides an unbalanced view. Usually the lines do not connect well or are just to complete parallel or horizontally aligned, providing hours of frustration to the stickler for details. 10: Relevance A picture is worth a thousand words, but don’t draw a giraffe after you wrote three paragraphs about the feeding habits of elephants. This is an extreme example, but use a diagram to help the reader to understand the aspect of the topic and assist him with the identification of the subject. Don’t allow a diagram to confuse your audience. I’ve seen countless diagrams in VCDX architecture designs of arrays connected to an FC architecture, when the candidate was using an iSCSI. If you use 6 LUNs, don’t use a diagram that shows an object with the words “LUN 1 …. LUN 99” in it. Allow the diagram relay information to strengthen the written word. Be consistent and have fun In almost all of the other guidelines I provide examples why consistency is important. It helps the reader to identify components and their relation more easily. Especially when you use a series of diagrams in the same presentation or publication. Do it well and allow it to become your trademark. But most of all have fun while creating diagrams. It shows! ================================================================================ Title: How to setup Multi-NIC vMotion on a distributed vSwitch URL: https://frankdenneman.ai/2013-02-01-how-to-setup-multi-nic-vmotion-on-a-distributed-vswitch/ Date: 2013-02-01 This article provides you an overview of the steps required to setup a Multi-NIC vMotion configuration on an existing distributed Switch with the vSphere 5.1 web client. This article is created to act as reference material for the designing your vMotion network series. Configuring Multi-NIC vMotion is done at two layers, first the distributed switch layer where we are going to create two distributed port groups and the second layer is the host layer. At the host layer we are going to configure two VMkernel NICs and connect them to the appropriate distributed port group. Before you start you need to have ready two ip-addresses for the VMkernel NICs, their respective subnet and their VLAN ID. Distributed switch level The first two steps are done at the distributed switch level, click on the networking icon in the home screen and select the distributed switch. Step 1: Create the vMotion distributed port groups on the distributed switch The initial configuration is pretty much basic, just provide a name and use the defaults: 1: Select the distributed switch, right click and select “New Distributed Port Group”. 2: Provide a name, call it “vMotion-01” and confirm it’s the correct distributed switch. 3: Keep the defaults at Configure settings and click next. 4: Review the settings and click finish. Do the same for the second distributed port group, name that vMotion-02 Step 2: Configuring the vMotion distributed port groups Configuring the vMotion distributed port groups consist of two changes. Enter the VLAN ID and set the correct failover order. 1: Select distributed Port Group vMotion-01 in the left side of your screen and right click and select edit settings. 2: Go to VLAN, select VLAN as VLAN type and enter the first VLAN used by the first VMkernel NIC. 3: Select “Teaming and failover” , move the second dvUplink down to mark it as a “Standby uplink”. Verify that load balancing is set to “Route based on originating virtual port”. 4: Click OK Repeat the instructions of step 2 for distributed Portgroup vMotion-02, but use the VLAN ID used by the IP-address of the second VMkernel NIC. Go to teaming and failover and configure the uplinks in an alternate order, ensuring that the second vMotion VMkernel NIC is using dvUplink2. Host level We are done at the distributed switch level, the distributed switch now updates all connected hosts and each host has access to the distributed port groups. Two vMotion enabled VMkernel NICs are configured at host level. Go to Hosts and Clusters view. Step 3: Create vMotion enabled VMkernel NICs 1: Select the first host in the cluster, go to manage, networking and “add host networking”. 2: Select VMkernel Network Adapter. 3: Select an existing distributed portgroup, click on Browse and select distributed Port Group “vMotion-01” Click on OK and click on Next. 4: Select vMotion traffic and click on Next. 5: Select static IPv4 settings, Enter the IP-address of the first VMkernel NIC corresponding with the VLAN ID set on distributed Port Group vMotion-01. 6: Click on next and review the settings. Create the second vMotion enabled VMkernel NIC. Configure identically except: 1: Select vMotion-02 portgroup 2: Enter IP-address corresponding with the VLAN ID on distributed Port Group vMotion-02. The setup of a Multi-NiC vMotion configuration on a single host is complete. Repeat Step 3 on each host in the cluster. ================================================================================ Title: Designing your vMotion network - 3 reasons why I use a distributed switch for vMotion networks URL: https://frankdenneman.ai/2013-01-30-designing-your-vmotion-network-3-reasons-why-i-use-a-distributed-switch-for-vmotion-networks/ Date: 2013-01-30 If your environment is licensed with the enterprise plus license you can choose to use a standard vSwitch or use a distributed switch for your vMotion network. Multi-NIC vMotion network is a complex configuration that consists out of many different components. Each component needs to be configured identically on each host in the cluster. Distributed switches can help you with that and in addition provide you with tools to prioritize traffic and allow other network streams to utilize available bandwidth when no vMotion traffic is active. 1. Use distributed portgroups consistent configuration across the cluster Consistently configuring two portgroups on each host in the cluster with alternating vmnic failover order is a challenging task. It’s a mere fact that humans are not good in performing a repetitive task consistently. Many virtual infrastructure health checks at various sites confirmed that fact. The beauty of distributed switches (VDS) is that it acts as profile configuration. Configure the portgroup once and the distributed switch propagates these settings to all the connected hosts of that distributed switch. A multi-NIC vMotion configuration is a perfect use-case to leverage the advantages of the distributed switch. As mentioned a Multi-NIC vMotion configuration is a complex configuration consisting of two portgroups with their own unique settings. By using the distributed switch, only two distributed portgroups need to be configured and the VDS distributes the portgroups and their settings to each host connected to the VDS. This saves a lot of work and you are ensured that each host is using the same configuration. Consistency in your cluster is important for to provide you reliable operations and consistent performance. 2. Set traffic priority with Network I/O Control Network I/O control can help you to consolidate the network connections into a single manageable switch, allowing you to utilize all the bandwidth available while still respecting requirements such as traffic isolation or traffic prioritization. This is applicable to both configurations containing a small number of 10GbE uplink as well for configurations that contain a high number of 1GbE ports. vMotion has a high bandwidth usage, performing optimally in high bandwidth environments. However vMotion traffic is not always present. Isolating NICs in order to protect other network traffic streams or provide a particular level of bandwidth can be uneconomical and may leave bandwidth idle and unused. By using Network I/O Control, you can control the priority of network traffic during contention. This allows you to specify the relative importance of traffic streams and provide bandwidth to particular traffic streams when other traffic competes for bandwidth. 3. Using Load Based Teaming to balance all traffic across uplinks Load based teaming, identified in the user interface as “Route based on physical NIC load” allows for ingress and egress traffic balancing. When consolidating all uplinks in one distributed switch, load based teaming (LBT) distributes the traffic streams across the available uplinks by taking into the utilization into account. Please note: Use Route based on originating virtual port load balancing policy for the two vMotion portgroup, but configure VM network with load based teaming load balancing policy. Route based on originating virtual port load balance policy creates a vNIC to pNIC relation during boot of a virtual machine. That vNIC is “bound” to that pNIC until the pNIC fails or the virtual machine is shutdown. When using a converged network or allowing all network traffic to use each uplink, a virtual machine could experience link saturation or latency due to vMotion using the same uplink. With LBT the virtual machine vNIC can be dynamically bound to a different pNIC with lesser utilization, providing better network performance. LBT monitors the utilization of each uplink and when the utilization is greater than 75 percent for a sustained period of time, LBT moves traffic to other underutilized uplinks. The benefits of a Distributed vSwitch Consistent configuration across hosts saves a lot effort, during configuration and troubleshooting. Consistent configuration is key when providing a stable and a performing environment. Multi-NIC vMotion allows you to use as much bandwidth as possible benefitting DRS in load balance and maintenance mode operations. LBT and Network I/O Control allow other network traffic streams to consume network traffic as much as possible. Load based teaming is a perfect partner to Network I/O Control. LBT attempts to balance out the network utilization across all available uplinks and Network I/O Control dynamically distributes network bandwidth during contention. Back to standard vSwitch when uplink isolation is necessary? Is this Multi-NIC vMotion/NetIOC/LBT configuration applicable to every customer? Unfortunately it isn’t. Converging all network uplinks into a single distributed switch and allowing all portgroups to utilize the uplinks require the VLANs to be available on every uplink. Some customers want to isolate vMotion traffic or other traffic and use dedicated links. For that scenario I would still use a distributed switch and create one for the vMotion configuration. In this particular scenario you do not leverage LBT and Network I/O Control but still benefit from the consistent configuration of distributed portgroups. Part 1 - Designing your vMotion network Part 2 - Multi-NIC vMotion failover order configuration Part 3 – Multi-NIC vMotion and NetIOC Part 4 – Choose link aggregation over Multi-NIC vMotion? ================================================================================ Title: vMotion and EtherChannel, an overview of the load-balancing policies stack URL: https://frankdenneman.ai/2013-01-28-vmotion-and-etherchannel-an-overview-of-the-load-balancing-policies-stack/ Date: 2013-01-28 After posting the article “Choose link aggregation over Multi-NIC vMotion” I received a couple of similar questions. Pierre-Louis left a comment that covers most of the questions. Let me use this as an example and clarify how vMotion traffic flows through the stack of multiple load balancing algorithms and policies: A question relating to Lee’s post. Is there any sense to you to use two uplinks bundled in an aggregate (LAG) with Multi-NIC vMotion to give on one hand more throughput to vMotion traffic and on the other hand dynamic protocol-driven mechanisms (either forced or LACP with stuff like Nexus1Kv or DVS 5.1)? Most of the time, when I’m working on VMware environment, there is an EtherChannel (when vSphere < v5.1) with access datacenter switches that dynamically load balance traffic based on IP Hash. If i’m using LAG, the main point to me is that load balancing is done independently from the embedded mechanism of VMware (Active/Standby for instance). Do you think that there is any issue on using LAG instead of using Active/Standby design with Multi-NIC vMotion? Do you feel that there is no interest on using LAG over Active/Standby (from VMware point of view and for hardware network point of view)? Pierre-Louis takes a bottom-up approach when reviewing the stack of virtual and physical load-balancing policies and although he is correct when stating that network load balancing is done independently from VMware’s network stack, it does not have the impact he thinks it has. Lets look at the starting point of vMotion traffic and how that impacts both the flow of packets and utilization of links. Please read the articles “Choose link aggregation over Multi-NIC vMotion” and “Designing your vMotion network” to review some of the requirements of Multi-NIC vMotion configurations Scenario configuration Lets assume you have two uplinks in your host, i.e. two physical NICs per ESX host. Each vmnic used by the VMkernel NIC (vmknic) is configured as active and both links are aggregated in a Link Aggregation Group (LAG) (EtherChannel in Cisco terms). First thing I want to clarify that the active/standby state of a vmknic is static and is controlled by the user, not a Load-Balancing policy. When using a LAG, both vmknics need to be configured active, as the load balancing policy needs to be able to send traffic across both links. Duncan explains the impact when using Standby NICs in an IP-Hash configuration. Load balancing stack A vMotion is initiated on host-level; therefor the first load balancer that comes in to play is vMotion itself. Then the portgroup load balancing policy will make a decision followed by the physical switch. Load balancing done by the physical switch/LAG is the last element in this stack. Step 1: vMotion load balancing, this is done on the application layer and it is vMotion process that selects which VMkernel NIC is used. As you are using a LAG and two NICs, only one vMotion VMkernel NIC should exist. The previous mentioned article explains why you should designate all vmnics as active in a LAG. By using one vmknic enabled for vMotion, vMotion is unable to load-balance at vmknic level and sends all the traffic to the single vmknic. Step 2: Next step is the load-balancing policy; IP-Hash will select one NIC after it hashes both source and destination IP. That means that this vMotion operation will use the same NIC until the vMotion operation is complete. It does not use two links, as a vMotion operation connection is setup by two VMkernel NICs and thus two IP-addresses (source IP address and destination IP). As IP-Hash determines the vmknic, traffic will be send out across the physical link. Step 3: is at the physical switch layer and determines which port to use to connect to a NIC of the destination host. Once the physical switch receives the packet, the load balancer of the LAG configuration comes into play. The physical switch determines which path to take to the destination host according to utilization or availability of a link. Each switch vendor has different types of load balancers, too many to describe, the article “Understanding EtherChannel Load Balancing and Redundancy on Catalyst Switches” describes the different load balancing operations within the Cisco Catalyst switch family. In short: Step 1 Load balancing by vMotion: has no direct control over which physical NICs are used, it load balances across available multiple vmknics. Step 2 Load balancing by IP-HASH: Outgoing connection is hashed based on its source and destination IP address; hash is used to select a physical NIC to use for network transmissions. Step 3 Load balancing by LAG: Physical switch connected to destination performs the hash to choose which physical NIC to send incoming connection to. To LAG or not to LAG, that’s the question By using a LAG configuration for vMotion traffic, you are limited to the available bandwidth of a single uplink per vMotion operation as only one uplink is used per vMotion operation. A Multi-NIC vMotion configuration balances vMotion traffic across both VMkernel NICs. It load balances traffic for a single vMotion operation as well as multiple concurrent vMotion operations across the links. Let me state that differently; It is able to use the bandwidth of two uplinks for both a single vMotion operation, as well as multiple vMotion operations. With Multi-NIC vMotion you will get a more equal load balance distribution than with any load-balancing policy operating at NIC level. I would always select multi-NIC vMotion over LAG. A LAG requires strict configuration on both virtual level as physical level. It’s a complex configuration on both technical and political level. Multiple departments need to be involved and throughout my years as an architect I seen many infrastructures fail due to inter-department politics. Troubleshooting a LAG configuration is not an easy task in an environment where there are communication-challenges between the server and the network department. Therefor I strongly prefer not to use LAG in a virtual infrastructure Multiple single uplinks can be used to provide more bandwidth to the vMotion process and other load-balancing policies available on the distributed switch keep track of link utilization (LBT). It’s less complex, and in most cases give you better performance. ================================================================================ Title: Designing your vMotion networking - Choose link aggregation over Multi-NIC vMotion? URL: https://frankdenneman.ai/2013-01-25-designing-your-vmotion-networking-choose-link-aggregation-over-multi-nic-vmotion/ Date: 2013-01-25 The “Designing your vMotion network” series have lead me to have some interesting conversations. A recurring question is why not use Link Aggregation technologies such as Etherchannel to increase bandwidth for vMotion operations. When zooming into vMotion load balancing operations and how the vNetwork load balancing operations work, it becomes clear that Multi-NIC vMotion network will provide a better performance that an aggregated link configuration. Anatomy of vMotion configuration In order to use vMotion a VMkernel network adapter needs to be configured. This network adapter needs to be enabled for vMotion and the appropriate load balancing policy and network adaptor failover mode needs to be selected. The vMotion load-balancing algorithm distribute vMotion traffic between the available VMkernel NICs, it does not consider the network configuration backing the VMkernel NIC (vmknic). vMotion expects that a vmknic is backed by a single active physical NIC, therefore when sending data to the vmknic the traffic will traverse a dedicated physical NIC. It’s important to understand that vMotion traffic flows between distinct vmknics! Migrating a virtual machine from a source host configured with Multi-NIC vMotion to a destination host with a single vmknic vMotion configuration result in the utilization of a single vmknic on the source host. Even though the single NIC vMotion is configured with two active uplinks, the source vMotion operation just selects one vmknic to transmit its data. Link aggregation My esteemed colleague Vyenkatesh “ Venky” Deshpande published an excellent article on the new LACP functionality on the vSphere blog. Let me highlight a very interesting section: Link aggregation allows you to combine two or more physical NICs together and provide higher bandwidth and redundancy between a host and a switch or between two switches. Whenever you want to create a bigger pipe to carry any traffic or you want to provide higher reliability you can make use of this feature. However, it is important to note that the increase in bandwidth by clubbing the physical NICs depends on type of workloads you are running and type of hashing algorithm used to distribute the traffic across the aggregated NICs. And the last part is the key, when you aggregate links into a single logical link, it depends on the load balancing / hashing algorithm how the traffic is distributed across the aggregated links. When using an aggregated link configuration, it’s required to select the IP-HASH load balancing operation on the portgroup. I’ve published an article in 2011 called “IP-hash versus LBT” explaining how the hashing and distribution of traffic across a link aggregation group works. Let’s assume you have Etherchannels in your environment and want to use it for your vMotion network. 2 x 1GB aggregated in one pipe, should beat 2 x 1GB right? As we learned vMotion deals only with vmknics and as vMotion detects a single vmknic it will send all the traffic to that vmknic. The vMotion traffic hits the load-balancing policy configured on the portgroup and the IP-Hash algorithm will select a vmnic to transmit the traffic to the destination host. Yes you read it correct, although the two links are aggregated the IP-Hash load balancing policy will always select a single NIC to send traffic. Therefor vMotion will use a single uplink (1GB in this example) to transfer vMotion traffic. With IP-hash a vMotion operation utilizes a single link, leaving the other NIC idle. Would you have used Multi-NIC vMotion, vMotion would have balanced the traffic across the multiple vmknics even for a single vMotion operation. Utilization aware Using the same scenario, vMotion determines that this host is allowed to have 4 concurrent vMotion operations. Unfortunately IP-Hash does not take utilization into account when selecting the NIC. The selection is done on a source-destination IP hash, decreasing the probability of load balancing across multiple NICs when using a small number of IP-addresses. This situation is often applicable to the vMotion subnet; this subnet contains a small number of IP-addresses used by the vMotion vmknics. Possibly resulting IP-HASH selecting the same NIC for the same concurrent vMotion operations. This in turn may lead to oversaturating a single uplink while leaving the other uplink idling. Would you have used Multi-NIC vMotion, vMotion would have balanced the traffic across the multiple vmknics, providing an overall utilization of both NICs. Key takeaways Link aggregation does not provide a big fat pipe to vMotion, due to the IP-Hash load balancing policy, a single nic will be used for a vMotion operation. IP-Hash is not utilization aware, possibly distributing traffic unevenly due to small number of source and destination IP-addresses. Multi-NIC vMotion distributes vMotion traffic across all available vmknics, for both single vMotion operations and multiple concurrent vMotion operations. Multi-NIC vMotion provides a better overall utilization of NICs allocated for the vMotion processes. Part 1 - Designing your vMotion network Part 2 - Multi-NIC vMotion failover order configuration Part 3 – Multi-NIC vMotion and NetIOC Part 5 – 3 reasons why I use a distributed switch for vMotion networks ================================================================================ Title: New technical paper: The CPU Scheduler in VMware vSphere 5.1 URL: https://frankdenneman.ai/2013-01-24-new-technical-paper-the-cpu-scheduler-in-vmware-vsphere-5-1/ Date: 2013-01-24 Today a new technical paper is available on vmware.com. Description The CPU scheduler is an essential component of vSphere 5.x. All workloads running in a virtual machine must be scheduled for execution and the CPU scheduler handles this task with policies that maintain fairness, throughput, responsiveness, and scalability of CPU resources. This paper describes these policies, and this knowledge may be applied to performance troubleshooting or system tuning. This paper also includes the results of experiments on vSphere 5.1 that show the CPU scheduler maintains or exceeds its performance over previous versions of vSphere. If you are interested in CPU scheduling and in particular NUMA, download the paper: The CPU Scheduler in VMware vSphere 5.1 ================================================================================ Title: Hide all Getting Started Pages in vSphere 5.1 webclient in 3 easy steps URL: https://frankdenneman.ai/2013-01-23-hide-all-getting-started-pages-in-vsphere-5-1-webclient/ Date: 2013-01-23 I’m rebuilding my lab and after I installed a new vCenter server I was confronted with those Getting Started tabs again. That reminded me that I promised someone at a VMUG to blog how to remove these tabs in one single operation. Go to Help (located in the blue bar top right of your screen) Click on the arrow Select Hide All Getting Started Pages ================================================================================ Title: Direct IP-storage and using NetIOC User-defined network resource pools for QoS URL: https://frankdenneman.ai/2013-01-21-direct-ip-storage-and-using-netioc-user-defined-network-resource-pools-for-qos/ Date: 2013-01-21 Some customers use iSCSI initiators inside the Guest OS to connect directly to a datastore on the array or using an NFS client inside the Guest OS to access remote NFS storage directly. Thereby circumventing the VMkernel storage stack of the ESX host. The virtual machine connects to the remote storage system via a VM network portgroup and therefore the VMkernel classifies this network traffic as virtual machine traffic. This “indifference” or non-discriminating behavior of the VMkernel might not suit you or might not help you to maintain service level agreements. Isolate traffic In the 1Gbe adapter world, having redundant and isolated uplinks assigned for different sorts of traffic is a simple way of not to worry about traffic congestion. However when using a small number of 10GbE adapters you need to be able to partition network bandwidth among the different types of network traffic flows. This is where NetIOC comes into play. Please read the “Primer on Network I/O Control” article to quickly brush up on your knowledge of NetIOC. System network resource pools By default NetIOC provides seven different system network resource pools. Six network pools are used to bind VMkernel traffic, such as NFS and iSCSI. One system network resource pool is used for virtual machine network traffic. The network adapters you use to connect your IP-Storage from within the Guest OS connect to a virtual machine network portgroup. Therefor NetIOC binds this traffic to the virtual machine network resource pool. In result this traffic shares the bandwidth and prioritization with “common” virtual machine network traffic. User-defined network resource pool Most customers tend to prioritize IP storage traffic over network traffic induced by applications and the guest-OS. To ensure the IP-Storage traffic created by the NFS client or iSCSI initiator inside the Guest OS create a user-defined network resource pool. User-defined network resource pools are available from vSphere 5.0 and upwards. Make sure your distributed switch is at least version 5.0. Shares: User-defined network resource pools are available to isolate and prioritize virtual machine network traffic. Configure the User-defined network resource pool with an appropriate number of shares. The number of shares will reflect the relative priority of this network pool compared to the other traffic streams using the same dvUplink. QoS tag: Another benefit of creating a separate User-defined network resource pool is the ability to set a QoS tag specifically for this traffic stream. If you are using IEEE 802.1p tagging end-to-end throughout your virtual infrastructure ecosystem, setting the QoS tag on the User-defined network resource pool helps you to maintain the service level for your storage traffic. Setup In a greenfield scenario setup the User-defined resource pool first, that allows you to select the correct network pool during the creation of the dvPortgroups. If you already created dvPortgroups, you can assign the correct network resource pool once you create the network resource pool. Create a user defined network resource pool: 1. Open your vSphere web client and go to networking. 2. Select the dvSwitch 3. Go to Manage 4. Select Resource Allocation tab 5. Click on the new icon. 6. Configure the network resource pool and click on OK I already made a User-defined network resource pool called dNFS, the overview of available network resource pools on the dvSwitch looks like this: To map the network resource pool to the Distributed Port Group, create a new Distributed Port group, or edit an existing one and select the appropriate network resource pool: ================================================================================ Title: Storage DRS Cluster shows error "Index was out of range. Must be non-negative and less than the size of the collection. URL: https://frankdenneman.ai/2013-01-21-storage-drs-cluster-shows-error-index-was-out-of-range-must-be-non-negative-and-less-than-the-size-of-the-collection-2/ Date: 2013-01-21 Recently I have noticed an increase of tweets mentioning the error “Index was out of range” Error message: When trying to edit the settings of a storage DRS cluster, or when clicking on SDRS scheduling settings an error pops up: “An internal error occurred in the vSphere Client. Details: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index” Impact: This error does not have any impact on Storage DRS operations and/or Storage DRS functionality and is considered a “cosmetic” issue. See KB article: KB 2009765. Solution: VMware vCenter Server 5.0 Update 2 fixes this problem. Apply Update 2 if you experience this error. As the fix is included in the Update 2 for vCenter 5.0, I expect that vCenter 5.1 Update 1 will include the fix as well, however I cannot confirm any deliverables. ================================================================================ Title: Storage DRS Cluster shows error "Index was out of range. Must be non-negative and less than the size of the collection. URL: https://frankdenneman.ai/2013-01-21-storage-drs-cluster-shows-error-index-was-out-of-range-must-be-non-negative-and-less-than-the-size-of-the-collection/ Date: 2013-01-21 Recently I have noticed an increase of tweets mentioning the error “Index was out of range” Error message: When trying to edit the settings of a storage DRS cluster, or when clicking on SDRS scheduling settings an error pops up: “An internal error occurred in the vSphere Client. Details: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index” Impact: This error does not have any impact on Storage DRS operations and/or Storage DRS functionality and is considered a “cosmetic” issue. See KB article: KB 2009765. Solution: VMware vCenter Server 5.0 Update 2 fixes this problem. Apply Update 2 if you experience this error. As the fix is included in the Update 2 for vCenter 5.0, I expect that vCenter 5.1 Update 1 will include the fix as well, however I cannot confirm any deliverables. ================================================================================ Title: Designing your vMotion network – Multi-NIC vMotion and NetIOC URL: https://frankdenneman.ai/2013-01-18-designing-your-vmotion-network-multi-nic-and-netioc/ Date: 2013-01-18 Designing a vMotion network can be quite a challenge. You want to provide vMotion as much bandwidth as possible but not at the expense of other network traffic streams. Network I/O Control (NetIOC) can provide you bandwidth management tools to shape and form your vMotion network. NetIOC provides a QoS that allows vMotion to utilize as much bandwidth as possible until contention occurs. The moment a physical NIC is saturated NetIOC distributes network bandwidth according to the relative share value of the network resource pool. In the article “A primer of Network I/O Control” I explain the various resource management constructs of Network I/O Control. How does NetIOC work with a Multi-NIC vMotion network? Multi-NIC vMotion network on a distributed switch NetIOC is only supported on a distributed switch therefor you need to create multiple vMotion portgroups on your distributed switch. In order to have a supported vMotion network, both distributed port groups need to be configured with an alternate failover order configuration. In my lab I’ve named the two dvPortgroups vMotion-01 and vMotion-02. dvPortgroups vMotion-01 is configured with a failover order where dvUplink1 is active and dvUplink2 is standby. vMotion-02 is configured with dvUplink2 as active and dvUplink1 as standby. If you wonder why I’m configuring the redundant uplink as Unused please review the article: “Multi-NIC vMotion – failover order configuration”. The reason why “Route Based on originating virtual port” is chosen is described in the initial article of this series: “Designing your vMotion network”. vMotion network resource pool Each host in my lab is connected to the distributed switch has two dvUplink portgroups. A virtual adapter with vMotion enabled is connected to a uplink and vMotion is able to leverage both physical adapters to send out vMotion traffic. When enabling NetIOC, 7 predefined system network resource pools become active. One system network resource pool is vMotion, vMotion traffic binds to the vMotion system network resource pool and if contention occurs the network resource pool competes for bandwidth with the other active streams. It’s important to realize that the shares apply at the physical adapter layer. Therefore as vMotion is able to utilize both uplinks it “receives” 50 shares per physical adapter. As mentioned in the NetIOC primer article, shares are only active when contention exists and they only count when it is transmitting. Consequently, if vMotion is not active, the shares are not counted when an adapter is congested. Also when vMotion uses a single link only 50 shares are active. Although the dvUplink portgroup is configured with 2 dvUplinks, one dvUplink is configured as standby. When the active NIC is operating normally, the dvPortgroups cannot utilize the standby link. This results in the utilization of only 1 link and therefor only 50 shares become active during congestion. Example Scenario 1; vmnic0 saturated The distributed switch is configured with 5 dvPortgroups, management, vMotion01, vMotion02, NFS and a virtual machine portgroup. Each network resource pool is configured with the default physical adapter shares. In this scenario vMotion is load-balancing traffic across both dvPortgroups, however the VMkernel decides to also send management and virtual machine traffic through dvUplink1, saturating vmnic0. In this scenario, bandwidth is distributed following relative share values. As NFS traffic isn’t transmitting across dvUplink1, its shares are not active. Available bandwidth for vMotion is reduced to 2.5Gb as long as this situation persists. vMotion is also using dvUplink2, as a result traffic flows through vmnic1 as well. Fortunately vmnic1 is not saturated and vMotion can utilize as much bandwidth it can allocate. Restricted by the CPU speed of the host, vMotion is able to utilize 6Gb of bandwidth on vmnic1 while having an additional bandwidth allocation of 2.5Gb on vmnic0. In total vMotion utilizes 8.5Gb at that moment. Example Scenario 2; Both NICs saturated The moment both NICs are saturated, the available bandwidth available to vMotion is calculated on a NIC basis. vMotion traffic wanting to use vmnic0 is assigned bandwidth relative to their share value, similar to example scenario 1. vMotion traffic wanting to use vmnic1 is assigned bandwidth relative to their share value compared to the active traffic streams, in this case NFS and vMotion are sending traffic to the dvUplink. The distributed switch receives traffic from NFS and vMotion destined for vmnic1, as a result NetIOC will assign half of the bandwidth to vMotion and NFS as each owns 50 shares of the total active 100 shares. In this case the amount of bandwidth vMotion can utilize is 2.5Gb on vmnic0 and 5 Gb on vmnic1, allowing vMotion to utilize 7.5Gb of the total available 20Gb. Host Limits vSphere 5.1 introduced a big adjustment to hosts limits, the host limits now applies to each individual uplink. This means that when setting a host limit on the network resource pool for vMotion of 3000Mbps, vMotion is limited to transmit a maximum of 3Gb per uplink. In the case of a Multi-NIC vMotion configuration (2NICs) the maximum traffic vMotion can issue to the vmnics is 6Gb. Divide the host limit by the number of active NICs if you decide to set a limit on a particular network resource pool. If you want to limit vMotion to 3Gb with a 2 NIC Multi-NIC configuration, set the host limit on the network resource pool to 1500 Mbps. As for limits, limits are always active! Host limits are always enforced on the physical adapter regardless of the utilization rate of the adapter. This means setting a host limit on a network resource pool restricts the traffic to utilize more bandwidth even if this bandwidth is available. Saturating an adapter is not by definition wrong you are utilizing the infrastructure. Taking precaution and limiting certain network streams unconditionally might hurt you more than your assumed gain. Before limiting a network stream my recommendation is always to measure traffic patterns over a longer period of time. Ingress only NetIOC shares and limits are only applied to ingress traffic. In ESX , the ingress and egress traffic are with respect to a distributed switch. Ingress traffic is traffic that flows from the VMkernel or vNic from the running VM towards the distributed switch. Egress is traffic that flows from the distributed vSwitch to the physical nic or to the vNIC It is important to know NetIOC only controls ingress traffic initiated from within the ESX host. This means that you can set a limit the vMotion network resource pool, this only affects traffic send towards the uplink inside the host. If you have a cluster with a small number of hosts, it can happen that multiple vMotion operations are inbound to that host. In that scenario, NetIOC cannot prevent the Uplinks to get saturated by incoming vMotion traffic. To avoid this situation, traffic shaping needs to be configured on the dvPortgroups. The upcoming article Designing your vMotion network - vMotion dvPortgroups traffic shaping explores this feature in depth. Part 1 - Designing your vMotion network Part 2 - Multi-NIC vMotion failover order configuration Part 4 – Choose link aggregation over Multi-NIC vMotion? Part 5 – 3 reasons why I use a distributed switch for vMotion networks ================================================================================ Title: A primer on Network I/O Control URL: https://frankdenneman.ai/2013-01-17-a-primer-on-network-io-control/ Date: 2013-01-17 Network I/O Control (NetIOC) provides controls to partition network capacity during contention. NetIOC provides additional control over the usage of network bandwidth in the form of network isolation and limits. vMotion operations introduce temporary network traffic that tries to consume as much bandwidth as possible. In a converged network vMotion operations may have a disruptive effect on other network traffic streams. Due to way NetIOC operates, NetIOC provides control for predictable networking performance while different network traffic streams are contending for the same bandwidth. This article serves as an introduction on Network I/O Control resource control before we dive into specifics on how to use NetIOC on multi-NIC vMotion networks on distributed switches. Please note that this article covers NetIOC in vSphere 5.1 Distributed Switch NetIOC is only available on the vNetwork Distributed Switch (vDS). To enable it, in the vSphere web client go to Networking, Manage, Settings, click on the third icon from the left (Edit distributed switch settings) and enable Network I/O Control. Once enabled go to the Resource Allocation tab there you will find an overview of the (predefined) Network Resource Pools, Host Limits and Physical Adapter Shares and the Shares value. Network Resource Pool The NetIOC network resource pool (NRP) construct is quire similar in many ways to the compute resource pools already existing for CPU and Memory. Each resource pool is assigned shares to define the relative priority of its workload against other workloads active on the same resource. In the case of NetIOC, network resource pools are used to differentiate between network traffic classes. NetIOC predefines 7 different system network resource pools: 1. Management Traffic 2. vMotion Traffic 3. Fault Tolerance (FT) Traffic 4. iSCSI Traffic 5. vSphere Storage Area Network Traffic * 6. NFS Traffic 7. vSphere Replication (VR) Traffic 8. Virtual Machine Traffic * The vSphere client marks this as a User defined network resource pool, while the web client marks this (correctly) as system network resource pool. NetIOC classifies incoming traffic and binds it automatically to the correct system network resource pool; therefor you do not have to assign the vMotion traffic network resource pool to the distributed port groups manually. Once a vMotion operation starts NetIOC “tags” it as vMotion traffic and assigns the appropriate share value to it. The user interface displays the term (default) in the Network resource pool settings screen. vSphere 5.0 introduced User-defined network resource pools and these are only applicable to virtual machine network traffic. User defined network pools are excellent to partition your network when multiple customers are using a shared network infrastructure. Physical adapter Shares NetIOC shares are comparable to the traditional CPU and memory shares. In the case of NetIOC, shares assigned to a network resource pool determine the portion of the total available bandwidth if contention occurs. Similar to compute shares, shares are only relative to the other active shares using the same resource. NetIOC provides 3 predefined share levels and a custom share level. The predefined share levels; low, normal and high provide an easy method of assigning a number of shares to the network resource pool. Low assigns 25 shares to the network resource pool, Normal 50 shares and High 100 shares. Custom allows you to assign the number of shares yourself within the supported range of 1 – 100. By default every system network resource pool is assigned 50 shares with the exception of the virtual machine traffic resource pool, this NRP gets 100 shares. The key to understand network resource pools and shares is that the shares apply on the physical adapter; hence the name physical adapter shares ;). This means that if the physical adapter of a host is saturated the shares of the network resource pools actively transmitting are in play. For example, your distributed switch is configured with a Management portgroup, a vMotion portgroup, NFS and virtual machine portgroup. All NRPs are configured with default shares and you use 2 uplinks. For the sake of simplicity, in this scenario both uplinks are configured as active uplinks Although this environment is configured with 4 portgroups, the default system network pools remain to exist. The existences of the non-utilized system-NRPs have no effect on the distribution of bandwidth during contention. As mentioned before the when the physical adapter is saturated, then the shares apply. In the following scenario vmnic0 of host ESX02 is saturated, as there are only 4 portgroups active on the distributed switch, the shares of the network pools are applied: This means that 50+50+50+100 (250) shares are active, in this scenario the virtual machine network resource pool gets to divide 40% (100/(50+50+50+100)) of the available physical network bandwidth. If vmnic0 was a 10GB NIC, the virtual machines network pool would receive 4GB to distribute amongst the actively transmitting virtual machines on that host. This was a worst-case scenario because usually not all portgroups are transmitting, as the shares are relative to other network pools actively using the physical adapter it might happen that Virtual Machine and vMotion traffic is only active on this NIC. In that case only the shares of the vMotion NRP and VM NRP are compared against each other to determine the available bandwidth for both network resource pools. The moment another traffic source transmits to the distributed switch a new calculation is made to determine the available bandwidth for the network resource pools. For example, HA is heart beating across the management LAN, using vmnic0, therefor the active transmitting network resource pools are Management, vMotion and virtual machines, generating a distribution of network bandwidth as follows: Host Limits Up to vSphere 5.1 NetIOC applies a limit per host. This means that the host limit enforces a traffic bandwidth limit on the overall set of dvUplinks for that particular network resource pool. The limit is expressed in an absolute unit of Mbps. This means that if you set a 3000Mpbs limit on the vMotion network resource pool, the traffic stream of the vMotion network resource pool will never exceed the given limit of 3000Mpbs for a distributed switch out of a particular ESX host. vSphere 5.1 introduced a big adjustment to hosts limits, the host limits now applies to each individual uplink. This means that when setting a host limit on the network resource pool for vMotion of 3000Mbps, vMotion is limited to transmit a maximum of 3Gb per uplink. In the case of a Multi-NIC vMotion configuration (2NICs) the maximum traffic vMotion can issue to the vmnics is 6Gb. Please note that limits only apply on ingress traffic (incoming traffic from vm to vds) meaning that a limit only affects native traffic coming from the active virtual machine running on the host or the vMotion traffic initiated on the host itself. Coming up next.. The next article in this series is how to use NetIOC for predictable network performance when using a Multi-NIC vMotion configuration on a distributed vSwitch. ================================================================================ Title: Storage DRS survey URL: https://frankdenneman.ai/2013-01-17-storage-drs-survey/ Date: 2013-01-17 If you have a few spare minutes, please fill out the survey about Storage DRS usage. We are very interested in your Storage DRS architecture and which options you use. But we are also seeking to identify adoption blockers preventing you to use Storage DRS. It takes about 5 minutes to get through. Storage DRS survey ================================================================================ Title: Two very interesting videos about how computer algorithms shapes today's world URL: https://frankdenneman.ai/2013-01-16-two-very-interesting-videos-about-how-computer-algorithms-shapes-todays-world/ Date: 2013-01-16 Resource management within virtual infrastructures relies on distributed algorithms, as a result I became more and more interested in the application of computer algorithms in other areas. Today I found an English version of the multiple award winning Dutch documentary which I can finally share with my non-dutch speaking friends. The documentary reviews the flash crash on the U.S. Stock Market on May 6th 2010. In particular it explores the application of black box trading (algo-trading) and how algorithms shaped and formed the architecture of not only of the trade market institutions but also city architectures and human terraforming. Money & Speed: Inside the Black Box (Marije Meerman, VPRO) Sit back and be amazed. Once viewed, view the TED presentation by Kevin Slavin: How algorithms shape our world. Kevin Slavin zooms in some of the algorithms applied in our lives and how it affect us and our surroundings. ================================================================================ Title: Adjusting the cost of vMotion – a word of caution URL: https://frankdenneman.ai/2013-01-15-adjusting-the-cost-of-vmotion-a-word-of-caution/ Date: 2013-01-15 Yesterday I posted an article on how to change the cost of vMotion in order to change the default number of concurrent vMotion. As I mentioned in the article, I’m not a proponent of changing advanced settings. Today Kris posted a very interesting question; How about the scenario where one uses multi NIC vMotion for against two 5Gbps virtual adapters)? I know a cost of 4 will be set for the network by the VMkernel, however as the aggregate bandwidth becomes 10Gbps is it safe enough to raise the limit? Perhaps not to the full 8 for 10Gbps, but 6? Please note that this article does not bash Kris. He provides a use case that I’ve heard a couple of times, making his comment an example use case. Although Kris’s scenario sounds like a very good use case to adjust the cost settings to circumvent the line-speed detection of the VMkernel to determine the max-cost of the network resource, it does not solve the other dynamic elements using line speed. DRS MaxMovesPerHost ESX 4.1 Introduces the MaxMovesPerHost setting, allowing the host to dynamically set the limit on moves. The limit is based on how many moves DRS thinks can be completed in one DRS evaluation interval. DRS adapts to the frequency it is invoked (pollPeriodSec, default 300 seconds) and the average migration time observed from previous migrations. However, this limit is still bound by the detected line speed and the associated Max cost. Although the proposed environment has 10GB line speed in total available, the VMkernel will still set the max cost to allow 4 vMotions on the host. Restricting the number of migrations, DRS can initiate during a load balance operation. vMotion system resource pool CPU reservation vMotion tries to move the used memory blocks as fast as possible. vMotion uses all the available bandwidth depending on the available CPU speed and bandwidth. Depending on the detected line speed, vMotion reserves an X amount of CPU speed at the start of a vMotion process. vMotion computes its desired host vMotion CPU reservation. For every 1GBe vMotion link speed it detects vMotion in vSphere 5.1 reserved 10% of a CPU core with a minimum desired CPU reservation of 30%. This means that if you use a single 1GBe, vMotion reserves 30% of a core, if you use 4 x 1GBe connections, that means vMotion reserves 40% of a core. A 10GBe link is special as vMotion reserves 100% of a single core. vMotion creates a (system) resource pool and sets the appropriate CPU reservation on the resource pool. It’s important to note that this is being done to the vMotion resource pool, which means that the reservation is shared across all vMotions happening on the host. Using two 5GB links, results in a 40% CPU core reservation (default 30% plus 10% for the extra link). However, this dynamic behavior might get unnoticed if you have enough spare CPU cycles in your source and destination host. Word of caution I hope these two examples show that there are multiple dynamic elements working together on various levels in your virtual infrastructure. Adjusting a setting might improve the performance of a specific use case, but to change the overall behavior, lots of settings have to be changed. Due to the lack of time and specific information correlating various settings is impossible for many of us most of the time. Therefore I would like to repeat my recommendation. Please do not adjust advanced settings only if VMware supports ask you to. ================================================================================ Title: Limiting the number of concurrent vMotions URL: https://frankdenneman.ai/2013-01-14-limiting-the-number-of-concurrent-vmotions/ Date: 2013-01-14 After explaining how to limit the number of concurrent Storage vMotions operations, I received multiple questions on how to limit the number of concurrent vMotion operations. This article will cover the cost and max cost constructs and show you how to calculate the correct config key values to limit the number of concurrent vMotion operations. Please note I usually do not post on configuration keys that change default behavior simply because I feel that most defaults are sufficient, and it should only be changed as a last resort when all other avenues are exhausted. I would like to mention that this is an unsupported configuration. Support will request to remove these settings before troubleshooting your environment! Cost To manage and limit the number of concurrent migrations either by vMotion or Storage vMotion, a cost and maximum cost (max cost) factor is applied. Think of the maximum cost as a limit. A resource has a max cost, and an operation is assigned a cost. A vMotion and Storage vMotion are considered operations, and the ESXi host, network, and datastore are considered resources. In order for a migration operation to be able to start, the cost cannot exceed the max cost. A resource has both a max cost and an in-use cost. When an operation is started, the resource records an in-use cost and allows additional operations until the maximum cost is reached. The in-use cost of an active operation and the new operation cost cannot exceed the max cost. As mentioned, there are three resources, host, network, and datastore. A vMotion operation interacts with the host, network, and datastore resource, while Storage vMotion interacts with the host and datastore resource. This means that changing the host or datastore-related cost can impact both vMotion and Storage vMotion. Let’s look at the individual costs and max cost before looking into which config key to change. Host Operation Config Key Cost vMotion Cost costPerVmotionESX41 1 Storage vMotion Cost costPerSVmotionESX41 4 Maximum Cost maxCostPerEsx41Host 8 Network Operation Config Key Cost vMotion Cost networkCostPerVmotion 1 Storage vMotion Cost networkCostPerSVmotion 0 Maximum Cost maxCostPerNic 2 maxCostPer1GNic 4 maxCostPer10GNic 8 Datastore Operation Config Key Cost vMotion Cost CostPerEsx41Vmotion 1 Storage vMotion Cost CostPerEsx41SVmotion 16 Maximum Cost maxCostPerEsx41Ds 128 Please note that because these values were not changed after 4.1, the advanced settings were unnecessary. Therefore these advanced settings apply to ESXi 5.0 and ESXi 5.1 as well. Default concurrent vMotion limit To limit vMotion we must identify which costs and max costs are involved; Datastore Cost: As we know, a vMotion transfers the memory from the source ESX host to the destination ESX host, sends over the pages stored in a non-shared page file is this exists, and finally transfers the ownership to the new VMX file. For the new host to run the new virtual machine, a new VMX file is created on the datastore; therefore, a vMotion process also includes the cost on the datastore resource. Although it generates overhead, the impact is very low. Therefore, the cost involved in the data store is 1. Datastore Max Cost: The maximum cost of a datastore is 128, therefore, a maximum of 128 concurrent vMotion operations can be active on a single datastore. Network: The cost for a vMotion on the network resource is 1 Network Max Cost: This config key is very interesting as it is set dynamically. The config key depends on the line speed detected by the VMkernel. If the VMkernel detects a line speed between 1 GB and 10GB, then the max cost value is set to 4. If the VMkernel detects 10GB, then the max cost value is set to 8. Please note that the VMkernel will set the max cost to 10GB ONLY if it detects 10GB line speed. It does not matter if you use 10GB Ethernet cards. It’s the line speed that counts. Please read the article “The impact of QoS network traffic on VM performance” and “Adaptive MaxMovesPerHost” if you apply a QOS on your converged network and wonder what impact this might have on vMotion performance and DRS load balancing. If the VMkernel detects a line speed below 1GB, it sets the max cost to 2, resulting in a maximum number of concurrent vMotions of 2 with the default network vMotion cost. Please note that the supported minimum required bandwidth is 1GB! This < 1gB line speeds setting is included for “just-in-case” scenarios where the vMotion network is temporarily incorrectly configured. It should not be used to justify a < 1gb line speed when designing the virtual infrastructure! Host cost: The cost for a vMotion on the host resource is 1 Host max cost: The host max config for all vMotion operations is 8. In-use cost If a vMotion is configured with a 1GB line speed, the max cost of the network allows for 4 concurrent vMotion, while the host max cost allows 8 concurrent vMotions. The most conservative max cost wins as the vMotion network does not allow the in-use cost to exceed the max cost. In the datastore cost section, I explained that the datastore allows for 128 concurrent vMotions. What usually is more common is to see multiple Storage vMotion operations active on a datastore due to Storage DRS Datastore Maintenance. If you are vMotioning a virtual machine that resides on the datastore to another host and you put a datastore into datastore maintenance mode, Storage DRS cannot initiate 8 storage vMotion because 8 Storage vMotion and the in-use cost of a vMotion exceeds the max cost of 128 of the data store. “Only” 7 concurrent Storage vMotions can be initiated while the vMotion is active. The in-use cost of the datastore is 7 x 16 = 112 + 1 (vMotion) = 113. Although it has 15 “points” left, it cannot start another Storage vMotion. Let’s assume the vMotion network is configured with 10GB line speed. This means that the host will allow for 8 concurrent vMotions. But if a Storage vMotion is already active, the in-use cost of the host is 4; therefore, the host can only allow for 1 additional Storage vMotion or 4 concurrent vMotions. networkCostPerVmotion As both Storage vMotion and vMotion use the host resource max cost, it is “recommended” to adjust the config key “networkCostPerVmotion”. Setting this config key to 2 allows for 2 concurrent vMotions on a 1GB vMotion network per host or 4 concurrent vMotions on a 10GB vMotion per host.The networkCostPerVmotion can be adjusted by editing the vpxd.cfg or via the advanced settings of the vCenter Server Settings in the administration view. If done via the vpxd.cfg, the value vpxd.ResourceManager.networkCostPerVmotion is added as follows: < config > < vpxd > < ResourceManager > < networkCostPerVmotion > new value < /networkCostPerVmotion > < /ResourceManager > < /vpxd > < /config > Word of caution Please note that cost and max values are applied to each migration process within vCenter! Therefore modification of costs impacts normal day-to-day DRS and Storage DRS load balancing operations as well as the manual vMotion and Storage vMotion operations occurring in the virtual infrastructure managed by the vCenter server. Adjusting the cost at the host side can be tricky as the costs of operation and limits are relative to each other and can even harm other host processes unrelated to migration processes. ================================================================================ Title: Adding new disk to an existing virtual machine in a Storage DRS Datastore Cluster URL: https://frankdenneman.ai/2013-01-11-adding-new-disk-to-an-existing-virtual-machine-in-a-storage-drs-datastore-cluster/ Date: 2013-01-11 Recently I had some discussions where I needed to clarify the behavior of Storage DRS when the user adds a new disk to a virtual machine that is already running in the datastore cluster. Especially what will happen if the datastore cluster is near its capacity? When adding a new disk, Storage DRS reviews the configured affinity rule of the virtual machine. By default Storage DRS applies an affinity rule (Keep VMDKs together by default) to all new virtual machines. vSphere 5.1 allows you to change the default behavior of the cluster, you can change the default affinity rule in the Datastore cluster settings: Adding a new disk to an existing virtual machine The first one to realize is that Storage DRS never can violate the affinity or anti-affinity rule of the virtual machine. For example, if the datastore cluster default affinity rule is set to “keep VMDKs together” then all the files are placed on the same datastore. Ergo if a new disk is added to the virtual machine, that disk must be stored on the same datastore in order not to violate the affinity rule. Let use an example, VM1 is placed in the datastore and is configured with the Intra-VM affinity rule (Keep the files inside the VM together). The virtual machine is configured with 2 hard disks and both reside on the datastore [nfs-f-07] of the datastore cluster When adding another disk, Storage DRS provides me with an recommendation Although all the other datastores inside the datastore cluster are excellent candidates as well, Storage DRS is forced to place the VMDK on the same datastore together with the rest of the virtual machine files. Datastore cluster defragmentation At one point you may find yourself in a situation where the datastores inside the datastore cluster are quite full. Not enough free space per datastore, but enough free space in the datastore cluster to host another disk of an exisiting virtual machine. In that situation, Storage DRS does not break the affinity rule but starts to “defragment” the datastore cluster. It will move virtual machines around to provide enough free space for the new VMDK. The article “Storage-DRS initial placement and datastore cluster defragmentation" can provide you more information about this mechanism. Key takeaway Therefore in a datastore cluster you will never see Storage DRS splitting up a virtual machine if the VM is configured with an affinity rule, but you will see pre-requisite moves, migrating virtual machines out of the datastore to make room for the new vmdk. ================================================================================ Title: Why do you manually select a datastore while using Storage DRS? URL: https://frankdenneman.ai/2013-01-10-why-do-you-manually-select-a-datastore-while-using-storage-drs/ Date: 2013-01-10 On the community forums a couple of threads are active about the Storage DRS Automation level behavior when trying to manually migrate a virtual machine between datastores in the same datastore cluster. When migrating within the datastore cluster, Storage DRS is disabled for that virtual machine. Some community members asked me why Storage DRS disables automation for this virtual machine when migrating between datastores inside a datastore cluster or when selecting a datastore during placement of a new virtual machine. Intent It is all about intent. When migrating a virtual machine into a datastore cluster, you are migrating the virtual machine into a load-balancing domain (the datastore cluster). You allow and trust Storage DRS to provide you an environment that provides an optimum load balanced state where the virtual machines receive the overall best I/O performance and the optimal placement regarding space utilization. If the user wants to migrate the virtual machine to a different datastore inside the datastore cluster, Storage DRS is capturing this intent, as “user knows best”. The way this is designed is that if a datastore is selected, then user is telling us that the selected datastore is the best, i.e. user knows something Storage DRS doesn’t. And to prohibit any future migration recommendation to other datastores, Storage DRS is disabled to ensure permanent placement. This behavior also applies when migrating a virtual machine into a datastore cluster. During initial placement it is expected that the user selects the datastore cluster, if the user wants to select a specific datastore it has to select “Disable Storage DRS for this virtual machine” in order to be able to select a member datastore. But this brings me to the question I have; what is the reason for not trusting Storage DRS? Why do you manually select a datastore while using Storage DRS? Apparently old habits die-hard and most tell me that their administrator feels like they could beat Storage DRS in placement. I’ve written a couple of articles about it (plus two 100+ page chapters featured in two books) about the working of Storage DRS and trust me it’s very difficult to beat Storage DRS placement and migration recommendations. During development the engineers try to run an equivalent experiment to IBM big blue versus Gary Kasparov and lined up two world-class storage experts versus Storage DRS. Although they received answers to all their questions they could not match the overall performance improvement Storage DRS could provide. Correlation of metrics Storage DRS has a lot of visibility into the environment, it measures space growth rates of existing virtual machine with thin disks, snapshots, etc. It selects destination datastore based on current utilization and growth rates. It builds device models to understand the performance of the devices backing the datastore as well as measuring the overall load on the datastores. It creates workload models of the existing virtual machine and measures on multiple metrics. Due to the insights Storage DRS can decide to migrate virtual machines to other datastores in order to make room or avoid the I/O threshold. It analyzes the environment and prefers moving virtual machines with low storage vMotion overhead. For more information please read the following articles: Storage DRS automation level and initial placement behavior Storage DRS Initial placement and datastore cluster defragmentation Avoiding VMDK level over commitment while using Thin disks and Storage DRS If you have a specific use case, for example to run some benchmark test on a datastore, then the option “Disable Storage DRS for this virtual machine” helps you to prevent Storage DRS from interrupting your test. However I would recommend selecting the datastore cluster as a destination instead of a specific datastore when migrating a virtual machine into a datastore cluster. Read the article Storage vMotion migration into a datastore cluster for more information. Remember Storage DRS always generate a recommendation that you can review during provisioning. After selecting the destination (datastore cluster), the user interface provides an overview of the current selections, at the right part of the screen a link “more recommendations” is provided. More recommendations After you click on the more recommendation link, the user interface provides you with a list of alternative recommendations. The order of the list is that the top recommendation provides the best placement, this is the same recommendation listed in the previous review selection screen. The list provides an overview of the space utilization before placement (2nd column), space utilization after placement (3rd column) and I/O latency before placement on the destination datastore (4th column). As the screenshot shows, the 2nd recommendation shows that placing the EMC-003 datastore provides the best placement. This datastore has the lowest utilization before and after placement and has the lowest I/O latency of all the other datastores inside the datastore cluster. Use this screen to educate your team responsible for provisioning and placement, show them that Storage DRS take multiple metrics into account and review the impact of the result if they picked the datastore of their choice. For my education, please share your thoughts on why you want to manually select a datastore that is a part of a datastore cluster? ================================================================================ Title: vSphere 5.1 web client: VM overrides -Storage DRS automation level overview URL: https://frankdenneman.ai/2013-01-09-vsphere-5-1-web-client-vm-overrides-storage-drs-automation-level-overview/ Date: 2013-01-09 Overall the vSphere 5.1 web client attempts to mimic the behavior of menus and settings workflows of the (old) vSphere client. When editing the settings of a datastore cluster, the web client provides the same set of options that can be edited as the vSphere client. However certain functions of overviews and menus are changed in the vSphere 5.1 web client. For example the VM overrides screen. The primary purpose of the VM overrides screen is to display deviant Storage DRS Automation level of the virtual machines inside the datastore cluster. VM overrides and Virtual Machine Settings screens The VM overrides screen is located in the storage view, select the datastore cluster, select the tab Manage and click on the Settings button. The VM overrides screen is the replacement of the virtual machine settings screen of the datastore cluster settings in the vSphere client. Difference in default behavior As you might have noticed, the web client is not listing any virtual machine while the Virtual Machine settings overview in the vSphere client shows 5 virtual machines and a VM template. Already mentioned in the introduction paragraph, the primary purpose of the VM overrides screen has changed from the Virtual Machine settings overview in the vSphere client. The VM Overrides screen only displays a virtual machine is set to a non-default automation level. To display the different behavior, I have change the Automation level of VM3, VM4, VM5. The datastore cluster is configured with a Manual Automation Mode. Therefor the default automation mode is Default (Manual). The previous screenshot shows that all virtual machines are configured with the Default (Manual) automation level, VM3 is changed to Fully Automated, VM4 to Manual and VM5 to disabled. If you want to reproduce this behavior in your own environment, change the automation level in the vSphere client and then go the VM overrides screen in the web client to see the modified virtual machines listed. The VM overrides screen displays the following: Even though VM4 is configured with the same automation level as the datastore cluster, the VM overrides screen displays VM4 as it is not configured with the default automation mode. By changing the automation mode back to Default (Manual) via the Edit screen, VM4 is removed from the VM overrides list. To be honest it took me a while to get used to the new functionality of this screen. I would like to know if you like this new behavior or if you rather prefer the way the virtual machine settings view in the old vSphere client works? ================================================================================ Title: Manual storage vMotion migrations into a datastore cluster URL: https://frankdenneman.ai/2013-01-08-manual-storage-vmotion-migration-into-datastore-cluster/ Date: 2013-01-08 Frequently I receive questions about the impact of a manual migration into a datastore cluster, especially about the impact of the VM disk file layout. Will Storage DRS take the initial disk layout into account or will it be changed? The short answer is that the virtual machine disk layout will be changed by the default affinity rule configured on the datastore cluster. The article describes several scenarios of migrating “distributed“ and “centralized” disk layout configurations into datastore cluster configured with different affinity rules. Test scenario architecture For the test scenarios I’ve build two virtual machines VM1 and VM2 Both virtual machines are of identical VM configuration, only the datastore location is different. VM1-centralized has a “centralized” configuration, storing all VMDKs on a single datastore, while VM2-distributed has a “distributed” configuration, storing all VMDKs on separate datastores. Hard disk Size VM 1 datastore VM 2 datastore Working directory 8GB FD-X4 FD-X4 Hard disk 1 60GB FD-X4 FD-X4 Hard disk 2 30GB FD-X4 FD-X5 Hard disk 1 10GB FD-X4 FD-X6 Two datastore clusters exists in the virtual infrastructure: Datastore cluster Default Affinity rule VMDK rule applied on VM Tier-1 VMs and VMDKs Do not keep VMDKs together Intra-VM Anti-affinity Tier-2 VMs and VMDKs Keep VMDKs together Intra-VM Affinity rule Test 1: VM1-centralized to Datastore Cluster Tier-2 VMs and VMDKs Since the virtual machine is stored on a single datastore is makes sense to start of migrating the virtual machine to the datastore cluster which applies a VMDK affinity rule, keeping the virtual machine disk files together on a single datastore in the datastore cluster.Select the virtual machine, right click the virtual machine to display the submenu and select the option “Migrate…”. The first step is to select the migration type, select change datastore. The second step is to select the destination datastore, as we are planning to migrate the virtual machine to a datastore cluster it is necessary to select the datastore cluster object. After clicking next, the user interface displays the Review Selection screen; notice that the datastore cluster applied the default cluster affinity rule. Storage DRS has evaluated the current load of the datastore cluster and the configuration of the virtual machine, it concludes that datastore nfs-f-05 is the best fit for the virtual machine, the existing virtual machines in the datastore cluster and the load balance state of the cluster. By clicking “more recommendations” other datastore destinations are presented. Test result: Intra-VM affinity rule applied and all virtual machine disk files are stored on a single datastore Selecting the Datastore cluster object The user interface provides you two options, select the datastore cluster object or a datastore that is part of the datastore cluster, however for that option you explicitly need to disable Storage DRS for this virtual machine. By selecting the datastore cluster, you fully leverage the strength of Storage DRS. Storage DRS initiates it’s algorithms and evaluate the current state of the datastore cluster. It reviews the configuration of the new virtual machine and is aware of the I/O load of each datastore as well as the space utilization. Storage DRS weigh both metrics and will weigh either space of I/O load heavier if the utilization is higher. Disable Storage DRS for this virtual machine By default it’s not possible to select a specific datastore that is a part of a datastore cluster during the second step “Select Datastore”. In order to do that, one must activate (tick the option box) the “Disable Storage for this virtual machine”. By doing so the datastores in the lower part of the screen are available for selection. However this means that the virtual machine will be disabled for any Storage DRS load balancing operation. Not only will it affect have an effect for the virtual machine itself, it also impacts other Storage DRS operations such as Maintenance Mode and Datastore Cluster defragmentation. As Storage DRS is not allowed to move the virtual machine, it cannot migrate the virtual machine to find an optimum load balance state when Storage DRS needs to make room for an incoming virtual machine. For more information about cluster defragmentation, read the following article: Storage DRS initial placement and datastore cluster defragmentation. Test 2: VM1-centralized to Datastore Cluster Tier-1 VMs and VMDKs Migrating a virtual machine stored on a single datastore to a datastore cluster with anti-affinity rules enabled results in a distribution of the virtual machine disk files: Test result: Intra-VM anti-affinity rule applied and the virtual machine disk files are placed on separate datastores. Working directory and default anti-affinity rules Please note that in the previous scenario the configuration file (working directory) is placed on the same datastore as Hard disk 3. Storage DRS does not forcefully attempt to place the working directory on a different datastore. It weighs the load balance state of the cluster heavier than separation from the virtual machine VMDK files. Test 3: VM2-distributed to Datastore Cluster Tier-1 VMs and VMDKs Following the example of VM1, I started off by migrating VM2-Distributed to Tier-1 as the datastore cluster is configured to mimic the initial state of the virtual machine and that is to distributed the virtual machine across as many datastores as possible. After selecting Datastore Cluster Tier-1 VM and VMDKs, Storage DRS provided the following recommendation: Test result: Intra-VM anti-affinity rule applied on VM and the virtual machine disk files are stored on separate datastores. A nice tidbit, as every virtual disk file is migrated between two distinct datastores, this scenario leverages the new functionality of parallel disk migration introduced in vSphere 5.1. Test 4: VM2-distributed to Datastore Cluster Tier-2 VMs and VMDKs What happens if you migrate a distributed virtual machine to a datastore cluster configured with a default affinity rule? Selecting Datastore Cluster Tier-2 VM and VMDKs, Storage DRS provided the following recommendation: Test result: Intra-VM affinity rule applied on VM and the virtual machines are placed on a single datastore cluster. Test 5: VM2-distributed to Multiple Datastore clusters A common use case is to distribute a virtual machine across multiple tiers of storage to provide performance while taken economics into account. This test simulates the exercise of placing the working directory and guest OS disk (Hard disk 1) on datastore cluster Tier 2 and the database and logging hard disk (Hard disk 2 and Hard disk 3) on datastore cluster Tier 1. In order to configure the virtual machine to use multiple datastores, click on the button Advanced during the second step of the migration: This screen shows the current configuration, by selecting the current datastore of a hard disk a browse menu appears: Select the appropriate datastore cluster for each hard disk and click on next to receive the destination datastore recommendation from Storage DRS. The working directory of the VM and Hard disk 1 are stored on datastore cluster Tier 2 and Hard disk 2 and Hard disk 3 are stored in datastore cluster Tier 1. As datastore cluster Tier 2 is configured to keep the virtual machine files together, both the working directory (designated as Configuration file in the UI) and Hard disk 1 are placed on datastore nfs-f-05. A default anti-affinity rule is applied to all new virtual machines in datastore cluster 2, therefore Storage DRS recommends to place Hard disk 2 on nfs-f-07 and Hard disk 3 on datastore nfs-f-01. Test result: Intra-VM anti-affinity rule applied on VM. The files stored in Tier-2 are placed on a single datastore, while the virtual machine disk files stored in the Tier-1 datastore are located on different datastores. Initial VM configuration Cluster default affinity rule Result Configured on: Centralized Affinity rule Centralized Entire VM Centralized Anti0Affinity rule Distributed Entire VM Distributed Anti-Affinity rule Distributed Entire VM Distributed Affinity rule Centralized Entire VM Distributed Affinity rule Centralized Working directory + Hard disk 1 Anti-Affinity rule Distributed Hard disk 2 and Hard disk 3 All types of migrations with the UI lead to a successful integration with the datastore cluster. Every migration results in an application of the correct affinity or anti-affinity rule set by the default affinity rule of the cluster. ================================================================================ Title: Storage DRS and Storage vMotion bugs solved in vSphere 5.0 Update 2. URL: https://frankdenneman.ai/2012-12-21-storage-drs-and-storage-vmotion-bugs-solved-in-vsphere-5-0-update-2/ Date: 2012-12-21 Today Update 2 for vSphere ESXI 5.0 and vCenter Server 5.0 were released. I would like to highlight two bugs that have been fixed in this update, one for Storage DRS and one for Storage vMotion Storage DRS vSphere ESXi 5.0 Update 2 was released today and it contains a fix that should be interesting to customers running Storage DRS on vSphere 5.0. The release note states the following bug: Adding a new hard disk to a virtual machine that resides on a Storage DRS enabled datastore cluster might result in Insufficient Disk Space error When you add a virtual disk to a virtual machine that resides on a Storage DRS enabled datastore and if the size of the virtual disk is greater than the free space available in the datastore, SDRS might migrate another virtual machine out of the datastore to allow sufficient free space for adding the virtual disk. Storage vMotion operation completes but the subsequent addition of virtual disk to the virtual machine might fail and an error message similar to the following might be displayed: Insufficient Disk Space In essence Storage DRS made room for the incoming virtual machine, but failed to place the new virtual machine. This update fixes a bug in the datastore cluster defragmentation process. For more information about datastore cluster defragmentation read the article: Storage DRS initial placement and datastore cluster defragmentation. Storage vMotion vCenter Server 5.0 Update 2 contains a fix that allows you to rename your virtual machine files with a Storage vMotion. vSphere 5 Storage vMotion is unable to rename virtual machine files on completing migration In vCenter Server , when you rename a virtual machine in the vSphere Client, the vmdk disks are not renamed following a successful Storage vMotion task. When you perform a Storage vMotion of the virtual machine to have its folder and associated files renamed to match the new name. The virtual machine folder name changes, but the virtual machine file names do not change. Duncan and I knew how many customers where relying on this feature for operational processes and pushed heavily to get it back in. We are very pleased to announce it’s back in vSphere 5.0, unfortunately this fix is not available in 5.1 yet! For more info about the fixes in the updates please review the release notes: ESXi 5.0 : https://www.vmware.com/support/vsphere5/doc/vsp_esxi50_u2_rel_notes.html vCenter 5.0: https://www.vmware.com/support/vsphere5/doc/vsp_vc50_u2_rel_notes.html ================================================================================ Title: Multi-NIC vMotion – failover order configuration URL: https://frankdenneman.ai/2012-12-20-multi-nic-vmotion-failover-order-configuration/ Date: 2012-12-20 After posting the article “designing your vMotion network” I quickly received the question which failover order configuration is better. Is it better to configure the redundant NIC(s) as standby or as unused? The short answer: always use standby and never unused! Tomas Fojta posted the comment that it does not make sense to place the NICs into standby mode: In the scenario as depicted in the diagram I prefer to use active/unused. If you think about it the standby option does not give you anything as when one of the NICs fails both vmknics will be on the same NIC which does not give you anything. Although it does not provide any performance benefits having vMotion routing the traffic across the same physical NIC during a NIC failure, there are two important reasons for providing redundant connection to each vmknic: Abstraction layer Using a default interface for management traffic Abstraction layer As mentioned in the previous article, vMotion operations are done at the vmknic level. Due to vMotion focussing on the vmknic layer instead of the physical layer, some details are abstracted from vMotion load balancing logic such as the physical NIC’s link state or health. Due to this abstraction vMotion just selects the appopriate vmknics for load balancing network packets and trusts that there is connectivity between the vmknics of the source and destination host. Default interface Although vMotion is able to use multiple vmknics to load balance traffic, vMotion assigns one vmknic as its default interface and prefers to use this for connection management and some trivial management transmissions. * As such, if you’ve got multiple physical NICs on a host that you plan to use for vMotion traffic, it makes sense to mark them as standby NICs for the other vMotion vmknics on the host. That way, even if you lose a physical NIC, you won’t see vMotion network connectivity issues. This means that if you have designated three physical NICs for vMotion your vmknic configuration should look as follows: VMknic Active NIC Standby NIC vmknic0 NIC1 NIC2, NIC3 vmknic1 NIC2 NIC1, NIC3 vmknic2 NIC3 NIC1, NIC2 By placing the redundant NICs into standby instead of unused you avoid the risk of having an unstable vMotion network. If a NIC fails, you might experience some vMotion performance degradation as the traffic gets routed through the same NIC, but you can trust your vMotion network to correclty migrate all virtual machines off the host in order for to replace the faulty NIC. * Word to the wise By writing about the fact that vMotion designates a vmknic to be the default interface I’m aware that this triggers and sparks the interest of some of the creative minds in our community. Please do not attempt to figure out which vmknic is designated as default interface and make that specific vmknic redundant and different from the rest. To paraphrase Albert Einstein: “Simplicity is the root of all genius”. Keep your Multi-NIC consistent and identical within the host and throughout all hosts. This saves you a lot of frustration during troubleshooting. Being able to depend on your vMotion network to migrate the virtual machines safely and correctly is worth its weight in gold. Part 1 - Designing your vMotion network Part 3 – Multi-NIC vMotion and NetIOC Part 4 – Choose link aggregation over Multi-NIC vMotion? Part 5 – 3 reasons why I use a distributed switch for vMotion networks ================================================================================ Title: Thin or thick disks? – it’s about management not performance URL: https://frankdenneman.ai/2012-12-19-thin-or-thick-disks-its-about-management-not-performance/ Date: 2012-12-19 This is my contribution to the debate Zero or Thick disks – debunking the performance myth. The last couple of years all sorts of VMware engineers worked very hard to reduce the performance difference between thin disks and thick disks. Many white-papers have been written by performance engineers to explain the improvements made on thin-disk. Therefore today the question whether to use Thin-provisioned disks or Eager zero thick is not about the difference in performance but the difference in management. When using Thin-provisioned VMDKs you need to have a very clear defined process. What to do, when your datastore, which stores the thin provisioned disks is getting full? You need to define a consolidation ratio, you need to understand which operational process might be dangerous to your environment (think Patch-Tuesday) and what space utilization threshold you need to define before migrating thin-provisioned disks to other datastores. Today Storage DRS can help you with many of the fore mentioned challenges. For more information please read the article: Avoiding VMDK level over-commitment while using Thin-provisioned disks and Storage DRS. If Storage DRS is not used, Thin-provisioned disks can require a seamless collaboration between virtualization teams (provisioning and architecture) and storage administrators. When this is not possible due to organizational cultural differences, thin provisioning is rather a risk, than bliss. Zero out process: Eager zero thick on the other hand might provide in some (corner) cases a marginal performance increase; the costs involved could outweigh the perceived benefits. First of all, Eager zero thick disks need to be zeroed out during creation, when your array doesn’t support the VAAI initiatives, this can take a hit on performance and the time to provision is extended. With terabyte sized disks becoming more common this will impact provisioning time immensely. Waste of space: Most virtualized environments use virtual machines, typically configured with oversized OS disks and over-specced data disks, resulting in wasted space full of zero’s. Thin-provisioned disks only occupy the space used for storing data, not zero’s. Migration: Storage vMotion goes out of its way to migrate every little bit of a virtual disk, this means it needs to copy over every zeroed out block. Combined with the oversized disks, you are creating unnecessary overhead on your hosts and storage subsystem copying and verifying the integrity of zeroed out blocks. Migrating thin disks only requires migrating the “user-data”, resulting in faster migration times, lesser overhead on hosts and storage subsystem. In essence, Thin-provisioned disks versus Eager zero thick is all about resource/time saving versus risk avoidance. Choose wisely ================================================================================ Title: vMotion Futures Survey URL: https://frankdenneman.ai/2012-12-19-vmotion-futures-survey/ Date: 2012-12-19 If you have a few spare minutes, please fill out the survey about vMotion futures. We are very interested in how you use vMotion and especially your opinion about use cases for Long-distance vMotion operations. It takes about 10 minutes to get through. http://tinyurl.com/VMwareVMotion2012 ================================================================================ Title: Designing your vMotion network URL: https://frankdenneman.ai/2012-12-18-designing-your-vmotion-network/ Date: 2012-12-18 A well designed vMotion network will benefit the environment in many ways. Before vSphere 5, designing a vMotion network was relative easy, select the fastest NIC and assign it to a vMotion vmknic. vSphere 4.x supports both 1GB and 10GB networks and since vSphere 5.0 vMotion is able to leverage multiple NICs. Multi-NIC in vSphere 5.x makes it a bit more challenging. The combination of NICs, the failover mode and which load balancing policy need to be taken into consideration when configuring your vMotion network. The benefit of multi-NIC vMotion network In vSphere 5.x vMotion balances the vMotion operations across all available NICs. Both for a single vMotion operation and for multiple concurrent vMotions operations. By using multiple NICs it reduces the duration of a vMotion operation. This benefits: Manual vMotion processes: Allocating more bandwidth to the vMotion process will result in faster migration times. The less time spend on monitoring a manual process means more time you can spend on other – more important - operations. DRS load balancing: Based on the average time per migration and the number of concurrent vMotion process, DRS restrict the number of load balancing operations it can process between two load balancing runs. By providing more bandwidth to the vMotion network, DRS is able to run more load balancing operations in between load balancing runs, which in turn benefits the load balance of the cluster. Better load balance means better resource availability for the virtual machines, which in turn means better performance for your applications. Maintenance mode: Faster migration times, means reducing the time a host enters maintenance mode. With the increase of consolidation ratio, the time it takes to migrate all the virtual machines off the host is increased as well. This can have impact on your SLA and the ability to service the host within the allowed time. Multi-NIC setup The vMotion process leverages the vmknic instead of sending packets directly to a physical NIC.In order to use multiple NICs for vMotion, multiple vmknics are required. Duncan wrote an excellent article how to setup multi-NIC vMotion network. Failover order and Load balancing policy Each vmknic should only use one active NIC, this result in only one valid load balancing policy and that is “Route based on originating virtual port”. This setup inhibits physical NIC load balancing such as IP-hash or the load balancing policy “based on physical load (LBT)”. The reason why the active/standby failover order is the only valid and supported configuration is because of the way vMotion load balancing works. The vMotion process itself handles load balancing, based on its own algorithm, vMotion picks a vmknic for a specific network packet. As vMotion expects the vmknic to be backed by a single physical NIC, sending out data through a given vmknic ensures vMotion that the data traverses that dedicated physical NIC. If the physical NICs were configured in a load-balancing mode, this could interfere with the vMotion level load balancing logic. vMotion would not be able to predict which physical NIC is used by the vmknic and possible sending all vMotion traffic over the same NIC, even if Motion sends the data to different vmknics. Consistent configuration Ensure that each physical NIC is configured in a consistent and correct way across the vMotion network on the host and across the hosts inside the cluster. vCenter is very conservative and if it detects a mismatch it drops the number of concurrent vMotion operations on that host back to 2 regardless of the available bandwidth. Therefore always check on each level if the link speed, MTU and duplex settings are identical, both on the NIC side as well as the switch side. Please keep in mind that the all the vMotion vmknics should exist in the same subnet. Although its possible to set static routes, “routable vMotion” configurations are not supported by VMware. Link speed By default vCenter allows 4 concurrent vMotion operations on a host with a 1 GB vMotion network and 8 concurrent vMotion operations on a host with a 10 GB vMotion network. Be aware that this is based on the detected link-speed. In other words, if the physical NIC reports at least 10GbE, link speed, vCenter allows 8 vMotions, but if the physical NIC reports less than 10GBe, vCenter allows a maximum of 4 concurrent vMotions on that host. To stress it again, the number of concurrent vMotions is based on the detected link speed. For example, take the HP Flex technology. This sets a hard limit on the flexnics, resulting in the reported link speed equal or less to the configured bandwidth on Flex virtual connect level. I’ve come across many Flex environments configured with more than 1GB bandwidth, ranging between 2GB to 8GB. Although they will offer more bandwidth per vMotion process, it will not offer an increase in the number of concurrent vMotions, limiting the number of concurrent vMotion operations to 4. As mentioned before, vCenter is very conservative; multi-NIC vMotion limits are currently determined by the slowest available vMotion NIC. This means that if you include a 1GB NIC in your 10GB vMotion network configuration, that host is restricted to maximum of 4 concurrent vMotion operations per host. I’ve seen a few vMotion network designs where a 1GB link was added to the 10GB vMotion network configuration, primarily used as a safety net. Just in case the 10GB network drops, well that safety net just restricted that host to 4 concurrent vMotion operations. Increase in NICS does not impact number of concurrent vMotions Using multiple NICs increases the bandwidth available for the vMotion process; it does not increase the number of concurrent vMotions. When 10Gb uplinks are used for the vMotion, the maximum concurrent vMotions allowed is eight, even with multiple NICs are assigned, the limit remains at eight. However the increase in bandwidth will decrease the duration of each vMotion process. A Multi-NIC vMotion configuration is slightly more complex that single NIC vMotion networks, but this setup will benefit you in many ways. Reducing vMotion operation times allows DRS to schedule more load balancing operations per invocation and the increased bandwidth allows the host to complete the transition to maintenance mode faster. Multi-VM is very helpful when leveraging the new vMotion possibility by migrating the virtual machine between hosts and datastores simultaneously. Part 2 - Multi-NIC vMotion failover order configuration Part 3 – Multi-NIC vMotion and NetIOC Part 4 – Choose link aggregation over Multi-NIC vMotion? Part 5 – 3 reasons why I use a distributed switch for vMotion networks ================================================================================ Title: Storage vMotion and the vSphere web-client URL: https://frankdenneman.ai/2012-12-18-storage-vmotion-and-the-vsphere-web-client/ Date: 2012-12-18 The new web client of vSphere 5.1 is my weapon of choice when working in my lab. It contains a lot of “hidden” gems, the UI team spends a lot of time crafting and aligning the user-interface to the administrator needs. One thing that drove me nuts was the lack of information when running a Storage vMotion operation. The Recent task doesn’t show anything, other than Storage vMotion operation itself and the target. When using Storage DRS, it only shows the name of the target Datastore cluster. Sometimes you just want to know which datastore the virtual machine migrated to. The new recent task window For example, take a look at the Recent Tasks window in the right size of the corner of the web client. When running a storage vMotion operation, it displays the Storage vMotion task similar to the vSphere client. However, when clicking on the task itself it shows the event info and task in one view. Helping you identify the source and destination datastore of the Storage vMotion process. Compare that to the workflow of the old vSphere client Select the task in the Recent task bar and double click it. This brings you in the Task and Events view of the target datastore cluster. By default the view is displaying events. Therefore you need to select Tasks, then select the task itself and then click on the related events in the bottom view. It may look trivial, but these things do speed up your work. Eradicating unnecessary clicks and wait time before a screen refreshes sure makes your job a little easier. ================================================================================ Title: SIOC on datastores backed by a single datapool URL: https://frankdenneman.ai/2012-12-06-sioc-on-datastores-backed-by-a-single-datapool/ Date: 2012-12-06 Duncan posted an article today in which he brings up the question: Should I use many small LUNs or a couple large LUNs for Storage DRS? In this article he explains the differences between Storage I/O Control (SIOC) and Storage DRS and why they work well together, to re-emphasize, the goal of Storage DRS load balancing is to fix long term I/O imbalances, while SIOC addresses short term burst and loads. SIOC is all about managing the queue’s while Storage DRS is all about intelligent placement and avoiding bottlenecks. Julian Wood makes an interesting remark, and both Duncan and I hear this remark when discussing SIOC. Don’t get me wrong I’m not picking on Julian, I’m merely stating the fact he made a frequently used argument. “There is far less benefit in using Storage IO Control to load balance IO across LUNs ultimately backed by the same physical disks than load balancing across separate physical storage pools. “ Well when you look at the way SIOC works I tend to disagree with this statement. As stated before, SIOC manages queues, queues to the datastores used by the virtual machines in the virtual datacenter. Typically speaking these virtual machines differ from workload types, from peak moments and also they differ in importance to the organization. With the use of disk shares, important virtual machine can be assigned a higher priority within the disk queue. When contention occurs, and this is important to realize, when contention occurs these business critical virtual machine get prioritized over other virtual machines. Not all important virtual machines generate a constant stream of I/O, while other virtual machines, maybe with a lower priority do generate a constant stream of IO. The disk shares provide the high priority low IO virtual machines to get a foot between the door and get those I/Os to the datastore and back. Without SIOC and disk shares you need to start thinking of increasing the queue depth of each hosts and think about smart placement of these virtual machines (both high and low I/O load) to avoid those high I/O load getting on the same host. These placement adjustment might impact DRS load balancing operations, possibly affecting other virtual machines along the way. Investing time in creating and managing a matrix of possible vm to datastore placement is not the way to go in this time with rapidly expanding datacenters. Because SIOC is a datastore-wide scheduler, SIOC determines the queue-depth of the ESX hosts connected to the datastores running virtual machines on those datastores. Hosts with higher priority virtual machines get “deeper” queue depths to the datastore and hosts with lower priority virtual machines running on the datastore receive shorter queue-depths. To be more precise, SIOC calculates the datastore wide latency and each local host scheduler determines the queue depth for the queues of the datastore. But remember queue depth changes only occur when there is contention, when the datastore exceeds the SIOC latency threshold. For more info about SIOC latency read “To which Host level latency statistic is the SIOC threshold related” Coming back to the argument, I firmly believe that SIOC has benefits in a shared diskpool structure, between the VMM and the datastore a lot of queue’s exists. Because SIOC takes the avg device latency off all hosts connected to the datastore into account, it understands the overall picture when determining the correct queue depth for the virtual machines. Keep in mind, queue depth changes occur only during contention. Now the best part of SIOC in 5.1 is that it has the Automatic Latency Threshold Computation. By leveraging the SIOC injector it understands the peak value of a datastore and adjust the SIOC threshold. The SIOC threshold will be set to 90% of its peak value, therefor having an excellent understanding of the performance capability of the datastore. This is done on a regular basis so it keeps actual workload in mind. This dynamic system will give you far more performance benefit that statically setting the queue-depth and DNSRO for each host. One of the main reasons of creating multiple datastores that are backed by a single datapool is because of creating a multi-path environment. Together with advanced multi-pathing policies and LUN to controller port mappings, you can get the most out of your storage subsystem. With SIOC, you can manage your queue depths dynamically and automatically, by understanding actually performance levels, while having the ability to prioritize on virtual machine level. ================================================================================ Title: Overlapping DRS VM-Host affinity rule in a vSphere Stretched Cluster URL: https://frankdenneman.ai/2012-12-05-overlapping-drs-vm-host-affinity-rule-in-a-vsphere-stretched-cluster/ Date: 2012-12-05 A question on the VMware community forum triggered me to validate DRS behavior of applying VM to Host group rules. The scenario describes a stretched HA cluster with overlapping DRS group rules, allowing to run particular VMs on hosts in a single site and a subset of hosts of both sides. How does DRS handle overlapping groups? The architecture In this scenario the stretched cluster contains four hosts; ESXi-01 and ESXi-02 are located in Site A, ESXi-03 and ESXi-04 are located in Site B. A collection of virtual machines are run by the hosts in the cluster, two management virtual machines, vCenter and vCenterDB are a part of this group. Storage housing the virtual machines are available for all hosts. Storage architecture is not the focus of this article, for more information about storage configurations and stretched vSphere clusters, please read the white paper: VMware vSphere Metro Storage Cluster Case Study Site DRS VM-Host groups ESXi-01 and ESXi-02 are grouped in Host DRS group Host-Site-A, ESXi-03 and ESXi-04 are grouped in Host DRS group Host-Site-B. All virtual machines running in Site-A are grouped in a VM DRS group VM-Site-A, all virtual machines running in Site-B are grouped in a VM DRS group VM-Site-B. All (Site) rules are configured as preferential rules (Should run on). Management DRS VM-Host group In the scenario described on the community forum, the virtual machines vCenter and vCenterDB are placed in an additional VM DRS group; MGMT-VMs. This group should be run on a select set of hosts of both sides, to simulate similar behavior the Host DRS group configured in my environment contains ESXi-02 of Site-A and ESXi-03 of Site-B. The group is named MGMT-Hosts. Overlapping rule-set Because the management virtual machines are a part of the VM-Site-A VM group an overlap of compatible hosts exists. Please note that DRS allows virtual machines to be a member of multiple VM groups. When reviewing the active affinity rules in a DRS cluster, DRS extracts a subset of compatible hosts for each virtual machine and uses this subset for placement and load balancing decisions. A Venn diagram shows the compatible host for the VMs vCenter and vCenterDB and specifically the host(s) listed in the compatibility set. This means that under normal operation DRS will choose to run the virtual machines on ESXi-02 as it satisfies both rules. With normal condition I want to indicate that there is not excessive load on any of the host and all hosts are configured identically without any hardware failures. Back to the scenario, what if there is a host failure or ESXi-02 is placed into maintenance mode? Maintenance mode Now what happens if ESXi-02 is placed into maintenance mode? As previously mentioned, DRS determines the set of compatible hosts. As ESXi-02 is placed into maintenance mode, DRS mark this host as a source host for migration. This way DRS knows which virtual machine to select for migration and excludes ESXi-02 as a valid destination for migrations. DRS must select other hosts listed in the Host DRS group. As ESXi-02 is not a valid destination, DRS needs to select either ESXi-01 or ESXi-03. Which host will it select? This depends on the creation date of the DRS rules, in other words, the newer DRS VM-Host rule is selected first. This is important to understand when applying overlapping VM-Host affinity rules in your environment. The last rule you create is applied first, overruling all other existing rules. In my lab, I created the MGMT rule as last, therefor having the youngest timestamp. There is no option of showing the timestamp in the user-interface of the web and vSphere client. I have provided this feedback to the engineering team. Hopefully they can include it in future releases. I placed ESXi-02 into maintenance mode and DRS migrated the virtual machines to ESXi-03, regulated by the MGMT VM-Host affinity rule. After placing ESXi-03 into maintenance mode, the virtual machines were moved to ESXi-01 as there were no compatible hosts available to satisfy the MGMT affinity rule. As ESXi-01 is listed in the Host-Site-A group of the Site-A affinity rule, DRS had no choice other than moving them to ESXi-01. After resetting the lab and destroying all the rules, I created the same set of host groups, VM groups and affinity rules, but I created the Site-A affinity rule as last. This resulted in the behavior that DRS moved the virtual machines to host ESXi-01 after placing ESXi-02 into maintenance mode, as DRS respected the Site-A affinity rule. Alarms As DRS supports non-contradicting overlapping affinity rules, no alarm was generated. During the scenario where both ESXi-02 and ESXi-03 were placed into maintenance mode, no alarm was triggered. It was expected to see an alarm that an affinity alarm is violated, however after digging through some code and contacting engineering this appears to be behavior by design. In the current release, the alarm is only triggered when mandatory rules (must run on) are violated. HA behavior All rules are configured with preferential rule sets (Should run on) and HA is not aware of DRS constructs. When a mandatory rule set (Must run on) is created, the hosts listed in the rule set are registered in the compatibility list of the virtual machine itself. Only those hosts registered in the compatibility list are viable destinations. During startup, HA checks the compatibility list and attempts to start up the virtual machine on any of these hosts listed. As the virtual machines are a part of a preferential affinity rule, all hosts are listed in the compatibility list and therefor HA could place them on a host outside the DRS Host Group. Conclusion If you want certain virtual machines to gravitate to specific hosts or a specific site, please take into account the way DRS sequence the active affinity rules. ================================================================================ Title: Calculating the bandwidth usage and duration of a vMotion process? URL: https://frankdenneman.ai/2012-12-04-calculating-the-bandwidth-usage-and-duration-of-a-vmotion-process/ Date: 2012-12-04 Every once in a while I get the question if I have a calculator that can determine the lead-time and the bandwidth consumption of a vMotion process. Unfortunately I haven’t got such a calculator, as there isn’t an easy way to calculate the consumed bandwidth and the duration of a vMotion process. CPU vMotion tries to move the used memory blocks as fast as possible. vMotion uses all the available bandwidth depending on the available CPU speed and bandwidth. Depending on the detected line speed, vMotion reserves an X amount of CPU speed at the start of a vMotion process. vMotion computes its desired host vMotion CPU reservation. For every 1GBe vMotion link speed it detects vMotion in vSphere 5.1 reserved 10% of a CPU core with a minimum desired CPU reservation of 30%. This means that if you use a single 1GBe, vMotion reserves 30% of a core, if you use 4 x 1GBe connections, that means vMotion reserves 40% of a core. A 10GBe link is special as vMotion reserves 100% of a single core. vMotion creates a (system) resource pool and sets the appropriate CPU reservation on the resource pool. It’s important to note that this is being done to the vMotion resource pool, which means that the reservation is shared across all vMotions happening on the host. Warning: DO NOT CHANGE the default settings of the system vMotion resource pool. This is set dynamically by the kernel depending on its memory state, manually adjusting this setting will likely hurt performance. Please do not attempt to be smarter then the kernel, many have tried, very few have succeeded. DRS When DRS is enabled, it can decide to migrate virtual machines as well. It might happen that this occurs at the same time your vMotion process is running. All vMotions will be placed into the vMotion resource pool contesting for the resources acquired by the resource pool of vMotion. If high priority for the manual vMotion is selected (User interface uses the term: Reserve CPU for optimal vMotion performance) then the vMotion process receives a higher priority within the vMotion resource pool. In which case the high priority vMotion will have double the relative CPU shares, and as a result probably complete more quickly than their lower priority counterparts. However it still need to share the resources claimed by the vMotion resource pool. Although it has a higher priority over DRS vMotions, sharing resources still may have an effect on the duration of the vMotion process. Memory vMotion copies only the used memory blocks, a virtual machine doesn’t always have to use all of its memory. Therefor its not easy to determine the required bandwidth. To make it more complex, as we are migrating a live virtual machine, the virtual machine can dirty (re-use) memory blocks that are already copied over, those blocks have to be sent again. Prolonging the duration of the process and the used bandwidth. Swap file If the swap file is located on a non-shared datastore and pages has been stored in the swap file, those pages are copied over to the new swap file on a location accessible by the destination host. This will increase the demand for bandwidth and increases the duration of the vMotion process. For more information about the impact of non-shared swap files, please read the following articles: (Alternative) VM swap file locations Q&A and (Alternative) VM swap file locations Q&A – part 2. Conclusion As you can see, it’s very difficult to determine the duration of a vMotion process and the actual bandwidth it consumes. ================================================================================ Title: How to create a "New Storage DRS recommendation generated" alarm URL: https://frankdenneman.ai/2012-11-30-how-to-create-a-new-storage-drs-recommendation-generated-alarm-2/ Date: 2012-11-30 It is recommended to configure Storage DRS in manual mode when you are new to Storage DRS. This way you become familiar with the decision matrix Storage DRS uses and you are able to review the recommendations it provides. One of the drawbacks of manual mode is the need to monitor the datastore cluster on a regular basis to discover if new recommendations are generated. As Storage DRS is generated every 8 hours and doesn’t provide insights when the next invocation run is scheduled, it’s becomes a bit of a guessing game when the next load balancing operation has occurred. To solve this problem, it is recommended to create a custom alarm and configure the alarm to send a notification email when new Storage DRS recommendations are generated. Here’s how you do it: Step 1: Select the object where the alarm object resides If you want to create a custom rule for a specific datastore cluster, select the datastore cluster otherwise select the Datacenter object to apply this rule to each datastore cluster. In this example, I’m defining the rule on the datastore cluster object. Step 2: Go to Manage and select Alarm Definitions Click on the green + icon to open the New Alarm Definition wizard Step 3: General Alarm options Provide the name of the alarm as this name will be used by vCenter as the subject of the email. Provide an adequate description so that other administrators understand the purpose of this alarm. In the Monitor drop-down box select the option “Datastore Cluster” and select the option “specific event occurring on this object, for example VM Power On”. Click on Next. Step 4: Triggers Click on the green + icon to select the event this alarm should be triggered by. Select “New Storage DRS recommendation generated”. The other fields can be left blank, as they are not applicable for this alarm. Click on next. Step 5: Actions Click on the green plus icon to create a new action. You can select “Run a Command”, “Send a notification email” and “Send a notification trap”. For this exercise I have selected “Send a notification email”. Specify the email address that will receive the messages containing the warning that Storage DRS has generated a migration recommendation. Configure the alarm so that it will send a mail once when the state changes from green to yellow and yellow to red. Click on Finish. The custom alarm is now listed between the pre-defined alarms. As I chose to define the alarm on this particular datastore cluster, vCenter list that the alarm is defined on “this Object”. This particular alarm is therefor not displayed at Datacenter level and cannot be applied to other datastore clusters in this vCenter Datacenter. Please note that you must configure a Mail server when using the option “send a notification email” and configure an valid SNMP receiver when using the option “Send a notification trap”. To configure a mail or SNMP server, select the vCenter server option in the inventory list, select manage, settings and click on edit. Go to Mail and provide a valid mail server address and an optional mail sender. To test the alarm, I moved a couple of files onto a datastore to violate the datastore cluster space utilization threshold. Storage DRS ran and displayed the following notifications on the datastore cluster summary screen and at the “triggered alarm” view: The moment Storage DRS generated a migration recommendation I received the following email: As depicted in the screenshot above, the subject of the email generated by vCenter contains the name of the alarm you specified (notice the exclamation mark), the event itself - New Storage DRS recommendation generated" and the datastore cluster in which the event occurred. ================================================================================ Title: How to create a "New Storage DRS recommendation generated" alarm URL: https://frankdenneman.ai/2012-11-30-how-to-create-a-new-storage-drs-recommendation-generated-alarm/ Date: 2012-11-30 It is recommended to configure Storage DRS in manual mode when you are new to Storage DRS. This way you become familiar with the decision matrix Storage DRS uses and you are able to review the recommendations it provides. One of the drawbacks of manual mode is the need to monitor the datastore cluster on a regular basis to discover if new recommendations are generated. As Storage DRS is generated every 8 hours and doesn’t provide insights when the next invocation run is scheduled, it’s becomes a bit of a guessing game when the next load balancing operation has occurred. To solve this problem, it is recommended to create a custom alarm and configure the alarm to send a notification email when new Storage DRS recommendations are generated. Here’s how you do it: Step 1: Select the object where the alarm object resides If you want to create a custom rule for a specific datastore cluster, select the datastore cluster otherwise select the Datacenter object to apply this rule to each datastore cluster. In this example, I’m defining the rule on the datastore cluster object. Step 2: Go to Manage and select Alarm Definitions Click on the green + icon to open the New Alarm Definition wizard Step 3: General Alarm options Provide the name of the alarm as this name will be used by vCenter as the subject of the email. Provide an adequate description so that other administrators understand the purpose of this alarm. In the Monitor drop-down box select the option “Datastore Cluster” and select the option “specific event occurring on this object, for example VM Power On”. Click on Next. Step 4: Triggers Click on the green + icon to select the event this alarm should be triggered by. Select “New Storage DRS recommendation generated”. The other fields can be left blank, as they are not applicable for this alarm. Click on next. Step 5: Actions Click on the green plus icon to create a new action. You can select “Run a Command”, “Send a notification email” and “Send a notification trap”. For this exercise I have selected “Send a notification email”. Specify the email address that will receive the messages containing the warning that Storage DRS has generated a migration recommendation. Configure the alarm so that it will send a mail once when the state changes from green to yellow and yellow to red. Click on Finish. The custom alarm is now listed between the pre-defined alarms. As I chose to define the alarm on this particular datastore cluster, vCenter list that the alarm is defined on “this Object”. This particular alarm is therefor not displayed at Datacenter level and cannot be applied to other datastore clusters in this vCenter Datacenter. Please note that you must configure a Mail server when using the option “send a notification email” and configure an valid SNMP receiver when using the option “Send a notification trap”. To configure a mail or SNMP server, select the vCenter server option in the inventory list, select manage, settings and click on edit. Go to Mail and provide a valid mail server address and an optional mail sender. To test the alarm, I moved a couple of files onto a datastore to violate the datastore cluster space utilization threshold. Storage DRS ran and displayed the following notifications on the datastore cluster summary screen and at the “triggered alarm” view: The moment Storage DRS generated a migration recommendation I received the following email: As depicted in the screenshot above, the subject of the email generated by vCenter contains the name of the alarm you specified (notice the exclamation mark), the event itself - New Storage DRS recommendation generated" and the datastore cluster in which the event occurred. ================================================================================ Title: vSphere 5.1 Clustering deepdive Cyber Monday deal URL: https://frankdenneman.ai/2012-11-26-vsphere-5-1-clustering-deepdive-cyber-monday-deal/ Date: 2012-11-26 We are long time fascinated by the whole Black Friday and Cyber Monday craze in the USA. Unfortunately we do not celebrate Thanksgiving in the Netherlands and none of the shops are participating in something similar as Black Friday. Similar to last year, we thought it was a great idea to participate in some form and what better than to offer our vSphere 5.1 Clustering Technical Deepdive book for a price you cannot resist. We just changed the price of the vSphere 5.1 Clustering Technical Deepdive to $17.95, Amazon Deutschland is offering the book for 16.00 EURO, while Amazon UK is selling the book for 11.01 Pounds sterling. The book has some amazing reviews, here is one we like to share with you: The book contains information critical to VMware administrators. Clustering is a critical technology, and the book covers the underlying concepts as well as the practical issues surrounding implementation. The information contained within will be important far past vSphere 5.1; the principles will apply for decades, and even the details of implementation are unlikely to change dramatically over the next few generations of the product. As with any good “deep dive,” the fundamental concepts discussed will ultimately help you in any clustering situation, even with non-VMware products. From a reading comprehension standpoint, the book is easy to grok. The information flows quickly and you can read the entire work cover to cover with relative ease. This is a must have for any systems administrator. What better way than recover from the madness of Black Friday and just sit back and relax reading this amazing piece of work? This is most definitely the deal of the year for all virtualization fanatics! Duncan and Frank ================================================================================ Title: (Alternative) VM swap file locations Q&A – part 2 URL: https://frankdenneman.ai/2012-11-19-alternative-vm-swap-file-locations-qa-part-2-2/ Date: 2012-11-19 After writing the article “(Alternative) VM swap file locations Q&A” I received a lot of questions about the destination of the swapped pages and reading back my article I didn’t do a good job clarifying that part of the process. Which network is used for copying swapped pages? As mentioned in the previous post the swap file itself is not copied over to the destination host, but only the swapped pages itself. Raphael Schitz (@hypervisor_fr) was the first to ask, which network is used to copy over the swapped pages? The answer is vMotion network. The reason why the vMotion network is used, is that the source host running the active virtual machine, pulls the swapped pages back in to memory when migrating the memory pages to the destination host. Are swapped pages on the source host swapped out on the destination host? As the pages are copied out from the swap file to the destination host, swapped pages are copied into the stream of the in-memory pages from the source host to the destination host. That means that the destination host is not aware which pages orginate from swap file and which pages come from in-memory, they are just memory pages that need to be stored and made available to the new virtual machine. To describe the behavior in a different way, the source host pulls the swapped pages from disk before sending them over, therefor the destination host sees a continues stream of memory pages, unmarked, all equal and are therefor stored in memory by the destination host. What if the destination host is experiencing contention? Well it’s up to the destination host to decide which pages to swap out to disk. During a vMotion process, the destination VM starts out with a clean slate, meaning that the memory target is not determined by the source host but by the destination host. Memory targets are local memory schedule metrics and thus not shared. The source host shares the percentage of active pages but it’s the destination hosts’ memory scheduler that determines the appropriate swap target for the new virtual machine. It can possibly push out memory pages back to its swap file as needed. The pages could be the same as the pages on the old host, but they can also completely different pages. What about compressed pages? For every rule there is an exception and the exception is compressed pages. During a vMotion process the destination host will maintain the compressed pages by keeping them compressed. This behavior occurs even with an unshared swap migration. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: (Alternative) VM swap file locations Q&A – part 2 URL: https://frankdenneman.ai/2012-11-19-alternative-vm-swap-file-locations-qa-part-2/ Date: 2012-11-19 After writing the article “(Alternative) VM swap file locations Q&A” I received a lot of questions about the destination of the swapped pages and reading back my article I didn’t do a good job clarifying that part of the process. Which network is used for copying swapped pages? As mentioned in the previous post the swap file itself is not copied over to the destination host, but only the swapped pages itself. Raphael Schitz (@hypervisor_fr) was the first to ask, which network is used to copy over the swapped pages? The answer is vMotion network. The reason why the vMotion network is used, is that the source host running the active virtual machine, pulls the swapped pages back in to memory when migrating the memory pages to the destination host. Are swapped pages on the source host swapped out on the destination host? As the pages are copied out from the swap file to the destination host, swapped pages are copied into the stream of the in-memory pages from the source host to the destination host. That means that the destination host is not aware which pages orginate from swap file and which pages come from in-memory, they are just memory pages that need to be stored and made available to the new virtual machine. To describe the behavior in a different way, the source host pulls the swapped pages from disk before sending them over, therefor the destination host sees a continues stream of memory pages, unmarked, all equal and are therefor stored in memory by the destination host. What if the destination host is experiencing contention? Well it’s up to the destination host to decide which pages to swap out to disk. During a vMotion process, the destination VM starts out with a clean slate, meaning that the memory target is not determined by the source host but by the destination host. Memory targets are local memory schedule metrics and thus not shared. The source host shares the percentage of active pages but it’s the destination hosts’ memory scheduler that determines the appropriate swap target for the new virtual machine. It can possibly push out memory pages back to its swap file as needed. The pages could be the same as the pages on the old host, but they can also completely different pages. What about compressed pages? For every rule there is an exception and the exception is compressed pages. During a vMotion process the destination host will maintain the compressed pages by keeping them compressed. This behavior occurs even with an unshared swap migration. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: (Alternative) VM swap file locations Q&A URL: https://frankdenneman.ai/2012-11-14-alternative-vm-swap-file-locations-qa-2/ Date: 2012-11-14 Lately I have received a couple of questions about Swap file placement. As I mentioned in the article “Storage DRS and alternative swap file locations”, it is possible to configure the hosts in the DRS cluster to place the virtual machine swapfiles on an alternative datastore. Here are the questions I received and my answer: Question 1: Will placing a swap file on a local datastore increase my vMotion time? Yes, as the destination ESXi host cannot connect to the local datastore, the file has to be placed on a datastore that is available for the new ESXi host running the incoming VM.Therefor the destination host creates a new swap file in its swap file destination. vMotion time will increase as a new file needs to be created on the local datastore of the destination host and swapped memory pages potentially need to be copied. Question 2: Is the swap file an empty file during creation or is it zeroed out? When a swap file is created an empty file equal to the size of the virtual machine memory configuration. This file is empty and does not contain any zeros. Please note that if the virtual machine is configured with a reservation than the swap file will be an empty file with the size of (virtual machine memory configuration – VM memory reservation). For example, if a 4GB virtual machine is configured with a 1024MB memory reservation, the size of the swap file will be 3072MB. Question 3: What happens with the swap file placed on a non-shared datastore during vMotion? During vMotion, the destination host creates a new swap file in its swap file destination. If the source swap file contains swapped out pages, only those pages are copied over to the destination host. Question 4: What happens if I have an inconsistent ESXi host configuration of local swap file locations in a DRS cluster? When selecting the option “Datastore specified by host”, an alternative swap file location has to be configured on each host separately. If one host is not configured with an alternative location, then the swap file will be stored in the working directory of the virtual machine. When that virtual machine is moved to another host configured with an alternative swap file location, the contents of the swap file is copied over to the specified location, regardless of the fact that the destination host can connect to the swap file in the working directory. Question 5: What happens if my specified alternative swap file location is full and I want to power-on a virtual machine? If the alternative datastore does not have enough space, the VMkernel tries to store the VM swap file in the working directory of the virtual machine. You need to ensure enough free space is available in the working directory otherwise the VM not allowed to be powered up. Question 6: Should I place my swap file on a replicated datastore? Its recommended placing the swap file on a datastore that has replication disabled. Replication of files increases vMotion time. When moving the contents of a swap file into a replicated datastore, the swap file and its contents need to replicated to the replica datastore as well. If synchronous replication is used, each block/page copied from the source datastore to the destination datastore, it needs to wait until the destination datastore receives an acknowledgement from its replication partner datastore (the replica datastore). Question 7: Should I place my swap file on a datastore with snapshots enabled? To save storage space and design for the most efficient use of storage capacity, it is recommended not to place the swap files on a datastore with snapshot enabled. The VMkernel places pages in a swap file if it’s there is memory pressure, either by an overcommitted state or the virtual machine requires more memory than it’s configured memory limit. It only retrieves memory from the swap file if it requires that particular page. The VMkernel will not transfer all the pages out of the swap file if the memory pressure on the host is resolved. It keeps unused swapped out pages in the swap file, as transferring unused pages is nothing more than creating system overhead. This means that a swapped out page could stay there as long as possible until the virtual machine is powered-off. Having the possibility of snapshotting idle and unused pages on storage could reduce the pools capacity used for snapshotting useful data. Question 8: Should I place my swap file on a datastore on a thin provisioned datastore (LUN)? This is a tricky one and it all depends on the maturity of your management processes. As long as thin provisioned datastore is adequately monitored for utilization and free space and controls are in place that ensures sufficient free space is available to cope with bursts of memory use, than it could be a viable possibility. The reason for the hesitation is the impact a thin provisioned datastores has on the continuity of the virtual machine. Placement of swap files by VMkernel is done at the logical level. The VMkernel determines if the swap file can be placed on the datastore based on its file size. That means that it checks the free space of a datastore reported by the ESX host, not the storage array. However the datastore could exist in a heavily over-provisioned datapool. Once the swap file is created the VMkernel assumes it can store pages in the entire swap file, see question 2 for swap file calculation. As the swap file is just an empty file until the VMkernel places a page in the swap file, the swap file itself takes up a little space on the thin disk datastore. Now this can go on for a long time and nothing will happen. But what if the total reservation consumed, memory overcommit-level and workload spikes on the ESXi host layer are not correlated with the available space in the thin provisioning storage pool? Understand how much space the datastore could possibly obtain and calculate the maximum configured size of all existing swap files on the datastore to avoid an Out-of space condition. (Alternative) VM swap file locations Q&A – part 2 Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: (Alternative) VM swap file locations Q&A URL: https://frankdenneman.ai/2012-11-14-alternative-vm-swap-file-locations-qa/ Date: 2012-11-14 Lately I have received a couple of questions about Swap file placement. As I mentioned in the article “Storage DRS and alternative swap file locations”, it is possible to configure the hosts in the DRS cluster to place the virtual machine swapfiles on an alternative datastore. Here are the questions I received and my answer: Question 1: Will placing a swap file on a local datastore increase my vMotion time? Yes, as the destination ESXi host cannot connect to the local datastore, the file has to be placed on a datastore that is available for the new ESXi host running the incoming VM.Therefor the destination host creates a new swap file in its swap file destination. vMotion time will increase as a new file needs to be created on the local datastore of the destination host and swapped memory pages potentially need to be copied. Question 2: Is the swap file an empty file during creation or is it zeroed out? When a swap file is created an empty file equal to the size of the virtual machine memory configuration. This file is empty and does not contain any zeros. Please note that if the virtual machine is configured with a reservation than the swap file will be an empty file with the size of (virtual machine memory configuration – VM memory reservation). For example, if a 4GB virtual machine is configured with a 1024MB memory reservation, the size of the swap file will be 3072MB. Question 3: What happens with the swap file placed on a non-shared datastore during vMotion? During vMotion, the destination host creates a new swap file in its swap file destination. If the source swap file contains swapped out pages, only those pages are copied over to the destination host. Question 4: What happens if I have an inconsistent ESXi host configuration of local swap file locations in a DRS cluster? When selecting the option “Datastore specified by host”, an alternative swap file location has to be configured on each host separately. If one host is not configured with an alternative location, then the swap file will be stored in the working directory of the virtual machine. When that virtual machine is moved to another host configured with an alternative swap file location, the contents of the swap file is copied over to the specified location, regardless of the fact that the destination host can connect to the swap file in the working directory. Question 5: What happens if my specified alternative swap file location is full and I want to power-on a virtual machine? If the alternative datastore does not have enough space, the VMkernel tries to store the VM swap file in the working directory of the virtual machine. You need to ensure enough free space is available in the working directory otherwise the VM not allowed to be powered up. Question 6: Should I place my swap file on a replicated datastore? Its recommended placing the swap file on a datastore that has replication disabled. Replication of files increases vMotion time. When moving the contents of a swap file into a replicated datastore, the swap file and its contents need to replicated to the replica datastore as well. If synchronous replication is used, each block/page copied from the source datastore to the destination datastore, it needs to wait until the destination datastore receives an acknowledgement from its replication partner datastore (the replica datastore). Question 7: Should I place my swap file on a datastore with snapshots enabled? To save storage space and design for the most efficient use of storage capacity, it is recommended not to place the swap files on a datastore with snapshot enabled. The VMkernel places pages in a swap file if it’s there is memory pressure, either by an overcommitted state or the virtual machine requires more memory than it’s configured memory limit. It only retrieves memory from the swap file if it requires that particular page. The VMkernel will not transfer all the pages out of the swap file if the memory pressure on the host is resolved. It keeps unused swapped out pages in the swap file, as transferring unused pages is nothing more than creating system overhead. This means that a swapped out page could stay there as long as possible until the virtual machine is powered-off. Having the possibility of snapshotting idle and unused pages on storage could reduce the pools capacity used for snapshotting useful data. Question 8: Should I place my swap file on a datastore on a thin provisioned datastore (LUN)? This is a tricky one and it all depends on the maturity of your management processes. As long as thin provisioned datastore is adequately monitored for utilization and free space and controls are in place that ensures sufficient free space is available to cope with bursts of memory use, than it could be a viable possibility. The reason for the hesitation is the impact a thin provisioned datastores has on the continuity of the virtual machine. Placement of swap files by VMkernel is done at the logical level. The VMkernel determines if the swap file can be placed on the datastore based on its file size. That means that it checks the free space of a datastore reported by the ESX host, not the storage array. However the datastore could exist in a heavily over-provisioned datapool. Once the swap file is created the VMkernel assumes it can store pages in the entire swap file, see question 2 for swap file calculation. As the swap file is just an empty file until the VMkernel places a page in the swap file, the swap file itself takes up a little space on the thin disk datastore. Now this can go on for a long time and nothing will happen. But what if the total reservation consumed, memory overcommit-level and workload spikes on the ESXi host layer are not correlated with the available space in the thin provisioning storage pool? Understand how much space the datastore could possibly obtain and calculate the maximum configured size of all existing swap files on the datastore to avoid an Out-of space condition. (Alternative) VM swap file locations Q&A – part 2 Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: VMware feature request URL: https://frankdenneman.ai/2012-11-12-vmware-feature-request/ Date: 2012-11-12 During presentations I always stress to submit a feature request if you have an idea how to enhance the product or if you feel you are missing a vital product feature. VMware is very interested to hear how the products can be enhanced and improved. Although it’s always good to talk to your local VMware rep or your favorite VMware blogger, submitted feedback might not reach the correct person on time. In order to have the feedback routed to the correct person using the shortest path available, it is best to submit a feature request via the VMware website. Unfortunately VMware.com doesn’t have an action button on the front-page, therefor I thought it might be a good idea to publish a short article with the link included. If you have any feedback go to the feature request page and submit your comments. Thanks! ================================================================================ Title: VAAI hw offload and Storage vMotion between two Storage Arrays URL: https://frankdenneman.ai/2012-11-06-vaai-hw-offload-and-storage-vmotion-between-two-storage-arrays/ Date: 2012-11-06 Recently I received a question about migrating virtual machines with Storage vMotion between two Storage Arrays. More specifically if VAAI is leveraged by Storage vMotion in this process. Unfortunately VAAI is an internal array based feature, the Clone Blocks VAAI feature Storage vMotion leverages is only used to copy and migrate data within the same physical array. Datamovers How does Storage vMotion work between two arrays? Storage vMotion uses a VMkernel component called the datamover. This component is moves the blocks from the source to the destination datastore, to be more precise; it handles the read and write blocks I/O from and to the source and destination datastores. The VMkernel used in vSphere 4.1 and up contains 2 different datamovers, software datamovers (FSDM and FS3DM) and a hardware offloading datamover (FS3DM-hardware offloading). The most efficient datamover is the FS3DM-hardware offload, followed by the FS3DM and as last the legacy datamover FSDM. FS3DM operates at kernel level, while the FSDM operates at the application level, the shorter the communication path the faster the operation. In essence Storage vMotion is travelling up to the stack of datamovers, trying the most efficient first, before reverting to a less optimal choice. To get an idea of difference in performance, please read the article “Storage vMotion performance difference” on Yellow-Bricks.com Traversing the datamover stack When a data movement operation is invoked (I.E. Storage vMotion) and the VAAI hardware offload operation is enabled, the data mover will first attempt to use the hardware offload. If the hardware offload operation fails, the data mover reverts to the software datamovers, first FS3DM, then FSDM. As you are migrating between arrays, hardware offloading will fail and the VMkernel selects a software datamover FS3DM. If the block-sizes of the datastore are not identical, then Storage vMotion has to revert to the FSDM datamover. If you are migrating data between NFS datastores than Storage vMotion immediately revert to the FSDM datamover. Impact on Storage DRS datastore cluster design Keep this in mind when designing Storage DRS datastore clusters. Storage DRS does not keep historical data of storage vMotion lead times, and thus it cannot incorporate these metrics when generating migration recommendations. Although no performance loss will occur within the virtual machine, migrating between arrays can create overhead on the supporting infrastructure. If possible design your datastores to contain datastores within the same array and use identical blocksizes (if VMFS is used) ================================================================================ Title: vSphere 5.1 Storage DRS Multi-VM provisioning improvement URL: https://frankdenneman.ai/2012-11-02-vsphere-5-1-storage-drs-multi-vm-provisioning-improvement/ Date: 2012-11-02 When a virtual machine is provisioned to the datastore cluster, Storage DRS algorithm runs to determine the best placement of the virtual machine. The interesting part of this process is the method Storage DRS determines the free space of a datastore or to be more precise the improvement made in vSphere 5.1 regarding free space calculation and the method of finding the optimal destination datastore. vSphere 5.0 Storage DRS behavior Storage DRS is designed to balance the utilization of the datastore cluster, it selects the datastore with the highest free space value to balance the space utilization of the datastores in the datastore cluster and avoids out-of-space situations. During the deployment of a virtual machine, Storage DRS initiates a simulation to generate an initial placement operation. This process is an isolated process and retrieves the current datastore free space values. However, when a virtual machine is deployed, the space usage of the datastore is updated once the virtual machine deployment is completed and the virtual machine is ready to power-on. This means that the initial placement process is unaware of any ongoing initial placement recommendations and pending storage space allocations. Let’s use an example that explains this behavior. In this scenario the datastore cluster contains 3 datastores, the size of each datastore is 1TB, no virtual machines are deployed yet, and therefor they each report a 100% free space. When deploying a 500GB virtual machine, storage DRS selects the datastore with the highest reported number of free space and as all three datastores are equal it will pick the first datastore, Datastore-01. Until the deployment process is complete the datastore remains reporting 1000GB of free space. When deploying single virtual machines this behavior is not a problem, however when deploying multiple virtual machines this might result in an unbalanced distribution of virtual machine across the datastores. As the available space is not updated during the deployment process, Storage DRS might select the same datastore, until one (or more) of the provisioning operations complete and the available free space is updated. Using the previous scenario, Storage DRS in vSphere 5.0 is likely to pick Datastore-01 again when deploying VM2 before the provisioning process of VM1 is complete, as all three datastore report the same free space value and Datastore-01 is the first datastore it detected. vSphere 5.1 Storage DRS behavior Storage DRS in vSphere 5.1 behaves differently and because Storage DRS in vSphere 5.1 supports vCloud Director, it was vital to support the provisioning process of a vApp that contains multiple virtual machines. Enter the storage lease Storage DRS in vSphere 5.1 applies a storage lease when deploying a virtual machine on a datastore. This lease “reserves” the space and making deployments aware of each other, thus avoiding suboptimal/invalid placement recommendations. Let’s use the deployment of a vApp as an example. The same datastore cluster configuration is used, each datastore if empty, reporting 1000GB free space. The vApp exists of 3 virtual machines, VM1, VM2 and VM3. Respectively they are 100GB, 200GB and 400GB in size. During the provisioning process, Storage DRS needs to select a datastore for each virtual machine. As the main goal of Storage DRS is to balance the utilization of the datastore cluster, it determines which datastore has the highest free space value after each placement during the simulation. During the simulation VM1 is placed on Datastore-01, as all three datastores report an equal value of free space. Storage DRS then applies the lease of 100GB and reduces the available free space to 900GB. When Storage DRS simulates the placement of VM2, it checks the free space and determines that Datastore-02 and Datastore-03 each have 1000GB of free space, while Datastore-01 reports 900GB. Although VM2 can be placed on Datastore-01 as it does not violate the space utilization threshold, Storage DRS prefers to select the datastore with the highest free space value. Storage DRS will choose Datastore-02 in this scenario as it picks the first datastore if multiple datastores report the same free space value. The simulations determines that the optimal destination for VM3 is Datastore-03, as this reports a free space value of 1000GB, while Datastore-02 reports 800 free space and Datastore-01 reports 900GB of space. This lease is applied during the simulation of placement for the generation of the initial placement recommendation and remains applied until the placement process of the virtual machine is completed or when the operation times out. This means that not only a vApp deployment is aware of the storage resource lease but also other deployment processes. Update to vSphere 5.1 This new behavior is extremely useful when deploying multiple virtual machines in batches such as vApp deployment or vHadoop environments with the use of Serengeti. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Add DRS cluster to existing Storage DRS Datastore Cluster? URL: https://frankdenneman.ai/2012-10-29-add-drs-cluster-to-existing-datastore-cluster/ Date: 2012-10-29 Lately I have seen the following question popping up at multiple places: “How can I add hosts of a DRS cluster to a Storage DRS datastore cluster after the datastore cluster is created?” This is an intriguing question as it gives insight to how datastore cluster construct is perceived and that a step in the “create datastore cluster” workflow in the user interface might be the culprit of this. The workflow: During the create datastore cluster, step 4 requires the use to connect the datastore cluster to a DRS cluster or stand-alone hosts. The reason for incorporating the step of selecting clusters and stand-alone hosts is to help narrow down the list of datastores that are presented in step 5. This way, one can create a datastore cluster that consists of datastores that are connected to all the hosts in a particular DRS cluster. The article “Partially connected datastore clusters” provide more information on the impact partially connected datastores on Storage DRS load balancing. In short, the screen “Select Clusters and Hosts” is just a filter, no host to datastore connectivity is altered by this step. To prove this theory, I attached the datastores nfs-f-04, nfs-f-05, nfs-f-06 and nfs-f-07 to all the hosts in Cluster01 and Cluster02. However I selected the Cluster 01 in step 4. Step 5 provides the following overview, indicating that all the available datastores are connected to the hosts in Cluster01. When you review the “Ready to Complete” screen, scroll down to the “Cluster and Host” overview, this screen shows that all the datastores are connected to Cluster01 and Cluster02, but only Cluster01 is selected. In my opinion this screen doesn’t make sense and I’m already working with the User Interface team and engineering to see if we can make some adjustments. (No promises though)! Although this Selected column can make this screen a little confusing, in essence it displays the user selection during the configuration process. The “Datastore Connection Status” is the key message of this view. Once complete, select the Storage view, select the new datastore cluster and select the Cluster tab. This view shows that both clusters are connected and can utilize the datastore cluster as destination for virtual machine placement. Another check is the “Standalone Hosts” tab. This displays the connectivity state of the hosts to the datastore cluster. Host *h09* and *h10* are part of Cluster01, host *h11* and *h12* are part of Cluster02. In essence Remember I didn’t select Cluster02 during the datastore cluster configuration, yet Cluster02 and its hosts are still connected. Long story short: DRS Cluster and Storage DRS Datastore clusters are independent load balancing domain constructs. In the end it drills down to the host to datastore connectivity, remember partially connected datastores can still be a part of a datastore cluster. DRS and Storage DRS take the connectivity of hosts and datastores into account during the initial placement and load balancing process, not the cluster constructs. Before creating a datastore cluster, ensure the following: • Check Storage adapter settings • Check Array configuration (masking, LUN ID, exports) • Check zoning • Rescan hosts that are going to use the datastores inside the datastore cluster. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: vMotion bug fixed in vCenter server 5.1.0a URL: https://frankdenneman.ai/2012-10-29-vmotion-bug-fixed-in-vcenter-server-5-1-0a/ Date: 2012-10-29 Looking for the “Designing your vMotion network”? Please follow the link. Last week VMware vCenter Server™ 5.1.0a was released which contains a bugfix for Essential plus license customers. A few readers provided me feedback about being unable to initiate the new vMotion that migrates both host and datastore state of the virtual machine. Due to the feedback and the filed SRs we got to the bottom of the bug pretty quickly and got the bugfix in this release. vMotion and Storage vMotion Unable to access the cross-host Storage vMotion feature from the vSphere Web Client with an Essentials Plus license If you start the migration wizard for a powered on virtual machine with an Essentials Plus license, the Change both host and datastore option in the migration wizard is disabled, and the following error message is displayed: Storage vMotion is not licensed on this host. To perform this migration without a license, power off the virtual machine. This issue is resolved in this release. https://www.vmware.com/support/vsphere5/doc/vsphere-vcenter-server-510a-release-notes.html Thanks for the feedback and above all, thanks for filing the SRs providing us useful data. You can download the update here. ================================================================================ Title: vSphere 5.1 Storage DRS load balancing and SIOC threshold enhancements URL: https://frankdenneman.ai/2012-10-19-vsphere-5-1-storage-drs-load-balancing-and-sioc-threshold-enhancements/ Date: 2012-10-19 Lately I have been receiving questions on best practices and considerations for aligning the Storage DRS latency and Storage IO Control (SIOC) latency, how they are correlated and how to configure them to work optimally together. Let’s start with identifying the purpose of each setting, review the enhancements vSphere 5.1 has introduced and discover the impact when misaligning both thresholds in vSphere 5.0. Purpose of the SIOC threshold The main goal of the SIOC latency is to give fair access to the datastores, throttling virtual machine outstanding I/O to the datastores across multiple hosts to keep the measured latency below the threshold. It can have a restrictive effect on the I/O flow of virtual machines. Purpose of the Storage DRS latency threshold The Storage DRS latency is a threshold to trigger virtual machine migrations. To be more precise, if the average latency (VMObservedLatency) of the virtual machines on a particular datastore is higher than the Storage DRS threshold, then Storage DRS will mark that datastore as “source” for load balancing migrations. In other words, that datastore provide the candidates (virtual machines) for Storage DRS to move around to solve the imbalance of the datastore cluster. This means that the Storage DRS threshold metric has no “restrictive” access limitations. It does not limit the ability of the virtual machines to send I/O to the datastore. It is just an indicator for Storage DRS which datastore to pick for load balance operations. SIOC throttling behavior When the average device latency detected by SIOC is above the threshold, SIOC throttles the outstanding IO of the virtual machines on the hosts connected to that datastore. However due to different number of shares, various IO sizes, random versus sequential workload and the spatial locality of the changed blocks on the array, we are almost certain that no virtual machine will experience the same performance. Some virtual machines will experience a higher latency than other virtual machines running on that datastore. Remember SIOC is driven by shares, not reservations, we cannot guarantee IO slots (reservations). Long story short, when the datastore is experiencing latency, the VMkernel manages the outbound queue, resulting in creating a buildup of I/O somewhere higher up in the stack. As the SIOC latency threshold is the weighted average of D/AVG per host, the weight is the number of IOPS on that host. For more information how SIOC calculates the Device average latency, please read the article: “To which host-level latency statistic is the SIOC congestion threshold related?” Has SIOC throttling any effect on Storage DRS load balancing? Depending on which vSphere version you run is the key whether SIOC throttling has impact on Storage DRS load balancing. As stated in the previous paragraph, if SIOC throttles the queues, the virtual machine I/O does not disappear, vSphere always allows the virtual machine to generate I/O to the datastore, it just builds up somewhere higher in the stack between the virtual machine and the HBA queue. In vSphere 5.0, Storage DRS measures latency by averaging the device latency of the hosts running VMs on that datastore. This is almost the same metric as the SIOC latency. This means that when you set the SIOC latency equal to the Storage DRS latency, the latency will be build up in the stack above the Storage DRS measure point. This means that in worst-case scenario, SIOC throttles the I/O, keeping it above the measure point of Storage DRS, which in turn makes the latency invisible to Storage DRS and therefore does not trigger the load balance operation for that datastore. Introducing vSphere 5.1 VMObservedLatency To avoid this scenario Storage DRS in vSphere 5.1 is using the metric VMObservedLatency. This metric measures the round-trip of I/O from the moment the VMkernel receives the I/O (Virtual Machine Monitor) to the datastore and all the way back to the VMM. This means that when you set the SIOC latency to a lower threshold than the Storage DRS latency, Storage DRS still observes the latency build up in the kernel layer. vSphere 5.1 Automatic latency SIOC To help you avoid building up I/O in the host queue, vSphere 5.1 offers automatic threshold computation for SIOC. SIOC sets the latency to 90% of the throughput level of the device. To determine this, SIOC derives a latency setting after a series of tests, mapping maximum throughput to a latency value. During the tests SIOC detects where the throughput of I/O levels out, while the latency keeps on increasing. To be conservative, SIOC derives a latency value that allows the host to generate up to 90% of the throughput, leaving a burst space of 10%. This provides the best performance of the devices, avoiding unnecessary restrictions by building up latency in the queues. In my opinion, this feature alone warrants the upgrade to vSphere 5.1. How to set the two thresholds to work optimally together SIOC in vSphere 5.1 allows the host to go up to 90% of the throughput before adjusting the queue length of each host, and generate queuing in the kernel instead of queuing on the storage array. As Storage DRS uses VMObservedLatency it monitors the complete stack. It observes the overall latency, disregarding the location of the latency in the stack and tries to move VMs to other datastores to level out the overall experienced latency in the datastore cluster. Therefore you do not need to worry about misaligning the SIOC latency and the Storage DRS I/O latency. If you are running vSphere 5.0 it’s recommended setting the SIOC threshold to a higher value than the Storage DRS I/O latency threshold. Please refer to your storage vendor to receive the accurate SIOC latency threshold. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Storage DRS and alternative swap file locations URL: https://frankdenneman.ai/2012-10-18-storage-drs-and-alternative-swap-file-locations/ Date: 2012-10-18 By default a virtual machine swap file is stored in the working directory of the virtual machine. However, it is possible to configure the hosts in the DRS cluster to place the virtual machine swapfiles on an alternative datastore. A customer asked if he could create a datastore cluster exclusively used for swap files. Although you can use the datastores inside the datastore cluster to store the swap files, Storage DRS does not load balance the swap files inside the datastore cluster in this scenario. Here’s why: Alternative swap location This alternative datastore can either be a local datastore or a shared datastore. Please note that placing the swapfile on a non-shared local datastore impacts the vMotion lead-time, as the vMotion process needs to copy the contents of the swap file to the swapfile location of the destination host. However a datastore that is shared by the host inside the DRS cluster can be used, a valid use case is a small pool of SSD drives, providing a platform that reduces the impact of swapping in case of memory contention. Storage DRS What about Storage DRS and using alternative swap file locations? By default a swap file is placed in the working directory of a virtual machine. This working directory is “encapsulated” in a DRMdisk. A DRMdisk is the smallest entity Storage DRS can migrate. For example when placing a VM with 3 hard disks in a datastore cluster, Storage DRS creates 4 DRMdisks, one DRMdisk for the working directory and separate DRMdisks for each hard disk. Extracting the swap file out of the working directory DRMdisk When selecting an alternative location for swap files, vCenter will not place this swap file in the working directory of the virtual machine, in essence extracting it from – or placing it outside - the working directory DRMdisk entity. Therefore Storage DRS will not move the swap file during load balancing operations or maintenance mode operations as it can only move DRMdisks. Alternative Swap file location a datastore inside a datastore cluster? Storage DRS does not encapsulate a swap file in its own DRMdisk and therefore it is not recommended to use a datastore that is part of a datastore cluster as a DRS cluster swap file location. As Storage DRS cannot move these files, it can impact load-balancing operations. The user interface actually reveals the incompatibility of a datastore cluster as a swapfile location because when you configure the alternate swap file location, the user interface only displays datastores and not a datastore cluster entity. DRS responsibility Storage DRS can operate with swap files placed on datastores external to the datastore cluster. It will move the DRMdisks of the virtual machines and leave the swap file on its specified location. It is the task of DRS moving the swap file if it’s placed on a non-shared swap file datastore when migrating the compute state between two hosts. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: HA admission control is not a capacity management tool. URL: https://frankdenneman.ai/2012-10-17-ha-admission-control-is-not-a-capacity-management-tool/ Date: 2012-10-17 I receive a lot of questions on why HA doesn’t work when virtual machines are not configured with VM-level reservations. If no VM-level reservations are used, the cluster will indicate a fail over capacity of 99%, ignoring the CPU and memory configuration of the virtual machines. Usually my reply is that HA admission control is not a capacity management tool and I noticed I have been using this statement more and more lately. As it doesn’t scale well explaining it on a per customer basis, it might be a good idea to write a blog article about it. The basics Sometimes it’s better to review the basics again and understand where the perception of HA and the actual intended purpose of the product part ways. Let’s start of what HA admission control is designed for. In the availability guide the two following statement can be found: Quote 1: “vCenter Server uses admission control to ensure that sufficient resources are available in a cluster to provide failover protection and to ensure that virtual machine resource reservations are respected.” Let’s dive in the first quote and especially this statement: “To ensure that sufficient resources are available in a cluster” is the key element, and in particular the word sufficient (resources). What sufficient means for customer A, does not mean sufficient for customer B. As HA does not have an algorithm decoding the meaning of the word sufficient for each customer, HA relies on the customer to set vSphere resource management allocation settings to indicate the importance of resource availability for the virtual machine during resource contention scenarios. As we are going back to the basics, lets have a quick look at the resource allocation settings that are used in this case, reservations and shares. A reservation indicates the minimum level of resources available to the virtual machine at all times. This reservation guarantees – or protect might be a better word –the availability of physical resources to the virtual machine regardless of the level of contention. No matter how high the contention in the system is, the reservation restricts the VMkernel from reclaiming that particular CPU cycle or memory page. This means that when a VM is powered on with a reservation, admission control needs to verify if the host can provide these resources at all times. As the VMkernel cannot reclaim those resources, admission control makes sure that when it lets the virtual machine in, it can hold its promise of providing these resources all the time, but also checks if it won’t introduce problems for the VMkernel itself and other virtual machines with a reservation. This is the reason why I like to call admission control the virtual bouncer. Besides reservation we have shares and shares indicates the relative priority of resource access during contention. A better word to describe this behavior is “opportunistic access”. As the virtual machine is not configured with a reservation, it provides the VMkernel with a more relaxed approach of resource distribution. When resource contention occurs, VMkernel does not need to provide the configured resources all the time, but can distribute the resources based on the activity and the relative priority based on the shares of the virtual machines requesting the resources. Virtual machines configured only with shares will just receive what they can get; there is no restrictive setting for the VMkernel to worry about when running out of resources. Basically the virtual machines will just get what’s left. In the case of shares, it’s the VMkernel that decides which VM gets how many resources in a relaxed and very social way, where virtual machines configured with a reservation DEMAND to have the reservations available at all times and do not care about the needs of others. In other words, the VMkernel MUST provide the resources to the virtual machine with reservation first and then divvy up the rest amongst the virtual machines who opted for a opportunistic distribution (shares). How does this tie in with HA admission control? The second quote gives us this insight: “vSphere HA: Ensures that sufficient resources in the cluster are reserved for virtual machine recovery in the event of host failure.” We know that admission control checks if there is enough resources are available to satisfy the VM-level reservation without interfering with VMkernel operations or VM-level reservations of other virtual machines running on that host. As HA is designed to provide an automated method of host failure recovery, we need to make sure that once a virtual machine is up and running it can continue to run on another host in the cluster if the current hosts fails. Therefor the purpose of HA admission control is to regulate and check if there are enough resources available in the cluster that can satisfy the virtual machine level reservations after a host failure occurs. Depending on the admission control policy it calculates the capacity required for a failover based on available resources and still comply with the VMkernel resource management rules. Therefor it only needs to look at VM-level reservations, as shares will follow the opportunistic access method. Semantics of sufficient resources while using shares-only design In essence, HA will rely on you to determine if the virtual machine will receive the resources you think are sufficient if you use shares. The VMkernel is designed to allow for memory overcommitment while providing performance. HA is just the virtual bouncer that counts the number of heads before it lets the virtual machine in “the club”. If you are on the list for a table, it will get you that table, if you don’t have a reservation HA does not care if you decide to need to sit at a 4-person table with 10 other people fighting for your drinks and food. HA relies on the waiters (resource management) to get you (enough) food as quickly as possible. If you wanted to have a good service and some room at your table, it’s up to you to reserve. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Want to have a vSphere 5.1 clustering deepdive book for free? URL: https://frankdenneman.ai/2012-10-17-want-to-have-a-vsphere-5-1-clustering-deepdive-book-for-free/ Date: 2012-10-17 Want to have a vSphere 5.1 clustering deepdive book for free? CloudPhysics are giving away some vSphere 5.1 clustering deepdive books, do the following if you want to receive a copy: Action required Email info@cloudphysics.com with a subject of “Book”. No message is needed. Register at http://www.cloudphysics.com/ by clicking “SIGN UP”. Install the CloudPhysics Observer vApp to activate your dashboard. Eligibility rules You are a new CloudPhysics user. You fully install the CloudPhysics ‘Observer’ vApp in your vSphere environment. The first 150 users gets a free book, but what’s even better, the Cloudphysics service gives you great insights on your current environment. For more info read the following blogposts: CloudPhysics in a nutshell and VM reservations and limits card - a closer look ================================================================================ Title: Partially connected datastore clusters - where can I find the warnings and how to solve it via the web client? URL: https://frankdenneman.ai/2012-10-15-partially-connected-datastore-clusters-where-can-i-find-the-warnings-and-how-to-solve-it-via-the-web-client/ Date: 2012-10-15 During my Storage DRS presentation at VMworld I talked about datastore cluster architecture and covered the impact of partially connected datastore clusters. In short – when a datastore in a datastore cluster is not connected to all hosts of the connected DRS cluster, the datastore cluster is considered partially connected. This situation can occur when not all hosts are configured identically, or when new ESXi hosts are added to the DRS cluster. The problem I/O load balancing does not support partially connected datastores in a datastore cluster and Storage DRS disables the IO load balancing for the entire datastore cluster. Not only on that single partially connected datastore, but the entire cluster. Effectively degrading a complete feature set of your virtual infrastructure. Therefore having an homogenous configuration throughout the cluster is imperative. Warning messages An entry is listed in the Storage DRS Faults window. In the web vSphere client: 1. Go to Storage 2. Select the datastore cluster 3. Select Monitor 4. Storage DRS 5. Faults. The connectivity menu option shows the Datastore Connection Status, in the case of a partially connected datastore, the message Datastore Connection Missing is listed. When clicking on the entry, the details are shown in the lower part of the view: Returning to a fully connected state To solve the problem, you must connect or mount the datastores to the newly added hosts. In the web client this is considered a host-operation, therefore select the datacenter view and select the hosts menu option. 1. Right-click on a newly added host 2. Select New Datastore 3. Provide the name of the existing datastore 4. Click on Yes when the warning “Duplicate NFS Datastore Name” is displayed. 5. As the UI is using existing information, select next until Finish. 6. Repeat steps for other new hosts. After connecting all the new hosts to the datastore, check the connectivity view in the monitor menu of the of the datastore cluster Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: vSphere 5.1 DRS advanced option LimitVMsPerESXHost URL: https://frankdenneman.ai/2012-10-10-vsphere-5-1-drs-advanced-option-limitvmsperesxhost/ Date: 2012-10-10 During the Resource Management Group Discussion here in VMworld Barcelona a customer asked me about limiting the number of VMs per Host. vSphere 5.1 contains an advanced option on DRS clusters to do this. If the advanced option: “LimitVMsPerESXHost” is set, DRS will not admit or migrate more VMs to the host than that number. For example, when setting the LimitVMsPerESXHost to 40, each host allows up to 40 virtual machines. No correction for existing violation Please note that DRS will not correct any existing violation if the advanced feature is set while virtual machines are active in the cluster. This means that if you set LimitVMsPerESXHost to 40 and at the time 45 virtual machines are running on an ESX host, DRS will not migrate the virtual machines out of that host. However It does not allow any more virtual machines on the host. DRS will not allow any power-ons or migration to the host, both manual (by administrator) and automatic (by DRS). High Availability As this is a DRS cluster setting, HA will not honor the setting during a host failover operation. This means that HA can power on as many virtual machines on a host it deems necessary. This is to avoid any denial of service by not allowing virtual machines to power-on if the “LimitVMsPerESXHost” is set too conservative. Impact on load balancing Please be aware that this setting can impact VM happiness. This setting can restrict DRS in finding a balance with regards to CPU and Memory distribution. Use cases This setting is primary intended to contain the failure domain. A popular analogy to describe this setting would be “Limiting the number of eggs in one basket”. As virtual infrastructures are generally dynamic, try to find a setting that restricts the impact of a host failure without restricting growth of the virtual machines. I’m really interested in feedback on this advanced setting, especially if you consider implementing it, the use case and if you want to see this setting to be further developed. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Changing your vCenter logging level using the webclient URL: https://frankdenneman.ai/2012-10-05-changing-your-vcenter-logging-level-using-the-webclient/ Date: 2012-10-05 In order to monitor some load behavior I needed to increase the logging level of the vCenter server. The logging level still is included in the vCenter Server Settings, however it takes a few more clicks to get the Statistics option compared to the old vSphere client. 1. In the home screen click on vCenter. 2. click on the vCenter Servers link in the Inventory list. 3. Select the vCenter (probably Localhost). 4. Select the Manage tab in the right-pane. 5. Select Setting. 6. Click on the Edit button located on the far end right side of the screen. 7. Change the appropriate Statistic setting. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Cloudphysics VM Reservation & Limits card – a closer look URL: https://frankdenneman.ai/2012-10-05-cloudphysics-vm-reservation-limits-card-a-closer-look-2/ Date: 2012-10-05 The VM Reservation and Limits card was released yesterday. CloudPhysics decided to create this card based on the popularity of this topic in the contest. So what does this card do? Let’s have a closer look. This card provides you an easy overview of all the virtual machines configured with any reservation or limits for CPU and memory. Reservations are a great tool to guarantee the virtual machine continuous access to physical resources. When running business critical applications reservations could provide a constant performance baseline that helps you meet your SLA. However reservations can impact your environment as the VM reservations impacts the resource availability of other virtual machines in your virtual infrastructure. It can lower your consolidation ratio: The Admission Control Family and it can even impact other vSphere features such as vSphere High Availability. The CloudPhysics HA Simulation card can help you understand the impact of reservations on HA. Besides reservations virtual machine limits are displayed. A limit restricts the use of physical access of the virtual machine. A limit could be helpful to test the application during various level of resource availability. However virtual machine limits are not visible to the Guest OS, therefor it cannot scale and size its own memory management (or even worse the application memory management) to reflect the availability of physical memory. For more information about memory limits, please read this post by Duncan: Memory limits. As the VMkernel is forced to provide alternative memory resources limits can lead to the increased use of VM swap files. This can lead to performance problems of the application but can also impact other virtual machines and subsystems used in the virtual infrastructure. The following article zooms into one of the many problems when relying on swap files: Impact of host local VM swap on HA and DRS. Color indicators As virtual machine level limits can impact the performance of the entire virtual infrastructure, the CloudPhysics engineers decided to add an additional indicator to help you easily detect limits. When a virtual machine is configured with a memory limit still greater than 50% of its configured size an Amber dot is displayed next to the configured limit size. If the limit is smaller or equal to 50% of its configured size than a red dot is displayed next to the limit size. Similar for CPU limits, an amber dot is displayed when the limit of a virtual machine is set but is more than 500MHz, a red dot indicates that the virtual machine is configured with a CPU limit of 500MHz or less. For example: Virtual Machine Load06 is configured with 16GB of memory. A limit is set to 8GB (8192MB), this limit is equal to 50% of the configured size. Therefore the VM reservation and Limits card displays the configured limit in red and presents an additional red dot. Flow of information The indicators are also a natural divider between the memory resource controls and the CPU controls. As memory resource control impacts the virtual infrastructure more than the CPU resource controls, the card displays the memory resource controls at the left side of the screen. We are very interested in hearing feedback about this card, please leave a comment. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Cloudphysics VM Reservation & Limits card – a closer look URL: https://frankdenneman.ai/2012-10-05-cloudphysics-vm-reservation-limits-card-a-closer-look/ Date: 2012-10-05 The VM Reservation and Limits card was released yesterday. CloudPhysics decided to create this card based on the popularity of this topic in the contest. So what does this card do? Let’s have a closer look. This card provides you an easy overview of all the virtual machines configured with any reservation or limits for CPU and memory. Reservations are a great tool to guarantee the virtual machine continuous access to physical resources. When running business critical applications reservations could provide a constant performance baseline that helps you meet your SLA. However reservations can impact your environment as the VM reservations impacts the resource availability of other virtual machines in your virtual infrastructure. It can lower your consolidation ratio: The Admission Control Family and it can even impact other vSphere features such as vSphere High Availability. The CloudPhysics HA Simulation card can help you understand the impact of reservations on HA. Besides reservations virtual machine limits are displayed. A limit restricts the use of physical access of the virtual machine. A limit could be helpful to test the application during various level of resource availability. However virtual machine limits are not visible to the Guest OS, therefor it cannot scale and size its own memory management (or even worse the application memory management) to reflect the availability of physical memory. For more information about memory limits, please read this post by Duncan: Memory limits. As the VMkernel is forced to provide alternative memory resources limits can lead to the increased use of VM swap files. This can lead to performance problems of the application but can also impact other virtual machines and subsystems used in the virtual infrastructure. The following article zooms into one of the many problems when relying on swap files: Impact of host local VM swap on HA and DRS. Color indicators As virtual machine level limits can impact the performance of the entire virtual infrastructure, the CloudPhysics engineers decided to add an additional indicator to help you easily detect limits. When a virtual machine is configured with a memory limit still greater than 50% of its configured size an Amber dot is displayed next to the configured limit size. If the limit is smaller or equal to 50% of its configured size than a red dot is displayed next to the limit size. Similar for CPU limits, an amber dot is displayed when the limit of a virtual machine is set but is more than 500MHz, a red dot indicates that the virtual machine is configured with a CPU limit of 500MHz or less. For example: Virtual Machine Load06 is configured with 16GB of memory. A limit is set to 8GB (8192MB), this limit is equal to 50% of the configured size. Therefore the VM reservation and Limits card displays the configured limit in red and presents an additional red dot. Flow of information The indicators are also a natural divider between the memory resource controls and the CPU controls. As memory resource control impacts the virtual infrastructure more than the CPU resource controls, the card displays the memory resource controls at the left side of the screen. We are very interested in hearing feedback about this card, please leave a comment. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: From the archives - An old Isometric diagram URL: https://frankdenneman.ai/2012-10-05-from-the-archives-an-old-isometric-diagram/ Date: 2012-10-05 While searching for a diagram I stumbled upon an old diagram I made in 2007. I think this diagram started my whole obsession with diagrams and to add “cleanness” to my diagrams. This diagram depicts a virtual infrastructure located in two datacenters with replication between them. This infrastructure is no longer in use, but to make absolutely sure, I changed the device names into generic text labels such as ESX host, array, SW switch, etc. Back then I really liked to draw Isometric style. Now I’m more focused onto block diagrams and trying to minimalize the number of components in a diagram. In essence I follow the words from Colin Chapman: Simplify, then add lightness. But then applied to diagrams :) The fact that this diagram is still stored on my system tells me that I’m still very proud of this diagram. So that made me wonder, which diagram did you design and are you proud of? Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Storage DRS automation level and initial placement behavior URL: https://frankdenneman.ai/2012-10-04-storage-drs-automation-level-and-initial-placement-behavior/ Date: 2012-10-04 Recently I was asked why Storage DRS was missing a “Partially Automated mode”. Storage DRS has two automation levels, no automation (Manual Mode) and Fully Automated mode. When comparing this with DRS, we notice that Storage DRS is missing a “Partially Automated mode”. But in reality the modes of Storage DRS cannot be compared to DRS at all. This article explains the difference in behavior. DRS automation modes: There are three cluster automation levels: Manual automation level: When a virtual machine is configured with the manual automation level, DRS generate both initial placement and load balancing migration recommendations, however the user needs to manual approve these recommendations. Partially automation level: DRS automatically places a virtual machine with a partially automation level, however it will generate a migration recommendation which requires manual approval. Fully automated level: DRS automatically places a virtual machine on a host and vCenter automatically applies migration recommendation generated by DRS Storage DRS automation modes: There are two datastore cluster automation levels: No Automation (Manual mode): Storage DRS will make migration recommendations for virtual machine storage, but will not perform automatic migrations. Fully Automated: Storage DRS will make migration recommendations for virtual machine storage, vCenter automatically confirms migration recommendations. No automatic Initial placement in Storage DRS Storage DRS does not provide placement recommendations for vCenter to automatically apply. (Remember that DRS and Storage DRS only generate recommendations, it is vCenter that actually approves these recommendations if set to Automatic). The automation level only applies to migration recommendation of exisiting virtual machines inside the datastore cluster. However, Storage DRS does analyze the current state of the datastore cluster and generates initial placement recommendations based on space utilization and I/O load of the datastore and disk footprint and affinity rule set of the virtual machine. When provisioning a virtual machine, the summary screen provided in the user interface displays a datastore recommendation. When clicking on the “more recommendations” less optimal recommendations are displayed. This screen provides information about the Space Utilization % before placement, the Space Utilization % after the virtual machine is placed and the measured I/O Latency before placement. Please note that even when I/O load balancing is disabled, Storage DRS uses overall vCenter I/O statistics to determine the best placement for the virtual machine. In this case the I/O Latency metric is a secondary metric, which means that Storage DRS applies weighing to the space utilization and overall I/O latency. It will satisfy space utilization first before selecting a datastore with an overall lower I/O latency. Adding new hard-disks to a existing VM in a datastore cluster As vCenter does not apply initial placement recommendations automatically, adding new disks to an existing virtual machine will also generate an initial placement recommendation. The placement of the disk is determined by the default affinity cluster rule. The datastore recommendation depicted below shows that the new hard disk is placed on datastore nfs-f-01, why? Because it needs to satisfy storage initial placement requests and in this case this means satisfying the datastore cluster default affinity rule. If the datastore cluster were configured with a VMDK anti-affinity rule, the datastore recommendation would show any other datastore except datastore nfs-f-01. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Storage DRS Device Modeling behavior URL: https://frankdenneman.ai/2012-10-03-storage-drs-device-modeling-behavior/ Date: 2012-10-03 During a recent meeting the behavior of Storage DRS device modeling was discussed. When I/O load balancing is enabled, Storage DRS leverages the SIOC injector to determine the device characteristics of the disks backing the datastore. Because the injector stops when there is activity detected on the datastore, the customer was afraid that Storage DRS wasn’t able to get a proper model of his array due to the high levels of activity seen on the array. Storage DRS was designed to cope with these environments, as the customer was reassured after explaining the behavior I thought it might be interesting enough for to share it with you too. The purpose of device modeling Device modeling is used by Storage DRS to characterize the performance levels of the datastore. This information is used when Storage DRS needs to predict the benefit of a possible migration of a virtual machine. The workload model provides information about the I/O behavior of the VM, Storage DRS uses that as input and mixes this with the device model of the datastore in order to predict the increase of latency after the move. The device modeling of the datastore is done with the SIOC injector The workload To get a proper model, the SIOC injector injects random read I/O to the disk. SIOC uses different amounts of outstanding IO to measure the latency. The duration of the complete cycle is 30 seconds and is trigger once a day per datastore. Although it’s a short-lived process, this workload does generate some overhead on the array and Storage DRS is designed to enable storage performance for your virtual machines, not to interfere with them. Therefor this workload will not run when activity is detected on the devices backing the datastore. Timer As mentioned, the device modeling process runs for 30 seconds in order to characterize the device. If the IO injector starts and the datastore is active or becomes active, the IO injector will wait for 1 minute to start again. If the datastore is still busy, it will try again in 2 minutes, after that it idles for 4 minutes, after that 8 minutes, 16 minutes, 32 minutes, 1 hour and finally 2 hours. When the datastore is still busy after two hours after the initial start it will try to start the device modeling with an interval of 2 hours until the end of the day. If SIOC is not able to characterize the disk during that day, it will use the average value of all the other datastores in other not to influence the load balancing operations with false information and provide information that would favor this disk over other datastores that did provide actual data. The next day SIOC injector will try model the device again, but uses a skew back and forth of 2 hours from the previous period, this way during the year, Storage DRS will retrieve info across every period of the day. Key takeway Overall we do not expect the array to be busy 24/7, there is always a window of 30 seconds where the datastore is idling. Having troubleshooting many storage related problems I know arrays are not stressed all day long, therefor I’m more than confident that Storage DRS will have accurate device models to use for its prediction models. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Apply User-defined Storage Capabilities to multiple datastore at once URL: https://frankdenneman.ai/2012-10-01-apply-user-defined-storage-capabilities-to-multiple-datastore-at-once/ Date: 2012-10-01 To get a datastore cluster to surface a (user-defined) storage capability, all datastores inside the datastore cluster must be configured with the same storage capability. When creating Storage Capabilities, the UI does not contain a view where to associate a storage capability with multiple datastores. However that does not mean the web client does not provide you with the ability to do so. Just use the multi-select function of the webclient. Go to Storage, select the datastore cluster, select Related Objects and go to Datastores view. To select all datastores, click the first datastore, hold shift and select the last datastore. Right click and select assign storage capabilities. Select the appropriate Storage capability and click on OK. The Datastore Cluster summary tab now shows the user-defined Storage Capability. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Avoiding VMDK level over-commitment while using Thin disks and Storage DRS URL: https://frankdenneman.ai/2012-10-01-avoiding-vmdk-level-over-commitment-while-using-thin-disks-and-storage-drs/ Date: 2012-10-01 The behavior of thin provisioned disk VMDKs in a datastore cluster is quite interesting. Storage DRS supports the use of thin provisioned disks and is aware of both the configured size and the actual data usage of the virtual disk. When determining placement of a virtual machine, Storage DRS verifies the disk usage of the files stored on the datastore. To avoid getting caught out by instant data growth of the existing thin disk VMDKs, Storage DRS adds a buffer space to each thin disk. This buffer zone is determined by the advanced setting “PercentIdleMBinSpaceDemand". This setting controls how conservative Storage DRS is with determining the available space on the datastore for load balancing and initial placement operations of virtual machines. IdleMB The main element of the advanced option “PercentIdleMBinSpaceDemand” is the amount of IdleMB a thin-provisioned VMDK disk file contains. When a thin disk is configured, the user determines the maximum size of the disk. This configured size is referred to as “Provisioned Space”. When a thin disk is in use, it contains an x amount of data. The size of the actual data inside the thin disk is referred to as “allocated space”. The space between the allocated space and the provisioned space is called identified as the IdleMB. Let’s use this in an example. VM1 has a single VMDK on Datastore1. The total configured size of the VMDK is 6GB. VM1 written 2GB to the VMDK, this means the amount of IdleMB is 4GB. PercentIdleMBinSpaceDemand The PercentIdleMBinSpaceDemand setting defines percentage of IdleMB that is added to the allocated space of a VMDK during free space calculation of the datastore. The default value is set to 25%. When using the previous example, the PercentIdleMBinSpaceDemand is applied to the 4GB unallocated space, 25% of 4GB = 1 GB. Entitled Space Use Storage DRS will add the result of the PercentIdleMBinSpaceDemand calculation to the consumed space to determine the “entitled space use”. In this example the entitled space use is: 2GB + 1GB = 3GB of entitled space use. Calculation during placement The size of Datastore1 is 10GB. VM1 entitled space use is 3GB, this means that Storage DRS determines that Datastore1 has 7GB of available free space. Changing the PercentIdleMBinSpaceDemand default setting Any value from 0% to 100% is valid. This setting is applied on datastore cluster level. There can be multiple reasons to change the default percentage. By using 0%, Storage DRS will only use the allocated space, allowing high consolidation. This is might be useful in environments with static or extremely slow data increase. There are multiple use cases for setting the percentage to 100%, effectively disabling over-commitment on VMDK level. Setting the value to 100% forces Storage DRS to use the full size of the VMDK in its space usage calculations. Many customers are comfortable managing over-commitment of capacity only at storage array layer. This change allows the customer to use thin disks on thin provisioned datastores. Use case 1: NFS datastores A use case is for example using NFS datastores. Default behavior of vSphere is to create thin disks when the virtual machine is placed on a NFS datastore. This forces the customer to accept a risk of over-commitment on VMDK level. By setting it to 100%, Storage DRS will use the provisioned space during free space calculations instead of the allocated space. Use case 2: Safeguard to protect against unintentional use of thin disks This setting can also be used as safeguard for unintentional use of thin disks. Many customers have multiple teams for managing the virtual infrastructure, one team for managing the architecture, while another team is responsible for provisioning the virtual machines. The architecture team does not want over-commitment on VMDK level, but is dependent on the provisioning team to follow guidelines and only use thick disks. By setting “PercentIdleMBinSpaceDemand” to 100%, the architecture team is ensured that Storage DRS calculates datastore free space based on provisioned space, simulating “only-thick disks” behavior. Use-case 3: Reducing Storage vMotion overhead while avoiding over-commitment By setting the percentage to 100%, no over-commitment will be allowed on the datastore, however the efficiency advantage of using thin disks remains. Storage DRS uses the allocated space to calculate the risk and the cost of a migration recommendation when a datastore avoids its I/O or space utilization threshold. This allows Storage DRS to select the VMDK that generates the lowest amount of overhead. vSphere only needs to move the used data blocks instead of all the zeroed out blocks, reducing CPU cycles. Overhead on the storage network is reduced, as only used blocks need to traverse the storage network. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Storage DRS demo available on VMware TV URL: https://frankdenneman.ai/2012-09-27-storage-drs-demo-available-on-vmware-tv/ Date: 2012-09-27 If you haven’t seen Storage DRS in action, check out the Storage DRS demo I’ve created for VMwareTV. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: How to create VM to Host affinity rules using the webclient URL: https://frankdenneman.ai/2012-09-21-how-to-create-vm-to-host-affinity-rules-using-the-webclient/ Date: 2012-09-21 This article shows you how to create a VM to Host affinity rule using the new webclient. 1. Select host and clusters in the home screen. 2. Select the appropriate cluster. 3. Select the tab Manage and click on Settings. 4. Click on the » to expand the Cluster setting menu. 5. Select DRS Groups. 6. Click on Add to create a DRS Group. The dropdown box provides the ability to create a VM DRS group and a Host DRS group. The behavior of this window is a little tricky. When you create a group, you need to click on OK to actually create the group. If you create a VM DRS group first and then select the Host DRS group in the dropdown box before you click OK, the VM DRS group configuration is discarded. 7. Create the VM DRS Group and provide the VM group a meaningful name. 8. Click on “Add” to select the virtual machines. 9. Click on OK to add the virtual machines to the group. 10. Review the configuration and click on OK to create the VM DRS Group. 11. Click on “Add” again to create the Host DRS Group. 12. Select Host DRS Group in the dropdown box and provide a name for the Host DRS Group. 13. Click on “Add” to select the hosts that participate in this group. 14. Click on OK to add the hosts to the group. 15. Review the configuration and click on OK to create the Host DRS Group. 16. The DRS Groups view displays the different DRS groups in a single view. The groups are created, now it’s time to create the rules. 17. Select DRS Rules in the Cluster settings menu. 18. Click on “Add” to create the rule. 19. Provide a name for this rule and check if the rule is enabled (default enabled) 20. Select the “Virtual Machines to Hosts” rule in the Type dropbox. 21. Select the appropriate VM Group and the corresponding Host Group. 22. Select the type affinity rule. For more information about the difference between should and must rule, read the article: “Should or Must VM-Host affinity rules?”. In this example I’m selecting the should rule. 23. Click on Ok to create the rule. 24. Review your configuration in DRS rules screen. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Technical paper: “VMware vCloud Director Resource Allocation Models” available for download URL: https://frankdenneman.ai/2012-09-20-technical-paper-vmware-vcloud-director-resource-allocation-models-available-for-download/ Date: 2012-09-20 Today the technical paper “VMware vCloud Director Resource Allocation Models” has been made available for download on VMware.com. This whitepaper covers the allocation models used by vCloud Director 1.5 and how they interact with the vSphere layer. This paper helps you correlate the vCloud allocation model settings with the vSphere resource allocation settings. For example what happens on the vSphere layer when I set a guarantee on an Org VDC configured with the Allocation Pool Model. It provides insight on the distribution of resources on both the vCloud layer and vSphere layer and illustrates the impact of various allocation model settings on vSphere admission control. The paper contains a full chapter about allocation model in practice and demonstrates the effect of using various combinations of allocation models within a single provider vDC. Please note that this paper is based on vCloud Director 1.5 http://www.vmware.com/resources/techresources/10325 ================================================================================ Title: VM Storage Profiles and Storage DRS - Part 2 - Distributed VMs URL: https://frankdenneman.ai/2012-09-19-storage-drs-and-storage-profiles-part-2-distributed-vms/ Date: 2012-09-19 Mentioned in part-1 of the Storage DRS and VM Storage Profiles series, Storage DRS expects “storage characteristics –alike” datastores inside a single datastore cluster. But what if you have multiple tiers of storage and you want to span the virtual machine across them? Storage profiles can assist in deploying the VMs across multiple datastore clusters safely and inline with your SLAs. Storage DRS Datastore architecture When you have multiple tiers of storage, its recommended to create multiple datastore clusters and each datastore cluster contains disks from a single tier. Let’s assume you have three different kinds of disks in your array: SSD, FC 15K and SATA. Datastores backed by disk out a single pool are aggregated into a single datastore, resulting in three datastore clusters. Having multiple datastore clusters can increase the complexity of the provisioning process, using VM storage profiles ensures you that virtual machines or disk files are placed in the correct datastore cluster. Assign storage capabilities to datastores All datastores within a single datastore cluster are associated with the same storage capability. Storage Capability Associated with datastores in Datastore cluster: SSD – Low latency disks (Tier 1 VMs and VMDKs) Datastore Cluster Tier 1 VMs and VMDKs FC 15K – Fast disks (Tier 2 VMs and VMDKs) Datastore Cluster Tier 2 VMs and VMDKs SATA – High Capacity Disks (Tier 3 VMs and VMDKs) Datastore Cluster Tier 3 VMs and VMDKs Please note that all datastores must be configured with the same storage capability. If one datastore is not associated with a storage capability or has a different storage capability than its sibling datastores, the datastore cluster will not surface a storage capability. One virtual machine – different levels of service required Generally faster disk have higher cost per gigabyte and have a lower maximum capacity per drive, this usually drives various design decisions and operational procedures. Typically Tier 1 applications and data caching mechanisms end up in on the fastest storage disk pools. Most virtual machines are configured with multiple hard disks. A system disk containing the Operating System and one or more disks containing log files and data. The footprint of the virtual machine is made up out of a working directory and its VMDK files. When reviewing the requirements of the virtual machine, it is common that only the VMDKs containing the log files and the databases require low latency disks while the system disk can be placed in a lower tier storage pool. And this is the reason why you can assign multiple different VM storage profiles to a virtual machine. Multiple VM storage Profiles Let’s use an example; in this scenario we are going to deploy the virtual machine vCenter02. The virtual machine is configured with three disks, Hard disk 1 contains the OS, Hard disk 2 contains the database and Hard disk 3 contains the log files. We associated the VM with two VM Storage Profiles. To avoid wasting precious low latency disk space in the Tier 1 datastore cluster, we are going to associate the VMs working directory (containing the VM swap file) and the 60GB system disk are to the Tier 2VM storage profile, which is connected to Tier 2 Storage capability. When selecting storage during the deployment process, click on the button advanced. To associate a VM storage profile to a Hard disk or the working directory, called Configuration file in this screen, double click the item in the storage column and select browse. The VM storage Profile screen appears and you can select the appropriate VM storage profile. The VM storage profile “Tier 2 VMs and VMDKs” is selected and will be associated with the Configuration file once we click ok. As Tier 2 storage profile is associated with Storage Capability “FC 15K – Fast disks (Tier 2 VMs and VMDKs)”, the UI list “Datastore Cluster – Tier 2 VMs and VMDKs” as the only compatible datastore cluster. These steps have to be repeated for every hard disk of the virtual machine. At this point the working directory (configuration file) and the System disk will be placed on Datastore Cluster Tier 2 and the Database disk and Log file disk will be placed on Datastore Cluster Tier 1 once the deployment process has completed. The ready to complete screen displays the associated VM storage profiles and the destinations of the working directory and Hard disks. Storage DRS generates placement recommendations and these can be changed if you want to select a different datastore. By selecting the option “more recommendations” a window is displayed and will show you alternative destination datastores. DRMDisks Storage DRS is able to generate these stand-alone recommendations due to the construct called DRMDisks. Storage DRS generates a construct called DRMDisk for each VM working directory and each VMDK. The DRMDisk is the smallest element Storage DRS can load balance (atomic level). Therefore Storage DRS can move a system disk VMDK to a different datastore in the datastore cluster without having to move the working directory or another disk. Depending on the default Affinity rule of the cluster, DRMdisks within the datastore cluster will be placed on the same datastore (affinity) or separated on different datastores (anti-affinity). For more information about load balancing based on DRMdisk instead of a complete VM, please read the article: Impact of Intra VM affinity rules on Storage DRS. Part 3 will cover applying Storage Profiles to virtual machine templates Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: VM Storage Profiles and Storage DRS - Part 1 URL: https://frankdenneman.ai/2012-09-18-vm-storage-profiles-and-storage-drs-part-1/ Date: 2012-09-18 In my previous article about how to configure storage profiles using the web client I stated that different storage profiles could be assigned to a single virtual machine. Storage profiles can be used together with Storage DRS. Let’s take a closer look on how to use storage profiles with Storage DRS. Architectural view VM storage profiles need to be connected to a storage capability to function. The storage capability itself needs to be associated to one or more datastores. A virtual machine in its whole can be associated with a storage profile, or you can use a more granular configuration and associate different storage profiles to the VM working directory and / or VMDK files. Datastore cluster storage capabilities You might have noticed that there isn’t a datastore cluster element depicted in the diagram. The storage capabilities of a datastore cluster are extracted from the associated storage capabilities of each datastore member. If all datastores are configured with the same storage capability, the datastore cluster surfaces this storage capability and becomes compliant with the connected VM storage profiles. For example, “Datastore cluster – Tier 1 VMs and VMDKs” contains 4 datastores. NFS-F-01, NFS-F-02, NFS-F-03 are associated with the storage capability “SSD low latency disk (Tier-1 VMs and VMDKs)” while datastore NFS-F-04 is associated with storage capability “FC 15K – High Speed disk (Tier 2 VMs and VMDKs)”. When reviewing the Storage Capabilities of the datastore cluster, no Storage Capability is displayed: The VM Storage Profile “Tier 1 VMs and VMDK” is connected to the Storage Capability “SSD low latency disk (Tier-1 VMs and VMDKs)”. When selecting storage during the deployment of a virtual machine, the datastore cluster is considered incompatible with the selected VM Storage Profile. Incompatible, but there are three datastores available with the correct Storage capabilities? Although this is true, Storage DRS does not incorporate storage profiles compliancy in its balancing algorithms. Storage DRS is designed with the assumption that all disks backing the datastores are “storage characteristics-alike”. Manually selecting a datastore in the datastore cluster is only possible if the option “Disable Storage DRS for this virtual machine” is selected. Placing the VM on the specific datastore and then enabling Storage DRS later on that VM is futile. Storage DRS will load balance the VM if necessary, but it doesn’t take the VM storage profile compatibility into account when load balancing. So if you have, please remove this “workaround” in your operation manuals :) After removing the datastore with the dissimilar storage capability (NFS-F-04), the Datastore cluster surfaces “SSD – Low Latency disk (Tier-1 VMs and VMDKs)” and becomes compatible with virtual machines associated with the Tier-1 VM storage Profile. Part 2 will cover distributing virtual machine across multiple datastores using Storage Profiles. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: VMworld Europe Sessions URL: https://frankdenneman.ai/2012-09-18-vmworld-europe-sessions/ Date: 2012-09-18 At VMworld Europe I will participate in Meet the Expert sessions, Group discussions and three breakout sessions. Here’s an overview of my public schedule. I hope to see you all in my sessions or participate in the group discussions. Or schedule a time slot in Meet the Expert if you have a question about Storage DRS or DRS that you want to ask me. Tuesday 9 October: 11:00 – 12:00: Group Discussion 28 Resource Management 12:30 - 13:30: INF-VSP1168 Architecting a Cloud Infrastructure 14:00 – 15:00: INF-VSP1683 Resource Pool Best Practices Wednesday 10 October: 11:00 – 12:00: Meet the Experts 05 12:30 – 13:30: Group Discussion 28 Resource Management 15:30 – 16:30: INF-STO1545 Architecting and designing (SDRS) Datastore Clusters Thursday 11 October: 10:30 – 11:30: Meet the Experts 11 ================================================================================ Title: How to attach VM storage profiles to a virtual machine using the web client URL: https://frankdenneman.ai/2012-09-14-how-to-attach-vm-storage-profiles-to-a-virtual-machine-using-the-web-client/ Date: 2012-09-14 Virtual Machine Storage Profiles are used to identify the storage capabilities necessary in order to properly run the application within the virtual machine. VM Storage Profiles need to enabled first on the hosts and/or Cluster before you are able to assign them to a virtual machine. 1. Select VM Storage Profiles in the Home screen 2. Select the Icon in the middle to enable VM Storage profiles 3. Select the cluster or host that you want to enable Check if the host has the appropriate license 4. Create Storage Profiles A VM storage profile is attached to a Storage capability. In turn a Storage Capability profile is attached to a datastore. For more information about storage capabilities please read the article: vSphere 5.0 Storage Features Part 11 – Profile Driven Storage by Cormac Hogan. 5. Go back to home 6. Select VM and Templates 7. Select Datacenter in the left pane 8. Select the menu option “Related Objects” in the right pane 9. Select the menu option “Virtual Machines” 10. Right click on a virtual machine 11. Select All vCenter Actions 12. Select Storage Profiles 13. Select Manage Storage Profiles 14. Apply the VM storage Profiles to the working directory and disk Please note that you can assign different storage profiles to the virtual machine working directory and each single VMDK. The working directory is where the .vmx, .nvram, .log resides. This is listed in the UI as the Home VM location. Each VMDK can be assigned with a different storage profile to align it with your SLA’s. ================================================================================ Title: How to create a datastore cluster using the new web client. URL: https://frankdenneman.ai/2012-09-12-how-to-create-a-datastore-cluster-using-the-new-web-client/ Date: 2012-09-12 vSphere 5.1 main user interface is provided by the web client, during beta testing I spend some time to get accustomed to the new user interface. In order to save you some time, I created this write-up on how to create a datastore cluster using the web client. I assume you already installed the new vCenter 5.1. If not, check out’s Duncan post on how to install the new vCenter Server Appliance. Before showing the eight easy steps that need to be taken when creating a datastore cluster, I want to list some constraints and the recommendations for creating datastore clusters. Constraints: • VMFS and NFS cannot be part of the same datastore cluster. • Similar disk types should be used inside a datastore cluster. • Maximum of 64 datastores per datastore cluster. • Maximum of 256 datastore clusters per vCenter Server. • Maximum of 9000 VMDKs per datastore cluster Recommendations: • Group disks with similar characteristics (RAID-1 with RAID-1, Replicated with Replicated, etc.) • Leverage information provided by vSphere Storage APIs - Storage Awareness The Steps 1. Go to the Home screen and select Storage 2. Select the Datastore Clusters icon in Related Objects view. 3. Name and Location The first steps are to enable Storage DRS, specify the datastore cluster name and check if the “Turn on Storage DRS” option is enabled. When “Turn on Storage DRS” is activated, the following functions are enabled: • Initial placement for virtual disks based on space and I/O workload • Space load balancing among datastores within a datastore cluster • IO load balancing among datastores within a datastore cluster The “Turn on Storage DRS” check box enables or disables all of these components at once. If necessary, I/O balancing functions can be disabled independently.If Storage DRS is not enabled, a datastore cluster will be created which lists the datastores underneath, but Storage DRS won’t recommend any placement action for provisioning or migration operations on the datastore cluster. When you want to disable Storage DRS on an active datastore cluster, please note that all the Storage DRS settings, e.g. automation level, aggressiveness controls, thresholds, rules and Storage DRS schedules are saved so they may be restored to the same state at the moment Storage DRS was disabled. 4. Storage DRS Automation Storage DRS offers two automation levels: No Automation (Manual Mode) Manual mode is the default mode of operation. When the datastore cluster is operating in manual mode, placement and migration recommendations are presented to the user, but are not executed until they are manually approved. Fully Automated Fully automated allows Storage DRS to apply space and I/O load-balance migration recommendations automatically. No user intervention is required. However, initial placement recommendations still require user approval. Storage DRS allows virtual machines to have individual automation level settings that override datastore cluster-level automation level settings. Similar to when DRS was introduced, I recommend to start using manual mode first and review the generated recommendations. If you are comfortable with the decision matrix of Storage DRS you can switch to fully automated. Please note that you can switch between modes on the fly and without incurring downtime. 5. Storage DRS Runtime Settings Keep the defaults for now. Future articles expand upon the Storage DRS thresholds and advanced options. 6. Select Clusters and Hosts The “Select Hosts and Clusters” view allows the user to select one or more (DRS) clusters to work with. Only clusters within the same vCenter datacenter can be selected, as the vCenter datacenter is the boundary for Storage DRS to operate in. 7. Select Datastores By default, only datastores connected to all hosts in the selected (DRS) cluster(s) are shown. The Show datastore dropdown menu provides the options to show partially connected datastores. The article partially connected datastore cluster gives you insight of the impact of this design decision. 8. Ready to Complete The “Ready to Complete” screen provides an overview of all the settings configured by the user. Review the configuration of your new datastore cluster and click on finish. ================================================================================ Title: Where is my new vMotion functionality? URL: https://frankdenneman.ai/2012-09-11-where-is-my-new-vmotion-functionality/ Date: 2012-09-11 Just a reminder as I received a lot of questions and comments about this: The new vMotion functionality - migrating virtual machines between host without shared storage - is only available via the web client. Please note that in the vSphere 5.1 release all new features are only visible via the web client and not in the old vSphere client. For more information about the vMotion functionality: vSphere 5.1 vMotion deepdive Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: vSphere 5.1 vMotion Deep Dive URL: https://frankdenneman.ai/2012-09-07-vsphere-5-1-vmotion-deepdive/ Date: 2012-09-07 vSphere 5.1 vMotion enables a virtual machine to change its datastore and host simultaneously, even if the two hosts don’t have any shared storage in common. For me this is by far the coolest feature in the vSphere 5.1 release, as this technology opens up new possibilities and lays the foundation of true portability of virtual machines. As long as two hosts have (L2) network connection we can live migrate virtual machines. Think about the possibilities we have with this feature as some of the current limitations will eventually be solved, think inter-cloud migration, think follow the moon computing, think big! The new vMotion provides a new level of ease and flexibility for virtual machine migrations and the beauty of this is that it spans the complete range of customers. It lowers the barrier for vMotion use for small SMB shops, allowing them to leverage local disk and simpler setups, while big datacenter customers can now migrate virtual machines between clusters that may not have a common set of datastores between them. Let’s have a look at what the feature actually does. In essence, this technology combines vMotion and Storage vMotion. But instead of either copying the compute state to another host or the disks to another datastore, it is a unified migration where both the compute state and the disk are transferred to different host and datastore. All is done via the vMotion network (usually). The moment the new vMotion was announced at VMworld, I started to receive questions. Here are the most interesting ones that allows me to give you a little more insight of this new enhancement. Migration type One of the questions I have received is, will the new vMotion always move the disk over the network? This depends on the vMotion type you have selected. When selecting the migration type; three options are available: This may be obvious to most, but I just want to highlight it again. A Storage vMotion will never move the compute state of a VM to another host while migrating the data to another datastore. Therefore when you just want to move a VM to another host, select vMotion, when you only want to change datastores, select Storage vMotion. Which network will it use? vMotion will use the designated vMotion network to copy the compute state and the disks to the destination host when copying disk data between non-shared disks. This means that you need to the extra load into account when the disk data is being transferred. Luckily the vMotion team improved the vMotion stack to reduce the overhead as much as possible. Does the new vMotion support multi-NIC for disk migration? The disk data is picked up by the vMotion code, this means vMotion transparently load balances the disk data traffic over all available vMotion vmknics. vSphere 5.1 vMotion leverages all the enhancements introduced in the vSphere 5.0 such as Multi-NIC support and SDPS. Duncan wrote a nice article on these two features. Is there any limitation to the new vMotion when the virtual machine is using shared vs. unshared swap ? No, either will work, just as with the traditional vMotion. Will the new vMotion features be leveraged by DRS/DPM/Storage DRS ? In vSphere 5.1 DRS, DPM and Storage DRS will not issue a vMotion that copies data between datastores. DRS and DPM remains to leverage traditional vMotion, while Storage DRS issues storage vMotions to move data between datastores in the datastore cluster. Maintenance mode, a part of the DRS stack, will not issue a data moving vMotion operation. Data moving vMotion operations are more expensive than traditional vMotion and the cost/risk benefit must be taken into account when making migration decisions. A major overhaul of the DRS algorithm code is necessary to include this into the framework, and this was not feasible during this release. How many concurrent vMotion operations that copies data between datastores can I run simultaneously? A vMotion that copies data between datastores will count against the limitations of concurrent vMotion and Storage vMotion of a host. In vSphere 5.1 one cannot perform more than 2 concurrent Storage vMotions per host. As a result no more than 2 concurrent vMotions that copy data will be allowed. For more information about the costs of the vMotion process, I recommend to read the article: “Limiting the number of Storage vMotions” How is disk data migration via vMotion different from a Storage vMotion? The main difference between vMotion and Storage vMotion is that vMotion does not “touch” the storage subsystem for copy operations of non-shared datastores, but transfers the disk data via an Ethernet network. Due to the possibilities of longer distances and higher latency, disk data is transferred asynchronously. To cope with higher latencies, a lot of changes were made to the buffer structure of the vMotion process. However if vMotion detects that the Guest OS issues I/O faster than the network transfer rate, or that the destination datastore is not keeping up with the incoming changes, vMotion can switch to synchronous mirror mode to ensure the correctness of data. I understand that the vMotion module transmits the disk data to the destination, but how are changed blocks during migration time handled? For disk data migration vMotion uses the same architecture as Storage vMotion to handle disk content. There are two major components in play – bulk copy and the mirror mode driver. vMotion kicks off a bulk copy and copies as much as content possible to the destination datastore via the vMotion network. During this bulk copy, blocks can be changed, some blocks are not yet copied, but some of them can already reside on the destination datastore. If the Guest OS changes blocks that are already copied by the bulk copy process, the mirror mode drive will write them to the source and destination datastore, keeping them both in lock-step. The mirror mode driver ignores all the blocks that are changed but not yet copied, as the ongoing bulk copy will pick them up. To keep the IO performance as high as possible, a buffer is available for the mirror mode driver. If high latencies are detected on the vMotion network, the mirror mode driver can write the changes to the buffer instead of delaying the I/O writes to both source and destination disk. If you want to know more about the mirror mode driver, Yellow bricks contains a out-take of our book about the mirror mode driver. What is copied first, disk data or the memory state? If data is copied from non-shared datastores, vMotion must migrate the disk data and the memory across the vMotion network. It must also process additional changes that occur during the copy process. The challenge is to get to a point where the number of changed blocks and memory are so small that they can be copied over and switch over the virtual machine between the hosts before any new changes are made to either disk or memory. Usually the change rate of memory is much higher than the change rate of disk and therefore the vMotion process start of with the bulk copy of the disk data. After the bulk data process is completed and the mirror mode driver processes all ongoing changes, vMotion starts copying the memory state of the virtual machine. But what if I share datastores between hosts; can I still use this feature and leverage the storage network? Yes and this is very cool piece of code, to avoid overhead as much as possible, the storage network will be leveraged if both the source and destination host have access to the destination datastore. For instance, if a virtual machine resides on a local datastore and needs to be copied to a datastore located on a SAN, vMotion will use the storage network to which the source host is connected. In essence a Storage vMotion is used to avoid vMotion network utilization and additional host CPU cycles. Because you use Storage vMotion, will vMotion leverage VAAI hardware offloading? If both the source and destination host are connected to the destination datastore and the datastore is located on an array that has VAAI enabled, Storage vMotion will offload the copy process to the array. Hold on, you are mentioning Storage vMotion, but I have Essential Plus license, do I need to upgrade to Standard? To be honest I try to keep away from the licensing debate as far as I can, but this seems to be the most popular question. If you have an Essential Plus license you can leverage all these enhancements of vMotion in vSphere 5.1. You are not required to use a standard license if you are going to migrate to a shared storage destination datastore. For any other licensing question or remark, please contact your local VMware SE / account manager. Update: Essential plus customers, please update to vCenter 5.1.0A. For more details read the follow article: “vMotion bug fixed in vCenter Server 5.1.0a”. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Storage DRS datastore cluster default affinity rule URL: https://frankdenneman.ai/2012-09-05-storage-drs-datastore-cluster-default-affinity-rule/ Date: 2012-09-05 In vSphere 5.1 you can configure the default (anti) affinity rule of the datastore cluster via the user interface. Please note that this feature is only available via the web client. The vSphere client does not contain this option. By default the Storage DRS applies an intra-VM vmdk affinity rule, forcing Storage DRS to place all the files and vmdk files of a virtual machine on a single datastore. By deselecting the option “Keep VMDKs together by default” the opposite becomes true and an Intra-VM anti-affinity rule is applied. This forces Storage DRS to place the VM files and each VDMK file on a separate datastore. Please read the article: “Impact of intra-vm affinity rules on storage DRS” to understand the impact of both types of rules on load balancing. ================================================================================ Title: vSphere 5.1 storage vMotion parallel disk migrations URL: https://frankdenneman.ai/2012-09-04-vsphere-5-1-storage-vmotion-parallel-disk-migrations/ Date: 2012-09-04 Where previous versions of vSphere copied disks serially, vSphere 5.1 allows up to 4 parallel disk copies per Storage vMotion operation When you migrate a virtual machine with five VMDK files, Storage vMotion copies of the first four disks in parallel, then starts the next disk copy as soon as one of the first four finishes. To reduce performance impact on other virtual machines sharing the datastores, parallel disk copies only apply to disk copies between distinct datastores. This means that if a virtual machine has multiple VMDK files on Datastore1 and Datastore2, parallel disk copies will only happen if destination datastores are Datastore3 and Datastore4. Let’s use an example to clarify the process. Virtual machine VM1 has four vmdk files. VMDK1 and VMDK2 are on Datastore1, VMDK3 and VMDK4 are on Datastore2. The VMDK files are moved from Datastore1 to Datastore4 and from Datastore2 to Datastore3. VMDK1 and VMDK3 are migrated in parallel, while VMDK2 and VMDK4 are queued. The migration process of VMDK2 is started the moment the migration of VMDK1 is complete, similar for VMDK4 as it will be started when the migration of VMDK3 is complete. A fan out disk copy, in other words copying two VMDK files on datastore A to datastores B and C, will not have parallel disk copies. The common use case of parallel disk copies is the migration of a virtual machine configured with an anti-affinity rule inside a datastore cluster. ================================================================================ Title: Storage DRS datastore correlation detector URL: https://frankdenneman.ai/2012-09-03-storage-drs-datastore-correlation-detector/ Date: 2012-09-03 One of the cool new features of Storage DRS in vSphere 5.1 is the datastore correlation detector used by the SIOC injector. Storage arrays have many ways to configure datastores from among the available physical disk and controller resources in the array. Some arrays allow sharing of back-end disks and RAID groups across multiple datastores. When two datastores share backend resources, their performance characteristics are tied together: when one datastore experiences high latency, the other datastore will also experience similar high latency since IOs from both datastore are being serviced by the same disks. These datastores are considered “performance-related”. I/O load balancing operations in vSphere 5.1 avoid recommending migration of virtual machines between two performance-correlated datastores. I/O load balancing algorithm Storage DRS collects several virtual machine metrics to analyze the workload generated by the virtual machines within the datastore cluster. These metrics are aggregated in a workload model. To effectively distribute the different load of the virtual machines across the datastores, Storage DRS needs to understand the performance (latency) of each datastore. When a datastore violates its I/O load threshold, Storage DRS moves virtual machines out of the datastore. By linking workload models to device models, Storage DRS is able to select a datastore with a low I/O load when placing a virtual machine with a high I/O load during load balance operations. Performance related datastores However if data is moved between datastores that are backed by the same disks, the move may not decrease the latency experienced on the source datastore as the same set of disks, spindles or RAID-groups service the destination datastore as well. I/O load balancing recommendations should avoid using two performance-correlated datastores, since moving a virtual machine from the source datastore to the destination datastore has no effect on the datastore latency. How does Storage DRS discover performance related datastores? How does it work? The datastore correlation detector measures performance during isolation and when concurrent IOs are pushed to multiple datastores. The basic mechanism of correlation detector is rather straightforward: compare the overall latency when two datastores are being used alone in isolation and when there are concurrent IO streams on both of the datastores. If there is no performance correlation, the concurrent IO to the other datastore should have no effect. Contrariwise, if two datastores are performance correlated, then concurrent IO stream should amplify the average IO latency on both datastores. Please note that datastores will be checked for correlation on a regular basis. This allows Storage DRS to detect changes to the underlying storage configuration. Example scenario In this scenario Datastore1 and Datastore2 are backed by disk devices grouped in Diskgroup1, while Datastore3 and Datastore4 are backed by disk devices grouped in Diskgroup2. All four datastores belong to a single datastore cluster. After SIOC has run the workload and device models on a datastore, SIOC picks a random datastore in the datastore cluster to check for correlations. If both datastores are idle, the datastore correlation detector uses the same workload to measure the average I/O latency in isolation and concurrent I/O mode. Isolation The SIOC injector measures the average IO latency of Datastore1 in isolation. This means it measures the latency of the outstanding I/O of Datastore1 alone. Next, it measures the average IO latency of Datastore2 in isolation. Concurrent I/Os The first two steps are used to establish the baseline for each datastore. In the third step the SIOC injector sends concurrent I/O to both datastores simultaneously. This results in the behavior that Storage DRS does not recommend any I/O load balancing operations between Datastore1 and 2 and Datastore3 and 4, but it can recommend for example to move virtual machines from Datastore1 to Datastore2 or from Datastore2 to Datastore3, etc. All moves are possible as long as the datastores are not correlated. Enable Storage DRS on performance-correlated datastores? When two datastores are marked as performance-correlated, Storage DRS does not generate IO load balancing recommendations between those two datastores. However Storage DRS can be used for initial placement and still generate recommendations to move virtual machines between two correlated datastores to address out of space situations or to correct rule violations. Please keep in mind that some arrays use a subset of disk out of a larger diskpool to back a single datastore. With these configurations, it appears that all disks in a diskpool back all the datastores but in reality they don’t. Therefor I recommend to set Storage DRS automation mode to manual and review the migration recommendations to understand if all datastores within the diskpool are performance-correlated. ================================================================================ Title: vSphere 5.1 Clustering Deepdive available URL: https://frankdenneman.ai/2012-08-28-vsphere-5-1-clustering-deepdive-available/ Date: 2012-08-28 Duncan and I released the vSphere 5.1 Clustering deepdive book this week. The book contains the new features of the vSphere 5.1 suite. We rewrote the Storage DRS chapter and have added a complete new chapter focusing on Stretched Clusters. Font changes The challenge for us was to include all the new content in the book without allowing the book to grow beyond its trademark dimensions. To achieve this we used a different font and decreased the font size, this resulted in a growth of 80 pages, making it 415 pages instead of the 505 pages if we used the previous font. Please note that although we decreased the font size, this did not decrease the legibility of the book. Special cover The cover is designed in such a way that you can actually have multiple copies with all different shades of orange, dare I say 50 shades of Orange. ;) We hope you enjoy the new version of the vSphere clustering deepdive series. It’s available in Paperback and Kindle format. Paper copy – $ 24.95 Kindle version – $ 7.49 ================================================================================ Title: CloudPhysics in a nutshell URL: https://frankdenneman.ai/2012-08-22-cloudphysics-in-a-nutshell/ Date: 2012-08-22 Disclaimer: I’m a technical advisor for CloudPhysics. I’m very happy to see CloudPhysics coming out of stealth mode this week and making their beta product available to the public. In a nutshell CloudPhysics is bringing Big Data analytics to the IT environment and it will provide you with tools to analyze your datacenter. How does it acquire this dataset and what benefit do you get from it? The Observer Appliance To gather all that data, an Observer Appliance needs to run in the virtual infrastructure. And in order to get a valuable dataset that is used for analytics and simulations the Observer needs to be active in as many as virtual infrastructures as possible. Running an appliance that sends operational data to a third party like CloudPhysics can be a security concern. Going into detail about how CloudPhysics designed the system to handle privacy, security and data sharing issues is outside the scope of this article. In short, data extracted from the virtual infrastructure are performance statistics and inventory and configuration settings. All environmental details are scrubbed and no log files or content of disk and memory is gathered. The User Interface The data acquired by the Observer Appliance is accessible at https://app.cloudphysics.com. Logging in will give you access to your own data. The beta product provides a user interface that allows you to dive into specific focus areas. The UI provides so-called cards that displays key data points and is a launch point to a more detailed view. This view can contain information about the relationship with other features of the vSphere stack. An example of such a card would be Virtual Machine level reservations. Not only does this card provide you information about the present virtual machine level reservations in your environment in a clear and concise manner, it also displays the impact the reservation has on the High Availability slot size and therefor the consolidation ratio of your cluster. All this information combined in a single screen, no need to navigate through multiple screens and correlate particular metrics. Correlation of metrics Correlation of particular settings and understanding the impact each setting has on a complex environment, such as a virtual infrastructure, is time consuming and above all very difficult. This correlation of metrics allows you to save time, but it also helps you understand behavior of your environment. Now you might ask how do you know you can trust if these correlations are correct and this is one of the most interesting things about this product. It’s a combination of product expertise and community driven input. The two pillars of knowledge The CloudPhysics team comprises of industry heavy hitters. Some of these persons invented core features of the vSphere stack while working for VMware, while others made their mark at other industry leading companies. The second pillar is the community involvement. In this beta program, registered users can suggest ideas for utility cards. Domain experts will verify the community provided cards on technical accuracy. Near-future developments One thing I’m very exited about is the upcoming High Availability and DRS simulation tools. Both HA and DRS can be a challenge to configure as some settings impact the virtual infrastructure on multiple levels. The HA and DRS simulation analyzes current settings and provides you a platform where you can predict the effects of a change on your environment. VMworld Challenge 2012 Now back to the current status. CloudPhysics is running a VMworld Challenge 2012. The contest allows you to describe the problems you are facing, such as “I’m applying different disk shares in my environment but I cannot see the worst case scenario allocation”. The more card you produce, the more points you score. To increase your score, download the Observer Appliance and take your environment for a test drive. The more activity you generate, the more points you accumulate. How will you benefit from this contest, first of all, if you are located in the U.S. you can win some great prices. (Due to U.S legislation, non-U.S. residents are excluded from winning prizes), but by submitting cards you improve the system and the quality of the reporting tool and simulation tool. Resource Management as a Service I started of with a disclaimer, I am a technical advisor to CloudPhysics and you can expect to see more articles about the development of CloudPhysics. As I’m able to work with the inventors of DRS and Storage DRS, a lot of my focus is on resource management. Together with the input of the community, the continuous analysis by domain experts you can expect that this might well turn out as Resource Management as a Service. ================================================================================ Title: VM templates and Storage DRS URL: https://frankdenneman.ai/2012-08-22-vm-templates-and-storage-drs/ Date: 2012-08-22 Please note that Storage DRS cannot move VM templates via storage vMotion. This can impact load balancing operations or datastore maintenance mode operations. When initiating Datastore Maintenance mode, the following message is displayed: As maintenance mode is commonly used for array migrations of datastore upgrade operations (VMFS-3 to VMFS-5), remember to convert the VM template to a virtual machine first before initiating maintenance mode. ================================================================================ Title: My public VMworld schedule URL: https://frankdenneman.ai/2012-08-20-my-public-vmworld-schedule/ Date: 2012-08-20 This year will be an action-packed VMWorld for me, presenting sessions, participating in two panel sessions, hosting a group discussion and available in two “Meet the expert” sessions. Presenting the following sessions: INF-STO1545 - Architecting Storage DRS Datastore Clusters INF-VSP1683 - VMware vSphere Cluster Resource Pools Best Practices Panel sessions: (TAM Day) - ASK THE EXPERTS INF-VSP1504 - Ask the Expert vBloggers Hosting the GD22 - Resource management (DRS/SDRS) group discussion. I invited Anne Holler (Lead engineer DRS) to host this session together with me. During Meet the Experts session 13 and session 17 I’m available for short meetings to answer your resource management (DRS\SDRS) questions. Here is the week schedule of the sessions/events/activities that I will be taking part of, be sure to sign up if you have not already: Sunday (TAM Day): 14:35 – 15:35 : ASK THE EXPERTS Monday: 14:30 – 15:30 : INF-VSP1504 - Ask the Expert vBloggers 16:00 – 17:00 : GD22 – Resource Management Tuesday: 12:30 – 13:30 : INF-STO1545 - Architecting Storage DRS Datastore Clusters (Repeat session) 15:00 – 16:00 : INF-VSP1683 - vSphere Cluster Resource Pools Best Practices Wednesday: 08:30 – 09:30 : INF-STO1545 - Architecting Storage DRS Datastore Clusters 12:30 – 13:30 : Expert 13 Thursday: 12:00 – 12:00 : Expert 17 ================================================================================ Title: DRS and memory balancing in non-overcomitted clusters URL: https://frankdenneman.ai/2012-08-10-drs-and-memory-balancing-in-non-overcomitted-clusters/ Date: 2012-08-10 First things first, I normally do not recommend changing advanced settings. Always try to tune system behavior by changing the settings provided by the user interface or try to understand system behavior and how it aligns with your design. The “problem” DRS load balancing recommendations could be sub-optimal when no memory overcommitment is preferred. Some customers prefer not to use memory overcommitment. The clusters contain (just) enough memory capacity to ensure all running virtual machines have their memory backed by physical memory. Nowadays it is not uncommon seeing virtual machines with fairly highly allocated (consumed) memory and due to the use of large pages on hosts with recent CPU architectures, little to no memory is shared. Common scenario with this design is a usual host memory load of 80-85% consumed. In this situation, DRS recommendations may have a detrimental effect on performance as DRS does not consider consumed memory but active memory. DRS behavior When analyzing the requirements of a virtual machine during load balancing operations, DRS calculates the memory demand of the virtual machine. The main memory metric used by DRS to determine the memory demand is memory active. The active memory represents the working set of the virtual machine, which signifies the number of active pages in RAM. By using the working-set estimation, the memory scheduler determines which of the allocated memory pages are actively used by the virtual machine and which allocated pages are idle. To accommodate a sudden rapid increase of the working set, 25% of idle consumed memory is allowed. Memory demand also includes the virtual machine’s memory overhead. Let’s use an 8 GB virtual machine as example on how DRS calculates the memory demand. The guest OS running in this virtual machine has touched 50% of its memory size since it was booted but only 20% of its memory size is active. This means that the virtual machine has consumed 4096 MB and 1639.2 MB is active. As mentioned, DRS accommodate a percentage of the idle consumed memory to accommodate a sudden increase of memory use. To calculate the idle consumed memory, the active memory 1639.2 MB is subtracted from the consumed memory, 4096 MB, resulting in a total 2456.8 MB. By default DRS includes 25% of the idle consumed memory, i.e. 614.2 MB. The virtual machine has a memory overhead of 90 MB. The memory demand DRS uses in it’s load balancing calculation is as follows: 1639.2 MB + 614.2 MB + 90 MB = 2343.4 MB. This means that DRS will select a host that has 2343.4 MB available for this machine and the move to this host improves the load balance of the cluster. DRS and corner stone of virtualization resource overcommitment Resource sharing and overcommitment of resources are primary elements of the virtualization. When designing virtual infrastructure it is a challenge to build the environment in such a way that it can handle virtual machine workloads while improving server utilization. Because every workload is not equal, applying resource allocation settings such as shares, reservations and limits can make distinction in priority. DRS is designed with this corner stone in mind. And that’s makes DRS sometimes a hard act to follow. DRS is all about solving imbalance and providing enough resources to the virtual machines aligned to their demand. This means that DRS balances workload on demand and trust in its core value that overcommitment is allowed. It then relies on the host local scheduler to figure out the priority of the virtual machines. And this behavior is sometimes not in line with the perception of DRS. A common perception is that DRS is about optimizing performance. This is partially true. As mentioned before DRS looks at the demand of the VM, and will try to mix and match activity of the virtual machines with the available resources in the cluster. As it relies on resource allocation settings, it assumes that priority is defined for each virtual machine and that the host local schedulers can reclaim memory safely. For this reason the DRS memory imbalance metric is tuned to focus on VM active memory to allow efficient sharing of host memory resources. Allowing to run with less cluster memory than the sum of all running virtual machine memory sizes and reclaiming idle consumed memory from lower priority virtual machines for other virtual machines’ active workloads. Unfortunately DRS does not know when the environment is designed in such a way to avoid overcommitment. Based on the input it can place a virtual machine on a host with virtual machine that have lots of idle consumed memory laying around. Instigating memory reclamation. In most cases this reclamation is hardly noticeable due to the use of the balloon driver. However in the case where all hosts are highly utilized, ballooning might not be as responsive as required, forcing the kernel to compress memory and swap. This means that migrations for the sole purpose of balancing active memory are not useful in environments like these and, if the target host memory is highly consumed, can cause a performance impact on the migrating virtual machine as it waits to obtain memory and on the other virtual machines on the target host as they do processing to allow reclamation of their idle memory. The solution? You might want to change the 25% idle consumed memory setting The solution I recommend to start with is to lower the migration threshold by moving the slider to the left. This allows the DRS cluster to have an higher imbalance and allows DRS to be more conservative when recommending migrations. If this is not satisfactory, then I would suggest changing the DRS advanced option called IdleTax. Please note that this DRS advanced option is not the same setting as the memory kernel setting. Mem.IdleTax. The DRS IdleTax advanced option (default 75) controls how much consumed idle memory should be added to active memory in estimating memory demand. The calculation is as follows: 100-IdleTax. Default caluculation = 100-75=25 This means that the smaller the value of IdleTax, more consumed idle memory is added to the active memory by DRS for load balancing. Be aware that the value of IdleTax is a heuristic, tuned to facilitate memory overcommitment; tuning it to a lower value is appropriate for environments not using overcommitment. Note that the option is set per cluster, and would need to be changed for all DRS clusters as appropriate. Again, try to use a lower migration threshold setting and monitor if this setting provides satisfying results before setting this advanced feature. ================================================================================ Title: Storage DRS enables SIOC on datastores only if I/O load balancing is enabled URL: https://frankdenneman.ai/2012-08-01-storage-drs-enables-sioc-on-datastores-only-if-io-load-balancing-is-enabled/ Date: 2012-08-01 Lately, I’ve received some comments why I don’t include SIOC in my articles when talking about space load balancing. Well, Storage DRS only enables SIOC on each datastore inside the datastore cluster if I/O load balancing is enabled. When you don’t enable I/O load balancing during the initial setup of the datastore cluster, SIOC is left disabled. Keep in mind when I/O load balancing is enabled on the datastore cluster and you disable the I/O load balancing feature, SIOC remains enabled on all datastores within the cluster. ================================================================================ Title: Considerations when modifying the individual VM automation level URL: https://frankdenneman.ai/2012-07-27-considerations-when-modifying-the-individual-vm-automation-level/ Date: 2012-07-27 Recently I received some questions about the behavior of DRS when the automation level of an individual virtual machine is modified. DRS allows customization of the automation levels for individual virtual machines to override the DRS cluster automation level. The most common reason for modifying the automation level is to prevent DRS move a virtual machine automatically. Selecting an automation level mode other than the default cluster automation level or fully automated impacts (daily) operational procedures. It might impact cluster balance and/or resource availability if the operational procedures are not adjusted to align with the “new” behavior of DRS when dealing with non-default automation levels. Before continuing with the impact and caveats of a non-default automation level, let’s zoom into their behavior. Level of automation There are five automation level modes: • Fully Automated • Partially Automated • Manual • Default • Disabled Each automation level behaves differently: Automation level Initial placement Load Balancing Fully Automated Automatic Placement Automatic execution of migration recommendation Partially Automated Automatic Placement Migration recommendation is displayed Manual Recommended host is displayed Migration recommendation is displayed Disabled VM powered-on on registered host No migration recommendation generated The default automation level is not listed in the table above as it aligns with the cluster automation level. When the automation level of the cluster is modified, the individual automation level is modified as well. Disabled automation level If the automation level of a virtual machine is set to disabled, then DRS is disabled entirely for the virtual machine. DRS will not generate a migration recommendation or generate an initial placement recommendation. The virtual machine will be powered-on on its registered host. A powered-on virtual machine with its automation level set to disabled will still impact the DRS load balancing calculation as its consumes cluster resources. During the recommendation calculation, DRS ignores the virtual machines set to disabled automation level and selects other virtual machines on that host. If DRS must choose between virtual machines set to the automatic automation levels and the manual automation level, DRS chooses the virtual machines set to automatic as it prefers them over virtual machines set to manual. Manual automation level When a virtual machine is configured with the manual automation level, DRS generate both initial placement and load balancing migration recommendations, however the user needs to manual approve these recommendations. Partially automation level DRS automatically places a virtual machine with a partially automation level, however it will generate a migration recommendation which requires manual approval. The impact of manual and partially automation level on cluster load balance When selecting any other automation level than disabled, DRS assumes that the user will manual apply the migration recommendation it recommends. This means that DRS will continue to include the virtual machines in the analysis of cluster balance and resource utilization. During the analysis DRS simulates virtual machine moves inside the cluster, every virtual machine that is not disabled will be included in the selection process of migration recommendations. If a particular move of a virtual machine offers the highest benefit and the least amount of cost and lowest risk, DRS generates a migration recommendation for this move. Because DRS is limited to a specific number of migrations, it might drop a recommendation of a virtual machine that provide almost similar goodness. Now the problem with this scenario is, that the recommended migration might be a virtual machine configured with a manual automation level, while the virtual machine with near-level goodness is configured with the default automation level. This should not matter if the user monitors each and every DRS invocation and reviews the migration recommendations when issued. This is unrealistic to expect as DRS runs each 5 minutes. I’ve seen a scenario where a group of the virtual machines where configured with manual mode. It resulted in a host becoming a “trap” for the virtual machines during an overcommitted state. The user did not monitor the DRS tab in vCenter and was missing the migration recommendations. This resulted in resource starvation for the virtual machines itself but even worse, it impacted multiple virtual machines inside the cluster. Because DRS generated migration recommendations, it dropped other suitable moves and could not achieve an optimal balance. For more information about the maximum number of moves, please read this article. Interested in more information about goodness values, please read this article. Disabled versus partially and manual automatic levels Disabling DRS on a virtual machines have some negative impact on other operation processes or resource availability, such as placing a host into maintenance mode or powering up a virtual machine after maintenance itself. As it selects the registered host, it might be possible that the virtual machine is powered on a host with ample available resources while more suitable hosts are available. However disabled automation level avoids the scenario described in the previous paragraph. Partially automatic level automatically places the virtual machine on the most suitable host, while manual mode recommends placing the host on the most suitable host available. Partially automated offers the least operational overhead during placement, but can together with manual automation level introduce lots of overhead during normal operations. Risk versus reward Selecting an automation level is almost a risk versus reward game. Setting the automation level to disabled might impact some operation procedures, but allows DRS to neglect the virtual machines when generating migration recommendations and come up with alternative solutions that provide cluster balance as well. Setting the automation level to partially or manual will offer you better initial placement recommendations and a more simplified maintenance mode process, but will create the risk of unbalance or resource starvation when the DRS tab in vCenter is left unmonitored. ================================================================================ Title: To which host-level latency statistic is the SIOC congestion threshold related? URL: https://frankdenneman.ai/2012-07-23-to-which-host-level-latency-statistic-is-the-sioc-congestion-threshold-related/ Date: 2012-07-23 Today someone asked if the congestion threshold of SIOC is related to which host latency threshold? Is it the Device average (DAVG), Kernel Average (KAVG) or Guest Average (GAVG)? Well actually it’s none of the above. DAVG, KAVG and GAVG are metrics in a host-local centralized scheduler that has complete control over all the requests to the storage system. SIOC main purpose is to manage shared storage resources across ESXi hosts, providing allocation of I/O resources independent of the placement of virtual machines accessing the shared datastore. And because it needs to regulate and prioritize access to shared storage that spans multiple ESXi hosts, the congestion threshold is not measured against a host-side latency metric. But to which metric is it compared? In essence the congestion threshold is compared with the weighted average of D/AVG per host, the weight is the number of IOPS on that host. Let’s expand on this a bit further. Average I/O latency To have an indication of the load of the datastore on the array, SIOC uses the average I/O latency detected by each host connected to that datastore. Average latency across hosts is used to cope with the variety of workloads, the characteristic of the active workloads, such as read versus writes, I/O size and degree of sequential I/Os in addition to array behavior such as block location, caching policies and I/O scheduling. To calculate and normalize the average latency across hosts, each host writes its average device latency and number of I/Os for that datastore in a file called IORMSTATS.SF stored on the same datastore. A common misconception about SIOC is that it’s compute cluster based. The process of determining the datastore-wide average latency really reveals the key denominator – hosts connected to the datastore - . All hosts connected to the datastore write to the IORMSTATS.SF file, regardless of cluster membership. Other than enabling SIOC, vCenter is not necessary for normal operations. Each connected host reads the IORMSTATS.SF file each 4 seconds and locally computes the datastore-wide average to use for managing the I/O stream. Therefor cluster membership is irrelevant. Datastore wide normalized I/O latency Back to the process of computing the datastore wide normalized I/O latency. The average device latencies of each host are normalized by SIOC based on the I/O request size. As mentioned before, not all storage related workloads are the same. Workloads issuing I/Os with a large request size result in longer device latencies due to way storage arrays process these workloads. For example, when using a larger I/O request size such as 256KB, the transfer might be broken up by the storage subsystem into multiple 64KB blocks. This operation can lead to a decline of transfer rate and throughput levels, increasing latency. This allows SIOC to differentiate high device latency from actual I/O congestion at the device itself. Number of I/O requests complete per second At this point SIOC has normalized the average latency across hosts based on I/O size, next step is to determine the aggregate number of IOPS accessing the datastore. As each host reports the number of I/O requests complete per second, this metric is used to compare and prioritize the workloads. I hope this mini-deepdive into the congestion thresholds explains why the congestion threshold could never be solely related to a single host-side metric . Because the datastore-wide average latency is a normalized value, the latency observed of the datastore per individual host may be different than the latency SIOC reports per datastore. . ================================================================================ Title: Removing the horizontal bar in the footer of a word doc URL: https://frankdenneman.ai/2012-07-20-removing-the-horizontal-bar-in-the-footer-of-a-word-doc/ Date: 2012-07-20 Now for something completely different, a tip how to extend your life with about 5 years - or how to remove the horizontal bar in the footer of a word document. Unfortunately I have to deal with the mark-up of word documents quite frequently and am therefor exposed to the somewhat unique abilities of the headers and footers feature of MS-Word. During the edit process of the upcoming book, Word voluntarily added a horizontal bar to my footer. Example depicted below. However word doesn’t allow you to highlight and select a horizontal bar and therefor cannot be easily removed by pressing the delete button. This means you have to explore the fantastic menu of word. To remove the bar: 1. Open the footers section, by clicking in that area in the document. 2. Go to menu option Format 3. Borders and Shading 4. The borders and shading menu shows the line that miraculous appeared in my footer, by selecting the option None at the right side of the window it removes the horizontal bar from the footer. 5. Click OK I hope this short tip helps you to keep the frustration to a minimum. ================================================================================ Title: Disabling MinGoodness and CostBenefit URL: https://frankdenneman.ai/2012-07-09-disabling-mingoodness-and-costbenefit/ Date: 2012-07-09 Over the last couple of months I’ve seen recommendations popping up on changing the MinGoodNess and CostBenefit settings to zero on a DRS cluster (KB1017291) . Usually after the maintenance window, when hosts where placed in maintenance mode, the hosts remain unevenly loaded and DRS won’t migrate virtual machines to the less loaded host. By disabling these adaptive algorithms, DRS to consider every move and the virtual machines will be distributed aggressively across the hosts. Although this sounds very appealing, MinGoodness and CostBenefit calculations are created for a reason. Let’s explore the DRS algorithm and see why this setting should only be used temporarily and not as a permanent setting. DRS load balance objectives DRS primary objective is to provide virtual machines their required resources. If the virtual machine is getting the resources it request (dynamic entitlement), than there is no need to find a better spot. If the virtual machines do not get their resources specified in their dynamic entitlement, then DRS will consider moving the virtual machine depending on additional factors. This means that DRS allow certain situations where the administrator feels like the cluster is unbalanced, such as an uneven virtual machine count on hosts inside the cluster. I’ve seen situations where one host was running 80% of the load while the other hosts where running a couple of virtual machines. This particular cluster was comprised of big hosts, each containing 1TB memory while the entire virtual machine memory footprint was no more than 800GB. One host could easily run all virtual machines and provide the resources the virtual machines were requesting. This particular scenario describes the biggest misunderstanding of DRS, DRS is not primarily designed to equally distribute virtual machines across hosts in the cluster. It distributes the load as efficient as possible across the resources to provide the best performance of the virtual machines. And this is the key to understand why DRS does or does not generate migration recommendation. Efficiency! To move virtual machines around, it cost CPU cycles, memory resources and to a smaller extent datastore operation (stun/unstun) virtual machines. In the most extreme case possible, load balancing itself can be a danger to the performance of virtual machines by withholding resources from the virtual machines, by using it to move virtual machines. This is worst-case scenario, but the main point is that the load balancing process cost resources that could also be used by virtual machines providing their services, which is the primary reason the virtual infrastructure is created for. To manage and contain the resource consumption of load balancing operations, MinGoodness and CostBenefit calculations were created. CostBenefit DRS calculates the Cost Benefit (and risk) of a move. Cost: How many resources does it take to move a virtual machine by vMotion? A virtual machine that is constantly updating its large memory footprint cost more CPU cycles and network traffic than a virtual machine with a medium memory footprint that is idling for a while. Benefit: how many resources will it free up on the source host and what will the impact be on the normalized entitlement on the destination host? The normalized entitlement is the sum of dynamic entitlement of all the virtual machines running on that host divided by the capacity of the host. Risk is predicted how the workload might change on both the source and destination host and if the outcome of the move of the candidate virtual machine is still positive when the workload changes. MinGoodness To understand which host the virtual machine must move to, DRS uses the normalized entitlement of the host as the key metric and will only consider hosts that have a lower normalized entitlement than the source host. MinGoodness helps DRS understand what effect the move has on the overall cluster imbalance. DRS awards every move a CostBenefit and MinGoodness rating and these are linked together. DRS will only recommend a move with a negative CostBenefit rating if the move has a highly positive MinGoodness rating. Due to the metrics used, CostBenefit ratings are usually more conservative than the MinGoodness ratings. Overpowering the decision to move virtual machine to host with a lower normalized entitlement due to the cost involved or risk of that particular move. When MinGoodness and CostBenefit are set to zero, DRS calculates the cluster imbalance and recommend any move* that increases the balance of the normalized entitlement of each host within the cluster without considering the resource cost involved. In oversized environments, where resource supply is abundant, setting these options temporarily should not create a problem. In environments where resource demand rivals resource supply, setting these options can create resource starvation. *The number of recommendations are limited to the MaxMovesPerHost calculation. This article contains more information about MaxMovesPerHost. Recommendation My recommendation is to use this advanced option sparingly, when host-load is extremely unbalanced and DRS does not provide any migration recommendation. Typically when the hosts in the cluster were placed in maintenance mode. Permanently activating this advanced option is similar to lobotomizing the DRS load balancing algorithm, this can do more harm in the long run as you might see virtual machines in an almost-constant state of vMotion. ================================================================================ Title: Limiting the number of Storage vMotions URL: https://frankdenneman.ai/2012-06-28-limiting-the-number-of-storage-vmotions/ Date: 2012-06-28 When enabling datastore maintenance mode, Storage DRS will move virtual machines out of the datastore as fast as it can. The number of virtual machines that can be migrated in or out of a datastore is 8. This is related to the concurrent migration limits of hosts, network and datastores. To manage and limit the number of concurrent migrations, either by vMotion or Storage vMotion, a cost and limit factor is applied. Although the term limit is used, a better description of limit is maximum cost. In order for a migration operation to be able to start, the cost cannot exceed the max cost (limit). A vMotion and Storage vMotion are considered operations. The ESXi host, network and datastore are considered resources. A resource has both a max cost and an in-use cost. When an operation is started, the in-use cost and the new operation cost cannot exceed the max cost. The operation cost of a storage vMotion on a host is “4”, the max cost of a host is “8”. If one Storage vMotion operation is running, the in-use cost of the host resource is “4”, allowing one more Storage vMotion process to start without exceeding the host limit. As a storage vMotion operation also hits the storage resource cost, the max cost and in-use cost of the datastore needs to be factored in as well. The operation cost of a Storage vMotion for datastores is set to 16, the max cost of a datastore is 128. This means that 8 concurrent Storage vMotion operations can be executed on a datastore. These operations can be started on multiple hosts, not more than 2 storage vMotion from the same host due to the max cost of a Storage vMotion operation on the host level. [caption id=“attachment_2099” align=“aligncenter” width=“366” caption=“Storage vMotion in progress”][/caption] How to throttle the number of Storage vMotion operations? To throttle the number of storage vMotion operations to reduce the IO hit on a datastore during maintenance mode, it preferable to reduce the max cost for provisioning operations to the datastore. Adjusting host costs is strongly discouraged. Host costs are defined as they are due to host resource limitation issues, adjusting host costs can impact other host functionality, unrelated to vMotion or Storage vMotion processes. Adjusting the max cost per datastore can be done by editing the vpxd.cfg or via the advanced settings of the vCenter Server Settings in the administration view. If done via the vpxd.cfg, the value vpxd.ResourceManager.MaxCostPerEsx41Ds is added as follows: < config > < vpxd > < ResourceManager > < MaxCostPerEsx41Ds > new value < /MaxCostPerEsx41Ds > < /ResourceManager > < /vpxd > < /config > As the max cost have not been increased since ESX 4.1, the value-name remains the same and is valid for all ESX 4.1+ hosts. Please remember to leave some room for vMotion when resizing the max cost of a datastore. The vMotion process has a datastore cost as well. During the stun/unstun of a virtual machine the vMotion process hits the datastore, the cost involved in this process is 1. For example, Changing the to 112, allows 7 concurrent Storage vMotions against a given datastore in the vCenter inventory. If 7 concurrent Storage vMotions are started on this datastore, a vMotion process of a virtual machine using this datastore is queued as the vMotion process would violate the max cost of the datastore. 7 x 16 = 112 + 1 vMotion = 113. The moment a Storage vMotion is completed, the vMotion process will resume as resources become available. Please note that cost and max values are applied to each migration process, impact normal day to day DRS and Storage DRS load balancing operations as well as the manual vMotion and Storage vMotion operations occuring in the virtual infrastructure managed by the vCenter server. As mentioned before adjusting the cost at the host side can be tricky as the costs of operation and limits are relative to each other and can even harm other host processes unrelated to migration processes. If you still have the urge to change the cost on the host, consider the impact on DRS! When increasing the cost of a Storage vMotion operation on the host, the available “slots” for vMotion operations are reduced. This might impact DRS load balancing efficiency when a storage vMotion process is active and should be avoided at all times. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Fab-four: VMWorld 2012 sessions approved URL: https://frankdenneman.ai/2012-06-27-fab-four-vmworld-2012-sessions-approved/ Date: 2012-06-27 This morning I found out that my four sessions are accepted. I’m really pleased and I am looking forward to presenting at each one of them. Two sessions, Architecting Storage DRS Datastore Clusters and vSphere Cluster Resource Pool Best Practices are also scheduled for VMWorld Barcelona. Session ID: STO1545 Session Title: Architecting Storage DRS Datastore Clusters Track: Infrastructure Presenting at: US and Barcelona Presenting with: Valentin Hamburger Session ID: VSP1504 Session Title: Ask the Expert vBloggers Track: Infrastructure Presenting at: US Presenting with: Duncan Epping, Scott Lowe, Rick Scherer and Chad Sakac Session ID: VSP1683 Session Title: vSphere Cluster Resource Pools Best Practices Track: Infrastructure Presenting at: US and Barcelona Presenting with Rawlinson Rivera Session ID: CSM1167 Session Title: Architecting for vCloud Allocation Models Track: Operations Presenting at: US Presenting with Chris Colotti Can’t wait to attend VMworld 2012! See you there. ================================================================================ Title: VMware vSphere Storage DRS Interoperability technical paper available URL: https://frankdenneman.ai/2012-06-05-vmware-vsphere-storage-drs-interoperability-technical-paper-available/ Date: 2012-06-05 Today my second white paper, VMware vSphere Storage DRS Interoperability, is made available for download at the Technical Resource Center at VMware.com. This white paper presents an overview of best practices for customers considering the implementation of VMware vSphere Storage DRS in combination with advanced storage device features or other VMware products. This document zooms in on Storage DRS interoperability with array based features, such as Auto-Tiering, Thin provisioning, Depulication but also explains VMware products such as Snapshots. A small preview: VMware vSphere Snapshots Storage DRS supports virtual machine snapshots. By default, it collocates them with the virtual machine disk file to prevent fragmentation of the virtual machine. Also by default, Storage DRS applies a VMDK affinity rule to each new virtual machine. If it migrates the virtual machine to another datastore, all the files, including the snapshot files, move with it. If the virtual machine is configured with an inter-VMDK affinity setting, the snapshot is placed in the directory of its related disk and is moved to the same destination datastore as when migrated by a Storage vMotion operation. VMware supports the use of vSphere snapshots in combination with Storage DRS. Go and download it here: http://www.vmware.com/resources/techresources/10286 ================================================================================ Title: VMworld Session proposals URL: https://frankdenneman.ai/2012-05-30-vmworld-session-proposals/ Date: 2012-05-30 Here is just a quick overview of the sessions I submitted for VMworld events in San Francisco and Barcelona. I’ve submitted three sessions in total, as my passion for resource management and Storage DRS is a public secret it should be no suprise that all sessions I participate in focus on either vSphere resource managagement or Storage DRS. :) I’ve split them up into two categories, vSphere centric and vCloud centric. The fourth session is the annual Ask the Expert vBloggers with the all-star crew Scott Lowe, Duncan Epping, Rick Scherer, and Chad Sakac. I hope to see you at VMworld! vSphere centric sessions Session 1545 Architecting Storage DRS Datastore Clusters Abstract: In this session Frank Denneman and Valentin Hamburger will cover and explain in great detail what to consider when building a Storage DRS datastore cluster. Introducing the concept of datastore clusters can affect or shift the paradigm of storage management in virtual infrastructures. The goal is to demonstrate the relationship between the datastore cluster and existing objects in the virtual infrastructure and how the introduction of datastore clusters can effect various design decisions. This session is a must for anyone implementing Storage DRS that wants to maximize their cluster and vSphere resource designs. Session 1683 vSphere Cluster Resource Pools Best Practices Abstract: In this session Frank Denneman and Rawlinson Rivera will cover and explain in great detail what to consider when using resource pool inside a vSphere cluster. Introducing the concept of resource pools can affect virtual machine performance and overall resource management in virtual infrastructures. Join Frank and Rawlinson and discover both common pitfalls and best practices of resource pool design. This session is a must for anyone implementing resource pools that wants to maximize their cluster and vSphere resource designs. vCloud Director centric sessions Session 1167 vCloud tracks Architecting for vCloud Allocation Models Abstract: In this session Frank Denneman and Chris Colotti will break down the three vCloud Director Allocation models in depth. Each model’s settings will be shown in detail to explain the effect on vSphere resource scheduling. They will then show how Allocation models of the same type with different configurations, as well as different allocation models could live on the same Provider vDC. The goal is to demonstrate that by not only fully understanding the allocation models, but the vSphere resource allocation together you can design for multiple allocation models on a single Provider vDC. This session is a must for anyone implementing vCloud Director that wants to maximize their cluster and vCloud resource designs. Session 504 Ask the Expert vBloggers - Scott Lowe, Duncan Epping, Rick Scherer, Frank Denneman, Chad Sakac Abstract: One of the highest rated sessions at VMworld is back for it’s fifth year! Come meet four VMware Certified Design Experts (VCDX) on stage answering your questions. We get the top Virtualization Bloggers in the industry and get them on stage answering your questions in a wide array of topics. Simon at Techhead.co.uk wrote a nice article about how to vote for your favorite session at the VMworld.com portal ================================================================================ Title: Blog post on blogs.vmware.com/vsphere/ URL: https://frankdenneman.ai/2012-05-23-blog-post-on/ Date: 2012-05-23 As part of the Technical Marketing team of VMware focusing on the vSphere platform I contribute to the vSphere blog on VMware.com. From this point forward I will post a link to new articles posted on the vSphere blog. SDRS maintenance mode impossible because “The virtual machine is pinned to a host.” ================================================================================ Title: VMware vSphere Metro Storage Cluster Case Study Technical Paper available URL: https://frankdenneman.ai/2012-05-22-vmware-vsphere-metro-storage-cluster-case-study-technical-paper-available/ Date: 2012-05-22 As of today the VMware vSphere Metro Storage Cluster Case Study Technical Paper is available at http://www.vmware.com/resources/techresources/10284 VMware vSphere Metro Storage Cluster (VMware vMSC) is a new configuration within the VMware Hardware Compatibility List. This type of configuration is commonly referred to as a stretched storage cluster or metro storage cluster. It is implemented in environments where disaster/downtime avoidance is a key requirement. This case study was developed to provide additional insight and information regarding operation of a VMware vMSC infrastructure in conjunction with VMware vSphere. This paper will explain how vSphere handles specific failure scenarios and will discuss various design considerations and operational procedures. For me this is a new milestone as this is my first published white paper. I had the honor and pleasure of collaborating with Duncan Epping (DuncanYB), Ken Werneburg (@vmken), Stuart Hardman (@shard_man) and Lee Dilworth (@LeeDilworth) on this paper. Working with industry-leading experts, testing all sorts of scenario’s and listing to them analyzing and brainstorming was inspiring and very educational. Not only is this content great for customers who are interested in vSphere Metro Storage Cluster solutions, but is very educational for people who are interested in HA in general. A must read! ================================================================================ Title: DRS clusters and allocating reserved memory URL: https://frankdenneman.ai/2012-05-21-drs-clusters-and-allocating-reserved-memory/ Date: 2012-05-21 As mentioned in the admission control family, multiple features on multiple layers check to there is enough unused reserved memory available. This article is a part of a short series of articles on how memory is being claimed and provided as reserved memory; other articles will be posted throughout the week. Refresher I’ve published two articles that describes memory reservation at the VM level and the resource pool level. These two sources are an excellent way to refresh your memory (no pun intended) on the reservation construct: • Impact of memory reservation (VM-level) • Resource pool memory reservations “Unclaimed” reserved memory? If a memory reservation is configured on a child object (virtual machine or resource pool) admission control checks if there is enough reserved memory available. Which memory can be claimed for reserved memory? And how about the host-level memory and cluster level memory? Let’s dissect the cluster tree of resource providers and resource consumers and start with a bottom-up approach. Host-level to DRS cluster Both the host and DRS cluster are resource providers to the resource consumers i.e. resource pools and virtual machines. When a host is made a member of a DRS cluster, all its available memory resources are placed at the DRS disposal. The available memory of a host is the memory that is left after the VMkernel claimed host memory. The DRS cluster, also called the root resource pool, reserves this remaining memory. As the DRS cluster reserves this memory per host, all the memory aggregated inside the root resource pool and is actually designated as reserved memory. However to prevent confusion, this reserved memory is labeled as unused reserved memory in the vSphere Client user interface and as such provided to the child resource pools and child virtual machines.At the Resource Allocation Tab of the cluster, the Total memory capacity of the cluster is listed as well as the reserved capacity. The Available capacity is the result of Total capacity – Reserved capacity. Note that if HA is configured the amount of resources reserved for failover is automatically added to the reserved capacity. Child resource pools Resource pools allow for hierarchical partitioning of the cluster, but they always span the entire cluster. Resource pools draw resources from the root resource pool and do not pick and select resources from a specific hosts. The root resource pool functions as an abstraction layer. When configuring a reservation on resource pool level the specified amount of memory is claimed by that specific resource pool and cannot be allocated by other resource pools. Note that the claim of reserved resources by the resource pool is done immediately during the creation of the resource pool. It does not matter if there are running virtual machine inside the resource pools or not. The total configured memory is withdrawn from the root resource pool and thus unavailable for other resource pools. Please keep this in mind when sizing resources pools. The next article will expand on virtual machines inside a resource pool. ================================================================================ Title: Admission control and vCloud Allocation Pool model URL: https://frankdenneman.ai/2012-05-16-admission-control-and-vcloud-allocation-pool-model/ Date: 2012-05-16 The previous article outlines the multiple admission controls active in a virtual infrastructure. One that always interested me in particular is the admission control feature that verifies resource availability. With the introduction of vCloud director another level of resource construct were introduced. Along with Provider virtual datacenter (vDC) and Organization vDCs, allocation models were introduced. An allocation model defines how resources are allocated from the provider vDC. An organization vDC must be configured with one of the following three allocation models: “Pay As You Go”, “Allocation Pool” and “Reservation Pool”. It is out of the scope to describe all three models, please visit Chris Colotti’s blog or Yellow Bricks to read more about allocation models. As mentioned before the distinction between each of the allocation models is how resources are consumed. Depending on the chosen allocation model reservations and limits will be set on resource pool, virtual machine level, or both. One of the most interesting allocation model is the Allocation Pool model as it sets reservations on both resource pool level and virtual machine level simultaneously. During configuration of the allocation pool model, an amount of guaranteed resources can be specified. (Guaranteed is the vCloud term for vSphere reservation). The question I was given is will lowering the default value of 100% guaranteed memory result in an increase of more virtual machines inside the Organization vCD? And the answer lies within the working of vSphere admission control. Allocation Pool model settings By default the Allocation Pool model sets a 100% memory reservation on both resource pool level and virtual machine level. By lowering the default guarantee, it allows for opportunistic memory allocation on both resource pool level and virtual machine level. Creating this burstable space (resources available for opportunistic access) usually provides an higher consolidation ratio of virtual machines, however due to the simultaneous configuration of reservation on both resource pool and virtual machine level, this is not the case. Virtual machine level reservation During power-on operation admission control checks if the resource pool can satisfy the virtual machine level reservation. Because expandable reservation is disabled in this model, the resource pool is not able to allocate any additional resources from the provider vDC. Therefor the virtual machine memory reservation can only be satisfied by the resource pool level reservation of the organization vDC itself. When a virtual machine is using memory protected by a virtual machine level reservation, this memory is withdrawn from the resource pool-level reservation. If the resource pool does not have enough available memory to guarantee the virtual machine reservation, the power-on operation fails. Let’s use a scenario to visualize the process a bit better. Scenario An organization vCD is created with the Allocation Pool model and the memory allocation is set to 20GB; the memory guarantee is set to 50%. These settings result in a resource pool memory limit of 20GB and a memory reservation of 10GB. When powering up a 2GB virtual machine, 1GB of reserved resources will be allocated to that virtual machine and withdrawn from the available reserved memory pool. Admission control allows to power-on virtual machines until the reserved memory pool is reduced to zero. Following the previous example, virtual machine 2 is powered on. The resource pool providing resources to the organization vDC has 9 GB available in its pool of reserved memory. Admission control allows the power-on operation of the virtual machine as this pool can provide the reserved resources specified by the virtual machine level reservation. During each power-on operation 1GB of reserved memory is withdrawn from the reserved memory pool available to the organization vDC. Resulting in admission control allowing to power on ten virtual machines. When attempting to deploy virtual machine 11, admission controls fails the power-on operation as the organization vDC has no available reserved memory to satisfy the virtual machine level reservation. Note: This scenario excludes the impact of memory overhead reservation of each virtual machine. Under normal circumstances, the number of virtual machines that could be powered on would be close to 8 instead of 10 as the reserved pool available to the organization vDC is used to satisfy the memory overhead reservation of each virtual machine as well. Because the guarantee setting of the Allocation Pool model configures resource pool and virtual machine memory reservation settings simultaneously, the supply and demand of reserved memory resources are always equal regardless of the configured percentage setting. Therefore offering opportunistic access to resources inside the organization vDC does not allow an increase of the number of virtual machines inside the organization vDC. The next question arises, why should you lower the percentage of guaranteed resources? Providing burstable space increases the number of Organization vCDs inside the Provider vDC. Resource pool memory reservation Upon creation resource pools claim and withdraw the configured reserved resources from their parent instantaneously. This memory cannot be provided or distributed to other organization vDCs regardless of utilization of these resources. Although new resource constructs are introduced in a vCloud environment, consolidation ratios and resource management still leverage traditional vSphere resource management constructs and rules. Chris Colotti and I are currently working on a technical paper describing the allocation models in details and the way they interact with vSphere resource management. We hope to see this published soon. ================================================================================ Title: The Admission Control Family URL: https://frankdenneman.ai/2012-05-10-the-admission-control-family/ Date: 2012-05-10 It’s funny how sometimes something, in this case a vSphere feature, becomes a “trending topic” on any given day or week. Yesterday I was discussing admission control policies with Rawlinson Riviera (@punchingclouds) and we discussed how to properly calculate a percentage for the percentage based admission control with keeping consolidation ratios in mind. And today Gabe published an article about his misconception of admission control, which triggered me to write an article of my own about admission control. When discussing admission control usually only HA admission control policies are mentioned. However, HA isn’t the only feature using some sort of admission control. Storage DRS as well as DRS and the ESX(i) host have each their own admission control. Let’s take a closer look what admission control actually is and see how each admission control fits in the process of a virtual machine power-on operation. What’s the function of admission control? I like to call it our team of virtual bouncers. Admission control is there to ensure that sufficient resources are available for the virtual machine to function within it’s specified parameters / resource requirements. The last part about the parameters and requirements is the key to understand admission control. During a virtual machine power-on or a move operation, admission control checks if sufficient unreserved resources are available before allowing a virtual machine to power on or moved into the cluster. If a virtual machine is configured with reservation, this could be CPU, memory or even both, admission control needs to make sure that the datastore cluster, compute cluster, resource pool and host can provide these resources. If one of these constructs cannot allocate and provide these resources, then the datastore cluster, compute cluster, resource pool or host cannot provide an environment where the virtual machine can operate within its required parameters. In other words, the moment a virtual machine is configured to have an X amount of resources guaranteed, you want the environment to actually oblige to that wish and that’s why admission control is developed. As a vSphere environment can be configured in many different ways, each feature sports its own admission control, as you do not want to introduce dependencies for such a crucial component. Let’s take a closer look at each admission control feature and their checkpoints. High Availability Admission control: During a virtual machine power-on operation, HA checks if the virtual machine can be powered-on without violating the required capacity to cope with a host failure event. Depending on the HA admission control policy, HA checks if the cluster can provide enough unreserved resources to satisfy the virtual machine reservation. The internals of each type Admission control policies is outside the scope of this article, more information can be found in the clustering deep dive books or online at Yellow-bricks. After HA admission control gives the green light, it’s up to Storage DRS admission control if the virtual machine is placed in a Storage DRS datastore cluster. Storage DRS admission control checks datastore connectivity amongst the hosts in the datastore cluster and selects the hosts with the highest datastore connectivity to ensure the highest portability of a virtual machine. If there are multiple hosts with the same number of datastore connected it selects the host with the lowest compute utilization. Up next is DRS admission control to review the cluster state. DRS ensures that sufficient unreserved resources are available in the cluster before allowing the virtual machine to power on. If the virtual machine is placed inside a resource pool, DRS checks if the resource pool can provide enough resources to satisfy the reservation. Depending on the setting “expandable reservation” the resource pool checks its own pool of unreserved resources or borrows resources from its parent. If a virtual machine is moved into the cluster and EVC is enabled in the DRS cluster, EVC admission control verifies if the applied EVC mode to the virtual machine does not exceed the current EVC baseline of the cluster. DRS selects a host based on configured VM-VM and VM-Host affinity rules. Last step is Host admission control. In the end it’s the host that actually needs to provide the compute environment to allow the virtual machine to operate in. A cluster can have enough unreserved resources available, but it can be in a fragmented stage, where there are not enough resources available per host to satisfy the virtual machine reservation. To solve this problem a DRS invocation is triggered to recommend virtual machine migrations to re-balance the cluster and free up space on a particular host for the new virtual machine. If DRS is not enabled, the Host rejects the virtual machine due to the inability to provide the required resources. Host admission control also verifies is the virtual machine configuration is compatible with the host. The VM networks and datastores must be available in order to accommodate the virtual machine. The virtual machine compatibility list also the suitable host if the virtual machine is placed inside a “must” VM-Host affinity rules, admission control checks if its listed in the compatibility list. The last check is if the host can create a VM-swap file on the designated VM swap location. So there you have it before a virtual machine is powered-on or moved into a cluster, all these admission controls will make sure the virtual machine can operate within its required parameters and no cluster feature requirement is being violated. ================================================================================ Title: Mixing Resource Pools and Virtual Machines on the same hierarchical level URL: https://frankdenneman.ai/2012-05-09-mixing-resource-pools-and-virtual-machines-on-the-same-hierarchical-level/ Date: 2012-05-09 One of most frequent questions I receive is about mixing resource pools and virtual machines at the same hierarchical level. In almost all of the cases we recommend to commit to one type of child objects. Either use resource pools and place virtual machines within the resource pools or place only virtual machines at that hierarchical level. The main reason for this is how resource shares work. Shares determine the relative priority of the child-objects (virtual machines and resource pools) at the same hierarchical level and decide how excess of resources (total system resources - total Reservations) made available by other virtual machines and resource pools are divided. Shares are level-relative, which means that the number of shares is compared between the child-objects of the same parent. Since, they signify relative priorities; the absolute values do not matter, comparing 2:1 or 20.000 to 10.000 will have the same result. Lets use an example to clarify. In this scenario the DRS cluster (root resource pool) has two child objects, resource pool 1 and a virtual machine 1. The DRS cluster issues shares amongst its children, 4000 shares issued to the resource pool, 2000 shares issued to the virtual machine. This results in 6000 shares being active on that particular hierarchical level. During contention the child-objects compete for resources as they are siblings and belong to the same parent. This means that the virtual machine with 2000 shares needs to compete with the resource pool that has 4000 shares. As 6000 shares are issued on that hierarchical level, the relative value of each child entity is (2000 of 6000 shares) = 33% for the virtual machine and (4000 shares of 6000=66%) for the resource pool. The problem with this configuration is that the resource pool is not only a resource consumer but also a resource provider. So that it must claim resources on behalf of its children. Expanding the first scenario, two virtual machines are placed inside the resource pool. The Resource Pool issues shares amongst its children, 1000 shares issued to virtual machine 2 (VM2) and 2000 shares issued to virtual machine 3 (VM3). This results in 3000 shares being active on that particular hierarchical level. During contention the child-objects compete for resources as they are siblings and belong to the same parent which is the resource pool. This means that VM2 owning 1000 shares needs to compete with VM3 that has 2000 shares. As 3000 shares are issued on that hierarchical level, the relative value of each child entity is (1000 of 3000 shares) = 33% for VM2 and (2000 shares of 3000=66%) for VM3. As the resource pool needs to compete for resources with the virtual machine on the same level, the resource pool can only obtain 66% of the cluster resources. These resources are divided between VM2 and VM3. That means that VM2 can obtain up to 22% of the cluster resources (1/3 of 66% of the total cluster resources is 22%). Forward to scenario 3, two additional virtual machines are created and are on the same level as Resource Pool 1 and virtual machine 1. The DRS cluster issues 1000 shares to VM4 and 1000 shares to VM5. As the DRS cluster issued an additional 2000 shares, the total issued shares is increased to 8000 shares. Resulting in dilution of the relative share values of Resource Pool 1 and VM1. Resource pool 1 now owns 4000 shares of a total 8000 bringing the relative value down from 66% to 50%. VM1 owns 2000 shares of 8000, bringing its value down to 25%. Both VM4 and VM5 own each 12.5% of shares. As resource pool 1 provides resources to its child-object VM2 and VM3, fewer resources are divided between VM2 and VM3. That means that in this scenario VM2 can obtain up to 16% of the cluster resources (1/3 of 50% of the total cluster resources is 16%). Introducing more virtual machines to the same sibling level as the Resource Pool 1, will dilute the resources available to virtual machines inside Resource Pool 1. This is the reason why we recommend to commit to a single type of entity at a specific sibling level. If you create resource pools, stick with resource pools at that level and provision virtual machines inside the resource pool. Another fact is that a resource pool receives shares similar to a 4-vcpu 16GB virtual machine. Resulting in a default share value of 4000 shares of CPU and 163840 shares of memory when selecting Normal share value level. When you create a monster virtual machine and place it next to the resource pool, the resource pool will be dwarfed by this monster virtual machine resulting in resource starvation. Note: Shares are not simply a weighting system for resources. All scenarios to demonstrate way shares work are based on a worst-case scenario situation: every virtual machine claims 100% of their resources, the system is overcommitted and contention occurs. In real life, this situation (hopefully) does not occur very often. During normal operations, not every virtual machine is active and not every active virtual machine is 100% utilized. Activity and amount of contention are two elements determining resource entitlement of active virtual machines. For ease of presentation, we tried to avoid as many variable elements as possible and used a worst-case scenario situation in each example. So when can it be safe to mix and match virtual machines and resource pools at the same level? When all child-objects are configured with reservation equal to their configuration and limits, this result in an environment where shares are overruled by reservation and no opportunistic allocation of resources exist. But this designs introduces other constraints to consider. ================================================================================ Title: Storage DRS load balance frequency URL: https://frankdenneman.ai/2012-05-07-storage-drs-load-balance-frequency/ Date: 2012-05-07 Storage DRS load balancing frequency differs from DRS load balance frequency, where DRS runs every 5 minutes to balance CPU and memory resources, Storage DRS uses a far more complex load balancing scheme. Let’s take a closer look at Storage DRS load balancing. Default invocation period The invocation period of Storage DRS is every 8 hours and uses what’s called past-day statistics. Storage DRS triggers a load balancing evaluation process if a datastore exceeds the space utilization threshold. Storage DRS load balances space utilization of the datastores and if I/O metric is enabled, load balances on I/O utilization as well. Space utilization and I/O load on a datastore are two different load patterns, therefore Storage DRS uses different methods to accumulate and analyze IO load patterns and space utilization of the datastores within the datastore cluster. Space load balancing statistic collection Analyzing space utilization is rather straightforward; Storage DRS needs to understand the growth rate of each virtual machine and the utilization of each datastore. It collects information from the vCenter database to understand the disk usage and file structure of each virtual machine. Each ESXi host reports datastore utilization at a frequent interval and this is stored in the vCenter database as well. Storage DRS checks whether the datastore utilization is above the user-set threshold. When generating a load balance recommendation, Storage DRS knows where to move a virtual machine as it knows the current space growth of virtual machines on destination datastores, preventing a threshold violation direct after the placement. I/O load utilization is a different beast. I/O load may grow over time, however the datastore can experience a temporary increase of load. How does Storage DRS handle these spikes? Enter past-day statistics! I/O load balancing statistic collection Storage DRS uses two main information sources for I/O load balancing statistic collection, vCenter and the SIOC injector. vCenter statistics is uses to understand the workload each virtual disk is generating, this is called workload modeling. SIOC injector is used to understand the device performance and this is called device modeling. See “Impact of load balancing on datastore cluster configuration” for more info about device and workload modeling. Data of workload and device modeling is aggregated in a performance snapshot and is used as input for generating migration recommendations. Migrating virtual machine disk files takes time and most of all it impacts the infrastructure, migrating based on peak value is the last thing you want to do when you are introducing long-term high impact workloads. Therefore Storage DRS starts to recommend I/O load-related recommendations once an imbalance persists for some period of time. To avoid being caught out by peak load moments, Storage DRS does not use real-time statistics. It aggregates all the data points collected over a period of time. By using 90th percentile values, Storage DRS filters out the extreme spikes while still maintaining a good view of the busiest period of that day as this value translate to the lowest edge of the busiest period. As workloads shift during the day enough information needs to be collected to make a good assessment of the workloads. Therefore Storage DRS needs at least 16 hours of data before recommendation are made. By using at least 16 hours worth of data Storage DRS has an enough data of the same timeslot so it can compare utilization of datastores for example: datastore 1 to datastore 2 on Monday morning at 11:00. As 16 hour is 2/3 of the day Storage DRS receives enough information to characterize the performance of datastore on that day. But how does this tie in with the 8 hour invocation period? 8-hour invocation period and 16 hours worth of data Storage DRS uses 16 hours of data, however this data must be captured in the current day otherwise the performance snapshot of the previous day is used. How is this combined with the 8-hour invocation periods? This means that technically, the I/O load balancing is done every 16 hours. Usually after midnight, after the day date, the stats are fixed and rolled up. This is called the rollover event. The first invocation period (08:00) after the rollover event uses the 24 hours statistics of the previous day. After 16 hours are passed of the current day, Storage DRS uses the new performance snapshot and may evaluate moves. ================================================================================ Title: Whiteboard desk URL: https://frankdenneman.ai/2012-05-03-whiteboard-desk/ Date: 2012-05-03 As I’m an avid fan of “post your desk topics \ workspace ” forum threads, I thought it might be nice to publish a blog article about my own workspace. I always love to see how other people design their work environment and how they customize furniture to suit their needs. Hopefully you can find some inspiration in mine. Last year I decided to refurbish my home office. To create a space that enables me to do my work in the most optimum way, and of course that is pleasing to the eye. The first thing that came to mind was a whiteboard and a really big one. So I needed to build a wall to hang the whiteboard, as the room didn’t had any wall that could hold a whiteboard big enough. After completion of the wall, a 6 x 3 feet wide whiteboard found its way to my office. Although its roughly 5 to 6 feet away from my desk, I realized I didn’t use it enough due to distance. Sitting behind the desk while on the phone or just using my computer, I found myself scribbling on pieces of paper instead of getting out of my chair and walk over to the whiteboard. Therefor I needed a small whiteboard I could grab and use at my desk. It seemed reasonable, however I like minimalistic designs where clutter is removed as much as possible. I needed to come up with something different; enter the whiteboard desk! Whiteboard desk Instead of buying a mini whiteboard that needs to be stored when not used, I decided to visit my local IKEA and see what’s available. Besides “show your desk” threads I hit ikeahackers.net on a daily basis. While looking at tables, I noticed that the IKEA kitchen department sells customized tabletops. Each dimension is possible in almost every shape. I decided to order a 7 feet by 3 feet high-gloss white tabletop with a stainless steel edge. The Ikea employee asked where to put the sink, she was surprised when I told her that the tabletop was going to function as a desk. I chose to order the 2 inch thick tabletop as I need to have a desktop that is sturdy enough to hold the weight of a 27” I-mac and a 30” TFT screen. The stainless steel edge fits snug around the desk and covers each side; it doesn’t stick out and is not noticeable when typing. It looks fantastic! However the downside is the price, it was more expensive than the tabletop itself. The alternative is a laminate cover that looks like it will be worn out easily. While spending most of my time behind my desk I thought it was worth the investment of buying the real thing. The high-gloss finish acts as the whiteboard surface and works like a charm with any whiteboard markers. I left notes on my desk for multiple days and could be removed without leaving a trace. The tabletop rest on two IKEA Vika Moliden stands, due to the cast of the shadow its very difficult to notice that the color of the stands do not exactly match the color as the stainless steel edge. The whiteboard desk is just an awesome piece of furniture. When on the phone I can take notes on my desk while immediately drawing diagrams next to it. It saves a lot of trees, saves a lot of time scrambling for a piece of paper, and a pen and decreases clutter on the desk. The only thing you need to do when building a whiteboard desk is to banish all permanent markers in your office. :) It would be awesome to see what your workspace looks like. What do you love about your workspace and maybe show your own customizations? I would love to see blogs articles pop up describing the workspace of bloggers. Please post a link to your blog article in the comment section. ================================================================================ Title: Aggregating datastores from multiple storage arrays into one Storage DRS datastore cluster. URL: https://frankdenneman.ai/2012-04-26-aggregating-datastores-from-multiple-storage-arrays-into-one-storage-drs-datastore-cluster/ Date: 2012-04-26 Combining datastores located on different storage arrays into a single datastore cluster is a supported configuration, such a configuration could be used during a storage array data migration project where virtual machines must move from one array to another array, using datastore maintenance mode can help speed up and automate this project. Recently I published an article about this method on the VMware vSphere blog. But what if multiple arrays are available to the vSphere infrastructure and you want to aggregate storage of these arrays to provide a permanent configuration? What are the considerations of such a configurations and what are the caveats? Key areas to focus on are homogeneity of configurations of the arrays and datastores. When combining datastores from multiple arrays it’s highly recommended to use datastores that are hosted on similar types of arrays. Using similar type of arrays, guarantees comparable performance and redundancy features. Although RAID levels are standardized by SNIA, implementation of RAID levels by different vendors may vary from the actual RAID specifications. An implementation used by a particular vendor may affect the read and write performance and the degree of data redundancy compared to the same RAID level implementation of another vendor. Would VASA (vSphere Storage APIs - Storage Awareness) and Storage profiles be any help in this configuration? VASA enables vCenter to display the capabilities of the LUN/datastore. This information could be leveraged to create a datastore cluster by selecting the datastores that have similar Storage capabilities details, however the actual capabilities that are surfaced by VASA are being left to the individual array storage vendors. The detail and description could be similar however the performance or redundancy features of the datastores could differ. Would it be harmful or will Storage DRS stop working when aggregating datastores with different performance levels? Storage DRS will still work and will load balance virtual machine across the datastores in the datastore cluster. However, Storage DRS load balancing is focused on distributing the virtual machines in such a way that the configured thresholds are not violated and getting the best overall performance out of the datastore cluster. By mixing datastores that provide different performance levels, virtual machine performance could not be consistent if it would be migrated between datastores belonging to different arrays. The article “Impact of load balancing on datastore cluster configuration” explains how storage DRS picks and selects virtual machine to distribute across the available datastores in the cluster. Another caveat to consider is when virtual machines are migrated between datastores of different arrays; VAAI hardware offloading is not possible. Storage vMotion will be managed by one of the datamovers in the vSphere stack. As storage DRS does not identify “locality” of datastores, it does not incorporate the overhead caused by migrating virtual machines between datastores of different arrays. When could datastores of multiple arrays be aggregated into a single datastore if designing an environment that provides a stable and continuous level of performance, redundancy and low overhead? Datastores and array should have the following configuration: • Identical Vendor. • Identical firmware/code. • Identical number of spindles backing diskgroup/aggregate. • Identical Raid Level. • Same Replication configuration. • All datastores connected to all host in compute cluster. • Equal-sized datastores. • Equal external workload (best non at all). Personally I would rather create a multiple datastore clusters and group datastores belonging to a single storage array into one datastore cluster. This will reduce complexity of the design (connectivity), no multiple storage level entities to manage (firmware levels, replication schedules) and will leverage VAAI which helps to reduce load on the storage subsystem. If you feel like I missed something, I would love to hear reasons or recommendations why you should aggregate datastores from multiple storage arrays. More articles in the architecting and designing datastore clusters series: Part1: Architecture and design of datastore clusters. Part2: Partially connected datastore clusters. Part3: Impact of load balancing on datastore cluster configuration. Part4: Storage DRS and Multi-extents datastores. Part5: Connecting multiple DRS clusters to a single Storage DRS datastore cluster. ================================================================================ Title: Connecting multiple DRS clusters to a single Storage DRS datastore cluster. URL: https://frankdenneman.ai/2012-04-19-connecting-multiple-drs-clusters-to-a-single-storage-drs-datastore-cluster/ Date: 2012-04-19 Recently I received the question if you can connect multiple compute (HA and DRS) clusters to a single Storage DRS datastore cluster and specifically how this setup might impact Storage IO Control functionality. Let’s cover sharing a datastore cluster by multiple compute clusters first before diving into details of the SIOC mechanism. Sharing datastore clusters Sharing datastore clusters across multiple compute clusters is a supported configuration. During virtual machine placement the administrator selects which compute cluster the virtual machine will run in, Storage DRS selects the host that can provide the most resources to that virtual machine. A migration recommendation generated by Storage DRS does not move the virtual machine at host level, consequently a virtual machine cannot move from one compute cluster to another compute cluster by any operation initiated by Storage DRS. Maximums Please remember that the maximum supported number of hosts connected to a datastore is 64. Keep this in mind when sizing the compute cluster or connecting multiple compute clusters to the datastore cluster. As the maximum number of datastores inside a datastore cluster is 32 I think that the number of host connected is the first limit you hit in such a design as the total supported number of paths is 1024 and a host can connect up to 255 LUNs. The VAAI-factor If the datastores are formatted with the VMFS, it’s recommended to enable VAAI on the storage Array if supported. One of the important VAAI primitive is the Hardware assisted locking, also called Atomic Test and Set (ATS). ATS replaces the need for hosts to place a SCSI-2 disk lock on the LUN while updating the metadata or growing a file. A SCSI-2 disk lock command locks out other host from doing I/O to the entire LUN, while ATS modifies the metadata or any other sector on the disk without the use of a SCSI-2 disk lock. This locking was the focus of many best practices around the connectivity of datastores. To reduce the amount of locking, the best practice was to reduce the number of host attached. By using newly formatted VMFS5 volumes in combination with a VAAI-enabled storage array, scsi-2 disk lock commands are a thing of the past. Upgraded VMFS5 volumes or VMFS3 volumes will fall back to using SCSI-2 disk locks if the ATS command fails. For more information about VAAI and ATS please read the KB article 1021976. Note: If your array doesn’t support VAAI, be aware that SCSI-2 disk lock commands can impact scaling of the architecture. Storage DRS IO Load balancing and Storage IO Control When enabling the IO Metric on the datastore cluster, Storage DRS automatically enables Storage IO Control (SIOC) on all datastores in the cluster. Storage DRS uses the IO injector from SIOC to determine the capabilities of a datastore, however by enabling SIOC it also provides a method to fairly distribute I/O resources during times of contention. SIOC uses virtual disk shares in order to distribute storage resources fairly and are applied on a datastore wide level. The virtual disk shares of the virtual machine running on that datastore are relative to the virtual disk shares of other virtual machines using that same datastore. To be more specific, SIOC is a host-level module and aggregates the per-host views into a single datastore view in terms of observed latency. If the observed latency exceeds the SIOC level latency threshold, each host sets its own IO queue length based on the total virtual disks shares of the virtual machines in that host using the datastore. As SIOC and its shares are datastore focused cluster membership of the host has no impact on detecting the latency threshold violation and managing the I/O stream to the datastore. Previous articles in the SDRS short series Architecture and design of Datastore clusters: Part1: Architecture and design of datastore clusters. Part2: Partially connected datastore clusters. Part3: Impact of load balancing on datastore cluster configuration. Part4: Storage DRS and Multi-extents datastores. ================================================================================ Title: I/O Analyzer v1.1 URL: https://frankdenneman.ai/2012-03-30-io-analyzer-v1-1/ Date: 2012-03-30 I/O Analyzer v1.1 is now live on the Flings site: http://labs.vmware.com/flings/io-analyzer I/O Analyzer is a virtual appliance tool for measuring storage performance. This version of I/O Analyzer adds the ability to run trace replay – a function which allows a user to replay an I/O trace that was captured elsewhere (with vscsistats) on the target test system. This version also has cool data visualization charts, both for the characteristics of an imported trace, and performance results on the test system. This is really cool stuff, go check it out. ================================================================================ Title: Impact of Intra VM affinity rules on Storage DRS URL: https://frankdenneman.ai/2012-02-21-impact-of-intra-vm-affinity-rules-on-storage-drs/ Date: 2012-02-21 By default Storage DRS applies an Intra-VM affinity rule to all new virtual machines in the datastore cluster. The Intra-VM affinity rule keeps the virtual machine files, such as VMX file, log files, vSwap and VMDK files together on one datastore. Keeping all files together on one datastore allows ease of troubleshooting. However Storage DRS load balance algorithms may benefit from distributing the virtual machine across datastores. Let’s zoom in how Storage DRS handles virtual machine with multiple disks when the Intra-VM affinity rule is removed from the virtual machine. DrmDisk Storage DRS uses the construct “DrmDisk” as the smallest entity it can migrate. A DrmDisk represent a consumer of datastore resources. This means that Storage DRS creates a DrmDisk for each VMDK belonging to the virtual machine. The interesting part is the collection of system files and swap file belonging to virtual machines. Storage DRS creates a single Drmdisk for all the system files, if an alternate swapfile location is specified, the vSwap file is represented as a separate DrmDisk and Storage DRS will be disabled on the swap DrmDisk. More info about alternate swapfile locations can be found here. For example a virtual machine with three VMDK’s and with no alternate swapfile locations configured, Storage DRS creates 4 DrmDisk: • A separate DrmDisk for each Virtual Machine Disk File • A DrmDisk for system files (VMX, Swap, logs, etc) Initial placement recommendation will look similar to this screenshot when the Intra-VM affinity rule is disabled. Notice the separate recommendation for the “virtual machine configuration file”? This is the DrmDisk containing the system files. Initial placement Space load balancing Initial placement and Space load balancing benefit from this increased granularity tremendously. Instead of searching a suitable datastore that can fit the virtual machine as a whole, Storage DRS is able to seek for appropriate datastores for each DrmDisk file separately. Recently I wrote an article about datastore cluster fragmentation and Storage DRS ability to issue prerequisite migrations. You can imagine due to the increased granularity, datastore cluster fragmentation is less likely to happen and if prerequisite migrations are required, the number of migrations is expected to be a lot less. IO load balancing Similar to initial placement and load balancing, I/O load balancing benefit from the deeper level of detail. It can find a better fit for each workload generated by the VMDK files. The system file DrmDisk will not be migrated quite often as it small in size and does not generate a lot of I/O often. Storage DRS analyzes the workload and generates a workload model for each DrmDisk, it then decides which datastore it needs to place the DrmDisk to keep the load balanced within the datastore cluster while offering enough performance for each DrmDisk. You can imagine this becomes a lot harder when Storage DRS is required to keep all the VMDK files together. Usually the datastore chosen is the datastore that provides the best performance for the most demanding workload AND is able to store all the virtual machine disk files and system files. Now let’s dig into this a little deeper, for example the virtual machine used in the previous example has two DrmDisk generating heavy workloads, while the DrmDisks containing the system files and VMDK2 are “cold”. If Intra-VM affinity rules are used, Space balancing is required to find a datastore that has 350+ GB free without exceeding the space utilization threshold. If I/O load balancing is enabled, this datastore also needs to provide enough performance to keep the latency below the I/O latency threshold (by default 15ms) after placing the 4 DrmDisks. You can imagine it’s a lot less complicated when space and I/O load balancing are allowed to place each DrmDisk on a datastore that suits their needs. How to change default datastore cluster behavior? Mentioned before, datastore cluster defaults in applying an Intra-VM affinity rule to each new virtual machine. Recently Duncan published an article on how to change the affinity rules on active virtual machines. Unfortunately there is not User-Interface option available that can disable this behavior, so I turned to my good friend and colleague Alan Renouf and he created some nice PowerCLI code to solve this problem: As I’m not a powerCLI user at all, I’m relaying Alan’s instructions: First you need to run the below code to put the function into memory: function Set-DatastoreClusterDefaultIntraVmAffinity{ param( [CmdletBinding()] [parameter(Position = 0, Mandatory = $true, ValueFromPipeline = $true)] [PSObject]$DSC, [Switch]$Enabled ) process{ $SRMan = Get-View StorageResourceManager if($DSC.GetType().Name -eq “string”){ $DSC = Get-DatastoreCluster -Name $DSC | Get-View } elseif($DSC.GetType().Name -eq “DatastoreClusterImpl”){ $DSC = Get-DatastoreCluster -Name $DSC.Name | Get-View } $spec = New-Object VMware.Vim.StorageDrsConfigSpec $spec.podConfigSpec = New-Object VMware.Vim.StorageDrsPodConfigSpec $spec.podConfigSpec.DefaultIntraVmAffinity = $Enabled $SRMan.ConfigureStorageDrsForPod($DSC.MoRef, $spec, $true) } } Once this has been run you can use this function…. Get-DatastoreCluster “Shared Datastores” | Set-DatastoreClusterDefaultIntraVmAffinity Shared datastores is the name of the Datastore cluster, you can change that into the name of your own datastore cluster. Or in the case you have multiple datastore clusters and want to disable the rule for all datastore clusters at once, omit the name of the datastore cluster at all. If ease of troubleshooting is not your first concern, than it might be beneficial to the performance of Storage DRS to disable the default Intra-VM affinity rule on the virtual machines in the datastore cluster. However I’m interested in reasons why you wouldn’t want to disable the default affinity rule besides troubleshooting effort. Note: Unfortunately I’m unaware why VMware decided to use the Intra-VM affinity rule as default and I do not know if a future release of vSphere will provide a UI setting to change the affinity rule behavior of the datastore cluster. Please leave a comment if you would like this option included in a new version of vSphere. All I can do is relay this to the appopriate product manager. ================================================================================ Title: Storage DRS I/O load balancing and Array-based Auto-Tiering URL: https://frankdenneman.ai/2012-02-09-storage-drs-io-load-balancing-and-array-based-auto-tiering/ Date: 2012-02-09 In its basic form Storage DRS can be used together with any array, however there are a few combinations of Storage array features and Storage DRS features that don’t mix easily. One of the most sought after question is can Storage DRS work with Array based Auto-tiering? And the answer is yes, yes you can use initial placement and out of space avoidance features that Storage DRS offers, however it is not recommended to enable the I/O metric feature. Modeling The main goal of the I/O metric function, popular called I/O load balancing, is to resolve the imbalance of performance delivered from datastores in the datastore cluster. To avoid hotspots in the datastore cluster and decrease overall latency imbalance, Storage DRS I/O load balancing uses device modeling and virtual machine workload modeling. Device modeling helps Storage DRS to understand the performance characteristics of the devices backing the datastores, while virtual machine workload modeling analyzes virtual machine workload running inside the datastore cluster. Both device and workload modeling assists Storage DRS to asses the improvement of I/O latency that will be achieved after a virtual machine migration. Device modeling and the SIOC injector To understand and learn the performance of the devices backing the datastore, Storage DRS uses the Storage IO Control (SIOC) workload injector. To characterize the datastore, SIOC injector opens and read random blocks of the datastore. As the SIOC injector does not open every block backing the datastore, we cannot ensure that the SIOC injector opens an identical number of blocks of each performance tier to characterize the disk. As multiple performance tiers of disk back the datastore there is a possibility that the SIOC injector might open blocks located on similar speed disks, either slow or fast, while the datastore is primarily backed by disk with a different performance level. Let’s use an example to clarify this further. In the diagram pictured above, SIOC opens random blocks and perform its tests. Unfortunately it doesn’t open blocks on other disks. While most of the blocks backing the datastore are located on faster performing disks, Storage DRS device modeling will characterize this disk with performance similar to 7.2K SATA disks. This inaccurate characterization of datastore performance might lead to an incorrect performance assessment and can lead to Storage DRS withholding a migration recommendation while there is sufficient performance available. Segment migration triggered by auto-tiering algorithms By using SIOC injector Storage DRS evaluate the performance of the disks, however Auto-tiering solutions migrate LUN segments (chunks) to different disk types based on the use pattern. Hot segments (frequently accessed) typically move to faster disks while cold segments move to slower disks. Depending on the array type and vendor there are different kind of policies and threshold for these migrations. By default Storage DRS is invoked every 8 hours and requires performance data over more than 16 hours to generate I/O load balancing decisions. Multiple storage vendors offer auto-tiering solutions, each using different time-cycles to collect and analyze workload before moving LUN segments. Some auto-tiering solutions move chunks based on real-time workload while other arrays move chunks after collecting performance data for 24 hours. This means that auto tiering solutions alter the landscape in which the SIOC injector performs its test. Let’s turn to another scenario for clarification. In this scenario, SIOC is primarily opening blocks located in the Tier-1 diskgroup belonging to the datastore. As the datastore isn’t using these segments that often (cold) the auto tiering solution decides to migrate these segments to a lower tier. In this case the segments are migrated to 15K disks instead of SSD devices. Storage DRS expects that the behavior of the device remains the same for at least 16 hours; it will base its calculation on these facts. Auto tiering solutions might change the underlying structure of the datastore based on its algorithm and timescales, conflicting with Storage DRS its calculation. The misalignment of Storage DRS invocation and auto-tiering algorithms cycles makes it unpredictable when LUN segments may be moved, potentially colliding with the Storage DRS calculations and recommendations. Together with the transparency of auto tiering algorithms to Storage DRS and the non-existing communication between Storage DRS and Auto-tiering algorithms create the basis of the recommendation to disable I/O metric on datastore clusters backed by devices participating in an auto-tiering solution. Always verify these recommendations with your storage vendor. Additional information: Duncan wrote an excellent article about the Storage IO Control workload injector, which can be found here. More info on device modeling and load balancing can be found in the article impact of load balancing on datastore cluster configuration. Note: This article is describing Storage DRS behavior based on vSphere 5. ================================================================================ Title: (Storage) DRS (anti-) affinity rule types and HA interoperability URL: https://frankdenneman.ai/2012-02-06-sdrs-anti-affinity-rule-types-and-ha-interoperability/ Date: 2012-02-06 Lately I have received many questions about the interoperability between HA and affinity rules of DRS and Storage DRS. I’ve created a table listing the (anti-) affinity rules available in a vSphere 5.0 environment. Technology Type Affinity Anti-Affinity Respected by VMware HA DRS VM-VM Keep virtual machines together Separate virtual machines No VM-Host Should run on hosts in group Should not run on hosts in group No Must run on hosts in group Must not run on hosts in group Yes SDRS Intra-VM VMDK affinity VMDK anti-affinity N/A VM-VM Not available VM Anti-Affinity N/A As the table shows, HA will ignore most of the (anti-) affinity rules in its placement operations after a host failure except the “Virtual Machine to Host - Must rules”. Every type of rule is part of the DRS ecosystem and exists in the vCenter database only. A restart of a virtual machine performed by HA is a host-level operation and HA does not consult the vCenter database before powering-on a virtual machine. Virtual machine compatibility list The reason why HA respect the “must-rules” is because of DRS’s interaction with the host-local “compatlist” file. This file contains a compatibility info matrix for every HA protected virtual machine and lists all the hosts with which the virtual machine is compatible. This means that HA will only restart a virtual machine on hosts listed in the compatlist file. DRS Virtual machine to host rule A “virtual machine to hosts” rule requires the creation of a Host DRS Group, this cluster host group is usually a subset of hosts that are member of the HA and DRS cluster. Because of the intended use-case for must-rules, such as honoring ISV licensing models, the cluster host group associated with a must-rule is directly pushed down in the compatlist. Note Please be aware that the compatibility list file is used by all types of power-on operations and load-balancing operations. When a virtual machine is powered-on, whether manual (admin) or by HA, the compatibility list is checked. When DRS performs a load-balancing operation or maintenance mode operation, it checks the compatibility list. This means that no type of operation can override must- type affinity rules. For more information about when to use must and should rules, please read this article: Should or Must VM-Host affinity rules. Contraint violations After HA powers-on a virtual machine, it might violate any VM-VM or VM-host should (anti-) affinity rule. DRS will correct this constraint violation in the first following invocation and restore “peace” to the cluster. Storage DRS (anti-) affinity rules When HA restarts a virtual machine, it will not move the virtual machine files. Therefore creation of Storage DRS (anti-) affinity rules do not affect virtual machine placement after a host failure. ================================================================================ Title: Retrospect of 2011 content due to Bloggers survey URL: https://frankdenneman.ai/2012-01-24-retrospect-of-2011-content-due-to-bloggers-survey/ Date: 2012-01-24 vSphere-land.com is running it’s annual Top 25 virtualization blog survey again and I’m really interested to see who are picked this year. Like previous year, some great bloggers disappear while other new great ones emerge. One guy I want to mention by name is Chris Colotti, his blog is a great source of information about vCloud Director. If you haven’t visited his blog yet, go do that right away!. Last year I’ve been pretty busy writing, shaping, designing, wrestling with publishers in order to get our (@DuncanYB) book “vSphere 5 Clustering technical deepdive out to the public. This meant it cut down on research time, which resulted in a smaller number of blogs being released than previous years. So after seeing other people’s blog about their top 10, I was curious to see what I’ve done last year. The articles that are listed are the ones I’m proud of, spending a lot of time on researching them, but most of all, enjoyed the most writing them. Storage DRS initial placement and datastore cluster defragmentation Impact of Load Balancing on datastore cluster configuration Partially Connected datastore clusters Mem minfreepct sliding scale function Upgrading vmfs datastores and Storage DRS Multi NIC vMotion support in vSphere 5.0 Contention on lightly Utilized Hosts Restart vCenter results in DRS load balancing IP-HASH versus Load Based Teaming Setting correct percentage of cluster resources reserved AMD Magny Cours and ESX Please take 5 minutes of your time and vote for your favorite blogger. I hope they will announce the winner like they did last year. 90 minutes of nerve wrecking but oh-so-enjoyable videoshow! Cast your vote now! ================================================================================ Title: Storage DRS initial placement and datastore cluster defragmentation URL: https://frankdenneman.ai/2012-01-24-storage-drs-initial-placement-and-datastore-cluster-defragmentation/ Date: 2012-01-24 Recently an interesting question was raised about what happens if enough free space is available in the datastore cluster but not enough space is available per datastore during placement of a virtual machine. This scenario is often referred as a defragmented datastore cluster. The short answer is that if not enough space available on any given datastore, then Storage DRS starts to consider migrating existing virtual machines from the datastore to free up space. This article zooms in on the process of generating such an initial placement recommendation. Rules and boundaries within a datastore cluster Storage DRS will not violate the configured space utilization and IO latency threshold of the datastore cluster. This means that Storage DRS will place virtual machines that consume space up to the configured space utilization threshold, for example setting the space utilization threshold to 80% on a 1000GB datastore will allow Storage DRS to place virtual machines that consume space up to 800 GB. Be aware of this when monitoring free space available on the datastores in the cluster. When creating or moving a virtual machine in the datastore, the first thing to consider is the affinity rules. By default virtual machine files are kept together in the working directory of the virtual machine. If the virtual machine needs to be migrated, all the files inside the virtual machines’ working directory are moved. This article features the use of the default affinity rule, however if the default affinity rule is disabled, Storage DRS will move the working directory and virtual disks separately allowing Storage DRS to distribute the virtual disk files on a more granular level. Prerequisite migrations During initial placement, if no datastore with enough space is available in the datastore cluster, Storage DRS starts by searching alternative locations for the existing virtual machines in the datastores and attempts to place the virtual machines to other datastores one by one. As a result Storage DRS may generate sets of migration recommendations of existing virtual machines that allow placement of the new virtual machine. These migrations generated are called prerequisite migrations and combined with the placement operations is called a recommendation set. Depth of recursion Storage DRS uses a recursive algorithm for searching alternative placements combinations. To keep Storage DRS from trying an extremely high number of combinations of virtual machine migrations, the “depth of recursion” is limited to 2 steps. What defines a steps and what counts towards a step? A step can be best defined as a set of migrations out of a datastore in preparation of (or to make room for) another migration into that same datastore. This step can contain one vmdk, but can also contain multiple virtual machines with multiple virtual disks attached. In some cases, room must be created on that target datastore first by moving a virtual machine out to another datastore, which results in an extra step. The following diagram visualizes the process. Storage DRS has calculated that a new virtual machine can be placed in Datastore 1 if VM2 and VM3 are migrated to Datastore 2, however, placing these two virtual machines on datastore 2 will violate the space utilization, therefore room must be created. VM4 is moved out of Datastore2 as part of a step of creating space. This results in Step 1, moving out to Datastore 3, followed by Step 2, moving VM2 and VM3 to Datastore 2 to finally placing the new virtual machine on Datastore 1. Storage DRS stops its search if there are no 2-step moves to satisfy the storage requirement of an initial placement. An advanced setting can be set to change the number of steps used by the search. As always, it is strongly discouraged to change the defaults, as many hours of testing has been invested in researching the setting that offers good performance while minimizing the impact of the operation. If you have a strong case of changing the number of steps, set the advanced configuration option “MaxRecursionDepth”. The default value is 1 the maximum value is 5. Because the algorithm starts counting at 0, default value of 1 allows 2 steps. Goodness value Storage DRS will cycle through all the datastores in the datastore cluster and initiates a search for space on each datastore. A search generates a set of prerequisites migration if it can provide space that allows the virtual machine placement within the depth of recursion. Storage DRS evaluates the generated sets and award each set a goodness value. The set with the least amount of cost (i.e. migrations) is the preferred migration recommendation and shown at the top of the list. Let’s explore this a bit more by using a scenario with 3 datastores. Scenario The datastore cluster contains 3 datastores; each datastore has a size of 1000GB and contains multiple virtual machines with various sizes. The space consumed on the datastores range from 550GB to 650GB, while the space utilization threshold is set to 80%. At this point the administrator creates a virtual machine that requests 350GB of space. Although the datastore cluster itself contains 1225GB of free space, Storage DRS will not go forward and place the virtual machine on any of the three datastores, because placing the virtual machine will violate the space utilization threshold of the datastores. Search process As each ESXi host provide information about the overall datastore utilization and the vmdk statistics, Storage DRS has a clear overview of the most up to date situation and will use these statistics as input for its search. In the first step it will simulate all the necessary migrations to fit VM10 in Datastore 1. The prerequisite migration process with least number of migrations to fit the virtual machine on to Datastore 1 looks as follows: Step 1: VM3 from Datastore 1 to Datastore 2 Step 1: VM4 from Datastore 1 to Datastore 3 Place new virtual machine on Datastore 1 Although VM3 and VM4 are each moved out to a different datastore, both migrations are counted as a one step prerequisite migration as both virtual machines are migrated OUT of Datastore 1. Next Storage DRS will evaluate Datastore 2. Due to the size of VM5, Storage DRS is unable to migrate VM5 out of Datastore 2 because it will immediately violate the utilization threshold of the selected destination datastore. One of the coolest parts of the algorithm is that it considers inbound migrations as valid moves. In this scenario, migrating virtual machines into Datastore 2 would free up space on another datastore to provide enough free space to place VM5, which in turn free up space on Datastore 2 allowing Storage DRS to place VM10 onto Datastore2. The prerequisite migration process with least number of migrations to fit the virtual machine on to datastore 2 looks as follows: Step 1: VM2 from Datastore 1 to Datastore 2 Step 1: VM3 from Datastore 1 to Datastore 3 Step 2: VM5 from Datastore 2 to Datastore 1 Place new virtual machine on Datastore 2 Datastore 3 generates a single prerequisite migration. By migrating VM8 from Datastore 3 to Datastore 2 it will free up enough space to allow placement of VM10. Selecting VM9 would not free up enough space and migrating VM7 generates more cost than migrating VM8. By default Storage DRS attempts to migrate the virtual machine or virtual machine disk which size is closest to the required space. The prerequisite migration process with least number of migrations to fit the virtual machine on to datastore 3 looks as follows: Step 1: VM8 from Datastore 3 to Datastore 2 Place new virtual machine on Datastore 3 After analyzing the cost and benefit of the three search results Storage DRS will assign the highest goodness factor to the migration set of Datastore3. Although each search result can provide enough free space after moves, recommendation set of Datastore 3 will result in the lowest number of moves and migrates the lowest amount of data. All three results will be shown; the recommended set will be placed at the top A example placement recommendation screen is displayed, note that you can only apply the complete recommendation set. Applying the recommendation results in triggering the prerequisite migrations before the initial placement of the virtual machine occurs. ================================================================================ Title: Storage DRS and Multi-extents datastores URL: https://frankdenneman.ai/2012-01-17-sdrs-and-multi-extents-datastores/ Date: 2012-01-17 Somebody asked me if VMFS3 multi-extents datastores are supported by Storage DRS. Although they are supported and fully operational in Storage DRS, one must ask if this construct of large datastores should be used in a datastore cluster. Resource aggregation and flexibility Storage DRS Datastore clusters offer flexibility in adding and removing datastores dynamically and allow the administrator to focus on macro management by reducing the number of entities to be managed. By using datastore cluster, micro management of single datastores is something from the past, such as the tedious task of virtual machine placement. The administrator no longer needs to find a datastore that provide adequate space, while still ensuring that placement of the virtual machine will not result in an I/O bottleneck. Let alone monitoring the current workload next to the ever-expanding workload; application lifecycles are changing drastically and virtual machine server sprawl is still one of the top concerns of the modern administrator. Keeping track and managing such an environment is very challenging. By allowing Storage DRS to manage (initial) placement of virtual machines, the administrator only needs to monitor overall available space and IO performance of the datastore cluster itself. If the cluster requires more space of more IO performance the administrator can dynamically add more datastores to the datastore cluster and allow Storage DRS to find an optimal distribution of the current workload. The option “Run Storage now” in the datastore cluster view allows the administrator to trigger a Storage DRS invocation immediately. Using Storage DRS and particularly space load-balancing can reduce the need of multi-extents as well. By allowing Storage DRS to monitor space utilization, the free space used as a safety buffer can be greatly reduced. Each ESXi host reports the virtual machine space utilization and the datastore utilization; Storage DRS will trigger an invocation if the configured space utilization is violated. A common practice is to assign a big chunk of space as safety buffer to avoid out of space situation of a datastore, which might lead to downtime of the active virtual machines. I’ve seen organization using requirements of 30% free space on datastores. By reducing slack space, a higher consolidation ratio can be achieved (if IO performance allows this), or a reduction in LUN sizes. Reducing LUN sizes can be used to provision additional datastores to the datastore cluster. More datastores benefits Storage DRS by offering more load balancing options, more datastores increase the number of queues, which benefits IO management at ESXi level and at SIOC at cluster level. Essentially this configuration is the complete opposite of VMFS extends. However if larger size datastores are necessary, vSphere 5 offers VMFS5. VMFS5 VMFS5 allows datastores up to 64 terabyte of contiguous space. ESXi 5.0 allows a VMDK size up to 2 terabyte of space, providing sufficient space for most virtual machines configurations. If the virtual machine requires more than 32 virtual machine disks of 2 terabyte it’s recommended to disable the default affinity rule (keep all disks together) and allow Storage DRS to distribute the virtual machine disk files across all datastores inside the datastore cluster. This granularity allows Storage DRS to find a suitable datastore for each virtual disk that aligns with the performance requirements of that specific virtual disk. ================================================================================ Title: vSphere 4.1 HA and DRS book for only $19.95 URL: https://frankdenneman.ai/2012-01-09-vsphere-4-1-ha-and-drs-book-for-only-19-95/ Date: 2012-01-09 We lowered the price of the vSphere 4.1 HA and DRS technical Deepdive book permanently. As of this week you can obtain one of the coolest books in the virtualization section at Amazon for only $19.95. 30 5-star reviews couldn’t be wrong. Here is just a random selection of two of those 5-star reviews: B. Riley: The term “deepdive” is regularly abused in the technology world these days. There’s nothing more disheartening than walking into a one hour session at a conference entitled deepdive, and finding out that it’s neither deep, nor a dive. It ends up being more like sitting in a couple inches of warm water in a plastic kiddie pool. When these guys say deepdive, they mean it. This book is packed with helpful information from the first, to the last page. Somehow, they even manage to read minds. They know what you’re thinking as a VMware administrator, and they’ll tell you the why, and the best practice. Lots of books have good overviews of HA and DRS, but none goes as deep as this. It’s very well-written, and highly recommended for anyone who is running, or thinking about running an HA/DRS environment. This book is, as Jeremy Clarkson would say, “absolutely brilliant”! Chris Dearden: Ever had a series of discombobulated thoughts and ideas that have suddenly clicked into place & the plans come into focus? That’s exactly what happened when I read Frank & Duncan’s book. Even though I have a fair few years experience with Enterprise virtualisation , my knowledge of what’s deeply under the covers of the availability options of vSphere was made up of blog posts I’d read , anecdotes from colleagues and a few slides from trainers. It was enough to get me by, but there was always that nagging feeling that I wasn’t fully in control of what was happening. After reading the book ( in a morning - for a tech book it’s one that you can work though in a short amount of time and still get value from ) I had a real epiphany / light bulb moment / matrix moment / and all of those concepts and ideas suddenly had a deeper meaning and the big picture was visible. For anyone who thinks they know about HA / DRS : read this and *really* know about it. Get your copy now at: vSphere 4.1 HA and DRS technical deepdive Amazon page ================================================================================ Title: Impact of load balancing on datastore cluster configuration URL: https://frankdenneman.ai/2012-01-02-impact-of-load-balancing-on-datastore-cluster-configuration/ Date: 2012-01-02 This article is a part of the series on architecture and design on datastore clusters. This article zooms in on why it’s recommended to use similar type disks in a datastore cluster. In-tier balancing solution SDRS can be considered as an “in-tier” balancing solution, suggesting that a datastore cluster should be populated with datastores that provide similar performance, continuity, capacity or service level. Although it’s not a technical requirement to have similar configured datastores, using heterogeneous configurations in a datastore cluster can lead to unexpected results. Understanding the SDRS’ main goal and the load balancing process can assist you in architecting your datastore cluster. SDRS load balancing goal The main focus of SDRS is to correct imbalance from both a space utilization and latency perspective on the datastore level. SDRS determines the imbalance level (space or latency) of the datastore cluster and migrates one or multiple virtual machine disk to solve the imbalance. In order to select an appropriate migration candidate (virtual machine) SDRS relies on device and workload modeling to understand the impact of a workload on the latency of the datastore, SDRS uses virtual machine statistics and datastore utilization to understand the impact of virtual machine placement on the space utilization of a datastore. Modeling Let’s take a closer look at modeling. SDRS captures device performance to create a performance model; by using the SIOC injector and a reference workload it understands and learns the performance of each device. This way SDRS gets a clear picture of the datastores inside the datastore cluster. Workload modeling is used by SDRS to understand and learn the virtual machine workloads inside the datastore cluster. The workload modeling process creates a workload metric of each virtual disk and analyzes the impact of the data points on latency. SDRS combines and correlates the outcome of device and workload modeling and space utilization into a unified recommendation. This means that when SDRS decides to migrate a specific VMDK, it considers the workload metric of the virtual disk and analyzes the impact of that specific workload on the latency of the destination datastore. If both IO metric and space utilization functions are enabled on the datastore cluster, SDRS combines the outcome of device modeling, workload modeling and space utilization and weights them regarding to violated threshold. Interesting enough, even when you disable IO load balancing, SDRS attempts to take overall IO statistics into account when finding a suitable datastore. Impact of load balancing construct on datastore cluster configuration Although SDRS analyzes devices and each virtual machines’ workload it’s is key to understand that SDRS’ main priority is to correct the threshold violation of datastore. Although it tries to find the best suitable datastore for a specific workload, modeling is still used as a metric to understand and achieve the goal of getting the best overall performance out of the datastore cluster. In other words, modeling is used for balancing the load on the datastores and not to respect specific wishes of a virtual machine disk. In one way you can argue that SDRS load balancing has somewhat of a socialistic nature. Benefit for the society (datastores inside a datastore cluster) outweighs the individual need (single virtual machine performance). Let’s look at an example to better understand this concept. Example scenario VM1 is running on a datastore1. SDRS determined that the normalized load* is 5ms latency. VM2 and VM3 are running on a datastore2. SDRS considers datastore2 to have a normalized load of 20ms latency, violating the default threshold of 15ms. Normalized load: SDRS aggregates the device modeling and workload modeling into a metric called normalized load. SDRS moves VM3 to datastore1; at this point the overall latency of the datastore2 is reduced 13ms. However due to moving VM3 to datastore1, the latency is increased from 5ms to 12ms. At this point the increase in latency will impact the workload of VM1, however the “society” benefits from the move because after the move no datastore is violating the latency threshold of the SDRS cluster anymore. In this scenario the overall IOPS will be higher, which aligns with the goal of SDRS utilizing overall capacity and performance. Note: As this subject is complex enough, I used a very simple example. In this scenario the latency “moved” with the VM. In real life this is not necessarily the fact, when a virtual machine is moved the latency will go up with the same amount at which the latency went down on the source. Load Balancing in a Heterogeneous configuration What if the datastore cluster contains a mix of datastores that are backed by different types of disks? For a moment, let’s focus on the performance impact of a heterogeneous configuration. As mentioned before, device and workload modeling helps SDRS to find the most suitable datastore for a specific workload, however when combining different types of disk, for example, SSD, FC and SATA, it is not uncommon to see the fastest datastore fill up first. If one of the smaller SSD’s run out of space, SDRS is required to solve the space utilization threshold violation and will migrate a workload from a faster datastore to a slower datastore, prioritizing space utilization over IO utilization. Although future invocations of the SDRS algorithm might solve the problem by moving VMDK’s around to find a more optimal balance, no priority or guarantees can be assigned to a specific virtual disk avoiding potential decrease in performance of a specific VMDK. Now at this point most of you wonder if VASA and storage profiles can be used in such a configuration to associate specific profiles to virtual machines and make these VMs compliant to specific datastores. SDRS does not incorporate storage profiles compliancy in the load balancing algorithms and unfortunately not every storage vendor offers VASA providers of their arrays. Some excellent articles about VASA and Profile driven storage can be found at [Yellow Bricks.com](http://Yellow Bricks.com) and blogs.vmware.com/vSphere/storage VASA: http://blogs.vmware.com/vsphere/2011/08/vsphere-50-storage-features-part-10-vasa-vsphere-storage-apis-storage-awareness.html Profile driven storage: [http://www.yellow-bricks.com/2011-07-13-vsphere-5-0-profile-driven-storage-what-is-it-good-for/](/ http://www.yellow-bricks.com/2011-07-13-vsphere-5-0-profile-driven-storage-what-is-it-good-for/ " http://www.yellow-bricks.com/2011-07-13-vsphere-5-0-profile-driven-storage-what-is-it-good-for/") To guarantee specific performance to virtual machines it is recommended to uses similar type disks to back the datastores of a datastore cluster. This configuration offers a stable and predictable service level to the virtual infrastructure. If multiple types of disks are available, it is recommended to split and create multiple datastore clusters each containing groups of identical types of disks. Previous articles in the SDRS short series Architecture and design of Datastore clusters: Part1: Architecture and design of datastore clusters Part2: Partially connected datastore clusters ================================================================================ Title: Cyber Monday deal! URL: https://frankdenneman.ai/2011-11-26-cyber-monday-deal/ Date: 2011-11-26 We are long time fascinated by the whole Black Friday and Cyber Monday craze in the USA. Unfortunately we do not celebrate Thanksgiving in the Netherlands and none of the shops are participating in something similar as Black Friday. This year we thought it was a great idea to participate in some form and what better than to offer our vSphere 5 Clustering Technical Deepdive e-book for a price you cannot resist. We just changed the price of the vSphere 5 Clustering Technical Deepdive to $ 4.99 and 3.99 for our European friends. Yes that is correct…. Less than 5 dollars for over 350 pages of deepdive material. What better way than recover from the madness of Black Friday and just sit back and relax reading this amazing piece of work? This is most definitely the deal of the year for all virtualization fanatics! Keep in mind that this is a limited offer, Tuesday the 29th the price will be back to “normal” again. US – ebook – $ 4.99 UK – ebook – £ 3.99 DE – ebook – € 3.99 FR – ebook – € 3.99 Pick it up, tell your friends / colleagues / family about it… Here are some snippets from Amazon reviews, but with 15 extremely positive reviews, all of them 5 out of 5, you know you can’t go wrong: “If you’re serious about VMware virtualization this book is a must have. Regardless of you responsibilities with a virtual infrastructure administrative, or from a architecture design stand point this book is for you. The level of knowledge and depth which Frank and Duncan cover in this book about the new clustering changes in vSphere 5 is priceless. The design tips and illustrations through the book are truly invaluable. There is no other book that gets into the core of all the different vSphere 5 cluster technologies like this one, ” “Whether you are longing to know about the transition from AAM to FDM, best practices for DRS and DPM, or are just curious to know what those acronyms are this is a great book! The technical detail, practical advice, and comparative analysis throughout make this book one of the most thorough yet concise technical books available.” “The book is clearly written, a special emphasis has been made on making it understandable even for professionals like me who use vSphere daily yet do not manage huge production environments. The book goes to great lengths to explain all possible scenarios and I found answers to all my questions. Not only sections cover HOW the technology works, but the authors go as far as explaining the way the algorithms are working, which will satisfy the curiosity of everyone.” “The complete explanations provide the reader all of the information needed to make informed decisions about their environment with excellent diagrams to provide strong visual reinforcements.” Please remember that we are offering the book for the price listed above, depending on your location Amazon might charge an additional cost! ================================================================================ Title: New job role URL: https://frankdenneman.ai/2011-11-21-new-job-role/ Date: 2011-11-21 The last two years I enjoyed working as an architect within the PSO organization of VMware, designing and reviewing the most interesting virtual infrastructures in Europe. However today I signed my new contract, accepting a position within the Technical Marketing team. Starting December I will focus on resource management and disaster avoidance technologies. My new role allows me to collaborate with the Product managers and the R&D organization on products such as DRS, Storage DRS, vMotion, Storage vMotion and FT. My main tasks will be developing best practices, white-papers, documentation and technical presentations, educating field organizations and of course the customers. Although I enjoyed working within the PSO organization, I can’t wait to get started. Thanks to all the people who made my move possible and offering me such an opportunity! ================================================================================ Title: FDM in mixed ESX and vSphere clusters URL: https://frankdenneman.ai/2011-10-17-fdm-in-mixed-esx-and-vsphere-clusters/ Date: 2011-10-17 Last couple of weeks I’ve been receiving questions about vSphere HA FDM agent in a mixed cluster. When upgrading vCenter to 5.0, each HA cluster will be upgraded to the FDM agent. A new FDM agent will be pushed to each ESX server. The new HA version supports ESX(i) 3.5 through ESXi 5.0 hosts. Mixed clusters will be supported so not all hosts have to be upgraded immediately to take advantage of the new features of FDM. Although mixed environments are supported we do recommend keeping the time you run difference versions in a cluster to a minimum. The FDM agent will be pushed to each hosts, even if the cluster contains identically configured hosts, for example a cluster containing only vSphere 4.1 update 1 will still be upgraded to the new HA version. The only time vCenter will not push the new FDM agent to a host if the host in question is a 3.5 host without the required patch. When using clusters containing 3.5 hosts, it is recommended to upgrade the ESX host to ESX350-201012401-SG PATCH (ESX 3.5) or ESXe350-201012401-I-BG PATCH (ESXi) patch first before upgrading vCenter to vCenter 5.0. If you still get the following error message: Host ’’ is of type ( ) with build , it does not support vSphere HA clustering features and cannot be part of vSphere HA clusters. Visit the VMware knowledgebase article: 2001833. ================================================================================ Title: Partially connected datastore clusters URL: https://frankdenneman.ai/2011-10-07-partially-connected-datastore-clusters/ Date: 2011-10-07 The first article in the series about architecture and design decisions series focuses on the connectivity of the datastores within the datastore cluster. Connectivity between ESXi hosts and datastores in the datastore cluster affects initial placement and load balancing decisions made by DRS and Storage DRS. Although connecting a datastore to all ESXi hosts inside a cluster is a common practice, we still come across partially connected datastores in virtual environments. What is the impact of a partially connected datastore, member of a datastore cluster, connected to a DRS cluster? What interoperability problems can you expect and what is the impact of this design on DRS load balancing operations and SDRS load balancing operations? Let’s start with the basic terminology. Fully connected datastore clusters A fully connected datastore cluster is when the storage is attached to all ESX servers in a cluster. This is a recommendation, but it is not enforced. Partially connected datastore clusters If a datastore is connected to a subset of ESXi hosts inside the DRS cluster, the datastore cluster is treated as a partially connected datastore cluster. Now what happens if the DRS cluster is connected to partially connected datastores? It’s important to understand that the goal of both DRS and SDRS is resource availability, key to offering resource availability is to provide or have as much as mobility as possible. SDRS will not generate any migration recommendations that will reduce the compatibility of a virtual machine regarding datastore connections. Virtual machine to host compatibility are captured in compatibility lists. Compatibility list Inside the cluster a vm-host compatibility list is generated for each virtual machine. The compatibility list determines which ESXi host in the cluster have network and storage configurations that allow the virtual machine to successfully come online. Membership of a Mandatory VM to host affinity rules are also listed in the compatibility list.If the network portgroup or datastore is not available on the host, or the host is not listed in the host group of the mandatory affinity rule, the ESXi server is deemed incompatible to host that virtual machine. As mentioned, both DRS and SDRS focus on resource availability and resource outage avoidance, therefore SDRS prefers a datastore that is connected to all hosts rather than selecting a datastore that is partially connected. Connecting datastores to a subset of hosts reduce the compatibility list impacting the mobility of the virtual machine reducing the efficiency of DRS and SDRS. Finding a suitable location or the ability to load balance becomes more challenging when the cluster and datastore cluster are partially connected. During initial placement a selection of a datastore may impact the mobility of the virtual machine amongst the hosts, while selecting a host impacts the mobility of a virtual machines amongst the datastores in the datastore cluster. [caption id=“attachment_1773” align=“aligncenter” width=“410” caption=“VM mobility in partially connected datastore clusters”][/caption] Let’s explore this impact a little bit further. During the process of migration recommendations, DRS selects a host for a virtual machine that can provide enough resources to satisfy the virtual machines resource entitlement, while lowering the imbalance of the cluster. DRS might come across a low utilized host; other hosts inside the cluster are highly utilized. Unfortunately the lightly utilized host is not connected to the datastore containing the virtual machine files (it might even be lowly utilized due to the poor connection state) and therefore DRS will not consider the host due to the incompatibility. While from a DRS resource load balancing perspective this host might be very attractive option to solve resource imbalance. Also keep in mind the impact of this behavior on VM-Host affinity rules, DRS will not migrate the virtual machine to the partially connected host inside the host group. Similar happens with SDRS load balancing. Partially connected datastores are not recommended when fully connected datastores are available that do not violate the space SDRS threshold. You might wonder why the space SDRS threshold is explicitly mentioned and not the IO load balanced but that’s because IO load balancing is disabled when a partially connected datastore is detected in the datastore cluster. IO load balancing It is important to understand the impact a single partially connected datastore has on the service level of an entire datastore cluster. As SDRS detects a partially connected datastore it will disable the IO load balancing on the entire datastore cluster. Not only on that single partially connected datastore, but the entire cluster. Effectively degrading a complete feature set of your virtual infrastructure. Temporary partially connectivity – a real threat? The connectivity status is important when the SDRS interval expires; during the migration recommendation calculation is checks the connectivity. A temporary all-paths-down status or a rezoning procedure might not have effect on SDRS load-balancing behavior, but what if good old murphy decides to give you a visit during the invocation period? Keep this behavior in mind when scheduling maintenance on the storage platform. Warning messages SDRS generates a warning and displays it at the SDRS faults tab in the datastores and datastore cluster view Benefits of partially connected We cannot identify any direct benefit of partially connecting a datastore of a cluster. Partially connected datastores impact initial placement, disable IO load-balancing and will affect DRS load balancing as well as SDRS space balancing. Therefore a basic design decision would be connect all datastores to all host in the cluster connected to the datastore cluster. If anyone has got a good reason for not connecting a datastore to all the hosts, please leave a comment. ================================================================================ Title: Architecture and design of Datastore clusters URL: https://frankdenneman.ai/2011-09-16-architecture-and-design-of-datastore-clusters/ Date: 2011-09-16 Storage DRS extends the DRS feature set to the storage space. The primary element used by SDRS is a datastore cluster. Introducing the concept of datastore clusters can affect or shift the paradigm of storage management in virtual infrastructures. This article is the start of a short series of articles focusing on the design considerations of datastore clusters Datastore cluster concept Let’s start with looking at the concept datastore cluster. Datastore clusters can be regarded as the equivalent of DRS clusters. A datastore cluster is the storage equivalent of an vCenter (DRS) cluster whereas a datastore is the equivalent of a ESXi host. As datastore clusters pool storage resources into one single logical pool it becomes a management object. This storage pooling allows the administrator to manage many individual datastores as one element, and depending on the enabled SDRS features, providing optimized usage of storage capacity and IO performance capability off all member datastores. SDRS settings are configured at datastore cluster level and are applied to each member datastore inside the datastore cluster. When SDRS is enabled the datastore cluster it becomes the storage load-balancing domain, requiring administrators and architects to treat the datastore cluster as a single entity for decision making instead of individual datastores. Datastore Clusters architecture and design Although datastore clusters offers an abstraction layer, one must keep in mind the relationship between existing objects like hosts, clusters, virtual machine and virtual disks. This new abstraction layer might even disrupt existing (organizational) processes and policies. Introducing datastore clusters can have impact on various design decisions such as VMFS datastores sizing, configuration of the datastore clusters, the variety in datastore clusters and the number of datastore clusters in the virtual infrastructure. In this series I will address these considerations more in depth. Stay tuned for the first part; the impact of connectivity of datastores in a datastore cluster. More articles in the architecting and designing datastore clusters series: Part2: Partially connected datastore clusters. Part3: Impact of load balancing on datastore cluster configuration. Part4: Storage DRS and Multi-extents datastores. Part5: Connecting multiple DRS clusters to a single Storage DRS datastore cluster. ================================================================================ Title: SDRS out of space avoidance URL: https://frankdenneman.ai/2011-09-13-sdrs-out-of-space-avoidance/ Date: 2011-09-13 During VMworld I noticed a lot of focus of the attendees was on the IO load balancing features of Storage DRS (SDRS), however SDRS is more than only IO load balancing. Both space load balancing feature and initial placement are just as incredible, powerful and as useful as IO load balancing. Actually the term space load balancing isn’t really doing the algorithm any justice as it sounds it makes “unnecessary” moves around space usage, whereas “out of space avoidance” suits more the nature of this SDRS algorithm, because it will make crucial recommendations that in my opinion bring a lot of value. Initial placement and IO load balancing will be featured in future articles, but in this post I would like to focus on the out of space avoidance feature of SDRS. When SDRS is enabled, it will automatically make recommendations based on space utilization and IO load. IO load balancing can be activated or deactivated when enabling or disabling the option “Enable I/O metric for SDRS recommendations”. SDRS does not offer the option to enable or disable out of space avoidance; out of space avoidance is enabled by default and can only be disabled by disabling SDRS in its entirety. Thresholds By default, SDRS monitors the datastore space utilization and generates migration recommendations if the datastore utilization is exceeding the “space utilization ratio threshold”. The utilized space threshold determines the maximum acceptable space load of the VMFS datastore. And this is a part of the SDRS settings of the datastore cluster. This threshold is set by default to 80% and can be set to any value between 50 and 100 percent. [caption id=“attachment_1715” align=“aligncenter” width=“619” caption=“Space utilization ratio threshold”][/caption] Be aware that this threshold applies to each datastore that is a member of the datastore cluster; if you want to have similar absolute space headroom, it is recommended to add similar sized datastores to the datastore cluster. If the threshold is reached, SDRS will not migrate a random virtual machine to a random datastore, it needs to adhere to certain rules. Besides running a cost-benefit analysis on the registered virtual machines in the datastore, it also takes the “space utilization ratio difference threshold” into account. [caption id=“attachment_1718” align=“aligncenter” width=“619” caption=“Space utilization ratio difference threshold”][/caption] The utilization difference setting allows SDRS to determine which datastores should be considered as destination for virtual machines migrations. The space utilization ratio difference threshold indicates the required difference of utilization ratio between the destination and source datastores. The difference threshold is an advanced option of the SDRS runtime rules and is set to a default value of 5%. Consequently SDRS will not move any virtual machine disk from an 83% utilized datastore to a 78% utilized datastore. The reason why SDRS uses this setting is to avoid recommending migrations of marginal value. SDRS also uses space growth rate to avoid risky migrations. A migration is considered risky if it has to be undone in the near future. SDRS defines near future as a time window that is longer than the lead-time of a storage vMotion and defaults to 30 hours. This option cannot be changed in any supported way. Hence, SDRS will avoid moving any virtual machine disk to a datastore that is expected, based on growth rate, to exceed the utilization threshold within the next 30 hours. Space Utilization How does SDRS determine if VMFS datastore has exceeded the threshold? It does this by comparing the “Space utilization” against the utilized space threshold. Space utilization is determined by dividing the total consumed space on the datastore by the datastore capacity. Space utilization = total consumed space on the datastore / datastore capacity To determine the space utilization of the datastore, SDRS requires the “per-VMDK” usage statistic and the VMFS datastore usage statistic. The per-VMDK statistic provides SDRS data about the allocated space in the VMDK, while the VMFS datastore statistic provides information about the datastore utilization. [caption id=“attachment_1719” align=“aligncenter” width=“297” caption=“Out of Space avoidance input”][/caption] Now this is one of the more cool parts of the algorithm, as mentioned, SDRS takes into account the allocated amount of disk space instead of the provisioned disk space using thin disk. SDRS receives the per-VMDK statistics and space utilization per datastore on an ongoing basis. If the utilization exceeds the threshold the SDRS algorithm is triggered immediately and does not have to wait to complete his invocation period. By receiving space utilization information frequently, SDRS is able to understand and trend-map the data-growth within the VMDK. The growth rate is estimated using historical usage samples, with recent samples weighing more than older historical usage samples. By including this information in the cost-benefit risk analysis SDRS attempts to avoid migrating virtual machines with data-growth rates that will likely cause the destination datastore to exceed the threshold in the near future. By including the estimated growth rate, SDRS is equipped with an outage avoiding strategy. This avoiding outage intelligence helps most organization to adopt thin provisioned disks located in the virtualization stack. You still need to think about the over-subscription level, but SDRS will helps to control the environment and avoid outage caused by out-of-space situations as much as possible. Migration candidate selection Now how does SDRS know which virtual machines to move? As mentioned before SDRS uses a cost-benefit risk analysis. Moving virtual machines is expensive, both on CPU and memory subsystems as well as the IO subsystems. Therefore SDRS aims to generate recommendations that have the lowest impact on the environment while delivering improvements and solve any violation. SDRS considers the size of the VMDK (allocated space) and the activity of the IO workload to calculate the cost aspect of the CB analysis. When a datastore exceeds the space utilization threshold, SDRS will try to move the number of megabytes out of the datastore to correct the space utilization violation. In other words, SDRS attempts to select a virtual machine that is closest in size required to bring the space utilization of the datastore to the space utilization ratio threshold. Caveats Before enabling SDRS on arrays configured to use deduplication or replication technologies you might want to check your array vendor on their recommendation of combining SDRS with their technology. Duncan has written an excellent article about the interop between SDRS and various technologies. Please read if you already haven’t: http://www.yellow-bricks.com/2011-07-15-storage-drs-interoperability/ Key Takeaway SDRS out of space avoidance alone is reason enough to use SDRS. Having an automated way of distributing virtual machines across your datastore landscape can result in a more efficient usage of available storage space, while reducing the management effort. Understanding the outage avoidance measures inside the algorithm might help to consider using thin-provisioned VMDK format to decrease the footprint of the virtual machines, speed up SDRS migrations and possibly increase the VM density per datastore. ================================================================================ Title: Mem.MinFreePct sliding scale function URL: https://frankdenneman.ai/2011-07-26-mem-minfreepct-sliding-scale-function/ Date: 2011-07-26 One of the cool “under the hood” improvements vSphere 5 offers is the sliding scale function of the Mem.MinFreePct. Before diving into the sliding scale function, let’s take a look at the Mem.MinFreePct function itself. MinFreePct determines the amount of memory the VMkernel should keep free. This threshold is subdivided in various memory thresholds, i.e. High, Soft, Hard and Low and is introduced to prevent performance and correctness issues. The threshold for the low state is required for correctness. In other words, it protects the VMkernel layer from PSOD’s resulting from memory starvation. The soft and hard thresholds are about virtual machine performance and memory starvation prevention. The VMkernel will trigger more drastic memory reclamation techniques when it approaches the Low state. If the amount of free memory is just a bit less than the Min.FreePct threshold, the VMkernel applies ballooning to reclaim memory. The ballooning memory reclamation technique introduces the least amount of performance impact on the virtual machine by working together with the Guest operating system inside the virtual machine, however there is some latency involved with ballooning. Memory compressing helps to avoid hitting the low state without impacting virtual machine performance, but if memory demand is higher than the VMkernels’ ability to reclaim, drastic measures are taken to avoid memory exhaustion and that is swapping. However swapping will introduce VM performance degradations and for this reason this reclamation technique is used when desperate moments require drastic measurements. For more information about reclamation techniques I recommend reading the “disable ballooning” article. vSphere 4.1 allowed the user to change the default MinFreePct value of 6% to a different value and introduced a dynamic threshold of the Soft, Hard and Low state to set appropriate thresholds and prevent virtual machine performance issues while protecting VMkernel correctness. By default vSphere 4.1 thresholds was set to the following values: Free memory state Threshold Reclamation mechanism High 6% None Soft 64% of MinFreePct Balloon, compress Hard 32% of MinFreePct Balloon, compress, swap Low 16% of MinFreePct Swap Using a default MinFreePct value of 6% can be inefficient in times where 256GB or 512GB systems are becoming more and more mainstream. A 6% threshold on a 512GB will result in 30GB idling most of the time. However not all customers use large systems and prefer to scale out than to scale up. In this scenario, a 6% MinFreePCT might be suitable. To have best of both worlds, ESXi 5 uses a sliding scale for determining its MinFreePct threshold. Free memory state threshold Range 6% 0-4GB 4% 4-12GB 2% 12-28GB 1% Remaining memory Let’s use an example to explore the savings of the sliding scale technique. On a server configured with 96GB RAM, the MinFreePct threshold will be set at 1597.6MB, opposed to 5898.24MB if 6% was used for the complete range 96GB. Free memory state Threshold Range Result High 6% 0-4GB 245.96MB 4% 4-12GB 327.68MB 2% 12-28GB 327.68MB 1% Remaining memory 696.32MB Total High Threshold 1597.60MB Due to the sliding scale, the MinFreePct threshold will be set at 1597.96MB, resulting in the following Soft, Hard and low threshold: Free memory state Threshold Reclamation mechanism Threshold in MB Soft 64% of MinFreePct Balloon 1022.69 Hard 32% of MinFreePct Balloon, compress 511.23 Low 16% of MinFreePct Balloon, compress, swap 255.62 Although this optimization isn’t as sexy as Storage DRS or one of the other new features introduced by vSphere5 it is a feature of vSphere 5 that helps you drive your environments to higher consolidation ratios. ================================================================================ Title: Upgrading VMFS datastores and SDRS URL: https://frankdenneman.ai/2011-07-22-upgrading-vmfs-datastores-and-sdrs/ Date: 2011-07-22 Among many new cool features introduced by vSphere 5 is the new VMFS file system for block storage. Although vSphere 5 can use VMFS-3, VMFS-5 is the native VMFS level of vSphere 5 and it is recommended to migrate to the new VMFS level as soon as possible. Jason Boche wrote about the difference between VMFS-3 and VMFS-5. vSphere 5 offers a pain free upgrade path from VMFS-3 to VMFS-5. The upgrade is an online and non-disruptive operation which allows the resident virtual machines to continue to run on the datastore. But upgraded VMFS datastores may have impact on SDRS operations, specifically virtual machine migrations. When upgrading a VMFS datastore from VMFS-3 to VMFS-5, the current VMFS-3 block size will be maintained and this block size may be larger than the VMFS-5 block size as VMFS-5 uses unified 1MB block size. For more information about the difference between native VMFS-5 datatstores and upgraded VMFS-5 datastore please read: Cormac’s article about the new storage features Although the upgraded VMFS file system leaves the block size unmodified, it removes the maximum file size related to a specific block size, so why exactly would you care about having a non-unified block size in your SDRS datastore cluster? In essence, mixing different block sizes in a datastore cluster may lead to a loss in efficiency and an increase in the lead time of a storage vMotion process. As you may remember, Duncan wrote an excellent post about the impact of different block sizes and the selection of datamovers. To make an excerpt, vSphere 5 offers three datamovers: • fsdm • fs3dm • fs3dm – hardware offload The following diagram depicts the datamover placement in the stack. Basically, the longer path the IO has to travel to be handled by a datamover, the slower the process. In the most optimal scenario, you want to leverage the VAAI capabilities of your storage array. vSphere 5 is able to leverage the capabilities of the array allowing hardware offload of the IO copy. Most IOs will remain within the storage controller and do not travel up the fabric to the ESXi host. But unfortunately not every array is VAAI capable. If the attached array is not VAAI capable or enabled, vSphere will leverage the FS3DM datamover. FS3DM was introduced in vSphere 4.1 and contained some substantial optimizations so that data does not travel through all stacks. However if a different block size is used, ESXi reverts to FSDM, commonly known as the legacy datamover. To illustrate the difference in Storage vMotion lead time, read the following article (once again) by Duncan: Storage vMotion performance difference. This article contains the result of a test in which a virtual machine was migrated between two different types of disks configured with deviating block sizes and at a different stage a similar block size. To emphasize; the results illustrates the lead time of the FS3DM datamover and the FSDM datamover. The results below are copied from the Yellow-Bricks.com article: From(MB) To Duration in Minutes FC datastore 1MB blocksize FATA datastore 4MB blocksize 08:01 FATA datastore 4MB blocksize FC datastore 1MB blocksize 12:49 FC datastore 4MB blocksize FATA datastore 4MB blocksize 02:36 FATA datastore 4MB blocksize FC datastore 4MB blocksize 02:24 As the results in the table show, using a different blocksize lead to an increase in Storage vMotion lead time. Using different block sizes in your SDRS datastore cluster will decrease the efficiency of Storage DRS. Therefore it’s recommended designing for performance and efficiency when planning to migrate to a storage DRS cluster. Plan ahead and invest some time the migration path. If the VMFS-3 datastore is formatted with a larger blocksize than 1 MB, it may be better to empty the VMFS datastore and reformat the LUN with a fresh coat of VMFS-5 file system. The effort and time put into the migration will have a positive effect on the performance of the daily operations of Storage DRS. ================================================================================ Title: Multi-NIC vMotion support in vSphere 5.0 URL: https://frankdenneman.ai/2011-07-18-multi-nic-vmotion-support-in-vsphere-5-0/ Date: 2011-07-18 There are some fundamental changes to vMotion scalability and performance in vSphere 5.0 one is the multi-nic support. One of the most visible changes is multi-NIC vMotion capabilities. In vSphere 5.0 vMotion is now capable of using multiple NICs concurrently to decrease lead time of a vMotion operation. With multi-NIC support even a single vMotion can leverage all of the configured vMotion NICs, contrary to previous ESX releases where only a single NIC was used. Allocating more bandwidth to the vMotion process will result in faster migration times, which in turn affects the DRS decision model. DRS evaluates the cluster and recommends migrations based on demand and cluster balance state. This process is repeated each invocation period. To minimize CPU and memory overhead, DRS limits the number of migration recommendations per DRS invocation period. Ultimately, there is no advantage recommending more migrations that can be completed within a single invocation period. On top of that, the demand could change after an invocation period that would render the previous recommendations obsolete. vCenter calculates the limit per host based on the average time per migration, the number of simultaneous vMotions and the length of the DRS invocation period (PollPeriodSec). PollPeriodSec: By default, PollPeriodSec – the length of a DRS invocation period – is 300 seconds, but can be set to any value between 60 and 3600 seconds. Shortening the interval will likely increase the overhead on vCenter due to additional cluster balance computations. This also reduces the number of allowed vMotions due to a smaller time window, resulting in longer periods of cluster imbalance. Increasing the PollPeriodSec value decreases the frequency of cluster balance computations on vCenter and allows more vMotion operations per cycle. Unfortunately, this may also leave the cluster in a longer state of cluster imbalance due to the prolonged evaluation cycle. Estimated total migration time: DRS considers the average migration time observed from previous migrations. The average migration time depends on many variables, such as source and destination host load, active memory in the virtual machine, link speed, available bandwidth and latency of the physical network used by the vMotion process. Simultaneous vMotions: Similar to vSphere 4.1, vSphere 5 allows you to perform 8 concurrent vMotions on a single host with 10GbE capabilities. For 1GbE, the limit is 4 concurrent vMotions. Design considerations When designing a virtual infrastructure leveraging converged networking or Quality of Service to impose bandwidth limits, please remember that vCenter determine the vMotion limits based on the vMotion uplink physical NIC reported link speed. In other words, if the physical NIC reports at least 10GbE, link speed, vCenter allows 8 vMotions, but if the physical NIC reports less than 10GBe, but at least 1 GbE, vCenter allows a maximum of 4 concurrent vMotions on that host. For example; HP Flex technology sets a hard limit on the flexnics, resulting in the reported link speed equal or less to the configured bandwidth on Flex virtual connect level. I’ve come across many Flex environments configured with more than 1GB bandwidth, ranging between 2GB to 8GB. Although they will offer more bandwidth per vMotion process, it will not offer an increase in the amount of concurrent vMotions. Therefore, when designing a DRS cluster, take the possibilities of vMotion into account and how vCenter determines the concurrent number of vMotion operations. By providing enough bandwidth, the cluster can reach a balanced state more quickly, resulting in better resource allocation (performance) for the virtual machines. **disclaimer: this article contains out-takes of our book: vSphere 5 Clustering Technical Deepdive** ================================================================================ Title: Black and white edition Clustering deepdive available URL: https://frankdenneman.ai/2011-07-16-black-and-white-edition-clustering-deepdive-available/ Date: 2011-07-16 It looks like Amazon is getting its game together. As of now the Black and White paperback edition is available at Amazon.com. Get it here: VMware vSphere Clustering Technical Deepdive We are still waiting for the Full color edition to become available, but hey it’s a start :) ================================================================================ Title: VMware vSphere 5 Clustering Technical Deepdive URL: https://frankdenneman.ai/2011-07-16-vmware-vsphere-5-clustering-technical-deepdive/ Date: 2011-07-16 As of today the paperback versions of the VMware vSphere 5 Clustering Technical Deepdive is available at Amazon. We took the feedback into account when creating this book and are offering a Full Color version and a Black and White edition. Initially we planned to release an Ebook and a Full Color version only, but due to the high production cost associated with Full color publishing, we decided to add a Black and White edition to the line-up as well. At this stage we do not have plans to produce any other formats. As this is self-publishing release we developed, edited and created everything from scratch. Writing and publishing a book based on new technology has serious impact on one’s life, reducing every social contact to a minimum even family life. As of this, our focus is not on releasing additional formats such as ibooks or Nook at this moment. Maybe at a later stage but VMworld is already knocking on our doors, so little time is left to spend some time with our families. When producing the book, the page count rapidly exceeded 400 pages using the 4.1 HA and DRS layout. As many readers told us they loved the compactness of the book therefor our goal was to keep the page count increase to a minimum. Adjusting the inner margins of the book was the way to increase the amount of space available for the content. A tip for all who want to start publishing, start with getting accustomed to publisher jargon early in the game, this will save you many failed proof prints! We believe we got the right balance between white-space and content in the book, reducing the amount of pages while still offering the best reading experience. Nevertheless the number of pages grew from 219 to 348. While writing the book, we received a lot of help and although Duncan listed all the people in his initial blog, I want to use take a moment to thank them again. First of all I want to thank my co-author Duncan for his hard work creating content, but also spending countless hours on communication with engineering and management. Anne Holler - DRS and SDRS engineer – Anne really went out of her way to help us understand the products. I frequently received long and elaborate replies regardless of time and day. Thanks Anne! Next up is Doug – its number Frank not amounts! – Baer. I think most of the time Doug’s comments equaled the amount of content inside the documents. Your commitment to improve the book impressed us very much. Gabriel Tarasuk-Levin for helping me understand the intricacies of vMotion. A special thanks goes out to our technical reviewers and editors: Keith Farkas and Elisha Ziskind (HA Engineering), Irfan Ahmad and Rajesekar Shanmugam (DRS and SDRS Engineering), Puneet Zaroo (VMkernel scheduling), Ali Mashtizadeh and Doug Fawley and Divya Ranganathan (EVC Engineering). Thanks for keeping us honest and contributing to this book. I want to thank VMware management team for supporting us on this project. Doug “VEEAM” Hazelman thanks for writing the foreword! Availability This weekend Amazon made both the black and white edition and the full color edition available. Amazon list the black and white edition as: VMware vSphere 5 Clustering Technical Deepdive (Volume 2) [Paperback], whereas the full color edition is listed with Full Color in its subtitle. Or select the following links to go the desired product page: Black and white paperback $29.95 Full Color paperback $49.95 For people interested in the ebook: VMware vSphere 5 Clustering Technical Deepdive (price might vary based on location) If you prefer a European distributor, ComputerCollectief has both books available: Black and White edition: http://www.comcol.nl/detail/74615.htm Full Color edition: http://www.comcol.nl/detail/74616.htm Pick it up, leave a comment and of course feel free to make those great mugshots again and ping them over via Facebook or our Twitter accounts! For those looking to buy in bulk (> 20) contact clusteringdeepdive@gmail.com. ================================================================================ Title: Amazon indexing problems URL: https://frankdenneman.ai/2011-07-15-amazon-indexing-problems/ Date: 2011-07-15 The entire week Amazon hasn’t been able to index both vSphere 5 Clustering technical deepdive editions properly. We are working with Createspace to fix these problems. In the meantime, both full color and black and white editions can be ordered at Createspace: Black and White: https://www.createspace.com/3641804 $29.95 Full Color: https://www.createspace.com/3586911 $49.95 An update follows as soon as Amazon list the paperbacks. ================================================================================ Title: A look inside the upcoming vSphere clustering book URL: https://frankdenneman.ai/2011-06-10-a-look-inside-the-upcoming-vsphere-clustering-book/ Date: 2011-06-10 I just received the second proof copy of the new book and I’m really (really) stoked about the book. The full color print is awesome and truly adds that special feeling to the book. I’m so excited how the diagrams turned out, that I must share some pictures of the inside of the book. (For the CSI-fan’s, yes I have blurred some text fields as they contain NDA material) Besides the diagrams, the whole interior is redesigned. The spread is reviewed, inner and outer margins are adjusted and we even taken the gutter space into account, providing a better and nicer reading experience. We decided that we are going to offer the book in Full color format and a (full-color) ebook. After seeing the full-color version, we believe publishing a black and white version will not do the content any justice.And due to time constraints we cannot invest time in offering a black and white version of the book. We are still finalizing the book but we hope to provide the possibility of pre-ordering near the publish date. Stay tuned for more information! ================================================================================ Title: Keep alive ping - some updates URL: https://frankdenneman.ai/2011-06-08-keep-alive-ping-some-updates/ Date: 2011-06-08 Lately the amount of content of frankdenneman.nl is getting a bit stale, so a small update from my side to show what I’ve been up to and that this blog is still alive. vSphere x Clustering Deepdive Duncan and I are working (feverishly) on a new book for a while. Calling it an update of the 4.1 HA and DRS technical deepdive won’t do the book any justice as the old chapters are complete rewritten. The HA section will cover the new HA stack of the upcoming vSphere version, while the focus of the DRS section leans more towards resource management. In addition this book covers Storage DRS and introduces a cool new feature called “supporting deep dives” These additional deep dives expand on supporting technologies of the main cluster feature set, it will contain in-depth information of technologies such as vMotion, Storage vMotion, EVC and certain new technologies introduced in the upcoming version of vSphere. I bet you guys will love this stuff. Speaking at VMworld 2011 Last Friday I received very good news. In my previous post I asked everyone to consider voting on the sessions I participate in and the good news is that both sessions were accepted. Session VSP1682 - vSphere 5 clustering Q&A, co-presenting with Duncan is accepted for both VMworld Las Vegas as well as VMworld Europe in Copenhagen. The second session, VSP1425 - Ask the Experts vBloggers, I’m proud to join the incredible line-up of Chad Sackac, Duncan Epping, Rick Scherer and Scott Lowe to help answer questions on virtualization design. After 5 years of visit VMworld as an attendee, I finally get to experience what it’s like to be at the other side of the room. ================================================================================ Title: VMworld Public Voting URL: https://frankdenneman.ai/2011-05-16-vmworld-public-voting/ Date: 2011-05-16 VMworld 2011 session voting opened a week ago and there a still a few days left to cast your vote. About 300 in-depth sessions will be presented at VMworld this year and this year two sessions are submitted in which I participate. Both sessions are not the typical PowerPoint slide sessions, but are based on interaction with the attending audience. TA 1682 – vSphere Clustering Q&A Duncan Epping and Frank Denneman will answer any question with regards to vSphere Clustering in this session. You as the audience will have the chance to validate your own environment and design decisions with the Subject Matter Experts on HA, DRS and Storage DRS. Topics could include for instance misunderstandings around Admission Control Policies, the impact of limits and reservations on your environment, the benefits of using Resource Pools, Anti-Affinity Rules Gotchas, DPM and of course anything regarding Storage DRS. This is your chance to ask what you’ve always wanted to know! Duncan and I conducted this very successful session at the Dutch VMUG. Audience participation led to a very informative session where both general principles and in-depth details were explained and misconceptions where addressed. TA1425 - Ask the Expert vBloggers Four VMware Certified Design Experts (VCDX) On Stage! Are you running a virtual environment and experiencing some problems? Are you planning your companies’ Private Cloud strategy? Looking to deploy VDI and have some last minute questions? Do you have a virtual infrastructure design and want it blessed by the experts? Come join us for a one hour panel session where your questions are the topic of discussion! Join the Virtualization Experts, Frank Denneman (VCDX), Duncan Epping (VCDX), Scott Lowe (VCDX) and Chad Sakac, VP-VMware Alliance within EMC, as they answer your questions on virtualization design. Moderated by VCDX #21, Rick Scherer from VMwareTips.com Many friends of the business have submitted great sessions and there are really too many to list them all, but there is one I would want to ask you to vote on and that is the ESXi Quiz Show. A 1956 – The ESXi Quiz Show Join us for our very first ESXi Quiz Show where teams of vExperts and VMware engineers will match expertise on technical facts, trivia related to all VMware ESXi and related products. You as the audience will get 40% of the vote. We will cover topics around ESXi migration, storage, networking security, and VMware products. As an attendee of this session you will get to see the experts battle each other. For the very first time at VMworld you get to decide who leaves the stage as a winner and who does not. This can become the most awesome thing that ever hit VMworld. Can you think about the gossip, the hype and the sensation will introduce during VMworld? As Top vExperts, bloggers, VMware engineers and the just the lone sys admin (no not you Bob Plankers ;) ) compete with each other. Will the usual suspect win or will there be upsets? Who will dethrone who? Really I think this will become the hit of VMworld 2011 and will be the talk of the day at every party during the VMworld week. Session Voting is open until May 18, the competition is very fierce and it’s very difficult to choose between the excellent submitted sessions, however I would like to ask your help and I hope you guys are willing to vote on these three sessions. http://www.vmworld.com/cfp.jspa ================================================================================ Title: Contention on lightly utilized hosts URL: https://frankdenneman.ai/2011-04-27-contention-on-lightly-utilized-hosts/ Date: 2011-04-27 Often I receive the question why a virtual machine is not receiving resources while the ESXi host is lightly utilized and is accumulating idle time. This behavior is observed while reviewing the DRS distribution chart or the Host summary tab in the vSphere Client. A common misconception is that low utilization (Low MHz) equals schedule opportunities. Before focusing on the complexities of scheduling and workload behavior, let’s begin by reviewing the CPU distribution chart. The chart displays the sum of all the active virtual machines and their utilization per host. This means that in order to have 100% CPU utilization of the host, every active vCPU on the host needs to consume 100% of their assigned physical CPU (pCPU). For example, an ESXi host equipped with two Quad core CPUs need to simultaneously run eight vCPUs and each vCPU must consume 100% of “their” physical CPU. Generally this is a very rare condition and is only seen during boot storms or incorrect configured scheduled anti-virus scanning. But what causes latency (ready time) during low host utilization? Lets take a closer look at some common factors that affect or prohibit the delivery of the entitled resources: Amount of physical CPUs available in the system Amount of active virtual CPUs VSMP CPU scheduler behavior and vCPU utilization Load correlation and load synchronicity CPU scheduler behavior Amount of physical CPUs: To schedule a virtual CPU (vCPU) a physical (pCPU) needs to be available. It is possible that the CPU scheduler needs to queue virtual machines behind other virtual machines if more vCPUs are active than available pCPUs. Amount of active virtual CPUs: The keyword is active, an ESXi host only needs to schedule if a virtual machine is actively requesting CPU resources, contrary to memory where memory pages can exist without being used. Many virtual machines can run on host without actively requesting CPU time. Queuing will be caused if the amount of active vCPUs exceeds the number of physical CPUs. vSMP: (Related to the previous bullit) Virtual machines can contain multiple virtual processors. In the past vSMP virtual machine could experience latency due to the requirement of co-scheduling. Co-scheduling is the process of scheduling a set of processes on different physical CPUs at the same time. In vSphere 4.1 an advanced co-scheduling (relaxed co-scheduling) was introduced which reduced the latency radically. However ESX still needs to co-schedule vCPUs occasionally. This is due to the internal working of the Guest OS. The guest OS expects the CPUs it manages to run at the same pace. In a virtualized environment, a vCPU is an entity that can be scheduled and unscheduled independently from its sibling vCPUs belonging to the same virtual machine. And it might happen that the vCPUs do not make the same progress. If the difference in progress of the VM sibling vCPUs is too large, it can cause problems in the Guest OS. To avoid this, the CPU scheduler will occasionally schedule all sibling vCPUs. This behavior usually occurs if a virtual machine is oversized and does not host multithreaded applications. The “impact of oversized virtual machine series” offer more info on right-sizing virtual machines. CPU scheduler behavior and vCPU utilization: The local-host CPU scheduler uses a default time slice (quantum) of 50 milliseconds. A quantum is the amount of time a virtual CPU is allowed to run on a physical CPU before a vCPU of the same priority gets scheduled. When a vCPU is scheduled, that particular pCPU is not useable for other vCPUs and can introduce queuing. A small remark is necessary, a vCPU isn’t necessarily scheduled for the full 50 milliseconds, it can block before using up its quantum, and reducing the effective time slice the vCPU is occupying the physical CPU. Load correlation and load synchronicity: Load correlation defines the relationship between loads running in different machines. If an event initiates multiple loads, for example, a search query on front-end webserver resulting in commands in the supporting stack and backend. Load synchronicity is often caused by load correlation but can also exist due to user activity. It’s very common to see spikes in workload at specific hours, for example think about log-on activity in the morning. And for every action, there is an equal and opposite re-action, quite often load correlation and load synchronicity will introduce periods of collective non-or low utilization, which reduce the displayed CPU utilization. Local-host CPU scheduler behavior: The behavior of the CPU scheduler can impact the on scheduling of the virtual CPU. The CPU scheduler prefers to schedule the vCPU on the same pCPU it was scheduled before, to improve the chance of cache-hits. It might choose to ignore an idle CPU and wait a little bit so it can schedule the vCPU on the same pCPU again. If ESXi operates on a “Non-Uniform Memory Access” (NUMA) architecture, the NUMA CPU scheduler is active and will have effect on certain schedule decisions. The local host CPU scheduler will adjust progress and fairness calculations when Intel Hyper Threading is enabled on the system. Understanding CPU scheduling behavior can help you avoid latency, although understanding workload behavior and right sizing your virtual machines can help to improve performance. Frankdenneman.nl hosts multiple articles about the CPU scheduler and can be found here, however the technical paper “vSphere 4.1 CPU scheduler” is a must read if you want to learn more about the CPU scheduler. ================================================================================ Title: Restart vCenter results in DRS load balancing URL: https://frankdenneman.ai/2011-04-08-restart-vcenter-results-in-drs-load-balancing/ Date: 2011-04-08 Recently I had to troubleshoot an environment which appeared to have a DRS load-balancing problem. Every time when a host was brought out of maintenance mode, DRS didn’t migrate virtual machines to the empty host. Eventually virtual machines were migrated to the empty host but this happened after a couple of hours had passed. But after a restart of vCenter, DRS immediately started migrating virtual machines to the empty host. Restarting vCenter removes the cached historical information of the vMotion impact. vMotion impact information is a part of the Cost-Benefit Risk analysis. DRS uses this Cost-Benefit Metric to determine the return on investment of a migration. By comparing the cost, benefit and risks of each migration, DRS tries to avoid migrations with insufficient improvement on the load balance of the cluster. When removing the historical information a big part of the cost segment is lost, leading to a more positive ROI calculation, which in turn results in a more “aggressive” load-balance operation. ================================================================================ Title: Kindle version of HA&DRS book and info on Amazon's regional price scheme URL: https://frankdenneman.ai/2011-04-06-kindle-version-of-hadrs-book-and-info-on-amazons-regional-price-scheme-2/ Date: 2011-04-06 Maybe you already seen the tweets flying by, seen the posts on Facebook or just heard it at the water cooler but we finally published the eBook (Kindle) version of the HA and DRS deepdive book. Duncan has the low down on how the eBook came to life: http://www.yellow-bricks.com/2011-04-05-what-an-ebook-is-this-a-late-april-fools-joke/. We know that many who wanted the eBook bought the paper version instead so we decided to make it cheap and are offering the book for only $7.50. Please be aware that Amazon is using a regional price scheme, so orders outside the US pay a bit more. However, Marcel van den Berg (@marcelvandenber) posted a workaround how to save some money. Disclaimer: we do guarantee this works and won’t support this in any form. So without further ado we present: vSphere 4.1 HA and DRS technical deepdive,. Pick it up. Duncan & Frank PS: it is also available in the UK Kindle Store for £5.36. ================================================================================ Title: Kindle version of HA&DRS book and info on Amazon's regional price scheme URL: https://frankdenneman.ai/2011-04-06-kindle-version-of-hadrs-book-and-info-on-amazons-regional-price-scheme/ Date: 2011-04-06 Maybe you already seen the tweets flying by, seen the posts on Facebook or just heard it at the water cooler but we finally published the eBook (Kindle) version of the HA and DRS deepdive book. Duncan has the low down on how the eBook came to life: http://www.yellow-bricks.com/2011-04-05-what-an-ebook-is-this-a-late-april-fools-joke/. We know that many who wanted the eBook bought the paper version instead so we decided to make it cheap and are offering the book for only $7.50. Please be aware that Amazon is using a regional price scheme, so orders outside the US pay a bit more. However, Marcel van den Berg (@marcelvandenber) posted a workaround how to save some money. Disclaimer: we do guarantee this works and won’t support this in any form. So without further ado we present: vSphere 4.1 HA and DRS technical deepdive,. Pick it up. Duncan & Frank PS: it is also available in the UK Kindle Store for £5.36. ================================================================================ Title: Full color version of the new book? URL: https://frankdenneman.ai/2011-03-24-full-color-version-of-the-new-book/ Date: 2011-03-24 If you are following us on twitter you may have seen some recent tweets regarding our forthcoming book. Duncan (@duncanyb) and I have already started work on a new version of the HA and DRS Technical Deepdive. The new book will cover HA and DRS topics for the upcoming vSphere release. We are also aiming to include information about SIOC and Storage DRS in this version. We received a lot of feedback about the vSphere 4.1 book, one of the main themes was the lack of color in the diagrams. We plan to use a more suitable grayscale color combination in the next version, but we wondered if our readers would be interested in a full color copy of the upcoming book. Obviously printing costs increase with full color printing and in addition, low volume cost of color printing can be quite high. We expect the price of the full color version to cost around $50 USD – $55 USD. [poll id=“1”] ================================================================================ Title: IP-Hash versus LBT URL: https://frankdenneman.ai/2011-02-24-ip-hash-versus-lbt/ Date: 2011-02-24 vSwitch configuration and load-balancing policy selection are major parts of a virtual infrastructure design. Selecting a load-balancing policy can have impact on the performance of the virtual machine and can introduce additional requirements at the physical network layer. Not only do I spend lots of time discussing the various options during design sessions, it is also an often discussed topic during the VCDX defense panels. More and more companies seem to use IP-hash as there load balancing policy. The main argument seems to be increased bandwidth and better redundancy. Even when the distributed vSwitch is used, most organizations still choose IP-hash over the new load balancing policy “Route based on physical NIC load”. This article compares both load-balancing policies and lists the characteristics, requirements and constraints of both load-balancing policies. IP-Hash The main reason for selecting IP-Hash seems to be increased bandwidth as you aggregate multiple uplinks, unfortunately adding more uplinks does not proportionally increase the available bandwidth for the virtual machines. How IP-Hash works Based on the source and destination IP address together the VMkernel distributes the load across the available NICs in the vSwitch. The calculation of outbound NIC selection is described in KB article 1007371. To calculate the IP-hash yourself convert both the source and destination IP-addresses to a Hex value and compute the modulo over the number of available uplinks in the team. For example Virtual Machine 1 opens two connections, one connection to a backup server and one connection server to an application server. Virtual Machine IP-Address Hex Value VM1 164.18.1.84 A4120154 Backup Server 164.18.1.160 A41201A0 Application Server 164.18.1.195 A41201C3 The vSwitch is configured with two uplinks. Connection 1: VM1 > Backup Server (A4120154 Xor A41201A0 = F4) % 2 = 0 Connection 2: VM1 > Application Server (A4120154 Xor A41201C3 = 97) % 2 = 1 IP-Hash treats each connection between a source and destination IP address as a unique route and the vSwitch will distributed each connection across the available uplinks in the vSwitch. However due to the pNIC to vNIC affiliation, any connection is on a per flow basis. A flow can’t overflow to another uplink; this means that a connection is still limited to the speed of a single physical NIC. A real-world user case for IP-hash would be a backup server which requires a lot of bandwidth across multiple connections other than that; there are very few workloads that require bandwidth that can’t be satisfied by a single adapter. Complexity –In order for IP-hash to function correctly additional configuration at the network layer is required: EtherChannel: IP-hash needs to be configured on the vSwitch if EtherChannel technology is used at the physical switch layer. With EtherChannel the switch will load balance connections over multiple ports in the EtherChannel. Without IP-hash, the VMkernel only expects to receive information on a specific MAC address on a single vNIC. Resulting in some sessions go through to the virtual machine while other sessions will be dropped. When IP-hash is selected, then the VMkernel will accept inbound mac addresses on both active NICs EtherChannel configuration: As vSphere does not support dynamic link aggregation (LACP), none of the members can be set up to auto-negotiate membership and therefore physical switches have to be configured with static EtherChannel. Switch configuration: vSphere supports EtherChannel from one switch to the vSwitch. This switch can be a single switch or a stack of individual switches that act as one, but vSphere does not support EtherChannel from two separate – non stacked – switches, when the EtherChannel connect to the same vSwitch. Additional overhead – For each connection the VMkernel needs to select the appropriate uplink. If a virtual machine is running a front-end application and communicates 95% of its time to the backend database, the IP-Hash calculation is almost pointless. The VMkernel needs to perform the math for every connection and 95% of the connections will use the same uplink because the Algorithm will always result in the same hash. Utilization-unaware - It is possible that a second virtual machine is assigned to use the same uplink as the virtual machine that is already saturating the link. Let’s use the first example and introduce a new virtual machine VM3. Due to the backup window, VM3 connects to the backup server. Virtual Machine IP-Address Hex Value VM3 164.18.1.86 A4120156 Connection 3: VM3> Backup Server (A4120156 Xor A41201A0 = F6) % 2 = 0 Due to IP-HASH load balancing policy being unaware of utilization it will not rebalance if the uplink is saturated or if virtual machine are added or removed due to power-on or (DRS) migrations. DRS is unaware of network utilization and does not initiate a rebalance if a virtual machine cannot send or receive packets due to physical NIC saturation. In worst-case scenario DRS can migrate virtual machines to other ESX servers, leaving all the virtual machine that are saturating a NIC while the other virtual machines utilizing the other NICs are migrated. Admitted it’s a little bit of a stretch, but being aware of this behavior allows you to see the true beauty of the Load-Based Teaming team policy. Possible Denial of Service –Due to the pNIC-to-vNIC affiliation per connection a misbehaving virtual machine generating many connections can cause some sort of denial of service on all uplinks on the vSwitch. If this application would connect to a vSwitch with “Port-ID” or “based on physical load” only one uplink would be affected. Network failover detection Beacon Probing – Beacon probe does not work correctly if EtherChannel is used. ESX broadcast beacon packets out of all uplinks in a team. The physical switch is expected to forward all packets to other ports. In EtherChannel mode, the physical switch will not send the packets because it’s considered as one link. No beacon packets will be received and can interrupt network connections. Cisco switches will report flapping errors. See KB article 1012819. Route based on physical NIC Load VMware vSphere 4.1 introduced a new load-balancing policy available on distributed vSwitches. Route based on physical NIC load, also known as Load Based Teaming (LBT) takes the virtual machine network I/O load into account and tries to avoid congestion by dynamically reassigning and balancing the virtual switch port to physical NIC mappings. How LBT works Load Based Teaming maps vNICs to pNICs and remaps the vNIC-to-PNIC affiliation if the load exceeds specific thresholds on an uplink. LBT uses the same initial port assignment as the “originating port id” load balancing policy, resulting in the first vNIC being affiliated to the first pNIC, the second vNIC to the second pNIC, etc. After initial placement, LBT examines both ingress and egress load of each uplink in the team and will adjust the vNIC to pNIC mapping if an uplink is congested. The NIC team load balancer flags a congestion condition if an uplink experiences a mean utilization of 75% or more over a 30-second period. Complexity – LBT requires standard Access or Trunk ports. LBT does not support EtherChannels. Because LBT is moving flows among the available uplinks of the vSwitch, it may create packets re-ordering. Even though the reshuffling process is not done often (worst case scenario every 30 seconds) it is recommended to enable PortFast or TrunkFast on the switch ports. Additional overhead – The VMkernel will examine the congestion condition after each time window, this calculation creates a minor overhead opposed to using the static load-balancing policy “originating port-id”. Utilization aware – vNIC to pNIC mappings will be adjusted if the VMkernel detects congestion on an uplink. In the previous example both VM1 and VM3 shared the same connection due to the IP-hash calculation. Both connections can share the same physical NIC as long as the utilization stays below the threshold. It is likely that both vNICs are mapped to separate physical NICs. In the next example a third virtual machine is powered up and is mapped to NIC1. Utilization of NIC1 exceeds the mean utilization of 70% over a period of more than 30 seconds. After identifying congestion LBT remaps VM2 to NIC2 to decrease the utilization of NIC1. Although LBT is not integrated in DRS it can be viewed as complimentary technology next to DRS. When DRS migrates virtual machines onto a host, it is possible that congestion is introduced on a particular physical NIC. Due to vNIC to pNIC mapping based on actual load, LBT actively tries to avoid congestion at physical NIC level and attempts to reallocate virtual machines. By remapping vNiCs to pNICs it will attempt to make as much bandwidth available to the virtual machine, which ultimately benefits the overall performance of the virtual machine. Recommendations When using distributed virtual Switches it is recommended to use Load-Based teaming instead of IP-hash. LBT has no additional requirements on the physical network layer, reduces complexity and is able to adjust to fluctuating workloads. Due to the remapping of vNICs to pNICs based on actual load, LBT attempts to allocate as much bandwidth possible where IP-hash just simply distributes connections across the available physical NICs. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Dutch vBeers URL: https://frankdenneman.ai/2011-01-31-dutch-vbeers-2/ Date: 2011-01-31 Simon Long of The SLOG is introducing vBeers to Holland. I’ve copied the text from his vBeers blog article. Every month Simon Seagrave and I try organise a social get together of like-minded Virtualization enthusiasts held in a pub in central London (and Amsterdam). We like to call it vBeers. Before I go on, I would just like to state, although it’s called vBeers, you do NOT have to drink beer or any other alcohol for that matter. This isn’t just an excuse to get blind drunk. We came up with idea whilst on the Gestalt IT Tech Field Day back in April. We were chatting and we both recognised that we don’t get together enough to catch-up, mostly do to busy work schedules and private lives. We felt that if we had a set date each month, the likely hood of us actually making that date would be higher than previous attempts. So the idea of vBeers was born. The second Amsterdam vBeers will be held on Thursday 3rd of February starting at 6:30pm in ‘Herengracht Cafe’ which is placed close to Leidseplein and Dam Square. This venue serves a fine of selection of beers along with soft drinks and bar food. Drinks will not be paid for, there will not be a tab. When you buy a drink please pay for it as no one else will be paying for your drinks. * Location: The ‘Herengracht Cafe’ Amsterdam * Address: Herengracht 435, Herengracht/Leidsestraat * Nearest Tram Station: Koningsplein – Lijn 1,2,5 * Time: 6:30pm * Location: Map ================================================================================ Title: Re: impact of large pages on consolidation ratios URL: https://frankdenneman.ai/2011-01-25-re-impact-of-large-pages-on-consolidation-ratios/ Date: 2011-01-25 Gabe wrote an article about the impact of large pages on the consolidation ratio, I want to make something clear before the wrong conclusions are being made. Large pages will be broken down if memory pressure occurs in the system. If no memory pressure is detected on the host, i.e the demand is lower than the memory available, the ESX host will try to leverage large pages to have the best performance. Just calculate how big the Translation lookaside Buffer (TLB)is when a 2GB virtual machine use small pages (2048MB/4KB=512.000) or when using large pages 2048MB/2.048MB =1000. The VMkernel need to traverse the TLB through all these pages. And this is only for one virtual machine, imagine if there are 50 VMs running on the host. Like ballooning and compressing, if there is no need to over-manage memory than ESX will not do it as it generates unnecessary load. Using Large pages shows a different memory usage level, but there is nothing to worry about. If memory demand exceeds the availability of memory, the VMkernel will resort to share-before-swap and compress-before-swap. Resulting in collapsed pages and reducing the memory pressure. ================================================================================ Title: Setting Correct Percentage of Cluster Resources Reserved URL: https://frankdenneman.ai/2011-01-20-setting-correct-percentage-of-cluster-resources-reserved/ Date: 2011-01-20 vSphere introduced the HA admission control policy “Percentage of Cluster Resources Reserved”. This policy allows the user to specify a percentage of the total amount of available resources that will stay reserved to accommodate host failures. When using vSphere 4.1 this policy is the de facto recommended admission control policy as it avoids the conservative slots calculation method. Reserved failover capacity The HA Deepdive page explains in detail how the “percentage resources reserved” policy works, but to summarize; the CPU or memory capacity of the cluster is calculated as followed;The available capacity is the sum of all ESX hosts inside the cluster minus the virtualization overhead, multiplied by (1-percentage value). For instance; a cluster exists out of 8 ESX hosts, each containing 70GB of available RAM. The percentage of cluster resources reserved is set to 20%. This leads to a cluster memory capacity of 448GB (70GB+70GB+70GB+70GB+70GB+70GB+70GB+70GB) * (1 – 20%). 112GB is reserved as failover capacity. Although the example zooms in on memory, the percentage set applies both CPU and memory resources. Once a percentage is specified, that percentage of resources will be unavailable for active virtual machines, therefore it makes sense to set the percentage as low as possible. There are multiple approaches for defining a percentage suitable for your needs. One approach, the host-level-approach is to use a percentage that corresponds with the contribution of one or host or a multiplier of that. Another approach is the aggressive approach which sets a percentage that equals less than the contribution of one host. Which approach should be used? Host-level In the previous example 20% was used to be reserved for resources in an 8-host cluster. This configuration reserves more resources than a single host contributes to the cluster. High Availability’s main objective is to provide automatic recovery for virtual machines after a physical server failure. For this reason, it is recommended to reserve resource equal to a single host or a multiplier of that. When using the per-host level of granularity in an 8-host cluster (homogeneous configured hosts), the resource contribution per host to the cluster is 12.5%. However, the percentage used must be an integer (whole number). Using a conservative approach it is better to round up to guarantee that the full capacity of one host is protected, in this example, the conservative approach would lead to a percentage of 13%. Aggressive approach I have seen recommendations about setting the percentage to a value that is less than the contribution of one host to the cluster. This approach reduces the amount of resources reserved for accommodating host failures and results in higher consolidation ratios. One might argue that this approach can work as most hosts are not fully loaded, however it eliminates the guarantee that after a failure all impacted virtual machines will be recovered. As datacenters are dynamic, operational procedures must be in place to -avoid or reduce- the impact of a self-inflicted denial of service. Virtual machine restart priorities must be monitored closely to guarantee that mission critical virtual machines will be restarted before virtual machine with a lower operational priority. If reservations are set at virtual machine level, it is necessary to recalculate the failover capacity percentage when virtual machines are added or removed to allow the virtual machine to power on and still preserve the aggressive setting. Expanding the cluster Although the percentage is dynamic and calculates capacity at a cluster-level, when expanding the cluster the contribution per host will decrease. If you decide to continue using the percentage setting after adding hosts to the cluster, the amount of reserved resources for a fail-over might not correspond with the contribution per host and as a result valuable resources are wasted. For example, when adding four hosts to an 8-host cluster while continue using the previously configured admission control policy value of 13% will result in a failover capacity that is equivalent to 1.5 hosts. The following diagram depicts a scenario where an 8 host cluster is expanded to 12 hosts; each with 8 2GHz cores and 70GB memory. The cluster was originally configured with admission control set to 13% which equals to 109.2 GB and 24.96 GHz. If the requirement is to be able to recover from 1 host failure 7,68Ghz and 33.6GB is “wasted”. Maximum percentage High availability relies on one primary node to function as the failover coordinator to restart virtual machines after a host failure. If all five primary nodes of an HA cluster fail, automatic recovery of virtual machines is impossible. Although it is possible to set a failover spare capacity percentage of 100%, using a percentage that exceeds the contribution of four hosts is impractical as there is a chance that all primary nodes fail. Although configuration of primary agents and configuration of the failover capacity percentage are non-related, they do impact each other. As cluster design focus on host placement and rely on host-level hardware redundancy to reduce this risk of failing all five primary nodes, admission control can play a crucial part by not allowing more virtual machines to be powered on while recovering from a maximum of four host node failure. This means that maximum allowed percentage needs to be calculated by summing the contribution per host x 4. For example the recommended maximum allowed configured failover capacity of a 12-host cluster is 34%, this will allow the cluster to reserve enough resources during a 4 host failure without over allocating resources that could be used for virtual machines. ================================================================================ Title: 'Draft' of the vSphere 4.1 Hardening guide released URL: https://frankdenneman.ai/2011-01-19-draft-of-the-vsphere-4-1-hardening-guide-released/ Date: 2011-01-19 The ‘Draft’ of the vSphere 4.1 Hardening guide has been released. This draft will remain posted for comments until approximately the end of February 2011.The official document will be released shortly after the draft period. Please see the following: http://communities.vmware.com/docs/DOC-14548 ================================================================================ Title: HA and DRS book in action URL: https://frankdenneman.ai/2011-01-13-ha-and-drs-book-in-action/ Date: 2011-01-13 ================================================================================ Title: Beating a dead horse - using CPU affinity URL: https://frankdenneman.ai/2011-01-11-beating-a-dead-horse-using-cpu-affinity/ Date: 2011-01-11 Lately the question about setting CPU affinity is rearing its ugly head again. Will it offer performance advantages for the virtual machine? Yes it can, but only in very specific cases. Additional settings and changes to the virtual infrastructure are required to obtain a performance increase over the default scheduling techniques. Setting CPU affinity by itself will not result in any performance gain, but usually a performance decrease. What does CPU affinity do? By setting a CPU affinity on the virtual machine you are limiting the available CPUs on which the virtual machine can run. It does not dedicate that CPU to that virtual machine and therefore does not restrict the CPU scheduler from using that CPU for other virtual machines. When will CPU-affinity help? Under a controlled environment some specific workloads can benefit from using CPU affinity. When the virtual machine workload is cache bound and has a larger cache footprint than the available cache of one CPU it can profit from aggregated caches. However, if this workload has high intra-thread communications and is running on specific CPU architectures setting CPU affinity can have the opposite effect and become detrimental to the performance of the application. CPU-affinity can also be used to isolate a physical CPU to a virtual CPU. But requires a lot of changes and increases management. It will never dedicate the physical CPU to the virtual machine as the VMkernel schedules all its processes across all available CPUs regardless of any custom setting a virtual machine has. Furthermore the scheduling overhead stays the same whether CPU-affinity is set on the virtual machine or not. To determine if you application fit this description can be a challenge and maintaining such configurations usually result in a nightmare. Generally CPU-affinity is only used for simulations and load testing and it is better left unused for every other cases. Setting CPU-affinity results in less choice for the CPU scheduler to schedule the virtual machine, but there is more to it as well: Controlled environment Already mentioned but this cannot be stressed enough, CPU affinity does not equal isolation of a physical CPU. In other words, when a virtual machine is pinned to a physical CPU it does not control or own that CPU. The VMkernel CPU scheduler still considers that physical CPU a valid CPU to schedule other virtual machines on. If isolation of a CPU is the end-goal, than all other residing virtual machines on the host (and virtual machine that will be created in the future) must be configured with CPU affinity as well and the specific CPU(s) assigned to the virtual machine must excluded from all other virtual machines. Setting CPU affinity results in manual CPU micro management and can be a nightmare to maintain. To make it worse, think of the impact a migration will have, the administrator needs to configure the virtual machines on the destination host to exclude the CPU from all active virtual machines as well. (Update: Recent vSphere versions offer the “Latency Sensitive” functionality, isolating cores for vCPUs) Virtual Machine worlds A virtual machine is made of multiple worlds (threads), besides the vCPU world, worlds are active for the virtual machine MKS subsystem, CD-ROM and VMX file. Although the vCPU world generates the greater part of the CPU load, sometimes a physical CPU is required to run the other worlds. If CPU affinity is set, then all the worlds that constitute the virtual machine can only run on the specified CPUs. If set incorrectly, it can reduce the throughput of the virtual machine as the worlds must compete between each other for CPU time. Therefore it is recommended to add an additional CPU for these worlds. For example; configure a CPU affinity setting that contains 3 physical CPUs for a 2 vCPU virtual machine. Resource entitlements As CPU affinity will not automatically isolate the CPU for that specific virtual machine, shares and reservations needs to be set to guarantee a specific performance level. Because the scheduler will attempt to maintain fairness for all virtual machines it is possible that other virtual machines will be scheduled on the set of CPU specified in the affinity set of the virtual machine. Adjust the shares and reservations of the virtual machine accordingly to ensure priority over other active virtual machines. Be aware that CPU reservations are friendly; although the vCPU is guaranteed a specific portion of physical resources, it might happen that an external thread/interloper (other virtual machine) is using the vCPU; this thread will not instantly be de-scheduled. Even when the waiting virtual machine has a 100% CPU reservation configured. To make it worse, in the case when multiple virtual machines are affinity-bound to the same processor it is possible that the CPU scheduler cannot meet the specified reservation. Be aware that admission control ignores affinity, so multiple virtual machines can have a full reservation equal to a full core but still need to compete with other affinity bound virtual machines. More information about how CPU reservations work can be found in the article: “Reservations and CPU Scheduling”. CPU reservations and HA admission control If the virtual machine with the reservation is running in a HA cluster with a “Host failures cluster tolerates” admission control policy, the CPU reservation will influence the Slot size of the Cluster and can therefore impact the consolidation ratio of the cluster. More info about slot-sizes can be found on the HA deepdive. CPU affinity and DRS clusters. Because vMotion is not allowed if a virtual machine is configured with CPU affinity, that virtual machine cannot be placed in a DRS cluster with automation mode set to fully automated. If a virtual machine needs to be configured with CPU affinity, the administrator has three choices: Place the virtual machine on a stand-alone host Set DRS automation level to manual / partially automated Set virtual machine automation mode to manual / partially automated Stand-alone host If the virtual machine is placed on the stand-alone host the performance of the virtual machine depends on the level of contention and the virtual machine resource entitlement. During resource contention it can only fall back on its resource entitlement and hopefully gain a higher priority than the other residing virtual machines. If the virtual machine was located on an ESXi host in a DRS cluster, the virtual machine could have been migrated to receive its resource entitlement on another host. By choosing CPU-affinity, you are betting only on one horse, the local CPU scheduler of one host instead of leveraging the full suite of resource management vSphere delivers today. DRS set to Manual or partially automated If the DRS automation level is set to manual or partially automated, the cluster will not automatically load balance virtual machines and DRS will recommend migrations. These recommendations must be applied manually by the administrator. DRS imbalance calculation will be invoked every 300 seconds but is also triggered if the cluster detects resource demand and supply changes, as well as changes in the resource settings in the cluster. As you can imagine, this behavior will create an incredible load on the administrator to let the cluster operate as efficiently as possible if he wants to ensure that the virtual machines are receiving their resource entitlements. Set Virtual machine automation mode to manual / partially automated By changing the automation mode on VM-level, the virtual machine can still be placed inside a fully automated DRS cluster. Although DRS will not automatically migrate this virtual machine, it can migrate other virtual machines to ensure every virtual machine will receive its resource entitlement. However additional measures (shares and reservations) must be taken to guarantee the virtual machine enough physical resources. CPU architectures Today new CPU architectures, such as the Intel Nehalem and AMD Opteron’s offer a variety of on-die caches, multiple cores \ logical CPUs and an optimized local\remote memory subsystem. These features can either helpful or be detrimental to the performance of a virtual machine with CPU affinity. Cache level If a virtual machine is spanned across two processors (packages) it effectively results in having two L3 caches available to the virtual machine. Today’s CPU architectures offer dedicated L1 and L2 cache per core and a shared last-level L3 cache for all cores inside the CPU package. Because access to Last level cache is faster than (normal) memory, it makes sense to span the virtual machine across two processor packages to increase the amount of available L3 cache. However the inter-socket communication speed can reduce –or remove- the positive effect of having low-latency cache available and if the workload can fit inside one cache (small cache footprint) and uses intensive intra-thread communication, than placement in one processor packaged is to be preferred over spanning multiple packages. HyperThreading If a virtual machine is running on a HyperThreading-enabled system it is best to set the CPU-affinity to logical CPUs not belonging to the same core. The HT threads on a core are translated by the VMkernel as logical CPUs and are consecutively numbers, for example Core 1 contains LCPU0 and LCPU1, Core 2 contains LCPU2 and LCPU3, etc. If CPU-affinity is set to logical CPUs belonging to the same core, both vCPUs of the virtual machine need to compete with each other for physical CPU resources. By scheduling a virtual machine on logical CPUs of different cores, it doesn’t have to compete and can benefit the vCPUs’ throughput because the VMkernel allows the vCPU to use the entire Cores’ resources if only one logical CPU residing on the core is active. NUMA If CPU affinity is set on a virtual machine running in a NUMA architecture (Intel Nehalem and AMD Opteron) the virtual machine is treated as a NON-NUMA client and gets excluded from NUMA scheduling. Therefore the NUMA scheduler will not set a memory affinity for the virtual machine to its current NUMA node and the VMkernel can allocate memory from every available NUMA node in the system Therefore the virtual machine may end up running on a different NUMA node than were its memory is residing, resulting in unnecessary memory latency and possibly higher %Ready time as the instruction must wait until the memory is fetched from a remote node. Bottomline The bottomline is that almost in every case CPU affinity is better left unused. Scheduling threads is very complex, scheduling threads belonging to multiple virtual machines with different priorities, activity, progress and still considering optimal use of the underlying CPU and memory architecture is mind-blowing complex. The CPU scheduler is aware of all these components and together with the global scheduler (DRS) it can see to it that the virtual machine will receive its resource entitlement. If the virtual machine must have access to physical resources at any time, other mechanisms such as resource allocation settings will have a better effect than using the advanced setting CPU-affinity. ================================================================================ Title: AMD Magny-Cours and ESX URL: https://frankdenneman.ai/2011-01-05-amd-magny-cours-and-esx/ Date: 2011-01-05 AMD’s current flagship model is the 12-core 6100 Opteron code name Magny-Cours. Its architecture is quite interesting to say at least. Instead of developing one CPU with 12 cores, the Magny Cours is actually two 6 core “Bulldozer” CPUs combined in to one package. This means that an AMD 6100 processor is actually seen by ESX as this: As mentioned before, each 6100 Opteron package contains 2 dies. Each CPU (die) within the package contains 6 cores and has its own local memory controllers. Even though many server architectures group DIMM modules per socket, due to the use of the local memory controllers each CPU will connect to a separate memory area, therefore creating different memory latencies within the package. Because different memory latency exists within the package, each CPU is seen as a separate NUMA node. That means a dual AMD 6100 processor system is treated by ESX as a four-NUMA node system: Impact on virtual machines Because the AMD 6100 is actually two 6-core NUMA nodes, creating a virtual machine configured with more than 6 vCPUs will result in a wide-VM. In a wide-VM all vCPUs are split across a multitude of NUMA clients. At the virtual machine’s power on, the CPU scheduler determines the number of NUMA clients that needs to be created so each client can reside within a NUMA node. Each NUMA client contains as many vCPUs possible that fit inside a NUMA node.That means that an 8 vCPU virtual machine is split into two NUMA clients, the first NUMA client contains 6 vCPUs and the second NUMA client contains 2 vCPUs. The article “ESX 4.1 NUMA scheduling” contains more info about wide-VMs. Distribution of NUMA clients across the architecture ESX 4.1 uses a round-robin algorithm during initial placement and will often pick the nodes within the same package. However it is not guaranteed and during load-balancing the VMkernel could migrate a NUMA client to another NUMA node external to the current package. Although the new AMD architecture in a two-processor system ensures a 1-hop environment due to the existing interconnects, the latency from 1 CPU to another CPU memory within the same package is less than the latency to memory attached to a CPU outside the package. If more than 2 processors are used a 2-hop system is created, creating different inter-node latencies due to the varying distance between the processors in the system. Magny-Cours and virtual machine vCPU count The new architecture should perform well, at least better that the older Opteron series due to the increased bandwidth of the HyperTransport interconnect and the availability of multiple interconnects to reduce the amounts of hops between NUMA nodes. By using Wide-VM structures, ESX reduces the amount of hops and tries to keep as much memory local. But –if possible- the administrator should try to keep the virtual machine CPU count beneath the maximum CPU count per NUMA node. In the 6100 Magny-Cours case that should be maximum 6 vCPUs per virtual machine ================================================================================ Title: Funny: HA and DRS technical deepdive audiobook URL: https://frankdenneman.ai/2011-01-03-funny-ha-and-drs-technical-deepdive-audiobook/ Date: 2011-01-03 During a conversation the idea of an audiobook of the HA and DRS book spawned. Within a couple of minutes, I found the following in my inbox…. Once a tiny little vm found himself in a big bad cluster filled with big vm’s…… Rapunzel, Rapunzel, set down your high shares! Then the admin installed the LittleBoyBlue patch on the Dike server, and plugged the memory leak. Odysseus set a CPU limit on the Cyclops-VM so low that Cyclops couldn’t even see. “Who are you?” yelled the Cyclops. Odysseus replied, “My name is No One!” When the Cyclops complained to the Scheduler, it asked, “Who has limited you so badly?” “No One has!” replied the Cyclops…. (BTW who makes a creature with one eye? What a horrible single point of failure to bake into your design!) But the third little VM had his very own Resource Pool, and he huffed and he puffed and he outcompeted the much bigger VMs who were all sharing their Resource Pool shares… Just let’s focus on publishing an ebook first….. ================================================================================ Title: Impact of oversized virtual machines part 3 URL: https://frankdenneman.ai/2011-01-03-impact-of-oversized-virtual-machines-part-3/ Date: 2011-01-03 In part 1 of the series of post on the impact of oversized virtual machines NUMA architecture, memory overhead reservation and share levels are reviewed, part 2 zooms in on the impact of memory overhead reservation and share levels on HA and DRS. This part looks at CPU scheduling, memory management and what impact oversized virtual machines have on the environment when a bootstorm occurs. Multiprocessor virtual machine In most cases, adding more CPUs to a virtual machine does not automatically guarantee increase throughput of the application, because some workloads cannot always take advantage of all the available CPUs. Sharing resources and scheduling these processes will introduce additional overhead. For example, a four-way virtual machine is not four times as productive as a single-CPU system. If the application is unable to scale than the application will not benefit from these additional available resource. Progress Although relaxed co-scheduling reduces the requirement of the VMkernel to simultaneous schedule all vCPUs of the virtual machine, periodically scheduling the unused or idle vCPUs is still necessary to keep the progress of each vCPU in the virtual machine acceptably synchronized. Esxtop also gives scheduling stats for SMP virtual machines; %CRUN: All VCPUs want to run at once. CRUN is the amount of time between when a PCPU is told to run a certain VCPU on an SMP VM and when it is actually able to run that VM. This should be almost 0. %CSTOP: If a VCPU gets ahead of another VCPU of the same SMP VM, then we ask the faster VCPU to stop until the other one can catch up. The time spent in this stopped state is CSTOP. Single thread application Only applications with multiple threads and allow them to be scheduled in parallel can benefit from multiprocessor systems. A single-threaded application can only be scheduled on one CPU at the time and will not benefit from the multiple CPUs available. The Guest OS is able to migrate the thread between the available CPUs, introducing unnecessary overhead such as interrupts or context switches and cache misses. Timer interrupts In older guest operating systems, the unused virtual CPUs still take timer interrupts, which consumes a small amount of additional CPU. Please refer to KB articles “High CPU Utilization of Inactive Virtual Machines - KB1077” Configured memory Oversizing the memory configuration of a virtual machine can impact the performance of the virtual machine itself or even worse, impact the other active virtual machines on the host and in the cluster. Using memory reservations on oversized virtual machines will make it go from bad to worse. Application memory management Excess memory is a problem when the application uses this memory opportunistically, in other words the application is hoarding memory. Java, SAP and often Oracle workloads assume it can use all the memory it detects. Because ESX cannot determine which memory is important to the virtual machine, it always backs memory pages of the virtual machine with physical pages. Besides creating a large memory footprint on the physical level, these kinds of applications add a third level of memory management as well. Due to this additional management level, the Guest OS does not understand which pages are important and which are not. And because the Guest OS isn’t aware, it can not return inactive pages to the balloon driver when requested, therefor impacting the performance of the application during contention even more. Setting memory reservation at virtual machine level will guarantee the availability of physical memory and will secure a certain level of application performance (if memory bound). However setting memory reservations at virtual machine level will impact the virtual infrastructure and the larger the memory reservation, the larger the impact. Visit “Impact of memory reservation” for more info. To avoid these effects, it is recommended to monitor the behavior of the application over time and tune the configuration of the virtual machine and its reservation to get proper performance and limit the impact of its configured memory and the memory reservation. NUMA node If the virtual machines mentioned in the previous paragraph are configured with more memory than available in their home NUMA node, the system needs to fetch the memory from remote NUMA nodes. Accessing memory from remote nodes introduces latencies and generally reduced throughput of the vCPU. ESX does not communicate any NUMA information to the Guest OS and therefore both the Guest OS as well as the application are unaware of the non-uniform latency characteristics of the underlying platform. The Guest OS and application are therefor unable to prioritize which memory it will use. If the virtual machine uses all the available memory of a NUMA node, it will lead to a higher degree of remote memory of all the other active virtual machines using the pCPU, leading to higher memory latencies and less throughput of the other virtual machines and eventually an intra-node migration. For more information about NUMA nodes, please read the articles: Sizing VMs and NUMA nodes and ESX 4.1 NUMA Scheduling. Attempt to configure virtual machine with less memory than available in a NUMA node. Swap file During boot a swap file is created that equals the virtual machines configured memory minus the configured memory reservation. If no memory reservation is set, the virtual machine swap file (.vswap) equals the configured memory. Large virtual machines will generate an additional requirement for storing these large swap files reducing the consolidation ratio of virtual machines per VMFS datastore. Bootstorms A bootstorm is the occurrence of powering on a multitude of virtual machines simultaneously. Virtual infrastructures running versions prior to ESX 4.1 can encounter memory contention when a bootstorm occurs of virtual machines running windows. Windows checks how much memory is available to the OS by zeroing out pages it detects. Transparent page sharing will collapse these pages but this will not occur immediately. Transparent Page Sharing is a cycle-driven process that tries to make a pass over the virtual machine memory with a timeframe of 3600 seconds. The level of contention will impact the speed of the TPS process. During a bootstorm, this zero-out behavior and delayed TPS process can introduce contention. Usually this contention is short-lived. Unfortunately during the startup phase of the guest OS the balloon driver will not be loaded and this situation can lead to compressing (10% of configured memory) and swapping useless data straight to disk. ESXTOP will display swapped out memory but due to the nature of the data will show little to none swap-in. ESX 4.1 uses a new technique called zero-page sharing. An in-depth post about this cool new technique will follow shortly. End-note This post concludes the three-part series about the impact of oversized virtual machines. The reason I wrote these articles is that I know many organizations still size their virtual machines on assumed peak loads happing somewhere in the (late) future of that service or application. Many organizations are using the same policy or method used for physical machines. The beauty of using virtual machines is the flexibility an organization has when it comes to determining the size of a machine during its lifecycle. Leverage these mechanisms and incorporate this in your service catalog and daily operations. Size the virtual machine according to its current or near-future workload. ================================================================================ Title: Node Interleaving: Enable or Disable? URL: https://frankdenneman.ai/2010-12-28-node-interleaving-enable-or-disable/ Date: 2010-12-28 There seems to be a lot of confusion about this BIOS setting, I receive lots of questions on whether to enable or disable Node interleaving. I guess the term “enable” make people think it some sort of performance enhancement. Unfortunately the opposite is true and it is strongly recommended to keep the default setting and leave Node Interleaving disabled. Node interleaving option only on NUMA architectures The node interleaving option exists on servers with a non-uniform memory access (NUMA) system architecture. The Intel Nehalem and AMD Opteron are both NUMA architectures. In a NUMA architecture multiple nodes exists. Each node contains a CPU and memory and is connected via a NUMA interconnect. A pCPU will use its onboard memory controller to access its own “local” memory and connects to the remaining “remote” memory via an interconnect. As a result of the different locations memory can exists, this system experiences “non-uniform” memory access time. Node interleaving disabled equals NUMA By using the default setting of Node Interleaving (disabled), the system will build a System Resource Allocation Table (SRAT). ESX uses the SRAT to understand which memory bank is local to a pCPU and tries* to allocate local memory to each vCPU of the virtual machine. By using local memory, the CPU can use its own memory controller and does not have to compete for access to the shared interconnect (bandwidth) and reduce the amount of hops to access memory (latency) * If the local memory is full, ESX will resort in storing memory on remote memory because this will always be faster than swapping it out to disk. Node interleaving enabled equals UMA If Node interleaving is enabled, no SRAT will be built by the system and ESX will be unaware of the underlying physical architecture. ESX will treat the server as a uniform memory access (UMA) system and perceives the available memory as one contiguous area. Introducing the possibility of storing memory pages in remote memory, forcing the pCPU to transfer data over the NUMA interconnect each time the virtual machine wants to access memory. By leaving the setting Node Interleaving to disabled, ESX can use System Resource Allocation Table to the select the most optimal placement of memory pages for the virtual machines. Therefore it’s recommended to leave this setting to disabled even when it does sound that you are preventing the system to run more optimally. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Impact of oversized virtual machines part 2 URL: https://frankdenneman.ai/2010-12-17-impact-of-oversized-virtual-machines-part-2/ Date: 2010-12-17 In part 1 of the series of post on the impact of oversized virtual machines NUMA architecture, memory overhead reservation and share levels are reviewed, part 2 zooms in of the impact of memory overhead reservation and share levels on HA and DRS. Impact of memory overhead reservation on HA Slot size The VMware High Availability admission control policy “Host failures cluster tolerates” calculates a slot size to determine the maximum amount of virtual machines active in the cluster without violating failover capacity. This admission control policy determines the HA cluster slot size by calculating the largest CPU reservation, largest memory reservation plus it’s memory overhead reservation. If the virtual machine with the largest reservation (which could be an appropriate sized reservation) is oversized, its memory overhead reservation still can substantial impact the slot size. The HA admission control policy “Percentage of Cluster Resources Reserved” calculate the memory component of its mechanism by summing the reservation plus the memory overhead of each virtual machine. Therefore allowing the memory overhead reservation to even have a bigger impact on admission control than the calculation done by the “Host Failures cluster tolerates” policy. DRS initial placement DRS will use a worst-case scenario during initial placement. Because DRS cannot determine resource demand of the virtual machine that is not running, DRS assumes that both the memory demand and CPU demand is equal to its configured size. By oversizing virtual machines it will decrease the options in finding a suitable host for the virtual machine. If DRS cannot guarantee the full 100% of the resources provisioned for this virtual machine can be used it will vMotion virtual machines away so that it can power on this single virtual machine. In case there are not enough resources available DRS will not allow the virtual machine to be powered on. Shares and resource pools When placing a virtual machine inside a resource pool, its shares will be relative to the other virtual machines (and resource pools) inside the pool. Shares are relative to all the other components sharing the same parent; easier way to put it is to call it sibling share level. Therefore the numeric share values are not directly comparable across pools because they are children of different parents. By default a resource pool is configured with the same share amount equal to a 4 vCPU, 16GB virtual machine. As mentioned in part 1, shares are relative to the configured size of the virtual machine. Implicitly stating that size equals priority. Now lets take a look again at the image above. The 3 virtual machines are reparented to the cluster root, next to resource pools 1 and 2. Suppose they are all 4 vCPU 16GB machines, their share values are interpreted in the context of the root pool and they will receive the same priority as resource pool 1 and resource pool2. This is not only wrong, but also dangerous in a denial-of-service sense – a virtual machine running on the same level as resource pools can suddenly find itself entitled to nearly all cluster resources. Because of default share distribution process we always recommend to avoid placing virtual machines on the same level of resource pools. Unfortunately it might happen that a virtual machine is reparented to cluster root level when manually migrating a virtual machine using the GUI. The current workflow defaults to cluster root level instead of using its current resource pool. Because of this it’s recommended to increase the number of shares of the resource pool to reflect its priority level. More info about shares on resource pools can be found in Duncan’s post on yellow-bricks.com. Go to Part 3: Impact of oversized virtual machine. ================================================================================ Title: Impact of oversized virtual machines part 1 URL: https://frankdenneman.ai/2010-12-16-impact-of-oversized-virtual-machines-part-1/ Date: 2010-12-16 Recently we had an internal discussion about the overhead an oversized virtual machine generates on the virtual infrastructure. An oversized virtual machine is a virtual machine that consistently uses less capacity than its configured capacity. Many organizations follow vendor recommendations and/or provision virtual machine sized according to the wishes of the customer i.e. more resources equals better performance. By oversizing the virtual machine you can introduce the following overhead or even worse decrease the performance of the virtual machine or other virtual machines inside the cluster. Note: This article does not focus on large virtual machines that are correctly configured for their workloads. Memory overhead Every virtual machine running on an ESX host consumes some memory overhead additional to the current usage of its configured memory. This extra space is needed by ESX for the internal VMkernel data structures like virtual machine frame buffer and mapping table for memory translation, i.e. mapping physical virtual machine memory to machine memory. The VMkernel will calculate a static overhead of the virtual machine based on the amount of vCPUs and the amount of configured memory. Static overhead is the minimum overhead that is required for the virtual machine startup. DRS and the VMkernel uses this metric for Admission Control and vMotion calculations. If the ESX host is unable to provide the unreserved resources for the memory overhead, the VM will not be powered on, in case of vMotion, if the destination ESX host must be able to back the virtual machine reservation and the static overhead otherwise the vMotion will fail. The following table displays a list of common static memory overhead encountered in vSphere 4.1. For example, a 4vCPU, 8GB virtual machine will be assigned a memory overhead reservation of 413.91 MB regardless if it will use its configured resources or not. Memory (MB) 2vCPUs 4vCPUs 8vCPUs 2048 198.20 280.53 484.18 4096 242.51 324.99 561.52 8192 331.12 413.91 716.19 16384 508.34 591.76 1028.07 The VMkernel treats virtual machine overhead reservation the same as VM-level memory reservation and it will not reclaim this memory once it has been used, furthermore memory overhead reservations will not be shared by transparent page sharing. Shares (size does not translate into priority) By default each virtual machine will be assigned a specific amount of shares. The amount of shares depends on the share level, low, normal or high and the amount of vCPUs and the amount of memory. Share Level Low Normal High Shares per CPU 500 1000 2000 Shares per MB 5 10 20 I.e. a virtual machine configured with 4CPUs and 8GB of memory with normal share level receives 4000 CPU shares and 81960 memory shares. Due to relating amount of shares to the amount of configured resources this “algorithm” indirectly implies that a larger virtual machine needs to receive a higher priority during resource contention. This is not true, as some business critical applications perfectly are run on virtual machines configured with low amounts of resources. Oversized VMs on NUMA architecture vSphere 4.1 CPU scheduler has undergone optimization to handle virtual machines which contains more vCPUs than available cores on one NUMA physical CPU. The virtual machine (wide-vm) will be spread across the minimum number of NUMA nodes, but memory locality will be reduced, as memory will be distributed among its home NUMA nodes. This means that a vCPU running on one NUMA node might needs to fetch memory from its other NUMA node. Leading to unnecessary latency, CPU wait states, which can lead to %ready time for other virtual machines in high consolidated environments. Wide-NUMA nodes are of great use when the virtual machine actually run load comparable to its configured size, it reduces overhead compared to the 3.5/4.0 CPU scheduler, but it still will be better to try to size the virtual machine equal or less than the available cores in a NUMA node. More information about CPU scheduling and NUMA architectures can be found here: https://frankdenneman.ai/2010/09/esx-4-1-numa-scheduling/ Go to Part 2: Impact of oversized virtual machine on HA and DRS ================================================================================ Title: Enhanced vMotion Compatibility URL: https://frankdenneman.ai/2010-12-14-enhanced-vmotion-compatibility/ Date: 2010-12-14 Enhanced vMotion Compatibility (EVC) is available for a while now, but it seems to be slowly adopted. Recently VMguru.nl featured an article “Challenge: vCenter, EVC and dvSwitches” which illustrates another case where the customer did not enable EVC when creating the cluster. There seem to be a lot of misunderstanding about EVC and the impact it has on the cluster when enabled. What is EVC? VMware Enhanced VMotion Compatibility (EVC) facilitates VMotion between different CPU generations through use of Intel Flex Migration and AMD-V Extended Migration technologies. When enabled for a cluster, EVC ensures that all CPUs within the cluster are VMotion compatible. What is the benefit of EVC? Because EVC allows you to migrate virtual machines between different generations of CPUs, with EVC you can mix older and newer server generations in the same cluster and be able to migrate virtual machines with VMotion between these hosts. This makes adding new hardware into your existing infrastructure easier and helps extend the value of your existing hosts. EVC forces newer processors to behave like old processors Well, this is not entirely true; EVC creates a baseline that allows all the hosts in the cluster that advertises the same feature set. The EVC baseline does not disable the features, but indicates that a specific feature is not available to the virtual machine. Now it is crucial to understand that EVC only focuses on CPU features, such as SSE or AMD-now instructions and not on CPU speed or cache levels. Hardware virtualization optimization features such as Intel VT-Flexmigration or AMD-V Extended Migration and Memory Management Unit virtualization such as Intel EPT or AMD RVI will still be available to the VMkernel even if EVC is enabled. As mentioned before EVC only focuses of the availability of features and instructions of the existing CPUs in the cluster. For example features like SIMD instructions such as the SSE instruction set. Let’s take a closer look, when selecting an EVC baseline, it will apply a baseline feature set of the selected CPU generation and will expose specific features. If an ESX host joins the cluster, only those CPU instructions that are new and unique to that specific CPU generation are hidden from the virtual machines. For example; if the cluster is configured with an Intel Xeon Core i7 baseline, it will make the standard Intel Xeon Core 2 feature plus SSE4.1., SSE4.2, Popcount and RDTSCP features available to all the virtual machines, when an ESX host with a Westmere (32nm) CPU joins the cluster, the additional CPU instruction sets like AES/AESNI and PCLMULQDQ are suppressed. As mentioned in the various VMware KB articles, it is possible, but unlikely, that an application running in a virtual machine would benefit from these features, and that the application performance would be lower as the result of using an EVC mode that does not include the features. DRS-FT integration and building block approach When EVC is enabled in vSphere 4.1, DRS is able to select an appropriate ESX host for placing FT-enabled virtual machines and is able to load-balance these virtual machines, resulting in a more load-balanced cluster which likely has positive effect on the performance of the virtual machines. More info can be found in the article “DRS-FT integration”. Equally interesting is the building block approach, by enabling EVC, architects can use predefined set of hosts and resources and gradually expand the ESX clusters. Not every company buys computer power per truckload, by enabling EVC clusters can grow clusters by adding ESX host with new(er) processor versions. One potential caveat is mixing hardware of different major generations in the same cluster, as Irfan Ahmad so eloquently put it “not all MHz are created equal”. Meaning that newer major generations offer better performance per CPU clock cycle, creating a situation where a virtual machine is getting 500 MHz on a ESX host and when migrated to another ESX host where that 500 MHz is equivalent to 300 MHz of the original machine in terms of application visible performance. This increases the complexity of troubleshooting performance problems. Recommendations? No performance loss will be likely when enabling EVC. By enabling EVC, DRS-FT integration will be supported and organizations will be more flexible with expanding clusters over longer periods of time, therefor recommending enabling EVC on clusters. But will it be a panacea to stream of new major CPU generation releases? Unfortunately not! A possibility is to treat the newest hardware (Major releases) as a higher service as the older hardware and because of this create new clusters ================================================================================ Title: European distributor for HA and DRS book URL: https://frankdenneman.ai/2010-12-09-european-distributor-for-ha-and-drs-book/ Date: 2010-12-09 As of today, our book “vSphere 4.1 HA and DRS Technical Deepdive” can be ordered via ComputerCollectief. Computercollectief is a dutch computer book and software reseller and ships to most European countries. Using Computercollectief, we hope to evade the long shipping times and accompanying costs. Go check it out. http://www.comcol.nl/detail/73133.htm Comcol expect to be able to deliver at the end of this month. ================================================================================ Title: Dutch vBeers URL: https://frankdenneman.ai/2010-12-06-dutch-vbeers/ Date: 2010-12-06 Simon Long of The SLOG is introducing vBeers to Holland. I’ve copied the text from his vBeers blog article. Every month Simon Seagrave and I try organise a social get together of like-minded Virtualization enthusiasts held in a pub in central London (and Amsterdam). We like to call it vBeers. Before I go on, I would just like to state, although it’s called vBeers, you do NOT have to drink beer or any other alcohol for that matter. This isn’t just an excuse to get blind drunk. We came up with idea whilst on the Gestalt IT Tech Field Day back in April. We were chatting and we both recognised that we don’t get together enough to catch-up, mostly do to busy work schedules and private lives. We felt that if we had a set date each month, the likely hood of us actually making that date would be higher than previous attempts. So the idea of vBeers was born. The first Amsterdam vBeers will be held on Thursday 16th of December starting at 6:30pm in ‘Herengracht Cafe’ which is placed close to Leidseplein and Dam Square. This venue serves a fine of selection of beers along with soft drinks and bar food. Drinks will not be paid for, there will not be a tab. When you buy a drink please pay for it as no one else will be paying for your drinks. * Location: The ‘Herengracht Cafe’ Amsterdam * Address: Herengracht 435, Herengracht/Leidsestraat * Nearest Tram Station: Koningsplein – Lijn 1,2,5 * Time: 6:30pm * Location: Map ================================================================================ Title: HA and DRS Technical deepdive available URL: https://frankdenneman.ai/2010-12-06-ha-and-drs-technical-deepdive-available/ Date: 2010-12-06 After spending almost a year on writing, drawing and editing, the moment Duncan and I waited for finally arrived… Our new book, the vSphere 4.1 HA and DRS technical deepdive is available on CreateSpace and Amazon.com. Early this year Duncan approached me and asked me if I was interested in writing a book together on HA and DRS, without hesitation I accepted the honor. Before discussing the contents of the book I would like take the opportunity to thank our technical reviewers for their time, their wisdom and their input: Anne Holler (VMware DRS Engineering), Craig Risinger (VMware PSO), Marc Sevigny (VMware HA Engineering) and Bouke Groenescheij (Jume.nl). And a very special thanks to Scott Herold for writing the foreword! But most of all I would like to thank Duncan for giving me this opportunity to work together with him on creating this book. The in-depth discussions we had are without a doubt the most difficult I have ever experienced and were very interesting, both most of all fun! Thanks! Now let’s take a look at the book.Please note that we are still working on an electronic version of the book and we expect to finish this early 2011. This is the description of the book that is up on CreateSpace: About the authors: Duncan Epping (VCDX 007) is a Consulting Architect working for VMware as part of the Cloud Practice. Duncan works primarily with Service Providers and large Enterprise customers. He is focused on designing Public Cloud Infrastructures and specializes in bc-dr, vCloud Director and VMware HA. Duncan is the owner of Yellow-Bricks.com, the leading VMware blog. Frank Denneman (VCDX 029) is a Consulting Architect working for VMware as part of the Professional Services Organization. Frank works primarily with large Enterprise customers and Service Providers. He specializes in Resource Management, DRS and storage. Frank is the owner of frankdenneman.ai which has recently been voted number 6 worldwide on vsphere-land.com VMware vSphere 4.1 HA and DRS Technical Deepdive zooms in on two key components of every VMware based infrastructure and is by no means a “how to” guide. It covers the basic steps needed to create a VMware HA and DRS cluster, but even more important explains the concepts and mechanisms behind HA and DRS which will enable you to make well educated decisions. This book will take you in to the trenches of HA and DRS and will give you the tools to understand and implement e.g. HA admission control policies, DRS resource pools and host affinity rules. On top of that each section contains basic design principles that can be used for designing, implementing or improving VMware infrastructures. Coverage includes: • HA node types • HA isolation detection and response • HA admission control • VM Monitoring • HA and DRS integration • DRS imbalance algorithm • Resource Pools • Impact of reservations and limits • CPU Resource Scheduling • Memory Scheduler • DPM We hope you will enjoy reading it as much as we did writing it. Thanks, Eric Sloof received a proof copy of the book and shot a video about it. ================================================================================ Title: Should or Must VM-Host affinity rules? URL: https://frankdenneman.ai/2010-12-01-vm-host-affinity-rules-should-or-must/ Date: 2010-12-01 VMware vSphere 4.1 introduces a new affinity rule, called “Virtual Machines to Hosts” (VM-Host), which I described in the article “VM to Host affinity rule”. A short recap: VM-Host affinity rules are available in two flavors: Must run rules (Mandatory) Should run rules (Preferential) By providing these two options a new problem arises for the administrator\architect, when will the need occur for using the mandatory rule and when is it desired to use preferential rules? I think it all depends on the risk and limitations introduced by each rule. Let’s review difference between the rules, the behavior of each rule and the impact they have on cluster services and maintenance mode. What is the difference between a mandatory and a preferential rule? A mandatory rule limits HA, DRS and the user in such a way that a virtual machine may not be powered on or moved to a ESX host that does not belong to the associated DRS host group. A preferential rule defines a preference to DRS to run virtual machine on the host specified in the associated DRS host group. How does HA treat preferential rules? VMware High Availability respects mandatory rules and obey mandatory rules when placing virtual machines after a host failover. It can only place virtual machines on the ESX hosts that are specified in the DRS host group. DRS does not communicate the existence of preferential rules to HA, therefore HA is not aware of these rules. HA cannot prevent placing the virtual machine on a ESX host that is not a part of the DRS host group, thereby violating the affinity rule. DRS will correct this violation during the next invocation. How does DRS treat preferential rules? During a DRS invocation, DRS runs the algorithm with preferential rules as mandatory rules and will evaluate the result. If the result contains violations of cluster constraints; such as over-reserving a host or over-utilizing a host leading to 100% CPU or Memory utilization, the preferential rules will be dropped and the algorithm is run again. Limitations In essence a VM-Host affinity rule restricts the number of hosts on which the virtual machines may be powered-on or to which virtual machines may migrate. Setting VM-Host affinity rules can limit load-balancing and evacuation for maintenance mode. Load-balancing limitations A certain level of risk is introduced when Mandatory VM-Host affinity rules are used. As a result of the restrictive behavior by only allowing virtual machines to start on ESX host associated in the DRS host group, it impacts HA’s ability to select a compatible ESX host to place the virtual machine. In addition, using mandatory VM-Host affinity rules reduce the virtual machine placement options used by DRS when defragmenting the cluster. When using the HA “Percentage based” admission control resource fragmentation could occur. During a failover a defragmentation will be requested by HA from DRS. DRS tries to migrate virtual machines to regain enough unfragmented resources to fit and start all virtual machines. Because DRS is allowed to use “multi-hop” migrations, DRS calculations usually creates “chain” of migrations during defragmentation of a host. For example: VM-A migrates to host 2 and VM-B migrates from host 2 to host 3. Mandatory rules narrow the playing field, allowing VM to only move around in associated DRS host group, reducing the overall options to transport virtual machines around the cluster, regardless of association with VM-host affinity rules. Maintenance mode DRS will not violate CPU and memory reservation to obey mandatory VM-Host affinity rules and it will not violate mandatory rules to allow reservations to be honored. During placement both requirements must be met and therefore DRS will only place a virtual machine if its reservation and the mandatory rule can be satisfied. This behavior will impact the ability of DRS to select a suitable compatible host to the place virtual machines during maintenance mode automated evacuations. Conclusion Well, it’s up to you to decide which rule is appropriate to use for separating workloads across the ESX hosts in the cluster. By knowing the impact and limitations introduced by mandatory rules, one might be able to make an informed decision. ================================================================================ Title: Disallowing multiple vm console sessions URL: https://frankdenneman.ai/2010-11-30-disallowing-multiple-vm-console-sessions/ Date: 2010-11-30 Currently I’m involved in a high-secure virtual infrastructure design and we are required to reduce the number of entry points to the virtual infrastructure. One of the requirements is to allow only a single session to the virtual machine console. Due to the increasing awareness \ demand of security in virtual infrastructure more organizations might want to apply this security setting. 1. Turn of the virtual machine. 2. Open Configuration parameters of the VM to edit the advanced configuration settings 3. Add Remote.Display.maxConnections with a value of 1 4. Power on virtual machine Update: Arne Fokkema created a Power-CLI function to automate configuring this setting throughout your virtual infrastructure. You can find the power-cli function on ICT-freak.nl. ================================================================================ Title: Disable ballooning? URL: https://frankdenneman.ai/2010-11-29-disable-ballooning/ Date: 2010-11-29 Recently, Paul Meehan submitted this question via a comment on the “Memory reclamation, when and how” article: Hi, we are currently considering virtualising some pretty significant SQL workloads. While the VMware best practices documents for SQL server inside VMware recommend turning on ballooning, a colleague who attended a deep dive with a SQL Microsoft MVP came back and the SQL guy strongly suggested that ballooning should always be turned off for SQL workloads. We have 165 SQL instances, some of which will need 5-10000 IOPS so performance and memory management is critical. Do you guys have a view on this from experience? Thx, Paul I receive this kind of question a lot, whether it is SQL, Oracle or Citrix. And there always seem to be expert that is recommending disabling ballooning. Now this statement can be interpreted in two ways. 1. Disable the memory reclaimation mechanism by adding a particular parameter (sched.mem.maxmemctl) in the settings of the virtual machine. - or - 2. Ensure that enough physical memory resources are available to the virtual machine to keep the VMkernel from reclaiming memory of that particular virtual machine. I always hope that they mean guarantee enough memory to the virtual machine to stop the VMkernel from reclaiming memory from that specific VM. But unfortunately most specialists insist on disabling the mechanism. Why is disabling the ballooning mechanism bad? Many organizations that deploy virtual infrastructures rely on memory overcommitment to reach a higher consolidation ratio and higher memory utilization. In a virtual infrastructure not every virtual machine is actively using its assigned memory at the same time and not every virtual machine is making use of its configured memory footprint. To allow memory overcommitment, the VMkernel uses different virtual machine memory reclamation mechanisms. 1. Transparent Page Sharing 2. Ballooning 3. Memory compression 4. Host swapping Except from Transparent Page Sharing, all memory reclamation techniques only become active when the ESX host experiences memory contention. The VMkernel will use a specific memory reclamation technique depending on the level of the host free memory. When the ESX host has 6% or less free memory available it will use the balloon driver to reclaim idle memory from virtual machines. The VMkernel selects the virtual machines with the largest amounts of idle memory (detected by the idle memory tax process) and will ask the virtual machine to select idle memory pages. Now to fully understand the beauty of the balloon driver, it’s crucial to understand that the VMkernel is not aware of the Guest OS internal memory management mechanisms. Guest OS’s commonly use an allocated memory list and a free memory list. When a guest OS makes a request for a page, ESX will back that page with physical memory. When the Guest OS stops using the page internally, it will remove the page from the allocated memory list and place it on the free memory list. Because no data is changed, ESX will keep storing this data in physical memory. When the Balloon driver is utilized, the balloon driver request the guest OS to allocated a certain amount of pages. Typically the Guest OS will allocate memory that has been idle or registered in the Guest OS free list. If the virtual machine has enough idle pages no guest-level paging or even worse kernel level paging is necessary. Scott Drummonds tested an Oracle database VM against an OLTP load generation tool and researched the (lack of) impact of the balloon driver on the performance of the virtual machine. The results are displayed in this image: [caption id=“attachment_1367” align=“aligncenter” width=“300” caption=“Impact on performance: Ballooning versus swapping”][/caption] Scott’s explanation: Results of two experiments are shown on this graph: in one memory is reclaimed only through ballooning and in the other memory is reclaimed only through host swapping. The bars show the amount of memory reclaimed by ESX and the line shows the workload performance. The steadily falling green line reveals a predictable deterioration of performance due to host swapping. The red line demonstrates that as the balloon driver inflates, kernel compile performance is unchanged. So the beauty of ballooning lies in the fact that it allows the guest OS itself to make the hard decision about which pages to be paged out without the hypervisor’s involvement. Because the Guest OS is fully aware of the memory state, the virtual machine will keep on performing as long as it has idle or free pages. When ballooning is disabled When we follow the recommendations of Non-VMware experts we would disable ballooning resulting in the following available memory reclamation techniques: 1. Transparent Page Sharing 2. Memory compression 3. Host-level swapping (.vswp) Memory compression Memory compression is offered in vSphere 4.1. The VMkernel will always try to compress memory before swapping. This feature is very helpful and a lot faster than swapping. However, the VMkernel will only compress a memory page if the compression ratio is 50% or more, otherwise the page will be swapped. Furthermore, the default size of the compression cache is 10%, if the compression cache is full, one compressed page must be replaced in order to make room for a new page. The older page will be swapped out. During heavy contention memory compression will become the first stop before ultimately ending up as a swapped page. Increasing the memory compression cache can have a contradictive effect, as the memory compression cache is a part of the virtual machine memory usage, introducing memory pressure or contention due to configuring large memory compression caches. Host-level Swapping In contrast to ballooning, host-level swapping does not communicate with the Guest OS. The VMkernel has no knowledge about the status of the page in the Guest OS only that the physical page belongs to a specific virtual machine. Because the VMkernel is unaware of the content of the stored data inside the page and its significance to the Guest OS, it could happen that the VMkernel decides to swap out specific Guest OS kernel pages. Guest OS kernel pages will never be swapped by the Guest OS itself as they are crucial to maintaining kernel performance. So by disabling ballooning, you have just deactivated the the most intelligent memory reclamation technique. Leaving the VMkernel with the option to either compress a memory page or just rip out complete random (and maybe crucial) page, significantly increasing the possibility of deteriorating the virtual machine performance. Which to me does not sound something worth recommending. Alternative to disabling the balloon driver while guaranteeing performance? The best option to guarantee performance is to use the resource allocation settings; shares and reservations. Use shares to define priority levels and use reservations to guarantee physical resources even when the VMkernel is experiencing resource contention. How reservations work are described in the articles: Setting reservations does have impact on the virtual infrastructure, described in the articles “Impact of Memory reservations” and “Resource Pools memory reservations” and “Reservations and CPU scheduling”. However setting reservations will impact the virtual infrastructure, a well know impact of setting a reservation is on the HA slot size if the cluster is configured with “Host failures cluster tolerates”. More info on HA can be found in the HA deep dive on yellow-bricks. To circumvent this impact one might choose to configure the HA cluster with the HA policy “Percentage of cluster resources reserved as fail over spare capacity”. Due to the HA-DRS integration introduced in vSphere 4.1 the main caveat of dealing with defragmented clusters is dissolved. Disabling the balloon-driver will likely worsen the performance of the virtual machine when the ESX host experiences resource contention. I suspect that the advice given by other-vendor-experts is to avoid memory reclamation and the only two build-in recommended mechanisms to help avoid memory reclaimation are the resource allocation unit settings: Shares and Reservations. ================================================================================ Title: The impact of QoS network traffic on VM performance URL: https://frankdenneman.ai/2010-11-18-the-impact-of-qos-network-traffic-on-vm-performance/ Date: 2010-11-18 A lot of interesting material is written about configuring Quality of Service (QoS) on 10GB (converged) networks in Virtual Infrastructures. With the release of vSphere 4.1, VMware introduced a network QoS mechanism called Network I/O Control (NetIOC). The two most popular Blade systems; HP with Flex10 technology and Cisco UCS both offer traffic shaping mechanisms at hardware level. Both NetIOC and Cisco UCS approach network Quality of Service with a sharing perspective, guaranteeing a minimum amount of bandwidth opposed to the HP Flex-10 technology, which isolates the available bandwidth and dedicate an X amount of bandwidth to a specified NIC. When allocating bandwidths to the various network traffic streams most admins try to stay on the safe side and over-allocate bandwidth to virtual machine traffic. Obviously it is essential to guarantee enough bandwidth to virtual machines but bandwidth is finite, resulting in less bandwidth available to other types of traffic such as vMotion. Unfortunately by reducing the available bandwidth used for vMotion traffic can ultimately have negative effect on the performance of the virtual machines. MaxMovesPerHost In vSphere 4.1 DRS uses an adaptive technique called MaxMovesPerHost. This technique allows DRS to decide the optimum concurrent vMotions per ESX host for Load-Balancing operations. DRS will adapt the maximum concurrent vMotions per host (8) based upon the average migration time observed from previous migrations. Decreasing bandwidth available for vMotion traffic can result in a lower number of allowed concurrent vMotions.In turn the amount of allowed concurrent vMotions affects the number of migration recommendations generated by DRS. DRS will only calculate and generate the amount of migration recommendation is believes it can complete before the next DRS invocation. It limits the amount of generated migration recommendations, as there is no advantage in generating recommending migrations that cannot be complement before the next DRS invocation. During the next re-evaluation cycle, virtual machine resource demand can have changed rendering the previous recommendations obsolete By limiting the amount of bandwidth available to vMotion, it can decrease the maximum amount of concurrent vMotions per host and could risk leaving the cluster imbalanced for a longer period of time. Both NetIOC and Cisco UCS Class of Service (COS) Quality of Service can be used to guarantee a minimum amount of bandwidth available to vMotion during contention. Both techniques allow vMotion traffic to use all the available bandwidth if no contention occurs. HP uses a different approach, isolating and dedicating a specific amount of bandwidth to an adapter and thereby possible restricting specific workloads. Bred Hedlund wrote an article explaining the fundamental differences in how bandwidth is handled between HP Flex-10 and Cisco UCS. Cisco UCS intelligent QoS vs. HP Virtual Connect rate limiting Recommendations for Flex-10 Due to the restrictive behavior of Flex-10, it is recommended to specifically take the adaptive nature of DRS into account and not restricting vMotion traffic too much when shaping network bandwidth for the configured FlexNics. It is recommended to monitor the bandwidth requirements of the virtual machines and adjust the rate limit for virtual machine traffic and vMotion traffic accordingly, reducing the possibility of delaying DRS to reach a steady state when a significant load imbalance in the cluster exits. Recommendations for NetIOC and UCS QoS Fortunately the sharing nature of NetIOC and UCS allow other network streams allocate bandwidth during periods without bandwidth contention. Despite this “plays well with other” nature, it is recommended to assign a minimum guarantee amount of bandwidth for vMotion traffic (NetIOC) or a custom Class of Service to the vMotion vNICs (UCS). Chances are that if virtual machines saturate the network, virtual machines are experiencing a high workload and DRS will try to provide the resources the virtual machines are entitled to. ================================================================================ Title: vSwitch Failback and High Availability URL: https://frankdenneman.ai/2010-10-22-vswitch-failover-and-high-availability/ Date: 2010-10-22 One setting most admins get caught off-guard is vSwitch Failback setting in combination with HA. If the management network vSwitch is configured with Active/Standby NICs and the HA isolation response is set to “Shutdown” VM or “Power-off” VM it is advised to set the vSwitch Failback mode to No. If left at default (Yes), all the ESX hosts in the cluster or entire virtual infrastructure might issue an Isolation response if one of the management network physical switches is rebooted. Here’s why: Just a quick rehash: Active\Standby One NIC (vmnic0) is assigned as active to the management\service console portgroup, the second NIC (vmnic1) is configured as standby. The vMotion portgroup is configured with the first NIC (vmnic0) in standby mode and the second NIC as Active (vmnic1). [caption id=“attachment_1344” align=“aligncenter” width=“316” caption=“Active Standby setup management network vSwitch0”][/caption] Failback The Failback setting determines if the VMkernel will return the uplink (NIC) to active duty after recovery of a downed link or failed NIC. If the Failback setting is set to Yes the NIC will return to active duty, when Failback is set to No the failed NIC is assigned the Standby role and the administrator must manually reconfigure the NIC to the active state. Effect of Failback yes setting on environment When using the default setting of Failback unexpected behavior can occur during maintenance of a physical switch. Most switches, like those from Cisco, initiate the port after boot, so called Lights on. The port is active but is still unable to receive or transmitting data. The process from Lights-on to forwarding mode can take up to 50 seconds; unfortunately ESX is not able to distinguish between Lights-on status and forwarding mode, there for treating the link as usable and will return the NIC to active status again. High Availability will proceed to transmit heartbeats and expect to receive heartbeats, after missing 13 seconds of heartbeats HA will try to ping its Isolation Address, due to the specified Isolation respond it will shut down or power-off the virtual machines two seconds later to allow other ESX hosts to power-up the virtual machines. But because it is common – recommended even – to configure each host in the cluster in an identical manner, each active NIC used by the management network of every ESX host connect to the same physical switch. Due to this design, once the switch is booted, a cluster wide Isolation response occurs resulting in a cluster wide outage. To allow switch maintenance, it’s better to set the vSwitch failback mode to No. Selecting this setting introduces an increase of manual operations after failure or certain maintenance operations, but will reduce the change of “false positives” and cluster-wide isolation responses. ================================================================================ Title: Best practices URL: https://frankdenneman.ai/2010-10-21-best-practices/ Date: 2010-10-21 Last week at VMworld and the VCDX defense panels I heard the term “Best Practices” a lot. The term best practice makes me feel happy, shudder and laugh at the same time. Now when it comes to applying best practice I always use the analogy of crossing the road: I am born and raised in the Netherlands and best practice is to look left first, then to the right and finally check left again before crossing the road. This best practice served me well and it helped me avoid being hit by a car/truck/crazy people on bikes and even trams and trolleys. But I ask you does this best practice still apply when I try to cross the street in London? Don’t get me wrong, best practice are useful and very valuable, but to apply a best practice blindly won’t be as lethal as my analogy but it can get you into a lot of trouble. ================================================================================ Title: VMworld vCloud Director Labs URL: https://frankdenneman.ai/2010-10-12-vmworld-vcloud-director-labs/ Date: 2010-10-12 Yesterday the VMworld Labs opened up to the public and if you want to take vCloud Director for a spin I’m recommending doing the following labs: Private cloud – Management: Lab 13 VMware vCloud Director Install and Config Lab 18 VMware vCloud Director Networking Private Cloud - Security: Lab20 VMware vShield Its best to complete Lab 18 vCloud Director Networking before doing VMware vShield Lab (Lab 20) because the terms and knowledge gained in Lab 18 will prepare you for Lab 20. Today the VMworld 2010 speaker sessions started and I strongly recommend Duncan’s session “BC7803 - Planning and Designing an HA Cluster that Maximizes VM Uptime” and Kit Colbert’s “ TA7750 - Understanding Virtualization Memory Management Concepts”. Go check it out. ================================================================================ Title: NUMA, Hyperthreading and NUMA.PreferHT URL: https://frankdenneman.ai/2010-10-07-numa-hyperthreading-and-numa-preferht/ Date: 2010-10-07 I received a lot of questions about Hyperthreading and NUMA in ESX 4.1 after writing the ESX 4.1 NUMA scheduling article. A common misconception is that Hyperthreading is ignored and therefore not used on a NUMA system. This is not entirely true and due to the improved Hyperthreading code on Nehalems, the CPU scheduler is programmed to use the HT feature more aggressively than the previous releases of ESX. The main reason why I think this misconception exists is the way the NUMA load balancer handles vCPU placement of vSMP virtual machine. Before continuing, let’s get our CPU elements nomenclature aligned, I’ve created a diagram showing all the elements: The Nehalem Hyperthreading feature is officially called Symmetric MultiThreading (SMT), the term HT and SMT are interchangeable. 1. An Intel Nehalem processor often called a CPU or package. 2. An Intel Nehalem processor contains 4 cores in one package. 3. Each core contains 2 threads if Hyperthreading is enabled. 4. A SMT Thread equals a logical processor. 5. A logical processor is translated in esxtop as a PCPU. 6. A vCPU is scheduled on a PCPU. 7. NUMA= Non-uniform Memory Access (Each Processor has its own local memory assigned) 8. LLC= Last Level Cache: Shared by Cores is last on-die cache memory before turning to Local memory. NUMA load balancer virtual machine placement During placement of a vSMP virtual machine, the NUMA load balancer assigns a single vCPU per CPU core and “ignores” the availability of SMT threads. As a result a 4-way vSMP virtual machine will be placed on four cores. In ESX 4.1 this virtual machine can be placed on one processor or on two processors, depending on the amount of cores on the processor or if set the advanced option numa.vcpu.maxPerMachineNode. When a virtual machine contains more vCPUs than the amount of cores the processor, this virtual machine will span across multiple processors (Wide-VM). The default policy is to span the virtual machine across as few processors (NUMA nodes) as possible, but this can be overridden by an advanced option called numa.vcpu.maxPerMachineNode, which defines the maximum amount of vCPUs of a virtual machine per NUMA client. But as always, only use advanced options if you know the full impact of this setting on your environment. But I digress; let’s go back to NUMA and Hyperthreading. Now the key to understand is that only during placement the SMT threads are ignored by the NUMA load balancer. It is the up to the CPU scheduler to decide in which way it will schedule the vCPUs within the core. It can allow the vCPU to use the full core or schedule it on a SMT thread depending on the workload, resource entitlement, the amount of active vCPUs and available pCPUs in the system. Because SMT threads share resources within a core will result into lesser performance than running a vCPU on a dedicated singe core. The ESX scheduler is designed in such a way that it will try to spread the load across all the cores in the NUMA node or in the server. But basically, If the workload is low it will try to schedule the vCPU on a complete core, if that’s not possible, it will schedule the vCPU on a SMT thread. As mentioned before, running a vCPU on a SMT thread will not offer the same progress than running on a complete core; therefore a different charging scheme is used for each scenario. This charging scheme is used to keep track of the delivered resources and to check if the VM gets it entitled resources, more on this topic can be found in the article “Reservations and CPU scheduling”. NUMA.preferHT=One NUMA node to rule them all? Although the CPU scheduler can decide how to schedule the vCPU within the core, it will only schedule one vCPU of a vSMP virtual machine onto one core. Scott Drummonds article about numa.preferHT might offer a solution. Setting the advanced parameter numa.preferHT=1 allows the NUMA load balancer to assign vCPU to SMT thread and if possible “contain” one vSMP VM into a single NUMA node. However the amount of vCPU must be less or equal than the amount of pCPUs within the NUMA node. By placing all vCPUs within a processor a virtual machine with a “intensive-cache-footprint” workload can benefit from a “warmed-up” cache. The vCPUs can fetch the memory from Last Level Cache instead of turning to local memory resulting in less latency. And this is exactly why this setting might not be beneficial to most environments. The numa.preferHT setting is a CPU scheduler wide setting, that means that the NUMA load balancer will place every vSMP virtual machine inside a processor i.e. both intensive cache workloads and low-cache footprint workloads. Currently the ESX 4.1 CPU scheduler does not detect different workloads so it cannot distinguish virtual machines from each other and select an appropriate placement method i.e. place the virtual machine within one processor and use SMT threads or use wide-VM numa placement and “isolate” a vCPU per core. It is crucial to know that by placing all vCPU on one processor doesn’t guarantee it to have all its memory in local memory, the main goal is to use LLC as much as possible, but if there is a cache miss (memory not available in cache) it will fetch it from local memory. The VMkernel tries to keep memory as local as possible but if there is not enough room inside local memory, it will place the memory into remote memory. Storing memory in remote memory is still faster than swapping it out to disk but inter-socket communication is noticeable slower than intra-socket communications. This brings me to migration of virtual machines between NUMA nodes, if a virtual machines home node is more heavily loaded than other NUMA nodes, it will be migrated to a less loaded NUMA node. During the migration phase, local memory turns into remote memory. This newly remote memory is moved gradually because moving memory has high overhead. By using the numa.preferHT option forces you to scope the maximum amount of memory assigned to a virtual machine and the consolidation ratio. Having multiple virtual machine traverse the quick path interlink to fetch memory stored in remote memory defeats the purpose of containing the virtual machines inside a processor. ================================================================================ Title: VM settings: Prefer Partially Automated over Disabled URL: https://frankdenneman.ai/2010-10-01-vm-settings-prefer-partially-automated-over-disabled/ Date: 2010-10-01 Due to requirements or constraints it might be necessary to exclude a virtual machine from automatic migration and stop it from moving around by DRS. Use the “Partially automated” setting instead of “Disabled” at the individual virtual machine automation level. Partially automated blocks automated migration by DRS, but keep the initial placement function. During startup, DRS is still able to select the most optimal host for the virtual machine. By selecting the “Disabled” function, the virtual machine is started on the ESX server it is registered and chances of getting an optimal placement are low(er). An exception for this recommendation might be a virtualized vCenter server, most admins like to keep track of the vCenter server in case a disaster happens. After a disaster occurs, for example a datacenter-wide power-outage, they only need to power-up the ESX host on which the vCenter VM is registered and manually power-up the vCenter VM. An alternative to this method is to keep track of the datastore vCenter is placed on and register and power-on the VM on a (random) ESX host after a disaster. Slightly more work than disabling DRS for vCenter, but offers probably better performance of the vCenter Virtual Machine during normal operations. Due to expanding virtual infrastructures and new additional features, vCenter is becoming more and more important for day-to-day operational management today. Assuring good performance outweighs any additional effort necessary after a (hopefully) rare occasion, but both methods have merits. ================================================================================ Title: Voted number 6 of top 25 VMware blogs, WOW! URL: https://frankdenneman.ai/2010-10-01-voted-number-6-of-top-25-vmware-blogs-wow/ Date: 2010-10-01 Eric Siebert of vSphere-land, together with David Davis, Simon Seagrave and John Troyer announced the results of the Top 25 blogs election this week. Using vChat is in my opinion a very cool format and I had the feeling that I was watching the academy awards for Bloggers. Gentlemen, thank you for taking the time and effort to create this entertaining show. Next time I will be viewing this from my TV instead of my 13" laptop screen and making sure I’ll have the popcorn ready. But most of all I want to thank everyone who voted for me. According to you my blog belongs in the top 10 and I’m very proud and honored to be ranked up that high. Thank you very much! ================================================================================ Title: Provider vDC: cluster or resource pool? URL: https://frankdenneman.ai/2010-09-24-provider-vdc-cluster-or-resource-pool/ Date: 2010-09-24 Duncan’s article on vCloud Allocation models states that: a provider vDC can be a VMware vSphere Cluster or a Resource Pool … Although vCloud Director offers the ability to map Provider vDCs to Clusters or Resource Pool, it might be better to choose for the less complex solution. This article zooms in on the compute resource management constructs and particularly on making the choice between assigning a VMware Cluster or a Resource Pool to a Provider vDC and placement of Organization vDCs. I strongly suggest visiting Yellow Bricks to read all vCloud Director posts, these posts explain the new environment / cloud model used by VMware very thoroughly. Let’s do a quick rehash of these elements before discussing whether to choose between a Cluster or Resource Pool based Provider vDC. Provider vDC and Organization vDC In the vCloud a construct named vDCs exist. vDCs stands for Virtual Data Center. Two types of vDCs exists; Provider vDCs and Organization vDCs. A Provider vDC is used to offer a single type of compute resources and a single type of storage resources. This means that Provider vDCs are created for segmenting resources based on resource characteristics (Tiering) or quantity of resources (Capacity). Basically a Provider vDC will function as a SLA construct in the vCloud. At the vSphere layer a VMware vSphere Cluster or Resource Pool can be used to provide the Provider vDC raw Virtual Infrastructure resources. Now the fun part is that using Resource Pools basically contradicts the whole idea behind a Provider vDC, but we will discuss that later. An Organization vDC (Org vDC) is an allocation out of the Provider vDC (pVDC), in other words the resources provided by the PvDC are consumed by the Org vDC. Organization vDCs inherit the resource types (Tiering\Capacity) from the Provider vDC. At the vSphere level this means that a Resource Pool is created per Org vDC and this will carve out resources from the Provider vDC using the resource allocation settings Reservation, Shares and Limit values for compute resources. Note: A vDC is not identical to a vSphere Resource Pool, a vDC provides storage additional to compute resources (leveraging resource pools) whether a resource pool only offers compute resources (CPU and Memory). Compute resource management is done at the vSphere level, Storage is enforced and maintained at the vCloud Director level. vCloud Director uses allocation models to define different usage levels of Reservation and Limits. The Share levels are identical throughout all allocation models and each model uses the normal share level setting. Allocation Models Each Organization vCD is configured with an allocation model, three models different types of allocation models exist. Pay As You Go Allocation Pool Reservation Pool Each allocation model has a unique set of resource allocation settings and each model uses both Resource Pool level and Virtual Machine level resource allocation settings differently. Read the vCD allocation models article on Yellow-Bricks.com. Note: Reservations on resource pool act differently than reservations on VM-level, for a refresher please read the articles: “Resource Pools memory reservations” and “Impact of memory reservations". In addition CPU type reservations behave differently from Memory reservations, please read the article “Reservations and CPU scheduling”. Now let’s visualize the difference between a PvDC aligned with a cluster and a pVDC aligned with a Resource Pool: Using Resource Pools instead of Clusters One thing immediately becomes obvious, when using a Resource Pool for providing Compute and Memory resources to the PvDC you share the cluster resources with other PvDCs. One might argue to create only one Resource Pool below Cluster level and create some sort of buffer, but creating a single Resource Pool below cluster level and assigning a PvDC to it will render a certain amount of cluster resources unused. By default, a Resource Pool can claim up to a maximum of 94% of its parent Resource Pool. By using multiple Provider vDCs in one cluster you abandon the idea of segmenting resources based on resource characteristics and quantity (Tiering and Capacity). Because a Resource Pool spans the entire cluster the PvDCs will schedule the virtual machine on every host available in the cluster. By using the Resource Pool model it introduces a whole new complex resource management construct all by itself. Let’s focus on the challenges this model will introduce: Resource Pool creation When creating a Provider vDC, a Cluster or Resource Pool must be selected, this means the Resource Pool must be manually configured before creating and mapping the Provider vDC to the Resource Pool. During the creation of this Resource pool, the admin must specify the resource allocation settings. The Reservation, Shares and Limit settings of a Resource Pool are not changed dynamically when adding additional ESX hosts to the cluster. The admin must change (increase) the reservation and Limit setting each time new hosts are added to the cluster. The second drawback of the RP model is sizing. Because multiple Provider vDC Resource Pools will exists beneath the Root Resource Pool (Cluster) level the admin/architect needs to calculate a proper resource allocation ratio for the existing Provider vDCs. Mapping a Provider vDC to a Resource pool result in manually recalculation the resource allocation settings each time a new tenant is introduced and when the new Org vDC joins the Provider vDC. Sibling Share Level If “Pay as You go” or “Allocation Pool” models are used, some resources might be provided via a “burstability” model. When creating an Organization vDC, a guaranteed amount of resources must be specified as well as an upper limit known as an “Allocation”. The difference between the total allocated resources and the specified guaranteed resources is a pool of resources that are available to that Organization vDC, however, it is important to note that those resources are not certain to be available at any given point in time. This is called the burstability space. These “burstable” resources are allocated based on Shares in times of contention. Shares specify the priority for the virtual machine or Resource Pool relative to other Resource Pools and/or virtual machines with the same parent in the resource hierarchy. The key point is that shares values can be compared directly only among siblings. This means that each Provider vDC is the sibling of another Provider vDC in the cluster and they will receive resources from its parent Resource Pool (Root Resource Pool) based on their Resource Entitlement. That means that this model: Translates into this model: Resource Entitlement Resource Pool and virtual machine resource entitlements are based on various statistics and some estimation techniques. DRS computes a resource entitlement for each virtual machine, based on virtual machine and Resource Pool configured shares, reservations, and limits settings, as well as the current demands of the virtual machines and Resource Pools, the memory size, its working set and the degree of current resource contention. As mentioned before, this burstable space is allocated based on the amount of shares and the active utilization (working set) when calculating the resource entitlement. Virtual machines who are idling aren’t competing for resources, so they won’t get any new resources assigned and therefore the Provider vDC will not demand it from the Root Resource Pool. Be aware that the resource entitlement is calculated at host level scheduling (VMkernel) and Global scheduling (DRS). DRS will create a pack (lump sum) of resources and divide this across the Resource pools and its children. This lump sum is recalculated every 5 minutes. Introducing an additional layer of Provider vDC Resource Pools between the cluster and the Organization vDC Resource Pools will not only complicate the resource entitlement calculation but will also create additional unnecessary overhead on DRS. Besides the 300 second invocation period, DRS also gets invocated each time when a virtual machine is powered-off, when a resource setting of a virtual machine or Resource Pool is changed or when a Resource Pool or a virtual machine is moved in or out the Resource Pool hierarchy. This is the reason why the Resource Pool tree must be as “flat” as possible; having additional layers will complicate the resource calculation and distribution. If you decide to map a Provider vDC to a Resource Pool is recommended allocating the amount of CPU and Memory resources of the pVDC Resource Pool identical to the combined amount of resources allocated to the Org vDCs. By accumulating all Org vDC allocation settings and setting the reservation on the Provider vDC equal to the result of that sum removes the burstable space on PvDC level. Only siblings inside the Provider vDC will have to compete for resources during contention. Placement of Organization vDCs in Provider vDCs Proper Resource management is very complicated in a Virtual Infrastructure or vCloud environment. Each allocation models uses a different combination of resource allocation settings on both Resource Pool and Virtual Machine level, therefore introducing different types of resource entitlement behavior. Mixing Allocation models inside a Provider vDC makes capacity management and capacity planning a true nightmare. It is advised to create a Provider vDC per Allocation Model. This means that (preferential) a Provider vDC is mapped to a Cluster and this cluster will host only “Pay As You Go”, “Allocation Pool” or “Reservation Pool” type Organization vDCs. Words of advice Using different allocation models within a Provider vDC can be a challenge to create a proper level of utilization and flexibility all by itself. Using Resource Pools to act as the compute Resource Pool construct for Provider vDCs makes it in my opinion incredibly complex. Using Resource Pools instead of Clusters deviates from the intention Provider vDCs are created (segmenting Tiering and Capacity). Although it’s possible to map Provider vDC to a Resource Pool it is wiser to map Provider vDCs to Cluster levels only. Avoid using different types of allocation models within a Provider vDC, mixing allocation models makes proper capacity management and capacity planning unnecessary difficult. Best practice: Map Provider vDC to a VMware vSphere Cluster. Usage of same type of Allocation model type Organization vDC inside a Provider vDC. ================================================================================ Title: Resource pools and simultaneous vMotions URL: https://frankdenneman.ai/2010-09-20-resource-pools-and-simultaneous-vmotions/ Date: 2010-09-20 Many organizations have the bad habit to use resource pools to create a folder structure in the host and cluster view of vCenter. Virtual machines are being placed inside a resource pool to show some kind of relation or sorting order like operating system or types of application. This is not reason why VMware invented resource pools. Resource pools are meant to prioritize virtual machine workloads, guarantee and/or limit the amount of resources available to a group of virtual machines. During design workshops I always try to convince the customer why resource pools should not to be used to create a folder structure. The main object I have for this is the sibling share level of resource pools and virtual machines. Shares specify the priority for the virtual machine or resource pool relative to other resource pools and/or virtual machines with the same parent in the resource hierarchy. The key point is that shares values can be compared directly only among siblings: the ratios of shares of VM6:VM7 tells which VM is higher priority, but the shares of VM4:VM6 does not tell which VM has higher priority. Many articles have been written about this, such as: “The resource pool priority-pie paradox”, (Craig Risinger) “Resource pools and shares” (Duncan Epping), “Don’t add resource pools for fun” (Eric Sloof) and “Resource pools caveats” (Bouke Groenescheij). But another reason not to use resource pools as a folder structure is the limitation resource pools inflict on vMotion operations. Depending on the network speed, vSphere 4.1 allows 8 simultaneous vMotion operations, however simultaneous migrations with vMotion can only occur if the virtual machine is moving between hosts in the same cluster and is not changing its resource pool. This is recently confirmed in Knowledge Base article [1026102](http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1026102&sliceId=1&docTypeID=DT_KB_1_1&dialogID=111208416&stateId=0 0 116148556). Fortunately simultaneous cross-resource-pool vMotions can occur if the virtual machines are migrating to different resource pools, but still one vMotion operation per target resource pool. Because clusters are actually implicit resource pools (the root resource pool), migrations between clusters are also limited to a single concurrent vMotion operation. Using resource pools to create a folder structure can not only impact the availability of resources for the virtual machines, but can also hinder your daily (maintenance) operations if batches of virtual machines are being migrated to other resource pools. ================================================================================ Title: 2 days left to vote URL: https://frankdenneman.ai/2010-09-19-5-days-to-vote/ Date: 2010-09-19 Eric Siebert owner of vSphere-land.com started the second round of the bi-annual top 25 VMware virtualization blogs voting. The last voting was back in January and this is your chance to vote for your favorite virtualization bloggers and help determining the top 25 blogs of 2010. This year my blog got nominated for the first time and entered the top 25 (no. 14) and I hope to stay in the top 25 after this voting round. My articles tend to focus primarily on resource management and cover topics such as DRS, CPU and memory scheduler to help you make an informed decision when designing or managing a virtual infrastructure. As noble as this may sound I know that these kinds of topics are not mainstream and I can understand that not everybody is interested to read about these topics week in week out. Fortunately I’ve managed to get a blog post listed in the Top 5 Planet V12n blog post list at least once every month and referred on a regular basis by sites like Yellow-bricks.com (Duncan Epping), Scott Lowe, NTpro.nl (Eric Sloof) and Chad Sakac and of course many others. So it seems I’m doing something right. This is my list of top 10 articles I’ve created this year: ESX 4.1 NUMA scheduling vCloud Director Architecture VM to Host Affinity rule Memory Reclaimation, when and how? Reservations and CPU scheduling Resource pools and memory reservations ESX ALUA, TPGS and HP CA Removing an orphaned Nexus DVS Impact of Host local swap on HA and DRS Sizing VMs and NUMA nodes Closing remarks This year I’ve got to personally know a lot of bloggers and one thing that amazed me was the time and effort that each of these bloggers put into their work in their spare time. Please take a couple of minutes to vote at vSphere-land whether it’s for me or any of the other bloggers listed and reward them for their hard work. ================================================================================ Title: ESX 4.1 NUMA Scheduling URL: https://frankdenneman.ai/2010-09-13-esx-4-1-numa-scheduling/ Date: 2010-09-13 VMware has made some changes to the CPU scheduler in ESX 4.1; one of the changes is the support for Wide virtual machines. A wide virtual machine contains more vCPUs than the total amount of available cores in one NUMA node. Wide VM’s will be discussed after a quick rehash of NUMA. NUMA NUMA stands for Non-Uniform Memory Access, which translates into a variance of memory access latencies. Both AMD Opteron and Intel Nehalem are NUMA architectures. A processor and memory form a NUMA node. Access to memory within the same NUMA node is considered local access, access to the memory belonging to the other NUMA node is considered remote access. Remote memory access is slower, because the instructions has to traverse a interconnect link which introduces additional hops. Like many other techniques and protocols, more hops equals more latency, therefore keeping remote access to a minimum is key to good performance. (More info about NUMA scheduling in ESX can be found in my previous article “Sizing VM’s and NUMA nodes".) If ESX detects its running on a NUMA system, the NUMA load balancer assigns each virtual machine to a NUMA node (home node). Due to assigning soft affinity rules, the memory scheduler preferentially allocates memory for the virtual machine from its home node. In previous versions (ESX 3.5 and 4.0) the complete virtual machine is treated as one NUMA client. But the total amount of vCPUs of a NUMA client cannot exceed the number of CPU cores of a package (physical CPU installed in a socket) and all vCPUs must reside within the NUMA node. If the total amount of vCPUs of the virtual machine exceeds the number of cores in the NUMA node, then the virtual machine is not treated as a NUMA client and thus not managed by the NUMA load balancer. Because the VM is not a NUMA client of the NUMA load balancer, no NUMA optimization is being performed by the CPU scheduler. Meaning that the vCPUs can be placed on any CPU core and memory comes from either a single CPU or all CPUs in a round-robin manner. Wide virtual machines tend to be scheduled on all available CPUs. Wide-VMs The ESX 4.1 CPU scheduler supports wide virtual machines. If the ESX4.1 CPU scheduler detects a virtual machine containing more vCPUs than available cores in one NUMA node, it will split the virtual machine into multiple NUMA clients. At the virtual machine’s power on, the CPU scheduler determines the number of NUMA clients that needs to be created so each client can reside within a NUMA node. Each NUMA client contains as many vCPUs possible that fit inside a NUMA node. The CPU scheduler ignores Hyper-Threading, it only counts the available number of cores per NUMA node. An 8-way virtual machine running on a four CPU quad core Nehalem system is split into a two NUMA clients. Each NUMA client contains four vCPUs. Although the Nehalem CPU has 8 threads 4 cores plus 4 HT “threads”, the CPU scheduler still splits the virtual machine into multiple NUMA clients. The advantage of wide VM The advantage of a wide VM is the improved memory locality, instead of allocating memory pages random from a CPU, memory is allocated from the NUMA nodes the virtual machine is running on. While reading the excellent whitepaper: “VMware vSphere: The CPU Scheduler in VMware ESX 4.1 VMware vSphere 4.1 whitepaper” one sentence caught my eye: However, the memory is interleaved across the home nodes of all NUMA clients of the VM. This means that the NUMA scheduler uses an aggregated memory locality of the VM to the set of NUMA nodes. Call it memory vicinity. The memory scheduler receives a list (called a node mask) of the NUMA node the virtual machine is scheduled on. The memory scheduler will preferentially allocate memory for the virtual machine from this set of NUMA nodes, but it can distributed pages across all the nodes within this set. This means that there is a possibility that the CPU from NUMA node 1 uses memory from NUMA node 2. Initially this looks like no improvement compared to the old situation, but fortunately supporting Wide VM makes a big difference. Wide-VM’s stop large VM’s from scattering all over the CPU’s with having no memory locality at all. Instead of distributing the vCPU’s all over the system, using a node mask of NUMA nodes enables the VMkernel to make better memory allocations decisions for the virtual machines spanning the NUMA nodes. ================================================================================ Title: vCloud Director Architecture URL: https://frankdenneman.ai/2010-09-08-vcloud-director-architecture/ Date: 2010-09-08 A vCloud infrastructure consists of several components. Many of you have deployed, managed, installed or designed vSphere environments. The vCloud Director architecture introduces a new and additional layer on top of the vSphere environment. I have created a diagram which depicts architecture mainly to be used by service providers. Service providers are likely to use an internal cloud for their own application and IT department operations and an external cloud environment for their service delivery. The purpose of this diagram is to create a clear overview of the new environment and showing a relational overview of all the components. In essence, which component connects with which other components? The environment consists of a management cluster, an internal resource group and an external resource group. Building blocks (VMware vSphere, SAN and networking) within a vCD environment are often referred to as “resource groups”. To run a VMware vCloud environment, the minimum components to run are: • VMware vCloud Director (vCD) • VMware vShield Manager • VMware Chargeback • VMware vCenter (supporting the Resource groups) • vSphere infrastructure • Supporting infrastructure elements (Networking, SAN) The management cluster contains all the vCD components, such as the vCenters and Update managers managing the resource groups, the vCD cells and Chargeback clusters. Due to the Oracle Database requirement, a physical Oracle cluster is recommended, as RAC clustering is not supported on VMware vSphere. No vCD is used to manage the management cluster, management is done by a dedicated vCenter. Within a resource pod a separate cluster is created for each pVCD to enable the service provider to deliver the different service levels. Duncan wrote a great introduction to VCD recently. http://www.yellow-bricks.com/2010-08-31-vmware-vcloud-director-vcd/. This diagram shows two additional vCenters, one for the internal resource pod and one for the external resource pod. It is advised to isolate internal IT resources from customer IT resources. Managing and deploying internal IT services and external IT services such as customers VM from one vCD can become obscure and complex. A single vCenter is used for both pVCDs as it expected that customer will deploy virtual machines in different service level offerings. By using one vCenter a single Distributed Virtual Switch can be used, spanning both clusters\service offerings. To rehash, this is a abstract high level diagram intended to show the involved elements of a vCloud environment and to show the relation or connections all the components. ================================================================================ Title: DRS 4.1 Adaptive MaxMovesPerHost URL: https://frankdenneman.ai/2010-08-27-drs-4-1-adaptive-maxmovesperhost/ Date: 2010-08-27 Another reason to upgrade to vSphere 4.1 is the DRS adaptive MaxMovesPerHost parameter. The MaxMovesPerHost setting determines the maximum amount of migrations per host for DRS load balancing. DRS evaluates a cluster and recommends migrations. By default this evaluation happens every 5 minutes. There are limits to how many migrations DRS will recommend per interval per ESX host because there’s no advantage to recommending so many migrations that they won’t all be completed by the next re-evaluation, by which time demand could have changed anyway. Be aware that there is no limit on max moves per host for a host entering maintenance or standby mode, but there’s a limit on max moves per host for load balancing. This can (but usually shouldn’t) be changed by setting the DRS Advanced Option “MaxMovesPerHost”. The default value is 8 and is set at the Cluster level. Remember, the MaxMovesPerHost is a cluster setting but configures the maximum migrations from a single host on each DRS invocation. This means you can still see 30 or 40 vMotion operations in the cluster during a DRS invocation. In ESX/ESXi 4.1, the limit on moves per host will be dynamic, based on how many moves DRS thinks can be completed in one DRS evaluation interval. DRS adapts to the frequency it is invoked (pollPeriodSec, default 300 seconds) and the average migration time observed from previous migrations. In addition DRS follows the new maximum number of concurrent vMotion operations per host over depending on the Network Speed (1GB – 4 vMotions, 10GB – 8 vMotions). Due to the adaptive nature the algorithm, the name of the setting is quite misleading as it’s no longer a maximum. The “MaxMovesPerHost” parameter will still exist, but its value might be exceeded by DRS. By leveraging the increased amount of concurrent vMotion operations per host and the evaluation of previous migration times DRS is able to rebalance the cluster in a fewer amount of passes. By using fewer amounts of passes, the virtual machines will receive their entitled resources much quicker which should positively affect virtual machine performance. ================================================================================ Title: vSphere 4.1 – HA and DRS deepdive book URL: https://frankdenneman.ai/2010-08-25-vsphere-4-1-ha-and-drs-deepdive-book/ Date: 2010-08-25 This is a complete repost of the article written by Duncan, as all publications about this book must be checked by VMware Legal, he wrote one article and speaks for the both of us. URL: http://www.yellow-bricks.com/2010-08-24-soon-in-a-bookstore-near-you-ha-and-drs-deepdive Over the last couple of months Frank Denneman and I have been working really hard on a secret project. Although we have spoken about it a couple of times on twitter the topic was never revealed. Months ago I was thinking about what a good topic would be for my next book. As I already wrote a lot of articles on HA it made sense to combine these and do a deepdive on HA. However a VMware Cluster is not just HA. When you configure a cluster there is something else that usually is enabled and that is DRS. As Frank is the Subject Matter Expert on Resource Management / DRS it made sense to ask Frank if he was up for it or not… Needless to say that Frank was excited about this opportunity and that was when our new project was born: VMware vSphere 4.1 - HA and DRS deepdive. As both Frank and I are VMware employees we contacted our management to see what the options were for releasing this information to market. We are very excited that we have been given the opportunity to be the first official publication as part of a brand new VMware initiative, codenamed Rome. The idea behind Rome along with pertinent details will be announced later this year. Our book is currently going through the final review/editing stages. For those wondering what to expect, a sample chapter can be found here. The primary audience for the book is anyone interested in high availability and clustering. There is no prerequisite knowledge needed to read the book however, the book will consist of roughly 220 pages with all the detail you want on HA and DRS. It will not be a “how to” guide, instead it will explain the concepts and mechanisms behind HA and DRS like Primary Nodes, Admission Control Policies, Host Affinity Rules and Resource Pools. On top of that, we will include basic design principles to support the decisions that will need to be made when configuring HA and DRS. I guess it is unnecessary to say that both Frank and I are very excited about the book. We hope that you will enjoy reading it as much as we did writing it. Stay tuned for more info, the official book title and url to order the book. Frank and Duncan ================================================================================ Title: VCDX tip: The application form URL: https://frankdenneman.ai/2010-08-23-vcdx-tip-the-application-form/ Date: 2010-08-23 Last week I reviewed some recently submitted designs and it appears that the requirements stated in the application form are too ambiguous. During this year I’ve seen many application forms and the same error are made by many candidates. Let’s go over the sections which contain the most errors and try to remove any doubts for future candidates. The VMware VCDX Handbook and application form is subject to change. So this article is based on version 1.0.5. The application form is available for candidates enrolled in the VCDX program. Section 4 Project References What deliverables were provided? (This should represent a comprehensive design package and include, at a minimum, the design, blueprints, test plan, assembly and configuration guide, and operations guide.) Ok so this requirement is not understood clearly by some. To meet this requirement you MUST submit at least: 1. the VMware VI 3.5 or vSphere design document. 2. blueprints (Visio drawings of physical and logical layout) 3. a documented test plan 4. a assembly and configuration guide 5. and a operation guide. This means you are required to submit those five listed documents otherwise your application is rejected (bad) or returned for rework (Still bad, but it doesn’t cost you 300 bucks and you might have a chance to defend during the upcoming defense panels). Section 5 Design Development Activities This section requires you to submit five requirements, assumptions and constrains that had to be followed within this design. This means you must submit at least five requirements, five assumptions and five constrains you encountered when working on the design. I’ve seen some application forms with requirements such as enough power, enough floor space and enough cables. Which are all genuine requirements if you are a project manager. We are requesting a list of requirements, assumptions and constraints which you as a virtual infrastructure architect had to deal with. The submitted design needs to align and deal with requirements and constraints listed in the application form. Design Deliverable Documentation: A small error made by many, no big deal if you miss this but it makes our live much easier if you do it correctly. This sections requires you to list the page numbers where the diagrams can be found not how many pages the document has. Design Decisions In this section you must provide four decision criteria for each of the decision areas, this means if you leave one field empty the application will be rejected. It’s just really simple; your application form is NOT completed when a field is empty. Not completed forms get rejected. Application form does not equal design document The application form is not a substitute for the design document. It is a part of the VCDX certification program and not a part of the VMware virtual infrastructure design. The two are not complimentary to each other. Everything stated in the application form must be included in the design document or any of the other documents. Just remember you are submitting a defense you have delivered to a real or imaginary customer! Ask yourself have you ever submitted a VCDX application form during a design project to your customer? ================================================================================ Title: Disable DRS and VM-Host rules URL: https://frankdenneman.ai/2010-07-22-disable-drs-and-vm-host-rules/ Date: 2010-07-22 vSphere 4.1 introduces DRS VM-Host Affinity rules and offer two types of rules, mandatory (must run on /must not run on) and preferential (should run on /should nor run on). When creating mandatory rules, all ESX hosts not contained in the specified ESX Host DRS Group are marked as “incompatible” hosts and DRS\VMotion tasks will be rejected if an incompatible ESX Host is selected. A colleague of mine ran into the problem that mandatory VM-Host affinity rules remain active after disabling DRS; the product team explained the reason why: By design, mandatory rules are considered very important and it’s believed that the intended user case which is licensing compliance is so important, that VMware decided to apply these restrictions to non-DRS operations in the cluster as well. If DRS is disabled while mandatory VM-Host rules still exist, mandatory rules are still in effect and the cluster continues to track, report and alert mandatory rules. If a VMotion would violate the mandatory VM-Host affinity rule even after DRS is disabled, the cluster still rejects the VMotion. Mandatory rules can only be disabled if the administrator explicitly does so. If it the administrator intent to disable DRS, remove mandatory rules first before disabling DRS. ================================================================================ Title: DRS-FT integration URL: https://frankdenneman.ai/2010-07-22-drs-ft-integration/ Date: 2010-07-22 Another new feature of vSphere 4.1 is the DRS-Fault Tolerance integration. vSphere 4.1 allows DRS not only to perform initial placement of the Fault Tolerance (FT) virtual machines, but also migrate the primary and secondary virtual machine during DRS load balancing operations. In vSphere 4.0 DRS is disabled on the FT primary and secondary virtual machines. When FT is enabled on a virtual machine in 4.0, the existing virtual machine becomes the primary virtual machine and is powered-on onto its registered host, the newly spawned virtual machine, called the secondary virtual machine is automatically placed on another host. DRS will refrain from generating load balancing recommendations for both virtual machines. The new DRS integration removes both the initial placement- and the load-balancing limitation. DRS is able to select the best suitable host for initial placement and generate migration recommendations for the FT virtual machines based on the current workload inside the cluster. This will result in a more load-balanced cluster which likely has positive effect on the performance of the FT virtual machines. In vSphere 4.0 an anti-affinity rule prohibited both the FT primary- and secondary virtual machine to run on the same ESX hosts based on an anti-affinity rule, vSphere 4.1 offers the possibility to create a VM-host affinity rule ensuring that the FT primary and secondary virtual machine do not run on ESX hosts in the same blade chassis if the design requires this. For more information about VM-Host affinity rules please visit this article. Not only has the DRS-FT integration a positive impact on the performance of the FT enabled virtual machines and arguably all other VMs in the cluster but it will also reduce the impact of FT-enabled virtual machines on the virtual infrastructure. For example, DPM is now able to move the FT virtual machine to other hosts if DPM decides to place the current ESX host in standby mode, in vSphere 4.0, DPM needs to be disabled on at least two ESX host because of the DRS disable limitation which I mentioned in this article. Because DRS is able to migrate the FT-enabled virtual machines, DRS can evacuate all the virtual machines automatically if the ESX host is placed into maintenance mode. The administrator does not need to manually select an appropriate ESX host and migrate the virtual machines to it, DRS will automatically select a suitable host to run the FT-enabled virtual machines. This reduces the need of both manual operations and creating very “exiting” operational procedures on how to deal with FT-enabled virtual machines during the maintenance window. DRS FT integration requires having EVC enabled on the cluster. Many companies do not enable EVC on their ESX clusters based on either FUD on performance loss or arguements that they do not intend to expand their clusters with new types of hardware and creating homogenous clusters. The advantages and improvement DRS-FT integration offers on both performance and reduction of complexity in cluster design and operational procedures shed some new light on the discussion to enable EVC in a homogeneous cluster. If EVC is not enabled, vCenter will revert back to vSphere 4.0 behavior and enables the DRS disable setting on the FT virtual machines. ================================================================================ Title: Load Based Teaming URL: https://frankdenneman.ai/2010-07-21-load-based-teaming/ Date: 2010-07-21 In vSphere 4.1 a new network Load Based Teaming (LBT) algorithm is available on the distributed virtual switch dvPort groups. The option “Route based on physical NIC load” takes the virtual machine network I/O load into account and tries to avoid congestion by dynamically reassigning and balancing the virtual switch port to physical NIC mappings. The three existing load-balancing policies, Port-ID, Mac-Based and IP-hash use a static mapping between virtual switch ports and the connected uplinks. The VMkernel assigns a virtual switch port during the power-on of a virtual machine, this virtual switch port gets assigned to a physical NIC based on either a round-robin- or hashing algorithm, but all algorithms do not take overall utilization of the pNIC into account. This can lead to a scenario where several virtual machines mapped to the same physical adapter saturate the physical NIC and fight for bandwidth while the other adapters are underutilized. LBT solves this by remapping the virtual switch ports to a physical NIC when congestion is detected. After the initial virtual switch port to physical port assignment is completed, Load Based teaming checks the load on the dvUplinks at a 30 second interval and dynamically reassigns port bindings based on the current network load and the level of saturation of the dvUplinks. The VMkernel indicates the network I/O load as congested if transmit (Tx) or receive (Rx) network traffic is exceeding a 75% mean over a 30 second period. (The mean is the sum of the observations divided by the number of observations). An interval period of 30 seconds is used to avoid MAC address flapping issues with the physical switches. Although an interval of 30 seconds is used, it is recommended to enable port fast (trunk fast) on the physical switches, all switches must be a part of the same layer 2 domain. ================================================================================ Title: DPM scheduled tasks URL: https://frankdenneman.ai/2010-07-20-dpm-scheduled-task/ Date: 2010-07-20 vSphere 4.1 introduces a lot of new features and enhancements of the existing features, one of the hidden gems I believe is the DPM enable and disable scheduled tasks. The DPM “Change cluster power settings” schedule task allows the administrator to enable or disable DPM via an automated task. If the admin selects the option DPM off, vCenter will disable all DPM features on the selected cluster and all hosts in standby mode will be powered on automatically when the scheduled task runs. This option removes one of the biggest obstacles of implementing DPM. One of the main concerns administrators have, is the incurred (periodic) latency when enabling DPM. If DPM place an ESX host in standby mode, it can take up to five minutes before DPM decides to power up the ESX host again. During this (short) period of time, the environment experiences latency or performance loss, usually this latency occurs in the morning. It’s common for DPM to place ESX hosts in standby mode during the night due to the decreased workloads, when the employees arrive in the morning the workload increases and DPM needs to power on additional ESX hosts. The period between 7:30 and 10:00 is recognized as one of the busiest periods of the day and during that period the IT department wants their computing power lock, stock and ready to go. This scheduled task will give the administrators the ability to disable DPM before the workforce arrive. Because the ESX hosts remain powered-on until the administrator or a DPM scheduled task enables DPM again, another schedule can be created to enable DPM after the periods of high workload demand ends. To create a scheduled task to disable DPM, open vCenter, go to Home>Management>Scheduled Tasks (CTRL-Shift-T) and select the task “Change cluster power settings”. Select the default power management for the cluster , On or Off and configure the task. For example, by scheduling a DPM disable task on every weekday at 7:00, the administrator is ensured that all ESX hosts are powered on before 8 o’clock every weekday in advance of the morning peak, rather than have to wait for DPM to react to the workload increase. By scheduling the DPM disable task more than one hour in advance of the morning peak, DRS will have the time to rebalance the virtual machine across all active hosts inside the cluster and Transparent Page Sharing process can collapse the memory pages shared by the virtual machines on the ESX hosts. By powering up all ESX hosts early, the ESX cluster will be ready to accommodate load increases. ================================================================================ Title: VM to Hosts affinity rule URL: https://frankdenneman.ai/2010-07-16-vm-to-hosts-affinity-rule/ Date: 2010-07-16 VMware vSphere 4.1 introduces a new affinity rule, called “Virtual Machines to Hosts” (VM-Host). This new rule is available in vSphere 4.1 DRS clusters in addition to the existing (anti) affinity rule, which is now called VM-VM affinity rule. The new VM-Host affinity rule provides the ability of placing a group of virtual machines on a subset of hosts inside the cluster. The new rule can very useful in blade system environments and for honoring ISV license requirements. Rules can be created to ensure that virtual machines run on ESX hosts in different blade chassis for availability reasons, or the complete opposite and limit the virtual machines to ESX hosts inside a blade chassis to optimize network speeds by keeping network traffic inside the blade chassis. VM-host are also very useful to fulfill the requirements of special ISV license models as well, for example restricting Oracle database virtual machines to run only on ESX hosts which are licensed by Oracle. Difference between VM-Host affinity rules and VM-VM rules The VM-host affinity rule differ from the VM-VM rule, A VM-Host (anti) affinity rule specify the (anti) affinity between a group of virtual machines and a group of ESX hosts inside the cluster, whether a VM-VM (anti) affinity rule only specify the (anti) affinity between individual virtual machines. Components. A virtual machine to host affinity rule exists out of three components: • Virtual machine DRS group • ESX host DRS group • Designation – “Must” affinity\anti-affinity or “Should” affinity\anti-affinity Virtual machine DRS groups and ESX host DRS Group are quite self-explanatory so let’s dive into the designations component straight away. Designations Two different types of VM-Host rules are available, a VM-Host affinity rule can either be a “must” rule or a “should” rule. The must-rule is a mandatory rule for HA, DRS and DPM, it confines or prevent the virtual machines to run on the ESX hosts specified in the ESX host DRS Group. The “should” rule is a preferential rule for DRS and DPM and expresses a preference. DRS and DPM use their best effort to try to confine or prevent the virtual machines from running on the ESX host they are affined to, but DRS and DPM can violate “should” rules if it compromises certain key operations, HA is not aware of preferential rules because DRS will not communicate these rules to HA. HA, DRS and DPM must take the mandatory rules into account when generating or executing operations. HA, DRS and DPM will never take any action that result in the violation of mandatory affinity rules. Because of this, mandatory rules place more constraints on VM mobility, making it more difficult for DRS to balance load and enforce resource allocation policies, HA and DPM operations are constrained as well, for example, mandatory rules will; • Limit DRS in selecting hosts to load-balance the cluster • Limit HA in selecting hosts to power up the virtual machines • Limit DPM in selecting hosts to power down Due its limiting behavior, it is recommended to use mandatory rules sparingly and only for specific cases, such as licensing requirements. Preferential rules can be used to meet availability requirements such as separating virtual machines between blade centers. DRS and mandatory rules DRS takes mandatory rules into account when generating load-balance recommendations. If a rule is created and the current virtual machine placement is in violation with the rule, DRS will create a priority one recommendation (five stars) and executes the recommendation if DRS is set to fully automatic. DRS will not generate recommendation that will violate the rule, it will not migrate virtual machines to or from an ESX server, even if places the source ESX host into maintenance mode. VMotion will reject the operation as well if it detects that the operation is in violation of the mandatory rule If a reservation is set on the virtual machine, DRS takes both reservation and mandatory affinity rule into account. Both requirements must be satisfied during placement or power on. If DRS is unable to honor either one of the requirements the virtual machine is not powered on or migrated to the proposed destination host. For example if a new rule is created and the current virtual machine placement is in violation of the rule, it can only migrate to a new host if the virtual machine memory reservation can be satisfied on the new host, if this is not possible, DRS will not generate the recommendation. If a rule is created that conflict with another active, the older rule overrules the newer rule and DRS will disable the new rule. As you can imagine that mandatory affinity rules can complicate troubleshooting in certain scenarios for example, why a virtual machine is not migrated from a highly utilized host to an alternative lightly utilized host in the cluster. DPM DPM does not place an ESX host into standby mode if it will violate the mandatory rule and will power-on ESX hosts if these are needed to meet the requirements of the mandatory riles. High Availability Due to the DRS-HA integration in vSphere 4.1, HA respects mandatory (must) rules. During an ESX host failure event, HA ask DRS to supply the list of hosts and places the virtual machines only on the compatible host, i.e. the host that are allowed by the mandatory rules. HA is unaware of the preferential (should) rules, so HA might unknowingly violate the rule during placement of virtual machines after an ESX failure, but the violation will be corrected by the next DRS invocation. Let’s take a look at a configuration which I think is going to be widely implemented soon, the Oracle Must affinity rule. 1. Place all Oracle virtual machines in a Cluster VM DRS group. (vm01, vm03, vm11, vm20) 2. Place all Oracle licensed ESX host in a Cluster Host DRS Group (ESX07, ESX08, ESX15, ESX16) 3. Select “Must run on Host in Group” In this scenario, DRS never places, migrates, or recommend placement of a host-affined virtual machine on a host to which is not listed in the Cluster Host DRS Group (ESX01 - ESX06 & ESX09-ESX14). This means that DRS will never ever place the virtual machine on an unlicensed host, not for maintenance mode, not for DPM power saving and not after an ESX host failure event. This virtual machine to host affinity rule make it possible to run oracle inside big clusters without having to license all the ESX host. I have been involved in a few projects where Oracle license was a constraint. Normally separate smaller clusters were deployed for Oracle database virtual machines, increasing both OPEX and CAPEX of the environment. These rules allows the Oracle virtual machines to run inside the cluster with other virtual machines without having to license all the ESX host inside the cluster. Hereby making the lives easier of both the architect and the administrator. vSphere 4.1, you gotta love it! Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: VMware Fault Tolerance and DPM URL: https://frankdenneman.ai/2010-06-20-vmware-fault-tolerance-and-dpm/ Date: 2010-06-20 Some requirements of the design I am working on is to be as “green” as possible and to offer the highest level of redundancy for business continuity. Enter VMware Fault Tolerance (FT) and Distributed Power Management (DPM)! When mixing multiple features, the requirements of one feature can have impact on- or even worse becomes a constraint of the other feature. DPM works together with DRS to VMotion virtual machines onto fewer ESX host servers when the resource demand drops below a specific threshold. In the current release of vSphere, DRS does not consider the FT-enabled virtual machines during load balancing operations and DRS will not migrate FT-enabled virtual machine automatically, because of this DPM cannot power down these hosts until the administrator will manually VMotion the primary or secondary virtual machines to another ESX host server. Fortunately when enabling DPM on the cluster, you can disable DPM at ESX host level. Due to the current limitations of DRS with VMware Fault Tolerance, it is recommended to disable DPM on at least two ESX server host to act as host for FT-enabled virtual machines. ================================================================================ Title: Memory reclamation, when and how? URL: https://frankdenneman.ai/2010-06-11-memory-reclaimation-when-and-how/ Date: 2010-06-11 After discussing with Duncan the performance problem presented by @heiner_hardt , we discussed the exact moment the VMkernel decides which reclamation technique it will use and specific behaviors of the reclamation techniques. This article supplements Duncan’s article on Yellow-bricks.com. Now let’s begin with when the kernel decides to reclaim memory and see how the kernel reclaims memory. So host physical memory is reclaimed based on four “free memory states”, each with a corresponding threshold. Based on the Threshold, the VMkernel chooses which reclamation technique it will use to reclaim memory from virtual machines. Free Memory state Threshold Reclamation technique High 6% None Soft 4% Ballooning Hard 2% Ballooning and Swapping Low 1% Swapping The high memory state has a threshold hold of 6%, that means that 6% of the ESX host physical memory minus the service console memory must be free. When the virtual machines use less than 94% of the host physical memory, the VMkernel will not reclaim memory because there is no need to, but when the memory usage starts to fall towards the free memory threshold the VMkernel will try to balloon memory. The VMkernel selects the virtual machines with the largest amounts of idle memory (detected by the idle memory tax process) and will ask the virtual machine to select it’s idle memory pages. Now to do this the guest os needs to swap those pages, so if the guest is not configured with sufficient swap space, ballooning can become problematic. Linux behaves pretty worse in this situation, invoking OOM (out-of memory) killer when its swap space is full and starts to randomly kill processes. Back to the VMkernel, in the High and Soft state, ballooning if favored over swapping. If it ESX server cannot reclaim memory by ballooning in time before it reaches the Hard state, the ESX turns to swapping. Swapping has proven to be a sure thing within a limited amount of time. Opposite of the balloon driver, which tries to understand the needs of the virtual machine let the guest decides whether and what to swap, the swap mechanism just brutally picks pages at random from the virtual machine, this impacts the performance of the virtual machine but will help the VMkernel to survive. Now the fun thing is, before the VMkernel detects the free memory is reaching the soft threshold, it will start to request pages through the balloon driver (vmmemctl), this is because it takes time for the Guest OS to respond to the vmmemctl driver with suitable pages. By starting prematurely, the VMkernel tries to avoid the situation that it will reach the Soft state or worse. So you can see ballooning occurring sometimes before the Soft state is reached. (between 6 and 4% free memory) One exception is the virtual machine memory limit, if a limit is set on the virtual machine, the VMkernel always tries to balloon or swap pages of the virtual machine after reaching its limit, even if the ESX host has enough free memory available. ================================================================================ Title: Reservations and CPU scheduling URL: https://frankdenneman.ai/2010-06-08-reservations-and-cpu-scheduling/ Date: 2010-06-08 Most of my resource management articles focus more on the behavior of memory management than on CPU management. Mainly because the Memory scheduler within ESX is such an interesting complex system which comprises of memory allocation, swapping and reclamation with algorithms such as Idle Memory Tax and mechanisms like ballooning and swapping. But lately it seems that CPU scheduling seems to attract more and more my attention. The discussion Duncan and I had prior to posting his article about how CPU limits actually sparked the interest how CPU scheduling works when setting reservations, so additional to Duncan excellent article, I want to take a closer look how the ESX CPU scheduler handles CPU reservations and shares and show why CPU scheduling is more fair that memory management. Similar to memory, the resource allocation settings, reservations, shares and limits can be set on CPU level. Limits and shares have similar behavior on CPU as well as Memory. Reservation act differently, let’s take a quick look at the resource allocation settings: **Shares:**Shares indicate the proportional value of the entity on the same hierarchical level. If everything else is equal, reservations, limits and active utilization, the virtual machine that is allocated twice as many shares as another virtual machine is entitled to consume twice as many CPU cycles. Limit: A limit is a mechanism to restrict physical resource usage of the virtual machine. A limit ensures that the VM will never receive more CPU cycles than specified, even if extra cycles are available on the host. Reservation: A reservation is a guarantee of the specified amount of physical resources regardless of the total number of shares in his environment. Now reservations act differently when setting it on a CPU than setting it on memory. When the virtual machine does not use its CPU cycles, these CPU cycles are redistributed to other active virtual machines, so unused reservations are not wasted. Contrary to memory management, when the memory will not be reclaimed by the scheduler once the virtual machine touched the pages. By redistributing available CPU cycles and not letting the virtual machine hoard CPU resources, the VMkernel tries to properly divide the resources and achieve better fairness among virtual machines and improve utilization of the resources. To achieve both goals and divide the CPU resources among virtual machines the CPU scheduler calculates a MHzPerShare metric. This metric tries to identify which virtual machines are “ahead” of their entitlement and which virtual machines are “behind” and do not fully utilize their entitlement. MHzPerShare = MHzUsed / Shares MHzUsed is the current utilization of the virtual machine measured in Megahertz. Shares is the current configured amount of shares of the virtual machine. For example, the virtual machine is using 2500 MHZ and has 1000 shares, this means that the MHzPershare value is 2.5.The VMkernel will calculate the MHzPerShare number of each active virtual machine and the virtual machine with the lowest MHzPerShare value will have the highest priority of running on the CPU. If the virtual machine with the lowest MHzPerShare value decides not to use it right to allocate the cycles, the cycles can be used by the virtual machine with the next lower MHzPerShare value. Although not shown, reservations play a important part in this calculation. As mentioned before, reservations overrule shares and guarantee the amount of physical resources regardless of the amount of shares. This means that the virtual machine always can use the CPU cycles specified in its reservation, even if the virtual machine has a greater MHzPerShare value. So how exactly do reservations and shares interact with each other when it comes to calculating the MHzPerShare value? For example: In a 6 GHz system, 1 virtual machine is running and 2 are powered on, VM1 is running a memory intensive app and doesn’t really care much about CPU cycles, the virtual machine is configured with 1000 CPU shares and no reservation. The 2 other virtual machines run CPU intensive apps and are currently competing for resources. VM2 has a reservation of 2250 MHz and has a default share setting of 1000 shares, the other CPU intensive virtual machine, VM3 is equipped with 2vcpu’s and therefore receives 2000 shares, but the administrator didn’t set any reservation. Now VM1 is running at 500 MHz, with its 1000 shares, the MHzPerShare value equals 0.5. Because VM2 is in need of CPU cycles, it immediately utilizes its reservations and “occupies” all 2250 MHz, its MHzPerShare value equals 2.25 (2250/1000). Now because VM3 doesn’t have any reservation and is in need of CPU cycles, the VMkernel looks at its MHzPerShare value to decide how many CPU cycles it can use before distributing excess CPU cycles to other virtual machines. The kernel will distribute cycles to VM3 until it reaches the same MHzPerShare value of VM2, which is 2.25. In theory this means that the VMkernel will allocate 2000 x 2.25 = 4500 MHz before looking at another VM. Due to the fact that CPU scheduler already allocated 500 MHz to VM1 and 2250 MHz to VM2 of the available 6GHz, it can allocate VM3 3250 Mhz. Because VM2 has a reservation it can allocate up to its reservation even when initially VM3 has a lower MHzPerShare value (0) and the CPU cycle requirements of VM1 are met at 500MHz. However due to the fairness principle VM2’s own MHzPerShare value influences the VMkernel’s decision how much cycles to allocate to VM3 before considering allocating additional cycles to vm2 again. Now for some reason the application in VM3 is leveling out at 2000 MHz, VM1 is still using 500 MHz and VM2 is in desperate need of extra CPU cycles. No settings are changed so VM1 and VM2 has a 1000 shares each and VM2 has a reservation of 2250MHz, VM3 has 2000 shares and no reservation is set. The VMkernel will satisfy the request of VM1, resulting in a MHzPerShare value of 0.5. VM2 claims its reservation and utilizes 2250 MHz resulting in a MHzPerShare value of 2.25, VM3 can allocate up to 4500 before reaching the MHzPerShare value of VM3, but stops consuming above 2000Mhz, ending up with a MHzPerShare value of 2000/2000 = 1, this means that inside the 6GHz host 1250 cycles are available. The CPU scheduler will shop around with these available cycles and see which VM is interested. Now the VMkernel will offer the cycles to the virtual machines in the increasing order of MHzPerShare, so first it will ask VM1 (0.5), because its CPU request is satisfied, it will forfeit its claim, VM2 also forfeits this claim, so VM3 will happily accepts the remaining cycles and its resource usage will increase to 3500 MHz. So here you have it, both shares and reservation interact or even battle with each other to allocate CPU cycles for the virtual machines. Shares are by many perceived as an inferior resource allocation setting, hopefully this demonstrates the power of shares, it can in combination with utilization become a very important factor in ESX resource management. ================================================================================ Title: Virtual Machine memory overhead URL: https://frankdenneman.ai/2010-05-31-virtual-machine-memory-overhead/ Date: 2010-05-31 Every virtual machine running on an ESX host consumes some memory overhead additional to the current usage of its configured memory. This extra space is needed by ESX for the internal VMkernel datastructures like virtual machine frame buffer and mapping table for memory translation (mapping physical virtual machine memory to machine memory). Two kinds of virtual machine overhead exists: Static overhead Static overhead is the minimum overhead that is required for the virtual machine startup. DRS and the VMkernel uses this metric for admission control and VMotion calculations. The destination host must be able to back the virtual machine reservation and the static overhead otherwise the VMotion will fail. Dynamic overhead Once the virtual machine has started up, the virtual machine monitor (VMM) can request additional memory space. The VMM will request the space, but the VMkernel is not required to supply it. If the VMM does not obtain the extra memory space, the virtual machine will continue to function but this can lead to performance degradation. The VMkernel treats virtual machine overhead reservation the same as VM-level memory reservation and it will not reclaim this memory once it used. Overhead memory used in admission control As mentioned before, DRS and the VMkernel will not allow the virtual machine to be powered up if reservations cannot be guaranteed, this means that the effective memory reservation for a virtual machine is the user configured memory reservation (VM-level reservation) plus the overhead reservation. Resource pool memory reservations This means that during the design phase of a resource pool, the memory overhead of a virtual machine must be included in the calculation of the memory reservation specified on the resource pool. The behavior of dynamic overhead must also be taken into account. Table 3.2 of the vSphere resource management guide list the overhead memory on virtual machines. VMware vSphere Online Library - Table 3.2 overhead memory Please be aware of the fact that memory overheads are growing with each new release of ESX, so keep this in mind when upgrading to a new version. Verify the documentation of the virtual machine memory overhead and check the specified memory reservation on the resource pool. ================================================================================ Title: Re: Swapping URL: https://frankdenneman.ai/2010-05-26-re-swapping/ Date: 2010-05-26 Recently we had a discussion about swapping, as Duncan mentioned in his article “Swapping” Swapped memory might not have impact on the performance of the virtual machine. There are scenarios when pages can be swapped out without experiencing performance problems. One common scenario is a bootstorm, i.e startup of many virtual machines at once. Bootstorms can happen when a host failure occurs and High Availability powers on the virtual machines on other host, but are also frequently encountered in windows shops after Patch Tuesday, when the operations team need to obey a limited maintenance window timeslot. When a virtual machine guest OS starts, there will be a period of time before the VMware tools is loaded and the vmmemctl (balloon driver) is operational. During this timeslot the operating system can access a large portion of its configured memory. Windows systems are notorious for this as they tend to touch every page until it reaches the end of their configured memory. Unfortunately page sharing due to Transparent Page Sharing (TPS) is also at a minimum. Redundant memory pages are not collapsed immediately when a virtual machine is started. TPS is a VMkernel background process and uses a cycle of 60 minutes (Mem.ShareScanTime) to scan a virtual machine for page sharing opportunities. During these bootstorms many virtual machines are powered on at the same time, all claiming lots of memory or even their maximum configured memory (windows). This behavior leads to a spike in memory usage and without the help of the balloon driver and TPS, the ESX host needs to resort to swapping out memory. When referring to windows startup, windows will touch every page and this forces ESX to back all in machine memory (physical memory). These pages are filled with useless information and chances are that this might never be accessed by the virtual machine again. Now ESX will not proactively swap memory back in to physical memory when the memory pressure disappears. These pages will remain swapped until it is accessed by the virtual machine, at that point ESX will swap it into memory. Swapping during the bootstorm will delay the boot process, but these swapped out pages will not cause any performance problems during normal operation. As mentioned in Duncan’s “Swapping article”, there are a few metrics that indicates that a virtual machine is swapping or has swapped before. When encountering swapped memory, check the metrics SWCUR (Swap current) and SWTGT (Swap target). If a bootstorm occurred it is likely to have a higher value at SWCUR than at the SWTGT. The SWTGT indicates the desired amount of memory to be swapped out, this is determined by ESX by the resource entitlement calculation of the virtual machine. If there is no memory pressure, the swaptarget will be equal to 0, but because pages remain in the swap file until accessed, the SWCUR will indicate the remaining swapped out pages. If memory contention does occur, ESX will attempt to make the SWCUR equal to the SWTGT (swap target). ================================================================================ Title: Resource pools memory reservations URL: https://frankdenneman.ai/2010-05-18-resource-pools-memory-reservations/ Date: 2010-05-18 After publishing the article “impact of memory reservations” I received a lot of questions about setting memory reservation at resource pool level. It seems there are several common facts about resource pools and memory reservations that are often misunderstood. Because reservations are used by the VMkernel\DRS resource schedulers and (HA) admission control, the behavior of reservation can be very confusing. Before memory reservation on resource pool is addressed, let look at which mechanisms uses reservations and when reservations are used. When are reservations actually used besides admission control? If a cluster is under-committed the VM resource entitlement will be the same as its demand, in other words, the VM will be allocated whatever it wants to consume within its configured limit. When a cluster is overcommitted, the cluster experiences more resource demand than its current capacity, at this point DRS and the VMkernel will allocate resources based on the resource entitlement of the virtual machine. Resource entitlement is covered later in the article. Is there any difference between resource pool level and virtual machine level memory reservation? To keep it short, VM level reservation can be rather evil, it will hoard memory if it has been used by the virtual machine once. Even if the virtual machine becomes idle, the VMkernel will not reclaim this memory and return it to the free memory set. This means that ESX can start swapping and ballooning if no free memory is available for other virtual machines while the owning VM’s aren’t using their claimed reserved memory. It also has influence on the slot size of High availability, for more information about HA slot sizes, please visit the HA deep dive page at yellow-bricks.com. For more information about virtual machine level memory reservation, please read the article “impact of memory management”. Behavior of resource pool memory reservation Now setting a memory reservation on a resource pool level has its own weaknesses, but it is much fairer and more along the whole idea of consolidation and sharing than virtual machine memory reservations. RP level reservations are immediately active, but are not claimed. This means it will only subtract the specified amount of memory from the unreserved capacity of the cluster. RP reservations are used when children of the resource pool uses memory and the system is under contention. Reservations are not wasted and the resources can be used by other virtual machines. Be aware, using and reserving are two distinct concepts! Virtual machines can use the resource, but they cannot reserve this as well if it is already reserved by another item. It appears that resource pool memory reservations work almost similar to CPU reservations, they won’t let any resource go to waste. And to top it off, resource pool reservations don’t flow to virtual machines, they will not influence HA slot sizes. Which unfortunately can lead to (temporary) performance loss if a host failover occurs. When a virtual machine is restarted by HA they are not restarted in the correct resource pool but in the root resource pool, which can lead to starvation. Until DRS is invoked, the virtual machine need to do it without any memory reservations. How to use resource pool memory reservation? Ok so two popular strategies exist when it comes to setting memory reservation on resource pool levels: 1. CPU and Memory reservations within the resource pool is never overcommitted i.e configured memory all VM’s (40Gb) equals reservation (40GB) 2. Percentage of Cluster resources reserved i.e. memory reservation resource pool (20GB) less than configured memory virtual machines inside RP (40GB) The process of divvying is rather straightforward if the memory reservation equals the configured memory of the virtual machines inside the resource pool. All pages by the virtual machines are backed by machine pages, the resource entitlement is at least as large as its memory reservation. What I find more interesting is what happens if the resource pool is configured with a memory reservation that is less than all virtual machine configured memory? DRS will divvy memory reservations based on the virtual machine resource entitlement. So how is resource entitlement calculated? A virtual machines resource entitlement is based on various statistics and some estimation techniques. DRS computes a resource entitlement for each virtual machine, based on virtual machine and resource pool configured shares, reservations, and limits settings, as well as the current demands of the virtual machines and resource pools, the memory size, its working set and the degree of current resource contention. Now by setting a reservation on the resource pool level, the virtual machines who are actively using memory profits the most of this mechanism. Basically if no reservation is set on the VM level, the “RP” reservation is granted to all virtual machines inside the resource pool who are actively using memory. DRS and the VMkernel calculates the resource pool and the virtual machine share levels. Please read the article “the resource pool priority-pie paradox” to get more information about share levels. and use this to specify the virtual machines priority. Besides the share level the active utilization (working set) and the configured memory size are both accounted when calculating the resource entitlement. Virtual machines who are idling aren’t competing for resources, so they won’t get any new resources. If the memory is also idle the allocation get adjusted by the idle memory tax. Idle memory tax uses a progressive tax rate, the more idle memory a VM has, the more tax it will generate, this is why the configured memory size is also taken into account. (nice ammo if your customer wants to configure the DHCP server with 64GB memory!) When we create a “Diva” VM (coined by Craig Risinger), that is setting VM level reservations, this allocation setting is passed to the VMkernel. It will subtract the specified amount of the reservation pool of the RP and it will not share it with others, i.e. the Diva VM is a special creature. As stated above, RP memory reservations flow more than VM-level reservation, it will not claim\hoard memory. So basically when setting a resource pool reservation, reservations are just a part of the computation of the virtual machines resource entitlement. When the host is overcommitted, the memory usage of the virtual machine is either above or below the resource entitlement. If the memory usage exceeds its resource entitlement, the memory is ballooned or swapped from the virtual machine until it is at or below its entitlement. Disclosure Now before you think I fabricated this article all by myself I am happily to admit that I’m in the lucky position to work for VMware and to call some of the world brightest minds my colleagues. Kit Colbert, Carl Waldspurger and Chirag Bhatt took the time and explain this theory very thoroughly to me. Luckily my colleagues and good friends Duncan Epping and Craig Risinger helped me decipher some out-of-this-world emails from the crew above and participated in some excellent discussions. ================================================================================ Title: VMware tools disk timeout value Linux GOS URL: https://frankdenneman.ai/2010-04-28-vmware-tools-disk-timeout-value-linux-gos/ Date: 2010-04-28 After I posted the “VMtools increases TimeOutValue article” I received a lot of questions if the VMware Tools automatically adjust the timeout value for Linux machines as well. Well, VMware Tools of versions ESX 3.5 Update 5 and ESX 4.0 install a udev rule file on Linux operating systems with kernel version equal or greater then 2.6.13. This rule file changes the default timeout value of VMware virtual disks to 180 seconds. This helps the guest operating system to better survive a SAN failure and keep the linux system disk from becoming read only. Because of the requirement of updates related to udev featured in the 2.6.13 kernel, the SCSI timeout value in other Linux kernels is not touched by the installation of VMware tools and the default value remains active. The two major Linux Kernel version each have a different timeout value: Linux 2.4 - 60 seconds Linux 2.6 - 30 seconds You can set the timeout value manually listed in /sys/block/disk/device/timeout. The problem is the distinction VMtools make between certain Linux Kernels, if you do not know this caveat you might end up with an Linux environment which is not configured exactly the same. This can lead to different behaviour during a SAN outage. Standardization is key when managing virtual infrastructure environments and a uniform environment eases troubleshooting A while ago Jason wrote an excellent article about the values and benefit of increasing the guest os timeout ================================================================================ Title: ESX4 ALUA, TPGS and HP CA URL: https://frankdenneman.ai/2010-03-25-esx4-alua-and-hp-continuous-access/ Date: 2010-03-25 In my blog post: “HP CA and the use of LUN balancing scripts” I tried to cover the possible impact of using HP continuous Access EVA on the LUN path load balancing scheme in ESX 3.x. I received a lot of questions about this and wanted to address some issues again and try to clarify them. Let’s begin with a recap of the HP CA article; The impact of CA on the load-balancing scheme is due to the fact that an EVA is an asymmetric Active-Active array that uses the Asymmetric Logical Unit Access protocol (ALUA). ESX3 is not ALUA aware and does not recognize the different specific access characteristics of the array’s target ports. VMware addressed this shortcoming and added ALUA support in the new storage stack of ESX4. The ALUA support is a great feature of the new storage architecture, it reduces a lot of extra manual steps of creating a proper load-balanced environment. But how exactly does ALUA identifies which path is optimized and will HP Continuous Access still have an impact on ESX4 environments as well? Asymmetric Logical Unit Access Asymmetric Logical Unit Access occurs when the access characteristics of a storage processor port relative to the LUN differs from another port in the array. This behavior occurs on Asymmetrical Active-Active arrays (AAA). Two well-know AAA arrays are the EMC CX and the HP EVA. In a AAA array both controllers can receive IO commands (active-active), but only one controller can issue IO to the LUN. This is the asymmetrical part. The opposite of a AAA array is a symmetric Active-Active array, like the EMC Symmetrix DMX, such an array can issue IO command to the LUN via both controllers. But for now let’s concentrate on AAA arrays and the HP EVA specifically. The controller in an AAA array who can issue commands is called the managing controller, paths to the LUN via ports of this controller are called optimized paths. IO sent to a port of the non-owning controller must be transferred to the owning controller internally and increases latency and have impact on the performance of the array. Due to this, paths leading to the LUN via the non-managing controller are called non-optimized paths. Pluggable Storage Architecture The default Native Multipathing Plugin (NMP) used by the storage stack in ESX4 uses two sub-plugins, the Storage Array Type Plug-in and Path Selection Plugins. Storage Array Type Plugins is used for path handling and the Path Selection Plugins is used for Path selection. It is interesting to know that the SATP is associated with all the physical paths to the array and is configured globally per array, but the PSP can be configured per LUN. Storage Array Type Plugins A default Storage Array Type Plugins (SATP) is included for each supported array as well as a generic SATP for non-specified storage arrays. The two SATP available for the HP EVA are: • VMW_SATP_EVA (array specific) • VMW_SATP_ALUA (generic SATP) The problem with the array specific VMW_SATP_EVA is that it doesn’t use Target Port Group Support (TPGS), Funny thing is that TPGS is critical when it comes to determining optimized and non-optimized paths. Therefore the VMware HCL list the generic VMW_SATP_ALUA as the supported SATP on EVA Active-Active arrays. Target Port Group Support ALUA uses Target Port Group Support to determine the access characteristics of the path to a LUN. A port on a storage processor is called a target port and belongs to a target port group (TPG). All target ports belonging to the same TPG are always in the same Asymmetric Access State (AAS) relative to a LUN. There are a multiple of access states for a Target port: • Active/Optimized • Active/Non-optimized • Standby • Unavailable • Transitioning Note the grouping of Target Ports per TPG per Controller are vendor specific! Active/Optimized A target port reporting the active/optimized state belongs to the managing owner of the LUN and communicates directly with the LUN. Active/Non-Optimized A target port reporting the active/non-optimized state belongs to the non-managing owner of the LUN and the controller must send the IO via the multilink-ports to the managing controller of the LUN. Transitioning If a target port is in a Transitioning state, the ownership of the LUN is being transferred between controllers, this can occur if a hardware failure happened on the managing controller or when the threshold of proxy IO’s for the specific LUN is reached, this is called an Implicit Lun Transfer (ILT) For example, a ESX host with two HBA’s connected to a EVA 8100, SP A is the managing controller of LUN1. TPGS will report that the Active Optimized paths are HBA1:1:1 and HBA2:1:1. HBA1:2:1 and HBA2:2:1 are listed as the non-optimized paths. Path Selection Plugin It’s up to the Path Selection Plugin (PSP) to actually select the path to be used by the ESX host to communicate with the LUN. There are three PSP’s are available: • Most Recently Used Path Selection • VMW_PSP_RR Round Robin Path Selection • VMW_PSP_FIXED Fixed Path Selection The Fibre Channel SAN Configuration Guide list the following: MRU: Selects the path the ESX/ESXi host used most recently to access the given device. If this path becomes unavailable, the host switches to an alternative path and continues to use the new path while it is available. Fixed: Uses the designated preferred path, if it has been configured. Otherwise, it uses the first working path discovered at system boot time. If the host cannot use the preferred path, it selects a random alternative available path. The host automatically reverts back to the preferred path as soon as that path becomes available. **Round Robin (RR):**Uses a path selection algorithm that rotates through all available paths enabling load balancing across the paths. ALUA aware PSPs Both MRU and Round Robin PSPs are ALUA aware.MRU selects the first working optimized path discovered at system boot time and uses non-optimized paths when all optimized paths are dead. When an optimized path becomes available again, MRU will switch back to the optimized path. This differs from traditional MRU behavoir where a administrator must manually fail back to a path. Per default Round Robin will issue IO across all optimized paths and will use non-optimized paths only if no optimized paths are available. Selecting the Fixed PSP on an ALUA enabled array defeats the purpose of ALUA support of the NMP architecture. If the preferred path is configured to use a non-optimized path, ESX will use this path to issue IO, even though optimized paths might be available. My advice is to stick with MRU or Round Robin when using a ALUA aware SATP. Note! When if you have a mixed environment of ESX 3.5 and ESX 4 set the preferred paths to the LUNs according to the ALUA optimized paths listed in ESX4. If preferred paths are configured to use the non-optimized paths, repeatedly Implicit LUN transfers can occur VMware list MRU as the default supported PSP in the HCL, I asked VMware Global Support Services if Round Robin is supported even if it is not listed in the HCL. If the partner recommends RR for their certified arrays on ESX 4.0, then they provide their customers with the directions for configuring it and they support them. We will support it but not list it in the HCL MRU or RR So the question arises, which PSP should I select? If you use MSCS configurations, using Round Robin is unsupported on the LUNs that are part of MSCS VMs. By selecting MRU ESX will only use the first optimized path discovered at boot, where Round Robin uses all optimized paths. MRU will use the first optimized path and does not load balance across HBA’s. ie. HBA1->LUN1, HBA2-LUN2, HBA1->LUN3, as where Round Robin will utilize all optimized paths of both HBA’s to send IO. Almost every vendor list Round Robin as best practice when using an ALUA array, generating better load-balance and aggregation of the available uplinks. Another point of interest is the IOOperationlimit, this setting specifies how much IO commands will be sent across one path before switching over to the next available optimized path. Before following best practise of your vendor, my advice is to read Chad’s post first: vSphere 4 NMP RR IOoperationsLimit bug and workaround. Continuous Access So how about the impact of Continuous Access on the path load balancing in ESX 4? When a LUN is being replicated it becomes a member of a CA Data Replication Group (DR Group). The DR Group is a Virtual Controller Software (VCS) construct and enforces the member LUNs to have the following behavior: • Disk in DR Group will fail over together • To preserve write order within the DR Group • Share a log disk • Replicate to the same destination array To meet these requirements, CA will assign one controller as managing controller for all the member LUNS in the DR Group. Because all the target ports of controller belong to the same TPG, these target ports will report to ESX as the Active/Optimized paths to the LUNs of the DR Group. This behavior will impact the design of the DR Groups, how many LUNs will become a part of one DR Group? This will impact the load across both controllers, quite easily an imbalance can occur when “DR-Group managed controller affiliation” is not taken into consideration. Please be aware of the limitation of the amount of DR-Groups and the use of DR tunnels. The path between the source and destination controller is called a DR Tunnel (DRT), to ensure IO consistency CA uses only one path per DR Group. A DR group is allocated a set of write and copy resources per DR Tunnel and these resources are not infinite. When heavy traffic occurs, DR groups can temporarily run out of allocated write resources. Running out of write resources will have impact on the write performance of the virtual machine. For these reasons designing a balanced DR Group is something you must do together with your SAN-administrators, because the ratio DR Groups\DR Tunnels can have impact of your virtual environment as well. Please read the following great posts as well: “what is ALUA exactly” and “a couple of important (ALUA and SRM) Notes” ================================================================================ Title: Identify storage performance issues URL: https://frankdenneman.ai/2010-03-25-identify-storage-performance-issues/ Date: 2010-03-25 VMware has recently updated the kb article " Using esxtop to identify storage performance issues Details" (KB1008205). The KB article provides information about how to use esxtop to determine the latency statistics across various devices. The article contain easy to follow, step-by-step instructions on how to setup ESXtop to monitor storage performance per HBA, LUN and virtual machine. It also list generic acceptable values to put your measured values in perspective. It’s a great article, bookmark it for future reference. If you want to learn about threshold of certain metrics in ESXtop, please check out the ESXtop metric bible featured on Yellow-bricks.com. ESXtop is a great tool to view and measure certain criteria in real time, but sometimes you want to collect metrics for later reference. If this is the case, the tool vscsiStats might be helpful. vscsiStats is a tool to profile your storage environment and collects info such as outstanding IO, seekdistance and many many more. Check out Duncan’s excellent article on how to use vscsiStats. Because vscsiStats will collect data in a .csv file you can create diagrams, Gabe written an article how to convert the vscsiStats data into excel charts. ================================================================================ Title: VCDX tip: VMtools increases TimeOutValue URL: https://frankdenneman.ai/2010-03-16-vcdx-tip-vmtools-increases-timeoutvalue/ Date: 2010-03-16 This is just a small heads-up post for all the VCDX candidates. Almost every VCDX application I read mentions the fact that they needed to increase the Disk TimeOutValue (HKEY_LOCAL_MACHINE/System/CurrentControlSet/Services/Disk) by to 60 seconds on Windows machines. The truth is that the VMware Tools installation (ESX version 3.0.2 and up) will change this registry value automatically. You might want to check your operational procedures documentation and update this! VMware KB 1014 ================================================================================ Title: Removing orphaned Nexus DVS URL: https://frankdenneman.ai/2010-03-11-removing-orphaned-nexus-dvs/ Date: 2010-03-11 During the test of the Cisco Nexus 1000V the customer deleted the VSM first without removing the DVS using commands from within the VSM, ending up with an orphaned DVS. One can directly delete the DVS from the DB, but there are bunch of rows in multiple tables that need to be deleted. This is risky and may render DB in some inconsistent state if an error is made while deleting any rows. Luckily there is a more elegant way to remove an orphaned DVS without hacking and possibly breaking the vCenter DB. A little background first: When installing the Cisco Nexus 1000V VSM, the VSM uses an extension-key for identification. During the configuration process the VSM spawns a DVS and will configure it with the same extension-key. Due to the matching extension keys (extension session) the VSM owns the DVS essentially. And only the VSM with the same extension-key as the DVS can delete the DVS. So to be able to delete a DVS, a VSM must exist registered with the same extension key. If you deleted the VSM and are stuck with an orphaned DVS, the first thing to do is to install and configure a new VSM. Use a different switch name than the first (deleted) VSM. The new VSM will spawn a new DVS matching the switch name configured within the VSM. The first step is to remove the new spawned DVS and do this the proper way using commands from within the VSM virtual machine. Removing DVS with Nexus VSM virtual machine: Log in VSM Ping the vCenter to make sure you have a connection. conf t svs connection connection_name The connection_name is created during the configuration of the connection between the VSM and the vCenter server. The default connection name is vCenter. To query the current connection name: show svs connections If the SVS connection output does not show a datacenter name, but the minus (-) sign, you must specify the vCenter datacenter where the DVS is created, with the following command: (In my case I needed to specify the datacenter even when the datacenter name was listed in the svs connection) vmware dvs datacenter-name name (case sensitive) e.g: vCenter datacenter name = DATACenter vmware dvs datacenter-name DATACenter Use the following command to remove the DVS: no vmware dvs The following warning appears: This will remove the DVS from the vCenter Server and any associated port-groups Do you really want to proceed (yes/no) When selecting Yes, the following output appears in the VSM command prompt: Note: Command execution in progress, please wait.. Simultaneously the recent task window in vCenter shows two tasks: Delete folder Delete vNetwork Distributed Switch Select Network Inventory view to check if the DVS is deleted. Removing DVS after destroying the VSM virtual machine. The first part was the easy part, removing a DVS after the corresponding VSM is removed is a bit trickier. First we must change the hostname of the VSM to reflect the switch name of the orphaned DVS. Log in VSM conf t hostname exit copy run start After setting the new hostname, the command prompt changes immediately to the “new” hostname. At this moment, the VSM is using the same switch name, but it still uses a different extension-key as the orphaned DVS. We need to change the extension key of the VSM to match the extension key of the orphaned DVS. Both old and new extension keys are listed in the vCenter database. First we need to know which extension-key the VSM is currently using. This key is going to be deleted from the vCenter DB. This is done by using the following command in the VSM: show vmware vc extension-key The command prompt returns with an extension-key e.g: Cisco_Nexus_1000V_1234101238 We need to remove this extension key from the vCenter DB, to do this we are going to use the managed Object Browser or (MOB. The Managed Object Browser is a web-based tool for working with the API. This tool enables you to browse managed objects on vCenter Server. Open an Internet Explorer window to access the vCenters’ MOB. Enter https:///mob See KB 568529 for more information about using the Managed Object Browser operations. Log in with user with administrator rights in vCenter. Before we are going to unregister the “new” extension-key, we need to know which extension-key the old VSM used. (the values listed in italic are examples, the values in your environment can be different) Go to: ServiceContent: content rootFolder: group-d8 childEntity: datacenter-76 networkFolder: group-n15 childEntity: group-n3456 childEntity: dvs-3457 DVSConfigInfo: config At this moment, two keys exists, one key matching the orphaned DVS and one used by the newly spawned DVS during configuration of the second VSM. Copy the old key matching the orphaned DVS to notepad file; we are using that key later on. Now copy the key matching the new DVS, this key must be removed using the ExtensionManager. For example: extensionKey: “Cisco_Nexus_1000V_1234101238” Go to: https://vcenterhostname/mob/?moid=ExtensionManager or follow the following path from the MOB home screen: ServiceContent: content extensionManger: ExtensionManager void: unregisterExtension extensionKey (required) string: paste key you copied from the DVSConfigInfo for example: extensionKey (required) string: Cisco_Nexus_1000V_8321457891 Click on InvokeMethod This will return with the status: Method invocation result: void Now return to the VSM command prompt and use the old extension id of the first VSM (saved in notepad file). vmware vc extension-key If the following error appears: Cannot change the extension key while a connection is already enabled the svs connection is still active and you must disconnect the current SVS connection by entering the following commands: svs connection vcenter no connect Issue the vmware vc extension-key again after closing the svs connection. At this point the VSM is configured with a matching extension-key as the orphaned DVS, but the VSM must registered within vCenter with this extension-key. To do this, you must use the extension.xml of the VSM, which is available for download on the webpage of the Nexus VSM. (Before I downloaded the xml file I restarted the vCenter, don’t know if this is necessary, but I just wanted to be sure that the prior configuration settings are saved and committed to the database.) Register the VSM by importing the xml with the extension key of the orphaned DVS switch: open internet explorer and enter the ip-address of the VSM. right click the link to save the cisco_nexus_1000v_extension.xml to your computer open vcenter Select Plug-ins Manage Plug-ins… Right click on whitespace inside Plug-in Manager and select the option New Plug-in… click the Browse button and select the saved xml file click on Ok. now the Cisco_Nexus_1000v_old extension key code appears in Available Plug-ins section. At this point the VSM is using the matching switch name and extension key of the orphaned DVS and is registered with the extension key in vCenter.Time to connect the VSM to vCenter. This will spawn a new DVS, this DVS will use the same extension key and switch name as the orphaned DVS and override all the info in the vCenter database. But because a extension session will exist we can remove the newly spawned DVS from the VSM using the commands mentioned in the first part of this article. Return to the command prompt of the VSM and issue the following commands; conf t svs connection vcenter vmware dvs datacenter_name datacenter The DVS is created with the same name as the orphaned dvs and overwriting the old configuration. When this is completed check the network inventory view. return to command promt and issue the command no vmware dvs to remove the new-old DVS. If you will receive an error follow the steps mentioned in the Removing DVS with Nexus VSM virtual machine section. The orphaned DVS is removed. More information about the Nexus commands go visit the online nexus command reference ================================================================================ Title: DRS Resource Distribution Chart URL: https://frankdenneman.ai/2010-03-08-drs-resource-distribution-chart/ Date: 2010-03-08 A customer of mine wanted more information about the new DRS Resource Distribution Chart in vCenter 4.0, so I thought after writing the text for the customer, why not share this? The DRS Resource Distribution Chart was overhauled in vCenter 4.0 and is quite an improvement over the resource distribution chart featured in vCenter 2.5. Not only does it use a better format, the new charts produce more in-depth information. Resource entitlement Before we dive into the old and the new chart, clarification of the term “resource entitlement” can be helpful. The charts show the resource entitlement of virtual machines running inside the cluster, but what are resource entitlements and what are they used for? To quote Minwen a VMware R&D engineer Based on various stats and some estimation techniques, DRS determines each VM’s *demand* for CPU & memory resources. It then computes each VM’s cpu & mem *entitlement* (how many resources it should get) based on resource settings (shares, limits, reservations) and the degree of resource contention there is. If there are enough resources in the cluster to satisfy every VM’s demand (and assuming no limits), then entitlement is equal to demand, meaning every VM gets as much CPU & memory as it wants. The old chart Let’s begin with the vCenter 2.5 chart and take a look how the old chart showed the info. I always disliked the way the old chart displayed the number of hosts. The chart used in this example belongs to a cluster with 3 hosts. At the top of the Y-axis a number 2 is shown and in the middle of the Y axis a horizontal line is displayed. This horizontal line in this example depicts 50% of two hosts, in other words: one host. The chart shows multiple bars, the leftmost bars represent the resource utilization of 2 hosts, which means that two hosts have a CPU and memory utilization between 0 and 10 percent. The orange bar in the 20-30 column stretches to the horizontal line on the Y-axis and that means that the memory of the third host is utilized between 20 and 30 percent, in this example the same third host has a CPU utilization of 90-100 percent. If the cluster is balanced, all orange or blue bars are closed to each other, the closer the blue and orange bars are to each other, the more balanced the cluster is. The bottom chart “percent of entitled resources delivered” uses a different scale in this example. Because all three hosts deliver 90-100 percent of the memory resources the top of the Y axis represents 3 hosts. 2 hosts deliver 90-100 percent of the CPU resources and the bar stretches to 66% of the Y axis. Because of the current statistics, the horizontal bar in the middle of the Y-axis cannot be translated into one host. The Y-axis has a dynamic nature due to the relation of the scale to the load; this behavior does not contribute to legibility of the charts. DRS Resource Distribution chart 2.0 The chart in vCenter 2.5 showed the CPU and Memory utilization of the host inside the cluster, load distribution across host and if the delivery of the resource entitlement of the virtual machines. The current chart used in vCenter 4 shows seven levels of information: Cluster info: • Relative balance of hosts X • Approximate spare capacity left in the cluster Host info: • Resource usage for each host X • Relative resource consumption of each VM on each host X VM info: • Resource usage for each VM X • Host assignment for each VM X • Percent resource entitlement for each VM X VMware chose not to combine the CPU and Memory into one chart, but used a toggle view. Let’s take a look at the new view: For this example I used the CPU chart, because of the pretty colors. You can choose between % or MHz view. I used the % view in all the examples for no specific reasons, it can be easily be substituted with the MHz view, both views are equally informative. The new chart displays the host on the Y-axis and bar along the X-axis shows the CPU (or memory) utilization. The Y-axis lists the hosts of the cluster with its hostname. This way the resource usage of a specific host and the relative balance between hosts inside the cluster is instantly recognizable. In this example all the hosts are utilized below the 25%, it shows that we have an enormous amount of spare CPU capacity left in the cluster. The horizontal bar behind the hostname shows the current resource usage of the host and is subdivided by the resource utilization of each virtual machine on that host. Each block in the bar stands for a specific virtual machine, the size of the block represent the relative resource consumption of the virtual machine on the host. When hovering over a block, the statistics of that specific virtual machine running on the host is displayed. The info block shows the consumed resources, the active resources and the resource entitlement of the virtual machine. Color vs Gray scale The CPU view uses a color scheme. At the bottom of the chart view a gradient bar displays the percentage of the entitle resources delivered. The VM block color corresponds to the amount of entitled resources that are delivered by the host to the virtual machine. The memory view uses a gray scale; actually it uses just one tone, just gray. :) I don’t have an authoritative answer why the memory chart view doesn’t use a color gradient, but I have an educated guess. I already sent the product manager of DRS some questions; hopefully I can soon update this article with a definitive answer. Stay tuned! ================================================================================ Title: Resource pools and avoiding HA slot sizing URL: https://frankdenneman.ai/2010-02-24-resource-pools-and-avoiding-ha-slot-sizing/ Date: 2010-02-24 Virtual machines configured with large amounts of memory (16GB+) are not uncommon these days. Most of the time these “heavy hitters” run mission critical applications so it’s not unusual setting memory reservations to guarantee the availability of memory resources. If such a virtual machine is placed in a HA cluster, these significant memory reservations can lead to a very conservative consolidation ratio, due to the impact on HA slot size calculation. (For more information about slot size calculation, please review the HA deep dive page on yellow-bricks.com.) There are options to avoid creation of large slot sizes. Such as not setting reservations, disabling strict admission control, using vSphere new admission control policy “percentage of cluster resources reserved” or creating a custom slot size by altering the advanced settings das.vmMemoryMinMB. But what if you are still using ESX 3.5, must guarantee memory resources for that specific VM, do not want to disable strict admission control or don’t like tinkering with the custom slot size setting? Maybe using the resource pool workaround can be an option. Resource pool workaround During a conversation with my colleague Craig Risinger, author of the very interesting article “The resource pool priority pie paradox”, we discussed the lack of relation between resource pools reservation settings and High Availability. As Craig so eloquently put it: “RP reservations will not muck around with HA slot sizes” High Availability ignores resource pools reservation settings when calculating the slot size, so if a single VM is placed in a resource pools with memory reservation configured, it will have the same effect on resource allocation as per VM memory reservation, but does not affect the HA slot size. By creating a resource pool with a substantial memory setting you can avoid decreasing the consolidation ratio of the cluster and still guarantee the virtual machine its resources. Publishing this article does not automatically mean that I’m advocating using this workaround on a regular basis. I recommend implementing this workaround very sparingly as creating a RP for each VM creates a lot of administrative overhead and makes the host and cluster view a very unpleasant environment to work in. A possible scenario to use this workaround can be when implementing MS Exchange 2010 mailbox servers. These mailbox servers are notorious for demanding a huge amount of memory and listed by many organizations as mission critical servers. To emphasize it once more, this is not a best practice! But it might be useful in certain scenarios to avoid large slots and therefore low consolidation ratios. ================================================================================ Title: Impact of host local VM swap on HA and DRS URL: https://frankdenneman.ai/2010-02-15-impact-of-host-local-vm-swap-on-ha-and-drs/ Date: 2010-02-15 On a regular basis I come across NFS based environments where the decision is made to store the virtual machine swap files on local VMFS datastores. Using host-local swap can affect DRS load balancing and HA failover in certain situations. So when designing an environment using host-local swap, some areas must be focused on to guarantee HA and DRS functionality. VM swap file Lets start with some basics, by default a VM swap file is created when a virtual machine starts, the formula to calculate the swap file size is: configured memory – memory reservation = swap file. For example a virtual machine configured with 2GB and a 1GB memory reservation will have a 1GB swap file. Reservations will guarantee that the specified amount of virtual machine memory is (always) backed by ESX machine memory. Swap space must be reserved on the ESX host for the virtual machine memory that is not guaranteed to be backed by ESX machine memory. For more information on memory management of the ESX host, please the article on the impact of memory reservation. During start up of the virtual machine, the VMkernel will pre-allocate the swap file blocks to ensure that all pages can be swapped out safely. A VM swap file is a static file and will not grow or shrink not matter how much memory is paged. If there is not enough disk space to create the swap file, the host admission control will not allow the VM to be powered up. Note: If the local VMFS does not have enough space, the VMkernel tries to store the VM swap file in the working directory of the virtual machine. You need to ensure enough free space is available in the working directory otherwise the VM is still not allowed to be powered up. Let alone ignoring the fact that you initially didn’t want the VM swap stored on the shared storage in the first place. This rule also applies when migrating a VM configured with a host-local VM swap file as the swap file needs to be created on the local VMFS volume of the destination host. Besides creating a new swap file, the swapped out pages must be copied out to the destination host. It’s not uncommon that a VM has pages swapped out, even if there is not memory pressure at that moment. ESX does not proactively return swapped pages back into machine memory. Swapped pages always stays swapped, the VM needs to actively access the page in the swap file to be transferred back to machine memory but this only occurs if the ESX host is not under memory pressure (more than 6% free physical memory). Copying host-swap local pages between source- and destination host is a disk-to-disk copy process, this is one of the reasons why VMotion takes longer when host-local swap is used. Real-life scenario A customer of mine was not aware of this behavior and had discarded the multiple warnings of full local VMFS datastores on some of their ESX hosts. All the virtual machines were up and running and all seemed well. Certain ESX servers seemed to be low on resource utilization and had a few active VMs, while other hosts were highly utilized. DRS was active on all the clusters, fully automated and a default (3 stars) migration threshold. It looked like we had a major DRS problem. DRS If DRS decide to rebalance the cluster, it will migrate virtual machines to low utilized hosts. VMkernel tries to create a new swap file on the destination host during the VMotion process. In my scenario the host did not contain any free space in the VMFS datastore and DRS could not VMotion any virtual machine to that host because the lack of free space. But the host CPU active and host memory active metrics were still monitored by DRS to calculate the load standard deviation used for its recommendations to balance the cluster. (More info about the DRS algorithm can be found on the DRS deepdive page). The lack of disk space on the local VMFS datastores influenced the effectiveness of DRS and limited the options for DRS to balance the cluster. High availability failover The same applies when a HA isolation response occurs, when not enough space is available to create the virtual machine swap files, no virtual machines are started on the host. If a host fails, the virtual machines will only power-up on host containing enough free space on their local VMFS datastores. It might be possible that virtual machines will not power-up at-all if not enough free disk space is available. Failover capacity planning When using host local swap setting to store the VM swap files, the following factors must be considered. • Amount of ESX hosts inside cluster. • HA configured host failover capacity. • Amount of active virtual machines inside cluster. • Consolidation ratio (VM per host). • Average swap file size. • Free disk space local VMFS datastores. Number of hosts inside cluster: 6 HA configured host failover capacity: 1 Active virtual machines: 162 Average consolidation ratio: 27:1 Average memory reservation: 0GB Average swap file size: 4GB For the sake of simplicity, let’s assume that DRS balanced the cluster load and that all (identical) virtual machines are spread evenly across every host. In case of a host failure, 27 VMs will be restarted on the remaining 5 hosts inside the cluster, HA will start 5.4 virtual machines per host, as it is impossible to start 0.4 VM, some ESX hosts will start 6 virtual machines, while other hosts will start 5 VM’s. The average swap file size is 4GB, this requires at least 24 GB of free space to be available on the local VMFS datastores to start the VM’s. Besides the 24GB, enough free space needs to be available to for DRS to move multiple VMs around to rebalance the load across the cluster. If the design of the virtual infrastructure incorporates site failover as well, enough free disk space on all the ESX hosts must be reserved to power-up all the affected virtual machines from the failed site. Closing remarks Using host local swap can be a valid option for some environments, but additional calculation of the factors mentioned above is necessary to ensure sustained HA and DRS functionality. ================================================================================ Title: VCDX number 029 URL: https://frankdenneman.ai/2010-02-12-vcdx-number-29/ Date: 2010-02-12 Monday 8th of February I was scheduled to participate in the defend session of the VCDX panel at Las Vegas. For people not familiar with the VCDX program, the defend panel is the final part of the extensive VCDX program. My defend session was the first session of the week, so my panel members where fresh and eager to get started. Besides the three panel members, an observer and a facilitator where also present in the room. The session consisted out of three parts; • Design defend session (75 minutes) • Design session (30 minutes) • Troubleshooting session (15 minutes) During the design defend session you are required to present your design, I used a twelve deck slide presentation and included all blueprints\Visio drawings as appendix. This helped me a lot, as I am not a native English speaker using diagrams helped me to explain the layout. There is no time limit on the duration of the presentation, but it is wise to keep it as brief as possible. During the session, the panel will try to address a number of sections and if they cannot address these sections this can impact your score. The design and troubleshooting session you need to show you are able to think on your feet. One of the goals is to understand your though process. Thinking out loud and using the whiteboard will help you a lot. So how was my experience? After meeting my panel members I started to get really nervous as one of the storage guru’s within VMware was on my panel. The other two panel members have an extreme good track record inside the company as well, so basically I was being judged by an all-star panel. I thought my presentation went well, but word of advice; read your submitted documentation on a regular basis before entering the defend panel as the smallest details can be asked. After completing the design defend pane, I was asked to step outside. After the short break the design session and troubleshooting scenarios were next. I did not solve the design and troubleshooting scenarios, but that is really not the goal of those sections. Thinking out loud in English can be challenging for non-native English speakers, so my advice is to try to practice this as much as possible. I did a test presentation for a couple of friends and discovered some areas to focus on before doing the defend part of the program. After completing my defend panel, I was scheduled to participate as an observer on the remaining defend panel sessions the rest of the week. After multiple sessions as an observer and receiving the news that I passed the VCDX defend panel, I participated as a panel member on a defend session. Hopefully I will be on a lot more panels in the upcoming year, because sitting on the other side of the table is so much better than standing in front of it sweating like a pig. :) ================================================================================ Title: Sizing VMs and NUMA nodes URL: https://frankdenneman.ai/2010-02-03-sizing-vms-and-numa-nodes/ Date: 2010-02-03 Note: This article describes NUMA scheduling on ESX 3.5 and ESX 4.0 platform, vSphere 4.1 introduced wide NUMA nodes, information about this can be found in my new article: ESX4.1 NUMA scheduling With the introduction of vSphere, VM configurations with 8 CPUs and 255 GB of memory are possible. While I haven’t seen that much VM’s with more than 32GB, I receive a lot of questions about 8-way virtual machines. With today’s CPU architecture, VMs with more than 4 vCPUs can experience a decrease in memory performance when used on NUMA enabled systems. While the actually % of performance decrease depends on the workload, avoiding performance decrease must always be on the agenda of any administrator. Does this mean that you stay clear of creating large VM’s? No need to if the VM needs that kind of computing power, but the reason why I’m writing this is that I see a lot of IT departments applying the same configuration policy used for physical machines. A virtual machine gets configured with multiple CPU or loads of memory because it might need it at some point during its lifecycle. While this method saves time, hassle and avoid office politics, this policy can create unnecessary latency for large VMs. Here’s why: NUMA node Most modern CPU’s, Intel new Nehalem’s and AMD’s veteran Opteron are NUMA architectures. NUMA stands for Non-Uniform Memory Access, but what exactly is NUMA? Each CPU get assigned its own “local” memory, CPU and memory together form a NUMA node. An OS will try to use its local memory as much as possible, but when necessary the OS will use remote memory (memory within another NUMA node). Memory access time can differ due to the memory location relative to a processor, because a CPU can access it own memory faster than remote memory. Figure 1: Local and Remote memory access Accessing remote memory will increase latency, the key is to avoid this as much as possible. How can you ensure memory locality as much as possible? VM sizing pitfall #1, vCPU sizing and Initial placement. ESX is NUMA aware and will use the NUMA CPU scheduler when detecting a NUMA system. On non-NUMA systems the ESX CPU scheduler spreads load across all sockets in a round robin manner. This approach improves performance by utilizing as much as cache as possible. When using a vSMP virtual machine in a non-NUMA system, each vCPU is scheduled on a separate socket. On NUMA systems, the NUMA CPU scheduler kicks in and use the NUMA optimizations to assigns each VM to a NUMA node, the scheduler tries to keep the vCPU and memory located in the same node. When a VM has multiple CPUs, all the vCPUs will be assigned to the same node and will reside in the same socket, this is to support memory locality as much as possible. Figure 2: NON-NUMA vCPU placement Figure 3: NUMA vCPU placement At this moment, AMD and Intel offer Quad Core CPU’s, but what if the customer decides to configure an 8-vCPU virtual machine? If a VM cannot fit inside one NUMA node, the vCPUs are scheduled in the traditional way again and are spread across the CPU’s in the system. The VM will not benefit from the local memory optimization and it’s possible that the memory will not reside locally, creating added latency by crossing the intersocket connection to access the memory. VM sizing pitfall #2: VM configured memory sizing and node local memory size NUMA will assign all vCPU’s to a NUMA node, but what if the configured memory of the VM is greater than the assigned local memory of the NUMA node? Not aligning the VM configured memory with the local memory size will stop the ESX kernel of using NUMA optimizations for this VM. You can end up with all the VM’s memory scattered all over the server. So how do you know how much memory every NUMA node contains? Typically each socket will get assigned the same amount of memory; the physical memory (minus service console memory) is divided between the sockets. For example 16GB will be assigned to each NUMA node on a two socket server with 32GB total physical. A quick way to confirm the local memory configuration of the NUMA nodes is firing up esxtop. Esxtop will only display NUMA statistics if ESX is running on a NUMA server. The first number list the total amount of machine memory in the NUMA node that is managed by ESX, the statistic displayed within the round brackets is the amount of machine memory in the node that is currently free. Figure 4: esxtop memory totals Let’s explore NUMA statistics in esxtop a little bit more based on this example. This system is a HP BL 460c with two Nehalem quad cores with 64GB memory. As shown, each NUMA node is assigned roughly 32GB. The first node has 13GB free; the second node has 372 MB free. It looks it will run out of memory space soon, luckily the VMs on that node still can get access remote memory. When a VM has a certain amount of memory located remote, the ESX scheduler migrates the VM to another node to improve locality. It’s not documented what threshold must be exceeded to trigger the migration, but its considered poor memory locality when a VM has less than 80% mapped locally, so my “educated” guess is that it will be migrated when the VM hit a number below the 80%. Esxtop memory NUMA statistics show the memory location of each VM. Start esxtop, press m for memory view, press f for customizing esxtop and press f to select the NUMA Statistics. Figure 5: Customizing esxtop Figure 6 shows the NUMA statistics of the same ESX server with a fully loaded NUMA node, the N%L field shows the percentage of mapped local memory (memory locality) of the virtual machines. Figure 6: esxtop NUMA statistics It shows that a few VMs access remote memory. The man pages of esxtop explain all the statistics: Metric Explanation NHN Current Home Node for virtual machine NMIG Number of NUMA migrations between two snapshots. It includes balance migration, inter-mode VM swaps performed for locality balancing and load balancing NRMEM (MB) Current amount of remote memory being accessed by VM NLMEM (MB) Current amount of local memory being accessed by VM N%L Current percentage memory being accessed by VM that is local GST_NDx (MB) The guest memory being allocated for VM on NUMA node x. “x” is the node number OVD_NDx (MB) The VMM overhead memory being allocated for VM on NUMA node x Transparent page sharing and memory locality. So how about transparent page sharing (TPS), this can increase latency if the VM on node 0 will share its page with a VM on node 1. Luckily VMware thought of that and TPS across nodes is disabled by default to ensure memory locality. TPS still works, but will share identical pages only inside nodes. The performance hit of accessing remote memory does not outweigh the saving of shared pages system wide. Figure 7: NUMA TPS boundaries This behavior can be changed by altering the setting VMkernel.Boot.sharePerNode. As most default settings in ESX, only change this setting if you are sure that it will benefit your environment, 99.99% of all environments will benefit from the default setting. Take away With the introduction of vSphere ESX 4, the software layer surpasses some abilities current hardware techniques can offer. ESX is NUMA aware and tries to ensure memory locality, but when a VM is configured outside the NUMA node limits, ESX will not apply NUMA node optimizations. While a VM still run correctly without NUMA optimizations, it can experience slower memory access. While the actually % of performance decrease depends on the workload, avoiding performance decrease if possible must always be on the agenda of any administrator. To quote the resource management guide: The NUMA scheduling and memory placement policies in VMware ESX Server can manage all VM transparently, so that administrators do not need to address the complexity of balancing virtual machines between nodes explicitly. While this is true, administrators must not treat the ESX server as a black box; with this knowledge administrators can make informed decisions about their resource policies. This information can help to adopt a scale-out policy (multiple smaller VMs) for some virtual machines instead of a scale up policy (creating large VMs) if possible. Beside the preference for scale up or scale out policy, a virtual environment will profit when administrator choose to keep the VMs as agile as possible. My advice to each customer is to configure the VM reflecting its current and near future workload and actively monitor its habits. Creating the VM with a configuration which might be suitable for the workload somewhere in its lifetime can have a negative effect on performance. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Top 25 Virtualization Bloggers URL: https://frankdenneman.ai/2010-01-21-top-25-virtualization-bloggers/ Date: 2010-01-21 This week Eric Siebert processed all the votes and published this year’s Top 25 VMware/Virtualization Blogger list. Over 700 people voted, each casting 10 votes. I can only imagine the work involved that is put in to producing this list, so a big thank you goes out to Eric for voluntary organizing this! Awesome! This year a lot of new names entered the top 25 including my blog. I never ever expected to see my name published in the top 25. I’m truly honored to make it to the list, let alone be voted number #14 so I would like to thank everyone for voting for me! I really appreciate it! Congratulations to all other people mentioned in the list and I would like to congratulate Duncan Epping specifically for taking the number 1 place again this year. The top 25 as published by Eric Siebert on vSphere-land: 1 Yellow Bricks Duncan Epping 2 Virtual Geek Chad Sakac 3 blog.scottlowe.org Scott Lowe 4 ntpro.nl Eric Sloof 5 rtfm-ed.co.uk Mike Laverick 6 [Boche.net](http://www.boche.net/blog/ /) Jason Boche 7 VM/etc Rich Brambley 8 [gabesvirtualworld.com](http://www.gabesvirtualworld.com/ /) Gabrie van Zanten 9 virtualstorageguy Vaughn Stewart 10 virtu-al.net Alan Renouf 1 1 [virtualization-pro](http:// http://itknowledgeexchange.techtarget.com/virtualization-pro/) Various 12 vcritical.com Eric Gray 13 vmwaretips.com Rick Scherer 14 frankdenneman.nl Frank Denneman 15 vmguy.com Dave Lawrence 16 [planetvm.net](http:// http://planetvm.net/blog/) Tom Howarth 17 The Slog Simon Long 18 vmguru.nl Various 19 Mike D’s Blog Mike DiPetrillo 20 Hypervizor.com Hany Michael 21 [techhead.co.uk](http:// www.techhead.co.uk/) Simon Seagrave 22 vreference.com Forbes Guthrie 23 Pivot Point Scott Drummonds 24 TechnoDrone Maish Saidel-Keesing 25 chriswolf.com Chris Wolf A new home Being voted one of the top 25 bloggers, puts a lot of pressure on one. I hope to continue blogging articles people find interesting. And to make a good start, frankdenneman.wordpress.com moved to www.frankdenneman.nl. ================================================================================ Title: Voting closes at Friday! URL: https://frankdenneman.ai/2010-01-11-get-your-votes-in-while-it-can-voting-closes-at-friday/ Date: 2010-01-11 Eric Siebert of vsphere-land.com started a new election of the best 20 bloggers in the VMware and Virtualization scene. Because even more top blogs got started in 2009, Eric decided to expand the top 20 to the top 25. To my suprise, Eric decided to nominate my blog as well. I’m really honored to be a nominee amongst the best virtualization bloggers out there. Unfortunately the list Eric is longer that the 10 votes one can cast, so good luck picking the ones who stand out above the excellent crowd. This is my top 10 blogs; Duncan Epping Chad Sakac Kenneth van Ditmarsch Alan Renouf Scott Lowe Scott Drummonds Hypervizor (Hany Michael) Arnim van Lieshout Arne Fokkema Gabe Virtual world. Go vote now before it’s too late! http://vsphere-land.com/news/time-to-vote-for-your-favorite-bloggers.html ================================================================================ Title: Joining VMware URL: https://frankdenneman.ai/2009-12-30-joining-vmware/ Date: 2009-12-30 The year 2009 has been an interesting year. After leaving a long-term position, I participated in some really awesome projects, got to fiddle around with the cutting edge technology and got to work with some really excellent and inspiring people. Begin August I got an e-mail from Duncan Epping If I would like to do some contractors work for VMware. As you can imagine, it didn’t take me long to respond with a Font size 72 YES. (Do you know you can make text blink in word?) After completing a few project VMware offered me a permanent job, being a contractor for 9 years made the decision a bit tougher, but getting such a job offer is something you can hardly refuse. Working with the best of the business, being able to access internal information and getting exposed to all the new stuff VMware is creating is just plain awesome. So on the 4th of January I will be joining VMware as the new Senior PSO Consultant. ================================================================================ Title: Wrap-up 2009 URL: https://frankdenneman.ai/2009-12-30-wrap-up-2009/ Date: 2009-12-30 Daniel Easons blog post inspired me to write a wrap-up of 2009 myself. Beside the career move described in the previous post, 2009 was a year of finding two new addictions. Blogging and twitter(@frankdenneman). Beginning of February I started blogging and the first article was received pretty well. The article got mentioned on Yellow Bricks the same day. Now 10 months later, more than 37.000 people visited the site. It cannot hold a candle to the great blogs out there, but it’s nice start. Some of my articles appeared in a few Top 5 Planetv12n lists, got mentioned on Yellow Bricks and were featured in Scott Lowe’s virtualization short takes. Trying to create in-depth articles is an excellent way to learn stuff. Most of the time describing a certain subject somehow challenged my current knowledge of that topic and ended up spending ridiculous amounts of time researching that particular subject. More often than not coming across very interesting material not really related to the subject but similarly interesting, consuming even more time. Some articles are more popular than others; these are the top five visited articles this year; 1. Increasing the queue depth 2. Lefthand SAN – Lessons learned 3. HP Continuous Access and the use of LUN balancing scripts 4. Impact of memory reservations 5. NFS and IP-HASH Load-Balancing Lately I’m running into a few limitations of the free wordpress blog themes, that’s why I’ve decided to move to another site, stay tuned for the URL. I’m aiming to release the new site at the beginning of next year. ================================================================================ Title: vSphere 4.0 Quick Start Guide review URL: https://frankdenneman.ai/2009-12-24-vsphere-4-0-quick-start-guide-review/ Date: 2009-12-24 un•put•down•a•ble Pronunciation: (un"poot-dou’nu-bul), [key] —adj. Informal. Adjective meaning consistently and irresistibly interesting. Typically refers to a book that is so well written and entertaining as to be difficult to (literally) put down and pause away from. Normally a term used to describe novels, but the vSphere Quick Start Guide certainly fits the definition. Last month I was finishing three major projects and needed to write my VCDX application in one week, but somehow it kept ending up in my hands. So what’s so special about this book and how does it distinguish itself from the competition? The book central theme is providing tips and ‘how to’s’ and it does this rather well. The book handles the traditional subjects, such like vCenter, Host, Virtual Machines, Networking and Storage. Besides the concise, easy to follow and non-ambiguous way the tips are written, I really like the minimal use of screenshots. This allowed using the (limited) space to contain as much content as possible. Besides describing how to change settings via the Service Console CLI and the GUI, most tips also list PowerCLI and RemoteCLI example scripts. Incorporating PowerCLI scripts allows this book to be of value to the more experienced administrator who is using PowerCLI or RemoteCLI to manage its environment. The examples certainly increased my interest of picking up PowerCLI. But what really makes this book shine is the short in-depth text accompanying most of the tips and how to’s. The text contains valuable information on how certain mechanism works, what impact changing a setting can have and field experience of using certain settings. Added bonus is addressing the possibility of using third-party tools such as Dell expart, EMC powerpath VE, vwire and many others, confirming that this book is written by authors with true field experience. I really recommend this book to anyone who is using VMware ESX. It doesn’t matter if you are a novice administrator or a seasoned consulting architect, you WILL learn something new by reading this book. During the ESX 2.5 era, anyone who was serious about his job owned the Advanced Technical Design Guide, in the current vSphere era it’s clear that this book must be on your desk. ================================================================================ Title: VMware updates Timekeeping best practices URL: https://frankdenneman.ai/2009-12-22-vmware-updates-timekeeping-best-practices-for-windows-vms/ Date: 2009-12-22 A couple of weeks ago I discovered that VMware updated its timekeeping best practices for Linux virtual machines. December 7th VMware published a new best practice of timekeeping in Windows VMs. (KB1318) VMware now recommends to use either W32Time or NTP for all virtual machines. This a welcome statement from VMware ending the age old question while designing a Virtual Infrastructure; Do we use VMware tools time sync or do we use W32time? If we use VMware tools, how do we configure the Active Directory controller VMs? VMware Tools can still be used and still function well enough for most non time sensitive application. VMware tools time sync is excellent in accelerating and catching up time if the time that is visible to virtual machines (called apparent time) is going slowly, but W32time and NTP can do one thing that VMware tools time sync can’t, that’s slowing down time. Page 15 of the (older) white paper: Timekeeping in VMware Virtual Machines http://www.vmware.com/pdf/vmware_timekeeping.pdf explains the issue. However, at this writing, VMware Tools clock synchronization has a serious limitation: it cannot correct the guest clock if it gets ahead of real time (except in the case of NetWare guest operating systems). For more info about timekeeping best practices for Windows VMs, please check out KB article 1318 http://kb.vmware.com/kb/1318 It appears that VMware updated the Timekeeping best practices for Linux guests as well. http://kb.vmware.com/kb/1006427 (9 december 2009) ================================================================================ Title: ESX 4i support Jumbo Frames URL: https://frankdenneman.ai/2009-12-19-esx-4i-support-jumbo-frames/ Date: 2009-12-19 Last week I blogged about jumbo frames being unsupported in ESX 4i. Yesterday Charu Chaubal, Sr. Technical Marketing Architect at VMware blogged the following; I am happy to say that this is merely an error in the documentation. In fact, ESXi 4.0 DOES support Jumbo Frames on VMkernel networking interfaces. The correction will hopefully appear in a new release of the documentation, but in the meantime, go ahead and configure Jumbo frames for your ESXi 4.0 hosts. http://blogs.vmware.com/esxi/2009/12/esxi-40-supports-jumbo-frames.html Although Jumbo frames being unsupported might not be an adopion blocker, this is quite good news for companies willing to use ESX 4i. ================================================================================ Title: Timesavers for VCDX application URL: https://frankdenneman.ai/2009-12-16-timesavers-for-vcdx-application/ Date: 2009-12-16 Last week VMware send out the invitations for the VCDX defend session at the Partner Exchange Las Vegas 2010. Like many others I’m trying to finish my application on time. So any help, shortcut and timesavers will help realize the goal. At this moment these tools and shortcuts save me lots of time: Puretext Steve miller created a simple but awesome tool. Puretext will strip any formatting while pasting text. This IS the lifesaver for me at the moment, because I’m copying text from older documents with different Fonttype and size. Run the small exe file and start loving the windows key + v command. http://www.stevemiller.net/puretext/ Visio shapes Besides using graphics from the official VMware Branding Team, I also use visio shapes from the Xtravirt Presentation Pack 2.1. (needs registration) http://viops.vmware.com/home/servlet/JiveServlet/download/1514-2-5957/VMware-Stencil1-vSphere.zip http://viops.vmware.com/home/servlet/JiveServlet/download/1514-2-5966/VMware-Stencil2-vSphere.ziph http://xtravirt.com/presentation-pack Visiocafe will offer some really sweet vendor shapes, this will make your presentation look even more impressive :) http://www.visiocafe.com/vsdfx.htm Visio shortcuts: Align Shapes: F8 Duplicate: CTRL + D Group: CTRL + G Ungroup: CTRL + Shift + U Fill: F3 Line format: ALT O L (ALT O = letter o) Pointer tool: CTRL + 1 Text tool: CTRL + 2 Line tool: CTRL + 6 Rectangle tool: CTRL + 8 Centre Text: CTRL+Shift+C Bring to Front: CTRL+Shift_F Actual size: CTRL + Shift + i Whole page: CTRL + W If you have a tip, please feel free to comment ================================================================================ Title: Impact of mismatch Guest OS type URL: https://frankdenneman.ai/2009-12-15-impact-of-mismatch-guest-os-type/ Date: 2009-12-15 During Healthchecks I frequently encounter virtual machines configured with the incorrect Guest OS type specified. Incorrect configuration of Guest OS of the virtual machine can lead to; • Reduction of performance • Different default type for the SCSI device * • Different defaults of devices • Wrong VMware Tools presented to the Guest OS resulting in failure to install • Inability to select virtual hardware such as enhanced vmxnet, vmxnet3 or number of vCPUs. • Inability to activate features such as CPU and Memory Hot Add. • Inability to activate Fault Tolerance. • VM burning up 100% of CPU when idling (rare occasions) Buslogic SCSI Device * Due to mismatch of Guest OS Type, windows 2000 and Windows 2003 can be configured with a Buslogic SCSI device. Using the Buslogic virtual adapter with Windows 2000 and 2003 will limit the effective IO queue depth of one. This limits disk throughput severely and lead to serious performance degradation. For more information visit KB article [1614](http://kb.vmware.com/selfservice/microsites/search.do?cmd=displayKC&docType=kc&externalId=1614&sliceId=1&docTypeID=DT_KB_1_1&dialogID=53698934&stateId=0 0 53706353) Virtual Machine Monitor and execution mode Selecting the wrong Guest OS type can be of influence of the selected execution mode. When a virtual machine is powering on, the VMM inspects the physical CPU’s features and the guest operating system type to determine the set of possible execution modes. This can have a slight impact on performance and in some extreme cases application crashes or BSODs. VMware published a Must-Read whitepaper about the VMM and execution modes http://www.vmware.com/files/pdf/software_hardware_tech_x86_virt.pdf How to solve the mismatch? vCenter only displays the configured Guest OS of the Virtual Machine, it will not check the installed operating system inside the virtual machine. Powercli offers the solution to this problem, today more and more people start to discover the beauty of Powercli and incorporate this in their day-to-day activities. So I’ve asked PowerCLI guru Alan Renouf if he could write a PowerCLI script which can detect the Guest OS mismatch. Get-View -ViewType VirtualMachine | Where { $_.Guest.GuestFullname} | Sort Name |Select-Object Name, @{N=“SelectedOS”;E={$_.Guest.GuestFullName}}, @{N=“InstalledOS”;E={$_.Summary.Config.GuestFullName}} | Out-GridView Alans “one-liner” checks the configured Gues Os Type in the VM (VM properties) and queries the VMtools to see which operating system it reports. Once the mismatch is identified, set the correct Guest OS Type in the VM properties as soon as possible. The best way to deal with the mismatch is to power-down the VM before changing the guest OS type. ================================================================================ Title: Impact of memory reservation URL: https://frankdenneman.ai/2009-12-08-impact-of-memory-reservation/ Date: 2009-12-08 I have a customer who wants to set memory reservation on a large scale. Instead of using resource pools they were thinking of setting reservations on VM level to get a guaranteed performance level for every VM. Due to memory management on different levels, using such a setting will not get the expected results. Setting aside the question if it’s smart to use memory reservation on ALL VM’s, it raises the question what kind of impact setting memory reservation has on the virtual infrastructure, how ESX memory management handles memory reservation and even more important; how a proper memory reservation can be set. Key elements of the memory system Before looking at reservations, let’s take a look what elements are involved. There are three memory layers in the virtual infrastructure: • Guest OS virtual memory - Virtual Page Number (VPN) • Guest OS physical memory - Physical Page Number (PPN) • ESX machine memory - Machine Page Number (MPN) The OS inside the guest maps virtual memory ( VPN) to physical memory(PPN). The Virtual Machine Monitor (VMM) maps the PPN to machine memory (MPN). The focus of this article is on mapping physical page numbers (PPN) to Machine Page Number (MPN). Impact of memory management on the VM Memory reservations guarantee that physical memory pages are backed by machine memory pages all the time, whether the ESX server is under memory pressure or not. Opposite of memory reservations are limits. When a limit is configured, the memory between the limit and the configured memory will never be backed by machine memory; it could either be reclaimed by the balloon driver or swapped even if enough free memory is available in the ESX sever. Next to reservations and limits, shares play an important factor in memory management of the VM. Unlike memory reservation, shares are only of interest when contention occurs. The availability of memory between memory reservation and configured memory depends on the entitled shares compared to the total shares allocated to all the VMs on the ESX server. This means that the virtual machine with the most shares can have its memory backed by physical pages. For the sake of simplicity, the vast subject of resource allocation based on the proportional share system will not be addressed in this article. One might choose to set the memory reservation equal to the configured memory, this will guarantee the VM the best performance all of the time. But using this “policy” will have its impact on the environment. Admission Control Configuring memory reservation has impact on admission control . There are three levels of admission control; • Host • High Availability • Distributed Resource Scheduler Host level When a VM is powered on, admission control checks the amount of available unreserved CPU and memory resources. If ESX cannot guarantee the memory reservation and the memory overhead of the VM, the VM is not powered on. VM memory overhead is based on Guest OS, amount of CPUs and configured memory, for more information about memory overhead review the Resource management guide. HA and DRS Admission control also exist at HA and DRS level. HA admission control uses the configured memory reservation as a part of the calculation of the cluster slot size.The amount of slots available equals the amount of VM’s that can run inside the cluster. To find out more about slot sizes, read the HA deepdive article of Duncan Epping. DRS admission control ignores memory reservation, but uses the configured memory of the VM for its calculations. To learn more about DRS and its algorithms read the DRS deepdive article at yellow-bricks.com Virtual Machine Swapfile Configuring memory reservation will have impact on the size of the VM swapfile; the swapfile is (usually) stored in the home directory of the VM. The virtual machine swapfile is created when the VM starts. The size of the swapfile is calculated as follows: Configured memory – memory reservation = size swap file Configured memory is the amount of “physical” memory seen by guest OS. For example; configured memory of VM is 2048MB – memory reservation of 1024MB = Swapfile size = 1024MB. ESX use the memory reservation setting when calculating the VM swapfile because reserved memory will be backed by machine memory all the time. The difference between the configured memory and memory reservation is eligible for memory reclamation. Reclaiming Memory Let’s focus a bit more on reclaiming. Reclaiming of memory is done by ballooning or swapping. But when will ESX start to balloon or swap? ESX analyzes its memory state. The VMkernel will try to keep 6% free (Mem.minfreepct) of its memory. (physical memory-service console memory) When free memory is greater or equal than 6%, the VMkernel is in a HIGH free memory state. In a high free memory state, the ESX host considers itself not under memory pressure and will not reclaim memory in addition to the default active Transparent Page sharing process. When available free memory drops below 6% the VMkernel will use several memory reclamation techniques. The VMkernel decides which reclamation technique to use depending on its threshold. ESX uses four thresholds high (6%), soft (4%) hard (2%) and low (1%). In the soft state (4% memory free) ESX prefers to use ballooning, if free system memory keeps on dropping and ESX will reach the Hard state (2% memory free) it will start to swap to disk. ESX will start to actively reclaim memory when it’s running out of free memory, but be aware that free memory does not automatically equal active memory. Memory reservation technique Let’s get back to memory reservation .How does ESX handle memory reservation? Page 17 of the Resource Management Guide states the following: Memory Reservation If a virtual machine has a memory reservation but has not yet accessed its full reservation, the unused memory can be reallocated to other virtual machines. Memory Reservation Used Used for powered‐on virtual machines, the system reserves memory resources according to each virtual machine’s reservation setting and overhead. After a virtual machine has accessed its full reservation, ESX Server allows the virtual machine to retain this much memory, and will not reclaim it, even if the virtual machine becomes idle and stops accessing memory. To recap the info stated in the Resource Management Guide, when a VM hits its full reservation, ESX will never reclaim that amount of reserved memory even if the machine idles and drops below its guaranteed reservation. It cannot reallocate that machine memory to other virtual machines. Full reservation But when will a VM hit its full reservation exactly? Popular belief is that the VM will hit full reservation when a VM is pushing workloads, but that is not entirely true. It also depends on the Guest OS being used by the VM. Linux plays rather well with others, when Linux boots it only addresses the memory pages it needs. This gives ESX the ability to reallocate memory to other machines. After its application or OS generates load, the Linux VM can hit its full reservation. Windows on the other hand zeroes all of its memory during boot, which results in hitting the full reservation during boot time. Full reservation and admission control This behavior will have impact on admission control. Admission control on the ESX server checks the amount of available unreserved CPU and memory resources. Because Windows will hit its full reservation at startup, ESX cannot reallocate this memory to other VMs, hereby diminishing the amount of available unreserved memory resources and therefore restricting the capacity of VM placement on the ESX server. But memory reclamation, especially TPS will help in this scenario, TPS (transparent page sharing) reduces redundant multiple guest pages by mapping them to a single machine memory page. Because memory reservation “lives” at machine memory level and not at virtual machine physical level, TPS will reduce the amount of reserved machine memory pages, memory pages that admission controls check when starting a VM. Transparant Page Sharing TPS cannot collapse pages immediately when starting a VM in ESX 3.5. TPS is a process in the VMkernel; it runs in the background and searches for redundant pages. Default TPS will have a cycle of 60 minutes (Mem.ShareScanTime) to scan a VM for page sharing opportunities. The speed of TPS mostly depends on the load and specs of the Server. Default TPS will scan 4MB/sec per 1 GHz. (Mem.ShareScanGHz). Slow CPU equals slow TPS process. (But it’s not a secret that a slow CPU will offer less performance that a fast CPU.) TPS defaults can be altered, but it is advised to keep to the default.TPS cannot collapse pages immediately when starting a VM in ESX 3.5. VMware optimized memory management in ESX 4; pages which Windows initially zeroes will be page-shared by TPS immediately. TPS and large pages One caveat, TPS will not collapse large pages when the ESX server is not under memory pressure. ESX will back large pages with machine memory, but installs page sharing hints. When memory pressure occurs, the large page will be broken down and TPS can do it’s magic. More info on Large pages and ESX can be found at Yellow Bricks. http://www.yellow-bricks.com/2009-05-31-nehalem-cpu-and-tps-on-vsphere/ Use resource pools Setting memory reservation has impact on the VM itself and its surroundings. Setting reservation per VM is not best practice; it is advised to create resource pools instead of per VM reservations. Setting reservations on a granular level leads to increased administrative and operational overhead. But when the situation demands to use per VM reservation, in which way can a reservation be set to guarantee as much performance as possible without wasting physical memory and with as less impact as possible. The answer: set reservation equal to the average Guest Memory Usage of the VMs. Guest Memory Usage Guest Memory Usage shows the active memory use of the VM. Which memory is considered active memory? If a memory page is accessed in mem.sampleperiod (60sec), it is considered active. To accomplish this you need to monitor each VM, but this is where vCenter comes to the rescue. vCenter logs performance data and does this for a period of time. The problem is that the counters average-, minimum and maximum active memory data is not captured on the default vCenter statistics. vCenter logging level needs to upgraded to a minimum level of 4. After setting the new level, vCenter starts to log the data. Changing the statistic setting can be done by Administration > VirtualCenter Management Server Configuration > Statistics. To display the average active memory of the VM, open the performance tab of the VM and change chart options, select memory Select the counters consumed memory and average-, minimum- and maximum active memory. The performance chart of most VMs will show these values close to each other. As a rule the average active memory figure can be used as input for the memory reservation setting, but sometimes the SLA of the VM will determine that it’s better to use the maximum active memory usage. Consumed memory is the amount of host memory that is being used to back guest memory. The images shows that memory consumed slowly decreases. The active memory use does not change that much during the monitored 24 hours. By setting the reservation equal to the maximum average active memory value, enough physical pages will be backed to meet the VM’s requests. My advice While memory reservation is an excellent mechanism to guarantee memory performance levels of a virtual machine, setting memory reservation will have a positive impact on the virtual machine itself and can have a negative impact on its surroundings. Memory reservation will ensure that virtual machine memory will be backed by physical memory (MPN) of the ESX host server. Once the VM hit its full reservation the VMkernel will not reclaim this memory, this will reduce the unreserved memory pool. This memory pool is used by admission control, admission control will power up a VM machine only if it can ensure the VMs resource request. The combination of admission control and the restraint of not able to allocate reserved memory to other VMs can lead to a reduced consolidation ratio. Setting reservations on a granular level leads to increased administrative and operational overhead and is not best practice. It is advised to create resource pools instead of per VM reservations. But if a reservation must be set, use the real time counters of VMware vCenter and monitor the average active memory usage. Using average active memory as input for memory reservation will guarantee performance for most of its resource requests. I recommend reading the following whitepapers and documentation; Carl A. Waldspurger. Memory Resource Management in VMware ESX Server: http://waldspurger.org/carl/papers/esx-mem-osdi02.pdf Understanding Memory Resource Management in VMware ESX: http://www.vmware.com/files/pdf/perf-vsphere-memory_management.pdf Description of other interesting memory performance counters can be found here http://www.vmware.com/support/developer/vc-sdk/visdk25pubs/ReferenceGuide/mem.html Software and Hardware Techniques for x86 Virtualization: http://www.vmware.com/files/pdf/software_hardware_tech_x86_virt.pdf Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: NFS and IP-HASH loadbalancing URL: https://frankdenneman.ai/2009-11-13-nfs-and-ip-hash-loadbalancing/ Date: 2009-11-13 My background is Fibre Channel and beginning 2009 I implemented a large iSCSI environment. The “other” storage protocol supported by VMware, NFS, is rather unknown to me. And to be honest I really tried to keep away from it as much as possible, thinking it was not a proper enterprise worthy solution. That changed this month as I was asked to perform a design review of an environment which relies completely of NFS storage. This customer decided to use IP-Hash as load-balancing policy for their NFS vSwitch, but what Impact does this have on the NFS environment? First of all, unlike the the default Port-based load balancing policy, IP-Hash has got some technical requirements. The physical switch must be configured with STATIC LACP (a.k.a. etherchannel, for Cisco switches) Dynamic LACP is not supported by VMware. KB article 1004048 focuses on the configuration of LACP between the ESX server and physical switches. Another technical requirement is that all uplinks in the vSwitch need to be active as the physicall switch is not aware of certain status of the uplink inside the vSwitch. David Marotta (VMware) explained the theory in such a way, I cannot forget this setting after hearing the story. Because the pSwitch thinks it can send traffic to a particular VM down any of the pNICs. So all pNICs should be Active. Otherwise traffic to VM-A could be sent by the pSwitch to pNIC-2, and if the vSwitch thinks pNIC-2 is not Active for the port group used by VM-A, the vSwitch will drop it. Then VM-A will never get the packet. And it will sit in a corner, lonely and depressed, wondering why nobody calls it anymore. Etherchannels and standby uplinks can introduce “Macflaps”, Duncan Epping (@DuncanYB) has written an excellent article on this as well Active / Standby etherchannels? Besides the technical requirements of IP-hash, It will not do perfect load-balancing out of the box. Remembering Ken Cline’s (@clinek) the great vSwitch debate series about vSwitches I knew you must dive in to the algorithm used by IP-Hash load balancing and pick specific IP-addresses to make IP-hash load-balancing work. But how does this IP-Hash algorithm work? As Ken cline so eloquently stated; Take an exclusive OR of the Least Significant Byte (LSB) of the source and destination IP addresses and then compute the modulo over the number of pNICs. OK, right! So how can we calculate it so we know if if the environment is balanced? Instead of the following the algorithm stated by Ken, I’ve used the method described in KB article 1007371 Waiver The ip-addresses which are used in this example are ficticious, the ip-addresses are not based on real-life addresses, if those are in use by some company, it’s purely coincidental. Step 1: Convert IP address to a HEX value Use a IP Hex Converter tool to convert the IP addresses to Hex. An online Hex converter tool can be found at http://www.kloth.net/services/iplocate.php In this example I use a vswitch with 2 uplinks. When using IP-hash, the first uplink has a IP binary representation of 0, the second uplink has a IP binary representation of 1. The IP-Hash is calculated on source IP address, the VMkernel NFS IP address and the destination address, the NFS array IP address. The VMknic has in this example the ip address of 145.10.44.10, The first IP address of the NAS is 145.10.44.80 and the second address is 145.10.44.90 HEX: ESX VMkernel: 145.10.44.10 = 910A2C0A HEX: NFS address 1: 145.10.44.80 = 910A2C50 HEX: NFS address 2: 145.10.44.90 = 910A2C5A Step 2 Calculate the binary representation of the HASH Now lets calculate the binary representation of the uplink’s IP address. I use windows for my desktop, so I use calc.exe for this example. Open calc.exe and select Programmer, select the option HEX and Qword and paste the HEX value of the VMkernel NIC Now press Xor , enter the (first) NFS IP address in HEX format (910A2C50) and click on the = button. The result is 5a, press the option Mod (modulo) and use the number of uplinks as value (2). Click on the = button to calculate the modulo. The result of this calculation is the number 0 (zero). This means that IP-hash chooses the first uplink because the hash and the uplink both have an binary representation of 0. Now lets calculate the second hash. In short; HEX value “VMkernel NIC” Xor HEX value “NFS address 2”: 910A2C0A Xor 910A2C5A = 50 MOD 2 = 0 The result of this calculation is also 0 (zero), this means that the VMkernel does not balance traffic and will send traffic to across one uplink. One ip-address of the NFS array needs to be changed to ensure that the VMkernel wil balance outbound traffic. For this example, IP-address 2: 145.10.44.90 is changed to 145.10.44.81, the HEX value of this address is 910A2C51. Now lets calculate the binary representation. 910A2C0A Xor 910A2C51 = 5B MOD 2 = 1. The result of this calculation is 1 (one) The VMkernel chooses the second uplink because it has the same binary representation of the Hash. Hereby balancing outbound NFS traffic across the two uplinks. Using IP-Hash to load-balance is a excellent choice, but you do need to fulfill certain technical requirements to get it supported by VMware and plan your IP-address scheme accordingly to get the most out of this load-balancing Policy. One last thing, because I knew little about NFS, I turned to my primary source of storage related VMware articles Chad Sakac (@sakacc) and he written an excellent article about the use of NFS and VMware together with Vaughn Stewart (@vstewed) of NetApp. Please read it if you haven’t already. A “Multivendor Post” to help our mutual NFS customers using VMware. Again it’s truely excellent! ================================================================================ Title: Upgrading to SRM 4 and SSL certificates URL: https://frankdenneman.ai/2009-11-08-impact-on-ssl-certificates-when-upgrading-srm-1-to-srm-4/ Date: 2009-11-08 Recently I started to work on a project implementing SRM 4. One of the project requirements is to use SSL certificates issued by a trusted CA. When upgrading to SRM 4, we ran into a small problem. Because of a change in the vCenter authentication protocol, a new certificate that complies with the new certificate content rules must be obtained. The requirements changed of the “Subject Alternative Name”, the SSL certificate issued for SRM 1 environments use the FQDN of the vCenter server host. In SRM 4 environments, the Subject Alternative Name field must contain the FQDN of the SRM server. This value will be different for each member of the SRM server pair. We installed the SRM server on a separate server, but If you have installed SRM on the vCenter server, then you do not need to acquire a new certificate. ================================================================================ Title: Lefthand SAN – Lessons learned URL: https://frankdenneman.ai/2009-10-11-lefthand-san-lessons-learned/ Date: 2009-10-11 Disclaimer: This article contains references to the words master and slave. I recognize these as exclusionary words. The words are used in this article for consistency because it’s currently the words that appear in the software, in the UI, and in the log files. When the software is updated to remove the words, this article will be updated to be in alignment. Please note that this article has been written in 2009. I do not know if Lefthand changed their solution. Please check with your HP representative for updates! I recently had the opportunity to deliver a virtual infrastructure that uses HP Lefthand SAN solution. Setting up a Lefthand SAN is not that difficult, but there are some factors to take into consideration when planning and designing a Lefthand SAN properly. These are my lessons learned. Lefthand, not the traditional Head-Shelf configuration HP lefthand SANs are based on iSCSI and are formed by Storage nodes. In traditional storage architectures, a controller manages arrays of disk drives. A Lefthand SAN is composed of storage modules. A Network Storage Module 2120 G2 (NSM node) is basically an HP DL185 server with 12 SAS or SATA drivers running SAN/iQ software. This architecture enables the aggregation of multiple storage nodes to create a storage cluster and this solves one of the toughest design questions when sizing a SAN. Instead of estimating growth and buying a storage array to “grow into”, you can add storage nodes to the cluster when needed. This concludes the sales pitch. But this technique of aggregating separate NSM nodes into a cluster raises some questions. Questions such as; Where will the blocks of a single LUN be stored; all on one node, or across nodes? How are LUNs managed? How is datapath load managed? What is the impact of failure of a NSM node ? Block placement and Replication level The placement of blocks of a LUN depends on the configured replication level. Replication level is a feature called Network RAID Level. Network RAID stripes and mirrors multiple copies of data across a cluster of storage nodes. Up to four levels of synchronous replication at LUN level can be configured; None 2-way 3-way 4-way Blocks will be stored on storage nodes according to the replication level. If a LUN is created with the default replication level of 2-way, two authoritative blocks are written at the same time to two different nodes. If a 3-way replication level is configured, blocks are stored on 3 nodes. 4-way = 4 nodes. (Replication cannot exceed the number of nodes in the cluster) SAN IQ will always start to write the next block to the second node containing the previous block. A picture is worth a thousand words. Node order The data in which blocks are written to the LUN is determined not by node hostname but by the order in which the nodes are added to the cluster. The order of the placement of the nodes is extremely important if the SAN will span two locations. More information on this design issue later. Virtual IP and VIP Load Balancing When setting up a Lefthand Cluster, a Virtual IP (VIP) needs to be configured. A VIP is required for iSCSI load balancing and fault tolerance. One NSM node will act as the VIP for the cluster, if this node fails, the VIP function will automatically failover to another node in the cluster. The VIP will function as the iSCSI portal, ESX servers use the VIP for discovery and to log in to the volumes. ESX servers can connect to volumes two ways. Using the VIP and using the VIP with the option load balancing (VIPLB) enabled. When enabling VIPLB on LUNs, the SAN/iQ software will balance connections to different nodes of the cluster. Configure the ESX iscsi inititiator with the VIP as a destination address. The VIP will supply the ESX servers with a target address for each LUN. VIPLB will transfer initial communication to the gateway connection of the LUN. Running the vmkiscsi-util command shows the VIP as a portal and another ip address as target address of the LUN root@esxhost00 vmfs]# vmkiscsi-util -i -t 58 -l *************************************************************************** Cisco iSCSI Driver Version … 3.6.3 (27-Jun-2005 ) *************************************************************************** TARGET NAME : iqn.2003-10.com.lefthandnetworks:lefthandcluster:1542:volume3 TARGET ALIAS : HOST NO : 0 BUS NO : 0 TARGET ID : 58 TARGET ADDRESS : 148.11.18.60:3260 SESSION STATUS : ESTABLISHED AT Fri Sep 11 14:51:13 2009 NUMBER OF PORTALS : 1 PORTAL ADDRESS 1 : 148.11.18.9:3260,1 SESSION ID : ISID 00023d000001 TSIH 3b1 Gateway Connection This target address is what Lefthand calls a gateway connection. The gateway connection is described in the Lefthand SAN User Manual (page 561) as follows; Use iSCSI load balancing to improve iSCSI performance and scalability by distributing iSCSI sessions for different volumes evenly across storage nodes in a cluster. ISCSI load balancing uses iSCSI Login-Redirect. Only initiators that support Login-Redirect should be used. When using VIP and load balancing, one iSCSI session acts as the gateway session. All I/O goes through this iSCSI session. You can determine which iSCSI session is the gateway by selecting the cluster, then clicking the iSCSI Sessions tab. The Gateway Connection column displays the IP address of the storage node hosting the load balancing iSCSI session. SAN/IQ will designate a node to act as a gateway connection for the LUN, the VIP will send the IP address of this node as a target address to all the ESX hosts. This means every host that uses the LUN will connect to that specific node and this storage node will handle all IO for this LUN. This leads to the question, how will the GC handle IO for blocks not locally stored on that node? When a block is requested that is stored on another node, the GC will fetch this block. All nodes are aware of which block is stored on which node. The GC node will fetch this block of one of the nodes it’s stored and will send the results back to the ESX host. Gateway Connection failover Most Clusters will host more LUNs than it has available nodes. This means that each node will host the gateway connection role of multiple LUNs. If a node fails, the GC role will be transferred to the other nodes in the cluster. But when a NSM node returns back online, the VIP will not failback the GC roles. This will create an unbalance it the cluster, which needs to be solved as quickly as possible. This can be done by issuing the RebalanceVIP for the volume from the cli. Ken Cline asked me the question: How do I know when I need to use this command? Is there a status indicator to tell me? Well actually there isn’t and that is exactly the problem! After a node failure, you need to be aware of this behavior and you will have to rebalance a volume yourself by running the RebalanceVIP command. The Lefthand CMC does not offer this option or some sort of alert. Network Interface Bonds How about the available bandwidth? Lefthand nodes come standard with two 1GB NICs. The two NICs can be placed in a bond. An NSM node has three NIC bond configurations; Active - Passive Link Aggregation (802.3 ad) Adaptive Load Balancing The most interesting is the Adaptive Load Balancing (ALB). Adaptive Load Balancing combines the benefits of the increased bandwidth of 802.3ad with the network redundancy of Active-Passive. Both NICS are made active and they can be connected to different switches, no additional configuration on physical switch level is needed. When an ALB bond is configured, it creates an interface. This interface balances traffic through both NICs. But how will this work with the iSCSI protocol? In RFC 3270 (http://www.ietf.org/rfc/rfc3720.txt) iSCSI uses command connection allegiance; For any iSCSI request issued over a TCP connection, the corresponding response and/or other related PDU(s) MUST be sent over the same connection. We call this “connection allegiance”. This means that the NSM node must use the same MAC address to send the IO back. How will this affect the bandwidth? As stated in the ISCSI SAN configuration guide; “ESX Server_‐based iSCSI initiators establish only one connection to each target.”._ It looks like ESX will communicate with the gateway connection of the LUN with only NIC. I asked Calvin Zito (http://twitter.com/HPstorageGuy) to educate me on ALB and how it handles connection allegiance. When you create a bond on an NSM, the bond becomes the ‘interface’ and the MAC address of one of the NICs becomes the MAC address for the bond. The individual NICs become ‘slaves’ at that point. I/O will be sent to and from the ‘interface’ which is the bond and the bonding logic figures out how to manage the 2 slaves behind the scenes. So with ALB, for transmitting packets, it will use both NICs or slaves, but they will be associated with the MAC of the bond interface, not the slave device. The bond uses the same IP and MAC address of the first onboard NIC. This means the node will use both interfaces to transmit data, but only one to receive. Chad Sakac (EMC), Andy Banta (VMware) and various other folks has written a multivendor post explaining how ESX and vSphere handles iSCSI traffic. A must-read! http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html#more Design issues; When designing a Lefthand SAN, these points are worth considering; Network RAID level Write performance When 2-way replication is selected, blocks will be written on two nodes simultaneously, if a LUN is configured with 3-way replication, then blocks must be replicated to three nodes . Acknowledgements are given when blocks are written in cache on all the participating nodes. When selecting the replication level, keep in mind that higher protection levels leads to less write performance. Raid Levels NSM Network RAID offers protection for storage node failure, but it does not protect against disk failure within a storage node. Disk RAID levels need to be configured at Storage Node level, unlike most traditional arrays where raid level can be configured per LUN level. It is possible to mix storage nodes with different configurations of RAID within a cluster, but the this can lead a lower useable capacity. For example, the cluster exists of 12 TB nodes running RAID 10. Each node will provide 6TB in usable storage. When adding two 12TB nodes running RAID 5, each provides 10 TB of usable storage. However, due to the restrictions of how the cluster uses capacity, the NSM nodes running RAID 5 will still be limited to 6 TB per storage node. This restriction is because the cluster operates at the smallest usable per-storage node capacity. The RAID level of the storage node must first be set before it can join a Cluster. Check RAID level of clusternodes before configuring the new node, because you cannot change the RAID configuration without deleting data. Combining Replication Levels with RAID levels RAID levels will ensure data redundancy inside the storage node, while Replication levels will ensure data redundancy on the storage node level. Both higher RAID levels and Replication levels offer greater data redundancy, but will have an impact on capacity and performance. RAID5 with 2-way replication seems to be the sweet spot for most implementations, but when high available data protection is needed, Lefthand recommends 3-way replication with raid 5, ensuring triple mirroring with 3 parity blocks available. I would not suggest RAID 0 with replication, because rebuilding a RAID set will always be quicker than copying an entire storage node over the network. Node placement Mentioned previously, the data in which blocks are written to the LUN is determined by the order in which the nodes are added to the cluster. When using 2-way replication, blocks are written to two consecutive nodes. When designing a cluster the order of the placement of the nodes is extremely important if the SAN will be placed in two separate racks or even better span two locations. Because the 2-way replication writes blocks on two consecutive nodes, adding the storage nodes to the cluster in alternating order will ensure that data is written to each rack or site. When nodes are added in the incorrect order or if a node is replaced, the general setting tab of the cluster properties allows you to “promote” or “demote” a storage node in the logical order. This list is the leading for the “write” order of the nodes. Management Group and Managers In addition to setting up data replication, it is important to setup managers. Managers play an important role in controlling data flow and access of clusters. Managers run inside a management group. Several storage nodes must be designated to run the manager’s service. Because managers use a voting algorithm, a majority of managers needs to be active to function. This majority is called a Quorum. If a quorum is lost, access to data is lost. Be aware that access to data is lost, not the data itself. An odd number of managers is recommended, as a (low) even a number of managers can get in a certain state where no majority is determined. The maximum number of managers is 5. Failover manager A failover manager is a special edition of a manager. Instead of running on a storage node, the failover manager runs as a virtual appliance. A failover manager only function is maintaining quorum. When designing a SAN spanning two sites, running a failover manager is recommended. The optimum placement of the failover manager is the third site. Place an even amount of managers in both sites and run the failover manager at an independent site. If a third site is not available, run the failover manager local on a server, creating a logical separated site Volumenames And the last design issue, volume names cannot be changed. The volume name is the only setting that can’t be edited after creation. Plan your naming convention carefully, otherwise, you will end up recreating volumes and restoring data. If someone of HP is reading this, please change this behavior! Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: Timekeeping best practices for Linux URL: https://frankdenneman.ai/2009-09-18-timekeeping-best-practices-for-linux/ Date: 2009-09-18 VMware KB article 1006427 presents best practices for Linux timekeeping. These recommendations include specifics on the particular kernel command line options to use for the Linux operating system of interest. There is also a description of the recommended settings and usage for NTP time sync, configuration of VMware Tools time synchronization, and Virtual Hardware Clock configuration, to achieve best timekeeping results. What surprised me is the recommendation done by VMware; “Note: In all cases use NTP instead of VMware Tools periodic time synchronization” http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1006427 ================================================================================ Title: Deploying Volumes with Lefthand (CLIQ) URL: https://frankdenneman.ai/2009-08-25-deploying-volumes-with-command-line-interface-lefthand-cliq/ Date: 2009-08-25 Due to my extreme busy schedule I haven’t blogged for a while. Besides studying for VCDX and preparing for VMworld I’m also involved in a couple of projects. One project is designing and implementing a vSphere 4 virtual infrastructure. The VI will host an Exchange 2010 environment. Due to the size of my client’s environment, 192 TB is used for hosting mailboxes. These datastores will be available thru RDM, which means creating 192 1-TB volumes and assigning them to every ESX host in the cluster. I’ve tried to use the Centralized Management Console, but it’s tedious and error prone work. Mind numbing repetitive exercises makes me ask really dumb questions on twitter such as where did SCSI id 7 go? D’oh!. So to protect myself from further bashing and being ridiculed I started to search for the Lefthand CLI to be able to automate the creation and assignment of volumes on a Lefthand SAN. CLIQ A CLI is available for the HP EVA series, but not much info is being published about the Lefthand Command Line Interface (CLIQ). But deep down in the bowels of the HP site a document about the CLIQ is published; The “User Manuals API CLI User Manual” http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01806770/c01806770.pdf?jumpid=reg_R1002_USEN (But help in the SSH session will offer almost the same amount of info.) Log in info Storage nodes configured with SAN/IQ 8.0 are standard equipped with the CLIQ and can be accessed by SSH. Log in to the CLIQ via SSH using the node IP address, but use port 16022 instead of the default port 22. When nodes are a part of a management group any node can be used to access the CLIQ. Open a SSH session to a random node and use the management group user and password information. Use of CLIQ The CLIQ is not case sensitive and it the ordering of parameters is not specified. Any order will do. Let create a volume createVolume volumeName=VMFS001 clusterName=ESX-CLUSTER01 size=1TB Replication=2 thinProvision=1 description=“VI Datastore VMFS001” Size: The following sizes can be used: MB, GB, and TB Replication: The replication level for the volume 1=none,2-way,3-way or 4-way ThinProvision: • 0 – Full-provisioning • 1 – Thin-provisioning Assigning the volume assignvolume volumeName=VMFS001 initiator=iqn.1998-01.com.vmware:esx001.acme.com;iqn.1998-01.com.vmware:esx002.acme.com An IQN is used to assign the server to the volume, if a server item is preconfigured in the CMC, the CMC will list the server item at the Assigned Server tab of the volume. Assigning multiple servers to a volume must be done with one command, if two separate commands are being used, the last command will overwrite the first command. Use a ; to delimit IQNs. Those two commands saved me a lot of trouble and lots of unnecessary tedious work, hopefully you can benefit from these as well. ================================================================================ Title: Deploying Volumes with Lefthand CLIQ URL: https://frankdenneman.ai/2009-08-25-deploying-volumes-with-command-line-interface-lefthand-cliq-2/ Date: 2009-08-25 Due to my extreme busy schedule I haven’t blogged for a while. Besides studying for VCDX and preparing for VMworld I’m also involved in a couple of projects. One project is designing and implementing a vSphere 4 virtual infrastructure. The VI will host an Exchange 2010 environment. Due to the size of my client’s environment, 192 TB is used for hosting mailboxes. These datastores will be available thru RDM, which means creating 192 1-TB volumes and assigning them to every ESX host in the cluster. I’ve tried to use the Centralized Management Console, but it’s tedious and error prone work. Mind numbing repetitive exercises makes me ask really dumb questions on twitter such as where did SCSI id 7 go? D’oh!. So to protect myself from further bashing and being ridiculed I started to search for the Lefthand CLI to be able to automate the creation and assignment of volumes on a Lefthand SAN. CLIQ A CLI is available for the HP EVA series, but not much info is being published about the Lefthand Command Line Interface (CLIQ). But deep down in the bowels of the HP site a document about the CLIQ is published; The “User Manuals API CLI User Manual” http://h20000.www2.hp.com/bc/docs/support/SupportManual/c01806770/c01806770.pdf?jumpid=reg_R1002_USEN (But help in the SSH session will offer almost the same amount of info.) Log in info Storage nodes configured with SAN/IQ 8.0 are standard equipped with the CLIQ and can be accessed by SSH. Log in to the CLIQ via SSH using the node IP address, but use port 16022 instead of the default port 22. When nodes are a part of a management group any node can be used to access the CLIQ. Open a SSH session to a random node and use the management group user and password information. Use of CLIQ The CLIQ is not case sensitive and it the ordering of parameters is not specified. Any order will do. Let create a volume createVolume volumeName=VMFS001 clusterName=ESX-CLUSTER01 size=1TB Replication=2 thinProvision=1 description=“VI Datastore VMFS001” Size: The following sizes can be used: MB, GB, and TB Replication: The replication level for the volume 1=none,2-way,3-way or 4-way ThinProvision: • 0 – Full-provisioning • 1 – Thin-provisioning Assigning the volume assignvolume volumeName=VMFS001 initiator=iqn.1998-01.com.vmware:esx001.acme.com;iqn.1998-01.com.vmware:esx002.acme.com An IQN is used to assign the server to the volume, if a server item is preconfigured in the CMC, the CMC will list the server item at the Assigned Server tab of the volume. Assigning multiple servers to a volume must be done with one command, if two separate commands are being used, the last command will overwrite the first command. Use a ; to delimit IQNs. Those two commands saved me a lot of trouble and lots of unnecessary tedious work, hopefully you can benefit from these as well. ================================================================================ Title: VMworld 2009 sessions URL: https://frankdenneman.ai/2009-07-25-vmworld-2009-sessions/ Date: 2009-07-25 After skipping both VMworld events in 2008 I’m attending the VMworld event in San Francisco. This is the first event after VMware released vSphere and I hope to see much in-depth information about the OS and it’s new features. Duncan Epping and Eric Sloof posted info about interesting sessions, so I started browsing the session catalog as well. The following sessions seems to be very interesting; Session ID: BC1500 Title: vCenter SRM “Up and Running” - Best Practices & Avoiding the Pitfalls Speaker: Lee Dilworth - VMware Session ID: BC2541 Title: Re-architecting Backup and Recovery for Virtual Environments: Best Practices Speaker: Chris Wolf – Burton Group Session ID: BC2761 Title: ESX Networking for High Availability and Disaster Recovery Speaker: Seva Semouchin - VMware Session ID: BC2961 Title: VMware Fault Tolerance Architecture and Performance Speaker: Krishna Raj Raja - VMware Session ID: BC3197 Title: High Availability - Internals and Best Practices Speaker: Marc Sevigny - VMware Session ID: BC3301 Title: DR Architecture Design Workshop with SRM Speakers: Andrew Hald & John Arrasjid - VMware Session ID: BC3370 Title: VMware Fault Tolerance - Overview and Best Practices Speaker: Lan Huang – VMware Session ID: TA1394 Title: vSphere 4.0 Advanced Storage Log Analysis Speaker: Mostafa Khalil – VMware Session ID: TA2259 Title: Ask the Experts - Virtualization Design Speakers: Duncan Epping, Tom Howarth, Scott Lowe, Rick Scherer, Chad Sakac (Panel Session) Session ID: TA2384 Title: Deploying Cisco Nexus 1000V in a VMware vSphere Environment Speaker: Han Yang – Cisco Session ID: TA2467 Title: Best Practices to Increase Availability and Throughput for the Future of VMware Speaker: Chad Sakac – EMC Session ID: TA2509 Title: Storage Best Practices for Scaling Virtualization Deployments Speakers: Mostafa Khalil, Lucas Nguyen, Bob Slovick – VMware (Panel Session) Session ID: TA2525 Title: VMware vSphere 4 Networking Deep Dive Speaker: Srinivas Neginhal – VMware Session ID: TA2627 Title: Understanding “Host” and “Guest” Memory Usage and Other Memory Management Concepts Speakers: Fei Guo & Kit Colbert – VMware Session ID: TA2731 Title: Tips for Planning and Upgrading to vSphere 4 Speaker: David Coligado - VMware Session ID: TA2942 Title: Performance Best Practices Speakers: Bhavjit Walha & Kaushik Banerjee - VMware Session ID: TA2945 Title: What vStorage means to a vSphere administrator Speaker: Adam Carter - HP Session ID: TA2963 Title: Esxtop for advance users Speakers: Krishna Raj Raja & Haiping Yang – VMware Session ID: TA3406 Title: What is new for storage in vSphere 4.0 Speaker: Paul Manning - VMware Session ID: TA3326 Title: Building an Internal Cloud-the Journey and the Details Speakers: Mike DiPetrillo, Andrew Hald & John Arrasjid – VMware Session ID: TA3603 Title: Getting The Most Out Of VMotion: EVC, Performance Tuning, and Troubleshooting Speaker: Kit Colbert & Joel Baxter – VMware Session ID: TA4341 Title: Virtual Network Performance Speaker: Boon Seong Ang – VMware Tip: Try to browse the intermediate session list as well, I usually tend to browse the advanced session list only, but there are some very interesting sessions at the intermediate level. See you at VMworld! ================================================================================ Title: Flex10 update URL: https://frankdenneman.ai/2009-07-08-flex10-update/ Date: 2009-07-08 In my first post I had a question about the path data travels when sent to a “standby” virtual connect module. To quote my own question : “What will happen if the VMkernel decides to use that nic to send IO? Is the Flexnic aware of the standby status of it “native” uplink? Will it send data to the uplink of the VC module it’s connected to or will it send data to the active uplink? How is this done? Will it send the IO through the midplane or CX-4 cable to the VC module with the active uplink? And if this occurs what will be the added latency of this behavior? HP describes the standby status as blocked, what does this mean? Will virtual connect discard IO send to the standby IO, will it not accept IO and how will it indicate this?” The virtualconnect module is not standby, it’s external( X1 thru x6) ports are standby. The blade can and will send I/O to the virtualconnect module. It is the only way, because the blade NIC is hardwired to that VC module. When the virtual connect module receives data, it will transfer the data to the Virtual Connect module with the active ports over its internal X0 port. The X0 port is also mentioned in the HP documentation as cross connect. HP beefed up the cross connects in theThe Flex10 module, it has 2 x 10 cross connects instead of the 1 x 10 cross connect found in the 1/10 virtual connect module BL460c G1 NC373i 1GB Nics bandwidth upgrade The new BL460cG6 blade has a Flex10 LOM adapter, when using G6 blades along G1 blades in a C7000 enclosure It might be nice to swap the 1/10 virtual connect module for Flex10 virtual connect modules. When a NC373i is connected to a flex10 VC module the link speed of the 1GB module will be upgraded to 2,5 GB. It does not automatically upgrade the bandwidth when installing the Flex10 module, a firmware upgrade of the NC373i nic is needed. Download the network firmware update tool from the HP support site (date 7 oct 2008, version 2.1.3.1). The updated version of the boot code, 4.4.1 that enables the 2.5 Gigabit support. Windows 2003 will not show the proper uplink speed. I haven’t checked it on a windows 2008 server. But to be certain Check the Flex10 virtual connect module to see which speed the nic is linked to. ================================================================================ Title: Windows 2008 disk alignment URL: https://frankdenneman.ai/2009-05-20-windows-2008-disk-alignment/ Date: 2009-05-20 Due to many performance studies about disk performance it is well known that disk alignment for both VMFS partitions and NTFS file systems improve IO performance such as reduced latency and increased throughput. Alignment of VMFS partitions are done when configuring storage via the VI client but aligning NTFS partitions in Windows system prior to Windows 2008 is a manual task. Windows Server 2008 use a partition starting offset of 1,048,576 bytes (1,024 KB) for disk larger than 4GB. This provides a well enough alignment for most disks. According to the official documentation, windows 2008 uses a different partition starting offset for disks smaller than 4GB. Or as MS states in the document Performance Tuning Guidelines for Windows Server 2008 “Note that Windows Server 2008 defaults to a smaller power-of-two offset for small drives.” But which starting offset does W2K8 exactly use for smaller disks? Registry The default setting is configurable and therefore it can be found in the registry. The key “HKLM\SYSTEM\CurrentControlSet\Services\VDS\Alignment.” holds several values: This means that Windows 2008 uses a 1,024KB boundary for every disk from 4GB to 2TB, a 64KB partition starting offset is used for disks smaller than 4 GB. Let’s check if this is true. Partition starting offset check You can examine the used partition starting offset of a basic disk by issuing the following command in the command prompt: wmic partition get Index, Name, StartingOffset I’ve created a virtual machine with five disks (vmdk’s): A primary partition of 512MB will be placed on disk 4. When opening Disk Management (diskmgmt.msc) the disk is shown as followed: When checking the disks in windows with the wmnic command it shows the following list: (w_mic partition get Index, Name, StartingOffset_) Wmic does not show disk 4. A partition of 512MB is placed on Disk 4: The wmic command shows the following: As expected the third and forth disk have a different Starting Offset than disks larger than 4 GB. The last disk, disk 5, or what wmnic refers to as disk #4 has an offset of 1024Kb despite the size (512mb) of the partition (less than 4GB) This is because Windows 2008 sets the starting partition offset according to the size of the “physical disk” in the Virtual world, i.e. the size of the VMDK and not the size of the first partition. So when creating a 8GB vmdk and installing a 512MB partition, Windows 2008 selects the 1024 partition starting offset. I have used the wmic command to check the starting offset because basic disks where used, if dynamic disks are used you can check the partition starting offset with the tool dmddiag.exe –v (windows 2003) or diskdiag.exe –v in windows 2008. Both tools are in the support tools directory. Do not use diskpart to check the disk alignment, diskpart rounds up values, windows uses a partition starting offset of 32,256 bytes, which equals to 31.5 KB. Diskpart shows the offset as 32 KB (32768 bytes) System disks Disk #0 is the system disk and Windows 2008 automatically aligns every disk out of the box, so even the system disk is now aligned. Prior to windows 2008, it was not recommend and according to Microsoft not even possible to align the system disk. Windows 2008 aligns the system disk as well. Disks created by older versions of windows and presented to a windows 2008 will be left untouched and the disks will maintain the settings which they were created with. Virtual Machine template I come across many VM’s where the guest OS disks are not properly aligned. This can have many reasons. Some administrators aren’t aware of the mis-alignment, or sometimes they forget aligning the newly added disk. Windows 2008 alleviates the problem by using a 1024KB boundary instead of the 31,5KB used by its predecessors. When dealing with deployment of VM’s by using templates this feature can be a blessing from heaven. Prior to windows 2008, NTFS partitions needed to be aligned manually by using the diskpart utility. All the diskpart commands are simple to perform, but partition alignment must be done at partition creation time, prior to partitions being formatted. Adjusting the partition alignment afterwards will destroy any data written on the disk! This additional work is really a pain in the butt, when creating a new virtual machine. The VM administrator must remember to align the partition and use the proper NTFS cluster size. And you all know, when deploying VM’s sometimes settings done by hand will not applied every time. To solve this problem some companies use a template VM with several dummy hard disks. Each the size of 1 GB, properly aligned and formatted with the right cluster size. When deploying the VM, the size of the disk (vmdk) will be increased and the extend command of the diskpart.exe file is used in windows to enlarge the NTFS partition to utilize the newly added disk space. This “workaround” is still viable, but beware, when using disk with less then 4 GB of space, windows 2008 will use a different starting partition offset. (64KB) The disk is still properly aligned, but you can end up different configured systems in your virtual environment. Because of the default alignment of Windows 2008, the template doesn’t have to be equipped with dummy disks. The only thing left to worry about is a proper cluster size of the NTFS partition. Cluster size (file allocation unit) When formatting the partition with a NTFS file system, a proper cluster size must be selected. Unfortunately there isn’t a one-size-fits-all cluster size. Some applications thrive when using a 64KB cluster size (SQL) some applications will perform at their best when formatting the partition with a 32KB cluster size. If you decide on the cluster size to use, you can automate the creation of the NTFS partitions by using diskpart in scripted mode. Diskpart Script For this example, a diskpart script is used to create a primary partition on the last disk of the VM. Insert the commands in a simple text document (script.txt) and issue the command: diskpart /s script.txt Here’s what the script.txt looks like: select disk 4 create partition primary assign letter=G format fs=ntfs unit=64K label=“scripted” nowait The nowait command forces the command to return immediately while the format process is still in progress, because of this, multiple formats can be issued at once. Windows 2008 saves the VM administrator a lot of trouble by auto aligning partitions, due to this fact some admins might have to take a look at there current deploy method and their templates. ================================================================================ Title: Flex-10 lessons learned URL: https://frankdenneman.ai/2009-04-26-flex-10-lessons-learned/ Date: 2009-04-26 One of my clients bought a couple of HP blade c7000 enclosures recently. Including the new dual port Flex-10 mezzanine cards (nc532m) and Flex-10 Virtual Connect modules. Due to the fact that this technology is quite new, not much inside-info is found on the web. I’ve had lots of discussions with Ken Cline and Scott Lowe, which will publish an Flex-10 article by it’s own pretty soon. This write-up is a quick overview of lessons learned by me but even more a call for answers. My client purchased the Flex-10 technology to use for iSCSI network traffic. Two uplinks are connected to the storage network. I’m aware about the iSCSI limitations in ESX, but this write-up will not contain info about the software iSCSI initiator. Some excellent articles are written about the behavior of the software iSCSI initiator and the ESX network stack, which I encourage you to read: • The legendary ISCSI multi-vendor post article of some major players; http://virtualgeek.typepad.com/virtual_geek/2009/01/a-multivendor-post-to-help-our-mutual-iscsi-customers-using-vmware.html#more • Ken Cline and his great vSwitch debate series; http://kensvirtualreality.wordpress.com/2009-04-05-the-great-vswitch-debate%e2%80%93part-3/ • And Scott Lowe’s; Understanding NIC Utilization in VMware ESX http://blog.scottlowe.org/2008-07-16-understanding-nic-utilization-in-vmware-esx/ When configuring HP blades, there are two areas to pay attention to. The uplink configuration and the server profile configuration. Uplinks can be used multiple blades in the enclosure, configuration of Blade Nics to use the uplinks are done in the server profile . Uplink Due to my clients’ redundancy requirements, two separate external switches are being used for iSCSI traffic. A simple point to point topology is being used and the Flex-10 modules are connected via one uplink port to a separate switch. The two uplink ports are placed into one “Shared Uplink Set” and assigned to a virtual Ethernet in virtual connect manager. In theory two connections of 10GB each will become available to use for iscsi traffic, but due to use of two separate switches and the loop prevent mechanism of virtual connect, one uplink port of the shared uplink set will be utilized as the active port. The other port is placed in Standby (blocked) mode. Loop prevent mechanism The connection mode of a shared uplink set determines the assignment of the uplinks. When multiple uplinks are assigned to a Shared Uplink Set, the default connection mode (auto) attempts to negotiate a port channel using LACP. If the LACP negotiation fails, auto connection mode places all uplink ports, except one, into standby (blocked) mode. The currently used physical switches at my clients’ environment do not support spanned Ethernet channel and therefore the LACP negotiation fails. As a result, one uplink port is assigned the active role, the uplink of the second virtual connect module is placed in the standby (blocked) mode. As a result of this, the useable bandwidth is reduced to 1 x 10 GB. Besides bandwidth reduction, the assignments of active and standby uplinks have impact on the configuration of the vswitch configuration. Flex-10 The behavior of a Flex-10 nic depends on which virtual connect module it’s connected to. When a Flex-10 nic is directly connected to a virtual connect Flex-10 module it enumerates as four Flexnics per port. When connected to other virtual connect modules, the Flex-10 nic will only enumerate 2 Flexnics, one Flexnic per port. The funny thing is that when a Flex-10 mezzanine card is “connected” to two empty interconnect bays, the Flex-10 nic will also enumerate eight Flexnics, because HP expects that the Flex-10 mezzanine card will eventually be connected to a Flex-10 virtual connect module. As a result of the internal port mapping, the dualport Flex-10 mezzanine card in my client blade servers are connected to the Flex-10 virtual connect modules. Therefore enumerating a total of eight Flexnics, but what exactly will be presented to the ESX host? Flexnic The ESX host will see a Flexnic as a device with an unique PCI Device ID that is connected to a 10 GB port. Due to this unique ID, it appears in ESX as separate nic with its own Broadcom 57711 driver instance. When mapping Ethernet networks to a Flex-10 nic inside virtual connect manager, eight Flexnics are being presented to the ESX server. Server profile Inside the server profile in virtual connect manager, nics can be mapped to Ethernet networks As a result of the enumeration of multiple Flexnics, the process of mapping Ethernet networks to Flexnics differs from dual port mezzanine- of onboard 1GB nics. The Flex10 port presents the Flexnics as four sub-devices, Flex10 port 1 presents the Flexnics as: 1a 1b 1c 1d Flex-10 port 2 presents the Flexnics as; 2a 2b 2c 2d When assigning an Ethernet network to a Flex-10 nic, it will alternate between the two Flex-10 ports. It will start by displaying the first Flexnic of the first Flex-10 port (1-a), then the first Flexnic of the second Flex-10 port (2-a) and so on. It is possible to map the same Ethernet network to two Flexnics of separate Flex-10 ports, for example, Ethernet network iscsi – 1a & 2 a, but it isn’t possible to map a single Ethernet network to multiple Flexnics on the same Flexnic port, Ethernet network iscsi – 1a & 1b. As a result of using one VLAN network for iSCSI traffic, only the first Flexnic of each Flex-10 port (1a & 2a) is mapped to the iSCSI Ethernet network. Because the Flex10 technique is used only for iSCSI traffic, the other Flexnic adapters are not mapped to a Ethernet network. Because of this, the unmapped Flexnic will appear as unconnected uplinks in ESX. Bandwidth throttle It is possible to throttle the bandwidth per Flexnic. Because one Flexnic per Flex10 port is being used, it will be configured with the maximum bandwidth. vSwitch The order in which the NICs are mapped to Ethernet networks in the server profile of the Virtual Connect manager determines the assignment of vmnic labels. Flexnic 1a is assigned the label vmnic4 and Flexnic 2a is assigned label vmnic5. Both uplinks have the UP status. Due to the internal port mapping of virtual connect, Flexnic 2a is mapped to the second Flex-10 VC module which uplink is assigned the standby (blocked) status of the Shared Uplink Set. This situation raised some questions, which I haven’t found any answers to (yet). What will happen if the VMkernel decides to use that nic to send IO? Is the Flexnic aware of the standby status of it “native” uplink? Will it send data to the uplink of the VC module it’s connected to or will it send data to the active uplink? How is this done? Will it send the IO through the midplane or CX-4 cable to the VC module with the active uplink? And if this occurs what will be the added latency of this behavior? HP describes the standby status as blocked, what does this mean? Will virtual connect discard IO send to the standby IO, will it not accept IO and how will it indicate this? The described situation can have impact on the vSwitch design. Just to be on the safe side of things, the 2a Flexnic is configured as standby adapter in the iSCSI vSwitch. Must read documents about Virtual Connect and Flex-10 technology: HP Virtual Connect for Cisco Network Administrators (c01386629.pdf) HP Virtual Connect for c-Class BladeSystem User Guide (c00865618.pdf) HP Virtual Connect Cookbook (c01471917.pdf) ================================================================================ Title: VMFS Volume Management document URL: https://frankdenneman.ai/2009-03-27-vmfs-volume-management-document/ Date: 2009-03-27 VMware published an excellent document about VMFS volume management a few days ago. VMware® VMFS Volume Management information guide The guide explains the VMFS volume header metadata mechanism and describes the new 3.5 setting SCSI.CompareLUNNumber. A must read if you use third-party storage snapshot and replication technology. ================================================================================ Title: My first lefthand ISCSI VI architecture URL: https://frankdenneman.ai/2009-03-23-my-first-lefthand-iscsi-vi-architecture/ Date: 2009-03-23 I’m currently reviewing a design of a new virtual infrastructure. The VI uses multiple 10GB links to connect to a very large HP Lefthand san. I’m more a Fibre Channel guy, but I believe that this solution will smoke most mid-range FC-sans. I cannot wait to deploy the VI on the SAN. But I need to get used to some differences between ISCSI and fibre channel configurations. The “problem” or my latest challenge is creating a LUN provisioning scheme where multiple clusters can connect to all the LUNs when a disaster occurs and a cluster has failed. Lefthand present the LUNs as targets instead using the LUN ID as a unique identifier. I’m used to design a LUN ID scheme per cluster, this way if a cluster fails, the “destination” cluster can connect to the LUNs of the failed cluster with the same LUN ID as the original cluster. But when a (lefthand) LUN is presented to the ESX server, it will use a unique target ID instead of a unique LUN ID. (vmhba1:2:0) I have done some testing and discovered that the assigned target ID can differ from ESX server to ESX server. I’m curious if the target ID is used when creating the UUID of the VMFS datastore. And I’m especially interested in what will happen if multiple ESX hosts are going to communicate with the LUN when all the ESX hosts will use a different “path” Maybe there isn’t a problem at all and different targets will work well, but is seems that I need to stop thinking in FC solutions and get used to iscsi Lefthand “quirks”. I’ve read the field guide for VMware infrastructures, I googled on terms like “iscsi lun scheme’s” but I cannot seem to find any real-life scenario’s. Maybe my Google skills are pitiful at the moment, and maybe someone can shed some lights on this and how they solved this “problem”. ================================================================================ Title: Increasing the queue depth? URL: https://frankdenneman.ai/2009-03-04-increasing-the-queue-depth/ Date: 2009-03-04 When it comes to IO performance in the virtual infrastructure one of the most recommended “tweaks” is changing the Queue Depth (QD). But most forget that the QD parameter is just a small part of the IO path. The IO path exists of layers of hardware and software components, each of these components can have a huge impact on the IO performance. The best results are achieved when the whole system is analysed and not just the ESX host alone. To be honest I believe that most environments will profit more from a balanced storage design than adjusting the default values. But if the workload is balanced between the storage controllers and IO queuing still occurs, adjusting some parameters might increase IO performance. Merely increasing the parameters can cause high latency up to the point of major slowdowns. Some factors need to be taking in consideration. LUN queue depth The LUN queue depth determines how many commands the HBA is willing to accept and process per LUN, if a single virtual machine is issuing IO, the QD setting applies but when multiple VM’s are simultaneously issuing IO’s to the LUN, the Disk.SchedNumReqOutstanding (DSNRO) value becomes the leading parameter. Increasing the QD value without changing the Disk.SchedNumReqOutstanding setting will only be beneficial when one VM is issuing commands. It is considered best practise to use the same value for the QD and DSNRO parameters! Read Duncan’s excellent article about the DSNRO setting. Qlogic Execution Throttle Qlogic has a firmware setting called „Execution Throttle" which specifies the maximum number of simultaneous commands the adapter will send. The default value is 16, increasing the value above 64 has little to no effect, because the maximum parallel execution of SCSI operations is 64. (Page 170 of ESX 3.5 VMware SAN System Design and Deployment Guide) If the QD is increased, execution throttle and the DSNRO must be set with similar values, but to calculate the proper QD the fan-in ratio of the storage port needs to be calculated. Target Port Queue Depth A queue exist on the storage array controller port as well, this is called the “Target Port Queue Depth”. Modern midrange storage arrays, like most EMC- and HP arrays can handle around 2048 outstanding IO’s. 2048 IO’s sounds a lot, but most of the time multiple servers communicate with the storage controller at the same time. Because a port can only service one request at a time, additional requests are placed in queue and when the storage controller port receives more than 2048 IO requests, the queue gets flooded. When the queue depth is reached, this status is called (QFULL), the storage controller issues an IO throttling command to the host to suspend further requests until space in the queue becomes available. The ESX host accepts the IO throttling command and decreases the LUN queue depth to the minimum value, which is 1! The VMkernel will check every 2 seconds to check if the QFULL condition is resolved. If it is resolved, the VMkernel will slowly increase the LUN queue depth to its normal value, usually this can take up to 60 seconds. Calculating the queue depth\Execution Throttle value To prevent flooding the target port queue depth, the result of the combination of number of host paths + execution throttle value + number of presented LUNs through the host port must be less than the target port queue depth. In short T => P * q * L T = Target Port Queue Depth P = Paths connected to the target port Q = Queue depth L = number of LUN presented to the host through this port Despite having four paths to the LUN, ESX can only utilize one (active) path for sending IO. As a result, when calculating the appropriate queue depth, you use only the active path for “Paths connected to the target port (P)” in the calculation, i.e. P=1. But in a virtual infrastructure environment, multiple ESX hosts communicate with the storage port, therefore the QD should be calculated by the following formula: T => ESX Host 1 (P * Q * L) + ESX Host 2 (P * Q * L) ….. + ESX Host n (P * Q * L) For example an 8 ESX host cluster connects to 15 LUNS (L) presented by an EVA8000 (4 target ports)* An ESX server issues IO through one active path (P), so P=1 and L=15. The execution throttle\queue depth can be set to 136,5=> T=2048 (1 * Q * 15) = 136,5 But using this setting one ESX host can fill the entire target port queue depth by itself, but the environment exists of 8 ESX hosts. 136,5/ 8 = 17,06 In this situation all the ESX Host communicate to all the LUNs through one port. Which does not happen in many situations if a proper load-balancing design is applied. Most arrays have two controllers and every controller has at least two ports. In the case of a controller failure, at least two ports are available to accept IO requests. It is possible to calculate the queue depth conservatively to ensure a minimum decrease of performance when losing a controller during a failure, but this will lead to underutilizing the storage array during normal operation, which will hopefully be 99,999% of the time. It is better to calculate a value which utlilize the array properly without flooding the target port queue. If you assume that multiple ports are available and that all LUNs are balanced across the available ports on the controllers, it will effectively quadruple the target port queue depth and therefore increase the values of the execution throttle in the example above to 68. Besides the fact that you cannot increase this value above 64, it is wise to decrease the value to a number below max value, it will create a buffer for safety What’s the Best Setting for Queue Depth? The examples mentioned are pure worst case scenario stuff, most of the time it is highly unlikely that all hosts perform at their maximum level at any one time. Changing the defaults can improve throughput, but most of the time it is just a shot in the dark. Although you are configuring your ESX hosts with the same values, not every load on the ESX server is the same. Every environment is different and so the optimal queue depths would differ. One needs to test and analyse its environment. Please do not increase the QD without analysing the environment; this can be more harmful than useful. Get notification of these blogs postings and more DRS and Storage DRS information by following me on Twitter: @frankdenneman ================================================================================ Title: HP CA and the use of LUN balancing scripts URL: https://frankdenneman.ai/2009-02-09-hp-continuous-access-and-the-use-of-lun-balancing-scripts/ Date: 2009-02-09 Some of my customers use HP Continuous Access to replicate VM data between storage arrays. Lately a couple of LUN balancing powershell- and Perl scripts were introduced in the VMware community. First of all, there is nothing wrong with those scripts. For example, Justin Emerson wrote an excellent script that balances the active paths to an active/active SAN. But using an auto balance scripts when Continuous Access is used in the Virtual Infrastructure can result in added IO latency and unnecessary storage processor load. Here’s why: AA type The HP EVA array range (4x,6x,8x) are categorized by VMware as active-active arrays. Active/active storage arrays are further divided into two categories: • Symmetrical Active-Active (SAA) • Asymmetric Active-Active (AAA) SAA arrays are considered by many as true active-active arrays. IO request can be issued over all paths and every controller in the array can accept and send IO to the LUN. EMC DMXs and HP XPs are SAA arrays. Asymmetric Active-Active and Asymmetric Logical Unit Access (ALUA) compliance The arrays from the EVA family are dual controller AAA arrays and are compliant with the SCSI Asymmetric Logical Unit Access (ALUA) standard for LUN access/failover and I/O processing. In an Asymmetric Active-Active Array both controllers are online and both can accept IO, but only one controller is assigned as the owning controller of the LUN. The owning controller can issue IO commands directly to the LUN, this is called an optimized path. The non-owning controller, or proxy controller can accept IO commands, but cannot communicate with the LUN. This is called an non-optimized path. If a read request reaches the array through the proxy controller, it will be forwarded to the owning controller of the LUN. This behavior is called a proxy read. It looks like I’m describing an active/passive array, but the main difference is that Active-Passive arrays transfer the ownership as soon as it receives IO on the non-owning controller. IO request are transferred between the controllers on the back-end of an AAA, making the process transparent to the ESX host. In a asymmetric Active-Active Array, storage processor ports have a certain port state with respect to a given LUN: • Active\Optimized • Active\Non-Optimized • Standby • Unavailable • Transitioning If the access characteristics of a port differs from another port asymmetric logical unit access occurs. ALUA provides a way to allow to report the states of the port to the host, the host can use the states of the port to prioritize paths. Unfortunately, ESX 3.x does not support ALUA, what that actually means is that ESX does not has the ability to identify, the LUN controller ownership, or to put it more precisely the the active\optimized or active non-optimized paths. ALUA support is implemented in the PSA (Pluggable Storage Architecture) of ESX4. Proxy reads Read IO requests received by the proxy controller (1) are sent to the owning controller (2), which retrieves the data from disk (3), caches the read data (4) and mirrors the data to the cache of the proxy controller (5). The proxy controller satisfies the host read request (6), making this process transparent to the ESX host. Proxy reads add unnecessary latency to the IO request. It also creates higher mirror port utilization. IO-Write commands to the proxy controller suffer less performance impact. Due to fault protection all writes are mirrored in both controllers’ caches, but the owning controller is still responsible for flushing the data to disk. Mirror Port The controller mirror ports are used for cache writes and proxy reads. If you setup your environment with correct multipathing, the mirror ports will only have to handle the write mirroring. Implicit LUN transition If the EVA array detects, in a period of 60 minutes, that at least 66% of the total read request to a LUN are proxy reads, ownership is transitioned to the non-owning proxy controller and making it the owning controller. Justin’s powershell script assigns the same path to every server the same way. This way the EVA should switch the managing controller within the hour. (If you have multiple ESX hosts run multiple VM’s on the LUN of course) Continuous Access DR Groups If HP Continuous Access (CA) is used to replicate LUNs between two arrays, extra care must be taken when planning to use a LUN balancing script. CA sets up replication relationships between LUNs on different arrays; this is called a Data Replication (DR) group. A DR group can be considered a consistency group, all LUNs in a DR group fail over together, share a log and preserve write order within the group. Because of this requirements, one controller is assigned as managing (owning) controller for all member LUNS. Implicit LUN transition Because one controller manages a group of LUNs, CA disables implicit LUN transitions for all DR group members. Mixing Implicit LUN transition (ILT) together with a large DR group can be a recipe for disaster. Imagine if ILT was switched on and the array detects too much proxy reads on a LUN in the group. The array will evoke an ILT for that LUN. Because all the LUN in the DR group must use the same controller to meet the consistency requirement all members are transitioned together. So far so good, but you can bet on it that a new proxy read situation appears, due to the multiple hosts communicating with the disks. This will evoke another Implicit LUN transition. And now we are back where we started. Enabling Implicit LUN transition can create some sort of bouncing group of LUNs between controllers. This is a sure way of giving you SAN administrator a small stroke. The downside of disabling ILT on a DR group is the possibility that LUNs may experience excessive proxy reads. Using auto balance script on CA managed LUNs Because ESX does not inquire about the status of the port when testing the path, the auto balance scripts cannot discover the optimized path. When paths are being initialized during the boot of the ESX host, it will just enumerate the paths available to it. Scanning the first controller and the lowest device number, this path might not necessarily be the path to the managing controller. Because all paths are active, the script will select the next path for a LUN. This can lead to IO requests arriving at the proxy controller and because implicit LUN transition is disabled, proxy reads will keep occurring. Custom load balancing If you have high workload intensity you might experience negative impact on IO performance when running a generic LUN balancing script. If you implement CA in your virtual infrastructure, it is better to take some time to design a custom load balance script. Using a well designed load balancing script along with fixed multipathing policy does not eliminate proxy reads, but it should only occur as a temporary condition during failures. In the example, each DR group contains 5 LUNS, because every LUN in the DR group share the same managing controller, the optimized path for LUN 1-5 is through Storage Processor A. Alternating HBA’s are used to load balance on the ESX side. Determining the managing controller To correctly load balance on EVA arrays you must know which controller owns the LUN. Command View EVA shows the managing controller of the LUN, but checking several LUNs via Command View EVA can be a lengthy process. Using Command view EVA (CVE) • On the presentation tab of the LUN (vdisk) properties • On the general tab of the DR Group properties HP Storage System Scripting Utility (SSSU) can help you to speed up discovering the management controller.Using SSSU: LS VDISK (Name will suffice, no need for complete path) The owning controller is listed as online controller (controller name) I’m not aware of any powershell tools to manage an EVA storage array. My contacts at HP cannot confirm if any new tooling except SSSU and CVE will appear soon