STORAGE DRS CLUSTER SHOWS ERROR "INDEX WAS OUT OF RANGE. MUST BE NON-NEGATIVE AND LESS THAN THE SIZE OF THE COLLECTION.
Recently I have noticed an increase of tweets mentioning the error “Index was out of range” Error message: When trying to edit the settings of a storage DRS cluster, or when clicking on SDRS scheduling settings an error pops up: “An internal error occurred in the vSphere Client. Details: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index” Impact: This error does not have any impact on Storage DRS operations and/or Storage DRS functionality and is considered a “cosmetic” issue. See KB article: KB 2009765. Solution: VMware vCenter Server 5.0 Update 2 fixes this problem. Apply Update 2 if you experience this error. As the fix is included in the Update 2 for vCenter 5.0, I expect that vCenter 5.1 Update 1 will include the fix as well, however I cannot confirm any deliverables.
STORAGE DRS CLUSTER SHOWS ERROR "INDEX WAS OUT OF RANGE. MUST BE NON-NEGATIVE AND LESS THAN THE SIZE OF THE COLLECTION.
Recently I have noticed an increase of tweets mentioning the error “Index was out of range” Error message: When trying to edit the settings of a storage DRS cluster, or when clicking on SDRS scheduling settings an error pops up: “An internal error occurred in the vSphere Client. Details: Index was out of range. Must be non-negative and less than the size of the collection. Parameter name: index” Impact: This error does not have any impact on Storage DRS operations and/or Storage DRS functionality and is considered a “cosmetic” issue. See KB article: KB 2009765. Solution: VMware vCenter Server 5.0 Update 2 fixes this problem. Apply Update 2 if you experience this error. As the fix is included in the Update 2 for vCenter 5.0, I expect that vCenter 5.1 Update 1 will include the fix as well, however I cannot confirm any deliverables.
DESIGNING YOUR VMOTION NETWORK – MULTI-NIC VMOTION AND NETIOC
Designing a vMotion network can be quite a challenge. You want to provide vMotion as much bandwidth as possible but not at the expense of other network traffic streams. Network I/O Control (NetIOC) can provide you bandwidth management tools to shape and form your vMotion network. NetIOC provides a QoS that allows vMotion to utilize as much bandwidth as possible until contention occurs. The moment a physical NIC is saturated NetIOC distributes network bandwidth according to the relative share value of the network resource pool. In the article “A primer of Network I/O Control” I explain the various resource management constructs of Network I/O Control. How does NetIOC work with a Multi-NIC vMotion network? Multi-NIC vMotion network on a distributed switch NetIOC is only supported on a distributed switch therefor you need to create multiple vMotion portgroups on your distributed switch. In order to have a supported vMotion network, both distributed port groups need to be configured with an alternate failover order configuration. In my lab I’ve named the two dvPortgroups vMotion-01 and vMotion-02. dvPortgroups vMotion-01 is configured with a failover order where dvUplink1 is active and dvUplink2 is standby. vMotion-02 is configured with dvUplink2 as active and dvUplink1 as standby. If you wonder why I’m configuring the redundant uplink as Unused please review the article: “Multi-NIC vMotion – failover order configuration”. The reason why “Route Based on originating virtual port” is chosen is described in the initial article of this series: “Designing your vMotion network”. vMotion network resource pool Each host in my lab is connected to the distributed switch has two dvUplink portgroups. A virtual adapter with vMotion enabled is connected to a uplink and vMotion is able to leverage both physical adapters to send out vMotion traffic. When enabling NetIOC, 7 predefined system network resource pools become active. One system network resource pool is vMotion, vMotion traffic binds to the vMotion system network resource pool and if contention occurs the network resource pool competes for bandwidth with the other active streams. It’s important to realize that the shares apply at the physical adapter layer. Therefore as vMotion is able to utilize both uplinks it “receives” 50 shares per physical adapter. As mentioned in the NetIOC primer article, shares are only active when contention exists and they only count when it is transmitting. Consequently, if vMotion is not active, the shares are not counted when an adapter is congested. Also when vMotion uses a single link only 50 shares are active. Although the dvUplink portgroup is configured with 2 dvUplinks, one dvUplink is configured as standby. When the active NIC is operating normally, the dvPortgroups cannot utilize the standby link. This results in the utilization of only 1 link and therefor only 50 shares become active during congestion. Example Scenario 1; vmnic0 saturated The distributed switch is configured with 5 dvPortgroups, management, vMotion01, vMotion02, NFS and a virtual machine portgroup. Each network resource pool is configured with the default physical adapter shares. In this scenario vMotion is load-balancing traffic across both dvPortgroups, however the VMkernel decides to also send management and virtual machine traffic through dvUplink1, saturating vmnic0. In this scenario, bandwidth is distributed following relative share values. As NFS traffic isn’t transmitting across dvUplink1, its shares are not active. Available bandwidth for vMotion is reduced to 2.5Gb as long as this situation persists. vMotion is also using dvUplink2, as a result traffic flows through vmnic1 as well. Fortunately vmnic1 is not saturated and vMotion can utilize as much bandwidth it can allocate. Restricted by the CPU speed of the host, vMotion is able to utilize 6Gb of bandwidth on vmnic1 while having an additional bandwidth allocation of 2.5Gb on vmnic0. In total vMotion utilizes 8.5Gb at that moment. Example Scenario 2; Both NICs saturated The moment both NICs are saturated, the available bandwidth available to vMotion is calculated on a NIC basis. vMotion traffic wanting to use vmnic0 is assigned bandwidth relative to their share value, similar to example scenario 1. vMotion traffic wanting to use vmnic1 is assigned bandwidth relative to their share value compared to the active traffic streams, in this case NFS and vMotion are sending traffic to the dvUplink. The distributed switch receives traffic from NFS and vMotion destined for vmnic1, as a result NetIOC will assign half of the bandwidth to vMotion and NFS as each owns 50 shares of the total active 100 shares. In this case the amount of bandwidth vMotion can utilize is 2.5Gb on vmnic0 and 5 Gb on vmnic1, allowing vMotion to utilize 7.5Gb of the total available 20Gb. Host Limits vSphere 5.1 introduced a big adjustment to hosts limits, the host limits now applies to each individual uplink. This means that when setting a host limit on the network resource pool for vMotion of 3000Mbps, vMotion is limited to transmit a maximum of 3Gb per uplink. In the case of a Multi-NIC vMotion configuration (2NICs) the maximum traffic vMotion can issue to the vmnics is 6Gb. Divide the host limit by the number of active NICs if you decide to set a limit on a particular network resource pool. If you want to limit vMotion to 3Gb with a 2 NIC Multi-NIC configuration, set the host limit on the network resource pool to 1500 Mbps. As for limits, limits are always active! Host limits are always enforced on the physical adapter regardless of the utilization rate of the adapter. This means setting a host limit on a network resource pool restricts the traffic to utilize more bandwidth even if this bandwidth is available. Saturating an adapter is not by definition wrong you are utilizing the infrastructure. Taking precaution and limiting certain network streams unconditionally might hurt you more than your assumed gain. Before limiting a network stream my recommendation is always to measure traffic patterns over a longer period of time. Ingress only NetIOC shares and limits are only applied to ingress traffic. In ESX , the ingress and egress traffic are with respect to a distributed switch. Ingress traffic is traffic that flows from the VMkernel or vNic from the running VM towards the distributed switch. Egress is traffic that flows from the distributed vSwitch to the physical nic or to the vNIC It is important to know NetIOC only controls ingress traffic initiated from within the ESX host. This means that you can set a limit the vMotion network resource pool, this only affects traffic send towards the uplink inside the host. If you have a cluster with a small number of hosts, it can happen that multiple vMotion operations are inbound to that host. In that scenario, NetIOC cannot prevent the Uplinks to get saturated by incoming vMotion traffic. To avoid this situation, traffic shaping needs to be configured on the dvPortgroups. The upcoming article Designing your vMotion network - vMotion dvPortgroups traffic shaping explores this feature in depth. Part 1 - Designing your vMotion network Part 2 - Multi-NIC vMotion failover order configuration Part 4 – Choose link aggregation over Multi-NIC vMotion? Part 5 – 3 reasons why I use a distributed switch for vMotion networks
A PRIMER ON NETWORK I/O CONTROL
Network I/O Control (NetIOC) provides controls to partition network capacity during contention. NetIOC provides additional control over the usage of network bandwidth in the form of network isolation and limits. vMotion operations introduce temporary network traffic that tries to consume as much bandwidth as possible. In a converged network vMotion operations may have a disruptive effect on other network traffic streams. Due to way NetIOC operates, NetIOC provides control for predictable networking performance while different network traffic streams are contending for the same bandwidth. This article serves as an introduction on Network I/O Control resource control before we dive into specifics on how to use NetIOC on multi-NIC vMotion networks on distributed switches. Please note that this article covers NetIOC in vSphere 5.1 Distributed Switch NetIOC is only available on the vNetwork Distributed Switch (vDS). To enable it, in the vSphere web client go to Networking, Manage, Settings, click on the third icon from the left (Edit distributed switch settings) and enable Network I/O Control. Once enabled go to the Resource Allocation tab there you will find an overview of the (predefined) Network Resource Pools, Host Limits and Physical Adapter Shares and the Shares value. Network Resource Pool The NetIOC network resource pool (NRP) construct is quire similar in many ways to the compute resource pools already existing for CPU and Memory. Each resource pool is assigned shares to define the relative priority of its workload against other workloads active on the same resource. In the case of NetIOC, network resource pools are used to differentiate between network traffic classes. NetIOC predefines 7 different system network resource pools: 1. Management Traffic 2. vMotion Traffic 3. Fault Tolerance (FT) Traffic 4. iSCSI Traffic 5. vSphere Storage Area Network Traffic * 6. NFS Traffic 7. vSphere Replication (VR) Traffic 8. Virtual Machine Traffic * The vSphere client marks this as a User defined network resource pool, while the web client marks this (correctly) as system network resource pool. NetIOC classifies incoming traffic and binds it automatically to the correct system network resource pool; therefor you do not have to assign the vMotion traffic network resource pool to the distributed port groups manually. Once a vMotion operation starts NetIOC “tags” it as vMotion traffic and assigns the appropriate share value to it. The user interface displays the term (default) in the Network resource pool settings screen. vSphere 5.0 introduced User-defined network resource pools and these are only applicable to virtual machine network traffic. User defined network pools are excellent to partition your network when multiple customers are using a shared network infrastructure. Physical adapter Shares NetIOC shares are comparable to the traditional CPU and memory shares. In the case of NetIOC, shares assigned to a network resource pool determine the portion of the total available bandwidth if contention occurs. Similar to compute shares, shares are only relative to the other active shares using the same resource. NetIOC provides 3 predefined share levels and a custom share level. The predefined share levels; low, normal and high provide an easy method of assigning a number of shares to the network resource pool. Low assigns 25 shares to the network resource pool, Normal 50 shares and High 100 shares. Custom allows you to assign the number of shares yourself within the supported range of 1 – 100. By default every system network resource pool is assigned 50 shares with the exception of the virtual machine traffic resource pool, this NRP gets 100 shares. The key to understand network resource pools and shares is that the shares apply on the physical adapter; hence the name physical adapter shares ;). This means that if the physical adapter of a host is saturated the shares of the network resource pools actively transmitting are in play. For example, your distributed switch is configured with a Management portgroup, a vMotion portgroup, NFS and virtual machine portgroup. All NRPs are configured with default shares and you use 2 uplinks. For the sake of simplicity, in this scenario both uplinks are configured as active uplinks Although this environment is configured with 4 portgroups, the default system network pools remain to exist. The existences of the non-utilized system-NRPs have no effect on the distribution of bandwidth during contention. As mentioned before the when the physical adapter is saturated, then the shares apply. In the following scenario vmnic0 of host ESX02 is saturated, as there are only 4 portgroups active on the distributed switch, the shares of the network pools are applied: This means that 50+50+50+100 (250) shares are active, in this scenario the virtual machine network resource pool gets to divide 40% (100/(50+50+50+100)) of the available physical network bandwidth. If vmnic0 was a 10GB NIC, the virtual machines network pool would receive 4GB to distribute amongst the actively transmitting virtual machines on that host. This was a worst-case scenario because usually not all portgroups are transmitting, as the shares are relative to other network pools actively using the physical adapter it might happen that Virtual Machine and vMotion traffic is only active on this NIC. In that case only the shares of the vMotion NRP and VM NRP are compared against each other to determine the available bandwidth for both network resource pools. The moment another traffic source transmits to the distributed switch a new calculation is made to determine the available bandwidth for the network resource pools. For example, HA is heart beating across the management LAN, using vmnic0, therefor the active transmitting network resource pools are Management, vMotion and virtual machines, generating a distribution of network bandwidth as follows: Host Limits Up to vSphere 5.1 NetIOC applies a limit per host. This means that the host limit enforces a traffic bandwidth limit on the overall set of dvUplinks for that particular network resource pool. The limit is expressed in an absolute unit of Mbps. This means that if you set a 3000Mpbs limit on the vMotion network resource pool, the traffic stream of the vMotion network resource pool will never exceed the given limit of 3000Mpbs for a distributed switch out of a particular ESX host. vSphere 5.1 introduced a big adjustment to hosts limits, the host limits now applies to each individual uplink. This means that when setting a host limit on the network resource pool for vMotion of 3000Mbps, vMotion is limited to transmit a maximum of 3Gb per uplink. In the case of a Multi-NIC vMotion configuration (2NICs) the maximum traffic vMotion can issue to the vmnics is 6Gb. Please note that limits only apply on ingress traffic (incoming traffic from vm to vds) meaning that a limit only affects native traffic coming from the active virtual machine running on the host or the vMotion traffic initiated on the host itself. Coming up next.. The next article in this series is how to use NetIOC for predictable network performance when using a Multi-NIC vMotion configuration on a distributed vSwitch.
STORAGE DRS SURVEY
If you have a few spare minutes, please fill out the survey about Storage DRS usage. We are very interested in your Storage DRS architecture and which options you use. But we are also seeking to identify adoption blockers preventing you to use Storage DRS. It takes about 5 minutes to get through. Storage DRS survey
TWO VERY INTERESTING VIDEOS ABOUT HOW COMPUTER ALGORITHMS SHAPES TODAY'S WORLD
Resource management within virtual infrastructures relies on distributed algorithms, as a result I became more and more interested in the application of computer algorithms in other areas. Today I found an English version of the multiple award winning Dutch documentary which I can finally share with my non-dutch speaking friends. The documentary reviews the flash crash on the U.S. Stock Market on May 6th 2010. In particular it explores the application of black box trading (algo-trading) and how algorithms shaped and formed the architecture of not only of the trade market institutions but also city architectures and human terraforming. Money & Speed: Inside the Black Box (Marije Meerman, VPRO) Sit back and be amazed. Once viewed, view the TED presentation by Kevin Slavin: How algorithms shape our world. Kevin Slavin zooms in some of the algorithms applied in our lives and how it affect us and our surroundings.
ADJUSTING THE COST OF VMOTION – A WORD OF CAUTION
Yesterday I posted an article on how to change the cost of vMotion in order to change the default number of concurrent vMotion. As I mentioned in the article, I’m not a proponent of changing advanced settings. Today Kris posted a very interesting question; How about the scenario where one uses multi NIC vMotion for against two 5Gbps virtual adapters)? I know a cost of 4 will be set for the network by the VMkernel, however as the aggregate bandwidth becomes 10Gbps is it safe enough to raise the limit? Perhaps not to the full 8 for 10Gbps, but 6?
LIMITING THE NUMBER OF CONCURRENT VMOTIONS
After explaining how to limit the number of concurrent Storage vMotions operations, I received multiple questions on how to limit the number of concurrent vMotion operations. This article will cover the cost and max cost constructs and show you how to calculate the correct config key values to limit the number of concurrent vMotion operations. Please note I usually do not post on configuration keys that change default behavior simply because I feel that most defaults are sufficient, and it should only be changed as a last resort when all other avenues are exhausted. I would like to mention that this is an unsupported configuration. Support will request to remove these settings before troubleshooting your environment!
ADDING NEW DISK TO AN EXISTING VIRTUAL MACHINE IN A STORAGE DRS DATASTORE CLUSTER
Recently I had some discussions where I needed to clarify the behavior of Storage DRS when the user adds a new disk to a virtual machine that is already running in the datastore cluster. Especially what will happen if the datastore cluster is near its capacity? When adding a new disk, Storage DRS reviews the configured affinity rule of the virtual machine. By default Storage DRS applies an affinity rule (Keep VMDKs together by default) to all new virtual machines. vSphere 5.1 allows you to change the default behavior of the cluster, you can change the default affinity rule in the Datastore cluster settings: Adding a new disk to an existing virtual machine The first one to realize is that Storage DRS never can violate the affinity or anti-affinity rule of the virtual machine. For example, if the datastore cluster default affinity rule is set to “keep VMDKs together” then all the files are placed on the same datastore. Ergo if a new disk is added to the virtual machine, that disk must be stored on the same datastore in order not to violate the affinity rule. Let use an example, VM1 is placed in the datastore and is configured with the Intra-VM affinity rule (Keep the files inside the VM together). The virtual machine is configured with 2 hard disks and both reside on the datastore [nfs-f-07] of the datastore cluster When adding another disk, Storage DRS provides me with an recommendation Although all the other datastores inside the datastore cluster are excellent candidates as well, Storage DRS is forced to place the VMDK on the same datastore together with the rest of the virtual machine files. Datastore cluster defragmentation At one point you may find yourself in a situation where the datastores inside the datastore cluster are quite full. Not enough free space per datastore, but enough free space in the datastore cluster to host another disk of an exisiting virtual machine. In that situation, Storage DRS does not break the affinity rule but starts to “defragment” the datastore cluster. It will move virtual machines around to provide enough free space for the new VMDK. The article “Storage-DRS initial placement and datastore cluster defragmentation" can provide you more information about this mechanism. Key takeaway Therefore in a datastore cluster you will never see Storage DRS splitting up a virtual machine if the VM is configured with an affinity rule, but you will see pre-requisite moves, migrating virtual machines out of the datastore to make room for the new vmdk.
WHY DO YOU MANUALLY SELECT A DATASTORE WHILE USING STORAGE DRS?
On the community forums a couple of threads are active about the Storage DRS Automation level behavior when trying to manually migrate a virtual machine between datastores in the same datastore cluster. When migrating within the datastore cluster, Storage DRS is disabled for that virtual machine. Some community members asked me why Storage DRS disables automation for this virtual machine when migrating between datastores inside a datastore cluster or when selecting a datastore during placement of a new virtual machine. Intent It is all about intent. When migrating a virtual machine into a datastore cluster, you are migrating the virtual machine into a load-balancing domain (the datastore cluster). You allow and trust Storage DRS to provide you an environment that provides an optimum load balanced state where the virtual machines receive the overall best I/O performance and the optimal placement regarding space utilization. If the user wants to migrate the virtual machine to a different datastore inside the datastore cluster, Storage DRS is capturing this intent, as “user knows best”. The way this is designed is that if a datastore is selected, then user is telling us that the selected datastore is the best, i.e. user knows something Storage DRS doesn’t. And to prohibit any future migration recommendation to other datastores, Storage DRS is disabled to ensure permanent placement. This behavior also applies when migrating a virtual machine into a datastore cluster. During initial placement it is expected that the user selects the datastore cluster, if the user wants to select a specific datastore it has to select “Disable Storage DRS for this virtual machine” in order to be able to select a member datastore. But this brings me to the question I have; what is the reason for not trusting Storage DRS? Why do you manually select a datastore while using Storage DRS? Apparently old habits die-hard and most tell me that their administrator feels like they could beat Storage DRS in placement. I’ve written a couple of articles about it (plus two 100+ page chapters featured in two books) about the working of Storage DRS and trust me it’s very difficult to beat Storage DRS placement and migration recommendations. During development the engineers try to run an equivalent experiment to IBM big blue versus Gary Kasparov and lined up two world-class storage experts versus Storage DRS. Although they received answers to all their questions they could not match the overall performance improvement Storage DRS could provide. Correlation of metrics Storage DRS has a lot of visibility into the environment, it measures space growth rates of existing virtual machine with thin disks, snapshots, etc. It selects destination datastore based on current utilization and growth rates. It builds device models to understand the performance of the devices backing the datastore as well as measuring the overall load on the datastores. It creates workload models of the existing virtual machine and measures on multiple metrics. Due to the insights Storage DRS can decide to migrate virtual machines to other datastores in order to make room or avoid the I/O threshold. It analyzes the environment and prefers moving virtual machines with low storage vMotion overhead. For more information please read the following articles: Storage DRS automation level and initial placement behavior Storage DRS Initial placement and datastore cluster defragmentation Avoiding VMDK level over commitment while using Thin disks and Storage DRS If you have a specific use case, for example to run some benchmark test on a datastore, then the option “Disable Storage DRS for this virtual machine” helps you to prevent Storage DRS from interrupting your test. However I would recommend selecting the datastore cluster as a destination instead of a specific datastore when migrating a virtual machine into a datastore cluster. Read the article Storage vMotion migration into a datastore cluster for more information. Remember Storage DRS always generate a recommendation that you can review during provisioning. After selecting the destination (datastore cluster), the user interface provides an overview of the current selections, at the right part of the screen a link “more recommendations” is provided. More recommendations After you click on the more recommendation link, the user interface provides you with a list of alternative recommendations. The order of the list is that the top recommendation provides the best placement, this is the same recommendation listed in the previous review selection screen. The list provides an overview of the space utilization before placement (2nd column), space utilization after placement (3rd column) and I/O latency before placement on the destination datastore (4th column). As the screenshot shows, the 2nd recommendation shows that placing the EMC-003 datastore provides the best placement. This datastore has the lowest utilization before and after placement and has the lowest I/O latency of all the other datastores inside the datastore cluster. Use this screen to educate your team responsible for provisioning and placement, show them that Storage DRS take multiple metrics into account and review the impact of the result if they picked the datastore of their choice. For my education, please share your thoughts on why you want to manually select a datastore that is a part of a datastore cluster?