HA/DRS/Upgrade/

What is HA, Explain HA ?

vSphere HA provides high availability for virtual machines by pooling the virtual machines and the hosts they reside on into a cluster. Hosts in the cluster are monitored and in the event of a failure, the virtual machines on a failed host are restarted on alternate hosts.

 

When you create a vSphere HA cluster, a single host is automatically elected as the master host. The master host communicates with vCenter Server and monitors the state of all protected virtual machines and of the slave hosts. Different types of host failures are possible, and the master host must detect and appropriately deal with the failure. The master host must distinguish between a failed host and one that is in a network partition or that has become network isolated. The master host uses network and datastore heart beating to determine the type of failure.

 

Master and Subordinate/Slave Hosts-

When you add a host to a vSphere HA cluster, an agent is uploaded to the host and configured to communicate with other agents in the cluster. Each host in the cluster functions as a master host or a subordinate host.

When vSphere HA is enabled for a cluster, all active hosts (that are not in standby, maintenance mode or not disconnected) participate in an election to choose the cluster’s master host.

 

The host that mounts the greatest number of datastores has an advantage in the election. Only one master host typically exists per cluster and all other hosts are subordinate hosts. If the master host fails, is shut down or put in standby mode, or is removed from the cluster a new election is held.

 

The master host in a cluster has several responsibilities:

  • Monitoring the state of subordinate hosts. If a subordinate host fails or becomes unreachable, the master host identifies which virtual machines must be restarted.
  • Monitoring the power state of all protected virtual machines. If one virtual machine fails, the master host ensures that it is restarted. Using a local placement engine, the master host also determines where the restart takes place.
  • Managing the lists of cluster hosts and protected virtual machines.
  • Acting as the vCenter Server management interface to the cluster and reporting the cluster health state.

 

The subordinate hosts primarily contribute to the cluster by running virtual machines locally, monitoring their runtime states, and reporting state updates to the master host. A master host can also run and monitor virtual machines. Both subordinate hosts and master hosts implement the VM and Application Monitoring features.

One of the functions performed by the master host is to orchestrate restarts of protected virtual machines. A virtual machine is protected by a master host after vCenter Server observes that the virtual machine’s power state has changed from powered off to powered on in response to a user action. The master host persists the list of protected virtual machines in the cluster’s datastores. A newly elected master host uses this information to determine which virtual machines to protect.

 

Hosts failure types –

The master host of a  High Availability cluster is responsible for detecting the failure of subordinate hosts. Depending on the type of failure detected, the virtual machines running on the hosts might need to be failed over.

In a vSphere HA cluster, three types of host failure are detected:

 

  • Failure. A host stops functioning.
  • Isolation. A host becomes network isolated.
  • Partition. A host loses network connectivity with the master host.

 

The master host monitors the liveness of the subordinate hosts in the cluster. This communication happens through the exchange of network heartbeats every second. When the master host stops receiving these heartbeats from a subordinate host, it checks for host liveness before declaring the host failed. The liveness check that the master host performs is to determine whether the subordinate host is exchanging heartbeats with one of the datastores. See Datastore Heartbeating. Also, the master host checks whether the host responds to ICMP pings sent to its management IP addresses.

 

If a master host cannot communicate directly with the agent on a subordinate host, the subordinate host does not respond to ICMP pings. If the agent is not issuing heartbeats, it is viewed as failed. The host’s virtual machines are restarted on alternate hosts. If such a subordinate host is exchanging heartbeats with a datastore, the master host assumes that the subordinate host is in a network partition or is network isolated. So, the master host continues to monitor the host and its virtual machines. See Network Partitions.

 

Host network isolation occurs when a host is still running, but it can no longer observe traffic from vSphere HA agents on the management network. If a host stops observing this traffic, it attempts to ping the cluster isolation addresses. If this pinging also fails, the host declares that it is isolated from the network.

The master host monitors the virtual machines that are running on an isolated host. If the master host observes that the VMs power off, and the master host is responsible for the VMs, it restarts them.

 

Respond to Host Failure

You can set specific responses to host failures that occur in your vSphere HA cluster.

 

Option Description
Failure Response If you select Disabled, this setting turns off host monitoring and VMs are not restarted when host failures occur. If Restart VMs is selected, VMs are failed over based on their restart priority when a host fails.
Default VM Restart Priority The restart priority determines the order in which virtual machines are restarted when the host fails. Higher priority virtual machines are started first. If multiple hosts fail, all virtual machines are migrated from the first host in order of priority, then all virtual machines from the second host in order of priority, and so on.
VM Dependency Restart Condition A specific condition must be selected as well as a delay after that condition has been met, before vSphere HA is allowed to continue to the next VM restart priority.

 

 

Host Isolation Response –

Host isolation response determines what happens when a host in a vSphere HA cluster loses its management network connections, but continues to run.

You can use the isolation response to have vSphere HA power off virtual machines that are running on an isolated host and restart them on a non-isolated host. Host isolation responses require that Host Monitoring Status is enabled. If Host Monitoring Status is disabled, host isolation responses are also suspended. A host determines that it is isolated when it is unable to communicate with the agents running on the other hosts, and it is unable to ping its isolation addresses. The host then executes its isolation response. The responses are Power off and restart VMs or Shutdown and restart VMs. You can customize this property for individual virtual machine

 

Note – If a virtual machine has a restart priority setting of Disabled, no host isolation response is made.

 

Respond to Host Isolation

You can set specific responses to host isolation that occurs in your vSphere HA cluster.

Procedure

  1. In the vSphere Web Client, browse to the vSphere HA cluster.
  2. Click the Configure tab.
  3. Select vSphere Availability and click Edit.
  4. Click Failures and Responses and expand Response for Host Isolation.
  5. To configure the host isolation response, select DisabledShut down and restart VMs, or Power off and restart VMs.
  6. Click OK.

Results

Your setting for the host isolation response takes effect.

 

 

Factors Considered for Virtual Machine Restarts

After a failure, the cluster’s master host attempts to restart affected virtual machines by identifying a host that can power them on. When choosing such a host, the master host considers a number of factors.

File accessibility

Before a virtual machine can be started, its files must be accessible from one of the active cluster hosts that the master can communicate with over the network

Virtual machine and host compatibility

If there are accessible hosts, the virtual machine must be compatible with at least one of them. The compatibility set for a virtual machine includes the effect of any required VM-Host affinity rules. For example, if a rule only permits a virtual machine to run on two hosts, it is considered for placement on those two hosts.

Resource reservations

Of the hosts that the virtual machine can run on, at least one must have sufficient unreserved capacity to meet the memory overhead of the virtual machine and any resource reservations. Four types of reservations are considered: CPU, Memory, vNIC, and Virtual flash. Also, sufficient network ports must be available to power on the virtual machine.

Host limits

In addition to resource reservations, a virtual machine can only be placed on a host if doing so does not violate the maximum number of allowed virtual machines or the number of in-use vCPUs.

Feature constraints

If the advanced option has been set that requires vSphere HA to enforce VM to VM anti-affinity rules, vSphere HA does not violate this rule. Also, vSphere HA does not violate any configured per host limits for fault tolerant virtual machines.

If no hosts satisfy the preceding considerations, the master host issues an event stating that there are not enough resources for vSphere HA to start the VM and tries again when the cluster conditions have changed. For example, if the virtual machine is not accessible, the master host tries again after a change in file accessibility.

 

VM and Application Monitoring

 

VM Monitoring restarts individual virtual machines if their VMware Tools heartbeats are not received within a set time. Similarly, Application Monitoring can restart a virtual machine if the heartbeats for an application it is running are not received. You can enable these features and configure the sensitivity with which vSphere HA monitors non-responsiveness.

When you enable VM Monitoring, the VM Monitoring service (using VMware Tools) evaluates whether each virtual machine in the cluster is running by checking for regular heartbeats and I/O activity from the VMware Tools process running inside the guest. If no heartbeats or I/O activity are received, this is most likely because the guest operating system has failed or VMware Tools is not being allocated any time to complete tasks. In such a case, the VM Monitoring service determines that the virtual machine has failed and the virtual machine is rebooted to restore service.

 

Occasionally, virtual machines or applications that are still functioning properly stop sending heartbeats. To avoid unnecessary resets, the VM Monitoring service also monitors a virtual machine’s I/O activity. If no heartbeats are received within the failure interval, the I/O stats interval (a cluster-level attribute) is checked. The I/O stats interval determines if any disk or network activity has occurred for the virtual machine during the previous two minutes (120 seconds). If not, the virtual machine is reset. This default value (120 seconds) can be changed using the advanced option das.iostatsinterval.

 

To enable Application Monitoring, you must first obtain the appropriate SDK (or be using an application that supports VMware Application Monitoring) and use it to set up customized heartbeats for the applications you want to monitor. After you have done this, Application Monitoring works much the same way that VM Monitoring does. If the heartbeats for an application are not received for a specified time, its virtual machine is restarted.

 

You can configure the level of monitoring sensitivity. Highly sensitive monitoring results in a more rapid conclusion that a failure has occurred. While unlikely, highly sensitive monitoring might lead to falsely identifying failures when the virtual machine or application in question is actually still working, but heartbeats have not been received due to factors such as resource constraints. Low sensitivity monitoring results in longer interruptions in service between actual failures and virtual machines being reset. Select an option that is an effective compromise for your needs.

You can also specify custom values for both monitoring sensitivity and the I/O stats interval by selecting the Custom checkbox.

 

VM Component Protection

If VM Component Protection (VMCP) is enabled, vSphere HA can detect datastore accessibility failures and provide automated recovery for affected virtual machines.

 

VMCP provides protection against datastore accessibility failures that can affect a virtual machine running on a host in a vSphere HA cluster. When a datastore accessibility failure occurs, the affected host can no longer access the storage path for a specific datastore. You can determine the response that vSphere HA will make to such a failure, ranging from the creation of event alarms to virtual machine restarts on other hosts.

 

Note: When you use the VM Component Protection feature, your ESXi hosts must be version 6.0 or higher.

 

Types of Failure

There are two types of datastore accessibility failure:

PDL

PDL (Permanent Device Loss) is an unrecoverable loss of accessibility that occurs when a storage device reports the datastore is no longer accessible by the host. This condition cannot be reverted without powering off virtual machines.

 

APD

APD (All Paths Down) represents a transient or unknown accessibility loss or any other unidentified delay in I/O processing. This type of accessibility issue is recoverable.

Configure VMCP Response

Configure the response that VM Component Protection (VMCP) makes when a datastore encounters a PDL or APD failure.

Procedure

  1. In the vSphere Web Client, browse to the vSphere HA cluster.
  2. Click the Configure tab.
  3. Select vSphere Availability and click Edit.
  4. Click Failures and Responses, and expand either Datastore with PDL or Datastore with APD .
  5. If you clicked Datastore with PDL, you can set the VMCP failure response for this type of issue, either DisabledIssue Events, or Power off and restart VMs.
  6. If you clicked Datastore with APD, you can set the VMCP failure response for this type of issue, either DisabledIssue EventsPower off and restart VMs–Conservative restart policy, or Power off and restart VMs–Aggressive restart policy. You can also set Response recovery, which is the number of minutes that VMCP waits before taking action.
  7. Click OK.

Results

Your settings for the VMCP failure response take effect.

 

Datastore Heartbeating

 

When the master host in a High Availability cluster cannot communicate with a subordinate host over the management network, the master host uses datastore heartbeating to determine whether the subordinate host has failed, is in a network partition, or is network isolated. If the subordinate host has stopped datastore heartbeating, it is considered to have failed and its virtual machines are restarted elsewhere.

 

VMware vCenter Server® selects a preferred set of datastores for heartbeating. This selection is made to maximize the number of hosts that have access to a heartbeating datastore and minimize the likelihood that the datastores are backed by the same LUN or NFS server.

You can use the advanced option das.heartbeatdsperhost to change the number of heartbeat datastores selected by vCenter Server for each host. The default is two and the maximum valid value is five.

 

vSphere HA creates a directory at the root of each datastore that is used for both datastore heartbeating and for persisting the set of protected virtual machines. The name of the directory is .vSphere-HA. Do not delete or modify the files stored in this directory, because this can have an impact on operations. Because more than one cluster might use a datastore, subdirectories for this directory are created for each cluster. Root owns these directories and files and only root can read and write to them. The disk space used by vSphere HA depends on several factors including which VMFS version is in use and the number of hosts that use the datastore for heartbeating. With vmfs3, the maximum usage is 2GB and the typical usage is 3MB. With vmfs5, the maximum and typical usage is 3MB. vSphere HA use of the datastores adds negligible overhead and has no performance impact on other datastore operations.

vSphere HA limits the number of virtual machines that can have configuration files on a single datastore. See Configuration Maximums for updated limits. If you place more than this number of virtual machines on a datastore and power them on, vSphere HA protects virtual machines only up to the limit.

Note:

A vSAN datastore cannot be used for datastore heartbeating. Therefore, if no other shared storage is accessible to all hosts in the cluster, there can be no heartbeat datastores in use. However, if you have storage that is accessible by an alternate network path independent of the vSAN network, you can use it to set up a heartbeat datastore.

 

vSphere HA Admission Control

vSphere HA uses admission control to ensure that sufficient resources are reserved for virtual machine recovery when a host fails.

 

The basis for vSphere HA admission control is how many host failures your cluster is allowed to tolerate and still guarantee failover. The host failover capacity can be set in three ways:

  • Cluster resource percentage
  • Slot policy
  • Dedicated failover hosts

 

Note: vSphere HA admission control can be disabled. However, without it you have no assurance that the expected number of virtual machines can be restarted after a failure. Do not permanently disable admission control.

 

Regardless of the admission control option chosen, a VM resource reduction threshold also exists. You use this setting to specify the percentage of resource degradation to tolerate, but it is not available unless vSphere DRS is enabled.

 

Cluster Resources Percentage Admission Control

 

You can configure vSphere HA to perform admission control by reserving a specific percentage of cluster CPU and memory resources for recovery from host failures.

With this type of admission control, vSphere HA ensures that a specified percentage of aggregate CPU and memory resources are reserved for failover.

With the cluster resources percentage option, vSphere HA enforces admission control as follows:

  1. Calculates the total resource requirements for all powered-on virtual machines in the cluster.
  2. Calculates the total host resources available for virtual machines.
  3. Calculates the Current CPU Failover Capacity and Current Memory Failover Capacity for the cluster.
  4. Determines if either the Current CPU Failover Capacity or Current Memory Failover Capacity is less than the corresponding Configured Failover Capacity (provided by the user).
    If so, admission control disallows the operation.

vSphere HA uses the actual reservations of the virtual machines. If a virtual machine does not have reservations, meaning that the reservation is 0, a default of 0MB memory and 32MHz CPU is applied.

 

 

Admission Control Using Cluster Resources Percentage

The way that Current Failover Capacity is calculated and used with this admission control policy is shown with an example. Make the following assumptions about a cluster:

  • The cluster is comprised of three hosts, each with a different amount of available CPU and memory resources. The first host (H1) has 9GHz of available CPU resources and 9GB of available memory, while Host 2 (H2) has 9GHz and 6GB and Host 3 (H3) has 6GHz and 6GB.
  • There are five powered-on virtual machines in the cluster with differing CPU and memory requirements. VM1 needs 2GHz of CPU resources and 1GB of memory, while VM2 needs 2GHz and 1GB, VM3 needs 1GHz and 2GB, VM4 needs 1GHz and 1GB, and VM5 needs 1GHz and 1GB.
  • The Configured Failover Capacity for CPU and Memory are both set to 25%.

Admission Control Example with Percentage of Cluster Resources Reserved Policy

 

 

The total resource requirements for the powered-on virtual machines is 7GHz and 6GB. The total host resources available for virtual machines is 24GHz and 21GB. Based on this, the Current CPU Failover Capacity is 70% ((24GHz – 7GHz)/24GHz). Similarly, the Current Memory Failover Capacity is 71% ((21GB-6GB)/21GB).

Because the cluster’s Configured Failover Capacity is set to 25%, 45% of the cluster’s total CPU resources and 46% of the cluster’s memory resources are still available to power on additional virtual machines.

 

Slot Policy Admission Control

With the slot policy option, vSphere HA admission control ensures that a specified number of hosts can fail and sufficient resources remain in the cluster to fail over all the virtual machines from those hosts.

Using the slot policy, vSphere HA performs admission control in the following way:

  • Calculates the slot size. A slot is a logical representation of memory and CPU resources. By default, it is sized to satisfy the requirements for any powered-on virtual machine in the cluster.
  • Determines how many slots each host in the cluster can hold.
  • Determines the Current Failover Capacity of the cluster.

 

This is the number of hosts that can fail and still leave enough slots to satisfy all of the powered-on virtual machines.

 

  • Determines whether the Current Failover Capacity is less than the Configured Failover Capacity (provided by the user).
    If it is, admission control disallows the operation.

 

Note: You can set a specific slot size for both CPU and memory in the admission control section of the vSphere HA settings in the vSphere Web Client.

 

Slot size is comprised of two components, CPU and memory.

 

vSphere HA calculates the CPU component by obtaining the CPU reservation of each powered-on virtual machine and selecting the largest value. If you have not specified a CPU reservation for a virtual machine, it is assigned a default value of 32MHz. You can change this value by using the das.vmcpuminmhzadvanced option.)

vSphere HA calculates the memory component by obtaining the memory reservation, plus memory overhead, of each powered-on virtual machine and selecting the largest value. There is no default value for the memory reservation.

 

Admission Control Using Slot Policy

The way that slot size is calculated and used with this admission control policy is shown in an example. Make the following assumptions about a cluster:

  1. The cluster is comprised of three hosts, each with a different amount of available CPU and memory resources. The first host (H1) has 9GHz of available CPU resources and 9GB of available memory, while Host 2 (H2) has 9GHz and 6GB and Host 3 (H3) has 6GHz and 6GB.
  2. There are five powered-on virtual machines in the cluster with differing CPU and memory requirements. VM1 needs 2GHz of CPU resources and 1GB of memory, while VM2 needs 2GHz and 1GB, VM3 needs 1GHz and 2GB, VM4 needs 1GHz and 1GB, and VM5 needs 1GHz and 1GB.
  3. The Host Failures Cluster Tolerates is set to one.
  4. Admission Control Example with Host Failures Cluster Tolerates Policy
  1. Slot size is calculated by comparing both the CPU and memory requirements of the virtual machines and selecting the largest.
    The largest CPU requirement (shared by VM1 and VM2) is 2GHz, while the largest memory requirement (for VM3) is 2GB. Based on this, the slot size is 2GHz CPU and 2GB memory.
  2. Maximum number of slots that each host can support is determined.
    H1 can support four slots. H2 can support three slots (which is the smaller of 9GHz/2GHz and 6GB/2GB) and H3 can also support three slots.
  3. Current Failover Capacity is computed.
    The largest host is H1 and if it fails, six slots remain in the cluster, which is sufficient for all five of the powered-on virtual machines. If both H1 and H2 fail, only three slots remain, which is insufficient. Therefore, the Current Failover Capacity is one.

The cluster has one available slot (the six slots on H2 and H3 minus the five used slots).

 

Dedicated Failover Hosts Admission Control

With dedicated failover hosts admission control, when a host fails, vSphere HA attempts to restart its virtual machines on any of the specified failover hosts. If restarting the virtual machines is not possible, for example the failover hosts have failed or have insufficient resources, then vSphere HA attempts to restart those virtual machines on other hosts in the cluster.

 

To ensure that spare capacity is available on a failover host, you are prevented from powering on virtual machines or using vMotion to migrate virtual machines to a failover host. Also, DRS does not use a failover host for load balancing.

 

Configure Admission Control

  1. After you create a cluster, you can configure admission control to specify whether virtual machines can be started if they violate availability constraints. The cluster reserves resources so that failover can occur for all running virtual machines on the specified number of hosts.
  2. The Admission Control page appears only if you enabled vSphere HA.
  3. Procedure
    1. In the vSphere Web Client, browse to the vSphere HA cluster.
    2. Click the Configure tab.
    3. Select vSphere Availability and click Edit.
    4. Click Admission Control to display the configuration options.
    5. Select a number for the Host failures cluster tolerates. This is the maximum number of host failures that the cluster can recover from or guarantees failover for.
    6. Select an option for Define host failover capacity by.
    1. Option
    1. Description
    1. Cluster resource percentage
    1. Specify a percentage of the cluster’s CPU and memory resources to reserve as spare capacity to support failovers.
    1. Slot Policy (powered-on VMs)
    1. Select a slot size policy that covers all powered on VMs or is a fixed size. You can also calculate how many VMs require multiple slots.
    1. Dedicated failover hosts
    1. Select hosts to use for failover actions. Failovers can still occur on other hosts in the cluster if a default failover host does not have enough resources.
    1. Disabled
    1. Select this option to disable admission control and allow virtual machine power ons that violate availability constraints.
  1. Set the percentage for the Performance degradation VMs tolerate.
    This setting determines what percentage of performance degradation the VMs in the cluster are allowed to tolerate during a failure.

 

 

 

 

 

Configure Proactive HA

 

You can configure how Proactive HA responds when a provider has notified its health degradation to vCenter, indicating a partial failure of that host.

This page is editable only if you have enabled vSphere DRS.

Procedure

  1. In the vSphere Web Client, browse to the Proactive HA cluster.
  2. Click the Configure tab.
  3. Select vSphere Availability and click Edit.
  4. Select Turn on Proactive HA.
  5. Click Proactive HA Failures and Responses.
  6. Select from the following configuration options.
Option Description
Automation Level Determine whether host quarantine or maintenance mode and VM migrations are recommendations or automatic.

  • Manual. vCenter Server suggests migration recommendations for virtual machines.
  • Automated. Virtual machines are migrated to healthy hosts and degraded hosts are entered into quarantine or maintenance mode depending on the configured Proactive HA automation level.
Remediation Determine what happens to partially degraded hosts.

  • Quarantine mode for all failures. Balances performance and availability, by avoiding the usage of partially degraded hosts provided that virtual machine performance is unaffected.
  • Quarantine mode for moderate and Maintenance mode for severe failure (Mixed). Balances performance and availability, by avoiding the usage of moderately degraded hosts provided that virtual machine performance is unaffected. Ensures that virtual machines do not run on severely failed hosts.
  • Maintenance mode for all failures. Ensures that virtual machines do not run on partially failed hosts.

Host.Config.Quarantine and Host.Config.Maintenance privileges are required to put hosts in Quarantine mode and Maintenance mode, respectively.

To enable Proactive HA providers for this cluster, select the check boxes. Providers appear when their corresponding vSphere Web Client plugin has been installed and the providers monitor every host in the cluster. To view or edit the failure conditions supported by the provider, click the edit link.

 

 

Election process of Master

The host that is participating in the election process having greatest number of datastores connected will be elected as master but if in case there are more than one or more host having the equal number of datastores connected then the one having the greatest Managed Object ID (MOID) will be chosen. This is done Lexically, means 99 beats 100 as 9 is greater than 1. You can see the HA host status in the summary tab.

 

What is MOID who assign it  ?

Every object within a vSphere environment is internally tracked, and referred to, by a unique identifier called moRef ID (“Managed Object Reference ID”).

 

Every virtual infrastructure object managed by vSphere (datacenters, clusters, hosts, datastores, VMs, vApps, vSwitches and so on) has a moRef ID.

 

What is the moRef ID?

This identifier is composed of a prefix stating the object type, followed by a numerical ID. For example:

vm-613

host-22

datastore-33

It is important to note that this identifier is guaranteed to be unique only within a single vCenter instance. In this article by William Lam it is explained that an additional identifier called InstanceUUID (unique for a given vCenter) was introduced in vSphere 5.0 that, when coupled with the moRef ID, gives a truly globally unique value.

Quick note: the moRef ID is not to be confused with VMs’ UUID. This universally unique value is stored in the SMBIOS and is conceptually similar to the unique BIOS system identifier in physical systems. This value, unlike the moRef ID, can be duplicated even within the same vCenter, for example as a result of a cloning operation, or a restore from backup.

If you ever need to examine logs from VMware components/solutions or 3rd party software integrating with vSphere, you’ll notice that objects often appear with their moRef ID rather than their name attribute.

For this reason, you may need from time to time to quickly match a moRef ID to the more “friendly” object name. As usual, there are several ways to do this.

 

More info click below ?

https://www.danilochiavari.com/2014/03/28/how-to-quickly-match-a-moref-id-to-a-name-in-vmware-vsphere/

DRS automation level –

Manual

Placement and migration recommendations are displayed, but do not run until you manually apply the recommendation.

Fully Automated

Placement and migration recommendations run automatically.

Partially Automated

Initial placement is performed automatically. Migration recommendations are displayed, but do not run.

Slot is logical representation of memory and cpu

1st – Slot size is calculated

2nd – HA determines how many slots each host inthe cluster can hold

3rd – calculate current failover capacity in of the cluster

4th – finaly it comapres this to the configured failover capacity by user

If the current failover capacity is lessthen configured addmision control will not allow furhet VM powered on

because suffecient slots wont exits incase of host failure

Vsphere 5.5 to 6.5 Upgrade complete infra (6.5 VCSA, ESXi, Vmware tool and Hardware version)

 

 

Source – https://sivasankar.org/2018/1288/upgrading-migrating-from-vsphere-5-x-to-6-x-best-practices/

High-level overview –

 

 

  • Know the Existing Environment (Eg – VC/ESX build version, ESX Hardware make/model/NIC/HBA card info,vswith/dvswitch/vlan/portgroups/uplinks, Cluste info HA/DRS rules/EVC details, All VM’s Name/OS/IP/Port group/DS details/Hardware version/VMware tool, RDM details SCSI ID/Physical/Virtual/Pointer Location, any backup and SRM Info.
  • Verify VMware and Hardware compatibility and upgrade matrix (Vmware VC to ESX compatibility, ESX host hardware support, SAN compatibility, All can be seen in Vmware site)
  • vSphere License and Support
  • Supported Drivers and Firmware’s for HW (NIC, FCOE, FC, ISCSI, Multipath whether MRU, Fixed, RR, Make sure firmware updated)

 

  • Migration/Upgrade approach and Steps :  Decide the vCenter topology Click here to find all topology, Decide whether VCSA or Windows based VC.

https://kb.vmware.com/s/article/2147672

Note – Windows based VC is not supported from 6.7 onwards  (I would prefer VCSA as it is more stable) from 6.7 no extended PSC, only Embedded

 

  • vCenter Upgrade/Migrate/Install : First thing to be migrated or upgrading is the vCenter server, after that only ESXi and VM’s will be migrated. After topology is decided find the upgrade path, (Eg – 5.5 to 6.5 U1, or 6.0 to 6.7, or migrate windows to 6.5 u1 or VCSA to VCSA)

 

  • ESXi Upgrade/Install :  Based on the upgrade path and hardware compatibility ESXi upgrade or fresh installation can be done. Its always good to do upgrade for existing servers as no need to do all configurations like vmotion, dns , Ip’s and all. For new Hardware of course fresh installation is the only approach.

 

  • Upgrade using Update Manager
  • Using VMware Orchestrator (Using workflow)
  • Using CD/Script etc
  • Install new ESX directly

 

  • ESXi 5.0 or 5.1 with updates cannot be directly upgraded to 6.5 an intermediate upgrade to 5.5 or 6.0 is required.
  • ESXi 5.5 or later can be directly upgraded to 6.5 or U1
  • ESXi 5.x  cannot be directly upgraded to 6.7 an intermediate upgrade to 6.0 is required.
  • ESXi 6.0 or later can be directly upgraded to 6.7

 

ESX Upgrade overview:

  1. verify the compatibility of the NIC, HBA and other for vSphere version.
  2. Install necessary supported BIOS and firmware version for hardware.
  3. Install/Update with OEM CD or fresh ESXi image from vmware.
  4. Check the correct version of drivers are present ( esxcli software vib list | grep -i driver_name )
  5. Install / Update Necessary compatible hardware drivers ( esxcli software vib install/update -d path_of_driver)
  6. Install necessary updates and security patches, i recommend using cli not update manager. cli takes few seconds to install.

 

 

Once ESX upgraded, Make sure all the VMs are connected and no issues,

Upgrade VM Hardware version and  Vmware tool, can be done using update manager.

 

***********************************************

How to Upgrade ESXi 6.0 to ESXi 6.5 using VMware Update Manager

vSphere 6.5 released with excellent new features to improve the security and high availability of critical applications running on the virtual machines. To make use of new vSphere features available with vSphere 6.5,

We need to upgrade the vCenter server 6.0 to vCenter server 6.5 and also Upgrade ESXi 6.0 to ESXi 6.5 along with the Virtual machine hardware upgrade & VMware Tools on the guest operating system.

There are multiple ways to upgrade ESXi 6.0 to ESXi 6.5 such as direct ISO image, upgrade via CLI (command Line) and upgrade via VMware Update Manager.

 

Method of upgrading ESXi 6.0 to ESXi 6.5 may depend on the number of the ESXi hosts in the infrastructure. Upgrade ESXi host via ISO or Command line may suitable for the infrastructure with less number of ESXi host. If you consider the enterprise infrastructure with more than hundreds of ESXi hosts may require to upgrade it via VMware Update Manager to simplify the upgrade procedure. In this article, I am going to explain the detailed procedure to upgrade ESXi 6.0 to ESXi 6.5 using VMware Update Manager.

 

Well Planning is required before we start the upgrade of ESXi hosts. Before upgrading ESXi hosts, We need to consider vCenter Server Upgrade and also need to validating that whether the existing hardware is compatible with ESXi 6.5. We also need to validate the compatible driver and firmware versions of network adapter,HBA adapters,etc. We also need to consider upgrading the firmware of your ESXi hardware before upgrading ESXi hosts.

 

All this needs to be planned prior to upgrading ESXi host. We can also combine drivers and other softwares such as mutipathing softwares such as Powerpath  in the baseline, once the hosts are upgraded. In this article, I will explain the detailed procedure to upgrade ESXi 6.0 to ESxi 6.5 using VMware Update Manager.

 

Pre-requisite to upgrade ESXi 6.0 to ESXi 6.5 using VMware Update Manager

 

  1. vCenter Server should be upgrade to vCenter 6.5. I have migrated my existing windows vCenter Server to vCSA 6.5. Take a look at how i did migration of Windows based vCenter 6.0 to vCenter Server appliance 6.5
  2. In vSphere 6.5, Update Manager is embedded with vCenter Server appliance. Ensure Update Manager is also upgraded to 6.5.
  3. Validate ESXi host hardware NIC’s, HBA and other devices compatibility with ESXi 6.5.
  4. Upgrade the hardware and BIOS firmware before starting the upgrade.

 

Upgrade ESXi 6.0 to ESXi 6.5 using VMware Update Manager

 

Before start upgrading the ESXi 6.0 to ESXi 6.5, We need to Import the ESXi image. You can import the hardware vendor customized ESXi 6.5 image based on the hardware make. Since this is my lab, I will be importing the VMware vanilla image of ESXi 6.5 . Login to your vCenter Server using vSphere Web Client,  Click on Update Manager -> Manage -> ESXi Images -> Import ESXi image

 

Click on Browse to browse towards the ESXi 6.5 ISO image location and Select the image to start uploading the ISO image into VMware Update Manager.

Once the import of ESXi 6.5 image is completed, Validate the version and Build number. Click on Close

Once the ESXi image is imported, Click on the ESXi 6.5 image and Select “Create Baseline”

Specify the name for this ESXi host upgrade baseline.  Click on Ok.

Newly created host upgrade baseline  “ESXi 6.5 Upgrade” will start appearing under host baselines.

To minimize the number of screenshots, I have captured the GIF image to explain the procedures in single image. In the vSphere Web client, Select Hosts & Clusters.

You can attach the created ESXi host upgrade baseline to either to individual ESXi host or even to cluster.

Select the ESXi to upgrade from ESXi 6.0 to ESXi 6.5, Click on Update Manager -> Attach Baseline -> Select “ESXi 6.5 Upgrade” baseline. 

Select the attached baseline and click on Scan for Upgrades. Once the scan reports that the attached image is “Non-Complaint”.

We are good to start with the upgrade.

You can place the ESXi host into maintenance mode. In the DRS enabled cluster, all the virtual machine in the host will be automatically migrate to different ESXi host in the cluster.

Once the ESXi host placed into maintenance mode, Click on “Remediate” -> Ensure ESXi 6.5 Upgrade baseline is selected.  -> Select the target ESXi host  -> click on Accept the license agreement -> Click on Next to accept all the default options before upgrading ESXi 6.0 to ESXi 6.5. -> Click on “Finish” to start the upgrade.

 

You can also notice that the ESXi host upgrade will be going on, when you monitor the ESXi  via host console.

Once the ESXi host upgrade is completed, You can notice that the attached ESXi 6.5 upgrade baseline become “Complaint” against the ESXi host.

 

 

Also you can validate the ESXi host is now upgraded to “ESXi 6.5.0”.

That’s it. We are done with the upgrade ESXi 6.0 to ESXi 6.5 using VMware Update Manager. I hope this is informative for you. Thanks for reading!!!. Be social and share it in social media, if you feel worth sharing it.

Distributed switch migration

Distributed switch is a major thing to consider when it comes to migration, As most of migrations will happen without downtime.

Fresh Installation of vCenter without Upgrade.

In this case first we need to manually configure the distributed switch in the new vCenter server however there is export and import option available for distributed switch which is made easy to export the switch and import as all the port groups and config will appear as is in the new vCenter else a manual config is required.

In this case while migrating the ESXi host from Old vCenter server to new vCenter server below to be considered.

  1. Select a Jump ESXi host residing in Old vCenter server ( this will be used for all VM’s migration)
  2. Remove one physical uplink from distributed switch on each ESXi host or the Jump ESXi host (in old vcenter)
  3. Create a Standard switch and port groups with same vLan info as distributed switch with the physical uplink removed added to it.
  4. Migrate the VM’s to the Jump ESXi Host using vMotion (old vCenter)
  5. Migrate the VM’s from Distributed to Standard on the Jump ESXi host (old vCenter)
  6. Move the Jump host from Distributed switch in old vCenter.
  7. Add the Jump ESXi host (Say ESXi 5.5) with VM’s running on it to the new vCenter server say 6.5 U1
  8. Jump Host ESXi host (Say ESXi 5.5) with VM’s running on it will be added to new vCenter say 6.5 U1 and shows as disconnected in old vCenter server say 5.5.
  9. Add the Jump host to Distributed switch in new vCenter.
  10. Migrate all VM’s on Jump host from Standard to Distributed switch.
  11. Move all VM’s from ESXi host say 5.5 to New ESXi 6.5 Hosts ( already added to DS) in new vCenter server.
  12. Disconnect the Jump host from New vCenter and add it back to Old vCenter , follow this until all VM’s are moved.

vCenter server Upgrade from existing VC 5.x.

In this case we no need to worry much as all the existing vCenter server 5.x information and configurations with ESXi hosts will come to the new vCenter server 6.x. Only thing to be considered is both Old ESXi hosts and New ESXi hosts are added to distributed switch after vCenter upgrade.

VM Migration

The important part during the migration is to make sure the VM is up and running. People always say no downtime, thanks to vmware migrations are easy. during the migration in most of the cases either of below or both will take place.

Migrating VM’s from Old vCenter to NEW vCenter ( no VC upgrade)

This is the case when a new vCenter server is built (not upgraded) and old vCenter is hosting the ESXi hosts and VM’s. As long as the vCenter supports ESXi version, we can add the ESXi host with VM’s running on it in the new vCenter server. This process will automatically disconnects from old VCenter and adds to new vCenter without interrupting the VM’s running on it.

If distributed switch is in use, move the VM’s from distributed switch to standard switch. this part is covered in Distributed switch section.

Note: when doing this if the ESXi host moved to new vCenter is having VM’s with RDM which are clustered with another VM. Try to move both hosts hosting the 2 Clustered RDM vm’s one after another and try to keep them on same vCenter server. Don’t leave one Clustered RDM VM on old vCenter and another on new vCenter. This will cause storage flapping issues.

Migrating VM’s between ESXi hosts of Different Versions under same vCenter server.

As long as both old version of ESXi host and new version of ESXi host are under the same vCenter server, Both hosts can host the VM resources like storage, RDM luns, Network port group and all. Its the matter of vMotion from the Old ESXi host to new ESXi host provided vMotion is configured. This way migration is possible even without a ping drop.

if distributed switch is there both old and new ESXi hosts needs to be added to the Distributed switch. If EVC mode is configured on the Cluster Hosting old ESXi host, make sure same EVC mode is configured on new ESXi host if wanted to do vMotion across them.

Things to verify:

  • Old ESXi host ( say ESXi 5.5 ) and New ESXi Host ( Say 6.5 U1) have visibility to all Datastores, RDM LUNs where VM’s are hosted.
  • Same Standard switch port groups are available on Old and New ESXi host.
  • Old and New ESXi host joined to Distributed switchs VM’s are using.
  • Same EVC Mode is configured on Cluster level of Old and New ESXi hosts for live vMotion.

Migrating VM’s with RDM (Physical / Virtual)

VM’s with RDM’s can also be migrated without downtime provided the destination host can see the RDM luns and the LUN’s where the pointers are saved.

However please note if the SCSI controller is shared by multiple RDM luns vMotion is not possible. Meaning to say suppose RDM lun 1 is on SCSI 1:0 and RDM Lun 2 is on SCSI 1:1 ports vMotion is not possible. This is the reason its always recommended to map each RDM lun with different SCSI controllers, like RDM lun 1 with SCSI 1:0 and RDM Lun 2 with SCSI 2:0.

Hope this post is useful, leave your suggestions and comments below.

 

How long it takes for HA master election process.

The HA master election takes approximately 15 seconds and is conducted using UDP

 

when a master is elected it will try to acquire ownership of all the datastores it can directly access or access by proxying requests to one of the slaves connected to it using the management network. For traditional storage architectures, it does this by locking a file called protectedlist that is stored on the datastores in an existing cluster. The master will also attempt to take ownership of any datastores it discovers along the way, and it will periodically retry any it could not take ownership of previously.

The naming format and location of this file is as follows: /<root of datastore>/.vSphere-HA/<cluster-specificdirectory>/protectedlist

 

The master uses this protectedlist file to store the inventory. It keeps track of which VMs are protected by HA. Calling it an inventory might be slightly overstating: it is a list of protected VMs and it includes information around VM CPU reservation and memory overhead. The master distributes this inventory across all datastores in use by the VMs in the cluster. The next screenshot shows an example of this file on one of the datastores.

 

What Slaves do ?

A slave has substantially fewer responsibilities than a master: a slave monitors the state of the VMs it is running and informs the master about any changes to this state.

The slave also monitors the health of the master by monitoring heartbeats. If the master becomes unavailable, the slaves initiate and participate in the election process. Last but not least, the slaves send heartbeats to the master so that the master can detect outages. Like the master to slave communication, all slave to master communication is point to point. HA does not use multicast.

 

HA files

Both the master and slave use files not only to store state, but also as a communication mechanism. We’ve already seen the protectedlist file used by the master to store the list of protected VMs. We will now discuss the files that are created by both the master and the slaves. Remote files are files stored on a shared datastore and local files are files that are stored in a location only directly accessible to that host.

Remote Files The set of powered on VMs is stored in a per-host “poweron” file. It should be noted that, because a master also hosts VMs, it also creates a “poweron” file. The naming scheme for this file is as follows: hostnumber-poweron

Tracking VM power-on state is not the only thing the “poweron” file is used for. This file is also used by the slaves to inform the master that it is isolated from the management network: the top line of the file will either contain a 0 or a 1. A 0 (zero) means not-isolated and a 1 (one) means isolated. The master will inform vCenter about the isolation of the host.

 

Local Files As mentioned before, when HA is configured on a host, the host will store specific information about its cluster locally.

Each host, including the master, will store data locally. The data that is locally stored is important state information. Namely, the VM-to-host compatibility matrix, cluster configuration, and host membership list. This information is persisted locally on each host. Updates to this information is sent to the master by vCenter and propagated by the master to the slaves. Although we expect that most of you will never touch these files – and we highly recommend against modifying them – we do want to explain how they are used:

clusterconfig This file is not human-readable. It contains the configuration details of the cluster.

vmmetadata This file is not human-readable. It contains the actual compatibility info matrix for every HA protected VM and lists all the hosts with which it is compatible plus a vm/host dictionary. ▪

fdm.cfg This file contains the configuration settings around logging. For instance, the level of logging and syslog details are stored in here. ▪

hostlist A list of hosts participating in the cluster, including hostname, IP addresses, MAC addresses and heartbeat datastores

 

This datastore heartbeat mechanism is used in case the master has lost network connectivity with one, or multiple, slaves. The datastore heartbeat mechanism is then used to validate whether a host has failed or is merely isolated/network partitioned. Isolation will be validated through the “poweron” file which, as mentioned earlier, will be updated by the host when it is isolated. Without the “poweron” file, there is no way for the master to validate isolatio

 

Let that be clear! Based on the results of checks of both files, the master will determine the appropriate action to take. If the master determines that a host has failed (no datastore heartbeats), the master will restart the failed host’s VMs. If the master determines that the slave is Isolated or Partitioned, it will only take action when it is appropriate to take action. With that meaning that the master will only initiate restarts when VMs are down or powered down / shut down by a triggered isolation response.

 

Datastore Heartbeat-

Datastore heartbeating enables a master to more determine the state of a host that is not reachable via the management network. This datastore heartbeat mechanism is used in case the master has lost network connectivity with one, or multiple, slaves. The datastore heartbeat mechanism is then used to validate whether a host has failed or is merely isolated/network partitioned. Isolation will be validated through the “poweron” file which, as mentioned earlier, will be updated by the host when it is isolated. Without the “poweron” file, there is no way for the master to validate isolation

 

By default, HA selects 2 heartbeat datastores

it will select datastores that are available on all hosts, or as many as possible. Although it is possible to configure an advanced setting (das.heartbeatDsPerHost) to allow for more datastores for datastore heartbeating we do not recommend configuring this option as the default should be sufficient for most scenarios, except for stretched cluster environments where it is recommended to have two in each site manually selected.

Isolated versus Partitioned

▪ An isolation event is the situation where a single host cannot communicate with the rest of the cluster. Note: single host!

▪A partition is the situation where two (or more) hosts can communicate with each other, but no longer can communicate with the remaining two (or more) hosts in the cluster. Note: two or more!

 

How failover ?

When the master stops receiving network heartbeats from a slave, it will check for host “liveness” for the next 15 seconds. Before the host is declared failed, the master will validate if it has actually failed or not by doing additional liveness checks. First, the master will validate if the host is still heartbeating to the datastore. Second, the master will ping the management IP address of the host. If both are negative, the host will be declared Failed. This doesn’t necessarily mean the host has PSOD’ed; it could be the network is unavailable, including the storage network, which would make this host Isolated from an administrator’s perspective but Failed from an HA perspective. As you can imagine, however, there are various combinations possible. The following table depicts these combinations including the “state”.

 

Note – In VMware vSphere 5.x and 6.x, if the agent is a master, then isolation is declared in 5 seconds. If it is a slave, isolation is declared in 30 seconds.

 

Host isolation address –

Sets the address to ping to determine if a host is isolated from the network. This address is pinged only when heartbeats are not received from any other host in the cluster. If not specified, the default gateway of the management network is used. This default gateway has to be a reliable address that is available, so that the host can determine if it is isolated from the network.

 

The Failure of a Slave

T0 – Slave failure.

▪ T3s – Master begins monitoring datastore heartbeats for 15 seconds.

▪ T10s – The host is declared unreachable and the master will ping the management network of the failed host. This is a continuous ping for 5 seconds.

▪ T15s – If no heartbeat datastores are configured, the host will be declared dead.

▪ T18s – If heartbeat datastores are configured, the host will be declared dead.

 

The Failure of a Master

Slaves receive network heartbeats from their master. If the master fails, let’s define this as T0 (T zero), the slaves detect this when the network heartbeats cease to be received. As every cluster needs a master, the slaves will initiate an election at T10s. The election process takes 15s to complete, which brings us to T25s. At T25s, the new master reads the protectedlist. This list contains all the VMs, which are protected by HA. At T35s, the master initiates the restart of all VMs that are protected but not currently running. The timeline depicted in the diagram below hopefully clarifies the process.

 

Isolation of a Slave –

 

Isolation of a master –

▪ T0 – Isolation of the host (master)

▪ T0 – Master pings “isolation addresses”

▪ T5s – Master declares itself isolated

▪ T35s – Master “triggers” isolation response

 

What is PDL ?

 

A PDL condition is a condition that is communicated by the array controller to ESXi via a SCSI sense code. This condition indicates that a device (LUN) has become unavailable and is likely permanently unavailable. An example scenario in which this condition would be communicated by the array would be when a LUN is set offline. This condition is used during a failure scenario to ensure ESXi takes appropriate action when access to a LUN is revoked. It should be noted that when a full storage failure occurs it is impossible to generate a PDL condition as there is no communication possible between the array and the ESXi host. This state will be identified by the ESXi host as an APD condition.

 

What is APD ?

As explained earlier, an APD condition is a situation where access to the storage is lost without receiving a SCSI sense code from the array. This for instance can happen when the network between the host and the storage system has failed, hence the name “all paths down.” When an APD condition occurs (access to a device is lost) the hypervisor starts a timer. After 140 seconds the APD condition is declared and the device is marked as APD time out. When the 140 seconds has passed, HA will start a timer. The HA time out is 3 minutes by default. When the 3 minutes has passed, HA will take the action defined within the UI. There are four options below

 

▪ Disabled ▪ Issue Events ▪ Power off and restart VMs – Conservative ▪ Power off and restart VMs – Aggressive

 

It is also good to know that if the APD is lifted and access to the storage is restored during the total of the approximate 5 minutes and 20 seconds it would take before the VM restart is initiated,

 

VM split brain –

A split-brain scenario is a scenario where a single VM is powered up multiple times, typically on two different hosts. This is possible in the scenario where the isolation response is set to “Disabled” and network based storage, like NFS / iSCSI and even Virtual SAN, is used. This situation can occur during a full network isolation, which may result in the lock on the VM’s VMDK being lost, enabling HA to actually power up the VM. As the VM was not powered off on its original host (isolation response set to “Disabled”), it will exist in memory on the isolated host and in memory with a disk lock on the host that was requested to restart the VM.

 

Admission Control Policy :

The Admission Control Policy dictates the mechanism that HA uses to guarantee enough resources are available for an HA initiated failover

 

3 policies –

Cluster Resource Percentage Algorithm :

The big change in vSphere 6.5 however is that there no longer is a need to specify a percentage manually, but you can now specify how many Host Failures the Cluster should Tolerate as shows in the below screenshot.

When you specify a number of Host Failures this number is then automatically calculated to a percentage. You can of course override this if you prefer to manually set the percentage, typically customers would keep CPU and memory equal. The big benefit of specifying the Host Failures is that when you add hosts to the cluster the percentage of resources saved for HA restarts is automatically calculated again and applied to the cluster, where in the past customers would need to manually calculate what the new percentage would be and configure this

 

Host failure cluster tolerates :

The Admission Control algorithm that has been around the longest is the slot size algorithm, formerly known as the “Host Failures Cluster Tolerates” policy. It is also historically the least understood Admission Control Policy due to its complex admission control mechanism.

 

Failover Hosts :

The third option one could choose is to select one or multiple designated Failover hosts. This is commonly referred to as a hot standby.It is “what you see is what you get”. When you designate hosts as failover hosts, they will not participate in DRS and you will not be able to run VMs on these hosts! Not even in a two-host cluster when placing one of the two in maintenance. These hosts are literally reserved for failover situations. HA will attempt to use these hosts first to failover the VMs. If, for whatever reason, this is unsuccessful, it will attempt a failover on

 

Slot is logical representation of memory and cpu

1st – Slot size is calculated

2nd – HA determines how many slots each host inthe cluster can hold

3rd – calculate current failover capacity in of the cluster

4th – finaly it comapres this to the configured failover capacity by user

 

If the current failover capacity is lessthen configured addmision control will not allow furhet VM powered on

because suffecient slots wont exits incase of host failure

 

DRS –

DRS deep dive ? Explain DRS process how it do resource balance ?

http://www.yellow-bricks.com/drs-deepdive/

 

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s