Mastering VMware® Infrastructure3

The goal of this chapter is to arm you with the most critical tools required for designing, managing, and troubleshooting a virtual infrastructure. Fluency in storage management, virtual machine provisioning, security, and backup are pointless if virtual machines cannot talk to the rest of the network. Server consolidation, simplified management, and greater return on investment are wasted efforts if production systems are not available. In this chapter you will learn to:

♦ Identify the components of virtual networking

♦ Create virtual switches and virtual switch port groups

♦ Create and manage NIC teams

♦ Create and manage virtual LANs (vLANs)

♦ Configure virtual switch security policies

Virtual Networking Components

When it comes to constructing the virtual networking infrastructure of your ESX Server hosts, you will notice some similar components and some not-so-similar components. The following list defines the various components involved in a virtual network architecture:

Virtual switch A switch that resides in the VMkernel and provides traffic management for virtual machines.

Port/port group A logical object on a virtual switch that provides specialized services for the Service Console, VMkernel, or hosted virtual machines. A virtual switch can contain a Service Console port, a VMkernel port, or a virtual machine port group.

Service Console port A specialized virtual switch port type that is configured with an IP address to allow access to the Service Console at the respective address. A Service Console port is also referred to as a vswif.

VMkernel port A specialized virtual switch port type that is configured with an IP address to allow VMotion, iSCSI storage access, or NAS/NFS storage access. A VMkernel port is also referred to as a vmknic.

Virtual Machine port group A specialized virtual switch port that is representative of a switch-to-switch connection and that allows virtual machines to access physical networks.

Virtual LAN (vLAN) A logical LAN configured on a virtual or physical switch that provides efficient traffic segmentation, security, and efficient bandwidth utilization by providing traffic only to the ports configured for a respective vLAN.

Trunk port (trunking) A trunk port on a switch is a port that listens for and knows how to pass traffic for all vLANs configured on the switch.

NIC team The aggregation of physical ports to form a single logical communication channel.

vmxnet adapter A virtualized network adapter operating inside a guest operating system. The vmxnet adapter is a high-performance virtual network adapter that operates only if VMware Tools have been installed. The vmxnet adapter is identified as "flexible" in the virtual machine properties.

vlance adapter A virtualized network adapter operating inside a guest operating system. The vlance adapter is the default adapter used until the VMware Tools installation has been completed.

e1000 adapter A virtualized network adapter that emulates the Intel e1000 network adapter. The e1000 network adapter is most common in 64-bit virtual machines.

Figure 3.1 Successful virtual networking is a blend of virtual and physical network adapters and switches.

The networking architecture of ESX revolves around the creation and configuration of virtual switches. Virtual switches are created and managed through the Service Console, but they operate within the VMkernel. Virtual switches provide the connectivity to provide communication:

♦ between virtual machines within an ESX Server host

♦ between virtual machines on different ESX Server hosts

♦ between virtual machines and physical machines on the network

♦ for Service Console access

♦ for VMkernel access to networks for VMotion, iSCSI, or NFS

Figure 3.1 details the various communication channels provided by virtual network adapters through virtual switches created in the VMkernel. The VMkernel then manages the virtual switch communication through a physical network adapter to connect the virtual and physical networking components.

As the virtual network implementation makes virtual machines accessible, it is essential that virtual switches be configured in a manner that supports reliable and efficient communication around the different network infrastructure components.

Creating Virtual Switches and Port Groups

The answers to the following questions are an integral part of the design of your virtual networking:

♦ Do you have a dedicated network for Service Console management?

♦ Do you have a dedicated network for VMotion traffic?

♦ Do you have an IP storage network? iSCSI? NAS/NFS?

♦ How many NICs are standard in your ESX Server host design?

♦ Is the existing physical network comprised of vLANs?

♦ Do you want to extend the use of vLANs into the virtual switches?

As a precursor to the setup of a virtual networking architecture, the physical network components and the security needs of the network will need to be identified and documented.

Virtual switches in ESX Server are constructed and operated in the VMkernel. Virtual switches (also known as vSwitches) are not managed switches and do not provide all the advanced features that many new physical switches provide. These vSwitches operate like a physical switch in some ways, but in other ways they are quite different. Like their physical counterparts, vSwitches operate at Layer 2, support vLAN configurations, prevent overdelivery, forward frames to other switch ports, and maintain MAC address tables. Despite the similarities to physical switches, vSwitches do have some differences. A vSwitch created in the VMkernel cannot be connected to another vSwitch, thereby eliminating a potential loop configuration and the need to offer support for Spanning Tree Protocol (STP). In physical switches, STP offers redundancy for paths and prevents loops in the network topology by locking redundant paths in a standby state. Only when a path is no longer available will STP activate the standby path.

ESX Server allows for the configuration of three types of virtual switches. The type of switch is dependent on the association of a physical network adapter with the virtual switch. The three types of vSwitches, as shown in Figure 3.2 and Figure 3.3, include:

♦ Internal-only virtual switch

♦ Virtual switch bound to a single network adapter

♦ Virtual switch bound to two or more network adapters

Figure 3.2 Virtual switches provide the cornerstone of virtual machine, VMkernel, and Service Console communication.

Figure 3.3 Virtual switches created on an ESX Server host manage all the different forms of communication required in a virtual infrastructure.

Without an uplink, the internal-only vSwitch allows communication only between virtual machines that exist on the same ESX Server host. Virtual machines that communicate through an internal-only vSwitch do not pass any traffic through a physical adapter on the ESX Server host. As shown in Figure 3.4, communication between virtual machines connected to an internal-only switch takes place entirely in software and happens at whatever speed the VMkernel can perform the task.

Figure 3.4 Virtual machines communicating through an internal-only vSwitch do not pass any traffic through a physical adapter.

For virtual machines to communicate with resources beyond the virtual machines hosted on the local ESX server, a vSwitch must be configured to use a physical network adapter, or uplink. A vSwitch bound to a single physical adapter allows virtual machines to establish communication with physical servers on the network, or with virtual machines on other ESX Server hosts that are also connected to a vSwitch bound to a physical adapter. The vSwitch associated with a physical network adapter provides virtual machines with the amount of bandwidth the physical adapter is configured to support. For example, a vSwitch bound to a network adapter with a 1Gbps maximum speed will provide up to 1Gbps worth of bandwidth for the virtual machines connected to it. Figure 3.5 displays the communication path for virtual machines connected to a vSwitch bound to a single network adapter. In the diagram, when VM01 on silo104 needs to communicate with VM02 on silo105, the traffic from the virtual machine is passed through the ProductionLAN virtual switch (port group) in the VMkernel on silo104 to the physical network adapter to which the virtual switch is bound. From the physical network adapter, the traffic will reach the physical switch (PhySw1). The physical switch (PhySw1) passes the traffic to the second physical switch (PhySw2), which will pass the traffic through the physical network adapter associated with the Production-LAN virtual switch on silo105. In the last stage of the communication, the virtual switch will pass the traffic to the destination virtual machine VM02.

Figure 3.5 The vSwitch with a single network adapter allows virtual machines to communicate with physical servers and other virtual machines on the network.

The last type of virtual switch is referred to as a NIC team. As shown in Figure 3.6 and Figure 3.7, a NIC team involves the association of multiple physical network adapters with a single vSwitch. A vSwitch configured as a NIC team can consist of a maximum of 32 uplinks. In other words, a single vSwitch can use up to 32 physical network adapters to send and receive traffic from the physical switches. NIC teams offer the advantage of redundancy and load distribution. Later in this chapter, we will dig deeper into the configuration and workings of the NIC team.

Figure 3.6 A vSwitch with a NIC team has multiple available adapters for data transfer. A NIC team offers redundancy and load distribution.

Figure 3.7 Virtual switches with a NIC team are identified by the multiple physical network adapters assigned to the vSwitch.

The vSwitch, as introduced in the first section of the chapter, allows several different types of communication, including communication to and from the Service Console, to and from the VMkernel, and between virtual machines. The type of communication provided by a vSwitch is dependent on the port (group), or connection type that is created on the switch. ESX Server hosts can have a maximum of 512 port groups, while the maximum number of ports (port groups) across all virtual switches is 4096.

During the virtual network design I am often asked why virtual switches should not be created with the largest number of ports to leave room to grow. To answer this question let's look at some calculations against the network maximums of an ESX Server 3.5 host.

The maximum number of ports in a virtual switch is 1016. The maximum number of ports across all switches on a host is 4096. This means that if virtual switches are created with the 1016 port maximum only 4 virtual switches can be created. If you're doing a quick calculation of 1016 x 4 and realizing it is not 4096, don't forget that virtual switches actually have 8 reserved ports. Therefore, the 1016 port switch actually has 1,024 ports. Calculate 1,024 x 4 and you will arrive at the 4096 port maximum for an ESX Server 3.5 host.

Create virtual switches with a number of ports to meet your goals. If you can anticipate growth it will save you from a seemingly needless reboot in the future should you have to alter the virtual switch, but if it comes to it, that is why we are thankful for VMotion. Virtual machines can be moved to another host in order to satisfy the rebooting needs of tasks like editing the number of ports on a virtual switch.

Port groups operate as a boundary for communication and/or security policy configuration. Each port group includes functionality for a specific type of traffic but can also be used to provide more or less security to the traffic passing through the respective port group. There are three different connection types or port (groups), shown in Figure 3.8 and Figure 3.9, that can be configured on a vSwitch:

♦ Service Console port

♦ VMkernel port

♦ Virtual Machine port group

A Service Console port on a vSwitch, shown in Figure 3.10 and Figure 3.11, acts as a passage into the management and monitoring capabilities of the console operating system. The Service Console port, also called a vswif, requires that an IP address be assigned. The vSwitch with a Service Console port must be bound to the physical network adapter connected to the physical switch on the network from which management tasks will be performed. In Chapter 2, we covered how the ESX Server installer creates the first vSwitch with a Service Console port to allow postinstallation access.

The console operating system (COS), or Service Console, includes a firewall that, by default, blocks all incoming and outgoing traffic except that required for basic server management. In Chapter 12 we will detail how to manage the firewall.

Figure 3.8 Virtual switches can contain three different connection types: Service Console, VMkernel, and virtual machine.

Figure 3.9 Virtual switches can be created with all three connection types on the same switch.

Figure 3.10 The Service Console port type on a vSwitch is assigned an IP address that can be used for access to the console operating system.

Figure 3.11 The Service Console port, known as a vswif, provides access to the console operating system.

A second Service Console connection provides redundancy in the form of a multihomed console operating system. This is not the same as a NIC team since this configuration will actually provide Service Console access on two different IP addresses. Perform the following steps to create a vSwitch with a Service Console connection using the VI Client:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then choose Networking from the Hardware menu list.

3. Click Add Networking to start the Add Network Wizard.

4. Select the Service Console radio button and click Next.

5. Select the checkbox that corresponds to the network adapter to be assigned to the vSwitch for Service Console communication, as shown in Figure 3.12.

Figure 3.12 Adding a second vSwitch with a Service Console port creates a multi-homed Service Console with multiple entry points.

6. Type a name for the port in the Network Label text box.

7. Enter an IP address for the Service Console port. Ensure the IP address is a valid IP address for the network to which the physical NIC from step 5 is connected. You do not need a default gateway for the new Service Console port if a functioning gateway has already been assigned on the Service Console port created during the ESX Server installation process.

8. Click Next to review the configuration summary and then click Finish.

Perform the following steps to create a vSwitch with a Service Console port using the command line:

1. Use putty.exe or a console session to log in to an ESX Server and establish root-level permissions. Use su - to elevate to root or log in as root if permitted.

2. Use the following command to create a vSwitch named vSwitch:

esxcfg-vswitch -avSwitchX

3. Use the following command to create a port group named SCX to a vSwitch named vSwitchX:

esxcfg-vswitch -A SCX vSwitchX

4. Use the following command to add a Service Console NIC named vswif99 with an IP address of 172.30.0.204 and a subnet mask of 255.255.255.0 to the SCX port group created in step 3:

esxcfg-vswif --add --ip=172.30.0.204 --netmask=255.255.255.0 --portgroup=SCXvswif99

5. Use the following command to assign the physical adapter vmnic3 to the new vSwitch:

esxcfg-vswitch -L vmnic3 vSwitchX

6. Use the following command to restart the VMware management service:

service mgmt-vmware restart

The VMkernel port, shown in Figure 3.13 and Figure 3.14, is used for VMotion, iSCSI, and NAS/NFS access. Like the Service Console port, the VMkernel port requires the assignment of an IP address and subnet mask. The IP addresses assigned to VMkernel ports are needed to support the source-to-destination type IP traffic of VMotion, iSCSI, and NAS. Unlike with the Service Console, there is no need for administrative access to the IP addresses assigned to the VMkernel. In later chapters we will detail the iSCSI and NAS/NFS configurations, as well as the details of the VMotion process. These discussions will provide insight into the traffic flow between VMkernel and storage devices (iSCSI/NFS) or other VMkernels (for VMotion).

Figure 3.13 A VMkernel port created on a vSwitch is assigned an IP address that can be used for accessing iSCSI or NFS storage devices or for performing VMotion with another ESX Server host.

Figure 3.14 A VMkernel port is assigned an IP address and a port label. The label should identify the use of the VMkernel port.

Perform these steps to add a VMkernel port using the VI Client:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then choose Networking from the Hardware menu list.

3. Click Properties for the virtual switch to host the new VMkernel port.

4. Click the Add button, select the VMkernel radio button option, and click Next.

5. Type the name of the port in the Network Label text box.

6. Select Use This Port Group for VMotion if this VMkernel port will host VMotion traffic; otherwise, leave the checkbox unselected.

7. Enter an IP address for the VMkernel port. Ensure the IP address is a valid IP address for the network to which the physical NIC is connected. You do not need to provide a default gateway if the VMkernel does not need to reach remote subnets.

8. Click Next to review the configuration summary and then click Finish.

Follow these steps to create a vSwitch with a VMkernel port using the command line:

1. Use the following command to add a port group named VMkernel to a virtual switch named vSwitch0:

esxcfg-vswitch -A VMkernel vSwitch0

2. Use the following command to assign an IP address and subnet mask to the VMkernel port group:

esxcfg-vmknic -a -i 172.30.0.114 -n 255.255.255.0 VMkernel

3. Use the following command to assign a default gateway of 172.30.0.1 to the VMkernel port group:

esxcfg-route 172.30.0.1

4. Use the following command to restart the VMware management service:

service mgmt-vmware restart

The last connection type (or port group) to discuss is the Virtual Machine port group. The Virtual Machine port group is much different than the Service Console or VMkernel. Whereas both of the other port types require IP addresses for the source-to-destination communication that they are involved in, the Virtual Machine port group does not require an IP address. For a moment, forget about vSwitches and consider standard physical switches. Let's look at an example, shown in Figure 3.15, with a standard 16-port physical switch that has 16 computers connected and configured as part of the 192.168.250.0/24 IP network. The IP network provides for 254 valid IP addresses, but only 16 IP addresses are being used because the switch only has 16 ports. To increase the number of hosts that can communicate on the same logical IP subnet, a second switch could be introduced. To accommodate the second switch, one of the existing computers would have to be unplugged from the first switch. The open port on the first switch could then be connected to a second 16-port physical switch. Once the unplugged computer is reconnected to a port on the new switch, all 16 computers will once again communicate. In addition, there will be 15 open ports on the second switch to which new computers could be added.

A vSwitch created with a virtual machine port group bound to a physical network adapter that is connected to a physical switch acts just as the second switch did in the previous example. As in the physical switch to physical switch example, an IP address does not need to be configured for a virtual machine port group to combine the ports of a vSwitch with those of a physical switch. Figure 3.16 shows the switch-to-switch connection between a vSwitch and a physical switch.

Figure 3.15 Two physical switches can be connected to increase the number of available hosts on the IP network.

Figure 3.16 A vSwitch with a virtual machine port group uses an associated physical network adapter to establish a switch-to-switch connection with a physical switch.

Perform the following steps to create a vSwitch with a virtual machine port group using the VI Client:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click Add Networking to start the Add Network Wizard.

4. Select the Virtual Machine radio button option and click Next.

5. Select the checkbox that corresponds to the network adapter to be assigned to the vSwitch. Select the NIC connected to the switch where production traffic will take place.

6. Type the name of the virtual machine port group in the Network Label text box.

7. Click Next to review the virtual switch configuration and then click Finish.

Follow these steps to create a vSwitch with a virtual machine port group using the command line:

1. Use the following command to add a virtual switch named vSwitch1:

esxcfg-vswitch -a vSwitch1

2. Use the following command to assign vSwitch1 to vmnic1:

esxcfg-vswitch -L vmnic1 vSwitch1

3. Use the following command to create a virtual machine port group named ProductionLAN on vSwitch1:

esxcfg-vswitch -A ProductionLAN vSwitch1

4. Use the following command to restart the VMware management service:

service mgmt-vmware restart

The creation of vSwitches and port groups and the relationship between vSwitches and uplinks is dependent on several factors, including the number of network adapters in the ESX Server host, the number of IP subnets to connect to, the existence of vLANs, and the number of physical networks to connect to. With respect to the configuration of the vSwitches and virtual machine port groups, there is no single correct configuration that will satisfy every scenario. It is true, however, to say that the greater the number of physical network adapters in an ESX Server host, the more flexibility you will have in your virtual networking architecture.

Later in the chapter we will discuss some advanced design factors, but for now let's stick with some basic design considerations. If the vSwitches created in the VMkernel are not going to be configured with multiple port groups or vLANs, you will be required to create a separate vSwitch for every IP subnet that you need to connect to. In Figure 3.17, there are five IP subnets that our virtual infrastructure components need to reach. The virtual machines in the production environment must reach the production LAN, the virtual machines in the test environment must reach the test LAN, the VMkernel needs to access the IP storage and VMotion LANs, and finally the Service Console must be on the management LAN. Notice that the physical network is configured with vLANs for the test and production LANs. They share the same physical network but are still separate IP subnets. Figure 3.17 displays a virtual network architecture that does not include the use of vLANs in the vSwitches and as a result requires one vSwitch for every IP subnet; five IP subnets means five vSwitches. In this case, each vSwitch consists of a specific connection type.

In this example, each connection type could be split because there were enough physical network adapters to meet our needs. Let's look at another example where an ESX Server host has only two network adapters to work with. Figure 3.18 shows a network environment with five IP subnets: management, production, test, IP storage, and VMotion, where the production, test, and IP storage networks are configured as vLANs on the same physical network. Figure 3.18 displays a virtual network architecture that includes the use of vLANs and that combines multiple connection types into a single vSwitch.

Figure 3.17 Without the use of port groups and vLANs in the vSwitches, each IP subnet will require a separate vSwitch with the appropriate connection type.

Figure 3.18 With a limited number of physical network adapters available in an ESX Server host, vSwitches will need multiple connection types to support the network architecture.

The vSwitch and connection type architecture of ESX Server, though robust and customizable, is subject to all of the following limits:

♦ An ESX Server host cannot have more than 4,096 ports.

♦ An ESX Server host cannot have more than 1,016 ports per vSwitch.

♦ An ESX Server host cannot have more than 127 vSwitches.

♦ An ESX Server host cannot have more than 512 virtual switch port groups.

By default, all virtual network adapters connected to a vSwitch have access to the full amount of bandwidth on the physical network adapter with which the vSwitch is associated. In other words, if a vSwitch is assigned a 1Gbps network adapter, then each virtual machine configured to use the vSwitch has access to 1Gbps of bandwidth. Naturally, if contention becomes a bottleneck hindering virtual machine performance, a NIC team would be the best option. However, as a complement to the introduction of a NIC team, it is also possible to enable and to configure traffic shaping. Traffic shaping involves the establishment of hard-coded limits for a peak bandwidth, average bandwidth, and burst size to reduce a virtual machine's outbound bandwidth capability.

As shown in Figure 3.19, the peak bandwidth value and the average bandwidth value are specified in Kbps, and the burst size is configured in units of KB. The value entered for the average bandwidth dictates the data transfer per second across the virtual vSwitch. The peak bandwidth value identifies the maximum amount of bandwidth a vSwitch can pass without dropping packets. Finally, the burst size defines the maximum amount of data included in a burst. The burst size is a calculation of bandwidth multiplied by time. During periods of high utilization, if a burst exceeds the configured value packets will be dropped in favor of other traffic; however, if the queue for network traffic processing is not full, the packets will be retained for transmission at a later time.

Perform the following steps to configure traffic shaping:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch, select the name of the virtual switch or port group from the Configuration list, and then click the Edit button.

4. Select the Traffic Shaping tab.

5. Select the Enabled option from the Status drop-down list.

6. Adjust the Average Bandwidth value to the desired number of Kbps.

7. Adjust the Peak Bandwidth value to the desired number of Kbps.

8. Adjust the Burst Size value to the desired number of KB.

Figure 3.19 Traffic shaping reduces the outbound bandwidth available to a port group.

With all the flexibility provided by the different virtual networking components, you can be assured that whatever the physical network configuration may hold in store, there will be several ways to integrate the virtual networking. What you configure today may change as the infrastructure changes or as the hardware changes. Ultimately the tools provided by ESX Server are enough to ensure a successful communication scheme between the virtual and physical networks.

Creating and Managing NIC Teams

In the previous section, we looked at some good examples of virtual network architectures needed to support the physical networking components. Now that you have some design and configuration basics under your belt, let's move on to extending the virtual networking beyond just establishing communication. A NIC team can support any of the connection types discussed in the previous section. Using NIC teams provides redundancy and load balancing of network communications to Service Console, VMkernel, and virtual machines.

A NIC team, shown in Figure 3.20 and Figure 3.21, is defined as a vSwitch configured with an association to multiple physical network adapters (uplinks). As mentioned in the previous section, the ESX Server host can either have a maximum of 32 uplinks spread across multiple vSwitches or be configured as a NIC team on one vSwitch.

Successful NIC teaming requires that all uplinks be connected to physical switches that belong to the same broadcast domain. As shown in Figure 3.22, all of the physical network adapters in the NIC team should be connected to the same physical switch or to physical network adapters connected to physical switches that are connected to one another.

Figure 3.20 Virtual switches, like vSwitch1, with multiple uplinks offer redundancy and load balancing.

Figure 3.21 A NIC team is identified by the association of multiple physical network adapters assigned to a vSwitch.

Figure 3.22 All of the physical network adapters that make up a NIC team must belong to same Layer 2 broadcast domain.

Constructing NIC Teams

NIC teams should be built on physical network adapters located on separate bus architectures. For example, if an ESX Server host contains two on-board network adapters and a PCI-based quad-port network adapter, a NIC team should be constructed using one on-board network adapter and one network adapter on the PCI bus. This design eliminates a single point of failure.

Perform the following steps to create a NIC team using the VI Client:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch that will be assigned a NIC team and select the Network Adapters tab.

4. Click Add and select the appropriate adapter from the Unclaimed Adapters list, as shown in Figure 3.23.

Figure 3.23 Create a NIC team using unclaimed network adapters that belong to the same Layer 2 broadcast domain as the original adapter.

5. Adjust the Policy Failover Order as needed to support an Active/Standby configuration.

6. Review the summary of the virtual switch configuration, click Next, and then click Finish.

The load-balancing feature of NIC teaming does not function like the load-balancing feature of advanced routing protocols. Load balancing across a NIC team is not a product of identifying the amount of traffic transmitted through a network adapter and shifting traffic to equalize data flow through all available adapters. The load-balancing algorithm for NIC teams in a vSwitch is a balance of the number of connections — not the amount of traffic. NIC teams on a VI vSwitch can be configured with one of the following three load-balancing policies:

♦ vSwitch port-based load balancing (default)

♦ Source MAC-based load balancing

♦ IP hash-based load balancing

Outbound Load Balancing

The load-balancing feature of NIC teams on a vSwitch only applies to the outbound traffic.

Virtual Switch Port Load Balancing

The vSwitch port-based load-balancing policy that is used by default uses an algorithm that ties each virtual switch port to a specific uplink associated with the vSwitch. The algorithm will maintain an equal number of port-to-uplink assignments across all uplinks to achieve load balancing. As shown in Figure 3.24, this policy setting ensures that traffic from a specific virtual network adapter connected to a virtual switch port will consistently use the same physical network adapter. In the event that one of the uplinks fails, the traffic from the failed uplink will failover to another physical network adapter.

Figure 3.24 The vSwitch port-based load-balancing policy assigns each virtual switch port to a specific uplink. Failover to another uplink occurs when one of the physical network adapters experiences failure.

You can see how this policy does not provide load balancing of the amount of traffic because each virtual machine can access only one physical network adapter at any given time. Since the port to which a virtual machine is connected does not change, each virtual machine is tied to a physical network adapter until failover occurs. Looking at Figure 3.24, imagine that the Linux virtual machine and the Windows virtual machine on the far left and far right are the two most network-intensive virtual machines. In this case, the vSwitch port-based policy has assigned both of the ports used by these virtual machines to the same physical network adapter. Meanwhile, the Linux and Windows virtual machines in the middle, which both might be processing very little traffic, are connected to ports assigned to individual physical network adapters.

The physical switch passing the traffic learns the port association and therefore sends replies back through the same physical network adapter from which the request initiated. The vSwitch port-based policy is best used when the number of virtual network adapters is greater than the number of physical network adapters. In the case where there are fewer virtual network adapters than physical adapters, some physical adapters will not be used. For example, if five virtual machines are connected to a vSwitch with six uplinks, only five used vSwitch ports will be assigned to exactly five uplinks, leaving one uplink with no traffic to process.

IP Hash Load Balancing

The third load-balancing policy available for NIC teams is the IP hash-based policy, also called the out-IP policy. This policy, shown in Figure 3.26, addresses the limitation of the other two policies that prevents a virtual machine from accessing two physical network adapters without having two virtual network adapters. The IP hash-based policy uses the source and destination IP addresses to determine the physical network adapter for communication. This algorithm then allows a single virtual machine to communicate over different physical network adapters when communicating with different destinations.

Figure 3.25 The source MAC-based load-balancing policy, as the name suggests, ties a virtual network adapter to a physical network adapter based on the MAC address.

Balancing for Large Data Transfers

Although the IP hash-based load-balancing policy can more evenly spread the transfer traffic for a single virtual machine, it does not provide a benefit for large data transfers occurring between the same source and destination systems. Since the source-destination hash will be the same for the duration of the data load, it will only flow through a single physical network adapter.

A vSwitch with a NIC team set to use the IP hash-based load-balancing policy should have all physical network adapters connected to the same physical switch to support link aggregation. ESX Server supports standard 802.3ad teaming in static (manual) mode, but does not support the Link Aggregation Control Protocol (LACP) or Port Aggregation Protocol (PAgP) commonly found on switch devices. Link aggregation will increase throughput by combining the bandwidth of multiple physical network adapters for use by a single virtual network adapter of a virtual machine.

Follow these steps to alter the load-balancing policy of a vSwitch with a NIC team:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch, select the name of virtual switch from the Configuration list, and then click the Edit button.

4. Select the NIC Teaming tab and then select the desired load-balancing strategy from the Load Balancing drop-down list.

5. Click OK and then click Close.

Figure 3.26 The IP hash-based policy is a more scalable load-balancing policy that allows virtual machines to use more than one physical network adapter when communicating with multiple destination hosts.

Now that the load-balancing policies are explained, let's take a deeper look at the failover and failback of uplinks in a NIC team. Failover detection in a NIC team can be configured to use either a link status method or a link status and beacon probing method.

The link status failover detection method works just as the name suggests. Failure of an uplink is identified by the link status provided by the physical network adapter. In this case, failure is identified for events like removed cables or power failures on a physical switch. The downside to the link status failover detection setting is its inability to identify misconfigurations or pulled cables that connect the switch to other networking devices (i.e., a cable connecting one switch to another switch.)

The beacon probing failover detection setting, which includes link status as well, sends Ethernet broadcast frames across all physical network adapters in the NIC team. These broadcast frames allow the vSwitch to detect upstream network connection failures and will force failover when ports are blocked by a spanning tree, are configured with the wrong vLAN, or a switch-to-switch connection has failed. When a beacon is not returned on a physical network adapter, the vSwitch triggers the failover notice and reroutes the traffic from the failed network adapter through another available network adapter based on the failover policy. Consider a vSwitch with a NIC team consisting of four physical network adapters, where each adapter is connected to a different physical switch and each physical switch is connected to a single physical switch, which is then connected to a router, as shown in Figure 3.27. When the NIC team is set to the beacon probing failover detection method, a beacon will be sent out over all four uplinks.

Figure 3.27 The beacon probing failover detection policy sends beacons out across the physical network adapters of a NIC team to identify upstream network failures or switch misconfigurations.

Once a failure has been detected, the vSwitch will use either a Rolling Failover or a Failover Order policy setting to continue passing traffic through another physical network adapter. By default, a NIC team is set to use a Rolling Failover policy of No, as shown in Figure 3.28. When the Rolling Failover policy is set to No, the NIC team not only provides a failover policy but also applies a fail-back policy. This means that if a failed physical network adapter is repaired and brought back online, the NIC team will reroute the traffic back to the adapter and relieve the standby adapter that assumed its role during failover. When the Rolling Failover policy is set to Yes, the NIC team will not perform a failback. In this case, the physical network adapter that assumed responsibility for the failed adapter will continue to process the traffic. Meanwhile, the failed adapter that is not back online will function as a standby until another adapter experiences failure.

Figure 3.28 By default, a NIC team is configured with a Rolling Failover policy setting of No.

As an alternative to the Rolling Failover policy, a Failover Order policy can be manually configured. The Failover Order policy involves an administrative assignment of priority to each of the physical network adapters in the NIC team. As shown in Figure 3.29, a vSwitch with a NIC team set to use a Failover Order policy can have physical network adapters designated as active and standby.

Perform the following steps to configure the Failover Order policy for a NIC team:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties for the virtual switch, select the name of virtual switch from the Configuration list, and then click the Edit button.

4. Select the NIC Teaming tab.

5. Use the Move Up and Move Down buttons to adjust the order of the network adapters and their location within the Active Adapters, Standby Adapters, and Unused Adapters lists, as shown in Figure 3.30.

6. Click OK and then click Close.

Figure 3.29 The Failover Order policy allows for the designation of active, standby, and unused network adapters for a NIC team.

Figure 3.30 Failover order for a NIC team is determined by the order of network adapters as listed in the Active Adapters, Standby Adapters, and Unused Adapters lists.

When a failover event occurs on a vSwitch with a NIC team, the vSwitch is obviously aware of the event. The physical switch that the vSwitch is connected to, however, will not know immediately. As shown in Figure 3.31, a NIC Team includes a Notify Switches configuration setting which, when set to Yes, will allow the physical switch to immediately learn of any of the following changes:

♦ A virtual machine is powered on (or any other time a client registers itself with the vSwitch)

♦ A VMotion occurs

♦ A MAC address is changed

♦ A NIC team failover or failback has occurred

Figure 3.31 The Notify Switches option allows physical switches to be notified of changes in NIC teaming configurations.

In any of these events, the physical switch is notified of the change using the Reverse Address Resolution Protocol (RARP). RARP updates the lookup tables on the physical switches and offers the shortest latency when a failover event occurs.

Although the VMkernel works proactively to keep traffic flowing from the virtual networking components to the physical networking components, VMware recommends taking the following actions to minimize networking delays:

♦ Disable Port Aggregation Protocol (PAgP) and Link Aggregation Control Protocol (LACP) on the physical switches

♦ Disable Dynamic Trunking Protocol (DTP) or trunk negotiation

♦ Disable Spanning Tree Protocol (STP)

Virtual Switches with Cisco Switches

VMware recommends configuring Cisco devices to use PortFast mode for access interfaces or PortFast trunk mode for trunking interfaces.

Creating and Managing VLANs

To vLAN or not to vLAN? That is the question. As defined in the first section, a virtual LAN (vLAN) is a logical LAN configured on a virtual or physical switch port that provides efficient traffic segmentation, security, and efficient bandwidth utilization by providing traffic only to the ports configured for a respective vLAN. In addition to the security and segmentation advantages, vLANs allow network administrators to exceed the physical distance limitations of standard cabling. Using vLANs is advantageous when an ESX Server host has a limited number of physical network adapters.

Figure 3.32 shows a typical vLAN configuration across physical switches.

Figure 3.32 Virtual LANs provide secure traffic segmentation without the cost of additional hardware.

Blade servers provide an excellent example of when vLANs offer tremendous benefit, because the blade servers offer limited expansion slots for physical network adapters due to the small form factor of the blade casing. Figure 3.33 shows a vSwitch architecture with vLANs as it integrates with a physical architecture also using vLANs. For a vSwitch to successfully send and receive packets tagged as one vLAN or another, a trunk port must be configured on the physical switch port to which the physical network adapter assigned to the vSwitch is connected.

Figure 3.33 The physical switch port to which a vSwitch's assigned physical network adapter is connected must be configured as a trunk port for vLAN tagging to work between virtual and physical switches.

Follow these steps to configure a vSwitch with a virtual machine port group with a vLAN using an ID of VLAN 117:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties link for the vSwitch where the new vLAN should be created.

4. Click the Add button, select the Virtual Machine radio button option, and then click Next.

5. Type the name of the virtual machine port group in the Network Label text box. In this case, vLAN117 would be appropriate.

6. Type 117 in the VLAN ID (Optional) text box, as shown in Figure 3.34.

Figure 3.34 The vLAN tagging support of vSwitches simplifies integration with existing physical hardware configured with vLANs.

7. Click Next to review the vSwitch configuration and then click Finish.

Although vLANs reduce the costs of constructing multiple logical subnets, keep in mind that the contention through physical switches and network adapters is still present. For bandwidth-intensive network operations, the disadvantage of the shared physical network might outweigh the scalability and cost savings of the vLAN.

Configuring Virtual Switch Security

Even though the vSwitches created in the VMkernel are considered to be “dumb switches”, they can be configured with vSwitch security policies to enhance or ensure Layer 2 security. Security policies can be applied at the vSwitch or at the lower-level connection types configured on a vSwitch and include the following three security options:

♦ Promiscuous Mode

♦ MAC Address Changes

♦ Forged Transmits

Applying a security policy to the vSwitch is effective, by default, for all connection types within the switch. However, if a connection type, or port group, is configured with a competing security policy, it will override the policy set at the vSwitch. As in the example in Figure 3.35, if a vSwitch is configured with a security policy that rejects the use of MAC address changes but a virtual machine port group on the switch is configured to accept MAC address changes, then any virtual machines connected to that port group will be allowed to communicate even though it is using a MAC address that differs from what is configured in its VMX file.

Figure 3.35 Security policies at the switch level are effective by default for all connection types on the switch. Security policies at the connection type (port group) level override the policy set at the virtual switch.

The default security profile for a vSwitch, shown in Figure 3.36, is set to reject Promiscuous mode and to accept MAC address changes and Forged transmits.

Promiscuous Mode

The Promiscuous Mode option is set to Reject by default to prevent virtual network adapters from observing any of the traffic submitted through the vSwitch. For enhanced security, allowing Promiscuous mode is not recommended because it is an insecure mode of operation that allows virtual adapters to access traffic other than its own. Despite the security concerns, there are valid reasons for permitting a switch to operate in Promiscuous mode. An intrusion detection system (IDS) requires the ability to identify all traffic to scan for anomalies and malicious patterns of traffic. To support the use of the IDS without overextending the reduced security of Promiscuous mode, you can create a dedicated virtual machine port group for use with the IDS. As shown in Figure 3.37, the virtual switch security policy will remain at the default setting of Reject for the Promiscuous Mode option, while the virtual machine port group for the IDS will be set to Accept. This setting will override the virtual switch, allowing the IDS to monitor all switch traffic.

Figure 3.36 The default security profile for a virtual switch prevents Promiscuous Mode but allows MAC Address Changes and Forged transmits.

MAC Address Changes and Forged Transmits

When a virtual machine is created with one or more virtual network adapters, a MAC address is generated for each virtual adapter. Just as Intel, Broadcom, and others manufacture network adapters and include unique MAC address strings, VMware is also a network adapter manufacturer that has its own MAC prefix to ensure uniqueness. Of course, VMware doesn't actually manufacture anything, since the product exists as a virtual NIC in a virtual machine. The six-byte, randomly generated MAC addresses for a virtual machine can be seen in the configuration file (.vmx) of the virtual machine, as shown in Figure 3.38. A VMware-assigned MAC address begins with the prefix 00:50:56 or 00:0C:29. The value of the fourth set (XX) cannot exceed 3F to prevent conflicts with other VMware products, while the fifth and sixth sets (YY:ZZ) are generated randomly based on the Universally Unique Identifier (UUID) of the virtual machine that is tied to the location of the virtual machine. For this reason, when a virtual machine location is changed a prompt will appear prior to successful boot. The prompt will inquire about keeping the UUID or generating a new UUID, which helps prevent MAC address conflicts.

Figure 3.37 Promiscuous mode, though a reduction in security, is required when using an intrusion detection system.

Figure 3.38 A virtual machine's initial MAC address is automatically generated and listed in the configuration file for the virtual machine.

Manually Setting a MAC

Manually configuring a MAC address in the configuration file of a virtual machine will not work unless the first three bytes are VMware-provided prefixes and the last three bytes are unique. If a non-VMware MAC prefix is entered in the configuration file, the virtual machine will not power on.

All virtual machines have two MAC addresses: the initial MAC and the effective MAC. The initial MAC address is the MAC discussed in the previous paragraph that is generated automatically and that resides in the configuration file. The guest operating system has no control over the initial MAC address. The effective MAC address is the MAC address configured by the guest operating system that is used during communication with other systems. The effective MAC address is included in network communication as the source MAC of the virtual machine. By default, these two addresses are identical. To force a non-VMware-assigned MAC address to a guest operating system, change the effective MAC address from within the guest operating system, as shown in Figure 3.39.

Figure 3.39 A virtual machine's source MAC address is the effective MAC address, which by default matches the initial MAC address configured in the VMX file. The effective MAC, however, can be changed in the guest operating system.

The ability to alter the effective MAC address cannot be removed from the guest operating system. However, the ability to let the system function with this altered MAC address is easily addressable through the security policy of a vSwitch. The remaining two settings of a virtual switch security policy are MAC Address Changes and Forged Transmits. Both of these security policies are concerned with allowing or denying differences between the initial MAC address in the configuration file and the effective MAC address in the guest operating system. As noted earlier, the default virtual switch security is to accept the differences and process traffic as needed.

The difference between the MAC Address Changes and Forged Transmits security settings involves the direction of the traffic. MAC Address Changes is concerned with the integrity of incoming traffic, while Forged Transmits oversees the integrity of outgoing traffic. If the MAC Address Changes option is set to Reject, traffic will not be passed through the vSwitch to the virtual machine (incoming) if the initial and the effective MAC addresses do not match. If the Forged Transmits option is set to Reject, traffic will not be passed from the virtual machine to the vSwitch (outgoing) if the initial and the effective MAC addresses do not match. Figure 3.40 highlights the security restrictions implemented when MAC Address Changes and Forged Transmits are set to Reject.

Figure 3.40 The MAC Address Changes and Forged Transmits security options deal with incoming and outgoing traffic respectively.

For the highest level of security, VMware recommends setting MAC Address Changes, Forged Transmits, and Promiscuous Mode on each vSwitch to Reject. When warranted or necessary, use port groups to loosen the security for a subset of virtual machines to connect to the port group.

Real World Scenario

Virtual Switch Policies for Microsoft Network Load Balancing

As with anything, there are, of course, exceptions. For virtual machines that will be configured as part of a Microsoft network load balancing (NLB) cluster set in Unicast mode, the virtual machine port group must allow MAC Address Changes and Forged Transmits. Systems that are part of an NLB cluster will share a common IP address and virtual MAC address, as shown here:

The shared virtual MAC address is generated by using an algorithm that includes a static component based on the NLB cluster's configuration of Unicast or Multicast mode plus a hexadecimal representation of the four octets that make up the IP address. This shared MAC address will certainly differ from the MAC address defined in the VMX file of the virtual machine. If the virtual machine port group does not allow for differences between the MAC addresses in the VMX and guest operating system, NLB will not function as expected. VMware recommends running NLB clusters in Multicast mode due to these issues with NLB clusters in Unicast mode.

Perform the following steps to edit the security profile of a vSwitch:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties link for the virtual switch.

4. Click the name of the virtual switch under the Configuration list and then click the Edit button.

5. Click the Security tab and make the necessary adjustments.

6. Click OK and then click Close.

Follow these steps to edit the security profile of a port group:

1. Use the VI Client to establish a connection to a VirtualCenter server or an ESX Server host.

2. Click the hostname in the inventory panel on the left, select the Configuration tab from the details pane on the right, and then select Networking from the Hardware menu list.

3. Click the Properties link for the virtual switch.

4. Click the name of the port group under the Configuration list and then click the Edit button.

5. Click the Security tab and make the necessary adjustments.

6. Click OK and then click Close.

Managing the security of a virtual network architecture is much the same as managing the security for any other portion of your information systems. Security policy should dictate that settings be configured as secure as possible to err on the side of caution. Only with proper authorization, documentation, and change management processes should security be reduced. In addition, the reduction in security should be as controlled as possible to affect the least number of systems if not just the systems requiring the adjustments.

The Bottom Line

Identify the components of virtual networking. Virtual networking is made up of a combination of relationships that exist between the logical networking components created in the VMkernel of ESX Server and the physical network devices. The virtual machines are configured on vSwitches bound to physical network adapters that are connected to physical switches.

Create virtual switches and virtual switch port groups. Virtual switches, ports, and port groups are the cornerstone of the virtual networking architecture. These virtual components provide the tools for connecting to the physical network components to allow communication between the virtual and physical environments.

Master It Virtual machines need to communicate with physical servers on the production network.

Master It Service console communication must occur on a dedicated management network.

Master It A dedicated network has been implemented to support VMotion.

Master It A dedicated storage network has been implemented to support communication to iSCSI and NFS storage devices.

Create and manage NIC teams. NIC teams offer the opportunity for redundancy and load balancing of network traffic. NIC teams offer three load-balancing policies: port-based, source MAC-based, and IP hash-based load balancing.

Master It Virtual machines with one virtual network adapter must be capable of using multiple physical network adapters when connecting to multiple network destinations.

Master It A vSwitch configured with a NIC team needs to experience failback when a physical network adapter is repaired after failover.

Master It Bandwidth available on multiple physical network adapters must be accessible to a single virtual network adapter on a virtual machine.

Master It Discovery time after a failover event on a NIC team needs to be minimized to prevent unnecessary delays.

Create and manage virtual LANs (VLANs). The use of vLANs in a virtual networking architecture offers security, scalability, and communication efficiency.

Master It A vSwitch needs to be configured with two vLANs named VLAN101 and VLAN102.

Master It A vSwitch is configured with vLANs identical to those configured on the physical switch to which it is connected; however, traffic between the two switches is not functioning.

Configure virtual switch security policies. Virtual switch security comes in a tight little package that includes three specific security settings that deal with identifying and processing traffic through a virtual switch. Promiscuous Mode, MAC Address Changes, and Forged Transmits each provides a securable vSwitch architecture, which ensures that only the right systems are sending and receiving traffic as expected.

Master It A virtual machine with an installed intrusion detection system (IDS) needs to "sniff" the traffic passing through a vSwitch but the vSwitch is not configured to allow virtual machines to identify all traffic on the switch. You need to allow the functionality of the IDS while minimizing the security impact on the network.

Master It An administrator of a Windows Server 2003 computer has changed the IP address of the guest operating system from the properties of the network adapter. The administrator now states that the Windows Server 2003 computer cannot communicate with requesting clients. You identify that the virtual machine port group to which the virtual machine is connected does not permit the vSwitch to send traffic when the effective and initial MAC addresses do not match.