Navigating through the VMware's forest of product offerings

Back in 2008 when I first started to learn about VMware's server virtualization technology, it is very confusing on the different products and how they are related together.  There are the ESX, ESXi, vSphere and vCenter server and there are older products such as VMware server or the GSX server.  With Google we can find lots of articles on these products but the problem is that most of these articles do not have a date and since the VMware virtualization technology is advancing in such a high pace, we do not know if the information was current or relevant to which version of vSphere.

Now in 2015, can you tell me all the current VMware products and how they are related?

To look at VMware's product offering, it is best to look at VMware's vision on data center - Software Defined Data Center architecture.

Data Center comprises of these functions:

1. Compute
2. Storage
3. Network
4. Management

VMware’s idea is to provide an abstraction layer for compute, storage and network hardware so that the entire data center is software driven.   

The advantage of a Software Defined Data Center architecture is that the data center can be more agile as the hardware resources are provisioned or withdraw on demand to the compute, storage and network resources.  This agility factor has huge implication to IT operations in a data center:

• More cost effective
• Ability to service user demand much faster
• Provide a stable platform for DevOps
• Eliminate or minimize human error
• Make Disaster Recovery easier thus able to improve RTO and RPO.

The benefits of a Software Defined Data Center architecture is not limited to the above mentioned points and it is not the main point for this article.

VMware vCloud Suite is VMware's solution to provide a "Software Defined Data Center" for customers. As shown in the diagram below vCloud Suite comprises of:

• Compute
• Storage and Availability
• Management and Automation
• Network and Security (add on feature)

Each of these different products under the above mentioned categories can be deployed separately and as a whole they formed the vCloud Suite.

image source: http://www.vmware.com/files/images/thumbnails/vmw-scrnsht-vcloud-suite-mgmt-lg.jpg

Compute
At the heart of the vCloud Suite is the vSphere Suite that provide the virtualization infrastructure for different business establishments from SMB to enterprise to cloud operators. I don't think I need to describe much about the vSphere Suite as this has been around for a number of years and are widely deployed.

Management and Automation

The VMware vRealize™ Suite provides the management and automation capability for the Software Defined Data Center architecture.

Basically, the vRealize Suite comprise of Automation, Operations and Business and Log Insight for monitoring.

image source: http://cloudmaniac.net/wp-content/uploads/2014/10/vmware-vrealize-suite.jpg

Note:

• vRealize Operations is formerly known as vCenter Operations Management Suite or vCops
• vRealize Automation is formerly known as vCloud Automation Center or vCAC
• vRealize Business is formerly known as IT Business Management Suite
• vRealize Log Insight is formerly know as vCenter Log Insight

For a more comprehensive and detailed description of the vRealize Suite, this VMware site is a good place to visit.

VMware vRealize Suite consists of the following products:

• VMware vRealize™ Automation™ Advanced or Enterprise
• VMware vRealize™ Operations™ Advanced or Enterprise
• VMware vRealize™ Log Insight™
• VMware vRealize™ Business™ Standard
• VMware vRealize™ Business™ Advanced or Enterprise

Storage and Availability
For all IT operations, disaster recovery is an important element.  Each business should defined the Recovery Time objective (RTO) and Recovery Point Objective (RPO) for acceptable level of disruption in case of a disaster event affecting the continuous operation of the IT infrastructure. Most importantly DR plans needs to be tested.  VMware vCenter Site Recovery Manager provide customer the ability to automate and orchestrate non-disruptive testing of recovery plans. With automation and orchestration, the RTO and RPO can be improved.

image source: http://ddf912383141a8d7bbe4-e053e711fc85de3290f121ef0f0e3a1f.r87.cf1.rackcdn.com/2.17%20vcenter.png

For more comprehensive and detailed description of VMware vCenter Site Recovery Manager visit here.

I am not sure why VMware group storage and availability into the same category because availability also include the compute and networking pieces that support the application that we are ultimately concern with.

Regarding storage, VMware had been working on abstracting the storage hardware and to provide an easy to configured Software defined Storage products.  In this area, VMware is taking 2 approaches:

1. Virtual Data Plane - Virtual SAN (VSAN) and vSphere Virtual Volume (vVol)
2. Policy Driven Control Plane - Storage Policy Based Management

There are many other blog post talking about these products. This blog post is meant to provide an overview and how these products related to each other.

Network and Security
This is an add-on to to the vCloud Suite.  In this category is the NSX - VMware's way of providing an abstraction layer to the physical network as well as some L4 - L7 network services.

NSX is also a security product as I have talked about in my previous post.

Brad Hedlund (@bradhedlund, Engineering Architect at VMware's Network and Security Business Unit) has an good article on what network virtualization is

 

image source: http://commondatastorage.googleapis.com/bradhedlund/blog/what-is-network-virtualization/What_is_Network_Virtualization.PNG

NSX virtualized the network as well as network functions such as load balancing, firewall and VPN. Any Cloud management platform including OpenStack can interact with NSX via RESTful API to provision network services for the cloud platform.  

NSX has two flavors as I have outlined in this post.  One flavor is tailor toward vSphere product and the other one is for non VMware hypervisors such as KVM.  

One point worth mentioning is that NSX can optimize the VXLAN multicast traffic because the NSX Controller is able to distribute the MAC address and VXLAN ID mapping to the various VTEPs instead of using bi-directional PIM as defined in RFC 7348 (I have a summary of this RFC here).  Multicast traffics takes up network bandwidths and network administrators always try to find ways to eliminate multicast traffic.

Other VMware products

For a complete list of VMware products we can find them here. In this post I have not talked about the End-user computing product of VMware - Horizon nor the VMware hybrid cloud product - VMware Air which is the VMware's offering of public cloud similar to Amazon's AWS, Google's Cloud platform or Microsoft's Azure.

I hope this will clarify VMware's product offerings and how they relate to each other.

A Quick Look at the VXLAN's specification - RFC 7348

VXLAN is being deployed in numerous production environments and is supported by quite a few networking equipment vendor as well as software vendors such as VMware and Red Hat.  Not until August 2014 that the specification is moved from IETF draft status to RFC 7348.

RFC stands for Request For Comments. As a software developer for networking equipment, I have to know the RFC in and out.  This RFC is the functional specification for the feature that I develop.  Test group will test the feature to make sure the feature function as what the RFC describes.

Today, let’s take a look at what is in RFC 7348.

The title of RFC 7348 is “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.”

RFC 7348 is relatively a short document compare to other RFC that specify other communication protocols.   The table of contents has 10 sections.  6 of them are standard sections for all RFC.  To break down this RFC to the core, we can look at the following sections:

• VXLAN Problem Statement
• VXLAN operation
• VXLAN Frame Format
• VXLAN Security Considerations

VXLAN Problem Statement

Basically RFC 7348 identified 3 problem areas in the networking infrastructure within the data center that VXLAN is designed to resolve.  Server virtualization in the data center works best in a flat layer 2 network.  Only until VMworld 2014 that VMware announced in vSphere 6 (as of this writing, it is still under beta) can support vMotion across vCenter and long distance.

Limitations Imposed by Spanning Tree and VLAN Ranges

Spanning Tree is used to safe guard loops in a Layer 2 network.  When there is a loop in a Layer 2 network, it will cause a broadcast storm and the network will be in-operable.  Spanning Tree Protocol is designed and used to protect the Layer 2 network.  While it is able to protect the Layer 2 network the price to pay is there are links in the network is not used as well as multipathing not an option in the network design.  Both of these problems ties back to ROI (Return On Investment) not to a point where it should be.

Another problem is that the VLAN ID is a 12-bit field thus causing data center in flat Layer 2 network limited to only 4094 VLANs.

Multi-tenant Environment

Multi-tenant is a virtualized data center is a definite requirement.  Tenant isolation is an absolute requirement.

In a Layer 2 network VLAN is a popular way to achieve tenant isolation.  The number of VLANs limited to 4094 as described in the previous section is a major stumbling block to data center design.  When each tenant requires to have more than one VLAN as in the example of a 3-tier web application, the number of VLANs available in a flat Layer 2 network become a more limited and yet required resource.

A layer 3 network is another possible way of network isolation for multi-tenant but it has it limitation.  This will limit each tenant to have unique IP subnets.  Also, this Layer 3 isolation solution limits user not able to use Layer 2 or non Layer 3 protocols for inter VM communication.

Inadequate Table Sizes at ToR Switch

The use of ToR switch to connect servers on a rack is a common data center design.  With multiple virtual machines running on the virtualized servers, the number of MAC address that a ToR switch that has to learn.  As the number of virtual machine that a virtualized server can host is increasing the MAC address table used by the ToR switch becomes inadequate

VXLAN Operations

Section 4 of RFC 7348 describes the operation of VXLAN.  As indicated in the title of this RFC, VXLAN is a framework for overlaying virtualized Layer 2 networks over Layer 3 networks.  It is designed specifically to overcome the problems that we face in the data center.

VXLAN is meant to extend a Layer 2 network over a Layer 3 network by the use of tunneling technology - encapsulating an UDP packet over the original Layer 2 frame.

RFC 7348 outline the following rules:

• Each overlay is termed a VXLAN segment.
• Only VMs within the same VXLAN segment can communicate with each other
• Each VXLAN segment is identified by a 24-bit segment ID (VNI).
• VNI identifies the scope of the inner MAC frame originated by the individual VM
• VNI is an outer header that encapsulates the inner MAC frame originated by the individual VM.
• VXLAN segment and VXLAN overlay network are interchangeable in the RFC.
• VXLAN tunnels are stateless connection between 2 end points.
• Each end point is called a VXLAN Tunnel End Point (VTEP)
• VTEP can be implemented on a virtual switch, physical switch or physical server either on hardware or software.
• Use of data plane learning.
• Multicast is used for carrying unknown destination, broadcast and multicast frames (BUM traffic).
• VTEPs MUST NOT fragment VXLAN packets.

The last 3 points are worth looked into a little bit more.

Data Plane Learning

Data plane learning means there is no control plane for VXLAN. Not until now that I realize I never truly understand what a control plane is.  Often time people will say generically that SDN is the separation of the control and data plane.  I work on networking area for almost 20 years and mostly Layer 2 networking features.  I say mostly because I worked on UDP Relay Agent and DHCP Snooping in which I have to know a little bit of IP forwarding.  Data plane is the forwarding operation of networking equipment.

So what exactly is a control plane?  For any network engineers working on Layer 3 networks this is very obvious to them.  Control plane is a Layer 3 networking concept.  BGP is an example of a control plane protocol.  Routers exchange routing information via the control plane.

VXLAN uses data plane learning just like the source learning on Ethernet (Layer 2) switches.  VTEP is responsible for learning the virtual machine’s MAC address and associate this with VXLAN segment/VXLAN Network Identifier.  This learning process is very important to the efficiency of the operation of VXLAN networks.

Multicast for BUM traffic

VXLAN is to extend Layer 2 segments to other data center.  In a Layer 2 network, unknown destination, broadcast and multicast frames are flooded to all the devices on the same broadcast domain.  With VXLAN, RFC 7348 specifically spell out IP multicast is used for sending BUM traffic to other VTEPs of the same VXLAN segment. 

While this works perfectly in the functional level, all Layer 3 network engineers always try to avoid IP multicast.

Cisco Nexus 1000V has Unicast-ONLY VXLAN and MAC Distribution Mode.  In the MAC Distribution Mode, there is a centralized controller.  This is same as introducing a control plane for VXLAN.

VMware NSX has its NSX Controller.  When a VM is provisioned, it will register itself to the NSX controller.  ARP request from the VMs are sent to the controller where the controller is aware of the MAC address and VTEP/VNI association. This again is introducing the control plane for VXLAN.

To work with other VTEPs, IP multicast still needs to be used.

RFC 7348 suggests to use bidirectional IP multicast protocol such as PIM-SM to build efficient multicast forwarding trees.

VTEPs MUST NOT fragment VXLAN packets

Due to encapsulation, VXLAN adds an extra 50 Bytes of overhead.  The “MUST NOT” is written in bold in the RFC.  This requirement has huge implication the MTU of the underlay Layer 3 network.  MTU for Ethernet v2 is 1500.  VMware recommend setting the MTU size to 1600 or to use jumbo frame option end to end. 

I know of an installation that ran into MTU problem and end up not deploying VXLAN.

VXLAN Frame Format

Section 5 of RFC 7348 details the frame format of VXLAN.  As a developer of network engineer that needs to troubleshoot VXLAN problem needs to know this frame format very well.

I believe WireShark is able to decode VXLAN traffic.  UDP port 4789 is assigned for VXLAN traffic.

VXLAN Security Considerations

While security consideration is a standard section for RFCs, it is also worth looking into. 

Quoting directly from the RFC:

Traditionally, Layer 2 networks can only be attacked from 'within' by rogue end points -- either

• by having inappropriate access to a LAN and snooping on traffic,
• by injecting spoofed packets to 'take over' another MAC address, or
• by flooding and causing denial of service. 

VXLAN increases the attack surface for these kinds of attacks.

While not going into detail, the security consideration section of this RFC suggests the following ways to safe guard VXLAN networks.

·     Continue to use the traditional way of mitigating rogue end points attack by limiting the management and administrative scope of who deploys and manages VM/gateways in a VXLAN environment.

• Use of 802.1X for admission control for individual end points.
• Use of 5-tuple-based ACL.
• Use of IPsec to authenticate and optionally encrypt VXLAN traffic.
• Use of designated VLAN for VXLAN traffic.
• Use of secure method on the management plane of the VTEP.

This summarize RFC 7348.

VXLAN in the contemporary data center

What is a Contemporary data center?

A contemporary data center is a virtualized data center.  At first it was only the server that was virtualized.  Virtualizing the servers alone changed the data center from static environment to a dynamic environment where servers running as virtual machines can be provisioned and deleted as well as moved from one physical machine to another.  The contemporary data center has changed into a dynamic/elastic environment.  

Later on, storage virtualization has made the data center more dynamic/elastic where beside the virtual machines, the data can be moved around. 

Virtual machines can move from one physical server to another server is very useful.  However, the limitation was that these physical servers have to be connected in a flat network (layer 2).

With multiple virtual servers running on a physical server allows for multi tenancy.  VLAN is a good way for traffic isolation among the various tenants.  The number of VLANs in a network is limited by the 12-bit field which is 4096 in which VLAN 0 is not a valid VLAN thus only 4095 VLANs can exist in a given layer-2 network.

To cope with the increased demand on the network from the virtualized data center the industry has come up with 3 different ways to alleviate the problems.  The most discussed technologies are:

• Network Virtualization
• Network Function Virtualization and
• Software Defined Networking

I will describe and compare these 3 technologies in another post.  This post will focus on VXLAN which is one version of Network Virtualization.

What is Network Virtualization?

Virtualization is the abstraction or decoupling of something from the physical entity.  In this case, for network virtualization it is the ability to abstract networking from the physical network. 

How does network virtualization abstract from the physical network?  One way is to use the technique of network overlay where tunnels between end points are created on existing physical networks.  The most common tunneling protocols are:

• VXLAN (Virtual Extensible LAN)
• Network Virtualization using Generic Encapsulation (NVGRE)
• Stateless Transport Tunneling (STT)
• Network Virtualization Overlay3 (NVO3)

Benefits of Network Overlay

The very first problem that network overlay can help solve is to extend the layer 2 domain across layer 3 subnets.  This resulted in physical servers are not confined to a single flat layer 2 network for virtual machines to move around.  With traffic tunneled between end points, it helps in traffic isolation among tenants.

Each overlay networks has its own network id and thus extend the 4095 VLAN limitation.  Furthermore, multi-tenants in the same data center can have the same private IP address.

What is VXLAN?

VXLAN (Virtual Extensible LAN) is a network tunneling technology by encapsulating UDP packet on top of a native Ethernet frame and transport over an IP network. 

It was jointly developed by VMware, Arista Networks and Cisco.  The latest specification which is moved to RFC status also has contribution from Storvisor, Broadcom, Citrix and Red Hat.  The title of this IETF draft is “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks”.  This technology was first announced at VMworld 2011. Since then there are tons of articles about this topic on different technical magazine and blog.

VXLAN Terminology

The best way to understand a technology, in my opinion is to start from the terminology being used.  It provides a framework of what the important elements are for a given technology. At the very least we can type in these key words in our favorite search engine and start researching.

The following are in my opinion the essential basic terminology used in the VXLAN world:

• Encapsulation
• VTEP
• VNI
• IP Multicast 

Encapsulation

The term encapsulation is used in Object Oriented Programming as well as in data communication.  The idea is the same in both cases. The concept of encapsulation is to put one object into another object and send to a destination. 

In the case of VXLAN, it is to put a layer-2 frame as the payload of an UDP packet and uses IP to reach the destination.  When reaching the destination, the packet is being de-capsulated.

I have a picture taken from the Cisco website that not only details the individual field of a VXLAN packet but also to explain the concept of encapsulation with color.  The yellow portion is the “original L2 frame” and is being put as the payload of an UDP packet as highlighted in blue.

Image source: http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.doc/_jcr_content/renditions/white-paper-c11-729383-02.jpg

VTEP

As we described in the above paragraph, packets are being encapsulated and de-capsulated from the source to the destination.  VTEP (VXLAN Tunnel End Point) is the entity that performs the encapsulation and de-capsulation. 

VTEP plays a vital role in the VXLAN operation.  It is these end points that the tunnel is created so that the “original L2 frame” can be transported back and forth thus achieving the goal of layer-2 communication over a layer-2 (IP) infrastructure when entities are in different IP subnets.  One constraint that we have mentioned in this post was that vMotions of virtual machine is limited to physical machines that are in the same layer-2 network.

VTEP can be implemented in virtual switches in the hypervisors or it can be on physical networking device such as switch and routers.

VNI

VXLAN is about layer-2 segments as inferred by its name – extensible.  Traditional VLAN segment is limited by the 12-bit VLAN ID to 4096 per network.  With VXLAN it is being expanded to 16 million logical segments.  This is done by the use of a 24-bit VNI or VNID (VXLAN Network ID) to uniquely identify logical segment within a VXLAN network.

Each device in the VXLAN network is uniquely identified by the combination of VNI and the MAC address.

IP Multicast

VXLAN operates on a layer-3/IP network.  Tunnels are created between VTEPs. IP multicast was specified in the VXLAN specification to be used to simulate a layer-2 broadcast to find the location of the destination device.  IP multicast can be IGMP or PIM.   I will have to find out if one method is being used more than the other.

Cisco has a proprietary implementation of using unicast to perform this function.  It is being called the unicast-mode.

Putting the terminology together to see how VXLAN works

Knowing the terminologies is like knowing the alphabets.  Now we are to make a sentence from the alphabets.

Image source: http://blogs.vmware.com/vsphere/files/2013/05/Learning-1.jpg

The above diagram that I have seen at the VMware blog explains the VXLAN operation very well:

• VTEP is implemented in VMware’s vSphere Distributed Switch.
• VTEP has an IP address and in this case they are on the same subnet.
• There is a Layer-3 infrastructure network.
• The 2 VTEPs are member of a multicast group and in this case IGMP (Internet Group Management Protocol) is used.
• The VNID is 5001.

When VM on the left wanted to communicate with VM on the right:

• VM sends out a destination unknown, broadcast or multicast packet
• VTEP on the left (IP = 10.20.10.10) encapsulate this layer-2 frame into an UDP packet and send it out to the multicast group.
• Other VTEPs in the multicast group (in this case there is only one) received the packet will de-capsulate and flood the packet on their local layer-2 domain.
• In this process the VNI and the MAC address of the VM on the left is learned by the VTEPs.
• VM on the right received the frame from VM on the left and reply.
• The reply frame will be send from VTEP on the right to the VTEP on the left as a unicast frame since the MAC address and VNI are learned.
• At this time the MAC address and the VNI of both VMs are learned by the VTEP and from this point onward, traffic between both VMs will be IP unicast between the 2 VTEPs

This is a brief overview of VXLAN in the contemporary data center.