Computer Networking 101 Part 2: "You Got Mail" - reaching the destination.

I am sure everyone knows what this icon is.  How many Email you have to read each day.  Are you able to keep your mail box at zero (0) unread mail?

Have you every wonder how Email reaches your mail box?

This post is not to explain how Email works.  With Email in mind I wanted touch another computer networking basic - reaching the destination.

Sending information from one host to another is just like sending a letter - it needs an address.  There are different kinds of addresses as I have mentioned in part 1, namely:

• MAC address
• IP address
• TCP or UDP port

There are other address with different communication protocol but in general are IP based.

In a computer network there are 3 kinds of traffic

1. Broadcast
2. Multicast
3. Unicast

These 3 kinds of traffic can be applied to layer-2 switching or layer-3 routing.  Layer-3 routing is much more complicated than layer-2 switching but the principle is the same and in this article I am going to treat layer-2 switching and layer-3 routing the same in the context of traffic type.

Broadcast traffic
These are the traffic that all devices on the same domain will received a copy.  In layer 2, it can be with destination MAC address ff:ff:ff:ff:ff:ff or destination unknown in which the switch has to flood the incoming packet to all port on the same domain.  In layer 3, the original packet will not be flooded but the router is going to send a layer-3 broadcast ARP packet to see if anyone knows where the destination is.  Layer-3 broadcast packet has the destination IP address as 255.255.255.255 or each IP subnet has a broadcast address.

image source: http://upload.wikimedia.org/wikipedia/commons/thumb/d/dc/Broadcast.svg/2000px-Broadcast.svg.png

All members on the same domain receives the broadcast traffic.

Multicast traffic
In layer 2 multicast is treated as the broadcast.  MAC address has a multicast bit and if it is set, the layer-2 frame is a multicast frame and layer-2 switch will not learn this address.
I work as a software developer for networking equipment company.  Often time we have to construct our Layer-2 frames.  I will always use 00:11:22:33;44:55 and I will not use 11:22:33:44:55:66 because the second MAC address the multicast bit is turned on and it will be treated as a multicast frame.  

image source: http://upload.wikimedia.org/wikipedia/commons/b/be/Multicast-multicast.png

The above diagram explains the concepts of multicast very well.  All circles are devices on the same domain but only the green circles registered to a multicast group and thus only the green circles will receive the multicast traffic.

In layer-3, there is the concept of multicast group and device will have to register to the multicast group to receive a copy of the layer-3 packet.  VXLAN relies heavily on this.  Multicast limits the flooding of the layer-3 packet to members of the multicast group.

Unicast traffic
This is the traffic type where networking equipment know exactly where the destination is and is the most efficient way of sending network traffic because it does not need to replicate the original traffic and to use unnecessary bandwidth.

image source: http://upload.wikimedia.org/wikipedia/commons/thumb/7/75/Unicast.svg/640px-Unicast.svg.png

Only one member receives the traffic.

Reaching the destination
Networking equipment makes forwarding decisions on these 3 kinds of traffic and send the traffic to the indent destination based on the unique MAC or IP address as well as the TCP/UDP ports. Traffic type has huge impact on the network resources and networking equipment vendors always wanted to find ways to streamline the traffic forwarding process as well as to maximize all the networking resources such as CPU cycles, buffers and/or memory usage.  In the case of VXLAN, since the early days of VXLAN, vendors or open source community tried to find ways to avoid IP multicast for “flooding” the IP packet to all VTEP in the same multicast group.

Networking is a huge subject and is not easy to cover it in one or two blog post.  I am hoping to lay down some important concept so that as I journey to the cloud and pick up new things in the networking area, I am able to refer to these "networking basic" posts.  I will come back to update and clarify these posts as I move on.  So stay tune.

A Quick Look at the VXLAN's specification - RFC 7348

VXLAN is being deployed in numerous production environments and is supported by quite a few networking equipment vendor as well as software vendors such as VMware and Red Hat.  Not until August 2014 that the specification is moved from IETF draft status to RFC 7348.

RFC stands for Request For Comments. As a software developer for networking equipment, I have to know the RFC in and out.  This RFC is the functional specification for the feature that I develop.  Test group will test the feature to make sure the feature function as what the RFC describes.

Today, let’s take a look at what is in RFC 7348.

The title of RFC 7348 is “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.”

RFC 7348 is relatively a short document compare to other RFC that specify other communication protocols.   The table of contents has 10 sections.  6 of them are standard sections for all RFC.  To break down this RFC to the core, we can look at the following sections:

• VXLAN Problem Statement
• VXLAN operation
• VXLAN Frame Format
• VXLAN Security Considerations

VXLAN Problem Statement

Basically RFC 7348 identified 3 problem areas in the networking infrastructure within the data center that VXLAN is designed to resolve.  Server virtualization in the data center works best in a flat layer 2 network.  Only until VMworld 2014 that VMware announced in vSphere 6 (as of this writing, it is still under beta) can support vMotion across vCenter and long distance.

Limitations Imposed by Spanning Tree and VLAN Ranges

Spanning Tree is used to safe guard loops in a Layer 2 network.  When there is a loop in a Layer 2 network, it will cause a broadcast storm and the network will be in-operable.  Spanning Tree Protocol is designed and used to protect the Layer 2 network.  While it is able to protect the Layer 2 network the price to pay is there are links in the network is not used as well as multipathing not an option in the network design.  Both of these problems ties back to ROI (Return On Investment) not to a point where it should be.

Another problem is that the VLAN ID is a 12-bit field thus causing data center in flat Layer 2 network limited to only 4094 VLANs.

Multi-tenant Environment

Multi-tenant is a virtualized data center is a definite requirement.  Tenant isolation is an absolute requirement.

In a Layer 2 network VLAN is a popular way to achieve tenant isolation.  The number of VLANs limited to 4094 as described in the previous section is a major stumbling block to data center design.  When each tenant requires to have more than one VLAN as in the example of a 3-tier web application, the number of VLANs available in a flat Layer 2 network become a more limited and yet required resource.

A layer 3 network is another possible way of network isolation for multi-tenant but it has it limitation.  This will limit each tenant to have unique IP subnets.  Also, this Layer 3 isolation solution limits user not able to use Layer 2 or non Layer 3 protocols for inter VM communication.

Inadequate Table Sizes at ToR Switch

The use of ToR switch to connect servers on a rack is a common data center design.  With multiple virtual machines running on the virtualized servers, the number of MAC address that a ToR switch that has to learn.  As the number of virtual machine that a virtualized server can host is increasing the MAC address table used by the ToR switch becomes inadequate

VXLAN Operations

Section 4 of RFC 7348 describes the operation of VXLAN.  As indicated in the title of this RFC, VXLAN is a framework for overlaying virtualized Layer 2 networks over Layer 3 networks.  It is designed specifically to overcome the problems that we face in the data center.

VXLAN is meant to extend a Layer 2 network over a Layer 3 network by the use of tunneling technology - encapsulating an UDP packet over the original Layer 2 frame.

RFC 7348 outline the following rules:

• Each overlay is termed a VXLAN segment.
• Only VMs within the same VXLAN segment can communicate with each other
• Each VXLAN segment is identified by a 24-bit segment ID (VNI).
• VNI identifies the scope of the inner MAC frame originated by the individual VM
• VNI is an outer header that encapsulates the inner MAC frame originated by the individual VM.
• VXLAN segment and VXLAN overlay network are interchangeable in the RFC.
• VXLAN tunnels are stateless connection between 2 end points.
• Each end point is called a VXLAN Tunnel End Point (VTEP)
• VTEP can be implemented on a virtual switch, physical switch or physical server either on hardware or software.
• Use of data plane learning.
• Multicast is used for carrying unknown destination, broadcast and multicast frames (BUM traffic).
• VTEPs MUST NOT fragment VXLAN packets.

The last 3 points are worth looked into a little bit more.

Data Plane Learning

Data plane learning means there is no control plane for VXLAN. Not until now that I realize I never truly understand what a control plane is.  Often time people will say generically that SDN is the separation of the control and data plane.  I work on networking area for almost 20 years and mostly Layer 2 networking features.  I say mostly because I worked on UDP Relay Agent and DHCP Snooping in which I have to know a little bit of IP forwarding.  Data plane is the forwarding operation of networking equipment.

So what exactly is a control plane?  For any network engineers working on Layer 3 networks this is very obvious to them.  Control plane is a Layer 3 networking concept.  BGP is an example of a control plane protocol.  Routers exchange routing information via the control plane.

VXLAN uses data plane learning just like the source learning on Ethernet (Layer 2) switches.  VTEP is responsible for learning the virtual machine’s MAC address and associate this with VXLAN segment/VXLAN Network Identifier.  This learning process is very important to the efficiency of the operation of VXLAN networks.

Multicast for BUM traffic

VXLAN is to extend Layer 2 segments to other data center.  In a Layer 2 network, unknown destination, broadcast and multicast frames are flooded to all the devices on the same broadcast domain.  With VXLAN, RFC 7348 specifically spell out IP multicast is used for sending BUM traffic to other VTEPs of the same VXLAN segment. 

While this works perfectly in the functional level, all Layer 3 network engineers always try to avoid IP multicast.

Cisco Nexus 1000V has Unicast-ONLY VXLAN and MAC Distribution Mode.  In the MAC Distribution Mode, there is a centralized controller.  This is same as introducing a control plane for VXLAN.

VMware NSX has its NSX Controller.  When a VM is provisioned, it will register itself to the NSX controller.  ARP request from the VMs are sent to the controller where the controller is aware of the MAC address and VTEP/VNI association. This again is introducing the control plane for VXLAN.

To work with other VTEPs, IP multicast still needs to be used.

RFC 7348 suggests to use bidirectional IP multicast protocol such as PIM-SM to build efficient multicast forwarding trees.

VTEPs MUST NOT fragment VXLAN packets

Due to encapsulation, VXLAN adds an extra 50 Bytes of overhead.  The “MUST NOT” is written in bold in the RFC.  This requirement has huge implication the MTU of the underlay Layer 3 network.  MTU for Ethernet v2 is 1500.  VMware recommend setting the MTU size to 1600 or to use jumbo frame option end to end. 

I know of an installation that ran into MTU problem and end up not deploying VXLAN.

VXLAN Frame Format

Section 5 of RFC 7348 details the frame format of VXLAN.  As a developer of network engineer that needs to troubleshoot VXLAN problem needs to know this frame format very well.

I believe WireShark is able to decode VXLAN traffic.  UDP port 4789 is assigned for VXLAN traffic.

VXLAN Security Considerations

While security consideration is a standard section for RFCs, it is also worth looking into. 

Quoting directly from the RFC:

Traditionally, Layer 2 networks can only be attacked from 'within' by rogue end points -- either

• by having inappropriate access to a LAN and snooping on traffic,
• by injecting spoofed packets to 'take over' another MAC address, or
• by flooding and causing denial of service. 

VXLAN increases the attack surface for these kinds of attacks.

While not going into detail, the security consideration section of this RFC suggests the following ways to safe guard VXLAN networks.

·     Continue to use the traditional way of mitigating rogue end points attack by limiting the management and administrative scope of who deploys and manages VM/gateways in a VXLAN environment.

• Use of 802.1X for admission control for individual end points.
• Use of 5-tuple-based ACL.
• Use of IPsec to authenticate and optionally encrypt VXLAN traffic.
• Use of designated VLAN for VXLAN traffic.
• Use of secure method on the management plane of the VTEP.

This summarize RFC 7348.

VXLAN in the contemporary data center

What is a Contemporary data center?

A contemporary data center is a virtualized data center.  At first it was only the server that was virtualized.  Virtualizing the servers alone changed the data center from static environment to a dynamic environment where servers running as virtual machines can be provisioned and deleted as well as moved from one physical machine to another.  The contemporary data center has changed into a dynamic/elastic environment.  

Later on, storage virtualization has made the data center more dynamic/elastic where beside the virtual machines, the data can be moved around. 

Virtual machines can move from one physical server to another server is very useful.  However, the limitation was that these physical servers have to be connected in a flat network (layer 2).

With multiple virtual servers running on a physical server allows for multi tenancy.  VLAN is a good way for traffic isolation among the various tenants.  The number of VLANs in a network is limited by the 12-bit field which is 4096 in which VLAN 0 is not a valid VLAN thus only 4095 VLANs can exist in a given layer-2 network.

To cope with the increased demand on the network from the virtualized data center the industry has come up with 3 different ways to alleviate the problems.  The most discussed technologies are:

• Network Virtualization
• Network Function Virtualization and
• Software Defined Networking

I will describe and compare these 3 technologies in another post.  This post will focus on VXLAN which is one version of Network Virtualization.

What is Network Virtualization?

Virtualization is the abstraction or decoupling of something from the physical entity.  In this case, for network virtualization it is the ability to abstract networking from the physical network. 

How does network virtualization abstract from the physical network?  One way is to use the technique of network overlay where tunnels between end points are created on existing physical networks.  The most common tunneling protocols are:

• VXLAN (Virtual Extensible LAN)
• Network Virtualization using Generic Encapsulation (NVGRE)
• Stateless Transport Tunneling (STT)
• Network Virtualization Overlay3 (NVO3)

Benefits of Network Overlay

The very first problem that network overlay can help solve is to extend the layer 2 domain across layer 3 subnets.  This resulted in physical servers are not confined to a single flat layer 2 network for virtual machines to move around.  With traffic tunneled between end points, it helps in traffic isolation among tenants.

Each overlay networks has its own network id and thus extend the 4095 VLAN limitation.  Furthermore, multi-tenants in the same data center can have the same private IP address.

What is VXLAN?

VXLAN (Virtual Extensible LAN) is a network tunneling technology by encapsulating UDP packet on top of a native Ethernet frame and transport over an IP network. 

It was jointly developed by VMware, Arista Networks and Cisco.  The latest specification which is moved to RFC status also has contribution from Storvisor, Broadcom, Citrix and Red Hat.  The title of this IETF draft is “VXLAN: A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks”.  This technology was first announced at VMworld 2011. Since then there are tons of articles about this topic on different technical magazine and blog.

VXLAN Terminology

The best way to understand a technology, in my opinion is to start from the terminology being used.  It provides a framework of what the important elements are for a given technology. At the very least we can type in these key words in our favorite search engine and start researching.

The following are in my opinion the essential basic terminology used in the VXLAN world:

• Encapsulation
• VTEP
• VNI
• IP Multicast 

Encapsulation

The term encapsulation is used in Object Oriented Programming as well as in data communication.  The idea is the same in both cases. The concept of encapsulation is to put one object into another object and send to a destination. 

In the case of VXLAN, it is to put a layer-2 frame as the payload of an UDP packet and uses IP to reach the destination.  When reaching the destination, the packet is being de-capsulated.

I have a picture taken from the Cisco website that not only details the individual field of a VXLAN packet but also to explain the concept of encapsulation with color.  The yellow portion is the “original L2 frame” and is being put as the payload of an UDP packet as highlighted in blue.

Image source: http://www.cisco.com/c/dam/en/us/products/collateral/switches/nexus-9000-series-switches/white-paper-c11-729383.doc/_jcr_content/renditions/white-paper-c11-729383-02.jpg

VTEP

As we described in the above paragraph, packets are being encapsulated and de-capsulated from the source to the destination.  VTEP (VXLAN Tunnel End Point) is the entity that performs the encapsulation and de-capsulation. 

VTEP plays a vital role in the VXLAN operation.  It is these end points that the tunnel is created so that the “original L2 frame” can be transported back and forth thus achieving the goal of layer-2 communication over a layer-2 (IP) infrastructure when entities are in different IP subnets.  One constraint that we have mentioned in this post was that vMotions of virtual machine is limited to physical machines that are in the same layer-2 network.

VTEP can be implemented in virtual switches in the hypervisors or it can be on physical networking device such as switch and routers.

VNI

VXLAN is about layer-2 segments as inferred by its name – extensible.  Traditional VLAN segment is limited by the 12-bit VLAN ID to 4096 per network.  With VXLAN it is being expanded to 16 million logical segments.  This is done by the use of a 24-bit VNI or VNID (VXLAN Network ID) to uniquely identify logical segment within a VXLAN network.

Each device in the VXLAN network is uniquely identified by the combination of VNI and the MAC address.

IP Multicast

VXLAN operates on a layer-3/IP network.  Tunnels are created between VTEPs. IP multicast was specified in the VXLAN specification to be used to simulate a layer-2 broadcast to find the location of the destination device.  IP multicast can be IGMP or PIM.   I will have to find out if one method is being used more than the other.

Cisco has a proprietary implementation of using unicast to perform this function.  It is being called the unicast-mode.

Putting the terminology together to see how VXLAN works

Knowing the terminologies is like knowing the alphabets.  Now we are to make a sentence from the alphabets.

Image source: http://blogs.vmware.com/vsphere/files/2013/05/Learning-1.jpg

The above diagram that I have seen at the VMware blog explains the VXLAN operation very well:

• VTEP is implemented in VMware’s vSphere Distributed Switch.
• VTEP has an IP address and in this case they are on the same subnet.
• There is a Layer-3 infrastructure network.
• The 2 VTEPs are member of a multicast group and in this case IGMP (Internet Group Management Protocol) is used.
• The VNID is 5001.

When VM on the left wanted to communicate with VM on the right:

• VM sends out a destination unknown, broadcast or multicast packet
• VTEP on the left (IP = 10.20.10.10) encapsulate this layer-2 frame into an UDP packet and send it out to the multicast group.
• Other VTEPs in the multicast group (in this case there is only one) received the packet will de-capsulate and flood the packet on their local layer-2 domain.
• In this process the VNI and the MAC address of the VM on the left is learned by the VTEPs.
• VM on the right received the frame from VM on the left and reply.
• The reply frame will be send from VTEP on the right to the VTEP on the left as a unicast frame since the MAC address and VNI are learned.
• At this time the MAC address and the VNI of both VMs are learned by the VTEP and from this point onward, traffic between both VMs will be IP unicast between the 2 VTEPs

This is a brief overview of VXLAN in the contemporary data center.