VXLAN is being deployed in numerous production environments and is supported by quite a few networking equipment vendor as well as software vendors such as VMware and Red Hat. Not until August 2014 that the specification is moved from IETF draft status to RFC 7348.
RFC stands for Request For Comments. As a software developer for networking equipment, I have to know the RFC in and out. This RFC is the functional specification for the feature that I develop. Test group will test the feature to make sure the feature function as what the RFC describes.
Today, let’s take a look at what is in RFC 7348.
The title of RFC 7348 is “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.”
RFC 7348 is relatively a short document compare to other RFC that specify other communication protocols. The table of contents has 10 sections. 6 of them are standard sections for all RFC. To break down this RFC to the core, we can look at the following sections:
- VXLAN Problem Statement
- VXLAN operation
- VXLAN Frame Format
- VXLAN Security Considerations
VXLAN Problem Statement
Basically RFC 7348 identified 3 problem areas in the networking infrastructure within the data center that VXLAN is designed to resolve. Server virtualization in the data center works best in a flat layer 2 network. Only until VMworld 2014 that VMware announced in vSphere 6 (as of this writing, it is still under beta) can support vMotion across vCenter and long distance.
Limitations Imposed by Spanning Tree and VLAN Ranges
Spanning Tree is used to safe guard loops in a Layer 2 network. When there is a loop in a Layer 2 network, it will cause a broadcast storm and the network will be in-operable. Spanning Tree Protocol is designed and used to protect the Layer 2 network. While it is able to protect the Layer 2 network the price to pay is there are links in the network is not used as well as multipathing not an option in the network design. Both of these problems ties back to ROI (Return On Investment) not to a point where it should be.
Another problem is that the VLAN ID is a 12-bit field thus causing data center in flat Layer 2 network limited to only 4094 VLANs.
Multi-tenant is a virtualized data center is a definite requirement. Tenant isolation is an absolute requirement.
In a Layer 2 network VLAN is a popular way to achieve tenant isolation. The number of VLANs limited to 4094 as described in the previous section is a major stumbling block to data center design. When each tenant requires to have more than one VLAN as in the example of a 3-tier web application, the number of VLANs available in a flat Layer 2 network become a more limited and yet required resource.
A layer 3 network is another possible way of network isolation for multi-tenant but it has it limitation. This will limit each tenant to have unique IP subnets. Also, this Layer 3 isolation solution limits user not able to use Layer 2 or non Layer 3 protocols for inter VM communication.
Inadequate Table Sizes at ToR Switch
The use of ToR switch to connect servers on a rack is a common data center design. With multiple virtual machines running on the virtualized servers, the number of MAC address that a ToR switch that has to learn. As the number of virtual machine that a virtualized server can host is increasing the MAC address table used by the ToR switch becomes inadequate
Section 4 of RFC 7348 describes the operation of VXLAN. As indicated in the title of this RFC, VXLAN is a framework for overlaying virtualized Layer 2 networks over Layer 3 networks. It is designed specifically to overcome the problems that we face in the data center.
VXLAN is meant to extend a Layer 2 network over a Layer 3 network by the use of tunneling technology - encapsulating an UDP packet over the original Layer 2 frame.
RFC 7348 outline the following rules:
- Each overlay is termed a VXLAN segment.
- Only VMs within the same VXLAN segment can communicate with each other
- Each VXLAN segment is identified by a 24-bit segment ID (VNI).
- VNI identifies the scope of the inner MAC frame originated by the individual VM
- VNI is an outer header that encapsulates the inner MAC frame originated by the individual VM.
- VXLAN segment and VXLAN overlay network are interchangeable in the RFC.
- VXLAN tunnels are stateless connection between 2 end points.
- Each end point is called a VXLAN Tunnel End Point (VTEP)
- VTEP can be implemented on a virtual switch, physical switch or physical server either on hardware or software.
- Use of data plane learning.
- Multicast is used for carrying unknown destination, broadcast and multicast frames (BUM traffic).
- VTEPs MUST NOT fragment VXLAN packets.
The last 3 points are worth looked into a little bit more.
Data Plane Learning
Data plane learning means there is no control plane for VXLAN. Not until now that I realize I never truly understand what a control plane is. Often time people will say generically that SDN is the separation of the control and data plane. I work on networking area for almost 20 years and mostly Layer 2 networking features. I say mostly because I worked on UDP Relay Agent and DHCP Snooping in which I have to know a little bit of IP forwarding. Data plane is the forwarding operation of networking equipment.
So what exactly is a control plane? For any network engineers working on Layer 3 networks this is very obvious to them. Control plane is a Layer 3 networking concept. BGP is an example of a control plane protocol. Routers exchange routing information via the control plane.
VXLAN uses data plane learning just like the source learning on Ethernet (Layer 2) switches. VTEP is responsible for learning the virtual machine’s MAC address and associate this with VXLAN segment/VXLAN Network Identifier. This learning process is very important to the efficiency of the operation of VXLAN networks.
Multicast for BUM traffic
VXLAN is to extend Layer 2 segments to other data center. In a Layer 2 network, unknown destination, broadcast and multicast frames are flooded to all the devices on the same broadcast domain. With VXLAN, RFC 7348 specifically spell out IP multicast is used for sending BUM traffic to other VTEPs of the same VXLAN segment.
While this works perfectly in the functional level, all Layer 3 network engineers always try to avoid IP multicast.
Cisco Nexus 1000V has Unicast-ONLY VXLAN and MAC Distribution Mode. In the MAC Distribution Mode, there is a centralized controller. This is same as introducing a control plane for VXLAN.
VMware NSX has its NSX Controller. When a VM is provisioned, it will register itself to the NSX controller. ARP request from the VMs are sent to the controller where the controller is aware of the MAC address and VTEP/VNI association. This again is introducing the control plane for VXLAN.
To work with other VTEPs, IP multicast still needs to be used.
RFC 7348 suggests to use bidirectional IP multicast protocol such as PIM-SM to build efficient multicast forwarding trees.
VTEPs MUST NOT fragment VXLAN packets
Due to encapsulation, VXLAN adds an extra 50 Bytes of overhead. The “MUST NOT” is written in bold in the RFC. This requirement has huge implication the MTU of the underlay Layer 3 network. MTU for Ethernet v2 is 1500. VMware recommend setting the MTU size to 1600 or to use jumbo frame option end to end.
I know of an installation that ran into MTU problem and end up not deploying VXLAN.
VXLAN Frame Format
Section 5 of RFC 7348 details the frame format of VXLAN. As a developer of network engineer that needs to troubleshoot VXLAN problem needs to know this frame format very well.
I believe WireShark is able to decode VXLAN traffic. UDP port 4789 is assigned for VXLAN traffic.
VXLAN Security Considerations
While security consideration is a standard section for RFCs, it is also worth looking into.
Quoting directly from the RFC:
Traditionally, Layer 2 networks can only be attacked from 'within' by rogue end points -- either
- by having inappropriate access to a LAN and snooping on traffic,
- by injecting spoofed packets to 'take over' another MAC address, or
- by flooding and causing denial of service.
VXLAN increases the attack surface for these kinds of attacks.
While not going into detail, the security consideration section of this RFC suggests the following ways to safe guard VXLAN networks.
· Continue to use the traditional way of mitigating rogue end points attack by limiting the management and administrative scope of who deploys and manages VM/gateways in a VXLAN environment.
- Use of 802.1X for admission control for individual end points.
- Use of 5-tuple-based ACL.
- Use of IPsec to authenticate and optionally encrypt VXLAN traffic.
- Use of designated VLAN for VXLAN traffic.
- Use of secure method on the management plane of the VTEP.
This summarize RFC 7348.