Microsegmentation – a perfect fit for Microservices security

On my last post, we have explored a new chapter in Docker networking. With the new and yet still under development libnetwork, Docker container is now able to take advantage of 3rd party networking and security solutions.  One of the security solution is Microsegmentation.

Microsegmentation is not a new concept but the implementation of Microsegmentation is not feasible until network virtualization had become mature.  Both VMware and Cisco offers Microsegmentation solutions.

What is Microsegmentation?

Segmentation is a security principle used to group entities within a network into one unit and to apply rules/polices to control the traffic in and out of the segment. Usually, this is done by a firewall.  Microsegmentation is to be able to provide a way to define rules/policies in a smaller granular way and sometimes as small as a Docker container.

Underlying Principles
An important principle for Microsegmentation is Zero Trust.  It is simply to say that within a network, nothing is trusted.  In the traditional IT security model, the assumption that attack to an IT infrastructure is from the "outside".  There is the familiar tools such as DMZ, Intrusion Detection/Protection System and the antivirus software that companies spend a lots of money on to stop hacker from entering into the system. Once inside the system, there is not much being done to check for the traffic inside the perimeter.

Perimeter based security measures are not enough for the modern day IT infrastructure especially when the work load is in the cloud in which the perimeter is not very well defined.  It is important to employ a Zero Trust model so as to provide the maximum level of security measure.

Major components for effective Microsegmentation
I read an article by Scott Lowe (I cannot find where it is.  Will put in the URL once I find it).  According to that article there are 3 main components that defines an effective Microsegmentation implementation

1. Network independent policy definition
2. Centralized policy definition repository
3. Distributed policy enforcement

1. Network independent policy definition
Traditional firewall rules use the 5 tuples to define access or deny rules.  It is enough when service is run in the monolithic process.  To provide a more granular way for Microsegmentation, the rules and policies has to be more than just network attributes.  One example of network independent definition can be the type of OS kernel or even path level of the same OS.

2. Centralized policy definition repository
Virtual Machine and/or Docker containers can move around.  To be effective, Microsegmentation has to have the ability to define the security policy in a central location so that no matter where the segment is moved, the policy can be easily retrieved.

3. Distributed policy enforcement
The enforcement point of the security policy has to be as close to the source as possible so that it will not be a bottleneck when applying the security policy. 

Microservices opens up Security risk

Microservices architecture is perfect for cloud native application, agile and with the ability to scale in and out depending on the need.  On the other hand, Microservices architecture opens up security risk that needs to be addressed:

• Frequent and short life span
• Increase of East-West traffic
• Services are not as isolated

Frequent and short life span
How is short life span a security risk? It is a security risk when the security rules cannot keep track on when and on which host machine the services are being deployed.  Over time the ACL which is a common form of security rule for segmentation can become unmanageable and thus create security risk. This problem is aggregated when the Microservices have frequent life span. A Docker container can be provisioned in a matter of millisecond.  We need to have the security measure to catch up with the pace otherwise there is going to create interval of security exposures.

Increase of East-West traffic
The traffic between client and server is defined as North-South traffic and traffic between servers are defined as East-West traffic.  Why would increase in east-west traffic present a new security threat?  Traditional security model has the assumption that attack to an IT infrastructure is from the "outside".  There are not much being done to stop hackers from hoping from server to server.  With the increase of East-West traffic, it amplifies this security risk that needs to be addressed.

Services are not as isolated
Microservices architecture is to break a monolithic service into smaller services.  In the monolithic model components of a service runs within the same process and it is much easier to provide service isolation.  Also the traffic to a monolithic service is much easier to control.  With Microservices, components of a service are not running as an individual process.  Individual processes that work together to perform a service needs to communicate with each other.  Traditional firewall rules uses the networking 5 tuples as the bases of the rules as to either allow or deny.  Now with Microservices, the use of networking attributes to setup up allow or deny rules are not enough thus making it difficult for the administrator of the IT infrastructure to use traditional firewall rules to provide service segmentation.

If Docker container is used to implement the Microservice architecture, we have the problem of all containers sharing the same OS kernel.  In this way, individual micro services are not truly isolated from each other.  Lots of development is done on Docker security but at least for now, it is a security risk that we have to address.

Microsegmentation and Microservices fits right in

Microsegmentation and Microservices is a perfect match.  How?

What Microsegmentation can offer for security is just what Microservices architecture implemented in the form of Docker containers needs. With Network virtualization as the foundation of Microsegmentation, the security policy is able to be ready to secure the fast provisioned Microservices.

The 3 major components of an effective Microsegmentation help to mitigate the security risk that Microservices opened up.

Network independent policy definition
In Microsegmentation, security rules are not limited to the traditional 5 tuples networking attributes and thus we can define a Docker container as a single segment and that the enforcement point of the policy is applied to the networking interface of the Docker container.  This can effective monitor and control the East-West traffic of the various Microservices.

Even if a single Docker container is compromised the hacker will not be able to break out from the Docker container because the security rule that is defined for a single Microservice will stop the unexpected traffic.  Depending on the implementation of the Microsegmentation, the Docker container once being found to have violated the defined security rule, it can be killed and being spawned back.

Centralized policy definition repository and distributed policy enforcement
Microservices architecture allows the application to be cloud native.  Common characteristic of a cloud native application is that they can be provisioned quickly and on-demand.  As these Microservices come and go they can be provisioned on different host machines.  For the purpose of balancing resources, these Microservices might be moved around different host machine during its life span.   Microsegmentation is able to adapt to dynamic and elastic nature of cloud native applications with distributed policy enforcement.  The policy defined in a centralized location by the IT infrastructure administrator is enforced regardless of where the Microservices is provisioned.

A New Chapter in Docker Networking

Docker networking is entering a new chapter.

Networking is one of the pillar for modern day IT infrastructure and lots of work are done by various networking equipment vendor to provide a stable and fast network.  Recently, there is also the movement of Software Defined Network (SDN) as well as the Network Function Virtualization (NFV). 

In the traditional client and server model, the traffic pattern is mostly "north-south traffic" (between the server and the clients). 

With Docker where most of the time it is being used to deploy Micro-Services, there is a need for the containers to talk to one another both within the same host or across multiple hosts.  This changed the traffic pattern and the demand for the network is changed to add "east-west traffic" (traffic between hosts).

Docker Inc has done a good job on Docker in packaging container but the networking support is a bit primitive. I had a blog post on Docker Networking options last year and before that I had another post describing what Docker container is and that VMware is not against but embracing this container technologies.   And of course there is Project Bonneville that is in technology preview state where VMware is making Docker containers to work just like a virtual machine in the vSphere environment so as to take advantage of the "enterprise ready" features of vSphere such as Distributed Resource Scheduler, vMotion and the benefit of the lightweight, fast provisioning characteristic of Docker container.

Native Docker Networking

• On startup Docker creates a Ethernet Bridge docker0 on the Linux Kernel
• docker0 creates a virtual subnet on the Docker host
• Docker creates a pair of virtual Ethernet interface on each container
• One of the Ethernet interface is the eth0 in the container
• Another Ethernet interface will have a unique name in the form of veth* (e.g.vethABI3IC) and is bind to docker0
• User can customize docker0
• Advanced Docker networking can be found here

image source: http://www.infrabricks.de/assets/images/docker_network_basics1.png

The native Docker networking was simple and is designed as a single-host solution. Native Docker networking does not scale well which is against Docker container use cases. 

Docker Networking from 3rd parties

As mentioned on my blog post from last year there are solutions/projects in development to solve or to improve Docker networking.  These solutions are:

• Weave
• Kubernetes
• Flannel
• Pipework
• SocketPlane <-now part of Docker Inc.

For detail description of these solutions you can take a look at here or here

While these solutions are useful and has its use cases, they are all external to Docker.

Docker's latest Networking Solution
On April 30, 2015, Docker announced an open source project - libnetwork.

libnetwork
Libnetwork is an open source project and can be found in GitHub here.

This "libnetwork" is a library that can provide native support for Docker container and its function is to connect containers.  This library is written in the Go language.  According to GitHub, "libnetwork project will follow Docker and Linux philosophy of developing small, highly modular and composable tools that works well independently. Libnetwork aims to satisfy that composable need for Networking in Containers."

Libnetwork implements the Container Network Model is is the work of various networking partners of Docker Inc such as Cisco, IBM, Microsoft, Joynet, Rancher, VMware and Weave.

The most important aspect for libnetwork is that it uses a driver/plugin model.  In the pass, Docker networking is handle by libcontainer and Docker Engine and now with libnetwork it can provide a single interface via the form of an API. 

Container Network Model
This model has 3 main components:

1. SandBox
2. Endpoint
3. Network

 image source: https://blog.docker.com/media/2015/04/cnm-model.jpg

This architecture diagram of Container Network Model is pretty self-explanatory.  Again, GitHub has good information about what these 3 elements are:

Sandbox
A Sandbox contains the configuration of a container's network stack. This includes management of the container's interfaces, routing table and DNS settings. An implementation of a Sandbox could be a Linux Network Namespace, a FreeBSD Jail or other similar concept. A Sandbox may contain many endpoints from multiple networks.

Endpoint
An Endpoint joins a Sandbox to a Network. An implementation of an Endpoint could be a veth pair, an Open vSwitch internal port or similar. An Endpoint can belong to only one network but may only belong to one Sandbox.

Network
A Network is a group of Endpoints that are able to communicate with each-other directly. An implementation of a Network could be a Linux bridge, a VLAN, etc. Networks consist of many endpoints.

Why is libnetwork so special?
Libnetwork is indeed very special that I called this a new chapter for Docker networking.

We have seen that libnetwork provides a single interface for networking. The significant of a single interface is that libnetwork can be present a plugin for external networking solutions. This is similar to the Neutron project for OpenStack where 3rd party networking solutions can be use.  

Both VMware and Cisco has already jump into this band wagon with their respective NSX and ACI networking solution to provide a robust networking solution for mulit-host container communication.  

Beside a robust networking solution, being able to use 3rd party networking solutions is also able to provide Docker containers security and layer 4 - 7 network functions features such as firewall and load-balancer.  

Security is an important aspect for all deployment in any environment. Both VMware's NSX and Cisco's ACI implements Micro-segmentation which is to provide a distributed firewall with extended rules.  These extended firewall rules allows user to define security policies beyond the traditional network attributes based rules.  My next post will be on Micro-segmentation.

Note: libnetwork is still under heavy development and is listed as experimental in Docker 1.7.  Please check GitHub for the latest status as things are going in a fact pace.

Reference:
https://github.com/docker/libnetwork/blob/master/docs/design.md
"Docker/libnetwork." GitHub. N.p., n.d. Web. 17 Aug. 2015.

Who do you trust? - nobody

It is not about me. I do have faith in the human race and there are people that I trust.

It is about a new security model proposed by Forrester Research in 2010.   

Traditional Network Security

The problem with the traditional network security model is that it assumes anything outside the network is untrusted while everything inside the network is trusted.  Heavy emphasis is put at the edge for network access control.  Once a user is in the network, there is not much control. 

There is the Role Based Network Control (RBAC) in which based on the credential of a user and sometimes based on where and when the user is trying to access the network, a role is assigned to the user after the user successfully authenticates with proper credential. It is more useful when RBAC is implemented at the application level. To implement RBAC at the network level, security control is still limited.

With the proliferation of server virtualization, virtual machine can move from one host to another host.  This makes the application of security control more difficult - where is the perimeter?

Before we go on we need to spell out 2 definitions: 

• East-West traffic: it is the traffic between servers within a datacenter
• North-South traffic: it is the traffic between client and server

Traditional security model mostly tailor to north-south traffic and not much is done for east-west traffic.

Zero Trust Security Model
The "Zero Trust" security model is proposed by John Kindervag, a senior research analyst at Forrester Research.  His report can be found here (you have to paid to read the full report).  Well, we can also listen to John Kindervag talk about this "Zero Trust" model here in YouTube.  Actually the name of this security model captured the essence - "Trust no one".  From the YouTube video, John Kindervag mentioned 3 concepts for "Zero Trust" security model:

1. All resources are accessed in a secure manner regardless of location
2. Access control is on a "need-to-know" basis and is strictly enforced
3. Inspect and log all traffic

To implement this on the traditional 3 tier network (access/aggregate/core) is not easy.

Today let's take a look at VMware and Cisco products that utilizes this "Zero Trust" security model.  This security model also protects east-west traffic between servers.

VMware
VMware implemented Zero Trust security model in its NSX product.

VMware NSX is well known as a Software Defined Network (SDN) feature.  I have in another post stating that NSX is also a security product and according to Chris King, vice president of product marketing for VMware's Networking and Security Business Unit, a lot of customers show interest in NSX because of its inherited security feature because of it design. 

NSX is a network virtualization platform and is able to automate, provision and managed network connectivity in a data center.  With NSX there are 3 levels of security that can be accomplished:

1. Isolation
2. Segmentation
3. Advance Segmentation with 3^rd party security partners

Isolation
In traditional network, Access Control List (ACL) is used for isolation.  With a virtualized network, the virtual network is by default isolated from the physical network.  Each virtualized network are also being isolated with one another.  This follows the zero trust principle a the virtualized network level.

Segmentation
In NSX, there is a concept of micro-segmentation.  In the traditional network segmentation is done through VLANs.  With a virtualized network, segmentation is not limited to a VLAN but can be fine tuned to smaller group of virtualized resource or even to an individual virtual machine.  In fact, as this will be explain again later in this post is that micro-segmentation is how VMware achieved the zero trust security principle.

Advanced Segmentation with 3rd party security partner
With service chaining, NSX in a virtualized network can direct the data traffic to 3rd party security appliances for deeper packet inspection and ACL parsing. 

The main idea for NSX to accomplish the zero trust security model is to have a distributed firewall (one on each ESXi host) and that traffic is inspected before being sent out to the traffic. Even if 2 VMs are connected to the same vSwitch, the distributed firewall is going to inspect the data traffic before sending to the destination VM. Without the distributed firewall, the 2 virtual machines connected to the same vSwitch are able to pass traffic between each other.

This diagram explain the concept that with the distributed firewall implemented at the hypervisor level, we can accomplished the zero trust security model where all traffic is being inspected and filtered according to the security policy defined:

image source: http://wahlnetwork.com/wn/wp-content/uploads/2014/08/nsx-firewall-yes.jpg

Cisco
Cisco's Application Centric Infrastructure (ACI) supports the concept of this Zero Trust security model.

As the name of this feature suggests it is all about - Application.

Traditional network security is network based, ACI decouples the security policy and segmentation from the network and defined "application friendly" policy model.  Security policy model in ACI is not only MAC address and IP address or its port number.  In ACI the security policy is defined by:

1. Endpoint Groups (EPG)
2. Policy Contract
3. Application Network Profile

image source: https://blogger.googleusercontent.com/img/b/R29vZ2xl/AVvXsEhW0Iql3smClfr6IFq9HOoNyFuW3HSslh5GjTGqyhkuZif90vZW8xjGCEMAMKErB3L6FfM8mnWrameSER9ChsJVQ2LKTLEZLR6JfR9Ko_PMuzftb21dZxObYRIfLh9WVjf4lD7052xXweE/s1600/cisco_aci_PolicyModelForSecurity.jpg

Endpoint Group

Devices with a common policy is put together as a group. It can be based on application friendly attributes such as OS, patch level, application type, application component or function.  Endpoint Group once created can be used to define security zones, trust boundaries or risk profile.  In ACI the default is no trust.

Policy Contract
The contract defines how data traffic is delivered between Endpoint Groups (EPG). This is is how the security rules are applied to devices regardless of where they are. In a virtualized environment, virtual machine migration is common. This contract defines filters and any associated action.  This is similar to our traditional firewall rules which based on the 5 tuples.  Policy contract enforcement for Endpoint Groups can be unidirectional or bidirectional.

Application Network Profile
In the diagram above this is stated as Service Chains.  Service chaining is a concept in which it defines the flow of the data traffic from one network service to another service.  Service chaining is a hot and important topic for Network Function Virtualization (NFV).

Trust and no trust
I believe the networking industry is catching up with the server and storage virtualization technology.  In a network we should trust no one but in our daily life we should have a certain trust level to other people that we come into contact with.  Everyday we are creating and updating out "Human Centric Profile" as to who and how much we can trust the people we know.

Reference:"Cisco ACI Security: A New Approach to Secure the Next-Generation Data Center." Cisco. N.p., n.d. Web. 13 Mar. 2015.
Egy, and White. Data Center Micro-Segmentation (n.d.): n. pag. Web.

OpenDaylight - an alternative to VMware NSX and Cisco XNC.

Before I worked on this blog post, I always think that OpenDaylight is a SDN Controller.  While it is correct to say that OpenDaylight is a SDN controller, it is in fact an Open Source project in which the controller is the core functionality.  In OpenDaylight wiki page, we can see a list of projects that is under the umbrella of OpenDaylight Project from the Linux Foundation. 

OpenDaylight Project is described by this web site as:

The OpenDaylight Project is a collaborative open source project that aims to accelerate adoption of Software-Defined Networking (SDN) and Network Functions Virtualization (NFV) for a more transparent approach that fosters new innovation and reduces risk. Founded by industry leaders and open to all, the OpenDaylight community is developing a common, open SDN framework consisting of code and blueprints

According to Neel Jacques, executive director of OpenDaylight:

“The OpenDaylight community has taken on the monumental task of bringing together all the disparate technologies, thoughts and ideas around SDN and forming it into a cohesive platform. The community has made amazing progress in a short amount of time as you can see in this second release which integrates more functionality, apps and use cases. Helium brings us one step closer to having one common platform the entire industry can standardize on.”

OpenStack name its release by city or street name.  OpenDaylight name its release by elements in the Periodic Table.  The first release was Hydrogen and the second release was Helium.  The OpenDaylight Helium release was available for download as of September 29, 1014.

Image source: https://www.sdncentral.com/wp-content/uploads/2014/09/opendaylight-project-helium-diagram.jpg

From OpenDaylight’s announcement “OpenDaylight paves way for innovation in SDN with latest open source software release”, I have summarized what’s new in the Helium release:

• 11 new protocols, applications and technologies
• New User Interface
• Simpler and customizable installation process
• User can build on-demand combinations of components and features to customize their solutions
• Open vSwitch Database Integration Project that provides technology preview of advanced OpenStack features
• High Availability
• Clustering
• Security
• OpenFlow Table Type Patterns
• Service Function Chaining

There are 3 things that I would like to highlight and comment:

1. Apache Karaf

One good feature for the Helium release is that user can build on-demand combinations of components and features for OpenDaylight.  This is done by Apache Karaf which is a small OSGi (Open Service Gateway initiative) based run time lightweight container where different components and applications can be deployed.  The best feature in my opinion is the ability to check for component dependencies.  Remember the days how we install packages onto a Linux system before apt-get or yum?

2. Integration with OpenStack

I believe for OpenDaylight to have more integration with OpenStack will entice more commercial IT vendors to embrace OpenDaylight.  Both VMware and Cisco announced integration with OpenStack and there are quite a few companies such as Rackspace, Metacloud, Mirantis, Cloudscaling, IBM and Red Hat that provide value added and easy installation for the Open Source OpenStack.

Work is done for the OVSDB (Open vSwitch Database) driver to provide features such as:

• Distributed L3 Forwarding
• Distributed ARP handling
• Security Group
• Load Balancing as a Service
• Firewall as a Service

This makes OpenDaylight an attractive choice as a OpenStack Neutron back end.  Also, there is a new feature to provide VLAN networking in addition to the tunnel-based networking option when a virtual network is created in OpenStack Neutron.

Interface with OpenStack with Keystone via the OpenDaylight AAA project is a big step forward between OpenDaylight and OpenStack integration.

3. Security

In the Hydrogen release, there is the Defense4All project for mitigating Distributed DoS attacks.  In the Helium there are 2 new projects for security:

1. Authentication, Authorization and Accounting (AAA)
2. Secure Network Bootstrapping Infrastructure (SNBi) project.

In this article, author Sean Michael Kerner stated that “Security is a particular area of focus in the Helium release”.

Authentication, Authorization and Accounting

AAA has been around for some time and is a popular and widely used security architecture where user’s credential is authenticated and based on the outcome of the authentication process, access for resources are granted/authorized while accounting will record this process so as to provide an audit trail.

According to a Red Hat blog, AAA project provides:

1. The ability to provide fine-grained permissions for resource usage
2. The ability to share an external identity service with other platforms

In our case as mentioned on the previous section, the external identity service will be OpenStack Keystone.

Secure Network Bootstrapping Infrastructure

This feature helps to solve the problem of having to manually distribute keys for the different networking device to communicate with each other. 

This page has more information about how this works.

Conclusion:

Besides, OpenStack integration and security both High Availability and clustering are important feature for the enterprise. OpenDaylight Helium is a big step forward from the Hydrogen release and hopefully more vendors will embrace OpenDaylight and provide this as an alternative to 

• Cisco - Extensible Network Controller
• HP - Virtual Application Networks (VAN) Controller
• NEC - ProgrammableFlow PF6800 Controller
• Nuage Networks - Virtualized Services Controller  
• VMware - NSX Controller

A Quick Look at the VXLAN's specification - RFC 7348

VXLAN is being deployed in numerous production environments and is supported by quite a few networking equipment vendor as well as software vendors such as VMware and Red Hat.  Not until August 2014 that the specification is moved from IETF draft status to RFC 7348.

RFC stands for Request For Comments. As a software developer for networking equipment, I have to know the RFC in and out.  This RFC is the functional specification for the feature that I develop.  Test group will test the feature to make sure the feature function as what the RFC describes.

Today, let’s take a look at what is in RFC 7348.

The title of RFC 7348 is “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.”

RFC 7348 is relatively a short document compare to other RFC that specify other communication protocols.   The table of contents has 10 sections.  6 of them are standard sections for all RFC.  To break down this RFC to the core, we can look at the following sections:

• VXLAN Problem Statement
• VXLAN operation
• VXLAN Frame Format
• VXLAN Security Considerations

VXLAN Problem Statement

Basically RFC 7348 identified 3 problem areas in the networking infrastructure within the data center that VXLAN is designed to resolve.  Server virtualization in the data center works best in a flat layer 2 network.  Only until VMworld 2014 that VMware announced in vSphere 6 (as of this writing, it is still under beta) can support vMotion across vCenter and long distance.

Limitations Imposed by Spanning Tree and VLAN Ranges

Spanning Tree is used to safe guard loops in a Layer 2 network.  When there is a loop in a Layer 2 network, it will cause a broadcast storm and the network will be in-operable.  Spanning Tree Protocol is designed and used to protect the Layer 2 network.  While it is able to protect the Layer 2 network the price to pay is there are links in the network is not used as well as multipathing not an option in the network design.  Both of these problems ties back to ROI (Return On Investment) not to a point where it should be.

Another problem is that the VLAN ID is a 12-bit field thus causing data center in flat Layer 2 network limited to only 4094 VLANs.

Multi-tenant Environment

Multi-tenant is a virtualized data center is a definite requirement.  Tenant isolation is an absolute requirement.

In a Layer 2 network VLAN is a popular way to achieve tenant isolation.  The number of VLANs limited to 4094 as described in the previous section is a major stumbling block to data center design.  When each tenant requires to have more than one VLAN as in the example of a 3-tier web application, the number of VLANs available in a flat Layer 2 network become a more limited and yet required resource.

A layer 3 network is another possible way of network isolation for multi-tenant but it has it limitation.  This will limit each tenant to have unique IP subnets.  Also, this Layer 3 isolation solution limits user not able to use Layer 2 or non Layer 3 protocols for inter VM communication.

Inadequate Table Sizes at ToR Switch

The use of ToR switch to connect servers on a rack is a common data center design.  With multiple virtual machines running on the virtualized servers, the number of MAC address that a ToR switch that has to learn.  As the number of virtual machine that a virtualized server can host is increasing the MAC address table used by the ToR switch becomes inadequate

VXLAN Operations

Section 4 of RFC 7348 describes the operation of VXLAN.  As indicated in the title of this RFC, VXLAN is a framework for overlaying virtualized Layer 2 networks over Layer 3 networks.  It is designed specifically to overcome the problems that we face in the data center.

VXLAN is meant to extend a Layer 2 network over a Layer 3 network by the use of tunneling technology - encapsulating an UDP packet over the original Layer 2 frame.

RFC 7348 outline the following rules:

• Each overlay is termed a VXLAN segment.
• Only VMs within the same VXLAN segment can communicate with each other
• Each VXLAN segment is identified by a 24-bit segment ID (VNI).
• VNI identifies the scope of the inner MAC frame originated by the individual VM
• VNI is an outer header that encapsulates the inner MAC frame originated by the individual VM.
• VXLAN segment and VXLAN overlay network are interchangeable in the RFC.
• VXLAN tunnels are stateless connection between 2 end points.
• Each end point is called a VXLAN Tunnel End Point (VTEP)
• VTEP can be implemented on a virtual switch, physical switch or physical server either on hardware or software.
• Use of data plane learning.
• Multicast is used for carrying unknown destination, broadcast and multicast frames (BUM traffic).
• VTEPs MUST NOT fragment VXLAN packets.

The last 3 points are worth looked into a little bit more.

Data Plane Learning

Data plane learning means there is no control plane for VXLAN. Not until now that I realize I never truly understand what a control plane is.  Often time people will say generically that SDN is the separation of the control and data plane.  I work on networking area for almost 20 years and mostly Layer 2 networking features.  I say mostly because I worked on UDP Relay Agent and DHCP Snooping in which I have to know a little bit of IP forwarding.  Data plane is the forwarding operation of networking equipment.

So what exactly is a control plane?  For any network engineers working on Layer 3 networks this is very obvious to them.  Control plane is a Layer 3 networking concept.  BGP is an example of a control plane protocol.  Routers exchange routing information via the control plane.

VXLAN uses data plane learning just like the source learning on Ethernet (Layer 2) switches.  VTEP is responsible for learning the virtual machine’s MAC address and associate this with VXLAN segment/VXLAN Network Identifier.  This learning process is very important to the efficiency of the operation of VXLAN networks.

Multicast for BUM traffic

VXLAN is to extend Layer 2 segments to other data center.  In a Layer 2 network, unknown destination, broadcast and multicast frames are flooded to all the devices on the same broadcast domain.  With VXLAN, RFC 7348 specifically spell out IP multicast is used for sending BUM traffic to other VTEPs of the same VXLAN segment. 

While this works perfectly in the functional level, all Layer 3 network engineers always try to avoid IP multicast.

Cisco Nexus 1000V has Unicast-ONLY VXLAN and MAC Distribution Mode.  In the MAC Distribution Mode, there is a centralized controller.  This is same as introducing a control plane for VXLAN.

VMware NSX has its NSX Controller.  When a VM is provisioned, it will register itself to the NSX controller.  ARP request from the VMs are sent to the controller where the controller is aware of the MAC address and VTEP/VNI association. This again is introducing the control plane for VXLAN.

To work with other VTEPs, IP multicast still needs to be used.

RFC 7348 suggests to use bidirectional IP multicast protocol such as PIM-SM to build efficient multicast forwarding trees.

VTEPs MUST NOT fragment VXLAN packets

Due to encapsulation, VXLAN adds an extra 50 Bytes of overhead.  The “MUST NOT” is written in bold in the RFC.  This requirement has huge implication the MTU of the underlay Layer 3 network.  MTU for Ethernet v2 is 1500.  VMware recommend setting the MTU size to 1600 or to use jumbo frame option end to end. 

I know of an installation that ran into MTU problem and end up not deploying VXLAN.

VXLAN Frame Format

Section 5 of RFC 7348 details the frame format of VXLAN.  As a developer of network engineer that needs to troubleshoot VXLAN problem needs to know this frame format very well.

I believe WireShark is able to decode VXLAN traffic.  UDP port 4789 is assigned for VXLAN traffic.

VXLAN Security Considerations

While security consideration is a standard section for RFCs, it is also worth looking into. 

Quoting directly from the RFC:

Traditionally, Layer 2 networks can only be attacked from 'within' by rogue end points -- either

• by having inappropriate access to a LAN and snooping on traffic,
• by injecting spoofed packets to 'take over' another MAC address, or
• by flooding and causing denial of service. 

VXLAN increases the attack surface for these kinds of attacks.

While not going into detail, the security consideration section of this RFC suggests the following ways to safe guard VXLAN networks.

·     Continue to use the traditional way of mitigating rogue end points attack by limiting the management and administrative scope of who deploys and manages VM/gateways in a VXLAN environment.

• Use of 802.1X for admission control for individual end points.
• Use of 5-tuple-based ACL.
• Use of IPsec to authenticate and optionally encrypt VXLAN traffic.
• Use of designated VLAN for VXLAN traffic.
• Use of secure method on the management plane of the VTEP.

This summarize RFC 7348.