VMUG - a platform for us to learn from each other.

This week I do not have any technical information to share because I have to prepare for the Southern California VMUG Conference that was held on Sept 24, 2014.

I think VMUG is a good platform for us to learn from each other.  I always like to say “I know some and you know some, let’s share what we have”.

 

Southern California VMUG Conference is a one-day event and we are fortunate to have Scott Lowe to be the Keynote speaker.   

Scott is a well-known author, speaker, and blogger.  His blog is here which has tons of useful information.  Scott said that he did not blog as much as before because of his busy schedule at VMware.  Scott is a very dynamic speaker.  A seasoned IT guy told me even if Scott talks about XML(a very boring subject), he will still listen with eagerness.

Scott has a slide to introduce himself.  This slide has his name, him being a VCDX, his twitter handle, blog URL and interestingly also include his life guiding principle – Col 3:17 NIV. 

The title of Scott’s keynote is “Closing the Cloud Skills Gap”.   The basic idea is that we need to venture out to sharpen skill to stay relevant.  According to IDC there will be 7 million cloud related jobs worldwide in 2015.

Scott asked what kind of skills that we are working on.  There were many answers from the audience.  Some say Docker, some say OpenStack and so on.  All the answers are technical skills.  Scott show us this slice:

This got us all by surprise.  I have never thought of this before.

The top 5 job skills that are needed in a cloud computing environment are:

1. Risk Management
2. IT service management
3.  Project/program management
4.  Business-IT alignment
5. Technical skills in cloud implementation

As you can see from the list technical skill that we got the answer from the audience is only the 5^th in the order.  Business leaders will not care if OpenStack or CloudStack is used.  As long as the ROI (Return On Investment) is met or business goal is met the tool used is not important to the business leader.

The cloud industry needs someone to bridge the gap between the technology and the businesses leaders.

After the keynote, there are other breakout sessions and the agenda can be found here.

After lunch was the vExpert panel and various things were discussed.  The hottest topic is EVO:rail and the hyper-convergence platform.  One panelist mentioned that Hyper-convergence is only good for SMB. One of the panelists Alastair Cooke mentioned that we need to understand the application and the work load before choosing a hyper convergence platform.  VVol was another topic being discussed.  When I looked up VVol I was surprised that this was around in 2012.  You can find a good blog about VVol here.

After the vExpert panel was other breakout sessions.

This year I did not attend any breakout sessions because I was fortunate enough to have 2 vBrownBag presentations at the conference.  I like the idea of sharing what we know. Check out my vBrownBag presentations on VXLAN and Virtual Design Master.

For me twitter is also one good platform to share ideas besides technical conferences.

After the conference there were parties hosted by Veeam at Marriott where the conference is held and Nutanix at Morton's Steakhouse.  I met different people at both parties and had good conversations.  Sometimes you can learn more by talking to different people than going to technical sessions in a conference. 

The sliders at Morton’s were very good.  I wish they have draft beers also.

A Quick Look at the VXLAN's specification - RFC 7348

VXLAN is being deployed in numerous production environments and is supported by quite a few networking equipment vendor as well as software vendors such as VMware and Red Hat.  Not until August 2014 that the specification is moved from IETF draft status to RFC 7348.

RFC stands for Request For Comments. As a software developer for networking equipment, I have to know the RFC in and out.  This RFC is the functional specification for the feature that I develop.  Test group will test the feature to make sure the feature function as what the RFC describes.

Today, let’s take a look at what is in RFC 7348.

The title of RFC 7348 is “Virtual eXtensible Local Area Network (VXLAN): A Framework for Overlaying Virtualized Layer 2 Networks over Layer 3 Networks.”

RFC 7348 is relatively a short document compare to other RFC that specify other communication protocols.   The table of contents has 10 sections.  6 of them are standard sections for all RFC.  To break down this RFC to the core, we can look at the following sections:

• VXLAN Problem Statement
• VXLAN operation
• VXLAN Frame Format
• VXLAN Security Considerations

VXLAN Problem Statement

Basically RFC 7348 identified 3 problem areas in the networking infrastructure within the data center that VXLAN is designed to resolve.  Server virtualization in the data center works best in a flat layer 2 network.  Only until VMworld 2014 that VMware announced in vSphere 6 (as of this writing, it is still under beta) can support vMotion across vCenter and long distance.

Limitations Imposed by Spanning Tree and VLAN Ranges

Spanning Tree is used to safe guard loops in a Layer 2 network.  When there is a loop in a Layer 2 network, it will cause a broadcast storm and the network will be in-operable.  Spanning Tree Protocol is designed and used to protect the Layer 2 network.  While it is able to protect the Layer 2 network the price to pay is there are links in the network is not used as well as multipathing not an option in the network design.  Both of these problems ties back to ROI (Return On Investment) not to a point where it should be.

Another problem is that the VLAN ID is a 12-bit field thus causing data center in flat Layer 2 network limited to only 4094 VLANs.

Multi-tenant Environment

Multi-tenant is a virtualized data center is a definite requirement.  Tenant isolation is an absolute requirement.

In a Layer 2 network VLAN is a popular way to achieve tenant isolation.  The number of VLANs limited to 4094 as described in the previous section is a major stumbling block to data center design.  When each tenant requires to have more than one VLAN as in the example of a 3-tier web application, the number of VLANs available in a flat Layer 2 network become a more limited and yet required resource.

A layer 3 network is another possible way of network isolation for multi-tenant but it has it limitation.  This will limit each tenant to have unique IP subnets.  Also, this Layer 3 isolation solution limits user not able to use Layer 2 or non Layer 3 protocols for inter VM communication.

Inadequate Table Sizes at ToR Switch

The use of ToR switch to connect servers on a rack is a common data center design.  With multiple virtual machines running on the virtualized servers, the number of MAC address that a ToR switch that has to learn.  As the number of virtual machine that a virtualized server can host is increasing the MAC address table used by the ToR switch becomes inadequate

VXLAN Operations

Section 4 of RFC 7348 describes the operation of VXLAN.  As indicated in the title of this RFC, VXLAN is a framework for overlaying virtualized Layer 2 networks over Layer 3 networks.  It is designed specifically to overcome the problems that we face in the data center.

VXLAN is meant to extend a Layer 2 network over a Layer 3 network by the use of tunneling technology - encapsulating an UDP packet over the original Layer 2 frame.

RFC 7348 outline the following rules:

• Each overlay is termed a VXLAN segment.
• Only VMs within the same VXLAN segment can communicate with each other
• Each VXLAN segment is identified by a 24-bit segment ID (VNI).
• VNI identifies the scope of the inner MAC frame originated by the individual VM
• VNI is an outer header that encapsulates the inner MAC frame originated by the individual VM.
• VXLAN segment and VXLAN overlay network are interchangeable in the RFC.
• VXLAN tunnels are stateless connection between 2 end points.
• Each end point is called a VXLAN Tunnel End Point (VTEP)
• VTEP can be implemented on a virtual switch, physical switch or physical server either on hardware or software.
• Use of data plane learning.
• Multicast is used for carrying unknown destination, broadcast and multicast frames (BUM traffic).
• VTEPs MUST NOT fragment VXLAN packets.

The last 3 points are worth looked into a little bit more.

Data Plane Learning

Data plane learning means there is no control plane for VXLAN. Not until now that I realize I never truly understand what a control plane is.  Often time people will say generically that SDN is the separation of the control and data plane.  I work on networking area for almost 20 years and mostly Layer 2 networking features.  I say mostly because I worked on UDP Relay Agent and DHCP Snooping in which I have to know a little bit of IP forwarding.  Data plane is the forwarding operation of networking equipment.

So what exactly is a control plane?  For any network engineers working on Layer 3 networks this is very obvious to them.  Control plane is a Layer 3 networking concept.  BGP is an example of a control plane protocol.  Routers exchange routing information via the control plane.

VXLAN uses data plane learning just like the source learning on Ethernet (Layer 2) switches.  VTEP is responsible for learning the virtual machine’s MAC address and associate this with VXLAN segment/VXLAN Network Identifier.  This learning process is very important to the efficiency of the operation of VXLAN networks.

Multicast for BUM traffic

VXLAN is to extend Layer 2 segments to other data center.  In a Layer 2 network, unknown destination, broadcast and multicast frames are flooded to all the devices on the same broadcast domain.  With VXLAN, RFC 7348 specifically spell out IP multicast is used for sending BUM traffic to other VTEPs of the same VXLAN segment. 

While this works perfectly in the functional level, all Layer 3 network engineers always try to avoid IP multicast.

Cisco Nexus 1000V has Unicast-ONLY VXLAN and MAC Distribution Mode.  In the MAC Distribution Mode, there is a centralized controller.  This is same as introducing a control plane for VXLAN.

VMware NSX has its NSX Controller.  When a VM is provisioned, it will register itself to the NSX controller.  ARP request from the VMs are sent to the controller where the controller is aware of the MAC address and VTEP/VNI association. This again is introducing the control plane for VXLAN.

To work with other VTEPs, IP multicast still needs to be used.

RFC 7348 suggests to use bidirectional IP multicast protocol such as PIM-SM to build efficient multicast forwarding trees.

VTEPs MUST NOT fragment VXLAN packets

Due to encapsulation, VXLAN adds an extra 50 Bytes of overhead.  The “MUST NOT” is written in bold in the RFC.  This requirement has huge implication the MTU of the underlay Layer 3 network.  MTU for Ethernet v2 is 1500.  VMware recommend setting the MTU size to 1600 or to use jumbo frame option end to end. 

I know of an installation that ran into MTU problem and end up not deploying VXLAN.

VXLAN Frame Format

Section 5 of RFC 7348 details the frame format of VXLAN.  As a developer of network engineer that needs to troubleshoot VXLAN problem needs to know this frame format very well.

I believe WireShark is able to decode VXLAN traffic.  UDP port 4789 is assigned for VXLAN traffic.

VXLAN Security Considerations

While security consideration is a standard section for RFCs, it is also worth looking into. 

Quoting directly from the RFC:

Traditionally, Layer 2 networks can only be attacked from 'within' by rogue end points -- either

• by having inappropriate access to a LAN and snooping on traffic,
• by injecting spoofed packets to 'take over' another MAC address, or
• by flooding and causing denial of service. 

VXLAN increases the attack surface for these kinds of attacks.

While not going into detail, the security consideration section of this RFC suggests the following ways to safe guard VXLAN networks.

·     Continue to use the traditional way of mitigating rogue end points attack by limiting the management and administrative scope of who deploys and manages VM/gateways in a VXLAN environment.

• Use of 802.1X for admission control for individual end points.
• Use of 5-tuple-based ACL.
• Use of IPsec to authenticate and optionally encrypt VXLAN traffic.
• Use of designated VLAN for VXLAN traffic.
• Use of secure method on the management plane of the VTEP.

This summarize RFC 7348.

VMware NSX enables Software Defined Security

A major reason used by user for not moving to the cloud is – SECURITY.  

 My opinion on this is that the main reason that the cloud is not as secure is because the mode of operation in the cloud is different thus causing the way to secure the cloud is different from the way one would secure the on-premises data center.  User that deploys cloud infrastructure must find new ways to implement security.  Security tools to secure the cloud is also different than the traditional on-premises data center.

Most people think of NSX as a product for Network Virtualization.  In fact NSX is also able to provide security to the virtualized network.  I do not know the business structure of VMware but I know that there is a NSBU – Networking and Security Business Unit.  With this business unit, we can see that VMware is putting networking and security together (from engineering and marketing’s point of view).

In VMware’s blog there is an article saying security is in fact a hidden gem of NSX.  In this article it explains how NSX can provide perfect security for a virtualized network.  This article stated that the following security features are inherent from how NSX operates:

• Isolation and multi-tenancy
• Segmentation
• Service insertion, chaining and steering

 Before we go on I think there are a few terminologies that we need to be clear of.

Isolation

All virtual networks are isolated from each other unless a router is explicitly configured to connect them together.  Virtual network are also isolated from the underlay physical network.  This is what I have written on my last blog post in which VMware and HP is in a joint development for orchestration and automation of both virtual and physical network.

Isolation is a good way to provide security.

Segmentation

It seems to me isolation and segmentation is interchangeable.  Often times we use segmentation to provide isolation.  Apparently, VMware is looking at isolation and segmentation as 2 different things.  It seems that isolation is when we look at the network itself while segmentation, VMware is looking at the application or tier that is utilizing the network.

Firewall is used allow or deny traffic flow between segments.

Network Services

In this context, network services refers to the Layer4-7 services such as DHCP, DNS, firewall, load balancer, IPS/IDS that are important to the operation of a network.

Service insertion, chaining and steering

These terms are fairly new to me and I have to do some research on them.  I found an article that describe these 3 terms and I quote from this article:

• Traffic Steering: directing and delivering traffic (flows/packets, tagged or otherwise) from one processing point to another
• Service Insertion: addition of some form of processing (terminated or mirrored,) delivered as a service, that is interposed dynamically between processing points
• Service Chaining: chaining (serialized or parallelized) and insertion of services with other services.

In summary these 3 terms used in the context of NSX, it refers to how we orchestrate or automate the various network services in a virtual network created by NSX.

Micro-segmentation

This term is seen a lot in NSX articles especially related to 3^rd party security vendors.  Micro-segmentation is the same as zero-trusted network.  What is a zero-trusted network?  Traditionally the network security perimeter is at the edge of the network. Once a user is gained access to a network he/she is free to move around.  Zero-trusted network means in a network nothing can be trusted event the east-west traffic between servers which is within the data center perimeter.  With server, storage and network virtualization or with the cloud this perimeter becomes burry or not easily defined.

NSX – Software Defined Security

At the core of NSX being a software defined security provider is the distributed vSwitch.  Security related network services are implemented at the distributed switch and thus with network service insertion, chaining and/or steering network security is done before it even enters the virtual network.  Virtual machines can be moved around and the security policy that is defined for each VM is applied wherever the VM is migrated to.

NSX is working with 3^rd party security vendors to implement network security.  There are cases that a physical security device is necessary to be deployed.  NSX provides API for 3^rd party vendor to integrate with NSX.  This page describes the “Network and Security Extensibility Framework API” in which NSX platform can be extended for various types of services e.g.

• Network Gateway services like ToR and EoR Fabric integration
• Network Security Services like Firewall and IDS
• Security Services like Anti-Virus, Vulnerability management
• Application Delivery services like L4-L7 Load Balancer

 image source: https://developercenter.vmware.com/nsx-for-vsphere/nsx-extensibility-framework

Security is my passion.  I will be looking at all the security features of NSX in the coming days.  Stay tuned.

VMware NSX and HP VAN SDN Controller - a value added solution

I am a software developer and for a long time when I implement a feature SNMP support for the feature is always done last and sometime delayed to a future release.  Back then, functionality of the feature was the most important thing in my mind.  As I venture into server virtualization, I start to see the perspective of a system administrator or a network administrator.  Monitoring and reporting is very important and sometimes more important than a feature set delivered by the vendor.  Of course maintaining the five 9s of uptime is always the highest priority.

These days I am looking into NSX.  Derek Seaman has a blog post on “VMworld 2014: Future Direction of NSX” where he summarize session NET1674.  Chris Wahl has a post on NSX 6.1 (4.2 for the NSX Multi-hypervisor).

I came across an article with the title “The industry’s east-west federated solution”.

After reading the title, 2 questions come to my mind.  What is:

• East-West traffic?
• The solution for what problem?

East-West Traffic

In simple term, east-west traffic refers to the traffic between servers in a data center.  There is another term – north south traffic and this refers to the traffic between clients and servers in a data center.  Traffic from client to the server will be northern bound traffic while traffic from server to the client is called southern traffic.

Image source: http://regmedia.co.uk/2011/08/15/xsigo_east_west.jpg

A Solution to a problem

First of all what is this solution?  What problem is this trying to solve?  The problem is that virtual network has not visibility to the physical network.  For a user or virtualization/network administrator this is not a big problem.  This will just be an inconvenience because there are tools to monitor, view and debug the virtual and physical network individual.  At the end of the day, user still has the tools to perform their job.

From the perspective of automation or orchestration, this is a big problem. 

A Federated Solution

Why federated?  The solution calls for a federation of 2 products to solve the problem.  The 2 products are VMware’s NSX and HP’s VAN Controller which one of HP’s SDN solution.

VMware NSX

Tons of information can be found about VMware’s NSX.  Not too many people have the luxury of having the opportunity to play around with this product.  In VMworld 2014, VMware announced a new certification track for Network Virtualization which has high NSX concentration.  Information on this certification track can be found here.

Image source: http://blogs.vmware.com/education/files/2014/08/VMW_14Q3_Cert_Roadmap_Network_Virtualization_R3.jpg

As of this writing, I see on twitter that a few people had sit for the VCP-NV certification and a few already achieved VCDX-NV status and they we being introduced on VMworld 2^nd day Keynote session.  The VCIX-NV is not available yet and this should be equal as getting a VCAP level certification and one of the requirements to sit for this certification is a person is a CCNP or CCIE holder.  This requirement shows that VMware is trying to get Cisco certification holders to get into VMware’s SDN solution.

When we look at VMware NSX, we can approach this from 2 angels.  Its capabilities and its components

VMware’s NSX has the following Capabilities:

• Logical Switches
• Logical Routers
• Logical Firewall
• Logical VPN
• Logical Load Balancer

VMware’s NSX has the following components

• NSX Manager
• NSX vSwitch
• NSX Controller
• NSX Edge

All the NSX components can be configured using vSphere Client, VMware command line interface (CLI) and REST API. 

The REST API is essential for 3^rd party software entities to interface with NSX.

HP’s SDN Technology

This page contains a very comprehensive description of HP’s SDN technology.  One of HP’s SDN offering is HP Virtual Application Network SDN Controller.  It is described as a control point in an OpenFlow-enabled network, simplifying management, provisioning, and orchestration. This enables delivery of a new generation of application-based network services and provides open application program interfaces (APIs) that allow third-party developers to deliver innovative solutions to dynamically link business requirements to network infrastructure via either custom Java programs or general-purpose RESTful control interfaces. HP VAN SDN Controller Software is designed to operate in campus, data center, or service provider environments

This HP VAN SDN controller is recommended to run on an Ubuntu 64-bit server with 12.04 LTS.  It requires a back end database.  The recommended database platform is PostgreSQL 9.1.  OpenJDK 7 JVM is also required.

Integrating VMware and HP’s solution

The 2 companies has entered into a joint development of a solution that will combine the strength of the 2 SDN solutions that according to a white paper to provide customers unified automation and visibility of the physical and virtual data center networks, enabling  business agility and improving business continuity.

Image source: https://pbs.twimg.com/media/BwJ2YaFIEAItUY9.png:large

The integration of the 2 controllers is using OSVDB (Open vSwitch Database Management Protocol).  The HP controller is acting as an OSVDB server while the VMware NSX controller is acting as an OSVDB client.  Both controller are able to communicate with each other via the OSVDB federated API.

HP blog is saying the integration is at the control plan.  When I look into OSVDB, I believe this should be the combination of the management and control plan where OSVDB is for management similar to NETCONF and control plan is where the OpenFlow protocol resides in which is being use to program traffic flows.

This solution will be available in the 4^th quarter of 2014.