Monday, August 24, 2015

Microsegmentation – a perfect fit for Microservices security

On my last post, we have explored a new chapter in Docker networking. With the new and yet still under development libnetwork, Docker container is now able to take advantage of 3rd party networking and security solutions.  One of the security solution is Microsegmentation.

Microsegmentation is not a new concept but the implementation of Microsegmentation is not feasible until network virtualization had become mature.  Both VMware and Cisco offers Microsegmentation solutions.

What is Microsegmentation?

Segmentation is a security principle used to group entities within a network into one unit and to apply rules/polices to control the traffic in and out of the segment. Usually, this is done by a firewall.  Microsegmentation is to be able to provide a way to define rules/policies in a smaller granular way and sometimes as small as a Docker container.

Underlying Principles
An important principle for Microsegmentation is Zero Trust.  It is simply to say that within a network, nothing is trusted.  In the traditional IT security model, the assumption that attack to an IT infrastructure is from the "outside".  There is the familiar tools such as DMZ, Intrusion Detection/Protection System and the antivirus software that companies spend a lots of money on to stop hacker from entering into the system. Once inside the system, there is not much being done to check for the traffic inside the perimeter.

Perimeter based security measures are not enough for the modern day IT infrastructure especially when the work load is in the cloud in which the perimeter is not very well defined.  It is important to employ a Zero Trust model so as to provide the maximum level of security measure.

Major components for effective Microsegmentation
I read an article by Scott Lowe (I cannot find where it is.  Will put in the URL once I find it).  According to that article there are 3 main components that defines an effective Microsegmentation implementation
  1. Network independent policy definition
  2. Centralized policy definition repository
  3. Distributed policy enforcement
1. Network independent policy definition
Traditional firewall rules use the 5 tuples to define access or deny rules.  It is enough when service is run in the monolithic process.  To provide a more granular way for Microsegmentation, the rules and policies has to be more than just network attributes.  One example of network independent definition can be the type of OS kernel or even path level of the same OS.

2. Centralized policy definition repository
Virtual Machine and/or Docker containers can move around.  To be effective, Microsegmentation has to have the ability to define the security policy in a central location so that no matter where the segment is moved, the policy can be easily retrieved.

3. Distributed policy enforcement
The enforcement point of the security policy has to be as close to the source as possible so that it will not be a bottleneck when applying the security policy. 

Microservices opens up Security risk

Microservices architecture is perfect for cloud native application, agile and with the ability to scale in and out depending on the need.  On the other hand, Microservices architecture opens up security risk that needs to be addressed:
  • Frequent and short life span
  • Increase of East-West traffic
  • Services are not as isolated
Frequent and short life span
How is short life span a security risk? It is a security risk when the security rules cannot keep track on when and on which host machine the services are being deployed.  Over time the ACL which is a common form of security rule for segmentation can become unmanageable and thus create security risk. This problem is aggregated when the Microservices have frequent life span. A Docker container can be provisioned in a matter of millisecond.  We need to have the security measure to catch up with the pace otherwise there is going to create interval of security exposures.

Increase of East-West traffic
The traffic between client and server is defined as North-South traffic and traffic between servers are defined as East-West traffic.  Why would increase in east-west traffic present a new security threat?  Traditional security model has the assumption that attack to an IT infrastructure is from the "outside".  There are not much being done to stop hackers from hoping from server to server.  With the increase of East-West traffic, it amplifies this security risk that needs to be addressed.

Services are not as isolated
Microservices architecture is to break a monolithic service into smaller services.  In the monolithic model components of a service runs within the same process and it is much easier to provide service isolation.  Also the traffic to a monolithic service is much easier to control.  With Microservices, components of a service are not running as an individual process.  Individual processes that work together to perform a service needs to communicate with each other.  Traditional firewall rules uses the networking 5 tuples as the bases of the rules as to either allow or deny.  Now with Microservices, the use of networking attributes to setup up allow or deny rules are not enough thus making it difficult for the administrator of the IT infrastructure to use traditional firewall rules to provide service segmentation.

If Docker container is used to implement the Microservice architecture, we have the problem of all containers sharing the same OS kernel.  In this way, individual micro services are not truly isolated from each other.  Lots of development is done on Docker security but at least for now, it is a security risk that we have to address.

Microsegmentation and Microservices fits right in

Microsegmentation and Microservices is a perfect match.  How?

What Microsegmentation can offer for security is just what Microservices architecture implemented in the form of Docker containers needs. With Network virtualization as the foundation of Microsegmentation, the security policy is able to be ready to secure the fast provisioned Microservices.

The 3 major components of an effective Microsegmentation help to mitigate the security risk that Microservices opened up.

Network independent policy definition
In Microsegmentation, security rules are not limited to the traditional 5 tuples networking attributes and thus we can define a Docker container as a single segment and that the enforcement point of the policy is applied to the networking interface of the Docker container.  This can effective monitor and control the East-West traffic of the various Microservices.

Even if a single Docker container is compromised the hacker will not be able to break out from the Docker container because the security rule that is defined for a single Microservice will stop the unexpected traffic.  Depending on the implementation of the Microsegmentation, the Docker container once being found to have violated the defined security rule, it can be killed and being spawned back.

Centralized policy definition repository and distributed policy enforcement
Microservices architecture allows the application to be cloud native.  Common characteristic of a cloud native application is that they can be provisioned quickly and on-demand.  As these Microservices come and go they can be provisioned on different host machines.  For the purpose of balancing resources, these Microservices might be moved around different host machine during its life span.   Microsegmentation is able to adapt to dynamic and elastic nature of cloud native applications with distributed policy enforcement.  The policy defined in a centralized location by the IT infrastructure administrator is enforced regardless of where the Microservices is provisioned.

Monday, August 17, 2015

A New Chapter in Docker Networking

Docker networking is entering a new chapter.

Networking is one of the pillar for modern day IT infrastructure and lots of work are done by various networking equipment vendor to provide a stable and fast network.  Recently, there is also the movement of Software Defined Network (SDN) as well as the Network Function Virtualization (NFV). 

In the traditional client and server model, the traffic pattern is mostly "north-south traffic" (between the server and the clients). 

With Docker where most of the time it is being used to deploy Micro-Services, there is a need for the containers to talk to one another both within the same host or across multiple hosts.  This changed the traffic pattern and the demand for the network is changed to add "east-west traffic" (traffic between hosts).

Docker Inc has done a good job on Docker in packaging container but the networking support is a bit primitive. I had a blog post on Docker Networking options last year and before that I had another post describing what Docker container is and that VMware is not against but embracing this container technologies.   And of course there is Project Bonneville that is in technology preview state where VMware is making Docker containers to work just like a virtual machine in the vSphere environment so as to take advantage of the "enterprise ready" features of vSphere such as Distributed Resource Scheduler, vMotion and the benefit of the lightweight, fast provisioning characteristic of Docker container.

Native Docker Networking
  • On startup Docker creates a Ethernet Bridge docker0 on the Linux Kernel
  • docker0 creates a virtual subnet on the Docker host
  • Docker creates a pair of virtual Ethernet interface on each container
  • One of the Ethernet interface is the eth0 in the container
  • Another Ethernet interface will have a unique name in the form of veth* (e.g.vethABI3IC) and is bind to docker0
  • User can customize docker0
  • Advanced Docker networking can be found here
image source:

The native Docker networking was simple and is designed as a single-host solution. Native Docker networking does not scale well which is against Docker container use cases. 

Docker Networking from 3rd parties

As mentioned on my blog post from last year there are solutions/projects in development to solve or to improve Docker networking.  These solutions are:
  • Weave
  • Kubernetes
  • Flannel
  • Pipework
  • SocketPlane <-now part of Docker Inc.
For detail description of these solutions you can take a look at here or here

While these solutions are useful and has its use cases, they are all external to Docker.

Docker's latest Networking Solution
On April 30, 2015, Docker announced an open source project - libnetwork.

Libnetwork is an open source project and can be found in GitHub here.

This "libnetwork" is a library that can provide native support for Docker container and its function is to connect containers.  This library is written in the Go language.  According to GitHub, "libnetwork project will follow Docker and Linux philosophy of developing small, highly modular and composable tools that works well independently. Libnetwork aims to satisfy that composable need for Networking in Containers."

Libnetwork implements the Container Network Model is is the work of various networking partners of Docker Inc such as Cisco, IBM, Microsoft, Joynet, Rancher, VMware and Weave.

The most important aspect for libnetwork is that it uses a driver/plugin model.  In the pass, Docker networking is handle by libcontainer and Docker Engine and now with libnetwork it can provide a single interface via the form of an API. 

Container Network Model
This model has 3 main components:
  1. SandBox
  2. Endpoint
  3. Network
 image source:

This architecture diagram of Container Network Model is pretty self-explanatory.  Again, GitHub has good information about what these 3 elements are:

A Sandbox contains the configuration of a container's network stack. This includes management of the container's interfaces, routing table and DNS settings. An implementation of a Sandbox could be a Linux Network Namespace, a FreeBSD Jail or other similar concept. A Sandbox may contain many endpoints from multiple networks.

An Endpoint joins a Sandbox to a Network. An implementation of an Endpoint could be a veth pair, an Open vSwitch internal port or similar. An Endpoint can belong to only one network but may only belong to one Sandbox.

A Network is a group of Endpoints that are able to communicate with each-other directly. An implementation of a Network could be a Linux bridge, a VLAN, etc. Networks consist of many endpoints.

Why is libnetwork so special?

Libnetwork is indeed very special that I called this a new chapter for Docker networking.

We have seen that libnetwork provides a single interface for networking. The significant of a single interface is that libnetwork can be present a plugin for external networking solutions. This is similar to the Neutron project for OpenStack where 3rd party networking solutions can be use.  

Both VMware and Cisco has already jump into this band wagon with their respective NSX and ACI networking solution to provide a robust networking solution for mulit-host container communication.  

Beside a robust networking solution, being able to use 3rd party networking solutions is also able to provide Docker containers security and layer 4 - 7 network functions features such as firewall and load-balancer.  

Security is an important aspect for all deployment in any environment. Both VMware's NSX and Cisco's ACI implements Micro-segmentation which is to provide a distributed firewall with extended rules.  These extended firewall rules allows user to define security policies beyond the traditional network attributes based rules.  My next post will be on Micro-segmentation.

Note: libnetwork is still under heavy development and is listed as experimental in Docker 1.7.  Please check GitHub for the latest status as things are going in a fact pace.

"Docker/libnetwork." GitHub. N.p., n.d. Web. 17 Aug. 2015.

Friday, August 14, 2015

My Personal Theme for VMworld 2015

VMworld is said to be the number one IT conference in the world. For the pass few years I was able to attend various technology conferences such as Interop, Citrix Synergy, Puppet Conference and OpenStack Summit.  But never to VMworld.

Ever since I got involved with VMware related technologies I have always wanted to attend this conference.  This year I was fortunate enough to secure a ticket and finally I am able to attend this conference.

Starting last week I get promotional Emails from various vendors with information about their booth location and/or any sessions that they are involved in.  One Email caught my attention. The title of the Email is - "Anthony, your tips for VMworld 2015."  Well first of all the Email subject is personalized and then it addressed one of my needs and that is tips for attending this number one IT conference that everybody is speaking of and wanting to attend.  Of course, it has the usual promotional information and then at the end it talks about packing light and to bring a comfortable shoe. There is one last item that prompted me to write this blog post.

  • Get Out of Your Comfort Zone: Remember, VMworld is about new experiences, so make sure you meet new people and learn about new technologies from key sponsors, like Coho Data in Booth #1713.   

"Get out of our comfort zone" has been one of my career's guiding principle -
image source:

Under this tip, the promotional Email mentioned that "VMworld is about new experiences - so make sure you meet new people ...". Wow, this also align with the suggestion that Cody Bunch (@cody_bunch) told me about what my objective should be at VMworld.  Cody told me (on Twitter) to meet 3 new persons and to help 3 persons.  Helping people can be as simple as point out where the nearest bathroom is at the Moscone convention center. Well I am hoping that I can do better than that.

My personal theme for VMworld 2015 is - EXPERIENCE
  1. To experience VMware's technologies
  2. To experience the people of VMware
  3. To experience the VMware community

To experience VMware Technologies
There must be new announcement at VMworld about new product or direction of VMware and I predict that there will be announcement on VMware View.  Container will be another topic that VMware is going to announce on what they will expand on Project Bonneville.

On existing VMware technology, I am hoping to experience NSX and VMware Integrated OpenStack (VIO) in a deeper level.

To experience the people of VMware
Last year the attendance of VMworld 2014 was 22,000.  Who can or will I meet during the conference.  At PuppetConf 2013, I met Gabriel Chapman (@Bacon_Is_King) at a lunch presentation and he inspired me to start this journey to the cloud.  At this time I do not have a plan as to how to met people and establish connection.  To me just shake hand and say hi is not enough, I need to be able to establish connection and maintain contact even after VMworld 2015.

On Twitter, I got to interact with others from time to time and I am hoping that I can me them in person at VMworld.  I wonder if the conference will print out our Twitter handle. If they do, my handle is vCloudernBeer. I will be at the hang space more than going to the breakout sessions or Hands-on-Lab as those can be view or done online after the conference.

To experience the VMware community
There is the VMware User Group VMUG and I had been saying for a long time VMUG is the best place to learn, network and to share.  I will have booth duty at VMUG booth (VMUG has not given me the time slot yet) and I hope I can get more people to join this wonderful VMware community.  This is also one way to experience people of VMware. Come by and say "Hi" if you are attending VMworld.

Another strong VMware community is vBrownBag. We got to learn a lot from the rich content that vBrownBag offers not just on VMware but also on DevOps, OpenStack and other relevant IT subjects. I am fortunate enough to get a presentation slot at the vBrownBag TechTalk on "Micro-Segmentation - a perfect fit for Micro-Service security".  There are 2 reasons that I sign up for this TeckTalk.  The first is to improve my soft skills and the second reason is that I believe in sharing.  My motto is "I know some, you know some, let share what we know".  In this way we can all grow as a community.

Waiting to see what will happen
In 3 weeks, I will be able to blog about what or how I experience in these 3 areas so stay tuned.  For those who are going to VMworld and have read this post, let see if we can bump into each other and establish connections.

image source: