vSphere 6 – VMware is heading toward VM-aware

VMware vSphere 6 was released in early February 2015.  The must read “Mastering VMware vSphere X” book series for vSphere 6 is already available.  There are tons of blog posts on what is new in vSphere 6 available on the Internet that we can search for.  I had been busy with doing bug fixes for OpenStack right before and after the OpenStack Summit in Vancouver.  My second OpenStack bug fix was submitted upstream and merged last week. It is difficult to have the mind switch between technologies for me.  This is also why I had not published any new blog post lately.  I will try to blog about my experience being an open source code committer in the coming days. 

VMworld 2015 is coming and I predict that there will be lots of new product or feature announcements in the End User Computing area.  Before more new things to learn I have made up my mind to at least catch up with what is new in vSphere 6.

What's new in vSphere 6
The official “what’s new” information page from VMware listed the following:

Compute

•  Increase Scalability
• Expanded Support for new chips sets, devices, drivers and guest OS
• Support for NVIDIA GRID vCPU
• Instant clone

Storage

• VM-aware Virtual Volumes
• Storage Policy-Based Management

Network

• Per-VM Distributed vSwitch bandwidth reservation
• Multicast Snooping (IGMP snooping for IPv4 and MLD Snooping for IPv6)
• Multiple TCP/IP stack for vMotion

Availability

• vMotion Enhancements
• Replication-Assisted vMotion
•  Expanded support for Fault Tolerance (up to 4 vCPUs instead of just one)

Management

• Content Library
• Cross-vCenter Clone and Migration
• Enhanced User Interface

Duncan Epping (@DunccanYB) had a much more detailed summary post on this subject.  For anyone who is involved in VMware related technologies, it is highly recommended to visit his blog "Yellow-Bricks" regularly as there are lots of good contents. According to him the vVol (Virtual Volume) is the “flagship feature” of the vSphere 6 release. I totally agree with Mr. Epping. 

VM-aware is the trend

In fact, if you look at the list of “what’s new”, lots of the items are feature enhancements.  Originally when I write this post the title was “Catching up on what’s new in vSphere 6”.  As I dig into “what’s new”, I see that VMware is making its product VM-aware.  I think it will eventually be heading toward application aware.  I am not an expert in Cisco product (yet).  Cisco is marketing ACI – Application Centric Infrastructure which is also heading toward the same direction.  

“There is no new thing under the sun”, both VMware and Cisco and among some other companies are seeing the need to have the data center infrastructure to be application aware so that we can provide the intelligence to run the infrastructure more effectively.  Another important benefit for the infrastructure to be application aware is – SECURITY. 

At the end of the day, the ultimate goal of having a data center infrastructure is to run business application so that a business entity can earn money.  It is the application that we want it to run efficiently and securely.

To have the infrastructure VM-aware or even application aware, it must be agile so that it can react to the dynamic changes.  vMotion is one example of dynamic changes.

Policy
Software Defined Data Center (SDDC) is the first step of providing support for VM-aware infrastructure.  With software providing an abstraction level to all the elements of the data center, operators/administrators can automation changes as well as to define policies which are the rules of how things should happen according to specified characteristics of a virtual machine.  The technology is still advancing and we can look at the defined policies as being the intelligence of the infrastructure.  The entire data center infrastructure, be it storage or networking reacts to changes according to the defined policies.  One common theme about the characteristics of a policy is that it is “declarative”.  Policy being “declarative” only specifies the end result and not how to attain the result.

Industry Convergence

In the OpenStack world, VMware is investing heavily on a project call “Congress” while Cisco is investing on “Group based policy”, it is interesting to see how the IT industry converge into a common way of providing an infrastructure for business application to run both efficiently and securely.

Reference:
 "VMware Virtualization for Desktop & Server, Application, Public & Hybrid Clouds | United States." VMware Virtualization for Desktop & Server, Application, Public & Hybrid Clouds | United States. N.p., n.d. Web. 03 July 2015.

From DevOps to Puppet Part 2

In part 1, we have looked at DevOps, Configuration Management Tools and a little bit of Puppet.  In this post we will continue to look into other aspect of Puppet.

Declarative vs Imperative
One important thing to know about Puppet is that it is a Declarative Language.  What it means is that user only specify the end state of the system that it is trying to configure.  User does not need to know the system specific.  A Puppet manifest can be applied n number of times and if the end state is already in the desire state, nothing will happen.

Another equally popular Configuration Management Tool - Chef, however, is an imperative language where user write out how to achieve the end result.

Puppet Terminology
Puppet looks at everything as a resource where each resource describes some aspect of a system

• The block of Puppet code that describes a resource is called a Resource Declaration.
• Each resource has a type, a title and a set of attributes

Example of a resource will be

• Linux package such as MongoDB
• Files
• Users on a system
• Services such as /etc/host entries; network interface or Windows registers

Example of a resource declaration:

As we can see there are 3 "file" type but the title and attributes are different.  The first example creates a file /tmp/test1 with content "Hi'.  The second one creates a file /tmp/test2 and the security mode is set to 0644 and the third example creates a symlink /tmp/test3 which links to the file /tmp/test1

As mentioned before Puppet is a declarative language.  If the file /tmp/test1 already existed and with content "Hi", nothing will happen even if user execute the manifest 100 times because the desired end state is reached.

There are built-in types for Puppet:

image source: https://docs.puppetlabs.com/references/latest/type.html

If you do not find what you need from the built-in types, take a look a the Puppet Forge and chances are someone else had already did the job for you and had developed a type that you can use or similar to what your needs are.

A Puppet Manifest is a Puppet program with the .pp file extension that contains the resource declaration.These are some of the operation/command that we can have on a Puppet Manifest:

puppet parser validate configWebSrv.pp

puppet apply –-noop configWebSrv.pp

puppet apply configWebSrv.pp

puppet-lint configWebSrv.pp

The "noop" parameter is a very useful command where we can test out the manifest without actually performing the action and Puppet will respond with what is the outcome when this manifest is executed.

Puppet can operate on a standalone device or it can operate on a "Master and Agent" model where the Puppet agent will pull from the Puppet Master on a regular interval.  The default pulling interval is 30 minutes. For emergency application of new configuration, user can trigger the pulling and I believe Puppet Enterprise has the option to trigger the pulling on the console.

Manifest is complied into catalog which is in the format that the Puppet Agent understand.  Puppet will configure the resource according to the content of the catalog to the desired system state.

In a standalone model it works this way:

 image source: https://docs.puppetlabs.com/learning/images/manifest_to_defined_state_unified.png

In a "Master and Agent" model, the manifests are stored in the Puppet Master and periodically, the agent will poll from the Puppet Master if there is any configuration changes.  If necessary, we can trigger the poll manually instead of waiting for the next poll interval.

image source: https://docs.puppetlabs.com/learning/images/manifest_to_defined_state_split.png

The difference between a standalone and "Master and Agent" model is how the manifests are store and where the catalog is complied.

Currently under preview is the Puppet Server.  This is to replace the Puppet Master and is to have many advantages in this new model because of a complete re-design.

May be in Part 3 we can look into this new Puppet Server.

OpenStack Series: Part 20 - Group-based Policy for Neutron

Group-based Policy is a new feature in OpenStack and is still in its infant state.  There is a page on this subject in the OpenStack wiki but not much information on what this feature is about. 

For 4 years I have developed and enhanced a feature on the Alcatel-Lucent Enterprise switch for a policy based network access control for network traffic.  I have with 3 other engineers submitted a patent in this area waiting for approval.  This is why I am particular interested to look into this Group-base Policy that is in OpenStack.  Actually, this also made it way into the SDN Controller OpenDaylight project.

This feature is supported by quite a few IT equipment vendors such as Cisco, Alcatel-Lucent (unfortunately I am not with this division such that I can work on this interesting project), Big Switch Network, Juniper, IBM, Red Hat and even Intel.

The main goal for Group-based policy is to provide support and abstraction for the application that is running on the OpenStack Infrastructure.

The whole idea is to provide an abstraction layer for the application so that it does not need to know the detail of the how the infrastructure that it is running on.   It is to use a declarative language to capture the intent of the application.  At this time the Group-based Policy is mainly for Neutron but it can also be developed to be applied to compute and storage.

Note: Declarative is an important concept. It is to defined the desired end state of a server or an infrastructure.  The popular Configuration Management Tool Puppet is a declarative language.  I have talked about OpenStack Congress and it is also a declarative model.

Currently the Neutron API is powerful in providing an abstraction to provide a logical network.  However, there is one drawback - it is very network-centric and user has to be very knowledgeable in networking.  The API may be a set of powerful API for network engineer but it may be too much for an application developer or operator to handle.

According to the OpenStack Documentation, Group-base Policy is to provide a intent-driven declarative policy model that presents simplified application-oriented interfaces to the user.

Application-oriented - Does this sound familiar?  For me I immediately think of the Cisco version of SDN (Software Defined Network) - Application Centric Infrastructure (ACI).  Of course, the APIC driver is part of Neutron's ML2 mechanism module.  ACI by itself is a big topic to be discussed.  I will most likely look into this in the near future as part of my quest to the cloud related technology.  When I look into this subject of Group-based Policy, I find the most informative information comes from Cisco's blog and/or white paper.  I think Cisco contribute to this Group-based policy feature and is driving the advancement of this feature as well.

While I think even Cisco is driving the advancement and acceptance of Group-based Policy, it has it value and is not a feature that is pushed by a vendor for their sole benefits in this case Cisco and ACI.

Group-based Policy Architecture

image source: http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.doc/_jcr_content/renditions/white-paper-c11-733126_0.jpg

I found that this Cisco white paper is a very good description of the Group-base Policy's architecture.  There are 4 main concepts that are essential to Group-based Policy:

• Groups: network endpoints are put together in a group and the properties of the endpoints are treated as a whole.
• Policy Rule Set: This describe how the groups are connected together
• Policy Layering: Policy can be layered on top of each  other forming a new/combined policy.
• Network Service Chaining: This is an important concept in NFV (Network Function Virtualization). Group-base Policy allows applications to define their requirements on how the packet flows.

According to the Cisco white paper, this Group-base policy has the following advantages:

• Automation and security are much easier to implement through Group-Based Policy.
• Offers a naturally flexible and extensible framework for capturing the requirements of a virtual machine in a single location.
• Makes consistency easier to achieve because only one step - becoming a member of the group - is required to inherit multiple policies 
• Easy for application developers to use and offers them a simple way to describe application requirements 
• Offers a means for allowing operator and user requirements to coexist cleanly 

Group-based Policy and OpenStack

image source: http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.doc/_jcr_content/renditions/white-paper-c11-733126_2.jpg

Also, according to the same Cisco white paper, Group-based policy is supposed to fit into OpenStack in a "non-disruptive" layer.  I take it as not or minimum changes is need in OpenStack.  When we look at the diagram above Group Policy is the orange layer and conceptually this is how it fits into the OpenStack.  We can see the Horizon and Heat module along with the OpenStack CLI sitting on top for configuration (manual or via a Heat Template).  The next layer is the application in which all our focus are in to making the deployment of application as easy as possible without have to "worry" about the supporting infrastructure.  Below the Group Policy layer are the other various OpenStack projects. At this time it is mainly focused on Networking.

Group-based Policy and OpenStack Congress
On the SDN front, we see that VMware and Cisco are going the opposite direction.  OpenStack Congress is heavily driven by VMware while Group-based Policy is heavily driven by Cisco.  Will the same thing happen here where the 2 technologies are competing with each other?

It seems that Congress covers governance and compliance while Group-based policy is to capture the application's intent and to provide an abstraction layer.  These 2 things compliment each other and is providing different services to the OpenStack users.


Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service  
OpenStack Series Part 18: Network Function Virtualization in OpenStack 
OpenStack Series Part 19: Storage policies for object storage

Reference:
"GroupBasedPolicy." - OpenStack. N.p., n.d. Web. 09 Nov. 2014.