Showing posts with label Orchestration. Show all posts
Showing posts with label Orchestration. Show all posts

Saturday, July 18, 2015

Marriage made in heaven - Google + Project Magnum


On July 16, Google announced that it is becoming a sponsor to the OpenStack Foundation.  This announcement immediately generated a lots of excitement and this news is all over the social media.  Immediately lots of blogs and technical news outlets published articles on this announcement.

My immediate reaction was this:
as I express myself on Twitter.  

In this blog post, Product Manager of Google, Craig McLuckie stated that by becoming a sponsor to the OpenStack Foundation, Google is expected to bring valuable knowledge, expertise and deployment experience of container-oriented computing to OpenStack. The blog also stated that Google is seeing 2 emerging trends in the enterprise computing sector:
  1. Moving toward the adoption of hybrid cloud (public and private)
  2. Moving toward the adoption of container-based computing
Quote directly from the blog post:

"We (Google) will be working over the coming months with the (OpenStack) community to integrate Kubernetes, as well as complementary container technologies, to create a stronger hybrid cloud."

The excitement about Google sponsoring OpenStack Foundation is not just just another "big" company joining IBM to endorse the future of OpenStack but what Google can bring and to enhance what OpenStack can provide for its consumers. Google has been running containers for over 10 years. Google's various applications such as Gmail, web search or map services are running as containers. It is said on the internet that Google launch 2 billions containers in one week.

Why Linux Container?
Linux Container is not a new technology and its origin can trace back to FreeBSD Jails in year 2000. Docker Inc, is the company behind the open source Docker platform. The Docker platform is an orchestration or packaging tool that allows application and their dependencies to run on container technology. Docker makes it easy for developer and sys admin to deploy applications with container technology.

The advantage of Linux container is light weight and fast provision as compare to a virtual machine.  It is also portable

Linux containers is just perfect for DevOps as well as Micro-services architecture.

Linux Container is part of the equation along with DevOps and Micro-services to answer the modern day business requirement – agility. In other words the ability to react to the change of the business environment fast or faster than the competitor.

Both DevOps and Micro-services are big topics that will take up an entire blog post to talk about.

In a nutshell, DevOps is not a technology but a culture and a form of application deployment methodology.  It relies on the continuous feedback model from operation to development so that applications can be deployed in a faster manner.
image source: http://www.appdynamics.com/blog/wp-content/uploads/2013/06/DevOps-infinity-loop2.png

Micro-services is the new and popular way of deploying applications.  Application used to be one monolithic process and is not as agile.  Micro-services breaks the application into smaller processes running together.

 image source: http://www.thoughtworks.com/sites/default/files/assets/MicroservicesFowler.png


DevOps, Linux container and micro-services architecture fits right into each other because they share one common characteristic and that is to break one complex problem into smaller problems which can be resolved easier and faster.

Linux containers are gaining traction in the enterprise IT environment.  In my other blog post, I talked about VMware embracing Docker with Project Bonneville making Docker to run just like a virtual machine in vSphere so that Docker container can take advantage of the matured high availability, security and networking infrastructure that VMware had developed.  

Microsoft is also embracing Docker in its public cloud Azure.

We can see that container is an emerging technology integrating with various enterprise ready technologies independently providing features for consumer of the IT infrastructure or cloud.

What is Kubernetes?
Kubernetes is developed by Google to manage containers in a clustered environment.  Google makes Kubernetes open source and can be found on GitHub here.  

The beauty of open source project is that there is always description of the project.  The  description of Kubernetes on GitHub is:

"Kubernetes is an open source system for managing containerized applications across multiple hosts, providing basic mechanisms for deployment, maintenance, and scaling of applications."

An interesting point to note is that Kubernetes is still considered as pre-production beta status and heavy development is still being done. (Well GitHub said it is in pre-production beta but on July 21 at oscon, Kubernetes 1.0 is released.  Release 1.0 and above is usually production ready).

Kubernetes works off the following concepts:
image source: 13110153-conversion-gate01/95/devnexus-2015-kubernetes-container-engine-17-638.jpg?cb=1426244619

Red Hat also used Kubernetes on their PaaS offering - OpenShift.

I will write up a more detailed blog post on Kubernetes later.  If you cannot wait, take a look at here.

Project Magnum
Before the July 16 Google announcement, OpenStack already has an ongoing project called Magnum - Containers as a Service. 

Project Magnum allows containers be deployed in an OpenStack infrastructure as easy as a virtual machine but in the fashion that container should be deployed.  This opens up a lot more business use case on DevOps and Micro-services for OpenStack.

In Nova, Docker is considered as one of the supported hypervisor type.  The main difference for Magnum over Nova is that Magnum uses container orchestration tool to deploy containers at scale and in a rapid manner.  Deployment characteristic of a container is different than a virtual machine such that container come and go more frequently and containers usually are deployed in group as a cluster and sometimes across multiple hosts.  These containers in the cluster needs interact with each other when they operate together to offer the service of one application to the user as in the Micro-service architecture.

There is not much description about the Magnum on GitHub.  Better description can be found on OpenStack wiki page for Magnum The OpenStack Magnum wiki page even has a 5 minute video on how Magnum works.  This project on OpenStack wiki is described as:

"OpenStack API service making container orchestration engines such as Docker Swarm and Kubernetes available as first class resources in OpenStack"

The main idea is to take advantage of the existing Docker orchestration engine and make it work on an OpenStack infrastructure.

I will write another blog post to describe Magnum in more detail.  For now we will look at Magnum at a very high level.  As a preview, the architecture of Magnum looks like this:
image source: https://wiki.openstack.org/wiki/File:Magnum_architecture.png

With this diagram, we can see the general idea on how Magnum works.  Magnum also takes advantage of the OpenStack Heat project.

This slide from the OpenStack Vancouver summit presentation summarize what is on the OpenStack Magnum wiki page:
image source:http://image.slidesharecdn.com/tuckeropenstacksummitvancouverpossibilities-150525221145-lva1-app6891/95/openstack-in-an-ever-expanding-world-of-possibilities-vancouver-2015-summit-25-638.jpg?cb=1432592195

What is ahead of us?
I see that Project Magnum is the reaction to the modern day business requirement and now with Google being able to inject its valuable knowledge, expertise and deployment experiences to the project, Linux container deployment at scale in OpenStack will become mature in due time. It will also integrate better with the other OpenStack projects such as Swift, Keystone or Neutron that more business use cases can be supported.

Google has over 15 years of deploying application at large scale and will certainly able to bring its experience and expertise to the Magnum project.

The most important question is how fast and what features Project Magnum can produce in a timely manner meaning if the market is ready to embrace what is offered. Sometimes the technology is way more advanced than what the market is ready for. I see in the 2000, there is a start-up already trying to offer SDN but at that time the market is not ready for it and that start-up closed down due to lack of funding around 2003.


Reference:
- "Containers + Private Cloud. Google Sponsors OpenStack Foundation." Google Cloud Platform Blog. N.p., n.d. Web. 17 July 2015. 
- "GoogleCloudPlatform/kubernetes." GitHub. N.p., n.d. Web. 18 July 2015. 
- "Magnum." - OpenStack. N.p., n.d. Web. 18 July 2015.     

 

Monday, November 10, 2014

OpenStack Series: Part 10 – Heat – Orchestration Service

The Orchestration service provides a way to create and manage cloud resources such as storage, networking, computer instances, or applications with a template.

Some documentation defines Heat as:

"A service to orchestrate multiple composite cloud applications using the .. _AWS CloudFormation: http://docs.aws.amazon.com/AWSCloudFormation/latest/APIReference/Welcome.html?r=7078 template format, through both an OpenStack-native ReST API and a CloudFormation-compatible Query API."

One important concept for Heat is that it is a declarative specification of application model.   Puppet is a declarative language in which user specify the end state of the server or device that it manages. In the case of Heat it specifies the end state of the state and does not specify how to attend the end state.

Orchestration Service Components
  • heat - A CLI tool that communicates with the heat-api to execute AWS CloudFormation APIs
  • heat-api - An OpenStack-native ReST API that processes API request by sending them to the heat-engine over RPC
  • heat-api-cfn - Provides an AWS-Query API that is compatible with AWS CloudFormation and processes API request by sending them to the heat-engine over RPC
  • heat-engine - Orchestrates the launching of templates and provide events back to the API consumer.
  • api-heat-cloudwatch - Provides monitoring (metrics collection) for the Orchestration Service.
  • heat-cfntools - A package of helper scripts (e.g. cfn-hup which handles updates to metadata and execute custom hooks).

From the components we can see that the heat-engine is the heart of the Orchestration Service in the sense that it launches the templates to create stacks (a collection of resources).

Another thing is that Heat is heavy related to Amazon Web Services CloudFormation. On the above service components there are 3 that are related to AWS -
  • heat-api-cfn, 
  • api-heat-cloudwatch and 
  • heat-cfntools.  

Note: Cloudwatch is a AWS monitoring feature that is the counter part of OpenStack Ceilometer.

We need to look at templates and the 2 versions of APIs (OpenStack native and CloudFormation-compatible).

Heat Template
A template is how we want the OpenStack infrastructure should look like.  Initially the format is compatible to AWS CloudFormation and HOT (Heat Orchestration Template) is to replace the CFN format to be the native format of Heat.

Heat template can be integrate with Configuration Management Tools such as Chef or Puppet.
Heat Template has the following structures:
  • Description
  • Parameters
  • Mappings
  • Resources
  • Outputs

Heat Engine
The Heat API is the vehicle from user to the Heat engine to perform orchestration task.  There are 2 set of API - OpenStack native REST API and AWS Query API

When we think of orchestration, we think of the ability to provision a set of resources and in the case of OpenStack Heat - to provision a stack.

There is another notable feature in OpenStack Heat that is worth looking into. It is - AutoScaling



image source: https://wiki.openstack.org/w/images/2/2a/Current.png

With the alarm provided by the OpenStack Ceilometer, Heat template can be specified to auto scale up or scale down resources.  Usually a load balancer is to be specified in the Heat template when autoscaling is configured.

Related Post:
OpenStack Series Part 1:  How do you look at OpenStack?
OpenStack Series Part 2:  What's new in the Juno Release?
OpenStack Series Part 3:  Keystone - Identity Service
OpenStack Series Part 4:  Nova - Compute Service
OpenStack Series Part 5:  Glance - Image Service
OpenStack Series Part 6:  Cinder - Block Storage Service
OpenStack Series Part 7:  Swift - Object Storage Service
OpenStack Series Part 8:  Neutron - Networking Service
OpenStack Series Part 9:  Horizon - a Web Based UI Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 19: Storage Polices for Object Storage
OpenStack Series Part 20: Group-based Policy for Neutron

Reference:

"Deploying OpenStack: Learning Environments (Manual Setup)." Deploying OpenStack: Learning Environments (Manual Setup). N.p., n.d. Web. 26 Oct. 2014.
"OpenStack." Welcome to the Heat Developer Documentation! — Heat 2015.1.dev217.gd62e13d Documentation. N.p., n.d. Web. 26 Oct. 2014.