OpenStack Series for Beginners

Blogging is new to me.  I started blogging in late July of this year.  To my surprise I was able to deliver 30 blog posts in 30 days. 

In the month of November I have participated in a community effort where each member publish 30 blogs in 30 days.  You can check out other blog posts with #vDM30in30 on Twitter or  go to http://www.virtualdesignmaster.com/page/2/

When I sign up I thought it was 30 blogs from the whole group.  Not until 2 weeks before Nov 1 that I have discovered that it was 30 blog post for each person.  I thought of backing off but then I wanted to push myself to see how far I can go.

Of the 30 blog posts, I have dedicated 20 of them to be OpenStack related in which I look at different parts of this popular open source cloud orchestration tool.  I would like to put all OpenStack related post into one for reference 

OpenStack Series Part 1:  How do you look at OpenStack?
OpenStack Series Part 2:  What's new in the Juno Release?
OpenStack Series Part 3:  Keystone - Identity Service
OpenStack Series Part 4:  Nova - Compute Service
OpenStack Series Part 5:  Glance - Image Service
OpenStack Series Part 6:  Cinder - Block Storage Service
OpenStack Series Part 7:  Swift - Object Storage Service
OpenStack Series Part 8:  Neutron - Networking Service
OpenStack Series Part 9:  Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service 
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 19: Storage policies for object storage
OpenStack Series Part 20: Group-based Policy for Neutron

Hope you enjoy this OpenStack for beginner series!

Nutanix - a web-scale solution provider

I got this t-shirt a few months back and the material of this t-shirt was so nice such that I put it neatly in the draw and had forgotten about it.

It was given to me for free for the Web-Scale Wednesday -An Global Online event where it brought together IT leaders, industry experts and enterprise customers to share their perspectives and experiences adopting web-scale IT and bringing it to the enterprise

What is Web-Scale?
According to this article from Gartner's blog, "Web-Scale" is a term that Gartner uses "in an effort to describe all of the things happening at large cloud services firms such as Google, Amazon, Rackspace, Netflix, Facebook, etc., that enables them to achieve extreme levels of service delivery as compared to many of their enterprise counterparts."  The article further identifies 6 elements that web scale has:

• Industrial data centers,
• Web-oriented architectures,
• Programmable management,
• Agile processes,
• A collaborative organization style and
• A learning culture.  

An interesting note to the word scale that most people will think of scaling in size, Gartner also stated that scale can refer to speed also.

Nutanix on the other hand suggest that a Web Scale Infrastructure has these 5 essential elements:

• Hyper-convergence on x86 servers
• Intelligence in Software
• Distributed Everything
• Self-Healing System
• API-based Automation and Rich Analytic

Nutanix has this video on "What Is Web-scale IT":

The main idea of a Web Scaled IT infrastructure is to follow how the huge web companies such as Google, Facebook or Netflix build, deploy and manage their data center.  Web Scale principle can be applied to enterprise and even SMBs (Small to Medium Business) to provide agility, scaling and better return on investment (RTO) on x86 hardware.

Nutanix

Nutanix is founded in 2009 with its headquarter in San Jose.  First product was shipped in 2011.

In marketing term, Nutanix offers the Nutanix Web-scale hyper-converged infrastructure solutions which "revolutionizing the enterprise datacenter by delivering efficient, radically simple physical, virtual and cloud environments."

 Nutanix's product offerings comes in a varieties of mix and match of it software editions and hardware platforms.

Nutanix hardware platform includes:

• NX-1000 series
• NX-3000 series
• NX-6000 series
• NX-7000 series
• NX-8000 series
• NX-9000 series

For detail specification and comparison, we can visit this page.

Nutanix software editions includes:

• Starter
• Pro
• Ultimate

For detail specification and description of the software, we can visit this page.

The Nutanix Solutions
According to the Nutanix web page: "The Nutanix Virtual Computing Platform is a web-scale converged infrastructure solution that consolidates the compute (server) tier and the storage tier into a single, integrated appliance.

The Nutanix Virtual Computing Platform integrates high-performance server resources with enterprise-class storage in a cost-effective 2U appliance. It eliminates the need for network-based storage architecture, such as a storage area network (SAN) or network-attached storage (NAS). The scalability and performance that the world’s largest, most efficient datacenters enjoy are now available to all enterprises and government agencies."

From the above paragraph, I believed that "web-scale converged infrastructure" is the most important words that describes Nutanix's solution which is web scale and with a converged infrastructure.  Providing to customer the ability to scale like the big web companies such as Google, Facebook or Netflix with a converged infrastructure bringing hypervisor, compute, storage and networking into a single appliance.

All the Nutanix hardware platforms can be "linked" together as a cluster.  The key to Nutanix's solution is distribution of operation thus making the infrastructure agile and resilience. .

Here is a "Simple Explanation of How Nutanix Works"

Nutanix Innovations
Nutanix does not have any special hardware, all their innovations are on the software - Nutanix Controller Virtual Machine.  At of now there are 3 flavors of virtual machines that are specially tuned to their respective hypervisor platform:

• VMware vSphere
• Microsoft Hyper-V
• Linux KVM

The Nutanix Controller Virtual Machine has 2 main functions:

• Nutanix Distributed File System
• Cluster management

image source: http://cdn1.stevenpoitras.com/wp-content/uploads/2013/09/NDFS_NodeDetail2.png

Nutanix Distributed File System
A Nutanix cluster consist of one or more appliance which has a minimum of 3 nodes.  Together it form the Nutanix Distributed File system (NDFS).

 

image source: http://cdn.stevenpoitras.com/wp-content/uploads/2013/09/CVM_Dist.png

This distributed file system is to provide data efficiency and data protection.  To the virtual machine in this web-scale converged infrastructure, the NDFS is a single data store.  The data efficiency and protection is abstracted from the user.  With this architecture, there is no need to have a separate and dedicated hardware to perform inline deduplication and compression.  According to Nutanix website NDFS has the following advantages:

• Self-healing
• Built-in converged backup and disaster recovery
• Scheduled snapshots to align with RPO and RTO
• Data localization in which data moves with the VM
• Elastic Deduplication Engine to perform deduplication in RAM
• Array-side compression

This page has more detailed description of NDFS.

Cluster Management
The other main function of the Nutanix Controller Virtual Machine is the management, coordination and application of the key Nutanix technologies in the cluster.  This diagram shows the high level components of a Nutanix cluster

image source: http://cdn.stevenpoitras.com/wp-content/uploads/2013/09/NDFS_ClusterComponents.png

Nutanix has a good document on its technologies  - Nutanix Bible.  This document is an ongoing updated document provided by Steven Poitaris for the Nutanix product.  It has so much detail on a lot of subjects.  It has a good description of each of these components and I extra the text from the Nutanix Bible:

Cassandra

• Key Role: Distributed metadata store
• Description: Cassandra stores and manages all of the cluster metadata in a distributed ring like manner based upon a heavily modified Apache Cassandra.  The Paxos algorithm is utilized to enforce strict consistency.  This service runs on every node in the cluster.  Cassandra is accessed via an interface called Medusa.

Zookeeper

• Key Role: Cluster configuration manager
• Description: Zeus stores all of the cluster configuration including hosts, IPs, state, etc. and is based upon Apache Zookeeper.  This service runs on three nodes in the cluster, one of which is elected as a leader.  The leader receives all requests and forwards them to the peers.  If the leader fails to respond a new leader is automatically elected.   Zookeeper is accessed via an interface called Zeus.

Stargate

• Key Role: Data I/O manager
• Description: Stargate is responsible for all data management and I/O operations and is the main interface from the hypervisor (via NFS, iSCSI or SMB).  This service runs on every node in the cluster in order to serve localized I/O.

Curator

• Key Role: Map reduce cluster management and cleanup
• Description: Curator is responsible for managing and distributing tasks throughout the cluster including disk balancing, proactive scrubbing, and many more items.  Curator runs on every node and is controlled by an elected Curator Master who is responsible for the task and job delegation.

Prism

• Key Role: UI and API
• Description: Prism is the management gateway for component and administrators to configure and monitor the Nutanix cluster.  This includes Ncli, the HTML5 UI and REST API.  Prism runs on every node in the cluster and uses an elected leader like all components in the cluster.

Genesis

• Key Role: Cluster component & service manager
• Description:  Genesis is a process which runs on each node and is responsible for any services interactions (start/stop/etc.) as well as for the initial configuration.  Genesis is a process which runs independently of the cluster and does not require the cluster to be configured/running.  The only requirement for genesis to be running is that Zookeeper is up and running.  The cluster_init and cluster_status pages are displayed by the genesis process.

Chronos

• Key Role: Job and Task scheduler
• Description: Chronos is responsible for taking the jobs and tasks resulting from a Curator scan and scheduling/throttling tasks among nodes.  Chronos runs on every node and is controlled by an elected Chronos Master who is responsible for the task and job delegation and runs on the same node as the Curator Master.

Cerebro

• Key Role: Replication/DR manager
• Description: Cerebro is responsible for the replication and DR capabilities of NDFS.  This includes the scheduling of snapshots, the replication to remote sites, and the site migration/failover.  Cerebro runs on every node in the Nutanix cluster and all nodes participate in replication to remote clusters/sites.

Pithos

• Key Role: vDisk configuration manager
• Description: Pithos is responsible for vDisk (NDFS file) configuration data.  Pithos runs on every node and is built on top of Cassandra.

Nutanix Use Cases
Being a web-scale converged infrastructure, Nutanix has the following but not limited to the following use cases:

• VDI
• Enterprise Branch Offices
• Big Data
• Private Cloud
• Disaster Recovery

Reference:
"Cameron Haight." Cameron Haight RSS. N.p., n.d. Web. 09 Dec. 2014 
"Architecture | Nutanix." Nutanix. N.p., n.d. Web. 10 Dec. 2014.
"The Nutanix Bible - StevenPoitras.com." StevenPoitrascom. N.p., n.d. Web. 10 Dec. 2014.

A look at SimpliVity – a Hyper-Convergence option

Compute, Network and Storage are the 3 pillars of a data center.  Storage had been my weakest point in terms of knowledge and experience. 

Today I had a great chat with Brian Knudtson (@bknudtson) who is very knowledgeable in different aspects of the technology field and had opened my mind in the area of “Hyper-convergence”.   In this post, I am stepping out of my comfort zone again and try to venture into this area and take a look at this emerging and expanding market.  In VMworld 2014, VMware announced a new product EVO:RAIL which reinforce the idea that this “Hyperconvergence” market has huge potential in the coming years.

What is Hyper-Convergence?

In its simplest term, Hyper-Convergence is the integration of compute, storage and network resource on a box in which virtualization technologies and X86 hardware platform is use.

For a converged system, compute and storage are put together into a single device/appliance.  As for Hyper-convergence, the hypervisor is added thus making the device as a mini Data Center.  

Being a mini Data Center, SimpliVity see that Hyper-Convergence or Convergence 3.0 should also deliver backup, Disaster Recovery, WAN Optimization and a Cloud Gateway.

SimpliVity sees Hyper-Convergence as "Data Center in a Box" and this is how they build their Hyper-Convergence product - OmniCube.

image source: https://www.simplivity.com/wp-content/uploads/Converged_Infrastructure_Evolution_1_2_3-1024x499.jpg

SimpliVity

SimpliVity is formed in 2009 with it headquarter based off Massachusetts U.S.A.  It mission statement is “Simplify IT”. 

To make IT simple, SimpliVity creates its hyperconverged infrastructure platform, OmniCube, by packaging OmniStack on an x86 platform that provides hypervisor, compute, storage services and network switching. OmniCube is a 2U rack mounted appliance.  The hypervisor is an integral part of hyperconverged infrastructure and the hypervisor in OmniCube is the VMware's ESXi.

OmniCube

OmniCube is to provide high availability and no single point of failure with the goal of making IT operations SIMPLE.  Besides putting compute, storage, networking and hypervisor onto a single appliance it is also to deliver enterprise features such as data protection and performance as well as cloud like scalability capability with its global federation with other OmniCubes managed via VMware’s vCenter.

OmniCube has the following versions:

• OmniCube CN-2000 series
• OmniCube CN-3000
• OmniCube CN-5000

image source: https://www.simplivity.com/wp-content/uploads/SIM_DS_FINAL_4.pdf

For a complete description of SimpliVity, you can visit their product page.  Recently SimpliVity announced partnership with Cisco to ship the “OmniStack Integrated Solution with Cisco UCS”.

OmniStack Integrated Solution with Cisco UCS

In August 25 2014, SimpliVity and Cisco had a press release announcing this OmniStack and UCS integration.   The title of the press release is “New SimpliVity Integrated Solution with Cisco UCS Delivers The Best of Both Worlds: Cloud Economics With Enterprise Performance, Protection and Functionality”.

Cloud economics means enterprise customer can enjoy the pooling of X86 resources and at the same time enjoy the enterprise level performance, protection and functionality such as high availability of the data center, data backup and restoration and the efficient handling of data offered by the SimpliVity OmniStack product.

The initial integration is to put the OmniStack controller and OmniStack Accelerator Card on the Cisco UCS C240 M3 Series Rack mount server.  Here is the reference architecture of this offering.

In Eric Wright’s blog post this integration is to be a win-win opportunity for SimpliVity and Cisco in which SimpliVity can reach into new market segments and Cisco can have more product offering to existing and/or new customers.

Image source: http://www.virtualizationsoftware.com/wp-content/uploads/2014/08/Cisco-OmniStack-Node-View.jpg

Image source : https://www.simplivity.com/wp-content/uploads/SimpliVity_OmniStack_Data_Sheet.pdf

SimpliVity Innovations

SimpliVity brings to the table 3 technology innovations:

• Accelerated Global Deduplication
• Global Unified Management 
• HyperConvergence

OmniStack

This is the software that runs as a controller on the VMware ESXi in which it is named as OVC (OmniStack Virtual Controller).

According to the SimpliVity official website this OmniStack has 10 patent pending.  The function of OmniStack is to:

• Combining the functions of up to 12 different products.
• Abstracts data from its underlying hardware, shifting the management paradigm from hardware resources to workloads/applications and providing unparalleled data mobility
• Work with the OmniStack Accelerator to perform inline data deduplication and compression
• Provide data protection
• Provide WAN Optimization
• Act as a cloud gateway

OmniStack Accelerator

This is the hardware module in every OmniCube appliance that is used to perform the actual inline compression, deduplication of the data.  This hardware module works closely with OmniStack Virtual Controller.

The OmniStack Accelerator card is to off load the compute resources so that the CPU cycle can be dedicated to the application and to use this dedicated hardware to dedupe, compress and optimize the data. With this accomplished for the data, OmniStack (controller portion) can take advantage of the optimized data to provide global fabric thus allowing VM and data to be moved from one geographic location to another in a efficient and seamless manner.

According to SimpliVity website, OmniStack Accelerator handles data in 4K to 8K blocks for any tier of storage media of a system as well as across different data center or even Amazon Web Services.  The main idea is in-line deduplication, compression and optimization of the data.  This is done before the data is written to storage thus reducing IOPS which translates to better performance.  The term “Once and forever” is used in various places at SimpliVity website to describe the inline data operation.  

This is a screen shot taken from the SimpliVity vCenter plugin and the deduplication ratio is 123.2:1 and the efficiency ratio is 183:1.

This is an important feature for more efficient operation to remote office branch office (ROBO), VDI and data protection.  The about screen shot is from the SimpliVity vCenter plugin, it show how the user deduplication and compress ration as well as how much storage usage in an easy to read user interface.

Global Unified Management

This innovation is to manage multiple instances of OmniCube as a single pool of resources.

Besides integrating the ESXi in OmniCube, SimpliVity has a plugin to VMware’s vCenter providing VM-centric management and reporting capability.

 Image source: http://www.esg-global.com/default/assets/Image/SimpliVityf8.png

While having a plugin to vCenter, provides user an easy and familiar user interface to manage the Hyper-convergence infrastructure there are additional features that SimpliVity had build on top of this vCenter interface.

Federation

On the right side of the SimpliVity vCenter console is the company logo.  On the left there is a button – Federation.  OmniCube has built in technology to “connect” to other OmniCube in the network.  This is very use for connecting remote office, data protection and most of all the ability to scale out.

Policy-based Data Protection

Another feature is the policy-based data protection in which user can configure the time and interval for data or even VM level backup at remote or DR sites.  Due to the in-line data deduplication, compression and optimization the backup and restoration of data and VM is made very efficient.

Connection to the cloud

SimpliVity has build-in support to interface with Amazon Web Service so that data or VM can be backup to the AWS storage. Again since the data are efficiently deduplicated, compressed and optimized this process is quite fast which helps user to meet better RPO and RTO.

This picture gives us a good view on Global Unified Management:

Image source: http://www.esg-global.com/default/assets/Image/SimpliVityf7.png

 

Reference:

"About SimpliVity - Simplivity." Simplivity About SimpliVity Comments. N.p., n.d. Web. 28 Nov. 2014.

"OmniCube Hyperconverged Infrastructure | OmniCube Differentiation." Simplivity Differentiation Comments. N.p., n.d. Web. 03 Dec. 2014.