Showing posts with label Storage Policy. Show all posts
Showing posts with label Storage Policy. Show all posts

Wednesday, June 3, 2015

Navigating through the VMware's forest of product offerings

Back in 2008 when I first started to learn about VMware's server virtualization technology, it is very confusing on the different products and how they are related together.  There are the ESX, ESXi, vSphere and vCenter server and there are older products such as VMware server or the GSX server.  With Google we can find lots of articles on these products but the problem is that most of these articles do not have a date and since the VMware virtualization technology is advancing in such a high pace, we do not know if the information was current or relevant to which version of vSphere.

Now in 2015, can you tell me all the current VMware products and how they are related?

To look at VMware's product offering, it is best to look at VMware's vision on data center - Software Defined Data Center architecture.

Data Center comprises of these functions:
  1. Compute
  2. Storage
  3. Network
  4. Management
VMware’s idea is to provide an abstraction layer for compute, storage and network hardware so that the entire data center is software driven.   

The advantage of a Software Defined Data Center architecture is that the data center can be more agile as the hardware resources are provisioned or withdraw on demand to the compute, storage and network resources.  This agility factor has huge implication to IT operations in a data center:
  • More cost effective
  • Ability to service user demand much faster
  • Provide a stable platform for DevOps
  • Eliminate or minimize human error
  • Make Disaster Recovery easier thus able to improve RTO and RPO.
The benefits of a Software Defined Data Center architecture is not limited to the above mentioned points and it is not the main point for this article.

VMware vCloud Suite is VMware's solution to provide a "Software Defined Data Center" for customers. As shown in the diagram below vCloud Suite comprises of:
  • Compute
  • Storage and Availability
  • Management and Automation
  • Network and Security (add on feature)
Each of these different products under the above mentioned categories can be deployed separately and as a whole they formed the vCloud Suite.
image source: http://www.vmware.com/files/images/thumbnails/vmw-scrnsht-vcloud-suite-mgmt-lg.jpg

Compute
At the heart of the vCloud Suite is the vSphere Suite that provide the virtualization infrastructure for different business establishments from SMB to enterprise to cloud operators. I don't think I need to describe much about the vSphere Suite as this has been around for a number of years and are widely deployed.

Management and Automation
The VMware vRealize™ Suite provides the management and automation capability for the Software Defined Data Center architecture.

Basically, the vRealize Suite comprise of Automation, Operations and Business and Log Insight for monitoring.
image source: http://cloudmaniac.net/wp-content/uploads/2014/10/vmware-vrealize-suite.jpg

Note:
  • vRealize Operations is formerly known as vCenter Operations Management Suite or vCops
  • vRealize Automation is formerly known as vCloud Automation Center or vCAC
  • vRealize Business is formerly known as IT Business Management Suite
  • vRealize Log Insight is formerly know as vCenter Log Insight
For a more comprehensive and detailed description of the vRealize Suite, this VMware site is a good place to visit.

VMware vRealize Suite consists of the following products:
  • VMware vRealize™ Automation™ Advanced or Enterprise
  • VMware vRealize™ Operations™ Advanced or Enterprise
  • VMware vRealize™ Log Insight™
  • VMware vRealize™ Business™ Standard
  • VMware vRealize™ Business™ Advanced or Enterprise
Storage and Availability
For all IT operations, disaster recovery is an important element.  Each business should defined the Recovery Time objective (RTO) and Recovery Point Objective (RPO) for acceptable level of disruption in case of a disaster event affecting the continuous operation of the IT infrastructure. Most importantly DR plans needs to be tested.  VMware vCenter Site Recovery Manager provide customer the ability to automate and orchestrate non-disruptive testing of recovery plans. With automation and orchestration, the RTO and RPO can be improved.

image source: http://ddf912383141a8d7bbe4-e053e711fc85de3290f121ef0f0e3a1f.r87.cf1.rackcdn.com/2.17%20vcenter.png

For more comprehensive and detailed description of VMware vCenter Site Recovery Manager visit here.

I am not sure why VMware group storage and availability into the same category because availability also include the compute and networking pieces that support the application that we are ultimately concern with.

Regarding storage, VMware had been working on abstracting the storage hardware and to provide an easy to configured Software defined Storage products.  In this area, VMware is taking 2 approaches:
  1. Virtual Data Plane - Virtual SAN (VSAN) and vSphere Virtual Volume (vVol)
  2. Policy Driven Control Plane - Storage Policy Based Management
There are many other blog post talking about these products. This blog post is meant to provide an overview and how these products related to each other.

Network and Security
This is an add-on to to the vCloud Suite.  In this category is the NSX - VMware's way of providing an abstraction layer to the physical network as well as some L4 - L7 network services.

NSX is also a security product as I have talked about in my previous post.

Brad Hedlund (@bradhedlund, Engineering Architect at VMware's Network and Security Business Unit) has an good article on what network virtualization is


image source: http://commondatastorage.googleapis.com/bradhedlund/blog/what-is-network-virtualization/What_is_Network_Virtualization.PNG

NSX virtualized the network as well as network functions such as load balancing, firewall and VPN. Any Cloud management platform including OpenStack can interact with NSX via RESTful API to provision network services for the cloud platform.  

NSX has two flavors as I have outlined in this post.  One flavor is tailor toward vSphere product and the other one is for non VMware hypervisors such as KVM.  

One point worth mentioning is that NSX can optimize the VXLAN multicast traffic because the NSX Controller is able to distribute the MAC address and VXLAN ID mapping to the various VTEPs instead of using bi-directional PIM as defined in RFC 7348 (I have a summary of this RFC here).  Multicast traffics takes up network bandwidths and network administrators always try to find ways to eliminate multicast traffic.

Other VMware products
For a complete list of VMware products we can find them here. In this post I have not talked about the End-user computing product of VMware - Horizon nor the VMware hybrid cloud product - VMware Air which is the VMware's offering of public cloud similar to Amazon's AWS, Google's Cloud platform or Microsoft's Azure.

I hope this will clarify VMware's product offerings and how they relate to each other.




Wednesday, November 19, 2014

OpenStack Series: Part 19 - Storage Policies for Object Storage

Storage Policy for object storage is an announced feature for the OpenStack Juno release.

Storage Policy is said to be the most significant feature for OpenStack Swift since its inception in 2010.

Why is Storage Policy so significant?
Storage policy allows user to decide where and how to store the data. Storage policy is introduced in Swift 2.0.  Before release 2.0 all data are stored the same way but in an OpenStack Infrastructure there can be multiple tenants and all will have different storage requirement.  We do not want to create a Swift cluster for each tenants.

In Swift entities are stored in a container and is determined by the ring structure.  I have an article on OpenStack Swift that talk about how each object is represented by:

/v1/account/container/object format.

With a modified MD5 hashing, the URI of the object is translated to where the data is stored in the physical device.

With Storage Policy, there are multiple rings and each rings is assigned a policy.  The policy can control:
  • Level of replication
  • Type Storage device used to store the object that has different I/O performance
  • Geographic location/availability Zone that the data is to be store
  • Backend Storage system to be used such as Kinetic or GlusterFS
With the ability for the user to tune how and where the data is store allows for better support for multiple tenants in an OpenStack Infrastructure.

Take for example the level of replication, there are some data that can be reproduced from another source and do not need to be replicated 3 times in the storage system.  User can now specify that the data to be replicated 2 times.  With this change the user is able to save 33% of the physical storage required.

Storage Policy
For detailed description of OpenStack Swift Storage Policy, we can go to this OpenStack Documentation.
I will highlight a few important points from the article:
  • Policies are implemented at the container level
  • This allows for minimum changes to the existing API
  • X-Storage-Policy header is used to set the policy for the container
  • Each container has a new special immutable metadata element - Storage Policy Index
  • Policies are assigned when a container is created
  • Once a policy is assigned to a container, it can not be changed, unless it is deleted and re-created
  • Containers have a many-to-one relationship with policies - many containers can have the same policy
  • For working with pre 2.0 version of Swift, there is the Policy-0 in which the index 0 is used
  • User can specify a default policy for the Swift cluster
  • Default policy is used when no policy is configured by the user.
  • Default policy is to be use for Swift version 2.0 and beyond
  • Policy-0 is to be used by pre 2.0 version of Swift where Storage Policy is not supported.
Erasure Code

Storage Policy paves the way for Erasure Code in OpenStack Swift which is another significant enhancement for the project. 

Erasure Code is one kind of forward error correction code which was invented by an American mathematician Richard Hamming in 1950.  It can be used as a replacement of RAID for data protection.  The advantage of Erasure Code over RAID is its ability to reduce the resource for data recovery in terms of speed and require less storage space.  The only drawback for Erasure code over RAID is that it is very computational intensive.  With Storage Policy, user can chose between erasure code or replication for data protection just as they can choose the level of replication be it 3 time or 2 times.


Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack

OpenStack Series Part 17: Congress - Policy Service  
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 20: Group-based Policy for Neutron

Reference:
"OpenStack." Storage Policies — Swift 2.2.0.34.g772dc5d Documentation. N.p., n.d. Web. 07 Nov. 2014.


Monday, November 17, 2014

OpenStack Series (Part 17): Congress – Policy Service

Congress is an OpenStack project to provide policy as a service across any collection of cloud services in order to offer governance and compliance for dynamic infrastructures.

The demand of a contemporary data center is agility.  Traditional policy enforcement done manually is not meeting this specific need.

We can see IT vendors is favoring to use policy for their products.  For example:

One objective for OpenStack Congress is to provide an abstraction layer/As a Service with a common interface to apply policy or policies to elements in the OpenStack Infrastructure.

There are 2 specific purpose/function outlined for OpenStack Congress - governance and compliance.

Governance
The first purpose of OpenStack Congress is governance which is to use a high level declarative language to define the stat of the cloud infrastructure.  Puppet is a declarative language and so is OpenStack Heat Template.  Declarative mean, only the desired end state is specified without giving detail or step by step instruction as to how to attain the desired end state.

The declarative language used by OpenStack Congress is Datalog which is basically SQL with syntax that is closer to traditional/procedural programming language.  Extracted from OpenStack Documentation the grammar of this declarative languages are:


<policy> ::= <rule>*
<rule> ::= <atom> COLONMINUS <literal> (COMMA <literal>)*
<literal> ::= <atom>
<literal> ::= NOT <atom>
<atom> ::= TABLENAME LPAREN <term> (COMMA <term>)* RPAREN
<term> ::= INTEGER | FLOAT | STRING | VARIABLE

Compliance
Another purpose of OpenStack Congress is policy enforcement.  There are 3 ways that the policy is enforced:


o    Proactively: preventing violations before they occur
o    Reactively: correcting violations after they occur
o    Interactively: give administrators insight into policy and its violations, e.g. identifying violations, explaining their causes, computing potential remediation, simulating a sequence of changes.


Monitoring is an important element of OpenStack Congress for enforcing policy reactively and interactively.  I can see that OpenStack is interacting with OpenStack Keystone, OpenStack Heat and OpenStack Mistral.  Not sure how Congress is doing the monitoring function.  I would think the best fit is to interact with OpenStack Ceilometer.  I will have to find out and update this section.

Use Cases for Congress
If you want to look into OpenStack Congress beside reading the official OpenStack document, you must take a look at this article (part 1) and this article (part 2).  As of this writing not much blog post or documentation is available for this subject.

OpenStack Document outlined a few use cases and this article by Tim Hinrichs and Scott Lowe has 4 specific use cases called for OpenStack Congress.



      In the coming days I am sure there will be more use cases as this project moves to maturity and integrate into the OpenStack release.  In the Juno release there is ground works done in Nova to support NFV which is again a hot topic.

      Related Post:
      OpenStack Series Part 1: How do you look at OpenStack?
      OpenStack Series Part 2: What's new in the Juno Release?
      OpenStack Series Part 3: Keystone - Identity Service
      OpenStack Series Part 4: Nova - Compute Service
      OpenStack Series Part 5: Glance - Image Service
      OpenStack Series Part 6: Cinder - Block Storage Service
      OpenStack Series Part 7: Swift - Object Storage Service
      OpenStack Series Part 8: Neutron - Networking Service
      OpenStack Series Part 9: Horizon - a Web Based UI Service
      OpenStack Series Part 10: Heat - Orchestration Service
      OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
      OpenStack Series Part 12: Trove - Database Service
      OpenStack Series Part 13: Docker in OpenStack
      OpenStack Series Part 14: Sahara - Data Processing Service
      OpenStack Series part 15: Messaging and Queuing System in OpenStack
      OpenStack Series Part 16: Ceph in OpenStack
      OpenStack Series Part 18: Network Function Virtualization in OpenStack
      OpenStack Series Part 19: Storage Polices for Object Storage
      OpenStack Series Part 20: Group-based Policy for Neutron

      Reference:
      "Congress." - OpenStack. N.p., n.d. Web. 30 Oct. 2014.