OpenStack Series: Part 19 - Storage Policies for Object Storage

Storage Policy for object storage is an announced feature for the OpenStack Juno release.

Storage Policy is said to be the most significant feature for OpenStack Swift since its inception in 2010.

Why is Storage Policy so significant?
Storage policy allows user to decide where and how to store the data. Storage policy is introduced in Swift 2.0.  Before release 2.0 all data are stored the same way but in an OpenStack Infrastructure there can be multiple tenants and all will have different storage requirement.  We do not want to create a Swift cluster for each tenants.

In Swift entities are stored in a container and is determined by the ring structure.  I have an article on OpenStack Swift that talk about how each object is represented by:

/v1/account/container/object format.

With a modified MD5 hashing, the URI of the object is translated to where the data is stored in the physical device.

With Storage Policy, there are multiple rings and each rings is assigned a policy.  The policy can control:

• Level of replication
• Type Storage device used to store the object that has different I/O performance
• Geographic location/availability Zone that the data is to be store
• Backend Storage system to be used such as Kinetic or GlusterFS

With the ability for the user to tune how and where the data is store allows for better support for multiple tenants in an OpenStack Infrastructure.

Take for example the level of replication, there are some data that can be reproduced from another source and do not need to be replicated 3 times in the storage system.  User can now specify that the data to be replicated 2 times.  With this change the user is able to save 33% of the physical storage required.

Storage Policy
For detailed description of OpenStack Swift Storage Policy, we can go to this OpenStack Documentation.

I will highlight a few important points from the article:

• Policies are implemented at the container level
• This allows for minimum changes to the existing API
• X-Storage-Policy header is used to set the policy for the container
• Each container has a new special immutable metadata element - Storage Policy Index
• Policies are assigned when a container is created
• Once a policy is assigned to a container, it can not be changed, unless it is deleted and re-created
• Containers have a many-to-one relationship with policies - many containers can have the same policy
• For working with pre 2.0 version of Swift, there is the Policy-0 in which the index 0 is used
• User can specify a default policy for the Swift cluster
• Default policy is used when no policy is configured by the user.
• Default policy is to be use for Swift version 2.0 and beyond
• Policy-0 is to be used by pre 2.0 version of Swift where Storage Policy is not supported.

Erasure Code

Storage Policy paves the way for Erasure Code in OpenStack Swift which is another significant enhancement for the project. 

Erasure Code is one kind of forward error correction code which was invented by an American mathematician Richard Hamming in 1950.  It can be used as a replacement of RAID for data protection.  The advantage of Erasure Code over RAID is its ability to reduce the resource for data recovery in terms of speed and require less storage space.  The only drawback for Erasure code over RAID is that it is very computational intensive.  With Storage Policy, user can chose between erasure code or replication for data protection just as they can choose the level of replication be it 3 time or 2 times.

Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service  
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 20: Group-based Policy for Neutron

Reference:
"OpenStack." Storage Policies — Swift 2.2.0.34.g772dc5d Documentation. N.p., n.d. Web. 07 Nov. 2014.