Showing posts with label Puppet. Show all posts
Showing posts with label Puppet. Show all posts

Sunday, November 30, 2014

From DevOps to Puppet Part 2

In part 1, we have looked at DevOps, Configuration Management Tools and a little bit of Puppet.  In this post we will continue to look into other aspect of Puppet.

Declarative vs Imperative
One important thing to know about Puppet is that it is a Declarative Language.  What it means is that user only specify the end state of the system that it is trying to configure.  User does not need to know the system specific.  A Puppet manifest can be applied n number of times and if the end state is already in the desire state, nothing will happen.

Another equally popular Configuration Management Tool - Chef, however, is an imperative language where user write out how to achieve the end result.

Puppet Terminology
Puppet looks at everything as a resource where each resource describes some aspect of a system
  • The block of Puppet code that describes a resource is called a Resource Declaration.
  • Each resource has a type, a title and a set of attributes
Example of a resource will be
  • Linux package such as MongoDB
  • Files
  • Users on a system
  • Services such as /etc/host entries; network interface or Windows registers
Example of a resource declaration:
As we can see there are 3 "file" type but the title and attributes are different.  The first example creates a file /tmp/test1 with content "Hi'.  The second one creates a file /tmp/test2 and the security mode is set to 0644 and the third example creates a symlink /tmp/test3 which links to the file /tmp/test1

As mentioned before Puppet is a declarative language.  If the file /tmp/test1 already existed and with content "Hi", nothing will happen even if user execute the manifest 100 times because the desired end state is reached.

There are built-in types for Puppet:

image source: https://docs.puppetlabs.com/references/latest/type.html

If you do not find what you need from the built-in types, take a look a the Puppet Forge and chances are someone else had already did the job for you and had developed a type that you can use or similar to what your needs are.


A Puppet Manifest is a Puppet program with the .pp file extension that contains the resource declaration.These are some of the operation/command that we can have on a Puppet Manifest:

puppet parser validate configWebSrv.pp

puppet apply –-noop configWebSrv.pp

puppet apply configWebSrv.pp

puppet-lint configWebSrv.pp

The "noop" parameter is a very useful command where we can test out the manifest without actually performing the action and Puppet will respond with what is the outcome when this manifest is executed.

Puppet can operate on a standalone device or it can operate on a "Master and Agent" model where the Puppet agent will pull from the Puppet Master on a regular interval.  The default pulling interval is 30 minutes. For emergency application of new configuration, user can trigger the pulling and I believe Puppet Enterprise has the option to trigger the pulling on the console.

Manifest is complied into catalog which is in the format that the Puppet Agent understand.  Puppet will configure the resource according to the content of the catalog to the desired system state.

In a standalone model it works this way:
 image source: https://docs.puppetlabs.com/learning/images/manifest_to_defined_state_unified.png

In a "Master and Agent" model, the manifests are stored in the Puppet Master and periodically, the agent will poll from the Puppet Master if there is any configuration changes.  If necessary, we can trigger the poll manually instead of waiting for the next poll interval.
image source: https://docs.puppetlabs.com/learning/images/manifest_to_defined_state_split.png

The difference between a standalone and "Master and Agent" model is how the manifests are store and where the catalog is complied.

Currently under preview is the Puppet Server.  This is to replace the Puppet Master and is to have many advantages in this new model because of a complete re-design.

May be in Part 3 we can look into this new Puppet Server.

Saturday, November 29, 2014

From DevOps to Puppet Part 1.

DevOps
DevOps is a emerging trend in the IT department. In various conferences you will find a DevOps track or sessions with standing audiences trying to listen to the speaker or to take pictures of the PowerPoint slides. 

DevOps is a new word derived from development and operations.

                      Development + Operations = DevOps

Why putting these 2 together together?

Thanksgiving 2014 just went by and yesterday was "Black Friday" plus 2 days later it will be the "Cyber Monday".  Demand on eCommerce increases tremendously during these few days and then comes Christmas which is also a high season for online shopping (although the true meaning of Christmas is not to give gifts). Consumers of the IT services is demanding for faster service to adjust to changes.  User are also demanding for faster resolutions of problems.

The response to this new IT demands is to put Development and Operations into a team and work together to get things resolves.  Developer is reward by putting in new features.  Operations is reward by providing a stable system.  Often times the goal of a developer and an operation person will conflict with each other.  This will affect how fast a new feature can be rolled out and how fast problem found in production can be resolved.

DevOps provide a new methodology in which problems are broken into smaller problem so that it can be resolved faster.  It also provide an consistent environment for development and production and this is where Configuration Management Tools comes into the picture.

There are 2 important concepts that we have know about DevOps. The first one is that DevOps is a culture.  It is about people's mentality more than a fix methodology or check list to follow.  Each organization is unique and thus how DevOps is implemented will be different. The second one is that there is no such position as a "DevOps Engineer".  There will continue be Development Engineer and there will continue to be Operations staff. DevOps is only to put them together to work as a team and change how things are done - breaking problem into smaller problems and to provide continuous integration/feedback and delivery of new feature/bug fixes in a timely manner.



image source: http://www.appdynamics.com/blog/wp-content/uploads/2013/06/DevOps-infinity-loop2.png

I highly recommend this book "The Phoenix Project" that talks about the subject of DevOps in a novel form.  I finished the book over one weekend.  I cannot put down the book because it was so interesting.  Instead of theory or opinion about DevOps, the book talks about a story of a company's IT problem and how different people solved that problem with the practice of DevOps.  It also went through different people objecting to the idea of DevOps and eventually saw the value of DevOps. After reading this book you will feel that you have went through implementing DevOps practice in your company's IT department with first hand experience and perspective. My wife thought something was wrong with me on that weekend.

Also if interested check out these "10 Must read DevOps books" article or this "DevOps Reading List" to find out if you want to read some of these books on the subject of DevOps.  Or check out this free "DevOps for Dummies" compliments of IBM.

Configuration Management Tools
Configuration Management Tools is an essential part of DevOps.

It is being used to automate tedious and/or repetitive tasks.  This help to avoid human errors.  All Configuration Management Tools has built-in version control and change history.  When things goes wrong we can easily roll back to the previous working version.  With change history we can also trace back to the person who made the changes and to see what is the reason that a change is necessary.

Another advantage of Configuration Management Tools is that it is the up-to-date documentation of the entire system.  A new member of the Operations team can just look at the Configuration Management Tool and will be able to grasp what devices are in the system and how they can be configured.

Configuration Management Tools can also shield the user from knowing the details of different computing platform such as Red Hat and Ubuntu.

Popular Configuration Management Tools are Puppet, Chef, Ansible, Salt.

What is Puppet?
Puppet is an open source Configuration Management Tool (CMT) used to configure mostly Linux system.  In the beginning, it is not very useful to use Puppet to configure a Windows system because the Windows system needs to be rebooted several times.  Recently this has been getting better and Puppet can work the the DSC (Desired State Configuration).  Puppet also has good support for configuring VMs in Microsoft's cloud offering - Azure.  Puppet can also be used to configure Docker containers.  It can also be used to configure networking devices such as F5, Juniper or Arista Network.

Note: some of the feature may only be available in the commercial version of Puppet - Puppet Enterprise from PuppetLabs.


PuppetLabs is based on Portland Oregon with more than 300 employees.  It packages the open source Puppet into a commercial product called Puppet Enterprise.  As the name indicate this commercial product has enterprise as the target user and add enterprise oriented feature such as Graphical User Interface, Security feature - Role Based Access Control and task orchestration capability.  The first 10 nodes of Puppet Enterprise is FREE.  As of this writing Puppet Enterprise is on version 3.7


image source: http://puppetlabs.com/wp-content/uploads/2013/05/puppet_node_manager.jpg

Puppet Enterprise also has the reporting capability such that user do not have to manually go through the log files to see might went wrong when there is a failure.  It can also generate a inventory of the system that is managed by Puppet Enterprise.

Will continue to look into other aspect of Puppet in the next blog post (Part 2).

Thursday, November 20, 2014

OpenStack Series: Part 20 - Group-based Policy for Neutron

Group-based Policy is a new feature in OpenStack and is still in its infant state.  There is a page on this subject in the OpenStack wiki but not much information on what this feature is about. 

For 4 years I have developed and enhanced a feature on the Alcatel-Lucent Enterprise switch for a policy based network access control for network traffic.  I have with 3 other engineers submitted a patent in this area waiting for approval.  This is why I am particular interested to look into this Group-base Policy that is in OpenStack.  Actually, this also made it way into the SDN Controller OpenDaylight project.

This feature is supported by quite a few IT equipment vendors such as Cisco, Alcatel-Lucent (unfortunately I am not with this division such that I can work on this interesting project), Big Switch Network, Juniper, IBM, Red Hat and even Intel.

The main goal for Group-based policy is to provide support and abstraction for the application that is running on the OpenStack Infrastructure.

The whole idea is to provide an abstraction layer for the application so that it does not need to know the detail of the how the infrastructure that it is running on.   It is to use a declarative language to capture the intent of the application.  At this time the Group-based Policy is mainly for Neutron but it can also be developed to be applied to compute and storage.

Note: Declarative is an important concept. It is to defined the desired end state of a server or an infrastructure.  The popular Configuration Management Tool Puppet is a declarative language.  I have talked about OpenStack Congress and it is also a declarative model.

Currently the Neutron API is powerful in providing an abstraction to provide a logical network.  However, there is one drawback - it is very network-centric and user has to be very knowledgeable in networking.  The API may be a set of powerful API for network engineer but it may be too much for an application developer or operator to handle.

According to the OpenStack Documentation, Group-base Policy is to provide a intent-driven declarative policy model that presents simplified application-oriented interfaces to the user.

Application-oriented - Does this sound familiar?  For me I immediately think of the Cisco version of SDN (Software Defined Network) - Application Centric Infrastructure (ACI).  Of course, the APIC driver is part of Neutron's ML2 mechanism module.  ACI by itself is a big topic to be discussed.  I will most likely look into this in the near future as part of my quest to the cloud related technology.  When I look into this subject of Group-based Policy, I find the most informative information comes from Cisco's blog and/or white paper.  I think Cisco contribute to this Group-based policy feature and is driving the advancement of this feature as well.

While I think even Cisco is driving the advancement and acceptance of Group-based Policy, it has it value and is not a feature that is pushed by a vendor for their sole benefits in this case Cisco and ACI.

Group-based Policy Architecture


image source: http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.doc/_jcr_content/renditions/white-paper-c11-733126_0.jpg
I found that this Cisco white paper is a very good description of the Group-base Policy's architecture.  There are 4 main concepts that are essential to Group-based Policy:
  • Groups: network endpoints are put together in a group and the properties of the endpoints are treated as a whole.
  • Policy Rule Set: This describe how the groups are connected together
  • Policy Layering: Policy can be layered on top of each  other forming a new/combined policy.
  • Network Service Chaining: This is an important concept in NFV (Network Function Virtualization). Group-base Policy allows applications to define their requirements on how the packet flows.
According to the Cisco white paper, this Group-base policy has the following advantages:

  • Automation and security are much easier to implement through Group-Based Policy.
  • Offers a naturally flexible and extensible framework for capturing the requirements of a virtual machine in a single location.
  • Makes consistency easier to achieve because only one step - becoming a member of the group - is required to inherit multiple policies 
  • Easy for application developers to use and offers them a simple way to describe application requirements 
  • Offers a means for allowing operator and user requirements to coexist cleanly 

Group-based Policy and OpenStack


image source: http://www.cisco.com/c/dam/en/us/solutions/collateral/data-center-virtualization/application-centric-infrastructure/white-paper-c11-733126.doc/_jcr_content/renditions/white-paper-c11-733126_2.jpg

Also, according to the same Cisco white paper, Group-based policy is supposed to fit into OpenStack in a "non-disruptive" layer.  I take it as not or minimum changes is need in OpenStack.  When we look at the diagram above Group Policy is the orange layer and conceptually this is how it fits into the OpenStack.  We can see the Horizon and Heat module along with the OpenStack CLI sitting on top for configuration (manual or via a Heat Template).  The next layer is the application in which all our focus are in to making the deployment of application as easy as possible without have to "worry" about the supporting infrastructure.  Below the Group Policy layer are the other various OpenStack projects. At this time it is mainly focused on Networking.

Group-based Policy and OpenStack Congress
On the SDN front, we see that VMware and Cisco are going the opposite direction.  OpenStack Congress is heavily driven by VMware while Group-based Policy is heavily driven by Cisco.  Will the same thing happen here where the 2 technologies are competing with each other?

It seems that Congress covers governance and compliance while Group-based policy is to capture the application's intent and to provide an abstraction layer.  These 2 things compliment each other and is providing different services to the OpenStack users.


Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack

OpenStack Series Part 17: Congress - Policy Service  
OpenStack Series Part 18: Network Function Virtualization in OpenStack 
OpenStack Series Part 19: Storage policies for object storage

Reference:
"GroupBasedPolicy." - OpenStack. N.p., n.d. Web. 09 Nov. 2014.