Tuesday, November 11, 2014

OpenStack Series: Part 11 Ceilometer – Monitoring and Metering Service

Project Goal/Objective
The Ceilometer project was started in 2012 with one simple goal in mind:to provide an infrastructure to collect any information needed regarding OpenStack projects. It was designed so that rating engines could use this single source to transform events into billable items.

Note: Billing involve 3 steps metering, rating and billing.  Ceilometer only deals with metering.  Application will have to be build on top of Ceilometer to perform the complete billing process.

As the project become mature a secondary goal is added: become a standard way to collect metric, regardless of the purpose of the collection

There is also opportunity to work with Heat to provide a tool to watch for variations in key values in order to trigger various reactions.
To summarize there are 3 tasks that OpenStack Ceilometer do:
  • Metering
  • Multi-Publishing
  • Alarming
Ceilometer Architecture
image source: https://wiki.openstack.org/w/images/7/75/Support-standard-audit-formats-diagram-options-push-pull.png

This diagram show not only the components of Ceilometer but also a very important concept in metering/data collection - how data is collected.

There are 3 ways for data collection:
Bus-Listener Agent
  • This is the most preferred method
  • Integrated with the OpenStack Oslo project
Push Agent
  • Data is "push" from the source into Ceilometer
  • Able to support resilience/High Availability
Polling Agent
  • Least preferred way
  • Ceilometer will poll the various entities and ask for the data
  • Polling is done in a configured interval
  • Very difficult to support resilience/High Availability
eNovance which is now part of Red Hat has a slide in SlideShare that depicts the work flow of Ceilometer:
 image source: http://image.slidesharecdn.com/ceiloplusheatequalsalarmingicehousesummit-131108053358-phpapp01/95/ceilometer-heat-alarming-4-638.jpg?cb=1383953649

Once data are store in the backend database, they can be retrieved via the Ceilometer REST API

image source: http://docs.openstack.org/developer/ceilometer/_images/2-accessmodel.png

From the above section we see that publishing is part of Ceilometer's work flow.  What does it mean by "Multi-Publishing"?
image source: http://docs.openstack.org/developer/ceilometer/_images/3-Pipeline.png

In Ceilometer data/meter is transformed and published to the receiver.

A single piece or a single set of data/meter can be used by multiple entities.  These entities that uses the data/meter has different requirement as to how the data is collected.  From the OpenStack Documentation it outlines 2 aspect of data collection that can pose conflict to the data depending on the requester of the data:

  • Frequency of publication. Typically a meter that you publish for billing need will need to be updated every 30 min while the same meter needed for performance tuning may be needed every 10 seconds.
  • Transport. In the case of data intended for a monitoring system, losing an update or not ensuring security (non-repudiability) of a message is not really a problem, while the same meter will need both security and guaranteed delivery in the case of data intended for rating and billing systems.

To solve this problem, come the multi-publisher allowing the same technical meter to be published multiple times to multiple destinations, each potentially using a different transport and frequency of publication. Three transports have been implemented so far:
  • notifier - a notification based publisher which pushes samples to a message queue.
  • rpc - he original and relatively secure RPC based publisher.
  • udp - which publishes samples using UDP packets.
This diagram show a single sample is sent to 2 of the transports - RPC and UDP which in turns publish the same data to 2 different target:
image source: http://docs.openstack.org/developer/ceilometer/_images/5-multi-publish.png

This component is introduced in the Havana release. It allows user to set alarms based on threshold evaluation for a collection of samples. An alarm can be set on a single meter, or on a combination.

There are 2 actions that can be associated with an alarm:
  1. HTTP callback: you provide a URL to be called whenever the alarm has been set off. The payload of the request contains all the details of why the alarm was triggered.
  2. Log: mostly useful for debugging, stores alarms in a log file.
The biggest use case for this is to integrate this with OpenStack Heat to provide autoscaling of resources in the OpenStack Infrastructure. For a more detailed description of how this work you can go to this eNorance blog.

Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 19: Storage Polices for Object Storage
OpenStack Series Part 20: Group-based Policy for Neutron

"OpenStack." System Architecture — Ceilometer 2015.1.dev146.g6eac318 Documentation. N.p., n.d. Web. 02 Nov. 2014.
"History, Objectives and Limits of the Ceilometer Project." ENovance Techs. N.p., n.d. Web. 02 Nov. 2014.


  1. +$3,624 profit last week...

    Subscribe For 5 Star verified winning bets on MLB, NHL, NBA & NFL + Anti-Vegas Smart Money Signals!!!

  2. It has been simply incredibly generous with you to provide openly what exactly many individuals would’ve marketed for an eBook to end up making some cash for their end, primarily given that you could have tried it in the event you wanted.

    AWS Training in Bangalore
    Python Training in Bangalore

  3. Great Article. Thank you for sharing! Really an awesome post for every one.

    IEEE Final Year projects Project Centers in Chennai are consistently sought after. Final Year Students Projects take a shot at them to improve their aptitudes, while specialists like the enjoyment in interfering with innovation. For experts, it's an alternate ball game through and through. Smaller than expected IEEE Final Year project centers ground for all fragments of CSE & IT engineers hoping to assemble. Final Year Project Domains for IT It gives you tips and rules that is progressively critical to consider while choosing any final year project point.

    Spring Framework has already made serious inroads as an integrated technology stack for building user-facing applications. Spring Framework Corporate TRaining the authors explore the idea of using Java in Big Data platforms.
    Specifically, Spring Framework provides various tasks are geared around preparing data for further analysis and visualization. Spring Training in Chennai