OpenStack Series: Part 11 Ceilometer – Monitoring and Metering Service

Project Goal/Objective
The Ceilometer project was started in 2012 with one simple goal in mind:to provide an infrastructure to collect any information needed regarding OpenStack projects. It was designed so that rating engines could use this single source to transform events into billable items.

Note: Billing involve 3 steps metering, rating and billing.  Ceilometer only deals with metering.  Application will have to be build on top of Ceilometer to perform the complete billing process.

As the project become mature a secondary goal is added: become a standard way to collect metric, regardless of the purpose of the collection

There is also opportunity to work with Heat to provide a tool to watch for variations in key values in order to trigger various reactions.
To summarize there are 3 tasks that OpenStack Ceilometer do:

• Metering
• Multi-Publishing
• Alarming

Metering
Ceilometer Architecture

 

image source: https://wiki.openstack.org/w/images/7/75/Support-standard-audit-formats-diagram-options-push-pull.png

This diagram show not only the components of Ceilometer but also a very important concept in metering/data collection - how data is collected.

There are 3 ways for data collection:
Bus-Listener Agent

• This is the most preferred method
• Integrated with the OpenStack Oslo project

Push Agent

• Data is "push" from the source into Ceilometer
• Able to support resilience/High Availability

Polling Agent

• Least preferred way
• Ceilometer will poll the various entities and ask for the data
• Polling is done in a configured interval
• Very difficult to support resilience/High Availability

eNovance which is now part of Red Hat has a slide in SlideShare that depicts the work flow of Ceilometer:

 image source: http://image.slidesharecdn.com/ceiloplusheatequalsalarmingicehousesummit-131108053358-phpapp01/95/ceilometer-heat-alarming-4-638.jpg?cb=1383953649

Once data are store in the backend database, they can be retrieved via the Ceilometer REST API

image source: http://docs.openstack.org/developer/ceilometer/_images/2-accessmodel.png

Multi-Publishing
From the above section we see that publishing is part of Ceilometer's work flow.  What does it mean by "Multi-Publishing"?

image source: http://docs.openstack.org/developer/ceilometer/_images/3-Pipeline.png

In Ceilometer data/meter is transformed and published to the receiver.

A single piece or a single set of data/meter can be used by multiple entities.  These entities that uses the data/meter has different requirement as to how the data is collected.  From the OpenStack Documentation it outlines 2 aspect of data collection that can pose conflict to the data depending on the requester of the data:

• Frequency of publication. Typically a meter that you publish for billing need will need to be updated every 30 min while the same meter needed for performance tuning may be needed every 10 seconds.
• Transport. In the case of data intended for a monitoring system, losing an update or not ensuring security (non-repudiability) of a message is not really a problem, while the same meter will need both security and guaranteed delivery in the case of data intended for rating and billing systems.

To solve this problem, come the multi-publisher allowing the same technical meter to be published multiple times to multiple destinations, each potentially using a different transport and frequency of publication. Three transports have been implemented so far:

• notifier - a notification based publisher which pushes samples to a message queue.
• rpc - he original and relatively secure RPC based publisher.
• udp - which publishes samples using UDP packets.

This diagram show a single sample is sent to 2 of the transports - RPC and UDP which in turns publish the same data to 2 different target:

image source: http://docs.openstack.org/developer/ceilometer/_images/5-multi-publish.png

Alarming
This component is introduced in the Havana release. It allows user to set alarms based on threshold evaluation for a collection of samples. An alarm can be set on a single meter, or on a combination.

There are 2 actions that can be associated with an alarm:

1. HTTP callback: you provide a URL to be called whenever the alarm has been set off. The payload of the request contains all the details of why the alarm was triggered.
2. Log: mostly useful for debugging, stores alarms in a log file.

The biggest use case for this is to integrate this with OpenStack Heat to provide autoscaling of resources in the OpenStack Infrastructure. For a more detailed description of how this work you can go to this eNorance blog.

Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series part 15: Messaging and Queuing System in OpenStack
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 19: Storage Polices for Object Storage
OpenStack Series Part 20: Group-based Policy for Neutron

Reference:
"OpenStack." System Architecture — Ceilometer 2015.1.dev146.g6eac318 Documentation. N.p., n.d. Web. 02 Nov. 2014.
"History, Objectives and Limits of the Ceilometer Project." ENovance Techs. N.p., n.d. Web. 02 Nov. 2014.