Saturday, November 15, 2014

OpenStack Series: Part 15 - Messaging and Queuing system in OpenStack

OpenStack uses a message broker to coordinate operations and status information among services. The Message Broker Service typically runs on the controller node

All OpenStack services has a RESTful API for external entities to communicate with them.  Within each services AMQP (Advanced Message Queue Protocol) is used for the sub task to communicate with each other.

image source: https://wiki.openstack.org/w/images/f/f8/ZonesArchitecture_sm.png

The above diagram show RabbitMQ and it is one of the 3 message queuing service backend supported in OpenStack.  The 3 backends are:
RabbitMQ seems to be the most accepted backend.  Red Hat used to favor using Qpid for its distribution of OpenStack but in the latest release Red Hat switch to using RabbitMQ as the default.

RabbitMQ and Qpid are AMQP brokers.  Explanation of the AMQP architecture can be found here. ZeroMQ is used in OpenStack as a message queuing service backend but is not exactly a AMQP broker.   

AMQP is a family of messaging protocols, while ZeroMQ/ØMQ is a library of messaging functionality. We do not use AMQP directly but rather download and use a specific AMQP implementation such as OpenAMQ or RabbitMQ.

It seems to me the OpenStack Oslo project is using ZeroQM while OpenStack Nova is using either RabbitMQ or Qpid.

Looking at a layering perspective, RPC is the network protocol used underneath the AMQP broker.  There are 2 mode of RPC:
  • rpc.cast (one way) – don’t wait for result
  • rpc.call (request+response) – wait for result (when there is something to return)
The message broker node has this internal elements:

· Topic Publisher:
· Direct Consumer:
· Topic Consumer:
· Direct Publisher:.
· Topic Exchange:.
· Direct Exchange:
· Queue Element:

Detailed description of these elements and RPC operation in OpenStack Nova can be found in OpenStack Documentation or this blog

What about Security
According to OpenStack Documentation:

The message queue is a critical piece of infrastructure that supports a number of OpenStack services but is most strongly associated with the Compute service


One of the larger concerns that remains is that many systems have access to this queue and there is no way for a consumer of the queue messages to verify which host or service placed the messages on the queue. An attacker who is able to successfully place messages on the queue is able to create and delete VM instances, attach the block storage of any tenant and a myriad of other malicious actions

Eric Bruno has a nice article on messaging security.
 image source: http://www.buildyourbestcloud.com/475/openstack-security-nugget-messaging

 In the above diagram the "Management" I believe is the Message Broker Service.

When we talk about security in OpenStack, Keystone is always in the picture.  User or services authenticate themselves to the Keystone server and obtain a token.  For messaging, this token is used so that the Message Broker Service will know if the sender of the message has the authorization to perform such an action.

It is also recommended to secure the transport layer of the message with SSL.
  • AMQP based solutions (Qpid and RabbitMQ) support transport-level security using SSL.
  • ZeroMQ messaging does not natively support SSL, but transport-level security is possible using labelled IPSec or CIPSO network labels.

Simple Authentication and Security Layer (SASL) is a framework for authentication and data security in Internet protocols. Both RabbitMQ and Qpid offer SASL and other pluggable authentication mechanisms beyond simple usernames and passwords that allow for increased authentication security. While RabbitMQ supports SASL, support in OpenStack does not currently allow for requesting a specific SASL authentication mechanism. RabbitMQ support in OpenStack allows for either username and password authentication over an unencrypted connection or username and password in conjunction with X.509 client certificates to establish the secure SSL connection.

In addition to authentication, RabbitMQ and Qpid offer access control mechanisms for controlling access to queues. ZeroMQ offers no such mechanisms.

Related Post:
OpenStack Series Part 1: How do you look at OpenStack?
OpenStack Series Part 2: What's new in the Juno Release?
OpenStack Series Part 3: Keystone - Identity Service
OpenStack Series Part 4: Nova - Compute Service
OpenStack Series Part 5: Glance - Image Service
OpenStack Series Part 6: Cinder - Block Storage Service
OpenStack Series Part 7: Swift - Object Storage Service
OpenStack Series Part 8: Neutron - Networking Service
OpenStack Series Part 9: Horizon - a Web Based UI Service
OpenStack Series Part 10: Heat - Orchestration Service
OpenStack Series Part 11: Ceilometer - Monitoring and Metering Service
OpenStack Series Part 12: Trove - Database Service
OpenStack Series Part 13: Docker in OpenStack
OpenStack Series Part 14: Sahara - Data Processing Service
OpenStack Series Part 16: Ceph in OpenStack
OpenStack Series Part 17: Congress - Policy Service
OpenStack Series Part 18: Network Function Virtualization in OpenStack
OpenStack Series Part 19: Storage Polices for Object Storage
OpenStack Series Part 20: Group-based Policy for Neutron

Reference:
"Zeromq." Welcome from AMQP -. N.p., n.d. Web. 02 Nov. 2014.
"OpenStack." AMQP and Nova — Nova 2015.1.dev637.g2a7f3af Documentation. N.p., n.d. Web. 02 Nov. 2014.

3 comments: