A project on Data Plane Development Kit

Project Title

A VPP plugin utilizing Intel® DPDK and QuickAssist Technology to perform hardware assisted compression operations.

Project Description

The Intel® QuickAssist Technology is a powerful hardware based solution to perform crypto and compression operations. QuickAssist offloads the operations from CPU to the 89XX communications chip. A VPP plugin that utilizes the QuickAssist feature for data compression can be used as a graph node and can be called by the packet processing graph.

Acceleration Enhancements for DPDK currently already has the ability to perform cryptographic operations either by software or hardware depending on the capabilities of the processor that the code is running on. This project will be adding compression to the Acceleration Enhancements for DPDK and at this time limit to hardware based only while later only add in software based compression similar to the crypto counter part.

This VPP plugin will use this compression feature that will be added to the Acceleration Enhancements for DPDK to perform the compression operation.

Below are 2 diagrams that I get from the Internet and included in the proposed project in Intel Developer Mesh for clarity:

image source: https://wiki.fd.io/images/c/c9/VPP_custom_application_packet_processing_graph.280.jpg

image source: http://images.slideplayer.com/39/10926601/slides/slide_5.jpg

Project Use Case

IP payload compression (RFC 3173) that help to save bandwidth can make good use of this hardware assist technology to speed up the compression operations.

Getting Started

To start this project I first looked at what Intel’s QuickAssist Technology is. Of course Google is the first place I went. I found this article from Admin Magazine very helpful in getting me started. It helped me to have a general understand of the features and a high level understand on how QuickAssist Technology works. One thing I like about this article and found it useful is the block diagrams and also it explains how this QuickAssist Technology can boost performance of different use cases and of course NFV is what I care about the most.

image source: http://www.admin-magazine.com/var/ezflow_site/storage/images/archive/2016/33/an-introduction-to-intel-quickassist-technology/figure-3/125991-1-eng-US/Figure-3_reference.png

Next this web site provides tons of useful resources and most of all it contains the source file for the Linux Driver for the hardware that supports this QuickAssist Technology as well as the programmer’s Guide along with the Cryptographic and Compression API Reference Manual.

This article is also a good resource in understand the uses cases of QuickAssist Technology and Network functions.

With this in place, I download the DPDK source code and the DPDK Programmer's Guide as well as the Getting started Guide.

Initially for this proposed project my idea was that currently DPDK already had the QAT related PMD (Pull Mode Driver) and a sample program and thus it should be simple to make "some" modification to add in compression support.  Both the Crypto PMD driver and dpdk-qat sample application has useful documentations.

With all the source code and documentations, I am all ready to go.

Hitting a Road Block

Out of my surprise, currently everything in the DPDK code base is geared specifically on crypto and no mention of compression. I found myself hitting a road block. I am not sure if I should modify the existing Crypto QAT driver or come up with a Compression QAT driver. First of all I see that this is a design decision and also being new to DPDK, I am not quite ready to write a new Pull Mode Driver from scratch.  I need more time and information.

Reaching out to the DPDK Community for help

I then reach out to the DPDK developer mailing list but unfortunately, this is over the Christmas holiday and only after 3 days that I got one reply saying compression is not currently supported as well as point me to a DPDK QAT documentation.

Another day passed, still no further reply on my query and I sent out another Email to the mailing list. This time I also mention not getting any help from the DPDK mailing list. In that afternoon, I get a reply from an Intel program manager. He told me his team worked on crypto portion of QAT and kept me engaged over Email.After explaining to him where I am coming from he said most of his team member are on vacation and when they come back in January 2017, they will see if they have the resources and priority to get this going and is happy to include me in making this to work.

Project Scope needs adjustment

It seems that the scope of this project is bigger than I have expected and takes longer time and more collaboration of the DPDK community to complete.

Current Status of this Project

While waiting for the DPDK design decision and collaboration, I am currently  looking at building a VPP plugin.

I am only able to complete this much by December 30, 2016. There is no code to show and nothing to demo yet.

I, however, even not able to finish this project before 2016, I am going to complete this proposed project in 2017. Will update the latest status as I move along.

Are you interested in working on this project?

Join me in this project if you are interested and make it to completion. Contact me at the Intel Dev Mesh website or ping me on Twitter - @vCloudernBeer.

Programming made simple.

With the advent of DevOps, more and more people are finding the needs to do basic programming.  While the need to do programming for "non-programmers" is mounting, I see that a lots of people is looking at programming as a tall mountain and they are finding this hurdle difficult to overcome.

This blog post has the intent to try to make programming a easy task for "non-programmers". This post is for programming in general and is not any programming language specific.

All programming language can be simplified into 3 basic operations or ingredients.  These are the building blocks for all programming languages be it the simple "Hello World" or as complicated as artificial intelligent software.

The 3 basic building blocks are:

1. Assignment
2. "If-then-else" / conditional statemen
3. Iterations

Assignment

Most computer program has to deal with data manipulation.  Often time a block of memory is reserved called a variable.  This variable should be named and also with a meaningful name to remind even the author of the program as to what the variable is for. Some programmers are lazy and name their variables x, y and/or foo instead of meaningful name such as return_code, username ... etc.  (As a side point, when developing a a software program, all the logics seem so obvious but then 3 months down the road in the middle of the night, we might be scratching our head asking, why did I write this logic).

Anyways, getting back to the topic, assignment is when we assign a value to a variable.  Depending on the programming language, some variable are very strict in the type of the variable.  If a variable is reserved/declared as integer, we can only assign a number value to this variable.  There are other programming language such as Python that the type is not checked by the interrupter/compiler.  Type checking is another big topic that we can look into and in this post we will just concentrate on the 3 building blocks of computer programming.

Example of variable assignment in go:

var string1 string
string1 = "Hello World!" 
fmt.Printf("I just wrote my first program: "%s", string1)

Depending on the context, the variable "string1" may not be a good name to use as it did not reveal how this variable is being used.

If-then-else or conditional statement

This is called the conditional statement and different programming language has different syntax to express the conditional statement but the concept is the same.  Depending of the condition (or the value of certain variables) the program will execute different sets of logic.


The format is:

if condition A {
 do something
} else {
 do some other thing
}

The "else" part is where condition A is not true.  Or we can be checking the condition in anther way:

if condition A is not true "
 do something
} else {
 do some other thing
}

For example:

if _, err := checkMonthIndexSize(i); err != nil {

fmt.Printf("\n%s\n", err)

} else {

fmt.Printf("\nSlice is initialized correctly (len = %d)\n", i)

}

The above logic is written in go and it is only for demonstrating purpose.  This is a more complex construct of the if-then-else logic where we we execute a function and depending on the return value or error condition of the function to decide if we should print out an error message or a informational message.


The if statement is checking for the return value of the function checkMonthIndexSize().  I have not include the function as I am only trying to illustrate the "if-then-else". The function checkMonthIndexSize() returns a value where we choose to ignore and the build in error checking feature of the go language.

In fact this is taken from the demo program that I have written for one of the episodes in "30-days-in-committmas 2015".  If you are interested you can find the podcast here in YouTube

Iterations

Programming usually involve in doing the same operation multiple times or depending on the value of a variable.

Most programming languages have the "for-loop" and the "while-loop".

Example of a for-loop:

for i := 6; i < 12; i++ {

fmt.Printf(" [%d] := %s\r\n", i, string1[i])

}

Usually a "for-loop" will do 3 things:
Assign a value to a variable and this case i := 6
check certain condition and in this case if ( value of the valuable is less than 12)
increment the value of i by one and in this case i++ or it can also be i = i + 1)

Example of a while-loop


while ( condition is true) {
 print a "dot" on the screen
}

This is good for user interface.  Let say we are doing a file transfer using ftp and we can see the progress by seeing dots being printed to the screen and we know the file transfer is still happening.  (Over simplified example for file transfer logic for illustration only).

So we can see this value assignment and condition checking is being used in the for-loop or while-loop.

Happy programming

Of course there are more to programming but this post is trying to help you get over the barrier and start programming.  You can see there 3 basic elements in very complex programs.

Of course there are data structure and object oriented approach plus other features to programming for the complex programs but we can use these 3 basic building blocks to start.  We got to start somewhere.

So: Happy Programming.