Thursday, December 29, 2016

A project on Data Plane Development Kit


Project Title

A VPP plugin utilizing Intel® DPDK and QuickAssist Technology to perform hardware assisted compression operations.

Project Description


The Intel® QuickAssist Technology is a powerful hardware based solution to perform crypto and compression operations. QuickAssist offloads the operations from CPU to the 89XX communications chip. A VPP plugin that utilizes the QuickAssist feature for data compression can be used as a graph node and can be called by the packet processing graph.

Acceleration Enhancements for DPDK currently already has the ability to perform cryptographic operations either by software or hardware depending on the capabilities of the processor that the code is running on. This project will be adding compression to the Acceleration Enhancements for DPDK and at this time limit to hardware based only while later only add in software based compression similar to the crypto counter part.

This VPP plugin will use this compression feature that will be added to the Acceleration Enhancements for DPDK to perform the compression operation.

Below are 2 diagrams that I get from the Internet and included in the proposed project in Intel Developer Mesh for clarity:

image source: https://wiki.fd.io/images/c/c9/VPP_custom_application_packet_processing_graph.280.jpg


image source: http://images.slideplayer.com/39/10926601/slides/slide_5.jpg

Project Use Case

IP payload compression (RFC 3173) that help to save bandwidth can make good use of this hardware assist technology to speed up the compression operations.

Getting Started

To start this project I first looked at what Intel’s QuickAssist Technology is. Of course Google is the first place I went. I found this article from Admin Magazine very helpful in getting me started. It helped me to have a general understand of the features and a high level understand on how QuickAssist Technology works. One thing I like about this article and found it useful is the block diagrams and also it explains how this QuickAssist Technology can boost performance of different use cases and of course NFV is what I care about the most.
image source: http://www.admin-magazine.com/var/ezflow_site/storage/images/archive/2016/33/an-introduction-to-intel-quickassist-technology/figure-3/125991-1-eng-US/Figure-3_reference.png

Next this web site provides tons of useful resources and most of all it contains the source file for the Linux Driver for the hardware that supports this QuickAssist Technology as well as the programmer’s Guide along with the Cryptographic and Compression API Reference Manual.

This article is also a good resource in understand the uses cases of QuickAssist Technology and Network functions.

With this in place, I download the DPDK source code and the DPDK Programmer's Guide as well as the Getting started Guide.

Initially for this proposed project my idea was that currently DPDK already had the QAT related PMD (Pull Mode Driver) and a sample program and thus it should be simple to make "some" modification to add in compression support.  Both the Crypto PMD driver and dpdk-qat sample application has useful documentations.

With all the source code and documentations, I am all ready to go.

Hitting a Road Block

Out of my surprise, currently everything in the DPDK code base is geared specifically on crypto and no mention of compression. I found myself hitting a road block. I am not sure if I should modify the existing Crypto QAT driver or come up with a Compression QAT driver. First of all I see that this is a design decision and also being new to DPDK, I am not quite ready to write a new Pull Mode Driver from scratch.  I need more time and information.

Reaching out to the DPDK Community for help

I then reach out to the DPDK developer mailing list but unfortunately, this is over the Christmas holiday and only after 3 days that I got one reply saying compression is not currently supported as well as point me to a DPDK QAT documentation.

Another day passed, still no further reply on my query and I sent out another Email to the mailing list. This time I also mention not getting any help from the DPDK mailing list. In that afternoon, I get a reply from an Intel program manager. He told me his team worked on crypto portion of QAT and kept me engaged over Email.After explaining to him where I am coming from he said most of his team member are on vacation and when they come back in January 2017, they will see if they have the resources and priority to get this going and is happy to include me in making this to work.

Project Scope needs adjustment

It seems that the scope of this project is bigger than I have expected and takes longer time and more collaboration of the DPDK community to complete.

Current Status of this Project

While waiting for the DPDK design decision and collaboration, I am currently  looking at building a VPP plugin.

I am only able to complete this much by December 30, 2016. There is no code to show and nothing to demo yet.

I, however, even not able to finish this project before 2016, I am going to complete this proposed project in 2017. Will update the latest status as I move along.

Are you interested in working on this project?

Join me in this project if you are interested and make it to completion. Contact me at the Intel Dev Mesh website or ping me on Twitter - @vCloudernBeer.


34 comments:

  1. Wonderful bloggers like yourself who would positively reply encouraged me to be more open and engaging in commenting.So know it's helpful.

    GMAT Classes Chennai

    ReplyDelete

  2. These provided information was really so nice,i agree our all points because all is very good information provided this through in the post.
    It is very helpful for me.

    Best Dental Clinic in Velachery

    ReplyDelete
  3. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lotfor all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    Web Development Company in India

    ReplyDelete

  4. Excellent article,it was helpful to us to learn more and useful to teach others.This like valuable information is very interesting to read

    Digital Marketing Company in Chennai

    ReplyDelete
  5. The site contains a very great blog. the information present in this site will be very useful for us. thank you for sharing the blog with us.
    SAT Training Centre in Chennai

    ReplyDelete
  6. Superb explanation & it's too clear to understand the concept as well, keep sharing admin with some updated information with right examples.Keep update more posts.

    Dental Hospital In Chennai

    ReplyDelete
  7. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
    Best Dental Clinic In Chennai

    ReplyDelete
  8. This is an awesome post.Really very informative and creative contents.These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    Digital Marketing Company in Chennai

    ReplyDelete

  9. "i am glad to visit your blog and i really like your writing this , its very interisting to read and helpful as well.
    i want to learn a lot from you .
    Thank You."!!!
    search engine marketing agency

    ReplyDelete
  10. Wow what a Great Information about World Day its very nice informative post. thanks for the post.
    role of digital marketing

    ReplyDelete
  11. Do you need free Instagram Likes?
    Did you know you can get them ON AUTOPILOT & TOTALLY FOR FREE by getting an account on Add Me Fast?

    ReplyDelete
  12. These are some great tools that i definitely use for SEO work. This is a great list to use in the future..
    digital marketing agency

    ReplyDelete
  13. Hallo,


    You make learning and reading addictive. All eyes fixed on you. Thank you being such a good and trust worthy guide.

    totally new here (and quite new in the kingdom of ML) and this is my first question;

    I am using XGBClassifier (latest version) in python training a dataset where the observations (should) have different weights. More specifically in am trying modelling the probability for an insurance policy to have a claim. The insurance policies may be on risk (have insurance cover) for different durations e.g. 1 month, 6 month 1 year creating the need to weight them according to how long they are on risk.

    Is it possible to pass this (N*1)
    vectors which sums up to 1
    to the XGBClassifier? If so, is the implication of this that obs. with weights are more likely to be picked by the sampling algo?

    Hope somebody can enlighten me. Please let me know if I need to clarify something.

    Thanks a lot. This was a perfect step-by-step guide. Don’t think it could have been done better.


    Ciao,
    Ajeeth

    ReplyDelete
  14. This is really too useful and has more ideas from your blog. Keep sharing more blog like this, thank you. We are waiting for your new blog and for useful information.Please contact us for Oracle Financials Training in Ameerpet details in our Erptree Training Institute

    ReplyDelete

  15. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.

    python Training in Bangalore | python Training in Bangalore

    ReplyDelete
  16. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  17. Jadi, kembali ke taruhan Anda. Jika Anda percaya bahwa mereka berada di imbang siram setelah kegagalan, maka Anda harus menghapus peluang mereka untuk menelepon. Pada intinya, Anda memaksa mereka harus menelepon taruhan yang “lebih” maka 36%.
    asikqq
    dewaqq
    sumoqq
    interqq
    hobiqq
    rajawaliqq
    http://192.254.236.33/hobiqq78/

    ReplyDelete
  18. In this manner my acquaintance Wesley Virgin's biography starts in this shocking and controversial video.

    As a matter of fact, Wesley was in the army-and soon after leaving-he unveiled hidden, "self mind control" secrets that the CIA and others used to get whatever they want.

    These are the same SECRETS many famous people (notably those who "became famous out of nowhere") and elite business people used to become rich and famous.

    You've heard that you utilize only 10% of your brain.

    Mostly, that's because most of your brainpower is UNCONSCIOUS.

    Maybe this thought has even occurred IN YOUR very own head... as it did in my good friend Wesley Virgin's head about 7 years ago, while driving an unregistered, beat-up garbage bucket of a vehicle with a suspended driver's license and $3 in his bank account.

    "I'm so frustrated with living check to check! Why can't I become successful?"

    You've taken part in those types of conversations, right?

    Your success story is waiting to start. You just need to take a leap of faith in YOURSELF.

    UNLOCK YOUR SECRET BRAINPOWER

    ReplyDelete
  19. Awesome article!!!! They have great explanations of the plugin utilizing intel.
    Germany VPS Server Hosting

    ReplyDelete
  20. Very nice article excellent information thanks for share.Keep updating thanks Mustache Transplantation Cost

    ReplyDelete
  21. Oh my goodness! an excellent article dude. Many thanks However I am experiencing trouble with ur rss . Do not know why Not able to enroll in it. Will there be any person obtaining identical rss dilemma? Anyone who knows kindly respond. Thnkx data science from scratch

    ReplyDelete
  22. Thank you for all your efforts in this It is very interesting Blog ...
    I believe there are many people who feel like I read this article!
    I hope you continue to have articles like this to share with everyone!
    wordpress
    blogspot
    youtube
    ភ្នាល់បាល់​ អនឡាញ

    ReplyDelete
  23. A must read post! Good way of describing and pleasure piece of writing. Thanks!
    BA Result Name Wise

    ReplyDelete
  24. Great Article… I love to read your articles because your writing style is too good, it is very helpful for all of us and I never get bored while reading your article. Australia VPS Server

    ReplyDelete
  25. This blog is a wonderful example of how the power of writing can make a difference in the world. Keep up the amazing work. Are you searching for SEO Agentur Wien that grows your business online presence in the world.. Then visit Fabians-Webdesign...........

    ReplyDelete