Thursday, December 29, 2016

A project on Data Plane Development Kit


Project Title

A VPP plugin utilizing Intel® DPDK and QuickAssist Technology to perform hardware assisted compression operations.

Project Description


The Intel® QuickAssist Technology is a powerful hardware based solution to perform crypto and compression operations. QuickAssist offloads the operations from CPU to the 89XX communications chip. A VPP plugin that utilizes the QuickAssist feature for data compression can be used as a graph node and can be called by the packet processing graph.

Acceleration Enhancements for DPDK currently already has the ability to perform cryptographic operations either by software or hardware depending on the capabilities of the processor that the code is running on. This project will be adding compression to the Acceleration Enhancements for DPDK and at this time limit to hardware based only while later only add in software based compression similar to the crypto counter part.

This VPP plugin will use this compression feature that will be added to the Acceleration Enhancements for DPDK to perform the compression operation.

Below are 2 diagrams that I get from the Internet and included in the proposed project in Intel Developer Mesh for clarity:

image source: https://wiki.fd.io/images/c/c9/VPP_custom_application_packet_processing_graph.280.jpg


image source: http://images.slideplayer.com/39/10926601/slides/slide_5.jpg

Project Use Case

IP payload compression (RFC 3173) that help to save bandwidth can make good use of this hardware assist technology to speed up the compression operations.

Getting Started

To start this project I first looked at what Intel’s QuickAssist Technology is. Of course Google is the first place I went. I found this article from Admin Magazine very helpful in getting me started. It helped me to have a general understand of the features and a high level understand on how QuickAssist Technology works. One thing I like about this article and found it useful is the block diagrams and also it explains how this QuickAssist Technology can boost performance of different use cases and of course NFV is what I care about the most.
image source: http://www.admin-magazine.com/var/ezflow_site/storage/images/archive/2016/33/an-introduction-to-intel-quickassist-technology/figure-3/125991-1-eng-US/Figure-3_reference.png

Next this web site provides tons of useful resources and most of all it contains the source file for the Linux Driver for the hardware that supports this QuickAssist Technology as well as the programmer’s Guide along with the Cryptographic and Compression API Reference Manual.

This article is also a good resource in understand the uses cases of QuickAssist Technology and Network functions.

With this in place, I download the DPDK source code and the DPDK Programmer's Guide as well as the Getting started Guide.

Initially for this proposed project my idea was that currently DPDK already had the QAT related PMD (Pull Mode Driver) and a sample program and thus it should be simple to make "some" modification to add in compression support.  Both the Crypto PMD driver and dpdk-qat sample application has useful documentations.

With all the source code and documentations, I am all ready to go.

Hitting a Road Block

Out of my surprise, currently everything in the DPDK code base is geared specifically on crypto and no mention of compression. I found myself hitting a road block. I am not sure if I should modify the existing Crypto QAT driver or come up with a Compression QAT driver. First of all I see that this is a design decision and also being new to DPDK, I am not quite ready to write a new Pull Mode Driver from scratch.  I need more time and information.

Reaching out to the DPDK Community for help

I then reach out to the DPDK developer mailing list but unfortunately, this is over the Christmas holiday and only after 3 days that I got one reply saying compression is not currently supported as well as point me to a DPDK QAT documentation.

Another day passed, still no further reply on my query and I sent out another Email to the mailing list. This time I also mention not getting any help from the DPDK mailing list. In that afternoon, I get a reply from an Intel program manager. He told me his team worked on crypto portion of QAT and kept me engaged over Email.After explaining to him where I am coming from he said most of his team member are on vacation and when they come back in January 2017, they will see if they have the resources and priority to get this going and is happy to include me in making this to work.

Project Scope needs adjustment

It seems that the scope of this project is bigger than I have expected and takes longer time and more collaboration of the DPDK community to complete.

Current Status of this Project

While waiting for the DPDK design decision and collaboration, I am currently  looking at building a VPP plugin.

I am only able to complete this much by December 30, 2016. There is no code to show and nothing to demo yet.

I, however, even not able to finish this project before 2016, I am going to complete this proposed project in 2017. Will update the latest status as I move along.

Are you interested in working on this project?

Join me in this project if you are interested and make it to completion. Contact me at the Intel Dev Mesh website or ping me on Twitter - @vCloudernBeer.


77 comments:

  1. Great post! I am actually getting ready to across this information, It's very helpful for this blog.Also great with all of the valuable information you have Keep up the good work you are doing well.

    digital marketing company in india

    ReplyDelete
  2. Wonderful bloggers like yourself who would positively reply encouraged me to be more open and engaging in commenting.So know it's helpful.

    GMAT Classes Chennai

    ReplyDelete
  3. i saw your blog it is really good and very much interesting too, thus i like your information what you have posted so please update latest information too.


    Best Laser Clinic In Chennai | Best Root Canal Treatment In Chennai | Best Implant clinic In Chennai

    ReplyDelete

  4. These provided information was really so nice,i agree our all points because all is very good information provided this through in the post.
    It is very helpful for me.

    Best Dental Clinic in Velachery

    ReplyDelete
  5. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lotfor all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    Web Development Company in India

    ReplyDelete

  6. Excellent article,it was helpful to us to learn more and useful to teach others.This like valuable information is very interesting to read

    Digital Marketing Company in Chennai

    ReplyDelete
  7. superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a greatblog with us. expecting for your..
    SEO Company in India
    Digital Marketing Company in India

    ReplyDelete
  8. The site contains a very great blog. the information present in this site will be very useful for us. thank you for sharing the blog with us.
    SAT Training Centre in Chennai

    ReplyDelete
  9. Superb explanation & it's too clear to understand the concept as well, keep sharing admin with some updated information with right examples.Keep update more posts.

    Dental Hospital In Chennai

    ReplyDelete
  10. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
    Hadoop Training in chennai

    ReplyDelete
  11. Thank you for taking the time to provide us with your valuable information. We strive to provide our candidates with excellent care and we take your comments to heart.As always, we appreciate your confidence and trust in us
    Best Dental Clinic In Chennai

    ReplyDelete
  12. This is an awesome post.Really very informative and creative contents.These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.

    Digital Marketing Company in Chennai

    ReplyDelete

  13. This is extremely helpful info!! Very good work. It is very interesting to learn and easy to understood. Thank you for giving information. Please let us know and more information get post to link.

    AWS Training in Chennai

    ReplyDelete
  14. This is an awesome post.Really very informative and creative contents. These concept is a good way to enhance the knowledge.I like it and help me to development very well.Thank you for this brief explanation and very nice information.Well, got a good knowledge.
    Manpower Consultancy in Chennai
    Hr Consultancy in Chennai

    ReplyDelete
  15. Superb. I really enjoyed very much with this article here. Really it is an amazing article I had ever read. I hope it will help a lot for all. Thank you so much for this amazing posts and please keep update like this excellent article.thank you for sharing such a great blog with us. expecting for your..
    SEO Company in India

    ReplyDelete

  16. "i am glad to visit your blog and i really like your writing this , its very interisting to read and helpful as well.
    i want to learn a lot from you .
    Thank You."!!!
    search engine marketing agency

    ReplyDelete
  17. Wow what a Great Information about World Day its very nice informative post. thanks for the post.
    role of digital marketing

    ReplyDelete
  18. If you need your ex-girlfriend or ex-boyfriend to come crawling back to you on their knees (no matter why you broke up) you must watch this video
    right away...

    (VIDEO) Why your ex will NEVER come back...

    ReplyDelete
  19. Good post, thanks for sharing this useful information.
    Click More Details Visit Our Website

    Texus
    Texus Systems

    ReplyDelete
  20. Do you need free Instagram Likes?
    Did you know you can get them ON AUTOPILOT & TOTALLY FOR FREE by getting an account on Add Me Fast?

    ReplyDelete
  21. QUANTUM BINARY SIGNALS

    Get professional trading signals delivered to your mobile phone every day.

    Start following our signals NOW and make up to 270% per day.

    ReplyDelete
  22. These are some great tools that i definitely use for SEO work. This is a great list to use in the future..
    digital marketing agency

    ReplyDelete
  23. Truly I discovered this article more useful, much obliged for sharing this article
    Inventory management cloud

    ReplyDelete
  24. Hallo,


    You make learning and reading addictive. All eyes fixed on you. Thank you being such a good and trust worthy guide.

    totally new here (and quite new in the kingdom of ML) and this is my first question;

    I am using XGBClassifier (latest version) in python training a dataset where the observations (should) have different weights. More specifically in am trying modelling the probability for an insurance policy to have a claim. The insurance policies may be on risk (have insurance cover) for different durations e.g. 1 month, 6 month 1 year creating the need to weight them according to how long they are on risk.

    Is it possible to pass this (N*1)
    vectors which sums up to 1
    to the XGBClassifier? If so, is the implication of this that obs. with weights are more likely to be picked by the sampling algo?

    Hope somebody can enlighten me. Please let me know if I need to clarify something.

    Thanks a lot. This was a perfect step-by-step guide. Don’t think it could have been done better.


    Ciao,
    Ajeeth

    ReplyDelete
  25. Informative Blog. Thank you for sharing
    aws training in chennai

    ReplyDelete
  26. This is really too useful and has more ideas from your blog. Keep sharing more blog like this, thank you. We are waiting for your new blog and for useful information.Please contact us for Oracle Financials Training in Ameerpet details in our Erptree Training Institute

    ReplyDelete

  27. Artificial Intelligence Companies in bangalore is the field of computer science dedicated to solving cognitive problems commonly associated with human intelligence, such as learning, problem solving, and pattern recognition.

    artificial intelligence companies in indiaMany business certainly expect AI to be disruptive Technology in the coming days or Testing the Waters currently, but why wait when the technology is already transforming every aspect of the way an organization operates?

    ReplyDelete
  28. Thank you so much for your generosity and for such a professional job! I had heard about your excellent work and performance, but today I was honored to experience it
    cheapest & best dental implants in madipakkam

    ReplyDelete

  29. Your good knowledge and kindness in playing with all the pieces were very useful. I don’t know what I would have done if I had not encountered such a step like this.

    python Training in Bangalore | python Training in Bangalore

    ReplyDelete
  30. Great Article… I love to read your articles because your writing style is too good, its is very very helpful for all of us and I never get bored while reading your article because, they are becomes a more and more interesting from the starting lines until the end.

    rpa training in chennai
    Best rpa training in bangalore
    rpa course in bangalore
    rpa training in marathahalli
    rpa training in btm
    best rpa training in chennai

    ReplyDelete
  31. Excellent blog, I wish to share your post with my folks circle. It’s really helped me a lot, so keep sharing post like this
    Best Devops training in sholinganallur
    Devops training in velachery
    Devops training in annanagar
    Devops training in tambaram

    ReplyDelete
  32. myTectra the Market Leader in Artificial intelligence training in Bangalore
    myTectra offers Artificial intelligence training in Bangalore using Class Room. myTectra offers Live Online Design Patterns Training Globally.Read More

    ReplyDelete

  33. Whoa! I’m enjoying the template/theme of this website. It’s simple, yet effective. A lot of times it’s very hard to get that “perfect balance” between superb usability and visual appeal. I must say you’ve done a very good job with this.

    AWS TRAINING IN BANGALORE|NO.1AWS TRAINING INSTITUTES IN BANGALORE

    ReplyDelete
  34. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  35. A bewildering web journal I visit this blog, it's unfathomably heavenly. Oddly, in this present blog's substance made purpose of actuality and reasonable. The substance of data is informative
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  36. A befuddling web diary I visit this blog, it's incredibly grand. Strangely, in this present blog's substance made motivation behind fact and sensible. The substance of information is instructive
    Oracle Fusion Financials Online Training
    Oracle Fusion HCM Online Training
    Oracle Fusion SCM Online Training

    ReplyDelete
  37. If you're looking for a date but not sure whether the dating site is a safe place, check our https://wizzlove.com/reviews/asiacharm-review review.

    ReplyDelete
  38. Jadi, kembali ke taruhan Anda. Jika Anda percaya bahwa mereka berada di imbang siram setelah kegagalan, maka Anda harus menghapus peluang mereka untuk menelepon. Pada intinya, Anda memaksa mereka harus menelepon taruhan yang “lebih” maka 36%.
    asikqq
    dewaqq
    sumoqq
    interqq
    hobiqq
    rajawaliqq
    http://192.254.236.33/hobiqq78/

    ReplyDelete
  39. Thanks for sharing information
    We are the best piping design course in Hyderabad, India. Sanjary academy Offers Piping Design Course and Best Piping Design Training Institute in Hyderabad. Piping Design Institute in India Piping Design Engineering.
    Piping Design Course
    Piping Design Course in india
    Piping Design Course in hyderabad

    ReplyDelete
  40. Good Information
    Sanjary kids is the best playschool, preschool in Hyderabad, India. Start your play school,preschool in Hyderabad with sanjary kids. Sanjary kids provides programs like Play group,Nursery,Junior KG,Serior KG,and Teacher Training Program.
    play school in hyderabad
    Preschool in hyderabad
    Preschool teacher training course in hyderabad
    pre and primary teacher training course in hyderabad

    ReplyDelete
  41. In this manner my acquaintance Wesley Virgin's biography starts in this shocking and controversial video.

    As a matter of fact, Wesley was in the army-and soon after leaving-he unveiled hidden, "self mind control" secrets that the CIA and others used to get whatever they want.

    These are the same SECRETS many famous people (notably those who "became famous out of nowhere") and elite business people used to become rich and famous.

    You've heard that you utilize only 10% of your brain.

    Mostly, that's because most of your brainpower is UNCONSCIOUS.

    Maybe this thought has even occurred IN YOUR very own head... as it did in my good friend Wesley Virgin's head about 7 years ago, while driving an unregistered, beat-up garbage bucket of a vehicle with a suspended driver's license and $3 in his bank account.

    "I'm so frustrated with living check to check! Why can't I become successful?"

    You've taken part in those types of conversations, right?

    Your success story is waiting to start. You just need to take a leap of faith in YOURSELF.

    UNLOCK YOUR SECRET BRAINPOWER

    ReplyDelete
  42. Awesome article!!!! They have great explanations of the plugin utilizing intel.
    Germany VPS Server Hosting

    ReplyDelete
  43. Very nice article excellent information thanks for share.Keep updating thanks
    GMAT classes chennai

    ReplyDelete
  44. Very nice article excellent information thanks for share.Keep updating thanks Mustache Transplantation Cost

    ReplyDelete
  45. Oh my goodness! an excellent article dude. Many thanks However I am experiencing trouble with ur rss . Do not know why Not able to enroll in it. Will there be any person obtaining identical rss dilemma? Anyone who knows kindly respond. Thnkx data science from scratch

    ReplyDelete
  46. I am very impressed with the article. The content of information is very informative. We are also providing the best services click on below links to visit our website.
    Snowflake Training
    Workday Training
    Okta Training
    AEM Training
    CyberArk Training

    ReplyDelete
  47. Thanks for sharing a useful information. When looking for the Famous Dental Clinic in Dwarka or advanced dental treatment then there can be no better place than Park Dental Clinic.

    Best Dentist in Dwarka Sec7
    Best implantologist in dwarka

    ReplyDelete
  48. Thank you for all your efforts in this It is very interesting Blog ...
    I believe there are many people who feel like I read this article!
    I hope you continue to have articles like this to share with everyone!
    wordpress
    blogspot
    youtube
    ភ្នាល់បាល់​ អនឡាញ

    ReplyDelete
  49. A must read post! Good way of describing and pleasure piece of writing. Thanks!
    BA Result Name Wise

    ReplyDelete
  50. Great Article… I love to read your articles because your writing style is too good, it is very helpful for all of us and I never get bored while reading your article. Australia VPS Server

    ReplyDelete