This Open-Source Library Accelerates AI Inference by 5-20x in a Few Lines of Code – hackernoon.com

The nebullvm library is an open-source tool to accelerate AI computing. It takes your AI model as input and outputs an optimized version that runs 5-20 times faster on your hardware. Nebullvm is quickly becoming popular, with 250+ GitHub stars on release day. The library aims to be: deep learning model agnostic. It takes a few lines of code to install the library and optimize your models. It is easy-to-use. It runs locally on your machine. Everything runs locally.

Your one stop-shop for AI acceleration.

How doesnebullvmwork?

It takes your AI model as input and outputs an optimized version that runs 5-20 times faster on your hardware. In other words, nebullvm tests multiple deep learning compilers to identify the best possible way to execute your model on your specific machine, without impacting the accuracy of your model.

And that's it. In just a few lines of code.

And a big thank you to everyone for supporting this open-source project! The library received 250+ Github stars on release day, and that's just amazing

Let's learn more about nebullvm and AI optimization. Where should we start? From...

Or let's jump straight to the library nebullvm

Finally, the adoption of Artificial Intelligence (AI) is growing rapidly, although we are still far from exploiting the full potential of this technology.

Indeed, what typically happens is that AI developers spend most of their time on data analysis, data cleaning, and model testing/training with the objective of building very accurate AI models.

Yet... few models make it into production. If they do, two situations arise:

AI models are developed by skilled data scientists and great AI engineers, who often have limited experience with cloud, compilers, hardware, and all the low-level matters. When their models are ready to be deployed, they select the first GPU or CPU they can think of on the cloud or their company/university server, unaware of the severe impact on model performance (i.e. much slower and more expensive computing) caused by uninformed hardware selection, poor cloud infrastructure configuration, and lack of model/hardware post-training optimization.

Other companies have developed in-house AI models that work robustly. AI inference is critical to these companies, so they often build a team of hardware/cloud engineers who spend hours looking for out-of-the-box methods to optimize model deployment.

Do you fall into one of these two groups? Then you might be interested in the nebullvm library, and below we explain why.

How does nebullvm work?

You import the library, nebullvm does some magic, and your AI model will run 5-20 times faster.

And that's it. In just a few lines of code.

The goal of nebullvm library is to let any developer benefit from deep learning compilers without having to waste tons of hours understanding, installing, testing and debugging this powerful technology.

Nebullvm is quickly becoming popular, with 250+ GitHub stars on release day and hundreds of active users from both startups and large tech companies. The library aims to be:

Deep learning model agnostic.nebullvmsupports all the most popular architectures such as transformers, LSTMs, CNNs and FCNs.

Hardware agnostic. The library now works on most CPUs and GPUs and will soon support TPUs and other deep learning-specific ASICs.

Framework agnostic.nebullvmsupports the most widely used frameworks (PyTorch, TensorFlow and Hugging Face) and will soon support many more.

Secure.Everything runs locally on your machine.

Easy-to-use. It takes a few lines of code to install the library and optimize your models.

Leveraging the best deep learning compilers. There are tons of DL compilers that optimize the way your AI models run on your hardware. It would take tons of hours for a developer to install and test them at every model deployment. The library does it for you!

Why is accelerating computing by 5-20x so valuable?

Tosave time Accelerate your AI services and make them real-time.

Tosave money Reduce cloud computing costs.

Tosave energy Reduce the electricity consumption and carbon footprint of your AI services.

Probably you can easily grasp how accelerated computing can benefit your specific use case. We'll also provide you with some use cases on how nebullvm is helping many in the community across different sectors:

Fast computing makes search and recommendation engines faster, which leads to a more enjoyable user experience on websites and platforms. Besides, near real-time AI is a strict requirement for many healthtech companies and for autonomous driving, when slow response time can put people's lives in danger. The metaverse and the gaming industry also require near-zero latency to allow people to interact seamlessly. Speed can also provide an edge in sectors such as crypto/NFT/fast trading.

Lowering costs with minimal effort never hurts anyone. There is little to explain about this.

Green AI is a topic that is becoming more popular over time. Everyone is well aware of the risks and implications of climate change and it is important to reduce energy consumption where possible. Widespread awareness of the issue is reflected in how purchasing behavior across sectors is moving toward greater sustainability. In addition, low power consumption is a system requirement in some cases, especially on IoT/edge devices that may not be connected to continuous power sources.

We suggest testing the library on your AI model right away by following the installation instructions onGithub. If instead you want to get a hands-on sense of the library's capabilities, check out thenotebooks at this linkwhere you can test nebullvm on popular deep learning models. Note that notebooks will still require you to install the library as you will to test nebullvm on your models, which will take several minutes. Once it's installed, nebullvm will optimize your models in a short time.

We have also tested nebullvm on popular AI models and hardware from leading vendors.

At first glance, we can observe that acceleration varies greatly across hardware-model couplings. Overall, the library provides great positive results, most ranging from 2 to 30 times speedup.

To summarize, the results are:

Nebullvm provides positive acceleration to non-optimized AI models

The table below shows the response time in milliseconds (ms) of the non-optimized model and the optimized model for the various model-hardware couplings as an average value over 100 experiments. It also displays thespeedupprovided by nebullvm, where speedup is defined as the response time of the optimized model over the response time of the non-optimized model.

Hardware used for the experiment is the following:

Nebullvm leverages the best deep learning compilers to accelerate AI models in inference.

So what exactly are deep learning compilers?

A deep learning compiler takes your model as input and produces an efficient version of it that runs the model computation graph faster on a specific hardware.

How?

There are several methods that, in principle, all attempt to rearrange the computations of neural networks to make better use of the hardware memory layout and optimize hardware utilization.

In very simplistic terms, deep learning optimization can be achieved by optimizing the entire end-to-end computation graph, as well as by restructuring operators (mainly for loops related to matrix multiplications) within the graph [1,2]. Here are some examples of optimization techniques:

Deep learning optimization depends greatly on the specific hardware-software coupling, and specific compilers work best on specific couplings. So it is difficult to know a priori the performance of the many deep learning compilers on the market for each specific use case and testing is necessary. This is exactly whatnebullvmdoes, saving programmers countless hours.

The team behind nebullvm are a group of former MIT, ETH, and EPFL folks who team up together and launchedNebuly. They developed this open-source library along with a lot of other great technologies to make AI more efficient. You can find out more about Nebuly on itswebsite,LinkedIn,TwitterorInstagram.

Many kudos go toDiego Fiori, the library's main contributor. Diego is a curious person and always thirsty for knowledge, which he likes to consume as much as good food and wine. He is a versatile programmer, very jealous of his code, and never lets his code look less than magnificent. In short, Diego is the CTO ofNebuly.

Huge thanks also go to the open-source community that has developed numerous DL compilers that enable to accelerate AI models.

And finally, many thanks to all those who are supporting thenebullvm open-source community, finding bugs and fixing them, and enabling the creation of a state-of-the-art, this super-powerful AI accelerator.

Papers and articles about deep learning compilers.

Documentation of deep learning compilers used by nebullvm.

View post:

This Open-Source Library Accelerates AI Inference by 5-20x in a Few Lines of Code - hackernoon.com