This is the second post in a series about tiny machine learning (TinyML) at the deep IoT edge. Read our earlier introduction to TinyMl as-a-Service, to learn how it ranks in respect to traditional cloud-based machine learning or the embedded systems domain.
TinyML is an emerging concept (and community) to run ML inference on Ultra Low-Power (ULP ~1mW) microcontrollers. TinyML as a Service will democratize TinyML, allowing manufacturers to start their AI business with TinyML running on microcontrollers.
In this article, we introduce the challenges behind the applicability of ML concepts within the IoT embedded world. Furthermore, we emphasize how these challenges are not simply due to the constraints added by the limited capabilities of embedded devices but are also evident where the computation capabilities of ML-based IoT deployments are empowered by additional resources confined at the network edge.
To summarize the nature of these challenges, we can say:
Below, we take a closer look at each of these challenges.
Edge computing promises higher performing service provisioning, both from a computational and a connectivity point of view.
Edge nodes support the latency requirements of mission critical communications thanks to their proximity to the end-devices, and enhanced hardware and software capabilities allow execution of increasingly complex and resource-demanding services in the edge nodes. There is growing attention, investments and R&D to make execution of ML tasks at the network edge easier. In fact, there are already several ML-dedicated "edge" hardware examples (e.g. Edge TPU by Google, Jetson Nano by Nvidia, Movidius by Intel) which confirm this.
Therefore, the question we are asking is: what are the issues that the edge computing paradigm has not been able to completely solve yet? And how can these issues undermine the applicability of ML concepts in IoT and edge computing scenarios?
We intend to focus on and analyze five areas in particular: (Note: Some areas we describe below may have solutions through other emerging types of edge computing but are not yet commonly available).
Figure 1
The web and the embedded worlds feature very heterogeneous characteristics. Figure 1 (above) depicts how this high heterogeneity is characterized, by comparing qualitatively and quantitively the capacities of the two paradigms both from a hardware and software perspective. Web services can rely on powerful underlying CPU architectures with high memory and storage capabilities. From a software perspective, web technologies can be designed to choose and benefit from a multitude of sophisticated operating systems (OS) and complex software tools.
On the other hand, embedded systems can rely on the limited capacity of microcontroller units (MCUs) and CPUs that are much less powerful when compared with general-purpose and consumer CPUs. The same applies with memory and storage capabilities, where 500KB of SRAM and a few MBs of FLASH memory can already be considered a high resource. There have been several attempts to bring the flexibility of Linux-based systems in the embedded scenario (e.g. Yocto Project), but nevertheless most of 32bit MCU-based devices owns the capacity for running real-time operating systems and no more complex distribution.
In simple terms, when Linux can run, system deployment is made easier since software portability becomes straightforward. Furthermore, an even higher cross-platform software portability is also made possible thanks to the wide support and usage of lightweight virtualization technologies such as containers. With almost no effort, developers can basically ship the same software functionalities between entities operating under Linux distributions, as happens in the case of cloud and edge.
The impossibility of running Linux and container-based virtualization in MCUs represents one of the most limiting issue and bigger challenge for current deployments. In fact, it appears clear how in typical "cloud-edge-embedded devices" scenarios, cloud and edge services are developed and deployed with hardware and software technologies, which are fundamentally different and easier to be managed if compared to embedded technologies.
TinyML as-a-Service tries to tackle this issue by taking advantage of alternative (and lightweight) software solutions.
Figure 2
In the previous section, we considered on a high-level how the technological differences between web and embedded domains can implicitly and significantly affect the execution of ML tasks on IoT devices. Here, we analyze how a big technological gap exists also in the availability of ML-dedicated hardware and software web, edge, and embedded entities.
From a hardware perspective, during most of computing history there have been only a few types of processor, mostly available for general use. Recently, the relentless growth of artificial intelligence (AI) has led to the optimization of ML tasks for existing chip designs such as graphics processing units (GPUs), as well as the design of new dedicated hardware forms such as application specific integrated circuits (ASICs), which embed chips designed exclusively for the execution of specific ML operations. The common thread that connects all these new devices is their usage at the edge. In fact, these credit-card sized devices are designed with the idea of operating at the network edge.
At the beginning of this article we mentioned a few examples of this new family of devices (Edge TPU, Jetson Nano, Movidius). We foresee that in the near future even more big and small chip and hardware manufacturers will increasingly invest resources into the design and production of ML-dedicated hardware. However, it appears clear how, at least so far, there has not been the same effort in the embedded world.
Such a lack of hardware availability undermines somehow a homogeneous and seamless ML "cloud-to-embedded" deployments. In many scenarios, the software can help compensate for hardware deficiencies. However, the same boundaries that we find in the hardware sphere apply for the development of software tools. Today, in the web domain, there are hundreds of ML-oriented application software. Such availability is registering a constant growth thanks also to the possibility given by the different open source initiatives that allow passionate developers all over the world to merge efforts. The result is more effective, refined, and niche applications. However, the portability of these applications into embedded devices is not so straightforward. The usage of high-level programming languages (e.g., Python), as well as the large sizes of the software runtime (intended as both runtime system and runtime program lifecycle phase) are just some of the reasons why the software portability is painful if not impossible.
The main rationale behind the TinyML as-a-Service approach is precisely the one to break the existing wall between cloud/edge and embedded entities. However, to expect exactly the same ML experience in the embedded domain as we have in the web and enterprise world would be unrealistic. It is still an irrefutable fact that size matters. The execution of ML inference is the only operation that we reasonably foresee to be executed in an IoT device. We are happy to leave all the other cumbersome ML tasks, such as data processing and training, to the more equipped and resourceful side of the scenario depicted in Figure 2.
In the next article, we will go through the different features which characterize TinyML as-a-Service and share the technological approach underlying the TinyML as-a-Service concept.
In the meantime, if you have not read it yet, we recommend reading our earlier introduction to TinyMl as-a-Service.
The IoT world needs a complete ML experience. TinyML as-a-service can be one possible solution for making this enhanced experience possible, as well as expanding potential technology opportunities. Stay tuned!
Continued here:
TinyML as a Service and machine learning at the edge - Ericsson