Bringing Machine Learning to the Edge with Microcontrollers

Machine learning (ML) has revolutionized many domains in recent years, from computer vision to natural language processing. However, much of this progress has relied on powerful processors, ample memory, and the ability to run complex software frameworks. So what about the billions of small, resource-constrained computing devices all around us? Can ML also run on the tiny microcontrollers powering everything from appliances to wearables to industrial sensors?

The answer is yes – ML is coming to microcontrollers and enabling smart capabilities on devices at the edge. However, significant challenges must be overcome to run ML workloads on devices with kilobytes of RAM rather than gigabytes. In this article, we‘ll explore what microcontrollers are, the unique constraints they impose, and the emerging techniques to run ML on these ubiquitous tiny computers.

Meet the Microcontroller

Microcontrollers (MCUs) are small, low-cost, low-power computers packaged as integrated circuits. They combine a processor, memory, and programmable I/O peripherals into a single chip. Unlike the general-purpose processors in PCs and phones, MCUs are designed for embedded applications to control equipment and devices.

MCUs are ubiquitous, with tens of billions sold each year. If a device has any smarts or automated functions, odds are it contains an MCU. Some examples include:

Appliances: Washing machines, microwaves, thermostats
Vehicles: Engine control, antilock brakes, power steering
Medical devices: Glucose monitors, pulse oximeters
Industrial equipment: Factory automation, process control
Consumer electronics: TV remotes, toys, wearables

Microcontrollers are used across industries. (Image source: Microcontroller Tips)

The most common MCUs are based on ARM Cortex-M cores, but other architectures like AVR, PIC, and RISC-V are also used. Popular MCU development platforms include Arduino, Particle, and the Raspberry Pi Pico.

According to industry reports, the global microcontroller market is projected to grow from $16.3B in 2021 to $21.7B by 2026, driven by the proliferation of smart devices, automation, and IoT. And with the rise of ML capabilities on MCUs, the market for AI-enabled MCUs is forecast to reach $27.7B by 2026, up from just $6.5B in 2021 (marketsandmarkets.com).

Microcontroller Constraints

While MCUs excel at real-time control and low power consumption, they have limited resources compared to the processors in PCs, servers and mobile devices:

Clock speeds of 16-300 MHz vs GHz
RAM in the kilobytes vs gigabytes range
Flash storage in megabytes vs gigabytes
No operating system, minimal software stack

To put this in perspective, a typical smartphone SoC like the Qualcomm Snapdragon 888 packs 8 CPU cores running up to 2.84 GHz, a GPU, a neural processing unit, 8-12 GB RAM, and 128-512 GB storage – orders of magnitude more than even high-end MCUs. The following table compares some popular MCUs:

MCU	Clock (MHz)	Flash (KB)	RAM (KB)
Atmel ATmega328P	16	32	2
STM32F405RG Cortex-M4	168	1024	192
Espressif ESP32-S3	240	384	512
Raspberry Pi RP2040	133	2048	264

Additionally, MCUs do not support common ML software libraries and frameworks. Tools like PyTorch, TensorFlow, NumPy, and pandas that are fundamental to ML development cannot run on MCUs.

These limitations make deploying ML on MCUs challenging. The training algorithms, model architectures, software, and development workflows that work for cloud and mobile ML must be heavily adapted for MCUs.

TinyML: ML for Microcontrollers

Despite the challenges, there is growing interest in running ML workloads such as computer vision, audio processing, and sensor analysis on MCUs. Executing ML at the edge on MCUs reduces latency, saves bandwidth, improves privacy, and opens up new offline use cases compared to the traditional approach of sending data to the cloud for ML processing.

"TinyML is a fast-growing field of machine learning technologies and applications including hardware (dedicated accelerators), algorithms, and software capable of performing on-device sensor (vision, audio, IMU, biomedical, etc.) data analytics at extremely low power, typically in the mW range and below, and hence enabling a variety of always-on use-cases and targeting battery operated devices," explains Evgeni Gousev, Senior Director at Qualcomm Technologies, Inc.

Two main approaches have emerged for running ML on MCUs: TensorFlow Lite Micro and the TinyML workflow.

TensorFlow Lite for Microcontrollers

Google has created a version of its TensorFlow Lite framework targeting MCUs. Models are trained in TensorFlow and converted to an optimized format to run on MCUs with a minimal C++ runtime. TF Lite Micro currently supports a limited set of operations and requires developer optimization. The framework has been ported to Arm Cortex-M, ESP32, and Arduino platforms.

While a promising effort to bring TensorFlow models to tiny devices, the workflow still requires ML and embedded expertise to navigate. And the runtime adds tens of kilobytes of code space.

TinyML Workflow

The TinyML approach, popularized by Harvard‘s TinyMLx program and companies like Edge Impulse, involves training compact models (often in TensorFlow or scikit-learn), converting them to optimized C code, and compiling them to run on MCUs with no additional runtime. This allows developers to access a wider variety of ML models and target ultra-constrained MCUs. We‘ll focus on this approach.

The TinyML workflow looks like this:

Gather and preprocess training data, often from sensors
Design and train a compact ML model (fully-connected, CNN, RNN, etc.)
Convert the model to C code using a tool like emlearn
Optimize the model via quantization, pruning, code tweaks
Test the model, ideally on the target MCU
Integrate the model code into an MCU firmware project
Deploy it to your device

Let‘s look at a few of these steps in more detail.

Compact Model Design

Fitting ML models within MCU resource constraints requires carefully limiting the model size. Some techniques:

Prefer simple, small layer types (fully-connected vs. conv, GRU vs. LSTM)
Use fewer and smaller layers (100s-1000s of weights, not millions)
Limit the size of the input data (downsample images, audio, etc.)
Quantize weights to 8 bits or less
Prune less important weights

Researchers have proposed TinyML model architectures like MicroNets and MCUNet that push the boundaries of accuracy vs. size. For example, MCUNet achieves over 70% accuracy on the ImageNet benchmark with a model size under 1 MB.

Benchmarks like MLPerf Tiny and TinyMLPerf have also emerged to systematically compare ML performance on MCUs. Initial benchmark results underscore the rapid progress in TinyML:

MLPerf Tiny v0.5 results. (Image source: MLCommons)

Code Generation

Converting a trained ML model to efficient C code is a key step for TinyML. Open source libraries like emlearn and EdgeML can convert scikit-learn and TensorFlow models to C. Commercial tools like Edge Impulse also offer this capability.

The generated code is designed to have minimal dependencies, with basic C math operations rather than library calls. This allows it to be easily integrated into MCU firmware projects and compiled for the target device.

For example, here‘s a snippet of decision tree code generated by emlearn:

if (features[12] <= 0.5) {
    if (features[14] <= 0.5) {
        if (features[19] <= 0.5) {
            scores[0] += 2.0;
        }
        else {
            scores[1] += 2.0;
        }
    }
    else {
        if (features[29] <= 0.5) {
            scores[1] += 10.0;
        }
        else {
            scores[1] += 1.0;
        }
    }
}

Optimization and Testing

Debugging embedded ML code can be tricky. It‘s important to validate your model‘s accuracy running on the MCU itself, not just in a simulator. Examining the model‘s weight ranges and outputs can help catch quantization and overflow bugs.

Measuring the model‘s execution time and peak memory usage is critical to ensure it performs adequately within the MCU‘s constraints. Commercial TinyML tools often include device-specific profiling and optimization guidance. Offloading some preprocessing to a separate digital signal processor (DSP) can also help.

Techniques like post-training quantization (PTQ) and quantization-aware training (QAT) can substantially reduce model size with minimal accuracy loss. And pruning weights below a certain threshold can provide further savings. These optimizations should be considered essential for many TinyML deployments.

Example TinyML Applications

So what can you actually do with ML on tiny devices today? Some interesting examples include:

Audio wake words – TinyML is powering the always-on keyword spotting in devices like smart speakers, remotely activating the full voice assistant when a certain phrase is heard. Researchers have shown how a GRU RNN model can detect "OK Google" with 95% accuracy on an Arm Cortex-M7 MCU.
Anomaly detection – Running ML on sensor data locally can detect problems earlier and avoid sending useless data to the cloud. Bosch used TinyML to catch anomalies in industrial equipment using vibration signals with CNNs on an Arm Cortex-M4 MCU.
Image classification – Computer vision is challenging on MCUs but initial results are promising. The MobileNetV2 CNN architecture was used to classify images on an STM32H7 MCU with 87% accuracy on CIFAR-10. And plant disease detection with over 95% accuracy on a Cortex-M7 was shown using a modified ResNet model.
Gesture recognition – ML is enabling new user interfaces for tiny devices. Researchers used accelerometer data and a novel binarized CNN to classify gestures on an Arduino Nano 33 BLE Sense with up to 98% accuracy.
Predictive maintenance – Analyzing the current signature of motors can detect faults before failure. An SVM model was shown to classify faults on an STM32 MCU by learning on frequency data with 95%+ accuracy while adding only 6 KB of code.

From wildlife monitoring to precision agriculture to smart cities, TinyML is enabling new intelligent sensing applications by pushing ML to the edge.

Remaining Challenges and Future Potential

While TinyML is a promising approach to running ML on MCUs, challenges remain:

Preprocessing data such as images enough to fit models in memory
Co-optimizing models, data pipelines, and power management
Updating ML models efficiently and securely post-deployment
Developing user-friendly tools to ease the embedded ML workflow

But with the rapid growth in tooling, frameworks, educational resources, and commercial solutions, deploying ML on tiny devices is becoming more accessible for developers. "TinyML is at the intersection of embedded systems and machine learning, and thus (for now) requires expertise in both domains," says Pete Warden, TinyML pioneer and lead of the TensorFlow Lite Micro effort at Google. "That‘s why I‘m so excited to see efforts like Harvard‘s TinyML course, which can help embedded engineers learn ML and bring the fields closer together."

And as the MCUs themselves evolve – for example, incorporating neural processing units and analog in-memory compute – the possibilities will only grow. Emerging ultra-low-power MCUs can run ML on only a few mW of power, unlocking battery-less designs that harvest their own energy. Combined with innovations like spiking neural networks, we‘re approaching the sci-fi vision of AI that can run anywhere for years without charging.

In the future, we can expect to see ML deployed on everything from billions of intelligent sensors to nanoscale devices inside the body. TinyML will help make this vision a reality.

"TinyML will be the future of AI in the coming decade," asserts Vijay Janapa Reddi, associate professor at Harvard and co-chair of the tinyML Foundation. "It‘s an exciting time to explore what is possible when machine learning models and algorithms are so tiny that they can run anywhere on next to no power."

So while squeezing ML onto MCUs may never be as easy as developing in Python, it‘s now achievable – and the upside is enormous. Those who master TinyML now can ride the next wave of AI out to the edge and beyond.

Additional Resources

Want to dive deeper into TinyML? Check out these resources:

TinyML Foundation – A neutral organization fostering the TinyML community
TinyML – New O‘Reilly book on the what, why and how of TinyML
TinyML Courses – Harvard‘s free online curriculum for learning TinyML
Awesome TinyML – Curated list of TinyML resources
TinyML Talks – Interviews with TinyML experts and practitioners
The Future of TinyML – Review paper on TinyML applications, techniques, and road ahead

Bringing Machine Learning to the Edge with Microcontrollers

Meet the Microcontroller

Microcontroller Constraints