Unleashing the Potential of On-Device Deep Learning with Core ML 3

Deep learning has transformed the world of mobile computing. With powerful frameworks like Apple‘s Core ML, developers can now harness the incredible power of convolutional neural networks and long short-term memory directly on the iPhone and iPad. The latest version of Core ML boasts an impressive array of features including on-device training, over 100 layer types, model quantization, and more.

In this post, we‘ll take an in-depth look at Core ML 3 from the perspective of a machine learning engineer. We‘ll compare its features and performance to other mobile ML frameworks, discuss use cases and examples, and share some thoughts on the future of on-device deep learning. Let‘s dive in!

The Rise of Mobile Machine Learning

The past few years have seen an explosion of interest in running deep learning models on mobile devices. This shift has been driven by a few key factors:

1. More powerful mobile hardware: High-end mobile processors like the A13 Bionic can now provide up to 1 trillion operations per second. Combined with fast GPU and Neural Engine cores, this is sufficient to run fairly complex neural networks in real-time.

2. Improved mobile ML frameworks: Tools like TensorFlow Lite, Core ML, and PyTorch Mobile have made it much easier to deploy models on a variety of mobile devices and edge hardware. These frameworks can optimize models for inference speed and power efficiency.

3. Growing demand for intelligent features: Users now expect app experiences to be highly personalized, responsive, and context-aware. Machine learning enables these intelligent features, from facial recognition to language translation to recommendation engines.

4. Privacy concerns with cloud AI: Running models on-device mitigates some of the privacy risks of sending sensitive user data to the cloud for processing. On-device ML allows for more privacy-preserving personalization.

As of 2019, there were over 900 million iPhones in active use globally, and millions more iPads ^1. All of those devices equipped with neural engines represents a massive opportunity for mobile machine learning.

Apple‘s ML Ecosystem

Let‘s step back and look at where Core ML fits in the landscape of Apple‘s machine learning tools:

At the lowest level are the device hardware and accelerators like the Neural Engine. The Neural Engine is a specialized processor designed for fast matrix math operations. It first appeared in the A11 Bionic chip in 2017 and has continued to grow in size and sophistication. The latest A14 Neural Engine boasts 16 cores and can perform 11 trillion operations per second ^2.

Built on top of the hardware are the low-level frameworks like Metal Performance Shaders and Accelerate with BNNS (Basic Neural Network Subroutines). These provide the essential computational primitives and building blocks for higher-level machine learning functionality.

The next layer up consists of Core ML itself, as well as the Vision and Natural Language frameworks. These frameworks provide higher-level APIs for common tasks like image classification, face detection, and language identification. Under the hood, they leverage Core ML for fast on-device inference.

Finally, at the highest level of abstraction, we have Create ML and Turi Create. These are designed to let you train some models on the Mac without writing any code. Create ML provides a graphical UI for assembling image and text datasets, training a model, and exporting to Core ML format. Turi Create offers a simple Python API for tasks like image classification, object detection, style transfer, and recommendations.

One notable omission from this diagram is Swift for TensorFlow, an ambitious project to integrate TensorFlow directly into the Swift language. While a promising effort, the future of Swift for TensorFlow is somewhat uncertain after Apple‘s recent announcement that they are no longer supporting development ^3.

What‘s New in Core ML 3

Core ML made its debut with iOS 11 in 2017 and has improved rapidly in the years since. The marquee feature of Core ML 3 is on-device training – prior to this release, Core ML models could only be used for inference.

On-device training has a few key benefits:

• Privacy: User data never has to leave the device
• Personalization: Models can be tuned to a specific user‘s preferences and behavior
• Offline updates: Models can be updated even without an internet connection

Core ML 3 utilizes transfer learning to achieve on-device training without the steep computational cost of training a model from scratch. The workflow looks like this:

1. Start with a pre-trained model (e.g. MobileNet or BERT)
2. Freeze most of the model weights
3. Train a new classifier head using on-device data
4. Personalization happens only in the final layer

This is much more efficient than full backpropagation through the whole model. Current iPhones and iPads don‘t have sufficient memory or compute to efficiently train large models like GPT-2 from scratch.

In addition to on-device training, Core ML 3 introduced model quantization which can reduce model size by up to 75% ^4. Quantization works by reducing the numerical precision of the model weights, from 32-bit floats to 8-bit integers. This makes the models much more compact with only a small degradation in accuracy.

Core ML 3 also expanded coverage of state-of-the-art neural network architectures like residual networks (ResNets), recurrent neural networks (LSTMs, GRUs), and generative adversarial networks (GANs). Over 100 new layer types were added including dilated convolutions, 3D convolutions, and pooling layers ^5.

Performance Comparison

So how does Core ML stack up against other on-device ML solutions? Let‘s look at some benchmark results.

In 2019, Fritz AI compared the inference speed of Core ML vs TensorFlow Lite across a range of iPhone hardware ^6:

Model	iPhone XR (Core ML)	iPhone XR (TF Lite)
MobileNet V2	4 ms	13 ms
DeepLabV3	36 ms	58 ms
Pose Estimation	15 ms	42 ms
Portrait Segmentation	31 ms	85 ms

On average, Core ML was 2-3x faster than TensorFlow Lite on the same hardware. This is likely due to Core ML‘s tight integration with the Neural Engine and GPU. TensorFlow Lite, by contrast, runs primarily on the CPU.

Another benchmark by Macrumors found that Core ML was up to 30% faster than TensorFlow Lite on an iPhone X for an image classification model ^7.

Core ML‘s on-device training capability is harder to benchmark, since TensorFlow Lite and other frameworks don‘t yet offer equivalent functionality. A 2019 Apple case study reported that personalizing a sentiment classification model with on-device training data improved accuracy from 78% to 96% ^8, but more real-world examples are needed to assess the benefits of this approach.

Ecosystem Comparison

In addition to performance, it‘s important to consider the broader ecosystem around each mobile ML framework. Factors like available models, developer tools, documentation, and community are all key.

Available Models

Core ML benefits from a large library of pre-trained models converted from popular Python frameworks:

• Apple provides around 50 pre-trained models for tasks like image classification, object detection, pose estimation, language translation, etc.
• The community-maintained coremltools GitHub repo has almost 100 additional models
• Fritz AI hosts a variety of Core ML models for image processing and other fields

By comparison, TensorFlow Lite has a smaller collection of hosted models, and TensorFlow Lite models seem to be less commonly shared by researchers and developers.

Conversion Tools

Core ML has official conversion tools for most major frameworks, including TensorFlow/Keras, PyTorch, Caffe, scikit-learn, XGBoost, LibSVM, and MATLAB. Using the coremltools Python package, developers can:

1. Load a trained model from one of these frameworks
2. Check compatibility and correctness
3. Convert to .mlmodel format
4. Run the model in an iOS simulator for testing
5. Integrate the model in an Xcode project

Google has also created the open-source TensorFlow Lite Converter which can convert TensorFlow models to TF Lite format, but it currently only supports a limited set of TensorFlow ops and layer types. PyTorch Mobile is still a young project and does not yet have a model converter tool.

On-Device Training Support

As mentioned above, Core ML 3 is currently unique in offering on-device training APIs. Neither TensorFlow Lite or PyTorch Mobile support updating models on the mobile device itself. Google is actively developing a framework called Federated Learning to fill this gap, but it‘s not yet ready for production use.

Quantization

All three frameworks now support post-training quantization from 32-bit float to 8-bit integer weights. Core ML and PyTorch Mobile can perform this quantization on-device, while TensorFlow Lite quantizes models as part of the initial conversion process before deployment. Core ML also supports 16-bit "reduced precision" quantization on newer devices with the A12 Bionic or later.

Framework	Post-Training Quantization	On-Device Quantization
Core ML	Yes	Yes (8 and 16-bit)
TF Lite	Yes	No
PyTorch Mobile	Yes	Yes (8-bit)

Developer Tools

This is one area where Core ML really shines. In addition to the coremltools Python package, developers can:

• Use Create ML to build simple models with a drag-and-drop UI and no code
• Use Turi Create‘s simple Python API for more advanced models
• Visualize .mlmodel files in Xcode for debugging and optimization insights
• Profile on-device performance with Instruments

TensorFlow Lite has some useful developer tools, but nothing quite as polished or intuitive as Apple‘s offerings. PyTorch Mobile is still a relative newcomer and its tooling story is even more sparse.

When to Use Core ML

Given all of these points of comparison, when does it make sense to use Core ML vs another mobile framework?

Core ML is likely the best choice when:

• You are building exclusively for iOS and macOS
• You want to integrate a wide variety of state-of-the-art models into your app
• You need the absolute fastest on-device performance
• You want to personalize models via on-device training
• You are most comfortable with Swift and Apple‘s developer tools

TensorFlow Lite may be preferable if:

• You are building for iOS and Android
• You are already familiar with the TensorFlow ecosystem
• You don‘t need on-device training (yet)

And PyTorch Mobile could be the way to go if:

• You have PyTorch models you want to deploy on mobile
• You want a framework backed by Facebook‘s AI research

Future Directions

Apple has shown a strong commitment to Core ML and on-device machine learning. At WWDC 2020, Apple announced a few key updates coming in Core ML 4 ^9:

• An API for GPU-accelerated inference of neural networks
• Support for model encryption and authentication
• A new compression technique called sparse compression (up to 5-10x size reduction)
• Per-layer profiling of models in Xcode

Looking further ahead, I suspect Apple may expand Create ML to support custom model architectures in addition to the current built-in templates. I could also imagine Core ML eventually supporting distributed on-device training across multiple devices for greater scale and efficiency.

It will be interesting to see how Core ML evolves in relation to Apple‘s other AI/ML projects like ​the Overton framework for weakly supervised learning ^10​. I wouldn‘t be surprised to see technologies incubated in research find their way into future iterations of Core ML.

One thing is clear: on-device machine learning is here to stay, and Apple is well-positioned to lead the charge with Core ML. As mobile ML moves from niche to mainstream, Core ML will be an increasingly indispensable tool for iOS developers.