Building Intelligent iOS Apps with Apple‘s CoreML Framework
Machine learning (ML) has emerged as one of the most transformative technologies of the past decade. From voice assistants to facial recognition to product recommendations, ML-powered features have quickly become ubiquitous in modern software. As users increasingly expect more intelligent, personalized app experiences, the ability to effectively incorporate ML has become a critical skill for mobile developers.
Fortunately, Apple has made it easier than ever to integrate ML into iOS apps with its powerful CoreML framework. By streamlining and accelerating the process of deploying ML models on-device, CoreML has opened up exciting new possibilities for iOS developers. In this article, we‘ll take a deep dive into CoreML, exploring its key features, benefits, and limitations. We‘ll walk through a practical, step-by-step example of using CoreML to build an app capable of detecting spam messages. Finally, we‘ll highlight some compelling real-world applications of CoreML and provide resources to help you get started in the exciting field of mobile ML development.
What is CoreML?
First introduced at Apple‘s Worldwide Developer Conference in 2017, CoreML is a framework that allows iOS developers to incorporate pre-trained machine learning models into their apps. CoreML handles all the complexity behind the scenes, automatically optimizing models for performance, minimizing memory footprint and power consumption. By making it simple to leverage powerful ML models in apps, CoreML enables developers to build highly intelligent features without deep expertise in ML.
At a high level, the CoreML workflow looks like this:
- Developer trains an ML model using popular tools/frameworks like TensorFlow, Keras, scikit-learn, XGBoost, etc.
- Developer converts their trained model to the CoreML format (.mlmodel file) using Apple‘s coremltools Python package.
- Developer drags the .mlmodel file into their Xcode project.
- Developer writes Swift/Objective-C code to invoke their model and utilize its predictions in their app.
- When the app runs on a user‘s device, CoreML optimizes and executes the model leveraging all available CPU and GPU resources.
The key to CoreML‘s performance is its ability to leverage the CPU, GPU, and Neural Engine in modern Apple systems-on-a-chip (SoCs). Models are optimized for each SoC‘s unique architecture and capabilities, enabling CoreML to choose the optimal compute path for every task. The result is fast, efficient on-device performance with minimal impact on battery life or system responsiveness.
The Benefits of On-Device ML
One of the most compelling aspects of CoreML is that it allows models to run entirely on-device without the need for a network connection. This enables several key benefits over the traditional approach of performing ML inference in the cloud:
- Improved Performance: With no round trip to a server, on-device ML delivers much faster, near-real-time predictions, enabling more responsive user experiences.
- Offline Functionality: Apps with on-device ML remain fully functional even without an internet connection.
- Enhanced Privacy: Since sensitive user data never leaves the device, on-device ML offers much stronger privacy guarantees and reduces the risk of data breaches.
- Reduced Costs: Processing data on-device eliminates the networking/cloud infrastructure costs associated with cloud-based ML.
As Apple‘s SVP of Software Engineering Craig Federighi explained in the CoreML launch keynote: "We‘re able to bring neural networks right onto the device so you can do incredible things like photo and video analysis, text analysis, and more without sacrificing your privacy or sending your data off to the cloud."
CoreML Use Cases
CoreML supports a wide and growing range of supervised machine learning tasks, including:
- Image classification, object detection, and style transfer
- Natural language processing tasks like text classification, language identification, and sentiment analysis
- Speech recognition and audio analysis
- Time series prediction and anomaly detection
- Recommender systems and ranking tasks
- General purpose regression, classification, and clustering
Some examples of compelling experiences powered by CoreML:
- The PetSmart app uses CoreML to allow users to snap a photo and automatically identify their pet‘s breed.
- The Homecourt app leverages CoreML to analyze basketball shots and provide real-time feedback to help players improve their game.
- The Yelp app uses CoreML to surface personalized restaurant recommendations based on users‘ previous dining experiences and preferences.
CoreML Example: Building a Spam Message Classifier
To illustrate the CoreML development process end-to-end, let‘s walk through a quick example of using it to build an app capable of detecting spam text messages.
Step 1: Train a Model
First, we‘ll train a simple logistic regression model to classify messages as spam/not spam using the Python scikit-learn library and the classic SMS Spam Collection dataset:
import pandas as pd
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.linear_model import LogisticRegression
# Load spam message data
url = ‘https://archive.ics.uci.edu/ml/machine-learning-databases/00228/smsspamcollection.zip‘
data = pd.read_csv(url, compression=‘zip‘, encoding=‘latin-1‘)
data.columns = [‘label‘, ‘message‘]
# Extract features and labels
X, y = data[‘message‘], data[‘label‘]
vectorizer = TfidfVectorizer()
X = vectorizer.fit_transform(X)
# Train classifier
classifier = LogisticRegression()
classifier.fit(X, y)Step 2: Convert the Model
Next, we‘ll convert our trained model to the CoreML .mlmodel format using coremltools:
import coremltools as ct
# Convert model
spam_classifier = ct.converters.sklearn.convert(
classifier,
‘message‘,
‘spam_probability‘)
# Set metadata
spam_classifier.short_description = ‘Classify whether an SMS message is spam‘
spam_classifier.input_description[‘message‘] = ‘Message to classify‘
spam_classifier.output_description[‘spam_probability‘] = ‘Probability message is spam‘
# Save model
spam_classifier.save(‘SpamClassifier.mlmodel‘)Step 3: Xcode Setup
Now we‘re ready to bring our model into an iOS app. After creating a new Xcode project, we simply drag the SpamClassifier.mlmodel file into our project navigator. Xcode automatically generates a Swift interface allowing us to interact with the model.
Step 4: User Interface
Using Xcode‘s Interface Builder, we lay out a simple UI with a text field for message entry, a button to trigger prediction, and a label to display the result:
Step 5: Making Predictions
Finally, we write a few lines of Swift code to pass the user‘s input to our model and display the resulting spam probability:
import CoreML
class ViewController: UIViewController {
@IBOutlet weak var textField: UITextField!
@IBOutlet weak var predictionLabel: UILabel!
let spamClassifier = SpamClassifier()
@IBAction func predictButtonTapped(_ sender: Any) {
if let message = textField.text {
guard let output = try? spamClassifier.prediction(message: message) else {
predictionLabel.text = "Error predicting spam"
return
}
let spamProbability = output.spam_probability
if spamProbability > 0.5 {
predictionLabel.text = "Message is likely SPAM (\(String(format:"%.0f", spamProbability * 100))%)"
} else {
predictionLabel.text = "Message is likely NOT SPAM (\(String(format:"%.0f", 100 - spamProbability * 100))%)"
}
}
}
}And that‘s it! We‘ve built a simple but useful on-device spam classification app with just a few lines of code. While this is a basic example, it illustrates just how quickly you can get up and running with CoreML.
The Evolution of CoreML
Since its 2017 debut, Apple has continued to expand CoreML‘s capabilities with each major release:
- CoreML 2 (2018) – Batch prediction, model quantization, flexible image sizes, custom model layers
- CoreML 3 (2019) – On-device personalization via transfer learning, support for unsupervised learning and generative models, model encryption
- CoreML 4 (2020) – Action classification, vision feature print, trajectory detection, contour detection
- CoreML 5 (2021) – Spatially adaptive learning rates, support for object detection and segmentation tasks
Perhaps the most exciting of these enhancements is on-device personalization via transfer learning introduced in CoreML 3. As Apple explains:
"CoreML 3 supports transfer learning for deep neural networks. With transfer learning, you can improve a model‘s accuracy for a given task by training on your own data, or create a model for an entirely new task (such as gesture classification). Your apps can personalize models on-device using transfer learning without compromising user privacy."
This approach starts with a base model trained to perform a generic task, then specializes that model for a specific user or context by tuning it with additional on-device training data. The updated model is kept on the user‘s device, ensuring privacy. Personalized models deliver significant accuracy improvements for many use cases.
Alternatives to CoreML
Of course, CoreML isn‘t the only option for machine learning on mobile. Two popular open source alternatives are:
- TensorFlow Lite – Google‘s solution for deploying TensorFlow models on mobile and edge devices. Offers a similar workflow to CoreML (train model, convert, integrate in app). Key advantages include flexibility to run on iOS, Android, and embedded systems, as well as a large community and ecosystem.
- PyTorch Mobile – Enables embedding PyTorch models in iOS and Android apps. Relative newcomer but rapidly gaining adoption. Unique ability to re-train models on-device is a key differentiator.
Other cloud-based options like Google‘s ML Kit and Amazon‘s SageMaker Neo provide APIs to run ML models in the cloud and return predictions to mobile apps. These solutions offer flexibility and scalability at the cost of added latency, connectivity requirements, and potential privacy concerns of sending user data off-device.
Ultimately, the right approach depends on your use case, target platforms, and machine learning framework preferences. But for iOS developers seeking to build compelling on-device ML experiences with minimal friction, it‘s hard to beat CoreML‘s ease of use and tight integration with Apple‘s ecosystem.
Conclusion
Machine learning offers immense potential to make apps smarter, more engaging, and more valuable to users. With CoreML, Apple has democratized ML for iOS developers, making it simpler than ever to build intelligent app experiences. By providing a user-friendly framework for embedding state-of-the-art ML models, CoreML empowers developers to create apps that can see, hear, and understand the world around them like never before.
As you explore opportunities to apply ML in your own apps, remember that successful projects start with a user-centric mindset. Focus first on defining a clear problem to be solved and key experience to be delivered. With a well-defined objective, the CoreML development process becomes a straightforward matter of acquiring relevant training data, experimenting with model architectures, evaluating performance, and iterating.
It‘s an exciting time to be an iOS developer. The possibilities for leveraging ML to build magical experiences are limitless and the barrier to entry has never been lower. Apple‘s continued investment in CoreML and on-device AI capabilities ensures that mobile ML will only accelerate from here. There‘s never been a better time to level up your skillset, experiment with these cutting-edge technologies, and push the boundaries of what‘s possible. Go out and create the next generation of advanced, intelligent apps with CoreML!
Additional Resources
To learn more about CoreML and mobile machine learning, check out these resources:
