A Deep Dive into the 3 Key Building Blocks of Successful Machine Learning Systems

Machine learning (ML) and artificial intelligence (AI) have become the hottest areas of technology over the past decade. The ability to extract insights from massive datasets and power intelligent systems has transformed almost every industry – from healthcare to finance to transportation and beyond.

However, building successful ML/AI applications requires much more than just training models. It requires carefully designing and orchestrating multiple components that enable the end-to-end data and inference flow.

In this article, we‘ll take a deep dive into the three fundamental building blocks that form the backbone of most modern machine learning systems:

1. Data Capture and Input
2. Data Processing, Storage and Compute
3. Output and Interaction

By the end, you‘ll have a solid framework for reasoning about ML systems as a whole and be better equipped to design and implement them as a data scientist. Let‘s jump in!

Building Block 1: Data Capture and Input

Figure 1: Data can be captured from many sources to feed ML models [1].

The first step in any ML project is sourcing the data that will be used to train and evaluate models. After all, the success of a model depends heavily on the quantity and quality of data it has access to. As the saying goes, "garbage in, garbage out."

Data can be captured from many different places – sensors, databases, data providers, web scraping, user interactions, and more (see Figure 1). Depending on the application, data may arrive in real-time streams or in batch loads. It may be structured with well-defined schemas or be totally unstructured.

Considerations for Data Selection

With so many potential data sources and formats, it‘s critical that data scientists put careful thought into what data to use for building ML models rather than just using every available signal. When selecting input data, you should consider:

• Predictive power – Is this data relevant for the task and likely to be predictive of the target?
• Cleanliness – How noisy and inconsistent is this data? What effort is needed to clean it?
• Availability – How complete is the data coverage? Are there gaps or sparseness issues?
• Timeliness – How up-to-date is the data? Is it refreshed frequently enough for your use case?
• Cost – How much does it cost to acquire or generate this data? Is there cheaper proxy data?
• Legal and ethical issues – Do you have rights to use this data? Are there privacy concerns?

Various feature selection techniques can be helpful for narrowing down the input space:

• Correlation analysis to identify and remove redundant features [2]
• Domain knowledge to pick variables that should impact the target
• Embedded methods like L1 regularization that automatically select features during model training [3]

Handling Different Data Types

Figure 2: ML models need to handle diverse data types [4]

Data comes in many flavors that each require their own special handling (see Figure 2):

• Numerical data is the easiest to work with and can be directly fed into most models
• Categorical data needs to be encoded as numbers, either via one-hot or embedding approaches
• Text data requires tokenization, embeddings, and other NLP techniques to extract features
• Image/video data is often processed using convolutional neural networks and deep learning
• Time-series data needs temporal models that account for trends and seasonality

See [5] for an overview of techniques for feature engineering across different data types.

Synthetic Data Generation

In some cases, you may not have enough volume or diversity of data to properly train a model. This is where data augmentation and synthetic data generation can help.

For example, in computer vision it‘s common practice to augment image training data by applying various transformations like scaling, cropping, rotation, and color distortions. This can significantly increase the effective dataset size.

Generative models like GANs and diffusion models have also become popular recently for synthesizing completely new data [6]. By learning the distribution of features, they can generate realistic text, images, and other modalities to supplement real data.

When using synthetic data, it‘s important to quantify how well it matches the distribution and properties of real data. Using poor quality synthetic data risks teaching models the wrong patterns.

Building Block 2: Data Processing, Storage and Compute

Figure 3: Typical edge-cloud architecture for ML systems [7]

Once you‘ve identified and captured relevant data to feed into ML models, the next building block is to process and store that data. This involves deciding what computations to perform on the data and where that processing should happen – on edge devices or in the cloud.

Edge vs Cloud Tradeoffs

There are pros and cons to processing data on the edge vs in the cloud (see typical architecture in Figure 3).

Benefits of edge processing include:

• Lower latency for real-time applications
• Reduced bandwidth costs to transmit data
• Improved privacy and security
• Enables offline functionality

The main downside of edge processing is that edge devices tend to have highly constrained hardware resources. Smartphones, wearables, and Internet-of-Things (IoT) devices have limited memory, storage, compute, and battery life.

This means you have to be very selective about what processing to perform on-device. Typically this includes:

• Filtering and downsampling input data
• Quick transformations and aggregations
• Lightweight models for real-time inference

Everything else gets sent to the cloud, which has virtually unlimited storage and compute power, but comes at the cost of higher latency and requiring network connectivity to work.

Federated Learning

An interesting middle ground between fully centralized (cloud) and fully decentralized (edge) processing is federated learning [8]. Under this paradigm, model training is distributed across a fleet of edge devices. Each device trains a local model on its own private data and only shares model updates, not the raw data, with a central server.

This enables a degree of privacy since data never leaves the local device, while still allowing for building models that learn from data across many users. It‘s an active area of research and has been deployed in production ML systems at Google, Apple, and elsewhere.

Real-time Decision Making

For many ML applications, there are strict latency requirements on how quickly the system needs to make predictions and take actions based on input data. Think of autonomous vehicles that need to make sub-second planning decisions, or fraud detection models that must block suspicious transactions before they complete.

Meeting these tight latency constraints requires careful design of the data flow and architecture [9]. As much processing as possible must be pushed to the edge to minimize round trip time. Streaming infrastructure like Kafka or Kinesis is often used to ingest real-time data. And the whole pipeline must be orchestrated to meet end-to-end deadlines.

This is very different than batch processing jobs that can be done asynchronously with no latency concerns. Real-time ML systems require specialized knowledge of streaming, queuing, load balancing and other distributed systems concepts.

Building Block 3: Output and Interaction

Figure 4: Principles of human-centered interface design for ML [10]

The final building block is the output of the ML system. This is where the rubber meets the road and all the data and models hopefully translate to real-world value. There are countless ways that ML system outputs can manifest:

• Predictions and estimated values
• Recommended actions and interventions
• Alerts and notifications
• Optimized configurations and control signals
• Generated text, images, and other media

But for the outputs to be useful, they need to be surfaced to users or other systems in the right way at the right time with the right context.

Human-Centered Design

A mistake that many technical ML teams make is focusing so much on optimizing model accuracy that they neglect the user experience. You could have the most accurate model in the world, but if users don‘t understand it or don‘t like interacting with it, you‘ve failed.

Techniques like participatory design, user research, and usability testing are crucial for building ML applications that solve real user needs [10]. ML teams need to collaborate closely with UX designers to create interfaces that are intuitive, explanatory, and actionable (see Figure 4).

Display the model‘s confidence and uncertainty in its predictions. Provide options to correct or override the model when needed. Continuously collect feedback on how well the model is working. The best ML systems keep humans in the loop.

Model Interpretability

One of the biggest challenges with complex ML models like deep neural networks is that they are "black boxes". Their internals are opaque and it‘s hard to explain how they arrive at specific outputs.

This lack of interpretability can erode user trust, make debugging difficult, and increase liability risk. Especially in high stakes domains like medicine and criminal justice, there are calls to make ML more transparent and accountable.

Research into techniques for explaining ML models is very active [11]. Current approaches include:

• Feature importance – quantify contribution of each input to the prediction
• Saliency maps – highlight regions of input that influence the output
• Case-based explanations – find similar examples the model has seen before
• Counterfactual explanations – show minimal changes needed to flip the prediction

While not perfect, these techniques can help users gain insight into model behavior and audit predictions. We are likely to see more regulation around AI explainability and transparency in the near future.

Monitoring and Maintenance

Figure 5: Models degrade over time as data distributions shift [12]

Deploying an ML model is not a one-and-done affair. Models degrade over time as the world changes and the data they were trained on becomes stale (see Figure 5). This is known as concept or data drift [12].

To keep models fresh, you need to continuously monitor their predictions and performance on new data. Track metrics like accuracy, precision/recall, and confusion matrices. When performance dips below a threshold, retrain models on more recent data.

It‘s also important to monitor the computational performance and cost of ML pipelines – memory usage, latency, throughput, server load, etc. Autoscaling infrastructure can help adjust resources based on demand.

Proper monitoring allows you to proactively detect and fix issues before they cause major problems. But it does require dedicated engineering effort to instrument pipelines and expose health metrics.

Conclusion

We covered a lot of ground in this deep dive on the three key building blocks of machine learning systems:

1. Data Capture and Input – source and select quality data to power models
2. Data Processing, Storage and Compute – design data pipelines for both edge and cloud
3. Output and Interaction – surface model predictions to drive value in a human-centered way

As an ML practitioner, you need to understand how all three blocks fit together to build successful real-world applications. Each block comes with its own unique challenges and considerations that are active areas of research and development.

Here are some key takeaways:

• Focus on capturing relevant high-quality data, not just large volumes of data
• Be smart about what to process on edge devices vs in the cloud
• Design user-centric interfaces that explain the model and keep humans in the loop
• Plan for continuous monitoring and maintenance to combat model drift

The field of ML is evolving incredibly quickly, but these foundational principles will serve you well no matter what the latest techniques and tools are. I hope this article gave you a solid conceptual framework to design and implement ML systems in your own projects.

Please let me know if you have any other questions! Happy building!

References

[1] Data sources for machine learning
[2] Feature selection techniques for machine learning
[3] Embedded methods for feature selection
[4] Structured vs unstructured data: The 5 key differences
[5] Feature engineering in machine learning
[6] Generative AI models — A beginner‘s guide
[7] Machine learning at the edge
[8] What is federated learning?
[9] How to design and architect a real-time streaming application
[10] Human-centered design for machine learning
[11] Machine learning explainability – Interpretable ML
[12] Machine learning: Adapting to data drift