# Designing Robust Data Ingestion Pipelines on AWS for AI/ML Workloads - Canonical: https://33rdsquare.com/data-ingestion-featuring-aws/ - Published: 2024-09-03 - Author: Jordan Brown - Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/) --- Data is growing at an exponential rate driven by the proliferation of digital devices, ubiquity of internet connectivity, and explosion of AI/ML applications. According to IDC, the global datasphere is predicted to grow to 175 zettabytes by 2025, a compounded annual growth rate of 61% from 2018 ([Source](https://www.idc.com/getdoc.jsp?containerId=prUS47560321)). Organizations that can harness this data to drive insights, innovation, and competitive advantage through AI/ML will be the winners in the digital economy. However, the journey from raw data to AI/ML-ready data involves many steps, the first of which is efficient and scalable data ingestion. Data ingestion, the process of collecting data from source systems and transporting it to target systems for storage and analysis, is both a critical and challenging piece of the AI/ML lifecycle. It needs to handle the 3 Vs of big data: 1. Volume – Large datasets in the order of terabytes to petabytes 2. Velocity – High throughput of data from streams and real-time sources 3. Variety – Structured, semi-structured, and unstructured data in different formats Moreover, data ingestion pipelines for AI/ML have some unique characteristics and challenges: - Quality and consistency of training data is paramount. ML models learn patterns from historical data and their predictive performance is highly dependent on the quality of data used for training. Data ingestion pipelines need to ensure completeness, accuracy, and consistency of data through schema validation, statistical checks, and anomaly detection. - Training datasets for ML models can get very large, sometimes 100s of terabytes. The ingestion layer needs to have the ability to process and transfer very high volumes of data and scale storage and network capacity to keep up with the ML training cycle. - Unstructured data such as text, images, videos, and speech are increasingly used in modern AI/ML applications. Data ingestion needs to handle these complex datatypes and apply transformations such as resizing images, transcoding videos, and extracting text from documents. - Real-time ML use cases such as fraud detection and predictive maintenance need model inference on streaming data with very low latency. Data ingestion pipelines need to be able to process streaming events and make them available for real-time scoring with pre-trained models. - Data lineage and traceability is very important for AI/ML workflows to ensure reproducibility and debuggability of models. Data ingestion pipelines need to capture and preserve the metadata about the provenance and processing of data. ## AWS – A Comprehensive Platform for AI/ML Data Ingestion Amazon Web Services (AWS) offers a broad and deep set of managed services to cover the entire data ingestion lifecycle for AI/ML projects. Let‘s look at some of the key AWS services and how they can be used to build data ingestion pipelines at scale. ### Storage and Data Lake At the foundation of any data architecture is cloud storage that acts as the centralized repository for all the raw and processed datasets used for AI/ML. On AWS, that service is [Amazon S3](https://aws.amazon.com/s3/) – an object storage service that provides virtually unlimited scalability, 99.999999999% durability, and rich security and compliance capabilities. Some of the key features of S3 that make it ideal for building data lakes for AI/ML: - Ability to store any type of data in its native format. This includes structured data like CSV/JSON as well as unstructured data like images, videos, and text. - Scalable performance for high throughput reads and writes. S3 automatically scales to handle very high request rates, on the order of millions of requests per second. - Server-side encryption and fine-grained access control with AWS Key Management Service (KMS) and AWS Identity and Access Management (IAM). - Intelligent Tiering that automatically moves data between different storage tiers based on access patterns, resulting in cost savings. - Strong consistency and atomic operations for maintaining data integrity in concurrent read-write scenarios. To optimally use S3 for AI/ML data ingestion, consider the following best practices: - Partition and store your data in S3 using a well-defined naming convention and layout that reflects the attributes of the data. This allows for easier discoverability and querying of data. For example, you could partition by date, geography, or data source. - Use columnar formats like Apache Parquet for storing structured data. Columnar formats enable faster queries and reduce storage costs through compression. - Use S3 Inventory to get a scheduled report of all the objects in your data lake. This can help in auditing and maintaining the data lifecycle. - Use S3 Event Notifications to trigger data ingestion workflows when new data is landed in S3. This allows for an event-driven architecture that reduces latency and cost. ### ETL and Data Processing Once the data has been ingested into a data lake, the next step is to process and transform it into a format suitable for AI/ML. This involves cleansing, normalization, joining disparate datasets, feature engineering, and converting to optimized file formats. AWS provides a fully-managed ETL service called [AWS Glue](https://aws.amazon.com/glue/) that makes it easy to prepare and load data for analytics and ML. Key components and features of AWS Glue include: - Crawlers that can scan data in S3, infer the schema, and store it in the Glue Data Catalog. The Data Catalog serves as the central metadata repository for all your datasets. - ETL jobs that can be created either visually using Glue Studio or programmatically using Python or Scala. Glue ETL jobs can perform complex data transformations at scale and load the results back into S3 or other data stores. - Support for a wide variety of data sources and targets including S3, Amazon Redshift, Amazon RDS, JDBC databases, and common file formats like CSV, JSON, Avro, Parquet, and ORC. - Automatic scaling of the underlying compute resources based on the size and complexity of the data processing workload. This provides a serverless experience and optimizes costs. - Built-in transforms for common ETL tasks such as join, pivot, mapping, filtering, and de-duplicating data. Glue also provides ML transforms for data cleaning and normalizing using techniques like FindMatches for deduplication and FindIncrementalMatches for record linkage. More recently, AWS has released new Glue features specifically targeting AI/ML data preparation challenges: - Glue DataBrew – A visual data preparation tool that enables data analysts and data scientists to clean and normalize data without writing code. It provides a point-and-click interface to combine datasets, handle missing values, perform advanced transformations, and validate data quality. - Glue Elastic Views – Allows running ETL jobs on a schedule and incrementally processing only new or modified files in S3 since the last run. This is particularly useful for ML feature store use cases where you need to backfill features from historical data. Some best practices to follow when building ETL pipelines with AWS Glue: - Use Glue Crawlers to automatically infer schema and evolve it as the underlying data changes. This keeps your metadata always up-to-date. - Perform coarse-grained transformations and joins in Spark ETL jobs and fine-grained transformations in DataBrew recipes. This provides a good balance between flexibility and ease of use. - Use Glue Job Bookmarks to process data incrementally and keep track of processed files across multiple ETL runs. - Monitor Glue job metrics such as success rate, duration, and data processed using Amazon CloudWatch. Set up alarms for proactively detecting issues such as job failures and data quality anomalies. ### Streaming Data Ingestion With the rise of IoT and event-driven architectures, more and more data is being generated as a continuous stream of events. Real-time ML use cases such as anomaly detection, predictive maintenance, and recommendation engines require the ability to ingest, process, and analyze streaming data with low latency. [Amazon Kinesis](https://aws.amazon.com/kinesis/) is a fully-managed service for real-time collection, processing, and analysis of streaming data at massive scale. Amazon Kinesis is composed of four distinct services that address different aspects of streaming data: - Kinesis Data Streams – Durable, elastic, real-time capture of streaming events. Streams can scale to handle millions of events per second and retain data for up to 365 days for replay and batch analysis. - Kinesis Data Firehose – Managed service to load streaming data into S3, Redshift, Elasticsearch, and 3rd party destinations. Firehose can batch, transform, and convert the format of streaming data before delivering it. - Kinesis Data Analytics – Enables running SQL queries or custom code on streaming data in real-time. Can ingest data from Kinesis Data Streams or Kinesis Data Firehose and output results to destinations such as S3 and Redshift. - Kinesis Video Streams – Real-time capture, process, and storage of video streams for analytics and ML. Provides integration with Amazon Rekognition Video for extracting objects, faces, and activities in real-time. Kinesis services integrate well with other AWS services for building end-to-end streaming AI/ML data pipelines. For example: - Use AWS IoT Core to ingest telemetry events from millions of devices and write them to a Kinesis Data Stream. - Use Amazon DynamoDB to persist a state store that is updated in real-time from a Kinesis Data Analytics application. - Use AWS Lambda to run custom ML inference code on streaming events and write the results to a Kinesis Data Firehose. - Use Amazon SageMaker to train ML models on historical data in S3 and deploy them on streaming data using Kinesis Data Analytics. - Use Amazon QuickSight to build real-time dashboards on streaming data aggregates computed by Kinesis Data Analytics. Here are some recommendations for optimizing performance, cost, and manageability of Kinesis data ingestion pipelines: - Use a partition key that ensures an even distribution of records across shards in a Kinesis data stream. This maximizes the parallelism and throughput of stream processing. - Batch records into larger payload sizes before writing to Kinesis Data Streams or Firehose. This reduces the number of HTTP requests and minimizes latency. - Compress data using formats like GZIP or Snappy before writing to Kinesis Firehose. This reduces data transfer and storage costs. - Use AWS IAM roles and policies to control access to Kinesis APIs and encrypt data using AWS KMS for secure data sharing in multi-tenant architectures. ### Data Lakes and Analytics The end goal of AI/ML data ingestion pipelines is to make data available for data scientists and analysts to build ML models and generate insights. A data lake architecture allows you to store massive amounts of data in a centralized repository and enable self-service analytics on top of it. [AWS Lake Formation](https://aws.amazon.com/lake-formation/) is a fully-managed service that makes it easy to build, secure, and manage data lakes on AWS. AWS Lake Formation provides the following key capabilities for building AI/ML data lakes: - Metadata Crawler – Crawls S3 and databases to capture technical metadata like schema and data types as well as operational metadata like data owners and update frequencies. Stores the metadata in the AWS Glue Data Catalog. - Centralized Access Controls – Provides a single pane for configuring fine-grained access policies across all the data in S3 and data catalog. Integrates with AWS IAM for user management and auditing. - Automated Data Deduplication – Utilizes ML techniques such as fuzzy matching to automatically identify and resolve duplicate records across disparate datasets. - Governed Tables – Allows data stewards to define data quality constraints, schema evolution rules, and data lifecycle policies. Enforces these governance rules on data writes and maintains audit logs. - Acceleration for Analytics Engines – Optimizes the performance of queries from analytics engines like Amazon Athena and Amazon Redshift Spectrum by automatically partitioning, compacting, and tuning datasets. Some patterns and practices for building successful AI/ML data lakes include: - Use a multi-layer data architecture with raw, cleaned, and conformed layers for decoupling storage from compute and allowing multiple use cases to access data. - Use AWS Glue jobs to perform regular compaction, optimization, and cataloging of new datasets that land in the data lake. - Define and enforce data schema and quality standards using a schema registry like AWS Glue Schema Registry and a data quality tool like AWS Deequ. - Build an enterprise-wide data catalog using AWS Glue and use it as the single source of truth for dataset discovery, lineage, and governance. - Use Amazon SageMaker Studio and SageMaker Data Wrangler for data science and ML workflows on data stored in the data lake. ## Conclusion In the age of AI/ML, data is the new oil. However, raw data needs to be refined through a sequence of pipelines before it can power AI/ML applications. A critical stage in that sequence is data ingestion – the process of collecting data from disparate source systems and moving it reliably and efficiently to target storage and processing systems. AWS provides a rich palette of managed services for building data ingestion pipelines for any scale and type of data. Services like Amazon S3 for storage, AWS Glue for data integration, Amazon Kinesis for streaming data, and AWS Lake Formation for governance and security can be composed to create best-of-breed data ingestion architectures for AI/ML use cases. However, building production-grade data pipelines involves much more than just picking the right services. It requires a deep understanding of data characteristics, quality requirements, security and compliance guardrails, and performance and scalability targets. It also requires robust development and operations practices such as source control, testing, monitoring, and incident response. As AWS continues to innovate and add new features in the data ingestion, we can expect these services to get more sophisticated and address emerging challenges in AI/ML data pipelines. One area of innovation I‘m personally excited about is the use of machine learning itself to make data ingestion more automated, intelligent, and self-optimizing. Imagine data ingestion pipelines that can automatically detect schema drift, correct data quality issues, optimize storage and network settings, and scale resources based on predicted workloads. Services like AWS Glue DataBrew and Lake Formation are already starting to introduce some of these smart capabilities. Organizations that crack the code on building agile, automated, and intelligent data ingestion pipelines will be able to democratize data at scale and accelerate their journey from PoC to production AI/ML. They will be able to empower their data scientists and analysts to focus more on high-value activities and less on data wrangling and infrastructure management. More importantly, they will be able to use their data assets to continuously innovate on behalf of their customers and leapfrog the competition. The future of AI/ML is bright and a robust data ingestion backbone built on AWS is a key enabler for that future. --- Source: [Designing Robust Data Ingestion Pipelines on AWS for AI/ML Workloads](https://33rdsquare.com/data-ingestion-featuring-aws/)