Building AI and Machine Learning Solutions with Azure Data Lake Storage Gen2

Azure Data Lake Storage Gen2 (ADLS Gen2) is becoming the de facto storage choice for cloud-based AI and machine learning projects. With its unmatched combination of scale, performance, and integration with Azure machine learning services, it provides an ideal foundation to build production AI solutions.

In this in-depth guide, we‘ll explore why ADLS Gen2 is so well-suited for AI/ML workloads and walk through best practices and code samples for connecting to and reading data in your ML projects. Whether you‘re a data scientist, ML engineer, or architect, you‘ll come away with a solid understanding of how to leverage ADLS Gen2 in your work. Let‘s dive in!

Why Azure Data Lake Storage Gen2 for AI and Machine Learning?

According to a recent survey by Gradient Flow^1, cloud file storage services are the most popular option for storing data for ML projects, used by over 60% of enterprises. Azure Data Lake Storage Gen2, launched in 2019 as an evolution of Gen1, has quickly become a leading choice in this segment and continues to see rapid adoption.

There are several key reasons why AI/ML professionals prefer ADLS Gen2:

1. Scalability for big data – Training ML models, especially deep learning models, often requires massive labeled datasets. Object detection models are trained on datasets like Open Images^2 with millions of images. Language models like GPT-3 are trained on hundreds of gigabytes of text^3. Linear scalability is essential, and ADLS Gen2 delivers with the ability to store billions of files and petabytes of data in a single filesystem.

2. Performance for data-intensive workloads – Aside from the raw volume of data, ML workloads involve highly data-intensive computations, often with frequent random reads. A resnet-50 training run in PyTorch on ImageNet data can involve 40M+ random reads^4. ADLS Gen2 is backed by fast SSD storage and provides high throughput for both small random reads and large sequential reads.

3. Flexible data format support – ML datasets come in a variety of formats. Images may be stored as JPEGs or PNGs, text as CSVs or JSONs, feature vectors as NumPy arrays or Parquet files. ADLS Gen2 natively supports any data format and provides optimizations for formats like Parquet that are common in ML feature stores.

4. Integration with Azure AI/ML platform – ADLS Gen2 is deeply integrated with Azure machine learning services like Azure Machine Learning, Azure Databricks, and Azure Synapse Analytics. You can seamlessly access data in ADLS Gen2 to train models with powerful GPUs/TPUs, build ML pipelines, and deploy them to production.

5. Security and compliance – Enterprise AI projects have strict security and compliance needs. Patient health records and financial data are subject to regulations like HIPAA and GDPR. ADLS Gen2 supports fine-grained POSIX ACLs and integrates with Azure Active Directory for role-based access control. All data is automatically encrypted at rest.

Connecting to ADLS Gen2 in Python

As a data scientist or ML engineer, you‘ll mostly be interacting with ADLS Gen2 through Python code to access data for training and deploying models. Here‘s how you can connect to ADLS Gen2 using the azure-storage-file-datalake library and retrieve a file:

from azure.storage.filedatalake import DataLakeServiceClient

account_name = "myadlsaccount"
container_name = "myfilesystem"
file_path = "path/to/data.csv"

service_client = DataLakeServiceClient(account_url=f"https://{account_name}.dfs.core.windows.net", credential="mykey")
file_system_client = service_client.get_file_system_client(file_system=container_name)
file_client = file_system_client.get_file_client(file_path)

with file_client.download_file() as reader:
    content = reader.readall()

The key steps are:

1. Import the necessary modules from the azure-storage-file-datalake package
2. Provide the ADLS Gen2 account name, container/filesystem name, and path to the file you want to read
3. Create a DataLakeServiceClient with the account URL and access key credential
4. Get a FileSystemClient for the target filesystem
5. Get a FileClient for the target file path
6. Download the file and read its contents

Reading Parquet Data in PyTorch or TensorFlow

While you can use ADLS Gen2 to store raw data like images and CSVs, it‘s recommended to convert your data into optimized formats like Parquet for better performance in ML training.

Parquet is a columnar storage format that provides efficient compression and encoding schemes. It‘s well-suited for ML feature data that is accessed in columnar fashion.

Here‘s how you can efficiently read a Parquet file from ADLS Gen2 into a PyTorch Dataset:

import dask.dataframe as dd
from torch.utils.data import Dataset

class ADLSParquetDataset(Dataset):
    def __init__(self, account_name, filesystem_name, file_path):
        self.url = f"abfs://{filesystem_name}@{account_name}.dfs.core.windows.net/{file_path}"
        self.df = dd.read_parquet(url, storage_options={"account_name": account_name, "account_key": "mykey"})

    def __len__(self):
        return len(self.df)

    def __getitem__(self, idx):
        return self.df.iloc[idx].to_dict()

ds = ADLSParquetDataset("myadlsaccount", "myfilesystem", "data/features.parquet")

This creates a custom torch.utils.data.Dataset that lazily reads a Parquet file from ADLS Gen2 using Dask. When accessed in a PyTorch DataLoader, it will efficiently load data batches during training.

You can similarly read Parquet data into TensorFlow using the pyarrow library:

import pyarrow.parquet as pq

table = pq.read_table(f"abfs://{filesystem_name}@{account_name}.dfs.core.windows.net/{file_path}", filesystem=fs)
df = table.to_pandas()
dataset = tf.data.Dataset.from_tensor_slices(dict(df))

AI/ML Solution Architectures with ADLS Gen2

When building an end-to-end AI/ML solution in Azure, ADLS Gen2 serves as the central storage layer that connects each stage of the workload. Here‘s a reference architecture^5:

1. Data ingestion – Raw data lands in ADLS Gen2 through Azure Data Factory pipelines, Kafka connectors, or direct uploads. This includes structured, semi-structured, and unstructured data from databases, apps, and IoT devices.

2. Data preparation – Raw data is cleaned, transformed, and labeled using Spark on Azure Databricks. Intermediate data is stored in Parquet format in ADLS Gen2. ADLS Gen2 scales to handle very large datasets.

3. Model training – Prepared feature data is loaded from ADLS Gen2 into Azure Machine Learning for model training. Deep learning models are trained on GPU/TPU clusters with hyperparameter tuning. Trained models are versioned in the Azure ML Model Registry.

4. Model deployment – Trained models are deployed as scalable web services in Azure Kubernetes Service. They read input data from and write predictions back to ADLS Gen2 for serving and logging.

5. Monitoring and retraining – Azure ML and Application Insights monitor deployed models for data drift and performance degradation. Retraining pipelines automatically kick off to retrain and deploy models on new data in ADLS Gen2.

Throughout this lifecycle, ADLS Gen2 provides the unified storage backend to support the data flows and artifacts in each stage. Data security and governance is provided through RBAC, ACLs, encryption, and auditing capabilities.

Best Practices for Using ADLS Gen2 in AI/ML Projects

To get the most out of Azure Data Lake Storage Gen2 in your machine learning projects, consider the following best practices:

1. Plan your directory structure – Organize your data in ADLS Gen2 according to how it will be used in ML pipelines. Use separate directories for raw data, prepared features, and model outputs. Partition data by time and/or source for easy querying.

2. Use optimized data formats – Convert your data into columnar formats like Parquet for faster scans and queries. Use Avro for large-scale data serialization. Consider optimized ML formats like TFRecord, PyTorch TensorDataset, and Petastorm.

3. Secure your data – Take advantage of ADLS Gen2‘s fine-grained security and identity management features. Use Azure AD authentication and role-based access control. Encrypt sensitive data and consider using a separate storage account for it.

4. Optimize for your access patterns – Choose the right access tier (hot, cold, archive) for your data based on how frequently it will be read in ML workloads. Use lifecycle management policies to automatically transition data. Prefer large files over many small files for better query performance.

5. Monitor and tune performance – Use Azure Monitor to collect metrics and set alerts on your ADLS Gen2 storage. Look out for throttling errors or high latency. Tune jobs for optimal parallelism and consider using performant runtimes like Hyperspace^6 where possible.

ADLS Gen2 Customer Stories in AI/ML

Innovative companies across industries are using Azure Data Lake Storage Gen2 to power their AI and machine learning initiatives. Here are a few examples:

• AGL Energy – Australian energy provider AGL uses ADLS Gen2 to store and analyze over 3 billion records per month including smart meter data^7. By training machine learning models on this data in Azure Databricks, they‘ve been able to better predict energy demand and prices.

• Anheuser-Busch InBev – The world‘s largest brewer AB InBev uses ADLS Gen2 to store quality inspection data from their manufacturing plants globally^8. They use ML in Azure Databricks to detect quality issues early in the supply chain, reducing waste and saving millions in cost.

• Discngine – French biotech company Discngine stores and processes petabytes of DNA sequencing data in ADLS Gen2^9. Their researchers use Azure Machine Learning to build deep learning models for drug design and discovery, enabled by the scalable storage.

Conclusion

Azure Data Lake Storage Gen2 has quickly become the preferred storage choice for AI and machine learning projects in the cloud, and for good reason. It provides the scale, performance, flexibility, and security that these data-intensive workloads demand.

As an AI/ML practitioner, you can take advantage of ADLS Gen2 to:

• Store and process massive volumes of structured and unstructured data
• Access data in optimized formats like Parquet from machine learning frameworks
• Build performant and scalable model training and deployment pipelines
• Secure your sensitive data and comply with regulatory requirements

By following best practices and learning from the success stories of leading enterprises, you‘ll be in a great position to deliver value with AI using Azure Data Lake Storage Gen2. The future of AI is data-driven, and ADLS Gen2 is the foundation you can trust.

References