Choosing the Right File Format for Storing Your AI/ML Data

In the era of artificial intelligence and machine learning, data is the new oil. According to IDC, the global datasphere will grow from 33 zettabytes in 2018 to a staggering 175 zettabytes by 2025. Data is being generated at breakneck speeds from an ever-expanding array of sources, including IoT sensors, mobile devices, social media, and enterprise systems.

For organizations looking to harness this data to drive AI/ML initiatives, the question of how to efficiently store and process massive volumes of data is top of mind. The choice of file format might seem like an inconsequential implementation detail, but as data scales into the petabytes and beyond, it becomes a critical factor in the performance, cost, and maintainability of AI/ML pipelines.

Why File Format Matters for AI/ML

In an AI/ML context, the main goal of data storage is to feed data to training and inference pipelines as efficiently as possible. The choice of file format impacts AI/ML workflows in several key ways:

• I/O performance: How quickly can the data be read from storage and fed to training/inference jobs? This is especially important when dealing with terabytes of unstructured data like images, video, and audio.

• Storage efficiency: How much space does the data consume on disk? More efficient storage means lower infrastructure costs and faster data transfer/processing.

• Preprocessing overhead: How much work is required to convert and preprocess data before feeding it to ML models? Certain formats can store data in a way that‘s closer to the ideal representation for training/inference.

• Interoperability: Is the format supported by the ML frameworks and libraries you use? Choosing a non-standard format may lead to integration headaches.

To illustrate the impact of file format, let‘s look at an example. Imagine you have a petabyte-scale image classification task. If you stored the images as uncompressed PNGs, each image might be 1-5 MB in size. In total, you‘d be looking at up to 5 PB of storage! Not only would this be unnecessarily expensive to store, but loading and processing a 5 MB image for each training example would be very inefficient.

By converting the images to a format like TFRecord (TensorFlow‘s native binary storage format), you could leverage techniques like compression and chunking to dramatically reduce storage requirements and enable more efficient I/O. In fact, Google has reported 10x-100x improvements in training performance on large image datasets by using TFRecord instead of raw images.

Benchmarking File Formats for AI/ML

To give a more concrete sense of how file format affects AI/ML workloads, let‘s look at some benchmarks. In one study by IBM Research, the performance of Parquet, ORC, and Avro was compared for a neural network training workload:

Parquet achieved the best compression ratio at 0.35, meaning the Parquet dataset was 35% of the original uncompressed size. ORC was close behind at 0.37, while Avro saw no storage size reduction. In terms of training performance, Parquet was the clear winner, with ORC and Avro coming in 15% and 30% slower respectively.

Another benchmark by Databricks compared JSON, CSV, and Parquet for a distributed Spark machine learning pipeline:

Again we see significant compression benefits with Parquet – nearly 20x compared to JSON. Even CSV was 7x larger than Parquet. When it came to distributed training and inference, Parquet‘s columnar format and query optimizations translated to a 3x speedup over JSON and 2x over CSV.

Of course, these benchmarks only illustrate a narrow slice of possible AI/ML use cases. Factors like data structure, schema evolution, and specific ML framework being used can all tip the scales in favor of different formats.

Recommendations for AI/ML File Formats

Based on the above benchmarks and analysis, here are some general recommendations for selecting a file format for common AI/ML use cases:

• Tabular data (structured): If your data is structured and tabular like a relational database, Parquet is an excellent choice. It compresses well, supports column-wise access, and integrates with a variety of distributed SQL query engines. If you need ACID transactions, consider using a modern table format like Iceberg or Delta Lake that builds on Parquet.

• Complex hierarchical data: For nested data with lots of variable length strings, JSON or Avro may be preferable to force-fitting it into a tabular format. With JSON you trade off storage efficiency and query speed for maximum flexibility. Avro gives you a more compact binary representation and schema evolution.

• Media data (images, audio, video): Media use cases call for specialized formats that support compression and chunking to minimize I/O bottlenecks. JPEG is a good baseline for images. TFRecord is ideal if you‘re using TensorFlow. For audio, WAV and AIFF are common choices. With video, compression is key – explore formats like MP4, AVI, MKV.

• Natural language data: If you‘re working with text corpora for NLP, efficiency really matters as you scale to billions of documents. Formats like RecordIO support compression and allow you to store metadata alongside the text data. Many NLP libraries have built-in support for optimized text formats, e.g. PyTorch‘s torch.utils.data.Dataset.

Ultimately, the "right" file format depends on your specific use case and tooling. When in doubt, defer to your machine learning framework‘s recommended format.

Chris Fregly, Founder at PipelineAI, emphasizes the importance of optimizing data access in AI/ML pipelines:

"When building machine learning pipelines, it‘s critical to consider how your data is being stored and accessed. Using a compressed columnar format like Parquet can dramatically speed up training and inference, especially for large structured datasets. Don‘t leave data storage as an afterthought!"

Conclusion

In the rapidly evolving world of AI and ML, it‘s easy to get caught up in model architectures and hyperparameters. But the foundation of any successful AI/ML initiative is high quality, efficiently stored data. By understanding the pros and cons of different big data file formats and aligning format selection with your ML use case, you can supercharge your AI/ML workflows.

Some key takeaways:

• Prioritize columnar formats like Parquet for large structured datasets
• Use binary formats like Avro and TFRecord over JSON and CSV for better storage efficiency and I/O performance
• Select media formats that support chunking and compression to accelerate training/inference on unstructured data
• Avoid reinventing the wheel – leverage file formats and utilities that integrate well with your ML framework of choice

As data continues to grow and AI/ML techniques evolve, the big data ecosystem will keep innovating on storage formats and compression techniques. Formats like Apache Iceberg and projects like IBM‘s Bosen show early promise for format-aware distributed training. By keeping an eye on these developments and continuously re-evaluating storage approaches, organizations can stay ahead of the curve and wring maximum value out of their AI/ML investments.