Just How Big is an Exabyte?

Hey there! As a fellow tech geek, I know you‘ll appreciate just how mind-bogglingly huge an exabyte of data really is. With global data expanding exponentially, terms like exabyte are becoming more common. But it‘s still hard to wrap your head around it without some real-world context. Let me break it down for you…

Defining this Behemoth

First, let‘s define this beast. An exabyte (EB) is an unfathomable 1,000 petabytes or 1 billion gigabytes. Let‘s compare:

• 1 GB could store a short HD movie
• 1 TB could store about 1,000 HD movies
• 1 PB could store over 1 million HD movies
• 1 EB could store over 1 BILLION HD movies!

That‘s over 2 million years of nonstop HD video playback. Crazy right?!

Here‘s another one:

• 1 GB could store a small library of ebooks
• 1 TB could store the US Library of Congress print collection
• 1 PB could store 50 Library of Congresses
• 1 EB could store the text of 50,000 Libraries of Congress!

That‘s a LOT of books. Like you could read for 100 lifetimes without repeating.

Exabyte Data Growth Going Exponential

Now that you have a sense of how ginormous an exabyte is, consider this – the global datasphere is predicted to grow from 33 zettabytes in 2018 to 175 zettabytes by 2025 according to IDC research.

For us data nerds, let‘s break that down:

• 1 zettabyte = 1,000 exabytes
• 175 zettabytes = 175,000 exabytes

That means in just 7 years we‘ll have 175 million exabytes of data! Just think of the possibilities and discoveries buried in those endless mountains of data!

But of course, incredible opportunities come with immense challenges. We‘re already struggling to store, manage and extract value from large datasets. How will we ever wrestle 175,000+ exabytes?!

Why So Much Data?

What‘s driving this breakneck expansion of the global datasphere? There are several key trends:

• Billions of Sensors – The Internet of Things (IoT) is connecting billions of devices and sensors gathering data nonstop about infrastructure, manufacturing, transportation, weather patterns, health metrics, and more. Intel predicts 200 billion IoT devices will be online by 2030!

• 4K/8K Video – Higher resolution cameras on smartphones and TVs means video data is multiplying quickly. Just look at the rise of bandwidth-hungry Netflix, Youtube and Twitch.

• AI/ML Datasets – Training advanced AI and machine learning models requires analyzing vast datasets, often petabytes in size. Autonomous vehicles gobble up exabytes of mapping data.

• Genomics Mapping – Sequencing human genomes creates enormous datasets. The Earth BioGenome Project aims to sequence the DNA of all 1.8 million known species on earth – producing an expected 10-100 exabytes of genetic data.

• Scientific Research – High-energy physics experiments like the Large Hadron Collider produce ridiculous amounts of sensor data – 15 petabytes annually. Radio telescopes scanning the cosmos create exabytes of sky maps.

The Data Storage Struggle is Real

With so much data flooding in, companies and research institutions are desperate for storage solutions. But exabyte-scale data is a beast to wrangle:

• Google processes over 3.5 billion searches per day. Indexing the entire internet is estimated to require 15-30 exabytes of data and counting. At some point searching the web risks hitting a storage wall.

• The Square Kilometer Array radio telescope under construction will gather over 1 exabyte of observational data every day. Where do you put decades of daily exabyte imports?

• Genomics researchers need to store vast databases of sequenced DNA. But established institutions lack the enormous data infrastructure required.

• Media giants like Netflix, Hulu, Spotify and Twitch have to store ever-growing libraries of 4K/8K video. Their demand for data center capacity is nearly insatiable.

Clearly, we need major breakthroughs in data storage tech if we hope to keep up…

Pushing Boundaries of Data Storage

With exabyte-scale data storage needs growing, both legacy tech giants and scrappy startups are racing to innovate:

DNA Storage – Using synthetic DNA to archive data by encoding 1s and 0s in genetic base pairs. 1 gram of DNA could theoretically store 1 exabyte! But it‘s complex and costly…for now.

Holographic Storage – Holograms etch data throughout a crystal medium using lasers. A 6 x 6 x 1 inch quartz cube could store 1 exabyte! But rewritable holographic storage faces physics challenges.

Cold Storage – Facebook is experimenting with submerging its cold storage data centers underwater to save cooling costs. Others use Antarctic air to near-freeze data warehouses. Smart, if you can stand the cold storage bills!

Quantum Computing – Will enable breakthroughs in speed, encryption and capacity that could remove exabyte-scale computing bottlenecks. But we‘re still years from useful quantum computers.

Exabyte Data Use Cases

While we wait for exotic storage tech to mature, some organizations already rely on exabyte data:

Academic Research – The Large Synoptic Survey Telescope will capture 20 terabytes of images nightly that must be stored and analyzed over a decade.

Government Security – The NSA‘s Utah data center stores exabytes of surveillance data. Redundancy and access controls are paranoid-grade.

Cloud Services – Amazon Web Services offers 100 petabyte data warehouse capacity via Snowmobile semi-trucks for physical data transfer.

Search Engines – Google processes over 3.5 billion searches per day, relying on exabyte-scale computing for indexing and retrieval from its sprawling data centers.

Social Networks – Facebook stores millions of photos and videos. Cold storage in underground Arctic vaults helps them affordably scale 100+ petabyte repositories.

Bracing for the Exabyte Era

As you can see, the world is hurtling towards a future measured in exabytes. The possibilities for discovery and innovation are incredibly exciting. But to harness massive datasets, we need new thinking on data storage, processing, analytics, privacy, and more.

What potential do you see in a world awash with exabytes of data? I expect amazing new datadriven technologies, but also challenges around data integrity, security, and ethics. One thing is certain – the exabyte era will require all of our collective creativity as technology architects to build a robust and responsible data infrastructure. It‘s a thrilling time to be a data geek! Let me know if you have any other big data questions.