5 Things Big Data Can and Cannot Do: Separating Hype from Reality

The big data revolution is well underway. Each day, we generate a staggering 2.5 quintillion bytes of data from our digital activities – from emails and social media posts, to digital photos, purchase transactions, and GPS signals. By 2025, the total amount of digital data created worldwide is projected to reach 175 zettabytes. That‘s more than triple the amount of data generated in 2020.

Clearly, we are living in the age of big data. And for good reason. Over the past decade, big data, coupled with advanced analytics, has demonstrated astonishing potential to transform businesses and societies:

• Enabling smarter decisions and breakthrough insights
• Improving operational efficiency through intelligent automation
• Hyper-personalizing products, services and experiences
• Uncovering new business models and revenue streams
• Accelerating the pace of scientific research and discovery

Given these remarkable capabilities, it‘s no surprise that the global big data and analytics market is booming – expected to grow from $138.9 billion in 2020 to $229.4 billion by 2025.

However, amid all the hype and hysteria surrounding big data, it‘s easy to lose sight of its real-world limitations. While big data can undoubtedly do great things, it is not a silver bullet. As with any technology, it has significant constraints.

In this article, we‘ll explore five key things that big data, for all its power, cannot do – and why the most effective analytics strategies combine big data with irreplaceable human capabilities.

1. Big Data Cannot Replace Human Judgment

One of the biggest misconceptions about big data is that it can provide clear-cut, data-driven answers that trump human judgment. The reality is far more nuanced.

Big data can certainly help inform decisions by providing valuable insights that we may otherwise miss. A retailer might use big data to optimize store layouts, a manufacturer to preventatively maintain equipment, or a baseball team to guide defensive positioning.

However, data insights alone are not enough. Human judgment is still essential for weighing difficult tradeoffs, considering ethics and values, and navigating uncertainty and ambiguity.

Imagine an AI system that predicts a defendant‘s likelihood to re-offend, to help determine sentencing or parole. While the big data insights could be useful, we can‘t simply outsource these weighty decisions to algorithms. Human judges must still interpret the data and carefully weigh things like extenuating circumstances and social costs.

Big data is ultimately a tool to augment human decision-making – not replace it entirely. The most effective approach is a "centaur model" that combines human leadership with machine insights. Like the hybrid chess teams that routinely beat both humans and computers.

2. Big Data Cannot Ask the Right Questions

Another key limitation of big data is that it cannot determine what problems to solve in the first place. It‘s up to humans to ask the right questions and define objectives that create value.

For example, big data could help a social media company optimize for engagement metrics like time-on-site. But is maximizing screen time what‘s best for users‘ well-being or for society? Solely optimizing for narrow engagement metrics risks unintended negative consequences.

Big data is tremendously powerful for answering the questions we ask of it. But it cannot determine what the right questions are to ask. It cannot tell us what metrics to optimize, what products to build, or what strategic bets to make. These are fundamentally human responsibilities that require sound judgment.

As analytics expert Cassie Kozyrkov puts it: "Data is like a lantern: It can illuminate the landscape and provide guidance, but it cannot choose the destination for us."

3. Big Data Cannot Interpret Itself

For all its remarkable capabilities, big data still cannot interpret itself. While advanced analytics can surface insights in the data, it takes human intelligence to understand the significance and determine implications.

Consider how an algorithm might find that ice cream sales and crime rates are correlated. Big data can detect this pattern, but it cannot grasp the meaning. It would take a person to reason that a lurking variable like temperature likely explains the relationship.

Similarly, big data could identify that a customer segment is highly profitable. But it cannot unpack the why behind it. Is it because the company has a unique value proposition for these customers, strong market positioning, or effective sales incentives? The data alone cannot tell the full story.

Interpreting big data insights requires deep domain knowledge of the business context, market dynamics, and customer needs. It demands critical thinking to see beyond surface-level correlations and develop nuanced understandings of root causes. While big data gets the conversation going, human analysts are needed to truly make sense of it.

4. Big Data Cannot Overcome Bad Data

As the saying goes, "garbage in, garbage out." Big data, for all its remarkable capabilities, cannot magically fix bad data. It cannot overcome incomplete, error-prone, or biased data sets.

Big data insights are only as good as the data that goes into them. If the input data is flawed, the output insights will be too. No amount of fancy algorithms can spin straw into gold.

This is a real problem, because most data in the wild is messy and imperfect. It may be missing values, have inconsistent entries, or reflect biased collection practices. For example:

• A medical algorithm trained primarily on data from white patients may not generalize well to other racial groups and exacerbate care inequities.

• A facial recognition model trained mostly on male faces may have higher error rates for women, leading to issues like innocent people being misidentified as criminals.

• An inventory forecast built on incomplete sales data could lead to costly stockouts or a massive surplus.

To overcome these challenges, significant human effort is required to integrate, validate and clean big data before it can be effectively analyzed. Even then, human judgment is critical for scrutinizing insights and considering potential biases.

5. Big Data Cannot Do Everything

Finally, it‘s crucial to recognize that there are many things big data simply cannot do, even with the most sophisticated analytical techniques. These tend to be tasks that rely on intrinsically human capabilities, such as:

Creativity and Innovation: Coming up with outside-the-box ideas or envisioning what doesn‘t yet exist. Big data can surface insights, but it cannot imagine radical new products, business models, or creative works.

Empathy and Social Skills: Connecting with others, building relationships, motivating teams, and persuading stakeholders. Big data cannot navigate complex social dynamics or inspire people to action.

Exploration and Sensemaking: Venturing into uncharted territory to discover new problems and opportunities. Big data is constrained by the parameters we set and the data we collect. It cannot explore the unknown in open-ended ways like humans can.

Holistic, Contextual Reasoning: Grasping the bigger picture, connecting the dots, and considering broader contexts. Big data can find granular patterns, but not the overall meaning and implications.

Ultimately, the power of big data is not about automating away humans, but rather empowering them to do what they do best. The magic happens when human and machine intelligence work together in complementary ways.

Putting It All Together

Big data has undoubtedly ushered in a new frontier of insight and innovation. From hyper-personalized medicine to self-driving cars, the possibilities are truly revolutionary.

However, in our collective exuberance, we must not overlook the very real limitations of big data. We must balance enthusiasm with healthy realism about what this technology can and cannot achieve.

Big data, for all its remarkable capabilities, cannot replace human judgment, ask the right questions, interpret itself, fully overcome data quality issues, or address tasks that rely on fundamentally human skills. Appreciating these constraints is critical for wielding this powerful tool responsibly and effectively.

Perhaps the real magic of big data is not what it does alone, but rather how it partners with human intelligence to enable incredible new possibilities. The best analytics is a centaur model – combining the lightning-fast calculation of big data with the slow hunch of human intuition.

The organizations that master this symbiotic partnership between humans and machines, while respecting the unique strengths of each, will be the ones that truly harness the power of big data – and shape the future.

As you embark on your own big data journey, keep sight of both the tremendous possibilities and the important limitations. Because pairing pragmatism with ambition is ultimately what separates hype from reality – and success from failure.