What is Big Data? A Quick Introduction for Analytics and Data Engineering Beginners

In today‘s digital world, data is being generated at an unprecedented pace from a multitude of sources. From web logs and mobile devices to IoT sensors and social media posts, the volume, variety and velocity of data has exploded in recent years. Welcome to the era of big data.

As an aspiring data analyst or engineer, it‘s crucial to understand what big data is, why it matters, and how it differs from the traditional structured datasets of the past. In this article, we‘ll provide a beginner-friendly overview of big data and share some key concepts and tools you should know from the perspective of artificial intelligence (AI) and machine learning (ML). Let‘s dive in!

The Exponential Growth of Data

The amount of digital data in the world has been growing exponentially. According to research by IDC, the global datasphere will grow from 33 zettabytes (ZB) in 2018 to 175 ZB by 2025.[^1] To put that in perspective, 175 ZB is equivalent to 175 billion terabytes (TB) – enough storage for over 2 trillion hours of 4K video![^2]

[^1]: IDC White Paper, sponsored by Seagate, "The Digitization of the World – From Edge to Core", November 2018
[^2]: Bernard Marr, "How Much Data Do We Create Every Day? The Mind-Blowing Stats Everyone Should Read", Forbes, May 2018

This data is coming from an ever-expanding number of sources. In 2020, over 50% of data is projected to come from IoT devices like industrial equipment sensors, smart home appliances, self-driving cars and more.[^3] Every minute, Google conducts 3.8 million searches, YouTube users watch 4.5 million videos, and Zoom hosts 208,333 participants in meetings.[^4] [^3]: IDC InfoBrief, sponsored by Seagate, "Data Age 2025: The Digitization of the World", November 2018
[^4]: Domo Report "Data Never Sleeps 8.0", June 2020

Defining Big Data

So what exactly is "big data"? While there is no strict criteria on how "big" the data needs to be, it generally refers to datasets that are too large or complex to be handled by traditional data processing tools and techniques. The unique characteristics of big data are often described by the "3 Vs":[^5] [^5]: Laney, Douglas. "3D data management: Controlling data volume, velocity and variety." Gartner. February 6, 2001.

1. Volume

The sheer scale and quantity of the data, often measured in terabytes, petabytes or beyond. For example, Facebook generates over 4 petabytes of data per day, including photos, videos, messages, and logs.[^6] The Large Hadron Collider (LHC) at CERN produces about 90 petabytes of data per year from particle collision experiments.[^7] [^6]: Josh Constine, "Facebook now has 2.5 billion monthly users…and responsibility", TechCrunch, July 2019
[^7]: Chris Jablonski, "Handling the Big Data of the Large Hadron Collider", ZDNet, September 2018

2. Variety

The diversity of data formats and structures, including structured, semi-structured and unstructured data. Big data often combines traditional structured data (like database tables), semi-structured data (like XML or JSON), and unstructured data (like text, images, audio, and video). Twitter data is a good example, with structured metadata fields like timestamp and geolocation combined with unstructured tweet text and images.

3. Velocity

The speed at which data is being generated, collected and analyzed. Many big data use cases involve real-time or near real-time processing of streaming data. For example, The New York Stock Exchange (NYSE) captures 1 terabyte of trade information during each trading session.[^8] Machines and sensors in manufacturing can generate thousands of readings per second for quality control and predictive maintenance.

[^8]: "The New York Stock Exchange is transforming financial markets with Azure", Microsoft Customer Story, May 2020

Some have proposed additional "V‘s" of big data, like Veracity (data quality and accuracy), Value (business benefits), and Variability (data consistency over time). But the core 3 Vs capture the key characteristics that distinguish big data from traditional datasets.

Why Big Data Matters

The ability to harness big data offers immense potential value for businesses and society. By analyzing these vast troves of data, we can surface hidden patterns, derive new insights, and make more informed decisions than ever before. According to McKinsey Global Institute, applying big data analytics to improve decision making could generate over $100B in value across the US healthcare system and €250 billion in value for Europe‘s public sector administration.[^9] [^9]: McKinsey Global Institute, "Big data: The next frontier for innovation, competition, and productivity", May 2011

Some common industry applications of big data include:

• Retail: Analyzing customer behavior data to optimize pricing, promotions, and product assortment. Walmart handles over 1 million customer transactions per hour.[^10]
• Manufacturing: Analyzing sensor data from production lines and equipment to predict maintenance needs and prevent quality issues.
• Healthcare: Combining genomic data, medical records, and wearable health trackers to enable precision medicine and personalized treatments.
• Finance: Detecting fraudulent transactions in real-time by analyzing patterns across millions of transactions.
• Telecom: Optimizing network capacity and predicting equipment failures by analyzing traffic data, service logs, and network throughput.

[^10]: Bernard Marr, "Really Big Data At Walmart: Real-Time Insights From Their 40+ Petabyte Data Cloud", Forbes, January 2017

The Role of AI and Machine Learning

Artificial intelligence (AI) and machine learning (ML) play an increasingly crucial role in deriving value from big data. With traditional statistical methods, the time and effort required to analyze massive datasets is often prohibitive. AI and ML provide more efficient and scalable ways to convert big data into insights and action.

Some key AI/ML techniques used with big data include:

• Supervised learning to train classification and regression models on labeled big data for prediction tasks. For example, predicting customer churn risk based on historical behavioral data and churn labels.

• Unsupervised learning to discover hidden patterns and segment big data without the need for labeling. Clustering algorithms can group similar customers, products or transactions based on common attributes.

• Deep learning with neural networks to learn hierarchical features from complex, unstructured data like images, audio, and video. Convolutional neural networks (CNNs) can be trained on millions of labeled images for object detection and facial recognition.

• Natural language processing (NLP) to analyze and extract meaning from large volumes of text data. Sentiment analysis models can classify social media posts as positive, neutral or negative to gauge brand perception.

• Anomaly detection to identify rare events or outliers in big data streams for applications like fraud detection and equipment health monitoring.

• Recommendation systems to suggest relevant content or products to users based on big data on past behavior, preferences, and similarities with other users.

By leveraging these AI and ML techniques, organizations can automate the discovery of insights from big data and build intelligent applications to drive value. IDC predicts that by 2025, nearly 30% of data will be consumed in real-time and over 10% of it will be "hypercritical" business data that is directly leveraged by AI/ML.[^11] [^11]: IDC White Paper "Data Age 2025: The Digitization of the World – From Edge to Core", November 2018

A Big Data Architecture Example

To illustrate how the various big data tools and technologies can fit together, let‘s walk through an example architecture for real-time fraud detection in credit card transactions.

1. Credit card transactions are captured in real-time from POS terminals, ATMs and online payment gateways.

2. The transaction data streams into an ingestion layer using Apache Kafka. Kafka enables buffering and low-latency pubsub messaging between producers and consumers at high scale.

3. The raw transaction data lands in a staging area in HDFS for initial preprocessing and validation using Spark Streaming jobs. Invalid transactions are filtered out and valid ones are enriched with additional reference data.

4. Clean transaction data is published to Kafka topics and consumed by Spark Streaming for real-time model scoring. The ML model predicts the probability of each transaction being fraudulent based on historical patterns.

5. Transactions with high fraud probability are written to a Cassandra database for investigation by analysts. Cassandra provides fast writes and time series queries for storing the flagged transactions.

6. A secondary pipeline consumes the transaction data from Kafka and loads it into a data warehouse like Amazon Redshift after applying batch ETL transformations using Spark SQL.

7. The data warehouse combines the streaming transaction data with other batch data sources and serves as the source of truth for BI, reporting, and ad-hoc historical analysis.

8. Summary data on transaction volumes and fraud trends are visualized in real-time on analyst dashboards built with Tableau connected to the streaming views in Cassandra.

Of course, this is a simplified architecture and actual fraud detection systems at major banks involve many more sub-components and optimization. The key point is to show how different big data tools can be integrated to enable the real-time processing and analysis of streaming data alongside historical data at scale.

Ethical Considerations in Big Data and AI

As we collect and analyze more granular data on human behavior, it‘s critical to consider the ethical implications. Some key issues in big data ethics include:

• Privacy and consent: Personal data collected from mobile apps, IoT devices and web browsing is often used for advertising and sold to third-party data brokers without clear consent. Regulations like GDPR aim to give consumers more control and transparency over their personal data.

• Algorithmic bias: AI/ML models trained on biased historical data can learn and amplify societal biases around gender, race, age and other attributes. It‘s important to audit training data and model outputs for bias and actively work to mitigate it.

• Transparency and accountability: The "black box" nature of complex AI/ML models like deep neural networks makes it difficult to explain how predictions are made. Explainable AI techniques aim to provide more transparency on model decision making.

• Fairness and discrimination: Big data-driven algorithms increasingly make high-stakes decisions like loan approvals, hiring, and criminal sentencing. Ensuring these systems are fair and don‘t discriminate against protected groups is an ongoing challenge.

These are complex issues with no easy answers. It‘s important for data professionals to be aware of the ethical considerations in their work and engage in the broader societal dialogue around data privacy, algorithmic accountability, and fairness.

Getting Started with Big Data

While the big data technology landscape may seem overwhelming at first, don‘t be intimidated! With the wealth of open source tools and cloud platforms available today, it‘s easier than ever to get hands-on experience with big data. Here are some learning suggestions to get started:

• SQL: Structured query language is the foundation for working with data. Practice querying and joining tables in a relational database like MySQL or PostgreSQL. Platforms like Mode Analytics and Leetcode offer interactive SQL practice problems.

• Python: Python is the most popular language for data analysis and machine learning. Learn libraries like Pandas, NumPy, Matplotlib, and Scikit-Learn for data manipulation, statistics, visualization, and modeling. Jupyter Notebooks provide an interactive coding environment.

• Spark: Apache Spark is the leading framework for distributed data processing. Work through the Spark documentation and examples using PySpark. Practice transforming and analyzing datasets using RDDs, DataFrames, and Spark SQL.

• Cloud Platforms: Major cloud providers like AWS, Azure, and GCP offer managed big data services that remove the need to setup and manage clusters. Explore Google BigQuery, Amazon EMR, and Databricks for cloud-based data analytics.

• Kaggle Competitions: Kaggle is the most popular online community for data science and machine learning. Participating in prediction competitions is a great way to practice big data skills on real-world datasets and learn from top practitioners.

• Data Engineering Books: Some helpful books for learning data engineering fundamentals include "Designing Data-Intensive Applications" by Martin Kleppmann, "Hadoop: The Definitive Guide" by Tom White, and "Spark: The Definitive Guide" by Bill Chambers and Matei Zaharia.

Big data and AI will only continue to grow in importance across industries. As an aspiring data professional, investing in these foundational skills will serve you well in the years ahead.

Conclusion

In this article, we‘ve covered the key concepts behind big data, its characteristics of volume, velocity, and variety, and how it differs from traditional data workloads. We explored common big data use cases and the role of AI and machine learning in extracting insights. We walked through an example big data architecture for real-time fraud detection and discussed important ethical considerations.

While big data presents immense opportunities, it also introduces new technical and organizational challenges. At a technical level, the tools and architectures for processing big data are rapidly evolving. Staying on top of the latest technologies is an ongoing learning process. At an organizational level, succeeding with big data requires new roles, skills, and processes. Driving alignment between data engineers, data scientists, analysts and business stakeholders is key.

As big data continues its exponential growth, so does the demand for data professionals that can harness it for insight and impact. Whether you aspire to be a data analyst, data scientist, or data engineer, I encourage you to continue learning about big data tools and techniques. We‘re still in the early innings of the big data revolution and the potential is immense.

You can find more articles on data engineering, data science and machine learning on the Analytics Vidhya blog. Subscribe to stay up to date on the latest guides and tutorials. If you have any questions or thoughts on big data, feel free to leave a comment below. Thanks for reading!