Four Data Engineering Fundamentals All Data Scientists Must Know

Introduction

In the world of data science, it‘s easy to get enamored with the latest modeling techniques or flashy visualizations. But the reality is, the success of any data science project hinges on the quality and reliability of the data fueling it. That‘s where data engineering comes in.

Data engineering is the foundation upon which data science is built. It encompasses the technologies, processes, and people that collect, store, process, and serve the data that powers analytics and machine learning.

While data scientists may not build data pipelines themselves, a strong understanding of data engineering principles is invaluable. It allows you to make informed decisions about what data to use, diagnose issues when they arise, and ensure your work is scalable, compliant, and ethical.

In this post, we‘ll dive deep into four data engineering concepts every data scientist should know:

1. Data Warehouses and Data Lakes
2. ETL Pipelines
3. Data Governance and Quality
4. Data Regulations and Ethics

For each topic, we‘ll cover the technical fundamentals, why it matters for data science and machine learning, current best practices, and expert insights. Let‘s get started!

Data Warehouses and Data Lakes

What They Are

A data warehouse is a centralized repository for structured, filtered data that has already been processed for a specific purpose. It integrates data from multiple sources and is optimized for fast SQL queries, typically in read-only mode. Leading data warehouse technologies include:

• Amazon Redshift
• Google BigQuery
• Microsoft Azure Synapse Analytics
• Snowflake

Here‘s an example of creating a table in Redshift:

CREATE TABLE users (
  id INT PRIMARY KEY,
  username VARCHAR(255),
  email VARCHAR(255),
  created_at TIMESTAMP
);

In contrast, a data lake is a vast pool of raw, unstructured data in its native format. Schema and data requirements are not defined until the data is needed. Data lakes offer more flexibility but require more effort to process data on read. Popular data lake solutions include:

• Amazon S3
• Azure Data Lake Storage
• Google Cloud Storage
• Databricks Delta Lake

Why They Matter for Data Science and Machine Learning

The choice of data warehouse vs data lake can significantly impact machine learning projects. Data warehouses are often used for traditional BI reporting on structured data, while data lakes are better suited for machine learning and AI use cases with unstructured data like images, video, and text.

Data lakes have become especially popular for machine learning because they allow data scientists to quickly ingest and experiment with new datasets without extensive upfront processing. According to a 2020 survey by Databricks, 41% of organizations now use data lakes as their primary data repository for data science and machine learning [1].

However, data warehouses still play a key role in many ML pipelines as a source of cleansed, integrated data for feature engineering and model training. The rise of SQL-based MPP data warehouses like BigQuery has made it easier for data scientists to build models directly on warehoused data.

Best practice is often a layered approach – land raw data in a lake, process it into structured datasets in a warehouse, then serve it to ML models via an analytical store or feature store. Tools like Databricks can help unify these layers.

ETL Pipelines

What They Are

ETL (Extract, Transform, Load) pipelines are the plumbing that moves and processes data from source systems to destination datastores like warehouses and lakes. The basic steps are:

1. Extract data from sources like databases, APIs, streaming systems
2. Transform data by filtering, aggregating, joining, cleaning, etc.
3. Load processed data into a destination system

Here‘s a simple example of an ETL pipeline using Python and SQL:

import pandas as pd
import sqlalchemy as sa 

# Extract data from CSV
data = pd.read_csv(‘users.csv‘)

# Transform data 
data[‘fullname‘] = data[‘first‘] + ‘ ‘ + data[‘last‘] 
data = data.drop([‘first‘, ‘last‘], axis=1)

# Load data to database
engine = sa.create_engine(‘postgresql://user:pass@host/db‘)
data.to_sql(‘users‘, engine, if_exists=‘replace‘, index=False)

In the cloud era, serverless, managed ETL services have become popular to reduce infrastructure overhead. Leading solutions include:

• AWS Glue
• Google Cloud Dataflow
• Azure Data Factory
• Fivetran

Why They Matter for Data Science and Machine Learning

Reliable ETL pipelines are critical for delivering high-quality data to train machine learning models. Poor data quality is a leading cause of ML project failure. A 2020 study by Anaconda found that 44% of data scientists cite poor data or data quality issues as the biggest barrier to success in AI and ML projects [2].

Common data issues that can derail ML models include:

• Missing values
• Duplicate records
• Inconsistent formats
• Incorrect or outdated values

Well-designed ETL processes catch and correct these issues early before they propagate downstream.

In addition, ML use cases often require complex data transformations like feature engineering, data normalization, and encoding categorical variables. ETL pipelines provide a framework to automate these transformations at scale.

As ML models are deployed to production, ETL pipelines also enable the flow of fresh data for continuous retraining and monitoring. Tools like TFX and MLflow can orchestrate ML-specific ETL workflows.

Data Governance and Quality

What They Are

Data governance defines the policies, procedures, and ownership around how an organization‘s data is acquired, stored, processed, and used. Key focus areas include:

• Metadata management
• Data quality
• Master data management
• Data security and privacy
• Data lifecycle management

Data quality is a critical component of governance that ensures data is accurate, complete, consistent, and timely. Common data quality dimensions and sample metrics include:

Dimension	Definition	Example Metric
Accuracy	Data reflects real-world correctly	Percent of records with correct values
Completeness	All required data is present	Percent of records with null values
Consistency	Data is in same format across systems	Percent of records with consistent formatting
Timeliness	Data is current and available on time	Avg time from data creation to availability

Popular data quality tools include Informatica Data Quality, Talend Data Quality, and Great Expectations.

Why They Matter for Data Science and Machine Learning

Poor data governance and quality can undermine trust in data and analytics. Inconsistent metric definitions, incomplete datasets, and stale data can all lead to flawed insights and decisions.

In machine learning, data quality issues can be even more devastating. ML models learn patterns from training data, so any biases or errors in that data can get amplified and perpetuated at scale. The classic "garbage in, garbage out" adage is especially true for ML.

Some real-world examples of data quality issues in ML include:

• Amazon‘s biased recruiting tool: Amazon trained an ML model to screen resumes but the training data contained gender bias, causing the model to favor male candidates [3].

• Apple Card‘s alleged gender discrimination: Some users reported Apple‘s credit limit algorithm gave much higher limits to men than women, even for couples with shared finances, raising concerns of gender bias [4].

• Zillow‘s home buying flop: Zillow‘s iBuying model severely overestimated home values, leading to a $500M loss and exit from the market. The CEO cited the algorithm‘s reliance on faulty data as a key issue [5].

To mitigate these risks, it‘s crucial to invest in robust data governance and implement rigorous data quality checks throughout your ML pipeline. Some best practices:

• Conduct exploratory data analysis (EDA) to uncover quality issues early
• Implement automated data quality tests with tools like Deequ or Great Expectations
• Use feature stores like Feast to serve only validated data to models
• Monitor production data and model performance for drift and degradation
• Document data lineage, provenance, and quality metrics

Data Regulations and Ethics

What They Are

The collection and use of personal data is increasingly regulated under laws like:

• GDPR (EU): Protects privacy of EU citizen data
• CCPA (California): Gives consumers rights to access and delete data
• HIPAA (US Healthcare): Mandates security standards for health data

Failing to comply with these regulations can result in hefty fines. GDPR violations can incur penalties up to €20M or 4% of annual global revenue. In 2021, Amazon was fined a record €746M under GDPR [6].

Beyond legal compliance, responsible data use is also a matter of ethics. Key ethical principles include:

• Transparency: Clearly communicate what data is collected and how it‘s used
• Consent: Obtain user consent and honor opt-outs
• Purpose Limitation: Only use data for specified purposes
• Fairness: Avoid biased or discriminatory models
• Accountability: Be able to explain and justify model decisions

Why They Matter for Data Science and Machine Learning

With the rise of big data and ML, data science teams have access to massive amounts of personal data and the ability to extract intimate insights from it. This comes with great responsibility.

Misuse of data can not only lead to regulatory penalties, but also erode user trust, damage brand reputation, and cause real-world harms. Examples include:

• Cambridge Analytica scandal: The political consulting firm harvested Facebook data from millions of users without consent to target them with psychographic profiles and influence elections [7].

• Racial bias in recidivism algorithms: An analysis found a widely used criminal risk assessment algorithm falsely labeled Black defendants as high-risk at twice the rate as White defendants [8].

• Clearview AI‘s facial recognition: The startup scraped billions of online photos without consent to power a facial recognition system used by law enforcement, raising major privacy concerns [9].

Data scientists must be proactive in considering the downstream impacts of their work and advocating for responsible practices. Some steps to take:

• Stay up-to-date on relevant data regulations
• Implement processes to fulfill data subject requests like deletion
• Assess models for fairness and mitigate bias
• Use privacy-preserving techniques like differential privacy and federated learning
• Establish ethical review processes for sensitive use cases
• Foster an organizational culture of ethics and accountability

Conclusion

We‘ve taken a deep dive into four critical data engineering concepts: data warehouses and lakes, ETL pipelines, data governance and quality, and data regulations and ethics.

Understanding these fundamentals allows data scientists to make sound decisions around data sourcing, processing, and use. It empowers you to build reliable, scalable, and responsible machine learning pipelines that drive real business impact.

But beyond the technical benefits, grasping these data engineering principles also enables data scientists to be better partners to data engineering teams and stronger stewards of data within their organizations.

As the famous saying goes, "With great power comes great responsibility." As data scientists, we wield the power to extract immense value from data, but also the responsibility to do so ethically and efficiently. By continually deepening our knowledge of data engineering best practices, we can ensure that our work is not only innovative, but also impactful and trusted.

References

[1] Databricks: The Data and AI Industry Survey
[2] Anaconda: 2020 State of Data Science Survey
[3] Reuters: Amazon scraps secret AI recruiting tool that showed bias against women
[4] The New York Times: Apple Card Investigated After Gender Discrimination Complaints
[5] CNN: Here‘s what went wrong with Zillow‘s home-buying algorithm
[6] CNBC: Amazon hit with $887 million fine by European privacy watchdog
[7] The New York Times: Cambridge Analytica and Facebook: The Scandal and the Fallout So Far
[8] ProPublica: Machine Bias
[9] The New York Times: The Secretive Company That Might End Privacy as We Know It