10 Common Bad Data Cases and Their Solutions

Bad Data: The Silent Killer of Business Success

Introduction
In today‘s data-driven business landscape, information is the most valuable asset an organization has. Companies rely on data to make critical decisions, drive innovation, and gain a competitive edge. However, the adage "garbage in, garbage out" has never been more relevant. Poor quality, inaccurate, or incomplete data – commonly referred to as "bad data" – is pervasive across industries and can have disastrous consequences for businesses.

A 2022 Gartner study found that the average financial impact of poor data quality on organizations is $9.7 million per year. Another survey by Experian revealed that 85% of organizations believe their revenue is affected by bad data in some way. Despite increasing awareness of the problem, the volumes of data being collected and the proliferation of data sources has made ensuring data quality more challenging than ever.

In this comprehensive guide, we‘ll take a deep dive into the world of bad data – what it is, where it comes from, why it matters so much, and most importantly, what your organization can do to combat it. By the end, you‘ll have a thorough understanding of data quality issues and a toolkit of best practices and strategies to make bad data a thing of the past.

What is Bad Data?
First, let‘s define what we mean by "bad data." Put simply, it is data that is inaccurate, incomplete, inconsistent, duplicated, or otherwise flawed in a way that makes it unreliable or unfit for its intended purpose. Some common characteristics of bad data include:

• Missing or null values where data should exist
• Duplicate records of the same entity
• Inconsistent formats or data types (e.g. phone numbers as text)
• Incorrect values that don‘t match the real world
• Outdated or stale information
• Violation of business rules or constraints
• Records with junk or meaningless values

The presence of any of these issues compromises the integrity of the dataset and leads to a host of downstream problems when that data is used for reporting, analysis, or input into other systems. A classic example is having an incorrect mailing address for a customer, which would result in invoices or marketing materials being undeliverable.

Bad data can exist at any stage of the data pipeline, from initial collection and entry through transformation, integration, and storage. It‘s a complex, multifaceted issue that requires a strategic approach to fully resolve.

The High Cost of Bad Data
The impacts of bad data extend far beyond just the inconvenience of having to clean it up. Poor data quality is tremendously expensive for organizations of all sizes and industries. Some of the costs and consequences include:

• Lost revenue and lost customers due to bad contact information
• Wasted marketing spend on inaccurate segmentation and targeting
• Fines for regulatory non-compliance or reporting errors
• Incorrect business decisions based on flawed data and analysis
• Productivity losses from time spent correcting errors
• Increased costs of data storage and processing
• Reputational damage and erosion of trust with customers/partners

IBM estimated that in the US alone, businesses lose $3.1 trillion annually to bad data. On an individual company basis, research by IBM and Gartner have found:

• Bad data costs businesses an average of 20-35% of their revenue
• Poor data can reduce a company‘s revenue by 6% due to inefficiency
• Every 100 employees wastes 8 hours per week on average dealing with data quality issues

Sources:

• Redman, Thomas C. "Bad Data Costs the U.S. $3 Trillion Per Year." Harvard Business Review, September 2016
• Hunter, David, and Larry Rosenberg. "Measuring the Business Impact of Poor Data Quality." Gartner, June 2017

The message is clear – bad data is a massive drain on resources and has tangible financial impacts no organization can afford to ignore. Proactively investing in data quality is critical to mitigating these costs.

Where Does Bad Data Come From?
To effectively solve data quality issues, you first need to understand the root causes. Bad data typically originates from one of four main sources:

1. Human Errors
   Simple mistakes made by the people entering, manipulating or querying the data is one of the most common causes of bad data. This could include things like transposing numbers, selecting the wrong value from a drop-down menu, or forgetting to update a field. Lack of training, confusing data entry UIs, and simple carelessness can all contribute to human error.

2. System Issues
   Bugs, glitches, misconfigurations, or lack of proper controls in the software and systems that collect and process data can introduce errors. This is especially problematic when data is being transformed or moved between systems, such as during an ETL (extract, transform, load) process. Incompatible file formats, inconsistent field names, or applying the wrong business logic can all mangle data.

3. Data Decay
   Data doesn‘t stay accurate forever. Over 3% of customer contact data becomes obsolete every month due to events like customers moving or changing their name. Product information, addresses, status fields, and other data that changes over time must be regularly updated, or it becomes stale and no longer reflects reality.

4. Lack of Data Governance
   The absence of clear policies, procedures, and accountability around how data is handled invites disorder. Without data quality standards, metadata management, defined data ownership and stewardship, and ongoing auditing, an organization has no mechanism to ensure data is complete, consistent, and accurate across systems.

Spotting these issues requires a keen eye and a combination of technical tools and human review. Some common red flags that indicate potential data quality problems include:

• Numbers that seem unusually high or low
• Text fields with inconsistent casing and formats
• Records with too many missing or blank values
• Values that violate known constraints (e.g. zip codes with too many digits)
• Duplicate records for the same person or entity
• Totals/subtotals that don‘t seem to add up
• Outliers and anomalies in distributions or patterns

Best Practices for Ensuring Data Quality
With the disastrous impacts of bad data, it‘s clear that ensuring high-quality data assets is a business imperative. But how can organizations actually achieve that? Here are some key best practices:

1. Establish data governance
   As mentioned before, lack of governance is a major cause of bad data. Instituting a robust data governance program is job one. That means documenting clear policies around data handling, assigning data stewards and owners, standardizing metadata, and embedding data quality reviews and remediation into operations.

2. Validate data at the point of entry
   Preventing errors from being introduced in the first place is far more efficient than trying to clean them up later. Implement strong validation, such as checking formats and applying business rules, as soon as data is collected or entered into your systems.

3. Automate data profiling and monitoring
   Regularly profiling your data to check for anomalies, inconsistencies, and integrity issues is key to proactive data quality management. Rule-based and ML-powered data profiling tools can automatically scan datasets and alert you to potential problems.

4. Create a data quality scorecard
   Develop metrics and KPIs to quantify and track data quality over time. Measures like number of errors, percent of fields with missing data, and time since last update will help gauge the health of your data assets. Report on this regularly to keep data quality top-of-mind.

5. Train and educate your team
   Anyone interacting with organizational data needs a strong grasp of data best practices and how their actions impact downstream quality. Invest in data literacy training for employees to raise awareness and build good data habits.

6. Perform regular data cleansing
   Even with solid prevention measures in place, some bad data is still inevitable given the scale and complexity of today‘s data ecosystem. Institute a regular data cleansing and enrichment process to programmatically identify and fix quality issues in your data stores.

With a focus on prevention, early detection, and continuous improvement driven by these best practices, companies can effectively combat their data quality challenges.

Cleaning Up Bad Data
Once you‘ve identified bad records in a dataset, you‘ll need to decide how to handle them. Data cleansing is the process of programmatically updating, correcting or removing flawed data to improve its quality. Some common cleansing techniques include:

• Normalizing formats and fixing spelling/grammar
• Parsing values into separate fields
• Removing duplicates or merging/consolidating records
• Enriching records with data from trusted external sources
• Applying business rules and transformations
• Flagging or removing records that can‘t be fixed

Data cleansing can be partially automated with scripts and algorithms, but often requires some manual review and remediation to get records to a high standard of quality. A typical data cleansing workflow involves:

1. Auditing the dataset and profiling the records to identify quality issues

2. Determining which issues can be fixed programmatically vs manually

3. Writing scripts/queries (e.g. SQL or Python) to automate the cleansing

4. Manually reviewing and fixing remaining low quality records

5. Validating completeness and accuracy of the cleansed dataset

6. Refreshing any downstream systems or reports with the clean data

Performing a deep cleanse of a large, messy dataset is a significant undertaking, but the improvements in accuracy and usability pay major dividends.

The Future of Data Quality
As the world becomes increasingly digitized, the challenges around maintaining high-quality data will only intensify. Some key trends and developments to watch include:

• Explosive growth in volume, velocity, and variety of big data
• More data being generated by IoT sensors, mobile devices, etc.
• Automated decision-making and predictive models that rely heavily on data
• Stricter data privacy regulations like GDPR and CCPA
• Blockchain and other new technologies to manage data sharing and lineage

To stay ahead of the curve, forward-thinking organizations are getting even more serious about data quality and governance. Building a strong foundation of clean, reliable data is becoming a core strategic priority. In particular, companies are keen to shift from reactive data cleansing to proactive data quality engineering and monitoring.

They‘re also doubling down on data literacy, making fluency in data skills a core competency for employees rather than the domain of a small group of experts. Embedding data validation and hygiene into every process is becoming the norm.

With the right mix of people, process, and technology aligned around ensuring quality data, businesses can accelerate their data maturity and unlock the full value of their data assets.

Conclusion
Bad data may be pervasive, but it doesn‘t have to be permanent. Armed with knowledge of the key issues, a solid data governance foundation, and a commitment to making data quality a priority, any organization can make major strides in wrangling their information chaos.

Data is the lifeblood of the digital economy – by taking control of data quality, you can position your company to thrive and make confident, insight-driven decisions for years to come. Don‘t let bad data kill your business. Implement these best practices and make trusted, reliable data the heart of your success.