In machine learning and data engineering, many teams pursue perfect data. They aim for flawless pipelines, zero defects, and ideal schemas. In practice, however, this ideal is rarely achievable or even necessary. The true objective is to produce data that is reliable enough for its intended users, whether they are analysts, data scientists, executives, or downstream systems.
Over the years, I have observed many organizations invest heavily in data quality initiatives without clear metrics or priorities. In this post, I share a practical framework for measuring data quality, understanding its costs, and focusing on the elements that matter most.
Leading and Lagging Indicators of Data Quality
We can measure data quality in ways similar to software reliability.
Leading indicators help predict and prevent problems:
- Data debt reflects the complexity of the environment. Useful heuristics include the percentage of assets with usable documentation, the median number of dependencies per dataset, the frequency of backfills, and the average number of filters per query.
- Trustworthiness is a strong signal. Regular surveys can assess whether teams believe the data represents reality and will remain stable. High levels of dataset replication often indicate low trust and drive unnecessary costs.
- Ownership measures the proportion of upstream sources with clear responsibility. When critical dashboards depend on many tables but few have assigned owners, the risk of prolonged incidents rises significantly.
Lagging indicators reveal issues after they occur:
- Data downtime tracks periods when critical data becomes unusable. Teams should monitor incident duration, mean time to detect, mean time to resolve, and the resulting business impact.
- Violated expectations cover changes in schema, semantics, service-level agreements, or handling of sensitive information. Logging these centrally provides valuable visibility.
- Quarterly incident counts should focus only on events that affect real decisions, such as incorrect resource allocation or flawed model outputs.
Understanding the Human Impact
Poor data quality affects different roles in distinct ways.
Data engineers often bear the heaviest load. They become the default team for every issue, spending significant time on root-cause analysis while receiving little recognition when systems run smoothly.
Data scientists encounter models that degrade silently. A prediction that appears successful may later prove inaccurate by a large margin.
Analysts, who work most closely with business logic, see their understanding disrupted by unexpected changes. They frequently face blame for problems outside their control.
Software engineers may introduce upstream modifications that cascade downstream, leading to friction across teams.
Business users have the least visibility and control. They may act on misleading numbers and commit resources to nonexistent problems.
The True Cost of Poor Data Quality
The expense extends far beyond storage or compute. It includes duplicated effort, reduced development velocity, eroded trust, and misguided business decisions. Data debt accumulates in a manner similar to technical debt, but its effects often appear more quickly in organizational performance.
The solution lies in pragmatic practices rather than perfection. Establish clear ownership for upstream sources. Define explicit expectations around schema, semantics, and reliability. Measure outcomes based on their influence on actual decisions.
Data quality represents a strategic trade-off. We must balance robustness with return on investment, always anchoring efforts in business value.
This perspective also explains why data contracts have become essential in modern data architectures, a topic I plan to explore in future writing.
What data quality challenges have you encountered in your work? I welcome your thoughts and experiences.
Newsletter
Keep reading.
One email when something new lands. No spam.