Data Quality and Data Governance have been hot topics lately with the increasing use of AI. I've noticed a fair amount of discussion and concern even in the last 5-10 years, but I'm still seeing the same issues and concerns. So why is data quality still an ongoing issue?
Why Data Quality Is an Ongoing Issue
Let's face it: Data Quality issues are not something that will just go away. Every organization has bad data, and every data team has different ways of dealing with it. As soon as the newest Data Quality issue is resolved, the next will rear its ugly head.
There are two primary causes of Data Quality issues: Processes and People.
Processes change constantly. Teams change how they work, and in turn, a downstream impact affects how data is collected or used. Maybe a data pipeline silently failed, causing missing data. Maybe the API changed, causing values to be stored incorrectly (or missing). Perhaps someone introduced a bug in the code.
People are also a big contributor to Data Quality issues. Humans make mistakes, and typos happen. In some of my past roles, one of the largest contributors to incorrect data was simply due to manual processes. From people interacting with customers and manually adding data to marketing teams managing campaigns in spreadsheets, there will always be some form of human element and manual processes.
Could AI resolve these issues? Perhaps, but doubtful.
Data Quality Isn't a Technical Issue
From what I've seen and heard from others, data quality and governance hasn't made much progress because many leaders think they can just implement a tool and that's it. I've sat in on a few calls where the salespeople can make the best of us think it's magic.
I keep reading about the latest tools designed to deal with data quality issues or teams implementing AI to "fix" all the issues. The truth is, this is not something that can be solved by technical solutions alone.
Yes, tools have excellent capabilities and can help teams greatly. Yes, AI can help too.
But AI is only as good as the data it's trained on, and automated processes are only as good as the governance behind them.
The piece that seems to be missing for most teams is the strategy behind how to implement it at a scalable level that works for the entire organization. Data Quality and Data Governance done correctly requires the entire organization, not just the Data Team or some magic tool.
Poor Data Quality Is a Culture Issue
Poor Data Quality is a Culture Issue. Yes, I just stated the same thing twice—that's how important it is to understand. I won't even say that being a "data-driven" organization will help, because many say that but really aren't. It's more about how the people within the organization see data and what they do with the data.
You also can't blame it on "fast-paced environments." I love working in a fast-paced environment, but there's a difference between just getting it done at all costs versus rapid execution within guardrails.
In general, I like to sum these issues up into three key Cultural Issues: Missing or Ignored Processes, Lack of Executive Sponsorship, and "Not My Job" Mentality.
Missing or Ignored Processes
When you think of implementing Data Governance, you most likely associate related processes. This includes the entire lifecycle of data from creation to executive usage.
Most issues happen right where data is created—manual entry is the worst! Sometimes teams will implement a process and wait until after the project is live to loop in any data people (only to find out they aren't creating the right data).
Next, someone needs to have ownership of the data being created (or systems it's pulled from). Who, you ask? Well, if you can answer who, you're already ahead of the game, as this is another spot that falls short. Hint: 99% of the time, the "who" is NOT someone on the data team.
Another spot that falls short is documenting the data needed and how it needs to be validated. This includes the fields you pull from to the KPIs created from them. Having this known, documented, and visible to the entire organization is key.
Ignored processes are huge. I couldn't tell you how many times I've heard "Just get me the data and we can figure out the rest later." The end result? A one-off request to see what some random data source looks like is now in production…and somehow made it to executive dashboards.
I could go into more, but I'm sure at this point you get where some of the biggest pain points are as they relate to processes.
Lack of Executive Sponsorship
Getting the executives to agree to a project is not executive sponsorship. Getting actual Executive Sponsorship is no easy task—they are all busy and each tasked with their own projects.
Getting the project approved is the first step, and typically the easiest one. But will they help ensure alignment from their teams to help make sure it's a success? As stated before, Data Quality and Data Governance is NOT a data problem—it's an organization-wide problem. Without executives leading by example and supporting the project, it will struggle to succeed.
"Not My Job" Mentality
The last cultural challenge is focused on things getting ignored because others assume or expect someone else to do it. This shows up in two critical ways:
Ownership: Someone (outside of data) needs to take ownership of data. I hear so many excuses about not understanding data, but that's not what's needed. What's needed is answering: What is the business definition for the data? What values look wrong to them? Who notices when something is off?
Ignoring The Issue: People will find or notice issues but just assume someone else will report it or fix it. This happens a lot when people find issues with dashboards or when Analysts notice bad data when diving into a project. The mentality of "someone else will handle it" creates a culture where problems persist.
How Do We Fix It?
Strategy Like anything else, start with a strategy. Every organization is different and has different needs and cultures. Create the strategy and make sure to highlight key risks and challenges. The strategy should be focused on the overall outcome and not even consider tools until the general strategy is set.
Lead Up Make sure every Executive understands the importance of Data Governance and Data Quality. Work with them on what role they and their teams play. What do they need for their teams to be successful? If it's an organizational issue to resolve, then you can't ignore what other teams/departments need in order for the project to be a success.
Keep It Simple I've always been a huge fan of keeping it simple. I find it's the best method to help make sure you don't over-engineer things and make others' lives easier. This could be reducing manual steps, adding drop-downs, or links to a form that has fields auto-filled. The easier it is for others to work with, the better it will be adopted.
Data Quality Performance Indicators Don't just track how long people take to answer a call—make sure to include accuracy as well. By having it as a performance indicator that is tracked, you show the organization how important it is to be as accurate as possible.
Reward Champions Highlight those within the organization who are helping make a big difference toward Data Quality and Data Governance. It's a great way to celebrate others and help them feel like their efforts are worth it. It also helps encourage others to do their part.
The Bottom Line
Data quality is a team sport. It's time we stopped expecting the Data Team to play every position.
The organizations that get this right aren't the ones with the fanciest tools or the biggest data teams—they're the ones where everyone, from executives to frontline staff, understands that data quality is part of their job. Start small, pick one cultural shift from this list, and make it happen. Your data team will thank you.

Comments
Post a Comment