"BI and Data Quality: challenges of secondary business processes"

Why is it that Business Intelligence professionals always seem to be whining about poor data quality? Business Intelligence (BI) projects are fraught with data quality issues. There are some structural reasons for this, which have a lot to do with the role BI plays in organizations. On top of that, in many organizations, a BI project or data warehouse may well be the place where data streams from independent (legacy) source systems are being confronted for the first time. In many cases the “surprising” data quality issues that arise, “belong” to the project team, despite the fact that these problems have existed long before the BI project was ever envisioned.

I find it ironic that data quality largely remains a “motherhood and apple pie” issue. Everybody agrees that it’s a problem, and everybody agrees that something should be done about it. However, when it comes the time to find volunteers who want to actually do something about it, everybody seems busy…

To clarify the context of data quality issues, I find it useful to distinguish between primary and secondary processes in an organization. What I refer to as primary processes, are business processes that directly create value for the customer. Secondary processes support primary ones. Examples of primary processes are Customer Service, Sales, Manufacturing, etc. Examples of secondary processes are Controlling, Human Resources, Corporate Strategy, and of course also Business Intelligence. On average, you can typically expect (significantly) lower data quality in secondary processes. Why might that be?

Download full article