Data warehouses are usually full with dirty data. So obviously the question about how clear is our data warehouse is wrong. Data represent as kind of source in company life as any other assets. So we have to take care of them. Are you addressing dirty data? Which tools are available and what we have to know about them?
How we have to address dirty data? Let’s clarify the tools.
- Data quality
Data quality usually focus on data movements from database A to database B. This data quality tools can make changes. The process it is very comparable to an ETL tool, but the difference is that Data quality make fixes in dirty data like fix form of zip codes os phone numbers. Additionally there is reference table matching and validation. That means the tool maintains a list of queen records (like short name of key supplier companies) and handle that these recorsd often come in dirty variation (for example full name of company or part of it). It is useful if your company have hundreds of product names or customer relation data.
- Master data management
MDM is very well suited for maintain consitency and handle syncronisation problems. Tipical problem at companies that customer’s contact data are stored in several systems (like in CRM, marketing tool, warehouse management tool) and we dont really know which data are updated and usable. What does MDM do in this case? Maintains a data set which is fed from all those above mentioned systems. MDM only keep track of the customer’s attributes (= master data). If a department gets a change in phone number from the customer, that information will get fed into the MDM tool. MDM store rules how added information have to look like. MDM control also which data need update and on which platform. These rules are super critical and MDM has a machine to control this process.
Another big difference is MDM’s ability to maintain and change hierarchies. A centralized logic. This allows organizations to define relationships between attributes like product name and ingredients.
What is needed for both solution?
A qualified Business User an Analyst who can make decision when there are conflict in records or logics.
Who do need DQ and MDM?
Each organisations have dirty data so Data Qualitiy is a must have after a size.
Small organizations may not need an MDM solution, while large organizations with sizable master data must consider implementing it. The path to MDM begins with quality of data, and the starting point for any data quality process is a discovery of master data, profiling, and analysis.
What is the relation between this two tools?
Organizations often use Data Quality Tools to pump the data into the Master Data Management environment. This allows the MDM tool to focus its efforts on the synchronization and consistency problem and not the low level data formatting issues. Data Quality Tool handle them.
Who is responsible for these processes? Who has to implement them?
MDM and DQ project is not something the IT department of any organization can be expected to execute in isolation. It requires the full involvement of both IT and business personnel bringing Data Governance teams, Data Stewards and business teams in all phases of the data quality and MDM implementation.
As a most important step our consultant focus on DQ while implementing ABUX. Read case studies about implementation.
Sources:
www.intricity.com/data-governance/dq-vs-mdm/
www.youtube.com/watch?v=dspdToaROn8
www.dataintegration.ninja/relationship-between-data-quality-and-master-data-management/
- Server oldal
- SQL server
- Analysis Services
- ABUX alkalmazás server
- Integrated Services vagy PENTAHO alkalmazása az ERP és az AB közötti adatintegrációban
- Kliens oldal
- Microsoft Excel
- Platformfüggetlen adatmegjelenítő kliens (opcionális)
- MS Power BI
