↓

Matière supérieure
Introduction
Le Qualité des données Gestion fonction defines the goals, approaches and plans of action that ensure data content is of sufficient quality to support defined business and strategic objectives of the organization. The fonction should be developed in alignment with business objectives, measured against defined la qualité des données (DQ) dimensions and based on an analysis of the current state of DQ. Qualité des données Management is a series of processes across the full data supply chain to ensure that the data provisioned meets the needs of its intended consumers.
DQ requires an understanding of how data is sourced, defined, transformed, provisioned and consumed. DQ is not a processus itself but describes the degree in which data is fit-for-purpose for a given business processus or operation.
Définition
Le Qualité des données Gestion de la qualité (DQM) est un ensemble de capacités permettant de définir profilage des données, Ces capacités permettent à l'organisation d'exécuter des processus dans l'ensemble de l'environnement de contrôle des données, en veillant à ce que les données soient adaptées à l'usage auquel elles sont destinées. Ces capacités permettent à l'organisation d'exécuter des processus dans l'ensemble de l'environnement de contrôle des données, en veillant à ce que les données soient adaptées à l'usage auquel elles sont destinées.
Champ d'application
- Mise en place d'un DQM fonction au sein de l'Office de gestion des données (ODM).
- Travailler avec la gestion des données (DM) Bureau de gestion des programmes (PMO) pour concevoir et mettre en œuvre des processus et des outils durables pour la gestion de la qualité des produits.
- Exécuter des processus DQM sur des données critiques pour l'entreprise. Les processus DQM comprennent profilage & classement, mesure, gestion des défauts, résolution des causes profondes, remédiation.
- Établir des mesures de QD et des routines d'établissement de rapports.
- Veiller à ce que la gouvernance du DQM soit intégrée dans la gouvernance des données (DG).
Proposition de valeur
Organizations that build, formalize and assign DQ responsibilities into daily routine and methodology achieve a sustainable organization-wide data culture.
Organizations that effectively implement Qualité des données Management and achieve the appropriate level of DQ across the data ecosystem get a return on investment from several areas:
- Une meilleure gestion des risques
- Analyse améliorée
- Amélioration du service à la clientèle et de l'innovation en matière de produits
- Amélioration de l'efficacité opérationnelle
Vue d'ensemble
DQ is a broad conceptual terme that needs to be understood in the context of how data is intended to be used. Perfect data is not always a viable objective. The quality of the data needs to be defined in terms that are relevant to the data consumers to ensure that it is fit for its intended purpose. The overall goal of DM is to ensure that data consumers have confidence in the data they receive. These consumers are using this data to support their business functions. For them to make accurate decisions the data must reflect the facts the data is designed to represent without the need for reconciliation or manual transformation.
The organization needs to develop a DQM strategy and establish the overall plans for managing the integrity and relevance of its data. One of the essential objectives is to create a shared culture of DQ stemming from executive management and integrated throughout the operations of the organization. To achieve this cultural shift, the organization must agree on both requirements and the measurement of DQ that can be applied across multiple business functions and applications. This will enable business sponsors, data producers, data consumers and technology stakeholders to link DQ management processes with objectives.
DQ can be segmented into dimensions:
- Précision: the relationship of the content with original intent
- Complétude: the availability of required data attributes
- Couverture: the availability of required data records
- Conformité: alignment of data content with required standards
- Cohérence: how well the data complies with the required formats/definitions
- Respect des délais: the currency of content representation as well as whether the data is available/can be used when needed
- Unicité: the degree that no record or attribute is recorded more than once
L'identification et la hiérarchisation des dimensions de la qualité des données foster effective communication about DQ expectations and are an essential prerequisite of the DM initiative.
Creating a profile of the current state of DQ is an important aspect of the overall DQM fonction. A new profile should be created periodically when data is transformed. The goal is to assess patterns in the data as well as to identify anomalies and commonalities as a baseline of what is currently stored in databases and how actual values may differ from expected values. Once the data profile is established, the organization needs to evaluate the data against the quality tolerances and thresholds defined by the DQ requirements. The evaluation also examines business requirements to validate that the data is fit-for-purpose.
The purpose of this evaluation processus is to measure the quality of the most important business attributes of the existing data and to determine what content needs remediation. A responsibility of the producteur de données et consommateur de données is to identify the data that is critical to the consommateur de données’s business processus. Prioritizing the data based on criticality then informs the DQM fonction which attributes require a heightened level of control and quality review. The designation of criticality requires that the highest level of précision and DQ treatment is applied. The assessment processus identifies the data that needs to be cleansed to meet consommateur de données requirements. Data cleansing should be performed against a predefined set of business rules to identify defects that can be linked to operational processes.
Data cleansing should be performed as close to the point of capture as possible. There should be clear accountability and a defined strategy for data cleansing to ensure that cleansing rules are known and to avoid duplicate cleansing processes at multiple points in the cycle de vie des données. The overall goal is to clean data once at the point of data capture based on verifiable documentation and business rules as well as to fix the processes that allowed defective data into the system at the root cause. Data corrections must be communicated to, and aligned with, all downstream repositories and upstream systems. It is important to have a consistent and documented processus for issue escalation and change verification for both data producers and data vendors.
It is also important to ensure that data meets quality standards throughout the lifecycle so that it can be integrated into operational data stores. This aspect of the DQ management processus is about the identification of data that is missing, determination of data that needs to be enriched and the validation of data against internal standards to prevent data errors before data is propagated into production environments.
For DQ to be sustained, a strong governance structure with the highest level of organizational support from senior executive management must be in place. This supports the DQM activities and ensures compliance to DQ processes. DQ processes need to be documented, operationalized and routinely validated via DM reviews and formal audit processes.
DQ cannot be achieved through central control. Organization-wide DQ requires the commitment and participation of a broad set of stakeholders. DQ is the result of a series of business processes creating a data supply chain. Therefore, stakeholders, along that chain must be in place, authorized and held responsible for the quality of data as it flows through their respective areas. DQ requires coordinated organizational support. DQM processes and objectives must be part of the operational culture of an organization for it to be sustained and successful.
Processes, Tools, & Constructs
- Élément d'entreprise/Élément de données Construire
- Élément d'entreprise, Business-Based Rules Construire
- Élément de données critique Criteria and Measurement Construire
- Qualité des données Rules Construire
- Profilage des données Construire
- Quality Metrics and Dashboards
- Defect Management Construire
- Root Cause Analysis Construire
- Capability Optimization
- Matrice RACI
- Processus Designs and End-to-End Processus Integration
- Procedures Guide
- Processus Performance Measurement
Questions fondamentales
- Is it understood that poor quality data is an indication of a broken business processus ou la technologie ?
- Is it understood that instituting a DQ system is a cultural shift that touches all aspects of business, operations and technology processes?
- Is the required training in place to sustain the DQM fonction?
- Are the necessary people and funding resources earmarked to implement and operate the DQM fonction?
- Are the necessary resources in place to provide organization-wide training to support a sustainable, DQ cultural change?
5.1 Data Quality Management (DQM) is Established
The DQM fonction strategy and approach must be defined and approved by stakeholders. Roles and responsibilities across the stakeholders must be established with operational processes in place and auditable.
5.1.1 The DQM strategy and approach are defined and adopted
Description
The strategy and approach must be defined for the DQM fonction and reflect the related vision and objectives of the Data Management Strategy (DMS). Once established, it must be formally empowered by senior management and its role communicated to all stakeholders.
Objectifs
- Formally establish the DQM strategy and approach within the organization.
- Get approval of the DQM strategy and approach from stakeholders.
- Assurer l'alignement des partie prenante plans and roadmaps with the DQM strategy and approach.
- Obtain executive management support for the DQM strategy.
- Communicate the role of the DQM fonction across the organization through formal channels.
- Operate the DQM fonction collaboratively with DM initiative stakeholders.
- Secure authority to enforce DQM compliance through politique et documenté procédure.
Conseil
The DQM strategy and approach encompasses the what, how and who of DQ. It needs to address ce que scope of data is to be scrutinized and reviewed; comment the DQ assessments will be performed with metrics defined; and who will be responsible with defined roles. DQM needs to be closely aligned with the organization’s business objectives to ensure that the most important data is properly maintained and monitored. DQM involves cultural change. It is critical that a documented DQM strategy and approach is socialized with business, data and technology stakeholders to ensure awareness, support and commitment.
The rapidly evolving focus on data ethics is introducing new requirements for the DQM fonction. These requirements include an ethical review as part of determining that data produced is fit-for-purpose. Additionally, DQM is one of the areas where the use of Machine Learning (ML) and Artificial Intelligence (AI) may assist in the processes used to achieve quality data. These requirements and opportunities should be evaluated in the strategy and approach of the DQM fonction.
Alignment of the DQM strategy and roadmap to the DMS vision and objectives is achieved by agreement between the operating level responsable des données and the individual responsible for delivering the DG fonction. The operating level responsable des données est responsable de l'établissement des priorités pour chacune des exigences des composantes du cadre.
Questions
- Has the DQM fonction a été formellement établie ?
- Is there a DQM strategy and approach in place?
- Is the DQM strategy and roadmap aligned to the DMS?
- Have innovative technologies such as ML and AI been considered as part of the DQM processus et l'infrastructure ?
- Has the review of data ethics been included in the DQM strategy and approach?
- Has the DQM fonction been formally communicated to business, technology, operations, finance and risk stakeholders?
- Comment la direction générale a-t-elle manifesté son soutien ?
- Has authority been granted to the DQM fonction mettre en œuvre et faire respecter les meilleures pratiques par le biais politique et des normes ?
- L'autorité a-t-elle été communiquée aux parties prenantes ?
- Existe-t-il un partenariat fonctionnel avec l'audit interne ?
Artéfacts
- The DQM plan
- Description of the roles and responsibilities of the DQM fonction
- Communication d'un soutien spécifique de la part de la direction générale au moyen de listes de distribution
- Policies and procedures associated with executing and enforcing DQM
- Bi-directional engagement with stakeholders on the DQM fonction authority
Notation
Non initié
No formal DQM strategy exists.
Conceptuel
No formal DQM strategy exists, but the need is recognized and the development is being discussed.
Développement
The formal DQM strategy is being developed.
Défini
The formal DQM strategy is defined and has been validated by the directly involved stakeholders.
Atteint
The formal DQM strategy is established and understood across the organization and is being followed by the stakeholders.
Améliorée
The formal DQM strategy is established as part of business-as-usual practice with a continuous improvement routine.
The strategy and approach are reviewed and updated at least annually.
5.1.2 The DQM stakeholder roles and responsibilities are defined and implemented
Description
DQM requires a network of data stewards and subject matter experts to ensure data is properly captured, processed and delivered. Accountable parties must be identified and the roles and responsibilities must be clearly communicated.
Objectifs
- Define and communicate the roles and responsibilities of the DQM fonction.
- Fund and staff the DQM fonction.
- Garantir et faire respecter l'alignement des activités et des projets sur les objectifs de l'UE. politique and standards through the authority of the DQM fonction.
- Hold individuals accountable for the DQM performance via annual reviews and compensation considerations.
Conseil
DQM involves numerous stakeholders who are responsible for data requirement capture, profilage des données, remédiation, définitions, métadonnées, transformation, root cause analysis, entitlement control and coordination across the full data ecosystem. These efforts involve the assignment and empowerment of owners, stewards, curators and custodians. These accountable parties need to be at the right levels of seniority as well as understand all the internal processes associated with DQM.
With the addition of a data ethics review in the DQM processus, subject matter expertise will be required either through the addition of experts or appropriate training of the data stewards. Similarly, to the extent that ML and AI are used to support the DQM processus, En outre, des compétences supplémentaires devront être ajoutées ou développées au sein des parties prenantes.
Questions
- Has the DQM fonction a été établie ?
- Is the DQM fonction des effectifs et un financement adéquats ?
- Does the DQM fonction disposent-ils de l'autorité nécessaire pour être efficaces ?
- Have the roles and responsibilities of the DQM fonction a été définie, documentée et socialisée ?
- Have the skills for data ethics review and execution of ML and AI tools been added or developed within the stakeholders?
- Have milestones and metrics associated with the DQM fonction a été établie ?
Artéfacts
- Preuve de partie prenante identification
- Matrice RACI ou toute autre preuve de l'obligation de rendre des comptes
- Description of the roles and responsibilities of the DQM fonction
- Staff assignments and qualifications
- Preuve de la responsabilité liée à l'évaluation des performances et à la rémunération
- Analyse des lacunes dans les compétences nécessaires et existantes
- Liste des parties prenantes et preuves d'une communication bidirectionnelle
Notation
Non initié
No formal DQM roles & responsibilities exist.
Conceptuel
No formal DQM roles & responsibilities exist, but the need is recognized and the development is being discussed.
Développement
The formal DQM roles & responsibilities are being developed.
Défini
The DQM roles & responsibilities are defined and have been validated by the directly involved stakeholders.
Atteint
The DQM roles & responsibilities are established and are recognized and used by stakeholders.
Améliorée
The DQM roles & responsibilities are established as part of business-as-usual practice with a continuous improvement routine.
The roles & responsibilities are reviewed and updated at least annually.
5.1.3 The DQM processes are defined and operational
Description
Formal processes must be established for the activities of the DQM fonction. These processes align with the DM politique et les normes de l'organisation et comprennent des procédures, des outils et des routines. Les routines sont nécessaires pour les opérations en régime permanent.
Objectifs
- Establish formal DQM processes in alignment with the DM politique et des normes.
- Integrate the DQM processes into the overall end-to-end processes of the DM initiative.
- Identify, schedule and maintain DCE routines, meetings and working sessions required for operational support.
Conseil
The DQM subject matter experts should work with the business processus design and optimization service within the Data Management Program (DMP) team. Together they will create and monitor the implementation of the DQM processes in alignment to the end-to-end processus across the full DM initiative.
The DQM processus design should include the requirements for ethical review as part of determining the data is fit-for-purpose. The design should also incorporate ML and AI into the processus if included in the DQM strategy and approach.
Questions
- Des processus formels ont-ils été définis et mis en œuvre ?
- Les procédures, outils et routines sont-ils en place pour la mise en œuvre des processus ?
- Have innovative technologies such as AI and ML been considered as part of the DQM processus et l'infrastructure ?
- Has the review of data ethics been included in the DQM strategy and approach?
- Are DQM activities part of the normal operational routine of stakeholders?
- Are there standing meetings, planning sessions and regular communications about data initiatives?
Artéfacts
- Processus des artefacts de conception, procédure guides et routines publiés
- Processus rapports sur les mesures de performance
- Meeting minutes, status reports and DMP announcements
Notation
Non initié
No formal DQM operational processes exist.
Conceptuel
No formal DQM operational processes exist, but the need is recognized and the development is being discussed.
Développement
The DQM operational processes are being developed.
Défini
The DQM operational processes are defined and have been validated by the directly involved stakeholders.
Atteint
The DQM operational processes are established and are recognized and used by stakeholders.
Améliorée
The DQM operational processes are established as part of business-as-usual practice with a continuous improvement routine.
5.1.4 The DQM processes are auditable
Description
DQ auditing must occur on three levels:
- Quality Assurance (QA): the accountable business performs self-assessments based on defined DQ thresholds, processes, and objectives.
- Quality Control (QC): the DM initiative performs a facilitated audit of the accountable business’ DQ and processes and is empowered to force the business to remediate any gaps found to ensure adherence to DQ thresholds and standards.
- Internal Audit: the accountable business’ DQ and processes are subject to audits. Failure to satisfy this review may result in formal escalated audit findings written against the business.
Objectifs
- Data Stewards have performed self-assessment of the accountable DQ and processes (QA).
- The DM initiative has performed facilitated assessments of the accountable business DQ and processes (QC).
- The DM initiative is empowered to force business teams to remediate gaps found in the operational DQ processes.
- Internal Audit performs routine examinations of the accountable business DQ and processes.
- Formal audit Issues are generated if operational gaps are uncovered.
Conseil
DQ processes, validation, root cause analysis, remediation, etc., should be routinely audited. Audit occurs on three levels. First, self-attestation – where the stakeholders evaluate and assert they are following the la qualité des données rules. Second, through the Office of Data Management (ODM) – where the DM initiative works with stakeholders to validate compliance. Third, through internal review where Internal Audit has formally validated that processes are being followed.
Questions
- What are the mechanisms to ensure validation, root cause analysis, and remediation?
- Is Internal Audit involved in DQM?
Artéfacts
- Preuve d'auto-attestation et entreprise ODM review
- Preuve de l'engagement et de l'examen de l'audit interne
Notation
Non initié
There is no oversight of the DQM processes.
Conceptuel
DQM oversight strategies and approaches are being discussed.
Développement
Three levels of la qualité des données review are being defined.
Défini
Three levels of la qualité des données review have been identified and are being shared with stakeholders for review and approval.
Atteint
Three levels of la qualité des données review have been implemented.
Améliorée
Le processus to ensure the auditability of DQM processes has a routine in place to identify opportunities for continuous improvement.
5.2 Les données sont profilées et mesurées
Profilage and measuring the data includes: 1) prioritizing the data in scope based on criticality and matérialité; 2) defining and testing la qualité des données rules based on business rules; and 3) measuring that the data is fit-for-purpose.
5.2.1 Data has been identified and prioritized
Description
Les données concernées, telles que définies par les objectifs de l'entreprise, doivent être classées par ordre de priorité en fonction de leur criticité et de leur importance pour l'entreprise. matérialité à la consommateur de données business processus.
Objectifs
- Définir un processus for prioritizing data.
- Identifier le champ d'application de la personne concernée to DQM, both current and historical.
- Prioritize the scope of data in alignment with the DMS and business priorities.
Conseil
An organization may establish data prioritization tiers. The DM politique et les normes devraient définir les niveaux de contrôle des données à appliquer à chaque niveau de priorité. Le niveau le plus élevé est un élément de données critique (CDE). Designated CDEs receive the highest-level of control to ensure the quality of these attributes is maintained. CDE La désignation est contrôlée processus to achieve agreement between the producteur de données et consommateur de données.
Questions
- Est-ce que le processus pour classer les données par ordre de priorité a été défini ?
- Le champ d'application de la personne concernée to DQM been identified, prioritized and verified?
Artéfacts
- Priorités domaine des données inventories
- Priorités CDE inventory
- Bi-directional communication about the inventories
Notation
Non initié
Data subject to DQM has not been identified or prioritized.
Conceptuel
Le champ d'application de la personne concernée to DQM is being discussed.
The concept of CDEs is being debated.
Développement
Le champ d'application de la personne concernée to DQM is being identified and shared with stakeholders.
CDEs are being defined.
Défini
Le champ d'application de la personne concernée to DQM is prioritized and aligned with both strategy and business priorities.
CDEs are verified.
Atteint
Le champ d'application de la personne concernée to DQM is approved.
CDEs are designated and actively maintained.
Améliorée
Le processus pour identifier et classer par ordre de priorité toutes les données pertinentes a mis en place une routine pour identifier les possibilités d'amélioration continue.
5.2.2 Data quality rules are defined and tested
Description
Qualité des données rules based on business rules must be defined and tested to confidently validate the data is fit-for-use.
Objectifs
- Définir un processus pour le développement de la qualité des données règles.
- Define business rules which can be interpreted into la qualité des données et utilisées pour mesurer la qualité des données.
- Définir un processus pour l'essai de la qualité des données règles.
- Establish an environment and tool set for the running and testing of rules.
- Socialize DQ rules and test output to stakeholders.
Conseil
Les règles de gestion constituent la base du développement la qualité des données rules necessary to profile the quality of the data. The dimensions de la qualité des données establish a range of potential rules that may be needed to determine DQ. A critical part of defining quality rules is to test the rule outcome. Testing is an iterative activity during the design of an individual rule. Testing and re-testing with each rule refinement is an essential activity for identification of the range of rules across the dimensions de la qualité des données. Ces dimensions sont nécessaires pour la la qualité des données rule-set to accurately measure the DQ.
Qualité des données des règles devraient être élaborées pour tester les différentes les dimensions de la qualité. Not every élément de données can be tested for each dimension. There is an art and a science to writing quality rules. The rules and the range of rules will evolve over time. As new quality defects surface they will guide the design of new rules to detect those issues in the future. A mature règle de qualité des données Le set est un atout convoité par une organisation.
The ability to test la qualité des données rules requires a testing environment. This may be a data sandbox where the data stewards can play with the data and experiment with various rules and their output. Additionally, the proper technical infrastructure to write, store, run and analyze rules is needed.
The skill set required to test rules may include business processus subject matter expertise, DQ management expertise and technical infrastructure and coding expertise. Make sure you assess the skills and available resources carefully as this range of skill may not be available through a single individual.
Questions
- Existe-t-il un processus for the design and testing of la qualité des données règles ?
- Les la qualité des données sur la base des règles de gestion définies ?
- Existe-t-il un bac à sable pour les données et des outils appropriés pour l'exploitation des données ? règle de qualité des données tests ?
- Les parties prenantes ont-elles validé le fait que l'éventail des dimensions de la qualité des données applied in the rule set is adequate to determine the DQ?
Artéfacts
- Processus des artefacts de conception, procédure guides et routines publiés
- Documenté dimensions de la qualité des données
- Criteria used to evaluate DQ
- Qualité des données référentiel de règles
- Qualité des données règles enregistrées comme métadonnées
- Rapports sur les résultats des tests et tableaux de bord
- Testing tools
- Testing environment
- Liste des parties prenantes et preuves d'une communication bidirectionnelle
Notation
Non initié
No DQ rules or testing capability exist.
Conceptuel
No DQ rules or testing capability exist, but the need is recognized and the development is being discussed.
Développement
DQ rules and testing capability are being developed.
Défini
DQ rules and testing capability have been defined and validated by directly involved stakeholders.
Atteint
DQ rules and testing capability are established and are recognized and used by stakeholders.
Améliorée
DQ rules and testing capability are established as part of business-as-usual practice with a continuous improvement routine.
5.2.3 The data is profiled, analyzed and graded
Description
Les données dans le champ de l'enquête doivent être profilées afin de déterminer l'ensemble du spectre des dimensions de la qualité des données (i.e., précision, exhaustivité, couverture, conformité, cohérence, respect des délais, unicité). This analysis must include both a row-based analysis examining the précision of the record and a column-based, statistical analysis. Métadonnées doit également être examinée pour s'assurer que la description et l'utilisation prévue des données sont correctement définies.
Objectifs
- Définir un processus pour profilage, analyzing and grading data.
- Profile and statistically analyze in-scope data.
- Review the métadonnées and perform gap analysis.
- Measure, monitor and grade in-scope data.
- Capture DQ metrics on a routine basis.
- Report DQ metrics to business, data and technology stakeholders.
Conseil
The DQM fonction est d'établir que les données sont adaptées à l'usage qui en est fait et qu'elles sont dignes de confiance. Profilage des données crée une référence de qualité pour l'organisation. Les preuves profilage des données Les données doivent être évaluées en fonction de critères d'adéquation à l'usage prévu et de critères d'évaluation de la qualité. Les données doivent être évaluées à la fois en fonction de critères d'adéquation à l'usage prévu et de l'objectif de l'audit interne. dimensions de la qualité des données. DQ business rules need to be defined and memorialized. A statistical and columnar analysis should be included to ensure that data is reasonable. Certain domaine des données types, such as time-series data, need to be evaluated against additional criteria like gaps, spikes and abnormalities.
The primary stakeholders involved in this processus sont les producteur de données et consommateur de données. Ultimately quality is defined by the business processus les exigences de la consommateur de données et doivent être formellement approuvées par le producteur de données. Creating a standard et automatisé processus for routinely executing the quality metrics and reporting the results is critical to meet the time constraints of the data supply chain.
Metrics are used to track DQ and drive data remediation efforts. Control points along the data supply chain capture DQ metrics that are used to produce DQ dashboards. The requirements of the consommateur de données are used to establish quality thresholds for the data. These thresholds permit the grading of the data as to the defined levels of acceptable DQ based on the minimal requirements of specific consommateur de données.
Le profilage des données, analyser et noter processus should include a periodic review of the ethical use and outcome of the data as part of determining fit-for-purpose of the data. Particular attention should be paid to the use of proxy values in a ensemble de données.
A mechanism for executing DQ rules and generating outcome reports are required to support the profilage des données, analyser et noter processus. The use of AI and ML may assist in the processus.
Questions
- Les données du champ d'application ont-elles été profilées, analysées et classées ?
- Is DQ profiled against business logic rules as well as for reasonableness against statistical expectations?
- Are the right business, operational, analytical, data and technical stakeholders involved in the processus?
- Have innovative technologies such as ML and AI been considered as part of the processus infrastructure?
- Has the review of data ethics been included in the processus?
- Sont standard criteria for measuring DQ defined and verified?
- Des mesures sont-elles collectées et communiquées régulièrement ?
- Les résultats de la mesure et de la classification des données sont-ils saisis en tant que métadonnées?
Artéfacts
- Règles de gestion et profilage des données and measurement criteria
- Statistical analysis results
- A mechanism for assigning and reporting grades for DQ
- DQ metric reports, dashboards, heat maps and other forms of output
- Liste des parties prenantes et preuves d'une communication bidirectionnelle
Notation
Non initié
Data is not profiled, analyzed or graded for the purpose of assessing DQ.
Conceptuel
Data is not profiled, analyzed or graded for the purpose of assessing DQ, but the need is recognized and the development is being discussed.
Développement
Profilage des données, analysis and grading, for the purpose of assessing DQ, is being developed.
Défini
Profilage des données, analysis and grading, for the purpose of assessing DQ, has been defined and validated by directly involved stakeholders.
Atteint
Profilage des données, analysis and grading, for the purpose of assessing DQ, is established and conducted by stakeholders.
Améliorée
Profilage des données, analysis and grading, for the purpose of assessing DQ, is established as part of business-as-usual practice with a continuous improvement routine.
It is recognized as the normal way of working.
5.3 DQ Issues are Remediated
Data remediation plans must be developed and executed to resolve the most pressing DQ issues. The remediation must include both correcting the existing data and performing root-cause-fix to eliminate future data defects.
5.3.1 Data remediation has been prioritized, planned and actioned
Description
Based on the current state analysis, remediation plans must be developed to address the most pressing DQ issues. Ongoing DQ evaluation and maintenance and timelines must also be established.
Objectifs
- Définir un processus for prioritizing and executing data remediation.
- Develop and prioritize data remediation plans.
- Prepare for immediate action to deal with high priority data remediation.
- Establish timelines for ongoing remediation.
Conseil
Data remediation is about correcting the defective data that has been identified. This data should be corrected as close to the source of data capture as possible. Make sure the remediation activities are not one-off processes, but rather established as part of the DQM routine. Data remediation needs to be implemented for both données au repos et données en mouvement.
Questions
- Is a DQ issue prioritization processus in place?
- Des plans d'assainissement des données ont-ils été élaborés, vérifiés et classés par ordre de priorité ?
- Has appropriate funding been allocated?
- Is there a communications processus related to data remediation?
Artéfacts
- DQ defect reports
- Plan de remédiation des données
- Preuve de la hiérarchisation des problèmes
- Preuve que des mesures correctives ont été prises
- Liste des parties prenantes et preuves d'une communication bidirectionnelle
Notation
Non initié
Data remediation is not prioritized, planned and actioned.
Conceptuel
Data remediation is not prioritized, planned and actioned, but the need is recognized and the development is being discussed.
Développement
Data remediation prioritization, planning and actioning is being developed.
Défini
Data remediation prioritization, planning and actioning has been defined and validated by directly involved stakeholders.
Atteint
Data remediation prioritization, planning and actioning is established, recognized and used by stakeholders.
Améliorée
Data remediation prioritization, planning and actioning is established as part of business-as-usual practice with a continuous improvement routine.
It is recognized as the normal way of working.
5.3.2 Root-cause analysis (RCA) process is defined
Description
Data remediation must include both correcting the existing data that is defective and determining the root-cause of the DQ deterioration to avoid the reoccurrence of defective data in the future.
Objectifs
- Définir un processus for conducting root-cause analysis and fixes.
- Determine the data defect root cause.
- Identify and implement corrective measures to business, data and/or technology processes.
Conseil
Remediating DQ issues is not merely an exercise in data correction. DQ issues can be systemic. Evaluate the depth and breadth of DQ to determine if the organization is focused more on tactical repair versus the upstream remediation of a root-cause fix. A strong reporting structure is needed to ensure that upstream systems are aware of repetitive or continuing DQ problems.
Data defects may have a people, processus, data or technical source. Having the right subject matter expertise from each of these areas will be important to the analysis of the root-cause.
Questions
- Are root cause analysis problems defined?
- Are corrective measures linked to root cause analysis?
Artéfacts
- Evidence of DQ defect reporting across the data supply chain
- Preuve de la réalisation d'une analyse des causes profondes et de la mise en œuvre de mesures correctives
Notation
Non initié
No Root-cause analysis (RCA) processus est définie.
Conceptuel
No RCA processus is defined, but the need is recognized and the development is being discussed.
Développement
The RCA processus est en cours d'élaboration.
Défini
The RCA processus is been defined and validated by directly involved stakeholders.
Atteint
The RCA processus is established, recognized and used by stakeholders.
Améliorée
The RCA processus est établi dans le cadre de la pratique habituelle des affaires avec une routine d'amélioration continue.
Le processus is reviewed and updated at least annually.
5.4 DQ is Monitored and Maintained
Monitoring and maintaining the data includes: 1) implementing la qualité des données control points; 2) capturing DQ metrics to identify defective data, and 3) continuous monitoring of the data.
5.4.1 DQ control points are in place
Description
Data control points must be developed to quantitatively measure the quality of data as it flows through business and technology processes.
Objectifs
- Définir un processus to define DQ control points.
- Put DQ control points in place and bring them to a fully operational state along the data supply chain.
- Record DQ controls as métadonnées.
Conseil
DQ is governed by developing control points along the data supply chain. DQ control points need to be applied at both the point of data entry into the organization and at the point of entry into the consuming application as well as when data moves and transforms along the supply chain. DQ controls include the implementation of business rules, establishing workflows, setting DQ tolerances and monitoring data movement.
Questions
- Les points de contrôle sont-ils définis, vérifiés et documentés ?
- Les règles de gestion sont-elles définies, vérifiées, documentées et approuvées ?
- Les entreprises processus définis et la manière dont ils gèrent les exceptions vérifiée ?
- Les points de contrôle, les règles de gestion et les processus flows operational?
Artéfacts
- Documentation on control points, business rules and processus flux
- Contrôle processus examen et signature
Notation
Non initié
No DQ control points are defined.
Conceptuel
No DQ control points are defined, but the need is recognized and the development is being discussed.
Développement
DQ control points are being developed.
Défini
DQ control points are defined and validated by directly involved stakeholders.
Atteint
DQ control points are established and recognized by stakeholders.
Améliorée
DQ control points are established as part of business-as-usual practice with a continuous improvement routine.
Control points are reviewed for relevance and précision at least annually and adjusted accordingly.
5.4.2 Data issues are managed
Description
Control points along the data supply chain capture DQ metrics that are used to produce DQ dashboards which are used to identify defective data. The DQ defects must be part of the issue management routine of the DM initiative. The DQ issue management processus doit assurer le suivi d'un problème jusqu'à sa résolution et fournir des partie prenante la communication.
Objectifs
- Définir un processus for managing data issues to resolution.
- Drive and prioritize remediation efforts using DQ metric reports.
- Establish an issue management reporting routine and infrastructure.
Conseil
Partie prenante l'engagement, y compris de la part de l'Union européenne, de l'Union européenne et de l'Union européenne. consommateur de données, is critical for successful DQ issue management. The issues need to be managed through all stages of resolution. These stages include defect triage, prioritization, root-cause analysis, root-cause fix and remediation of defective data. Partie prenante communication throughout this processus is critical and must include communication with the consommateur de données. Les consommateur de données must be made aware of the defective data and its impact on their business processus. They may need to participate in the analysis and determination of an acceptable resolution.
Important tools to support the resolution processus sont un journal des problèmes et un système de suivi de l'état d'avancement. Le lien vers l'enregistrement du problème doit faire partie de la page d'accueil de l métadonnées for all instances of defective data. This record will communicate to all users across an organization and can be used to help minimize duplication of effort when an issue is uncovered at multiple points along the data supply chain.
Often, particularly in the early stages of the DM initiative, the volume of defective data may be greater than the resources required to resolve the issues. Documenting the prioritization processus in the issue log even when it results in a backlog of issues is evidence that the issue was known and evaluated rather than new issues being uncovered as part of an audit.
Questions
- Are DQ metric reports and dashboards distributed on a routine basis?
- Are metrics used to identify DQ issues and drive remediation?
- Are the DQ issues captured as métadonnées?
- Are the right business, operational, analytical, data and technology resources involved in defining DQ requirements?
Artéfacts
- DQ dimension metrics
- DQ metric reports, dashboards, heat maps and other forms of output
- Liste des parties prenantes et preuves d'une communication bidirectionnelle
Notation
Non initié
Data issues are not managed.
Conceptuel
Data issues are not managed, but the need is recognized and the development is being discussed.
Développement
Data issue management is being developed.
Défini
Data issue management has been defined and validated by directly involved stakeholders.
Atteint
Data issue management is established, recognized and used by stakeholders.
Améliorée
Data issue management is established as part of business-as-usual practice with a continuous improvement routine.
It is recognized as the normal way of working.
5.4.3 Continuous monitoring is performed
Description
Data is monitored at control points. Control points must be established where data enters a business processus or when it enters a consuming application. To achieve continuous monitoring the data must be checked anytime there is data entering either type of control point. This monitoring may be real-time, a batch processus or on demand.
Objectifs
- Définir un processus for continuous monitoring of DQ.
- Establish an infrastructure for continuous monitoring of DQ.
Conseil
Le processus of continuous monitoring has costs, benefits and operational challenges. Some form of automation is required to achieve continuous monitoring. Legacy systems often are not capable of continuous monitoring. It would be cost prohibitive to add these quality checks at the point of data capture or data use. The challenge is to then define a technical solution that allows execution of the quality checks as close to the point of data capture or load that is an acceptable cost.
Questions
- Existe-t-il un processus for continuous monitoring of DQ?
- Is the infrastructure in place to support continuous monitoring of DQ?
- Les points de contrôle définis et les la qualité des données rules being monitored??
Artéfacts
- Schedule of DQ monitoring
- DQ defect reports
Notation
Non initié
Continuous monitoring at DQ control points is not performed.
Conceptuel
Continuous monitoring at DQ control points is not performed, but the need is recognized and the development is being discussed.
Développement
Continuous monitoring at DQ control points is being developed.
Défini
Continuous monitoring at DQ control points is defined and validated by directly involved stakeholders.
Atteint
Continuous monitoring at DQ control points is established and recognized by stakeholders.
Améliorée
Continuous monitoring at DQ control points is established as part of business-as-usual practice with a continuous improvement routine.
It is recognized as the normal way of working.
“The issues need to be managed through all stages of resolution. These stages include defect triage, prioritization, root-cause analysis, root-cause fix and remediation of defective data.” Are these stages of issue resolution defined anywhere? Specifically interested in defining the difference between ‘Defect Management’ and ‘Issue Management’ and how they relate to the generic term ‘Remediation’.
In reference to Component: 5.0.0
Issues Management is the overall process of taking an issue and seeing it through to resolution (even if the resolution is a conscious decision not to resolve).
The quotes you provided do not use the term Defect Management but Defect Triage. Defect Management is a synonym for Issues Management. Defect Triage, however, is a sub-process of Issues Management. The sub-process is investigating the issue to determine a potential cause and the SMEs that are required to resolve the issue. My experience with defect triage for data quality issues is to decide whether or not it is a technical issue, data architecture issue, or a process issue. If you can determine the source of the problem to be one of these, then you know what SMEs need to be involved in the root cause fix of the issue. Remediation is two-fold. First, I have bad data in my data set I need to cleanse it. The second part is why and where did the bad data get in my data set – this is the root cause fix process. Where did it break, why did it break and what can I do to fix it so it won’t break in the future.
In reference to Component: 5.0.0
“5.3.2 The ODM has an executive owner”
It seems that the title of (sub)capability was mistakenly copied from 2.3.2.
In reference to Component: 5.3.2
Jun,
Thank you for bringing this to our attention.
The issue has been resolved and the sub-heading has been renamed “Root-cause analysis (RCA) process is defined)”.
In reference to Component: 5.3.2