Data Quality Gates
Data quality gates provide process-level validation within customer data and identity resolution workflows, ensuring the integrity of data used in your first-party identity graph. Data quality gates can validate the data preparation, resolution, and export processes of your workflows, detect any issues, halt the workflow at any process that needs further review, and provide an error message in the process indicating any data quality problems.
You can review data quality failures in the Data Quality Gate Failures page and have the option to override the data quality gates and resume the workflow.
Types of Data Quality Gates
Data quality gates are available for data preparation, resolution, and export processes. A data quality gate type is determined by the metric the gate uses to evaluate a data field. The following table lists the types of data quality gates. For information on configuring a data quality gate for each process type, see data preparation, resolution, and export processes.
Note
The number of data quality gates in a process can vary and depends on the process type and number of data fields mapped. The data quality gates for data preparation and resolution processes are set to “Active” by default. For export processes, the default setting depends on the configurations in the upstream processes.
Data Quality Gate | Metric | Description | Available Configuration |
|---|---|---|---|
Corrupted | corrupted | Only present in data preparation processes. Checks the percentage of the data file that is corrupted (e.g., unexpected characters in rows, etc.). Default threshold is 0.5 percent. |
|
Mandatory | mandatory | Checks if required fields are populated. | Set as active or inactive |
Overconsolidation | overconsolidation | Only present in resolution processes. Checks for one or both of the following indications of overconsolidation:
| Set as active or inactive |
Uniqueness | uniqueness | Checks if specific fields are unique in the dataset. | Set as active or inactive |
Uniqueness by Group | uniquenessByGroup | Checks if a specific group of fields is unique in the dataset. | Set as active or inactive |
Volume Change | volumeChange | Checks if the number of values added or removed in the current workflow process run when compared to the previous is under a specified threshold. Default threshold is 0.5 percent. |
|
Configure a Data Quality Gate
Data quality gates can be configured to optimize your workflow. Configurable data quality gates are available for data preparation, resolution, and export processes. You can configure the data quality gates of a process when you create or edit the process in a workflow. For information on configuring the data quality gates for each type of process, see “Configure a Data Preparation Process”, “Configure a Resolution Process”, and “Configure an Export Process”.
Note
The number of data quality gates in a process can vary and depends on the process type and number of data fields mapped.
Begin creating the process as indicated in the configuration topic for each type of process (see “Configure a Data Preparation Process”, “Configure a Resolution Process”, and “Configure an Export Process”).
Follow the instructions for the "Data Quality Gates" step in the process..
In the Workflows page, select the workflow that has the process with the data quality gate you want to edit.
Click the tab to edit the workflow.
Select the More Options menu (
) of the desired process in the workflow and click .Click until you arrive at the “Data Quality Gates” step.
Update the appropriate data quality gates (see “Configure a Data Preparation Process”, “Configure a Resolution Process”, and “Configure an Export Process”).
Click , review the summary of the process, and click .