Match Identity

Basic (username/password) and API authentication for the Elasticsearch cluster are supported.

When you obtain Identity Deduplication, you will receive the following files:

The conf directory contains the following configuration files:

The mapping file is a yaml file describing the data type and the weight of each field in your data. The file should be in the following format:

fields:
  <field_name_1>:
    type: <type>
    weight: <weight>
  <field_name_2>:
    type: <type>
    weight: <weight>
  ...

Example:

# MAPPING file:
name:
  type: "PERSON"
  weight: 1
dob:
  type: "DATE"
  weight: 1

where the data file contains the following sample data:

# JSONL file:
{"name": "Sara Connor", "dob": "2004-05-06"}
{"name": "Alice Smith", "dob": "1988-07-08"}
{"name": "Bob Johnson", "dob": "1975-09-10"}
{"name": "Adam Smith", "dob": "1988-07-08"}
{"name": "Robert Johnson", "dob": "1975-09-10"}
{"name": "Emily Parker", "dob": "1995-03-15"}

The weight controls how strongly each field influences the similarity score between the two records. The weight is an integer. A larger weight will have a larger impact on the score. A weight of 0 means the field will be ignored. The weights are relative to each other; that is, a field with a weight of 2 impacts the score twice as much as a field with a weight of 1.

Example: Given the following mapping file, the weights will be calculated from a full weight of 8. The name field is 4/8 or 50% of the score, the date field is 1/8 or 12.5% of the score, and the address field is 3/8 or 37.5% of the score.

name:
  type: "PERSON"
  weight: 4
date:
  type: "DATE"
  weight: 1
address:
  type: "ADDRESS"
  weight: 3

LSH weights

You can specify a separate LSH weight in the mapping file if you want the field treated differently by the LSH algorithm than in final scoring. By default, LSH uses the same weight as regular scoring. The LSH algorithm only cares if the field is included or not; the only valid values for lshweight are 0 and 1.

To set the LSH value, add a third line under type and weight: lshWeight: <0/1>.

Example:

To remove the date from the LSH calculation:

name:
  type: "PERSON"
  weight: 4
date:
  type: "DATE"
  weight: 1
  lshWeight: 0
address:
  type: "ADDRESS"
  weight: 3

The data type determines how each field is handled during processing and scoring. The supported data types are:

The deduplication_report_<timestamp>.jsonl file contains detailed information about the duplicate clusters identified:

{
  "records": [
    {
      "recordId": "9DITU5kB8ADBDGQaM4mA",
      "similarityScores": {
        "8jITU5kB8ADBDGQaM4mA": 0.95867
      },
      "customerRecord": {
        "fields": {
          "dob": "01/23/1965",
          "name": "Jim Jones"
        }
      }
    },
    {
      "recordId": "8zITU5kB8ADBDGQaM4mA",
      "similarityScores": {
        "8jITU5kB8ADBDGQaM4mA": 0.90151066
      },
      "customerRecord": {
        "fields": {
          "dob": "01/23/1965",
          "name": "James Earl Jones"
        }
      }
    },
    {
      "recordId": "8jITU5kB8ADBDGQaM4mA",
      "similarityScores": {
        "8zITU5kB8ADBDGQaM4mA": 0.90151066,
        "9DITU5kB8ADBDGQaM4mA": 0.95867
      },
      "customerRecord": {
        "fields": {
          "dob": "01/23/1965",
          "name": "James Jones"
        }
      }
    }
  ]
}

The deduplication_statistics_<timestamp>.json file contains summary statistics about the deduplication process, including the percentage of duplicates identified:

{
  "totalDuplicateClusters" : 2,
  "percentageDuplicatesFound" : 0.2727272727272727,
  "totalDuplicatesFound" : 3,
  "averageDuplicateGroupSize" : 2.5,
  "totalRecordsProcessed" : 11
}