Match Identity

Each RNI query is processed in two passes to provide the best combination of speed and accuracy.

The first pass gives the system the speed necessary for high-transaction environments, eliminating values in the index from consideration. The slower second pass re-compares each selected value directly in their original script using enhanced scoring algorithms.

The scores from the first pass are discarded and the match candidates are re-ranked according to the similarity scores returned by the second pass. The match scores for all search terms are combined to generate a match score. All entries with a match score equal to or greater than the display threshold are displayed in a list. Those values that are equal to or greater than the match threshold will be highlighted with their match score.

The number of scores moved along to the second pass is determined by the Window Size setting. See Configure for more information on adjusting this setting.

Before searching with Rosette Match Studio, you must create an index by uploading a recordset containing your searchable data.

Rosette Match Studio imports structured data. Supported file formats are:

For .csv files, the first row must be a header row, containing the names of the fields in the source file. For other file types the key names are the field names. The field names must be unique. For .json and .xml files, multi-value fields are automatically detected. For .csv and .tsv files, multi-valued fields must use a delimiter to separate values (for example, John|Jon|Jonathan).

To create a new index:

As part of the the process of creating a new index, you will have to map fields to the columns in the recordset. For more information, see Mapping.

Mapping

Mapping is the process of assigning data types, or fields, to the columns in your dataset. Each column must have a field type assigned to it.

The mapping process is different depending on the data file format.

Mapping for .csv, .tsv, and .txt

Review the following when mapping fields to your index:

• Enable: Only enabled fields will be imported to RMS. Disabled fields will not be used for matching, nor will they appear in search results.

• Data Field: The name of the field as it appears in the index file.

• Field Name: In this column, enter the name of the field as you want it to appear in RMS.

• Use for Match: When this is enabled, the field will be considered when determining a match. When disabled, the field will not be considered when determining a match, but will still appear in search results.

• Grouped Field: Enable this if the data file contains one of the following group field types:

  • Name split into first name, middle name, and last name in separate columns.

  • Date split into day, month, and year

  • Address split into different fields supported by RNI Address matching

  • Multi-value grouped fields

  For more information, see Grouped Fields.

• Data Type: For fields used for match, provide the data type to tell RMS how it should match this field. For more information, see Data Types.

  • For fields that use the Name - General (RNI) data type, you must also specify the values used for the the Name Type (RNI) field later in the wizard.

• Multi-Value: Enable this if the data file contains fields with multiple values. Such fields with multiple aliases must separate values with a delimiter. You will be prompted to specify the type of delimiter used later in the wizard.

Match for .json and .xml

Enable a field by selecting its corresponding checkbox on the left side of the screen. Only enabled fields will be imported to RMS. Disabled fields will not be used for matching, nor will they appear in search results. For each field, the following options are available:

• Field name: In this column, enter the name of the field as you want it to appear in RMS.

• Use for match: When this is enabled, the field will be considered when determining a match. When disabled, the field will not be considered when determining a match, but will still appear in search results.

• Match type: For fields used for match, provide the match type to tell RMS how it should match this field. For more information, see Data Types.

  • For fields that use the Name - General (RNI) match type, you must also specify the values used for the the Name Type (RNI) field on the next step of the wizard.

Grouped Fields

Rosette allows for fields to be grouped together when multiple fields in an index map to a single field for searching. For example, if an index contains separate fields for the first name, middle name, and last name, these fields should be mapped to a single full name field for improved matching. The same is true for an index that separates dates into month, day, and year fields.

When mapping an index that contains group fields, you will have to do a bit of setup to create those groups. After the initial mapping step of the wizard, you will reach the Create Field Groups step.

1. Enter a name for the grouped field under Combined Field Name. For example, for a group consisting of first, middle, and last name, you might name the grouped field "Full Name".

   

2. Select the appropriate field type under Combined Field Type. For example, for a group consisting of first, middle, and last name, you should select Name - Person.

3. Select the checkbox next to all items in the Items Available box which should be included in the group.

4. Select Add to Group.

5. Select Create Group.

6. Map each field included in the group. For example, for a group consisting of first, middle, and last name, you should map them to Name Field 1, Name Field 2, and Name Field 3, respectively.

   Tip

   If the data contains a title field, it should be indexed as 1 with the other fields indexed as 2, 3, and 4.

   

7. Select Save.

8. Select Create.

Search returns a list of records from your index which are potential matches for a query, as determined by the calculated match score.

To perform a search:

Scores higher than the display threshold are shown. They are listed in descending order by match score. Results with match values greater than the match threshold are highlighted in green.

For each result:

Batch search performs multiple searches in a single task, using each record in a file as the query value against the index.

To perform a batch search:

For each value in the batch file, Batch Search displays the search term from the file. To see the matching records from the index, select the plus sign next to a name. This will show the top results with a match score above the display threshold. All match scores above the match threshold will be highlighted in green.

To initiate a new batch search with another dataset, select Upload New Batch.

To view the results from another set, select that set's box in the Query Data list.

The Configure section allows you to change how your data can be searched and how the results are displayed.

To access the Configure section:

You can also access this section by selecting the gear icon beneath the search pane.

Configurations

Use this tab to control the matching behavior of an index, including the weight of each field and the details of how match scores are calculated.

Matching Weights

A field's weight value represents the magnitude of its impact on the final match score. When determining a match, some fields are more important than others. For example, the person name is likely more important in determining a match between two people than the location name. Adjust the weight slider for each field based on its relative importance.

Weight is distributed equally among all fields by default. If a field is missing from a record, that field is ignored and its weight evenly distributed across other fields.

Match Configuration

Individual name tokens are scored by a number of algorithms. These algorithms can be optimized by modifying configuration parameters, thus changing the final match score.

A match configuration contains a set of parameters. Each named match configuration contains parameter values for a specific language pair and entity type. A single named match configuration can contain multiple language pairs and entity types.

Use the Match Configuration dropdown menu to set the default match configuration for search and batch search. You can use the default configuration (RMS-<version> Default) or create a new match configuration. See New Match Configuration for more information on creating a new match configuration.

New match configurations can also be selected directly from the search pane. Display of this value on the search pane is disabled by default. Use the toggle in the Display column to enable display.

Window Size

Each RNI query is processed in two passes to provide the best combination of speed and accuracy.

1. The first pass is designed to quickly generate a set of candidates for the second pass to consider.

2. The second pass compares every value returned by the first pass against the value in the query and computes a similarity score. Multiple scorers are applied in the second pass, to generate the best possible score.

Window size determines the number of scores moved along to the second pass. Increasing window size improves recall, but results in a slower search.

Window size can also be adjusted directly from the search pane. Display of this value on the search pane is disabled by default. Use the toggle in the Display column to enable display.

Match Threshold

Once the match score is calculated for all values in the index, those with scores greater than or equal to the match threshold are highlighted in the search results.

Display Threshold

Once the match score is calculated for all values in the index, only those with scores greater than or equal to the display threshold are returned in the search results. If you aren't seeing results you expect, try lowering the display threshold value to return more results.

When comparing two names, a match score will be displayed below the names being compared, along with a table including additional data about the names such as language, script, entity type, and what the names look like after normalization.

Below the match score, you will find the computation table matrix, a tabular representation of component scores and match types. For more details on how how name matching works, see Understanding name match scores.

For Name fields, the computation table matrix includes:

The tokens from Name 1 are listed down the first column, while the tokens from Name 2 are along the top row. The shaded boxes highlight the token pairs selected during matching that produce the best score. A token pair is a token from Name 1 and its matching token from Name 2. The matchtype column lists the match phenomenon for each token pair. The context score takes into consideration the placement of the token in the score calculation. A penalty is applied if the tokens are out of order. When the tokens line up on the diagonal, they are all in order.

Beneath the computation table matrix, the Tokens tab displays additional information about the weight of each token. Weight determines how important the token pair match is in calculating the final score. For example, unusual tokens get a higher weighting than common names because it is more significant when they match and initials are weighted less than full names. The bin value reflects how "unusual" a token is; the lower the bin, the higher the weight.

The Match Pairs tab displays a simplified list of token pairs along with their final score and match phenomena. This tab also displays whether a pair was used in the match (pairs are not used in a match if their score is too low).

The Score Calculation tab provides detailed information on the process of determining the match scores. This includes a breakdown of the parameters which played a part in determining the score, and what their exact impact on the score was.

Rosette Match Studio compares addresses by comparing the fields within each address. You can enter the address as a single field or as separate fields.

RMS optimizes the matching algorithm to the field type. Named entity fields, such as street name, city, and state are matched using an algorithm similar to name matching. Numeric and alphanumeric fields such as house number, postal code, and unit, are matched using numerically-based methods.

The Match Score Computation displays the match matrix for the address fields. Each field in address 1 is compared with each field in address 2, similar to how name tokens are scored.

You can also use the Match Configuration and Advanced Configuration tabs to apply different match configurations or edit individual address matching parameters, respectively, similar to name and date matching.

Matching records which include names is a challenging problem because name spellings can differ in so many ways, including simple misspellings, to nicknames, truncations, variable spaces (Mary Ellen, Maryellen), spelling variations, and names written in different languages. Nicknames have a strong cultural component. The terminology itself is problematic because matching implies two things that are the same or equal, but name matching is more about how similar two entities are. Once you have a measure of similarity, you may need additional rules or human analysis to determine if it is a match. It is important to understand these challenges when evaluating name accuracy.

NAME1,NAME1_LANG,NAME2,NAME2_LANG,ENTITY_TYPE,MATCH
Peter Harding,eng,Pete Harding,eng,PERSON,Y
Peter Harding,eng,Harding Peter,eng,PERSON,Y
Peter Harding,,Pete Michael Harding,eng,PERSON,Y
Peter Harding,eng,P. M. Harding,eng,PERSON,N
Peter Harding,eng,Pat Harding,,PERSON,N
Peter Harding,eng,P. B. Harding,eng,PERSON,N
Peter Harding,eng,Pietro Hardin,eng,PERSON,N

123 Fake Street Springfield MO,,123 Fake St Springfield IL,,ADDRESS,N
820 Forest Road,,820 Forrest Rd,,ADDRESS,Y

Creates a new evaluation to run RNI against a data set along with a selected configuration, helping you configure RNI for your environment.

Before you can run a new evaluation, you must:

Select View Evaluations to display all evaluations performed for a given evaluation data file. Use the Display settings to view the evaluation results for the best threshold, or for a specific threshold. Once you have specified the threshold, you can download the evaluation results for that threshold. Use Threshold Report to visualize how the results change as the threshold increases or decreases.

A key feature of RNI is that it returns a normalized score between 0 and 1 indicating how similar two names are. This makes integrating RNI into existing workflows powerful. However, it leads to the inevitable question: “What threshold should I use?” Without quantitative error analysis, this question is difficult to answer. RMS determines the correct value by processing your evaluation data against various thresholds, calculating precision, recall, and F1 for each data type.

The threshold report graphs the precision, recall, and F1 for each threshold value and each data type. As you can see from the graph, the threshold can be adjusted to favor precision or recall. With a high threshold, false positives will be less likely, leading to a higher precision. With a low threshold, false negatives will be less likely, leading to a higher recall. For applications involving border checking, you may be inclined to favor recall to avoid allowing potential threats into a country. Conversely, for Know Your Customer(KYC) applications, leaning your threshold towards precision may be best to reduce false positives. Your choice of threshold should ultimately be based on an analysis of accuracy as a function of threshold for your particular data, as well as your business requirements.

The threshold report page contains a table listing the Precision, Recall, and F1 values for each data type, along with the threshold value, along with a threshold graph. The threshold is the value which produces the best F1 score.

Figure 1. Threshold Report Graph

From this page, you can download the following reports as .csv files:

Hover over the Configure tab and select Match Configurations to view the Match Configurations page.

Individual name tokens are scored by a number of algorithms. These algorithms can be optimized by modifying configuration parameters, thus changing the final match score. RMS provides an easy way to compare and test different parameter settings based on your data.

A match configuration contains a set of parameters. Each named match configuration contains parameter values for a specific language pair and entity type. A single named match configuration can contain multiple language pairs and entity types.

The Configure page contains a list of existing match configuration files. You can:

Configuring Language Pairs and Types

When editing a match configuration, you can edit parameters for all language pairs and entity types, or for specific language pairs and entity types. Control which pairs and types you are configuring using the tree in the upper-left corner of the Match Configuration page.

If you want to edit parameters for a specific language pair and entity type that is not included in the tree, you can add it using the Add New Pair section in the lower-left corner of the Match Configuration page.

When you edit a parameter in your match configuration, that parameter's value will be inherited by any language-pair-specific or type-specific parameters "below" it in the hierarchy. For example, if you change adjustOneSidedDeletionScores to "1" for all language pairs and types, the adjustOneSidedDeletionScores parameter will also be set to "1" for English to English matching, and that parameter will be marked as "Inherited" when viewing it in the context of that language pair. Select Jump To to go where the value is inherited from.

If you then set that parameter to "1.2" for English to English Person matching, the parameter value will not change for any other language pairs or types. The parameter will be marked as "Local" to indicate that the value was specified for this language pair and type, and not inherited from anywhere. Select Reset to reset the parameter to the value it would inherit from "above" in the hierarchy.

Hover over the Configure tab and select Overrides to view the Overrides page. This page allows you to add or remove token pairs that are considered a match regardless of similarity. You can use this list for things like names with unusual nicknames, such as "Margaret" and "Peggy."

To add a new override set:

To find an override set:

To delete an override set:

To add a token to an existing override set:

To remove a token from an existing override set:

Hover over the Configure tab and select Stop Words to view the Stop Words page. This page allows you to add or remove words that are ignored during string matching so that they do not affect the match score. You can use this list for parts of names or titles that are not significant when determining a match, such as "Mr." or "Prime Minister."

You can create new stop words as prefixes to names. To add a new prefix stopword:

To find a stop word:

To delete a stopword:

For servers without authentication, the following parameters must be set in elasticsearch-<version>/config/elasticsearch.yml:

For servers with basic authentication, the following parameters must be set in elasticsearch-<version>/config/elasticsearch.yml:

For all servers, the following parameters must be set in elasticsearch-<version>/plugins/rni/bt_root/rlpnc/data/etc/parameter_defs.yaml:

To add a new external server:

After you finish adding the server, RMS will automatically switch to it, and it will be visible in both the Manage Servers page and the server dropdown menu on the navigation bar. Any parameter universes will automatically be imported into RMS as Match Configurations. If you are adding a server to RMS Lite, you will next be asked to link the desired indices from the new server.

If there are external servers added to your instance of RMS, you can connect to them at any time by completing the following steps:

You can also connect to a server from the Manage Servers page by hovering over the vertical ellipsis in the Action column for that server and selecting Connect.

While RMS Lite does not allow you to create new indices within RMS, you can link indices from the connected server(s). You will be prompted to do this the first time you use RMS Lite, but you can also do it at any time afterwards.