Match Identity

The entityType field identifies the type of name being matched and to select the algorithms to use for matching. Where supported, stop words and override files are specific to an entity type. Parameters can be set for specific languages and entity types.

You can process fielded names by separating the fields with "|". RNI assigns no explicit semantics to each field (such as given name or surname), but it does pay attention to the order of the fields when comparing two fielded names. RNI assigns lower scores to matches that cross field boundaries (e.g., the first field in name1 matches the second field in name2). Fields within a name can be empty.

When scoring a potential match between a name with fields and a name without fields, RNI treats the name without fields as if it were a name with a single field.

RNI treats trailing empty fields as if they were not present. For example "Rosanne|Taylor Smith|" is treated the same as "Rosanne|Taylor Smith".

Alternatively, you have the option of specifying that there is an unknown value in a field. To specify an unknown name field, replace the field with *?*.

When using JSON objects with RNI, special characters must be properly escaped when used in strings. RNI requires a backslash to escape the special character and then JSON requires another backslash to escape the first backslash. Thus, In RNI, the proper escape character for names containing a special character is a double backslash (\\).

The | used in fielded names is one example of a special character embedded within a name, where | is used to separate the fields. For proper processing of the vertical bar character, RNI needs to be able to distinguish when the user intends to build a fielded name and a name which contains the vertical bar character.

Let's assume we have a name that includes a |; it is not indicating a fielded name: "John|Smith". RNI requires that you escape the vertical bar with a backslash; e.g. "John\|Smith". Then, JSON requires that the backslash character be escaped with a backlash. The correct syntax for the name "John|Smith" is "John\\|Smith". If the entry were representing a fielded name, the correct syntax would be "John|Smith" without any backslashes.

We're going to continue the example that we started. The index is rni-test. The mapping defines a primary_name, aka, and occupation.

Let's consider an example of a database that includes alias names along with a primary name.

Nested Mapping:

"properties" : {
   "primary_name" : {"type" : "rni_name"},
   "aliases" : { 
      "type" : "nested",
      "properties" : { 
        "alias_name" : { "type" : "rni_name" } 
      }
   }
}

The curl command to create the mapping:

curl -XPUT "http://localhost:9200/rni-test/_mapping" -H 'Content-Type: application/json' -d '{
    "properties" : {
        "primary_name" : { "type" : "rni_name"},
        "aliases" : {
            "type" : "nested",
            "properties": { "alias_name" : { "type" : "rni_name" }
         }
       }
    }
}'

Each record includes a primary name. Each primary name can have multiple aliases.

"primary_name" : "John Smith", 
"aliases": [
  {"alias_name": "John Shark"},
  {"alias_name": "Smithy"},
  {"alias_name": "Johnny boy"}
]

The curl command to add the data:

curl -XPUT "http://localhost:9200/rni-test/_doc/null" -H 'Content-Type: application/json' -d '{
    "primary_name" : "John Smith", 
    "aliases" : [
        {"alias_name": "John Shark"},  
        {"alias_name": "Smithy"},
        {"alias_name": "Johnny boy"}
    ]
}'

The query will try to match one of the aliases. Specify score_mode: max to return the highest match score of the aliases.

curl -XGET "http://localhost:9200/rni-test/_search" -H 'Content-Type: application/json' -d '{
    "query" : {
        "nested" : {
            "path" : "aliases",
            "query" : {
                "match" : { "aliases.alias_name": "Johnny" }
            }
        }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "nested" : {
                    "path" : "aliases",
                    "score_mode" : "max",
                    "query" : {
                        "function_score" : {
                            "name_score" : {
                                "field" : "aliases.alias_name",
                                "query_name" : "Johnny"
                            }
                        }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

Individual name tokens are scored by a number of algorithms or rules. These algorithms can be optimized by modifying configuration parameters, thus changing the final similarity score.

The parameter files are contained in two .yaml files located in plugins/rni/bt_root/rlpnc/data/etc. The parameters are defined in parameter_defs.yaml and modified in parameter_profiles.yaml.

parameterUniverseOne/spa_eng_PERSON:
    reorderPenalty: 0.4
    HMMUsageThreshold: 0.8
    stringDistanceThreshold: 0.1
    useEditDistanceTokenScorer: true
parameterUniverseOne/eng_eng:
    reorderPenalty: 0.6
    

curl -XPOST "http://localhost:9200/rni-test/_search" -H 'Content-Type: application/json' -d'{ 
  "query": {
    "match": {
      "full_name": "A Ely Taylor"
    }
  },
  "rescore": {
    "window_size": 3,
    "rni_query": {
      "rescore_query": {
        "rni_function_score": {
          "name_score": {
            "field": "full_name",
            "query_name": "A Ely Taylor",
            "score_to_rescore_restriction": 1,
            "window_size_allowance": 0.5,
            "universe": "parameterUniverseOne"
          }
        }
      },
      "query_weight": 0,
      "rescore_query_weight": 1
    }
  }
}'

curl -XPOST "http://localhost:9200/rni-test/_search" -H 'Content-Type: application/json' -d'{ 
  "query": {
    "match": {
      "full_name": "{ \"data\": \"A Ely Taylor\", \"universe\": \"parameterUniverseOne\"}"
    }
  },
  "rescore": {
    "window_size": 3,
    "rni_query": {
      "rescore_query": {
        "rni_function_score": {
          "name_score": {
            "field": "full_name",
            "query_name": "A Ely Taylor",
            "score_to_rescore_restriction": 1,
            "window_size_allowance": 0.5,
            "universe": "parameterUniverseOne"
          }
        }
      },
      "query_weight": 0,
      "rescore_query_weight": 1
    }
  }
}'

curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'{ 
  "profiles": [
    {
      "name": "parameterUniverseOne/spa_eng_PERSON",
      "parameters": {
        "reorderPenalty": 0.4,
        "HMMUsageThreshold": 0.8,
        "stringDistanceThreshold": 0.1,
        "useEditDistanceTokenScorer": true
      }
    }
  ]
}'

To start tuning the parameters, run the RNI pairwise match on the test set and look at the match reasons in the response. These match reasons will serve as a guide for which parameters to tune, which are defined in parameter_defs.yaml. For additional support on tuning the parameters, contact support@rosette.com.

Once you define a profile and set a parameter value, rerun the RNI pairwise match, scoring the match with the edited parameter_profiles.yaml file.

zho_zho_PERSON:  
  haniFourCornerCodeMismatchPenalty: 1

You can index null values and empty strings by updating the allowNullValue parameter. If the allowNullValue parameter is enabled, any document containing null values and empty strings for the fields rni_name, rni_address, and rni_date types will be successfully indexed, but search capabilities will be limited to valid values.

You can direct RNI to index documents with malformed strings of language by updating the ignoreBadData parameter. If the ignoreBadData parameter is enabled, any document containing a malformed language string will be successfully indexed, but search capabilities will be limited to valid languages.

By default these parameters are disabled. These features are useful when performing bulk operations in Elasticsearch.

The file name is parameter_profiles.yaml, located in plugins/rni/bt_root/rlpnc/data/etc/.

To turn any of these features on, set the value of the parameter ignoreBadData or allowNullValue in the above file to true.

To evaluate the newly tuned parameter values, query a large dataset of names or addresses that does not include your test set. For an exact evaluation, query an annotated dataset that includes the correct answers for a number of queries. For a general evaluation, measure the number of pair matches that have scores above your threshold, compared to before tuning the parameter values. If there were too many matches before, now there should be fewer matches. If there were too few matches before, there should be more now. If the number of matches increases or decreases dramatically, then there is a higher chance of missing correct matches below the threshold or including incorrect matches above the threshold.

If you find new pair matches that you want to score above or below your threshold, collect them into a test set to retune the parameters. Then evaluate the parameters again using a large dataset to review results. It is important to frequently evaluate new parameter settings on separate test data to ensure the parameters continue to return correct results.

Before running any matching algorithms, the names are transformed into tokens that can be compared. RNI uses stop patterns and stop word prefixes to remove patterns, including titles such as Mr., Senator, or General, that you do not want to include in name matching. Both stop patterns and stop word prefixes are used to strip matching name elements during indexing and querying. Stop words are string literals and are processed much more quickly than stop patterns, which are regular expressions. You should use stop words for the most efficient removal of prefixes, such as titles. Stop words are language-dependent.

For each name, RNI performs the following steps in order:

RNI cycles its way through the stop patterns then the stop words, each cycle removing the patterns and words that strip nothing, until the list of stop patterns and stop words is empty.

stopregexes_LANG[_TYPE].txt

^fnu\b
\blnu$ 

\blnu$    2
\blnu$    3

stopprefixes_LANG[_TYPE].txt 

You can create UTF-8 text files that specify the scores to be assigned for specified full-name pairs. The filename uses the ISO 639-3 three-letter language codes to designate the language of each full name in each of the full-name pairs:

fullnames_LANG1_LANG2[_TYPE].txt

where LANG1 is the three-letter language code for the first name and LANG2 is the three letter language code for the second name.

Each row in the file, except for rows that begin with #, is a tab-delimited full-name pair and score:

name1 Tab name2 Tab score

The scores must be between 0 and 1.0, where 0 indicates no match, and 1.0 indicates a perfect match.

The installation includes a sample file with sample entries commented out: plugins/rni/bt_root/rlpnc/data/rnm/ref/override/fullnames_eng_eng.txt. Any non-commented-out entries in this file assign scores to English queries applied to English names in an RNI index. For example,

John Doe	Joe Bloggs	1.0

indicates that the query name John Doe matches the index name Joe Bloggs (both used in different regions to indicate 'person unknown') with a score of 1.0.

These match patterns are commutative. The previous entry also specifies a match score of 1.0 if the query name is Joe Bloggs and the index includes a document with an rni_name field containing John Doe.

You can add entries for English to English name matches to fullnames_eng_eng.txt, and create additional override files, using the filename to specify the languages. For example the following entries could appear in fullnames_jpn_eng.txt:

外山恒	   Toyama Koichi    1.0
ヒラリークリントン    Hillary Clinton    1.0

You can create text files that specify token (name-element) pairs that match. Token pair overrides are supported^[4] for English-English, Japanese-English, Chinese-English, Russian-English, Spanish-English, Japanese-Japanese, Russian-Russian, English-Korean, Korean-Korean, Spanish-Spanish, Greek-English and Hungarian-English token pairs. Such pairs may include proper name and nickname, such as Peter and Pete, and cognate names such as Peter and Pedro. When RNI evaluates two names, each of which contains an element from the pair, it enhances the value of the resulting name match score. For example, if Abigail and Abby constitute a token pair, then the match score for Abigail Harris and Abby Harris will be higher than it would be if the token pair had not been specified.

The token pairs may be within a language or cross-lingual, as indicated by the file name:

tokens_LANG1_LANG2_[TYPE].txt

where LANG1 is the three-letter language code for the first token in each pair and LANG2 is the three-letter language code for the second token in each pair. Each entry in the file, except for rows that begin with #, is a tab-delimited token pair and may include a raw score between 0.0 and 1.0 or an indicator that at least one of the tokens is a nickname or that the tokens are cognates:

Token1 Tab Token2 Tab [[0.0-1.0]|NICKNAME|COGNATE|VARIANT]

A token pair override score (raw score or indicator) serves as a minimum score, but you can write "/force" after a token score to force it to be exactly that value:

Token1 Tab Token2 Tab [([0.0-1.0]|NICKNAME|COGNATE|VARIANT)/force]

If you would like to prevent a token pair from matching, you can use the SUPPRESS indicator as an alias for "0.0/force". If you do not include NICKNAME, COGNATE, VARIANT, or SUPPRESS, RNI assumes NICKNAME.

RNI includes plugins/rni/bt_root/rlpnc/data/rnm/ref/override/tokens_eng_eng.txt, which contains a list of English/English token pairs. For example:

Peter    Pete    NICKNAME
Peter    Pedro   COGNATE

This directory also contains Chinese to English token overrides for LOCATION and ORGANIZATION: tokens_zho_eng_LOCATION.txt, tokens_zho_eng_ORGANIZATION.txt.

When you create an additional file in the same location, use the ISO 639-3 three-letter language name in the filename to identify the language of each name element in the pair. For example tokens_eng_eng.txt indicates that the contents match English names to English names; tokens_eng_eng_ORGANIZATION.txt indicates that the contents match English ORGANIZATION names to English ORGANIZATION names. The SDK includes a sample file for matching English/English tokens in LOCATION entities: tokens_eng_eng_LOCATION.txt.

We recommend that you enter the language names in alphabetical order in the filename and token pairs. Keep in mind that the order has no influence on the resulting score, since the scoring is commutative.

You can create text files that specify the normalized form for tokens (name elements) and variants to normalize to that form. The file name indicates the language and optionally the entity type for the tokens to be normalized:

equivalenceclasses_LANG_[TYPE].txt

For example, equivalenceclasses_jpn.txt would contain entries for normalizing Japanese token variants for any entity type to a normalized form.

Each entry in the file contains a normalized form followed by one or more variant forms. The syntax is as follows:

[normal_form1]
variant1_1
variant1_2
variant1_3
[normal_form2]
variant2_1
variant2_2
variant2_3
...

RNI includes plugins/rni/bt_root/rlpnc/data/rnm/ref/override/equivalenceclasses_eng_PERSON.txt, which contains a list of variant renderings to normalize to muhammad:

[muhammad]
mohammed
mahamed
mohamed
mohamad
mohammad
muhammed
muhamed
muhammet
muhamet
md
mohd
muhd

You can add lists of variants to this file, including the normalized form in square brackets to start each list.

You can edit the list of tokens that are given low influence in RNI. These low weight tokens are parts of a name (such as suffixes) that don't contribute much to the name matching accuracy.

The file name is lowWeightTokens_LANG.txt.

For example, plugins/rni/bt_root/rlpnc/data/rnm/ref/lowWeightTokens_eng.txt contains entries for tokens in English that you may want to put less emphasis on: "jr", "sr", "ii", "iii", "iv", "de".

Many companies have their own file of organizations with their different names. To improve matching between organization names, you can supplement the real world IDs provided in RNI and build your own file of real world IDs. The provided file will build a binary file in the specified output directory named <LANG>_ORGANIZATION_ids.bin where <LANG> is the three-letter language code of the file.

The input file is a tab separated file (.tsv). Each line contains an organization name and a corresponding alphanumeric ID. The file can only contain a single language and script. You must create a separate file for each language.

IBM    WE1X92
Big Blue    WE1X92
International Business Machines    WE1X92

Unzip the file realWorldIDBuilder.zip found in the plugins/rni/bt_root directory and run the build command. Instructions on how to run the program are in the README.md file in the zip file.

You may want to use real world ID matching even if there are some entities which you do not want to match via real world IDs. You can omit specific organizations and QIDs (Wikidata's identifier for entities) from matching by creating an omit file listing the organization names and QIDs you would like to omit.

The omit file is a tab separated file (.tsv) named <LANG>_ORGANIZATION_ids.tsv where <LANG> is the three-letter language code of the file. Each omit file can only contain names in one language and separate files must be made for each language. There are three types of lines that can appear in an omit file, which have different effects on omission: pairs, lone names, and lone QIDs.

Example:

IBM    Q37156
Nintendo    *
*    Q45700

To enable an omit file in RNI:

When an address is parsed into address fields, values can get put into the wrong field. Address field groups encapsulate common transpositions between fields. When scoring matching values in fields, RNI uses address field groups to group related or similar fields. If two field values match, but they are dissimilar fields, RNI applies a penalty to that match, reducing the score for that pair.

When matching two fields, the following penalties are applied:

RNI compares the fields in the query with the fields in the index, matching each non-blank field. Addresses do not have to contain all the same fields to be compared and matched.

As with other objects, the query for an address consists of two parts: the base query and the RNI pairwise address match rescore query.

Base Query. The base query is a standard query against the address field. Refer to Query the Index.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query" : {
    "match" : {
      "residence" : "{\"road\" : \"Main\", \"state\" : \"MA\"}"
    }
  }
}'

RNI Rescore with Addresses. Refer to Rescoring with RNI Pairwise Name Match.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query" : {
    "match" : { "residence" : "{\"road\" : \"Main\", \"state\" : \"MA\"}" }
  },
  "rescore" : {
    "query" : {
      "rescore_query" : {
        "function_score" : {
          "address_score" : {
            "field" : "residence",
            "query_address" : {
              "road" : "Main", 
              "state" : "MA"
            }
          }
        }
      }, 
      "query_weight" : 0.0,
      "rescore_query_weight" : 1.0
    }
  }
}'

The query returns a hit with the RNI address match score.

"hits": {
 "total" : 1,
  "max_score" : 0.6057692,
  "hits" : [
    {
      "_index" : "rni-test",
      "_type" : "_doc",
      "_id" : "1",
      "_score" : 0.6057692,
      "_source" : {
        "primary_name" : "Joe Schmoe",
        "residence" : {
          "houseNumber" : "123",
          "road" : "Main St",
          "city" : "Boston",
          "state" : "Massachusetts",
          "postCode" : "02110"
        }
      }
    }
  ]
}

The address match score is a measure of how similar the addresses are. Similar addresses have a stronger match and their address match score is closer to 1.

The address can be structured as a string for queries. The address structure for the query is independent of the format of the address in the original document. A string can be used in the query regardless of whether the indexed address was formatted with fields or as a string.

Base Query. The base query constructed with an address string.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query" : {
    "match" : {"residence" : "Main, MA"}
  }
}'

RNI Rescore with Addresses. The rescore query with an address string.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query" : {
     "match" : { "residence" : "Main, MA" }},
   "rescore" : {
     "query" : {
       "rescore_query" : {
         "function_score" : {
           "address_score" : {
             "field" : "residence",
             "query_address" : "Main, MA"
         }
       }
     },
     "query_weight" : 0.0,
     "rescore_query_weight" : 1.0
     }
   }
}'

The response displayed here returns the address as a string because the indexed document used in this example represented the address as strings. The response will return the address in the same format as the indexed document. The format of the query does not have to match the format of the indexed documents.

"hits" : {
    "total" : {
      "value" : 1,
      "relation" : "eq"
    },
    "max_score" : 0.4552421,
    "hits" : [
      {
        "_index" : "rni-test",
        "_type" : "_doc",
        "_id" : "1",
        "_score" : 0.4552421,
        "_source" : {
          "primary_name" : "Joe Schmoe",
          "residence" : "123 Main St, Boston, Massachusetts, 02110"
        }
      }
    ]
  }

The address match score is a measure of how similar the addresses are. Similar addresses have a stronger match and their address match score is closer to 1.

A stop pattern is a regular expression that excludes matching address field elements during indexing and queries. You can use any regular expression supported by the Java java.util.regex.Pattern class; see the Javadoc for detailed documentation.

Stop patterns for a given address field are specified in a UTF-8 file with the AddressField name:

stopregexes_LANG_ADDRESS_FIELD__FIELD.txt

where LANG is a three-letter language code and FIELD is an AddressField name. Currently, the only supported values for LANG are eng and zho. Each row in the file, except for rows that begin with #,^[6] is a regular expression. Leading and trailing whitespace is removed from regex lines, so use \s at beginning and end where needed.

Elements in the address fields matching any of these regular expressions are removed. Longer stop patterns are applied before shorter stop patterns, so the presence of a shorter stop pattern does not prevent the stripping of a longer pattern that includes the shorter pattern.

Stop pattern files are arranged by field in plugins/rni/bt_root/rlpnc/data/addresses/ref/stopwords. You can add patterns to existing files, or if the file doesn't exist, create a UTF-8 file in the directory and include the full address field identifier (ADDRESS_FIELD__FIELD) in the filename. For example, stopregexes_eng_ADDRESS_FIELD__CITY.txt would include regular expressions to remove elements from the CITY address field for English.

Use of complex patterns may increase processing time. When possible, use stop word prefixes.

A stop word prefix is a string literal that strips the matching prefix from address field elements during indexing and queries.

Stop word prefixes for a given address field are specified in a UTF-8 file with the AddressField name:

stopprefixes_LANG_ADDRESS_FIELD__FIELD.txt

where LANG is a three-letter language code and FIELD is an AddressField name. Currently, the only supported values for LANG are eng and zho. Each row in the file, except for rows that begin with #,^[7] is a string literal.

Prefixes in the address field matching any of these string literals are removed.

Like stop patterns, longer stop word prefixes take precedence over shorter prefixes that the longer stop word contains.

RNI includes files with stop word prefixes for selected address fields in English and Chinese. These files are in plugins/rni/bt_root/rlpnc/data/addresses/ref/stopwords. You can modify the contents of these files. To add stop word prefixes for a different address field, create an additional UTF-8 file in the same subdirectory and include the full address field identifier (ADDRESS_FIELD__FIELD) in the filename. For example, stopprefixes_eng_ADDRESS_FIELD__CITY.txt would include stopword prefixes for use on CITY address field for English.

RNI supports a wide variety of date formats. 

The following table shows different acceptable formats for the date March 7, 1984.

There are many ways to incorporate date matching within your query. Here are two examples, one with date matching by itself, and one with date and name matching.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "match" : {
            "birth_date" : "08-1955-25"
        }
    }
}'

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "match" : { "birth_date" : "08-1955-25" }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "date_score" : {
                        "field" : "birth_date",
                        "query_date" : "08-1955-25"
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

"hits": {
   "total": 1,
   "max_score": 1.618923,
   "hits": [
     {
       "_index": "test",
       "_type": "_doc",
       "_id": "AVXMepnorGuybmuiQtQr",
       "_score": 0.8120856,
       "_source": {
         "primary_name": "Joe Schmoe",
         "birth_date": "07-1955-24"
       }
     }
   ]
 }

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "primary_name": "Joe S."
          }
        },
        {
          "match": {
            "birth_date": "08-1955-25"
          }
        }
      ]
    }
  }'

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
  "query": {
    "bool": {
      "should": [
        {
          "match": {
            "primary_name": "Joe S."
          }
        },
        {
          "match": {
            "birth_date": "08-1955-25"
          }
        }
      ]
    }
 },
"rescore": {
    "query": {
      "rescore_query": {
        "function_score": {
          "doc_score": {
            "fields": {
              "primary_name": {
                "query_value": "Joe S."
              },
              "birth_date": {
                "query_value": "08-1955-25"
              }
            }
          }
        }
      },
      "query_weight" : 0.0,
      "rescore_query_weight" : 1.0
    }
  }
}'

The query can be a record containing multiple fields. The fields in the query record must be mapped to those of the indexed documents.

Base Query. The base query is a standard Elasticsearch query containing multiple fields that will return candidates for rescoring.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : { 
            "should" : [ 
                { "match" : { "name" : "{\"data\" : \"Brian McDonough\", \"entityType\": \"PERSON\"}" } }, 
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{\"houseNumber\":\"48\",\"road\":\"Parker St\",\"city\":\"Boston\",\"state\": \"MA\" } } 
            ]
        }
    }
}'

RNI Rescore with Records. Use the doc_score function to rescore the indexed documents against a query record.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : { 
            "should" : [ 
                { "match" : { "name" : "Brian McDonough" } }, 
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{\"houseNumber\":\"48\",\"road\":\"Parker St\",\"city\":\"Boston\",\"state\": "MA\" } } 
            ]
        }
    },
    "rescore" : {
        "rni_query" : {
            "rescore_query" : {
                "function_score" : {
                    "doc_score" : {
                         "fields" : {
                             "name" : { "query_value": "Brian McDonough" },
                             "dob" : { "query_value": "10/19/87" },
                             "address" : {
                                  "query_value" : { 
                                      "houseNumber" : "48", 
                                      "road" : "Parker St", 
                                      "city" : "Boston", 
                                      "state" : "MA" 
                                   }
                             },
                             "height" : { "query_value": 67 },
                             "nationality" : { "query_value": "CANADA" }
                         }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

As with addresses, the query_value of names can be an object to match additional name information. The rescore query above can easily be modified to additionally match against a name's entityType field:

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : { 
            "should" : [ 
                { "match" : { "name" : "Brian McDonough" } }, 
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{\"houseNumber\":\"48\",\"road\":\"Parker St\",\"city\":\"Boston\",\"state\": \"MA\" } } 
            ]
        }
    },
    "rescore" : {
        "rni_query" : {
            "rescore_query" : {
                "function_score" : {
                    "doc_score" : {
                         "fields" : {
                             "name" : {
                                  "query_value": {
                                      "data": "Brian McDonough",
                                      "entityType": "PERSON"
                                  }
                             },
                             "dob" : { "query_value": "10/19/87" },
                             "address" : {
                                  "query_value" : { 
                                      "houseNumber" : "48", 
                                      "road" : "Parker St", 
                                      "city" : "Boston", 
                                      "state" : "MA" 
                                   }
                             },
                             "height" : { "query_value": 67 },
                             "nationality" : { "query_value": "CANADA" }
                         }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

Each field can be given a weight to reflect its importance in the overall matching logic.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : { 
            "should" : [ 
                { "match" : { "name" : "Brian McDonough" } }, 
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{ \"houseNumber\" : \"48\", 
                                             \"road\" : \"Parker St\", 
                                             \"city\" : \"Boston\", \"state\" : \"MA\" }" } } 
            ]
        }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "doc_score": {
                         "fields": {
                             "name": { "query_value": "Brian McDonough", "weight": 4 },
                             "dob": { "query_value": "10/19/87", "weight": 2 },
                             "address" : {
                                  "query_value" : { 
                                      "houseNumber" : "48", 
                                      "road" : "Parker St", 
                                      "city" : "Boston", 
                                      "state" : "MA" 
                                   },
                                   "weight" : 2
                             },
                             "height" : { "query_value": 67, "weight": 0.5},
                             "nationality" : { "query_value": "CANADA", "weight": 1 }
                         }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

By default, if a queried-for field is null in the index, the field is removed from the score calculation, and the weights of the other fields are redistributed. However, you can override this behavior by using the score_if_null option to specify what score should be returned for this field if it is null in the index document.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : {
            "should" : [
                { "match" : { "name" : "Brian McDonough" } },
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{ \"houseNumber\" : \"48\", 
                                             \"road\" : \"Parker St\", 
                                             \"city\" : \"Boston\", \"state\" : \"MA\" }" } } 
            ]
        }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "doc_score" : {
                         "fields" : {
                             "name" : { "query_value": "Brian McDonough", "weight": 4, "score_if_null" : 0.0  },
                             "dob": { "query_value": "10/19/87", "weight": 2 },
                             "address" : {
                                  "query_value" : { 
                                      "houseNumber" : "48", 
                                      "road" : "Parker St", 
                                      "city" : "Boston", 
                                      "state" : "MA" 
                                   },
                                   "weight" : 2
                             },
                             "height" : { "query_value": 67, "weight": 0.5},
                             "nationality" : { "query_value": "CANADA", "weight": 1 ,  "score_if_null" : 1.0  }
                         }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

The doc_score function for rescoring does not currently support search queries containing multiple nested fields. To perform these queries, chain multiple rescorers and adjust the query_weight and rescore_query_weight parameters to control the relative importance of the original query and of the rescore query, respectively. When chaining multiple RNI advanced rescorers, be sure to add "score_mode":"total" to each rni_query object to ensure the final score is properly accumulated.

This example expands the previous examples, adding alias names and modifying the single date of birth (dob field) to contain a list of dates of birth, one for each alias (dob field).

To calculate the rescore_query_weight for each nested field, you have to work from bottom to top, dividing each field's desired weight by the product of the already-calculated query_weight values. The query_weight is calculated by subtracting the rescore_query_weight from 1.

If there are no previous query_weight values, the rescore_query_weight is simply the desired field weight.

In this example, the desired field weights are 0.4, 0.2, 0.2, 0.1, and 0.1 for the alias, dob, address, height, and country fields, respectively.

While the doc_score function has built-in similarity functions for many core field types, a custom similarity function can be provided at query time. In this manufactured example, we'll use a simple script_score function that matches CANADA and USA with a high score. Refer to the Elasticsearch documentation for more details about Elasticsearch scripting. Any other function can also be used.

curl -XGET 'http://localhost:9200/rni-test/_search' -H'Content-Type: application/json' -d '{
    "query" : {
        "bool" : { 
            "should" : [ 
                { "match" : { "name" : "Brian McDonough" } }, 
                { "match" : { "dob" : "10/19/87" } },
                { "match" : { "address" : "{ \"houseNumber\" : \"48\", 
                                             \"road\" : \"Parker St\", \"city\" : \"Boston\", 
                                             \"state\" : \"MA\" }" } } 
            ]
        }
    },
    "rescore" : {
        "query" : {
            "rescore_query" : {
                "function_score" : {
                    "doc_score": {
                        "fields": {
                            "name": { "query_value": "Brian McDonough", "weight": 4 },
                            "dob": { "query_value": "10/19/87", "weight": 2 },
                            "address" : {
                                  "query_value" : { 
                                      "houseNumber" : "48", 
                                      "road" : "Parker St", 
                                      "city" : "Boston", 
                                      "state" : "MA" 
                                   },
                                   "weight" : 2
                            },
                            "height": { "query_value": 67, "weight": 0.5 },
                            "nationality": { 
                               "function": {
                                   "function_score": {
                                       "script_score": {
                                           "script": {
                                               "lang": "painless",
                                               "params": {
                                                   "query_value": "CANADA"
                                               },
                                               "inline": "if (params.query_value == '\''CANADA'\'' &&
                                                doc['\''nationality'\''].value == '\''USA'\'') {return 0.8} 
                                                else {return 0.2}"
                                           }
                                       }
                                   }
                                },
                                "weight": 1
                            }
                        }
                    }
                }
            },
            "query_weight" : 0.0,
            "rescore_query_weight" : 1.0
        }
    }
}'

The POST_stopword adds one or more stop words. The entity_type field is optional, but the other fields are all required.

curl -XPOST "http://localhost:9200/rni_plugin/_stopwords" -H 'Content-Type: application/json' -d '{
    "lang": "eng",
    "stopword_type": "prefixes",
    "entity_type": "PERSON",
    "stopwords": [
        "honorable",
        "senior correspondent"
    ]
}'

The GET _stopwords method returns all stop words for a given language and stop word type. You can search by just language or by language and type.

When no entity type is specified, the stop word is applied to all names in the language, those with and without entity types. Therefore, calls that specify a type such as PERSON or ORGANIZATION will also return all stop words that don't have an entity type specified.

Returns all prefix stop words for PERSON types in English:

curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON"

Returns all regex stop words for ORGANIZATION types in Spanish:

curl -XGET "http://localhost:9200/rni_plugin/_stopwords/regexes_spa_ORGANIZATION"

Returns all prefix stop words in English with no type specified. For some languages, this list is empty by default. In these cases, data will only be returned if you've populated the file with values:

curl -XGET "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng"

The DELETE _stopwords method deletes a specified stop word. Deleting a stop word from a specific profile will also delete it from the any profile.

curl -XDELETE "http://localhost:9200/rni_plugin/_stopwords/prefixes_eng_PERSON/doctor"

The override index must exist before you can start adding token overrides. To create the index:

curl -s -XPOST "localhost:9200/rni_plugin/_overrides/_create"

To force a refresh of the dynamic override index:

curl -s -XPOST "localhost:9200/rni_plugin/_overrides/_refresh"

The POST _overrides adds one or more token overrides. As shown in the table above, entity_type, force, and score are optional, but the other fields are required.

curl -XPOST "http://localhost:9200/rni_plugin/_overrides" -H 'Content-Type: application/json' -d'{
        "lang1": "eng",
        "lang2": "eng",
        "entity_type": "PERSON",
        "token_pairs": 
        [{
        "token1": "Abigail",
        "token2": "Abbey",
        "score": 0.74,
        "force": true},
        {
        "token1": "Aleksander",
        "token2": "Alex",
        "score": 0.74},
        {
        "token1": "Alfonso",
        "token2": "Alphonse",
        "type": "COGNATE"},
        {
        "token1": "Frederica",
        "token2": "Federica",
        }]}'

The GET _overrides method returns the overrides of a given language profile.

curl -XGET "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON"

You can also retrieve the score of a given override pair.

curl -XGET "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON?token1=abigel&token2=abigail"

The DELETE _overrides method deletes a given override pair. Deleting an override from a specific profile will also delete it from the any profile.

curl -XDELETE "http://localhost:9200/rni_plugin/_overrides/hun_eng_PERSON/abigel+abigail"

The POST _parameter_universe method creates a parameter universe and the parameter profiles within the universe. Use this method to add or update a parameter value in a parameter universe. If you try to add a parameter universe that already exists, it overrides it with the new values. The parameter universe method uses the following syntax:

SomeParameterUniverseName/xxx_yyy where xxx_yyy is the language profile the parameters belong to, expressed in ISO 639-3 codes. The parameters field expects a list of parameters for the given profile, where the naming of the parameters should match the ones declared in parameter_defs.yaml.

curl -XPOST "http://localhost:9200/rni_plugin/_parameter_universe" -H 'Content-Type: application/json' -d'
  {
    "profiles": [
      {
        "name": "SomeParameterUniverseName/any",
        "parameters": {
          "translatorResultsToKeep": 4,
          "deletionScore": 0.269,
          "doQueryTokenOverrides": true,
          "fieldDeletionScore": 0.27,
          "yearDistanceWeight": 0.2
        }
      },
      {
        "name": "SomeParameterUniverseName/eng_eng",
        "parameters": {
          "HMMUsageThreshold": 0.8,
          "stringDistanceThreshold": 0.1,
          "useEditDistanceTokenScorer": true,
          "finalBias": 2.4,
          "reorderPenalty": 0.2
        }
      }
    ]
  }'

The GET _parameter_universe method retrieves parameter universes.

To retrieve a given parameter universe, the name of the parameter universe is provided as a path parameter:

curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"

If you include the name of the profile and a parameter, it returns the value of the parameter:

curl -XGET "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"

The DELETE _parameter_universe method deletes parameter universes.

To delete a specific parameter universe:

curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName"

To delete a parameter for a specific profile within a parameter universe:

curl -XDELETE "http://localhost:9200/rni_plugin/_parameter_universe/SomeParameterUniverseName/eng_eng.reorderPenalty"