Match Identity

The natural language processing algorithms employed by RNI use machine learning and cutting-edge NLP techniques to perform name matching. The match scores produced are a relative indication of how similar two names are, or a search name is to a name in an index; the higher the score the stronger the match. Customizations are available to tune and configure RNI to fit your business and data.

There are two common usage patterns in name and address matching: pairwise and index.

Index matching includes pairwise matching. When querying an index RNI performs a two-pass search:

RNI can match names in any language. For the languages listed in Fully supported text domains for name matching, RNI calculates a match score using a variety of techniques, as described in Understanding name match scores. For names not listed in those tables, RNI provides limited support, as described in Language support parameters.

This guide provides information on installing, running the sample applications included in RNI-RNT, setting up a development environment, and creating applications that use the runtime environment to incorporate RNI-RNT functionality.

The Java API is documented in HTML Javadoc pages generated from the source code, found at api-reference/index.html .

For instructions on using RNI-RNT with RLP, see Installing RLP with RNI.

You must install an SDK package that is appropriate for your platform with respect to operating system and CPU. Since the public API for RNI-RNT is Java, the C++ compiler that appears in the following list is irrelevant.

The compressed SDK package file names take the form:

rni-rnt-<version>-sdk-$BT_BUILD.<ext>

where <version> is the RNI-RNT version ( x.xx.x.cxx.x is the format), $BT_BUILD is in the table above, and <ext> is .zip for Windows or Java-only, and tar.gz for Unix platforms.

When you obtain RNI-RNT, you should receive the following files:

Expand the SDK into the install directory, which we will call $BT_ROOT, and copy the license to the $BT_ROOT/rlp/rlp/licenses subdirectory.

Once you have installed RNI-RNT, you can install RLP. See instructions for Installing RLP with RNI-RNT .

\rlp\bin\*

RNI uses the Logging Facade for Java (SLF4J) to log RNI activities. See http://www.slf4j.org/.

SFL4J is a facade for various logging APIs. Using SFL4J, the developer or an administrator can determine which one of many popular logging systems to use at runtime.

This is done by including one and only one adapter jar on the classpath, such as slf4j-log4j-1.17.36.jar, for the logging system of your choice, and the jar for that logging system (such as log4j-2.19.0.jar). You also need to include the SLF4J API jar, slf4j-api-1.17.36.jar, on the classpath.

By default, all activity is logged to the console. To log to a file and to control the level of logging, place an adapter jar, a logging library, an SLF4J API jar, and the appropriate properties file (e.g., log4j.properties if you are using log4j) on your classpath.

The adapter, logging, and API jars mentioned above are in samples/java/lib. A copy of log4j.properties, which is used by our samples, is in samples/java/logging. You should adjust the copy of log4j.properties that you place on your classpath to meet your specific runtime logging needs.

RNI uses libpostal to parse addresses; libpostal is a C library for parsing/normalizing street addresses around the world using statistical NLP and open data.

RNI packages libpostal data in plugins/rni/bt_root/rlpnc/data/libpostal. The data directory is relatively large (~2G). If you are certain that you won't be utilizing address matching of unfielded addresses, you can safely delete the libpostal data directory without impacting any other RNI functionalities.

If you are using both RLP and RNI-RNT together, first install RNI-RNT. Next, install RLP in the same location. When you are installing RLP there will be some overlap of directories, which is expected. Allow the RLP directories and files to replace the existing directories and files.

When building or running an RNI-RNT application, you must include the following JAR files on your classpath:

If using the seq2seq model for Katakana-English matching, you must also include the following JAR files on your classpath:

These files are in $BT_ROOT/rlpnc/lib/jvm.

For information about $BT_ROOT (the Basis root directory) and $BT_BUILD (the platform designator), see Installing RNI-RNT.

To use the Ant scripts described in Building and running the sample applications, make sure you have Ant (1.7.1 or later), the JAVA_HOME environment variable is set to the root of your Java SDK, and the Java SDK bin directory is on your PATH.

The API provides two ways of performing this action:

You can also set up an overlay directory. This directory must have an identical structure to the normal root directory outside of the rlp/lib and rlp/bin directories. License files will only be considered from BT_ROOT and should not be moved over to the overlay root.

Before you use Rosette Name Translator (RNT), you must instantiate a com.basistech.rnt.RNTEnvironment object. For example:

RNTEnvironment rntEnv = new RNTEnvironment();

The RNTEnvironment uses data files stored in the file system according to the standard RLP release hierarchy. Accordingly, you must set the Basis root directory prior to instantiating RNTEnvironment.

If your RLP license is not found in the appropriate location (rlp/rlp/licenses/rlp-license.xml) under your BT_ROOT directory, RNIConfiguration and RNTEnvironment include a setLicenseXML() method that you can use to provide the license as a string.

When you have finished performing translations, you should close the RNTEnvironment object to free resources. For example:

rntEnv.close();

When you use Rosette Name Indexer, RNI-RNT instantiates an RNTEnvironment object as required. If RNI-RNT instantiates an RNTEnvironment object, it also closes it at the appropriate time.

To build and run the sample applications, you must have the Java SDK (11 or later). To use the Ant build files we provide to build and run the samples, you need Ant (1.7.1 or later) with the JAVA_HOME environment variable set to the root of your Java SDK. For more information, see http://ant.apache.org.

The source files for these applications and the Ant build file for compiling and running them (build.xml) are located in $BT_ROOT/rlpnc/samples/java.

Change directory to $BT_ROOT/rlpnc/samples/java and run Ant:

ant -Dbt.arch=$BT_BUILD target

where target is one of the Ant build targets in the following table.

As you create your own applications, you can use the Ant build file as the starting point for establishing your own build procedures.

Names are complex to match because of the large number of variations that occur within a language and across languages. RNI breaks a name into tokens and compares the matching tokens. RNI can identify variations between matching tokens including, but not limited to, typographical errors, phonetic spelling variations, transliteration differences, initials, and nicknames.

RNI scores range from 0 to 1. The higher the score, the greater the confidence that this a relevant match. A score of 1.0 indicates that the query name string and result name string are identical (including all name properties).

The match score is a relative indication of how similar the match is; it is not an absolute value. When comparing different name matches, the relative matches of the scores are more relevant than the actual score. Similar name matches in different languages may generate different match scores. To understand how RNI calculates the score, see Understanding name match scores.

Scores less than 1.0 for similar names indicate the query name and index name vary with respect to one or more properties (such as language of origin) and/or one or more of the following:

Scoring is commutative: the scores for two given names are always the same, regardless of which name is in the index and which name is in the query.

You can configure RNI to customize how it scores different match phenomena.

The score weighting associated with a token may vary depending on the token's characteristics, such as the frequency with which it appears in the language model (the more frequent, the lower the weighting).

The entityType field identifies the type of name being matched and to select the algorithms to use for matching. Where supported, stop words and override files are specific to an entity type. Parameters can be set for specific languages and entity types.

By using a string array (such as String[] nameData = {"John", "Smith"};), you can create a name with data fields. The maximum number of data fields is 5. We assign no explicit semantics to each field (such as given name or surname), but the order of the fields does matter when comparing two names that have fields. RNI assigns lower scores to matches that cross field boundaries (e.g., the first field in one name matches the second field in another name). The use of fields may enhance accuracy when you are performing queries and matches with PERSON names in languages where standard name ordering is not the norm. By dictating a consistent name ordering, you can avoid penalties for mis-ordered tokens.

For consistency, you may want to adopt a paradigm for name fields, such as {title, given names, surname, suffix}. Include empty fields in the appropriate position for names that do not contain all these elements. If a trailing field is empty, you can leave it out. For example:

Identify two names to compare. They may be in different languages (languages of use) and writing scripts.

Use MatchScorer to score the similarity of two Name objects. MatchScorer and Name are in the com.basistech.rni.match package.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/match_2names.java

For the Arabic name نايف أبو شرخ and its IC transliteration Nayif Abu-Sharakh, this comparison returns a score of 0.99.

If you want to compare one name to many names, for improved efficiency you can cache the scorer with the one name (the query name) and used the cached scorer to compare that name to multiple names. As illustrated in the following code snippet, you must prepare each name that you use with the cached scorer.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/match_1name_tomany.java

For a sample Java application that matches two names and matches a query name against multiple reference names, see MatchNamesSample.

The default values of the RNI match parameters are tuned to perform well on most queries and datasets. However, every use case uses different data with distinct match requirements. You can modify match parameters to optimize match results for your data and business case.

The typical process for tuning parameters is as follows:

parameterUniverseOne/spa_eng_PERSON:
    reorderPenalty: 0.4
    HMMUsageThreshold: 0.8
    stringDistanceThreshold: 0.1
    useEditDistanceTokenScorer: true
parameterUniverseOne/eng_eng:
    reorderPenalty: 0.6
    

zho_zho_PERSON:  
  haniFourCornerCodeMismatchPenalty: 1

RNI includes override files (UTF-8 encoded) to improve name matching. There are different types of override files:

The name matching override files are in the plugins/rni/bt_root/rlpnc/data/rnm/ref/override directory.

You can modify these files and add additional files in the same subdirectory to extend coverage to additional supported languages. You can also create files that only apply to a specified entity type, such as PERSON.

stopregexes_LANG[_TYPE].txt

^fnu\b
\blnu$ 

\blnu$    2
\blnu$    3

stopprefixes_LANG[_TYPE].txt 

fullnames_LANG1_LANG2[_TYPE].txt

name1 Tab name2 Tab score

John Doe	Joe Bloggs	1.0

外山恒	   Toyama Koichi    1.0
ヒラリークリントン    Hillary Clinton    1.0

tokens_LANG1_LANG2_[TYPE].txt

Token1 Tab Token2 Tab [[0.0-1.0]|NICKNAME|COGNATE|VARIANT]

Token1 Tab Token2 Tab [([0.0-1.0]|NICKNAME|COGNATE|VARIANT)/force]

Peter    Pete    NICKNAME
Peter    Pedro   COGNATE

equivalenceclasses_LANG_[TYPE].txt

[normal_form1]
variant1_1
variant1_2
variant1_3
[normal_form2]
variant2_1
variant2_2
variant2_3
...

[muhammad]
mohammed
mahamed
mohamed
mohamad
mohammad
muhammed
muhamed
muhammet
muhamet
md
mohd
muhd

Organizations and companies often have nicknames which are very different from the company's official name. For example, International Business Machines, or IBM, is known by the nickname Big Blue. As there is no phonetic similarity between the two names, a match query between those two organization names would result in a low score. A real world identifier associates companies, along with their associated nicknames and permutations, with an identifier. When enabled, a search between two company names will include a comparison between the real world identifiers for the two names, thus matching dissimilar names for the same corporate entity.

RNI contains real world identifiers for corporations, which pair an entity id with nicknames and common permutations of the corporation name. Name matching within a language lists the languages with provided real-world ID dictionaries. Customers can also generate their own real-world ID dictionaries to supplement the provided dictionaries.

IBM    WE1X92
Big Blue    WE1X92
International Business Machines    WE1X92

IBM    Q37156
Nintendo    *
*    Q45700

You can train a language model on your own name data. RNI uses language models in which common names score differently than rare names. For example, "John Jingleheimer" should match "Jingleheimer" better than "John", because Jingleheimer is a rarer name than John. RNI already comes with language models for many supported languages, but you might find it best to train a new language model so that it reflects the statistics of your data. Please note that a large amount of full names are required to train an effective language model.

Installation

Unpack frequencyModelTrainer.zip to any desired location. Ensure that the JAVA_HOME environment variable is set and points to a Java version of 11 or higher.

Simple usage example

bin/buildLM.sh -root rni-rnt -in eng_PER_LM.tsv 
-out rni-rnt/data/rnm/ref/user_models/eng_PERSON_unigram.bin
-lang eng -script Latn

See README.txt in frequencyModelTrainer.zip for more details, including the full description of arguments.

A name index is an indexed list of names. The list includes a collection of Name objects and associated keys.

The Name object includes the name, language, ^[10] script, (script and language will be inferred if not included in the name definition) and may include entity type (such as person or place), language of origin, and additional information (with place names, for example, you may want to store the geocoordinates).

To create an indexed list of names on disk, you must specify a pathname for the data store, and you must use a IndexStoreDataModelFlags object (the default is fine).

Example:

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/create_index.java

Once the index is created, use NameBuilder to create Name objects and add them to the index. NameBuilder provides a fluent interface that supports method chaining. The following fragment illustrates the syntax for creating and adding a name to the index.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/add_name.java

When you are finished adding names, close the name index, as in the preceding fragment.

When you are adding a large number of names to an index, you can use an INameIndexSession object to batch these additions into a single transaction. A single transaction is faster than adding each name in a separate transaction. For information, see RNI Sessions and Transactions, and for a sample application that adds multiple names in a single transaction, see AddNamesSample.

Once you have an index created, you can use queries to search the index for similar names.

INameIndex index = StandardNameIndex.open(String indexPathname);

index.optimize();

index.close();

// Define a query.
NameIndexQuery defineQuery(Name queryName)
 throws NameIndexException, NameIndexStoreException, RNTException {
 NameIndexQuery query = new NameIndexQuery(queryName);
 query.setNameDataMinimumMatchScore(.30);
 return query;
}

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/query_index.java

index.close();

You may want to retrieve a group of names that share some common characteristic other than name similarity. Perhaps you even want to retrieve all the names in an RNI index.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/name_groups.java

The query returns all the names for which the Extra field contains the token used in the query.

By adjusting NameIndexQuery parameters, you can optimize queries for your use case.

In addition to using the INameIndex API for performing operations on an RNI Index, you can use the INameIndexSession API for finer-grained control. Sessions allow a set of operations to happen atomically (all occur or nothing occurs), and, especially for write operations, more efficiently. For those familiar with relational databases and SQL, the RNI concept of a session is similar to the JDBC concept of a connection with auto-commit mode off.

While INameIndexSession provides many of the same operations as INameIndex, such as query() and addName(), the difference is when changes to the index become permanent. INameIndex update operations are immediately flushed to disk, but INameIndexSession operations are not made permanent until you call commit(). At any time, you can invoke rollback() to undo all the operations since the last commit(). If you call rollback() before ever calling commit(), all of the operations of the session are undone.

You can run multiple sessions concurrently by having multiple threads call openSession() on the same INameIndex object. When multiple sessions are acting concurrently in separate threads, they are logically isolated from each other in order to not interfere with each other's operations. The isolation level is equivalent to READ COMMITTED, as outlined in the SQL-1992 Specification. This guarantees that one session will not see any uncommitted changes to the index performed by another session. In addition, a session will not see any uncommitted changes that it has made itself. For example, if a session adds a name to the index and then searches for that name before committing, it will not find the name it has added. You can also perform INameIndex auto-commit operations in the midst of one or more sessions; each INameIndex update or query is performed in its own session.

The session objects themselves are thread-safe; a session object may be shared by multiple threads.

The INameIndexSession API is recommended for doing bulk adds to the index. It is much more efficient to create a single session for adding all the names of a bulk add than to use the INameIndex API. The following fragment shows an example.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/add_names.java

A Sample. For a sample application that adds multiple names in a single transaction, see AddNamesSample.

Local vs. Distributed Transactions. A local transaction is a set of operations performed atomically (all occur or nothing occurs) on a single index. A distributed transaction is a set of operations performed atomically on multiple data sources, such as a relational database and an RNI index. All the operations on all the data sources must take place, or none of the operations take place.

For local transactions, use the INameIndexSession API, as illustrated above. The transaction object is managed internally and is not visible to the user.

In order to participate in a distributed transaction, an INameIndexTransaction object must be created from the session by calling INameIndexSession.startTransaction(). This transaction object is linked with the session internally. There is a division of labor between the two objects: the session object can only be used for adding/removing/searching, and the transaction object can only be used for committing or rolling back. A typical use case would be to provide the session object to the user application while handing over the transaction object to a transaction manager.

One side effect of this division of labor between the session and transaction objects is that a session cannot call commit() or rollback() once it is associated with a distributed transaction. These operations are only allowed by the linked transaction object. Specifically, after calling INameIndexSession.startTransaction(), you should not call INameIndexSession.commit(). You must call INameIndexTransaction.commit() instead.

A session can be associated with multiple distributed transactions, one at a time. When the work for one transaction is finished, you may call INameIndexSession.startTransaction() again to start a new one.

Two-Phase Commit. INameIndexTransaction supports two-phase commits, a standard protocol for managing transactions robustly among multiple data sources. INameIndexTransaction provides the prepare(), commit(), and rollback() operations necessary for a transaction manager to effectively execute the protocol. RNI does not include a transaction manager.

The following simplified example illustrates the use of INameIndexTransaction in a distributed transaction with a two-phase commit. In this example, both transactions are RNI transactions.

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/distributed_transaction.java

A Sample. For a sample application that illustrates a distributed transaction with a two-phase commit involving two RNI indexes, see DistributedTransactionSample.

Many companies have their own file of organizations with their different names. To improve matching between organization names, you can supplement the real world IDs provided in RNI and build your own file of real world IDs. The provided file will build a binary file in the specified output directory named <LANG>_ORGANIZATION_ids.bin where <LANG> is the three-letter language code of the file.

The input file is a tab separated file (.tsv). Each line contains an organization name and a corresponding alphanumeric ID. The file can only contain a single language and script. You must create a separate file for each language.

IBM    WE1X92
Big Blue    WE1X92
International Business Machines    WE1X92

Unzip the file realWorldIDBuilder.zip found in the plugins/rni/bt_root directory and run the build command. Instructions on how to run the program are in the README.md file in the zip file.

You may want to use real world ID matching even if there are some entities which you do not want to match via real world IDs. You can omit specific organizations and QIDs (Wikidata's identifier for entities) from matching by creating an omit file listing the organization names and QIDs you would like to omit.

The omit file is a tab separated file (.tsv) named <LANG>_ORGANIZATION_ids.tsv where <LANG> is the three-letter language code of the file. Each omit file can only contain names in one language and separate files must be made for each language. There are three types of lines that can appear in an omit file, which have different effects on omission: pairs, lone names, and lone QIDs.

Example:

IBM    Q37156
Nintendo    *
*    Q45700

To enable an omit file in RNI:

Addresses can be defined either as a set of address fields or as a single string. When defined as a string, the jpostal library^[11] is used to parse the address string into address fields.

When entered as a set of fields, the address may include any of the fields in Table 19, “Supported Address Fields”. At least one field must be specified, but no specific fields are required.

RNI optimizes the matching algorithm to the field type. Named entity fields, such as street name, city, and state, are matched using a linguistic, statistically-based algorithm that handles name variations. Numeric and alphanumeric fields, such as house number, postal code, and unit, are matched using character-based methods.

Identify two addresses to compare.

Use MatchScorer to score the similarity of two AddressSpec objects. MatchScorer and AddressSpec are in the com.basistech.rni.match and com.basistech.rni.match.address packages respectively.

// Use MatchScorer to match two addresses.
void match2Addresses(AddressSpec addr1, AddressSpec addr2) {
    MatchScorer ms = new MatchScorer();
    double score = ms.score(addr1, addr2);
    // Handle the score.
    System.out.println("Score: " + score);
    // Release resources used by the match scorer.
    ms.close();
}

The address match score is a value between 0.0 and 1.0; the higher the score, the stronger the match. The score is a relative indication of how similar two addresses are; it is not an absolute value. Calculating the match score is a complex process that utilizes multiple matching techniques and algorithms, as explained below.

To start tuning the parameters, run address matching on the test set and look for any unexpected results. Tunable parameters are defined in parameter_defs.yaml. The parameter files are described in Parameter configuration files.

An example parameter to tune is addressJoinedTokenLimit, which controls leniency towards joining or separating tokens. For some use cases, you may decide that joining many tokens within a field is acceptable. To adjust this parameter, find an existing parameter profile or define a new one, add the parameter and modify the value. By increasing the parameter value, the addressJoinedTokenLimit will be allowed to merge more tokens.

Another example parameter is houseNumberAddressFieldWeight, which controls the weight of the houseNumber score when calculating the overall score. This type of parameter is available for all address fields, and is weighted evenly at 1 by default. For example, cityAddressFieldWeight controls the weight of the city field when matching addresses.

Once you define a profile and set a parameter value, rerun the address pairwise match, scoring the match with the edited parameter_profiles.yaml file.

RNI uses stop patterns and stop word prefixes to remove patterns from address fields during indexing and queries before matching algorithms are applied. Using string literals to strip prefixes can be performed more quickly than the application of stop patterns (regular expressions), so you should use stop words for the efficient removal of prefixes, such as the, that you do not want to include in address matching.

For each address field, RNI performs the following steps in order:

stopregexes_LANG_ADDRESS_FIELD__FIELD.txt

stopprefixes_LANG_ADDRESS_FIELD__FIELD.txt

You can create text files that specify token (address field element) pairs that match. Token pair overrides are supported for English-English, Chinese-English, and Chinese-Chinese. When RNI evaluates two address fields, each of which contains an element from the pair, it enhances the value of the resulting address match score. For example, if road and rd constitute a token pair, then the match score for Stuart Road and Stuart Rd will be higher than it would be if the token pair had not been specified.

The token pairs may be within a language or cross-lingual, as indicated by the file name:

LANG1_LANG2_FIELD.txt

where LANG1 is the three-letter language code for the first token in each pair, LANG2 is the three letter language code for the second token in each pair, and FIELD is the AddressField name. Each entry in the file, except for rows that begin with #, is a tab-delimited token pair and may include a raw score between 0.0 and 1.0. If no score is provided, the addressOverrideDefaultScore parameter value will be used.

Token1 Tab Token2 Tab [0.0-1.0]

A token pair override score serves as a minimum score, but you can write /force after a token score to force it to be exactly that value:

Token1 Tab Token2 Tab [0.0-1.0]/force

If you would like to prevent a token pair from matching, you can use the SUPPRESS indicator as an alias for "0.0/force".

RNI includes plugins/rni/bt_root/rlpnc/data/addresses/ref/override/eng_eng_state.txt, which contains a list of U.S. state abbreviations. For example:

Massachusetts  MA
California  CA

When you create an additional file in the same location, use the respective AddressField name in the filename to identify the address field each token element in the pair pertains to. For example zho_eng_cityDistrict.txt indicates that the contents match Chinese - English cityDistrict address fields.

An address index is an indexed list of addresses. The list includes a collection of AddressSpec objects and associated keys.

The AddressSpec object may include house, house number, road, unit, level, staircase, entrance, suburb, city district, city, island, state district, state, country region, country, world region, post code, post office box and additional fields.

To create an indexed list of addresses on disk, you must specify a pathname for the data store.

For example:

// Create an Address index.
// indexPathname specifies the directory where the index will be created.
StandardAddressIndex createIndex(String indexPathname) throws NameIndexStoreException,
        RNTException {
    StandardAddressIndex index = StandardAddressIndex.create(indexPathname);
    return index;
}

Now you can use AddressSpecBuilder to create AddressSpec objects and add them to the index. AddressSpecBuilder provides a fluent interface that supports method chaining.

You can also create an AddressSpec object by parsing an address using AddressSpecBuilder.parse(String str) which internally utilizes the jpostal library. The following fragment illustrates the syntax for creating and adding an AddressSpec to the index.

// Add an address to the index.
void addAddress(StandardAddressIndex index, Integer id) throws NameIndexException, IOException {
    // Give the address a unique identifier. Must be a string.
    String uid = Integer.toString(id);
    // AddressSpecBuilder provides methods for adding address fields,
    // and a build method that returns the AddressSpec.
    AddressSpec addr = new AddressSpecBuilder()
            .house("101")
            .road("Stuart Street")
            .city("Boston")
            .state("MA")
            .countryRegion("New England")
            .uid(uid)
            .build();
// AddressSpecBuilder also provides a method for parsing addresses which uses jpostal,
// and a build method that returns the AddressSpec.
AddressSpec addr2 = AddressSpecBuilder.parse("101 Stuart Street, Boston, MA").build();

index.addAddress(addr);
index.close();}

When you are done adding addresses, be sure to close the address index, as in the preceding fragment.

You can define and run queries that search an index for similar addresses.

StandardAddressIndex index = StandardAddressIndex.open(String indexPathname);

index.optimize();

index.close();

// Define a query.
AddressIndexQuery defineQuery(AddressSpec address){
    AddressIndexQuery query = new AddressIndexQuery(address);
    query.setAddressDataMinimumMatchScore(.30);   
    return query;
}

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/address_query_index.java

index.close();

No more than one StandardAddressIndex object may exist for a given address index on disk at any time.

Queries and updates may be performed in multiple threads on a single StandardAddressIndex object.

A date contains a year, month, and day, but not all fields are required for matching. All common delimiters for English dates are supported, and dates can be expressed with various orderings. RNI will filter out some non-date related words. Formats that include time of day are not supported.

You can specify an Elasticsearch date format that includes time information in the mapping. The time component will be ignored.

RNI supports a wide variety of date formats. The best date format will always be the ISO standard of YYYY-MM-DD, where March 7, 1984 is written as 1984-03-07. RNI will attempt to interpret any date provided, although the less standard the format, the less guarantee that its interpretation will be the one you might expect.

Dates can be represented as YYYY-MM-DD. When some fields are unspecified, the letters represent the unknown values. For example, March 7 is YYYY-03-07, since the year in unspecified. Two digit years will be assumed to have unknown centuries. 3/7/84 is interpreted as YY84-03-07. March 7, 1984 will be an equally good match as March 7, 2084 and March 7, 1884.

When a date is provided, RNI will attempt to identify the year, month, and day within it, leaving blank any fields it cannot determine. You can omit fields if you do not have the value for one or more fields. For example: 1955-12-30, 1955--03, 12/30, -12-, --30, 1955, 1955-12- are all valid dates.

If RNI encounters an invalid date in an acceptable format, such as March 38, 1984, it will not return an error. Rather it will replace the impossible value as an unknown, March 1984.

Similarly to the name matching parameters, there are a series of date matching parameters. The parameter values can be edited in the plugins/rni/bt_root/rlpnc/data/etc/parameter_defs.yaml file.

Use RecordScorer to score the similarity of two records that include multiple field types, instead of the functions for a single type, such as the MatchScorer, DateScorer and AddressScorer functions for names, dates and address matching respectively.

The RecordScorer has default support for the RecordFieldType.RNI_NAME, RecordFieldType.RNI_DATE, and RecordFieldType.RNI_ADDRESS field types. All default similarity scores are between 0.0 and 1.0.

To index and search documents with RNI in a Solr application, you must add JARs to the Solr classpath, add the name fields to the schema.xml, and modify the solrconfig.xml.

 <!--Adjust the sharedlib setting if you move bt-rni-solr9x-plugins.jar to a different location.-->  
<str name="sharedLib">${bt.root}/rlpnc/data/rnm/sample/solr_shared_lib</str>

<fieldType name="bt_rni_name" class="com.basistech.rni.solr.NameField" needNameStore="true"/>

<fieldType name="bt_rni_addr" class="com.basistech.rni.solr.AddressField" needAddressStore="true"/>

<fieldType name="bt_rni_date" class="com.basistech.rni.solr.DateField" needDateStore="true"/>

<field name="primaryName" type="bt_rni_name" indexed="true" stored="true" multiValued="false"/>
<field name="aka" type="bt_rni_name" indexed="true" stored="true" multiValued="true"/>
<field name="residence" type="bt_rni_addr" indexed="true" stored="true" multiValued="false"/>
<field name="dateOfBirth" type="bt_rni_date" indexed="true" stored="true" multiValued="false"/>

<queryParser name="rniRerank" class="com.basistech.rni.solr.RNIReRankQParserPlugin"/>
<queryParser name="rniAddrRerank" class="com.basistech.rni.solr.RNIAddressReRankQParserPlugin"/>
<queryParser name="rniDateRerank" class="com.basistech.rni.solr.RNIDateReRankQParserPlugin"/>
<valueSourceParser name="rniMatch" class="com.basistech.rni.solr.NameMatchValueSourceParser"/>
<valueSourceParser name="rniAddrMatch" class="com.basistech.rni.solr.AddressMatchValueSourceParser"/>
<valueSourceParser name="rniDateMatch" class="com.basistech.rni.solr.DateMatchValueSourceParser"/>

<updateRequestProcessorChain name="RNIName">
  <processor class="com.basistech.rni.solr.MultiValueNameUpdateRequestProcessorFactory"/>
  <processor class="solr.LogUpdateProcessorFactory"/>
  <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

<requestHandler name="/update" 
      class="solr.UpdateRequestHandler">
  <lst name="defaults">
    <str name="update.chain">RNIName</str>
  </lst>
</requestHandler>

<updateRequestProcessorChain name="RNIAddr">
 <!--Custom processor required when using multivalued address fields-->
  <processor class="com.basistech.rni.solr.MultiValueAddressUpdateRequestProcessorFactory"/>
  <processor class="solr.LogUpdateProcessorFactory"/>
  <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

<requestHandler name="/update"
      class="solr.UpdateRequestHandler">
 <lst name="defaults">
   <str name="update.chain">RNIAddr</str>
 </lst>
</requestHandler>

<updateRequestProcessorChain name="RNIDate">
  <!--Custom processor required when using multivalued date fields-->
  <processor class="com.basistech.rni.solr.MultiValueDateUpdateRequestProcessorFactory"/>
  <processor class="solr.LogUpdateProcessorFactory"/>
  <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

<requestHandler name="/update"
      class="solr.UpdateRequestHandler">
 <lst name="defaults">
  <str name="update.chain">RNIDate</str>
 </lst>
</requestHandler>

<updateRequestProcessorChain name="RNI">
  <!--Custom processor required when using multivalued name fields-->
  <processor class="com.basistech.rni.solr.MultiValueNameUpdateRequestProcessorFactory"/>
  <!--Custom processor required when using multivalued address fields-->
  <processor   class="com.basistech.rni.solr.MultiValueAddressUpdateRequestProcessorFactory"/>
  <!--Custom processor required when using multivalued date fields-->
  <processor class="com.basistech.rni.solr.MultiValueDateUpdateRequestProcessorFactory"/>
  <processor class="solr.LogUpdateProcessorFactory"/>
  <processor class="solr.RunUpdateProcessorFactory"/>
</updateRequestProcessorChain>

<requestHandler name="/update"
      class="solr.UpdateRequestHandler">
 <lst name="defaults">
  <str name="update.chain">RNI</str>
 </lst>
</requestHandler>

bin/solr -a "-Dbt.root=$BT_ROOT -Djava.security.manager=allow" -m 2g

It is often necessary to query on other fields besides names fields, such as date of birth and address. The plugin enables the seamless integration of RNI into your Solr queries. To apply Boolean logic to queries, combine multiple fields with Boolean operators. The plugin supports all Boolean operators supported by the standard Lucene query parser (AND, OR, NOT, + , -). The OR operator is the default conjunction operator; if there is no Boolean operator between two terms (fields), the OR operator is used.

Example of a query with name and date fields:

primaryName:"Chuy Lopez A Deyas~entityType=PERSON" AND dateOfBirth:"1960-09-30"

Example of a query including an address field:

primaryName:"Chuy Lopez A Deyas~entityType=PERSON" AND residence:"road<Avenida Const. Pedro L Zavala 1957>
 house<Colonia Libertad>city<Culiacan>region<Sinaloa>postalCode<80180>country<Mexico>~fielded=true"

You can include name fields and other fields in your base query in conjunction with an RNI Solr reRank query and a custom valueSourceParser. The base query identifies candidate documents. The reRank query sends the top N candidates to the rniMatch valueSourceParser for pairwise matching. You can combine multiple fields in function queries which enable you to generate a relevancy score of those fields. The plugin supports all the functions available for function queries in Solr.

In a pairwise name match we return the maximum score of querying for primaryName and contactName:

&rq={!rniRerank reRankQuery=$rrq reRankMode=replace reRankWeight=1.0}
  &rrq={!func}max(rniMatch(primaryName, "Chuy Lopez A Deyas"), rniMatch(contactName, "Chuy Lopez"))

In a pairwise address match we return the maximum score of querying for primaryAddress and residence:

&rq={!rniAddrRerank reRankQuery=$rrq reRankMode=replace reRankWeight=1.0}
  &rrq={!func}max(rniAddrMatch(primaryAddress, "road<Calle Lago Cuitzeo 1394>house<Colonia Las Quintas>
 city<Culiacan>region<Sinaloa>postalCode<80060>country<Mexico>~fielded=true"),
rniAddrMatch(residence, "road<Avenida Const. Pedro L Zavala 1957>house<Colonia Libertad>
 city<Culiacan>region<Sinaloa>postalCode<80180>country<Mexico>~fielded=true"))

In a pairwise date match we return the maximum score of querying for dateOfBirth and dob:

&rq={!rniDateRerank reRankQuery=$rrq reRankMode=replace reRankWeight=1.0}
&rrq={!func}max(rniDateMatch(dateOfBirth, "01/07/42~format=dd/MM/yy"), rniDateMatch(dob, "1/7/1940"))

You can combine the score of multiple RNI fields of the same type where each field can be given a weight to reflect its importance in the overall matching logic.

For example, in a pairwise name match we can return the combined score of querying for aka and primaryName where aka has a weight of 0.3 and the remaining 0.7 is assigned to primaryName field:

&rq={!rniRerank reRankQuery=$rrq reRankMode=replace reRankWeight=1.0} 
  &rrq={!func}sum(linear(rniMatch(aka, "Jesus Alfonso Diaz"), 0.3, 0),
  linear(rniMatch(primaryName, "Jesus Diaz"), 0.7, 0))

q=primaryName:"Lopez Diaz"
fl=primaryName,aka,score
&rq={!rniRerank reRankQuery=$rrq reRankDocs=200 reRankWeight=3}
&rrq={!func}rniMatch(primaryName, "Lopez Diaz")

This example walks you through the steps for using the Solr 9 Admin example to perform queries.

Solr Admin displays a response.

For this query, Solr returns the appropriate Diaz document.

{
{
  "responseHeader": {
    "status": 0,
    "QTime": 387,
    "params": {
      "rrq": "{!func}rniMatch(name, \"Chuy Lopez A Deyas~entityType=PERSON\\")",
      "q": "name:\"Chuy Lopez A Deyas~entityType=PERSON\" AND dateOfBirth:[1960-09-30T00:00:00Z TO *]",
      "fl": "name,aka,dateOfBirth,address,nationality,score",  
      "_": "1631115490163",   
      "rq": "{!rniRerank reRankQuery=$rrq reRankMode=replace reRankWeight=1.0} "
    }
  },
  "response": {
    "numFound": 145,
    "start": 0,
    "maxScore":0.6820811,
    "numFoundExact":true,
    "docs": [
      {
        "name": "Jesus Alfonso LOPEZ DIAZ~uid=10353,entityType=PERSON",
        "address": [
          "c/o ESTABLO PUERTO RICO S.A. DE C.V.\nCuliacan Sinaloa\nMexico",
          "Avenida Const. Pedro L Zavala 1957\nColonia Libertad\nCuliacan Sinaloa 80180\nMexico"
        ],
        "nationality": [
          "Mexico"
        ],
        "dateOfBirth": [
          "1962-09-30T00:00:00Z"
        ],
        "score": 0.6820811
      },
      ...
    ]
  }
}

The RNI-RNT SDK ships with an example that illustrates the use of the org.apache.solr.client.solrj API to integrate the RNI Solr plugin into a Solr application. See RNISolrjSample. This sample also illustrates a procedure for posting Solr documents from an xml file.

You can use the org.apache.solr.client.solrj API to integrate the RNI Solr plugin into a Solr application.

The basic steps are as follows:

The following sample code snippets use these imports:

import org.apache.solr.client.solrj.SolrQuery;
import org.apache.solr.client.solrj.embedded.EmbeddedSolrServer;
import org.apache.solr.client.solrj.response.QueryResponse;
import org.apache.solr.common.SolrDocument;
import org.apache.solr.common.SolrInputDocument;
import org.apache.solr.common.params.CommonParams;
import org.apache.solr.core.CoreContainer;

Setup:

// Set the bt.root property to point to the RNI installation
String btRoot = args[0];
System.setProperty("bt.root", btRoot);
// Set solr.solr.home to the parent of a collection1/conf directory that contains
// a modified schema.xml and solrconfig.xml.
String solrHome = btRoot + "/rlpnc/data/rnm/sample/solr8x_home";
System.setProperty("solr.solr.home", solrHome);
CoreContainer coreContainer = new CoreContainer(solrHome);
coreContainer.load();
// For simplicity, use an embedded SolrServer rather than an HTTPSolrServer.
EmbeddedSolrServer server = new EmbeddedSolrServer(coreContainer, "");

Add a Solr document with fields of interest, including name fields:

SolrInputDocument doc = new SolrInputDocument();
// Primary name field
doc.addField("primaryName", "Midiam Patricia ZAMBADA NIEBLA");
// Multivalued also-known-as name field
doc.addField("aka", "Midian Patricia ZAMBADA NIEBLA");
doc.addField("aka", "Miriam ZAMBADA NIEBLA");
doc.addField("aka", "Midian Patricia LOPEZ LANDEY");
doc.addField("id", "3");
// Entity id field.
doc.addField("uid", "10358");
// Date field
doc.addField("dob", "1971-03-04");
// Address field
doc.addField("address", "road<Calle Lago Cuitzeo 1394>house<Colonia Las Quintas>"
             + "city<Culiacan>region<Sinaloa>postalCode<80060>"
             + "country<Mexico>~fielded=true");
doc.addField("nationality", "Mexico");

// Add the document to the index.
server.add(createInputDoc());

Commit updates:

// When you have completed updates, commit the updates.
server.commit();

Define and run a query against name and other fields, using RNI pairwise matching to rerank the documents returned:

// Define a query that combines name fields and other fields, and uses RNI
// pairwise matching to rerank the documents returned.
String queryName = "Chuy A Lopez";
SolrQuery solrQuery = new SolrQuery("aka" + ":\"" + queryName +
 "\" AND dob:[1960-09-30T00:00:00Z TO *]");
// Set the rerank query parser parameters
solrQuery.set(CommonParams.RQ,
 "{!rniRerank reRankQuery=$rrq reRankDocs=100 reRankWeight=1}");
// Create a rerank query that uses the RNI pairwise matching function
solrQuery.set("rrq", "{!func}rniMatch(" + "aka" + ", \"" + queryName + "\")");
// Set which fields to include in the results
solrQuery.setFields("uid", "primaryName", "address", "dob", "score");
QueryResponse qResults = server.query(solrQuery);
//QueryResponse qResults = server.query(createQuery());

Define and run a query against name, address, and other fields, using RNI pairwise matching to rerank the documents returned:

// Define a query that combines name, address and other fields, and uses RNI
// pairwise matching to rerank the documents returned.
String queryName = "Chuy A Lopez";
String queryAddress = "road<Avenida Const. Pedro L Zavala 1957>house<Colonia Libertad>"
           + "city<Culiacan>country<Mexico>~fielded=true";
SolrQuery solrQuery = new SolrQuery("aka" + ":\"" + queryName +
                    "\" AND dob:\"1960-09-30\"");
// Set the rerank query parser parameters
solrQuery.set(CommonParams.RQ,
        "{!rniAddrRerank reRankQuery=$rrq reRankMode=replace reRankDocs=100 reRankWeight=1}");
// Create a rerank query that uses the RNI pairwise matching function
solrQuery.set("rrq", "{!func}rniAddrMatch(" + "address" + ", \"" + queryAddress + "\")");
// Set which fields to include in the results
solrQuery.setFields("uid", "primaryName", "address", "dob", "score");
QueryResponse qResults = server.query(solrQuery);
//QueryResponse qResults = server.query(createQuery());

Define and run a query against name, date, and other fields, using RNI pairwise matching to rerank the documents returned.

// Define a query that combines name, address and other fields, and uses RNI
// pairwise matching to rerank the documents returned.
String queryName = "Chuy A Lopez";
String queryDate = "04/03/1971~format=dd/MM/yyyy";
SolrQuery solrQuery = new SolrQuery("aka" + ":\"" + queryName +
                    "\" AND dob:\"1960-09-30\"");
// Set the rerank query parser parameters
solrQuery.set(CommonParams.RQ,
          "{!rniDateRerank reRankQuery=$rrq reRankMode=replace reRankDocs=100 reRankWeight=1}");
// Create a rerank query that uses the RNI pairwise matching function
solrQuery.set("rrq", "{!func}rniDateMatch(" + "dob" + ", \"" + queryDate + "\")");
// Set which fields to include in the results
solrQuery.setFields("uid", "primaryName", "address", "dob", "score");
QueryResponse qResults = server.query(solrQuery);
//QueryResponse qResults = server.query(createQuery());

Display the results:

// Print information about the documents returned with their Solr score.
for (SolrDocument rdoc : qResults.getResults()) {
 System.out.println("Returned Entity: " + rdoc.getFieldValue("uid")+ 
       "\n Name: " + rdoc.getFieldValue("primaryName") + 
       "\n Address: " + rdoc.getFieldValue("address") + 
       "\n DOB: " + rdoc.getFieldValue("dob") + 
       "\n Document Score: " + rdoc.getFieldValue("score"));
}

For convenience utilities for working with RNI names in a Solrj environment, see the Javadoc for com.basistech.rni.solr.index.

Rosette Name Translator translates a name from one text domain to another. A text domain is specified by three parameters:

The source domain is the text domain of the document in which the name is found. The target domain is the text domain to which the name is to be translated.

Supported translation domains provides a list of supported source and target domains.

The type of translation depends on the characteristics of the source and target domains and the language of origin of the name to be translated. RNT supports the following types of translations:

With automated translation, the client provides one or more names, input and output text domains, and types of translation desired. For each name, the application generates a list of translations and associated confidence scores.

Automated usage model for performing RNT translations:

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/create_translator.java

translator.close();

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/translate.java

https://raw.githubusercontent.com/basis-technology-corp/rosette-sample-code/master/rni-rnt/inspect_translation_results.java

fullnames_SRCLANG_TARGETLANG[_TYPE].txt

tokens_SRCLANG_TARGETLANG[_TYPE].txt

source name Tab target name[ Tab target_script] [Tab confidence_score]

Ho Lide贺 利得Hans0.99

علي سعيد'Ali Sa'id

RNT provides an API that you can use to build interactive applications to translate Arabic, Iranian Persian, Afghan Persian, Pashto, Chinese, Korean, and Russian names from English or native script to standardized English. For supported transliteration schemes, see Supported Translation Domains.

The input is an Arabic, Iranian Persian, Afghan Persian, Pashto, Chinese, Korean, or Russian name in native script or in English that the user wants to translate. In the common case, the name is in English but may not conform to the desired transliteration standard.

To take full advantage of the resources that RNT Interactive provides, the user should have some familiarity with Arabic, Iranian Persian, Afghan Persian, Pashto, Chinese, Korean, and/or Russian.

For detailed information about the API, see the com.basistech.rnt.assistant package in the Java API Reference.

The first table identifies the languages, and for each language the writing scripts that Rosette Name Indexer fully supports.

This table identifies the range of cross-language searching and matching that Rosette Name Indexer and name matching fully support. If your query is a name in an Arabic document in Arabic script, the query may return one or more names in English documents in Latin script, in addition to names from Arabic documents in Arabic script. If the query is a name in English and Latin script, it may return documents from any of the supported languages and their native scripts.

The following table identifies the translations (target text domains) that Rosette Name Translator supports for each source text domain.

For each source domain, the target domains for each language and script combination are presented in a single row.

Translation. If the languages in the source domain and target domain do not match, Rosette Name Translator translates the name to the target language.

Language of Origin. For Arabic, Chinese, Hebrew, Japanese, and Korean, if the target language is English and the language of origin does not match the source language, Rosette Name Translator attempts to translate the name to its standard English representation. If the language of origin is the source language, Rosette Name Translator transliterates the name with the specified transliteration scheme. If the language of origin for the name object is unspecified (UNKNOWN), Rosette Name Translator guesses the language of origin. For the supported languages of origin for each domain pair, see Supported Languages of Origin.

Orthographic Enhancement. When the source and target domains match (the transliteration schemes are native), the translations are orthographic enhancements in the native script (vocalization in Arabic and Hebrew script languages, segmentation in Chinese, Japanese, Korean, or Thai).

Name Variants. When the target transliteration scheme is folk, Rosette Name Translator generates a list of variant representations of the name in Latin script.

The following table displays the languages of origin that are supported for each source and target domain. For the full English names for the scripts, languages, and transliteration schemes, see the preceding table.