Release Notes

Added new parameters, allowNullValue and ignoreBadData, in rlpnc/data/etc/parameter_defs.yaml. Both parameters will be used by the RNI elasticsearch plugin and are set to false by default.

The reorderPenalty parameter now controls an exponentially decaying penalty instead of a linear one, in order to improve matching of longer names. (RLPNC-3661)

Normalized some Katakana "small" characters into their full sized counterparts to improve Japanese name matching and translation. (RLPNC-5028)

Normalized Extension A Chinese characters into their variants to improve Chinese name matching and translation. (RLPNC-5045)

Fixed shading of Lucene 7 dependencies.

Added a frequency language model for LOCATION entity type and retrained the language model for PERSON entity type, improving Hungarian name matching. (RLPNC-5003, RLPNC-5004)

Upgraded RNI to use Lucene 7.6.0. (RLPNC-5000, RLPNC-5020)

Upgraded the RNI Solr plugin to support Solr 7.6.0. (RLPNC-5000)

Continued support for English address matching. (RLPNC-4352)

Added English address matching. (RLPNC-4329, RLPNC-4342, RLPNC-4351, RLPNC-4354, RLPNC-4812, RLPNC-4931, RLPNC-4941, RLPNC-4942, RLPNC-4943, RLPNC-4966, RLPNC-4968, RLPNC-4970, RLPNC-4973)

Improved Hungarian name matching, including support for multi-letter initials. (RLPNC-4958, RLPNC-4959, RLPNC-4989)

Continued improvement for English/Japanese and English/Chinese organization name matching. (RLPNC-4937, RLPNC-4938, RLPNC-4939, RLPNC-4940, RLPNC-4952, RLPNC-4962, RLPNC-4971)

Upgraded RNI to use Lucene 7.4. (RLPNC-4892)

Upgraded the RNI Solr plugin to support Solr 7.4. (RLPNC-4893)

Improved matching of Chinese organization names. (RLPNC-4925)

Continued support for Hungarian/English and Hungarian/Hungarian name matching. (RLPNC-4906)

Continued support for Hungarian/English and Hungarian/Hungarian name matching. (RLPNC-4744, RLPNC-4754, RLPNC-4821, RLPNC-4879, RLPNC-4886)

Fixed a bug involving incorrect order of setting entity type and language of origin which depends on the former during the process of name building. (RLPNC-4858)

Fixed a bug involving duplicate readings produced when transliterating chinese names. (RLPNC-4796, RLPNC-4838)

Fixed a bug which produced a confidence score of 0 for name translations. (RLPNC-4871)

Began adding support for Hungarian/English and Hungarian/Hungarian name matching. (RLPNC-4728, RLPNC-4734, RLPNC-4735, RLPNC-4736, RLPNC-4737, RLPNC-4738, RLPNC-4739, RLPNC-4740, RLPNC-4741, RLPNC-4742, RLPNC-4743, RLPNC-4745, RLPNC-4751, RLPNC-4757, RLPNC-4789, RLPNC-4818, RLPNC-4828, RLPNC-4840)

Added a detector that differentiates between Katakana of foreign origin vs. Katakana of Japanese origin, which improves accuracy for Japanese-English name matching. (RLPNC-4811)

Fixed the greek stop words file to get better matching. (RLPNC-4826)

Fixed the error which occurred if a stopprefix file was added for a language with no stopregex file. (RLPNC-4761)

Added a new parameter, includeExtraKatakanaPersonReadings, in rlpnc/data/etc/internal_param_defs.yaml. If true, it will include the foreign person name readings even when the language-of-origin is unknown. By default it is set to false. (RLPNC-4841)

Improved segmentation of Persian, Urdu, Pushto, and Dari names written in Arabic script. (RLPNC-4782)

No new features other than changing the versioning convention of the artifacts to include Basis' parent pom version. (RLPNC-4721)

Added co, corp, inc, corporation, and incorporated to stop word list for organizations in English. (RLPNC-4725)

Fixed issue in which the same file was being used multiple times at runtime. (RLPNC-4748)

Continued adding support for Greek by reviewing the match score adjustments. (RLPNC-4629)

Added a new parameter (see below) that controls the use of different segmentation schemes for Japanese. (RLPNC-4677)

Improved segmentation of Arabic names written in Arabic script. (RLPNC-4700)

Added a new parameter, useOldAndNewNameSegmentationForJapanese, that allows multiple segmentation schemes to be applied to Japanese names. (RLPNC-4677)

Adjusted the finalBias parameter for Greek/Greek and Greek/English name pairs to ensure scores for these matches are in line with those of other languages supported by RNI. (RLPNC-4629)

Adjusted parameters for matching names of Arabic origin to English names. (RLPNC-4700)

Added the capability to identify language of origin for Latin-script names. Currently this feature will categorize names as Arabic, Chinese, English, Japanese, and Korean. (RLPNC-4546)

Continued adding support for Greek by setting the default transliteration scheme to ISO-843, adding a Greek/English statistical model, gender identification, name overrides, language model, edit distance scoring, as well as adding support for initials and initialisms. (RLPNC-4622, RLPNC-4676, RLPNC-4628, RLPNC-4626, RLPNC-4651, RLPNC-4630, RLPNC-4631, RLPNC-4689, RLPNC-4674)

Improved the accuracy of language identification for Hani-script person names. (RLPNC-4532)

Improved segmentation of long Japanese katakana names. (RLPNC-4680)

Adjusted the reorderPenalty in the jpn_jpn_ORGANIZATION profile to 0.070 to improve performance for this profile. (RLPNC-4659)

Enabled useEditDistanceTokenScorer for Greek/Greek and Greek/English. (RLPNC-4665)

Improved the behavior of Western Farsi matching by tokenizing sooner in the process. (RLPNC-4576)

Added a new parameter (see below) that allows one to tune a penalty applied to pairwise match scores if the two names involved are of different lengths. (RLPNC-4554)

Added a new parameter (see below) that allows one to tweak the resulting match score of the case in which two identical names with unknown field are matched against each other. (RLPNC-4582)

Improved gemination rules of ISO11940-2 to improve Thai translation accuracy. (RLPNC-4579)

Relocated jar files distributed previously under rlp/lib/BT_BUILD to now be in a more central, platform-neutral location of rlpnc/lib/jvm. (RLPNC-4583)

Began adding support for Greek/English and Greek/Greek name matching. Note: Greek support is currently extremely minimal and will be more full in a future release. (RLPNC-4600, RLPNC-4611, RLPNC-4615, RLPNC-4621)

Fixed a bug involving the presence of digits in non-person names. Non-person names that contain digits will now have better match performance. (RLPNC-4590)

Added a new parameter, nameLengthMismatchPenalty, in rlpnc/data/etc/parameter_defs.yaml. The penalty is off (0) by default and only overridden for the zho_zho profile to be 0.55. Lower values will penalize the score less, and higher values will apply a more drastic penalty. As part of the work of adding this parameter, we also adjusted the zho_zho deletionScore from 0.250 to 0.314.

Adjusted the expensiveScorerJoinedTokenLimit in the jpn_jpn_VEHICLE profile to 5 to improve performance for this profile.

Added a new parameter, sameNameUnknownFieldMatchInterpolator, in rlpnc/data/etc/parameter_defs.yaml. This parameter affects a rare case in which two identical names with unknown fields are matched against each other. Usually, RNI would score identical names as 1.0. However, unknown tokens have their own penalty that applies, defined by unknownVsUnknownScore. This new parameter interpolates between the would-be score and 1.0. The default value is 1, which means that 1.0 will be returned in these cases. Turning the parameter to 0 will fall back to the would-be score, and values in-between will interpolate between these two numbers.

Added a new Thai/English statistical model for matching to improve Thai/English name match performance. (RLPNC-4429)

Added a new Thai name segmentation dictionary, improving segmentation and match performance. (RLPNC-4421)

Improved Thai transliteration, benefiting translation and match performance. (RLPNC-4547)

Enhanced the Thai stop word list, providing better stop word removal from Thai names during matching. (RLPNC-4461)

Tuned the finalBias value for Thai/Thai and Thai/English name pairs to ensure scores for these matches are in line with those of other languages supported by RNI. (RLPNC-4445)

Greatly improved Arabic/Arabic match performance by adding an edit distance metric. (RLPNC-4508)

Improved match performance for name pairs in which the names are identical when spaces are removed. (RLPNC-4495)

Added a few new entries to English/English token overrides. (RLPNC-4529)

Modified the names of RNI's internal Lucene fields so that they are simpler and standardized. (RLPNC-2506)

Added the ability to disable support for individual languages. See internal_param_defs.yaml in rlpnc/data/etc for more information on how to use this feature. (RLPNC-4558)

Removed support for Solr 5. (RLPNC-4531)

Added preview support for Thai in name matching and name translation. (RLPNC-4444, RLPNC-4420, RLPNC-4419, RLPNC-4417, RLPNC-4490, RLPNC-4424, RLPNC-4423, RLPNC-4493, RLPNC-4479, RLPNC-4418)

Upgraded RNI to use Lucene 6.6. (RLPNC-4292)

Upgraded the RNI Solr plugin to support Solr 6.6. Removed support for Solr 4. (RLPNC-4450, RLPNC-4456)

Changed default behavior of Chinese names during Chinese / English matching so that they are assumed to be of Chinese origin unless otherwise specified. (RLPNC-4375)

Improved Russian / English name matching in that Russian names now include multiple translations. (RLPNC-4496)

Greatly improved Chinese / Japanese organization name language detection. (RLPNC-4477)

Improved match performance when engEngFastMode is enabled. (RLPNC-4488)

Improved name matching to account for more substring matches. (RLPNC-4498)

Fixed a thread contention issue that could slow down large numbers of threads.

Upgraded the native libraries of the native Linux-compatible release of RNI. We are now using CentOS 6 to build these libraries, as CentOS 5 has reached EOL. The new BT_BUILD value for the Linux package is amd64-glibc212-gcc44. (RLPNC-4278)

Added a new config parameter, engEngFastMode which improves speed for English-English matching by turning off HMM and simplifying queries. For more information, check the documentation in internal_param_defs.yaml. (RLPNC-4357)

Included simple serialize and deserialize methods on Name and DateSpec. (RLPNC-4344)

Added two new config parameters, doQueryFuzzy and doQueryPhrase which affect components of RNI's internal Lucene queries. (RLPNC-4312)

Improved the automatic inference of BT_BUILD values. (RLPNC-4368)

Added static and deprecated attributes for config parameters. A "static" parameter is one that always has the value loaded in the default parameter profile; setting a static parameter to a different value in other profiles has no effect whatsoever. A "deprecated" parameter is one that we are proposing to eliminate; binding its value to anything other than the default results in a warning. (RLPNC-4193)

Improved the efficiency of when the HMM is used in the case of English-English name pairs. (RLPNC-2988)

Fixed the 'ignoreBadData' flag in RNICLI to function as intended. Also improved exception handling in RNICLI when the data of the input contains languages unsupported by RNI. (RLPNC-4319)

Reduced the size of the package through the word embeddings datafiles (RLPNC-4245)

Enhanced semantic matching of tokens in Organization names through use of word embeddings.

Note: This drastically increases the size of the SDK package. To reduce the size, the embeddings dictionaries in rlpnc/data/tvec/multilingual can be removed as long as the corresponding language pairs in parameter_profiles.yaml have 'useEmbedded' set to false. (RLPNC-4173, RLPNC-4201, RLPNC-4219, RLPNC-4244)

Added the ability for specific token overrides to always override the score between tokens even if a different method of matching generates a higher score. This can be used to prevent specific token pairs from matching. (RLPNC-3951)

Enhanced the segmentation of Japanese Organization names through decomposing compounds. (RLPNC-2910)

Improved the accuracy of fuzzy phonetic matching between Japanese and English. (RLPNC-2444)

Implemented support for multi-token "token" overrides. (RLPNC-4080)

Added token override dictionary for matching Organization names between Japanese and English. (RLPNC-4172)

Ensured compatibility with RLP version 7.15. (RLPNC-4190)

Upgraded Solr plugin to support Solr 6.2. (RLPNC-4179)

Fixed an issue introduced in 7.15.0 that caused a significant slowdown in English-English querying and matching. (RLPNC-4217)

Fixed an issue where NameBuilder.hintLanguage() would return the incorrect value. (RLPNC-4189)

Digits are no longer stripped from non-English Organization names. (RLPNC-4211)

Restored missing Katakana segmentation data. (RLPNC-4210)

Added support to RNI for Japanese-Chinese, Japanese-Korean, and Korean-Chinese name matching. (RLPNC-3900, RLPNC-4096, RLPNC-4095)

Added a new query parameter, namesToCheckAllowance, which sets the general proportion of names to pass to the high-precision filter. This is used at query time to determine the number of names to check based on the commonality of the query name in the index, allowing for more efficient querying. As a result, generally a higher setting of maximumNamesToCheck can be used. This involved making a breaking change to the INameIndexFilter interface. This behavior was also added to the Solr plugin with a parameter called reRankDocsAllowance. (RLPNC-4059, RLPNC-4150)

Added a new query parameter, scoreToCheckRestriction, that acts as a more efficient replacement for minimumScoreToCheck and improves query speed. The minimumScoreToCheck parameter has been deprecated. This behavior was also added to the Solr plugin with a parameter called scoreToRerankRestriction. (RLPNC-3665, RLPNC-4166)

Enhanced our Japanese name segmentation logic and expanded our inventory of Japanese segmentation data, improving the accuracy of both Japanese name translation and name matching. (RLPNC-4117, RLPNC-4118, RLPNC-4144, RLPNC-4145)

Enhanced the Lucene query logic in our first-pass filter to improve accuracy on sparse fuzzy queries. (RLPNC-4081)

Improved the speed of some non-English name matching by pruning unlikely translation alternatives. This is controlled by a new config paramater, alternativePairsToCheck(RLPNC-4138)

Added a new config parameter, queryAlternativeOriginLanguages, which controls the set of query languages where transliterations of alternate origins are made part of the query. If matching Chinese, Japanese, and Korean names this can be adjusted to improve accuracy at the cost of speed. (RLPNC-4158)

Changed CachedScorer to accept both completed and uncompleted names. Users should no longer have to complete a Name object. In any case, obtainCompletedName and checkComplete methods have been added to Name and the StandardNameIndex.completeName method has been deprecated. (RLPNC-4149)

Modified the method StandardNameIndex.generateHighRecallKeys to no longer mutate the given Name object. (RLPNC-4147)

Fixed an issue that could result in an NullPointerException when matching a name consisting solely of a fullwidth semicolon and likely other rare forms of punctuation. (RLPNC-4134)

Setting maximumNamesToConsider to Unlimited (eg. -99) will no longer result in a write lock exception when querying with multiple threads. (RLPNC-3589)

Special characters like '#' are now normalized out of Arabic names to prevent unwanted effects on match scores. (RLPNC-3961)

Completing a name no longer alters the original language-of-origin. This was occasionally causing Names to have different results when completed multiple times. Instead, Names have a derivedLanguageOfOrigin() method. (RLPNC-4132)

Fixed an issue that was causing the Solr 6 plugin to use the slower query that supports multivalued fields when not necessary. (RLPNC-4116)

Fixed a concurrency bug involving parameter profiles used by the HMM. Also fixed in 7.14.1. (RLPNC-4128)

Removed support for Java 1.7. Java 1.8 or higher is required. (RLPNC-4077)

Converted many thrown checked exceptions to runtime exceptions. This includes the removal of two exception classes, RNILicenseException and RNTLicenseException. This will likely require code changes. (RLPNC-3982)

Increased our inventory of segmentation and reading data of Japanese Kanji, increasing the accuracy of both Japanese name translation and name matching. (RLPNC-3985, RLPNC-4032, RLPNC-4084)

Improved RNI's handling of Japanese punctuation to better match how it normalizes English punctuation. (RLPNC-4030)

Added a small inventory of common organization and company words to RNI's Japanese-English overrides. (RLPNC-4064)

Added the ability to specify a reRankFilter in the Solrj utility code. (RLPNC-4057)

The rws-names web services have been deprecated in favor of the Rosette API and the Elasticsearch/Solr plugins. (RLPNC-4076)

Upgraded internally to Lucene 6.0. This may require changes to settings of minimumScoreToCheck. (RLPNC-4077)

Added plugin support for Solr 6.0. Support for Solr 4.x has been deprecated. (RLPNC-4078)

Improved public support and documentation for RNI match parameter tuning. (RLPNC-4068)

Added support for custom language model training for RNI. (RLPNC-4028)

Modified Rosette API RNT service to throw an exception when the language of the input name cannot be translated into the specified target language. (RLPNC-4086)

Fixed a concurrency bug involving parameter profiles used by the HMM. (RLPNC-4128)

Added plugin support for Solr 5.5. (RLPNC-3929)

Added a single parameter to turn off all case-dependent logic in RNI. (RLPNC-3963)

Improved the error message when attempting to match names in an unsupported language with the java only platform. (RLPNC-3997)

Modified the frequency weighting in fielded names. There is now a minimum total weight for each field regardless of how frequent are the tokens it contains. (RLPNC-3984)

Improved support for macro scripts (Eg. Jpan and Kore) in RuleSetTranslator. (RLPNC-3981)

Added method for determining successful installation and configuration of the RNI plugin in an Elasticsearch installation. (RLPNC-3890)

Removed dependency on btrlp.jar and btutil.jar. (RLPNC-3976)

Added support for user supplied low weight tokens. (RLPNC-3950)

Improved accuracy of RNI's frequency weighting by adding character level language model features. (RLPNC-3946, RLPNC-3969)

Improved RNI support for matching unknown field markers. (RLPNC-3949, RLPNC-3968)

Modified closable objects to implement Java's AutoClosable. (RLPNC-3944)

Added support for specifying the use of different RNI language models in parameter profiles. (RLPNC-3817)

Added the ability to configure multiple parameter universes that can be activated dynamically at match time and when querying via the Elasticsearch plugin. (RLPNC-3814, RLPNC-3932)

Improved the accuracy of our Hani script language guesser, especially for Japanese. (RLPNC-3831)

Added plugin support for Elasticsearch 2.2.1. (RLPNC-3862, RLPNC-3959, RLPNC-4000)

Removed Chinese and Korean readings of Japanese Kanji from consideration for matching if the name's language of origin is specified as Japanese. (RLPNC-3874)

Greatly improved the speed of our first-pass filter by optimizing our Lucene query. This also applied to both our Solr and Elasticsearch plugins. (RLPNC-3805, RLPNC-3875, RLPNC-3931)

Moved RNI's tunable matching parameters to a config file rlpnc/data/etc/parameter_defs.yaml. (RLPNC-3692, RLPNC-3938)

Upgraded to Lucene 5.2.1. (RLPNC-3708)

Improved RNI's ability to add missing spaces by adding some fuzziness. For example, when matching RobertJohnson Smith and Robbert Smith(RLPNC-3826)

Added the ability to specify match parameters to the pairwise match demo via the URL. For example: http://localhost:9022/rnipm/?name1=Robert%20Smith&name2=Bob%20Smith (RLPNC-3838, RLPNC-3839)

Excluded two letter tokens, like Al and Jo, from consideration for the gender conflict penalty as well as adjusted some heuristics that often cause it to be applied incorrectly. (RLPNC-3825, RLPNC-3808)

Boosted scores of subsumptions, where there are deletions in one name and there are no deletions in the other name and each name has more than one non-deleted token. (RLPNC-3769)

Revised the implementation of Persian IC to match the specification. I.e., eliminated variations from the specification that we had introduced in response to customer requests. (RLPNC-3815)

Fixed a bug when supplying RNI a license via the API before performing Chinese-Chinese name matching. (RLPNC-3844)

Fixed an issue where the RNI Elasticsearch plugin was not escaping some special characters (eg. '|', '\'). (RLPNC-3852)

Corrected an issue where stop prefixes would not be loaded if the stop regex file was empty. (RLPNC-3853)

Added the ability for the RNI Elasticsearch plugin to verify a document is of the correct type before attempting to rescore it using RNI. The plugin will also report some potential errors with the rescore name such as unescaped curly braces. (RLPNC-3889, RLPNC-3974)

Fixed Chinese matching to treat whitespace the same as middle dot to prevent further segmentation. (RLPNC-3913)

Fixed an error that could occur when translating certain names from English to Russian. (RLPNC-3927)

Added support for some Japanese ideographs used in company names that were being normalized away. (RLPNC-3943)

Added an experimental Rosette Name Indexer Elasticsearch^[1] plugin for building fuzzy name retrieval and matching applications for persons, locations, and organizations. This Elasticsearch plugin is distributed in a separate package, is java only, and supports English-English name matching. (RLPNC-3747)

Removed support for Java 1.6. Java 1.7 or higher is required. (RLPNC-3506)

Refactored RNI queries to perform approximately 50% faster. (RLPNC-3740, RLPNC-3714, RLPNC-3700, RLPNC-3699, RLPNC-3660, RLPNC-3596, RLPNC-3490, RLPNC-3734)

Dramatically improved the speed of Korean-English and English-Korean name matching. (RLPNC-3658)

Enhanced RNI matching between names with and without data fields. (RLPNC-3644)

Added a mechanism that uses text files to normalize token variants, ensuring high RNI match scores where variants are involved. Included a file (equivalenceclasses_eng_PERSON.txt) to normalize variant spellings and abbreviations of Muhammad. (RLPNC-3798, RLPNC-3802)

To improve accuracy of RNI match scoring, revised the use of gender penalties so they do not apply to full name overrides, token overrides, or exact token matches. Also removed entries mismatching gender from the English tokens override file: tokens_eng_eng.txt.(RLPNC-3767, RLPNC-3788, RLPNC-3773)

Added names/nicknames to the English tokens override file: tokens_eng_eng.txt.(RLPNC-3787)

Enhanced RNI support for supplying missing spaces when matching names. This strategy improves the score, for example, when matching JohnFitzgerald Kennedy and John Kennedy. (RLPNC-3781)

Enhanced the RNI token scorer to match similar tokens with minor typographical differences, such as Jong and Jxng. This feature adds a SpanMatch.Reason: STRING_SIMILARITY. (RLPNC-3772)

Added a digital signature to the Windows MSI for installing the RWS-Names web service. (RLPNC-3641)

Added a .bat file (Windows) for running RNICLI. (RLPNC-3795)

Added a Java sample that illustrates the different match phenomena that RNI supports: MatchPhenomenaSample.java.

For our Solr 4.9+ plugin, added an RNIReRankQParser, which also provides support for setting a minimum score to check and for replacing the Solr document score with the RNI match score. (RLPNC-3650, RLPNC-3651)

Added Solr 4.9+ plugin support for processing fielded names. (RLPNC-3592)

Added convenience utilities for RNI solrj users. See Javadoc for com.basistech.rni.solr.index. (RLPNC-3615)

For efficiency and to avoid problems sometimes encountered with fielded names, moved titles to be stripped from names from the stopregexes file (rlpnc/data/rnm/ref/override/stopregexes_eng_PERSON.txt) to stopprefixes (rlpnc/data/rnm/ref/override/stopprefixes_eng_PERSON.txt). Added support for specifying which field a stopregex should be applied to when handling a fielded name. (RLPNC-3800)

To enhance security when using RWS-Names, upgraded Tomcat to version 8.0.21 (RLPNC-3819)

Corrected RNI failure to apply stop patterns and stop word prefixes when matching fielded names. (RLPNC-3766)

Fixed the shell script (rnicli.sh) for running RNICLI. (RLPNC-3792)

Corrected a bug introduced in 7.12. with NameIndexQuery#setTestPrimaryData(true). This method no longer returns names where isPrimary() is false for both the query Name and the candidate Name. (RLPNC-3682)

Fixed bug loading binary RNI dictionaries when RNI-RNT is installed in a directory with rnt in its name. (RLPNC-3793)

Revised the RNI Pairwise Matching Demo to report that it cannot generate a score if the names constitute an unsupported language pair and to correctly handle fielded names with empty fields. (RLPNC-3702, RLPNC-3640)

Fixed a problem launching RWS-Names on Mac OS X with the launch.sh script by ugrading Tanuki to 3.5.26. (RLPNC-3819)

Enhanced the classpath so the command-line scripts (rnicli and rntcli) run without logging warnings. (RLPNC-3804)

Adjusted the specification of com.basistech.rni.solr.NameField and associated subfields to enable the correct display of this field in the Solr schema browser. (RLPNC-3806)

The bt_rni_Name_Store subfield was incorrectly appearing in Solr plugin query results. This subdfield is no longer indexed as part of the document and stored, hence no longer appears in query results. (RLPNC-3807)

Deprecated support for Java 1.6. Java 1.7 is required for RNI interaction with Solr 4.8 and above. (RLPNC-3534)

Added support for MAC OS X v10.7 Darwin 11 (amd64-darwin11-xcode4) and 32-bit and 64-bit Windows for Visual Studio 2012 (ia32-w32-msvc110, amd64-w64-msvc110).

Removed support for platforms that do not support Java 1.7: MAC OS X v10.5 Darwin 9 (universal-darwin9-gcc40) and Red Hat Enterprise Linux (ia32-glibc23-gcc32, ia32-glibc23-gcc34, amd64-glibc23-gcc34). (RLPNC-3601)

Added a sample RNI Index, Presidents, which contains the names of the presidents of the United States. (RLPNC-3562)

Added RNI support for normalizing � to ss for queries and matches in English. For example, Russland now matches Ru�land with a score of 0.99. (RLPNC-3560)

Added translators for translating from Korean to English that provide FOLK transliterations that more closely resemble the conventional English spelling of Korean names. (RLPNC-3521)

Changed the RNI default minimumScoreToCheck from 0 to 0.05, which considerably improves speed without a significant decrease in accuracy. If you observe an undesirable loss in recall, you can change this threshold back to 0. (RLPNC-3513)

Moved some English stop words, such as "general" and "mr", to a non-generic stop words file that only applies to PERSON entities, which improves handling of such non-PERSON names as "General Electric" and "MR electric units". (RLPNC-3508)

RNM token matching adjusted to give lower scores when initialisms fail to match (such as "NCTA" and "NICTA"), thereby reducing the number of false positives. (RLPNC-3495)

Dropped support for Solr 3.x. Support for Solr 4.x applies to all versions of 4.x. (RLPNC-3484)

Added an enhanced Solr plugin that supports Solr 4.9 and above and allows RNI to be integrated seamlessly into Solr applications. Our existing Solr 4.x plugin is deprecated. Replaced RNIinSolr4xSample with RNISolrjSample which uses the new plugin and simple org.apache.solr.client.solrj calls to demonstrate how to index and query documents with multiple and multivalued name fields. (RLPNC-3465, RLPNC-3556, RLPNC-3603)

Added an OFAC index with persons and organization (and an XML source file for the index) to illustrate the type of index with which the enhanced Solr plugin can be used. (RLPNC-3610)

Changed name of the RNI requestHandler to '/RNI' in solrconfig.xml. Solr 4.x expects a '/' in front of the requestHandler name. (RLPNC-3533)

To facilitate the use of RNI and our Solr plugin in the SolrCloud with a possibly heterogeneous collection of documents, removed the requirement that bt_rni_NAME_UID be designated as the uniqueKey and that the shard.qt parameter (for the custom request handler) be set to /RNI. (RLPNC-3534)

Revised to RNI Pairwise Matching Demo to provide a more usable explanation of the score that is returned and to support fielded names, with match information for each field. (RLPNC-3459, RLPNC-3590)

Improved role of initialisms in RNI first-pass queries to only match based on initials if both the index name and query name has an initial. (RLPNC-1779)

Configured RNI to better handle Cyrillic initialisms, and Cyrillic initials for Russian names are indexed for the first-pass RNI query, which improves recall. (RLPNC-3325, RLPNC-3442)

Added Russian gender data to improve Russian-Russian and Russian-English name matches. (RLPNC-35450

Enabled users to optionally specify language of origin in the RNT Demo (Interactive). (RLPNC-3427)

Enhanced the RNT Demo to process names with characters from multiple supported scripts. The demo uses a macro-script identifier and translator, such as Jpan for a name that contains Kanji, Hiragana, and Katakana characters. (RLPNC-3590)

RNT now honors whitespace in Japanese input when translating names to English. (RLPNC-3423)

Moved Chinese LOCATION and ORGANIZATION stop words to Chinese to English token overrides, and removed the Chinese LOCATION and ORGANIZATION files. As a result, RNI queries still take advantage of matches, and RNI is not sidetracked when one of the tokens appears in a name (such as Taiwan). (RLPNC-3406)

Expanded use of the Arabic to English LOCATION dictionary to be used by RNTAssistant. (RLPNC-3384)

Added support for reverse transliteration From English / Latin / language of origin Chinese / Chinese Telegraph Code to Chinese / Hani / Native. (RLPNC-2638)

Added the com.basistech.names.parameters. package to suport the runtime configuration of RNI parameters. (RLPNC-3177).

Added a NameCommonService Restful service with support for guessing the language of a string. This functionality is useful in both RNI and RNT. For convenience, guesslang supports JSON and plain-string input: e.g., {"text": "John"} or "John". The previous support for this functionality in NameTranslation is deprecated. (RLPNC-3370, RLPNC-3565))

Added support to the Restful Webservice to process XML input and produce XML output. (RLPNC-3470)

Enabled matching of names consisting solely of digits, such 07 in the OFAC_PERSONS RNI index. (RLPNC-3584)

Fixed RNT error sometimes stripping the space between name elements when translating from Korean to English. (RLPNC-3542)

Fixed a script guesser failure to correctly classify some Japanese strings mixing Kanji and Katakana as Jpan. (RLPNC-3516).

Characters that don't occur in the training data were removed from the English to Arabic RNT model file, eliminating garbage results. (RLPNC-3448)

Eliminated an RNT infinite loop that was leading to an out of memory exception. (RLPNC-3477)

Corrected a bug in assigning weight for scoring to a name element based on its frequency (the greater the frequency the lower the weight) in the associated language model. (RLPNC-3471)

Fixed a memory leak that occurred when running queries with the Rosette Web Services for Names. (RLPNC-3608)

Fixed a TranslationAssistant bug returning Pinyin from English when Chinese Telegraph Code was specified. (RLPNC-3606)

Fixed a TranslationAssistant error dropping the terminal token representing the definite article (al) from names in English of Arabic origin, such as Abedin Zain Ul. (RLPNC-3485)

The nameDataMinimumMatchScore setting was incorrectly influencing the number of names submitted for the second pass of an RNI query, which in turn could lead to inconsistent results. This anomaly has been eliminated. (RLPNC-3599)

Fixed synchronization of RNI web service when opening indexes during initialization. (RLPNC-3620)

Fixed a StringIndexOutOfBoundsException error that occurred when handling a fielded name with an empty field. (RLPNC-3633)

Added KORDA transliterations of Korean names to our training data to improve RNI matching of Korean names in Korean to Korean names in English. (RLPNC-3444)

Added support for handling the ᆻ jamo in the Korean IC and KORDA transliteration schemes. (RLPNC-3443)

For improved performance, added prefix stop words that use string literals rather than regular expresions to strip prefixes during name matching. A longer stop pattern or prefix takes precedence over shorter patterns or prefixes that the longer pattern or prefix contains. For example, the lieutenant colonel stop word prefix is applied where applicable when colonel is also a stop word prefix. (RLPNC-3436)

With English input of Muhammed and Language of Origin of Pushto, TranslationAssistant incorrectly returned the English output alternative as StatisticallyInferred. The output altenerative is now correctly reported as HumanlyAttested. (RLPNC-3464)

Fixed a bug in the RNI Solr clients and improved performance by eliminating duplicate translations. (RLPNC-3454)

Fixed errors translating Chinese names including special punctuation or delimiter characters such as "|" and ";" to English. For example can now translate 新|w2ㄙ垂O and 画面;广场. Non-Chinese characters are included unchanged in the translation.(RLPNC-33453 and RLPNC-3445)

Fixed an error constructing the correct NameDomain when running RNI queries in the NameIndex web service. (RLPNC-3446)

Removed support for integrating RNI into a Solr 1.4 Application. (RLPNC-3232)

Added support for making concurrent calls to the RNI web service querying and/or updating the same RNI index. (RLPNC-3397)

Enabled translation from Chinese to English of name containing a bullet (U+2022) in place of the middle dot (U+00B7). The middle dot is used to separate the words in a non-Chinese name. (RLPNC-3395)

Fixed a crash bug that appeared when attempting to translate a single Korean jamo to English. The jamo is now transliterated as the appropriate Latin script letter(s). (RLPNC-3394)

In addition to dictionary lookups, added support to the com.basistech.rnt.assistant interactive API to provide statistically inferred English alternatives for Arabic, Korean, Russian, and Chinese input, as well as statistically inferred Arabic, Korean, and Russian alternatives for English input When a result is statistically inferred, that information is included in the result (DataSourceType.StatisticallyInferred). (RLPNC-3283, RLPNC-3309, RLPNC-3063)

Fixed failure to log the exception when the RNI web service cannot open an index. (RLPNC-3389)

Added a DefaultTranslationPairs class to enable translations by specifying the source and target language rather than complete language domains. (RLPNC-3372)

Added a hintLanguage property to the Name object that NameBuilder can use as a suggestion when it guesses the language. In standard usage, the hint will be the language already identified for the document containing the names. If the hinted language is compatible with the script, which NameBuilder can also guess, NameBuilder returns the hinted language, otherwise the language it guessed. This feature is also available through the NameTranslation web service. (RLPNC-3369, RLPNC-3311))

The RWS-Names web service was sometimes returning UNKNOWN as the language for a name. Now it always guesses a language if the user did not supply it. (RLPNC-3316)

Make the LanguageOfOrigin results returned as part of a translation available through the NameTranslation web service ResultAnnotations object. (RLPNC-3352)

Translations from Russian to English now include language of origin (Russian or English) for each word in each result. (RLPNC-3352

Optimized the script guessing algorithm, approximately doubling the speed. (RLPNC-3314)

Fixed the Solr plugins to include language of origin when adding names to an index. (RLPNC-3305)

Enhanced RNI scoring of name matches when the names are the same except for reordering of regions of the name (such as Carnera Baer Braddock Louis Charles Walcott and Charles Walcott, Carnera Baer Braddock Louis). (RLPNC-3304)

Normalized l with stroke (U+142) for better RNI handling of Polish names (such as Michał, which now matches Michal). (RLPNC-3303)

Corrected exception thrown when user attempts to transliterate a particular person name (اسامة نين اع بن لادن) from Arabic to English. (RLPNC-3301)

Extended the RWS-Names web interface to support pairwise name matching and added the RNI Pairwise Matcher Demo to illustrate this feature. (RLPNC-3012)

Modified the RNI Demo to enable users to specify language of origin and entity type for a query name. (RLPNC-3295, RLPNC-3104)

Established a static method (RNIConfiguration.setLicenseXML(String licenseXML)) for passing in the RNI user license, matching the behavior for RNT (RNTEnvironment.setLicenseXML(String licenseXML). (RLPNC-3344)

To improve accuracy, trained an RNI Russian language model and a Korean language model for determining the frequency of person name tokens. (RLPNC-3289, RLPNC-3241)

Fixed an error handling the Persian ezafe (-e) during IC transliterations. (RLPNC-3273)

Added ability to extract debug info from each of the MatchResult objects returned by a query. In NameIndexQuery call setIncludeDebugInfo(true), then in each MatchResult, call getDebugInfo() to return a string that describes how the result was derived. (RLPNC-3266)

Added a .bat file (Windows) and shell script (Unix) for running RNICLI from the command line without using Ant. (RLPNC-3203)

Increased accuracy of Korean English matching of native names by including more Korean name data in training. (RLPNC-3341)

Added "corp" to our eng_eng_ORGANIZATION token override file so RNI returns a better score when matching an organization name like Korea National Oil Corporation with Korea National Oil Corp. (RLPNC-3340)

Adjusted the phonetic algorithm that RNI uses for generating search keys for Korean names so that 김 provides a better match for Kim. (RLPNC-3334)

Provided more complete normalization of extended Latin characters, such as đ with stroke (U+0111), to improve accuracy handling names with such characters. (RLPNC-3322)

Fixed support for specifing the Korean script Kore, an alias for Hangul + Han. (RLPNC-3318)

Fixed an RNI threading issue that was causing crashes during initialization. (RLPNC-3315)

Added support for performing the reverse transliteration of Korean names in English using the BGN scheme. (RLPNC-3281)

Enhanced ISO15924Utils.scriptForString to know to return a macro-script (Hrkt or Jpan) for mixed script Japanese strings. Hrkt is a mixture of Katakana (Kana) and Hiragana (Hira) (e.g., トイザらス). Jpan is a mixture of Kana, Hira, and Kanji (Hani) (e.g., トヨタ自動車株式会社). (RLPNC-3279)

Fixed the accuracy of RNI span matches when matching a hyphenated name in English against a name in Korean. (RLPNC-3277)

NameBuilder now uses an Unchecked Exception in place of InvalidNameException, so users do not have to use a try/catch every time they create a name. (RLPNC-3276)

Added guessLanguage and guessScript utility methods to NameBuilder, so users can determine language and script without having to create the Name object. (RLPNC-3275)

Added an English to Russian (Latn, eng, folk to Cyrl, rus, native) translator. (RLPNC-3269)

Added an RNT option to specify which transliteration scheme should be used for BGN when transliterating Korean. If com.basistech.rnt.options.KorGeographyOption is set to NORTHKOREAN (the default), RNT uses McCune-Reischauer . If set to SOUTHKOREAN, RNT uses Revised Romanization of Korean. (RLPNC-3261)

Added support for transliterating Korean into Undiacritized BGN. (RLPNC-3253)

Improved ability of RNI to match truncated names. (RLPNC-3200)

The RNT Japanese translator now use Korean segmentation when given Hani name with Korean language of origin. (RLPNC-3173)

Extended the use of the RNI simililarty score of 1.0, meaning exact match, to indicate the strings are equal, the languages of use and origin match, and the entity types match. (RLPNC-2636)

System.(err|out).print and printStackTrace() statements in RNI and RNT have been replaced with slf4j (logging) calls. (RLPNC-2688)

The com.basistech.rni.match.Name constructors are deprecated. Use the com.basistech.rni.match.NameBuilder Build() method to construct and return a name object with all value set via fluent methods. (RLPNC-3234)

Removed support for Java 1.5. Java 1.6 or higher is required. (RLPNC-2489)

All translation domain pairs now include language of origin. (RLPNC-3078, RLPNC-2939)

RNI name matching for Chinese to Chinese now includes variants for both simplified and traditional Chinese. (RLPNC-3092)

Added support for translating Korean and English names from English (folk) to Korean. (RLPNC-3211)

When RNI is integrated into a Solr application, RNI will guess language and script if you do not include those settings. (RLPNC-2523)

To improve efficiency, all Name fields in an RNI document are serialized in a JSON object and stored in a single field. (RLPNC-3225)

Added a boolean flag to com.basistech.rni.index.IndexStoreDataModelFlags: usingCachingCodec. When true, (the defaults to false) this flag instructs Lucene to use our CachingCodec which at read time loads and stores documents in RAM. Warning: This is exceptionally RAM intensive, but may significantly improve performance. (RLPNC-3218)

RNI caches query data to enhance query performance. (RLPNC-3223)

Added Windows releases for Microsoft Visual Studio 10. (RLPNC-3163)

Removed unrequired JAR files. (RLPNC-3185)

Fixed a bug that caused an ArrayIndexOutOfBoundsException when indexing names with greater than 6 RNT overrides. (RLPNC-3164)

Added support to the RNT web demo for specifying the language of origin. (RLPNC-3103)

Fixed a slowdown in English queries against English names that was introduced in the 7.9.0 release. (RLPNC-3141)

Fixed a slowdown caused by a significant increase in calls for supported translation pairs that was introduced in the 7.9.0 release. (RLPNC-3176)

Expanded RNI query results to indicate spans (one or more tokens) in the query name and result name that match or do not match. (RLPNC-2878)

Added a RESTful interface to the Rosette Web Services for Names (RWS-Names). The SOAP interface is still in place. The RNI and RNT Web Demos use the RESTful interface. (RLPNC-2184)

Extended RNI Solr plugin to support Solr 4.3. (RLPNC-4.3)

Added support for translating names from English to Chinese. (RLPNC-2174)

Expanded the scope of the normalization translation option to convert Chinese names in traditional script to the simplified Chinese script, and to convert Japanese Kanji variants (including old Kanji) to their standard form. (RLPNC-2914, 2846)

Updated implementation of IC transliteration scheme for Chinese to conform to May 2013 deliberative draft of the IC Chinese Standardized Transliteration System for Personal Names. (RLPNC-3068)

Improved the accuracy of Western Farsi, Dari, and Pushto translations. (RLPNC-2972, 2973, 2976, 3001, 3002)

Added support for transforming BGN to Undiacritized BGN for Arabic, Western Farsi, Dari, and Pashto. (RLPNC-2980)

Added support for translating non-Chinese person names in the Chinese language and to their traditional English representation. (RLPNC-1451)

Added support for translating non-Korean person names in the Korean language and to their traditional English representation. (RLPNC-2979)

Added support for transliterating Korean person names in accordance with the IC standared. (RLPNC-2720)

Added programmatic access to RNI and RNT overrides, which lets you define your own override tables (character streams) in place of the tables in the default override directories. (RLPNC-2689)

In the API documentation, see the following methods in com.basistech.rni.index:

and the following method in com.basistech.rnt:

Deprecated the use of translation domains that combine Latn with a non-Latn-script language, such as Arabic or Chinese. In an upcoming release, language of origin will be used to clarify the nature of the translation. For example, a translation from Arabic, Arabic script, native transliteration to English, Latin script, IC, will be a translation, not a transliteration, if the language of origin is English. (RLPNC-2954)

Extended supported domain pairs to include language of origin for Arabic and Chinese. (RLPNC-3067)

Changed Rosette Web Services For Names default port to 9022. (RLPNC-3048)

Extended RNI Web demo to support language guessing. (RLPNC-3042)

Fixed RNT problem handing some Japanese characters. (RLPNC-3052)

Fixed Null Pointer Exception that occurred with some Solr RNI queries. (RLPNC-3015)

Deprecated com.basistech.rni.match.setMaximumNumTokens(int maxNumTokens). (RLPNC-2788)

Improved RNI accuracy matching Japanese names, including native names, names of Chinese or Korean origin, and other non-native names with English translations. These improvements in accuracy do entail a slowdown in RNI operations with Japanese. (RLPNC-2606, RLPNC-2608, RLPNC-2422)

Extended RNI support for matching Japanese name variations such as nicknames and cognates (Katakana), and reordered name components. (RLPNC-2088)

Added RNI sample index with Japanese names (professional baseball players) in Japanese scripts and sample queries in Latin script.

Improved the accuracy of Pushto translations using the IC standard. (RLPNC-2800)

Fixed bug transliterating Arabic heh (/U0647) in Pushto names. (RLPNC-2783)

Refactored enforcement of constraints on setting maximum number of names to consider, to check, and to return for RNI queries, thereby enabling support for setting these constraints when using RNI with Solr. (RLPNC-2828)

Numbers are no longer stripped from entity types other than PERSON. (RLPNC-2185)

Override files for token matches may include a third item for each entry which provides RNI with additional context for handling the override: NICKNAME or COGNATE. If no value is included, the default is NICKNAME. (RLPNC-2050)

Having determined that some classes and methods in internal packages should be available to users, and that some publicly available APIs return a class in an internal package, we have refactored the API as indicated in the following table. We have also refactored the API for handling a collection of names. (RLPNC-1768)

com.basistech.rni.index.
RNILicenseException

com.basistech.rni.index.internal.
RNILicenseException

com.basistech.rni.index.
SupportedDomainPairs

com.basistech.rni.index.internal.
SupportedDomainPairs

com.basistech.rni.match.Name
com.basistech.rni.match.MatchScorer.
prepNameForCachedScorer(Name n)

com.basistech.rni.match.internal.translate.
CommonNameTranslators.translate(Name n)

com.basistech.rni.index.internal.
INameIndexStore com.basistech.rni.index.
StandardNameIndex.getStore()

com.basistech.rni.index.internal.
NameIndexFilterComparator com.basistech.rni.index.
INameIndexFilter.getComparator(NameIndexQuery query) 

To improve performance using linguistic analysis to return high-precision results from a high-recall list of names that match a query name, deprecated the static com.basistech.rni.index.StandardNameIndex queryList method in favor of a non-static com.basistech.rni.index.StandardNameIndex filterCollection method. (RLPNC-2560)

Per the IC specification for transliterating Person names from Pushto, we provide special handling when the language of origin is Dari. We also provide variant spelling and regional options to control the transliteration. (RLPNC-2652, RLPNC-2677, RLPNC-2448)

Added support for defining a set of text domains that filter results returned by an RNI query. For example, if the RNI index contains names in English, Western Farsi, Arabic, and Pushto, you can set text domains to only return names in English/Latin script, and Western Farsi/Arabic script. In the HTML API documentation, see com.basistech.rni.index.NameIndexQuery.setTargetNameDomains(Set<TextDomain>) and com.basistech.rni.index.NameIndexQuery.testTargetNameDomains(boolean). (RLPNC-2718)

Removed support for the JDEC-Afghanistan Pushto transliteration scheme. (RLPNC-2734)

Support and an associated sample have been added for running RNI with Solr 4x. (RLPNC-2742)

When the Segmentation Option is turned off for Japanese, RNT now assumes the names it is processing have been segmented by the user (tokens are space delimited). Prior to this fix, RNT treated the entire name as a single token, which produced incomplete results for names with space delimited tokens. (RLPNC-2750)

Verified RLPNC support for Redhat 6.0. For 32-bit platforms, use the ia32-glibc23-gcc34 package; for 64-bit platforms, use the amd64-glibc23-gcc34 package. (RLP-3649)

Improved support for vocalizing Pushto and Western Farsi names. (RLPNC-2512, RLPNC-2232)

Improved handling of the ezafe in Western Farsi, Dari, Pushto (izafat), and Urdu names. (RLPNC-343)

Added RNI support for indexing and querying names identified as Persian (fas) or Dari (prs). Persian is the metalanguage that includes both Western Farsi (pes) and Dari. Queries for Persian names may return names indexed as Persian or Western Farsi. (RLPNC-2711)

Added override file for translation of LOCATION names from Russian to English. The package now contains override files for translating LOCATION names from Arabic, Japanese, and Russian to English. (RLPNC-2176)

Fixed a bug that prevented Solr 1.4 users from setting the isPrimary attribute for a Name. (RLPNC-2667)

Fixed the inclusion of incorrect results from name pair (fullname) override files during the high-recall phase of queries. (RLPNC-2693)

Fixed a NullPointerException or the inclusion of results of the incorrect entity type from name pair (fullname) override files. (RLPNC-2676)

Starting with this release, the RLPNC release number matches the release number of the RLP with which it should be installed (the third number may vary).

Added TranslationAssistant and NameIndex web services. For Windows users, added .NET clients for each of the Rosette Web Services for Names. (RLPNC-2243, RLPNC-2272, RLPNC-2233)

To clarify translation support for Western Farsi and Dari (both members of the Persian macro-language), we have replaced the Persian ISO 639 language code ("fas") with the Western Farsi language code ("pes"). For BGN, which does not distinguish between Western Farsi and Dari, we support the use of all three language codes ("fas", "pes", and "prs") for Dari. (RLPNC-2505)

Improved the vocalization maps for the translation of Western Farsi and Dari names. (RLPNC-2257)

Enforce requirement that the maximum number of names considered in the first pass must be greater than or equal to the maximum number of names evaluated in the second pass and the maximum number of names returned by the query. The code no longer resets these settings under the covers if the user makes a setting that violates this constraint. (RLPNC-2160)

Changed the default setting for testing entity type during queries from false to true. As a result, a query only returns names that match the entity type of the query name (such as PERSON, LOCATION, ORGANIZATION, VEHICLE, or NONE). (RLPNC-2149)

Upgraded to Lucene 3.6. (RLPNC-2470)

Extended support for using RNI with Solr 3.x and Solr 4.x. Added an RNI sample that runs with Solr 3.x, and provided instructions in the documentation for using RNI with the Solr 3.5 Admin Example to post Name documents and perform queries. (RLPNC-2462)

For queries performed with Lucene or Solr 3.x, adopted DisjunctiveMaxQuery to improve first-pass scores with names for which multiple alternatives (such as nicknames) are defined in token override files. For a given minimumScoreToCheck, this strategy provides more accurate recall for names submitted to the second pass. (RLPNC-2473)

Added support for indexing and querying Spanish names. (RLPNC-2478)

Restructured HighRecallKeys. The StandardNameIndex generateHighRecallKeys method is no longer available and HighRecallKeys has been refactored. Removed the GenerateAndUseHighRecallKeys Java sample and the RNICLI -generate-keys option. Our working assumpation is that RNI provides HighRecallKeys to assist in Name queries using Lucene or Solr. The underlying structure is in a state of evolution. If you want to use some other infrastructure to store Names and perform queries, please discuss this issue with us. (RLPNC-2499)

Added com.basistech.rni.match.NameBuilder as the preferred mechanism for creating Name objects. NameBuilder provides a fluent interface that supports method chaining. (RLPNC-2515)

Added language of origin as a Name field. The default value is LanguageCode.UNKNOWN. RNT uses this value when translating foreign names from Arabic, Japanese, and Russian. (RLPNC-1717)

Added an RNT translation option (MinimizeOrthographyOption) to remove short vowel diacritics from Arabic, Western Farsi, Dari, Pushto, and Urdu names in Arabic script. The default setting for this option is false. (RLPNC-2320)

Removed RNT translators for Dari and the JDEC-Afghanistan transliteration scheme. (RLPNC-2558)

Extended support for applying the IC transliteration standard for Pushto to include the special rules and special cases defined in the Pashto Standardized Transliteration System for Personal Names, 01 June 2011. (RLPNC-2565)

Adjusted MatchScorer settings for each language to establish a Precision/Recall crossover near the 0.55 threshold for all language pairs. The crossover may still vary, depending on the input data. If you are interested in customizing these settings, please contact support@rosette.com. (RLPNC-2555)

RLPNC support for Java 1.5 is deprecated. RLPNC users of Java 1.5 should move to Java 1.6. In this release, Java 1.6 is required to use The Rosette Web Services for Names, and to use RNI with Solr 3.x or 4.x. (RLPNC-2486)

Fixed a runtime error that occurred when translating Japanese Katakana names. (RLPNC-2435)

Added missing stop pattern and token override files to improve accuracy of Japanese RNI queries. (RLPNC-2477)

Upgraded the keys stored with Names. Accordingly you must recreate any existing RNI indexes that you intend to use with this release.

Enhanced support for translating Japanese Kanji names and foreign names in Katakana.

Users can now compile and run the Solr sample (RNIinSolr14Sample) without providing the path to an Apache-Solr-1.4 distribution. The required .jar files have been placed in the samples lib directory. (RLPNC-2113)

If an override file for RNT translations includes source names with multiple target names, and the file does not include confidence scores, RNT sets the confidence score for each translation to 1 divided by the number of translations for that name. (RLPNC-2122)

To discourage performance degradation processing oversized names (probably bad data), the Name object issues a warning if you exceed 10 tokens for the data in a Name. You can use the static Name.setMaximumNumTokens(int maxTokens) method to change or eliminate the limit. RNICLI and RNTCLI now include a -maxTokens parameter. (RLPNC-2124, RLPNC-2204)

Extended support for stop regular-expression patterns to apply to all supported languages, for fullname overrides to apply to all supported text domain pairs, and for token overrides to apply to English, Japanese, Chinese, and Russian. (RLPNC-1922, RLPNC-2152, RLPNC-2085)

Enabled multi-threaded RNI update operations. Multiple threads may share an INameIndexSession object. The write operation for each update is handled in a single thread, but other portions of an RNI update, such as the name completion required for adding a name, are multi-threadable. (RLPNC-1957)

To run the RNI and RNT command-line interfaces or examples, you no longer need to set the LD_LIBRARY_PATH or DYLD_LIBRARY_PATH environment variable on Linux, Solaris, and Mac OS X platforms. (RLPNC-2158)

Incorporated the NameTranslation web service into the RNI-RNT SDK. Other Rosette Web Services for Names will be added in future releases. (RLPNC-1577)

This release requires RLP 7.4.

Documented RNI support for running in a Solr application. (RLPNC-1918)

Added gender as a consideration for matching English names. (RLPNC-1851)

RLPNC is no longer built for the following platforms: sparc-solaris9-cc58 and sparc-solaris9-cc58-64. (RLPNC-1950)

Added support for translating Japanese Kanji names to English and for segmenting Kanji names. (RLPNC-1947)

Added preliminary support for indexing, querying, and matching names in Russian and Japanese Kanji. (RLPNC-2003)

Refined support for English LOCATION, ORGANIZATION, and VEHICLE entity types. Accordingly, the results when processing these types may differ from the results processing PERSON entities or Name objects with no entity. For most accurate results, specify the entity type when defining a Name. (RLPNC-1941)

Added -maxToCheck parameter to RNICLI to specify how many potential candidates the query should check with its high-precision linguistic filter. Use this parameter to adjust the speed/accuracy tradeoff. Also deprecated the -top parameter for specifying the maximum number of results the query should return. Use -max. (RLPNC-2032)

Optimized handling of duplicate candidate names in the RNI index when running the com.basistech.rni.index.StandardNameIndexFilter query() method. (RLPNC-1656)

Added support for defining Name objects with data fields for indexing and querying. With this facility, you can potentially enhance the accuracy of queries with English, Japanese, and Chinese PERSON names. Scores are higher when a field in the query is similar to the same field (as determined by the order in which the fields appear) in a candidate index name. Fields have no explicit semantic definition (such as family name or given name). When translating a name with fields, RNT handles the name as a single string with a space between each field. (RLPNC-1864)

Modifed the samples ant script to support building and running the Solr Connector sample, provided you have access to Solr 1.4. (RLPNC-2068)

To improve error handling with interactive translations, RNT now uses com.basistech.rnt.assistant.InitialInput to throw a com.basistech.rnt.UnsupportedBasicTranslatorException if the input or output domain is not supported, or a com.basistech.rnt.InvalidNameContentException if the input string is empty or not in the correct input script. (RLPNC-1955)

In response to customer feedback, modified the content of the stop patterns file and token override file for English. (RLPNC-2075)

Fixed RNI entity-type-specific overrides to work as documented. (RLPNC-1935)

Refined the Dari transliteration maps for vocalization to be in line with and as complete as the Pushto transliteration maps. (RLPNC-1774)

In StandardNameIndex deprecated generateHighRecallKeys in favor of two new methods: generateHighRecallIndexKeys and generateHighRecallQueryKeys. (RLPNC-1595)

Improved accuracy of RNI queries as measured with test data. (RLPNC-1766)

Fixed bugs and improved the accuracy of handling of variable segmentation in RNI's English-to-English matching. (RLPNC-1692)

Added support for designating entity-type-specific overrides for RNI stop patterns, name pair matches, and token pair matches. (RLPNC-1780)

Optimized handling of duplicate names returned by an RNI query. (RLPNC-1656)

Extended out-of-the-box token overrides file for English-to-English matching with cognate or "cousin" name pairs, such as Pierre and Pedro. (RLPNC-1681)

Fixed error generating Arabic sun letters. (RLPNC-1715)

Improved performance of RNTCLI and fixed a bug in its invocation from the Ant build.xml script. (RLPNC-1711)

Revised implementation of the IC standard for Western Farsi transliteration as detailed in the footnote in the Appendix: Supported Translation Domains.

Enabled statistical inference for adding diacritization to Dari, Pushto, and Urdu native names not found in the corresponding dictionary. (RLPNC-1622)

Introduced caching of RNI name scores to enhance performance when querying English names against large English language name sets. (RLPNC-1677)

Enabled RNT to return multiple choices for multiple-token input when translating Arabic names. (RLPNC-1650)

Enabled RNTCLI to begin writing output while processing is still in progress. (RLPNC-1648)

Fixed a race condition that sometimes occurred in RNICLI or RNTCLI when processing with multiple threads. (RLPNC-1671)

Fixed an RNT bug handling the complete range of Arabic characters and processing names in Arabic script that end with a Latin character. (RLPNC-1673)

Fixed errors and a potential crash in the IC transliteration of Western Farsi. (RLPNC-1665, RLPNC-1657, RLPNC-1658)

Fixed memory leaks in Arabic name normalization, in the translation of Russian names, and in the procedure for inferring language when not specified by the user. (RLPNC-1661, RLPNC-1663, RLPNC-1660)

Added constructor for creating a Name object with just a String argument (script and language are inferred). You can add the Name to an index, use it in a query, and use it in name matching. Accordingly, RNICLI can now process Name objects that are constructed solely from a String. To translate the Name, you must still supply a target transliteration scheme. (RLPNC-887)

Added statistical support for translating foreign (non-Russian) names from Russian to English. (RLPNC-1558)_

Improved recall in RNI first pass key search to do a better job of presenting all potential similarity matches to the linguistics filter. (RLPNC-1459)

Modified the RNICLI utility to enable the addition of names to an existing index, as well as the creation of (and optionally adding names to) an index. (RLPNC-1574)

Enhanced ability to return a reasonable translation to English, rather than no translation, for unknown foreign names, provided the user supplies a sufficiently low translation threshold. (RLPNC-1581)

Enabled statistical inference for adding diacritization to Persian, Pushto, and Urdu native names not found in the Persian, Pushto, or Urdu dictionary. (RLPNC-1597, RLPNC-1622)

Revised the implementation of IC transliteration for Persian to conform to the IC standard for those languages. (RLPNC-64)

Enabled normalization of Arabic native names. (RLPNC-1412)

Renamed the Rosette Name Indexer packages: com.basistech.rni contains the RNICLI command-line interface. The Rosette Name Indexer API, including the utility for loading the contents of an XML gazetteer into an RNI index, is in com.basistech.rni.index and the name matching API is in com.basistech.rni.match. As a result, the com.basistech.rnm packages no longer exist. (RLPNC-37)

For consistency and clarity, replaced "lookup" with "query" and "NameLookupKey" with "PhoneticHighRecallKey" in the RNI API com.basistech.rnm.index (RLPNC-1538):

Removed the com.basistech.rlp.pipeline.name package and the associated sample (NamePipelineSample). (RLPNC-1522)

Deleted com.basistech.rnt.SimpleTranslatable. Use com.basistech.rni.match.Name as the implementation of the com.basistech.rnt.ITranslatable interface. (RLPNC-14927)

Removed support for the C++ API for name translation. The corresponding sample application and API documentation have also been removed. This change results in a public API that is entirely in Java. (RLPNC-1409)

Added support for using regular expressions in a "stop-words" file to exclude specified strings from indexing and queries. (RLPNC-1486)_

Added support for using fullname files to specify the similarity scores to assign to specified name pairs. (RLPNC-1455)

Added support for using token files with pairs of name elements. When RNI evaluates two names, each of which contains a token from a pair in the tokens file, it enhances the similarity score for the two names. (RLPNC-1457)

Improved Arabic to Arabic name matching. (RLPNC-1447)_

Refined the matching algorithm to guarantee that matches for a given pair of names are commutative. For two names (a and b), the similarity score is identical, whether a is in the index and b is in the query, or b is in the index and a is in the query. (RLPNC-1446)

Improved the handling of variable segmentation in English to English PERSON name matching, such that the match between two names that differ only by the presence or absence of a space (such as Van Dick and VanDick) receives a higher score than in prior releases. (RLPNC-1430)

Added support for handling titles and initials in Arabic to English and English to Arabic translations of PERSON names. (RLPNC-68, 92)

Added support for standardizing names of Arabic origin in Latin script, according to the transliteration standard specified in the target domain. If the name is not of Arabic origin, it is unchanged. (RLPNC-1370)

To better incorporate the interactive translation assistant into RNT, the com.basistech.xa package has been deprecated in favor of a new package: com.basistech.rnt.assitant. The TranslationAssistant class in this package replaces TransliterationAssistant in the deprecated package. (RLPNC-1377)

Deprecation of some empty constructors, associated set methods, and name normalization without a Name object. (RLPNC-1497, RLPNC-1476)

In com.basistech.rnm.Name, the empty constructor is deprecated in favor of public Name(String data, LanguageCode language, ISO15924 script).

In com.basistech.rnm.Transliteration, the empty constructor, setScheme, setScript, and setTransliteration are deprecated in favor of public Transliteration(TransliterationScheme scheme, ISO15924 script, String transliteration).

In com.basistech.rnm.index.NameIndexQuery, the empty constructor is deprecated in favor of public NameIndexQuery(Name qname)

In com.basistech.rnm.index.NameStringNormalizer, normalize is deprecated in favor of public static String normalize(Name n).

Plugged a memory leak that sometimes occurred processing Pushto or Dari names. (RLPNC-1460)

All RNI indexes should be rebuilt using this release.

Modified the RNICLI utility to take a file pathname as -name argument when performing the generate-keys option. Each line in the UTF-8 file contains a name for which RNICLI generates keys.

Added support for generating and using high recall keys while storing names in something other than an RNI index, such as a relational SQL database. The GenerateAndUseKeys.java sample illustrates this process using HashMaps to store keys and names.

TranslationAssistant, data transfer objects are now serializable and com.basistech.xa.TranslationAssistant includes a new select method to accommodate the processing of serialized data:

public Output select(int alternativeIndex, InitialInput initial, Segmentation segmentation)

Renamed a few methods on com.basistech.xa.Segmentation (with deprecation).

Plugged some holes in the Urdu transliteration schemes. (RLPNC-1359)

Fixed a bug with RNT's command line interface. (RLPNC-1360)

Upgraded from Apache Lucene 2.1.0 to 2.9.1. Accordingly, you must recreate any exiting RNI indexes that you intend to use with this release.

Added RNI support for local and distributed transactions. If you use the INameIndex interface to perform updates and queries, each operation is automatically executed in its own transaction. To explicitly control local transactions, use the INameIndexSession interface; for distributed transactions, use the INameIndexTransaction interface. As a consequence of this change, the APIs for setting and getting batch mode, and flushing updates to an index have been removed, and the INameIndez open() method no longer includes a boolean argument specifying whether the index is to be opened for updates.

Added two Java samples. AddNamesSample illustrates the use of a transaction to add a number of names to an RNI index. DistributedTransactionSample illustrates a distributed transaction with a two-phase commit involving two RNI indexes.

Removed support for the C++ API for name matching. The corresponding sample application and API documentation have also been removed.

Removed the com.basistech.rnm.index.adv package from the accessible API. The public API for RNI indexes is in the com.basistech.rnm.index package

Extended the TranslationAssistant (RNT interactive mode) Java API to handle Dari and Pushto names in addition to Arabic names. Refactored the API to simplify access to TranslationAssistant functionality. The infrastructure is now in place to support variable segmentation of the input string, but for this release, TranslationAssistant does not yet support overlapping segments.

Added support for retrieving a group of names (even all names in an RNI index) that share some common characteristic other than name similarity.

Changed the means of specifying multi-threaded behavior for the RNI command-line interface.

Fixed a thread-safety problem performing RNT translations in multiple threads. (RLPNC-2551)

Fixed the occasional omission of leading portion of a Persian, Pushto, or Urdu name in Arabic script during RNT translation. (RLPNC-295)

Significantly sped up the operation of creating or opening an RNI index. (RLPNC-1288)

Fixed a failure in some cases to generate output with long foreign names during RNT translation from Arabic to English. (RLPNC-1297)

Significant speed improvements in RNI queries. (RLPNC-1081)

Greater separation between scores for good and bad matches when matching names in Chinese Han characters to names in Latin script. (RLPNC-871)

Updated the dictionary of names that appear in English. (RLPNC-113)

RNI queries now return name data results with scores greater than or equal to the value set with setNameDataMinimumMatchScore. Prior to this adjustment results had to be greater than the minimum match score to be returned. The minimum match score must be greater than 0.

The RNI command-line interface includes an optional parameter for minimum match score and an optional parameter for maximum number of names to return.

The RNI and RNT command-line interfaces support the concurrent processing of multiple files in separate threads.

Added support for segmenting (determining the boundary between) Korean surnames and given names. (RLPNC-338-, RLPNC-1069)

Updated documentation to use ISO639-3 three-letter language codes, rather than ISO639-1 two-letter language codes. The ISO639-3 codes enable finer language distinctions, such as between Western Farsi ('pes') and Dari ('prs'), both members of the Persian macrolanguage, for which the ISO639-3 code is 'fas' and the ISO639-1 code is 'fa'.

Fixed the capitalization of Latin-script translations of Japanese names. (RLPNC-1217)

Added preliminary RNI and name-matching support for excluding titles from names during English-language queries and name matches.

Activated support for handling nicknames in RNI queries and name matches.

Tuned support for handling initials in RNI queries and name matches

Added support for the Windows 64-bit platform and the Linux IA32 glibc23 gcc40 platform.

Expanded support for performing RNI queries and name matches with names in the English language. In addition to handling matches that involve phonetic/orthographic differences and missing name components, queries and name matches are designed to handle names with initials, nicknames, and word order variations.

Added preliminary RNI and name-matching support for handling names in the Japanese language rendered with the Hiranaga and Katakana scripts. At the current level of support , queries and name matches involving Hiragana and Katakana are most effective when the names have been segmented (the words that make up each name are space delimited).

New transliteration schemes for the translation of Arabic names in Arabic script to Latin script: Extended IC (handles all characters in Arabic script, including characters used in Persian, Urdu, Pushto, and Dari) and undiacritized BGN (removes diacritics or non-ASCII characters from BGN transliterations).

New transliteration schemes for the translation of Korean names in Hangul and Hanja to Latin script: the Revised Romanization of Korean (MOCT) and the Korean Romanization for Data Applications (KORDA).

Added support for translating Russian names in Cyrillic to Latin script in accordance with the ISO 9:1995 transliteration standard.

For RNI queries, changed the default setting of testNameData to true, so that users are not required to make this setting for performing name-match queries.

Added a sample RNI Index, OFAC_PERSONS, which contains the names in Latin script of individuals in the Office of Foreign Asset Control watch list. (RLPNC-994)

More robust handling of input names that contain non-alphabetic characters. (RLPNC-968)

Removed the restriction on the number of names an RNI query can return. (RLPNC-909)

Improved the performance of RNI queries for Arabic names. (RLPNC-924)

Changed the default setting for NameIndexQuery.testNameData from false to true in order to simplify the process of issuing standard RNI queries. (RLPNC-960)

Return 0 rather than throw an exception when querying with a Chinese Hani name that contains data in another script. (RLPNC-970)

Modified the statistical model for translating Arabic names, leading to minor differences in the results returned. (RLPNC-974)

Addition of TranslationAsistant (RNT in interactive mode), a Java API for building interactive applications to transliterate Arabic names.

Clarification of the pairing of language/script combinations for RNI index queries and name matching. Language is language of use in which a name appears, not the language of origin. So, for example, the language of a name of Arabic origin in Latin script, is English, not Arabic. When adding names to an RNI index, performing index queries, or matching names, the script must be a native script for the language. You may use LanguageCode.UNKNOWN for the language. A name in any legal language/script pairing may match a name in any legal pairing for that language or English (or unknown) in Latin script, and vice versa. For the details, see the appendix titled Supported Text Domains for Rosette Name Indexer and Name Matching. (RLPNC-819)

Fixed memory leak triggered by repeated opening and closing of an RNI index. (RLPNC-917)

Fixed problem generating match score > 1.0 for some Latin to Latin name pairs. (RLPNC-681)

Removed support for the Windows 32-bit Visual Studio 7.1 platform. For Windows, use the Windows 32-bit Visual Studio 8.0 release.

RNI-RNT now throws an UnsupportedNameDomainException when an application attempts to index or lookup a name with a script/language combination that is not in a supported text domain. (RLPNC-533)

Fixed a multithreading problem that appeared sporadically in RNT and RNI. (RLPNC-559)

Improved Latin to Latin name matching to score partial matches for individual words in each name. (RLPNC-355)

Provided higher recall in RNI queries at the cost of speed. In a future release, we plan to provide users with control over the tradeoff between speed and accuracy.

Improved handling of Korean names when an eumjeol in the name contains 2 (not 3) jamo.

Improved performance of name index (RNI) queries involving names in Latin script.

Upgraded IndexQuerySample.java and the C++ sample applications to extract input data from a file.

Improved RNI handling of names with whitespace and/or hyphens.

Enhanced support for translating personal names from Arabic to English.

Support for transliterating Japanese names in Hiragana and Katakana.

Options added to RNT for adjusting the performance tradeoff between speed and precision and for turning off the use of statistical methods to establish information that was not found in a dictionary.

New static method for returning a map of all the text domain pairs suported by RNI and name matching: com.basistech.rnm.index.StandardNameIndex.getMapOfSupportedTextDomainPairs(). The key for each map entry is a query text domain. The value is a list of reference text domains supported for that query text domain.

RNI name indexes.

Utility for loading the contents of XML gazetteers into RNI indexes.

Translation of non-native personal names in Arabic documents to their standard English form in Latin script.

The Folk transliteration scheme to generate variant names in Latin script from Arabic, Persian, Pushto and Urdu names in Arabic script.

Move from Java 1.4 to Java 1.5.

Renamed the Java com.basistech.NameMatching.* packages to com.basistech.rnm.*. For RNI functionality, see com.basistech.rnm.index.*.

New sample applications to illustrate usage of the RNI-RNT APIs.