Release Notes

Wikidata refreshed: We've updated the knowledge base data. The QID assigned to some extracted entities may differ from previous versions.

Bug fix: We fixed a bug where long words inside long spans of text without punctuation would sometimes be tokenized incorrectly. (ETROG-3716)

Updated real world ID dictionaries: The real world ID dictionaries have been updated. You may see some differences in matching when using real world IDs. (RLPNC-8170)

1 letter misspellings: We added new parameters to control the token score when there is 1 letter difference between 2 versions of a name, or when the edit distance equals 1. To use, set alternateEditDistanceTokenScorerMechanism to true. The token score will then take the value of the parameter alternateEditDistanceTokenScorerMechanismScore, the default value of which is 0.95. (RLPNC-8227)

Bug fixes: 

Bug fixes: 

Updated real world ID dictionaries: The real world ID dictionaries have been updated. You may see some differences in matching when using real world IDs. (RLPNC-8170)

1 letter misspellings: We added new parameters to control the token score when there is 1 letter difference between 2 versions of a name, or when the edit distance equals 1. To use, set alternateEditDistanceTokenScorerMechanism to true. The token score will then take the value of the parameter alternateEditDistanceTokenScorerMechanismScore, the default value of which is 0.95. (RLPNC-8227)

New date parameter: We added a new parameter, boostSwappedDigits, to control whether dates that are identical apart from two adjacent digits being swapped (e.g. '1958-05-02' and '1958-05-20') are scored higher than other dates with an edit distance of 2. This is set to true by default to match previous behavior. (RLPNC-8114)

Bug fix: We fixed a bug where long words inside long spans of text without punctuation would sometimes be tokenized incorrectly. (ETROG-3716)

Wikidata refreshed: We've updated the knowledge base data. The QID assigned to some extracted entities may differ from previous versions.

Bug fix: We fixed a bug with in-document coreference server. In-document coreference chains of entity mentions will now be correct. (TEJ-2655)

Dutch support: Dutch is now supported for event extraction.

New name: The /info endpoint now returns "Babel Street Analytics Server" instead of "Rosette".

Improved Hebrew-English ORG matching: We've improved name matching for organizations for names containing affixes. (RLPNC-7944)

Example: AL-QAID IN IRAQ vs אלקאעדה עירא

Improved Korean name matching: We've improved Korean matching for PERSONS and ORGANIZATIONS by updating the stop word list. (RLPNC-7951)

New parameter: We've added a parameter, maxExpansions to control the number of phonetically similar terms considered during the first-pass fuzzy matching. Increasing this parameter can improve first-pass results, ensuring that the correct name will be sent to the second pass, but may impact performance. (RLPNC-7967)

Improved stop words: We now support stop word prefixes and stop patterns that contain the forward slash (/) characters. This is especially useful for Indian and Malaysian names that include titles which are acronyms, such as A/P, A/L, S/O, and D/O. (RLPNC-7919)

English to Chinese name translation: The English to Chinese name translation is now implemented in Java instead of C++. You may see some differences in translations. (RLPNC-8003)

New character support: Match now supports CJK Unified Ideographs Extension B, which includes rare and historical Chinese characters. This update ensures that characters from U+20000 to U+2A6DF are correctly recognized and processed, improving compatibility with Chinese, Cantonese, Korean, and Japanese data. (RLPNC-7956)

Bug fix: We fixed a bug in Cantonese where the confidence score was always 0.0 if a token had a special character at the end. (RLPNC-7893)

Bug fix We fixed an issue where left and right names in the explain info could appear in the wrong order (RLPNC-7922)

Bug fix: We fixed an issue where removing stop words produced improper segmentation results for Chinese, leading to poor match scores. (RLPNC-8036)

Bug fix: We fixed a bug in Chinese name translation where a StringIndexOutOfBoundsException occurred due to incorrect handling of token dictionary matches. (RLPNC-8009)

Bug fix: We fixed a bug with the handling of transliteration schemes in Cantonese to English translations. You can now set the language of origin and language of use to yue and receive the correct translation without error. (RLPNC-7865)

New output value: The value processedByBabel was added to the response. This is a Boolean value which is set to True when the input value is processed in any way for translation. (RLPNC-7961)

Example: If an input of ABC returns abc, the value will be true.

English to Chinese name translation: The English to Chinese name translation is now implemented in Java instead of C++. You may see some differences in translations. (RLPNC-8003)

Bug fix: We fixed a bug with the handling of transliteration schemes in Cantonese to English translations. You can now set the language of origin and language of use to yue and receive the correct translation without error. (RLPNC-7865)

Bug fix: We fixed a bug in native Chinese name translation where too many results could be returned if a name had multiple segmentations (RLPNC-8058)

Bug fix: We fixed a bug with the handling of transliteration schemes in Cantonese to English translations. You can now set the language of origin and language of use to yue and receive the correct translation without error. (RLPNC-7865)

Known issue: When translating from English to Chinese, the source and target domain information (sourceScript, sourceLanguageOfUse, targetLanguage, targetScript, targetScheme) are not returned in the results. The translation is returned. (RLPNC-8101)

New name: The /ping endpoint now returns "Babel Street Analytics Server at your service" instead of "Rosette at your service".

Wikidata refreshed: We've updated the knowledge base data. The QID assigned to some extracted entities may differ from previous versions.

Unicode update: Unicode 16.0 is now supported. (ETROG-3694)

New name: Rosette Name Indexer has been renamed to Babel Street Match.

Improved L337/OCR scorer: We've improved the GeneralizedEditDistanceTokenScorer for L337 and OCR type errors. (RLPNC-7776)

New parameter added: We've added a parameter charactersToAlwaysNormalizeToSpace to define a set of characters to replace with a space when normalizing names and addresses. (RLPNC-7808)

Improved performance: We've improved performance of English and Spanish name matching. (RLPNC-7782)

Gender ignored by L337/OCR scorer: We no longer apply a genderConflictPenalty in the case of L337 matching.

Updated real world ID dictionaries: We've updated the real world ID dictionary. You may see some differences in matches when using real world IDs, especially for Arabic organization names containing numerals. (RLPNC-7642)

Bug fix: We now use the correct value for sourceLanguageOfUse when translating Cantonese names. The yue/hani/jyutping domain is now used instead of zho/hani/hypy. (RLPNC-7837)

Bug fix: An error is no longer thrown when one of the search terms is normalized away. (RLPNC-7870)

Bug fix: We now use the correct value for sourceLanguageOfUse when translating Cantonese names. The yue/hani/jyutping domain is now used instead of zho/hani/hypy. (RLPNC-7837)

Unicode update: Unicode 16.0 is now supported. (ETROG-3694)

Wikidata refreshed: We've updated the knowledge base data. The QID assigned to some extracted entities may differ from previous versions.

Bug fix: Fixed a bug where REX would accept indoc-coref-server entity mentions which violated sentence boundaries. REX will now reject mentions from indoc-coref-server that are not contained within document sentences. (TEJ-2451)

Bug fix: Trailing decimal points in Chinese are no longer treated as part of a decimal fraction. When “点” or “點” ends a number, it is no longer segmented as part of the number. (ETROG-3680)

l337/OCR error token scorer added: We've added a new feature to handle leetspeak (l337) and OCR errors. The token scorer contains a series of rules files defining symbol substitutions, along with the penalty for the substitution.

Improved explainability: Explain info now contains alternative Latin readings (alternativeLatnData) for translated names. This field displays the alternative Latin readings of translated names from the input-info. (RLPNC-7609)

Example:

{
    "leftInput": {
        "data": "温家宝",
        "normalizedData": "温家宝",
        "latnData": "wen jiabao",
        "script": "Hani",
        "languageOfUse": "CHINESE",
        "languageOfOrigin": "UNKNOWN",
        "entityType": "PERSON",
        "alternativeLatnData": [
            "on kahou",
            "on ka-po"
        ]
    },
    "rightInput": {
        "data": "عبد الله بن عبد الرحمن بن جبرين",
        "normalizedData": "عبد الله بن عبد الرحمن بن جبرين",
        "latnData": "abd allah bin abd alrahman bin jabrin",
        "script": "Arab",
        "languageOfUse": "ARABIC",
        "languageOfOrigin": "UNKNOWN",
        "entityType": "PERSON",
        "alternativeLatnData": [
            "abid allah bin abd alrahman bin jabrin",
            "abd allah bin abid alrahman bin jabrin",
            "abd allah bunn abd alrahman bin jabrin",
            "abd allah bin abd alrahman bunn jabrin"
        ]
    },
    "finalScore": 0.0
}

Improved unsupported language scoring: We added a new parameter editDistanceFalloff to control scoring of names in unsupported languages. The parameter is set to 0 (disabled) by default; a value between 0 and 1 will reduce the dropoff in edit distance. When set to a value between 0 and 1, you may now find matches in scripts and languages that failed to match in previous releases. (RLPNC-7481)

Improved performance: Significant improvement of the HMM used in the high-recall phase of name matching. (RLPNC-7686, RLPNC-7679)

Improved performance: We've added a parameter, disableHMMMatching, which disables the second pass HMM matching. Enabling this parameter greatly increasing performance, but at the cost of reducing accuracy. By default, this parameter is off. (RLPNC-7605, RLPNC-7604, RLPNC-7603)

Faster Arabic translation: We've added a new parameter, tokenByTokenArabLatnFolkTranslation to enable faster translation of Arabic-script, Arabic-language names to Latin-script English names with the Folk transliteration scheme. This speeds up translation by processing the name token-by-token. (RLPNC-7719)

New parameters added: You can now boost a specified number of tokens from the left, right, or both ends. This can be useful when scoring multi-token given names or surnames. The provided value sets the number of tokens from the left and/or right to give a boost. This feature is supported by adding the parameters leftBoostTokens, rightBoostTokens, and bothEndsBoostTokens to the parameter_defs.yaml file. (RLPNC-7606)

Improved Hebrew transliteration: We've improved the FOLK transliteration scheme for Hebrew-English. (RLPNC-7623)

Improved Persian/English matching: We've improved accuracy of Persian-English matching through tuning of parameters. (RLPNC-7631)

Persian organization improvements: We've added stop words for organizations for Persian. (RLPNC-6890)

Chinese migration: Chinese name translation and matching is now implemented completely in Java, instead of C++. You may notice some small fluctuations in Chinese name matching. (RLPNC-6768)

Improved token span definitions: Token spans now attempt to include boundary characters removed from the original input during normalization (RLPNC-7730)

Gender added to pairwise matching: We've added the gender property as a request parameter for /name-similarity. (RLPNC-7451)

Bug fix: We fixed a bug where the seq2seq neural model could not be loaded properly. The model now loads. (RLPNC-7754)

Bug fix: We fixed a bug where if you set haniFourCornerCodeMismatchPenalty globally, it didn't work for any profiles. Users can now either set the haniFourCornerCodeMismatchPenalty globally (enabling it for all Hani matching) or using the language profiles (i.e. specifically for zho_zho).(RLPNC-7717)

Bug fix We removed some erroneous overrides. (RLPNC-7758)

Bug fix We fixed a memory leak in RNT that was causing slowdowns in Rosette Cloud and Rosette Server. (RLPNC-7703)

Faster Arabic translation: We've added a new parameter, tokenByTokenArabLatnFolkTranslation to enable faster translation of Arabic-script, Arabic-language names to Latin-script English names with the Folk transliteration scheme. This speeds up translation by processing the name token-by-token. (RLPNC-7719)

Improved Hebrew transliteration: We've improved the FOLK transliteration scheme for Hebrew-English. (RLPNC-7623)

Bug fix We fixed a memory leak in RNT that was causing slowdowns in Rosette Cloud and Rosette Server. (RLPNC-7703)

Improved explainability in record matching: Request-level information is now included in the information block of the response. Messages concerning the mapping or properties of a request are now part of the info field of the response (RLPNC-7653)

Improved error handling in record matching: Record matching will now throw an exception if a mapping does not contain any valid fields. (RLPNC-7653)

Bug fix: Trailing decimal points in Chinese are no longer treated as part of a decimal fraction. When “点” or “點” ends a number, it is no longer segmented as part of the number. (ETROG-3680)

Wikidata refreshed: We've updated the knowledge base data for the provided knowledge base. The QID assigned to some extracted entities may differ from previous versions. You should see large improvements in entity linking. (RWIKI-454, RWIKI-507)

We've made some changes as to how some entity types are linked to the provided knowledge base:

Linking improvements: We've changed the conflict resolution algorithm to one which tries to link using the longest possible mentions. You should see better linking, especially in cases where the mention of a popular entity is embedded within the mention of interest. (RWIKI-404)

Example: I studied at the University of Chicago

Linking improvements: We've added a heuristic to help stop generic, unnamed entities, such as "mortgage law", from being linked. (RWIKI-475)

New endpoint for supported languages: We've added a new endpoint, entities/indoc-coref-server/supported-languages, which returns the supported languages for indocument coreference. (WS-3176)

Bug fix: We fixed a bug when running REX with multiple threads where stop word files were being loaded and not closed, causing the system to run out of file handles and memory. (TEJ-2347)

Bug fix: We fixed a bug where Chinese characters were normalized when looking up knowledge base artifacts for linking in Japanese. You should see improved entity linking in Japanese. (RWIKI-406)

Bug fix: English terms for half(s) and quarter(s) were removed from the Russian (RUS) and German (DEU) regexes for time. (TEJ-1817)

Bug fix: Aliases are no longer filtered by low normalized link probability; it is now possible to link entities where abbreviations like "MIT", "LA", "WHO", "UN" are the mention text. (RWIKI-389)

Bug fix: We fixed a NullPointerException while writing log entry when processing empty tokens. (TEJ-2361)

Bug fix: We fixed a bug where news media, such as television programs, were typed as ORG. (RWIKI-483).

Known issue: If the indoc-coref-server is enabled, errors may be returned from the /sentiment, /topics, and /relationships endpoints if multiple entity mentions are returned for the same entity. (RQA-1345)

CLA lexicon: Two terms have been added to the CLA lexicon: 喷码机 inkjet printer and 管理器 manager (in the context of software) (ETROG-3678)

Improved Readings: Readings are now returned for numeric words in Chinese and Japanese when tokenizerType is set to SPACELESS_LEXICAL. (ETROG-3684)

Bug fix: When tokenizerType is set to SPACELESS_LEXICAL, Japanese tokens for verbs in lemma form have had their readings fixed to cover the entire token. (ETROG-3640)

Example: Input: 食べる

Bug fix: When tokenizerType is set to SPACELESS_LEXICAL, Japanese lemmatization has been corrected for numeric tokens containing both decimal points and multiplier characters.

Example: Input: 2.5亿

Bug fix: We fixed a bug where a token whose surface form was the empty string could be returned when fragmentBoundaryDetection was set to true (the default). (ETROG-3686)

Japanese translation improvements: We've improved Japanese translation by updating the custom reading dictionary (RLPNC-7539).

Hebrew translation overrides: Added additional overrides for Hebrew translation. (RLPNC-7471)

Non-Latin numeric characters: Numeric characters in certain languages are now normalized to their Latin-script counterparts. Supported languages currently include Thai (RLPNC-7562), Arabic, Burmese, Pashto (RLPNC-7564), Persian (including Iranian and Afghan Persian), Urdu, and Khmer (RLPNC-7565).

Pashto organization improvements: We've added stop words for organizations for Pashto. (RLPNC-6889)

Spanish organization improvements: We've added stop words for organizations for Spanish (RLPNC-6893)

Bug fix: We fixed a bug with cross-entity-type matching: cross-entity-type match scoring is now commutative. As a result of this, cross-entity-type matching will now ignore entity-type-specific parameters and overrides. (RLPNC-7485).

Bug fix: We fixed a bug where khm-khm was returned as a supported language pair for name translation. (RLPNC-7556)

Bug fix: We fixed a bug, that caused the left and right input fields in the explain info to swap places, when a Japanese organization name is matched against a single character, that is normalized away (like '*') and the language of use is not defined on that side. (RLPNC-7554)

Japanese translation improvements: We've improved Japanese translation by updating the custom reading dictionary (RLPNC-7539).

Hebrew translation overrides: Added additional overrides for Hebrew translation. (RLPNC-7471)

Bug fix: We fixed a bug where khm-khm was returned as a supported language pair for name translation. (RLPNC-7556)

Fielded dates supported: We've added support for fielded dates. (RLPNC-7520)

Fielded addresses supported: We've added support for fielded addresses. (RLPNC-7519)

Blank fields supported: We've added support for specifying a scoreIfNull for fields in record similarity. You can now specify a value for when a field is missing in a record. (RLPNC-7516, RLPNC-7517)

Parameter support improved: We've added support for specifying either a parameter universe or a mapping of parameter names to parameter values. (RLPNC-7497)

Validation improved: We've improved validation of field mappings and field weights. (RLPNC-7545)

Info messages added: Record matching responses now return info messages when default property values are used.(RLPNC-7509)

Error reporting improved: Fields not included in the mapping or fields with unknown types no longer cause an error in record matching. Instead, this information is returned in the "info" block. A record with only non-included fields or fields with unknown types will lead to an error in the response. (RLPNC-7512, RLPNC-7514)

Partial records supported: Record similarity now supports partial request success if some record pairs contain unmapped or unknown fields, or encounter other scoring errors. (RLPNC-7502)

More records and fields supported: We've removed hard limits on the number of records or mapping fields in record-similarity requests (RLPNC-7601)

Known issue: If the indoc-coref-server is enabled, errors may be returned from the /sentiment, /topics, and /relationships endpoints if multiple entity mentions are returned for the same entity. (RQA-1345)

Known issue: If the indoc-coref-server is enabled, errors may be returned from the /sentiment, /topics, and /relationships endpoints if multiple entity mentions are returned for the same entity. (RQA-1345)

Bug fix: We fixed a bug where the Chinese word “星期四” would be tokenized incorrectly in certain contexts. (ETROG-3582)

Known issue: If the indoc-coref-server is enabled, errors may be returned from the /sentiment, /topics, and /relationships endpoints if multiple entity mentions are returned for the same entity. (RQA-1345)

Bug fix: Fixed the error being returned when matching addresses. Only English and Chinese are supported for address matching; RNI will now throw an unsupported language exception when matching non-English, non-Chinese addresses if allLanguageSupport is disabled. (RLPNC-7416)

Indoc Coreference: We've added a server to provide in-document coreference (indoc coref). With indoc coref enabled, all entity mentions, including pronouns, titles, and other references to an entity, are returned in the ADM output. To enable indoc coref, set the option useIndocServer to true. By default, indoc coref is disabled. (TEJ-2244)

New event extractor: Event extractor analyzes unstructured text and extracts event mentions and the roles (event mentions) which add detail to the event. We've included two simple event models, travel and meet, to demonstrate how events works.

Unicode update: Unicode 15.1 is now supported. (ETROG-3595)

Bug fix: Upper case input text is now supported. Previously, the endpoint would send an error message Language uen not supported. (WS-3163)

Malay support expanded: We have improved name Malay matching by expanding the stop word list. (RLPNC-7175, RLPNC-7176)

Hebrew improved: We have improved name matching and translation for Hebrew by expanding translation override lists. (RLPNC-7234)

Explain info improved:

Bug fix: Fixed a bug in which stop words were not being applied properly for Greek. (RLPNC-7144)

Bug fix: Names in Han script with unknown language and unknown language of origin now give appropriate Japanese, Chinese, and Korean readings. (RLPNC-7367)

Bug fix: Fixed token tagging in names that end with a suffix. (RLPNC-7417)

Bug fix: Names that get normalized to empty now return an empty list of Real World IDs. (RLPNC-7202)

Bug fix: Overrides are no longer considered for the gender penalty. (RLPNC-7346)

Bug fix: Confidence scores for fullname overrides are now correctly calculated when at least one token has a confidence score specified. (RLPNC-7456)

Bug fix: Fixed token tagging in names that end with a suffix. (RLPNC-7417)

Multiple translations: We've added a parameter, maximumResults, to return multiple translations along with their confidence scores. The default is to return a single translation. (RLPNC-7350)

Hebrew improved: We have improved name matching and translation for Hebrew by expanding translation override lists. (RLPNC-7234)

Record Similarity: We've added a new endpoint to compare two lists of records and return a similarity score for each pair. Each record can contain one to five fields of mixed data types. Check it out. (RLPNC-7372)

New embeddings for French and Italian: The GEN_2 embeddings (originally released in June2023) are now available for French and Italian. These embeddings provide more accurate results and are debiased compared to the previous embeddings. You may see differences in returned values for these languages. To use the previous embeddings, set embeddingsMode to GEN_1. (RD-2632) 

Korean embeddings: North Korean and South Korean embeddings are no longer distinguished by default. The default embedding mode (GEN_2) treats them the same. If you need to distinguish between North Korean and South Korean embeddings, set embeddingsMode to GEN_1.

Bug fix: Upper case input text is now supported. Previously, the endpoint would send an error message Language uen not supported. (WS-3163)

Unicode update: Unicode 15.1 is now supported. (ETROG-3595)

Bug fix: Upper case input text is now supported. Previously, the endpoint would send an error message Language uen not supported. (WS-3163)

Bug fix: We've added a reject gazetteer to ensure that the string USA, Canada is extracted correctly as 2 location entities: USA and Canada. This is active by default. (TEJ-2114)

Japanese improvements: We have improved and augmented the Japanese lexicon. (ETROG-3532, 3581, 3535, 3668)

Cyrillic support expanded: Extended Cyrillic characters are now supported. This will improve performance for non-Russian, Cyrillic languages. (RLPNC-7236)

Added parameter hmmNormalizationAlternative: This parameter adjusts normalization for more accurate HMM match scores in certain languages. It is available for Russian, Hebrew, Korean, Japanese, Arabic, and Greek. When this parameter is enabled, HMM scores may be lowered. It is disabled for all languages by default. This can lower the probability of names being translated, resulting in them being transliterated instead, which may be more accurate for names in some languages. (RLPNC-7192).

Cyrillic support expanded: Extended Cyrillic characters are now supported. This will improve performance for non-Russian, Cyrillic languages. (RLPNC-7236)

Source language returned: The response now always includes sourceLanguageOfUse and sourceScript. If these values are not included in the request, they are the values determined by the endpoint. (RLPNC-7170)

Cyrillic support expanded: Extended Cyrillic characters are now supported. This will improve performance for non-Russian, Cyrillic languages. (RLPNC-7236)

Japanese improvements: We have improved and augmented the Japanese lexicon. (ETROG-3532, 3581, 3535, 3668)

Expanded Chinese lexicon: We've expanded the lexicon of multi-character Chinese surnames when tokenizerType is set to spaceless_lexical. (ETROG-3616)

Expanded Japanese lexicon: We have expanded the Japanese lexicon that is used when tokenizerType is set to spaceless_lexical. (ETROG-3632)

Improved support for Chinese readings when tokenizerType is set to spaceless_lexical:

Bug fix: We fixed a bug where an ArrayIndexOutOfBoundsException occurred when the Chinese dictionaries produced more than 6 matches and tokenizerType was set to spaceless_lexical. (ETROG-3635)

Bug fix: When Chinese readings are constructed by character and tokenizerType is set to spaceless_lexical, an apostrophe is now inserted before pinyin syllables that start with "a", "e", or "o" which are not the first syllable. (ETROG-3637)

Bug fix: We fixed a bug where the UPT-16 conversion where some Japanese particles part of speech were not tagged correctly. The particles are now tagged correctly as ADP. (ETROG-3526)

Malay support added: Added support for Malay-English and Malay-Malay name matching. (RLPNC-7079)

Known issue: When performing address matching, the setting allLanguageSupport must be set to true (the default value). If it is set to false, an unsupported language exception will be thrown, regardless of the language of the address. By default, it is set to true.

Bug fix: The parameter regexCurrencySplit has been fixed. When set to true, currency values will now extract into two entity types: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE instead of IDENTIFIER:MONEY. (TEJ-1960)

Improved organization name matching The stop word list for organization entities has been expanded in the following languages (RLPNC-7025):

New embeddings: We've replaced the embeddings used by the /semantics/similar and /semantics/vector endpoints. The new embeddings (GEN_2) provide more accurate results and the results are debiased compared to the previous embeddings. You may see some differences in returned values. The new embeddings are the default. French and Italian continue to use the original GEN_1 embeddings. To use the previous embeddings, set embeddingsMode to GEN_1. (RD-2575)

Bug fix: The deprecated genre parameter is now allowed in document requests.

We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

Turkish support added: Added support for Turkish-English and Turkish-Turkish name matching. We have also added person and organization overrides, stopwords, and language detection to improve matching in Turkish. (RLPNC-6499)

Improved person name matching: RNI-RNT now has the ability to detect given names and surnames in Latin script when the name is of English origin. When the enableAdditionalOnomastics parameter is true, gender mismatch penalty is only applied to the detected given name, as opposed to the first name token in a query. (RLPNC-6719, RLPNC-6720)

Improved Arabic person name matching: The new TRAILING_PATRONYMIC_DELETION match phenomenon provides improved scores for matches which contain a deletion that is caused by truncation of a patronymic. The score of this deletion is controlled by the trailingPatronymicDeletionScore parameter. This only applies to Latin script names of Arabic origin when enableAdditionalOnomastics is true. (RLPNC-6756)

Russian support migrated from C++ to Java: Translations from Russian to English may differ slightly. (RLPNC-6764)

We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

Improved Chinese address matching: We've expanded the list of Chinese stop words for addresses. (RLPNC-6587)

Wikidata refreshed: We've updated the knowledge base data for the provided linking knowledge base. The QID assigned to some extracted entities may differ from previous versions. (RWIki-119, ELK-274, ELK-276)

New currency regex: We've introduced a new option, regexCurrencySplit, that, when set to true, will attempt to split entities extracted with the regex engine of type IDENTIFIER:MONEY into two new entities: IDENTIFIER:CURRENCY_AMT and IDENTIFIER:CURRENCY_TYPE. These two new types represent the amount of the currency (50,000) and the currency type ($), respectively. By default, regexCurrencySplit is set to false. (TEJ-1792)

Tagalog support: We've added case-insensitive NER support for Tagalog. Previously we released a case-sensitive model and we've now added the case-insensitive model as well. (TEJ-1858)

Parameter removed: We've removed the deprecated genre extraction option. This option was used to turn the linker on which has been, and will still be, available by the linkEntities option. The genre option is no longer available in the rex-factory-config.yaml file, as an option in the call, or in the Rosette API bindings (TEJ-1855).

Ukrainian support added: Tokenization, sentence boundary detection, segmentation user dictionaries, and many-to-one normalization dictionaries are supported for Ukrainian. (ETROG-3594)

Improved part of speech tags: Language-neutral tokens (numbers, symbols, and punctuation) now get part of speech tags in Indonesian, Standard Malay, and Tagalog. (ETROG-3574)

Emoji support: Emoji 15.0 is now supported. (ETROG-3577)

Bug fix: The Japanese POS tag VN (verbal noun) is now mapped to the UPT-16 POS tag NOUN. It was previously mapped to VERB. (ETROG-3583)

Improved Japanese organization matching: カンパニー (company) added to the Japanese organization stopwords list. (RLPNC-6545)

Improved Chinese organization matching: We've expanded the list of Chinese stop words for organizations. (RLPNC-6615)

Improved name matching results: When no entityType is specified, the type PERSON will be applied. Previously, the type NONE was applied. (RLPNC-6576)

Bug fix: Horizontal tabs are now removed as part of normalization in English. (RLPNC-6541)

Bug fix: Control characters are now removed from Arabic names before matching. (RLPNC-6543)

Bug fix Fixed a case where unexpected name inputs could lead to a null pointer exception. (RLPNC-6634)

Bug fix Fixed an issue where name-similarity could return match scores greater than 1. (RLPNC-6595)

Ukrainian support added: Sentence boundary detection is now supported for Ukrainian. (ETROG-3594)

Ukrainian support added: Tokenization is now supported for Ukrainian. (ETROG-3594)

Tagalog (tgl) support: We've added Tagalog to our list of languages. The following processors are supported: gazetteer, regex, statistical NER, linking. (TEJ-1812, TEJ-1822, TEJ-1785, TEJ-1786)

New linking option: We've added a new option for entity linking. When linkMentionMode is set to entities the linker will attempt to link the entities extracted by other processors (regex, gazetters, and the statistical processor) instead of using its own processor to extract entity candidates. Depending on your data, this may provide higher accuracy and speed. (TEJ-1806)

Bug fix: An exception is no longer emitted when token normalization produces an empty string. (TEJ-1803)

Bug fix: When looking for candidate mentions in text, if there is an overlap between these mentions, the linker now resolves the longest spanning mention before disambiguation. (ELK-277)

Tagalog (tgl) support:

Indonesian (ind) support: RBL now supports lemmatization for Indonesian, which is the standardized form of Malay spoken in Indonesia. (ETROG-3563)

Standard Malay (zsm) support: RBL now supports lemmatization for Standard Malay, the standardized form of Malay spoken in Malaysia. (ETROG-3563)

Bug fix: We fixed a bug in Russian where certain uncommon consonant–vowel sequences in words in the lexicon were incorrectly replaced with more common sequences with different vowel letters. (ETROG-3541)

Example: брошюра

Complete CJK Ext A support: We now have full support of CJK Unified Ideographs Extension A. (RLPNC-6324)

Improved Spanish name matching: We have improved Spanish surname detection. (RLPNC-6294)

Improved Japanese location matching: All prefectures of Japan are now included in the override list. (RLPNC-6326)

Example: 北海道 vs. Hokkaido Prefecture

Bug fix: The string "luiz arlos da silva bueno" is no longer in the Greek, English, and Vietnamese stop word lists. (RLPNC-6358)

Complete CJK Ext A support: We now have full support of CJK Unified Ideographs Extension A. (RLPNC-6324)

Tagalog (tgl) support: The Tagalog sentence-breaker now recognizes certain abbreviations that end with periods and doesn’t break sentences after them. The tokenizer keeps the period in the token with the rest of the abbreviation. (ETROG-3573)

Tagalog (tgl) support: The Tagalog sentence-breaker now recognizes certain abbreviations that end with periods and doesn’t break sentences after them. The tokenizer keeps the period in the token with the rest of the abbreviation. (ETROG-3573)

Improved Chinese - English address matching: We expanded overrides for ethnic minority regions, particularly from Xinjiang, Tibet, and Inner Mongolia. (RLPNC-6077)

Bug fix: An error is no longer generated when there are null prefixes in Arabic morphological analyses. (TEJ-1765)

Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)

Indonesian support added: Rosette now supports part of speech (POS) tagging in Indonesian. (ETROG-3543)

Malay (standard) support added: Rosette now supports part of speech (POS) tagging in Malay (standard). (ETROG-3545)

Russian lexicon improved: We've added many words related to computer technology to the Russian lexicon. (ETROG-3523, ETROG-3538)

Bug fix: In Japanese, negative forms of Ichidan verbs written all in Hiragana are no longer lemmatized to end with “なう”. (ETROG-3534)

Example: Input: くれない

Khmer - English added: Khmer - Khmer and Khmer - English are now supported name matching pairs. (RLPNC-5712)

Improved language detection: We've improved language detection for languages that use Han characters (Chinese, Japanese, Korean). (RLPNC-6059)

Improved ORG matching: We've expanded the list of known organization names in our real world ID tables to improve ORG matching in Arabic (ara), Burmese (mya), Chinese (zho), French (fra), German (deu), Greek (ell), Hebrew (heb), Hungarian (hun), Italian (ita), Japanese (jpn), Korean (kor), Portuguese (por), Russian (rus), Spanish (spa), Thai (tha), and Vietnamese (vie). (RLPNC-6090)

Improved Chinese - English ORG matching: We added override mappings for Chinese numerals in Hanzi to Arabic numbers from zero through twenty-one. (RLPNC-6028)

Improved matching of Spanish names and names of Spanish origin: Name similarity now has a deeper understanding of Spanish surnames. For example: "JOSE JORGE RIOS TORRES" now gets a higher score when matched against "JOSE RIOS" than it does when matched against "JOSE TORRES", since "RIOS" is recognized as the primary surname. (RLPNC-6037)

The following parameters impact how Spanish names are matched:

Improved matching of English organization names: We've added ordinal numbers to the override list for English organizations. (RLPNC-6225)

Example: 1st National Bank vs. First National Bank

Improved Vietnamese name matching: We've expanded the Vietnamese stop word lists for PERSON and ORGANIZATION entity types. (RLPNC-5694)

Example: Chủ tịch Hồ Chí Minh vs. Hồ Chí Minh (translation: President Ho Chi Minh)

New parameter for organization names: We've added the parameter tokenizeOrganizationsWithNumbers that prevents tokenization of names with numbers within the name. When set to true (default), the number is left within the token and the name will get a higher value from the edit distance scorer. This is desirable if your data contains organization names which intersperse alphabetic and numeric characters or if your data often contains typographical errors with numerals inserted into otherwise valid tokens. (RLPNC-6200)

Improved Japanese-English location name matching: We've expanded the Japanese-English overrides list for location names. (RLPNC-6268)

Example: 大阪府 vs. Osaka Prefecture

Bug fix: Pairwise match now works with all languages that have limited language support. Previously, an error was returned for unidentified languages. (RLPNC-6100)

Bug Fix: Name similarity will no longer return match scores above 1.0. (RLPNC-6254)

Khmer - English added: Khmer to English is now a supported name translation language pair. (RLPNC-5708)

Cantonese support added: We can now transliterate Han into Latin characters using the Jyutping transliteration scheme for Cantonese. (RLPNC-6232)

New Language support: The Semantic Similarity endpoints now support Tagalog (tgl). (RD-2546)

Spaceless Korean tokenizer: We've added an option to select the spaceless Korean tokenizer in the call. The default tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. To use this tokenizer, set the option modelType to DNN. Previously, this option was not available on a per-call basis. (ETROG-3513)

Chinese address matching: We now support Chinese-Chinese and Chinese-English address matching. (RLPNC-5822)

Improved address matching: We've modified the field weight values to provide more accurate address match scores. Weightings were determined by evaluating US and UK address data. (RLPNC-5893)

Example: "85 Court Road Newton Ferrers, Plymouth PL8 1DE1B Devon, England UK” vs “85 Court Road Newton Ferrers PL8 1DE UK"

Bug fix: The supported entity types info of the DNN processor now spells PERSON correctly. (TEJ-1670)

Bug fix: Hungarian dates are now extracted correctly. Previously, dates with embedded periods followed by a space were not being extracted. (TEJ-1681)

Bug fix: The entities/info endpoint now returns the complete list of valid entity types. (WS-2360)

Bug fix: The entities/info endpoint no longer lists TEMPORAL types by default for SWEDISH. (TEJ-1687)

Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)

Example: Input: 三・一四

Emojis: U+3030 and U+303D are now tagged as emojis even when not followed by U+FE0F. (ETROG-3478)

Emoji support: We now support the emoji in Unicode 14.0 (ETROG-3476)

Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)

Improved POS tags: Many number, punctuation, and symbol characters are now POS-tagged appropriately as numbers, punctuations, and symbols instead of being marked as unknown or some other tag. This applies to all languages with POS tags. (ETROG-3481)

Hungarian improvements: We've added some Hungarian abbreviations and improved sentence boundary detection around Hungarian abbreviations. (ETROG-3479, ETROG-3484)

Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

Bug fix: We've reverted two of the POS changes made in version 1.19.0 as they introduced regressions in Chinese and Japanese. (ETROG-3466)

The values are now:

Bug fix: Morphology no longer detects characters as emoji when followed by the text presentation selector (U+FE0E). (ETROG-3480)

Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

Burmese transliteration: We added a Basis Technology-created Folk transliteration scheme for Burmese name matching that is similar to how Burmese names are commonly transliterated to English. (RLPNC-5892)

Basic support for all languages: Name similarity can now match names in any language. Languages which previously would have returned an "unsupported language" error now return a match score. The score is either 1 for a perfect match, or a value based on edit distance. (RLPNC-5979)

Improved ORG matching: We added real world ID tables or organizational names to improve ORG matching in the following languages: Thai (tha), Greek (ell), Hebrew (heb), Burmese (mya), German (deu), French (fra), Hungarian (hun), Italian (ita), Portuguese (por), Spanish (spa), and Vietnamese (vie). (RLPNC-5986)

Example: "International Astronomical Union" vs. "האיגוד האסטרונומי הבינלאומי"

Improved Hebrew name matching: We've added a rule-based vocalization checker for the statistical-model vocalizer to improve Hebrew-Hebrew and Hebrew-English name matching. (RLPNC-5990)

Improved Hebrew transliteration: The Hebrew character ח used to be transliterated as “h” in some cases and “kh” in others (if it was followed by a geresh). It is now transliterated as “ch"when not followed by a geresh. The Hebrew character כ used to be transliterated as “h” in some cases and “k” in others (if it has a dagesh). Now, it is transliterated to “ch” in the cases when it used to be transliterated to “h”. (RLPNC-5928)

Example: נחמן

Example: מיכל

Bug fix: Burmese-English transliteration has been improved by revising the Folk and MLCTS transliteration schemes. (RLPNC-5950)

Bug fix: Hebrew Folk transliteration has been improved, especially for the letters vav and yod. (RLPNC-5916)

Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts in Japanese. (ETROG-3474)

Example: Input: 三・一四

Japanese tokenization: In Japanese, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. (ETROG-3475)

Bug fix: In Japanese, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

Bug fix: In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

Improved address matching: We've expanded the override tables for UK, U.S., and Canadian addresses. (RLPNC-5886)

Example: houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<Aberdeenshire>postCode<AB25 1XT> vs. houseNumber<47>road<Albert Street>city<Aberdeen>stateDistrict<ABD>postCode<AB25 1XT>

Bug fix: Overrides for alphanumeric address fields (houseNumber, unit, poBox, postCode) are now being applied. (RLPNC-5863)

Example: “3710 W Martin Luther King Blvd STE #121” vs. “3710 W Martin Luther King Blvd Suite #121”

Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions. (TEJ-1657, TEJ-1658)

New /info endpoint: The entities/info endpoint returns a list of supported entity types available by processor type and language.

New option for structured regions: The option structuredRegionProcessingType has been added to specify the type of processing for structured regions. By default, the statistical/DNN model is turned off for structured regions, which increases precision, but may result in reduced recall in structured regions.

The existing option enableStructuredRegion does not determine how structured regions get processed. When enableStructuredRegion is set to true and the input is contentUri (i.e. set to retrieve content from a URL), then HTML lists and tables will be extracted by Tika as structured regions. The value of structuredRegionProcessingType will then determine how those structured regions are processed.

Bug fix: Confidence scores for language regions are now always within the range [0,1]. (RLIJE-552)

Improved Hebrew-English name matching:

Improved Hebrew-Hebrew name matching: We expanded the token overrides for person entity types. (RLPNC-5891)

Example: סולומונתאס vs. סולונאס

Improved English-English name matching: We added the token override pair Alex/Aleksandar. (RLPNC-5871)

Example: Alex vs. Aleksandar

Improved matching for identifiers: We improved matching and added support for three new subtypes: IDENTIFIER_DRIVERS_LICENSE, IDENTIFIER_LICENSE_PLATE, IDENTIFIER_NATIONAL_ID_NUM, along with IDENTIFIER_GENERIC. (RLPNC-5852)

Example: NH123456789DL vs. NH123456789DN (as IDENTIFIER_DRIVERS_LICENSE entity type)

Improved Japanese Segmentation: We've expanded the segmentation dictionary to improve Japanese name segmentation. (RLPNC-5835)

Example: ミロシェヴィッチスロボダン

Bug fix: Hebrew tokens containing diacritics are now identified in the override table. (RLPNC-5882)

Example: אֲבִי vs. Abigail

New language support: Sentence tagging is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

New language support: Tokenization is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

Improved handling of postal codes: Better match scores result from this enhancement. (RLPNC-5639)

Example: houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140 1234> vs. houseNumber<123>road<Clifton St>city<Cambridge>state<MA>postCode<02140-1234>

Expanded UK and CA address override tables: We expanded override tables for UK and Canadian addresses. (RLPNC-5607)

Example: houseNumber<100>road<Main Ave>city<Shellbrook>state<Saskatchewan>postCode<S0J 2E0> vs houseNumber<100>road<Main Ave>city<Shellbrook>state<Sask>postCode<S0J 2E0>

New default processing for structured text regions (lists, tables): Because structured text is often just words or phrases, and thus missing the syntactic context that REX was trained on, some REX users would pre-process input text to remove structured regions, on which REX performed poorly. Users no longer have to pre-process the input as now the statistical/DNN model is turned off by default for structured regions. This mode increases precision but may result in reduced recall in these regions. Note, the other REX processors (pattern match, exact match, entity linking) which do not rely on context will continue to analyze the structured regions. (TEJ-1502) (TEJ-1502)

New name classifier model for structured regions (LABS): We've added a new model for processing structured regions. Each sentence or fragment (the structured region) is classified as a single entity. The name classifier classifies the entity as PERSON, LOCATION, or NONE. It is disabled by default. It can be enabled by using the enableStructuredRegion option in the call. (TEJ-1613, TEJ-1621)

New option for structured regions: The option enableStructuredRegion has been added to configure how structured regions are processed. The default value is false. When set to true:

Japanese organization gazetteers: The gazetteers for Japanese organizations has been updated to improve extraction of Japanese organizations. (TEJ-1612)

Bug fix: Invalid whitespace handling by the DNN processor no longer causes a runtime exception. (TEJ-1614)

Bug fix: A previous entity offset alignment issue involving very short regex-captured entities followed by \r is fixed.

Bug fix: We reverted some changes to Korean tokenization from the 1.18.0 release. You may see minor differences in extracted entities in Korean when comparing results from the previous release.

Bug fix: Entities are no longer extracted when they cross a sentence boundary. To enable entity linking across sentence boundaries, set disableApplySentenceBoundaries to true. (ELK-259)

Bug fix: Entities are now checked to ensure they are normalized. (TEJ-1615)

Greek lexicon: The Greek lexicon has additional nouns. (ETROG-3288)

New Greek disambiguator: There is a new, more accurate Greek disambiguator. (ETROG-3304)

Greek disambiguation improved: Certain Greek forms are now disambiguated to prefer a modern analysis over an archaic analysis. (ETROG-3289)

Example: δείξε

Emoji normalization: Morphology no longer normalizes certain emoji zero-width joiner (ZWJ) sequences to U+1F48F KISS, U+1F491 COUPLE WITH HEART, and U+1F46A FAMILY, to be consistent with Unicode’s efforts to make emoji more gender-neutral by default. (ETROG-3350)

Improved Hebrew tokenizer and new analyzer: The Hebrew tokenizer is now more consistent with the tokenizers of other languages. (ETROG-3290)

Normalized characters: Normalized half-width and full-width characters are processed the same as their counterparts. (ETROG-3351)

Bug fix: Consecutive punctuation characters are no longer returned as a single token in Chinese. Now each character is its own token. (ETROG-3402)

Example: Input: 天津？？

Bug fix: We reverted some changes to Korean disambiguation from the 1.18.0 release as the changes introduced new disambiguation errors. You may see minor differences in Korean disambiguation when comparing results from earlier releases.

Bug fix: Hebrew tokens containing a geresh are now tokenized properly. Previously, only the part up to the geresh would be returned as the token, and the part after the geresh would sometimes be considered a suffix. Now all the characters of the token are returned as the token. ETROG-3290)

Example: מע'רב

Improved name deduplication: We added support for name overrides so that, for example, the nickname “Mike” is included in the same cluster with “Michael”. (RLPNC-5600)

Burmese-English added: Burmese-Burmese and Burmese-English are now supported name matching pairs. (RLPNC-5660)

Hebrew-English added: Hebrew-Hebrew and Hebrew-English are now supported name matching pairs. (RLPNC-5339)

Vietnamese-English added: Vietnamese-Vietnamese and Vietnamese-English are now supported name matching pairs. (RLPNC-5687)

Improved Chinese-English: Name similarity scores for Chinese-English now leverage English translations from a list of Chinese name translations. (RLPNC-5643)

Example: 汤姆 vs. Tom

Improved English-English organizations: We expanded the overrides list with numbers and their written form from 1 to 21. (RLPNC-5644)

Example: Channel One Russia vs. Channel 1 Russia

Improved name matching for organizations: Name similarity scores for organizations has been improved by adding new frequency models in English, Chinese, Arabic, Japanese, and Russian. (RLPNC-5416)

Updated frequency model: We updated the English frequency model for PERSON entity type by adding birth names from 1920-2019 to the existing model. (RLPNC-5592)

Character normalization improved: We've improved character normalization for all supported languages. All languages except for Japanese, Korean, and Arabic-script languages now have NFKC normalizations performed. Japanese, Korean, and Arabic-script languages have NFKD normalizations performed.

Japanese ORGANIZATION type matching improved:  We've improved the accuracy of ORGANIZATION matching for Japanese-English and Japanese-Japanese names by expanding the stop word list for the ORGANIZATION entity type.

Example: コダック合同会社 vs. Kodak Limited

Better entity resolution for Russian ORGANIZATION type matching:  We've improved Russian-English and Russian-Russian name matching by adding support for Russian organizations in the entity resolution engine.

Example: Ура́льские авиали́нии vs. Ural Airlines

Expanded stop word list for Russian ORGANIZATION type matching:  We've improved the accuracy of ORGANIZATION matching for Russian-English and Russian-Russian names by expanding the stop word list for the ORGANIZATION entity type.

Example: Балтийский федеральный университет имени Иммануила Канта vs. Immanuel Kant Baltic Federal University

Korean ORGANIZATION type matching improved:  We've improved Korean-English and Korean-Korean name matching by adding support for Korean organizations in the entity resolution engine.

Example: 현대자동차 vs. Hyundai Motor Company

Bug fix: We fixed a bug where Arabic-Arabic name matching was returning a low score in some cases for names that were seemingly very similar.

Example: عبدلخالق الحوثي vs. عبدالخالق الحوثي

Burmese-English added: Burmese to English is now a supported name translation language pair. (RLPNC-5662)

Example: မင်း အောင် လှိုင် ⟹ Maang Aaung Lhuing

Wikidata refreshed: The internal database for Wikidata linking has been refreshed and re-indexed. QIDs for some entities may change from previous versions.

Bug fix: The sample for the SQLite-kb-connector now works correctly. Runtime issues with SQLite dependencies have been corrected.

Bug fix: Extraction no longer fails when a custom processor returns a NULL annotator; a warning is issued instead.

Bug fix: Mentions normalized by a custom processor are no longer ignored.

Bug fix: Newlines \r and \r\n are now handled correctly.

Deprecated option: The alternative tokenization option deliverExtendedAttributes is now deprecated. Previously it delivered an unsupported extended property.

Bug fix: Combining characters in Hebrew which were being erroneously split into tokens separated from their bases are no longer being split.

Bug fix: Certain patterns of white-space within Chinese and Japanese tokens no longer cause an internal server error.

Bug fix: The mappings of default Basis POS tags to universal POS tags ("options": {"partOfSpeechTagSet": "upt16"}) have been corrected for Greek.

Bug fix: When fstTokenize is enabled, inputs in Spanish are no longer truncated.

Improved ORGANIZATION type matching: We improved the accuracy of ORGANIZATION name matching by integrating name completion with an internal entity resolution engine.

Example: ソニー株式会社 vs. Sony Corporation

Improved Japanese: We've improved the segmentation of Japanese PERSON names.

Example: スズキタロウ

Spanish token overrides: The Spanish-Spanish token overrides for the PERSON entity type have been expanded.

Example: Francisco vs. Paco

Languages added: The semantic similarity endpoints now also supports Hebrew, Hungarian, Italian, Persian, Portuguese, and Urdu. 

License expiration: The license expiration date has been added to the retrieved API information.

Bug fix: We fixed a bug in the Arabic statistical model and retrained the model.

Latvian support added: Latvian lemmatization is supported. 

Latin-script tokens in Russian: Latin-script regions within Russian inputs are now tokenized and analyzed as English. Example input: “мой новый iPhone”:

Bug fix: When Rosette guessed German compounds they were sometimes lemmatized as verbs but tagged as nouns.

Bug fix: There was a pairwise matching bug where Arabic name A vs Arabic name B produced a different result than Arabic name B vs Arabic name A.

The fragment boundary detector now marks a boundary after any spaces following a fragment boundary delimiter.

Bug fix: There was a sentence break after every Windows newline (i.e. carriage return + line feed).

Bug fix: Multi-script Russian text would have a sentence break each time the script changed.

Bug fix: There were unexpected sentence breaks after some short lines which did not end in whitespace.

Bug fix: Sentence breaks were missing when the sentence break did not align with a token boundary.

Latvian support added: Latvian tokenization is supported. 

Configurable fragment boundaries: The delimiters for the fragment boundary detector are now configurable. A delimiter is restricted to a single character. To configure, edit the value of fragmentBoundaryDelimiters in the rbl-factory-config.yaml file. The string should contain all values that should be recognized as a fragment boundary, including any of the default values you want to keep.

Bug fix: An underscore (U+005F) is no longer treated as a token separator in German when fstTokenize is enabled.

Bug fix: Tokens from multi-script Russian text sometimes had incorrect offsets if fstTokenize was enabled.

Improved address matching: We've boosted the score of tokens that differed only by an inserted or missing space.

Improved address matching: We've added support for additional Spanish and English address abbreviations, such as:

Swedish: We have added entity extraction and linking support for Swedish (swe).

Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on. Previously it was off. 

New imperative Arabic verbs: We've added the imperative forms of 2000 Arabic verbs to the Arabic lexicon.

Bug fix: In Russian, embedded spaces have been removed from the lemmas of numbers containing spaces. For example, the token "1 234" is now lemmatized to "1234" instead of "1 234".

Bug fix: In Japanese, if a middle dot appeared in the input text immediately before a newline character, it was missing from the tokenized output. Now the middle dot gets a token in the output.

Improved Japanese-English matching for organizations: We've expanded the Japanese-English token overrides for organizations. Overrides explicitly map nicknames, cognates, and variants to improve matching accuracy.

Rosette Enterprise can now support multiple profiles for users that have different processing needs. Each profile consists of a specific set of parameter and configuration settings (e.g., case-insensitive on/off, entity linking on/off) and different data domains (e.g., enabling specific regex and gazetteer files). Custom profiles do not apply to the /address-similarity, /name-similarity, /name-deduplication, and /name-translation endpoints.

Refer to the section Custom Profiles in the Enterprise User Guide for more details.

Dynamic gazetteers: We’ve added a new endpoint, /entities/configuration/gazetteer/add, to add gazetteer entries to the /entities endpoint without having to stop and restart Rosette Enterprise.

Refer to the section Adding Dynamic Gazetteers in the Enterprise User Guide for more details.

Fragment boundary detection: The fragment boundary detector, which is used to better process data in lists and tables, is now on by default. Previously it was off by default.

To turn off fragment boundary detection:

New model for Katakana-Latin name matching: We've added a new deep learning model that improves Katakana-Latin name matching. It is enabled by setting enableSeq2SeqTokenScorer to true. Note that when true, all Japanese names, not just those in Katakana, will be scored with this new model, which may result in lower accuracy for Japanese names in Hiragana or Kanji. Improvements are planned.

New parameter: We've added a new static parameter, katakanaTransliterationOnly, which defaults to false. Setting it to true will cause Japanese names written in Katakana to only be transliterated, not translated. This goes into the jpn_eng: section in the parameter_defs.yaml file.

Refer to the section Name Similarity Configuration Files in the Enterprise User Guide for more details.

Fixed a bug where translating names from Arabic to English could cause a crash.

New endpoint Address Similarity: We’ve added a new endpoint, /address-similarity, which performs a field by field comparison of two addresses and returns a similarity score between 0 and 1. Note that this endpoint is still in LABS and subject to change. Send us your feedback!

Bug fix: Incorporates Chinese tokenization fix from /morphology.

Bug fix: The Chinese tokenizer could create tokens at the end of the input string for temporal nouns without checking that the context was valid for temporal nouns.

Additional token overrides: We’ve added token overrides to the English-English override file and a new fullname override to the Japanese-English fullname override file.

Bug fix: The Rosette install directory name can now contain spaces.

Bug fix: When fstTokenizeis enabled, Russian words hyphenated with a number are now tagged with the part of speech of the word without the number.

Bug fix: We fixed a bug where transliterations from Arabic to English did not have all expected vowels.

New Feature - PermID Linking (LABS): We have augmented QID (Wikidata) linked entities by also linking them to Thomson Reuters Permanent Identifiers (PermIDs) when PermIDs are included in the Wikidata entry. To access these identifiers, add {"options": {"includePermID": true}} to your call. For more information, see the Features and Functions. Note that this feature is still in LABS and subject to change. Please try it out and send us your feedback! 

Wikidata update: We’ve refreshed and re-indexed the internal database for Wikidata linking. As a result, QIDs for some entities may have changed from previous versions.

DBpedia update: A QID can now be associated with more than one DBpedia subtype. A new option parameter, includeDBpediaTypes, has been added. A new response field, dbpediaTypes, is a list that can contain one or more string values. The original response field, dbpediaType, will continue to return a single string: the first value on the list. The original option and response field are now deprecated and will be removed in a future release. If you are using DBpedia values, please migrate to the new fields.

Bug fix: We fixed a bug in entity extraction and linking where it would crash when the modelType option is set to DNN, the linkEntities option is set to true, and the Entity Linker and DNN mentions overlap.

Malaysian update: Language identification now returns zsm for Standard Malay (Malaysian) when using both long string and short string algorithms. Previously, zsm was returned  from the short string algorithm and msa (Malay macrolanguage) from the long string algorithm. This may require updates if you have code looking for the language code msa.

Increased short string language coverage: Language identification's standard and short-string algorithms now have identical language coverage (except for transliterated Arabic-script languages). Added support for Albanian, Bulgarian, Catalan, Croatian, Estonian, Icelandic, Kurdish (Arabic script), Kurdish (Latin script), Latvian, Lithuanian, Macedonian, Polish, Serbian (Cyrillic script), Serbian (Latin script), Slovak, Slovenian, Somali, Tagalog, Ukrainian, Urdu (Arabic script), Uzbek (Cyrillic script), Uzbek (Latin script), and Vietnamese to the short string language identification algorithm.

Multilingual short string support: For sufficiently short strings, language identification will now use its short-string algorithm in multilingual mode. 

Polish update: Some Polish words ending in -cku, -ska, or -sku are now (following a different convention) lemmatized to forms ending in -cki or -ski. They had previously been lemmatized to their surface forms.

Bug fix: The Japanese POS tag NE remained NE and was not converted correctly to UPT-16 (Universal POS tagset). NE is now correctly converted to the UPT-16 PART.

Bug fix: The French POS tag CONJQUE was converted to UPT-16 CONJ. It is now converted to the more appropriate SCONJ.

Bug fix: Chinese punctuation was tagged as GUESS when alternativeTokenization was disabled. Chinese punctuation is now tagged as PUNCT or EOS.

Improved Chinese-Chinese name matching: We now give greater weight to names whose romanization is identical.

Improved name matching and translation of Japanese by normalizing small Katakana characters into their full-sized counterparts.

Improved name matching and translation in Chinese by normalizing Extension A characters.

Token frequency used for document vectors all languages: For all languages, document vectors are now calculated using token frequency information. This information gives more weight to less common words, which are more significant to the overall meaning of the document. Previously, only English vectors were weighted this way.

Entity linking is disabled (linkEntities: false) 

No supplemental regular expressions are loaded (supplementalRegularExpressionPaths null)

Pronominal resolution is not enabled (resolvePronouns: false)

caseSensitivity: caseSensitive

Japanese Update:  Japanese data regex now includes the new era 令和 (Reiwa), which started May 1, 2019.

Bug fix: We've fixed additional cases where Japanese characters were wrongly normalized into their simplified Chinese equivalents in entity linking. This issue was also addressed in the 1.13.0 release.

Bug fix: We've added missing parameter files for linking models, which may improve accuracy.

Updated lexicons: We've made various small updates to the lexicons of German, English, and Swedish. 

Improved Arabic analyzer: The Arabic analyzer will attempt to replace leading hamzated alefs with plain alefs for unrecognized tokens, to see if the version with the plain alef is recognized.

Improved Hebrew normalization: U+2019 RIGHT SINGLE QUOTATION MARK and U+201D RIGHT DOUBLE QUOTATION MARK are now normalized to U+0027 APOSTROPHE and U+0022 QUOTATION MARK in Hebrew, to support their use as geresh and gershayim.

Bug fix: We've fixed a bug which occurred when Rosette encountered a Hebrew token consisting of multiple prefixes without a base. In this case, Rosette returned an incomplete surface form, i.e., only containing the first prefix. For example, the surface form of the token “מה” should be  “מה” but was being returned as “מ”. With the bug fix the surface form returned now is “מה”.

Bug fix: We’ve fixed a problem where closing parentheses, brackets, and braces that follow URLs were merged into the URLs.

Bug fix: The disambiguator is now more likely to select analyses for @mentions, email addresses, hashtags, and URLs over other analyses.

Bug fix: When the Hebrew tokenizer encounters a character not used in Hebrew immediately following a character used in Hebrew, it now starts a new token. Formerly, it would delete that character and any following characters up to the next token separator (e.g. whitespace).

Bug fix: Hebrew tokens consisting of multiple prefixes without a base are now tagged with the part of speech “unknown”, to match single-prefix tokens. Previously, multi-prefix tokens were not tagged with any part of speech.

Bug fix: When fstTokenize is enabled, we’ve fixed a problem where Russian hyphenated words that end in numbers, like “Аполлона-11”, were not tagged as DIG. They are now tagged with the same parts of speech they had before Rosette API 1.12.2

rex-root-<version>.tar.gz

rex-root-<version>-eng.tar.gz for English language files

rex-root-<version>-deu.tar.gz for German language files

tvec-root-<version>-eng.tar.gz for English language files

tvec-root-<version>-deu.tar.gz for German language files

 Expanded English support: We can now process English with a case-insensitive model by providing (uen) as the input language.

Bug fix: We’ve fixed a problem in Japanese where entity names that include the middle dot were not being handled correctly. Entity names that include a middle dot are no longer split into two entities.

Bug fix: We’ve fixed a problem in Japanese entity linking where, in some cases, Japanese characters were being replaced with Chinese characters.

Bug fix: We’ve fixed a problem where, in some cases, entities were mislabeled when the includeDbPediaTypes option was not flagged.

New Hebrew disambiguator: We’ve added a new default disambiguator (perceptron) for Hebrew. Use the disambiguatorType option to enable another disambiguator (DNN or dictionary). For example, to return to the previous default, add {"options": {"disambiguatorType": "DNN"}} to your call.

Bug fix: We’ve fixed an error where some Chinese tokens included spurious white space characters.

Bug fix: We’ve fixed a problem where extremely long tokens (thousands of characters) would slow down the tokenizer.

Bug fix: We’ve fixed a  problem where Polish tokens that can appear in multiword expressions were lemmatized to full expressions, even when the full expression wasn’t in the input.

Bug fix: We've fixed a problem where the non-final components of Russian compound words with more than one hyphen were not lemmatized correctly.

Hungarian: We’ve added a Hungarian frequency language model for LOCATION entity type and retrained the language model for PERSON entity type, improving Hungarian name matching.

PERSON:

LOCATION:

 Expanded English support: We can now process English with a case-insensitive model by providing (uen) as the input language. Be aware that when using DNN for the modelType the accuracy of the results may be lower than when analyzing standard, sentence-cased, English input.

Bug fix: We’ve fixed a bug where, under some circumstances, the head mention for an extracted entity was not correctly identified.

Bug fix: We've fixed a bug where the Persian lemmatizer did not add lemmas to the first analyses of many tokens, especially verbs. 

Bug fix: We've fixed a bug where after some sequences of 4096 characters, containing mostly white space and at most one token, any following tokens had incorrect original offsets.

Bug fix: We've fixed a bug where capitalized tokens could return an out-of-vocabulary token-level embedding instead of the embedding consistent with their lowercase form.

Bug fix: Previously, the Semantic Vectors endpoint did not always return vectors of a consistent length. Now, returned vectors will always be normalized to have a length of one.

New endpoint Similar Terms:  We've added a new endpoint, /semantics/similar, which uses text vectors to generate multilingual related terms with numerical similarity scores for any input word(s) in Arabic, English, Chinese, German, Japanese, North or South Korean, Russian, or Spanish. For more information, see the Features and Functions.

Semantic Vectors: The /text-embedding endpoint has been renamed to /semantics/vector. While the /text-embedding endpoint will remain accessible through April, we encourage you to migrate as soon as possible to avoid missing any important updates.

Improved linking confidence: We've updated the linking confidence calculation and thresholds to improve accuracy.

Supported languages: We've removed xxx from the list of languages returned when using /entities/supported-languages.

Bug fix: We’ve fixed a bug where the salience score was not always returned for entities with pronominal mentions, when requested.

Bug fix: We've fixed a bug where some later entity mentions that were chained to the first mention of a given entity were not always properly returned.

Bug fix: We've fixed a bug where sometimes a null pointer exception was returned when resolving pronouns.

Bug fix: We’ve fixed a bug where some English words were automatically getting tagged as proper nouns when capitalized. 

Bug fix: We’ve fixed a bug in English where “people” was not being properly lemmatized to “person”. It now has the lemma candidate “person” when appropriate.

Bug fix: We’ve fixed a bug where ordinal numbers and comparative adjectives in English like “second” and “lower” were analyzed as verbs.

Hungarian: We’ve added support for multi-letter initials in Hungarian, improving Hungarian name matching.

Japanese: We’ve improved the accuracy of matching between katakana and kanji versions of Japanese organization names.

Chinese: We’ve improved the accuracy of matching Chinese organization names.

We've decreased warm-up time by only loading licensed languages when Rosette is set to pre-warm.

We’ve added the ability for on-premises users to have more control over configuration options when using custom-trained models.

We've reduced the minimum memory requirements to 16GB of RAM for the entity extraction and linking, sentiment analysis, and topic extraction endpoints.

We’ve improved the efficiency of the initial load time for the entity extraction and linking endpoint.

All client bindings have been updated to support the /semantics/vector and /semantics/similar endpoints.

The examples have been modified in the Python bindings to demonstrate how to set options.

New Addition

Spotify Annoy Java v0.2.5 - Apache 2.0

Version Changes

Basis Technology Annotated Data Model v2.5.2 - Apache 2.0