Release Notes

Solr and Lucene support: Solr 9.8.1 is now supported. (ETROG- 3714)

We fixed a bug where long words inside long spans of text without punctuation would sometimes be tokenized incorrectly. (ETROG-3716)

Solr and Lucene support: Solr 8.11.4 and Lucene 8.11.4 are now supported. (ETROG-3703)

Solr and Lucene support: Solr 9.8.0 and Lucene 10.1.0 are now supported. (ETROG- 3704)

Solr and Lucene support: Lucene 9.11.1 and Solr 9.7.0 are now supported (ETROG-3695)

Java 21 support: Java 21 is now supported. Java 11 and 17 are still supported. (ETROG-3698)

Unicode update: Unicode 16.0 is now supported. (ETROG-3694)

We fixed a bug where long compound words (70-100 characters) in Danish, Norwegian (Bokmål and Nynorsk), and Swedish could cause an OutofMemoryError. (ETROG-3696)

Neural model support improved: We upgraded TensorFlow Java to version 1.0.0-rc.1, which adds support for macOS ARM64. Neural models are now supported for macOS ARM64. (ETROG-3554)

Trailing decimal points in Chinese are no longer treated as part of a decimal fraction. When “点” or “點” ends a number, it is no longer segmented as part of the number. (ETROG-3680)

Solr and Lucene support: Lucene 9.10 and Solr 9.6 are now supported (ETROG-3683)

CLA lexicon: Two terms have been added to the CLA lexicon: 喷码机 inkjet printer and 管理器 manager (in the context of software) (ETROG-3678)

Improved Readings: Readings are now returned for numeric words in Chinese and Japanese when tokenizerType is set to SPACELESS_LEXICAL. (ETROG-3684)

When tokenizerType is set to SPACELESS_LEXICAL, Japanese tokens for verbs in lemma form have had their readings fixed to cover the entire token. (ETROG-3640)

Example: Input: 食べる

When tokenizerType is set to SPACELESS_LEXICAL, Japanese lemmatization has been corrected for numeric tokens containing both decimal points and multiplier characters.

Example: Input: 2.5亿

We fixed a bug where the Chinese word “星期四” would be tokenized incorrectly in certain contexts. (ETROG-3582)

We fixed a bug where a token whose surface form was the empty string could be returned when fragmentBoundaryDetection was set to true (the default). (ETROG-3686)

Unicode update: Unicode 15.1 is now supported. (ETROG-3595)

Solr and Lucene support: Lucene 9.9 and Solr 9.5 are now supported. (ETROG-3673)

Japanese improvements: We have improved and augmented the Japanese lexicon that is used when tokenizerType is set to spaceless_lexical. (ETROG-3532, 3581, 3535, 3668)

Solr and Lucene support: Lucene 9.5 – 9.8 and Solr 9.4 are now supported. (ETROG-3665)

Expanded Chinese lexicon: We've expanded the lexicon of multi-character Chinese surnames when tokenizerType is set to spaceless_lexical. (ETROG-3616)

Expanded Japanese lexicon: We have expanded the Japanese lexicon that is used when tokenizerType is set to spaceless_lexical. (ETROG-3632)

Added secondary parts of speech: We've added support for secondary parts of speech to Chinese and Japanese when tokenizerType is set to spaceless_lexical. (ETROG-3636)

Improved support for Chinese readings when tokenizerType is set to spaceless_lexical:

Solr and Lucene support: Lucene 9.5 - 9.7 and Solr 9.3 are now supported (ETROG-3643)

We fixed a bug where an ArrayIndexOutOfBoundsException occurred when the Chinese dictionaries produced more than 6 matches and tokenizerType was set to spaceless_lexical. (ETROG-3635)

When Chinese readings are constructed by character and tokenizerType is set to spaceless_lexical, an apostrophe is now inserted before pinyin syllables that start with "a", "e", or "o" which are not the first syllable. (ETROG-3637)

We fixed a bug where the UPT-16 conversion where some Japanese particles part of speech were not tagged correctly. The particles are now tagged correctly as ADP. (ETROG-3526)

The plugin for Solr 8.11 may throw an exception when using multiple neural models.

Lucene and Solr support: RBL-JE now supports Lucene 9.2 to 9.4 and Solr 9.2. (ETROG-3631)

We fixed a bug where processing an empty string as Chinese or Japanese with tokenizerType set to spaceless_lexical would throw a NullPointerException. (ETROG-3629)

In Solr 8.11.2, it is not possible to consistently load more than one neural model at a time. (ETROG-3608)

We've improved the time it takes to tokenize extremely long (> 10K characters) Japanese sentences. (ETROG-3602)

In Solr 8.11.2, it is not possible to consistently load more than one neural model at a time. (ETROG-3608)

Annotating Ukrainian with universalPosTags set to true no longer results in a RosetteUnsupportedLanguageException being thrown. (ETROG-3604)

Ukrainian support added: Tokenization, sentence boundary detection, segmentation user dictionaries, and many-to-one normalization dictionaries are supported for Ukrainian. (ETROG-3594)

Improved part of speech tags: Language-neutral tokens (numbers, symbols, and punctuation) now get part of speech tags in Indonesian, Standard Malay, and Tagalog. (ETROG-3574)

GPU support: Features that use TensorFlow now use a GPU if available. (ETROG-3564)

Emoji support: Emoji 15.0 is now supported. (ETROG-3577)

New option for Katakana: We've added the option joinKatakanaNextToMiddleDot to control whether sequences of Japanese Katakana tokens adjacent to a middle dot should be merged into a single Katakana token. By default, it is true, which matches the behavior in previous versions of RBL-JE. (ETROG-3592)

Solr 9.1 support: Lucene and Solr 9.1 are supported. (ETROG-3597)

The Japanese POS tag VN (verbal noun) is now mapped to the UPT-16 POS tag NOUN. It was previously mapped to VERB. (ETROG-3583)

Tagalog support:

Indonesian (ind) support: RBL now supports lemmatization for Indonesian, which is the standardized form of Malay spoken in Indonesia. (ETROG-3563)

Standard Malay (zsm) support: RBL now supports lemmatization for Standard Malay, the standardized form of Malay spoken in Malaysia. (ETROG-3563)

We fixed a bug in Russian where certain uncommon consonant–vowel sequences in words in the lexicon were incorrectly replaced with more common sequences with different vowel letters. (ETROG-3541)

Example: брошюра

An open source package (JavaCPP) has been updated to allow the Elasticsearch plugin to use TensorFlow. (ESPI-168)

RBL-JE no longer crashes when loading TensorFlow fails with a NoClassDefFoundError. It now throws an exception. (ESPI-169)

Indonesian support added: RBL now supports Part of Speech (POS) tagging in Indonesian. (ETROG-3543)

Malay (Standard) support added: RBL now supports Part of Speech (POS) tagging in Malay (Standard). (ETROG-3545)

Russian lexicon improved: We've added many words related to computer technology to the Russian lexicon. (ETROG-3523, ETROG-3538)

Java 17 support added: Java 8 and 9 support has been removed. (ETROG-3524)

Solr 9 support added: RBL now supports Lucene and Solr 9. (ETROG-3549)

Solr 6 support deprecated: RBL no longer supports Lucene or Solr 6 or earlier. (ETROG-3519)

In Japanese, negative forms of ichidan verbs written all in hiragana are no longer lemmatized to end with “なう”. (ETROG-3534)

Example: Input: くれない

Solr 8.11 support: This release supports Solr 8.11 (ETROG-3502)

Deprecated methods: Token#getType has been deprecated as token types are not used in RBL-JE without the Lucene/Solr plugins and the plugins use a different API. (ETROG-3503)

Solr 6 support deprecated: Support for Solr versions 6.x and earlier is deprecated as of this release and will be removed in the next version.

Permission changes: We removed group and other write permissions from model files. All files are now only writable by the owner. (ETROG-3516)

RBL no longer crashes in Arabic when emoticons is enabled. This fixes a bug introduced in 7.42.1. (ETROG-3493)

In Korean, emoji and other language-neutral tokens no longer cause a ClassCastException to be thrown when using the ADM API. (ETROG-3488)

Deprecated factories: TokenizerFactory, AnalyzerFactory, and CSCAnalyzerFactory have been deprecated in favor of BaseLinguisticsFactory. (ETROG-3453)

Katakana tokenization: The fullwidth and halfwidth Katakana middle dots (U+30FB and U+FF65) are now treated as decimal points in numeric contexts, for Japanese with tokenizerType set to spaceless_lexical. (ETROG-3474)

Example: Input: 三・一四

Emojis: U+3030 and U+303D are now tagged as emojis even when not followed by U+FE0F. (ETROG-3478)

Emoji support: We now support the emoji in Unicode 14.0 (ETROG-3476)

Japanese tokenization: In Japanese, when tokenizerType is set to spaceless_lexical, numeric tokens tagged NN are lemmatized to their ASCII values. For example, “七” is lemmatized to “7”. This is consistent with the default algorithm, spaceless_statistical. (ETROG-3475)

Solr 8.10 support: This release supports Solr 8.10. (ETROG-3482)

Improved POS tags: Many number, punctuation, and symbol characters are now POS-tagged appropriately as numbers, punctuations, and symbols instead of being marked as unknown or some other tag. This applies to all languages with POS tags. (ETROG-3481)

Hungarian improvements: We've added some Hungarian abbreviations and improved sentence boundary detection around Hungarian abbreviations. (ETROG-3479, ETROG-3484)

In Japanese, when tokenizerType is set to spaceless_lexical, the combining marks U+3099 and U+309A are now tokenized with the preceding character as a single token. Previously, they were tokenized as 2 separate tokens. (ETROG-3472)

We've reverted two of the POS changes made in version 7.39.0.63.0 as they introduced regressions in Chinese and Japanese. (ETROG-3466)

The values are now:

RBL-JE no longer detects characters as emoji when followed by the text presentation selector (U+FE0E). (ETROG-3480)

In English, the lowercase abbreviations of the titles “dr.”, “drs.”, “mr.”, and “mrs.” are now tokenized the same as the uppercase “Dr.”, “Drs.”, “Mr.”, and “Mrs.”. (ETROG-3485)

Enabling universalPosTags for Indonesian, Tagalog, or Standard Malay no longer throws a RosetteUnsupportedLanguageException. POS tags are not supported for these languages, so the universalPosTags option is ignored. (ETROG-3465)

New language support: Tokenization is now supported for Indonesian, Standard Malay, and Tagalog. (ETROG-3443)

Fragment definition: A single line followed by an empty line is no longer always considered a fragment. They are still considered fragments if the line is short, as specified by the maxTokensForShortLine parameter. (ETROG-3431)

Solr 8.9 support: This release supports Solr 8.9. (ETROG-3457)

We fixed a bug where enabling alternativeSpanishDisambiguation for Spanish caused a NullPointerException to be thrown. (ETROG-3435)

We fixed a bug where setting disambiguatorType to DNN for Hebrew caused a RosetteRuntimeException to be thrown. (ETROG-3437)

New option for tokenizers: We've added a new option, tokenizerType to specify which tokenizer to use. The options alternativeTokenization and fstTokenize are deprecated in favor of tokenizerType. (ETROG-3419)

New Korean tokenizer: We've added a new tokenizer for spaceless Korean input. The previous tokenizer was not trained on spaceless Korean and did not perform well without spaces between tokens. Activate it by setting tokenizerType to spaceless_statistical. (ETROG-3392)

Multiple punctuation characters are no longer returned as a single token in Chinese when alternativeTokenization is true or tokenizerType is set to spaceless_lexical. Now each character is its own token. (ETROG-3402)

Example: Input: 天津？？

Improved Hebrew tokenizer and new analyzer: The Hebrew tokenizer is now more consistent with the tokenizers of other languages. Hebrew tokenization and analysis are now done in separate steps.(ETROG-3290)

Normalized characters: Normalized half-width and full-width characters are processed the same as their counterparts. (ETROG-3351)

Solr 8.8: This release supports Solr 8.8. (ETROG-3369)

Improved CSCAnnotator output: The CSCAnnotator now emits tokens in addition to translations, even if no tokens were specified in the input. (ETROG-3356)

Improved directory structure: The contents of the models/ directory are now separated into subdirectories by language. (ETROG-1218)

Statistical models moved to models/ directory: The following files have been moved from dicts/ to models/: (ETROG-1218)

Hebrew tokens containing a geresh are now tokenized properly. Previously, only the part up to the geresh would be returned as the token text, and the part after the geresh would sometimes be considered a suffix. Now the whole token is returned as the token's text. (ETROG-3262, ETROG-3290)

Example: מע'רב

A structured region containing two new lines is now properly labeled as STRUCTURED. Previously, the layout region would be labeled as UNSTRUCTURED. (ETROG-3378)

Example: * item\n* item\n* item\n\n

RBL-JE no longer normalizes certain emoji ZWJ sequences to U+1F48F KISS, U+1F491 COUPLE WITH HEART, and U+1F46A FAMILY, to be consistent with Unicode’s efforts to make emoji more gender-neutral by default. (ETROG-3350)

We reverted some changes to Korean disambiguation from the 7.37.0.c62.2 release as the changes introduced new disambiguation errors. (ETROG-3349)

Greek lexicon: The Greek lexicon has additional words. (ETROG-3288)

Greek disambiguation improved: Certain Greek forms are now disambiguated to prefer a modern analysis over an archaic analysis. alternativeGreekDisambiguation must be set to false, which is the default. (ETROG-3289)

Example: δείξε

New Greek disambiguator added: The new Greek disambiguator is more accurate, but slower. The new disambiguator is enabled by default. To use the old disambiguator, set alternativeGreekDisambiguation to true. (ETROG-3304)

Deprecated classes: The classes BufferWordBreaker and WordBreakResults have been deprecated. (ETROG-3318)

The Greek guesser now handles tokens with non-alphanumeric characters. (ETROG-3286)

Example: Start+

GenericTokenizer#hasNext is now implemented to be consistent with the documentation for Iterator#hasNext. Previously it always returned false. (ETROG-2140)

Performance improvement: Spanish disambiguation with alternativeSpanishDisambiguation set to false is now faster. (ETROG-3271)

Performance improvement: Korean disambiguation is now faster. (ETROG-3280, ETROG-3282)

Support for unknown language: If the language is unknown (xxx), tokenization and sentence breaking is supported. (ETROG-3278)

Solr 8.7.0: We now support Solr 8.7.0. (ETROG-3315)

Tokenization rule preprocessor: The preprocessor command !!btinclude is supported in tokenization rule files, supporting inclusion of files in rule files. (ETROG-2497)

Updated sample: The tokenize-analyze sample has been changed from two applications running in sequence to a single application that both tokenizes and analyzes. (ETROG-3291)

New sample: The sample csc-annotate demonstrates using CSC with the ADM API. (ETROG-3317)

Deprecated option: TokenizerOption#includeRoots has been deprecated and replaced with TokenizerOption#includeHebrewRoots. (ETROG-3314)

Deprecated option: The alternative tokenization option deliverExtendedAttributes is now deprecated. Previously it delivered an unsupported extended property. (ETROG-3311)

Combining characters in Hebrew which were being erroneously split into tokens separate from their bases are now not being split. (ETROG-3277)

A clear exception (RosetteUnsupportedLanguageException) is now thrown when tokenizing some unsupported languages. Previously, these languages appeared to work. The same tokenizer is still available by specifying the unknown language (xxx). The languages impacted are Albanian, Bulgarian, Croatian, Indonesian, Malay, Slovenian, Standard Malay, and Ukrainian. (ETROG-3278, ETROG-3326)

RBL no longer crashes when alternativeTokenization and fragmentBoundaryDetection are both enabled for some inputs in Japanese and Chinese. (ETROG-3285)

Correct start and end offsets are now produced when fstTokenize is set to true. Previously, some Spanish inputs would produce tokens with start and end offsets of 0. (ETROG-3292)

The mappings of default Basis POS tags to universal POS tags (UPT-16) have been corrected for Greek. (ETROG-3306)

Tokens no longer have null token types. (ETROG-3316)

When an NFKC normalized character results in multiple tokens, those tokens no longer have equal start and end offsets. Previously this could occur when nfkcNormalize was set to true. (ETROG-2505)

Example: ﷺ

Lucene/Solr: Versions up through 8.6.0 are now supported. (ETROG-3250)

Decompose compounds: The option to control decomposition of compounds is now available in Dutch, German, Hungarian, Danish, Bokmål, Nynorsk, Swedish, and Korean. The default for decomposeCompounds is true. (ETROG-3263, ETROG-3264, ETROG-3265)

Performance improvement: English and Spanish disambiguation with is now faster. Alternate disambiguation (alternateEnglishDisambiguation or alternateSpanishDisambiguation) must be set to false. (ETROG-3246, ETROG-3243)

In Hebrew, prefixes in some acronym tokens are now listed correctly in the list of prefixes, instead of being duplicated in the lemma. (ETROG-3214)

Example: “ומש"ס”

Sentence breaks are now correct when there are two line breaks and fragmentBoundaryDetection is enabled. (ETROG-3241)

Example: "a very very very very long line\nshort\n\n"

In Hebrew, lemmas starting or ending with spaces now have the spaces removed. (ETROG-3248)

Example: "אאורקה"

Analysis of unknown Hebrew words with guessed prefixes no longer have duplicate prefixes in their prefix list. (ETROG-3253)

Example: "בפיירפוקס"

In Chinese and Japanese, the system no longer crashes when both fragmentBoundaryDetection and alternativeTokenization are enabled. (ETROG-3260)

In Japanese, adjacent tokens are no longer erroneously joined when alternativeTokenization is enabled. (ETROG-3261)

When universalPosTags are enabled the UPT-16 POS tags are now marked as having the tag set UPT16_V1 instead of the default tag set of the language. (ETROG-3273)

Example: French

We've fixed the tokenize-analyze example in the samples directory. It now correctly produces results for Hebrew analysis. (ETROG-3252)

Layout regions added: Layout regions, describing each section of input text as STRUCTURED or UNSTRUCTURED, are now identified by the annotator. In order to detect layout regions, fragment boundary detection must be enabled. (ETROG-3172)

New short line parameter: The option maxTokensForShortLine has been added to configure how many tokens can be in a line for it to be considered short for fragment boundary detection. The default value is 6. (ETROG-3179)

Greek time abbreviations: The time abbreviations "π.μ." and "μ.μ." are now identified and annotated in Greek. The option fstTokenize must be set to true. (ETROG-3226)

Greek coverage expanded: POS tags and lemmas are now recognized for some Greek words previously not identified. (ETROG-3225)

Hebrew user-defined dictionaries added: Static and dynamic user-defined Hebrew analysis dictionaries are now supported. (ETROG-3230)

Deprecated method: HebrewAnalysis#characteristicString is now deprecated. (ETROG-3209)

Order of user-defined dictionaries: The order in which user-defined dictionaries are consulted has been standardized. Refer to the RBL-JE Application Developer's Guide for details. (ETROG-3148)

Whitespace-delimited fragment boundaries are no longer skipped when they fall within tokens. This only occurred when fstTokenize was enabled and in some languages. (ETROG-3159)

Example: "1\n234" (embedded newline within the number string)

This example assumes fstTokenize is enabled and the language is French.

Fragment detection now counts tokens correctly to determine short lines. This mostly impacts languages without spaces: Chinese, Japanese, and Thai. (ETROG-3177)

Tokens with digits are now eligible for the Greek guesser. (ETROG-3231)

In Hebrew, tokens with an unknown part of speech are no longer assigned the part of speech of one of their prefixes. This only occured when the guessHebrewPrefixes option is set to true.(ETROG-3221)

Example: "ומפיפרנו"

Russian perfective verbs are now lemmatized correctly. Previously some were lemmatized to their imperfective counterparts' lemmas or other incorrect lemmas. (ETROG-3112)

Example: "разложу" where "разложу" is perfective and its lemma is "разложить". Its imperfective counterpart’s lemma is "раскладывать"

German lemmas that consist of a separable prefix and a noun are now correctly capitalized. (ETROG-3235)

Example: Input "Mitbehandlung"; "mit" is a separable prefix

In Hebrew, terminal combining characters are no longer getting split into their own tokens. (ETROG-3224)

Example: "1" (keycap)

Hebrew tokens that have prefixes but not stems now get appropriate parts of speech. Previously, they got the POS tag "unknown". (ETROG-3207)

Example: “ה” from the string “ה70”

Lucene/Solr up through version 8.5.1 is now supported. (ETROG-3208)

When guessHebrewPrefixes is true, unrecognized Hebrew tokens will now get analyses with and without potential prefixes. Previously, they would only get analyses with potential prefixes. (ETROG-3188)

Example: Token: "ומפיפרנו"

Minimally-qualified emoji are no longer split apart. (ETROG-3185)

Example: The emoji for "man tipping hand" (<U+1F481, U+200D, U+2642>: )

Capitalized nouns are no longer being detected as verbs. (ETROG-3186)

Example: The noun "Service" from the phrase "Price and Quality of Service"

When creating multiple analyzers for Chinese, Japanese, or Thai with alternateTokenization set to false (the default), the analyzers will now share the same model data. This will improve memory usage when creating multiple analyzers. (ETROG-3200)

Note: While memory usage has been improved, the process is still memory intensive. If RBL throws an OutOfMemoryError, increase the heap space.

We removed some incorrect entries from the Hebrew lexicon that were added in 7.34.0.c62.2. (ETROG-3182, ETROG-3183)

Lucene/Solr: RBL-JE now supports Lucene/Solr up through version 8.4.1. (ETROG-3156)

Unicode 13.0 emojis: Unicode 13.0 emoji sequences are now tokenized. (ETROG-3164)

Additional emoji support: Emoji hair components are now lemmatized. (ETROG-3167)

German professions: Additional German professions have been added to the German lexicon. (ETROG-3163)

Spanish performance improvements: Spanish disambiguation is now faster when alternativeSpanishDisambiguation is false. (ETROG-3169)

Hebrew lemmatization: We increased proper noun coverage in the Hebrew lexicon. (ETROG-3161, ETROG-3162)

Low surrogates are no longer stripped from the ends of tokens in Hebrew. (ETROG-3165)

Number tokens with embedded spaces are no longer split into multiple tokens when preceded or followed by a symbol when fstTokenize is true. (ETROG-3158)

The delimiters for the fragment boundary detector are now configurable. (ETROG-3116)

The fragment boundary detector now marks a boundary after any spaces following the fragment boundary delimiter. (ETROG-3116)

An underscore (U+005F) is no longer treated as a token separator in German when fstTokenize is enabled. (ETROG-3144)

We fixed a bug where tokens from multi-script Russian text sometimes had incorrect offsets if fstTokenize was enabled. (ETROG-3142)

We fixed a bug where multi-script Russian text would have a sentence break each time the script changed. (ETROG-3145)

We fixed a bug where there were unexpected sentence breaks after some short lines not ending in whitespace. (ETROG-3146)

We fixed a bug where sentence breaks were missing when the sentence break did not align with a token boundary. (ETROG-3140)

Added support for Lucene/Solr up through version 8.3.0. (ETROG-3128)

Added support for tokenizing and lemmatizing Latvian. (ETROG-2798)

Latin-script regions within Russian documents are now tokenized and analyzed as English. (ETROG-3126)

TokenizerOption.licenseString, AnalyzerOption.licenseString, and BaseLinguisticsOption.licenseString may now be passed into a create method. Previously, these options had to be set on the factory itself. (ETROG-3134)

We fixed a bug where guessed German compounds were sometimes lemmatized as verbs but tagged as nouns. (ETROG-3094)

We fixed a bug where the fragment boundary detector would mark a sentence break after every Windows newline. (ETROG-3133)

The Hebrew files dinflections.bin, dprefixes.data, and gimatria.data have been moved from the root/models directory to root/dicts/heb. (ETROG-3088)

Specifying the universalPosTags option now adds the deliverExtendedTags option as well. (ETROG-2185)

Dynamic user dictionaries can now be created and populated at runtime. See the section User-Defined Dictionaries in the Application Developer's Guide for details. (ETROG-3086, ETROG-3100, ETROG-3109, ETROG-3110, ETROG-3111)

Fragment boundary detection is now enabled by default. Previously it was disabled by default. (ETROG-3108)

TokenizerOption.alternativeTokenizationOptions has been deprecated in favor of a separate options for each YAML key. See the Javadoc for details. (ETROG-3109)

The UPT-16 files upt-16-pes.yaml and upt-16-prs.yaml have been removed from the distribution package, as they were unused. (ETROG-3122)

The -order option in rbl-build-csc-dictionary has been removed. All dictionaries are now built as LE, as LE dictionaries still work on BE machines. (ETROG-3120)

We've added imperative forms for 2000 verbs to the Arabic lexicon. (ETROG-3090)

Fragment boundary detection is now enabled for Hebrew. (ETROG-1442)

When lemmatizing numbers in Russian, numbers containing spaces will now be lemmatized without the space. For example, "1 234" will now be lemmatized as "1234" instead of "1 234". (ETROG-3101)

We fixed a bug introduced in 7.30.1.c61.0 which raised an ArrayIndexOutOfBoundsException when processing Japanese with alternativeTokenization and favorUserDictionary set to true. (ETROG-3118)

We fixed a bug where a middle dot would be ignored if it preceded white space when using alternativeTokenization in Japanese. (ETROG-3113)

We fixed a bug where an AssertionError might be thrown when analyzing Hungarian with Java assertions enabled.

Russian words hyphenated with a number are now tagged with the part of speech of the word without the number.

Correct token offsets are now returned from a Japanese annotator where a non-katakana character precedes a user-defined katakana token and alternativeTokenization and favorUserDictionary are enabled.

We fixed a bug where constructors of factory classes in the Lucene/Solr plugin would throw an UnsupportedOperationException if passed a Map that did not support the remove method.

Added support for Lucene/Solr up through version 8.2.0.

Dictionaries and models that are used on both big- and little-endian machines no longer include LE in their file names.

When alternativeTokenization was set to true, the Chinese tokenizer could create tokens at the end of the input string with the part of speech NT without checking that the context was valid for NT

Analyzing Chinese and Japanese with alternativeTokenization enabled is now much faster on sentences that are thousands of characters long.

Segmentation user dictionaries can be used for all languages, not just Chinese, Japanese, and Thai.

The option compoundComponentSurfaceForms has been added to return the surface forms of the components of compound words. By default, RBL-JE only returns the lemmas.

Added support for Lucene/Solr up through version 8.1.1.

Some Polish words ending in “-cku”, “-ska”, or “-sku” are lemmatized to forms ending in “-cki” or “-ski”.

The Japanese POS tag NE was not converted correctly to UPT-16.

The French POS tag CONJQUE was converted to UPT-16 CONJ instead of the more appropriate SCONJ.

When alternativeTokenization was disabled, Chinese punctuation was tagged as GUESS instead of PUNCT or EOS.

Setting alternativeTokenization to true enables an alternative tokenizer for Thai, for parity with the Thai tokenizer in Basis Technology's C++ API (RLP).

All Hebrew tokens have analyses. The main change was adding the new part of speech punctuation. Non-punctuation tokens that formerly had empty analysis lists now have the part of speech unknown.

U+2019 RIGHT SINGLE QUOTATION MARK and U+201D RIGHT DOUBLE QUOTATION MARK are now normalized to U+0027 APOSTROPHE and U+0022 QUOTATION MARK in Hebrew words, to match the normalization of U+05F3 HEBREW PUNCTUATION GERESH and U+05F4 HEBREW PUNCTUATION GERSHAYIM.

Updated the English lexicon.

Added support for Lucene/Solr up through version 8.0.0.

Updated the German lexicon.

Updated the Swedish lexicon.

Arabic analysis will attempt to replace leading hamzated alefs with plain alefs for unrecognized tokens.

The surface forms of Hebrew tokens consisting of multiple prefixes without a base, like “מה”, are now the entire token text, instead of just the first prefix.

Russian hyphenated words that end in numbers, like “Аполлона-11”, are no longer tagged as DIG. They are now tagged with the same parts of speech they had before 7.27.2.c60.0.

Closing parentheses, brackets, and braces that follow URLs when urls is enabled are no longer merged into the URLs.

The disambiguator is now more likely to select analyses with the POS tags ATMENTION, EMAIL, HASHTAG, and URL over other analyses.

When the Hebrew tokenizer encounters a character not used in Hebrew immediately following a character used in Hebrew, it starts a new token. Formerly, it would delete that character and any following characters up to the next token separator (e.g. white space).

RBL-JE can now successfully read in ICU tokenization rule files that begin with a BOM.

Hebrew tokens consisting of multiple prefixes without a base are now tagged with the part of speech “unknown”, to match single-prefix tokens.

The English token “than” is tagged only as COTHAN. The candidate part of speech COORD has been removed for this token.

A perceptron-based disambiguator is available for Hebrew. It is used by default and when the option disambiguatorType is set toDisambiguatorType.PERCEPTRON. It was measured to have higher lemma and part of speech accuracies than the alternatives. To use the previous default, set disambiguatorType to DisambiguatorType.DICTIONARY.

Added support for Lucene/Solr up through version 7.6.0.

Running on Java 11 is now supported.

Some white space characters could be part of Chinese tokens when alternativeTokenization was enabled.

Tokens that are thousands of characters long slow down the tokenizer.

Polish tokens that can appear in multiword expressions are no longer lemmatized to the full expressions. For example, “dzień” is not lemmatized to “dzień_dobry”.

The non-final components of Russian compound words with more than one hyphen were not lemmatized. The non-final components of Russian hyphenated compound words with the interfix “е” or “о” that coincidentally looked like the short forms of adjectives were lemmatized as if they were short forms.