Release Notes

Running on Java 11 is now supported. (RLIJE-511)

Added the option koreanDialects to specify that RLI-JE should return North Korean (qkp) or South Korean (qkr) instead of Korean (kor). By default, this option is turned off. (RLIJE-480)

Added support for Indonesian and Malaysian to the short string algorithm. (RLIJE-448)

Rescaled confidence scores for the default (not short string) algorithm such that high-confidence results get scores around 0.9 instead of around 0.03. (RLIJE-447)

Added the option minNonScriptioContinuaRegionLength to control when short regions of scripts like Latin are merged into regions of scripts like Han. The default is to never merge the regions. To return to the behavior of previous versions, set this option to 10. (RLIJE-454)

Exposed more features through OSGi (internal use only). (RLIJE-423)

Exposed more features through OSGi (internal use only). (RLIJE-423)

Added the -output-json option to RLICmd to write the output ADM as JSON.

RLI-JE now returns results for raw input in encodings not supported by the JVM. In previous versions, it threw an exception. (RLIJE-288)

Sped up the language region detection algorithm. It now completes in O(n) time. (RLIJE-397)

To become file-system-agnostic, the use of Path in the API is now supported. (RLIJE-331)

Version of OSGi (internal use only) upgraded. (RLIJE-380)

Added -dontBreakRegionOnScriptBoundary to RLICmd. (RLIJE-324)

Added debug and trace log messages to the multilingual detection code. (RLIJE-388)

Language weight adjustments can now be used to boost any given language, not just demote it. (RLIJE-111)

The weight adjustment API now works for the short string algorithm. (RLIJE-204)

Refactored RLICmd's command line options. (RLIJE-230)

The short string and legacy algorithms now consistently handle cases when a language cannot be determined. Both now return language Unknown. NoMatchException, NotEnoughDataException, and LanguageIdentificationException are deprecated. (RLIJE-272, RLIJE-304)

Achieved a modest accuracy gain by changing the internals of the matching algorithm to be more tolerant of noise. (RLIJE-277)

Added an option to return only one result per language. See LanguageIdentifierBuilder#uniqueLanguages(boolean). (RLIJE-298)

The languageHint and encodingHint methods of LanguageIdentifierBuilder are deprecated. Use the weight adjustment API instead. (RLIJE-320)

Profiles for transliterated languages (e.g. Arabic in Latin script) are disabled by default. To enable them, see LanguageIdentifierBuilder#languageWeightAdjustment(LanguageCode, ISO15924, int). (RLIJE-225)

Added an option to read models used by the short string algorithm from a JAR file. See LanguageIdentifierBuilder#useModelsInJar(boolean). (RLIJE-276)

RLICmd can continue running if it fails to find a file in a provided list of files to analyze. (RLIJE-216)

RLICmd can use multiple threads to analyze documents in parallel. (RLIJE-221)

icu4j and args4j have been shaded into the rli-je-shaded jar. (RLIJE-237)

Basis' common-lib jar has been shaded into the rli-je-shaded jar. (RLIJE-248)

RLI-JE now depends on adm-model instead of adm-shaded. With this change, RLI-JE no longer depends on Apache Commons Betwixt or Javassist. (RLIJE-252)

The short string detection algorithm is 20% faster than 7.13.0. (RLIJE-254)

Added alternative analysis for improved accuracy when detecting the language of short strings. (RLIJE-96)

Renamed the license directory from license to licenses to be consistent with other Basis products. (RLIJE-124)

Moved the command line utility, RLICmd, from tools/bin to bin. (RLIJE-200)

Refactored the identification of Chinese to separate language and script. Chinese language (zho) is now detected when script is Han, Simplified (Hans) and when script is Han, Traditional (Hant). RLI-JE used to identify these variants as Simplified Chinese (zhs) and Traditional Chinese (zht). (RLIJE-152)

Added support for identifying language regions in a document that contains blocks of text in multiple languages. (RLIJE-88)

Deprecated LanguageIdentifierFactory and LanguageIdentifierin favor of LanguageIdentifierBuilder, which you can use to set options and create Annotator objects that detect language, encoding, and writing script, as well as language regions in multilingual input. This implementation employs the new data model package (com.basistech.rosette.dm), which is used in a variety of Rosette products. (RLIJE-93)

Added a factory class (LanguageIdentifierFactory) for creating instances of the LanguageIdentifier. (RLIJE-15)

Moved the command line utility from LanguageIdentifier to RLICmd. (RLIJE-32)

When returning UTF-16 encoding, RLI-JE now identifies whether the encoding is Little Endian (UTF-16LE) or Big Endian (UTF-16BE). (RLIJE-51)

Extended support for specifying the license. LanguageIdentifierFactoryincludes constructors for getting the license from a file, and input stream or an .xml string.

With the LanguageIdentifier setLanguageWeight Adjustment methods, added support for reducing the weight associated with a specific language to a percentage of its original weight in order to assist in the detection of other languages in documents that mix multiple languages.

Adjusted the default weights assigned to Pushto/Latn and Urdu/Latn to return more accurate results. Lowered the default weight for Serbian/Latn to 0, so that Croation is returned. Pushto/Arab, Urdu/Arab, and Serbian/Cyrillic are not affected by these adjustments. (RLIJE-37)

Placed new .jar files in the lib directory. mahout-collections-1.0.jar is no longer used by RLI-JE.

Replaced trove-2.0.4.jar with mahout-collections-1.0.jar.

Removed utilities-no-jni-7.1.jar and replaced rlp-common-1.3.6.jar with btcommon-3.jar.