Analyze Language

You can use RBL in your own JVM application, use its Apache Lucene compatible API in a Lucene application, or integrate it directly with either Apache Solr or Elasticsearch. ^[1]

RBL performs multiple types of analysis. Depending on the language, one or more of the following may be identified in the input text:

For some languages, the analyzer can disambiguate between multiple analysis objects and return the disambiguated analysis object.

In the ADM API, use the BaseLinguisticsFactory to set the linguistic options and instantiate an Annotator which annotates the input text. The ADM API creates an annotator for all linguistic objects, including tokens.Using the Annotated Data Model API

In the classic API, use the BaseLinguisticsFactory to configure and create tokenizers, analyzers, and CSC analyzers. The classic API creates separate tokenizers and analyzers.Using the Classic API

The following table indicates the type of support that RBL provides for each supported language. The RBL tokenizer provides normalization, tokenization, and sentence boundary detection. The RBL analyzer provides lemma lookup (including orthographic normalization for Japanese), lemma guessing (when the lookup fails), decompounding, and supports lemma, segmentation, and many-to-one normalization user dictionaries.

For unknown languages (language code xxx), generic rules, such as whitespace and punctuation delimitation, are used to tokenize. It will also identify some common acronyms and abbreviations, as well as sentence boundaries. Segmentation user dictionaries are supported for unknown languages.

Some of the features in RBL are supported using deep learning, or neural models which require the native library, TensorFlow. The neural network is used for the following features:

BasisTech has created a collection of Java classes that generate and represent Rosette's linguistic analyses as a set of annotations. This collection of Java classes is called the Annotated Data Model (ADM) and may be used in RLI and Entity Extractor as well as in RBL.

The ADM provides advanced functionality not available with the classic API.

When using the ADM API, you use BaseLinguisticsFactory to set the options and instantiate an Annotator. Depending on the analysis and the language, you may get information about sentences, layout regions, tokens, compounds, and readings.

The ADM API supports more options than the classic API. Whenever possible, we recommend using this API.

For complete API documentation of the ADM, consult the Javadoc for the package:

AnnotatedText results = annotator.annotate(getInput(inputFilePathname));
int index = 0;
for (Token token : results.getTokens()) {
    outputData.format("token %d:\t%s%n", index, token.getText());
    int aindex = 0;
    List<MorphoAnalysis> analyses = token.getAnalyses();
    if (null != analyses) {
        outputData.format("\tindex\tlemma\tpart-of-speech%n");
        for (MorphoAnalysis ma : analyses) {
            outputData.format("\t%d    %s\t%s%n", aindex, ma.getLemma(),
                     ma.getPartOfSpeech());
             aindex++;
        }
    }
    index++;
}

When using the classic API, you instantiate separate factories for tokenizers and analyzers.

If your application requires streaming, use this API. The Lucene, Solr, and Elasticsearch integrations use these methods.

For the complete API documentation, consult the Javadoc for BaseLinguisticsFactory.

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);

factory.setOption(BaseLinguisticsOption.nfkcNormalize, "true");

Tokenizer tokenizer = factory.createTokenizer();

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(BaseLinguisticsOption.licensePath, 
        new File(rootPath, "licenses/rlp-license.xml").getAbsolutePath());
Analyzer analyzer = factory.createAnalyzer();

RBL is a Java SDK and works on any system with Java, including OpenJDK, installed.

The minimum Java heap required to run RBL smoothly is 1.5 GB. It has been observed that a -Xmx setting of 8 GB allows for optimal performance in significantly multithreaded environments. This memory setting accounts for the simultaneous use of multiple languages.

We also recommend that you set -Xms equal to your -Xmx setting. This prevents the JVM from having to grow the heap, which is time-consuming.

RBL uses TensorFlow, a native library, when using a neural network. Ubuntu 14.04+, Windows 7+, and macOS 10.11+ are supported, but you should be able to run TensorFlow successfully on other modern Linux flavors as well.

The TensorFlow library for Linux requires a system with:

The version of TensorFlow included with RBL is configured to work on a broad range of systems, which requires it to avoid some features not available everywhere. Compiling TensorFlow for your specific system and using it on the classpath instead of the default libtensorflow_jni-<tensorflowversion>.jar will likely improve performance. The version of TensorFlow included supports CUDA on Windows and Linux. Our neural models may perform better after installing CUDA on machines with supported GPUs.

Your installation of RBL will include the following files:

RBL contains version-specific files for the Lucene and Solr integrations. These files support multiple versions of Lucene/Solr, as indicated below.

RBL supports Lucene versions 7.0 - 10.1.0 and Solr versions 7.0 - 9.8 with the following files, where <version> is the RBL version:

RBL uses the Logging Facade for Java (SLF4J) to log activities. The SLF4J API JAR is in lib/.

SFL4J is a facade for various logging APIs. Using SFL4J, the developer or an administrator can determine which one of many popular logging systems to use at runtime. In the tools/lib directory, we include the SLF4J binding JARs:

These JARs are used by our samples and RBLCmd. When you place these JARs on your classpath, the logging facade is bound to the implementation, and RBL logging is turned on. As defined in the file etc/log4j2.properties, by default, WARN messages are output to the console (System.err). As the Javadoc for org.slf4j.impl.SimpleLogger explains, you can use system properties or this properties file to output to a file and control other logging parameters.

If you want to use SLF4J with a different implementation, put the appropriate binding JAR files and properties file on your classpath.

Depending on the scope of your application, you may wish to remove unnecessary files to reduce the size of your application.

The JAR jakarta.annotation-api-<version>.jar is included in RBL but cannot be shaded. There may be conflicts if this JAR is used elsewhere in your processing. The JAR is used in disambiguation of Arabic, English, Greek, Hebrew, Japanese, Korean, and Spanish. If you are not using this functionality, the JAR can be removed.

If the option rootDirectory is specified, then the string ${rootDirectory} takes that value in the dictionaryDirectory, modelDirectory, and licensePath options.

Enum Classes:

A sample application that illustrates the use of ADM is in rbl-je-<version>/samples/annotator-tokenize.

Your license (rlp-license.xml) must be in the licenses subdirectory of the RBL installation.

AnnotatorTokenize tokenizes the English string and provides one or more analyses with lemma and part-of-speech for each token.

The output appears in annotator-tokenize.txt.

length: 29
------
Some members spoke yesterday.

------
token 0: Some
index lemma part-of-speech
0 some QUANT
token 1: members
index lemma part-of-speech
0 member NOUN
token 2: spoke
index lemma part-of-speech
0 speak VPAST
1 spoke VI
2 spoke VPRES
3 spoke NOUN
token 3: yesterday
index lemma part-of-speech
0 yesterday ADV
1 yesterday NOUN
token 4: .
index lemma part-of-speech
0 . SENT

A sample application illustrating the use of the classic API is in rbl-je-<version>/samples/tokenize-analyze.

Your license (rlp-license.xml) must be in the licenses subdirectory of the RBL installation.

TokenizeAnalyze tokenizes the sample German document and provides a disambiguated analysis of each token.

The output appears in two files: deu-tokenized.txt and deu-analyzed.txt. The first file contains a token on each line, with a blank line following the end of a sentence.

The second file contains the token, lemma, part of speech, and compound components (where relevant) on each line. For those languages for which disambiguation is not supported,^[2] there may be multiple rows for each token (the token appearing in the first column), one for each analysis. Here is a fragment with a sentence from deu-analyzed.txt:

TOKEN                    LEMMA                    POS       COMPOUNDS 
-----                    -----                    ---       ---------
3.11.06                  3.11.06                  CARD   
-                       -                         PUNCT     
Not                      Not                      NOUN      
und                      und                      COORD     
Elend                    Elend                    NOUN      
in                       in                       PREP      
ihren                    ihr                      POSDET    
Heimatländern            Heimatland               NOUN      [Heimat, Land]
lassen                   lassen                   VVFIN     
immer                    immer                    ADV       
mehr                     mehr                     INDADJ    
Afrikaner                Afrikaner                NOUN      
die                      der                      ART       
Reise                    Reise                    NOUN      
nach                     nach                     PREP      
Europa                   Europa                   NOUN      
antreten                 antreten                 VVINF     
.                        .                        SENT     

To run the samples with sample text in a different language, set the test.language parameter with the language code. For example to tokenize and analyze the Spanish sample, call

ant -Dtest.language=spa run

RBLCmd is a general-purpose command line utility for RBL. It provides a simple way to produce RBL output without writing code. It is also useful for ad hoc speed and thread testing.

A Bash shell script (RBLCmd) and Windows script (RBLCmd.bat) for running this utility are in rbl-je-<version>/tools/bin. For more information, see RBLCmd's on-line help, RBLCmd -h.

The command:

echo 'Hola' | ./tools/bin/RBLCmd -outputJson --language spa --rootDirectory . | jq

produces the following output:

{
  "version": "1.1.0",
  "data": "Hola\n",
  "attributes": {
    "sentence": {
      "type": "list",
      "itemType": "sentence",
      "items": [
        {
          "startOffset": 0,
          "endOffset": 5
        }
      ]
    },
    "scriptRegion": {
      "type": "list",
      "itemType": "scriptRegion",
      "items": [
        {
          "startOffset": 0,
          "endOffset": 5,
          "script": "Latn"
        }
      ]
    },
    "layoutRegion": {
      "type": "list",
      "itemType": "layoutRegion",
      "items": [
        {
          "startOffset": 0,
          "endOffset": 5,
          "layout": "STRUCTURED"
        }
      ]
    },
    "token": {
      "type": "list",
      "itemType": "token",
      "items": [
        {
          "startOffset": 0,
          "endOffset": 4,
          "text": "Hola",
          "analyses": [
            {
              "partOfSpeech": "INTERJ",
              "lemma": "hola",
              "raw": "hola[+INTERJ]",
              "tagSet": "BT_SPANISH"
            }
          ]
        }
      ]
    }
  },
  "documentMetadata": {}
}

A document may contain tables and lists in addition to regular sentences. Structured text is composed of fragments, such as list items, table cells, and short lines of text. The tokenizer emits sentence offsets for each fragment it encounters.

One way fragments are identified is by detecting fragment delimiters. A delimiter is restricted to one character; the default delimiters are U+0009 (tab), U+000B (vertical tab), and U+000C (form feed). To modify the set of recognized delimiters, pass a string containing all desired delimiter values to the fragmentBoundaryDelimiters option. The string must include any default values you want to keep. Non-whitespace delimiters within a token will be ignored.

The following rules determine where fragments are identified, in descending priority:

Fragments always include trailing whitespace.

Example:

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory)
factory.setOption(BaseLinguisticsOption.language, "eng");
EnumMap<BaseLinguisticsOption, String> options = Maps.newEnumMap(BaseLinguisticsOption.class);
options.put(BaseLinguisticsOption.fragmentBoundaryDelimiters, "|~");
options.put(BaseLinguisticsOption.maxTokensForShortLine, "5");
factory.createSingleLanguageAnnotator(options);

By default, fragment detection is enabled. Use the fragmentBoundaryDetection option to disable it.

Enum Classes:

RBL supports POS-tagging of emoji, emoticons, @mentions, email addresses, hashtags, and URLs in all supported languages.

Tokenization of emoji is always enabled. The other options are disabled by default but can be enabled through the options listed. When tokenization is disabled, the characters may be split into multiple tokens.

Enum Classes:

Sample input and output

Type	Input	Tokenization (when option is enabled)	Tokenization (when option is disabled)	POS tag
emoji			Tokenization of emoji cannot be disabled.	EMO
emoticon	:)	:)	: )	EMO
@mention	@basistechnology	@basistechnology	@ basistechnology	ATMENTION
hashtag	#basistechnology	#basistechnology	# basistechnology	HASHTAG
email address	info@basistech.com	info@basistech.com	info @ basistech.com	EMAIL
URL	http://www.babelstreet.com	http://www.babelstreet.com	http : / / www.babelstreet.com	URL

The tokenization when the option is disabled depend on the options language and tokenizerType. The samples provided here are for when language=eng and tokenizerType=ICU.

Emoji Normalization & Lemmatization

RBL normalizes emoji, placing the result into the lemma field. The simplest example is when an emoji presentation selector follows a character that is already an emoji. In this case, RBL will simply remove the emoji presentation selector.

Lemmatization applies to an emoji character in multiple ways.

Emoji that depict people or body parts may be followed by an emoji modifier indicating skin tone. Lemmatization simply removes the emoji modifier skin tone from the emoji character. The reasoning is that the skin tone is of secondary importance to the meaning of the emoji.

Surface form	Lemmatized form
( Boy + Medium Skin Tone)

Emoji depicting people may be followed by an emoji component indicating hair color or style. Lemmatizing removes the hair component from the emoji character.

Surface form	Lemmatized form
( Adult + ZWJ + Red Hair)

Where a gender symbol has been added to create a gendered occupation emoji, lemmatization removes the gender symbol.

Surface form	Lemmatized form
( Police Officer + ZWJ + ♀ Female Sign + VS16)

Finally, RBL can normalize non-fully-qualified emoji ZWJ sequences to fully-qualified emoji ZWJ sequences. In the above example, it is possible to omit the VS16 (though discouraged by Unicode): since Police Officer is an emoji, anything joined to it by a ZWJ is implicitly an emoji too. RBL adds the missing VS16.

The ICU tokenizer is the default tokenizer used for European languages. It works based on behavior defined in a rule file. If the default behavior is not exactly what is desired, RBL allows custom rule files to be supplied that will determine the behavior of the tokenizer. How to make these customizations is briefly outlined here. Be careful with any changes you make to the tokenizer behavior; BasisTech does not support customizations made by the user.

BaseLinguisticsFactory has a method addCustomTokenizerRules which can be used to specify a custom rule file. RBLCmd also has the -ctr option to specify a path on the command line. All of these methods accept a case-sensitivity value (for -ctr, cs and ci mean case-sensitive and case-insensitive), which is important because only when BaseLinguisticsOption.caseSensitive is the same as the value for a rule file will it be selected. Custom rule files are not cumulative, i.e. only one set of rules may be used at a time for any one combination of case sensitivity and language.

...
$Smiley = [\:=][)}\]];

!!forward;
$Smiley;
...

=)          

=
)       

=)       

RBL provides basic tokenization support when the language is "Unknown" (xxx). The tokenizer uses generic rules to tokenize, such as whitespace and punctuation delimitation.

Supported Features when language is unknown (xxx):

Using the language code of xxx will provide basic tokenization support for languages not supported by RBL.

For each token and normalized form in the token stream, the analyzer performs a dictionary lookup starting with any user dictionaries followed by the RBL dictionary. During lookup, RBL ignores the context in which the token or normalized form appears.

Once the analyzer has found one or more lemmas in a dictionary, it does not consult additional dictionaries. In other words, if two user dictionaries are specified, and the filter finds a lemma in the first dictionary, it does not consult the second user dictionary or the RBL dictionary.

Unless overridden by an analysis dictionary, the only lemmatization done in Chinese and Thai is number normalization. Other Chinese and Thai tokens' lemmas are equal to their surface forms.

There is no analysis dictionary available for Finnish, Pashto, or Urdu. All other languages are supported.

No dictionary can ever be complete: new words get added to languages, languages change and borrow. So, in general, analysis for each language includes some sort of guessing capability. The job of a guesser is to take a word and to come up with some analysis of it. Whatever facts we generate for a language, those are all possible outputs of a guesser.

In European languages, guessers deliver lemmas and parts of speech. In Korean, guessers provide morphemes, morpheme tags, compound components, and parts of speech.

By default, the analyzer returns any lemma that contains whitespace as multiple lemmas (each with no whitespace). To allow lemmas with whitespace (such as International Business Machines as a lemma for the token IBM) to be placed as such in the token stream, you can create a user analysis dictionary with an entry that defines the lemma. For example:

IBM	International[^_]Business[^_]Machines[+PROP]

The analyzer decomposes Chinese, Danish, Dutch, German, Hungarian, Japanese, Korean, Norwegian, and Swedish compounds, returning the lemmas of each of the components.

The lemmas may differ from their surface form in the compound, such that the concatenation of the components is not the same as the original compound (or its lemma). Components are often connected by elements that are present only in the compound form.

For example, the German compound Eingangstüren (entry doors) is made up of two components, Eingang (entry) and Tür (door), and the connecting 's' is not present in the component list. For this input token, the RBL tokenizer and analyzer return the following entries:

Other German examples include letter removal (Rennrad ⇒ rennen + Rad), vowel changes (Mängelliste ⇒ Mangel + Liste), and capitalization changes (Blaugrünalge ⇒ blau + grün + Alge).

Enum Classes:

For some languages, the analyzer can disambiguate between multiple analysis objects and return the disambiguated analysis object. The disambiguate option enables the disambiguator. When true, the disambiguator determines the best analysis for each word given the context in which it appears.

When using an annotator, the disambiguated result is at the head of all possible analyses. The remainder of the list is ordered randomly. When using a tokenizer/analyzer, use the method getSelectedAnalysis to return the disambiguated result.

For all languges except Japanese, disambiguation is enabled by default. For performance reasons, disambiguation is disabled by default for Japanese when using the statistical model.

Enum Classes:

In RBL, each language has its own set of POS tags and a few languages have multiple tag sets. Each tag set is identified by an identifier, which is a value of the TagSet enum. When RBL outputs a POS tag, it also lists the identifier for the tag set it came from. Output from a single language may contain POS tags from multiple tag sets, including the language-neutral set.

POS tags are defined for Arabic, Chinese, Czech, Dutch, English, French, German, Greek, Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Malay (Standard), Persian, Polish, Portuguese, Russian, Spanish, Tagalog, and Urdu.Part-of-Speech Tags

-
  - NUM_VOC
  -
    - { m: \+Total, t: PRON }
    - { s: moc, t: DET }
    - { s: oba, t: DET }
    - { t: NUM }

You can split contractions and return analyses with tokens, lemmas, and POS tags for each constituent. For example, given the English contraction can't, RBL returns analyses for can and for not. To split contractions, set tokenizeContractions to true.

Contractions are defined by contraction rule files. By default, the tokenizer uses the rules in rootDirectory/contractions/contraction-rules-<language>.yaml, where <language> is the language code. RBL comes with contraction rules for English, German, and Portuguese. To add rules for these languages or add rules to support another language, edit the default files or create a custom rule file. The URI for the custom file is defined by customTokenizeContractionRulesUri.

Enum Classes:

For a sample, see rbl-je-<version>/samples/contractions.

-
  - [ "ain't", "VBPRES" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]
-
  - [ "amn't", "ADJ" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]
-
  - [ "amn't", "NOUN" ]
  -
    - [ "am", "VBPRES", "be", null ]
    - [ "not", "NOT", "not", null ]

Enum Classes:

The ignoreStopwords option uses a stop words list to define stop words. The path to the stop words list is language-dependent: Chinese uses root/dicts/zho/cla/zh_stop.utf8 and Japanese uses root/dicts/jpn/jla/JP_stop.utf8.

You can add stop words to these files. When you edit one of these files, you must follow these rules:

RBL includes multiple disambiguators for Hebrew. Set the value for the option disambiguatorType to select which type to use. The valid values for DisambiguatorType are:

Enum Classes:

Generic Arabic script normalization includes the following:

All static and dynamic user dictionaries, except for many-to-one normalization dictionaries, are consulted in reverse order of addition. In cases of conflicting information, dictionaries added later take priority over those added earlier. Once a token is found in a user dictionary, RBL stops and will consult neither the remaining user dictionaries nor the RBL dictionary.

Many-to-one normalization dictionaries are consulted in the following order:

Example of non-many-to-one user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, statDict3, statDict2, dynDict1

Example of many-to-one normalization user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, dynDict1, statDict2, statDict3

The Chinese and Japanese language analyzers load all dictionaries with the user dictionaries loaded at the end of the list. To prioritize the user dictionaries and put them at the front of the list, guaranteeing the matches in the user dictionaries will be used, set the option favorUserDictionary to true.

The following formatting rules apply to user dictionary source files.

Once complete, the source file is compiled into a binary format for use in RBL.

norm1        var1        var2        var3

dictionary.add("norm1", "var1", "var2", "var3");

In the tools/bin directory, RBL includes a shell script for Unix

rbl-build-user-dictionary

and a .bat file for Windows.

rbl-build-user-dictionary.bat

To compile a user dictionary, from the RBL root directory:

tools/bin/rbl-build-user-dictionary -type TYPE_ARGUMENT LANG INPUT_FILE OUTPUT_FILE

where TYPE_ARGUMENT is the dictionary type, LANG is the language code, INPUT_FILE is the pathname of the source file you have created, and OUTPUT_FILE is the pathname of the binary compiled dictionary the tool creates. For example:

tools/bin/rbl-build-user-dictionary -type analysis jpn jpn_lemmadict.txt jpn_lemmadict.bin

The script uses Java to compile the user dictionary. The operation is performed in memory, so you may require more than the default heap size. You can set heap size with the JAVA_OPTS environment variable. For example, to provide 8 GB of heap size, set JAVA_OPTS to -Xmx8g.

Unix shell:

export JAVA_OPTS=-Xmx8g

Windows command prompt:

set JAVA_OPTS=-Xmx8g

The format for a segmentation dictionary source file is very simple. Each word is written on its own line, and that word is guaranteed to be segmented as a single token when seen in the input text, regardless of context. Japanese example:

三菱UFJ銀行
酸素ボンベ

Each entry is a word, followed by a tab and an analysis. The analysis must end with a lemma and a part-of-speech (POS) tag.Part-of-Speech Tags

word	lemma[+POS]

For those languages for which RBL does not return POS tags, use DUMMY.

Variations. You can provide more than one analysis for a word or more than one version of a word for an analysis.

The following example includes two analyses for "telephone" (noun and verb), and two renditions of "dog" for the same analysis (noun).

telephone	telephone[+NOUN]
telephone	telephone[+VI]
dog	dog[+NOUN]
Dog	dog[+NOUN]               

For some languages, the analysis may include special tags and additional information.

Contracted forms. For English, French, Italian, and Portuguese, ^= is a separator for a contraction or elision.

English example:

doesn't	does[^=]not[+VDPRES]

Multi-Word Analysis. For English, Italian, Spanish, and Dutch, ^_ indicates a space.

English example:

IBM	International[^_]Business[^_]Machines[+PROP]

Compound Boundary. For Danish, Dutch, Norwegian, German, and Swedish, ^# indicates the boundary between elements in a compound word. For Hungarian, the compound boundary tag is ^CB+.

German example:

heimatländern	Heimat[^#]Land[+NOUN]

Compound Linking Element. For German ^/, indicates a compound linking element. For Dutch, use ^//.

German example:

arbeitskreis	Arbeit[^/]s[^#]Kreis[+NOUN]

Derivation Boundary or Separator for Clitics. For Italian, Portuguese, and Spanish, ^| indicates a derivation boundary or separator for clitics.

Spanish example with derivation boundary:

duramente	duro[^|][+ADV]

Italian example with separator for clitics:

farti	fare[^|]tu[+VINF_CLIT]

Japanese Readings and Normalized Forms. For Japanese, [^r] precedes a reading (there may be more than one), and [^n] precedes a normalization. For example:

行わ	行う[^r]オコナワ[+V]
tv	テレビ[^r]テレビ[^n]テレビ[+NC]
アキュムレータ	アキュムレーター[^r]アキュムレータ[^n]アキュムレーター[+NC]

Korean Analysis. A Korean analysis uses a different pattern than the analysis for other languages. The pattern for an analysis in a user Korean dictionary is as follows:

Token	Mor1[/Tag1][^+]Mor2[/Tag2][^+]Mor3[/Tag3]

Where each MorN is a morpheme, consisting of one or more Korean characters, and TagN is the POS tag for that morpheme. [^+] indicates the boundary between morphemes.

Here's an example:

유전자이다	유전자[/NPR][^+]이[/CO][^+]다[/ECS]

If the analysis contains one noun morpheme, that morpheme is the lemma and the POS tag is the POS tag for that morpheme. If more than one of the morphemes are nouns, the lemma is the concatenation of those nouns (a compound). Example:

정보검색	정보[/NNC][^+]검색/[NNC]

Otherwise, the lemma is the first morpheme, and the POS tag is the POS tag associated with that morpheme.

You can override this algorithm for identifying the lemma and/or POS tag in a user dictionary entry by placing [^L]lemma and/or [^P][/Tag] at the end of the analysis. The lemma may or may not correspond to one of the morphemes in the analysis. For example:

유전자이다	유전자[/NNC][^+]이[/CO][^+]다[/ECS][^L]유전[^P][/NPR]

The KoreanAnalysis interface provides access to the morphemes and tags associated with a given token in either the standard Korean dictionary or a user Korean dictionary.

A many-to-one normalization dictionary maps one or more variants to a normalized form. The first value on each line is the normalized form. The remainder of the entries on the line are the variants to be mapped to the normalized form. All values on the line are separated by tabs.

Example:

Use the option normalizationDictionaryPaths to specify the static user normalization dictionaries.

The source file for a Chinese or Japanese user dictionary is UTF-8 encoded (see Valid Characters for Chinese or Japanese User Dictionary Entries). The file may begin with a byte order mark (BOM). Empty lines are ignored. A comment line begins with #. The first line of a Japanese dictionary may begin !DICT_LABEL followed by Tab and an arbitrary string to set the dictionary's name, which is not currently used anywhere.

Each entry in the dictionary source file is a single line:

word Tab POS Tab DecompPattern Tab Reading1,Reading2,...

where word is the entry or surface form, POS is one of the user-dictionary part-of-speech tags listed below, DecompPattern is the decomposition pattern: a comma-delimited list of numbers that specify the number of characters from word to include in each component of the compound (0 for no decomposition), and Reading1,... is a comma-delimited list of one or more transcriptions rendered in Hiragana or Katakana (only applicable to Japanese).

The decomposition pattern and readings are optional, but you must include a decomposition pattern if you include readings. In other words, you must include all elements to include the entry in a reading user dictionary, even though the reading user dictionary does not use the POS tag or decomposition pattern. To include an entry in a segmentation (tokenization) user dictionary, you only need POS tag and an optional decomposition pattern. Keep in mind that those entries that include all elements can be included in both a segmentation (tokenization) user dictionary and a reading user dictionary.

For more information on Chinese and Japanese POS tags, refer to the following sections:

The tables below list the primary and secondary tags emitted when the user POS tag is specified.

Examples (the last three entries include readings):

!DICT_LABEL New Words 2014
デジタルカメラ NOUN
デジカメ NOUN 0
東京証券取引所 ORGANIZATION 2,2,3
狩野 SURNAME 0
安倍晋三 PERSON 2,2 あべしんぞう
麻垣康三 PERSON 2,2 あさがきこうぞう
商人 NOUN 0 しょうにん,あきんど
        

The POS and decomposition pattern can be in full-width numerals and Roman letters. For example:

東京証券取引所 ｏｒｇａｎｉｚａｔｉｏｎ ２,２,３

The "2,2,3" decomposition pattern instructs the tokenizer to decompose this compound entry into

東京
証券
取引所
        

There are two standard forms of written Chinese: Simplified Chinese (SC) and Traditional Chinese (TC). SC is used in Mainland China. TC is used in Taiwan, Hong Kong, and Macau.

Conversion from one script to another is a complex matter. The main problem of SC to TC conversion is that the mapping is one-to-many. For example, the simplified form 发 maps to either of the traditional forms 發 or 髮. Conversion must also deal with vocabulary differences and context-dependence.

The Chinese Script Converter converts text in simplified script to text in traditional script, or vice versa. The conversion can be on any of three levels, listed here from simplest to the most complex:

The Chinese input may contain a mixture of TC and SC, and even some non-Chinese text. The Chinese Script Converter converts to the target (SC or TC), leaving any tokens already in the target form and any non-Chinese text unchanged.

See Initial and Path Options for additional options

Enum Classes:

This example uses the BaseLinguisticsFactory and CSCAnalyzer.

The RBL distribution includes a sample (CSCAnalyze) that you can compile and run with an ant build script.

In a Bash shell script (Unix) or Command Prompt (Windows), navigate to rbl-je-<version>/samples/csc-analyze and call

ant run

The sample reads an input file in SC and prints each token with its TC conversion to standard out.

This example tokenizes Chinese test and converts from TC to SC.

CSC user dictionaries support orthographic and lexemic conversions between Simplified Chinese and Traditional Chinese. They are not used for codepoint conversion.

CSC user dictionaries follow the same format as other user dictionaries:

Once complete, the source file is compiled into a binary format for use in RBL.

Each entry contains two or three tab-delimited elements:

input_token orthographic_translation [lexemic_translation]

The input_token is the form you are converting from and the orthographic_translation and optional lexemic_translation are the form you are converting to.

Sample entries for a TC to SC user dictionary:

電腦	电脑	计算机
宇宙飛船	宇宙飞船
            

Compiling a CSC User Dictionary. In the tools/bin directory, RBL includes a shell script for Unix

rbl-build-csc-dictionary

and a .bat file for Windows

rbl-buld-csc-dictionary.bat

The script uses Java to compile the user dictionary. The operation is performed in memory, so you may require more than the default heap size. You can set heap size with the JAVA_OPTS environment variable. For example, to provide 8 GB of heap size, set JAVA_OPTS to -Xmx8g.

Unix shell:

export JAVA_OPTS=-Xmx8g

Windows command prompt:

set JAVA_OPTS=-Xmx8g

Compile the CSC user dictionary from the RBL root directory:

tools/bin/rbl-build-csc-dictionary INPUT_FILE OUTPUT_FILE

INPUT_FILE is the pathname of the source file you have created, and OUTPUT_FILE is the pathname of the binary compiled dictionary the tool creates. For example:

tools/bin/rbl-build-csc-dictionary my_tc2sc.txt my_tc2sc.bin

RBL provides an API for integrating with Apache Lucene. See the Javadoc that accompanies this package for the complete API documentation.

A Lucene analyzer examines the text of fields and generates a token stream. Rosette provides an RBL base linguistics analyzer for Lucene, which uses RBL's linguistic tools.

RBL supports two usage patterns for incorporating Rosette Base Linguistics in a Lucene application.

Samples are included in the use cases below. The code comes from the Lucene 7.0 samples found in rbl-je-<version>/samples/lucene-7_0.

RBL supports Lucene versions 7.0 - 10.1.0 and Solr versions 7.0 - 9.8 with the following files, where <version> is the RBL version:

The following options are passed in through an options Map.

Enum Classes:

The RBL Lucene base linguistics analyzer (BaseLinguisticsAnalyzer) provides an analysis chain with a language-specific tokenizer and token filter.

You can configure the analyzer with a number of tokenizer and token filter options.

The analysis chain includes the Lucene filters LowerCaseFilter and CJKWidthFilter,. It also provides support for including a StopFilter.

Japanese Analyzer Sample. This Lucene sample, JapaneseAnalyzerSample.java, uses the base linguistics analyzer to generate an enriched token stream from Japanese text.

To run the sample, in a Bash shell (Unix) or Command Prompt (Windows), navigate to rbl-je-<version>/samples/lucene-5_0 and use the Ant build script:

The sample reads rbl-je-<version>/samples/data/jpn-text.txt and writes the output to jpn-Analyzed-byAnalyzer.txt.

The output includes each token, lemma, and reading in the token stream on a separate line with the type attribute for each element. There may be more than one analysis (hence lemma and reading in the sample output) for a token; lemmas are not put into the token stream when identical to the token.

For example:

When creating an analysis chain, you can do the following:

One of the tasks of the token filter is to set tokens’ token types. For example, given a lemma token, the token filter gives it the type <LEMMA>. Given a contraction component token, the token filter has a choice: by default, the type is set to <LEMMA>, but when the identifyContractionComponents option is enabled, the type is set to <CONT>.

ant runTokenizerAndFilter

You can use the com.basistech.rosette.lucene.AnalysesAttribute object to gather linguistic data about the text in a document. Depending on the language, the data may include tokens, normalized tokens, lemmas, part-of-speech tags, readings, compound components, and Semitic roots.

The Lucene sample, AnalysesAttributeSample.java, illustrates this.

To run the sample with the German sample file, navigate to rbl-je-<version>/samples/lucene-<version>, and call ant as follows:

ant -Dtest.language=deu runAnalysesAttribute

The sample writes the output to deu-analysesAttributes.txt.

In some languages, case distinctions are meaningful. For example, in German, a word may be a noun if it begins with an upper-case letter, and not a noun if it does not. As a result, RBL delivers higher accuracy in selecting lemmas and splitting compounds when it can process text with correct casing. On the other hand, users typing in queries may be sloppy with capital letters.

For this reason, the default behavior of the Lucene integration is to perform the following analysis steps:

The result is that the index contains the lowercase form of the most accurately selected lemma.

However, some applications work with text in which case distinctions are not reliably present, even in languages where they are important. These applications need to determine lemmas and compound components even though the spelling is nominally incorrect with respect to case.

To support these applications, RBL provides a 'case-insensitive' mode of operation. In this mode, RBL performs the following analysis steps:

The mapping is still required to ensure that the index or query ends up with uniformly lowercase text.

To specify case sensitivity for the analysis, set com.basistech.rosette.bl.AnalyzerOption.caseSensitive to true or false. By default, the setting is true, except for Danish, Norwegian, and Swedish, for which our dictionaries are lowercase and the setting is false irrespective of the user setting.

When you are making this setting in the com.basistech.rosette.lucene package, include the caseSensitive option as a string. For example:

Map<String, String> options = new HashMap<>();
options.put("language", LanguageCode.ITALIAN.ISO639_30);
options.put("caseSensitive", "true");
TokenFilterFactory factory = new BaseLinguisticsTokenFilterFactory(options);
        

User Dictionaries can be used when using RBL with Lucene.

In the com.basistech.rosette.lucene package, BaseLinguisticsTokenizerFactory and BaseLinguisticsTokenFilterFactory can load segmentation and analysis dictionaries respectively.

The path options are provided as a list of paths, separated by semicolons or the OS-specific path separator.

BaseLinguisticsTokenizerFactory provides the method addUserDefinedDictionary for adding a segmentation dictionary. For example:

Map<String, String> args = new HashMap<>();
args.put(TokenizerOption.language.name(), LanguageCode.JAPANESE.ISO639_30);
args.put(TokenizerOption.nfkcNormalize.name(), "true");
BaseLinguisticsTokenizerFactory factory = new BaseLinguisticsTokenizerFactory(args);
factory.addUserDefinedDictionary(LanguageCode.JAPANESE, "/path/to/my/jpn-dict.bin");

The constructor for BaseLinguisticsTokenFilterFactory takes a Map of options. Use the userDefinedDictionaryPath option to load an analysis dictionary:

Map<String, String> options = new HashMap<>();
options.put("language", LanguageCode.ITALIAN.ISO639_30);
options.put("userDefinedDictionaryPath", "/path/to/my/ita-dict.bin");
options.put("caseSensitive", "true");
TokenFilterFactory factory = new BaseLinguisticsTokenFilterFactory(options);

RBL/Lucene Distribution Sample. For supported versions of Lucene, the RBL distribution includes a sample (CSCCharTermAttributeSample) that you can compile and run with an ant build script.

In a Bash shell script (Unix) or Command Prompt (Windows), navigate to the samples directory (rbl-je-<version>/samples) and the file for your version of lucene (/csc-analyze-<luceneversion> and call:

ant run

The sample reads an input file in SC and prints the TC conversion for each token to standard out.

You can configure Apache Solr search to use RBL for both indexing documents as well as for queries.

To index and search documents with RBL in a Solr application, you must add JARs to the Solr classpath and define Solr analysis chains that apply the RBL analysis components to process text at index and query time.

In Solr 9.0.0 and later, you must update the policy file at server/etc/security.policy to grant Solr permissions to read the files in RBL-JE's root directory. Add the following line to the file:

grant {
  permission java.io.FilePermission "<RBLJE_ROOT>${/}-", "read";
};

where you replace <RBLJE_ROOT> with the path to the root of your RBL installation.

Add the following lib elements to the solrconfig.xml for each Solr collection you are using.

<lib path="<RBLJE_ROOT>/lib/btrbl-je-<version>.jar"/>
<lib path="<RBLJE_ROOT>/lib/btcommon-api-<version>.jar"/>
<lib path="<RBLJE_ROOT>/lib/slf4j-api-<version>.jar"/>
<lib path="<RBLJE_ROOT>/lib/btrbl-je-lucene-solr-<version>-<version>.jar"/>

where you replace <RBLJE_ROOT> with the path to the root of your RBL installation and take the full file names with correct <version> values from the RBL <lib path>. They can change with each new release.

For example, if the root of the RBL installation is /opt/local/bt/rbl-je , the version of Solr is 8.6, and the version of RBL is 7.44.0.c67.0, the lib paths are:

<lib path="/opt/local/bt/rbl-je/lib/btrbl-je-7.44.0.c67.0.jar"/>
<lib path="/opt/local/bt/rbl-je/lib/btcommon-api-37.0.1.jar"/>
<lib path="/opt/local/bt/rbl-je/lib/slf4j-api-1.7.28.jar"/>
<lib path="/opt/local/bt/rbl-je/btrbl-je-lucene-solr-7_0-7.44.0.c67.0.jar"/>

The correct Lucene/Solr version files are listed in the section Lucene/Solr Versions.

The SLF4J JARs enable logging.

In the Solr schema.xml or managed-schema.xml, add a fieldType element and a corresponding field element for the language of the documents processed by the application.

Field Type. The fieldType includes two analyzers: one for indexing documents and one for querying documents. Each analyzer contains a tokenizer and a token filter.

Here, for example, is a fieldType for Japanese:

<fieldtype name="basis-japanese" class="solr.TextField">
  <analyzer type="index">
    <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory"
               language="jpn"
               rootDirectory="<RBLJE_ROOT>"
    />
    <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory"
            language="jpn"
            rootDirectory="<RBLJE_ROOT>"
    />
  </analyzer>
  <analyzer type="query">
    <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory"
               language="jpn"
               rootDirectory="<RBLJE_ROOT>"
    />
    <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory"
            language="jpn"
            rootDirectory="<RBLJE_ROOT>"
            query="true"
    />
  </analyzer>
</fieldtype>

where you replace <RBLJE_ROOT> with the path to the root of your RBL installation. The fieldType name indicates the language, and each language attribute is set to the ISO 693-3 language code for Japanese.

Field. The analysis chain requires a field definition with a type attribute that maps to the fieldType. For the Japanese example above, add the following field definition to schema.xml.

<field name="text-japanese" type="basis-japanese" indexed="true" stored="true"/>

In your Solr application, you can now index and query Japanese documents placed in the text-japanese field.

Most API options can be used in a Solr analysis chain. In Solr, you do not directly use the option enum classes. Instead, options are specified in the schema using the format option="value".

When specifying options for the tokenizer (class BaseLinguisticsTokenizerFactory), use options in the TokenizerOption enum class. When specifying options for a filter (class BaseLinguisticsTokenFilterFactory), use options in the AnalyzerOption and FilterOption enum classes.

Example:

<fieldType class="solr.TextField" name="basis-french">
    <analyzer type="index">
        <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory" language="fra" 
        rootDirectory="${bt_root}"/>
            <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory" language="fra" 
        rootDirectory="${bt_root}"/>
            <filter class="solr.LowerCaseFilterFactory"/>
            <filter class="org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory"/>
    </analyzer>
    <analyzer type="query">
         <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory" language="fra" 
    query="true" rootDirectory="${bt_root}"/>
         <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory" language="fra" 
    query="true" rootDirectory="${bt_root}"/>
         <filter class="solr.LowerCaseFilterFactory"/>
         <filter class="org.apache.solr.analysis.RemoveDuplicatesTokenFilterFactory"/>
    </analyzer>
</fieldType>

When using the neural models with Solr, you must grant the appropriate permissions and may want to increase the JVM memory.

The security manager is enabled by default in Solr 9. To use any of the neural models, grant the following permissions in server/etc/security.policy:

grant {
   permission java.io.FilePermission \"$HOME/.javacpp/cache\", \"read,write,delete,execute\";
   permission java.lang.RuntimePermission \"loadLibrary.*\";
};

Neural models may require more memory. We recommend increasing the memory to at least 1 GB by editing bin/solr/in.sh:

SOLR_JAVA_MEM="-Xms1g -Xmx1g"

User Dictionaries can be used when using RBL with Solr.

Use the option userDefinedDictionaryPath as shown in this example:

<fieldType class="solr.TextField" name="basis-japanese">
    <analyzer>
        <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory"
            tokenizerType="spaceless_lexical"
            language="jpn"
            rootDirectory="${bt_root}"
            userDefinedDictionaryPath= "/path/to/my/jpn-udd.bin"/>
        <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory"
            tokenizerType="spaceless_lexical"  
            language="jpn"
            rootDirectory="${bt_root}"/>
    </analyzer>
</fieldType>

Here is an example of using a JLA reading dictionary:

<fieldType class="solr.TextField" name="basis-japanese-rd">
    <analyzer>
        <tokenizer class="com.basistech.rosette.lucene.BaseLinguisticsTokenizerFactory"
            tokenizerType="spaceless_lexical"
            readings="true"
            language="jpn"
            rootDirectory="${bt_root}"
            userDefinedReadingDictionaryPath="/path/to/my/readings.bin"/>
        <filter class="com.basistech.rosette.lucene.BaseLinguisticsTokenFilterFactory"
            tokenizerType="spaceless_lexical"
            language="jpn"
            rootDirectory="${bt_root}"/>
    </analyzer>
</fieldType>

RBL canonicalizes some language codes to other language codes. These canonicalization rules apply to all APIs.

LanguageCode.NORWEGIAN is canonicalized to LanguageCode.NORWEGIAN_BOKMAL immediately upon any input to the API. This means that there can be no distinguishing between them. In particular, an Analyzer built from a factory configured to use Norwegian will report its language as Norwegian Bokmål instead.

Similarly, LanguageCode.SIMPLIFIED_CHINESE and LanguageCode.TRADITIONAL_CHINESE are canonicalized to LanguageCode.CHINESE immediately. The one exception is that they are not canonicalized as inputs to or outputs from the Chinese Script Converter.

Those are the only language code canonicalizations. Although RBL internally treats Afghan Persian and Iranian Persian as Persian, they are not considered the same language. This makes it possible to configure different user dictionaries for each variety of Persian, even though they are otherwise processed identically.

RBL uses ISO 639-3 language codes to specify languages as strings. There are a few nonstandard language codes, as indicated. RBL also accepts 2-letter codes specified by the ISO-639-1 standard. See the Javadoc for more details on language codes.