Analyze Language

You can use RBL in your own JVM application, use its Apache Lucene compatible API in a Lucene application, or integrate it directly with either Apache Solr or Elasticsearch. ^[6]

RBL performs multiple types of analysis. Depending on the language, one or more of the following may be identified in the input text:

For some languages, the analyzer can disambiguate between multiple analysis objects and return the disambiguated analysis object.

In the ADM API, use the BaseLinguisticsFactory to set the linguistic options and instantiate an Annotator which annotates the input text. The ADM API creates an annotator for all linguistic objects, including tokens.Using the Annotated Data Model API

In the classic API, use the BaseLinguisticsFactory to configure and create tokenizers, analyzers, and CSC analyzers. The classic API creates separate tokenizers and analyzers.Using the Classic API

When using the classic API, you instantiate separate factories for tokenizers and analyzers.

If your application requires streaming, use this API. The Lucene, Solr, and Elasticsearch integrations use these methods.

For the complete API documentation, consult the Javadoc for BaseLinguisticsFactory.

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);

factory.setOption(BaseLinguisticsOption.nfkcNormalize, "true");

Tokenizer tokenizer = factory.createTokenizer();

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory);
File rootPath = new File(rootDirectory);
factory.setOption(BaseLinguisticsOption.licensePath, 
        new File(rootPath, "licenses/rlp-license.xml").getAbsolutePath());
Analyzer analyzer = factory.createAnalyzer();

A document may contain tables and lists in addition to regular sentences. Structured text is composed of fragments, such as list items, table cells, and short lines of text. The tokenizer emits sentence offsets for each fragment it encounters.

One way fragments are identified is by detecting fragment delimiters. A delimiter is restricted to one character; the default delimiters are U+0009 (tab), U+000B (vertical tab), and U+000C (form feed). To modify the set of recognized delimiters, pass a string containing all desired delimiter values to the fragmentBoundaryDelimiters option. The string must include any default values you want to keep. Non-whitespace delimiters within a token will be ignored.

The following rules determine where fragments are identified, in descending priority:

Fragments always include trailing whitespace.

Example:

BaseLinguisticsFactory factory = new BaseLinguisticsFactory();
factory.setOption(BaseLinguisticsOption.rootDirectory, rootDirectory)
factory.setOption(BaseLinguisticsOption.language, "eng");
EnumMap<BaseLinguisticsOption, String> options = Maps.newEnumMap(BaseLinguisticsOption.class);
options.put(BaseLinguisticsOption.fragmentBoundaryDelimiters, "|~");
options.put(BaseLinguisticsOption.maxTokensForShortLine, "5");
factory.createSingleLanguageAnnotator(options);

By default, fragment detection is enabled. Use the fragmentBoundaryDetection option to disable it.

Enum Classes:

RBL supports POS-tagging of emoji, emoticons, @mentions, email addresses, hashtags, and URLs in all supported languages.

Tokenization of emoji is always enabled. The other options are disabled by default but can be enabled through the options listed. When tokenization is disabled, the characters may be split into multiple tokens.

Enum Classes:

Sample input and output

Type	Input	Tokenization (when option is enabled)	Tokenization (when option is disabled)	POS tag
emoji			Tokenization of emoji cannot be disabled.	EMO
emoticon	:)	:)	: )	EMO
@mention	@basistechnology	@basistechnology	@ basistechnology	ATMENTION
hashtag	#basistechnology	#basistechnology	# basistechnology	HASHTAG
email address	info@basistech.com	info@basistech.com	info @ basistech.com	EMAIL
URL	http://www.babelstreet.com	http://www.babelstreet.com	http : / / www.babelstreet.com	URL

The tokenization when the option is disabled depend on the options language and tokenizerType. The samples provided here are for when language=eng and tokenizerType=ICU.

Emoji Normalization & Lemmatization

RBL normalizes emoji, placing the result into the lemma field. The simplest example is when an emoji presentation selector follows a character that is already an emoji. In this case, RBL will simply remove the emoji presentation selector.

Lemmatization applies to an emoji character in multiple ways.

Emoji that depict people or body parts may be followed by an emoji modifier indicating skin tone. Lemmatization simply removes the emoji modifier skin tone from the emoji character. The reasoning is that the skin tone is of secondary importance to the meaning of the emoji.

Surface form	Lemmatized form
( Boy + Medium Skin Tone)

Emoji depicting people may be followed by an emoji component indicating hair color or style. Lemmatizing removes the hair component from the emoji character.

Surface form	Lemmatized form
( Adult + ZWJ + Red Hair)

Where a gender symbol has been added to create a gendered occupation emoji, lemmatization removes the gender symbol.

Surface form	Lemmatized form
( Police Officer + ZWJ + ♀ Female Sign + VS16)

Finally, RBL can normalize non-fully-qualified emoji ZWJ sequences to fully-qualified emoji ZWJ sequences. In the above example, it is possible to omit the VS16 (though discouraged by Unicode): since Police Officer is an emoji, anything joined to it by a ZWJ is implicitly an emoji too. RBL adds the missing VS16.

The ICU tokenizer is the default tokenizer used for European languages. It works based on behavior defined in a rule file. If the default behavior is not exactly what is desired, RBL allows custom rule files to be supplied that will determine the behavior of the tokenizer. How to make these customizations is briefly outlined here. Be careful with any changes you make to the tokenizer behavior; BasisTech does not support customizations made by the user.

BaseLinguisticsFactory has a method addCustomTokenizerRules which can be used to specify a custom rule file. RBLCmd also has the -ctr option to specify a path on the command line. All of these methods accept a case-sensitivity value (for -ctr, cs and ci mean case-sensitive and case-insensitive), which is important because only when BaseLinguisticsOption.caseSensitive is the same as the value for a rule file will it be selected. Custom rule files are not cumulative, i.e. only one set of rules may be used at a time for any one combination of case sensitivity and language.

...
$Smiley = [\:=][)}\]];

!!forward;
$Smiley;
...

=)          

=
)       

=)       

RBL provides basic tokenization support when the language is "Unknown" (xxx). The tokenizer uses generic rules to tokenize, such as whitespace and punctuation delimitation.

Supported Features when language is unknown (xxx):

Using the language code of xxx will provide basic tokenization support for languages not supported by RBL.

For each token and normalized form in the token stream, the analyzer performs a dictionary lookup starting with any user dictionaries followed by the RBL dictionary. During lookup, RBL ignores the context in which the token or normalized form appears.

Once the analyzer has found one or more lemmas in a dictionary, it does not consult additional dictionaries. In other words, if two user dictionaries are specified, and the filter finds a lemma in the first dictionary, it does not consult the second user dictionary or the RBL dictionary.

Unless overridden by an analysis dictionary, the only lemmatization done in Chinese and Thai is number normalization. Other Chinese and Thai tokens' lemmas are equal to their surface forms.

There is no analysis dictionary available for Finnish, Pashto, or Urdu. All other languages are supported.

No dictionary can ever be complete: new words get added to languages, languages change and borrow. So, in general, analysis for each language includes some sort of guessing capability. The job of a guesser is to take a word and to come up with some analysis of it. Whatever facts we generate for a language, those are all possible outputs of a guesser.

In European languages, guessers deliver lemmas and parts of speech. In Korean, guessers provide morphemes, morpheme tags, compound components, and parts of speech.

By default, the analyzer returns any lemma that contains whitespace as multiple lemmas (each with no whitespace). To allow lemmas with whitespace (such as International Business Machines as a lemma for the token IBM) to be placed as such in the token stream, you can create a user analysis dictionary with an entry that defines the lemma. For example:

IBM	International[^_]Business[^_]Machines[+PROP]

The analyzer decomposes Chinese, Danish, Dutch, German, Hungarian, Japanese, Korean, Norwegian, and Swedish compounds, returning the lemmas of each of the components.

The lemmas may differ from their surface form in the compound, such that the concatenation of the components is not the same as the original compound (or its lemma). Components are often connected by elements that are present only in the compound form.

For example, the German compound Eingangstüren (entry doors) is made up of two components, Eingang (entry) and Tür (door), and the connecting 's' is not present in the component list. For this input token, the RBL tokenizer and analyzer return the following entries:

Other German examples include letter removal (Rennrad ⇒ rennen + Rad), vowel changes (Mängelliste ⇒ Mangel + Liste), and capitalization changes (Blaugrünalge ⇒ blau + grün + Alge).

Enum Classes:

For some languages, the analyzer can disambiguate between multiple analysis objects and return the disambiguated analysis object. The disambiguate option enables the disambiguator. When true, the disambiguator determines the best analysis for each word given the context in which it appears.

When using an annotator, the disambiguated result is at the head of all possible analyses. The remainder of the list is ordered randomly. When using a tokenizer/analyzer, use the method getSelectedAnalysis to return the disambiguated result.

For all languges except Japanese, disambiguation is enabled by default. For performance reasons, disambiguation is disabled by default for Japanese when using the statistical model.

Enum Classes:

In RBL, each language has its own set of POS tags and a few languages have multiple tag sets. Each tag set is identified by an identifier, which is a value of the TagSet enum. When RBL outputs a POS tag, it also lists the identifier for the tag set it came from. Output from a single language may contain POS tags from multiple tag sets, including the language-neutral set.

POS tags are defined for Arabic, Chinese, Czech, Dutch, English, French, German, Greek, Hebrew, Hungarian, Indonesian, Italian, Japanese, Korean, Malay (Standard), Persian, Polish, Portuguese, Russian, Spanish, Tagalog, and Urdu.Part-of-Speech Tags

-
  - NUM_VOC
  -
    - { m: \+Total, t: PRON }
    - { s: moc, t: DET }
    - { s: oba, t: DET }
    - { t: NUM }

Enum Classes:

The ignoreStopwords option uses a stop words list to define stop words. The path to the stop words list is language-dependent: Chinese uses root/dicts/zho/cla/zh_stop.utf8 and Japanese uses root/dicts/jpn/jla/JP_stop.utf8.

You can add stop words to these files. When you edit one of these files, you must follow these rules:

RBL includes multiple disambiguators for Hebrew. Set the value for the option disambiguatorType to select which type to use. The valid values for DisambiguatorType are:

Enum Classes:

Generic Arabic script normalization includes the following:

All static and dynamic user dictionaries, except for many-to-one normalization dictionaries, are consulted in reverse order of addition. In cases of conflicting information, dictionaries added later take priority over those added earlier. Once a token is found in a user dictionary, RBL stops and will consult neither the remaining user dictionaries nor the RBL dictionary.

Many-to-one normalization dictionaries are consulted in the following order:

Example of non-many-to-one user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, statDict3, statDict2, dynDict1

Example of many-to-one normalization user dictionary priority:

Dictionaries are prioritized in the following order:

dynDict4, dynDict1, statDict2, statDict3

The Chinese and Japanese language analyzers load all dictionaries with the user dictionaries loaded at the end of the list. To prioritize the user dictionaries and put them at the front of the list, guaranteeing the matches in the user dictionaries will be used, set the option favorUserDictionary to true.

The following formatting rules apply to user dictionary source files.

Once complete, the source file is compiled into a binary format for use in RBL.

norm1        var1        var2        var3

dictionary.add("norm1", "var1", "var2", "var3");

In the tools/bin directory, RBL includes a shell script for Unix

rbl-build-user-dictionary

and a .bat file for Windows.

rbl-build-user-dictionary.bat

To compile a user dictionary, from the RBL root directory:

tools/bin/rbl-build-user-dictionary -type TYPE_ARGUMENT LANG INPUT_FILE OUTPUT_FILE

where TYPE_ARGUMENT is the dictionary type, LANG is the language code, INPUT_FILE is the pathname of the source file you have created, and OUTPUT_FILE is the pathname of the binary compiled dictionary the tool creates. For example:

tools/bin/rbl-build-user-dictionary -type analysis jpn jpn_lemmadict.txt jpn_lemmadict.bin

The script uses Java to compile the user dictionary. The operation is performed in memory, so you may require more than the default heap size. You can set heap size with the JAVA_OPTS environment variable. For example, to provide 8 GB of heap size, set JAVA_OPTS to -Xmx8g.

Unix shell:

export JAVA_OPTS=-Xmx8g

Windows command prompt:

set JAVA_OPTS=-Xmx8g

The format for a segmentation dictionary source file is very simple. Each word is written on its own line, and that word is guaranteed to be segmented as a single token when seen in the input text, regardless of context. Japanese example:

三菱UFJ銀行
酸素ボンベ

Each entry is a word, followed by a tab and an analysis. The analysis must end with a lemma and a part-of-speech (POS) tag.Part-of-Speech Tags

word	lemma[+POS]

For those languages for which RBL does not return POS tags, use DUMMY.

Variations. You can provide more than one analysis for a word or more than one version of a word for an analysis.

The following example includes two analyses for "telephone" (noun and verb), and two renditions of "dog" for the same analysis (noun).

telephone	telephone[+NOUN]
telephone	telephone[+VI]
dog	dog[+NOUN]
Dog	dog[+NOUN]               

For some languages, the analysis may include special tags and additional information.

Contracted forms. For English, French, Italian, and Portuguese, ^= is a separator for a contraction or elision.

English example:

doesn't	does[^=]not[+VDPRES]

Multi-Word Analysis. For English, Italian, Spanish, and Dutch, ^_ indicates a space.

English example:

IBM	International[^_]Business[^_]Machines[+PROP]

Compound Boundary. For Danish, Dutch, Norwegian, German, and Swedish, ^# indicates the boundary between elements in a compound word. For Hungarian, the compound boundary tag is ^CB+.

German example:

heimatländern	Heimat[^#]Land[+NOUN]

Compound Linking Element. For German ^/, indicates a compound linking element. For Dutch, use ^//.

German example:

arbeitskreis	Arbeit[^/]s[^#]Kreis[+NOUN]

Derivation Boundary or Separator for Clitics. For Italian, Portuguese, and Spanish, ^| indicates a derivation boundary or separator for clitics.

Spanish example with derivation boundary:

duramente	duro[^|][+ADV]

Italian example with separator for clitics:

farti	fare[^|]tu[+VINF_CLIT]

Japanese Readings and Normalized Forms. For Japanese, [^r] precedes a reading (there may be more than one), and [^n] precedes a normalization. For example:

行わ	行う[^r]オコナワ[+V]
tv	テレビ[^r]テレビ[^n]テレビ[+NC]
アキュムレータ	アキュムレーター[^r]アキュムレータ[^n]アキュムレーター[+NC]

Korean Analysis. A Korean analysis uses a different pattern than the analysis for other languages. The pattern for an analysis in a user Korean dictionary is as follows:

Token	Mor1[/Tag1][^+]Mor2[/Tag2][^+]Mor3[/Tag3]

Where each MorN is a morpheme, consisting of one or more Korean characters, and TagN is the POS tag for that morpheme. [^+] indicates the boundary between morphemes.

Here's an example:

유전자이다	유전자[/NPR][^+]이[/CO][^+]다[/ECS]

If the analysis contains one noun morpheme, that morpheme is the lemma and the POS tag is the POS tag for that morpheme. If more than one of the morphemes are nouns, the lemma is the concatenation of those nouns (a compound). Example:

정보검색	정보[/NNC][^+]검색/[NNC]

Otherwise, the lemma is the first morpheme, and the POS tag is the POS tag associated with that morpheme.

You can override this algorithm for identifying the lemma and/or POS tag in a user dictionary entry by placing [^L]lemma and/or [^P][/Tag] at the end of the analysis. The lemma may or may not correspond to one of the morphemes in the analysis. For example:

유전자이다	유전자[/NNC][^+]이[/CO][^+]다[/ECS][^L]유전[^P][/NPR]

The KoreanAnalysis interface provides access to the morphemes and tags associated with a given token in either the standard Korean dictionary or a user Korean dictionary.

A many-to-one normalization dictionary maps one or more variants to a normalized form. The first value on each line is the normalized form. The remainder of the entries on the line are the variants to be mapped to the normalized form. All values on the line are separated by tabs.

Example:

Use the option normalizationDictionaryPaths to specify the static user normalization dictionaries.

The source file for a Chinese or Japanese user dictionary is UTF-8 encoded (see Valid Characters for Chinese or Japanese User Dictionary Entries). The file may begin with a byte order mark (BOM). Empty lines are ignored. A comment line begins with #. The first line of a Japanese dictionary may begin !DICT_LABEL followed by Tab and an arbitrary string to set the dictionary's name, which is not currently used anywhere.

Each entry in the dictionary source file is a single line:

word Tab POS Tab DecompPattern Tab Reading1,Reading2,...

where word is the entry or surface form, POS is one of the user-dictionary part-of-speech tags listed below, DecompPattern is the decomposition pattern: a comma-delimited list of numbers that specify the number of characters from word to include in each component of the compound (0 for no decomposition), and Reading1,... is a comma-delimited list of one or more transcriptions rendered in Hiragana or Katakana (only applicable to Japanese).

The decomposition pattern and readings are optional, but you must include a decomposition pattern if you include readings. In other words, you must include all elements to include the entry in a reading user dictionary, even though the reading user dictionary does not use the POS tag or decomposition pattern. To include an entry in a segmentation (tokenization) user dictionary, you only need POS tag and an optional decomposition pattern. Keep in mind that those entries that include all elements can be included in both a segmentation (tokenization) user dictionary and a reading user dictionary.

For more information on Chinese and Japanese POS tags, refer to the following sections:

The tables below list the primary and secondary tags emitted when the user POS tag is specified.

Examples (the last three entries include readings):

!DICT_LABEL New Words 2014
デジタルカメラ NOUN
デジカメ NOUN 0
東京証券取引所 ORGANIZATION 2,2,3
狩野 SURNAME 0
安倍晋三 PERSON 2,2 あべしんぞう
麻垣康三 PERSON 2,2 あさがきこうぞう
商人 NOUN 0 しょうにん,あきんど
        

The POS and decomposition pattern can be in full-width numerals and Roman letters. For example:

東京証券取引所 ｏｒｇａｎｉｚａｔｉｏｎ ２,２,３

The "2,2,3" decomposition pattern instructs the tokenizer to decompose this compound entry into

東京
証券
取引所
        

There are two standard forms of written Chinese: Simplified Chinese (SC) and Traditional Chinese (TC). SC is used in Mainland China. TC is used in Taiwan, Hong Kong, and Macau.

Conversion from one script to another is a complex matter. The main problem of SC to TC conversion is that the mapping is one-to-many. For example, the simplified form 发 maps to either of the traditional forms 發 or 髮. Conversion must also deal with vocabulary differences and context-dependence.

The Chinese Script Converter converts text in simplified script to text in traditional script, or vice versa. The conversion can be on any of three levels, listed here from simplest to the most complex:

The Chinese input may contain a mixture of TC and SC, and even some non-Chinese text. The Chinese Script Converter converts to the target (SC or TC), leaving any tokens already in the target form and any non-Chinese text unchanged.

See Initial and Path Options for additional options

Enum Classes:

This example uses the BaseLinguisticsFactory and CSCAnalyzer.

The RBL distribution includes a sample (CSCAnalyze) that you can compile and run with an ant build script.

In a Bash shell script (Unix) or Command Prompt (Windows), navigate to rbl-je-<version>/samples/csc-analyze and call

ant run

The sample reads an input file in SC and prints each token with its TC conversion to standard out.

This example tokenizes Chinese test and converts from TC to SC.

CSC user dictionaries support orthographic and lexemic conversions between Simplified Chinese and Traditional Chinese. They are not used for codepoint conversion.

CSC user dictionaries follow the same format as other user dictionaries:

Once complete, the source file is compiled into a binary format for use in RBL.

Each entry contains two or three tab-delimited elements:

input_token orthographic_translation [lexemic_translation]

The input_token is the form you are converting from and the orthographic_translation and optional lexemic_translation are the form you are converting to.

Sample entries for a TC to SC user dictionary:

電腦	电脑	计算机
宇宙飛船	宇宙飞船
            

Compiling a CSC User Dictionary. In the tools/bin directory, RBL includes a shell script for Unix

rbl-build-csc-dictionary

and a .bat file for Windows

rbl-buld-csc-dictionary.bat

The script uses Java to compile the user dictionary. The operation is performed in memory, so you may require more than the default heap size. You can set heap size with the JAVA_OPTS environment variable. For example, to provide 8 GB of heap size, set JAVA_OPTS to -Xmx8g.

Unix shell:

export JAVA_OPTS=-Xmx8g

Windows command prompt:

set JAVA_OPTS=-Xmx8g

Compile the CSC user dictionary from the RBL root directory:

tools/bin/rbl-build-csc-dictionary INPUT_FILE OUTPUT_FILE

INPUT_FILE is the pathname of the source file you have created, and OUTPUT_FILE is the pathname of the binary compiled dictionary the tool creates. For example:

tools/bin/rbl-build-csc-dictionary my_tc2sc.txt my_tc2sc.bin

RBL canonicalizes some language codes to other language codes. These canonicalization rules apply to all APIs.

LanguageCode.NORWEGIAN is canonicalized to LanguageCode.NORWEGIAN_BOKMAL immediately upon any input to the API. This means that there can be no distinguishing between them. In particular, an Analyzer built from a factory configured to use Norwegian will report its language as Norwegian Bokmål instead.

Similarly, LanguageCode.SIMPLIFIED_CHINESE and LanguageCode.TRADITIONAL_CHINESE are canonicalized to LanguageCode.CHINESE immediately. The one exception is that they are not canonicalized as inputs to or outputs from the Chinese Script Converter.

Those are the only language code canonicalizations. Although RBL internally treats Afghan Persian and Iranian Persian as Persian, they are not considered the same language. This makes it possible to configure different user dictionaries for each variety of Persian, even though they are otherwise processed identically.

RBL uses ISO 639-3 language codes to specify languages as strings. There are a few nonstandard language codes, as indicated. RBL also accepts 2-letter codes specified by the ISO-639-1 standard. See the Javadoc for more details on language codes.