Release Notes

In Japanese, a middle dot comes in the middle of Western names and acts as a sort of whitespace separating words in the name. Previously, some of the entities with middle dot have been split into two entities, extracting only a part of the name. This is now handled correctly, and entities with middle dot are not split. (TEJ-1341)

In Japanese entity linking, in some cases the last character of the Japanese word changed into a Chinese character. This is now fixed. (ELK-118)

Previously, when includeDbPediaTypes option was off, entity linking occassionally extracted an inaccurate entity type. Now, fine types of linked entities are identified also when includeDbPediaTypes option is off. (ELK-115)

Provided a distribution package per language. (TEJ-1361)

Reduced linker data package size (TEJ-1306, TEJ-1321)

Updated the linking confidence calculation and thresholds to improve accuracy (TEJ-1343)

Improved the accuracy of Korean extraction, largely through better handling of Josa (postpositions) and compound words.

Added support for Entity linking to Wikipedia for both the top level types (PERSON, LOCATION, ORGANIZATION, ETC.) as well as the over 700 DBpedia types in the remaining 16 languages supported by Entity Extraction. This is in addition to the languages currently supported by entity linking: Chinese, English, Japanese, and Spanish.

The linker process now has the option of returning over 700 new entity types drawn from the DBpedia ontology. To access these entity types, turn on the kbLinker processor and add the includeDBpediaType flag to the factory configuration. You’ll notice more than 10 additional primary types in the type field as well as the all new DBpedia type field. Note that this is a LABS (experimental) api and subject to change. Send us your feedback!

New language: Entity extraction now supports Hungarian.

Replaced Japanese tokenizer to improve accuracy (TEJ-1176, TEJ-1180)

Enabled string normalization for Hebrew based on DNN disambiguation model to improve indoc-coref results (chaining mention`s into a single `Entity) and to present a more proper form of the name (TEJ-1139, TEJ-1173)

Social-media characters such as '@' and '#' are removed from Mention`s normalized string, offsets to the original string `data field remain the same. This feature can be disabled by EntityExtractor.setRetainSocialMediaSymbols() (TEJ-418)

Improved statistical model confidence score to emit maximal confidence less frequently (TEJ-1146)

Added static and dynamic capabilities to adding entries to the custom knowledge base for entity linking (ELK-30, ELK-41, ELK-44, TEJ-1150)

Added a new deep neural network processor (BETA) as an alternative entity extraction processor, which can be used in place of the standard statistical extractor for English, Arabic and Korean (TEJ-1132, TEJ-1142, TEJ-1150)

Added support for entity type "Title" in Hebrew (APE-1641)

Added new option MaxResolvedEntities to REXCmd (TEJ-1137)

Accuracy of Korean statistical model is improved (APE-1737)

Default linking confidence thresholds are set (TEJ-1080, TEJ-1068)

The method setUseDeepNeuralNetworkProcessor() in com.basistech.rosette.rex.EntityExtractor is part of a new experimental API to replace the statistical model by new deep learning model. Another option to use it is to provide ProcessorType.deepNeuralNetwork for the method setProcessors. Currently available only for English and Arabic. Some operating systems do not support the deep neural network model, and some do not provide good latency.

FTK supports training a disambiguation model for custom knowledge base (ELK-13, ELK-14, ELK-16, ELK-22, ELK-34)

Added default linking confidence threshold for linked entities (TEJ-1068)

Updated RBL version (TEJ-1048)

Application developer’s guide and customization guide are merged into a single manual (ELK-22)

Added confidence score for linking results, named linkingConfidence, in addition to statistical model confidence score. Different thresholds apply for the different confidence scores. (TEJ-974)

The new salience classifier has been incorporated. The salience calculation is enabled via EntityExtractor.setCalculateSalience() or by setting calculateSalience in either REXFactoryConfiguration or REXAnnotatorConfiguration. (TEJ-936)

Added manual custom processor registration API. (TEJ-972, TEJ-982)

Deduped partial duplicated regex. (TEJ-785)

Improved multiple gazetteer. (exact-match) processor support (TEJ-960, TEJ-1005)

Confidence score calculation is improved to correlate well with precision, may be used for thresholding and removal of false positives (TEJ-910, TEJ-919)

Statistical models are trained with new emoticon-sensitive tokenizer (TEJ-924)

New script allows repacking REX with minimal configuration per language (TEJ-893)

Automatic case sensitivity mode prefers case-sensitive for short text by default (TEJ-931)

Added Custom Processor for rejection. (TEJ-840, TEJ-841, TEJ-843, TEJ-880)

Redactor improvements: dynamic rules prioritization and subtypes handling (TEJ-863, TEJ-858)

Pronominal resolver is fully supported for English. Added as a processor type, as well as indoc-coref (TEJ-867)

Indoc-coref allows partial match for ORGANIZATION type (INDOC-26)

Added full support for Vietnamese. (APE-1691)

Added an example of how to use REX over Hadoop DFS. The README illustrating how to use this example can be found at ./samples/MapReduceExample/README.md. (TEJ-807)

The method setResolvePronouns() in com.basistech.rosette.rex.EntityExtractor is part of a new experimental API to resolve pronouns like 'he' and 'she' to entities of type Person. It may be changed or removed in future releases. Available only in English. (TEJ-831)

Reject regex and gazetteers allow wildcard entity type. (TEJ-853, TEJ-817)

The kb-linker experimental processor now supports Chienese and Japanese in addition to English. This functionality may be changed or removed in future minor releases. (TEJ-857)

Automatic case sensitivity improved (English only). (TEJ-861)

Added an example of how to use REX over Hadoop DFS. The README illustrating how to use this example can be found at ./samples/SparkEntityCount/README.md. (TEJ-155)

Information about what languages are licensed can now be accessed. The methods getLanguageInformation() and getSupportedEntityTypes() in com.basistech.rosette.rex.EntityExtractor now take in a flag for whether or not to return information on all languages REX supports, or just those that are licensed. Additionally, REXCmd info now has a -onlyLicensed option. (TEJ-767)

Added partial support for Vietnamese to extract phone numbers and dates using regexes. (TEJ-740)

REX annotators are now created faster and can feasibly be created on a document level when using the new com.basistech.rosette.rex.REXAnnotatorFactory API. The current com.basistech.rosette.rex.EntityExtractor API now also has a faster startup time. See the javadocs, the API Overview section in the Application Developer’s guide, and the sample program at ./samples/EntityAnnotatorFactorySample.java for additional details. (TEJ-773)

REX now reports its results using two new classes, Entity and Mention, such that each Entity in a document has one or more Mentions that refer to the same real-world identity. Moving forward, this API will replace EntityMention and its coreferenceChainId. This version of REX is backwards-compatible and still supports the deprecated EntityMention. (TEJ-702)

Improved Malaysian statistical model and added a new Malaysian gazetteer. (TEJ-711, TEJ-715)

kbLinker (flinx) is part of a new experimental API to to link entities from social media text to knowledge bases and may be changed or removed in future minor releases. (TEJ-722, TEJ-725, TEJ-757)

REX has been upgraded to Rosette Platform compatibility level 58.2. If you intend to use more than one Rosette JVM SDK in a single application, then you should choose versions that have the same compatibility number. (TEJ-702)

Improved case-insensitivity detection in European languages. (TEJ-687)

Entity mentions extracted with the statistical model now also specify the model’s path as a subsource. (TEJ-724)

Added a new method, EntityExtractor setOverlayDataDirectory(Path overlayDataDirectory), that allows you to specify an additional data directory for REX to use. (TEJ-731)

The REXCmd command line utility now allows you to specify any additional regex files you want to use, besides just the default. (TEJ-628)

Added full support for Standard Malay to extract standard entities using regexes, gazetteers, and statistical processors. (TEJ-704)

Added support for extraction using two statistical models operating in tandem. (TEJ-674)

In order to reduce disk footprint, Big Endian binaries are no longer shipped. REX will correctly memory map Little Endian models and dictionaries even on Big Endian systems. (TEJ-664)

Optional new packaging: RBL and REX classes are available in one combined jar. (TEJ-692)

This is a maintenance release to address SUPPO-569.

Added new customization for statistical processors with unsupervised field training. See Section 4.5 of the Application Developer Guide.

Added partial support for Malaysian. (TEJ-626)

Added a new setting for REX to automatically choose the most accurate CaseSensitivity model (case-insensitive or case-sensitive) for the input text. This is not activated by default, see the sample programs or javadocs for reference on how to enable this feature. (TEJ-568)

Added case-insensitive models for German, Italian, Dutch, and Spanish. (TEJ-566)

REX is now built with JDK 1.7, so users can no longer run REX on Java Virtual Machines versioned 1.6 and earlier. (TEJ-551)

Improved accuracy of the English statistical model by using multiple Brown clusters. (TEJ-396, TEJ-559)

New disableStatisticalCleaner option added to REXCmd and EntityExtractor. (TEJ-379)

You can now reactivate regular expression-based entities that are disabled by default by instructing REX to load the regex files in each language’s supplemental directory. See the Javadoc for EntityExtractor.addRegularExpressions(). (TEJ-587)

Refined redaction rules for PERSON entities. (TEJ-115)

EntityMentions and the returned fields are now documented in the Application Developer Guide. (TEJ-623)

Added a boolean caseSensitive parameter to the EntityExtractor’s `addGazetteer and addGazetteerEntity method, to allow case-insensitive string matching of user-provided textual gazetteer entries. (TEJ-56)

A RosetteUnsupportedLanguageException is now thrown when REX cannot find data for the requested language, instead of a generic runtime exception. (TEJ-536)

Adding a duplicate gazetteer entry will overwrite the existing one. (TEJ-74)

Support for script-insensitive Chinese added: Entities are now extracted from Chinese input documents for which the 'Simplified' or 'Traditional' writing system is not specified. Applications may now submit text using the zho language code instead of specifying zhs or zht. (TEJ-525)

Version 7.14.0.c56.6 introduced the use of the "compatibility" version number extension (c56.6 in this case). If you intend to use more than one Basis Technology JVM SDK in a single application, then choose versions that have the same compatibility number. (TEJ-524)

Indonesian (Bahasa Indonesia) support added. (TEJ-441)

To enhance performance, and in response to customer feedback, we deactivated the regular expressions for extracting the following entity types: IDENTIFIER:DISTANCE, IDENTIFIER:LATITUDE_LONGITUDE, IDENTIFIER:UTM, TEMPORAL:DATE, and TEMPORAL:TIME. You can restore support for any of these entity types by removing the @ignore=rex-je attribute value that appears in front of the relevant regular expressions in the regexes.xml files. (TEJ-510)

Improved speed and reduced memory consumption of regex and gazetteer matching. (TEJ-489)

Improved the accuracy of the statistical models for case-insensitive Portuguese and French. (TEJ-475)

Added the ability to modify the indoc shut-off threshold: setMaxResolvedEntities(). (TEJ-456)

Introduced an experimental EntityExtractor API for excluding entity types in which you are not interested. See the Javadoc for {get,set}ExcludedEntityTypesfor details. Note that this is an experimental API which can change or be removed in future minor versions of REX. (TEJ-480)

REX emits an exception when asked to extract in a language it has no data for. (TEJ-447)

Missing values for confidence and coreferenceChainId are now represented as nulls instead of -1. (TEJ-466, TEJ-467)

Improved Korean Entity Extraction and In-Document Co-reference Resolution

REX uses a new statistical model that achieves higher accuracy (about 25% overall error reduction). (APE-1111)

In-document co-reference resolution now recognizes Korean prefixes and suffixes, and will attempt to chain morphological variations of Korean entity mentions. (TEJ-366)

Added features to the REXCmd utility

Plaintext output now pretty-prints the chain ID for each entity mention it returns. Mentions with the same chain ID refer to the same entity. (TEJ-410)

The -context option marks the entities in their original text context, with embedded entity type and chain ID. (TEJ-410)

REXCmd now supports pre-annotated input in json-serialized Annotated Text format. (TEJ-455)

Modified the reporting of text offsets for partial regular-expression and gazetteer matches to align with the token boundaries of the tokens that contain the matched text. (TEJ-393)

The REX Tcl implementation of regular expressions now supports characters in the Supplementary Multilingual Plane (SMP) (TEJ-454). Previous releases represented SMP codepoints as two characters each. (TEJ-330)

Provided an EntityExtractor option to instruct the REX statistical processor to ignore the lowercase/uppercase distinction. This feature is currently supported for English, French, and Portuguese. (TEJ-458)

Added the EntityExtractor.createDispatchAnnotator method to allow annotating documents from a predefined set of languages. (TEJ-432)

Added Portuguese regular expressions for temporal and monetary expressions. (TEJ-444)

Added the Fragment Boundary Detector to enable the extractor to separate entities in text fragments that do not form sentences. (TEJ-117)

Refined the usage pattern for the com.basistech.rosette.rex.REXCmd command-line utility. The JSON output this utility generates has been trimmed to represent the serialization of an AnnotatedText object. (TEJ-374)

Deprecated com.basistech.rosette.rex.EntityCursor. Use com.basistech.rosette.rex.EntityExtractor and com.basistech.rosette.dm.Annotator to extract entities from an input document.

Re-established pattern matcher support for the regular expressions disabled in 2.1.0.

Added a document-level API for extracting entities. Over time, we expect to deprecate EntityCursor (streaming) in favor of EntityExtractor and Annotator (document-level extraction).

Added an API (the EntityExtractor setPostConfidence method) for extracting a confidence floating point value for each entity that REX Java Edition finds. A potential entity is ignored if its confidence score is below the threshold set with the setConfidenceThreshold method. (TEJ-60)

REX Java Edition returns normalized entities from all sources: statistical, pattern matching, and exact matches (gazetteers). (TEJ-288)

We have disabled the use of pattern matcher regular expressions that do not terminate properly, consuming large amounts of CPU time. This change could cause REX to miss some temporal, distance and long/lat expressions. If your use case requires high recall on these numeric types, please contact analyticssupport@babelstreet.com for assistance on enabling these regular expressions.(TEJ-283)

The REXCmd command-line utility now includes the REX Java Edition version number in the output. (TEJ-261)

Added a shell script (Unix) and .bat file (Windows) to simplify the running of the RexCmd command-line utility. (TEJ-247)

The JSON document generated by the command-line utility is much more verbose than in previous releases. Accordingly, if you are not writing the output to a file, you may want to pipe the output through a JSON parser (e.g., | python -mjson.tool) and concentrate on the EntityMention elements. (TEJ-260)

Added support for Arabic, Simplified Chinese, Traditional Chinese, Korean, and Japanese.

Support for using the REX Field Training Kit to enhance accuracy handling a particular category of documents and to return new entity types. (TEJ-220)

Added support for Dutch, Hebrew, Persian (Western Farsi and Dari), Portuguese, Pashto, and Urdu.

Incorporated improvements to the statistical language models, gazetteers, and regular expressions introduced in the Rosette C++ REX implementation since the release of REX Java Edition 1.1.

Enhanced the command-line utility with new options. (TEJ-212)

Added support for resetting the maximum number of tokens that an entity may include, which defaults to 8. Use the EntityExtractor setMaxEntityTokens(int) method. (TEJ-188)

Added support for Uppercase English, French, German, Italian, Russian, and Spanish.

Added support for resolving coreferences to the same entity. Use EntityExtractor setResolveNamedEntities(true) to put coreferences to the same entity in an entity chain: see EntityCursor getChainId(). (TEJ-52)

In response to customer feedback, removed IDENTIFIER:NUMBER from the default set of entity types returned by regular expressions. We commented out the IDENTIFIER:NUMBER entries in the regexes.xml files in data/regex/lang/accept, so you can re-activate any of these entries if you wish. (TEJ-171)

Added a public EntityCursor hasNext() method that can be used to determine whether there are any more entities in the result set, without advancing to the next entity. (TEJ-150)

Added the following EntityExtractor methods:

TEJ-1349 Fixed linker head mention bug

TEJ-1353 Fixed missing entity types in linker results

TEJ-1281 Set log level to Debug instead of Warn when linkEntities and genre don’t agree

TEJ-1319, TEJ-1346, ELK-114 Picked up new TVEC to improve the efficiency of the initial load time

TEJ-1324 Fixed a bug where the salience score was not always returned for entities with pronominal mentions, when requested.

TEJ-1327 Consumed new RBL to fix null pointer exception with pronoun resolver

TEJ-1331 Removed xxx from reported supported languages

APE-1766 Fixed a bug where all entity mentions were not always returned by statistical model

TEJ-1282 Relocated LIBLINEAR and TVEC, tested with rli

TEJ-1283 Fixed cases in which QID was not returned although DBPedia result was available, when using REX’s Kblinker

TEJ-1292 Custom processors made configurable via REXFactoryConfiguration

ELK-82 Fixed an acronym feature issue.

TEJ-1160 Moved ORGANIZATION eng regexes to supplemental directory to improve performance

TEJ-1167 Improved failure message for missing flinx data directory

TEJ-1168 Fixed null point exception with long chains

TEJ-1099 Added additional currency symbols to regex

TEJ-1108 Fixed hexadecimal number string incorrectly extracted as product

TEJ-1067 Fixed calculateConfidence configuration bug

TEJ-1020 Added regex for Israeli ID number

TEJ-1054 Cleaned MD5 codes extracted as PRODUCT entity type

TEJ-1049 Fixed case-insensitive text file gazetteer treated as case-sensitive

TEJ-1050 Fixed annotator configuration object being modified while creating.

TEJ-1041 Improved redactor rules for extractor and linking overlaps and indoc-coref to prefer chaining based on linking ID.

TEJ-1039 Enabled emoticon mode for RBL for RosAPI

TEJ-1042 Fixed error handling for requesting salience score while indoc-coref is disabled

TEJ-950 Improved custom processor examples.

TEJ-951 Apply American SSN regex for English only.

TEJ-962 Fixed personal ID entity type in Vietnamese regex.

TEJ-1005 Fixed source and subsource for static user gazetteer.

TEJ-1006 Improved handling of the case Arabic prefix is null.

TEJ-1014 Fixed shading of dependencies.

TEJ-971 Fixed a concurrency issue in kb-linker.

TEJ-977 Fixed to return null for confidence from non-statistical entities.

TEJ-952, TEJ-963 Fixed lookup issue in wordclasses

TEJ-693 setStatisticalModel without caseSensitivity assumes case-sensitive

TEJ-813 Relocated more RBL classes in REX to prevent version conflicts when using RBL in conjunction with REX.

TEJ-477 REXCmd now emits a usage error when 'info' is used with no info command.

TEJ-755 Relocated RBL classes in REX to prevent version conflicts when using RBL in conjunction with REX.

TEJ-735, TEJ-738 Fixed a bug in which Entity Type Survey didn’t support non-default field training models or customer-defined types.

TEJ-747 Fixed a bug in which REXCmd produced unexpected offsets for files with DOS-style line endings.

TEJ-718 Added Malaysian sample text.

TEJ-717 Minor update to zsm model in 'single' distribution form.

TEJ-694 Added a missing file resource release in addGazetteer.

TEJ-676, TEJ-691 Shaded and relocated all 3rd party dependencies.

TEJ-683 Removed extraneous pom.xml’s from META-INF in distro jar.

SUPPO-569 Fixed incompatibility issues with RBL.

TEJ-659 Fixed support for custom tags in statistical model parser.

APE-1608 Fixed REXCmd’s behavior with case-insensitive models and the -model command line option.

SUPPO-536 Fixed partial regex extraction for Korean.

TEJ-573 REXCmd now ignores a BOM at the beginning of its input file.

TEJ-574 Lookbehind assertions are not supported, and this is now included in the Application Developer Guide.

TEJ-512 Regex xml parser misses some CDATA sections (URL extraction issues)

TEJ-459, TEJ-507 indoc chaining applied only to PER/LOC/ORG

TEJ-498 Application Developer Guide claims that REX extracts full postal addresses

TEJ-486 EntityAnnotator reuse problems

TEJ-479 RBL dictionary markers in distro

TEJ-460 Partial match regex produces wrong offsets

TEJ-438 ICU dependency not relocated/shaded

TEJ-415 REX requires an RBL license