Babel Street Match
Match
Explainable AI to accurately fuzzy match names (of people and organizations), dates and addresses against vast databases.
Documentation and Resources
Application Developer's Guide - Babel Street Match
Application Developer's Guide - Babel Street Match for Elasticsearch
Application Developer's Guide - Babel Street Match for OpenSearch
Match Evaluation and Configuration Guide
Try the demo: demo.babelstreet.com/match-demo
Try Analytics Cloud: developer.babelstreet.com
Key features
Cross-lingual name matching
Intuitive match confidence score from 0 to 1 to enable automation of workflow
Explainable AI provides explanation of match score calculations
Intelligent address and date matching
Flexible and extensive configurations to optimize accuracy on your data
See impact of configuration changes in real-time through administrative interface, Match Studio
Supports matching of just a name or a name plus any number of identifying attributes (address, date of birth, ID number, etc.)
Languages
Match matches names within the following languages and scripts.
This table identifies the range of cross-language matching that Match fully supports.
Name variations
These are some of the many name variations that Name Match considers in every name comparison.
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Elasticsearch OpenSearch Solr
Visit our GitHub pages for bindings and documentation.
{
"translation": "Mu'ammar Muhammad Abu-Minyar al-Qadhaf",
"targetLanguage": "eng",
"targetScript": "Latn",
"targetScheme": "IC",
"confidence": 0.06856099342585828
}Name Translator
Translates names consistently according to transliteration standards using knowledge of language-specific name conventions.
Documentation and Resources
Application Developer's Guide - Babel Street Match
Try the demo: demo.babelstreet.com/match-demo
Try Analytics Cloud: developer.babelstreet.com
Key features
Transliterates names consistently according to user-selected transliteration standards.
Recognizes language of origin of names to produce “conventional spellings” of well-known names when possible (Example, from the Arabic جورج دبليو بوش translating to “George Bush” instead of transliterating to “Jurj Bush”).
Recognizes when to translate words (such as titles) instead of transliterating.
Arabic translation sample
Example of Arabic name translation from Name Translator:
Transliteration type | Input | Output |
|---|---|---|
Person name – Arabic-origin | ابو يوسف يعقوب | Abu-Yusif Ya’qub |
Person name – English-origin | رذرفورد بي هايز | Rutherford B. Hayes |
Place name – Arabic-origin | باقة الشرقية | Baqah al-Sharqiyyah |
Organization Acronym – English-origin | بي بي سي | B.B.C. |
Scripts and transliterations
Rosette Name Translator translates names between these writing systems and transliteration standards.
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Elasticsearch Solr
{
"name1": {
"text": "Влади́мир Влади́мирович Пу́тин",
"language": "rus",
"entityType": "PERSON"
},
"name2": {
"text": "Vladimir Putin",
"language": "eng",
"entityType": "PERSON"
}
}
{
"result": {
"score": 0.9486632809417912
}
}Match Studio
Match Studio is a user-friendly administrative interface to help the Name Match user try out different configurations and see the impact on match behavior and scores in real-time.
Key features
Analyze precision and recall to assist user in choosing a match threshold; the score above which two names are considered “a match”
User-friendly configuration testing in a GUI environment
Compare match results using differently weighted fields when comparing records
Languages
Match Studio matches names within the following languages and scripts.
This table identifies the range of cross-language matching that Match fully supports.
Windows macOS Docker Linux
Text Analytics
Entity Extractor
Documentation and Resources
Try the demo: demo.babelstreet.com/text-analytics-demo
Try Analytics Cloud: developer.babelstreet.com
Fast, accurate named entity recognition using an ensemble of algorithms – statistical models and deep learning models (AI), pattern matching (regular expressions), and entity lists (gazetteers) for high accuracy across languages. Internally, an adjudication module scores and remediates conflicting results between the different processors.
Linking to knowledge bases enables Babel Street Analytics to distinguish between similarly named entities, such as Neil Armstrong (astronaut) and Neil Armstrong (hockey referee).
Key features
Entity extraction of people, location, organization and more entity types.
Entity linking to Wikidata by default, but customizable to other knowledge bases.
Coreference resolution – linking entities and their pronoun mentions.
Salience scoring – highlighting entities most relevant to document content.
Languages
Entity Extractor supports the languages listed below.
Entity types
Entity Extractor is pre-trained to extract the following entity types:
Location
Organization
Person
Title
Nationality
Religion
Credit Card
Distance
Email
Latitude/Longitude
Money
Currency
ID number
Phone number
URL
UTM
Date
Time
In addition to the entity types above, Analytics recognizes over 450 sub-entity types and will link to a WikiData QID and DBpedia parse tree when it is available. As an example: “Ibuprofen” will be tagged as “SUBSTANCE”, linked to the WikiData ID: Q186969, and assigned the DBpedia tree ”ChemicalSubstance/Drug”.
Linked knowledge bases
By default, Entity Extractor is pre-trained to link entities to entries in these knowledge bases, which enables it to distinguish between similarly named entities by examining the context in which the entity appears. Users can also specify their own knowledge bases for linking.
WikiData
DBpedia
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Elasticsearch Solr
{
"entities": [
{
"type": "ORGANIZATION",
"mention": "Securities and Exchange Commission",
"normalized": "Securities and Exchange Commission",
"count": 3,
"mentionOffsets": [
{
"startOffset": 4,
"endOffset": 38
},
{
"startOffset": 166,
"endOffset": 169
},
{
"startOffset": 536,
"endOffset": 539
}
],
"entityId": "Q953944",
"confidence": 0.67070782,
"linkingConfidence": 0.27190905,
"dbpediaType": "Agent/Organisation/GovernmentAgency"
},
{
"type": "PERSON",
"mention": "Bridget Fitzpatrick",
"normalized": "Bridget Fitzpatrick",
"count": 2,
"mentionOffsets": [
{
"startOffset": 99,
"endOffset": 118
},
{
"startOffset": 287,
"endOffset": 298
}
],
"entityId": "T1",
"confidence": 0.92063326
},
{
"type": "PERSON",
"mention": "David Gottesman",
"normalized": "David Gottesman",
"count": 2,
"mentionOffsets": [
{
"startOffset": 174,
"endOffset": 189
},
{
"startOffset": 307,
"endOffset": 316
}
],
"entityId": "Q5234268",
"confidence": 0.92488831,
"linkingConfidence": 0.47211223,
"dbpediaType": "Agent/Person"
},
{
"type": "TITLE",
"mention": "Chief Litigation Counsel",
"normalized": "Chief Litigation Counsel",
"count": 1,
"mentionOffsets": [
{
"startOffset": 134,
"endOffset": 158
}
],
"entityId": "T2",
"confidence": 0.3306601
},
{
"type": "TITLE",
"mention": "Deputy Chief Litigation Counsel",
"normalized": "Deputy Chief Litigation Counsel",
"count": 1,
"mentionOffsets": [
{
"startOffset": 229,
"endOffset": 260
}
],
"entityId": "T5",
"confidence": 0.81287289
},
{
"type": "TEMPORAL:DATE",
"mention": "December 2016",
"normalized": "December 2016",
"count": 1,
"mentionOffsets": [
{
"startOffset": 268,
"endOffset": 281
}
],
"entityId": "T6"
},
{
"type": "TITLE",
"mention": "Ms.",
"normalized": "Ms.",
"count": 1,
"mentionOffsets": [
{
"startOffset": 283,
"endOffset": 286
}
],
"entityId": "T7",
"confidence": 0.76600134
},
{
"type": "TITLE",
"mention": "Mr.",
"normalized": "Mr.",
"count": 1,
"mentionOffsets": [
{
"startOffset": 303,
"endOffset": 306
}
],
"entityId": "T9",
"confidence": 0.72353458
},
{
"type": "TITLE",
"mention": "Co-Acting Chief Litigation Counsel",
"normalized": "Co-Acting Chief Litigation Counsel",
"count": 1,
"mentionOffsets": [
{
"startOffset": 332,
"endOffset": 366
}
],
"entityId": "T11",
"confidence": 0.03582656
},
{
"type": "LOCATION",
"mention": "Washington D.C.",
"normalized": "Washington D.C.",
"count": 1,
"mentionOffsets": [
{
"startOffset": 460,
"endOffset": 475
}
],
"entityId": "Q61",
"linkingConfidence": 0.66086622,
"dbpediaType": "Place/PopulatedPlace/Settlement"
}
]
}Event Extractor
Effectively triage the overflow of data to get alerts and key details about critical events -- specific to your use case -- when you need them, even from data in languages that your analysts don’t know.
Documentation and Resources
Try Analytics Cloud: developer.babelstreet.com
Key features
Quick training of AI models for events specific to you -- in less than a day with Model Training Suite (MTS).
Extraction of the who, what, when and where for each event – not just the source text.
True context understanding, more accurate than keyword alerting.
Languages
Event Extractor supports event extraction models for the following languages. Model Training Suite enables users to train models for events they require in as little as a day.
Arabic
Chinese
Dutch
English
German
Hungarian
Japanese
Korean
Russian
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Request
{"content": "I want flights from Boston to New York",
"language": "eng",
"options": {
"workspaceId": "multi-1"
}Response
{
"events": [
{
"eventType": "flight_booking_schema.flight_booking",
"mentions": [
{
"startOffset": 7,
"endOffset": 38,
"roles": [
{
"startOffset": 7,
"endOffset": 14,
"name": "key",
"id": "E1",
"dataSpan": "flights",
"obsolete": false,
"roleType": "flight_booking_schema.flight_booking_key",
"extractorName": "flight_booking_schema.flight-key-morphological"
},
{
"startOffset": 20,
"endOffset": 26,
"name": "origin",
"id": "T0",
"dataSpan": "Boston",
"obsolete": false,
"roleType": "generic_schema.location",
"extractorName": "generic_schema.location-entity"
},
{
"startOffset": 30,
"endOffset": 38,
"name": "destination",
"id": "T1",
"dataSpan": "New York",
"obsolete": false,
"roleType": "generic_schema.location",
"extractorName": "generic_schema.location-entity"
}
]
}
],
"confidence": 0.93891401,
"workspaceId": "multi-1"
}
]
}Model Training Suite
Model Training Suite (MTS) is a framework with user-friendly graphical user interface for quickly creating machine learning models or adapting existing models to a particular domain or data set. Compared to traditional data annotation methods, MTS reduces the number of training documents that need to be annotated to achieve a reasonable level of accuracy, shortening the annotation process.
Key features
Model Training Suite is a framework to quickly annotate, train, and deploy NLP models. Inside, Adaptation Studio provides a user-friendly interface for annotators to annotate and for project managers to easily check cross-annotation agreement and adjudicate conflicts to produce high-quality training data.
AI-assisted data preprocessing — Instead of starting each model from scratch, MTS builds on top of existing natural language models.
Iterative model evaluation —Project managers can continually monitor the accuracy of an interim model built from documents tagged “so far,” so that tagging can be halted as soon as the model reaches the target accuracy or hits a point of diminishing returns.
Efficient annotation — Based on the interim model’s confidence score on untagged documents, active learning recommends the most likely “informative” documents to be tagged first.
Computer-assisted tagging — The interim model will pre-tag documents for the human annotator to accept, reject, or correct; adjusting tags is much faster than tagging from scratch.
NLP models
Model Training Suite is set up to train models for the following NLP tasks. It is extensible to train models for other tasks, too.
Entity extraction
Event extraction
Languages
Model Training Suite trains custom models for entity extraction and events in the following languages. The language support is easily extensible to all the languages supported by Base Linguistics.
Server
Semantic Similarity
Semantic Similarity can compare the meaning of words and text within the following languages (orchestra, symphony) and between these languages (king, roi). It can also find semantically similar words given an input word. See the sample output in the righthand column of the word “spy” and its semantically similar output in Spanish, German, and Japanese.
Documentation and Resources
Try Analytics Cloud: developer.babelstreet.com
Supported languages
Arabic (
ara)Chinese (
zho)English (
eng)French (
fra)German (
deu)Hebrew (
heb)Hungarian (
hun)Italian (
ita)Japanese (
jpn)Korean (
kor)Korean - North (
qkp)Korean - South (
qkr)Persian (
fas)Portuguese (
por)Russian (
rus)Spanish (
spa)Tagalog (
tgl)Urdu (
urd)
Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
/semantics/similar){"content": "spy", "options": {"resultLanguages": ["spa", "deu", "jpn"]}}
{
"similarTerms": {
"spa": [
{
"term": "espía",
"similarity": 0.61295485
},
{
"term": "cia",
"similarity": 0.46201307
},
{
"term": "desertor",
"similarity": 0.42849663
},
{
"term": "cómplice",
"similarity": 0.36646274
},
{
"term": "subrepticiamente",
"similarity": 0.36629659
},
{
"term": "asesino",
"similarity": 0.36264464
},
{
"term": "misterioso",
"similarity": 0.35466132
},
{
"term": "fugitivo",
"similarity": 0.35033143
},
{
"term": "informante",
"similarity": 0.34707013
},
{
"term": "mercenario",
"similarity": 0.34658083
}
],
"jpn": [
{
"term": "スパイ",
"similarity": 0.5544399
},
{
"term": "諜報",
"similarity": 0.46903181
},
{
"term": "MI6",
"similarity": 0.46344957
},
{
"term": "殺し屋",
"similarity": 0.41098994
},
{
"term": "正体",
"similarity": 0.40109193
},
{
"term": "プレデター",
"similarity": 0.39433435
},
{
"term": "レンズマン",
"similarity": 0.3918637
},
{
"term": "S.H.I.E.L.D.",
"similarity": 0.38338536
},
{
"term": "サーシャ",
"similarity": 0.37628397
},
{
"term": "黒幕",
"similarity": 0.37256041
}
],
"deu": [
{
"term": "Deckname",
"similarity": 0.51391315
},
{
"term": "GRU",
"similarity": 0.50809389
},
{
"term": "Spion",
"similarity": 0.50051737
},
{
"term": "KGB",
"similarity": 0.49981388
},
{
"term": "Informant",
"similarity": 0.48774603
},
{
"term": "Geheimagent",
"similarity": 0.48700801
},
{
"term": "Geheimdienst",
"similarity": 0.48512384
},
{
"term": "Spionin",
"similarity": 0.47224587
},
{
"term": "MI6",
"similarity": 0.46969846
},
{
"term": "Decknamen",
"similarity": 0.44730526
}
]
}Base Linguistics
Text analytics fundamentals to prepare your data for multilingual search and advanced NLP analysis.
Documentation and Resources
Try the demo: demo.babelstreet.com/text-analytics-demo
Try Analytics Cloud: developer.babelstreet.com
Key features
Base Linguistics outputs these morphological analyses, some of which are language-specific.
Tokenization
Sentence boundary detection
Part of speech tagging
Lemmatization
Noun decompounding
Chinese readings (Pinyin pronunciation)
Chinese script conversion (traditional <=> simplified
Japanese readings (pronunciation)
Japanese spelling normalization (Katakana and modern/old-style Kanji)
Arabic script languages; text and token normalization, semitic root and stem analysis
Languages and analyses
Base Linguistics outputs the following analyses for each supported language as shown in the table below.
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Elasticsearch Solr
{
"tokens": [
"The",
"fact",
"is",
"that",
"the",
"geese",
"just",
"went",
"back",
"to",
"get",
"a",
"rest",
"and",
"I",
"'m",
"not",
"banking",
"on",
"their",
"return",
"soon"
],
"lemmas": [
"the",
"fact",
"be",
"that",
"the",
"goose",
"just",
"go",
"back",
"to",
"get",
"a",
"rest",
"and",
"I",
"be",
"not",
"bank",
"on",
"they",
"return",
"soon"
]
}Language Identifier
Instantly identify the language of whole documents or multiple language regions within each document.
Documentation and Resources
Try the demo: demo.babelstreet.com/text-analytics-demo
Try Analytics Cloud: developer.babelstreet.com
Key features
Detects North Korean v. South Korean text, and transliterated Arabic, Kurdish, Persian, Pashto, and Urdu
Detects language of short strings (3 words to a sentence)
Detects sections of text in different languages within a single multilingual document
Languages
Language Identifier detects text in the following languages and encodings.
Java SDK For on-premises systems that need the low-latency, high-speed integration of an SDK, Java is the way to go. It has been deployed in the most demanding, high-transaction environments, including web search engines, financial compliance, and border security. Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
Elasticsearch Solr
{
"languageDetections": [
{
"language": "spa",
"confidence": 0.38719602327387076
},
{
"language": "eng",
"confidence": 0.32699986625091865
},
{
"language": "por",
"confidence": 0.05569054210624943
},
{
"language": "deu",
"confidence": 0.030069489878380328
},
{
"language": "swe",
"confidence": 0.027734757034048835
}
]
}Sentiment Analyzer
Detects positive, negative or neutral sentiment of a whole document or emotional hotspots in text about companies, people, and products. Sentiment Analyzer rates sentiment on a scale of 1 (positive) to –1 (negative) with 0 being neutral.
Documentation and Resources
Try the demo: demo.babelstreet.com/text-analytics-demo
Try Analytics Cloud: developer.babelstreet.com
Languages
Sentiment Analyzer identifies the emotion within text written in these languages.
Arabic (
ara)English (
eng)French (
fra)Japanese (
jpn)Persian (
fas)Spanish (
spa)
Entity types
Sentiment Analyzer can identify sentiment about a particular entity. Entity-centric sentiment analysis is possible for the entity types listed below and any other type supported by Entity Extractor.
Person
Organization
Location
Nationality
Religion
Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
For the document, and each identified entity, only the highest scoring sentiment is returned, along with a confidence value between 0 and 1. For each entity, detail about the entity is also returned.
{
"document": {
"label": "string",
"confidence": number
},
"entities": [
{
"type": "string",
"mention": "string",
"normalized": "string",
"count": 0,
"mentionOffsets": [
{
"startOffset": number,
"endOffset": number
}
],
"entityId": "string",
"confidence": 0,
"linkingConfidence": 0,
"sentiment": {
"label": "string",
"confidence": number
}
]
}
Topic Extractor
Identify keywords and significant phrases in your text, and the topics not explicitly named to capture the essence of a document’s content.
Documentation and Resources
Try the demo: demo.babelstreet.com/text-analytics-demo
Try Analytics Cloud: developer.babelstreet.com
Key features
Extracts keyphrases from the text that best represent its content.
Extracts concepts contained in the text, but which may not be explicitly stated.
Languages
Out of the box, Analytics extracts topics from these languages.
English
Sample extraction
Topic Extractor might find these concepts in the text below which are not explicitly mentioned.
“Substance abuse"
"Rex Tillerson”
“Harm reduction”
“Heroin”
“Controlled Substances Act”
"Opioid”
"Jeff Sessions"
"Drug policy reform”
"Infinite Crisis”
But also extract these keyphrases which are representative of the text.
Analytics Server This on-premises private cloud deployment puts all the functionality of the Analytics API behind your secure firewall, and enables advanced user settings, access to custom profiles (user-specific configuration setups), and deployment of custom models. Hosted Services The SaaS version of Babel Street Analytics is rapidly implemented, low maintenance and ideal for users who wish to pay based on monthly call volume. Numerous bindings through a RESTful API are supported.
{"content": "To Sleep John Keats, 1795 - 1821
O soft embalmer of the still midnight!
Shutting with careful fingers and benign
Our gloom-pleased eyes, embower’d from the light,
Enshaded in forgetfulness divine;
O soothest Sleep! if so it please thee, close,
In midst of this thine hymn, my willing eyes,
Or wait the amen, ere thy poppy throws
Around my bed its lulling charities;
Then save me, or the passèd day will shine
Upon my pillow, breeding many woes;
Save me from curious conscience, that still lords
Its strength for darkness, burrowing like a mole;
Turn the key deftly in the oilèd wards,
And seal the hushèd casket of my soul. - John Keats
This poem is in the public domain.
John Keats
Born in 1795, John Keats was an English Romantic poet and author of three poems considered to be among the finest in the English language."}{"keyphrases":
[{"phrase": "lulling charities"},
{"phrase": "O soothest Sleep"},
{"phrase": "John Keats"},
{"phrase": "O soft embalmer"},
{"phrase": "hushèd casket"},
{"phrase": "English Romantic poet"},
{"phrase": "forgetfulness divine"},
{"phrase": "pleased eyes"},
{"phrase": "passèd day"},
{"phrase": "oilèd wards"}],
"concepts":
[{"phrase": "John Keats",
"conceptId": "Q82083"}]}Resources
Documentation
Match Names
Text Analytics
API Documentation
Try a demo
Matching people, organizations, locations, addresses, and dates: demo.babelstreet.com/match-demo
Unstructured text analysis (entities, morphology, sentiment, and topics): demo.babelstreet.com/text-analytics-demo
Sign up for Analytics Cloud: developer.babelstreet.com