Extract and Link Information

Each user can edit their personal information, including changing their password.

Once Adaptation Studio is installed and running, go to Users to create users. User annotations are tied to the user id. If you change a user id, the user will lose access to any annotations they’ve made.

The superuser can create users and perform all tasks. At install, the superuser is set to admin.

The first time you log in as admin, you will be forced to change the password. The initial login information is:

The superuser password can be reset by running the reset_admin.sh in /basis/ras/scripts. After reset, you will be prompted to change the password when you log in the first time.

./reset_admin.sh '2^$4bu\Qm$4C'

Before starting annotation, add new labels, if necessary.

It may take a few minutes for an imported project to appear in the Project list, especially if it is a large project. The imported project will only be visible in the Project list to superusers and users assigned to the project.

Comments allow you to pose questions or provide clarifications about the sample being edited to adjudicators and managers. Adjudicators and managers can view the comments for each sample via View Annotations. Comments do not affect model training in any way. To add a comment while annotating, follow these steps:

You can edit comments you have made by selecting the commented annotation and selecting Edit Comment. From here you can also delete the comment by selecting the trash can icon.

When the Enable link annotation option is selected, you can add a link ID to the entity, along with the label. The link IDs are used to train a linking knowledge base. Entity extractor supports linking to a default Wikidata knowledge base. You can also train it to link to a custom knowledge base with Model Training Suite.

The annotations in Adaptation Studio are sent to the Entity Training Server and provide the data for training a new model. Only adjudicated annotations are included in training. The system will auto-adjudicate samples.

The rules for auto adjudication are:

Select Filters on the View Annotations page to expand the Filters panel. To view the filter criteria, click on the down arrow to expand the section. The Filters panel allows you to select which samples to display in the sample list.

Select Update results to filter the sample list so that it only displays samples with all chosen criteria.

All samples in the project that match the applied filters are listed below the filters panel.

Each sample is displayed next to a table showing the tags selected by each annotator for each entity. If you are a project manager, this view is useful for identifying and inspecting specific points of high annotator disagreement. Samples with disagreements are highlighted in red. Adjudicators and managers can also see any comments added by annotators on the rightmost column of the table.

The following are displayed below each sample:

If there are two or more identical entities, each will be labeled with a corresponding id in the sample text and the Entity column. Mouse over any entity in the Entity column to highlight that entity in the sample text.

The following management tasks and information are accessed by selecting Manage from the project navigation bar.

{
  "export_time": "2021-11-15 11:37:49",
  "language": "eng",
  "labels": [
    {
      "type": "PERSON",
      "label": "Person"
    },
    {
      "type": "LOCATION",
      "label": "Location"
    },
    {
      "type": "ORGANIZATION",
      "label": "Organization"
    },
    {
      "type": "TITLE",
      "label": "Title"
    },
    {
      "type": "PRODUCT",
      "label": "Product"
    }
  ],
  "training_count": 795,
  "eval_count": 75,
  "scores": {
    "precision": 1,
    "recall": 1,
    "f1": 1
  }
}

These tasks are available from the project menu in the upper right-hand corner of the project dashboard.

From the project menu, you can export a project or extract ADMs. The annotated data model (ADM) is a format used to represent the objects in a model.

Reports provide information to project managers pertaining to the quantity and quality of the data being created by annotators and adjudicators. When training a model, it is critical to have the correct volume and distribution of high-quality data. Only managers have access to reports.

Most NLP tools are evaluated based on their precision and recall. Accuracy is measured as a combination of precision, recall, and F1 measure.

The correct measure depends on your application. In applications where you can only handle a few responses, such as voice applications (e.g. Amazon's Alexa), high precision with low recall is ideal, because the system can only present the user with a few options in a reasonable time frame. However, other applications, such as redacting text to remove personally identifiable information, redacting too much (low precision) is much better than missing even one item that should have been redacted. In that case, high recall is preferred.

The values displayed on the project dashboard are calculated using the annotated validation data as the gold data. As the model is trained, it generates new predictions and the scores are recalculated. The predictions generated by the model for the validation samples are compared with the annotated values in the samples.

Let's look at how precision, recall, and f-score are calculated. Let's assume a search system, where a number of items were retrieved.

TP: True positive. A correctly extracted entity.

FP: False positive. An incorrectly extracted entity.

FN: False negative. Number of entities that should have been extracted, but weren't.

Retrieved: All entities retrieved = TP + FP

Relevant: All entities that should have been retrieved = TP + FN

Precision is the fraction of correct retrievals among all retrieved instances.

Recall is the fraction of relevant entities that are successfully retrieved.

F-score is the harmonic mean of precision and recall

The project dashboard displays precision, recall, and F1 measures for the labels. There are several ways of calculating these scores across multiple labels.

An important concept here is support: the frequency of a label in the dataset.

Named entity recognition models are trained to recognize and extract entity mentions. Each annotated sample may contain multiple entities.

The statistics displayed on the project dashboard are the macro averages of the precision, recall, and F1 measures for the labels.

Machine-learning models are completely dependent on the quality of the data used for training. Inconsistent annotation or lack of adherence to the project annotation guidelines will lead to less accurate models. Especially when starting a new project or on-boarding new human annotators, check for reliable annotations by having a subset of data annotated in parallel by multiple human annotators.

Krippendorff’s alpha is a statistical inter-rater reliability metric used to measure inter-annotator agreement. Krippendorff’s alpha scores range from -1.0 to 1.0 with 1.0 indicating perfect agreement between annotators. A score of 0.0 indicates agreement no better than random chance (as if your annotators picked their labels randomly out of a hat). A reliability score of 0.80 or greater is generally considered sufficient (though, the higher the better). Lower scores may indicate potential issues with your data, your annotation guidelines, or your annotators’ understanding of the task. A low level of inter-annotator agreement will ultimately lead to a less accurate model, so we recommend repeatedly measuring the reliability of your annotators until they achieve a satisfactory level of agreement. The cases where annotators disagree are usually good examples to include in your annotation guidelines. It can be useful to have a discussion about points of disagreement with your annotators as a group to reach a consensus.

The IAA history report displays the progress of the inter-annotator agreement over time.

Event modeling can be complex. To help you learn about event modeling, look for tutorials on our support site. Events tutorials guide you through creating a schema and annotating data. They include sample data and basic annotation guidelines.

There are two types of event schema, they both contain the same objects and definitions. The difference is whether the schema is attached to a project.

Events schema template

To use the schema editor to create a template schema, select Manage from the top navigation bar. Then select Create under Event Schema Template.

Note

You must be a manager or superuser.

You can create a schema from nothing, or modify an existing schema as a starting point. The schema created in this dialogue should be considered a template for an events project. When you create a project, the project schema will use the template.

Any updates to the project schema will modify the project version only; the template will not be changed. The same template can be used by multiple projects, each of which will have its own version.

Once complete, Save the schema definition. You can then use the schema template in a project. To modify a project schema, select Project Schema from the project dashboard options.

Modify an existing schema

Schemas already defined may contain event types, roles, role types, and/or extractors which you'd like to use in a new schema template. You can take some or all of the objects in an existing schema template and create a new template. You can also import a .json file definition of an existing schema.

1. From the top navigation bar, select Manage then select Modify from the Event Schema Template section.

2. Select an existing schema.

3. Select a schema from the list in the Existing schemas box. The objects in the schema will be displayed in the Preview box.

4. At the bottom of the screen, choose which elements from the schema you would like in your new schema. Select Everything to edit the entire schema or Let me select to choose a subset of schema objects to use.

5. Next.

6. Select the objects you want to import. If you selected Everything, all objects will be selected. You can modify your choices at this time.

7. Load.

8. Complete defining the schema. If you want to retain the original schema, modify the schema name in the General Information section. Otherwise, you will overwrite the original schema template definition.

Both the schema template and project schema use the same dialog to define and configure the schema.

The schema definition is broken up into 3 sections:

Once the objects in a section are defined completely, the indicator in the right corner will change to a green checkmark.

Before starting annotation, modify the project schema.

It may take a few minutes for an imported project to appear in the Project list, especially if it is a large project. The imported project will only be visible in the Project list to superusers and users assigned to the project.

A sample may contain multiple event mentions, but each mention must be annotated individually. Let's look at an example:

My flight to San Diego took 6 hours, but my flight back to Boston only took 5.

In this example, we're extracting flight events. The simplified schema includes:

There are 2 event mentions in the above sample:

Each event mention gets its own annotation.

Both event mention annotations will be saved.

Comments allow you to pose questions or provide clarifications about the sample being edited to adjudicators and managers. Adjudicators and managers can view the comments for each sample via View Annotations. Comments do not affect model training in any way. To add a comment while annotating, follow these steps:

You can edit comments you have made by selecting the commented annotation and selecting Edit Comment. From here you can also delete the comment by selecting the trash can icon.

The adjudication panel displays the sample with the selected event on the left side of the window and with the annotations by annotator on the right side of the window. The full paragraph is displayed below for context.

The annotations frame has the following columns:

Event adjudication has multiple steps:

During annotation, an annotator may identify key phrases or role mentions that the event model has not identified as candidates. The annotator can label a key phrase or a role that was not suggested. The Studio then creates a tentative role type enabling the annotator to continue annotating documents.

At some point, the tentative role types must be reviewed and approved as part of the adjudication process. If the adjudicator Accepts the new role type, it is added to the schema and the extractor is modified. If it shouldn't have been defined, the adjudicator can Reject the tentative role type.

Tentatives are resolved as part of the adjudication process.

To review tentative role types:

Tentative extractors are useful for supplementing what is being extracted by the model. They can be used to fill in the "blind spots" of a model that is identifying some, but not all, of the desired phrases as candidate roles. However, if you find yourself making new exact entity extractors for almost every role, you likely need to train a new entity model.

The annotations in Adaptation Studio are sent to the Event Training Server and provide the data for training a new model. Only adjudicated annotations are included in training. The system will auto-adjudicate samples.

The rules for auto adjudication are:

Select Filters on the View Annotations page to expand the Filters panel. To view the filter criteria, click on the down arrow to expand the section. The Filters panel allows you to select which samples to display in the sample list using the following criteria:

Select Update results to filter the sample list so that it only displays samples with all chosen criteria.

All samples in the project that match the applied filters are listed below the filters panel.

Each sample is displayed next to a table showing the events and roles selected by each annotator. If you are a project manager, this view is useful for identifying and inspecting multiple annotations at a time. Managers can also see any comments added by annotators on the rightmost column of the table.

The following are displayed below each sample:

Each event is identified down arrow (v) with the roles listed underneath. There may be multiple events listed in a single sample. You can expand or hide the roles for each event.

The base event algorithm extracts events when a key phrase and any required role mentions are detected in the document. It does not recognize whether the event happened or didn't happen, also known as the polarity of the event. For example, in a travel event, the following two sentences will both be extracted by the key phrase "travel":

In the example above, "didn't" is an example of a negation cue. The existence of the cue signifies the event is negated.

You can choose to include or ignore negation in Test Document. The Negation option has 4 values:

This example shows the negation cue:

The following management tasks and information are accessed by selecting Manage from the navigation bar.

These tasks are available from the project menu in the upper right-hand corner of the project dashboard.

From the project menu, you can export a project or extract ADMs. The annotated data model (ADM) is a format used to represent the objects in a model.

{
  "export_time": "2021-11-18 08:17:53",
  "language": "eng",
  "labels": [
    {
      "type": "key",
      "label": "flight_booking"
    },
    {
      "type": "origin",
      "label": "origin"
    },
    {
      "type": "when",
      "label": "when"
    },
    {
      "type": "destination",
      "label": "destination"
    }
  ],
  "training_count": 59,
  "eval_count": 56,
  "scores": {
    "precision": 0.4,
    "recall": 0.47058823529411764,
    "f1": 0.4324324324324324
  }
}

Adaptation Studio is the training platform for training event models. To use the model in production, the trained model must be copied to the production instance of Analytics Server. There are two ways to deploy an events model to productions:

Requirements

The deployment server and model are configured for each project.

Shows the distribution of annotations and adjudications among all event types. Select Download CSV to download the data.

The Evaluation section includes two options: Event Detection and Event Extraction.

There are two other rows in the detection and extraction tables that merit some additional explanation: Model Macro and Model Weighted. Both of these provide a holistic, rather than event-specific, indication of the model's performance. The key difference is how much each event is weighted when the average is calculated.

Machine-learning models are completely dependent on the quality of the data used for training. Inconsistent annotation or lack of adherence to the project annotation guidelines will lead to less accurate models. Especially when starting a new project or on-boarding new human annotators, check for reliable annotations by having a subset of data annotated in parallel by multiple human annotators.

Krippendorff’s alpha is a statistical inter-rater reliability metric used to measure inter-annotator agreement. Krippendorff’s alpha scores range from -1.0 to 1.0 with 1.0 indicating perfect agreement between annotators. A score of 0.0 indicates agreement no better than random chance (as if your annotators picked their labels randomly out of a hat). A reliability score of 0.80 or greater is generally considered sufficient (though, the higher the better). Lower scores may indicate potential issues with your data, your annotation guidelines, or your annotators’ understanding of the task. A low level of inter-annotator agreement will ultimately lead to a less accurate model, so we recommend repeatedly measuring the reliability of your annotators until they achieve a satisfactory level of agreement. The cases where annotators disagree are usually good examples to include in your annotation guidelines. It can be useful to have a discussion about points of disagreement with your annotators as a group to reach a consensus.

The IAA history report displays the progress of the inter-annotator agreement over time.

Most NLP tools are evaluated based on their precision and recall. Accuracy is measured as a combination of precision, recall, and F1 measure.

The correct measure depends on your application. In applications where you can only handle a few responses, such as voice applications (e.g. Amazon's Alexa), high precision with low recall is ideal, because the system can only present the user with a few options in a reasonable time frame. However, other applications, such as redacting text to remove personally identifiable information, redacting too much (low precision) is much better than missing even one item that should have been redacted. In that case, high recall is preferred.

The values displayed on the project dashboard are calculated using the annotated validation data as the gold data. As the model is trained, it generates new predictions and the scores are recalculated. The predictions generated by the model for the validation samples are compared with the annotated values in the samples.

Let's look at how precision, recall, and f-score are calculated. Let's assume an event model, where a number of events were extracted.

TP: True positive. A correctly extracted event.

FP: False positive. An incorrectly extracted event.

FN: False negative. Number of events that should have been extracted, but weren't.

Retrieved: All events retrieved = TP + FP

Relevant: All events that should have been retrieved = TP + FN

Precision is the fraction of correct retrievals among all retrieved instances.

Recall is the fraction of relevant events that are successfully retrieved.

F-score is the harmonic mean of precision and recall

The project dashboard displays precision, recall, and F1 measures for the event model. There are several ways of calculating these scores across multiple event types.

An important concept here is support: the frequency of an event type in the dataset.

Event extraction models are trained to recognize and extract multiple components. Each event mention contains a key phrase and one or more roles. A single annotated sample may contain multiple event mentions, and therefore, multiple key phrases and multiple roles.

The statistics displayed on the project dashboard are the macro averages of precision, recall, and f1 for event detection.

More detail is available on the reports page, where you can see both macro and weighted averages for detection (just the key phrase) and extraction (key phrase and role mentions).