Text Annotation

 

Overview

The Annotator Lab is designed to keep a human expert as productive as possible. It minimizes the number of mouse clicks, keystrokes, and eye movements in the main workflow, based on iterative feedback from daily users.

Keyboard shortcuts are supported for all annotations – this enables having one hand on keyboard, one hand on mouse, and eyes on screen at all time. One-click completion and automated switching to the next task keeps experts in the loop.

On the upper side of the Labeling screen, you can find the list of labels defined for the project. In the center of the screen the content of the task is displayed. On the right side there are several widgets:

  • Completions
  • Predictions
  • Results

Completions

A completion is a list of annotations manually defined by a user for a given task. When the work on a task is done (e.g. all entities have been highlighted in the document or the task has been assigned one or more classes in the case of classification projects) the user clicks on the Save button.

Starting Annotation Lab 1.2.0, we introduced the idea of completion submission. In the past, annotators could change or delete completions as many times as they wanted with no restriction. From now on, a submitted completion is no longer editable and cannot be deleted. Creating a new copy of the submitted completion is the only option to edit it. An annotator can modify or delete his/her completions only if the completions are not submitted yet.

This is an important feature for ensuring a complete audit trail of all user actions. Now, it is possible to track the history and details of any deleted completions, which was not possible in previous releases. This means it is possible to see the name of the completion creator, date of creation, and deletion.

Predictions

A prediction is a list of annotations created automatically, via the use of Spark NLP pretrained models. Predictions are created using the “Preannotate” button form the Task view. Predictions are read only - users can see them but cannot modify them in any way.

Results

The results widget has two sections.

The first section - Regions - gives a list overview of all annotated chunks. When you click on one annotation it gets automatically highlighted in the document. Annotations can be edited or removed if necessary.

The second section - Relations - lists all the relations that have been labeled. When the user moves the mouse over one relation it is highlighted in the document.

NER Labels

To extract information using NER labels, first click on the label to select it or press the shortcut key assigned to it and then, with the mouse, select the relevant part of the text. Wrong extractions can be easily edited by clicking on them to select them and then by selecting the new label you want to assign to the text.

For deleting a label, select it by clicking on it and press backspace.

Assertion Labels

To add an assertion label to an extracted entity, select the label and use it to do the same exact extraction as the NER label. After this, the extracted entity will have two labels: one for NER and one for assertion. In the example below, the chunks “heart disease”, “kidney disease”, “stroke” etc. ware extracted using first a blue yellow label and then a red assertion label (Absent).

Relation Extraction

Creating relations with the Annotation Lab is very simple. First select one labeled entity, then press r and select the second labeled entity.

You can add a label to the relation, change its direction or delete it using the relation box.

Pre-annotations

When you setup a project to use existing Spark NLP models for pre-annotation, you can run the designated models on all of your tasks by pressing the Preannotate button located on the upper side of the task view.

As a result, all predicted labels for a given task will be available in the Prediction widget, on the main annotation screen. The predictions are not editable, you can only view them and navigate them or compare them with older predictions. However, you can create a new completion based on a given prediction. All labels and relations from such a new completion are now editable.

Last updated