Inter-Annotator Agreement Charts (IAA)

Inter Annotator Agreement charts can be used by annotators, reviewers, and managers for identifying contradictions or disagreements within the stared completions. When multiple annotators work on the same tasks, IAA charts are handy to measure how well the annotations created by different annotators align. IAA chart can also be used to identify outliers in the labeled data or to compare manual annotations with model predictions. To get access to the IAA charts, navigate on the third tab of the Analytics Dashboard of NER projects, called “Inter-Annotator Agreement”. Several charts should appear on the screen with a default selection of annotators to compare. The dropdown boxes below each chart allow you to change annotators for comparison purposes. It is also possible to download the data generated for some charts in CSV format by clicking the download button present at the bottom right corner of each of them.

Note: Only the Submitted and starred (Ground Truth) completions are used to render these charts.

  • Shows agreement with Label wise breakdown
  • Shows whether two Annotators agree on every annotated Chunks
  • Shows agreement between one Annotator and Preannotation result on every annotated Chunks
  • Shows Labels of each Chunk by one Annotator and context in the tasks
  • Shows the frequency of Chunk-Label by one Annotator
  • Shows the frequency of Chunk-Annotatory by one Label

Disabled Analytics

By default, the Analytics dashboards page is disabled. Users can request Admin to enable the Analytics page. The request can be seen by the Admin user’s account, on the Settings page. Once the Admin user approves the request, users can access the analytics page.


Refresh Analytics

A refresh button is added to this analytics page. Changes made by the annotators (like creating tasks, adding new completion, etc.) will not be automatically reflected in the Analytics charts. The latter can be updated by pressing the refresh button.

Download data used for charts

CSV file for specific charts can be downloaded using the new download button which will call specific API endpoints: /api/projects/{project_name}/charts/{chart_type}/download_csv

Screen Recording 2022-03-08 at 3 47 49 PM

IAA between manager and annotator

Starting from version 2.8.0, Inter Annotator Agreement(IAA) charts allow the comparison between annotations produced by annotators, project managers, or reviewers.

Screen Recording 2022-03-14 at 1 45 03 PM

Last updated