Edit on GitHub
  • John Snow Labs
  • Apr 27, 2023

Legal NER for MAPA(Multilingual Anonymisation for Public Administrations)

  • lt
  • licensed
  • ner
  • legal
  • mapa

Description

The dataset consists of 12 documents taken from EUR-Lex, a multilingual corpus of court decisions and legal dispositions in the 24 official languages of the European Union.

This model extracts ADDRESS, AMOUNT, DATE, ORGANISATION, and PERSON entities from Lithuanian documents.

Predicted Entities

ADDRESS, AMOUNT, DATE, ORGANISATION, PERSON

Download Copy S3 URI

How to use

document_assembler = nlp.DocumentAssembler()\
        .setInputCol("text")\
        .setOutputCol("document")

sentence_detector = nlp.SentenceDetectorDLModel.pretrained("sentence_detector_dl", "xx")\
        .setInputCols(["document"])\
        .setOutputCol("sentence")

tokenizer = nlp.Tokenizer()\
        .setInputCols(["sentence"])\
        .setOutputCol("token")

embeddings = nlp.BertEmbeddings.pretrained("bert_embeddings_base_lt_cased", "lt")\
        .setInputCols(["sentence", "token"])\
        .setOutputCol("embeddings")\
        .setMaxSentenceLength(512)\
        .setCaseSensitive(True)

ner_model = legal.NerModel.pretrained("legner_mapa", "lt", "legal/models")\
        .setInputCols(["sentence", "token", "embeddings"])\
        .setOutputCol("ner")

ner_converter = nlp.NerConverter()\
        .setInputCols(["sentence", "token", "ner"])\
        .setOutputCol("ner_chunk")

nlpPipeline = nlp.Pipeline(stages=[
        document_assembler,
        sentence_detector,
        tokenizer,
        embeddings,
        ner_model,
        ner_converter])

empty_data = spark.createDataFrame([[""]]).toDF("text")

model = nlpPipeline.fit(empty_data)

text = ["""Iš pagrindinės bylos matyti, kad Martin-Meat darbuotojai buvo komandiruoti į Austriją laikotarpiu nuo 2007 m iki 2012 m mėsos išpjaustymo darbams Alpenrind patalpose atlikti."""]

result = model.transform(spark.createDataFrame([text]).toDF("text"))

Results

+-----------+------------+
|chunk      |ner_label   |
+-----------+------------+
|Martin-Meat|ORGANISATION|
|Austriją   |ADDRESS     |
|2007 m     |DATE        |
|2012 m     |DATE        |
|Alpenrind  |ORGANISATION|
+-----------+------------+

Model Information

Model Name: legner_mapa
Compatibility: Legal NLP 1.0.0+
License: Licensed
Edition: Official
Input Labels: [sentence, token, embeddings]
Output Labels: [ner]
Language: lt
Size: 1.4 MB

References

The dataset is available here.

Benchmarking

label         precision  recall  f1-score  support 
ADDRESS       0.86       0.75    0.80      8       
AMOUNT        1.00       0.64    0.78      11      
DATE          0.97       0.97    0.97      65      
ORGANISATION  0.81       0.86    0.83      35      
PERSON        0.87       0.84    0.85      56      
macro-avg     0.90       0.87    0.89      175     
macro-avg     0.90       0.81    0.85      175     
weighted-avg  0.90       0.87    0.89      175