Legal Proposal Summarization

document_assembler = nlp.DocumentAssembler()\
    .setInputCol("text")\
    .setOutputCol("document")

t5 = nlp.T5Transformer().pretrained("legsum_proposal", "en", "legal/models")\
    .setTask("summarize")\
    .setMaxOutputLength(512)\
    .setInputCols(["document"])\
    .setOutputCol("summaries")

text = """The main reason for migration is poverty, and often times it is down to corruption in the leadership of poor countries. What people in such countries demand time and again is that the EU does not engage with their government, and does not supply financial support (which tends to end up in the wrong hands). The EU needs a strict line of engagement. One could envision a rating list by the EU that defines clear requirements support receiving nations must fulfill. Support should be granted in the form of improved economic conditions, such as increased import quota, discounted machinery, and technical know-how injection, not in terms of financial support. Countries failing to fulfill the requirements, especially those with indications of corruption must be put under strict embargoes."""

data_df = spark.createDataFrame([[text]]).toDF("text")

pipeline = nlp.Pipeline().setStages([document_assembler, t5])

results = pipeline.fit(data_df).transform(data_df)

results.select("summaries.result").show(truncate=False)