Packages

package dl

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. class AssertionDLApproach extends AnnotatorApproach[AssertionDLModel] with Logging with Licensed

    Contains all the methods for training an AssertionDLModel.

    Contains all the methods for training an AssertionDLModel. For pretrained models please use AssertionDLModel and see the Models Hub for available models.

    Example

    First, pipeline stages for pre-processing the dataset (containing columns for text and label) are defined.

    val document = new DocumentAssembler()
      .setInputCol("text")
      .setOutputCol("document")
    val chunk = new Doc2Chunk()
      .setInputCols("document")
      .setOutputCol("chunk")
    val token = new Tokenizer()
      .setInputCols("document")
      .setOutputCol("token")
    val embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
      .setInputCols("document", "token")
      .setOutputCol("embeddings")

    Define AssertionDLApproach with parameters and start training

    val assertionStatus = new AssertionDLApproach()
      .setLabelCol("label")
      .setInputCols("document", "chunk", "embeddings")
      .setOutputCol("assertion")
      .setBatchSize(128)
      .setDropout(0.012f)
      .setLearningRate(0.015f)
      .setEpochs(1)
      .setStartCol("start")
      .setEndCol("end")
      .setMaxSentLen(250)
    
    val trainingPipeline = new Pipeline().setStages(Array(
      document,
      chunk,
      token,
      embeddings,
      assertionStatus
    ))
    
    val assertionModel = trainingPipeline.fit(data)
    val assertionResults = assertionModel.transform(data).cache()
    See also

    AssertionDLModel for using pretrained models

    AssertionLogRegModel for non deep learning based extraction

  2. class AssertionDLModel extends AnnotatorModel[AssertionDLModel] with WriteTensorflowModel with HasStorageRef with HasSimpleAnnotate[AssertionDLModel] with ParamsAndFeaturesWritable with Licensed

    This is a main class in AssertionDL family.

    This is a main class in AssertionDL family. A Deep Learning based approach is used to extract Assertion Status from extracted entities and text. AssertionDLModel requires DOCUMENT, CHUNK and WORD_EMBEDDINGS type annotator inputs, which can be obtained by e.g a DocumentAssembler, NerConverter and WordEmbeddingsModel. The result is an assertion status annotation for each recognized entity. Possible values include “present”, “absent”, “hypothetical”, “conditional”, “associated_with_other_person” etc.

    For pretrained models please see the Models Hub for available models.

    Example

    Define pipeline stages to extract NER chunks first

    val data = Seq(
      "Patient with severe fever and sore throat",
      "Patient shows no stomach pain",
      "She was maintained on an epidural and PCA for pain control.").toDF("text")
    val documentAssembler = new DocumentAssembler().setInputCol("text").setOutputCol("document")
    val sentenceDetector = new SentenceDetector().setInputCols("document").setOutputCol("sentence")
    val tokenizer = new Tokenizer().setInputCols("sentence").setOutputCol("token")
    val embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
      .setOutputCol("embeddings")
    val nerModel = MedicalNerModel.pretrained("ner_clinical", "en", "clinical/models")
      .setInputCols("sentence", "token", "embeddings").setOutputCol("ner")
    val nerConverter = new NerConverter().setInputCols("sentence", "token", "ner").setOutputCol("ner_chunk")

    Then a pretrained AssertionDLModel is used to extract the assertion status

    val clinicalAssertion = AssertionDLModel.pretrained("assertion_dl", "en", "clinical/models")
      .setInputCols("sentence", "ner_chunk", "embeddings")
      .setOutputCol("assertion")
    
    val assertionPipeline = new Pipeline().setStages(Array(
      documentAssembler,
      sentenceDetector,
      tokenizer,
      embeddings,
      nerModel,
      nerConverter,
      clinicalAssertion
    ))
    
    val assertionModel = assertionPipeline.fit(data)

    Show results

    val result = assertionModel.transform(data)
    result.selectExpr("ner_chunk.result", "assertion.result").show(3, truncate=false)
    +--------------------------------+--------------------------------+
    |result                          |result                          |
    +--------------------------------+--------------------------------+
    |[severe fever, sore throat]     |[present, present]              |
    |[stomach pain]                  |[absent]                        |
    |[an epidural, PCA, pain control]|[present, present, hypothetical]|
    +--------------------------------+--------------------------------+
    See also

    AssertionDLApproach for training a custom AssertionDLModel

    AssertionLogRegModel for non deep learning based extraction

  3. trait ReadablePretrainedAssertionDL extends ParamsAndFeaturesReadable[AssertionDLModel] with HasPretrained[AssertionDLModel]
  4. trait ReadsAssertionGraph extends ParamsAndFeaturesReadable[AssertionDLModel] with ReadTensorflowModel

Value Members

  1. object AssertionDLApproach extends DefaultParamsReadable[AssertionDLApproach] with Serializable
  2. object AssertionDLModel extends ReadsAssertionGraph with ReadablePretrainedAssertionDL with Serializable

Ungrouped