Type Members

1.  case class DistanceResult(distance: Double, weightedDistance: Double) extends Product with Serializable

   Class that contains distance in both representations: weighted and non-weighted, for using later in DistancePooling

2.  class JDataReader extends AnyRef
3.  case class JTreeComponent(embeddings: Array[Float], data: JTreeData) extends Product with Serializable
4.  case class JTreeData(code: String, trained: Array[String], normalized: String) extends Product with Serializable
5.  class JTreeReader extends StorageReader[JTreeComponent]
6.  class JTreeWriter extends StorageBatchWriter[JTreeComponent]
7.  trait ReadablePretrainedSentenceEntityResolver extends ParamsAndFeaturesReadable[SentenceEntityResolverModel] with HasPretrained[SentenceEntityResolverModel] with EvalEntityResolver
8.  class Resolution2Chunk extends AnnotatorModel[Resolution2Chunk] with HasSimpleAnnotate[Resolution2Chunk]

   A feature transformer that converts the input array of strings (annotatorType CHUNK) into an array of chunk-based tokens (annotatorType TOKEN).

   A feature transformer that converts the input array of strings (annotatorType CHUNK) into an array of chunk-based tokens (annotatorType TOKEN).

   When the input is empty, an empty array is returned.

   This Annotator is specially convenient when using NGramGenerator annotations as inputs to WordEmbeddingsModels

   Example

   Define a pipeline for generating n-grams

   val data = Seq(("A 63-year-old man presents to the hospital ...")).toDF("text")
   val document = new DocumentAssembler().setInputCol("text").setOutputCol("document")
   val sentenceDetector = new SentenceDetector().setInputCols("document").setOutputCol("sentence")
   val token = new Tokenizer().setInputCols("sentence").setOutputCol("token")
   val ngrammer = new NGramGenerator()
    .setN(2)
    .setEnableCumulative(false)
    .setInputCols("token")
    .setOutputCol("ngrams")
    .setDelimiter("_")

   Stage to convert n-gram CHUNKS to TOKEN type

   val chunk2Token = new Chunk2Token().setInputCols("ngrams").setOutputCol("ngram_tokens")
   val trainingPipeline = new Pipeline().setStages(Array(document, sentenceDetector, token, ngrammer, chunk2Token)).fit(data)
   
   val result = trainingPipeline.transform(data).cache()
   result.selectExpr("explode(ngram_tokens)").show(5, false)
     +----------------------------------------------------------------+
     |col                                                             |
     +----------------------------------------------------------------+
     |{token, 3, 15, A_63-year-old, {sentence -> 0, chunk -> 0}, []}  |
     |{token, 5, 19, 63-year-old_man, {sentence -> 0, chunk -> 1}, []}|
     |{token, 17, 28, man_presents, {sentence -> 0, chunk -> 2}, []}  |
     |{token, 21, 31, presents_to, {sentence -> 0, chunk -> 3}, []}   |
     |{token, 30, 35, to_the, {sentence -> 0, chunk -> 4}, []}        |
     +----------------------------------------------------------------+

   See also

   NGramGenerator

9.  class ResolverMerger extends AnnotatorModel[ResolverMerger] with HasSimpleAnnotate[ResolverMerger] with CheckLicense
10.  class SentenceEntityResolverApproach extends AnnotatorApproach[SentenceEntityResolverModel] with SentenceResolverParams with HasCaseSensitiveProperties with CheckLicense

    Contains all the parameters and methods to train a SentenceEntityResolverModel.

    Contains all the parameters and methods to train a SentenceEntityResolverModel. The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g. BertSentenceEmbeddings and returns the normalized entity for a particular trained ontology / curated dataset. (e.g. ICD-10, RxNorm, SNOMED etc.)

    To use pretrained models please use SentenceEntityResolverModel and see the Models Hub for available models.

    Example

    Training a SNOMED resolution model using BERT sentence embeddings

    Define pre-processing pipeline for training data. It needs consists of columns for the normalized training data and their labels.

    val documentAssembler = new DocumentAssembler()
       .setInputCol("normalized_text")
       .setOutputCol("document")
     val bertEmbeddings = BertSentenceEmbeddings.pretrained("sent_biobert_pubmed_base_cased")
       .setInputCols("sentence")
       .setOutputCol("bert_embeddings")
     val snomedTrainingPipeline = new Pipeline().setStages(Array(
       documentAssembler,
       bertEmbeddings
     ))
     val snomedTrainingModel = snomedTrainingPipeline.fit(data)
     val snomedData = snomedTrainingModel.transform(data).cache()

    Then the Resolver can be trained with

    val bertExtractor = new SentenceEntityResolverApproach()
      .setNeighbours(25)
      .setThreshold(1000)
      .setInputCols("bert_embeddings")
      .setNormalizedCol("normalized_text")
      .setLabelCol("label")
      .setOutputCol("snomed_code")
      .setDistanceFunction("EUCLIDIAN")
      .setCaseSensitive(false)
    
    val snomedModel = bertExtractor.fit(snomedData)

    See also

    SentenceEntityResolverModel

11.  class SentenceEntityResolverModel extends AnnotatorModel[SentenceEntityResolverModel] with SentenceResolverParams with HasStorageModel with HasEmbeddingsProperties with HasCaseSensitiveProperties with HasSimpleAnnotate[SentenceEntityResolverModel] with CheckLicense

    The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g.

    The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g. BertSentenceEmbeddings and returns the normalized entity for a particular trained ontology / curated dataset. (e.g. ICD-10, RxNorm, SNOMED etc.)

    To use pretrained models please see the Models Hub for available models.

    Example

    Resolving CPT

    First define pipeline stages to extract entities

    val documentAssembler = new DocumentAssembler()
      .setInputCol("text")
      .setOutputCol("document")
    val sentenceDetector = SentenceDetectorDLModel.pretrained()
      .setInputCols("document")
      .setOutputCol("sentence")
    val tokenizer = new Tokenizer()
      .setInputCols("sentence")
      .setOutputCol("token")
    val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
      .setInputCols("sentence", "token")
      .setOutputCol("embeddings")
    val clinical_ner = MedicalNerModel.pretrained("jsl_ner_wip_clinical", "en", "clinical/models")
      .setInputCols("sentence", "token", "embeddings")
      .setOutputCol("ner")
    val ner_converter = new NerConverter()
      .setInputCols("sentence", "token", "ner")
      .setOutputCol("ner_chunk")
      .setWhiteList("Test","Procedure")
    val c2doc = new Chunk2Doc()
      .setInputCols("ner_chunk")
      .setOutputCol("ner_chunk_doc")
    val sbert_embedder = BertSentenceEmbeddings
      .pretrained("sbiobert_base_cased_mli","en","clinical/models")
      .setInputCols("ner_chunk_doc")
      .setOutputCol("sbert_embeddings")

    Then the resolver is defined on the extracted entities and sentence embeddings

    val cpt_resolver = SentenceEntityResolverModel.pretrained("sbiobertresolve_cpt_procedures_augmented","en", "clinical/models")
      .setInputCols("sbert_embeddings")
      .setOutputCol("cpt_code")
      .setDistanceFunction("EUCLIDEAN")
    val sbert_pipeline_cpt = new Pipeline().setStages(Array(
      documentAssembler,
      sentenceDetector,
      tokenizer,
      word_embeddings,
      clinical_ner,
      ner_converter,
      c2doc,
      sbert_embedder,
      cpt_resolver))

    Show results

    sbert_outputs
      .select("explode(arrays_zip(ner_chunk.result ,ner_chunk.metadata, cpt_code.result, cpt_code.metadata, ner_chunk.begin, ner_chunk.end)) as cpt_code")
      .selectExpr(
        "cpt_code['0'] as chunk",
        "cpt_code['1'].entity as entity",
        "cpt_code['2'] as code",
        "cpt_code['3'].confidence as confidence",
        "cpt_code['3'].all_k_resolutions as all_k_resolutions",
        "cpt_code['3'].all_k_results as all_k_results"
      ).show(5)
    +--------------------+---------+-----+----------+--------------------+--------------------+
    |               chunk|   entity| code|confidence|   all_k_resolutions|         all_k_codes|
    +--------------------+---------+-----+----------+--------------------+--------------------+
    |          heart cath|Procedure|93566|    0.1180|CCA - Cardiac cat...|93566:::62319:::9...|
    |selective coronar...|     Test|93460|    0.1000|Coronary angiogra...|93460:::93458:::9...|
    |common femoral an...|     Test|35884|    0.1808|Femoral artery by...|35884:::35883:::3...|
    |   StarClose closure|Procedure|33305|    0.1197|Heart closure:::H...|33305:::33300:::3...|
    |         stress test|     Test|93351|    0.2795|Cardiovascular st...|93351:::94621:::9...|
    +--------------------+---------+-----+----------+--------------------+--------------------+

    See also

    SentenceEntityResolverApproach for training a custom model

12.  case class TreeData(code: String, trained: Array[String], normalized: String) extends Product with Serializable