Packages

package resolution

Ordering
  1. Alphabetic
Visibility
  1. Public
  2. All

Type Members

  1. case class DistanceResult(distance: Double, weightedDistance: Double) extends Product with Serializable

    Class that contains distance in both representations: weighted and non-weighted, for using later in DistancePooling

  2. class JDataReader extends AnyRef
  3. case class JTreeComponent(embeddings: Array[Float], data: JTreeData) extends Product with Serializable
  4. case class JTreeData(code: String, trained: Array[String], normalized: String) extends Product with Serializable
  5. class JTreeReader extends StorageReader[JTreeComponent]
  6. class JTreeWriter extends StorageBatchWriter[JTreeComponent]
  7. trait ReadablePretrainedSentenceEntityResolver extends ParamsAndFeaturesReadable[SentenceEntityResolverModel] with HasPretrained[SentenceEntityResolverModel] with EvalEntityResolver
  8. class Resolution2Chunk extends AnnotatorModel[Resolution2Chunk] with HasSimpleAnnotate[Resolution2Chunk]

    A feature transformer that converts the input array of strings (annotatorType CHUNK) into an array of chunk-based tokens (annotatorType TOKEN).

    A feature transformer that converts the input array of strings (annotatorType CHUNK) into an array of chunk-based tokens (annotatorType TOKEN).

    When the input is empty, an empty array is returned.

    This Annotator is specially convenient when using NGramGenerator annotations as inputs to WordEmbeddingsModels

    Example

    Define a pipeline for generating n-grams

    val data = Seq(("A 63-year-old man presents to the hospital ...")).toDF("text")
    val document = new DocumentAssembler().setInputCol("text").setOutputCol("document")
    val sentenceDetector = new SentenceDetector().setInputCols("document").setOutputCol("sentence")
    val token = new Tokenizer().setInputCols("sentence").setOutputCol("token")
    val ngrammer = new NGramGenerator()
     .setN(2)
     .setEnableCumulative(false)
     .setInputCols("token")
     .setOutputCol("ngrams")
     .setDelimiter("_")

    Stage to convert n-gram CHUNKS to TOKEN type

    val chunk2Token = new Chunk2Token().setInputCols("ngrams").setOutputCol("ngram_tokens")
    val trainingPipeline = new Pipeline().setStages(Array(document, sentenceDetector, token, ngrammer, chunk2Token)).fit(data)
    
    val result = trainingPipeline.transform(data).cache()
    result.selectExpr("explode(ngram_tokens)").show(5, false)
      +----------------------------------------------------------------+
      |col                                                             |
      +----------------------------------------------------------------+
      |{token, 3, 15, A_63-year-old, {sentence -> 0, chunk -> 0}, []}  |
      |{token, 5, 19, 63-year-old_man, {sentence -> 0, chunk -> 1}, []}|
      |{token, 17, 28, man_presents, {sentence -> 0, chunk -> 2}, []}  |
      |{token, 21, 31, presents_to, {sentence -> 0, chunk -> 3}, []}   |
      |{token, 30, 35, to_the, {sentence -> 0, chunk -> 4}, []}        |
      +----------------------------------------------------------------+
    See also

    NGramGenerator

  9. class ResolverMerger extends AnnotatorModel[ResolverMerger] with HasSimpleAnnotate[ResolverMerger] with CheckLicense
  10. class SentenceEntityResolverApproach extends AnnotatorApproach[SentenceEntityResolverModel] with SentenceResolverParams with HasCaseSensitiveProperties with CheckLicense

    Contains all the parameters and methods to train a SentenceEntityResolverModel.

    Contains all the parameters and methods to train a SentenceEntityResolverModel. The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g. BertSentenceEmbeddings and returns the normalized entity for a particular trained ontology / curated dataset. (e.g. ICD-10, RxNorm, SNOMED etc.)

    To use pretrained models please use SentenceEntityResolverModel and see the Models Hub for available models.

    Example

    Training a SNOMED resolution model using BERT sentence embeddings

    Define pre-processing pipeline for training data. It needs consists of columns for the normalized training data and their labels.

    val documentAssembler = new DocumentAssembler()
       .setInputCol("normalized_text")
       .setOutputCol("document")
     val bertEmbeddings = BertSentenceEmbeddings.pretrained("sent_biobert_pubmed_base_cased")
       .setInputCols("sentence")
       .setOutputCol("bert_embeddings")
     val snomedTrainingPipeline = new Pipeline().setStages(Array(
       documentAssembler,
       bertEmbeddings
     ))
     val snomedTrainingModel = snomedTrainingPipeline.fit(data)
     val snomedData = snomedTrainingModel.transform(data).cache()

    Then the Resolver can be trained with

    val bertExtractor = new SentenceEntityResolverApproach()
      .setNeighbours(25)
      .setThreshold(1000)
      .setInputCols("bert_embeddings")
      .setNormalizedCol("normalized_text")
      .setLabelCol("label")
      .setOutputCol("snomed_code")
      .setDistanceFunction("EUCLIDIAN")
      .setCaseSensitive(false)
    
    val snomedModel = bertExtractor.fit(snomedData)
    See also

    SentenceEntityResolverModel

  11. class SentenceEntityResolverModel extends AnnotatorModel[SentenceEntityResolverModel] with SentenceResolverParams with HasStorageModel with HasEmbeddingsProperties with HasCaseSensitiveProperties with HasSimpleAnnotate[SentenceEntityResolverModel] with CheckLicense

    The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g.

    The model transforms a dataset with Input Annotation type SENTENCE_EMBEDDINGS, coming from e.g. BertSentenceEmbeddings and returns the normalized entity for a particular trained ontology / curated dataset. (e.g. ICD-10, RxNorm, SNOMED etc.)

    To use pretrained models please see the Models Hub for available models.

    Example

    Resolving CPT

    First define pipeline stages to extract entities

    val documentAssembler = new DocumentAssembler()
      .setInputCol("text")
      .setOutputCol("document")
    val sentenceDetector = SentenceDetectorDLModel.pretrained()
      .setInputCols("document")
      .setOutputCol("sentence")
    val tokenizer = new Tokenizer()
      .setInputCols("sentence")
      .setOutputCol("token")
    val word_embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
      .setInputCols("sentence", "token")
      .setOutputCol("embeddings")
    val clinical_ner = MedicalNerModel.pretrained("jsl_ner_wip_clinical", "en", "clinical/models")
      .setInputCols("sentence", "token", "embeddings")
      .setOutputCol("ner")
    val ner_converter = new NerConverter()
      .setInputCols("sentence", "token", "ner")
      .setOutputCol("ner_chunk")
      .setWhiteList("Test","Procedure")
    val c2doc = new Chunk2Doc()
      .setInputCols("ner_chunk")
      .setOutputCol("ner_chunk_doc")
    val sbert_embedder = BertSentenceEmbeddings
      .pretrained("sbiobert_base_cased_mli","en","clinical/models")
      .setInputCols("ner_chunk_doc")
      .setOutputCol("sbert_embeddings")

    Then the resolver is defined on the extracted entities and sentence embeddings

    val cpt_resolver = SentenceEntityResolverModel.pretrained("sbiobertresolve_cpt_procedures_augmented","en", "clinical/models")
      .setInputCols("sbert_embeddings")
      .setOutputCol("cpt_code")
      .setDistanceFunction("EUCLIDEAN")
    val sbert_pipeline_cpt = new Pipeline().setStages(Array(
      documentAssembler,
      sentenceDetector,
      tokenizer,
      word_embeddings,
      clinical_ner,
      ner_converter,
      c2doc,
      sbert_embedder,
      cpt_resolver))

    Show results

    sbert_outputs
      .select("explode(arrays_zip(ner_chunk.result ,ner_chunk.metadata, cpt_code.result, cpt_code.metadata, ner_chunk.begin, ner_chunk.end)) as cpt_code")
      .selectExpr(
        "cpt_code['0'] as chunk",
        "cpt_code['1'].entity as entity",
        "cpt_code['2'] as code",
        "cpt_code['3'].confidence as confidence",
        "cpt_code['3'].all_k_resolutions as all_k_resolutions",
        "cpt_code['3'].all_k_results as all_k_results"
      ).show(5)
    +--------------------+---------+-----+----------+--------------------+--------------------+
    |               chunk|   entity| code|confidence|   all_k_resolutions|         all_k_codes|
    +--------------------+---------+-----+----------+--------------------+--------------------+
    |          heart cath|Procedure|93566|    0.1180|CCA - Cardiac cat...|93566:::62319:::9...|
    |selective coronar...|     Test|93460|    0.1000|Coronary angiogra...|93460:::93458:::9...|
    |common femoral an...|     Test|35884|    0.1808|Femoral artery by...|35884:::35883:::3...|
    |   StarClose closure|Procedure|33305|    0.1197|Heart closure:::H...|33305:::33300:::3...|
    |         stress test|     Test|93351|    0.2795|Cardiovascular st...|93351:::94621:::9...|
    +--------------------+---------+-----+----------+--------------------+--------------------+
    See also

    SentenceEntityResolverApproach for training a custom model

  12. case class TreeData(code: String, trained: Array[String], normalized: String) extends Product with Serializable

Value Members

  1. object ConfidenceFunction

    Helper object to use while setting confidenceFunction parameter

  2. object DistanceFunction

    Helper object to use while setting distanceFunction parameter

  3. object PoolingStrategy

    Helper object to use while setting poolingStrategy parameter

  4. object Resolution2Chunk extends DefaultParamsReadable[Resolution2Chunk] with Serializable
  5. object ResolverMerger extends ParamsAndFeaturesReadable[ResolverMerger] with Serializable
  6. object SentenceEntityResolverModel extends ReadablePretrainedSentenceEntityResolver with Serializable

Ungrouped