class EntityChunkEmbeddings extends BertSentenceEmbeddings with CheckLicense

Entity Chunk Embeddings uses BERT Sentence embeddings to compute a weighted average vector represention of related entity chunks. The input the model consists of chunks of recognized named entities. One or more entities are selected as target entities and for each of them a list of related entities is specified (if empty, all other entities are assumed to be related). The model looks for chunks of the target entities and then tries to pair each target entity (e.g. DRUG) with other related entities (e.g. DOSAGE, STRENGTH, FORM, etc). The criterion for pairing a target entity with another related entity is that they appear in the same sentence and the maximal syntactic distance is below a predefined threshold. The relationship between target and related entities is one-to-many, meaning that if there multiple instances of the same target entity (e.g.) within a sentence, the model will map a related entity (e.g. DOSAGE) to at most one of the instances of the target entity. For example, if there is a sentence "The patient was given 125 mg of paracetamol and metformin", the model will pair "125 mg" to "paracetamol", but not to "metformin". The output of the model is an average embeddings of the chunks of each of the target entities and their related entities. It is possible to specify a particular weight for each entity type. An entity can be defined both as target a entity and as a related entity for some other target entity. For example, we may want to compute the embeddings of SYMPTOMs and their related entities, as well as the embeddings of DRUGs and their related entities, one of each is also SYMPTOM. In such cases, it is possible to use the TARGET_ENTITY:RELATED_ENTITY notation to specify the weight of an related entity (e.g. "DRUG:SYMPTOM" to set the weight of SYMPTOM when it appears as an related entity to target entity DRUG). The relative weights of entities for particular entity chunk embeddings are available in the annotations metadata.

This model is a subclass of BertSentenceEmbeddings and shares all parameters with it. It can load any pretrained BertSentenceEmbeddings model. Available models can be found at Models Hub.

Two input columns are required - chunks and dependency annotations.

val embeddings = EntityChunkEmbeddings.pretrained()
  .setInputCols("sentence", "dependencies")
  .setOutputCol("entity_chunk_embeddings")

The default model is "sbiobert_base_cased_mli" from "clinical/models".

Sources :

BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks

Example

import spark.implicits._
import com.johnsnowlabs.nlp.base.DocumentAssembler
import com.johnsnowlabs.nlp.annotator.SentenceDetector
import com.johnsnowlabs.nlp.annotators.parser.dep.DependencyParserModel
import com.johnsnowlabs.nlp.annotators.pos.perceptron.PerceptronModel
import com.johnsnowlabs.nlp.annotators.ner.{MedicalNerModel, NerConverterInternal}
import com.johnsnowlabs.nlp.annotators.embeddings.EntityChunkEmbeddings
import org.apache.spark.ml.Pipeline

val documentAssembler = new DocumentAssembler()
   .setInputCol("text")
   .setOutputCol("document")

val sentenceDetector = new SentenceDetector()
   .setInputCols("document")
   .setOutputCol("sentence")

val tokenizer = new Tokenizer()
   .setInputCols("sentence")
   .setOutputCol("tokens")

 val wordEmbeddings = WordEmbeddingsModel
   .pretrained("embeddings_clinical", "en", "clinical/models")
   .setInputCols(Array("sentences", "tokens"))
   .setOutputCol("word_embeddings")

 val nerModel = MedicalNerModel
   .pretrained("ner_posology_large", "en", "clinical/models")
   .setInputCols(Array("sentence", "tokens", "word_embeddings"))
   .setOutputCol("ner")

 val nerConverter = new NerConverterInternal()
   .setInputCols("sentence", "tokens", "ner")
   .setOutputCol("ner_chunk")

 val posTager = PerceptronModel
   .pretrained("pos_clinical", "en", "clinical/models")
   .setInputCols("sentences", "tokens")
   .setOutputCol("pos_tags")

 val dependencyParser = DependencyParserModel
   .pretrained("dependency_conllu", "en")
   .setInputCols(Array("sentences", "pos_tags", "tokens"))
   .setOutputCol("dependencies")

 val drugChunkEmbeddings = EntityChunkEmbeddings
   .pretrained("sbiobert_base_cased_mli","en","clinical/models")
   .setInputCols(Array("ner_chunks", "dependencies"))
   .setOutputCol("drug_chunk_embeddings")
   .setMaxSyntacticDistance(3)
   .setTargetEntities(Map("DRUG" -> List()))
   .setEntityWeights(Map[String, Float]("DRUG" -> 0.8f, "STRENGTH" -> 0.2f, "DOSAGE" -> 0.2f, "FORM" -> 0.5f))

val pipeline = new Pipeline()
     .setStages(Array(
         documentAssembler,
         sentenceDetector,
         tokenizer,
         wordEmbeddings,
         nerModel,
         nerConverter,
         posTager,
         dependencyParser,
         drugChunkEmbeddings))

val sampleText = "The patient was given metformin 125 mg, 250 mg of coumadin and then one pill paracetamol."

val testDataset = Seq("").toDS.toDF("text")
val result = pipeline.fit(emptyDataset).transform(testDataset)

result
   .selectExpr("explode(drug_chunk_embeddings) AS drug_chunk")
   .selectExpr("drug_chunk.result", "slice(drug_chunk.embeddings, 1, 5) AS drugEmbedding")
   .show(truncate=false)

+-----------------------------+-----------------------------------------------------------------+
|                       result|                                                    drugEmbedding|
+-----------------------------+-----------------------------------------------------------------+
|metformin 125 mg             |[-0.267413, 0.07614058, -0.5620966, 0.83838946, 0.8911504]       |
|250 mg coumadin              |[0.22319649, -0.07094894, -0.6885556, 0.79176235, 0.82672405]    |
|one pill paracetamol          |[-0.10939768, -0.29242, -0.3574444, 0.3981813, 0.79609615]      |
+-----------------------------+----------------------------------------------------------------+
See also

BertEmbeddings for token-level embeddings

BertSentenceEmbeddings for sentence-level embeddings

Annotators Main Page for a list of transformer based embeddings

Linear Supertypes
CheckLicense, BertSentenceEmbeddings, HasEngine, HasCaseSensitiveProperties, HasStorageRef, HasEmbeddingsProperties, HasProtectedParams, WriteOnnxModel, WriteOpenvinoModel, WriteTensorflowModel, HasBatchedAnnotate[BertSentenceEmbeddings], AnnotatorModel[BertSentenceEmbeddings], CanBeLazy, RawAnnotator[BertSentenceEmbeddings], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[BertSentenceEmbeddings], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. EntityChunkEmbeddings
  2. CheckLicense
  3. BertSentenceEmbeddings
  4. HasEngine
  5. HasCaseSensitiveProperties
  6. HasStorageRef
  7. HasEmbeddingsProperties
  8. HasProtectedParams
  9. WriteOnnxModel
  10. WriteOpenvinoModel
  11. WriteTensorflowModel
  12. HasBatchedAnnotate
  13. AnnotatorModel
  14. CanBeLazy
  15. RawAnnotator
  16. HasOutputAnnotationCol
  17. HasInputAnnotationCols
  18. HasOutputAnnotatorType
  19. ParamsAndFeaturesWritable
  20. HasFeatures
  21. DefaultParamsWritable
  22. MLWritable
  23. Model
  24. Transformer
  25. PipelineStage
  26. Logging
  27. Params
  28. Serializable
  29. Serializable
  30. Identifiable
  31. AnyRef
  32. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new EntityChunkEmbeddings()
  2. new EntityChunkEmbeddings(uid: String)

Type Members

  1. type AnnotationContent = Seq[Row]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  2. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType
  3. implicit class ProtectedParam[T] extends Param[T]
    Definition Classes
    HasProtectedParams

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[T](feature: StructFeature[T]): T
    Attributes
    protected
    Definition Classes
    HasFeatures
  5. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[T](feature: SetFeature[T]): Set[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: ArrayFeature[T]): Array[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  10. def afterAnnotate(dataset: DataFrame): DataFrame
    Attributes
    protected
    Definition Classes
    BertSentenceEmbeddings → AnnotatorModel
  11. def areSameTargetEntityAnnotations(targetEntityAnno1: Annotation, targetEntityAnno2: Annotation): Boolean
    Attributes
    protected
  12. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  13. def averageEmbeddings(embeddings: Array[Array[Float]], weights: Array[Float]): Array[Float]
    Attributes
    protected
  14. def batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
    Definition Classes
    EntityChunkEmbeddings → BertSentenceEmbeddings → HasBatchedAnnotate
  15. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotate
  16. val batchSize: IntParam
    Definition Classes
    HasBatchedAnnotate
  17. def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  18. val caseSensitive: BooleanParam
    Definition Classes
    HasCaseSensitiveProperties
  19. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  20. def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
    Definition Classes
    CheckLicense
  21. def checkValidScope(scope: String): Unit
    Definition Classes
    CheckLicense
  22. def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  23. def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  24. final def clear(param: Param[_]): EntityChunkEmbeddings.this.type
    Definition Classes
    Params
  25. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  26. val configProtoBytes: IntArrayParam
    Definition Classes
    BertSentenceEmbeddings
  27. def copy(extra: ParamMap): BertSentenceEmbeddings
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  28. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  29. def createDatabaseConnection(database: Name): RocksDBConnection
    Definition Classes
    HasStorageRef
  30. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  31. val dimension: ProtectedParam[Int]
    Definition Classes
    HasEmbeddingsProperties
  32. val engine: Param[String]
    Definition Classes
    HasEngine
  33. val entityWeights: MapFeature[String, Float]

    The relative weights of drug related entities.

    The relative weights of drug related entities. If not set, all entities have equal weights. If the list is non-empty and some entity is not in it, then its weight is set to 0. The notation TARGET_ENTITY:RELATED_ENTITY can be used to specify the weight of a entity which is related to specific target entity (e.g. "DRUG:SYMPTOM" -> 0.3f). Entity names are case insensitive.

  34. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  35. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  36. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  37. def explainParams(): String
    Definition Classes
    Params
  38. def extraValidate(structType: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  39. def extraValidateMsg: String
    Attributes
    protected
    Definition Classes
    RawAnnotator
  40. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  41. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  42. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  43. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  44. def get[T](feature: StructFeature[T]): Option[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  45. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  46. def get[T](feature: SetFeature[T]): Option[Set[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  47. def get[T](feature: ArrayFeature[T]): Option[Array[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  48. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  49. def getAnnotationEmbeddingWeight(targetEntityAnno: Annotation, anno: Annotation): Float
    Attributes
    protected
  50. def getBatchSize: Int
    Definition Classes
    HasBatchedAnnotate
  51. def getCaseSensitive: Boolean
    Definition Classes
    HasCaseSensitiveProperties
  52. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  53. def getConfigProtoBytes: Option[Array[Byte]]
    Definition Classes
    BertSentenceEmbeddings
  54. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  55. def getDimension: Int
    Definition Classes
    HasEmbeddingsProperties
  56. def getEngine: String
    Definition Classes
    HasEngine
  57. def getEntityWeight(entity: String): Float
  58. def getEntityWeights: Map[String, Float]
  59. def getInputCols: Array[String]
    Definition Classes
    HasInputAnnotationCols
  60. def getIsLong: Boolean
    Definition Classes
    BertSentenceEmbeddings
  61. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  62. def getMaxSentenceLength: Int
    Definition Classes
    BertSentenceEmbeddings
  63. def getMaxSyntacticDistance: Int

    Maximal syntactic distance, as threshold (Default: 0)

  64. def getModelIfNotSet: Bert
    Definition Classes
    BertSentenceEmbeddings
  65. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  66. final def getOutputCol: String
    Definition Classes
    HasOutputAnnotationCol
  67. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  68. def getSignatures: Option[Map[String, String]]
    Definition Classes
    BertSentenceEmbeddings
  69. def getStorageRef: String
    Definition Classes
    HasStorageRef
  70. def getSyntacticDistance(anno1: Annotation, anno2: Annotation, deps: Array[Annotation]): Int
    Attributes
    protected
  71. def getTargetEntities: Map[String, List[String]]
  72. def getTargetEntity(entity: String): Option[List[String]]
  73. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  74. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  75. def hasParent: Boolean
    Definition Classes
    Model
  76. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  77. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  78. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  79. val inputAnnotatorTypes: Array[AnnotatorType]

    Input annotator types: CHUNK, DEPENDENCY

    Input annotator types: CHUNK, DEPENDENCY

    Definition Classes
    EntityChunkEmbeddings → BertSentenceEmbeddings → HasInputAnnotationCols
  80. final val inputCols: StringArrayParam
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  81. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  82. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  83. val isLong: ProtectedParam[Boolean]
    Definition Classes
    BertSentenceEmbeddings
  84. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  85. def isTargetEntityAnnotation(anno: Annotation): Boolean
    Attributes
    protected
  86. def isTargetEntityRelatedAnnotation(targetEntityAnno: Annotation, anno: Annotation): Boolean
    Attributes
    protected
  87. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  88. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  89. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  90. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  91. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  92. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  93. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  94. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  95. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  96. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  97. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  98. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  99. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  100. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  101. val maxSentenceLength: IntParam
    Definition Classes
    BertSentenceEmbeddings
  102. val maxSyntacticDistance: IntParam

    Maximal syntactic distance between the drug entity and the other drug related entities.

    Maximal syntactic distance between the drug entity and the other drug related entities. Default value is 2.

  103. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  104. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  105. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  106. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  107. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    BertSentenceEmbeddings → ParamsAndFeaturesWritable
  108. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  109. val outputAnnotatorType: AnnotatorType

    Output annotator types: CHUNK

    Output annotator types: CHUNK

    Definition Classes
    EntityChunkEmbeddings → BertSentenceEmbeddings → HasOutputAnnotatorType
  110. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  111. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  112. var parent: Estimator[BertSentenceEmbeddings]
    Definition Classes
    Model
  113. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  114. def sentenceEndTokenId: Int
    Definition Classes
    BertSentenceEmbeddings
  115. def sentenceStartTokenId: Int
    Definition Classes
    BertSentenceEmbeddings
  116. def set[T](param: ProtectedParam[T], value: T): EntityChunkEmbeddings.this.type
    Definition Classes
    HasProtectedParams
  117. def set[T](feature: StructFeature[T], value: T): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  118. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  119. def set[T](feature: SetFeature[T], value: Set[T]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  120. def set[T](feature: ArrayFeature[T], value: Array[T]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  121. final def set(paramPair: ParamPair[_]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    Params
  122. final def set(param: String, value: Any): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    Params
  123. final def set[T](param: Param[T], value: T): EntityChunkEmbeddings.this.type
    Definition Classes
    Params
  124. def setBatchSize(size: Int): EntityChunkEmbeddings.this.type
    Definition Classes
    HasBatchedAnnotate
  125. def setCaseSensitive(value: Boolean): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings → HasCaseSensitiveProperties
  126. def setConfigProtoBytes(bytes: Array[Int]): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  127. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  128. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  129. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  130. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  131. final def setDefault(paramPairs: ParamPair[_]*): EntityChunkEmbeddings.this.type
    Attributes
    protected
    Definition Classes
    Params
  132. final def setDefault[T](param: Param[T], value: T): EntityChunkEmbeddings.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  133. def setDimension(value: Int): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings → HasEmbeddingsProperties
  134. def setEntityWeights(w: HashMap[String, Double]): EntityChunkEmbeddings.this.type
  135. def setEntityWeights(weights: Map[String, Float]): EntityChunkEmbeddings.this.type

    Sets the wieght of the chunk embeddings relative to the sentence embeddings.

    Sets the wieght of the chunk embeddings relative to the sentence embeddings. The value should between 0 and 1.

  136. final def setInputCols(value: String*): EntityChunkEmbeddings.this.type
    Definition Classes
    HasInputAnnotationCols
  137. def setInputCols(value: Array[String]): EntityChunkEmbeddings.this.type
    Definition Classes
    HasInputAnnotationCols
  138. def setIsLong(value: Boolean): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  139. def setLazyAnnotator(value: Boolean): EntityChunkEmbeddings.this.type
    Definition Classes
    CanBeLazy
  140. def setMaxSentenceLength(value: Int): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  141. def setMaxSyntacticDistance(maxSyntacticDistance: Int): EntityChunkEmbeddings.this.type

    Set the ,aximal syntactic distance

  142. def setModelIfNotSet(spark: SparkSession, tensorflowWrapper: Option[TensorflowWrapper], onnxWrapper: Option[OnnxWrapper], openvinoWrapper: Option[OpenvinoWrapper]): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  143. final def setOutputCol(value: String): EntityChunkEmbeddings.this.type
    Definition Classes
    HasOutputAnnotationCol
  144. def setParent(parent: Estimator[BertSentenceEmbeddings]): BertSentenceEmbeddings
    Definition Classes
    Model
  145. def setSignatures(value: Map[String, String]): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  146. def setStorageRef(value: String): EntityChunkEmbeddings.this.type
    Definition Classes
    HasStorageRef
  147. def setTargetEntities(entities: HashMap[String, List[String]]): EntityChunkEmbeddings.this.type
  148. def setTargetEntities(entities: Map[String, List[String]]): EntityChunkEmbeddings.this.type
  149. def setVocabulary(value: Map[String, Int]): EntityChunkEmbeddings.this.type
    Definition Classes
    BertSentenceEmbeddings
  150. val signatures: MapFeature[String, String]
    Definition Classes
    BertSentenceEmbeddings
  151. val storageRef: Param[String]
    Definition Classes
    HasStorageRef
  152. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  153. val targetEntities: MapFeature[String, List[String]]

    The target entities mapped to lists of their related entities.

    The target entities mapped to lists of their related entities. A target entity with an empty list of related entities means all other entities are assumed to be related to it. Entity names are case insensitive.

  154. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  155. def tokenize(sentences: Seq[Sentence]): Seq[WordpieceTokenizedSentence]
    Definition Classes
    BertSentenceEmbeddings
  156. final def transform(dataset: Dataset[_]): DataFrame
    Definition Classes
    AnnotatorModel → Transformer
  157. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  158. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  159. final def transformSchema(schema: StructType): StructType
    Definition Classes
    RawAnnotator → PipelineStage
  160. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  161. val uid: String
    Definition Classes
    EntityChunkEmbeddings → BertSentenceEmbeddings → Identifiable
  162. def validate(schema: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  163. def validateStorageRef(dataset: Dataset[_], inputCols: Array[String], annotatorType: String): Unit
    Definition Classes
    HasStorageRef
  164. val vocabulary: MapFeature[String, Int]
    Definition Classes
    BertSentenceEmbeddings
  165. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  166. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  167. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  168. def wrapColumnMetadata(col: Column): Column
    Attributes
    protected
    Definition Classes
    RawAnnotator
  169. def wrapEmbeddingsMetadata(col: Column, embeddingsDim: Int, embeddingsRef: Option[String]): Column
    Attributes
    protected
    Definition Classes
    HasEmbeddingsProperties
  170. def wrapSentenceEmbeddingsMetadata(col: Column, embeddingsDim: Int, embeddingsRef: Option[String]): Column
    Attributes
    protected
    Definition Classes
    HasEmbeddingsProperties
  171. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  172. def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOnnxModel
  173. def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOnnxModel
  174. def writeOpenvinoModel(path: String, spark: SparkSession, openvinoWrapper: OpenvinoWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOpenvinoModel
  175. def writeOpenvinoModels(path: String, spark: SparkSession, ovWrappersWithNames: Seq[(OpenvinoWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOpenvinoModel
  176. def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit
    Definition Classes
    WriteTensorflowModel
  177. def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit
    Definition Classes
    WriteTensorflowModel
  178. def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit
    Definition Classes
    WriteTensorflowModel

Inherited from CheckLicense

Inherited from BertSentenceEmbeddings

Inherited from HasEngine

Inherited from HasCaseSensitiveProperties

Inherited from HasStorageRef

Inherited from HasEmbeddingsProperties

Inherited from HasProtectedParams

Inherited from WriteOnnxModel

Inherited from WriteOpenvinoModel

Inherited from WriteTensorflowModel

Inherited from HasBatchedAnnotate[BertSentenceEmbeddings]

Inherited from AnnotatorModel[BertSentenceEmbeddings]

Inherited from CanBeLazy

Inherited from RawAnnotator[BertSentenceEmbeddings]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[BertSentenceEmbeddings]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

anno

getParam

param

setParam

Ungrouped