Class/Object

com.johnsnowlabs.nlp.annotators.resolution

EnsembleEntityResolverModel

Related Docs: object EnsembleEntityResolverModel | package resolution

Permalink

class EnsembleEntityResolverModel extends Model[EnsembleEntityResolverModel] with RawAnnotator[EnsembleEntityResolverModel] with EnsembleModelResolverParams with EnsembleModelParams with StringFunctions with CanBeLazy with Licensed

Contains all the parameters to transform a dataset with two Input Annotations of types TOKEN and WORD_EMBEDDINGS, coming from ChunkTokenizer and ChunkEmbeddings Annotators and return the Normalized Entity for a particular trained ontology / curated dataset.

This EnsembleChunkEntityResolverModel consists of two layers: First a TFIDF + OvrLogRegClassifier on top of the TOKEN Annotations Second a set of ChunkEntityResolversModels, one per each different class from the first layer

This architecture allows Spark NLP's Entity Resolution Architecture to scale to a few millions of rows [codes]

Linear Supertypes
Licensed, CanBeLazy, StringFunctions, EnsembleModelParams, HasFeaturesJsl, EnsembleModelResolverParams, RawAnnotator[EnsembleEntityResolverModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[EnsembleEntityResolverModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. EnsembleEntityResolverModel
  2. Licensed
  3. CanBeLazy
  4. StringFunctions
  5. EnsembleModelParams
  6. HasFeaturesJsl
  7. EnsembleModelResolverParams
  8. RawAnnotator
  9. HasOutputAnnotationCol
  10. HasInputAnnotationCols
  11. HasOutputAnnotatorType
  12. ParamsAndFeaturesWritable
  13. HasFeatures
  14. DefaultParamsWritable
  15. MLWritable
  16. Model
  17. Transformer
  18. PipelineStage
  19. Logging
  20. Params
  21. Serializable
  22. Serializable
  23. Identifiable
  24. AnyRef
  25. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new EnsembleEntityResolverModel()

    Permalink
  2. new EnsembleEntityResolverModel(uid: String)

    Permalink

    uid

    a unique identifier for the instanced Annotator

Type Members

  1. type AnnotatorType = String

    Permalink
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[K <: String, V <: ChunkEntityResolverModel](feature: ResolverMapFeature[K, V]): Map[K, V]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeaturesJsl
  5. def $$[T](feature: StructFeature[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: SetFeature[T]): Set[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. def $$[T](feature: ArrayFeature[T]): Array[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  9. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  10. val alternatives: IntParam

    Permalink

    number of results to return in the metadata after sorting by last distance calculated

    number of results to return in the metadata after sorting by last distance calculated

    Definition Classes
    EnsembleModelResolverParams
  11. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  12. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  13. lazy val classCol: String

    Permalink
    Definition Classes
    EnsembleModelParams
  14. lazy val classificationPipeline: Pipeline

    Permalink
    Definition Classes
    EnsembleModelParams
  15. lazy val classifierLabels: Array[String]

    Permalink
    Definition Classes
    EnsembleModelParams
  16. final def clear(param: Param[_]): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    Params
  17. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. def copy(extra: ParamMap): EnsembleEntityResolverModel

    Permalink
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  19. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  20. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  21. val distanceFunction: Param[String]

    Permalink

    what distance function to use for KNN: 'EUCLIDEAN' or 'COSINE'

    what distance function to use for KNN: 'EUCLIDEAN' or 'COSINE'

    Definition Classes
    EnsembleModelResolverParams
  22. val distanceWeights: DoubleArrayParam

    Permalink

    distance weights to apply before pooling: [WMD, TFIDF, Jaccard, SorensenDice, JaroWinkler, Levenshtein]

    distance weights to apply before pooling: [WMD, TFIDF, Jaccard, SorensenDice, JaroWinkler, Levenshtein]

    Definition Classes
    EnsembleModelResolverParams
  23. lazy val docCol: String

    Permalink
    Definition Classes
    EnsembleModelParams
  24. lazy val embeddingsCol: String

    Permalink
    Definition Classes
    EnsembleModelParams
  25. val enableJaccard: BooleanParam

    Permalink

    whether or not to use Jaccard token distance

    whether or not to use Jaccard token distance

    Definition Classes
    EnsembleModelResolverParams
  26. val enableJaroWinkler: BooleanParam

    Permalink

    whether or not to use Jaro-Winkler character distance

    whether or not to use Jaro-Winkler character distance

    Definition Classes
    EnsembleModelResolverParams
  27. val enableLevenshtein: BooleanParam

    Permalink

    whether or not to use Levenshtein character distance

    whether or not to use Levenshtein character distance

    Definition Classes
    EnsembleModelResolverParams
  28. val enableSorensenDice: BooleanParam

    Permalink

    whether or not to use Sorensen-Dice token distance

    whether or not to use Sorensen-Dice token distance

    Definition Classes
    EnsembleModelResolverParams
  29. val enableTfidf: BooleanParam

    Permalink

    whether or not to use TFIDF token distance

    whether or not to use TFIDF token distance

    Definition Classes
    EnsembleModelResolverParams
  30. val enableWmd: BooleanParam

    Permalink

    whether or not to use WMD token distance

    whether or not to use WMD token distance

    Definition Classes
    EnsembleModelResolverParams
  31. val entityResolverModels: ResolverMapFeature[String, ChunkEntityResolverModel]

    Permalink

    Map of trained ClassifierLabel -> ChunkEntityResolverModels

    Map of trained ClassifierLabel -> ChunkEntityResolverModels

    Definition Classes
    EnsembleModelParams
  32. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  33. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  34. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  35. def explainParams(): String

    Permalink
    Definition Classes
    Params
  36. def extraValidate(structType: StructType): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  37. def extraValidateMsg: String

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  38. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  39. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  40. val extramassPenalty: DoubleParam

    Permalink

    penalty for extra words in the knowledge base match during WMD calculation

    penalty for extra words in the knowledge base match during WMD calculation

    Definition Classes
    EnsembleModelResolverParams
  41. val features: ArrayBuffer[Feature[_, _, _]]

    Permalink
    Definition Classes
    HasFeaturesJsl → HasFeatures
  42. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  43. def get[K <: String, V <: ChunkEntityResolverModel](feature: ResolverMapFeature[K, V]): Option[Map[K, V]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeaturesJsl
  44. def get[T](feature: StructFeature[T]): Option[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  45. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  46. def get[T](feature: SetFeature[T]): Option[Set[T]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  47. def get[T](feature: ArrayFeature[T]): Option[Array[T]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  48. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  49. def getAlternatives: Int

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  50. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  51. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  52. def getDistanceFunction: String

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  53. def getDistanceWeights: Array[Double]

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  54. def getEnableJaccard: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  55. def getEnableJaroWinkler: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  56. def getEnableLevenshtein: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  57. def getEnableSorensenDice: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  58. def getEnableTfidf: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  59. def getEnableWmd: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  60. def getEntityResolverModels: Map[String, ChunkEntityResolverModel]

    Permalink
    Definition Classes
    EnsembleModelParams
  61. def getExtramassPenalty: Double

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  62. def getHashingTFModel: HashingTF

    Permalink
    Definition Classes
    EnsembleModelParams
  63. def getIdfModel: IDFModel

    Permalink
    Definition Classes
    EnsembleModelParams
  64. def getInputCols: Array[String]

    Permalink
    Definition Classes
    HasInputAnnotationCols
  65. def getLabels: Array[String]

    Permalink
    Definition Classes
    EnsembleModelParams
  66. def getLazyAnnotator: Boolean

    Permalink
    Definition Classes
    CanBeLazy
  67. def getMergeChunks: Boolean

    Permalink
    Definition Classes
    EnsembleModelParams
  68. def getMissAsEmpty: Boolean

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  69. def getNeighbours: Int

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  70. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  71. final def getOutputCol: String

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  72. def getOvrModel: OneVsRestModel

    Permalink
    Definition Classes
    EnsembleModelParams
  73. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  74. def getPoolingStrategy: String

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  75. def getThreshold: Double

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  76. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  77. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  78. def hasParent: Boolean

    Permalink
    Definition Classes
    Model
  79. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  80. val idfModel: StructFeature[IDFModel]

    Permalink

    Trained IDF Spark ML Model

    Trained IDF Spark ML Model

    Definition Classes
    EnsembleModelParams
  81. lazy val idfModelValue: IDFModel

    Permalink
    Definition Classes
    EnsembleModelParams
  82. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  83. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  84. val inputAnnotatorTypes: Array[String]

    Permalink

    inputAnnotatorTypes are TOKEN coming from ChunkTokenizer and WORD_EMBEDDINGS coming from ChunkEmbeddings

    inputAnnotatorTypes are TOKEN coming from ChunkTokenizer and WORD_EMBEDDINGS coming from ChunkEmbeddings

    Definition Classes
    EnsembleEntityResolverModel → HasInputAnnotationCols
  85. final val inputCols: StringArrayParam

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  86. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  87. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  88. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  89. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  90. def label2path(label: String): String

    Permalink

    Convenience function to use when naming folders after a string that eventually does not comply with filesystem requirements

    Convenience function to use when naming folders after a string that eventually does not comply with filesystem requirements

    label

    string with special characters to transform into lowercase letters and numbers

    returns

    lowercase letters and numbers replacing special characters with _

    Definition Classes
    StringFunctions
  91. val labels: StringArrayParam

    Permalink

    array used to output the label in the original form

    array used to output the label in the original form

    Definition Classes
    EnsembleModelParams
  92. val lazyAnnotator: BooleanParam

    Permalink
    Definition Classes
    CanBeLazy
  93. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  94. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  95. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  96. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  97. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  98. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  99. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  100. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  101. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  102. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  103. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  104. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  105. val mergeChunks: BooleanParam

    Permalink

    whether to merge all chunks in a document or not when training the Classifier

    whether to merge all chunks in a document or not when training the Classifier

    Definition Classes
    EnsembleModelParams
  106. val missAsEmpty: BooleanParam

    Permalink

    whether or not to return an empty annotation on unmatched chunks

    whether or not to return an empty annotation on unmatched chunks

    Definition Classes
    EnsembleModelResolverParams
  107. def msgHelper(schema: StructType): String

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  108. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  109. val neighbours: IntParam

    Permalink

    number of neighbours to consider in the KNN query to calculate WMD

    number of neighbours to consider in the KNN query to calculate WMD

    Definition Classes
    EnsembleModelResolverParams
  110. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  111. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  112. def onWrite(path: String, spark: SparkSession): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    ParamsAndFeaturesWritable
  113. val outputAnnotatorType: AnnotatorType

    Permalink

    oututAnnotatorType is ENTITY

    oututAnnotatorType is ENTITY

    Definition Classes
    EnsembleEntityResolverModel → HasOutputAnnotatorType
  114. final val outputCol: Param[String]

    Permalink
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  115. val ovrModel: StructFeature[OneVsRestModel]

    Permalink

    Trained OvrLogReg Spark ML Model

    Trained OvrLogReg Spark ML Model

    Definition Classes
    EnsembleModelParams
  116. lazy val ovrModelValue: OneVsRestModel

    Permalink
    Definition Classes
    EnsembleModelParams
  117. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  118. var parent: Estimator[EnsembleEntityResolverModel]

    Permalink
    Definition Classes
    Model
  119. val poolingStrategy: Param[String]

    Permalink

    pooling strategy to aggregate distances: AVERAGE or SUM

    pooling strategy to aggregate distances: AVERAGE or SUM

    Definition Classes
    EnsembleModelResolverParams
  120. lazy val resolverModels: Map[String, ChunkEntityResolverModel]

    Permalink
    Definition Classes
    EnsembleModelParams
  121. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  122. def set[K <: String, V <: ChunkEntityResolverModel](feature: ResolverMapFeature[K, V], value: Map[K, V]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeaturesJsl
  123. def set[T](feature: StructFeature[T], value: T): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  124. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  125. def set[T](feature: SetFeature[T], value: Set[T]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  126. def set[T](feature: ArrayFeature[T], value: Array[T]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  127. final def set(paramPair: ParamPair[_]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  128. final def set(param: String, value: Any): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  129. final def set[T](param: Param[T], value: T): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    Params
  130. def setAlternatives(a: Int): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  131. def setDefault[K <: String, V <: ChunkEntityResolverModel](feature: ResolverMapFeature[K, V], value: () ⇒ Map[K, V]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeaturesJsl
  132. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  133. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  134. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  135. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  136. final def setDefault(paramPairs: ParamPair[_]*): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  137. final def setDefault[T](param: Param[T], value: T): EnsembleEntityResolverModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  138. def setDistanceFunction(value: String): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  139. def setDistanceWeights(v: Array[Double]): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  140. def setEnableJaccard(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  141. def setEnableJaroWinkler(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  142. def setEnableLevenshtein(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  143. def setEnableSorensenDice(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  144. def setEnableTfidf(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  145. def setEnableWmd(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  146. def setEntityResolverModels(value: Map[String, ChunkEntityResolverModel]): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  147. def setExtramassPenalty(emp: Double): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  148. def setHashingTFModel(value: HashingTF): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  149. def setIdfModel(value: IDFModel): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  150. final def setInputCols(value: String*): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  151. final def setInputCols(value: Array[String]): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  152. def setLabels(value: Array[String]): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  153. def setLazyAnnotator(value: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    CanBeLazy
  154. def setMergeChunks(value: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  155. def setMissAsEmpty(v: Boolean): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  156. def setNeighbours(k: Int): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  157. final def setOutputCol(value: String): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  158. def setOvrModel(value: OneVsRestModel): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelParams
  159. def setParent(parent: Estimator[EnsembleEntityResolverModel]): EnsembleEntityResolverModel

    Permalink
    Definition Classes
    Model
  160. def setPoolingStrategy(value: String): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  161. def setThreshold(dist: Double): EnsembleEntityResolverModel.this.type

    Permalink
    Definition Classes
    EnsembleModelResolverParams
  162. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  163. val tfModel: StructFeature[HashingTF]

    Permalink

    HashingTF Spark ML Model for reference

    HashingTF Spark ML Model for reference

    Definition Classes
    EnsembleModelParams
  164. lazy val tfModelValue: HashingTF

    Permalink
    Definition Classes
    EnsembleModelParams
  165. val threshold: DoubleParam

    Permalink

    threshold value for the aggregated distance

    threshold value for the aggregated distance

    Definition Classes
    EnsembleModelResolverParams
  166. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  167. lazy val tokenAnnotationCol: String

    Permalink
    Definition Classes
    EnsembleModelParams
  168. lazy val tokenRawCol: String

    Permalink
    Definition Classes
    EnsembleModelParams
  169. def transform(dataset: Dataset[_]): DataFrame

    Permalink

    Transforms the input array of anything into the DataFrame this method will check upfront the input array before applying the projection pipeline also, it will resolve the appropriate projection before applying it

    Transforms the input array of anything into the DataFrame this method will check upfront the input array before applying the projection pipeline also, it will resolve the appropriate projection before applying it

    dataset

    dataset a Dataset containing ChunkTokens and ChunkEmbeddings

    returns

    a DataFrame, with the result of the entity resolution for each chunk and the following metadata all_k_results -> Sorted ResolverLabels in the top alternatives that match the distance threshold all_k_resolutions -> Respective ResolverNormalized strings all_k_distances -> Respective distance values after aggregation all_k_wmd_distances -> Respective WMD distance values all_k_tfidf_distances -> Respective TFIDF Cosinge distance values all_k_jaccard_distances -> Respective Jaccard distance values all_k_sorensen_distances -> Respective SorensenDice distance values all_k_jaro_distances -> Respective JaroWinkler distance values all_k_levenshtein_distances -> Respective Levenshtein distance values all_k_confidences -> Respective normalized probabilities based in inverse distance values all_k_confidence_ratios -> Convenience indicator calculated as the ratio between the i_th and the (i+1)_th probability target_text -> The actual searched string resolved_text -> The top ResolverNormalized string confidence -> Top probability confidence_ratio -> Top confidence ratio distance -> Top distance value sentence -> Sentence index chunk -> Chunk Index token -> Token index

    Definition Classes
    EnsembleEntityResolverModel → Transformer
  170. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  171. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  172. final def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    RawAnnotator → PipelineStage
  173. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  174. val uid: String

    Permalink

    a unique identifier for the instanced Annotator

    a unique identifier for the instanced Annotator

    Definition Classes
    EnsembleEntityResolverModel → Identifiable
  175. def validate(schema: StructType): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  176. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  177. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  178. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  179. def wrapColumnMetadata(col: Column): Column

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  180. def write: MLWriter

    Permalink
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable

Inherited from Licensed

Inherited from CanBeLazy

Inherited from StringFunctions

Inherited from EnsembleModelParams

Inherited from HasFeaturesJsl

Inherited from RawAnnotator[EnsembleEntityResolverModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[EnsembleEntityResolverModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Ungrouped