Class/Object

com.johnsnowlabs.nlp.embeddings

WordEmbeddingsModel

Related Docs: object WordEmbeddingsModel | package embeddings

Permalink

class WordEmbeddingsModel extends AnnotatorModel[WordEmbeddingsModel] with HasEmbeddingsProperties with HasStorageModel with ParamsAndFeaturesWritable

Word Embeddings lookup annotator that maps tokens to vectors

See https://github.com/JohnSnowLabs/spark-nlp/blob/master/src/test/scala/com/johnsnowlabs/nlp/annotators/ner/NerConverterTest.scala for example usage of this API.

Linear Supertypes
HasStorageModel, HasExcludableStorage, HasStorageReader, HasCaseSensitiveProperties, HasStorageRef, HasEmbeddingsProperties, AnnotatorModel[WordEmbeddingsModel], CanBeLazy, RawAnnotator[WordEmbeddingsModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[WordEmbeddingsModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. WordEmbeddingsModel
  2. HasStorageModel
  3. HasExcludableStorage
  4. HasStorageReader
  5. HasCaseSensitiveProperties
  6. HasStorageRef
  7. HasEmbeddingsProperties
  8. AnnotatorModel
  9. CanBeLazy
  10. RawAnnotator
  11. HasOutputAnnotationCol
  12. HasInputAnnotationCols
  13. HasOutputAnnotatorType
  14. ParamsAndFeaturesWritable
  15. HasFeatures
  16. DefaultParamsWritable
  17. MLWritable
  18. Model
  19. Transformer
  20. PipelineStage
  21. Logging
  22. Params
  23. Serializable
  24. Serializable
  25. Identifiable
  26. AnyRef
  27. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new WordEmbeddingsModel()

    Permalink

    Annotator reference id.

    Annotator reference id. Used to identify elements in metadata or to refer to this annotator type

  2. new WordEmbeddingsModel(uid: String)

    Permalink

Type Members

  1. type AnnotationContent = Seq[Row]

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  2. type AnnotatorType = String

    Permalink
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[T](feature: StructFeature[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  5. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[T](feature: SetFeature[T]): Set[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: ArrayFeature[T]): Array[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  9. def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  10. def afterAnnotate(dataset: DataFrame): DataFrame

    Permalink
    Attributes
    protected
    Definition Classes
    WordEmbeddingsModel → AnnotatorModel
  11. def annotate(annotations: Seq[Annotation]): Seq[Annotation]

    Permalink

    takes a document and annotations and produces new annotations of this annotator's annotation type

    takes a document and annotations and produces new annotations of this annotator's annotation type

    annotations

    Annotations that correspond to inputAnnotationCols generated by previous annotators if any

    returns

    any number of annotations processed for every input annotation. Not necessary one to one relationship

    Definition Classes
    WordEmbeddingsModel → AnnotatorModel
  12. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  13. def beforeAnnotate(dataset: Dataset[_]): Dataset[_]

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  14. val caseSensitive: BooleanParam

    Permalink
    Definition Classes
    HasCaseSensitiveProperties
  15. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  16. final def clear(param: Param[_]): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    Params
  17. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  18. def copy(extra: ParamMap): WordEmbeddingsModel

    Permalink
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  19. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  20. def createDatabaseConnection(database: Name): RocksDBConnection

    Permalink
    Definition Classes
    HasStorageRef
  21. def createReader(database: Name, connection: RocksDBConnection): WordEmbeddingsReader

    Permalink
    Attributes
    protected
    Definition Classes
    WordEmbeddingsModel → HasStorageReader
  22. val databases: Array[Name]

    Permalink
    Definition Classes
    WordEmbeddingsModel → HasStorageModel
  23. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  24. def deserializeStorage(path: String, spark: SparkSession): Unit

    Permalink
    Definition Classes
    HasStorageModel
  25. def dfAnnotate: UserDefinedFunction

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  26. val dimension: IntParam

    Permalink
    Definition Classes
    HasEmbeddingsProperties
  27. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  28. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  29. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  30. def explainParams(): String

    Permalink
    Definition Classes
    Params
  31. def extraValidate(structType: StructType): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  32. def extraValidateMsg: String

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  33. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  34. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  35. val features: ArrayBuffer[Feature[_, _, _]]

    Permalink
    Definition Classes
    HasFeatures
  36. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  37. def get[T](feature: StructFeature[T]): Option[T]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  38. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  39. def get[T](feature: SetFeature[T]): Option[Set[T]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  40. def get[T](feature: ArrayFeature[T]): Option[Array[T]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  41. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  42. def getCaseSensitive: Boolean

    Permalink
    Definition Classes
    HasCaseSensitiveProperties
  43. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  44. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  45. def getDimension: Int

    Permalink
    Definition Classes
    HasEmbeddingsProperties
  46. def getIncludeStorage: Boolean

    Permalink
    Definition Classes
    HasExcludableStorage
  47. def getInputCols: Array[String]

    Permalink
    Definition Classes
    HasInputAnnotationCols
  48. def getLazyAnnotator: Boolean

    Permalink
    Definition Classes
    CanBeLazy
  49. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  50. final def getOutputCol: String

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  51. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  52. def getReader[A](database: Name): StorageReader[A]

    Permalink
    Attributes
    protected
    Definition Classes
    HasStorageReader
  53. def getStorageRef: String

    Permalink
    Definition Classes
    HasStorageRef
  54. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  55. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  56. def hasParent: Boolean

    Permalink
    Definition Classes
    Model
  57. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  58. val includeStorage: BooleanParam

    Permalink
    Definition Classes
    HasExcludableStorage
  59. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  60. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  61. val inputAnnotatorTypes: Array[String]

    Permalink

    Input annotator type : DOCUMENT, TOKEN

    Input annotator type : DOCUMENT, TOKEN

    Definition Classes
    WordEmbeddingsModel → HasInputAnnotationCols
  62. final val inputCols: StringArrayParam

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  63. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  64. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  65. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  66. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. val lazyAnnotator: BooleanParam

    Permalink
    Definition Classes
    CanBeLazy
  68. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  70. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  71. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  72. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  73. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  74. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  75. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  76. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  77. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  78. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  79. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  80. def msgHelper(schema: StructType): String

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  81. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  82. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  83. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  84. def onWrite(path: String, spark: SparkSession): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    HasStorageModel → ParamsAndFeaturesWritable
  85. val outputAnnotatorType: AnnotatorType

    Permalink

    Output annotator type : WORD_EMBEDDINGS

    Output annotator type : WORD_EMBEDDINGS

    Definition Classes
    WordEmbeddingsModel → HasOutputAnnotatorType
  86. final val outputCol: Param[String]

    Permalink
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  87. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  88. var parent: Estimator[WordEmbeddingsModel]

    Permalink
    Definition Classes
    Model
  89. val readCacheSize: IntParam

    Permalink

    cache size for items retrieved from storage.

    cache size for items retrieved from storage. Increase for performance but higher memory consumption

  90. var readers: Map[Name, StorageReader[_]]

    Permalink
    Attributes
    protected
    Definition Classes
    HasStorageReader
  91. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  92. def saveStorage(path: String, spark: SparkSession, withinStorage: Boolean): Unit

    Permalink
    Definition Classes
    HasStorageModel
  93. def serializeStorage(path: String, spark: SparkSession): Unit

    Permalink
    Definition Classes
    HasStorageModel
  94. def set[T](feature: StructFeature[T], value: T): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  95. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  96. def set[T](feature: SetFeature[T], value: Set[T]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  97. def set[T](feature: ArrayFeature[T], value: Array[T]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  98. final def set(paramPair: ParamPair[_]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  99. final def set(param: String, value: Any): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  100. final def set[T](param: Param[T], value: T): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    Params
  101. def setCaseSensitive(value: Boolean): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasCaseSensitiveProperties
  102. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  103. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  104. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  105. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    HasFeatures
  106. final def setDefault(paramPairs: ParamPair[_]*): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  107. final def setDefault[T](param: Param[T], value: T): WordEmbeddingsModel.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  108. def setDimension(value: Int): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasEmbeddingsProperties
  109. def setIncludeStorage(value: Boolean): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasExcludableStorage
  110. final def setInputCols(value: String*): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  111. final def setInputCols(value: Array[String]): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  112. def setLazyAnnotator(value: Boolean): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    CanBeLazy
  113. final def setOutputCol(value: String): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  114. def setParent(parent: Estimator[WordEmbeddingsModel]): WordEmbeddingsModel

    Permalink
    Definition Classes
    Model
  115. def setReadCacheSize(value: Int): WordEmbeddingsModel.this.type

    Permalink

    Set cache size for items retrieved from storage.

    Set cache size for items retrieved from storage. Increase for performance but higher memory consumption

  116. def setStorageRef(value: String): WordEmbeddingsModel.this.type

    Permalink
    Definition Classes
    HasStorageRef
  117. val storageRef: Param[String]

    Permalink
    Definition Classes
    HasStorageRef
  118. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  119. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  120. final def transform(dataset: Dataset[_]): DataFrame

    Permalink
    Definition Classes
    AnnotatorModel → Transformer
  121. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  122. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

    Permalink
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  123. final def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    RawAnnotator → PipelineStage
  124. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  125. val uid: String

    Permalink
    Definition Classes
    WordEmbeddingsModel → Identifiable
  126. def validate(schema: StructType): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  127. def validateStorageRef(dataset: Dataset[_], inputCols: Array[String], annotatorType: String): Unit

    Permalink
    Definition Classes
    HasStorageRef
  128. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  129. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  130. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  131. def wrapColumnMetadata(col: Column): Column

    Permalink
    Attributes
    protected
    Definition Classes
    RawAnnotator
  132. def wrapEmbeddingsMetadata(col: Column, embeddingsDim: Int, embeddingsRef: Option[String]): Column

    Permalink
    Attributes
    protected
    Definition Classes
    HasEmbeddingsProperties
  133. def wrapSentenceEmbeddingsMetadata(col: Column, embeddingsDim: Int, embeddingsRef: Option[String]): Column

    Permalink
    Attributes
    protected
    Definition Classes
    HasEmbeddingsProperties
  134. def write: MLWriter

    Permalink
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable

Inherited from HasStorageModel

Inherited from HasExcludableStorage

Inherited from HasStorageReader

Inherited from HasCaseSensitiveProperties

Inherited from HasStorageRef

Inherited from HasEmbeddingsProperties

Inherited from AnnotatorModel[WordEmbeddingsModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[WordEmbeddingsModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[WordEmbeddingsModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Required input and expected output annotator types

Members

Parameter setters