com.johnsnowlabs.nlp.annotators.windowed

WindowedSentenceModel

Companion object WindowedSentenceModel

class WindowedSentenceModel extends AnnotatorModel[WindowedSentenceModel] with CheckLicense with HasSimpleAnnotate[WindowedSentenceModel] with WindowedParams with HasFeatures with FilteringParams

The WindowedSentenceModel class is used to combine a series of sentences based on specific window configurations. Returns the joined results after windowing its inputs

Example

  val documentAssembler = new DocumentAssembler()
    .setInputCol("text")
    .setOutputCol("document")
  val sentenceDetector = new SentenceDetector()
    .setInputCols("document")
    .setOutputCol("sentence")
  val windowedSentence = new WindowedSentenceModel()
    .setWindowSize(1)
    .setInputCols("sentence")
    .setOutputCol("windowed")
    .setGlueString(":::")
  val flattener = new Flattener()
    .setInputCols("five")

  val pipeline = new Pipeline().setStages(Array(
      documentAssembler,
      sentenceDetector,
      windowedSentence,
      flattener))
      val text = "A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years. " +
                "Two weeks prior to presentation respiratory tract infection . " +
                 "She was on for HTG . " +
                "She had been of presentation ." +
                "examination benign with no  or rigidity . "
  val data = spark.createDataFrame(Array(Tuple1(text))).toDF("text")

  val model = pipeline.fit(data).transform(data)
  model.show(false)
+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+
 |windowed_result                                                                                                                                                                       |five_begin|five_end|five_metadata_sentence|
 +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+
 |A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years.:::Two weeks prior to presentation respiratory tract infection .                       |0         |152     |0                     |
 |A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years.:::Two weeks prior to presentation respiratory tract infection .:::She was on for HTG .|0         |173     |1                     |
 |Two weeks prior to presentation respiratory tract infection .:::She was on for HTG .:::She had been of presentation .                                                             |92        |204     |2                     |
 |She was on for HTG .:::She had been of presentation .:::examination benign with no  or rigidity .                                                                                 |154       |245     |3                     |
 |She had been of presentation .:::examination benign with no  or rigidity .                                                                                                        |175       |245     |4                     |
 +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+

Linear Supertypes

FilteringParams, WindowedParams, HasSimpleAnnotate[WindowedSentenceModel], CheckLicense, AnnotatorModel[WindowedSentenceModel], CanBeLazy, RawAnnotator[WindowedSentenceModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[WindowedSentenceModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

WindowedSentenceModel
FilteringParams
WindowedParams
HasSimpleAnnotate
CheckLicense
AnnotatorModel
CanBeLazy
RawAnnotator
HasOutputAnnotationCol
HasInputAnnotationCols
HasOutputAnnotatorType
ParamsAndFeaturesWritable
HasFeatures
DefaultParamsWritable
MLWritable
Model
Transformer
PipelineStage
Logging
Params
Serializable
Serializable
Identifiable
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new WindowedSentenceModel()
new WindowedSentenceModel(uid: String)

Type Members

type AnnotationContent = Seq[Row]

Attributes
protected
Definition Classes
AnnotatorModel
type AnnotatorType = String

Definition Classes
HasOutputAnnotatorType

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
def $$[T](feature: StructFeature[T]): T

Attributes
protected
Definition Classes
HasFeatures
def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: SetFeature[T]): Set[T]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: ArrayFeature[T]): Array[T]

Attributes
protected
Definition Classes
HasFeatures
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def afterAnnotate(dataset: DataFrame): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def annotate(annotations: Seq[Annotation]): Seq[Annotation]
Returns the joined results after windowing its inputs
Returns the joined results after windowing its inputs
annotations
a sequence of elements to window
returns
result elements containing the neighbors based on the windowSize

Definition Classes
WindowedSentenceModel → HasSimpleAnnotate
final def asInstanceOf[T0]: T0

Definition Classes
Any
def beforeAnnotate(dataset: Dataset[_]): Dataset[_]

Definition Classes
WindowedSentenceModel → AnnotatorModel
val blackList: StringArrayParam
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed

Definition Classes
FilteringParams
val caseSensitive: BooleanParam
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. If the filterValue is 'entity', 'caseSensitive' is always false. The default value is true, except: com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer

Definition Classes
FilteringParams
final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

Attributes
protected
Definition Classes
HasInputAnnotationCols
def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit

Definition Classes
CheckLicense
def checkValidScope(scope: String): Unit

Definition Classes
CheckLicense
def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit

Definition Classes
CheckLicense
def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit

Definition Classes
CheckLicense
final def clear(param: Param[_]): WindowedSentenceModel.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def contextify(annotations: Seq[Annotation]): IndexedSeq[Annotation]

Definition Classes
WindowedParams
def copy(extra: ParamMap): WindowedSentenceModel

Definition Classes
RawAnnotator → Model → Transformer → PipelineStage → Params
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
val criteria: Param[String]
It is used to how to compare black and white listed values with the result of the Annotation.
It is used to how to compare black and white listed values with the result of the Annotation. Possible values are the following: 'isin', 'regex'. Default: isin
- isin : Filter by the chunk
- regex : Filter by using a regex
Definition Classes
FilteringParams
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def dfAnnotate: UserDefinedFunction

Definition Classes
HasSimpleAnnotate
val entitiesConfidence: MapFeature[String, Float]
Pairs (entity,confidenceThreshold).
Pairs (entity,confidenceThreshold). Filter the chunks with entities which have confidence lower than the confidence threshold.

Definition Classes
FilteringParams
lazy val entitiesConfidenceMap: Map[String, Float]

Definition Classes
FilteringParams
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
def extraValidate(structType: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def extraValidateMsg: String

Attributes
protected
Definition Classes
RawAnnotator
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
val features: ArrayBuffer[Feature[_, _, _]]

Definition Classes
HasFeatures
val filterValue: Param[String]
Possible values are 'result' and 'entity'.
Possible values are 'result' and 'entity'. If the value is 'entity', it filters the ner chunks by the ner label that you want to filter. If the value is 'result', it will filter chunks by the result of the Annotation.

Definition Classes
FilteringParams
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def get[T](feature: StructFeature[T]): Option[T]

Attributes
protected
Definition Classes
HasFeatures
def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: SetFeature[T]): Option[Set[T]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: ArrayFeature[T]): Option[Array[T]]

Attributes
protected
Definition Classes
HasFeatures
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getInputCols: Array[String]

Definition Classes
HasInputAnnotationCols
def getLazyAnnotator: Boolean

Definition Classes
CanBeLazy
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCol: String

Definition Classes
HasOutputAnnotationCol
def getParam(paramName: String): Param[Any]

Definition Classes
Params
val glueString: Param[String]
string to use to join the neighboring elements together
string to use to join the neighboring elements together

Definition Classes
WindowedParams
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hasParent: Boolean

Definition Classes
Model
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
val inputAnnotatorTypes: Array[String]
Input annotator types: DOCUMENT
Input annotator types: DOCUMENT

Definition Classes
WindowedSentenceModel → HasInputAnnotationCols
final val inputCols: StringArrayParam

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
val lazyAnnotator: BooleanParam

Definition Classes
CanBeLazy
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def msgHelper(schema: StructType): String

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def onWrite(path: String, spark: SparkSession): Unit

Attributes
protected
Definition Classes
ParamsAndFeaturesWritable
val optionalInputAnnotatorTypes: Array[String]

Definition Classes
HasInputAnnotationCols
val outputAnnotatorType: AnnotatorType
Output annotator types: DOCUMENT
Output annotator types: DOCUMENT

Definition Classes
WindowedSentenceModel → HasOutputAnnotatorType
final val outputCol: Param[String]

Attributes
protected
Definition Classes
HasOutputAnnotationCol
lazy val params: Array[Param[_]]

Definition Classes
Params
var parent: Estimator[WindowedSentenceModel]

Definition Classes
Model
val regex: StringArrayParam
If defined, list of regex to process the chunks (Default: Array())
If defined, list of regex to process the chunks (Default: Array())

Definition Classes
FilteringParams
def resolveFilter(chunkerAnnotations: Seq[Annotation]): Seq[Annotation]

Attributes
protected
Definition Classes
FilteringParams
def resolveMergeFilter(a: Annotation, entityValue: String, falsePositivesArray: Array[(String, String, String)], replaceDictMap: Map[String, String] = Map.empty): Option[Annotation]

Attributes
protected
Definition Classes
FilteringParams
def resolveWhiteListBlackListFilter(annotations: Seq[Annotation]): Seq[Annotation]

Attributes
protected
Definition Classes
FilteringParams
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
def set[T](feature: StructFeature[T], value: T): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: SetFeature[T], value: Set[T]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: ArrayFeature[T], value: Array[T]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def set(paramPair: ParamPair[_]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): WindowedSentenceModel.this.type

Definition Classes
Params
def setAllowList(list: String*): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setAllowList(list: Array[String]): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setBlackList(list: String*): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setBlackList(list: Array[String]): WindowedSentenceModel.this.type
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed.

Definition Classes
FilteringParams
def setCaseSensitive(value: Boolean): WindowedSentenceModel.this.type
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. If the filterValue is 'entity', 'caseSensitive' is always False. The default value is true, except: com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer

Definition Classes
FilteringParams
def setCriteria(s: String): WindowedSentenceModel.this.type
Sets criteria for how to compare black and white listed values with the result of the Annotation.
Sets criteria for how to compare black and white listed values with the result of the Annotation. Possible values are the following: 'isin', 'regex'. Default: isin.
- 'isin' : Filter by the chunk.
- 'regex' : Filter by using a regex.
- You can use 'assertion' in com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer and 'assertion' option is default value for com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer
Definition Classes
FilteringParams
def setDefault[T](feature: StructFeature[T], value: () ⇒ T): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def setDefault(paramPairs: ParamPair[_]*): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): WindowedSentenceModel.this.type

Attributes
protected
Definition Classes
Params
def setDenyList(list: String*): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setDenyList(list: Array[String]): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setEntitiesConfidence(value: Map[String, Float]): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setFilterEntity(v: String): WindowedSentenceModel.this.type
Possible values are 'result' and 'entity'.
Possible values are 'result' and 'entity'. If the value is 'entity', it filters the ner chunks by the ner label that you want to filter. If the value is 'result', it will filter chunks by the result of the Annotation.

Definition Classes
FilteringParams
def setGlueString(value: String): WindowedSentenceModel.this.type

Definition Classes
WindowedParams
final def setInputCols(value: String*): WindowedSentenceModel.this.type

Definition Classes
HasInputAnnotationCols
def setInputCols(value: Array[String]): WindowedSentenceModel.this.type

Definition Classes
HasInputAnnotationCols
def setLazyAnnotator(value: Boolean): WindowedSentenceModel.this.type

Definition Classes
CanBeLazy
final def setOutputCol(value: String): WindowedSentenceModel.this.type

Definition Classes
HasOutputAnnotationCol
def setParent(parent: Estimator[WindowedSentenceModel]): WindowedSentenceModel

Definition Classes
Model
def setRegex(list: String*): WindowedSentenceModel.this.type
Sets the list of regexes to process the chunks.
Sets the list of regexes to process the chunks.

Definition Classes
FilteringParams
def setWhiteList(list: String*): WindowedSentenceModel.this.type

Definition Classes
FilteringParams
def setWhiteList(list: Array[String]): WindowedSentenceModel.this.type
Sets the list of entities to process.
Sets the list of entities to process. The rest will be ignored. Do not include IOB prefix on labels.

Definition Classes
FilteringParams
def setWindowSize(value: Int): WindowedSentenceModel.this.type

Definition Classes
WindowedParams
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
final def transform(dataset: Dataset[_]): DataFrame

Definition Classes
AnnotatorModel → Transformer
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" )
def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" ) @varargs()
final def transformSchema(schema: StructType): StructType

Definition Classes
RawAnnotator → PipelineStage
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
val uid: String

Definition Classes
WindowedSentenceModel → Identifiable
def validate(schema: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val whiteList: StringArrayParam
If defined, list of entities to process.
If defined, list of entities to process. The rest will be ignored. Does not include IOB prefix on labels (Default: Array())

Definition Classes
FilteringParams
val windowSize: IntParam
number of neighbors to grab from each side (actual full window is [windowSize * 2 + 1])
number of neighbors to grab from each side (actual full window is [windowSize * 2 + 1])

Definition Classes
WindowedParams
def wrapColumnMetadata(col: Column): Column

Attributes
protected
Definition Classes
RawAnnotator
def write: MLWriter

Definition Classes
ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable

Packages

WindowedSentenceModel

Companion object WindowedSentenceModel

class WindowedSentenceModel extends AnnotatorModel[WindowedSentenceModel] with CheckLicense with HasSimpleAnnotate[WindowedSentenceModel] with WindowedParams with HasFeatures with FilteringParams

Example

Instance Constructors

Type Members

Value Members

Inherited from FilteringParams

Inherited from WindowedParams

Inherited from HasSimpleAnnotate[WindowedSentenceModel]

Inherited from CheckLicense

Inherited from AnnotatorModel[WindowedSentenceModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[WindowedSentenceModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[WindowedSentenceModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Packages

WindowedSentenceModel 

Companion object WindowedSentenceModel

class WindowedSentenceModel extends AnnotatorModel[WindowedSentenceModel] with CheckLicense with HasSimpleAnnotate[WindowedSentenceModel] with WindowedParams with HasFeatures with FilteringParams

Example

Instance Constructors

Type Members

Value Members

Inherited from FilteringParams

Inherited from WindowedParams

Inherited from HasSimpleAnnotate[WindowedSentenceModel]

Inherited from CheckLicense

Inherited from AnnotatorModel[WindowedSentenceModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[WindowedSentenceModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[WindowedSentenceModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

WindowedSentenceModel