com.johnsnowlabs.nlp.annotators.windowed
WindowedSentenceModel
Companion object WindowedSentenceModel
class WindowedSentenceModel extends AnnotatorModel[WindowedSentenceModel] with CheckLicense with HasSimpleAnnotate[WindowedSentenceModel] with WindowedParams with HasFeatures with FilteringParams
The WindowedSentenceModel
class is used to combine a series of sentences based on specific window configurations.
Returns the joined results after windowing its inputs
Example
val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val sentenceDetector = new SentenceDetector() .setInputCols("document") .setOutputCol("sentence") val windowedSentence = new WindowedSentenceModel() .setWindowSize(1) .setInputCols("sentence") .setOutputCol("windowed") .setGlueString(":::") val flattener = new Flattener() .setInputCols("five") val pipeline = new Pipeline().setStages(Array( documentAssembler, sentenceDetector, windowedSentence, flattener)) val text = "A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years. " + "Two weeks prior to presentation respiratory tract infection . " + "She was on for HTG . " + "She had been of presentation ." + "examination benign with no or rigidity . " val data = spark.createDataFrame(Array(Tuple1(text))).toDF("text") val model = pipeline.fit(data).transform(data) model.show(false) +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+ |windowed_result |five_begin|five_end|five_metadata_sentence| +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+ |A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years.:::Two weeks prior to presentation respiratory tract infection . |0 |152 |0 | |A 28-year-old female with a history of gestational diabetes mellitus diagnosed eight years.:::Two weeks prior to presentation respiratory tract infection .:::She was on for HTG .|0 |173 |1 | |Two weeks prior to presentation respiratory tract infection .:::She was on for HTG .:::She had been of presentation . |92 |204 |2 | |She was on for HTG .:::She had been of presentation .:::examination benign with no or rigidity . |154 |245 |3 | |She had been of presentation .:::examination benign with no or rigidity . |175 |245 |4 | +----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+----------+--------+----------------------+
- Grouped
- Alphabetic
- By Inheritance
- WindowedSentenceModel
- FilteringParams
- WindowedParams
- HasSimpleAnnotate
- CheckLicense
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Type Members
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
annotate(annotations: Seq[Annotation]): Seq[Annotation]
Returns the joined results after windowing its inputs
Returns the joined results after windowing its inputs
- annotations
a sequence of elements to window
- returns
result elements containing the neighbors based on the windowSize
- Definition Classes
- WindowedSentenceModel → HasSimpleAnnotate
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Definition Classes
- WindowedSentenceModel → AnnotatorModel
-
val
blackList: StringArrayParam
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed
- Definition Classes
- FilteringParams
-
val
caseSensitive: BooleanParam
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. If the filterValue is 'entity', 'caseSensitive' is always false. The default value is true, except: com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer
- Definition Classes
- FilteringParams
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
def
checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScope(scope: String): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
final
def
clear(param: Param[_]): WindowedSentenceModel.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
contextify(annotations: Seq[Annotation]): IndexedSeq[Annotation]
- Definition Classes
- WindowedParams
-
def
copy(extra: ParamMap): WindowedSentenceModel
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
criteria: Param[String]
It is used to how to compare black and white listed values with the result of the Annotation.
It is used to how to compare black and white listed values with the result of the Annotation. Possible values are the following: 'isin', 'regex'. Default: isin
- isin : Filter by the chunk
- regex : Filter by using a regex
- Definition Classes
- FilteringParams
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
def
dfAnnotate: UserDefinedFunction
- Definition Classes
- HasSimpleAnnotate
-
val
entitiesConfidence: MapFeature[String, Float]
Pairs (entity,confidenceThreshold).
Pairs (entity,confidenceThreshold). Filter the chunks with entities which have confidence lower than the confidence threshold.
- Definition Classes
- FilteringParams
-
lazy val
entitiesConfidenceMap: Map[String, Float]
- Definition Classes
- FilteringParams
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
val
filterValue: Param[String]
Possible values are 'result' and 'entity'.
Possible values are 'result' and 'entity'. If the value is 'entity', it filters the ner chunks by the ner label that you want to filter. If the value is 'result', it will filter chunks by the result of the Annotation.
- Definition Classes
- FilteringParams
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getBlackList: Array[String]
Gets blackList parameter
Gets blackList parameter
- Definition Classes
- FilteringParams
-
def
getCaseSensitive: Boolean
Gets caseSensitive parameter
Gets caseSensitive parameter
- Definition Classes
- FilteringParams
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getInputCols: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
def
getWhiteList: Array[String]
Gets whiteList parameter
Gets whiteList parameter
- Definition Classes
- FilteringParams
-
val
glueString: Param[String]
string to use to join the neighboring elements together
string to use to join the neighboring elements together
- Definition Classes
- WindowedParams
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[String]
Input annotator types: DOCUMENT
Input annotator types: DOCUMENT
- Definition Classes
- WindowedSentenceModel → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Attributes
- protected
- Definition Classes
- ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Output annotator types: DOCUMENT
Output annotator types: DOCUMENT
- Definition Classes
- WindowedSentenceModel → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[WindowedSentenceModel]
- Definition Classes
- Model
-
val
regex: StringArrayParam
If defined, list of regex to process the chunks (Default:
Array()
)If defined, list of regex to process the chunks (Default:
Array()
)- Definition Classes
- FilteringParams
-
def
resolveFilter(chunkerAnnotations: Seq[Annotation]): Seq[Annotation]
- Attributes
- protected
- Definition Classes
- FilteringParams
-
def
resolveMergeFilter(a: Annotation, entityValue: String, falsePositivesArray: Array[(String, String, String)], replaceDictMap: Map[String, String] = Map.empty): Option[Annotation]
- Attributes
- protected
- Definition Classes
- FilteringParams
-
def
resolveWhiteListBlackListFilter(annotations: Seq[Annotation]): Seq[Annotation]
- Attributes
- protected
- Definition Classes
- FilteringParams
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
set[T](feature: StructFeature[T], value: T): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): WindowedSentenceModel.this.type
- Definition Classes
- Params
-
def
setAllowList(list: String*): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setAllowList(list: Array[String]): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setBlackList(list: String*): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setBlackList(list: Array[String]): WindowedSentenceModel.this.type
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed.
- Definition Classes
- FilteringParams
-
def
setCaseSensitive(value: Boolean): WindowedSentenceModel.this.type
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. If the filterValue is 'entity', 'caseSensitive' is always False. The default value is true, except: com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer
- Definition Classes
- FilteringParams
-
def
setCriteria(s: String): WindowedSentenceModel.this.type
Sets criteria for how to compare black and white listed values with the result of the Annotation.
Sets criteria for how to compare black and white listed values with the result of the Annotation. Possible values are the following: 'isin', 'regex'. Default: isin.
- 'isin' : Filter by the chunk.
- 'regex' : Filter by using a regex.
- You can use 'assertion' in com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer and 'assertion' option is default value for com.johnsnowlabs.nlp.annotators.chunker.AssertionFilterer
- Definition Classes
- FilteringParams
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): WindowedSentenceModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): WindowedSentenceModel.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setDenyList(list: String*): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setDenyList(list: Array[String]): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setEntitiesConfidence(value: Map[String, Float]): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setFilterEntity(v: String): WindowedSentenceModel.this.type
Possible values are 'result' and 'entity'.
Possible values are 'result' and 'entity'. If the value is 'entity', it filters the ner chunks by the ner label that you want to filter. If the value is 'result', it will filter chunks by the result of the Annotation.
- Definition Classes
- FilteringParams
-
def
setGlueString(value: String): WindowedSentenceModel.this.type
- Definition Classes
- WindowedParams
-
final
def
setInputCols(value: String*): WindowedSentenceModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): WindowedSentenceModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): WindowedSentenceModel.this.type
- Definition Classes
- CanBeLazy
-
final
def
setOutputCol(value: String): WindowedSentenceModel.this.type
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[WindowedSentenceModel]): WindowedSentenceModel
- Definition Classes
- Model
-
def
setRegex(list: String*): WindowedSentenceModel.this.type
Sets the list of regexes to process the chunks.
Sets the list of regexes to process the chunks.
- Definition Classes
- FilteringParams
-
def
setWhiteList(list: String*): WindowedSentenceModel.this.type
- Definition Classes
- FilteringParams
-
def
setWhiteList(list: Array[String]): WindowedSentenceModel.this.type
Sets the list of entities to process.
Sets the list of entities to process. The rest will be ignored. Do not include IOB prefix on labels.
- Definition Classes
- FilteringParams
-
def
setWindowSize(value: Int): WindowedSentenceModel.this.type
- Definition Classes
- WindowedParams
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- WindowedSentenceModel → Identifiable
-
def
validate(schema: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
whiteList: StringArrayParam
If defined, list of entities to process.
If defined, list of entities to process. The rest will be ignored. Does not include IOB prefix on labels (Default:
Array()
)- Definition Classes
- FilteringParams
-
val
windowSize: IntParam
number of neighbors to grab from each side (actual full window is [windowSize * 2 + 1])
number of neighbors to grab from each side (actual full window is [windowSize * 2 + 1])
- Definition Classes
- WindowedParams
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
Inherited from FilteringParams
Inherited from WindowedParams
Inherited from HasSimpleAnnotate[WindowedSentenceModel]
Inherited from CheckLicense
Inherited from AnnotatorModel[WindowedSentenceModel]
Inherited from CanBeLazy
Inherited from RawAnnotator[WindowedSentenceModel]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[WindowedSentenceModel]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
Annotator types
Required input and expected output annotator types