com.johnsnowlabs.finance.chunk_classification.assertion

AssertionDLModel

Companion object AssertionDLModel

class AssertionDLModel extends nlp.annotators.assertion.dl.AssertionDLModel

This is a main class in AssertionDL family. A Deep Learning based approach is used to extract Assertion Status from extracted entities and text. AssertionDLModel requires DOCUMENT, CHUNK and WORD_EMBEDDINGS type annotator inputs, which can be obtained by e.g a DocumentAssembler, NerConverter and WordEmbeddingsModel. The result is an assertion status annotation for each recognized entity. Possible values include “present”, “absent”, “hypothetical”, “conditional”, “associated_with_other_person” etc.

For pretrained models please see the Models Hub for available models.

Example

Define pipeline stages to extract NER chunks first

val data = Seq(
  "Patient with severe fever and sore throat",
  "Patient shows no stomach pain",
  "She was maintained on an epidural and PCA for pain control.").toDF("text")
val documentAssembler = new DocumentAssembler().setInputCol("text").setOutputCol("document")
val sentenceDetector = new SentenceDetector().setInputCols("document").setOutputCol("sentence")
val tokenizer = new Tokenizer().setInputCols("sentence").setOutputCol("token")
val embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models")
  .setOutputCol("embeddings")
val nerModel = MedicalNerModel.pretrained("ner_clinical", "en", "clinical/models")
  .setInputCols("sentence", "token", "embeddings").setOutputCol("ner")
val nerConverter = new NerConverter().setInputCols("sentence", "token", "ner").setOutputCol("ner_chunk")

Then a pretrained AssertionDLModel is used to extract the assertion status

val clinicalAssertion = AssertionDLModel.pretrained("assertion_dl", "en", "clinical/models")
  .setInputCols("sentence", "ner_chunk", "embeddings")
  .setOutputCol("assertion")

val assertionPipeline = new Pipeline().setStages(Array(
  documentAssembler,
  sentenceDetector,
  tokenizer,
  embeddings,
  nerModel,
  nerConverter,
  clinicalAssertion
))

val assertionModel = assertionPipeline.fit(data)

Show results

val result = assertionModel.transform(data)
result.selectExpr("ner_chunk.result", "assertion.result").show(3, truncate=false)
+--------------------------------+--------------------------------+
|result                          |result                          |
+--------------------------------+--------------------------------+
|[severe fever, sore throat]     |[present, present]              |
|[stomach pain]                  |[absent]                        |
|[an epidural, PCA, pain control]|[present, present, hypothetical]|
+--------------------------------+--------------------------------+

See also: AssertionDLApproach for training a custom AssertionDLModel
AssertionLogRegModel for non deep learning based extraction

Linear Supertypes

nlp.annotators.assertion.dl.AssertionDLModel, CheckLicense, HandleExceptionParams, HasSimpleAnnotate[nlp.annotators.assertion.dl.AssertionDLModel], HasStorageRef, WriteTensorflowModel, WhiteAndBlackListParams, AssertionDLParams, AnnotatorModel[nlp.annotators.assertion.dl.AssertionDLModel], CanBeLazy, RawAnnotator[nlp.annotators.assertion.dl.AssertionDLModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[nlp.annotators.assertion.dl.AssertionDLModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

AssertionDLModel
AssertionDLModel
CheckLicense
HandleExceptionParams
HasSimpleAnnotate
HasStorageRef
WriteTensorflowModel
WhiteAndBlackListParams
AssertionDLParams
AnnotatorModel
CanBeLazy
RawAnnotator
HasOutputAnnotationCol
HasInputAnnotationCols
HasOutputAnnotatorType
ParamsAndFeaturesWritable
HasFeatures
DefaultParamsWritable
MLWritable
Model
Transformer
PipelineStage
Logging
Params
Serializable
Serializable
Identifiable
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new AssertionDLModel()
new AssertionDLModel(uid: String)
uid
a unique identifier for the instantiated AnnotatorModel

Type Members

type AnnotationContent = Seq[Row]

Attributes
protected
Definition Classes
AnnotatorModel
type AnnotatorType = String

Definition Classes
HasOutputAnnotatorType

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
def $$[T](feature: StructFeature[T]): T

Attributes
protected
Definition Classes
HasFeatures
def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: SetFeature[T]): Set[T]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: ArrayFeature[T]): Array[T]

Attributes
protected
Definition Classes
HasFeatures
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def afterAnnotate(dataset: DataFrame): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def annotate(annotations: Seq[Annotation]): Seq[Annotation]
This is main point of interest.
This is main point of interest. It annotates the input dataset applying properties of the class above This method utilises chunk groups, word piece embeddings and conversion to chunk indexes
annotations
a sequence of Annotations
returns
a sequence of projected Annotations

Definition Classes
AssertionDLModel → AssertionDLModel → HasSimpleAnnotate
final def asInstanceOf[T0]: T0

Definition Classes
Any
val batchSize: IntParam
Size for each batch in the optimization process (Default: 64)
Size for each batch in the optimization process (Default: 64)

Definition Classes
AssertionDLParams
def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
This method validates the dataset before annotation
This method validates the dataset before annotation
dataset
a collection of inputs to validate
returns
a Dataset after validation

Attributes
protected
Definition Classes
AssertionDLModel → AssertionDLModel → AnnotatorModel
val blackList: StringArrayParam
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
val caseSensitive: BooleanParam
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. Default: true

Definition Classes
WhiteAndBlackListParams
def checkEntityAssertion(entity: String, assertion: String): Boolean

Definition Classes
AssertionDLModel
final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

Attributes
protected
Definition Classes
HasInputAnnotationCols
def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit

Definition Classes
CheckLicense
def checkValidScope(scope: String): Unit

Definition Classes
CheckLicense
def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit

Definition Classes
CheckLicense
def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit

Definition Classes
CheckLicense
val chunkCol: Param[String]
Column with extracted NER chunks
Column with extracted NER chunks

Definition Classes
AssertionDLParams
val classes: StringArrayParam
List of internal copies of classes used to train for Python
List of internal copies of classes used to train for Python

Definition Classes
AssertionDLModel
final def clear(param: Param[_]): AssertionDLModel.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val configProtoBytes: IntArrayParam
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()

Definition Classes
AssertionDLParams
def copy(extra: ParamMap): nlp.annotators.assertion.dl.AssertionDLModel
requirement for annotators copies
requirement for annotators copies

Definition Classes
AssertionDLModel → RawAnnotator → Model → Transformer → PipelineStage → Params
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def createDatabaseConnection(database: Name): RocksDBConnection

Definition Classes
HasStorageRef
val datasetInfo: Param[String]
Descriptive information about the dataset being used.
Descriptive information about the dataset being used.

Definition Classes
AssertionDLParams
val datasetParams: StructFeature[DatasetEncoderParams]
Collection of Parameters, which are used by method annotate()
Collection of Parameters, which are used by method annotate()

Definition Classes
AssertionDLModel
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def dfAnnotate: UserDefinedFunction

Definition Classes
HasSimpleAnnotate
val doExceptionHandling: BooleanParam
If true, exceptions are handled.
If true, exceptions are handled. If exception causing data is passed to the model, a error annotation is emitted which has the exception message. Processing continues with the next one. This comes with a performance penalty.

Definition Classes
HandleExceptionParams
val dropout: FloatParam
Dropout at the output of each layer (Default: 0.05f)
Dropout at the output of each layer (Default: 0.05f)

Definition Classes
AssertionDLParams
val enableOutputLogs: BooleanParam
Whether to output to annotators log folder (Default: false)
Whether to output to annotators log folder (Default: false)

Definition Classes
AssertionDLParams
val endCol: Param[String]
Column with token number for last target token
Column with token number for last target token

Definition Classes
AssertionDLParams
var entityAssertionCaseSensitive: BooleanParam
Determines whether entity labels and assertion labels are case sensitive
Determines whether entity labels and assertion labels are case sensitive

Definition Classes
AssertionDLModel
val epochs: IntParam
Number of epochs for the optimization process (Default: 5)
Number of epochs for the optimization process (Default: 5)

Definition Classes
AssertionDLParams
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def evaluateFilter(filter: String): Boolean
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
def extraValidate(structType: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def extraValidateMsg: String

Attributes
protected
Definition Classes
RawAnnotator
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
val features: ArrayBuffer[Feature[_, _, _]]

Definition Classes
HasFeatures
def filterByEntityField(annotation: Annotation): Boolean
Filter annotation by blackList and whiteList, taking into account the caseSensitive param.
Filter annotation by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.metadata.getOrElse("entity", annotation.metadata.getOrElse("identifier", "")).toString
returns
Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByEntityField(annotations: Seq[Annotation]): Seq[Annotation]
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.metadata.getOrElse("entity", annotation.metadata.getOrElse("identifier", "")).toString

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByWhiteAndBlackList(annotation: Annotation): Boolean
Filter annotation by blackList and whiteList, taking into account the caseSensitive param.
Filter annotation by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.result
returns
Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByWhiteAndBlackList(annotations: Seq[Annotation]): Seq[Annotation]
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.result

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
def get[T](feature: StructFeature[T]): Option[T]

Attributes
protected
Definition Classes
HasFeatures
def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: SetFeature[T]): Option[Set[T]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: ArrayFeature[T]): Option[Array[T]]

Attributes
protected
Definition Classes
HasFeatures
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
def getBlackList: Array[String]
Gets blackList param
Gets blackList param

Definition Classes
WhiteAndBlackListParams
def getCaseSensitive: Boolean
Gets caseSensitive param
Gets caseSensitive param

Definition Classes
WhiteAndBlackListParams
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
def getClasses: Array[String]
get the tags used to trained this NerDLModel
get the tags used to trained this NerDLModel

Definition Classes
AssertionDLModel
def getConfigProtoBytes: Option[Array[Byte]]
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()

Definition Classes
AssertionDLParams
def getConfigProtoBytesAsInt: Option[Array[Int]]

Definition Classes
AssertionDLModel
def getDatasetInfo: String
get descriptive information about the dataset being used
get descriptive information about the dataset being used

Definition Classes
AssertionDLParams
def getDatasetParams: DatasetEncoderParams

Definition Classes
AssertionDLModel
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getEnableOutputLogs: Boolean
Whether to output to annotators log folder
Whether to output to annotators log folder

Definition Classes
AssertionDLParams
def getEntityAssertion: Map[String, Array[String]]
Get the lists of assertion labels allowed for a given entity
Get the lists of assertion labels allowed for a given entity

Definition Classes
AssertionDLModel
def getEntityAssertionCaseSensitive: Boolean
Gets the case sensitivity of entities and assertion labels
Gets the case sensitivity of entities and assertion labels

Definition Classes
AssertionDLModel
def getEntityAssertionStr: String
Get the string representation of the lists of assertion labels allowed for a given entity
Get the string representation of the lists of assertion labels allowed for a given entity

Definition Classes
AssertionDLModel
def getIncludeConfidence: Boolean
whether to include confidence scores in annotation metadata
whether to include confidence scores in annotation metadata

Definition Classes
AssertionDLParams
def getInputCols: Array[String]

Definition Classes
HasInputAnnotationCols
def getLazyAnnotator: Boolean

Definition Classes
CanBeLazy
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCol: String

Definition Classes
HasOutputAnnotationCol
def getOutputLogsPath: String
Folder path to save training logs
Folder path to save training logs

Definition Classes
AssertionDLParams
def getParam(paramName: String): Param[Any]

Definition Classes
Params
def getReplaceLabels: Map[String, String]
Gets the assertion labels to be replaced for the specified new labels.
Gets the assertion labels to be replaced for the specified new labels.

Definition Classes
AssertionDLModel
def getReplaceLabelsStr: String
Gets the string presentation of the assertion labels to be replaced for the specified new labels.
Gets the string presentation of the assertion labels to be replaced for the specified new labels.

Definition Classes
AssertionDLModel
def getScopeWindow: (Int, Int)
Get scope window
Get scope window

Definition Classes
AssertionDLParams
def getStorageRef: String

Definition Classes
HasStorageRef
def getWhiteList: Array[String]
Gets whiteList param
Gets whiteList param

Definition Classes
WhiteAndBlackListParams
val graphFile: Param[String]
File path that contain external graph file.
File path that contain external graph file. When specified, the provided file will be used, and no graph search will happen. The path can be a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).

Definition Classes
AssertionDLParams
val graphFolder: Param[String]
Folder path that contain external graph files.
Folder path that contain external graph files.
Folder path that contain external graph files. The path can a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).

Definition Classes
AssertionDLParams
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hasParent: Boolean

Definition Classes
Model
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
val includeConfidence: BooleanParam
Whether to include confidence scores in annotation metadata
Whether to include confidence scores in annotation metadata

Definition Classes
AssertionDLParams
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
val inputAnnotatorTypes: Array[String]
Input annotator types: DOCUMENT, CHUNK, WORD_EMBEDDINGS
Input annotator types: DOCUMENT, CHUNK, WORD_EMBEDDINGS

Definition Classes
AssertionDLModel → HasInputAnnotationCols
final val inputCols: StringArrayParam

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
def isValueInList(value: String, list: Array[String]): Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def isWhiteListAndBlacklistEmpty: Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
val labelCol: Param[String]
Column with one label per document.
Column with one label per document. Example of possible values: “present”, “absent”, “hypothetical”, “conditional”, “associated_with_other_person”, etc.

Definition Classes
AssertionDLParams
val lazyAnnotator: BooleanParam

Definition Classes
CanBeLazy
val learningRate: FloatParam
Learning rate for the optimization process (Default: 0.0012f)
Learning rate for the optimization process (Default: 0.0012f)

Definition Classes
AssertionDLParams
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
val maxSentLen: IntParam
Max possible length of a sentence, must match graph model (Default: 250)
Max possible length of a sentence, must match graph model (Default: 250)

Definition Classes
AssertionDLParams
def model: TensorflowAssertion
Tensorflow model for the AssertionDLModel.
Tensorflow model for the AssertionDLModel. This is used to generate the predictions.

Definition Classes
AssertionDLModel
def msgHelper(schema: StructType): String

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def onWrite(path: String, spark: SparkSession): Unit

Definition Classes
AssertionDLModel → ParamsAndFeaturesWritable
val optionalInputAnnotatorTypes: Array[String]

Definition Classes
HasInputAnnotationCols
val outputAnnotatorType: AnnotatorType
Output annotator types: ASSERTION
Output annotator types: ASSERTION

Definition Classes
AssertionDLModel → HasOutputAnnotatorType
final val outputCol: Param[String]

Attributes
protected
Definition Classes
HasOutputAnnotationCol
val outputLogsPath: Param[String]
Folder path to save training logs.
Folder path to save training logs. If no path is specified, the logs won't be stored in disk. The path can be a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).

Definition Classes
AssertionDLParams
lazy val params: Array[Param[_]]

Definition Classes
Params
var parent: Estimator[nlp.annotators.assertion.dl.AssertionDLModel]

Definition Classes
Model
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
val scopeWindow: IntArrayParam
The scope window of the assertion (whole sentence by default)
The scope window of the assertion (whole sentence by default)

Definition Classes
AssertionDLParams
def set[T](feature: StructFeature[T], value: T): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: SetFeature[T], value: Set[T]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: ArrayFeature[T], value: Array[T]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def set(paramPair: ParamPair[_]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): AssertionDLModel.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): AssertionDLModel.this.type

Definition Classes
Params
def setAllowList(list: String*): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setAllowList(list: Array[String]): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setBatchSize(size: Int): AssertionDLModel.this.type
Size for each batch in the optimization process
Size for each batch in the optimization process

Definition Classes
AssertionDLParams
def setBlackList(list: String*): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setBlackList(list: Array[String]): AssertionDLModel.this.type
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
def setCaseSensitive(value: Boolean): AssertionDLModel.this.type
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. Default: true

Definition Classes
WhiteAndBlackListParams
def setChunkCol(c: String): AssertionDLModel.this.type
Column with extracted NER chunks
Column with extracted NER chunks

Definition Classes
AssertionDLParams
def setConfigProtoBytes(bytes: Array[Int]): AssertionDLModel.this.type
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()

Definition Classes
AssertionDLParams
def setDatasetInfo(value: String): AssertionDLModel.this.type
set descriptive information about the dataset being used
set descriptive information about the dataset being used

Definition Classes
AssertionDLParams
def setDatasetParams(params: DatasetEncoderParams): nlp.annotators.assertion.dl.AssertionDLModel
Collection of Parameters, which are used by method annotate()
Collection of Parameters, which are used by method annotate()

Definition Classes
AssertionDLModel
def setDefault[T](feature: StructFeature[T], value: () ⇒ T): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): AssertionDLModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def setDefault(paramPairs: ParamPair[_]*): AssertionDLModel.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): AssertionDLModel.this.type

Attributes
protected[org.apache.spark.ml]
Definition Classes
Params
def setDenyList(list: String*): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setDenyList(list: Array[String]): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setDoExceptionHandling(value: Boolean): AssertionDLModel.this.type
If true, exceptions are handled.
If true, exceptions are handled. If exception causing data is passed to the model, a error annotation is emitted which has the exception message. Processing continues with the next one. This comes with a performance penalty.

Definition Classes
HandleExceptionParams
def setDropout(factor: Float): AssertionDLModel.this.type
Dropout at the output of each layer
Dropout at the output of each layer

Definition Classes
AssertionDLParams
def setEnableOutputLogs(v: Boolean): AssertionDLModel.this.type
Whether to output to annotators log folder
Whether to output to annotators log folder

Definition Classes
AssertionDLParams
def setEndCol(e: String): AssertionDLModel.this.type
Column with token number for last target token
Column with token number for last target token

Definition Classes
AssertionDLParams
def setEntityAssertion(assertionEntities: HashMap[String, List[String]]): AssertionDLModel.this.type
Set the lists of assertion labels allowed for a given entity.
Set the lists of assertion labels allowed for a given entity.

Definition Classes
AssertionDLModel
Note
entityAssertion functionality is processed earlier than replaceLabels
def setEntityAssertion(assertionEntities: Map[String, Array[String]]): AssertionDLModel.this.type
Set the lists of assertion labels allowed for a given entity.
Set the lists of assertion labels allowed for a given entity.

Definition Classes
AssertionDLModel
Note
entityAssertion functionality is processed earlier than replaceLabels
def setEntityAssertionCaseSensitive(value: Boolean): nlp.annotators.assertion.dl.AssertionDLModel
Sets the case sensitivity of entities and assertion labels
Sets the case sensitivity of entities and assertion labels

Definition Classes
AssertionDLModel
def setEpochs(number: Int): AssertionDLModel.this.type
Number of epochs for the optimization process
Number of epochs for the optimization process

Definition Classes
AssertionDLParams
def setGraphFile(path: String): AssertionDLModel.this.type
Folder path that contain external graph files
Folder path that contain external graph files

Definition Classes
AssertionDLParams
def setGraphFolder(path: String): AssertionDLModel.this.type
Folder path that contain external graph files
Folder path that contain external graph files

Definition Classes
AssertionDLParams
def setIncludeConfidence(value: Boolean): AssertionDLModel.this.type
Whether to include confidence scores in annotation metadata
Whether to include confidence scores in annotation metadata

Definition Classes
AssertionDLParams
final def setInputCols(value: String*): AssertionDLModel.this.type

Definition Classes
HasInputAnnotationCols
def setInputCols(value: Array[String]): AssertionDLModel.this.type

Definition Classes
HasInputAnnotationCols
def setLabelCol(label: String): AssertionDLModel.this.type
Column with one label per document
Column with one label per document

Definition Classes
AssertionDLParams
def setLazyAnnotator(value: Boolean): AssertionDLModel.this.type

Definition Classes
CanBeLazy
def setLearningRate(rate: Float): AssertionDLModel.this.type
Learning rate for the optimization process
Learning rate for the optimization process

Definition Classes
AssertionDLParams
def setMaxSentLen(len: Int): AssertionDLModel.this.type
Max possible length of a sentence, must match graph model
Max possible length of a sentence, must match graph model

Definition Classes
AssertionDLParams
final def setOutputCol(value: String): AssertionDLModel.this.type

Definition Classes
HasOutputAnnotationCol
def setOutputLogsPath(v: String): AssertionDLModel.this.type
Folder path to save training logs
Folder path to save training logs

Definition Classes
AssertionDLParams
def setParent(parent: Estimator[nlp.annotators.assertion.dl.AssertionDLModel]): nlp.annotators.assertion.dl.AssertionDLModel

Definition Classes
Model
def setReplaceLabels(value: Map[String, String]): AssertionDLModel.this.type
Sets the replaceLabels to be replaced for the specified new labels.
Sets the replaceLabels to be replaced for the specified new labels.
value
A map containing the replacement labels.

Definition Classes
AssertionDLModel
Note
replaceLabels functionality runs later than entityAssertion
def setReplaceLabels(value: HashMap[String, String]): AssertionDLModel.this.type
Sets the replaceLabels to be replaced for the specified new labels.
Sets the replaceLabels to be replaced for the specified new labels.
value
A map containing the replacement labels.

Definition Classes
AssertionDLModel
Note
replaceLabels functionality is processed later than entityAssertion
def setScopeWindow(window: (Int, Int)): AssertionDLModel.this.type
Max possible length of a sentence.
Max possible length of a sentence.

Definition Classes
AssertionDLParams
def setStartCol(s: String): AssertionDLModel.this.type
Column with token number for first target token
Column with token number for first target token

Definition Classes
AssertionDLParams
def setStorageRef(value: String): AssertionDLModel.this.type

Definition Classes
HasStorageRef
def setTensorflow(spark: SparkSession, tf: TensorflowWrapper): nlp.annotators.assertion.dl.AssertionDLModel

Definition Classes
AssertionDLModel
def setValidationSplit(validationSplit: Float): AssertionDLModel.this.type
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.

Definition Classes
AssertionDLParams
def setVerbose(verbose: Level): AssertionDLModel.this.type
Level of verbosity during training
Level of verbosity during training

Definition Classes
AssertionDLParams
def setWhiteList(list: String*): AssertionDLModel.this.type

Definition Classes
WhiteAndBlackListParams
def setWhiteList(list: Array[String]): AssertionDLModel.this.type
Sets the list of entities to process.
Sets the list of entities to process. The rest will be ignored. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
val startCol: Param[String]
Column with token number for first target token
Column with token number for first target token

Definition Classes
AssertionDLParams
val storageRef: Param[String]

Definition Classes
HasStorageRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
final def transform(dataset: Dataset[_]): DataFrame

Definition Classes
AnnotatorModel → Transformer
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" )
def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" ) @varargs()
final def transformSchema(schema: StructType): StructType

Definition Classes
RawAnnotator → PipelineStage
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
val uid: String

Definition Classes
AssertionDLModel → AssertionDLModel → Identifiable
def validate(schema: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def validateStorageRef(dataset: Dataset[_], inputCols: Array[String], annotatorType: String): Unit

Definition Classes
HasStorageRef
val validationSplit: FloatParam
The proportion of training dataset to be used as validation set.
The proportion of training dataset to be used as validation set.
The model will be validated against this dataset on each Epoch and will not be used for training. The value should be between 0.0 and 1.0.

Definition Classes
AssertionDLParams
val verbose: IntParam
Level of verbosity during training
Level of verbosity during training

Definition Classes
AssertionDLParams
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val whiteList: StringArrayParam
If defined, list of entities to process.
If defined, list of entities to process. The rest will be ignored. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
def wrapColumnMetadata(col: Column): Column

Attributes
protected
Definition Classes
RawAnnotator
def write: MLWriter

Definition Classes
ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit

Definition Classes
WriteTensorflowModel

Packages

AssertionDLModel 

Companion object AssertionDLModel

class AssertionDLModel extends nlp.annotators.assertion.dl.AssertionDLModel

Example

Instance Constructors

Type Members

Value Members

Inherited from nlp.annotators.assertion.dl.AssertionDLModel

Inherited from CheckLicense

Inherited from HandleExceptionParams

Inherited from HasSimpleAnnotate[nlp.annotators.assertion.dl.AssertionDLModel]

Inherited from HasStorageRef

Inherited from WriteTensorflowModel

Inherited from WhiteAndBlackListParams

Inherited from AssertionDLParams

Inherited from AnnotatorModel[nlp.annotators.assertion.dl.AssertionDLModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[nlp.annotators.assertion.dl.AssertionDLModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[nlp.annotators.assertion.dl.AssertionDLModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

AssertionDLModel