Class/Object

com.johnsnowlabs.nlp.annotators.assertion.dl

AssertionDLApproach

Related Docs: object AssertionDLApproach | package dl

Permalink

class AssertionDLApproach extends AnnotatorApproach[AssertionDLModel] with Logging with Licensed

Contains all the methods for training the DL Approach, together with trainWithChunk, trainWithStartEnd

Linear Supertypes
Licensed, Logging, AnnotatorApproach[AssertionDLModel], CanBeLazy, DefaultParamsWritable, MLWritable, HasOutputAnnotatorType, HasOutputAnnotationCol, HasInputAnnotationCols, Estimator[AssertionDLModel], PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. AssertionDLApproach
  2. Licensed
  3. Logging
  4. AnnotatorApproach
  5. CanBeLazy
  6. DefaultParamsWritable
  7. MLWritable
  8. HasOutputAnnotatorType
  9. HasOutputAnnotationCol
  10. HasInputAnnotationCols
  11. Estimator
  12. PipelineStage
  13. Logging
  14. Params
  15. Serializable
  16. Serializable
  17. Identifiable
  18. AnyRef
  19. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new AssertionDLApproach()

    Permalink
  2. new AssertionDLApproach(uid: String)

    Permalink

    uid

    a unique identifier for the instanced AssertionDLApproach

Type Members

  1. type AnnotatorType = String

    Permalink
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  4. final def ==(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  5. def _fit(dataset: Dataset[_], recursiveStages: Option[PipelineModel]): AssertionDLModel

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  6. final def asInstanceOf[T0]: T0

    Permalink
    Definition Classes
    Any
  7. val batchSize: IntParam

    Permalink
  8. def beforeTraining(spark: SparkSession): Unit

    Permalink
    Definition Classes
    AnnotatorApproach
  9. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  10. val chunkCol: Param[String]

    Permalink
  11. val classes: IntParam

    Permalink
  12. final def clear(param: Param[_]): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    Params
  13. def clone(): AnyRef

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  14. val configProtoBytes: IntArrayParam

    Permalink
  15. final def copy(extra: ParamMap): Estimator[AssertionDLModel]

    Permalink
    Definition Classes
    AnnotatorApproach → Estimator → PipelineStage → Params
  16. def copyValues[T <: Params](to: T, extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  17. final def defaultCopy[T <: Params](extra: ParamMap): T

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  18. val description: String

    Permalink
    Definition Classes
    AssertionDLApproach → AnnotatorApproach
  19. val dropout: FloatParam

    Permalink
  20. val enableOutputLogs: BooleanParam

    Permalink
  21. val endCol: Param[String]

    Permalink
  22. val epochs: IntParam

    Permalink
  23. final def eq(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  24. def equals(arg0: Any): Boolean

    Permalink
    Definition Classes
    AnyRef → Any
  25. def explainParam(param: Param[_]): String

    Permalink
    Definition Classes
    Params
  26. def explainParams(): String

    Permalink
    Definition Classes
    Params
  27. final def extractParamMap(): ParamMap

    Permalink
    Definition Classes
    Params
  28. final def extractParamMap(extra: ParamMap): ParamMap

    Permalink
    Definition Classes
    Params
  29. def finalize(): Unit

    Permalink
    Attributes
    protected[java.lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  30. final def fit(dataset: Dataset[_]): AssertionDLModel

    Permalink
    Definition Classes
    AnnotatorApproach → Estimator
  31. def fit(dataset: Dataset[_], paramMaps: Array[ParamMap]): Seq[AssertionDLModel]

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  32. def fit(dataset: Dataset[_], paramMap: ParamMap): AssertionDLModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" )
  33. def fit(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): AssertionDLModel

    Permalink
    Definition Classes
    Estimator
    Annotations
    @Since( "2.0.0" ) @varargs()
  34. final def get[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  35. final def getClass(): Class[_]

    Permalink
    Definition Classes
    AnyRef → Any
  36. def getConfigProtoBytes: Option[Array[Byte]]

    Permalink
  37. final def getDefault[T](param: Param[T]): Option[T]

    Permalink
    Definition Classes
    Params
  38. def getEnableOutputLogs: Boolean

    Permalink
  39. def getInputCols: Array[String]

    Permalink
    Definition Classes
    HasInputAnnotationCols
  40. def getLazyAnnotator: Boolean

    Permalink
    Definition Classes
    CanBeLazy
  41. def getLogName: String

    Permalink
    Definition Classes
    Logging
  42. final def getOrDefault[T](param: Param[T]): T

    Permalink
    Definition Classes
    Params
  43. final def getOutputCol: String

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  44. def getOutputLogsPath: String

    Permalink
  45. def getParam(paramName: String): Param[Any]

    Permalink
    Definition Classes
    Params
  46. val graphFolder: Param[String]

    Permalink
  47. final def hasDefault[T](param: Param[T]): Boolean

    Permalink
    Definition Classes
    Params
  48. def hasParam(paramName: String): Boolean

    Permalink
    Definition Classes
    Params
  49. def hashCode(): Int

    Permalink
    Definition Classes
    AnyRef → Any
  50. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  51. def initializeLogIfNecessary(isInterpreter: Boolean): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  52. val inputAnnotatorTypes: Array[String]

    Permalink
    Definition Classes
    AssertionDLApproach → HasInputAnnotationCols
  53. final val inputCols: StringArrayParam

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  54. final def isDefined(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  55. final def isInstanceOf[T0]: Boolean

    Permalink
    Definition Classes
    Any
  56. final def isSet(param: Param[_]): Boolean

    Permalink
    Definition Classes
    Params
  57. def isTraceEnabled(): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  58. val labelCol: Param[String]

    Permalink
  59. val lazyAnnotator: BooleanParam

    Permalink
    Definition Classes
    CanBeLazy
  60. val learningRate: FloatParam

    Permalink
  61. def log(value: ⇒ String, minLevel: Level): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  62. def log: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  63. def logDebug(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  64. def logDebug(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  65. def logError(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  66. def logError(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  67. def logInfo(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  68. def logInfo(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  69. def logName: String

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  70. def logTrace(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  71. def logTrace(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  72. def logWarning(msg: ⇒ String, throwable: Throwable): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  73. def logWarning(msg: ⇒ String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  74. val logger: Logger

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  75. val maxSentLen: IntParam

    Permalink
  76. def msgHelper(schema: StructType): String

    Permalink
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  77. final def ne(arg0: AnyRef): Boolean

    Permalink
    Definition Classes
    AnyRef
  78. final def notify(): Unit

    Permalink
    Definition Classes
    AnyRef
  79. final def notifyAll(): Unit

    Permalink
    Definition Classes
    AnyRef
  80. def onTrained(model: AssertionDLModel, spark: SparkSession): Unit

    Permalink
    Definition Classes
    AnnotatorApproach
  81. val outputAnnotatorType: AnnotatorType

    Permalink
    Definition Classes
    AssertionDLApproach → HasOutputAnnotatorType
  82. final val outputCol: Param[String]

    Permalink
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  83. def outputLog(value: ⇒ String, uuid: String, shouldLog: Boolean, outputLogsPath: String): Unit

    Permalink
    Attributes
    protected
    Definition Classes
    Logging
  84. val outputLogsPath: Param[String]

    Permalink
  85. lazy val params: Array[Param[_]]

    Permalink
    Definition Classes
    Params
  86. def save(path: String): Unit

    Permalink
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  87. final def set(paramPair: ParamPair[_]): AssertionDLApproach.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  88. final def set(param: String, value: Any): AssertionDLApproach.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  89. final def set[T](param: Param[T], value: T): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    Params
  90. def setBatchSize(size: Int): AssertionDLApproach.this.type

    Permalink
  91. def setChunkCol(c: String): AssertionDLApproach.this.type

    Permalink
  92. def setClasses(k: Int): AssertionDLApproach.this.type

    Permalink
  93. def setConfigProtoBytes(bytes: Array[Int]): AssertionDLApproach

    Permalink
  94. final def setDefault(paramPairs: ParamPair[_]*): AssertionDLApproach.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  95. final def setDefault[T](param: Param[T], value: T): AssertionDLApproach.this.type

    Permalink
    Attributes
    protected
    Definition Classes
    Params
  96. def setDropout(factor: Float): AssertionDLApproach.this.type

    Permalink
  97. def setEnableOutputLogs(v: Boolean): AssertionDLApproach.this.type

    Permalink

    Whether to output to annotators log folder

  98. def setEndCol(e: String): AssertionDLApproach.this.type

    Permalink
  99. def setEpochs(number: Int): AssertionDLApproach.this.type

    Permalink
  100. def setGraphFolder(path: String): AssertionDLApproach

    Permalink
  101. final def setInputCols(value: String*): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  102. final def setInputCols(value: Array[String]): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    HasInputAnnotationCols
  103. def setLabelCol(label: String): AssertionDLApproach.this.type

    Permalink
  104. def setLazyAnnotator(value: Boolean): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    CanBeLazy
  105. def setLearningRate(rate: Float): AssertionDLApproach.this.type

    Permalink
  106. def setMaxSentLen(len: Int): AssertionDLApproach.this.type

    Permalink
  107. final def setOutputCol(value: String): AssertionDLApproach.this.type

    Permalink
    Definition Classes
    HasOutputAnnotationCol
  108. def setOutputLogsPath(v: String): AssertionDLApproach.this.type

    Permalink
  109. def setStartCol(s: String): AssertionDLApproach.this.type

    Permalink
  110. def setTargetCol(target: String): AssertionDLApproach.this.type

    Permalink
  111. def setValidationSplit(validationSplit: Float): AssertionDLApproach.this.type

    Permalink
  112. def setVerbose(verbose: Level): AssertionDLApproach

    Permalink
  113. val startCol: Param[String]

    Permalink
  114. final def synchronized[T0](arg0: ⇒ T0): T0

    Permalink
    Definition Classes
    AnyRef
  115. val testDataset: ExternalResourceParam

    Permalink

    val testDataset = new ExternalResourceParam(this, "testDataset", "Path to test dataset.

    val testDataset = new ExternalResourceParam(this, "testDataset", "Path to test dataset. If set used to calculate statistic on it during training.")

  116. def toString(): String

    Permalink
    Definition Classes
    Identifiable → AnyRef → Any
  117. def train(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): AssertionDLModel

    Permalink

    This is a main point of interest of this class.

    This is a main point of interest of this class. It trains the dataset with recursive pipeline and uses methods trainWithChunk() and trainwithStartEnd() The choice of training happens based on the startCol value of the DL Approach

    dataset

    a collection of inputs to train

    recursivePipeline

    an instance of PipelineModel

    returns

    an instance of trained AssertionDLModel

    Definition Classes
    AssertionDLApproach → AnnotatorApproach
  118. final def transformSchema(schema: StructType): StructType

    Permalink
    Definition Classes
    AnnotatorApproach → PipelineStage
  119. def transformSchema(schema: StructType, logging: Boolean): StructType

    Permalink
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  120. val uid: String

    Permalink

    a unique identifier for the instanced AssertionDLApproach

    a unique identifier for the instanced AssertionDLApproach

    Definition Classes
    AssertionDLApproach → Identifiable
  121. def validate(schema: StructType): Boolean

    Permalink
    Attributes
    protected
    Definition Classes
    AnnotatorApproach
  122. val validationSplit: FloatParam

    Permalink

    Choose the proportion of training dataset to be validated against the model on each Epoch.

    Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.

  123. val verbose: IntParam

    Permalink
  124. val verboseLevel: Level

    Permalink
    Definition Classes
    AssertionDLApproach → Logging
  125. final def wait(): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  126. final def wait(arg0: Long, arg1: Int): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  127. final def wait(arg0: Long): Unit

    Permalink
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  128. def write: MLWriter

    Permalink
    Definition Classes
    DefaultParamsWritable → MLWritable

Inherited from Licensed

Inherited from Logging

Inherited from AnnotatorApproach[AssertionDLModel]

Inherited from CanBeLazy

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from HasOutputAnnotatorType

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from Estimator[AssertionDLModel]

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

getParam

param

Ungrouped