com.johnsnowlabs.nlp.annotators.classification

FewShotAssertionClassifierModel

Companion object FewShotAssertionClassifierModel

class FewShotAssertionClassifierModel extends GenericClassifierModel with HasStorageRef with WhiteAndBlackListParams with InternalWriteOnnxModel

FewShotAssertionClassifierModel does assertion classification using can run large (LLMS based) few shot classifiers based on the SetFit approach.

Example

Define pipeline stages to prepare the data

val documentAssembler = new DocumentAssembler()
  .setInputCol("text")
  .setOutputCol("document")

val sentenceDetector = new SentenceDetector()
   .setInputCols(Array("document"))
   .setOutputCol("sentences")

val tokenizer = Tokenizer()
   .setInputCols(Array("sentence"))
   .setOutputCol("token")

val embeddings = WordEmbeddingsModel
   .pretrained("embeddings_clinical", "en", "clinical/models")
   .setInputCols(Array("sentence", "token"))
   .setOutputCol("embeddings")
   .setCaseSensitive(False)

val ner = MedicalNerModel
   .pretrained("ner_jsl", "en", "clinical/models")
   .setInputCols(["sentence", "token", "embeddings"])
   .setOutputCol("ner")

val nerConverter = NerConverter()
   .setInputCols(Array("sentence", "token", "ner"))
   .setWhiteList("Disease_Syndrome_Disorder", "Hypertension")
   .setOutputCol("ner_chunk")

 val fewShotAssertionClassifier = LargeFewShotClassifierModel
   .pretrained("clinical_assertion")
   .setInputCols(Array("sentence"))
   .setBatchSize(1)
   .setOutputCol("label")

 val pipeline = new Pipeline().setStages(Array(
  documentAssembler, sentenceDetector, tokenizer, embeddings, ner, nerConverter, fewShotAssertionClassifier))

 val model = pipeline.fit(Seq().toDS.toDF("text"))
 val results = model.transform(
   Seq("Includes hypertension and chronic obstructive pulmonary disease.").toDS.toDF("text"))

 results
   .selectExpr("explode(assertion) as assertion")
   .selectExpr("assertion.result", "assertion.metadata.chunk", "assertion.metadata.confidence")
   .show(truncate = false)

+-------+-------------------------------------+----------+
|result |chunk                                |confidence|
+-------+-------------------------------------+----------+
|present|hypertension                         |1.0       |
|present|chronic obstructive pulmonary disease|1.0       |
|absent |arteriovenous malformations          |1.0       |
|absent |vascular malformation                |0.9999997 |
+-------+-------------------------------------+----------+

See also: LargeFewShotClassifierModel for instantiated models
https://arxiv.org/abs/2209.11055 for details about the SetFit approach

Linear Supertypes

InternalWriteOnnxModel, WhiteAndBlackListParams, GenericClassifierModel, CheckLicense, HasSafeAnnotate[GenericClassifierModel], HandleExceptionParams, HasSimpleAnnotate[GenericClassifierModel], WriteTensorflowModel, HasStorageRef, GenericClassifierParams, AnnotatorModel[GenericClassifierModel], CanBeLazy, RawAnnotator[GenericClassifierModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[GenericClassifierModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any

Ordering

Grouped
Alphabetic
By Inheritance

Inherited

FewShotAssertionClassifierModel
InternalWriteOnnxModel
WhiteAndBlackListParams
GenericClassifierModel
CheckLicense
HasSafeAnnotate
HandleExceptionParams
HasSimpleAnnotate
WriteTensorflowModel
HasStorageRef
GenericClassifierParams
AnnotatorModel
CanBeLazy
RawAnnotator
HasOutputAnnotationCol
HasInputAnnotationCols
HasOutputAnnotatorType
ParamsAndFeaturesWritable
HasFeatures
DefaultParamsWritable
MLWritable
Model
Transformer
PipelineStage
Logging
Params
Serializable
Serializable
Identifiable
AnyRef
Any

Hide All
Show All

Visibility

Public
All

Instance Constructors

new FewShotAssertionClassifierModel()
new FewShotAssertionClassifierModel(uid: String)
uid
a unique identifier for the instantiated AnnotatorModel

Type Members

type AnnotationContent = Seq[Row]

Attributes
protected
Definition Classes
AnnotatorModel
type AnnotatorType = String

Definition Classes
HasOutputAnnotatorType

Value Members

final def !=(arg0: Any): Boolean

Definition Classes
AnyRef → Any
final def ##(): Int

Definition Classes
AnyRef → Any
final def $[T](param: Param[T]): T

Attributes
protected
Definition Classes
Params
def $$[T](feature: StructFeature[T]): T

Attributes
protected
Definition Classes
HasFeatures
def $$[K, V](feature: MapFeature[K, V]): Map[K, V]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: SetFeature[T]): Set[T]

Attributes
protected
Definition Classes
HasFeatures
def $$[T](feature: ArrayFeature[T]): Array[T]

Attributes
protected
Definition Classes
HasFeatures
final def ==(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def afterAnnotate(dataset: DataFrame): DataFrame

Attributes
protected
Definition Classes
AnnotatorModel
def annotate(annotations: Seq[Annotation]): Seq[Annotation]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
annotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
returns
any number of annotations processed for every input annotation. Not necessary one to one relationship

Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → HasSimpleAnnotate
final def asInstanceOf[T0]: T0

Definition Classes
Any
val batchSize: IntParam
Batch size
Batch size

Definition Classes
GenericClassifierParams
def beforeAnnotate(dataset: Dataset[_]): Dataset[_]

Attributes
protected
Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → AnnotatorModel
val blackList: StringArrayParam
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
val caseSensitive: BooleanParam
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. Default: true

Definition Classes
WhiteAndBlackListParams
final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean

Attributes
protected
Definition Classes
HasInputAnnotationCols
def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String], metadata: Option[Map[String, Value]]): Unit

Definition Classes
CheckLicense
def checkValidScope(scope: String): Unit

Definition Classes
CheckLicense
def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean, metadata: Option[Map[String, Value]]): Unit

Definition Classes
CheckLicense
def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean, metadata: Option[Map[String, Value]]): Unit

Definition Classes
CheckLicense
val classes: StringArrayParam

Definition Classes
GenericClassifierModel
final def clear(param: Param[_]): FewShotAssertionClassifierModel.this.type

Definition Classes
Params
def clone(): AnyRef

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
def copy(extra: ParamMap): GenericClassifierModel

Definition Classes
RawAnnotator → Model → Transformer → PipelineStage → Params
def copyValues[T <: Params](to: T, extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def createDatabaseConnection(database: Name): RocksDBConnection

Definition Classes
HasStorageRef
val datasetInfo: Param[String]
Descriptive information about the dataset being used.
Descriptive information about the dataset being used.

Definition Classes
GenericClassifierParams
final def defaultCopy[T <: Params](extra: ParamMap): T

Attributes
protected
Definition Classes
Params
def dfAnnotate: UserDefinedFunction

Definition Classes
HasSimpleAnnotate
val doExceptionHandling: BooleanParam
If true, exceptions are handled.
If true, exceptions are handled. If exception causing data is passed to the model, a error annotation is emitted which has the exception message. Processing continues with the next one. This comes with a performance penalty.

Definition Classes
HandleExceptionParams
val dropout: FloatParam
Dropout coefficient.
Dropout coefficient.
It is applied to the output of the last hidden layer of the graph. Graphs without hidden layer (GenericLogRegClassifier, FewShotClassifier, GenericSVMClassifier) apply it directly to the input feature vector, so a value greater than 0 drops input features there: set it to 0 for those annotators unless input dropout is what you want.

Definition Classes
GenericClassifierParams
val epochsN: IntParam
Maximum number of epochs to train
Maximum number of epochs to train

Definition Classes
GenericClassifierParams
final def eq(arg0: AnyRef): Boolean

Definition Classes
AnyRef
def equals(arg0: Any): Boolean

Definition Classes
AnyRef → Any
def evaluateFilter(filter: String): Boolean
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def explainParam(param: Param[_]): String

Definition Classes
Params
def explainParams(): String

Definition Classes
Params
final val extraInputCols: StringArrayParam

Attributes
protected
Definition Classes
HasInputAnnotationCols
def extraValidate(structType: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def extraValidateMsg: String

Attributes
protected
Definition Classes
RawAnnotator
final def extractParamMap(): ParamMap

Definition Classes
Params
final def extractParamMap(extra: ParamMap): ParamMap

Definition Classes
Params
val featureScaling: Param[String]
Feature scaling method.
Feature scaling method. Possible values are 'zscore', 'minmax' or empty (no scaling)

Definition Classes
GenericClassifierParams
val features: ArrayBuffer[Feature[_, _, _]]

Definition Classes
HasFeatures
def filterByEntityField(annotation: Annotation): Boolean
Filter annotation by blackList and whiteList, taking into account the caseSensitive param.
Filter annotation by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.metadata.getOrElse("entity", annotation.metadata.getOrElse("identifier", "")).toString
returns
Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByEntityField(annotations: Seq[Annotation]): Seq[Annotation]
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.metadata.getOrElse("entity", annotation.metadata.getOrElse("identifier", "")).toString

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByWhiteAndBlackList(annotation: Annotation): Boolean
Filter annotation by blackList and whiteList, taking into account the caseSensitive param.
Filter annotation by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.result
returns
Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def filterByWhiteAndBlackList(annotations: Seq[Annotation]): Seq[Annotation]
Filter annotations by blackList and whiteList, taking into account the caseSensitive param.
Filter annotations by blackList and whiteList, taking into account the caseSensitive param. It filters by annotation.result

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def finalize(): Unit

Attributes
protected[lang]
Definition Classes
AnyRef
Annotations
@throws( classOf[java.lang.Throwable] )
val fixImbalance: BooleanParam
Fix the imbalance in the training set by replicating examples of under represented categories.
Fix the imbalance in the training set by replicating examples of under represented categories.
The class weights are only used by the loss of graphs with a softmax output (the GenericClassifier/RelationExtraction family). Graphs with a sigmoid output (logistic regression, SVM) compute their loss without them, so this parameter has no effect there.

Definition Classes
GenericClassifierParams
def get[T](feature: StructFeature[T]): Option[T]

Attributes
protected
Definition Classes
HasFeatures
def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: SetFeature[T]): Option[Set[T]]

Attributes
protected
Definition Classes
HasFeatures
def get[T](feature: ArrayFeature[T]): Option[Array[T]]

Attributes
protected
Definition Classes
HasFeatures
final def get[T](param: Param[T]): Option[T]

Definition Classes
Params
def getBatchSize: Int
Batch size
Batch size

Definition Classes
GenericClassifierParams
def getBlackList: Array[String]
Gets blackList param
Gets blackList param

Definition Classes
WhiteAndBlackListParams
def getCaseSensitive: Boolean
Gets caseSensitive param
Gets caseSensitive param

Definition Classes
WhiteAndBlackListParams
def getCategories(): Array[String]

Definition Classes
GenericClassifierModel
def getCategoryName(id: Int): String

Definition Classes
GenericClassifierModel
final def getClass(): Class[_]

Definition Classes
AnyRef → Any
Annotations
@native()
def getClasses: Array[String]
Returns labels used to train this model
def getDatasetInfo: String
get descriptive information about the dataset being used
get descriptive information about the dataset being used

Definition Classes
GenericClassifierParams
final def getDefault[T](param: Param[T]): Option[T]

Definition Classes
Params
def getDropout: Float
Dropout coefficient
Dropout coefficient

Definition Classes
GenericClassifierParams
def getEncoder: GenericClassifierDataEncoder

Definition Classes
GenericClassifierModel
def getFeatureScaling: String
Get feature scaling method
Get feature scaling method

Definition Classes
GenericClassifierParams
def getFixImbalance: Boolean
Fix imbalance in training set
Fix imbalance in training set

Definition Classes
GenericClassifierParams
def getGraphFolder: String
Folder with the TensorFlow graphs to choose the model graph from
Folder with the TensorFlow graphs to choose the model graph from

Definition Classes
GenericClassifierParams
def getHasDifferentiableHead: Boolean
Whether the model has a differentiable head or not
def getInputCols: Array[String]

Definition Classes
HasInputAnnotationCols
def getIsTrainable: Boolean
def getLabelColumn: String
Column with label per each document
Column with label per each document

Definition Classes
GenericClassifierParams
def getLazyAnnotator: Boolean

Definition Classes
CanBeLazy
def getLearningRate: Float
Learning Rate
Learning Rate

Definition Classes
GenericClassifierParams
def getLicenseScopes: Seq[String]

Attributes
protected
def getMaxEpochs: Int
Maximum number of epochs to train
Maximum number of epochs to train

Definition Classes
GenericClassifierParams
def getModelFile: String
Path of the TensorFlow graph to train with, empty when it is resolved automatically
Path of the TensorFlow graph to train with, empty when it is resolved automatically

Definition Classes
GenericClassifierParams
def getMultiClass: Boolean
Gets the model multi class prediction mode
Gets the model multi class prediction mode

Definition Classes
GenericClassifierParams
def getOnnxModel: Option[OnnxSetFitClassifierHead]
final def getOrDefault[T](param: Param[T]): T

Definition Classes
Params
final def getOutputCol: String

Definition Classes
HasOutputAnnotationCol
def getOutputLogsPath: String
Get output logs path
Get output logs path

Definition Classes
GenericClassifierParams
def getParam(paramName: String): Param[Any]

Definition Classes
Params
def getStorageRef: String

Definition Classes
HasStorageRef
def getValidationSplit: Float
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.

Definition Classes
GenericClassifierParams
def getWhiteList: Array[String]
Gets whiteList param
Gets whiteList param

Definition Classes
WhiteAndBlackListParams
val graphFolder: Param[String]
Folder with TensorFlow graphs to choose the model graph from.
Folder with TensorFlow graphs to choose the model graph from.
When it is not set, the graphs embedded in the library are used. It is ignored when modelFile is set explicitly.

Definition Classes
GenericClassifierParams
final def hasDefault[T](param: Param[T]): Boolean

Definition Classes
Params
val hasDifferentiableHead: BooleanParam
Whether the model has a differentiable head or not
def hasParam(paramName: String): Boolean

Definition Classes
Params
def hasParent: Boolean

Definition Classes
Model
def hashCode(): Int

Definition Classes
AnyRef → Any
Annotations
@native()
val inExceptionMode: Boolean

Attributes
protected
Definition Classes
HasSafeAnnotate
def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean

Attributes
protected
Definition Classes
Logging
def initializeLogIfNecessary(isInterpreter: Boolean): Unit

Attributes
protected
Definition Classes
Logging
val inputAnnotatorTypes: Array[AnnotatorType]
Input annotator types : DOCUMENT, CHUNK
Input annotator types : DOCUMENT, CHUNK

Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → HasInputAnnotationCols
final val inputCols: StringArrayParam

Attributes
protected
Definition Classes
HasInputAnnotationCols
final def isDefined(param: Param[_]): Boolean

Definition Classes
Params
final def isInstanceOf[T0]: Boolean

Definition Classes
Any
final def isSet(param: Param[_]): Boolean

Definition Classes
Params
def isTraceEnabled(): Boolean

Attributes
protected
Definition Classes
Logging
val isTrainable: BooleanParam
def isValueInList(value: String, list: Array[String]): Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
def isWhiteListAndBlacklistEmpty: Boolean

Attributes
protected
Definition Classes
WhiteAndBlackListParams
val labelColumn: Param[String]
Column with label per each document
Column with label per each document

Definition Classes
GenericClassifierParams
val labels: MapFeature[Int, String]
Labels used to decode predicted IDs back to string tags
val lazyAnnotator: BooleanParam

Definition Classes
CanBeLazy
val learningRate: FloatParam
Learning Rate
Learning Rate

Definition Classes
GenericClassifierParams
def log: Logger

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logDebug(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logError(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logInfo(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logName: String

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logTrace(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String, throwable: Throwable): Unit

Attributes
protected
Definition Classes
Logging
def logWarning(msg: ⇒ String): Unit

Attributes
protected
Definition Classes
Logging
def model: TensorflowGenericClassifier

Definition Classes
GenericClassifierModel
val modelFile: Param[String]
Path of the TensorFlow graph to train with.
Path of the TensorFlow graph to train with.
Empty by default: a graph with suitable dimensions is then selected automatically, either from graphFolder or from the graphs embedded in the library. Set it to train with one specific graph.

Definition Classes
GenericClassifierParams
def msgHelper(schema: StructType): String

Attributes
protected
Definition Classes
HasInputAnnotationCols
val multiClass: BooleanParam
If multiClass is set, the model will return all the labels with corresponding scores.
If multiClass is set, the model will return all the labels with corresponding scores. By default, multiClass is false.

Definition Classes
GenericClassifierParams
final def ne(arg0: AnyRef): Boolean

Definition Classes
AnyRef
final def notify(): Unit

Definition Classes
AnyRef
Annotations
@native()
final def notifyAll(): Unit

Definition Classes
AnyRef
Annotations
@native()
def onWrite(path: String, spark: SparkSession): Unit

Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → ParamsAndFeaturesWritable
val optionalInputAnnotatorTypes: Array[String]

Definition Classes
HasInputAnnotationCols
val outputAnnotatorType: AnnotatorType
Output Annotator Types: WORD_EMBEDDINGS
Output Annotator Types: WORD_EMBEDDINGS

Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → HasOutputAnnotatorType
final val outputCol: Param[String]

Attributes
protected
Definition Classes
HasOutputAnnotationCol
val outputLogsPath: Param[String]
Folder path to save training logs.
Folder path to save training logs. If no path is specified, the logs won't be stored in disk. The path can be a local file path, a distributed file path (HDFS, DBFS), or a cloud storage (S3).

Definition Classes
GenericClassifierParams
lazy val params: Array[Param[_]]

Definition Classes
Params
var parent: Estimator[GenericClassifierModel]

Definition Classes
Model
def safeAnnotate(annotations: Seq[Annotation]): Seq[Annotation]
A protected method designed to safely annotate a sequence of Annotation objects by handling exceptions.
A protected method designed to safely annotate a sequence of Annotation objects by handling exceptions.
annotations
A sequence of Annotation.
returns
A sequence of Annotation objects after processing, potentially containing error annotations.

Attributes
protected
Definition Classes
HasSafeAnnotate
def save(path: String): Unit

Definition Classes
MLWritable
Annotations
@Since( "1.6.0" ) @throws( ... )
def scaleFeatures(features: Array[Array[Float]]): Array[Array[Float]]

Attributes
protected
Definition Classes
GenericClassifierModel
def set[T](feature: StructFeature[T], value: T): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: SetFeature[T], value: Set[T]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def set[T](feature: ArrayFeature[T], value: Array[T]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def set(paramPair: ParamPair[_]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
Params
final def set(param: String, value: Any): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
Params
final def set[T](param: Param[T], value: T): FewShotAssertionClassifierModel.this.type

Definition Classes
Params
def setAllowList(list: String*): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setAllowList(list: Array[String]): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setBatchSize(batch: Int): FewShotAssertionClassifierModel.this.type
Batch size
Batch size

Definition Classes
GenericClassifierParams
def setBlackList(list: String*): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setBlackList(list: Array[String]): FewShotAssertionClassifierModel.this.type
If defined, list of entities to ignore.
If defined, list of entities to ignore. The rest will be processed. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
def setCaseSensitive(value: Boolean): FewShotAssertionClassifierModel.this.type
Determines whether the definitions of the white listed and black listed entities are case sensitive or not.
Determines whether the definitions of the white listed and black listed entities are case sensitive or not. Default: true

Definition Classes
WhiteAndBlackListParams
def setCategoryNames(categoryNames: Array[String]): FewShotAssertionClassifierModel.this.type

Definition Classes
GenericClassifierModel
def setDatasetInfo(value: String): FewShotAssertionClassifierModel.this.type
set descriptive information about the dataset being used
set descriptive information about the dataset being used

Definition Classes
GenericClassifierParams
def setDefault[T](feature: StructFeature[T], value: () ⇒ T): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
HasFeatures
final def setDefault(paramPairs: ParamPair[_]*): FewShotAssertionClassifierModel.this.type

Attributes
protected
Definition Classes
Params
final def setDefault[T](param: Param[T], value: T): FewShotAssertionClassifierModel.this.type

Attributes
protected[org.apache.spark.ml]
Definition Classes
Params
def setDenyList(list: String*): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setDenyList(list: Array[String]): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setDoExceptionHandling(value: Boolean): FewShotAssertionClassifierModel.this.type
If true, exceptions are handled.
If true, exceptions are handled. If exception causing data is passed to the model, a error annotation is emitted which has the exception message. Processing continues with the next one. This comes with a performance penalty.

Definition Classes
HandleExceptionParams
def setDropout(dropout: Float): FewShotAssertionClassifierModel.this.type
Dropout coefficient
Dropout coefficient

Definition Classes
GenericClassifierParams
def setEncoder(encoder: GenericClassifierDataEncoder): FewShotAssertionClassifierModel.this.type

Definition Classes
GenericClassifierModel
def setEpochsNumber(epochs: Int): FewShotAssertionClassifierModel.this.type
Maximum number of epochs to train
Maximum number of epochs to train

Definition Classes
GenericClassifierParams
def setExtraInputCols(value: Array[String]): FewShotAssertionClassifierModel.this.type

Definition Classes
HasInputAnnotationCols
def setFeatureScaling(featureScaling: String): FewShotAssertionClassifierModel.this.type
Set the feature scaling method.
Set the feature scaling method. Possible values are 'zscore', 'minmax' or empty (no scaling)

Definition Classes
GenericClassifierParams
def setFixImbalance(fix: Boolean): FewShotAssertionClassifierModel.this.type
Fix imbalance of training set
Fix imbalance of training set

Definition Classes
GenericClassifierParams
def setGraphFolder(path: String): FewShotAssertionClassifierModel.this.type
Set the folder with the TensorFlow graphs to choose the model graph from.
Set the folder with the TensorFlow graphs to choose the model graph from. Ignored when modelFile is set.

Definition Classes
GenericClassifierParams
def setHasDifferentiableHead(value: Boolean): FewShotAssertionClassifierModel.this.type
Whether the model has a differentiable head or not
final def setInputCols(value: String*): FewShotAssertionClassifierModel.this.type

Definition Classes
HasInputAnnotationCols
def setInputCols(value: Array[String]): FewShotAssertionClassifierModel.this.type

Definition Classes
HasInputAnnotationCols
def setIsTrainable(value: Boolean): FewShotAssertionClassifierModel.this.type
def setLabelColumn(column: String): FewShotAssertionClassifierModel.this.type
Column with label per each document
Column with label per each document

Definition Classes
GenericClassifierParams
def setLabels(value: Map[Int, String]): FewShotAssertionClassifierModel.this.type
def setLazyAnnotator(value: Boolean): FewShotAssertionClassifierModel.this.type

Definition Classes
CanBeLazy
def setModelFile(modelFile: String): FewShotAssertionClassifierModel.this.type
Set the model file name
Set the model file name

Definition Classes
GenericClassifierParams
def setMultiClass(value: Boolean): FewShotAssertionClassifierModel.this.type
Sets the model in multi class prediction mode
Sets the model in multi class prediction mode

Definition Classes
GenericClassifierParams
def setOnnxModelIfNotSet(spark: SparkSession, classifierOnnxWrapper: InternalOnnxWrapper): FewShotAssertionClassifierModel.this.type
final def setOutputCol(value: String): FewShotAssertionClassifierModel.this.type

Definition Classes
HasOutputAnnotationCol
def setOutputLogsPath(outputLogsPath: String): FewShotAssertionClassifierModel.this.type
Set the output log path
Set the output log path

Definition Classes
GenericClassifierParams
def setParent(parent: Estimator[GenericClassifierModel]): GenericClassifierModel

Definition Classes
Model
def setStorageRef(value: String): FewShotAssertionClassifierModel.this.type

Definition Classes
HasStorageRef
def setTensorflowModel(spark: SparkSession, tf: TensorflowWrapper): FewShotAssertionClassifierModel.this.type

Definition Classes
GenericClassifierModel
def setValidationSplit(validationSplit: Float): FewShotAssertionClassifierModel.this.type
Choose the proportion of training dataset to be validated against the model on each Epoch.
Choose the proportion of training dataset to be validated against the model on each Epoch. The value should be between 0.0 and 1.0 and by default it is 0.0 and off.

Definition Classes
GenericClassifierParams
def setWhiteList(list: String*): FewShotAssertionClassifierModel.this.type

Definition Classes
WhiteAndBlackListParams
def setWhiteList(list: Array[String]): FewShotAssertionClassifierModel.this.type
Sets the list of entities to process.
Sets the list of entities to process. The rest will be ignored. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
def setlearningRate(lr: Float): FewShotAssertionClassifierModel.this.type
Learning Rate
Learning Rate

Definition Classes
GenericClassifierParams
val storageRef: Param[String]

Definition Classes
HasStorageRef
final def synchronized[T0](arg0: ⇒ T0): T0

Definition Classes
AnyRef
def toString(): String

Definition Classes
Identifiable → AnyRef → Any
def trainedLabelIds: Option[Array[Int]]
Output units of the graph which correspond to a class learned during training.
Output units of the graph which correspond to a class learned during training. It is None for models without category names, in which case every output unit is considered valid.

Attributes
protected
Definition Classes
GenericClassifierModel
final def transform(dataset: Dataset[_]): DataFrame

Definition Classes
AnnotatorModel → Transformer
def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" )
def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame

Definition Classes
Transformer
Annotations
@Since( "2.0.0" ) @varargs()
final def transformSchema(schema: StructType): StructType

Definition Classes
RawAnnotator → PipelineStage
def transformSchema(schema: StructType, logging: Boolean): StructType

Attributes
protected
Definition Classes
PipelineStage
Annotations
@DeveloperApi()
val uid: String

Definition Classes
FewShotAssertionClassifierModel → GenericClassifierModel → Identifiable
def validate(schema: StructType): Boolean

Attributes
protected
Definition Classes
RawAnnotator
def validateStorageRef(dataset: Dataset[_], inputCols: Array[String], annotatorType: String): Unit

Definition Classes
HasStorageRef
val validationSplit: FloatParam
The proportion of training dataset to be used as validation set.
The proportion of training dataset to be used as validation set.
The model will be validated against this dataset on each Epoch and will not be used for training. The value should be between 0.0 and 1.0.

Definition Classes
GenericClassifierParams
final def wait(): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long, arg1: Int): Unit

Definition Classes
AnyRef
Annotations
@throws( ... )
final def wait(arg0: Long): Unit

Definition Classes
AnyRef
Annotations
@throws( ... ) @native()
val whiteList: StringArrayParam
If defined, list of entities to process.
If defined, list of entities to process. The rest will be ignored. Should not include IOB prefix on labels. Default: Array()

Definition Classes
WhiteAndBlackListParams
def wrapColumnMetadata(col: Column): Column

Attributes
protected
Definition Classes
RawAnnotator
def write: MLWriter

Definition Classes
ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: InternalOnnxWrapper, suffix: String, fileName: String, encrypt: Boolean): Unit

Definition Classes
InternalWriteOnnxModel
def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: InternalOnnxWrapper, suffix: String, fileName: String): Unit

Definition Classes
InternalWriteOnnxModel
def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(InternalOnnxWrapper, String)], suffix: String, encrypt: Boolean = false): Unit

Definition Classes
InternalWriteOnnxModel
def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(InternalOnnxWrapper, String)], suffix: String): Unit

Definition Classes
InternalWriteOnnxModel
def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit

Definition Classes
WriteTensorflowModel
def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit

Definition Classes
WriteTensorflowModel

Packages

FewShotAssertionClassifierModel 

Companion object FewShotAssertionClassifierModel

class FewShotAssertionClassifierModel extends GenericClassifierModel with HasStorageRef with WhiteAndBlackListParams with InternalWriteOnnxModel

Example

Instance Constructors

Type Members

Value Members

Inherited from InternalWriteOnnxModel

Inherited from WhiteAndBlackListParams

Inherited from GenericClassifierModel

Inherited from CheckLicense

Inherited from HasSafeAnnotate[GenericClassifierModel]

Inherited from HandleExceptionParams

Inherited from HasSimpleAnnotate[GenericClassifierModel]

Inherited from WriteTensorflowModel

Inherited from HasStorageRef

Inherited from GenericClassifierParams

Inherited from AnnotatorModel[GenericClassifierModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[GenericClassifierModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[GenericClassifierModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

Annotator types

Members

Parameter setters

Parameter getters

FewShotAssertionClassifierModel