class FinanceBertForSequenceClassification extends MedicalBertForSequenceClassification
MedicalBertForSequenceClassification can load Bert Models with sequence classification/regression head on top (a linear layer on top of the pooled output) e.g. for multi-class document classification tasks.
Pretrained models can be loaded with pretrained
of the companion object:
val sequenceClassifier = MedicalBertForSequenceClassification.pretrained() .setInputCols("token", "document") .setOutputCol("label")
The default model is "bert_sequence_classifier_ade"
, if no name is provided.
For available pretrained models please see the Models Hub.
Models from the HuggingFace 🤗 Transformers library are also compatible with Spark NLP 🚀. The Spark NLP Workshop example shows how to import them https://github.com/JohnSnowLabs/spark-nlp/discussions/5669.
Example
import spark.implicits._ import com.johnsnowlabs.nlp.base._ import com.johnsnowlabs.nlp.annotator._ import org.apache.spark.ml.Pipeline val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val tokenizer = new Tokenizer() .setInputCols("document") .setOutputCol("token") val sequenceClassifier = MedicalBertForSequenceClassification.pretrained() .setInputCols("token", "document") .setOutputCol("label") .setCaseSensitive(true) val pipeline = new Pipeline().setStages(Array( documentAssembler, tokenizer, sequenceClassifier )) val data = Seq("John Lenon was born in London and lived in Paris. My name is Sarah and I live in London").toDF("text") val result = pipeline.fit(data).transform(data) result.select("label.result").show(false) +--------------------+ |result | +--------------------+ |[True, False] | +--------------------+
- See also
MedicalBertForSequenceClassification for sequnece-level classification
Annotators Main Page for a list of transformer based classifiers
- Grouped
- Alphabetic
- By Inheritance
- FinanceBertForSequenceClassification
- MedicalBertForSequenceClassification
- CheckLicense
- HasEngine
- HasCaseSensitiveProperties
- WriteOnnxModel
- WriteTensorflowModel
- HasBatchedAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Type Members
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
takes a document and annotations and produces new annotations of this annotator's annotation type
takes a document and annotations and produces new annotations of this annotator's annotation type
- batchedAnnotations
Annotations that correspond to inputAnnotationCols generated by previous annotators if any
- returns
any number of annotations processed for every input annotation. Not necessary one to one relationship
- Definition Classes
- FinanceBertForSequenceClassification → MedicalBertForSequenceClassification → HasBatchedAnnotate
-
def
batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotate
-
val
batchSize: IntParam
- Definition Classes
- HasBatchedAnnotate
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- FinanceBertForSequenceClassification → MedicalBertForSequenceClassification → AnnotatorModel
-
val
caseSensitive: BooleanParam
- Definition Classes
- HasCaseSensitiveProperties
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
def
checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScope(scope: String): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
final
def
clear(param: Param[_]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
coalesceSentences: BooleanParam
Instead of 1 class per sentence (if inputCols is sentence) output 1 class per document by averaging probabilities in all sentences.
Instead of 1 class per sentence (if inputCols is sentence) output 1 class per document by averaging probabilities in all sentences. Due to max sequence length limit in almost all transformer models such as BERT (512 tokens), this parameter helps feeding all the sentences into the model and averaging all the probabilities for the entire document instead of probabilities per sentence. (Default: false)
- Definition Classes
- MedicalBertForSequenceClassification
-
val
configProtoBytes: IntArrayParam
ConfigProto from tensorflow, serialized into byte array.
ConfigProto from tensorflow, serialized into byte array. Get with
config_proto.SerializeToString()
- Definition Classes
- MedicalBertForSequenceClassification
-
def
copy(extra: ParamMap): MedicalBertForSequenceClassification
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
engine: Param[String]
- Definition Classes
- HasEngine
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getBatchSize: Int
- Definition Classes
- HasBatchedAnnotate
-
def
getCaseSensitive: Boolean
- Definition Classes
- HasCaseSensitiveProperties
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getClasses: Array[String]
Returns labels used to train this model
Returns labels used to train this model
- Definition Classes
- MedicalBertForSequenceClassification
-
def
getCoalesceSentences: Boolean
- Definition Classes
- MedicalBertForSequenceClassification
-
def
getConfigProtoBytes: Option[Array[Byte]]
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getEngine: String
- Definition Classes
- HasEngine
-
def
getInputCols: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
def
getLicenseScopes: Seq[String]
- Attributes
- protected
- Definition Classes
- MedicalBertForSequenceClassification
-
def
getMaxSentenceLength: Int
- Definition Classes
- MedicalBertForSequenceClassification
-
def
getModelIfNotSet: MedicalBertClassification
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
def
getSignatures: Option[Map[String, String]]
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[String]
Input Annotator Types: DOCUMENT, TOKEN
Input Annotator Types: DOCUMENT, TOKEN
- Definition Classes
- MedicalBertForSequenceClassification → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
labels: MapFeature[String, Int]
Labels used to decode predicted IDs back to string tags
Labels used to decode predicted IDs back to string tags
- Definition Classes
- MedicalBertForSequenceClassification
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
maxSentenceLength: IntParam
Max sentence length to process (Default:
128
)Max sentence length to process (Default:
128
)- Definition Classes
- MedicalBertForSequenceClassification
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- MedicalBertForSequenceClassification → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Output Annotator Types: CATEGORY
Output Annotator Types: CATEGORY
- Definition Classes
- MedicalBertForSequenceClassification → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[MedicalBertForSequenceClassification]
- Definition Classes
- Model
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
sentenceEndTokenId: Int
- Definition Classes
- MedicalBertForSequenceClassification
-
def
sentenceStartTokenId: Int
- Definition Classes
- MedicalBertForSequenceClassification
-
def
set[T](feature: StructFeature[T], value: T): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): FinanceBertForSequenceClassification.this.type
- Definition Classes
- Params
-
def
setBatchSize(size: Int): FinanceBertForSequenceClassification.this.type
- Definition Classes
- HasBatchedAnnotate
-
def
setCaseSensitive(value: Boolean): FinanceBertForSequenceClassification.this.type
- Definition Classes
- HasCaseSensitiveProperties
-
def
setCoalesceSentences(value: Boolean): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setConfigProtoBytes(bytes: Array[Int]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): FinanceBertForSequenceClassification.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): FinanceBertForSequenceClassification.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
final
def
setInputCols(value: String*): FinanceBertForSequenceClassification.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setLabels(value: Map[String, Int]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setLazyAnnotator(value: Boolean): FinanceBertForSequenceClassification.this.type
- Definition Classes
- CanBeLazy
-
def
setMaxSentenceLength(value: Int): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setModelIfNotSet(spark: SparkSession, onnxWrapper: OnnxWrapper, sentenceSeparator: Option[String]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setModelIfNotSet(spark: SparkSession, tensorflowWrapper: TensorflowWrapper, sentenceSeparator: Option[String] = None): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
setOutputCol(value: String): FinanceBertForSequenceClassification.this.type
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[MedicalBertForSequenceClassification]): MedicalBertForSequenceClassification
- Definition Classes
- Model
-
def
setSignatures(value: Map[String, String]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
def
setVocabulary(value: Map[String, Int]): FinanceBertForSequenceClassification.this.type
- Definition Classes
- MedicalBertForSequenceClassification
-
val
signatures: MapFeature[String, String]
It contains TF model signatures for the laded saved model
It contains TF model signatures for the laded saved model
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- FinanceBertForSequenceClassification → MedicalBertForSequenceClassification → Identifiable
-
def
validate(schema: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
val
vocabulary: MapFeature[String, Int]
Vocabulary used to encode the words to ids with WordPieceEncoder
Vocabulary used to encode the words to ids with WordPieceEncoder
- Definition Classes
- MedicalBertForSequenceClassification
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
-
def
writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit
- Definition Classes
- WriteTensorflowModel
Inherited from MedicalBertForSequenceClassification
Inherited from CheckLicense
Inherited from HasEngine
Inherited from HasCaseSensitiveProperties
Inherited from WriteOnnxModel
Inherited from WriteTensorflowModel
Inherited from HasBatchedAnnotate[MedicalBertForSequenceClassification]
Inherited from AnnotatorModel[MedicalBertForSequenceClassification]
Inherited from CanBeLazy
Inherited from RawAnnotator[MedicalBertForSequenceClassification]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[MedicalBertForSequenceClassification]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Annotator types
Required input and expected output annotator types