sparknlp_jsl.annotator.classification.large_few_shot_classifier
#
Module Contents#
Classes#
LargeFewShotClassifierModel annotator can run large (LLMS based) few shot classifiers based on the SetFit approach. |
- class LargeFewShotClassifierModel(classname='com.johnsnowlabs.nlp.annotators.classification.LargeFewShotClassifierModel', java_model=None)#
Bases:
sparknlp_jsl.common.AnnotatorModelInternal
,sparknlp.common.HasCaseSensitiveProperties
,sparknlp.common.HasBatchedAnnotate
,sparknlp.common.HasMaxSentenceLengthLimit
LargeFewShotClassifierModel annotator can run large (LLMS based) few shot classifiers based on the SetFit approach.
- batchSize
Batch size
- caseSensitive
Whether the classifier is senstivie to text casing
- maxSentenceLength
The maximum length of the input text
>>> document_assembler = sparknlp.DocumentAssembler() ... .setInputCol("text") ... .setOutputCol("document")
>>> large_few_shot_classifier = LargeFewShotClassifierModel().pretrained() ... .setInputCols("document") ... .setOutputCol("label") >>> data = spark.createDataFrame( ... [["Hypertension:includes hypertension and chronic obstructive pulmonary disease."]] ... ).toDF("text") >>> results = sparknlp.base.Pipeline() ... .setStages([document_assembler, large_few_shot_classifier]) ... .fit(data) ... .transform(data) >>> results ... .selectExpr("explode(label) as label") ... .selectExpr("label.result", "label.metadata.confidence") ... .show()
result|confidence|conditional
0.34774542
- batchSize#
- caseSensitive#
- getter_attrs = []#
- hasDifferentiableHead#
- inputAnnotatorTypes#
- inputCols#
- lazyAnnotator#
- maxSentenceLength#
- max_length_limit = 512#
- modelArchitecture#
- name = LargeFewShotClassifierModel#
- optionalInputAnnotatorTypes = []#
- outputAnnotatorType#
- outputCol#
- skipLPInputColsValidation = True#
- clear(param)#
Clears a param from the param map if it has been explicitly set.
- copy(extra=None)#
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.
- Parameters:
extra (dict, optional) – Extra parameters to copy to the new instance
- Returns:
Copy of this instance
- Return type:
JavaParams
- explainParam(param)#
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
- explainParams()#
Returns the documentation of all params with their optionally default values and user-supplied values.
- extractParamMap(extra=None)#
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
- Parameters:
extra (dict, optional) – extra param values
- Returns:
merged param map
- Return type:
dict
- getBatchSize()#
Gets current batch size.
- Returns:
Current batch size
- Return type:
int
- getCaseSensitive()#
Gets whether to ignore case in tokens for embeddings matching.
- Returns:
Whether to ignore case in tokens for embeddings matching
- Return type:
bool
- getClasses()#
Returns labels used to train this model
- getInputCols()#
Gets current column names of input annotations.
- getLazyAnnotator()#
Gets whether Annotator should be evaluated lazily in a RecursivePipeline.
- getMaxSentenceLength()#
Gets max sentence of the model.
- Returns:
Max sentence length to process
- Return type:
int
- getOrDefault(param)#
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
- getOutputCol()#
Gets output column name of annotations.
- getParam(paramName)#
Gets a param by its name.
- getParamValue(paramName)#
Gets the value of a parameter.
- Parameters:
paramName (str) – Name of the parameter
- hasDefault(param)#
Checks whether a param has a default value.
- hasParam(paramName)#
Tests whether this instance contains a param with a given (string) name.
- inputColsValidation(value)#
- isDefined(param)#
Checks whether a param is explicitly set by user or has a default value.
- isSet(param)#
Checks whether a param is explicitly set by user.
- classmethod load(path)#
Reads an ML instance from the input path, a shortcut of read().load(path).
- static loadSavedModel(folder, spark_session, model_architecture, has_differentiable_head=False)#
Loads a locally saved model.
- Parameters:
folder (str) – Folder of the saved model
spark_session (pyspark.sql.SparkSession) – The current SparkSession
model_architecture (str) – The model architecture of the underlying sentence embeddings model, e.g. MPNet or Bert
has_differentiable_head (bool) – A flag indicating whether the classifier is differentiable
- Returns:
The restored model
- Return type:
- static pretrained(name='', lang='en', remote_loc='clinical/models')#
Downloads and loads a pretrained model.
- Parameters:
name (str, optional) – Name of the pretrained model, by default “”
lang (str, optional) – Language of the pretrained model, by default “en”
remote_loc (str, optional) – Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.
- Returns:
The restored model
- Return type:
- classmethod read()#
Returns an MLReader instance for this class.
- save(path)#
Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
- set(param, value)#
Sets a parameter in the embedded param map.
- setBatchSize(v)#
Sets batch size.
- Parameters:
v (int) – Batch size
- setCaseSensitive(value)#
Sets whether to ignore case in tokens for embeddings matching.
- Parameters:
value (bool) – Whether to ignore case in tokens for embeddings matching
- setForceInputTypeValidation(etfm)#
- setInputCols(*value)#
Sets column names of input annotations.
- Parameters:
*value (List[str]) – Input columns for the annotator
- setLazyAnnotator(value)#
Sets whether Annotator should be evaluated lazily in a RecursivePipeline.
- Parameters:
value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline
- setMaxSentenceLength(value)#
Sets max sentence length to process.
Note that a maximum limit exists depending on the model. If you are working with long single sequences, consider splitting up the input first with another annotator e.g. SentenceDetector.
- Parameters:
value (int) – Max sentence length to process
- setOutputCol(value)#
Sets output column name of annotations.
- Parameters:
value (str) – Name of output column
- setParamValue(paramName)#
Sets the value of a parameter.
- Parameters:
paramName (str) – Name of the parameter
- setParams()#
- transform(dataset, params=None)#
Transforms the input dataset with optional parameters.
New in version 1.3.0.
- Parameters:
dataset (
pyspark.sql.DataFrame
) – input datasetparams (dict, optional) – an optional param map that overrides embedded params.
- Returns:
transformed dataset
- Return type:
- write()#
Returns an MLWriter instance for this ML instance.