sparknlp_jsl.annotator.classification.few_shot_assertion_classifier
#
Module Contents#
Classes#
FewShotAssertionClassifierModel does assertion classification using can run large (LLMS based) |
- class FewShotAssertionClassifierModel(classname='com.johnsnowlabs.nlp.annotators.classification.LargeFewShotClassifierModel', java_model=None)#
Bases:
sparknlp_jsl.annotator.classification.large_few_shot_classifier.LargeFewShotClassifierModel
FewShotAssertionClassifierModel does assertion classification using can run large (LLMS based) few shot classifiers based on the SetFit approach.
Input Annotation types
Output Annotation type
DOCUMENT
,CHUNK
ASSERTION
- batchSize
Batch size
- caseSensitive
Whether the classifier is senstivive to text casing
- maxSentenceLength
The maximum length of the input text
>>> document_assembler = sparknlp.DocumentAssembler() ... .setInputCol("text") ... .setOutputCol("document") ... >>> sentence_detector = SentenceDetector() ... .setInputCol("document") ... .setOutputCol("sentence") ... >>> tokenizer = Tokenizer() ... .setInputCols(["sentence"]) ... .setOutputCol("token") ... >>> embeddings = WordEmbeddingsModel.pretrained("embeddings_clinical", "en", "clinical/models") ... .setInputCols(["sentence", "token"]) ... .setOutputCol("embeddings") ... .setCaseSensitive(False) ... >>> ner = MedicalNerModel.pretrained("ner_jsl", "en", "clinical/models") ... .setInputCols(["sentence", "token", "embeddings"]) ... .setOutputCol("ner") ... >>> ner_converter = NerConverter() ... .setInputCols(["sentence", "token", "ner"]) ... .setWhiteList("Disease_Syndrome_Disorder", "Hypertension") ... .setOutputCol("ner_chunk") ... >>> few_shot_assertion_classifier = FewShotAssertionClassifierModel().pretrained() ... .setInputCols(["sentence", "ner_chunk"]) ... .setOutputCol("assertion") ... >>> data = spark.createDataFrame( ... [["Includes hypertension and chronic obstructive pulmonary disease."]] ... ).toDF("text") ... >>> results = sparknlp.base.Pipeline() ... .setStages([ ... document_assembler, sentence_detector, tokenizer, embeddings, ner, ner_converter, ... few_shot_assertion_classifier]) ... .fit(data) ... .transform(data) ... >>> results ... .selectExpr("assertion.result", "assertion.metadata.chunk", "assertion.metadata.confidence") ... .show()
result
chunk
confidence
- present
absent
hypertension
arteriovenous malformations
1.0 1.0
- batchSize#
- caseSensitive#
- getter_attrs = []#
- hasDifferentiableHead#
- inputAnnotatorTypes#
- inputCols#
- lazyAnnotator#
- maxSentenceLength#
- max_length_limit = 512#
- modelArchitecture#
- name = FewShotAssertionClassifierModel#
- optionalInputAnnotatorTypes = []#
- outputAnnotatorType#
- outputCol#
- skipLPInputColsValidation = True#
- clear(param)#
Clears a param from the param map if it has been explicitly set.
- copy(extra=None)#
Creates a copy of this instance with the same uid and some extra params. This implementation first calls Params.copy and then make a copy of the companion Java pipeline component with extra params. So both the Python wrapper and the Java pipeline component get copied.
- Parameters:
extra (dict, optional) – Extra parameters to copy to the new instance
- Returns:
Copy of this instance
- Return type:
JavaParams
- explainParam(param)#
Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.
- explainParams()#
Returns the documentation of all params with their optionally default values and user-supplied values.
- extractParamMap(extra=None)#
Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.
- Parameters:
extra (dict, optional) – extra param values
- Returns:
merged param map
- Return type:
dict
- getBatchSize()#
Gets current batch size.
- Returns:
Current batch size
- Return type:
int
- getCaseSensitive()#
Gets whether to ignore case in tokens for embeddings matching.
- Returns:
Whether to ignore case in tokens for embeddings matching
- Return type:
bool
- getClasses()#
Returns labels used to train this model
- getInputCols()#
Gets current column names of input annotations.
- getLazyAnnotator()#
Gets whether Annotator should be evaluated lazily in a RecursivePipeline.
- getMaxSentenceLength()#
Gets max sentence of the model.
- Returns:
Max sentence length to process
- Return type:
int
- getOrDefault(param)#
Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.
- getOutputCol()#
Gets output column name of annotations.
- getParam(paramName)#
Gets a param by its name.
- getParamValue(paramName)#
Gets the value of a parameter.
- Parameters:
paramName (str) – Name of the parameter
- hasDefault(param)#
Checks whether a param has a default value.
- hasParam(paramName)#
Tests whether this instance contains a param with a given (string) name.
- inputColsValidation(value)#
- isDefined(param)#
Checks whether a param is explicitly set by user or has a default value.
- isSet(param)#
Checks whether a param is explicitly set by user.
- classmethod load(path)#
Reads an ML instance from the input path, a shortcut of read().load(path).
- static loadSavedModel(folder, spark_session, model_architecture, has_differentiable_head=False)#
Loads a locally saved model.
- Parameters:
folder (str) – Folder of the saved model
spark_session (pyspark.sql.SparkSession) – The current SparkSession
model_architecture (str) – The model architecture of the underlying sentence embeddings model, e.g. MPNet or Bert
has_differentiable_head (bool) – A flag indicating whether the classifier is differentiable
- Returns:
The restored model
- Return type:
- static pretrained(name='assertion_fewshotclassifier', lang='en', remote_loc='clinical/models')#
Downloads and loads a pretrained model.
- Parameters:
name (str, optional) – Name of the pretrained model, by default “assertion_fewshotclassifier”
lang (str, optional) – Language of the pretrained model, by default “en”
remote_loc (str, optional) – Optional remote address of the resource, by default None. Will use Spark NLPs repositories otherwise.
- Returns:
The restored model
- Return type:
- classmethod read()#
Returns an MLReader instance for this class.
- save(path)#
Save this ML instance to the given path, a shortcut of ‘write().save(path)’.
- set(param, value)#
Sets a parameter in the embedded param map.
- setBatchSize(v)#
Sets batch size.
- Parameters:
v (int) – Batch size
- setCaseSensitive(value)#
Sets whether to ignore case in tokens for embeddings matching.
- Parameters:
value (bool) – Whether to ignore case in tokens for embeddings matching
- setForceInputTypeValidation(etfm)#
- setInputCols(*value)#
Sets column names of input annotations.
- Parameters:
*value (List[str]) – Input columns for the annotator
- setLazyAnnotator(value)#
Sets whether Annotator should be evaluated lazily in a RecursivePipeline.
- Parameters:
value (bool) – Whether Annotator should be evaluated lazily in a RecursivePipeline
- setMaxSentenceLength(value)#
Sets max sentence length to process.
Note that a maximum limit exists depending on the model. If you are working with long single sequences, consider splitting up the input first with another annotator e.g. SentenceDetector.
- Parameters:
value (int) – Max sentence length to process
- setOutputCol(value)#
Sets output column name of annotations.
- Parameters:
value (str) – Name of output column
- setParamValue(paramName)#
Sets the value of a parameter.
- Parameters:
paramName (str) – Name of the parameter
- setParams()#
- transform(dataset, params=None)#
Transforms the input dataset with optional parameters.
New in version 1.3.0.
- Parameters:
dataset (
pyspark.sql.DataFrame
) – input datasetparams (dict, optional) – an optional param map that overrides embedded params.
- Returns:
transformed dataset
- Return type:
- write()#
Returns an MLWriter instance for this ML instance.