class ZeroShotNerModel extends RoBertaForQuestionAnswering with CheckLicense
ZeroShotNerModel implements zero shot named entity recognition by utilizing RoBERTa transformer models fine tuned on a question answering task. Its input is a list of document annotations and it automatically generates questions which are used to recognize entities. The definitions of entities is given by a dictionary structures, specifying a set of questions for each entity. The model is based on RoBertaForQuestionAnswering from the open source SparkNLP project.
Pretrained models can be loaded with pretrained
of the companion object:
val zeroShotRE = ZeroShotNerModel.pretrained() .setInputCols("document") .setOutputCol("zer_shot_ner")
For available pretrained models please see the Models Hub.
Example
val documentAssembler = new DocumentAssembler() .setInputCol("text") .setOutputCol("document") val sentenceDetector = new SentenceDetector() .setInputCols(Array("document")) .setOutputCol("sentences") val zeroShotNer = ZeroShotNerModel .pretrained() .setEntityDefinitions( Map( "NAME" -> Array("What is his name?", "What is her name?"), "CITY" -> Array("Which city?"))) .setPredictionThreshold(0.01f) .setInputCols("sentences") .setOutputCol("zero_shot_ner") val pipeline = new Pipeline() .setStages(Array( documentAssembler, sentenceDetector, zeroShotNer)) val model = pipeline.fit(Seq("").toDS.toDF("text")) val results = model.transform( Seq("Clara often travels between New York and Paris.").toDS.toDF("text")) results .selectExpr("document", "explode(zero_shot_ner) AS entity") .select( col("entity.result"), col("entity.metadata.word"), col("entity.metadata.sentence"), col("entity.begin"), col("entity.end"), col("entity.metadata.confidence"), col("entity.metadata.question")) .show(truncate=false) +------+-----+--------+-----+---+----------+------------------+ |result|word |sentence|begin|end|confidence|question | +------+-----+--------+-----+---+----------+------------------+ |B-CITY|Paris|0 |41 |45 |0.78655756|Which is the city?| |B-CITY|New |0 |28 |30 |0.29346612|Which city? | |I-CITY|York |0 |32 |35 |0.29346612|Which city? | +------+-----+--------+-----+---+----------+------------------+
- See also
https://arxiv.org/abs/1907.11692 for details about the RoBERTa transformer
RoBertaForQuestionAnswering for the SparkNLP implementation of RoBERTa question answering
- Grouped
- Alphabetic
- By Inheritance
- ZeroShotNerModel
- CheckLicense
- RoBertaForQuestionAnswering
- HasEngine
- HasCaseSensitiveProperties
- WriteOnnxModel
- WriteTensorflowModel
- HasBatchedAnnotate
- AnnotatorModel
- CanBeLazy
- RawAnnotator
- HasOutputAnnotationCol
- HasInputAnnotationCols
- HasOutputAnnotatorType
- ParamsAndFeaturesWritable
- HasFeatures
- DefaultParamsWritable
- MLWritable
- Model
- Transformer
- PipelineStage
- Logging
- Params
- Serializable
- Serializable
- Identifiable
- AnyRef
- Any
- Hide All
- Show All
- Public
- All
Instance Constructors
Type Members
Value Members
-
final
def
!=(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
final
def
##(): Int
- Definition Classes
- AnyRef → Any
-
final
def
$[T](param: Param[T]): T
- Attributes
- protected
- Definition Classes
- Params
-
def
$$[T](feature: StructFeature[T]): T
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[K, V](feature: MapFeature[K, V]): Map[K, V]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: SetFeature[T]): Set[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
$$[T](feature: ArrayFeature[T]): Array[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
==(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
_transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
def
afterAnnotate(dataset: DataFrame): DataFrame
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
final
def
asInstanceOf[T0]: T0
- Definition Classes
- Any
-
def
batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering → HasBatchedAnnotate
-
def
batchProcess(rows: Iterator[_]): Iterator[Row]
- Definition Classes
- HasBatchedAnnotate
-
val
batchSize: IntParam
- Definition Classes
- HasBatchedAnnotate
-
def
beforeAnnotate(dataset: Dataset[_]): Dataset[_]
- Attributes
- protected
- Definition Classes
- AnnotatorModel
-
val
caseSensitive: BooleanParam
- Definition Classes
- HasCaseSensitiveProperties
-
final
def
checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
def
checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScope(scope: String): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
def
checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
- Definition Classes
- CheckLicense
-
final
def
clear(param: Param[_]): ZeroShotNerModel.this.type
- Definition Classes
- Params
-
def
clone(): AnyRef
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
val
configProtoBytes: IntArrayParam
- Definition Classes
- RoBertaForQuestionAnswering
-
def
copy(extra: ParamMap): RoBertaForQuestionAnswering
- Definition Classes
- RawAnnotator → Model → Transformer → PipelineStage → Params
-
def
copyValues[T <: Params](to: T, extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
final
def
defaultCopy[T <: Params](extra: ParamMap): T
- Attributes
- protected
- Definition Classes
- Params
-
val
engine: Param[String]
- Definition Classes
- HasEngine
-
final
def
eq(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
def
equals(arg0: Any): Boolean
- Definition Classes
- AnyRef → Any
-
def
explainParam(param: Param[_]): String
- Definition Classes
- Params
-
def
explainParams(): String
- Definition Classes
- Params
-
def
extraValidate(structType: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
extraValidateMsg: String
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
final
def
extractParamMap(): ParamMap
- Definition Classes
- Params
-
final
def
extractParamMap(extra: ParamMap): ParamMap
- Definition Classes
- Params
-
val
features: ArrayBuffer[Feature[_, _, _]]
- Definition Classes
- HasFeatures
-
def
finalize(): Unit
- Attributes
- protected[lang]
- Definition Classes
- AnyRef
- Annotations
- @throws( classOf[java.lang.Throwable] )
-
def
get[T](feature: StructFeature[T]): Option[T]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: SetFeature[T]): Option[Set[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
get[T](feature: ArrayFeature[T]): Option[Array[T]]
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
get[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getBatchSize: Int
- Definition Classes
- HasBatchedAnnotate
-
def
getCaseSensitive: Boolean
- Definition Classes
- HasCaseSensitiveProperties
-
final
def
getClass(): Class[_]
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
def
getConfigProtoBytes: Option[Array[Byte]]
- Definition Classes
- RoBertaForQuestionAnswering
-
final
def
getDefault[T](param: Param[T]): Option[T]
- Definition Classes
- Params
-
def
getEngine: String
- Definition Classes
- HasEngine
-
def
getEntities: Array[String]
Get the list of entities which are regonized
- def getEntityDefinitionsStr: Array[String]
-
def
getIgnoreEntities: Array[String]
Get the list of questions to catch the distractor entity
-
def
getInputCols: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
def
getLazyAnnotator: Boolean
- Definition Classes
- CanBeLazy
-
def
getMaxSentenceLength: Int
- Definition Classes
- RoBertaForQuestionAnswering
-
def
getModelIfNotSet: RoBertaClassification
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering
-
final
def
getOrDefault[T](param: Param[T]): T
- Definition Classes
- Params
-
final
def
getOutputCol: String
- Definition Classes
- HasOutputAnnotationCol
-
def
getParam(paramName: String): Param[Any]
- Definition Classes
- Params
-
def
getPredictionThreshold: Float
Get the minimum entity prediction score
-
def
getSignatures: Option[Map[String, String]]
- Definition Classes
- RoBertaForQuestionAnswering
-
final
def
hasDefault[T](param: Param[T]): Boolean
- Definition Classes
- Params
-
def
hasParam(paramName: String): Boolean
- Definition Classes
- Params
-
def
hasParent: Boolean
- Definition Classes
- Model
-
def
hashCode(): Int
- Definition Classes
- AnyRef → Any
- Annotations
- @native()
-
var
ignoreEntities: StringArrayParam
List of entities to ignore
-
def
initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
def
initializeLogIfNecessary(isInterpreter: Boolean): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
val
inputAnnotatorTypes: Array[String]
Input Annotator Types: DOCUMENT
Input Annotator Types: DOCUMENT
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering → HasInputAnnotationCols
-
final
val
inputCols: StringArrayParam
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
isDefined(param: Param[_]): Boolean
- Definition Classes
- Params
-
final
def
isInstanceOf[T0]: Boolean
- Definition Classes
- Any
-
final
def
isSet(param: Param[_]): Boolean
- Definition Classes
- Params
- def isTokenInEntity(token: Annotation, entity: Annotation): Boolean
-
def
isTraceEnabled(): Boolean
- Attributes
- protected
- Definition Classes
- Logging
-
val
lazyAnnotator: BooleanParam
- Definition Classes
- CanBeLazy
-
def
log: Logger
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logDebug(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logError(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logInfo(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logName: String
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logTrace(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String, throwable: Throwable): Unit
- Attributes
- protected
- Definition Classes
- Logging
-
def
logWarning(msg: ⇒ String): Unit
- Attributes
- protected
- Definition Classes
- Logging
- def maskEntity(document: Annotation, entity: Annotation): String
- val maskSymbol: String
-
val
maxSentenceLength: IntParam
- Definition Classes
- RoBertaForQuestionAnswering
-
val
merges: MapFeature[(String, String), Int]
- Definition Classes
- RoBertaForQuestionAnswering
-
def
msgHelper(schema: StructType): String
- Attributes
- protected
- Definition Classes
- HasInputAnnotationCols
-
final
def
ne(arg0: AnyRef): Boolean
- Definition Classes
- AnyRef
-
final
def
notify(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
final
def
notifyAll(): Unit
- Definition Classes
- AnyRef
- Annotations
- @native()
-
def
onWrite(path: String, spark: SparkSession): Unit
- Definition Classes
- RoBertaForQuestionAnswering → ParamsAndFeaturesWritable
-
val
optionalInputAnnotatorTypes: Array[String]
- Definition Classes
- HasInputAnnotationCols
-
val
outputAnnotatorType: AnnotatorType
Output Annotator Types: NAMED_ENTITY
Output Annotator Types: NAMED_ENTITY
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering → HasOutputAnnotatorType
-
final
val
outputCol: Param[String]
- Attributes
- protected
- Definition Classes
- HasOutputAnnotationCol
-
def
padTokenId: Int
- Definition Classes
- RoBertaForQuestionAnswering
-
lazy val
params: Array[Param[_]]
- Definition Classes
- Params
-
var
parent: Estimator[RoBertaForQuestionAnswering]
- Definition Classes
- Model
-
var
predictionThreshold: FloatParam
Minimal score of predicted entity
- def recognizeMultipleEntities(document: Annotation, nerDefs: Map[String, Array[Annotation]], recognizedEntities: Seq[Annotation] = Seq()): Seq[Annotation]
-
def
save(path: String): Unit
- Definition Classes
- MLWritable
- Annotations
- @Since( "1.6.0" ) @throws( ... )
-
def
sentenceEndTokenId: Int
- Definition Classes
- RoBertaForQuestionAnswering
-
def
sentenceStartTokenId: Int
- Definition Classes
- RoBertaForQuestionAnswering
-
def
set[T](feature: StructFeature[T], value: T): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[K, V](feature: MapFeature[K, V], value: Map[K, V]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: SetFeature[T], value: Set[T]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
set[T](feature: ArrayFeature[T], value: Array[T]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
set(paramPair: ParamPair[_]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set(param: String, value: Any): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
set[T](param: Param[T], value: T): ZeroShotNerModel.this.type
- Definition Classes
- Params
-
def
setBatchSize(size: Int): ZeroShotNerModel.this.type
- Definition Classes
- HasBatchedAnnotate
-
def
setCaseSensitive(value: Boolean): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering → HasCaseSensitiveProperties
-
def
setConfigProtoBytes(bytes: Array[Int]): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering
-
def
setDefault[T](feature: StructFeature[T], value: () ⇒ T): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
def
setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- HasFeatures
-
final
def
setDefault(paramPairs: ParamPair[_]*): ZeroShotNerModel.this.type
- Attributes
- protected
- Definition Classes
- Params
-
final
def
setDefault[T](param: Param[T], value: T): ZeroShotNerModel.this.type
- Attributes
- protected[org.apache.spark.ml]
- Definition Classes
- Params
-
def
setEntityDefinitions(definitions: HashMap[String, List[String]]): ZeroShotNerModel.this.type
Set definitions of named entities
-
def
setEntityDefinitions(definitions: Map[String, Array[String]]): ZeroShotNerModel.this.type
Set definitions of named entities
-
def
setIgnoreEntities(value: Array[String]): ZeroShotNerModel.this.type
Set the list of questions to catch the distractor entity
-
final
def
setInputCols(value: String*): ZeroShotNerModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setInputCols(value: Array[String]): ZeroShotNerModel.this.type
- Definition Classes
- HasInputAnnotationCols
-
def
setLazyAnnotator(value: Boolean): ZeroShotNerModel.this.type
- Definition Classes
- CanBeLazy
-
def
setMaxSentenceLength(value: Int): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering
-
def
setMerges(value: Map[(String, String), Int]): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering
-
def
setModelIfNotSet(spark: SparkSession, tensorflowWrapper: Option[TensorflowWrapper], onnxWrapper: Option[OnnxWrapper] = None): ZeroShotNerModel
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering
-
final
def
setOutputCol(value: String): ZeroShotNerModel.this.type
- Definition Classes
- HasOutputAnnotationCol
-
def
setParent(parent: Estimator[RoBertaForQuestionAnswering]): RoBertaForQuestionAnswering
- Definition Classes
- Model
-
def
setPredictionThreshold(value: Float): ZeroShotNerModel.this.type
Set the minimum entity prediction score
-
def
setSignatures(value: Map[String, String]): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering
-
def
setVocabulary(value: Map[String, Int]): ZeroShotNerModel.this.type
- Definition Classes
- RoBertaForQuestionAnswering
-
val
signatures: MapFeature[String, String]
- Definition Classes
- RoBertaForQuestionAnswering
-
final
def
synchronized[T0](arg0: ⇒ T0): T0
- Definition Classes
- AnyRef
-
def
toString(): String
- Definition Classes
- Identifiable → AnyRef → Any
-
final
def
transform(dataset: Dataset[_]): DataFrame
- Definition Classes
- AnnotatorModel → Transformer
-
def
transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" )
-
def
transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
- Definition Classes
- Transformer
- Annotations
- @Since( "2.0.0" ) @varargs()
-
final
def
transformSchema(schema: StructType): StructType
- Definition Classes
- RawAnnotator → PipelineStage
-
def
transformSchema(schema: StructType, logging: Boolean): StructType
- Attributes
- protected
- Definition Classes
- PipelineStage
- Annotations
- @DeveloperApi()
-
val
uid: String
- Definition Classes
- ZeroShotNerModel → RoBertaForQuestionAnswering → Identifiable
-
def
validate(schema: StructType): Boolean
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
val
vocabulary: MapFeature[String, Int]
- Definition Classes
- RoBertaForQuestionAnswering
-
final
def
wait(): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long, arg1: Int): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... )
-
final
def
wait(arg0: Long): Unit
- Definition Classes
- AnyRef
- Annotations
- @throws( ... ) @native()
-
def
wrapColumnMetadata(col: Column): Column
- Attributes
- protected
- Definition Classes
- RawAnnotator
-
def
write: MLWriter
- Definition Classes
- ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
-
def
writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
- Definition Classes
- WriteOnnxModel
-
def
writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit
- Definition Classes
- WriteTensorflowModel
-
def
writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit
- Definition Classes
- WriteTensorflowModel
Inherited from CheckLicense
Inherited from RoBertaForQuestionAnswering
Inherited from HasEngine
Inherited from HasCaseSensitiveProperties
Inherited from WriteOnnxModel
Inherited from WriteTensorflowModel
Inherited from HasBatchedAnnotate[RoBertaForQuestionAnswering]
Inherited from AnnotatorModel[RoBertaForQuestionAnswering]
Inherited from CanBeLazy
Inherited from RawAnnotator[RoBertaForQuestionAnswering]
Inherited from HasOutputAnnotationCol
Inherited from HasInputAnnotationCols
Inherited from HasOutputAnnotatorType
Inherited from ParamsAndFeaturesWritable
Inherited from HasFeatures
Inherited from DefaultParamsWritable
Inherited from MLWritable
Inherited from Model[RoBertaForQuestionAnswering]
Inherited from Transformer
Inherited from PipelineStage
Inherited from Logging
Inherited from Params
Inherited from Serializable
Inherited from Serializable
Inherited from Identifiable
Inherited from AnyRef
Inherited from Any
Parameters
A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.
Annotator types
Required input and expected output annotator types