Packages

class MedicalQuestionAnswering extends MedicalTextGenerator

MedicalQuestionAnswering is a GPT based model for answering questions given a context. Unlike span based models, it generates the answers to the questions, rather than selecting phrases from the given context. The model is capable of answering various types of questions, including yes-no or full text ones.

Pretrained models can be loaded with pretrained of the companion object:

val medQA = MedicalQuestionAnswering.pretrained()
  .setInputCols("question_document", "context_document")
  .setOutputCol("answer")

For available pretrained models please see the Models Hub.

Example

 import spark.implicits._

 val documentAssembler = new MultiDocumentAssembler()
   .setInputCols("question", "context")
   .setOutputCols("document_question", "document_context")

val medQA = MedicalQuestionAnswering.pretrained()
  .setInputCols("question_document", "context_document")
  .setOutputCol("answer")
  .setQuestionType("short")

 val pipeline = new Pipeline()
   .setStages(Array(
     documentAssembler,
     medQA))

 val model = pipeline.fit(Seq("", "").toDS.toDF("question", "context"))
 val results = model.transform(
   Seq(
   ("Should chest wall irradiation be included after mastectomy and negative node breast cancer?",
   """
       |This study aims to evaluate local failure patterns in node negative breast cancer patients treated with
       |post-mastectomy radiotherapy including internal mammary chain only. Retrospective analysis of 92 internal
       |or central-breast node-negative tumours with mastectomy and external irradiation of the internal mammary
       |chain at the dose of 50 Gy, from 1994 to 1998. Local recurrence rate was 5 % (five cases). Recurrence
       |sites were the operative scare and chest wall. Factors associated with increased risk of local failure
       |were age<or = 40 years and tumour size greater than 20mm, without statistical significance.
       |""".stripMargin)
   ).toDS.toDF("question", "context"))

 results
   .selectExpr("answer.result")
   .show(truncate = false)

 +-------+
 |result |
 +-------+
 |[yes]  |
 +-------+
See also

https://academic.oup.com/bib/article/23/6/bbac409/6713511 for details about using medical text generation with GPT

Linear Supertypes
MedicalTextGenerator, CheckLicense, HasEngine, WriteSentencePieceModel, WriteOnnxModel, WriteTensorflowModel, HasBatchedAnnotate[MedicalQuestionAnswering], HasCaseSensitiveProperties, GPTGenerationParams, AnnotatorModel[MedicalQuestionAnswering], CanBeLazy, RawAnnotator[MedicalQuestionAnswering], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[MedicalQuestionAnswering], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Known Subclasses
FinanceQuestionAnswering, LegalQuestionAnswering
Ordering
  1. Grouped
  2. Alphabetic
  3. By Inheritance
Inherited
  1. MedicalQuestionAnswering
  2. MedicalTextGenerator
  3. CheckLicense
  4. HasEngine
  5. WriteSentencePieceModel
  6. WriteOnnxModel
  7. WriteTensorflowModel
  8. HasBatchedAnnotate
  9. HasCaseSensitiveProperties
  10. GPTGenerationParams
  11. AnnotatorModel
  12. CanBeLazy
  13. RawAnnotator
  14. HasOutputAnnotationCol
  15. HasInputAnnotationCols
  16. HasOutputAnnotatorType
  17. ParamsAndFeaturesWritable
  18. HasFeatures
  19. DefaultParamsWritable
  20. MLWritable
  21. Model
  22. Transformer
  23. PipelineStage
  24. Logging
  25. Params
  26. Serializable
  27. Serializable
  28. Identifiable
  29. AnyRef
  30. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MedicalQuestionAnswering()
  2. new MedicalQuestionAnswering(uid: String)

    uid

    required uid for storing annotator to disk

Type Members

  1. type AnnotationContent = Seq[Row]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  2. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[T](feature: StructFeature[T]): T
    Attributes
    protected
    Definition Classes
    HasFeatures
  5. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[T](feature: SetFeature[T]): Set[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: ArrayFeature[T]): Array[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. val DOCUMENT_VARIABLE_NAME: String
    Attributes
    protected
    Definition Classes
    MedicalTextGenerator
  10. def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  11. val additionalTokens: MapFeature[Int, String]

    Additional tokens

    Additional tokens

    Definition Classes
    MedicalTextGenerator
  12. def afterAnnotate(dataset: DataFrame): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  13. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  14. def batchAnnotate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]

    takes a document and annotations and produces new annotations of this annotator's annotation type

    takes a document and annotations and produces new annotations of this annotator's annotation type

    batchedAnnotations

    Annotations that correspond to inputAnnotationCols generated by previous annotators if any

    returns

    any number of annotations processed for every input annotation. Not necessary one to one relationship

    Definition Classes
    MedicalQuestionAnswering → MedicalTextGenerator → HasBatchedAnnotate
  15. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotate
  16. val batchSize: IntParam
    Definition Classes
    HasBatchedAnnotate
  17. def batchedAnnotateWithoutPromptTemplate(batchedAnnotations: Seq[Array[Annotation]]): Seq[Seq[Annotation]]
    Definition Classes
    MedicalTextGenerator
  18. def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  19. val caseSensitive: BooleanParam
    Definition Classes
    HasCaseSensitiveProperties
  20. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  21. def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
    Definition Classes
    CheckLicense
  22. def checkValidScope(scope: String): Unit
    Definition Classes
    CheckLicense
  23. def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  24. def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  25. final def clear(param: Param[_]): MedicalQuestionAnswering.this.type
    Definition Classes
    Params
  26. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  27. val configProtoBytes: IntArrayParam

    ConfigProto from tensorflow, serialized into byte array.

    ConfigProto from tensorflow, serialized into byte array. Get with config_proto.SerializeToString()

    Definition Classes
    MedicalTextGenerator
  28. def copy(extra: ParamMap): MedicalQuestionAnswering
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  29. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  30. val customPrompt: Param[String]

    Custom model prompt

    Custom model prompt

    Definition Classes
    MedicalTextGenerator
  31. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  32. val doSample: BooleanParam

    Whether or not to use sampling, use greedy decoding otherwise (Default: false)

    Whether or not to use sampling, use greedy decoding otherwise (Default: false)

    Definition Classes
    GPTGenerationParams
  33. val engine: Param[String]
    Definition Classes
    HasEngine
  34. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  35. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  36. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  37. def explainParams(): String
    Definition Classes
    Params
  38. def extraValidate(structType: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  39. def extraValidateMsg: String
    Attributes
    protected
    Definition Classes
    RawAnnotator
  40. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  41. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  42. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  43. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  44. def get[T](feature: StructFeature[T]): Option[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  45. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  46. def get[T](feature: SetFeature[T]): Option[Set[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  47. def get[T](feature: ArrayFeature[T]): Option[Array[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  48. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  49. def getAdditionalTokens: Map[Int, String]

    Get additional tokens

    Get additional tokens

    Definition Classes
    MedicalTextGenerator
  50. def getAdditionalTokensStr: String

    Get additional tokens in string format

    Get additional tokens in string format

    Definition Classes
    MedicalTextGenerator
  51. def getBatchSize: Int
    Definition Classes
    HasBatchedAnnotate
  52. def getCaseSensitive: Boolean
    Definition Classes
    HasCaseSensitiveProperties
  53. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  54. def getConfigProtoBytes: Option[Array[Byte]]

    Definition Classes
    MedicalTextGenerator
  55. def getCustomPrompt: String

    Custom model prompt

    Custom model prompt

    Attributes
    protected
    Definition Classes
    MedicalTextGenerator
  56. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  57. def getDoSample: Boolean

    Definition Classes
    GPTGenerationParams
  58. def getEngine: String
    Definition Classes
    HasEngine
  59. def getIgnoreTokenIds: Array[Int]

    Definition Classes
    GPTGenerationParams
  60. def getInputCols: Array[String]
    Definition Classes
    HasInputAnnotationCols
  61. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  62. def getMaxContextLength: Int

    Definition Classes
    GPTGenerationParams
  63. def getMaxNewTokens: Int

    Definition Classes
    GPTGenerationParams
  64. def getMaxTextLength: Int

    Definition Classes
    MedicalTextGenerator
  65. def getMlFrameworkType: String

    Get ML framework type

    Get ML framework type

    Definition Classes
    MedicalTextGenerator
  66. def getModelIfNotSet: MedicalEncoderDecoderModel

    Definition Classes
    MedicalTextGenerator
  67. def getModelType: String

    Get model type

    Get model type

    Definition Classes
    MedicalTextGenerator
  68. def getNoRepeatNgramSize: Int

    Definition Classes
    GPTGenerationParams
  69. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  70. final def getOutputCol: String
    Definition Classes
    HasOutputAnnotationCol
  71. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  72. def getQuestionTypes: Array[String]
  73. def getRandomSeed: Option[Int]

    Definition Classes
    GPTGenerationParams
  74. def getSignatures: Option[Map[String, String]]
    Definition Classes
    MedicalTextGenerator
  75. def getStopAtEos: Boolean

    Checks whether text generation stops when the end-of-sentence token is encountered.

    Checks whether text generation stops when the end-of-sentence token is encountered.

    Definition Classes
    MedicalTextGenerator
  76. def getTopK: Int

    Definition Classes
    GPTGenerationParams
  77. def getUseCache: Boolean
    Definition Classes
    MedicalTextGenerator
  78. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  79. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  80. def hasParent: Boolean
    Definition Classes
    Model
  81. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  82. val ignoreTokenIds: IntArrayParam

    A list of token ids which are ignored in the decoder's output (Default: Array())

    A list of token ids which are ignored in the decoder's output (Default: Array())

    Definition Classes
    GPTGenerationParams
  83. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  84. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  85. val inputAnnotatorTypes: Array[String]

    Input annotator type : DOCUMENT, DOCUMENT

    Input annotator type : DOCUMENT, DOCUMENT

    Definition Classes
    MedicalQuestionAnswering → MedicalTextGenerator → HasInputAnnotationCols
  86. final val inputCols: StringArrayParam
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  87. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  88. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  89. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  90. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  91. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  92. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  93. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  94. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  95. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  96. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  97. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  98. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  99. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  100. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  101. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  102. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  103. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  104. val maxContextLength: IntParam

    Maximum length of context text.

    Maximum length of context text. (Default: 1024)

    Definition Classes
    GPTGenerationParams
  105. val maxNewTokens: IntParam

    Maximum number of new tokens to be generated (Default: 30)

    Maximum number of new tokens to be generated (Default: 30)

    Definition Classes
    GPTGenerationParams
  106. val maxTextLength: IntParam

    Maximum length of context text.

    Maximum length of context text. (Default: 1024)

    Definition Classes
    MedicalTextGenerator
  107. val merges: MapFeature[(String, String), Int]

    Holding merges.txt coming from RoBERTa model

    Holding merges.txt coming from RoBERTa model

    Definition Classes
    MedicalTextGenerator
  108. val mlFrameworkType: Param[String]

    ML framework type

    ML framework type

    Definition Classes
    MedicalTextGenerator
  109. val modelType: Param[String]

    Model type

    Model type

    Definition Classes
    MedicalTextGenerator
  110. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  111. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  112. val noRepeatNgramSize: IntParam

    If set to int > 0, all ngrams of that size can only occur once (Default: 0)

    If set to int > 0, all ngrams of that size can only occur once (Default: 0)

    Definition Classes
    GPTGenerationParams
  113. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  114. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  115. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    MedicalQuestionAnswering → MedicalTextGenerator → ParamsAndFeaturesWritable
  116. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  117. val outputAnnotatorType: String

    Output annotator type : DOCUMENT

    Output annotator type : DOCUMENT

    Definition Classes
    MedicalQuestionAnswering → MedicalTextGenerator → HasOutputAnnotatorType
  118. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  119. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  120. var parent: Estimator[MedicalQuestionAnswering]
    Definition Classes
    Model
  121. val questionSkipLastToken: MapFeature[String, Boolean]
  122. val questionType: Param[String]

    Question type, e.g.

    Question type, e.g. 'short' or 'long').

  123. val randomSeed: Option[Int]

    Optional Random seed for the model.

    Optional Random seed for the model. Needs to be of type Long.

    Definition Classes
    GPTGenerationParams
  124. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  125. def set[T](feature: StructFeature[T], value: T): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  126. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  127. def set[T](feature: SetFeature[T], value: Set[T]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  128. def set[T](feature: ArrayFeature[T], value: Array[T]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  129. final def set(paramPair: ParamPair[_]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    Params
  130. final def set(param: String, value: Any): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    Params
  131. final def set[T](param: Param[T], value: T): MedicalQuestionAnswering.this.type
    Definition Classes
    Params
  132. def setAdditionalTokens(values: HashMap[Int, String]): MedicalQuestionAnswering.this.type

    Set additional tokens

    Set additional tokens

    Definition Classes
    MedicalTextGenerator
  133. def setAdditionalTokens(value: Map[Int, String]): MedicalQuestionAnswering.this.type

    Set additional tokens

    Set additional tokens

    Definition Classes
    MedicalTextGenerator
  134. def setBatchSize(size: Int): MedicalQuestionAnswering.this.type
    Definition Classes
    HasBatchedAnnotate
  135. def setCaseSensitive(value: Boolean): MedicalQuestionAnswering.this.type
    Definition Classes
    HasCaseSensitiveProperties
  136. def setConfigProtoBytes(bytes: Array[Int]): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  137. def setCustomPrompt(value: String): MedicalQuestionAnswering.this.type

    Set custom model prompt

    Set custom model prompt

    Definition Classes
    MedicalTextGenerator
  138. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  139. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  140. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  141. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  142. final def setDefault(paramPairs: ParamPair[_]*): MedicalQuestionAnswering.this.type
    Attributes
    protected
    Definition Classes
    Params
  143. final def setDefault[T](param: Param[T], value: T): MedicalQuestionAnswering.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  144. def setDoSample(value: Boolean): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  145. def setIgnoreTokenIds(tokenIds: Array[Int]): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  146. final def setInputCols(value: String*): MedicalQuestionAnswering.this.type
    Definition Classes
    HasInputAnnotationCols
  147. def setInputCols(value: Array[String]): MedicalQuestionAnswering.this.type
    Definition Classes
    HasInputAnnotationCols
  148. def setLazyAnnotator(value: Boolean): MedicalQuestionAnswering.this.type
    Definition Classes
    CanBeLazy
  149. def setMaxContextLength(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  150. def setMaxNewTokens(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  151. def setMaxTextLength(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  152. def setMerges(value: Map[(String, String), Int]): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  153. def setMlFrameworkType(value: String): MedicalQuestionAnswering.this.type

    Set ML framework type

    Set ML framework type

    Definition Classes
    MedicalTextGenerator
  154. def setModelIfNotSet(spark: SparkSession, model: MedicalEncoderDecoderModel): MedicalQuestionAnswering.this.type
    Definition Classes
    MedicalTextGenerator
  155. def setModelIfNotSet(spark: SparkSession, encoder: OnnxWrapper, decoder: OnnxWrapper, spp: SentencePieceWrapper): MedicalQuestionAnswering.this.type
    Definition Classes
    MedicalTextGenerator
  156. def setModelIfNotSet(spark: SparkSession, tfWrapper: TensorflowWrapper, spp: SentencePieceWrapper, useCache: Boolean): MedicalQuestionAnswering.this.type
    Definition Classes
    MedicalTextGenerator
  157. def setModelIfNotSet(spark: SparkSession, tfWrapper: TensorflowWrapper): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  158. def setModelType(value: String): MedicalQuestionAnswering.this.type

    Set model type

    Set model type

    Definition Classes
    MedicalTextGenerator
  159. def setNoRepeatNgramSize(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  160. final def setOutputCol(value: String): MedicalQuestionAnswering.this.type
    Definition Classes
    HasOutputAnnotationCol
  161. def setParent(parent: Estimator[MedicalQuestionAnswering]): MedicalQuestionAnswering
    Definition Classes
    Model
  162. def setQuestionAnswerTerminals(value: HashMap[String, List[Int]]): MedicalQuestionAnswering.this.type

    Set question terminals

  163. def setQuestionAnswerTerminals(value: Map[String, Array[Int]]): MedicalQuestionAnswering.this.type

    Set question terminals

  164. def setQuestionPrompts(value: HashMap[String, String]): MedicalQuestionAnswering.this.type

    Set question prompts

  165. def setQuestionPrompts(value: Map[String, String]): MedicalQuestionAnswering.this.type

    Set question prompts

  166. def setQuestionSkipLastToken(value: HashMap[String, Boolean]): MedicalQuestionAnswering.this.type

    Set question last tokens to skip

  167. def setQuestionSkipLastToken(value: Map[String, Boolean]): MedicalQuestionAnswering.this.type

    Set question last tokens to skip

  168. def setQuestionType(value: String): MedicalQuestionAnswering.this.type

  169. def setRandomSeed(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  170. def setSignatures(value: Map[String, String]): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  171. def setStopAtEos(value: Boolean): MedicalQuestionAnswering.this.type

    Determines whether text generation stops when the end-of-sentence token is encountered.

    Determines whether text generation stops when the end-of-sentence token is encountered.

    Definition Classes
    MedicalTextGenerator
  172. def setTopK(value: Int): MedicalQuestionAnswering.this.type

    Definition Classes
    GPTGenerationParams
  173. def setUseCache(value: Boolean): MedicalQuestionAnswering.this.type
    Definition Classes
    MedicalTextGenerator
  174. def setVocabulary(value: Map[String, Int]): MedicalQuestionAnswering.this.type

    Definition Classes
    MedicalTextGenerator
  175. val signatures: MapFeature[String, String]

    It contains TF model signatures for the laded saved model

    It contains TF model signatures for the laded saved model

    Definition Classes
    MedicalTextGenerator
  176. val stopAtEos: BooleanParam

    Stop text generation when the end-of-sentence token is encountered.

    Stop text generation when the end-of-sentence token is encountered.

    Definition Classes
    MedicalTextGenerator
  177. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  178. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  179. val topK: IntParam

    The number of highest probability vocabulary tokens to consider

    The number of highest probability vocabulary tokens to consider

    Definition Classes
    GPTGenerationParams
  180. final def transform(dataset: Dataset[_]): DataFrame
    Definition Classes
    AnnotatorModel → Transformer
  181. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  182. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  183. final def transformSchema(schema: StructType): StructType
    Definition Classes
    RawAnnotator → PipelineStage
  184. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  185. val uid: String
    Definition Classes
    MedicalQuestionAnswering → MedicalTextGenerator → Identifiable
  186. val useCache: BooleanParam

    Cache internal state of the model to improve performance

    Cache internal state of the model to improve performance

    Definition Classes
    MedicalTextGenerator
  187. def validate(schema: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  188. val vocabulary: MapFeature[String, Int]

    Vocabulary used to encode the words to ids with bpeTokenizer.encode

    Vocabulary used to encode the words to ids with bpeTokenizer.encode

    Definition Classes
    MedicalTextGenerator
  189. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  190. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  191. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  192. def wrapColumnMetadata(col: Column): Column
    Attributes
    protected
    Definition Classes
    RawAnnotator
  193. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  194. def writeOnnxModel(path: String, spark: SparkSession, onnxWrapper: OnnxWrapper, suffix: String, fileName: String): Unit
    Definition Classes
    WriteOnnxModel
  195. def writeOnnxModels(path: String, spark: SparkSession, onnxWrappersWithNames: Seq[(OnnxWrapper, String)], suffix: String): Unit
    Definition Classes
    WriteOnnxModel
  196. def writeSentencePieceModel(path: String, spark: SparkSession, spp: SentencePieceWrapper, suffix: String, filename: String): Unit
    Definition Classes
    WriteSentencePieceModel
  197. def writeTensorflowHub(path: String, tfPath: String, spark: SparkSession, suffix: String): Unit
    Definition Classes
    WriteTensorflowModel
  198. def writeTensorflowModel(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]]): Unit
    Definition Classes
    WriteTensorflowModel
  199. def writeTensorflowModelV2(path: String, spark: SparkSession, tensorflow: TensorflowWrapper, suffix: String, filename: String, configProtoBytes: Option[Array[Byte]], savedSignatures: Option[Map[String, String]]): Unit
    Definition Classes
    WriteTensorflowModel

Inherited from MedicalTextGenerator

Inherited from CheckLicense

Inherited from HasEngine

Inherited from WriteSentencePieceModel

Inherited from WriteOnnxModel

Inherited from WriteTensorflowModel

Inherited from HasBatchedAnnotate[MedicalQuestionAnswering]

Inherited from HasCaseSensitiveProperties

Inherited from GPTGenerationParams

Inherited from AnnotatorModel[MedicalQuestionAnswering]

Inherited from CanBeLazy

Inherited from RawAnnotator[MedicalQuestionAnswering]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[MedicalQuestionAnswering]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Parameters

A list of (hyper-)parameter keys this annotator can take. Users can set and get the parameter values through setters and getters, respectively.

Members

Parameter setters

Parameter getters