class MedicalVisionLLM extends AutoGGUFVisionModel with CheckLicense

Multimodal annotator to generate text completions with large language models. It supports ingesting images for captioning.

At the moment only CLIP based models are supported.

If the parameters are not set, the annotator will default to use the parameters provided by the model.

This annotator expects a column of annotator type AnnotationImage for the image and Annotation for the caption. Note that the image bytes in the image annotation need to be raw image bytes without preprocessing. We provide the helper function ImageAssembler.loadImagesAsBytes to load the image bytes from a directory.

Pretrained models can be loaded with pretrained of the companion object:

val medicalVisionLLM = MedicalVisionLLM.pretrained()
  .setInputCols("image', "document")
  .setOutputCol("completions")

Note

To use GPU inference with this annotator, make sure to use the Spark NLP GPU package and set the number of GPU layers with the setNGpuLayers method.

When using larger models, we recommend adjusting GPU usage with setNCtx and setNGpuLayers according to your hardware to avoid out-of-memory errors.

Example

import com.johnsnowlabs.nlp.ImageAssembler
import com.johnsnowlabs.nlp.annotator._
import com.johnsnowlabs.nlp.jsl.annotator._
import com.johnsnowlabs.nlp.base._
import org.apache.spark.ml.Pipeline
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.lit

val documentAssembler = new DocumentAssembler()
  .setInputCol("caption")
  .setOutputCol("caption_document")

val imageAssembler = new ImageAssembler()
  .setInputCol("image")
  .setOutputCol("image_assembler")

val imagesPath = "IMAGES_PATH"
val data: DataFrame = ImageAssembler
  .loadImagesAsBytes(spark, imagesPath)
  .withColumn("caption", lit("Caption this image.")) // Add a caption to each image.

val model = MedicalVisionLLM.pretrained()
  .setInputCols("caption_document", "image_assembler")
  .setOutputCol("completions")
  .setBatchSize(4)
  .setNGpuLayers(99)
  .setNCtx(4096)
  .setMinKeep(0)
  .setMinP(0.05f)
  .setNPredict(40)
  .setNProbs(0)
  .setPenalizeNl(false)
  .setRepeatLastN(256)
  .setRepeatPenalty(1.18f)
  .setStopStrings(Array("</s>", "Llama:", "User:"))
  .setTemperature(0.05f)
  .setTfsZ(1)
  .setTypicalP(1)
  .setTopK(40)
  .setTopP(0.95f)

val pipeline = new Pipeline().setStages(Array(documentAssembler, imageAssembler, model))
pipeline
  .fit(data)
  .transform(data)
  .selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result")
  .show(truncate = false)
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|image_name       |result                                                                                                                                                                                        |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|palace.JPEG      |[ The image depicts a large, ornate room with high ceilings and beautifully decorated walls. There are several chairs placed throughout the space, some of which have cushions]               |
|egyptian_cat.jpeg|[ The image features two cats lying on a pink surface, possibly a bed or sofa. One cat is positioned towards the left side of the scene and appears to be sleeping while holding]             |
|hippopotamus.JPEG|[ A large brown hippo is swimming in a body of water, possibly an aquarium. The hippo appears to be enjoying its time in the water and seems relaxed as it floats]                            |
|hen.JPEG         |[ The image features a large chicken standing next to several baby chickens. In total, there are five birds in the scene: one adult and four young ones. They appear to be gathered together] |
|ostrich.JPEG     |[ The image features a large, long-necked bird standing in the grass. It appears to be an ostrich or similar species with its head held high and looking around. In addition to]              |
|junco.JPEG       |[ A small bird with a black head and white chest is standing on the snow. It appears to be looking at something, possibly food or another animal in its vicinity. The scene takes place out]  |
|bluetick.jpg     |[ A dog with a red collar is sitting on the floor, looking at something. The dog appears to be staring into the distance or focusing its attention on an object in front of it.]              |
|chihuahua.jpg    |[ A small brown dog wearing a sweater is sitting on the floor. The dog appears to be looking at something, possibly its owner or another animal in the room. It seems comfortable and relaxed]|
|tractor.JPEG     |[ A man is sitting in the driver's seat of a green tractor, which has yellow wheels and tires. The tractor appears to be parked on top of an empty field with]                                |
|ox.JPEG          |[ A large bull with horns is standing in a grassy field.]                                                                                                                                     |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Linear Supertypes
CheckLicense, AutoGGUFVisionModel, CompletionPostProcessing, HasProtectedParams, HasLlamaCppInferenceProperties, HasLlamaCppModelProperties, HasEngine, HasBatchedAnnotateTextImage[AutoGGUFVisionModel], AnnotatorModel[AutoGGUFVisionModel], CanBeLazy, RawAnnotator[AutoGGUFVisionModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[AutoGGUFVisionModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MedicalVisionLLM
  2. CheckLicense
  3. AutoGGUFVisionModel
  4. CompletionPostProcessing
  5. HasProtectedParams
  6. HasLlamaCppInferenceProperties
  7. HasLlamaCppModelProperties
  8. HasEngine
  9. HasBatchedAnnotateTextImage
  10. AnnotatorModel
  11. CanBeLazy
  12. RawAnnotator
  13. HasOutputAnnotationCol
  14. HasInputAnnotationCols
  15. HasOutputAnnotatorType
  16. ParamsAndFeaturesWritable
  17. HasFeatures
  18. DefaultParamsWritable
  19. MLWritable
  20. Model
  21. Transformer
  22. PipelineStage
  23. Logging
  24. Params
  25. Serializable
  26. Serializable
  27. Identifiable
  28. AnyRef
  29. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MedicalVisionLLM()

    Annotator reference id.

    Annotator reference id. Used to identify elements in metadata or to refer to this annotator type

  2. new MedicalVisionLLM(uid: String)

    uid

    required uid for storing annotator to disk

Type Members

  1. type AnnotationContent = Seq[Row]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  2. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType
  3. implicit class ProtectedParam[T] extends Param[T]
    Definition Classes
    HasProtectedParams

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[T](feature: StructFeature[T]): T
    Attributes
    protected
    Definition Classes
    HasFeatures
  5. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[T](feature: SetFeature[T]): Set[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: ArrayFeature[T]): Array[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  10. def afterAnnotate(dataset: DataFrame): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  11. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  12. def batchAnnotate(batchedAnnotations: Seq[(Annotation, AnnotationImage)]): Seq[Seq[Annotation]]

    Completes the batch of annotations.

    Completes the batch of annotations.

    batchedAnnotations

    The single batch of annotations

    returns

    Completed text sequences sentences that belong to the same original row !! (challenging)

    Definition Classes
    MedicalVisionLLM → AutoGGUFVisionModel → HasBatchedAnnotateTextImage
  13. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotateTextImage
  14. val batchSize: IntParam
    Definition Classes
    HasBatchedAnnotateTextImage
  15. def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  16. val cachePrompt: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  17. val chatTemplate: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  18. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  19. def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
    Definition Classes
    CheckLicense
  20. def checkValidScope(scope: String): Unit
    Definition Classes
    CheckLicense
  21. def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  22. def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  23. final def clear(param: Param[_]): MedicalVisionLLM.this.type
    Definition Classes
    Params
  24. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. def close(): Unit
    Definition Classes
    AutoGGUFVisionModel
  26. def copy(extra: ParamMap): AutoGGUFVisionModel
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  27. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  28. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  29. val defragmentationThreshold: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  30. val disableLog: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  31. val disableTokenIds: IntArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  32. val dynamicTemperatureExponent: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  33. val dynamicTemperatureRange: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  34. val embedding: BooleanParam
  35. val engine: Param[String]
    Definition Classes
    HasEngine
  36. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  37. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  38. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  39. def explainParams(): String
    Definition Classes
    Params
  40. def extraValidate(structType: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  41. def extraValidateMsg: String
    Attributes
    protected
    Definition Classes
    RawAnnotator
  42. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  43. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  44. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  45. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  46. val flashAttention: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  47. val frequencyPenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  48. def get[T](feature: StructFeature[T]): Option[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  49. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  50. def get[T](feature: SetFeature[T]): Option[Set[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  51. def get[T](feature: ArrayFeature[T]): Option[Array[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  52. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  53. def getBatchSize: Int
    Definition Classes
    HasBatchedAnnotateTextImage
  54. def getCachePrompt: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  55. def getChatTemplate: String
    Definition Classes
    HasLlamaCppModelProperties
  56. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  57. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  58. def getDefragmentationThreshold: Float
    Definition Classes
    HasLlamaCppModelProperties
  59. def getDisableLog: Boolean
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  60. def getDisableTokenIds: Array[Int]
    Definition Classes
    HasLlamaCppInferenceProperties
  61. def getDynamicTemperatureExponent: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  62. def getDynamicTemperatureRange: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  63. def getEngine: String
    Definition Classes
    HasEngine
  64. def getFlashAttention: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  65. def getFrequencyPenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  66. def getGrammar: String
    Definition Classes
    HasLlamaCppInferenceProperties
  67. def getIgnoreEos: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  68. def getInferenceParameters: InferenceParameters
    Attributes
    protected
    Definition Classes
    HasLlamaCppInferenceProperties
  69. def getInputCols: Array[String]
    Definition Classes
    HasInputAnnotationCols
  70. def getInputPrefix: String
    Definition Classes
    HasLlamaCppInferenceProperties
  71. def getInputSuffix: String
    Definition Classes
    HasLlamaCppInferenceProperties
  72. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  73. def getLogVerbosity: Int
    Definition Classes
    HasLlamaCppModelProperties
  74. def getMainGpu: Int
    Definition Classes
    HasLlamaCppModelProperties
  75. def getMetadata: String
    Definition Classes
    HasLlamaCppModelProperties
  76. def getMetadataMap: Map[String, Map[String, String]]
    Definition Classes
    HasLlamaCppModelProperties
  77. def getMinKeep: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  78. def getMinP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  79. def getMiroStat: String
    Definition Classes
    HasLlamaCppInferenceProperties
  80. def getMiroStatEta: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  81. def getMiroStatTau: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  82. def getModelDraft: String
    Definition Classes
    HasLlamaCppModelProperties
  83. def getModelIfNotSet: GGUFWrapperMultiModal
    Definition Classes
    AutoGGUFVisionModel
  84. def getModelParameters: ModelParameters
    Attributes
    protected
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  85. def getNBatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  86. def getNCtx: Int
    Definition Classes
    HasLlamaCppModelProperties
  87. def getNDraft: Int
    Definition Classes
    HasLlamaCppModelProperties
  88. def getNGpuLayers: Int
    Definition Classes
    HasLlamaCppModelProperties
  89. def getNGpuLayersDraft: Int
    Definition Classes
    HasLlamaCppModelProperties
  90. def getNKeep: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  91. def getNPredict: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  92. def getNProbs: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  93. def getNThreads: Int
    Definition Classes
    HasLlamaCppModelProperties
  94. def getNThreadsBatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  95. def getNUbatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  96. def getNoKvOffload: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  97. def getNuma: String
    Definition Classes
    HasLlamaCppModelProperties
  98. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  99. final def getOutputCol: String
    Definition Classes
    HasOutputAnnotationCol
  100. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  101. def getPenalizeNl: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  102. def getPenaltyPrompt: String
    Definition Classes
    HasLlamaCppInferenceProperties
  103. def getPresencePenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  104. def getRemoveThinkingTag: Option[String]
    Definition Classes
    CompletionPostProcessing
  105. def getRepeatLastN: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  106. def getRepeatPenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  107. def getRopeFreqBase: Float
    Definition Classes
    HasLlamaCppModelProperties
  108. def getRopeFreqScale: Float
    Definition Classes
    HasLlamaCppModelProperties
  109. def getRopeScalingType: String
    Definition Classes
    HasLlamaCppModelProperties
  110. def getSamplers: Array[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  111. def getSeed: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  112. def getSplitMode: String
    Definition Classes
    HasLlamaCppModelProperties
  113. def getStopStrings: Array[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  114. def getSystemPrompt: String
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  115. def getTemperature: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  116. def getTfsZ: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  117. def getTokenBias: Map[String, Float]
    Definition Classes
    HasLlamaCppInferenceProperties
  118. def getTokenIdBias: Map[Int, Float]
    Definition Classes
    HasLlamaCppInferenceProperties
  119. def getTopK: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  120. def getTopP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  121. def getTypicalP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  122. def getUseChatTemplate: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  123. def getUseMlock: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  124. def getUseMmap: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  125. def getYarnAttnFactor: Float
    Definition Classes
    HasLlamaCppModelProperties
  126. def getYarnBetaFast: Float
    Definition Classes
    HasLlamaCppModelProperties
  127. def getYarnBetaSlow: Float
    Definition Classes
    HasLlamaCppModelProperties
  128. def getYarnExtFactor: Float
    Definition Classes
    HasLlamaCppModelProperties
  129. def getYarnOrigCtx: Int
    Definition Classes
    HasLlamaCppModelProperties
  130. val gpuSplitMode: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  131. val grammar: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  132. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  133. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  134. def hasParent: Boolean
    Definition Classes
    Model
  135. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  136. val ignoreEos: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  137. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  138. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  139. val inputAnnotatorTypes: Array[AnnotatorType]
    Definition Classes
    AutoGGUFVisionModel → HasInputAnnotationCols
  140. final val inputCols: StringArrayParam
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  141. val inputPrefix: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  142. val inputSuffix: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  143. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  144. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  145. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  146. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  147. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  148. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  149. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  150. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  151. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  152. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  153. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  154. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  155. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  156. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  157. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  158. val logVerbosity: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  159. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  160. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  161. val logger: Logger
    Attributes
    protected
    Definition Classes
    HasLlamaCppModelProperties
  162. val mainGpu: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  163. val metadata: ProtectedParam[String]
    Definition Classes
    HasLlamaCppModelProperties
  164. val minKeep: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  165. val minP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  166. val miroStat: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  167. val miroStatEta: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  168. val miroStatTau: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  169. val modelDraft: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  170. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  171. val nBatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  172. val nCtx: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  173. val nDraft: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  174. val nGpuLayers: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  175. val nGpuLayersDraft: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  176. val nKeep: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  177. val nPredict: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  178. val nProbs: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  179. val nThreads: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  180. val nThreadsBatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  181. val nUbatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  182. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  183. val noKvOffload: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  184. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  185. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  186. val numaStrategy: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  187. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    AutoGGUFVisionModel → ParamsAndFeaturesWritable
  188. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  189. val outputAnnotatorType: AnnotatorType
    Definition Classes
    AutoGGUFVisionModel → HasOutputAnnotatorType
  190. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  191. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  192. var parent: Estimator[AutoGGUFVisionModel]
    Definition Classes
    Model
  193. val penalizeNl: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  194. val penaltyPrompt: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  195. val presencePenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  196. def processCompletions(results: Array[String]): Array[String]
    Attributes
    protected
    Definition Classes
    CompletionPostProcessing
  197. val removeThinkingTag: Param[String]
    Definition Classes
    CompletionPostProcessing
  198. val repeatLastN: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  199. val repeatPenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  200. val ropeFreqBase: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  201. val ropeFreqScale: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  202. val ropeScalingType: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  203. val samplers: StringArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  204. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  205. val seed: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  206. def set[T](param: ProtectedParam[T], value: T): MedicalVisionLLM.this.type
    Definition Classes
    HasProtectedParams
  207. def set[T](feature: StructFeature[T], value: T): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  208. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  209. def set[T](feature: SetFeature[T], value: Set[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  210. def set[T](feature: ArrayFeature[T], value: Array[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  211. final def set(paramPair: ParamPair[_]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  212. final def set(param: String, value: Any): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  213. final def set[T](param: Param[T], value: T): MedicalVisionLLM.this.type
    Definition Classes
    Params
  214. def setBatchSize(size: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasBatchedAnnotateTextImage
  215. def setCachePrompt(cachePrompt: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  216. def setChatTemplate(chatTemplate: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  217. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  218. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  219. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  220. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  221. final def setDefault(paramPairs: ParamPair[_]*): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  222. final def setDefault[T](param: Param[T], value: T): MedicalVisionLLM.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  223. def setDefragmentationThreshold(defragThold: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  224. def setDisableLog(disableLog: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  225. def setDisableTokenIds(disableTokenIds: Array[Int]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  226. def setDynamicTemperatureExponent(dynatempExponent: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  227. def setDynamicTemperatureRange(dynatempRange: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  228. def setFlashAttention(flashAttention: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  229. def setFrequencyPenalty(frequencyPenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  230. def setGpuSplitMode(splitMode: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  231. def setGrammar(grammar: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  232. def setIgnoreEos(ignoreEos: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  233. final def setInputCols(value: String*): MedicalVisionLLM.this.type
    Definition Classes
    HasInputAnnotationCols
  234. def setInputCols(value: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasInputAnnotationCols
  235. def setInputPrefix(inputPrefix: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  236. def setInputSuffix(inputSuffix: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  237. def setLazyAnnotator(value: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    CanBeLazy
  238. def setLogVerbosity(logVerbosity: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  239. def setMainGpu(mainGpu: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  240. def setMetadata(metadata: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  241. def setMinKeep(minKeep: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  242. def setMinP(minP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  243. def setMiroStat(mirostat: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  244. def setMiroStatEta(mirostatEta: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  245. def setMiroStatTau(mirostatTau: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  246. def setModelDraft(modelDraft: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  247. def setModelIfNotSet(spark: SparkSession, wrapper: GGUFWrapperMultiModal): MedicalVisionLLM.this.type
    Definition Classes
    AutoGGUFVisionModel
  248. def setNBatch(nBatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  249. def setNCtx(nCtx: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  250. def setNDraft(nDraft: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  251. def setNGpuLayers(nGpuLayers: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  252. def setNGpuLayersDraft(nGpuLayersDraft: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  253. def setNKeep(nKeep: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  254. def setNParallel(nParallel: Int): MedicalVisionLLM.this.type
    Definition Classes
    AutoGGUFVisionModel
  255. def setNPredict(nPredict: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  256. def setNProbs(nProbs: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  257. def setNThreads(nThreads: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  258. def setNThreadsBatch(nThreadsBatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  259. def setNUbatch(nUbatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  260. def setNoKvOffload(noKvOffload: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  261. def setNumaStrategy(numa: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  262. final def setOutputCol(value: String): MedicalVisionLLM.this.type
    Definition Classes
    HasOutputAnnotationCol
  263. def setParent(parent: Estimator[AutoGGUFVisionModel]): AutoGGUFVisionModel
    Definition Classes
    Model
  264. def setPenalizeNl(penalizeNl: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  265. def setPenaltyPrompt(penaltyPrompt: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  266. def setPresencePenalty(presencePenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  267. def setRemoveThinkingTag(value: String): MedicalVisionLLM.this.type
    Definition Classes
    CompletionPostProcessing
  268. def setRepeatLastN(repeatLastN: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  269. def setRepeatPenalty(repeatPenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  270. def setRopeFreqBase(ropeFreqBase: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  271. def setRopeFreqScale(ropeFreqScale: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  272. def setRopeScalingType(ropeScalingType: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  273. def setSamplers(samplers: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  274. def setSeed(seed: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  275. def setStopStrings(stopStrings: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  276. def setSystemPrompt(systemPrompt: String): MedicalVisionLLM.this.type
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  277. def setTemperature(temperature: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  278. def setTfsZ(tfsZ: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  279. def setTokenBias(tokenBias: HashMap[String, Double]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  280. def setTokenBias(tokenBias: Map[String, Float]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  281. def setTokenIdBias(tokenIdBias: HashMap[Integer, Double]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  282. def setTokenIdBias(tokenIdBias: Map[Int, Float]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  283. def setTopK(topK: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  284. def setTopP(topP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  285. def setTypicalP(typicalP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  286. def setUseChatTemplate(useChatTemplate: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  287. def setUseMlock(useMlock: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  288. def setUseMmap(useMmap: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  289. def setYarnAttnFactor(yarnAttnFactor: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  290. def setYarnBetaFast(yarnBetaFast: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  291. def setYarnBetaSlow(yarnBetaSlow: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  292. def setYarnExtFactor(yarnExtFactor: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  293. def setYarnOrigCtx(yarnOrigCtx: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  294. val stopStrings: StringArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  295. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  296. val systemPrompt: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  297. val temperature: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  298. val tfsZ: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  299. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  300. val tokenBias: StructFeature[Map[String, Float]]
    Definition Classes
    HasLlamaCppInferenceProperties
  301. val tokenIdBias: StructFeature[Map[Int, Float]]
    Definition Classes
    HasLlamaCppInferenceProperties
  302. val topK: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  303. val topP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  304. final def transform(dataset: Dataset[_]): DataFrame
    Definition Classes
    AnnotatorModel → Transformer
  305. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  306. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  307. final def transformSchema(schema: StructType): StructType
    Definition Classes
    RawAnnotator → PipelineStage
  308. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  309. val typicalP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  310. val uid: String
    Definition Classes
    MedicalVisionLLM → AutoGGUFVisionModel → Identifiable
  311. val useChatTemplate: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  312. val useMlock: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  313. val useMmap: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  314. def validate(schema: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  315. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  316. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  317. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  318. def wrapColumnMetadata(col: Column): Column
    Attributes
    protected
    Definition Classes
    RawAnnotator
  319. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  320. val yarnAttnFactor: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  321. val yarnBetaFast: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  322. val yarnBetaSlow: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  323. val yarnExtFactor: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  324. val yarnOrigCtx: IntParam
    Definition Classes
    HasLlamaCppModelProperties

Inherited from CheckLicense

Inherited from AutoGGUFVisionModel

Inherited from CompletionPostProcessing

Inherited from HasProtectedParams

Inherited from HasLlamaCppInferenceProperties

Inherited from HasLlamaCppModelProperties

Inherited from HasEngine

Inherited from HasBatchedAnnotateTextImage[AutoGGUFVisionModel]

Inherited from AnnotatorModel[AutoGGUFVisionModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[AutoGGUFVisionModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[AutoGGUFVisionModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Members