class MedicalVisionLLM extends AutoGGUFVisionModel with CheckLicense

Multimodal annotator to generate text completions with large language models. It supports ingesting images for captioning.

At the moment only CLIP based models are supported.

If the parameters are not set, the annotator will default to use the parameters provided by the model.

This annotator expects a column of annotator type AnnotationImage for the image and Annotation for the caption. Note that the image bytes in the image annotation need to be raw image bytes without preprocessing. We provide the helper function ImageAssembler.loadImagesAsBytes to load the image bytes from a directory.

Pretrained models can be loaded with pretrained of the companion object:

val medicalVisionLLM = MedicalVisionLLM.pretrained()
  .setInputCols("image', "document")
  .setOutputCol("completions")

Note

To use GPU inference with this annotator, make sure to use the Spark NLP GPU package and set the number of GPU layers with the setNGpuLayers method.

When using larger models, we recommend adjusting GPU usage with setNCtx and setNGpuLayers according to your hardware to avoid out-of-memory errors.

Example

import com.johnsnowlabs.nlp.ImageAssembler
import com.johnsnowlabs.nlp.annotator._
import com.johnsnowlabs.nlp.jsl.annotator._
import com.johnsnowlabs.nlp.base._
import org.apache.spark.ml.Pipeline
import org.apache.spark.sql.DataFrame
import org.apache.spark.sql.functions.lit

val documentAssembler = new DocumentAssembler()
  .setInputCol("caption")
  .setOutputCol("caption_document")

val imageAssembler = new ImageAssembler()
  .setInputCol("image")
  .setOutputCol("image_assembler")

val imagesPath = "IMAGES_PATH"
val data: DataFrame = ImageAssembler
  .loadImagesAsBytes(spark, imagesPath)
  .withColumn("caption", lit("Caption this image.")) // Add a caption to each image.

val model = MedicalVisionLLM.pretrained()
  .setInputCols("caption_document", "image_assembler")
  .setOutputCol("completions")
  .setBatchSize(4)
  .setNGpuLayers(99)
  .setNCtx(4096)
  .setMinKeep(0)
  .setMinP(0.05f)
  .setNPredict(40)
  .setNProbs(0)
  .setPenalizeNl(false)
  .setRepeatLastN(256)
  .setRepeatPenalty(1.18f)
  .setStopStrings(Array("</s>", "Llama:", "User:"))
  .setTemperature(0.05f)
  .setTfsZ(1)
  .setTypicalP(1)
  .setTopK(40)
  .setTopP(0.95f)

val pipeline = new Pipeline().setStages(Array(documentAssembler, imageAssembler, model))
pipeline
  .fit(data)
  .transform(data)
  .selectExpr("reverse(split(image.origin, '/'))[0] as image_name", "completions.result")
  .show(truncate = false)
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|image_name       |result                                                                                                                                                                                        |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
|palace.JPEG      |[ The image depicts a large, ornate room with high ceilings and beautifully decorated walls. There are several chairs placed throughout the space, some of which have cushions]               |
|egyptian_cat.jpeg|[ The image features two cats lying on a pink surface, possibly a bed or sofa. One cat is positioned towards the left side of the scene and appears to be sleeping while holding]             |
|hippopotamus.JPEG|[ A large brown hippo is swimming in a body of water, possibly an aquarium. The hippo appears to be enjoying its time in the water and seems relaxed as it floats]                            |
|hen.JPEG         |[ The image features a large chicken standing next to several baby chickens. In total, there are five birds in the scene: one adult and four young ones. They appear to be gathered together] |
|ostrich.JPEG     |[ The image features a large, long-necked bird standing in the grass. It appears to be an ostrich or similar species with its head held high and looking around. In addition to]              |
|junco.JPEG       |[ A small bird with a black head and white chest is standing on the snow. It appears to be looking at something, possibly food or another animal in its vicinity. The scene takes place out]  |
|bluetick.jpg     |[ A dog with a red collar is sitting on the floor, looking at something. The dog appears to be staring into the distance or focusing its attention on an object in front of it.]              |
|chihuahua.jpg    |[ A small brown dog wearing a sweater is sitting on the floor. The dog appears to be looking at something, possibly its owner or another animal in the room. It seems comfortable and relaxed]|
|tractor.JPEG     |[ A man is sitting in the driver's seat of a green tractor, which has yellow wheels and tires. The tractor appears to be parked on top of an empty field with]                                |
|ox.JPEG          |[ A large bull with horns is standing in a grassy field.]                                                                                                                                     |
+-----------------+----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
Linear Supertypes
CheckLicense, AutoGGUFVisionModel, HasProtectedParams, HasLlamaCppInferenceProperties, HasLlamaCppModelProperties, HasEngine, HasBatchedAnnotateTextImage[AutoGGUFVisionModel], AnnotatorModel[AutoGGUFVisionModel], CanBeLazy, RawAnnotator[AutoGGUFVisionModel], HasOutputAnnotationCol, HasInputAnnotationCols, HasOutputAnnotatorType, ParamsAndFeaturesWritable, HasFeatures, DefaultParamsWritable, MLWritable, Model[AutoGGUFVisionModel], Transformer, PipelineStage, Logging, Params, Serializable, Serializable, Identifiable, AnyRef, Any
Ordering
  1. Alphabetic
  2. By Inheritance
Inherited
  1. MedicalVisionLLM
  2. CheckLicense
  3. AutoGGUFVisionModel
  4. HasProtectedParams
  5. HasLlamaCppInferenceProperties
  6. HasLlamaCppModelProperties
  7. HasEngine
  8. HasBatchedAnnotateTextImage
  9. AnnotatorModel
  10. CanBeLazy
  11. RawAnnotator
  12. HasOutputAnnotationCol
  13. HasInputAnnotationCols
  14. HasOutputAnnotatorType
  15. ParamsAndFeaturesWritable
  16. HasFeatures
  17. DefaultParamsWritable
  18. MLWritable
  19. Model
  20. Transformer
  21. PipelineStage
  22. Logging
  23. Params
  24. Serializable
  25. Serializable
  26. Identifiable
  27. AnyRef
  28. Any
  1. Hide All
  2. Show All
Visibility
  1. Public
  2. All

Instance Constructors

  1. new MedicalVisionLLM()

    Annotator reference id.

    Annotator reference id. Used to identify elements in metadata or to refer to this annotator type

  2. new MedicalVisionLLM(uid: String)

    uid

    required uid for storing annotator to disk

Type Members

  1. type AnnotationContent = Seq[Row]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  2. type AnnotatorType = String
    Definition Classes
    HasOutputAnnotatorType
  3. implicit class ProtectedParam[T] extends Param[T]
    Definition Classes
    HasProtectedParams

Value Members

  1. final def !=(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  2. final def ##(): Int
    Definition Classes
    AnyRef → Any
  3. final def $[T](param: Param[T]): T
    Attributes
    protected
    Definition Classes
    Params
  4. def $$[T](feature: StructFeature[T]): T
    Attributes
    protected
    Definition Classes
    HasFeatures
  5. def $$[K, V](feature: MapFeature[K, V]): Map[K, V]
    Attributes
    protected
    Definition Classes
    HasFeatures
  6. def $$[T](feature: SetFeature[T]): Set[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  7. def $$[T](feature: ArrayFeature[T]): Array[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  8. final def ==(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  9. def _transform(dataset: Dataset[_], recursivePipeline: Option[PipelineModel]): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  10. def afterAnnotate(dataset: DataFrame): DataFrame
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  11. final def asInstanceOf[T0]: T0
    Definition Classes
    Any
  12. def batchAnnotate(batchedAnnotations: Seq[(Annotation, AnnotationImage)]): Seq[Seq[Annotation]]

    Completes the batch of annotations.

    Completes the batch of annotations.

    batchedAnnotations

    The single batch of annotations

    returns

    Completed text sequences sentences that belong to the same original row !! (challenging)

    Definition Classes
    MedicalVisionLLM → AutoGGUFVisionModel → HasBatchedAnnotateTextImage
  13. def batchProcess(rows: Iterator[_]): Iterator[Row]
    Definition Classes
    HasBatchedAnnotateTextImage
  14. val batchSize: IntParam
    Definition Classes
    HasBatchedAnnotateTextImage
  15. def beforeAnnotate(dataset: Dataset[_]): Dataset[_]
    Attributes
    protected
    Definition Classes
    AnnotatorModel
  16. val cachePrompt: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  17. val chatTemplate: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  18. final def checkSchema(schema: StructType, inputAnnotatorType: String): Boolean
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  19. def checkValidEnvironment(spark: Option[SparkSession], scopes: Seq[String]): Unit
    Definition Classes
    CheckLicense
  20. def checkValidScope(scope: String): Unit
    Definition Classes
    CheckLicense
  21. def checkValidScopeAndEnvironment(scope: String, spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  22. def checkValidScopesAndEnvironment(scopes: Seq[String], spark: Option[SparkSession], checkLp: Boolean): Unit
    Definition Classes
    CheckLicense
  23. final def clear(param: Param[_]): MedicalVisionLLM.this.type
    Definition Classes
    Params
  24. def clone(): AnyRef
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  25. def copy(extra: ParamMap): AutoGGUFVisionModel
    Definition Classes
    RawAnnotator → Model → Transformer → PipelineStage → Params
  26. def copyValues[T <: Params](to: T, extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  27. final def defaultCopy[T <: Params](extra: ParamMap): T
    Attributes
    protected
    Definition Classes
    Params
  28. val defragmentationThreshold: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  29. val disableLog: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  30. val disableTokenIds: IntArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  31. val dynamicTemperatureExponent: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  32. val dynamicTemperatureRange: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  33. val embedding: BooleanParam
  34. val engine: Param[String]
    Definition Classes
    HasEngine
  35. final def eq(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  36. def equals(arg0: Any): Boolean
    Definition Classes
    AnyRef → Any
  37. def explainParam(param: Param[_]): String
    Definition Classes
    Params
  38. def explainParams(): String
    Definition Classes
    Params
  39. def extraValidate(structType: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  40. def extraValidateMsg: String
    Attributes
    protected
    Definition Classes
    RawAnnotator
  41. final def extractParamMap(): ParamMap
    Definition Classes
    Params
  42. final def extractParamMap(extra: ParamMap): ParamMap
    Definition Classes
    Params
  43. val features: ArrayBuffer[Feature[_, _, _]]
    Definition Classes
    HasFeatures
  44. def finalize(): Unit
    Attributes
    protected[lang]
    Definition Classes
    AnyRef
    Annotations
    @throws( classOf[java.lang.Throwable] )
  45. val flashAttention: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  46. val frequencyPenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  47. def get[T](feature: StructFeature[T]): Option[T]
    Attributes
    protected
    Definition Classes
    HasFeatures
  48. def get[K, V](feature: MapFeature[K, V]): Option[Map[K, V]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  49. def get[T](feature: SetFeature[T]): Option[Set[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  50. def get[T](feature: ArrayFeature[T]): Option[Array[T]]
    Attributes
    protected
    Definition Classes
    HasFeatures
  51. final def get[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  52. def getBatchSize: Int
    Definition Classes
    HasBatchedAnnotateTextImage
  53. def getCachePrompt: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  54. def getChatTemplate: String
    Definition Classes
    HasLlamaCppModelProperties
  55. final def getClass(): Class[_]
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  56. final def getDefault[T](param: Param[T]): Option[T]
    Definition Classes
    Params
  57. def getDefragmentationThreshold: Float
    Definition Classes
    HasLlamaCppModelProperties
  58. def getDisableLog: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  59. def getDisableTokenIds: Array[Int]
    Definition Classes
    HasLlamaCppInferenceProperties
  60. def getDynamicTemperatureExponent: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  61. def getDynamicTemperatureRange: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  62. def getEngine: String
    Definition Classes
    HasEngine
  63. def getFlashAttention: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  64. def getFrequencyPenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  65. def getGrammar: String
    Definition Classes
    HasLlamaCppInferenceProperties
  66. def getIgnoreEos: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  67. def getInferenceParameters: InferenceParameters
    Attributes
    protected
    Definition Classes
    HasLlamaCppInferenceProperties
  68. def getInputCols: Array[String]
    Definition Classes
    HasInputAnnotationCols
  69. def getInputPrefix: String
    Definition Classes
    HasLlamaCppInferenceProperties
  70. def getInputSuffix: String
    Definition Classes
    HasLlamaCppInferenceProperties
  71. def getLazyAnnotator: Boolean
    Definition Classes
    CanBeLazy
  72. def getLogVerbosity: Int
    Definition Classes
    HasLlamaCppModelProperties
  73. def getMainGpu: Int
    Definition Classes
    HasLlamaCppModelProperties
  74. def getMetadata: String
    Definition Classes
    HasLlamaCppModelProperties
  75. def getMetadataMap: Map[String, Map[String, String]]
    Definition Classes
    HasLlamaCppModelProperties
  76. def getMinKeep: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  77. def getMinP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  78. def getMiroStat: String
    Definition Classes
    HasLlamaCppInferenceProperties
  79. def getMiroStatEta: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  80. def getMiroStatTau: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  81. def getModelDraft: String
    Definition Classes
    HasLlamaCppModelProperties
  82. def getModelIfNotSet: GGUFWrapperMultiModal
    Definition Classes
    AutoGGUFVisionModel
  83. def getModelParameters: ModelParameters
    Attributes
    protected
    Definition Classes
    HasLlamaCppModelProperties
  84. def getNBatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  85. def getNCtx: Int
    Definition Classes
    HasLlamaCppModelProperties
  86. def getNDraft: Int
    Definition Classes
    HasLlamaCppModelProperties
  87. def getNGpuLayers: Int
    Definition Classes
    HasLlamaCppModelProperties
  88. def getNGpuLayersDraft: Int
    Definition Classes
    HasLlamaCppModelProperties
  89. def getNKeep: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  90. def getNPredict: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  91. def getNProbs: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  92. def getNThreads: Int
    Definition Classes
    HasLlamaCppModelProperties
  93. def getNThreadsBatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  94. def getNUbatch: Int
    Definition Classes
    HasLlamaCppModelProperties
  95. def getNoKvOffload: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  96. def getNuma: String
    Definition Classes
    HasLlamaCppModelProperties
  97. final def getOrDefault[T](param: Param[T]): T
    Definition Classes
    Params
  98. final def getOutputCol: String
    Definition Classes
    HasOutputAnnotationCol
  99. def getParam(paramName: String): Param[Any]
    Definition Classes
    Params
  100. def getPenalizeNl: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  101. def getPenaltyPrompt: String
    Definition Classes
    HasLlamaCppInferenceProperties
  102. def getPresencePenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  103. def getRepeatLastN: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  104. def getRepeatPenalty: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  105. def getRopeFreqBase: Float
    Definition Classes
    HasLlamaCppModelProperties
  106. def getRopeFreqScale: Float
    Definition Classes
    HasLlamaCppModelProperties
  107. def getRopeScalingType: String
    Definition Classes
    HasLlamaCppModelProperties
  108. def getSamplers: Array[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  109. def getSeed: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  110. def getSplitMode: String
    Definition Classes
    HasLlamaCppModelProperties
  111. def getStopStrings: Array[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  112. def getSystemPrompt: String
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  113. def getTemperature: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  114. def getTfsZ: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  115. def getTokenBias: Map[String, Float]
    Definition Classes
    HasLlamaCppInferenceProperties
  116. def getTokenIdBias: Map[Int, Float]
    Definition Classes
    HasLlamaCppInferenceProperties
  117. def getTopK: Int
    Definition Classes
    HasLlamaCppInferenceProperties
  118. def getTopP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  119. def getTypicalP: Float
    Definition Classes
    HasLlamaCppInferenceProperties
  120. def getUseChatTemplate: Boolean
    Definition Classes
    HasLlamaCppInferenceProperties
  121. def getUseMlock: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  122. def getUseMmap: Boolean
    Definition Classes
    HasLlamaCppModelProperties
  123. def getYarnAttnFactor: Float
    Definition Classes
    HasLlamaCppModelProperties
  124. def getYarnBetaFast: Float
    Definition Classes
    HasLlamaCppModelProperties
  125. def getYarnBetaSlow: Float
    Definition Classes
    HasLlamaCppModelProperties
  126. def getYarnExtFactor: Float
    Definition Classes
    HasLlamaCppModelProperties
  127. def getYarnOrigCtx: Int
    Definition Classes
    HasLlamaCppModelProperties
  128. val gpuSplitMode: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  129. val grammar: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  130. final def hasDefault[T](param: Param[T]): Boolean
    Definition Classes
    Params
  131. def hasParam(paramName: String): Boolean
    Definition Classes
    Params
  132. def hasParent: Boolean
    Definition Classes
    Model
  133. def hashCode(): Int
    Definition Classes
    AnyRef → Any
    Annotations
    @native()
  134. val ignoreEos: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  135. def initializeLogIfNecessary(isInterpreter: Boolean, silent: Boolean): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  136. def initializeLogIfNecessary(isInterpreter: Boolean): Unit
    Attributes
    protected
    Definition Classes
    Logging
  137. val inputAnnotatorTypes: Array[AnnotatorType]
    Definition Classes
    AutoGGUFVisionModel → HasInputAnnotationCols
  138. final val inputCols: StringArrayParam
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  139. val inputPrefix: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  140. val inputSuffix: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  141. final def isDefined(param: Param[_]): Boolean
    Definition Classes
    Params
  142. final def isInstanceOf[T0]: Boolean
    Definition Classes
    Any
  143. final def isSet(param: Param[_]): Boolean
    Definition Classes
    Params
  144. def isTraceEnabled(): Boolean
    Attributes
    protected
    Definition Classes
    Logging
  145. val lazyAnnotator: BooleanParam
    Definition Classes
    CanBeLazy
  146. def log: Logger
    Attributes
    protected
    Definition Classes
    Logging
  147. def logDebug(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  148. def logDebug(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  149. def logError(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  150. def logError(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  151. def logInfo(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  152. def logInfo(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  153. def logName: String
    Attributes
    protected
    Definition Classes
    Logging
  154. def logTrace(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  155. def logTrace(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  156. val logVerbosity: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  157. def logWarning(msg: ⇒ String, throwable: Throwable): Unit
    Attributes
    protected
    Definition Classes
    Logging
  158. def logWarning(msg: ⇒ String): Unit
    Attributes
    protected
    Definition Classes
    Logging
  159. val logger: Logger
    Attributes
    protected
    Definition Classes
    HasLlamaCppModelProperties
  160. val mainGpu: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  161. val metadata: ProtectedParam[String]
    Definition Classes
    HasLlamaCppModelProperties
  162. val minKeep: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  163. val minP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  164. val miroStat: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  165. val miroStatEta: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  166. val miroStatTau: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  167. val modelDraft: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  168. def msgHelper(schema: StructType): String
    Attributes
    protected
    Definition Classes
    HasInputAnnotationCols
  169. val nBatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  170. val nCtx: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  171. val nDraft: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  172. val nGpuLayers: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  173. val nGpuLayersDraft: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  174. val nKeep: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  175. val nPredict: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  176. val nProbs: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  177. val nThreads: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  178. val nThreadsBatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  179. val nUbatch: IntParam
    Definition Classes
    HasLlamaCppModelProperties
  180. final def ne(arg0: AnyRef): Boolean
    Definition Classes
    AnyRef
  181. val noKvOffload: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  182. final def notify(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  183. final def notifyAll(): Unit
    Definition Classes
    AnyRef
    Annotations
    @native()
  184. val numaStrategy: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  185. def onWrite(path: String, spark: SparkSession): Unit
    Definition Classes
    AutoGGUFVisionModel → ParamsAndFeaturesWritable
  186. val optionalInputAnnotatorTypes: Array[String]
    Definition Classes
    HasInputAnnotationCols
  187. val outputAnnotatorType: AnnotatorType
    Definition Classes
    AutoGGUFVisionModel → HasOutputAnnotatorType
  188. final val outputCol: Param[String]
    Attributes
    protected
    Definition Classes
    HasOutputAnnotationCol
  189. lazy val params: Array[Param[_]]
    Definition Classes
    Params
  190. var parent: Estimator[AutoGGUFVisionModel]
    Definition Classes
    Model
  191. val penalizeNl: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  192. val penaltyPrompt: Param[String]
    Definition Classes
    HasLlamaCppInferenceProperties
  193. val presencePenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  194. val repeatLastN: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  195. val repeatPenalty: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  196. val ropeFreqBase: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  197. val ropeFreqScale: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  198. val ropeScalingType: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  199. val samplers: StringArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  200. def save(path: String): Unit
    Definition Classes
    MLWritable
    Annotations
    @Since( "1.6.0" ) @throws( ... )
  201. val seed: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  202. def set[T](param: ProtectedParam[T], value: T): MedicalVisionLLM.this.type
    Definition Classes
    HasProtectedParams
  203. def set[T](feature: StructFeature[T], value: T): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  204. def set[K, V](feature: MapFeature[K, V], value: Map[K, V]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  205. def set[T](feature: SetFeature[T], value: Set[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  206. def set[T](feature: ArrayFeature[T], value: Array[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  207. final def set(paramPair: ParamPair[_]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  208. final def set(param: String, value: Any): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  209. final def set[T](param: Param[T], value: T): MedicalVisionLLM.this.type
    Definition Classes
    Params
  210. def setBatchSize(size: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasBatchedAnnotateTextImage
  211. def setCachePrompt(cachePrompt: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  212. def setChatTemplate(chatTemplate: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  213. def setDefault[T](feature: StructFeature[T], value: () ⇒ T): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  214. def setDefault[K, V](feature: MapFeature[K, V], value: () ⇒ Map[K, V]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  215. def setDefault[T](feature: SetFeature[T], value: () ⇒ Set[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  216. def setDefault[T](feature: ArrayFeature[T], value: () ⇒ Array[T]): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    HasFeatures
  217. final def setDefault(paramPairs: ParamPair[_]*): MedicalVisionLLM.this.type
    Attributes
    protected
    Definition Classes
    Params
  218. final def setDefault[T](param: Param[T], value: T): MedicalVisionLLM.this.type
    Attributes
    protected[org.apache.spark.ml]
    Definition Classes
    Params
  219. def setDefragmentationThreshold(defragThold: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  220. def setDisableLog(disableLog: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  221. def setDisableTokenIds(disableTokenIds: Array[Int]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  222. def setDynamicTemperatureExponent(dynatempExponent: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  223. def setDynamicTemperatureRange(dynatempRange: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  224. def setFlashAttention(flashAttention: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  225. def setFrequencyPenalty(frequencyPenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  226. def setGpuSplitMode(splitMode: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  227. def setGrammar(grammar: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  228. def setIgnoreEos(ignoreEos: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  229. final def setInputCols(value: String*): MedicalVisionLLM.this.type
    Definition Classes
    HasInputAnnotationCols
  230. def setInputCols(value: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasInputAnnotationCols
  231. def setInputPrefix(inputPrefix: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  232. def setInputSuffix(inputSuffix: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  233. def setLazyAnnotator(value: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    CanBeLazy
  234. def setLogVerbosity(logVerbosity: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  235. def setMainGpu(mainGpu: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  236. def setMetadata(metadata: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  237. def setMinKeep(minKeep: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  238. def setMinP(minP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  239. def setMiroStat(mirostat: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  240. def setMiroStatEta(mirostatEta: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  241. def setMiroStatTau(mirostatTau: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  242. def setModelDraft(modelDraft: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  243. def setModelIfNotSet(spark: SparkSession, wrapper: GGUFWrapperMultiModal): MedicalVisionLLM.this.type
    Definition Classes
    AutoGGUFVisionModel
  244. def setNBatch(nBatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  245. def setNCtx(nCtx: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  246. def setNDraft(nDraft: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  247. def setNGpuLayers(nGpuLayers: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  248. def setNGpuLayersDraft(nGpuLayersDraft: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  249. def setNKeep(nKeep: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  250. def setNParallel(nParallel: Int): MedicalVisionLLM.this.type
    Definition Classes
    AutoGGUFVisionModel
  251. def setNPredict(nPredict: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  252. def setNProbs(nProbs: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  253. def setNThreads(nThreads: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  254. def setNThreadsBatch(nThreadsBatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  255. def setNUbatch(nUbatch: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  256. def setNoKvOffload(noKvOffload: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  257. def setNumaStrategy(numa: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  258. final def setOutputCol(value: String): MedicalVisionLLM.this.type
    Definition Classes
    HasOutputAnnotationCol
  259. def setParent(parent: Estimator[AutoGGUFVisionModel]): AutoGGUFVisionModel
    Definition Classes
    Model
  260. def setPenalizeNl(penalizeNl: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  261. def setPenaltyPrompt(penaltyPrompt: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  262. def setPresencePenalty(presencePenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  263. def setRepeatLastN(repeatLastN: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  264. def setRepeatPenalty(repeatPenalty: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  265. def setRopeFreqBase(ropeFreqBase: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  266. def setRopeFreqScale(ropeFreqScale: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  267. def setRopeScalingType(ropeScalingType: String): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  268. def setSamplers(samplers: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  269. def setSeed(seed: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  270. def setStopStrings(stopStrings: Array[String]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  271. def setSystemPrompt(systemPrompt: String): MedicalVisionLLM.this.type
    Definition Classes
    MedicalVisionLLM → HasLlamaCppModelProperties
  272. def setTemperature(temperature: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  273. def setTfsZ(tfsZ: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  274. def setTokenBias(tokenBias: HashMap[String, Double]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  275. def setTokenBias(tokenBias: Map[String, Float]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  276. def setTokenIdBias(tokenIdBias: HashMap[Integer, Double]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  277. def setTokenIdBias(tokenIdBias: Map[Int, Float]): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  278. def setTopK(topK: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  279. def setTopP(topP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  280. def setTypicalP(typicalP: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  281. def setUseChatTemplate(useChatTemplate: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppInferenceProperties
  282. def setUseMlock(useMlock: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  283. def setUseMmap(useMmap: Boolean): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  284. def setYarnAttnFactor(yarnAttnFactor: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  285. def setYarnBetaFast(yarnBetaFast: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  286. def setYarnBetaSlow(yarnBetaSlow: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  287. def setYarnExtFactor(yarnExtFactor: Float): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  288. def setYarnOrigCtx(yarnOrigCtx: Int): MedicalVisionLLM.this.type
    Definition Classes
    HasLlamaCppModelProperties
  289. val stopStrings: StringArrayParam
    Definition Classes
    HasLlamaCppInferenceProperties
  290. final def synchronized[T0](arg0: ⇒ T0): T0
    Definition Classes
    AnyRef
  291. val systemPrompt: Param[String]
    Definition Classes
    HasLlamaCppModelProperties
  292. val temperature: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  293. val tfsZ: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  294. def toString(): String
    Definition Classes
    Identifiable → AnyRef → Any
  295. val tokenBias: StructFeature[Map[String, Float]]
    Definition Classes
    HasLlamaCppInferenceProperties
  296. val tokenIdBias: StructFeature[Map[Int, Float]]
    Definition Classes
    HasLlamaCppInferenceProperties
  297. val topK: IntParam
    Definition Classes
    HasLlamaCppInferenceProperties
  298. val topP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  299. final def transform(dataset: Dataset[_]): DataFrame
    Definition Classes
    AnnotatorModel → Transformer
  300. def transform(dataset: Dataset[_], paramMap: ParamMap): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" )
  301. def transform(dataset: Dataset[_], firstParamPair: ParamPair[_], otherParamPairs: ParamPair[_]*): DataFrame
    Definition Classes
    Transformer
    Annotations
    @Since( "2.0.0" ) @varargs()
  302. final def transformSchema(schema: StructType): StructType
    Definition Classes
    RawAnnotator → PipelineStage
  303. def transformSchema(schema: StructType, logging: Boolean): StructType
    Attributes
    protected
    Definition Classes
    PipelineStage
    Annotations
    @DeveloperApi()
  304. val typicalP: FloatParam
    Definition Classes
    HasLlamaCppInferenceProperties
  305. val uid: String
    Definition Classes
    MedicalVisionLLM → AutoGGUFVisionModel → Identifiable
  306. val useChatTemplate: BooleanParam
    Definition Classes
    HasLlamaCppInferenceProperties
  307. val useMlock: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  308. val useMmap: BooleanParam
    Definition Classes
    HasLlamaCppModelProperties
  309. def validate(schema: StructType): Boolean
    Attributes
    protected
    Definition Classes
    RawAnnotator
  310. final def wait(): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  311. final def wait(arg0: Long, arg1: Int): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... )
  312. final def wait(arg0: Long): Unit
    Definition Classes
    AnyRef
    Annotations
    @throws( ... ) @native()
  313. def wrapColumnMetadata(col: Column): Column
    Attributes
    protected
    Definition Classes
    RawAnnotator
  314. def write: MLWriter
    Definition Classes
    ParamsAndFeaturesWritable → DefaultParamsWritable → MLWritable
  315. val yarnAttnFactor: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  316. val yarnBetaFast: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  317. val yarnBetaSlow: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  318. val yarnExtFactor: FloatParam
    Definition Classes
    HasLlamaCppModelProperties
  319. val yarnOrigCtx: IntParam
    Definition Classes
    HasLlamaCppModelProperties

Inherited from CheckLicense

Inherited from AutoGGUFVisionModel

Inherited from HasProtectedParams

Inherited from HasLlamaCppInferenceProperties

Inherited from HasLlamaCppModelProperties

Inherited from HasEngine

Inherited from HasBatchedAnnotateTextImage[AutoGGUFVisionModel]

Inherited from AnnotatorModel[AutoGGUFVisionModel]

Inherited from CanBeLazy

Inherited from RawAnnotator[AutoGGUFVisionModel]

Inherited from HasOutputAnnotationCol

Inherited from HasInputAnnotationCols

Inherited from HasOutputAnnotatorType

Inherited from ParamsAndFeaturesWritable

Inherited from HasFeatures

Inherited from DefaultParamsWritable

Inherited from MLWritable

Inherited from Model[AutoGGUFVisionModel]

Inherited from Transformer

Inherited from PipelineStage

Inherited from Logging

Inherited from Params

Inherited from Serializable

Inherited from Serializable

Inherited from Identifiable

Inherited from AnyRef

Inherited from Any

Members