sparkocr.base.light_pipelines#

Classes#

class AnnotationTypeInt#

class LightPipeline(pipelineModel: pyspark.ml.pipeline.PipelineModel)#

Creates a LightPipeline from a Spark OCR PipelineModel.

LightPipeline is a Spark OCR specific Pipeline class equivalent to Spark ML Pipeline. The difference is that it’s execution does not hold to Spark principles, instead it computes everything locally (but in parallel) in order to achieve fast results when dealing with small amounts of data. This means, we do not input a Spark Dataframe, but a string or an Array of strings instead, to be annotated. To create Light Pipelines, you need to pass an already trained (fit) Spark ML Pipeline.

Parameters:

pipelineModel (pyspark.ml.PipelineModel) – The PipelineModel containing Spark OCR Annotators

Notes

Use fullAnnotate() to also output the result as Annotation, with metadata.

Examples

>>> from sparkocr.base import LightPipeline
>>> light = LightPipeline(pipeline.fit(data))
>>> light.fromLocalPath("./documents/pdfs")

pipeline_model#

fromBinary(buffer)#

Runs the provided pipeline against in memory buffer.

The buffer must be byte array of PDFs, DOCX, or images.

fromLocalPath(local_folder)#

Runs the provided pipeline against files in local_folder.

The files can be PDFs, DOCX, or images.

Parameters:

local_folder (list or str) – The data to be annotated

Returns:

Ocr Output as a Spark NLP Annotation ? Can be :)

Return type:

List[Annotation]

fromString(buffer)#

Runs the provided pipeline against in memory buffer.

The buffer must be base64 encoding of PDFs, DOCX, or images.

getIgnoreUnsupported()#

Gets whether to ignore unsupported AnnotatorModels.

Returns:

Whether to ignore unsupported AnnotatorModels.

Return type:

bool

setIgnoreUnsupported(value)#

Sets whether to ignore unsupported AnnotatorModels.

Parameters:

value (bool) – Whether to ignore unsupported AnnotatorModels.

Returns:

The current LightPipeline

Return type:

LightPipeline

transform(dataframe)#

Transforms a dataframe provided with the stages of the LightPipeline.

Parameters:

dataframe (pyspark.sql.DataFrame) – The Dataframe to be transformed

Returns:

The transformed DataFrame

Return type:

pyspark.sql.DataFrame