sparkocr.pretrained.pretrained_pipeline#

Classes#

class PretrainedPipeline(name, lang='en', remote_loc=None, parse_embeddings=False, disk_location=None)#

Loads a fully constructed and trained Spark OCR/NLP pipeline, ready to be used.

This way, a whole pipeline can be defined in 1 line. Additionally, the LightPipeline version of the model can be retrieved with member light_model.

Parameters:

• name (str) – Name of the PretrainedPipeline. These can be gathered from the Pipelines Page.

• lang (str, optional) – Langauge of the model, by default ‘en’

• remote_loc (str, optional) – Link to the remote location of the model (if it was already downloaded), by default None

• parse_embeddings (bool, optional) – Whether to parse embeddings, by default False

• disk_location (str , optional) – Path to locally stored PretrainedPipeline, by default None

fromLocalPath(local_folder)#

Runs the provided pipeline against files in local_folder.

The files can be PDFs, DOCX, or images.

Parameters:

local_folder (list or str) – The data to be annotated

Return type:

List[Annotation]

static from_disk(path, parse_embeddings=False)#

supports_lp(pipeline)#

Check whether a pipeline supports light pipelines.

Parameters:

pipeline (PipelineModel) – The pipeline to check.

Returns:

True if pipeline supports light pipeline, False otherwise.

Return type:

bool

transform(data)#

Transforms the input dataset with Spark.

Parameters:

data (pyspark.sql.DataFrame) – input dataset

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame