sparkocr.transformers.ocr_resource_downloader#

Classes#

class OcrResourceDownloader#

Bases: sparknlp.pretrained.ResourceDownloader

Downloads and manages resources, pretrained models/pipelines.

Usually you will not need to use this class directly. It is called by the pretrained() function of annotators.

However, you can use this class to list the available pretrained resources.

Examples

If you want to list all NerDLModels for the english language you can run:

>>> ResourceDownloader.showPublicModels("NerDLModel", "en")
+-------------+------+---------+
| Model       | lang | version |
+-------------+------+---------+
| onto_100    | en   | 2.1.0   |
| onto_300    | en   | 2.1.0   |
| ner_dl_bert | en   | 2.2.0   |
|  ...        | ...  | ...     |

Similarly for Pipelines:

>>> ResourceDownloader.showPublicPipelines("en")
+------------------+------+---------+
| Pipeline         | lang | version |
+------------------+------+---------+
| dependency_parse | en   | 2.0.2   |
| check_spelling   | en   | 2.1.0   |
| match_datetime   | en   | 2.1.0   |
|  ...             | ...  | ...     |

static clearCache(name, language, remote_loc=None)#

Clears the cache entry of a model.

Parameters:

• name (str) – Name of the model

• language (en) – Language of the model

• remote_loc (str, optional) – Directory of the remote Spark NLP Folder, by default None

static downloadEndpointModel(fileName, modelType)#

static downloadModel(reader, name, language, remote_loc=None, j_dwn='OcrPythonResourceDownloader')#

Downloads and loads a model with the default downloader. Usually this method does not need to be called directly, as it is called by the pretrained() method of the annotator.

Parameters:

• reader (obj) – Class to read the model for

• name (str) – Name of the pretrained model

• language (str) – Language of the model

• remote_loc (str, optional) – Directory of the Spark NLP Folder, by default None

• engine (str, optional) – Preferred Deep Learning engine used to resolve and download the model, by default “onnx”

• skip_preferred_engine (bool, optional) – Whether to skip the preferred engine resolution and use the default engine priority, by default False

• j_dwn (str, optional) – Which java downloader to use, by default ‘PythonResourceDownloader’

Returns:

Loaded pretrained annotator/pipeline

Return type:

AnnotatorModel

static downloadModelDirectly(name, remote_loc='public/models', unzip=True)#

Downloads a model directly to the cache folder. You can use to copy-paste the s3 URI from the model hub and download the model. For available s3 URI and models, please see the Models Hub. :param name: Name of the model or s3 URI :type name: str :param remote_loc: Directory of the remote Spark NLP Folder, by default “public/models” :type remote_loc: str, optional :param unzip: Used to unzip model, by default ‘True’ :type unzip: Bool, optional

static downloadPipeline(name, language, remote_loc=None)#

Downloads and loads a pipeline with the default downloader.

Parameters:

• name (str) – Name of the pipeline

• language (str) – Language of the pipeline

• remote_loc (str, optional) – Directory of the remote Spark OCR Folder, by default None

Returns:

The loaded pipeline

Return type:

PipelineModel

static showAvailableAnnotators()#

Shows all available annotators in Spark NLP.

static showPublicModels(annotator=None, lang=None, version=None)#

Prints all pretrained models for a particular annotator model, that are compatible with a version of Spark NLP. If any of the optional arguments are not set, the filter is not considered.

Parameters:

• annotator (str, optional) – Name of the annotator to filer, by default None

• lang (str, optional) – Language of the models to filter, by default None

• version (str, optional) – Version of Spark NLP to filter, by default None

static showPublicPipelines(lang=None, version=None)#

Prints all pretrained models for a particular annotator model, that are compatible with a version of Spark NLP. If any of the optional arguments are not set, the filter is not considered.

Parameters:

• lang (str, optional) – Language of the models to filter, by default None

• version (str, optional) – Version of Spark NLP to filter, by default None

static showUnCategorizedResources()#

Shows models or pipelines in the metadata which has not been categorized yet.