sparkocr.transformers.readers.rvlcdip_reader#

Module Contents#

Classes#

RvlCdipReader

class RvlCdipReader#
readTestDataset(spark, path, partitions=8, storage_level=pyspark.StorageLevel.DISK_ONLY)#

Reads the RVL-CDP train dataset from an external resource.

Parameters:

  • spark (pyspark.sql.SparkSession) – Initiated Spark Session with Spark NLP

  • path (str) – the path where you unzip the files for RvlCdip Test

  • partitions (sets the minimum number of partitions for the case of lifting multiple files in parallel into a single dataframe. Defaults to 8.) –

  • storage_level (sets the persistence level according to PySpark definitions. Defaults to StorageLevel.DISK_ONLY.) –

Returns:

Spark Dataframe with the data

Return type:

pyspark.sql.DataFrame

readTrainDataset(spark, labels_path, images_path, partitions=8, storage_level=pyspark.StorageLevel.DISK_ONLY)#

Reads the dataset from an external resource.

Parameters:

  • spark (pyspark.sql.SparkSession) – Initiated Spark Session with Spark NLP

  • labels_path (str) – The path to the labels file, i.e., labels/train.txt

  • images_path (str) –

    the path where you unzipped the files for RvlCdip Train Images partitions : int

    sets the minimum number of partitions for the case of lifting multiple files in parallel into a single dataframe. Defaults to 8.

  • storage_level (sets the persistence level according to PySpark definitions. Defaults to StorageLevel.DISK_ONLY.) –