sparkocr.transformers.readers.docvqa_reader#

Classes#

class DocVQA(questionsCol='questions')#

Bases: sparknlp.internal.ExtendedJavaWrapper

Reads Doc VQA datasets{train, valid, test} into a Spark Dataframe to be consume by Visual NLP

java_obj#

sc = None#

apply()#

getDataFrame(spark, jdf)#

new_java_array(pylist, java_class)#

ToDo: Inspired from spark 2.0. Review if spark changes

new_java_array_integer(pylist)#

new_java_array_string(pylist)#

new_java_obj(java_class, *args)#

readDataset(spark, jsonPath, imagesFolder, partitions=8, storage_level=pyspark.StorageLevel.DISK_ONLY)#

Reads the dataset from an external resource.

Parameters:

• spark (pyspark.sql.SparkSession) – Initiated Spark Session with Spark NLP

• jsonPath (str) – the path where you unzip the files for DocVQA

• imagesFolder (str) – the folder containing all the images

• partitions (sets the minimum number of partitions for the case of lifting multiple files in parallel into a single dataframe. Defaults to 8.) –

• storage_level (sets the persistence level according to PySpark definitions. Defaults to StorageLevel.DISK_ONLY.) –

Returns:

Spark Dataframe with the data

Return type:

pyspark.sql.DataFrame

spark_version()#