sparkocr.transformers.dicom.dicom_metadata_deidentifier#

Classes#

class DicomMetadataDeidentifier#

Bases: pyspark.ml.Transformer, pyspark.ml.param.shared.HasInputCols, pyspark.ml.param.shared.HasOutputCol, pyspark.ml.util.DefaultParamsReadable, pyspark.ml.util.DefaultParamsWritable, sparkocr.transformers.params.HasKeepInput, sparkocr.transformers.dicom.base_dicom.BaseDicom

De-identify metadata of dicom document.

groupStrategyFileContent#

inputCols :Param[List[str]]#

keepInput#

keepMapping#

outputCol :Param[str]#

placeholderText#

readMaxBytes#

regenerateSeries#

removePrivateTags#

strategyFileContent#

tagCleaned#

tagMapping#

uid = ''#

userSeed#

valid_actions#

valid_group_actions#

static add_tag(tag_str, vr, value)#

build_dicom_reference_map(ds, parent_path=None, parent=None, refs=None)#

static clean_tag_name(tag)#

clear(param: Param) → None#

Clears a param from the param map if it has been explicitly set.

copy(extra: pyspark.ml._typing.ParamMap | None = None) → P#

Creates a copy of this instance with the same uid and some extra params. The default implementation creates a shallow copy using copy.copy(), and then copies the embedded and extra parameters over and returns the copy. Subclasses should override this method if the default approach is not sufficient.

Parameters:

extra (dict, optional) – Extra parameters to copy to the new instance

Returns:

Copy of this instance

Return type:

Params

explainParam(param: str | Param) → str#

Explains a single param and returns its name, doc, and optional default value and user-supplied value in a string.

explainParams() → str#

Returns the documentation of all params with their optionally default values and user-supplied values.

extractParamMap(extra: pyspark.ml._typing.ParamMap | None = None) → pyspark.ml._typing.ParamMap#

Extracts the embedded default param values and user-supplied values, and then merges them with extra values from input into a flat param map, where the latter value is used if there exist conflicts, i.e., with ordering: default param values < user-supplied values < extra.

Parameters:

extra (dict, optional) – extra param values

Returns:

merged param map

Return type:

dict

static generate_series_number(old_study_uid, old_series_uid)#

getInputCols() → List[str]#

Gets the value of inputCols or its default value.

getOrDefault(param: str) → Any#

getOrDefault(param: Param[T]) → T

Gets the value of a param in the user-supplied param map or its default value. Raises an error if neither is set.

getOutputCol() → str#

Gets the value of outputCol or its default value.

getParam(paramName: str) → Param#

Gets a param by its name.

getRules(rulesFileContent)#

get_all_values_by_tag_and_vr(tag_str, vr)#

get_all_values_private_or_group(ds, type='private', group_prefix=None)#

get_input_col(dataset, col_type, required=False, position=0)#

get_reference_from_map(path, return_parent=False)#

get_s3_creds(sql_ctx)#

hasDefault(param: str | Param[Any]) → bool#

Checks whether a param has a default value.

hasParam(paramName: str) → bool#

Tests whether this instance contains a param with a given (string) name.

isDefined(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user or has a default value.

isSet(param: str | Param[Any]) → bool#

Checks whether a param is explicitly set by user.

classmethod load(path: str) → RL#

Reads an ML instance from the input path, a shortcut of read().load(path).

classmethod read() → DefaultParamsReader[RL]#

Returns a DefaultParamsReader instance for this class.

static read_dicom(dicom, params, read_max_bytes, origin='', stop_before_pixels=True)#

save(path: str) → None#

Save this ML instance to the given path, a shortcut of ‘write().save(path)’.

set(param: Param, value: Any) → None#

Sets a parameter in the embedded param map.

setGroupStrategyFile(value)#

Sets the value of groupStrategyFileContent.

setInputCols(value)#

Sets the value of inputCols.

setKeepInput(value)#

Sets the value of keepInput.

setKeepMappingCols(value)#

Sets the value of keepMapping.

setOutputCol(value)#

Sets the value of outputCol.

setPlaceholderText(value)#

Sets the value of placeholderText.

setReadMaxBytes(value)#

Sets the value of readMaxBytes.

setRegenerateSeries(value)#

Sets the value of regenerateSeries.

setRemovePrivateTags(value)#

Sets the value of removePrivateTags.

setStrategyFile(value)#

Sets the value of strategyFileContent.

setTagCleanedCol(value)#

Sets the value of tagCleaned.

setTagMappingCol(value)#

Sets the value of tagMapping.

setUserSeed(value)#

Sets the value of userSeed.

transform(dataset: pyspark.sql.dataframe.DataFrame, params: pyspark.ml._typing.ParamMap | None = None) → pyspark.sql.dataframe.DataFrame#

Transforms the input dataset with optional parameters.

New in version 1.3.0.

Parameters:

• dataset (pyspark.sql.DataFrame) – input dataset

• params (dict, optional) – an optional param map that overrides embedded params.

Returns:

transformed dataset

Return type:

pyspark.sql.DataFrame

transform_dicom(dicom, exception, tag_mapping, cleaned_ner)#

write() → MLWriter#

Returns a DefaultParamsWriter instance for this class.

class Obfuscator(names_file_path, surnames_file_path, placeholder=None, user_seed=None)#

Bases: object

default_value#

add_empty_tag(tag, vr, value, option=None)#

If the tag exists and has a value NO_ACTION is returned. If the tag exists but is empty we return a default value. If the tag is missing value should be None, so we return the default for that VR

cap_age(tag, vr, value, option=None)#

Medtronic CST Profile requires that if Age is greater than 90 Set it to 99 else leave the age as is. In case the age from medtronic is not aligned with Dicom Standard will generate random age.

clean_phi(tag, vr, value, option=None)#

static clean_tag(tag, vr, value, option=None)#

static delete_tag(tag, vr, value, option=None)#

dummy_graphic_annotation_sequence(tag, vr, value, option=None)#

dummy_person_identification_code_sequence(tag, vr, value, option=None)#

dummy_verifying_observer_identification_sequence(tag, vr, value, option=None)#

static generate_delta(date_format, range=60)#

Generates a delta for DA and DT VR tags. In case when VR is DA, and value is YYYY, YYYYMM the delta should be large enough In all other cases the above should not be a problem.

date_format is a string representing the format of the date_value

generate_random_dates(vr)#

In case Value for DA/DT VR is not valid, date parsing failed or delta date generation failed We fallback here, and based on VR we generate a random date

generate_random_string(tag, vr, value, option=None)#

generate_random_time(tag, vr, value, option='')#

Generates a random DICOM Time for VR TM If Option is fractional we generate the fractional part of the TM

get_random_name(tag, vr, value, option=None)#

get_random_name_with_seed(tag, vr, value, option='coherent')#

hash_uid(tag, vr, value, option=None)#

static infer_date_format(value, date_length)#

# https://dicom.nema.org/dicom/2013/output/chtml/part05/sect_6.2.html Infers the date format based on the length of the value passed, used for both DA and DT VR In case the value itself is not present DicomToMetadata will set it to <EMPTY>

static literal(tag, vr, value, option=None)#

patient_hash_gen(tag, vr, value, option)#

static read_newline_separated(filename)#

static remove_binary(tag, vr, value, option=None)#

Removes binary VRs by replacing with empty bytes. OB, OD, OF, OW, UN.

static remove_date(tag, vr, value, option=None)#

static remove_numeric(tag, vr, value, option=None)#

Removes numeric VRs by replacing them with zero. Works for SL, SS, UL, US, FL, FD, AT, etc.

static remove_sequence(tag, vr, value, option=None)#

remove_str(tag, vr, value, option=None)#

resolve_method(vr, action)#

resolve the method to be called according to CSV configs

return_empty_if_value_empty()#

shift_age_randomly(tag, vr, value, option=None)#

shift_date_or_time_fixed(tag, vr, value, option=None)#

Works for both DA and DT VR In case the date_format is None, either the value is empty or we failed to handle the format Option is carried from the strategy file

shift_date_or_time_random(tag, vr, value, option=None)#

Works for both DA and DT VR In case the date_format is None, either the value is empty or we failed to handle the format. Option is carried from the strategy file

shift_date_time_days(tag, vr, value, option, dt_format=None)#

Generates a new date based on value, option (range) and inferred date format

shift_past_unix_timestamp(tag, vr, value, option=None)#

Generates a random past unix timestamp based on the input.