ketl.loader package

Submodules

ketl.loader.Loader module

class ketl.loader.Loader.BaseLoader(destination: Union[pathlib.Path, str], **kwargs)[source]

Bases: object

The base loader class. Not intended to be instantiated directly.

abstract finalize()[source]

Abstract method that must be implemented by any subclass. Finalizes any closing that needs to happen.

abstract load(data_frame: pandas.core.frame.DataFrame)[source]

Abstract method that must be implemented by any subclass. Performs the actual load.

class ketl.loader.Loader.DatabaseLoader(destination: str, **kwargs)[source]

Bases: ketl.loader.Loader.BaseLoader

A loader that writes data to a database table. The table is presumed to already exist.

finalize()[source]

Abstract method that must be implemented by any subclass. Finalizes any closing that needs to happen.

load(data_frame: pandas.core.frame.DataFrame)[source]

Write the data to a table. Deletes data if it hasn’t been deleted already.

Parameters

data_frame –

Returns

class ketl.loader.Loader.DelimitedFileLoader(destination: Union[pathlib.Path, str], naming_func: Optional[Callable] = None, clean: bool = True, **kwargs)[source]

Bases: ketl.loader.Loader.LocalFileLoader

A loader that writes delimited data to a text file.

load(data_frame: pandas.core.frame.DataFrame)[source]

Write the data to a delimited text file. :param data_frame: a Pandas data frame. :return: None.

class ketl.loader.Loader.HashLoader(destination: Union[pathlib.Path, str], **kwargs)[source]

Bases: ketl.loader.Loader.BaseLoader

finalize()[source]

Abstract method that must be implemented by any subclass. Finalizes any closing that needs to happen.

load(data_frame: pandas.core.frame.DataFrame, **kwargs)[source]

Writes a hash to a file that contains the value of a data frame.

Parameters

  • data_frame – a Pandas data frame.

  • kwargs – ignored.

Returns

None

exception ketl.loader.Loader.InvalidLoaderConfiguration[source]

Bases: Exception

class ketl.loader.Loader.LocalFileLoader(destination: Union[pathlib.Path, str], naming_func: Optional[Callable] = None, clean: bool = True, **kwargs)[source]

Bases: ketl.loader.Loader.BaseLoader

finalize()[source]

Abstract method that must be implemented by any subclass. Finalizes any closing that needs to happen.

full_path(df: pandas.core.frame.DataFrame)[source]
class ketl.loader.Loader.ParquetLoader(destination: Union[pathlib.Path, str], naming_func: Optional[Callable] = None, clean: bool = True, **kwargs)[source]

Bases: ketl.loader.Loader.LocalFileLoader

A loader that writes data to a Parquet file.

finalize()[source]

If a writer is open, close it.

Returns

None

load(data_frame: pandas.core.frame.DataFrame)[source]

Write data to a Parquet file.

Parameters

data_frame – a Pandas data frame.

Returns

None

class ketl.loader.Loader.PickleLoader(destination: Union[pathlib.Path, str], naming_func: Optional[Callable] = None, pickler=None, **kwargs)[source]

Bases: ketl.loader.Loader.LocalFileLoader

A loader to write the data to a pickle file.

finalize()[source]

Abstract method that must be implemented by any subclass. Finalizes any closing that needs to happen.

load(obj: Any)[source]

Write the data to the file using the pickler.

Parameters

obj –

Returns

Module contents