pyevoc.data.dataset#

Dataset ingestion and canonical corpus preparation.

The canonical PyEvoc schema is domain-agnostic: user_id, doc_id, time, source, text.

class pyevoc.data.dataset.DatasetConfig(column_map, start_date=None, end_date=None, timezone='UTC', drop_missing_text=True, keep_extra_columns=False, read_kwargs=<factory>, show_progress=True)[source]#

Bases: object

Configuration for converting a table into a canonical PyEvoc corpus.

Parameters:

  • column_map (Mapping[str, str]) – Mapping from input column names to canonical column names. Required canonical names are user_id, doc_id, time, and text.

  • start_date (str | None) – Optional inclusive date bounds in the same ISO-like format, for example YYYY-MM-DD. The end date is inclusive up to the end of the day.

  • end_date (str | None) – Optional inclusive date bounds in the same ISO-like format, for example YYYY-MM-DD. The end date is inclusive up to the end of the day.

  • timezone (str) – Time zone used when parsing timestamps. UTC is recommended for social-media corpora.

  • drop_missing_text (bool) – Whether to remove records with missing or empty text.

  • keep_extra_columns (bool)

  • read_kwargs (dict)

  • show_progress (bool)

column_map: Mapping[str, str]#
start_date: str | None = None#
end_date: str | None = None#
timezone: str = 'UTC'#
drop_missing_text: bool = True#
keep_extra_columns: bool = False#
read_kwargs: dict#
show_progress: bool = True#
pyevoc.data.dataset.standardise_dataset(df, config)[source]#

Return a canonical PyEvoc corpus from an existing DataFrame.

Parameters:
Return type:

DataFrame

pyevoc.data.dataset.load_dataset(path, config)[source]#

Load a table from disk and convert it to the canonical PyEvoc schema.

Parameters:
Return type:

DataFrame

pyevoc.data.dataset.corpus_summary(df)[source]#

Return basic corpus-level counts for a canonical PyEvoc corpus.

Parameters:

df (DataFrame)

Return type:

dict