pyevoc.preprocessing.corpus_statistics

Corpus-level descriptive statistics.

This module computes descriptive statistics on raw textual data before linguistic preprocessing. The tokenisation is Unicode-aware and preserves words, contractions, numbers, hashtags, mentions, and emojis.

class pyevoc.preprocessing.corpus_statistics.CorpusStatisticsConfig(text_col='text', doc_col='doc_id', user_col='user_id', source_col='source', time_col='time', lowercase=True, show_progress=True, as_percent=True)

Bases: object

Configuration for corpus-level descriptive statistics.

Parameters:

• text_col (str)

• doc_col (str)

• user_col (str)

• source_col (str)

• time_col (str)

• lowercase (bool)

• show_progress (bool)

• as_percent (bool)

text_col: str = 'text'

doc_col: str = 'doc_id'

user_col: str = 'user_id'

source_col: str = 'source'

time_col: str = 'time'

lowercase: bool = True

show_progress: bool = True

as_percent: bool = True

pyevoc.preprocessing.corpus_statistics.simple_tokenise(text)

Tokenise raw text using a Unicode-aware regular expression.

The function preserves: - words; - contractions; - numbers; - hashtags; - mentions; - emojis and other Unicode grapheme clusters.

Parameters:

text (object)

Return type:

list[str]

pyevoc.preprocessing.corpus_statistics.is_valid_stat_token(token)

Return True if a token should be retained for corpus statistics.

Empty strings, whitespace-only tokens, and punctuation-only tokens are removed. Emojis and symbolic Unicode grapheme clusters are retained.

Parameters:

token (object)

Return type:

bool

pyevoc.preprocessing.corpus_statistics.extract_tokens(texts, *, lowercase=True, show_progress=True)

Extract valid tokens from a sequence of raw texts.

Parameters:

• texts (Iterable[object])

• lowercase (bool)

• show_progress (bool)

Return type:

list[str]

pyevoc.preprocessing.corpus_statistics.corpus_statistics(df, text_col='text', *, doc_col='doc_id', user_col='user_id', time_col='time', lowercase=True, show_progress=True, as_dataframe=True)

Compute corpus-level descriptive statistics.

Parameters:

• df (DataFrame) – Input dataframe.

• text_col (str) – Name of the raw text column.

• doc_col (str) – Name of the document identifier column.

• user_col (str) – Name of the user identifier column.

• source_col – Name of the source/community/platform column.

• time_col (str) – Name of the timestamp column.

• lowercase (bool) – If True, tokens are lowercased before computing lexical statistics.

• show_progress (bool) – If True, display a progress bar during token extraction.

• as_dataframe (bool) – If True, return a display-ready pandas DataFrame. If False, return a dictionary.

Returns:

Corpus-level descriptive statistics.

Return type:

pandas.DataFrame or dict