pyevoc.features.unigram_selection

Unigram cleaning and selection.

This module implements the unigram filtering stage used before PyEvoc term-level indicator computation.

The selection logic follows the original PyEvoc analytical workflow:

• retain focal UPOS categories;

• construct analytical terms from lemmas or emoji tokens;

• remove stop words, interactional markers and low-content lexical items;

• remove URLs, punctuation-only tokens, numeric tokens and malformed clitics;

• retain valid emoji tokens;

• apply minimum document/user reliability filters;

• preserve token-level metadata columns.

The output remains a token-level dataframe enriched with the term column.

class pyevoc.features.unigram_selection.UnigramSelectionConfig(focal_upos=<factory>, token_col='text', display_col=None, lemma_col='lemma', upos_col='upos', doc_col='doc_id', user_col='user_id', term_col='term', stop_words=<factory>, min_chars=2, min_docs_per_term=3, min_users_per_term=3, exclude_url_token=True, require_latin_terms=True, remove_numeric_terms=True, remove_punct_only=True, lowercase_non_emoji=True, return_diagnostics=False, verbose=True)

Bases: object

Configuration for unigram cleaning and selection.

Parameters:

• focal_upos (set[str])

• token_col (str)

• display_col (str | None)

• lemma_col (str)

• upos_col (str)

• doc_col (str)

• user_col (str)

• term_col (str)

• stop_words (set[str])

• min_chars (int)

• min_docs_per_term (int)

• min_users_per_term (int)

• exclude_url_token (bool)

• require_latin_terms (bool)

• remove_numeric_terms (bool)

• remove_punct_only (bool)

• lowercase_non_emoji (bool)

• return_diagnostics (bool)

• verbose (bool)

focal_upos: set[str]

token_col: str = 'text'

display_col: str | None = None

lemma_col: str = 'lemma'

upos_col: str = 'upos'

doc_col: str = 'doc_id'

user_col: str = 'user_id'

term_col: str = 'term'

stop_words: set[str]

min_chars: int = 2

min_docs_per_term: int = 3

min_users_per_term: int = 3

exclude_url_token: bool = True

require_latin_terms: bool = True

remove_numeric_terms: bool = True

remove_punct_only: bool = True

lowercase_non_emoji: bool = True

return_diagnostics: bool = False

verbose: bool = True

pyevoc.features.unigram_selection.is_punct_only(value)

Return True if a value consists only of punctuation.

Parameters:

value (object)

Return type:

bool

pyevoc.features.unigram_selection.is_latin_term(value)

Return True if a value contains only Latin letters, apostrophes or hyphens.

Parameters:

value (object)

Return type:

bool

pyevoc.features.unigram_selection.is_valid_emoji(value)

Return True if a value is a valid emoji token.

Parameters:

value (object)

Return type:

bool

pyevoc.features.unigram_selection.build_unigram_terms(tokens, *, config=None)

Append the analytical unigram term column.

Parameters:

• tokens (DataFrame)

• config (UnigramSelectionConfig | None)

Return type:

DataFrame

pyevoc.features.unigram_selection.unigram_selection_diagnostics(*, input_tokens, base_mask, lexical_mask, emoji_mask, initial_mask, cleaned, config)

Return diagnostic tables for the unigram selection step.

Parameters:

• input_tokens (int)

• base_mask (Series)

• lexical_mask (Series)

• emoji_mask (Series)

• initial_mask (Series)

• cleaned (DataFrame)

• config (UnigramSelectionConfig)

Return type:

tuple[DataFrame, DataFrame]

pyevoc.features.unigram_selection.clean_unigram_tokens(tokens, *, config=None)

Clean and select unigram tokens.

Parameters:

• tokens (DataFrame)

• config (UnigramSelectionConfig | None)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame]

pyevoc.features.unigram_selection.select_unigrams(tokens, *, lemma_col='lemma', upos_col='upos', keep_upos=None, min_chars=2, lowercase=True, token_col='text', doc_col='doc_id', user_col='user_id', min_docs_per_term=3, min_users_per_term=3, return_diagnostics=False)

Backward-compatible wrapper for unigram selection.

Parameters:

• tokens (DataFrame)

• lemma_col (str)

• upos_col (str)

• keep_upos (set[str] | None)

• min_chars (int)

• lowercase (bool)

• token_col (str)

• doc_col (str)

• user_col (str)

• min_docs_per_term (int)

• min_users_per_term (int)

• return_diagnostics (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame]