pyevoc.preprocessing

Language filtering, thematic filtering, corpus statistics, text cleaning, and linguistic annotation.

PyEvoc preprocessing layer.

This subpackage contains document-level and token-level preprocessing routines: emoji-safe text cleaning, corpus diagnostics, language filtering, thematic subsetting, and Stanza-based Universal Dependencies annotation.

The public names are loaded lazily so that API documentation can be generated without importing optional runtime dependencies such as emoji, fasttext or stanza unless the corresponding function is actually requested.

class pyevoc.preprocessing.AnnotationConfig(text_col='text_clean', doc_id_col='post_id', user_col='user_id', time_col='time', language='en', processors='tokenize,pos,lemma,depparse', tokenize_no_ssplit=False, use_gpu=False, batch_size=128, show_progress=True, save_parquet=False, output_dir='annotated_outputs', output_name=None, save_emoji_sample=True)

Bases: object

Configuration for Stanza-based UD annotation.

Parameters:

• text_col (str)

• doc_id_col (str)

• user_col (str)

• time_col (str)

• language (str)

• processors (str)

• tokenize_no_ssplit (bool)

• use_gpu (bool)

• batch_size (int)

• show_progress (bool)

• save_parquet (bool)

• output_dir (str | Path)

• output_name (str | None)

• save_emoji_sample (bool)

batch_size: int = 128

doc_id_col: str = 'post_id'

language: str = 'en'

output_dir: str | Path = 'annotated_outputs'

output_name: str | None = None

processors: str = 'tokenize,pos,lemma,depparse'

save_emoji_sample: bool = True

save_parquet: bool = False

show_progress: bool = True

text_col: str = 'text_clean'

time_col: str = 'time'

tokenize_no_ssplit: bool = False

use_gpu: bool = False

user_col: str = 'user_id'

class pyevoc.preprocessing.CleaningConfig(lowercase=False, decode_html=True, replace_ampersand=True, normalise_apostrophes=True, expand_contractions=True, remove_urls=True, url_placeholder='URL', space_emojis=True, reduce_elongated_vowels=True, remove_control_chars=True, space_punctuation=True, normalise_whitespace=True, show_progress=True)

Bases: object

Configuration for basic text cleaning.

Parameters:

• lowercase (bool)

• decode_html (bool)

• replace_ampersand (bool)

• normalise_apostrophes (bool)

• expand_contractions (bool)

• remove_urls (bool)

• url_placeholder (str)

• space_emojis (bool)

• reduce_elongated_vowels (bool)

• remove_control_chars (bool)

• space_punctuation (bool)

• normalise_whitespace (bool)

• show_progress (bool)

decode_html: bool = True

expand_contractions: bool = True

lowercase: bool = False

normalise_apostrophes: bool = True

normalise_whitespace: bool = True

reduce_elongated_vowels: bool = True

remove_control_chars: bool = True

remove_urls: bool = True

replace_ampersand: bool = True

show_progress: bool = True

space_emojis: bool = True

space_punctuation: bool = True

url_placeholder: str = 'URL'

class pyevoc.preprocessing.CorpusStatisticsConfig(text_col='text', doc_col='doc_id', user_col='user_id', source_col='source', time_col='time', lowercase=True, show_progress=True, as_percent=True)

Bases: object

Configuration for corpus-level descriptive statistics.

Parameters:

• text_col (str)

• doc_col (str)

• user_col (str)

• source_col (str)

• time_col (str)

• lowercase (bool)

• show_progress (bool)

• as_percent (bool)

as_percent: bool = True

doc_col: str = 'doc_id'

lowercase: bool = True

show_progress: bool = True

source_col: str = 'source'

text_col: str = 'text'

time_col: str = 'time'

user_col: str = 'user_id'

class pyevoc.preprocessing.LanguageFilterConfig(text_col='text', target_language='en', min_probability=0.9, ambiguous_min_probability=0.5, ambiguous_max_probability=0.9, use_lingua_fallback=True, lingua_min_confidence=0.8, min_text_length=3, fasttext_batch_size=50000, lingua_batch_size=10000, prediction_col='language', probability_col='language_probability', lingua_prediction_col='language_lingua', lingua_confidence_col='language_lingua_confidence', keep_probability=True, keep_diagnostics=False, show_progress=True, verbose=True)

Bases: object

Configuration for language filtering.

Parameters:

• text_col (str)

• target_language (str)

• min_probability (float)

• ambiguous_min_probability (float)

• ambiguous_max_probability (float)

• use_lingua_fallback (bool)

• lingua_min_confidence (float)

• min_text_length (int)

• fasttext_batch_size (int)

• lingua_batch_size (int)

• prediction_col (str)

• probability_col (str)

• lingua_prediction_col (str)

• lingua_confidence_col (str)

• keep_probability (bool)

• keep_diagnostics (bool)

• show_progress (bool)

• verbose (bool)

ambiguous_max_probability: float = 0.9

ambiguous_min_probability: float = 0.5

fasttext_batch_size: int = 50000

keep_diagnostics: bool = False

keep_probability: bool = True

lingua_batch_size: int = 10000

lingua_confidence_col: str = 'language_lingua_confidence'

lingua_min_confidence: float = 0.8

lingua_prediction_col: str = 'language_lingua'

min_probability: float = 0.9

min_text_length: int = 3

prediction_col: str = 'language'

probability_col: str = 'language_probability'

show_progress: bool = True

target_language: str = 'en'

text_col: str = 'text'

use_lingua_fallback: bool = True

verbose: bool = True

class pyevoc.preprocessing.ThematicFilterConfig(text_col='text', min_anchor_hits=1, similarity_threshold=0.1, min_expansion_hits=2, top_n_expansion_terms=150, min_df=10, max_df=0.5, max_features=50000, stop_words='english', lowercase=True, ngram_range=(1, 2), anchor_col='anchor_hit', anchor_hits_col='anchor_hits', expansion_hits_col='expansion_hits', expansion_col='expansion_hit', subset_flag_col='theme_subset_flag', filter_type_col='filter_type', retain_columns=None, verbose=True, show_progress=True)

Bases: object

Configuration for anchor-based thematic filtering.

Parameters:

• text_col (str)

• min_anchor_hits (int)

• similarity_threshold (float)

• min_expansion_hits (int)

• top_n_expansion_terms (int)

• min_df (int | float)

• max_df (int | float)

• max_features (int | None)

• stop_words (str | list[str] | None)

• lowercase (bool)

• ngram_range (tuple[int, int])

• anchor_col (str)

• anchor_hits_col (str)

• expansion_hits_col (str)

• expansion_col (str)

• subset_flag_col (str)

• filter_type_col (str)

• retain_columns (tuple[str, ...] | None)

• verbose (bool)

• show_progress (bool)

anchor_col: str = 'anchor_hit'

anchor_hits_col: str = 'anchor_hits'

expansion_col: str = 'expansion_hit'

expansion_hits_col: str = 'expansion_hits'

filter_type_col: str = 'filter_type'

lowercase: bool = True

max_df: int | float = 0.5

max_features: int | None = 50000

min_anchor_hits: int = 1

min_df: int | float = 10

min_expansion_hits: int = 2

ngram_range: tuple[int, int] = (1, 2)

retain_columns: tuple[str, ...] | None = None

show_progress: bool = True

similarity_threshold: float = 0.1

stop_words: str | list[str] | None = 'english'

subset_flag_col: str = 'theme_subset_flag'

text_col: str = 'text'

top_n_expansion_terms: int = 150

verbose: bool = True

pyevoc.preprocessing.annotate_dataframe(df, df_name='df', *, doc_id_col='post_id', text_col='text_clean', user_col='user_id', time_col='time', batch_size=128, save_parquet=True, output_dir='annotated_outputs', show_progress=True, use_gpu=False)

Notebook-compatible wrapper around annotate_with_stanza().

This mirrors the original notebook call style:

anno_df = annotate_dataframe(df, "df")

and saves <df_name>_stanza_anno.parquet when save_parquet=True.

Parameters:

• df (DataFrame)

• df_name (str)

• doc_id_col (str)

• text_col (str)

• user_col (str)

• time_col (str)

• batch_size (int)

• save_parquet (bool)

• output_dir (str | Path)

• show_progress (bool)

• use_gpu (bool)

Return type:

DataFrame

pyevoc.preprocessing.annotate_with_stanza(df, config=AnnotationConfig())

Annotate a document-level dataframe with Stanza.

The returned dataframe is the correct input for dependency-based collocation extraction. In particular, it contains token, head_token_id and dep_rel.

Parameters:

• df (DataFrame)

• config (AnnotationConfig)

Return type:

DataFrame

pyevoc.preprocessing.annotation_diagnostics(anno_df, *, doc_col='doc_id', user_col='user_id', time_col='time')

Return a compact diagnostic table for an annotated token dataframe.

Parameters:

• anno_df (DataFrame)

• doc_col (str)

• user_col (str)

• time_col (str)

Return type:

DataFrame

pyevoc.preprocessing.build_thematic_subset(df, anchor_file=None, config=ThematicFilterConfig(), expansion_terms=None, anchors=None)

Build a thematic subcorpus using anchor and expansion filters.

Parameters:

• df (DataFrame) – Input document-level dataframe.

• anchor_file (str | Path | None) – Optional external anchor file. Required when anchors is not given.

• config (ThematicFilterConfig) – Filtering configuration.

• expansion_terms (set[str] | list[str] | None) – Optional pre-computed expansion terms. If not supplied, expansion terms are derived from the input corpus.

• anchors (list[str] | None) – Optional in-memory anchor inventory. If supplied, anchor_file is not required.

Return type:

tuple[DataFrame, dict]

pyevoc.preprocessing.clean_corpus(df, text_col='text', output_col='clean_text', *, config=None, lowercase=None, remove_urls=None)

Apply basic cleaning to a dataframe.

Parameters:

• df (DataFrame) – Input dataframe.

• text_col (str) – Name of the source text column.

• output_col (str) – Name of the output cleaned-text column.

• config (CleaningConfig | None) – Optional cleaning configuration.

• lowercase (bool | None) – Optional override for config.lowercase.

• remove_urls (bool | None) – Optional override for config.remove_urls.

Returns:

Copy of the input dataframe with an additional cleaned-text column.

Return type:

pandas.DataFrame

pyevoc.preprocessing.clean_text(text, *, config=None, lowercase=None, remove_urls=None)

Clean a single text.

Parameters:

• text (object) – Raw textual input.

• config (CleaningConfig | None) – Optional cleaning configuration.

• lowercase (bool | None) – Optional override for config.lowercase.

• remove_urls (bool | None) – Optional override for config.remove_urls.

Returns:

Cleaned text.

Return type:

str

pyevoc.preprocessing.contains_emoji_grapheme(value)

Return True if a string contains at least one emoji grapheme.

Parameters:

value (object)

Return type:

bool

pyevoc.preprocessing.count_emoji_documents(df, *, text_col='text_clean')

Count documents containing at least one emoji grapheme.

Parameters:

• df (DataFrame)

• text_col (str)

Return type:

int

pyevoc.preprocessing.count_emoji_tokens(anno_df, *, token_col='token')

Count annotated tokens containing at least one emoji grapheme.

Parameters:

• anno_df (DataFrame)

• token_col (str)

Return type:

int

pyevoc.preprocessing.dependency_diagnostics(anno_df, *, dep_rel_col='dep_rel', head_col='head_token_id')

Return diagnostics for dependency columns used by collocations.

Parameters:

• anno_df (DataFrame)

• dep_rel_col (str)

• head_col (str)

Return type:

DataFrame

pyevoc.preprocessing.derive_expansion_terms(df, anchors, config)

Derive expansion terms using notebook-style term-term cosine similarity.

The function returns expansion_terms, a diagnostic dataframe, the fitted vectorizer, and the document-term matrix.

Parameters:

• df (DataFrame)

• anchors (list[str])

• config (ThematicFilterConfig)

Return type:

tuple[set[str], DataFrame, CountVectorizer, object]

pyevoc.preprocessing.direct_anchor_filter(df, anchors, config)

Add anchor hit indicators to a dataframe.

Parameters:

• df (DataFrame)

• anchors (list[str])

• config (ThematicFilterConfig)

Return type:

DataFrame

pyevoc.preprocessing.download_fasttext_lid_model(compact=False, overwrite=False)

Download the official fastText language-identification model.

Parameters:

• compact (bool) – If True, download lid.176.ftz instead of lid.176.bin.

• overwrite (bool) – Re-download even if the file already exists.

Returns:

Local model path.

Return type:

pathlib.Path

pyevoc.preprocessing.emoji_token_sample(anno_df, *, n=30, token_col='token')

Return a sample of emoji-bearing annotated tokens.

Parameters:

• anno_df (DataFrame)

• n (int)

• token_col (str)

Return type:

DataFrame

pyevoc.preprocessing.extract_tokens(texts, *, lowercase=True, show_progress=True)

Extract valid tokens from a sequence of raw texts.

Parameters:

• texts (Iterable[object])

• lowercase (bool)

• show_progress (bool)

Return type:

list[str]

pyevoc.preprocessing.fasttext_predict_batch(texts, model, *, batch_size=50000, show_progress=True, verbose=True)

Predict languages with fastText in batches.

The function first attempts fastText’s list-based batch prediction. If that fails because of a fastText/NumPy compatibility issue, it falls back to row-wise prediction. Display is limited to one global progress bar.

Parameters:

• texts (Series)

• model (Any)

• batch_size (int)

• show_progress (bool)

• verbose (bool)

Return type:

tuple[list[str | None], list[float | None]]

pyevoc.preprocessing.filter_by_language(df, config=LanguageFilterConfig(), model_path=None)

Return rows matching the target language.

Decision rule

A document is retained when:

1. fastText predicts the target language with probability greater than or equal to min_probability;

or, if Lingua fallback is enabled,

2. fastText predicts the target language with probability in the ambiguous interval and Lingua confirms the same target language with confidence greater than or equal to lingua_min_confidence.

Parameters:

• df (DataFrame)

• config (LanguageFilterConfig)

• model_path (str | Path | None)

Return type:

DataFrame

pyevoc.preprocessing.is_emoji_grapheme(value)

Return True if a Unicode grapheme cluster is an emoji.

The function is robust to multi-codepoint emojis and modifiers.

Parameters:

value (object)

Return type:

bool

pyevoc.preprocessing.is_valid_stat_token(token)

Return True if a token should be retained for corpus statistics.

Empty strings, whitespace-only tokens, and punctuation-only tokens are removed. Emojis and symbolic Unicode grapheme clusters are retained.

Parameters:

token (object)

Return type:

bool

pyevoc.preprocessing.lingua_detect_batch(texts, *, batch_size=10000, show_progress=True, verbose=True)

Apply Lingua fallback to a series of texts.

Display is limited to one global progress bar. The implementation appends exactly one language label and one confidence value per input text.

Parameters:

• texts (Series)

• batch_size (int)

• show_progress (bool)

• verbose (bool)

Return type:

tuple[list[str | None], list[float | None]]

pyevoc.preprocessing.load_anchor_terms(path, column=None)

Load anchor terms from TXT, CSV/TSV, or JSON.

TXT files should contain one term per line. CSV/TSV files use column if provided; otherwise the first column is used. JSON files may contain a list or a dictionary with an anchors key.

Parameters:

• path (str | Path)

• column (str | None)

Return type:

list[str]

pyevoc.preprocessing.predict_language_fasttext(texts, model_path=None, *, batch_size=50000, show_progress=True, verbose=True)

Predict language labels and probabilities using fastText.

Parameters:

• texts (Series)

• model_path (str | Path | None)

• batch_size (int)

• show_progress (bool)

• verbose (bool)

Return type:

DataFrame

pyevoc.preprocessing.prepare_for_langid(text)

Prepare text for language identification.

Parameters:

text (object)

Return type:

str

pyevoc.preprocessing.separate_emojis(text)

Add spaces around emoji grapheme clusters without splitting them.

Examples

‘great😂’ -> ‘great 😂’

‘ok👍🏽’ -> ‘ok 👍🏽’

Parameters:

text (object)

Return type:

str

pyevoc.preprocessing.simple_tokenise(text)

Tokenise raw text using a Unicode-aware regular expression.

The function preserves: - words; - contractions; - numbers; - hashtags; - mentions; - emojis and other Unicode grapheme clusters.

Parameters:

text (object)

Return type:

list[str]

pyevoc.preprocessing.validate_annotation_input(df, config=AnnotationConfig())

Validate the input dataframe before annotation.

Parameters:

• df (DataFrame)

• config (AnnotationConfig)

Return type:

None

Modules

• pyevoc.preprocessing.annotation
  • Canonical output columns
  • AnnotationConfig
  • annotate_with_stanza()
  • annotate_dataframe()
  • annotation_diagnostics()
  • dependency_diagnostics()
  • contains_emoji_grapheme()
  • count_emoji_documents()
  • count_emoji_tokens()
  • emoji_token_sample()
  • validate_annotation_input()
• pyevoc.preprocessing.cleaning
  • CleaningConfig
  • is_emoji_grapheme()
  • separate_emojis()
  • clean_text()
  • clean_corpus()
• pyevoc.preprocessing.corpus_statistics
  • CorpusStatisticsConfig
  • simple_tokenise()
  • is_valid_stat_token()
  • extract_tokens()
  • corpus_statistics()
• pyevoc.preprocessing.language_filtering
  • LanguageFilterConfig
  • download_fasttext_lid_model()
  • prepare_for_langid()
  • fasttext_predict_batch()
  • predict_language_fasttext()
  • lingua_detect_batch()
  • filter_by_language()
• pyevoc.preprocessing.thematic_filtering
  • ThematicFilterConfig
  • normalise_anchor_terms()
  • load_anchor_terms()
  • compile_anchor_pattern()
  • direct_anchor_filter()
  • build_count_vectorizer()
  • derive_expansion_terms()
  • add_expansion_hits()
  • build_thematic_subset()