pyevoc.preprocessing.language_filtering#

Language filtering utilities for large-scale textual corpora.

PyEvoc uses fastText language identification as the primary detector and can optionally apply a Lingua fallback to uncertain cases.

Large pretrained fastText models are not bundled with PyEvoc. Use download_fasttext_lid_model to download lid.176.bin or lid.176.ftz from the official fastText repository into ~/.pyevoc/models.

class pyevoc.preprocessing.language_filtering.LanguageFilterConfig(text_col='text', target_language='en', min_probability=0.9, ambiguous_min_probability=0.5, ambiguous_max_probability=0.9, use_lingua_fallback=True, lingua_min_confidence=0.8, min_text_length=3, fasttext_batch_size=50000, lingua_batch_size=10000, prediction_col='language', probability_col='language_probability', lingua_prediction_col='language_lingua', lingua_confidence_col='language_lingua_confidence', keep_probability=True, keep_diagnostics=False, show_progress=True, verbose=True)[source]#

Bases: object

Configuration for language filtering.

Parameters:

  • text_col (str)

  • target_language (str)

  • min_probability (float)

  • ambiguous_min_probability (float)

  • ambiguous_max_probability (float)

  • use_lingua_fallback (bool)

  • lingua_min_confidence (float)

  • min_text_length (int)

  • fasttext_batch_size (int)

  • lingua_batch_size (int)

  • prediction_col (str)

  • probability_col (str)

  • lingua_prediction_col (str)

  • lingua_confidence_col (str)

  • keep_probability (bool)

  • keep_diagnostics (bool)

  • show_progress (bool)

  • verbose (bool)

text_col: str = 'text'#
target_language: str = 'en'#
min_probability: float = 0.9#
ambiguous_min_probability: float = 0.5#
ambiguous_max_probability: float = 0.9#
use_lingua_fallback: bool = True#
lingua_min_confidence: float = 0.8#
min_text_length: int = 3#
fasttext_batch_size: int = 50000#
lingua_batch_size: int = 10000#
prediction_col: str = 'language'#
probability_col: str = 'language_probability'#
lingua_prediction_col: str = 'language_lingua'#
lingua_confidence_col: str = 'language_lingua_confidence'#
keep_probability: bool = True#
keep_diagnostics: bool = False#
show_progress: bool = True#
verbose: bool = True#
pyevoc.preprocessing.language_filtering.download_fasttext_lid_model(compact=False, overwrite=False)[source]#

Download the official fastText language-identification model.

Parameters:

  • compact (bool) – If True, download lid.176.ftz instead of lid.176.bin.

  • overwrite (bool) – Re-download even if the file already exists.

Returns:

Local model path.

Return type:

pathlib.Path

pyevoc.preprocessing.language_filtering.prepare_for_langid(text)[source]#

Prepare text for language identification.

Parameters:

text (object)

Return type:

str

pyevoc.preprocessing.language_filtering.fasttext_predict_batch(texts, model, *, batch_size=50000, show_progress=True, verbose=True)[source]#

Predict languages with fastText in batches.

The function first attempts fastText’s list-based batch prediction. If that fails because of a fastText/NumPy compatibility issue, it falls back to row-wise prediction. Display is limited to one global progress bar.

Parameters:

  • texts (Series)

  • model (Any)

  • batch_size (int)

  • show_progress (bool)

  • verbose (bool)

Return type:

tuple[list[str | None], list[float | None]]

pyevoc.preprocessing.language_filtering.predict_language_fasttext(texts, model_path=None, *, batch_size=50000, show_progress=True, verbose=True)[source]#

Predict language labels and probabilities using fastText.

Parameters:

  • texts (Series)

  • model_path (str | Path | None)

  • batch_size (int)

  • show_progress (bool)

  • verbose (bool)

Return type:

DataFrame

pyevoc.preprocessing.language_filtering.lingua_detect_batch(texts, *, batch_size=10000, show_progress=True, verbose=True)[source]#

Apply Lingua fallback to a series of texts.

Display is limited to one global progress bar. The implementation appends exactly one language label and one confidence value per input text.

Parameters:

  • texts (Series)

  • batch_size (int)

  • show_progress (bool)

  • verbose (bool)

Return type:

tuple[list[str | None], list[float | None]]

pyevoc.preprocessing.language_filtering.filter_by_language(df, config=LanguageFilterConfig(), model_path=None)[source]#

Return rows matching the target language.

Decision rule#

A document is retained when:

  1. fastText predicts the target language with probability greater than or equal to min_probability;

or, if Lingua fallback is enabled,

  1. fastText predicts the target language with probability in the ambiguous interval and Lingua confirms the same target language with confidence greater than or equal to lingua_min_confidence.

Parameters:
Return type:

DataFrame