pyevoc.preprocessing.thematic_filtering

Generic thematic corpus filtering using anchor terms and distributional expansion.

This module implements the notebook-style thematic-subsetting procedure used in PyEvoc:

1. normalise an external anchor inventory;

2. identify direct anchor matches with a compiled regular expression;

3. build a document-term matrix with CountVectorizer;

4. identify vectorizer terms containing at least one anchor string;

5. compute cosine similarity between all terms and anchor-related terms;

6. select the most similar expansion terms;

7. retain documents with either direct anchor hits or enough expansion hits.

The implementation is deliberately generic. It does not assume a renewable-energy domain and therefore uses neutral column names such as theme_subset_flag and filter_type.

class pyevoc.preprocessing.thematic_filtering.ThematicFilterConfig(text_col='text', min_anchor_hits=1, similarity_threshold=0.1, min_expansion_hits=2, top_n_expansion_terms=150, min_df=10, max_df=0.5, max_features=50000, stop_words='english', lowercase=True, ngram_range=(1, 2), anchor_col='anchor_hit', anchor_hits_col='anchor_hits', expansion_hits_col='expansion_hits', expansion_col='expansion_hit', subset_flag_col='theme_subset_flag', filter_type_col='filter_type', retain_columns=None, verbose=True, show_progress=True)

Bases: object

Configuration for anchor-based thematic filtering.

Parameters:

• text_col (str)

• min_anchor_hits (int)

• similarity_threshold (float)

• min_expansion_hits (int)

• top_n_expansion_terms (int)

• min_df (int | float)

• max_df (int | float)

• max_features (int | None)

• stop_words (str | list[str] | None)

• lowercase (bool)

• ngram_range (tuple[int, int])

• anchor_col (str)

• anchor_hits_col (str)

• expansion_hits_col (str)

• expansion_col (str)

• subset_flag_col (str)

• filter_type_col (str)

• retain_columns (tuple[str, ...] | None)

• verbose (bool)

• show_progress (bool)

text_col: str = 'text'

min_anchor_hits: int = 1

similarity_threshold: float = 0.1

min_expansion_hits: int = 2

top_n_expansion_terms: int = 150

min_df: int | float = 10

max_df: int | float = 0.5

max_features: int | None = 50000

stop_words: str | list[str] | None = 'english'

lowercase: bool = True

ngram_range: tuple[int, int] = (1, 2)

anchor_col: str = 'anchor_hit'

anchor_hits_col: str = 'anchor_hits'

expansion_hits_col: str = 'expansion_hits'

expansion_col: str = 'expansion_hit'

subset_flag_col: str = 'theme_subset_flag'

filter_type_col: str = 'filter_type'

retain_columns: tuple[str, ...] | None = None

verbose: bool = True

show_progress: bool = True

pyevoc.preprocessing.thematic_filtering.normalise_anchor_terms(anchors)

Lowercase, strip, deduplicate and length-sort anchor terms.

Parameters:

anchors (Iterable[object])

Return type:

list[str]

pyevoc.preprocessing.thematic_filtering.load_anchor_terms(path, column=None)

Load anchor terms from TXT, CSV/TSV, or JSON.

TXT files should contain one term per line. CSV/TSV files use column if provided; otherwise the first column is used. JSON files may contain a list or a dictionary with an anchors key.

Parameters:

• path (str | Path)

• column (str | None)

Return type:

list[str]

pyevoc.preprocessing.thematic_filtering.compile_anchor_pattern(anchors)

Compile the notebook-style anchor regex pattern.

Parameters:

anchors (list[str])

Return type:

Pattern

pyevoc.preprocessing.thematic_filtering.direct_anchor_filter(df, anchors, config)

Add anchor hit indicators to a dataframe.

Parameters:

• df (DataFrame)

• anchors (list[str])

• config (ThematicFilterConfig)

Return type:

DataFrame

pyevoc.preprocessing.thematic_filtering.build_count_vectorizer(config)

Construct the CountVectorizer used for expansion-term discovery.

Parameters:

config (ThematicFilterConfig)

Return type:

CountVectorizer

pyevoc.preprocessing.thematic_filtering.derive_expansion_terms(df, anchors, config)

Derive expansion terms using notebook-style term-term cosine similarity.

The function returns expansion_terms, a diagnostic dataframe, the fitted vectorizer, and the document-term matrix.

Parameters:

• df (DataFrame)

• anchors (list[str])

• config (ThematicFilterConfig)

Return type:

tuple[set[str], DataFrame, CountVectorizer, object]

pyevoc.preprocessing.thematic_filtering.add_expansion_hits(df, expansion_terms, vectorizer, X, config)

Add expansion hit counts and boolean expansion-hit flags.

Parameters:

• df (DataFrame)

• expansion_terms (set[str])

• vectorizer (CountVectorizer)

• X (Any)

• config (ThematicFilterConfig)

Return type:

DataFrame

pyevoc.preprocessing.thematic_filtering.build_thematic_subset(df, anchor_file=None, config=ThematicFilterConfig(), expansion_terms=None, anchors=None)

Build a thematic subcorpus using anchor and expansion filters.

Parameters:

• df (DataFrame) – Input document-level dataframe.

• anchor_file (str | Path | None) – Optional external anchor file. Required when anchors is not given.

• config (ThematicFilterConfig) – Filtering configuration.

• expansion_terms (set[str] | list[str] | None) – Optional pre-computed expansion terms. If not supplied, expansion terms are derived from the input corpus.

• anchors (list[str] | None) – Optional in-memory anchor inventory. If supplied, anchor_file is not required.

Return type:

tuple[DataFrame, dict]