pyevoc.analysis.collocations_entities#

Dependency collocations and PROPN named-entity n-grams.

This module implements the PyEvoc collocation and named-entity extraction workflow used after token-level annotation.

It provides one integrated interface that can extract:

  • dependency-based lexical collocations;

  • proper-noun named-entity n-grams;

  • both tables with optional overlap adjudication;

  • compact HTML reports.

The implementation follows the original PyEvoc notebook logic while adapting it to a package-ready API.

class pyevoc.analysis.collocations_entities.CollocationEntityConfig(output_dir='evoc_outputs', entity_n=2, min_freq=3, min_docs=3, min_users=3, g2_alpha=0.001, caps_thr=0.6, min_caps_obs=20, include_collocations=True, include_entities=True, resolve_overlap=True, prefer_overlap='evidence', write_html=True, html_max_colloc_rows=60, html_max_entity_rows=60, html_max_overlap_rows=50, doc_col='doc_id', user_col='user_id', sentence_col='sentence_id', token_id_col='token_id', token_col='token', lemma_col='lemma', upos_col='upos', surface_col='token', head_col='head_token_id', dep_rel_col='dep_rel', colloc_relations=<factory>, false_entity_terms=<factory>, stop_words=<factory>, verbose=True)[source]#

Bases: object

Configuration for collocation and named-entity extraction.

Parameters:

  • output_dir (str | Path)

  • entity_n (int)

  • min_freq (int)

  • min_docs (int)

  • min_users (int)

  • g2_alpha (float)

  • caps_thr (float)

  • min_caps_obs (int)

  • include_collocations (bool)

  • include_entities (bool)

  • resolve_overlap (bool)

  • prefer_overlap (str)

  • write_html (bool)

  • html_max_colloc_rows (int)

  • html_max_entity_rows (int)

  • html_max_overlap_rows (int)

  • doc_col (str)

  • user_col (str)

  • sentence_col (str)

  • token_id_col (str)

  • token_col (str)

  • lemma_col (str)

  • upos_col (str)

  • surface_col (str)

  • head_col (str)

  • dep_rel_col (str)

  • colloc_relations (set[str])

  • false_entity_terms (set[str])

  • stop_words (set[str])

  • verbose (bool)

output_dir: str | Path = 'evoc_outputs'#
entity_n: int = 2#
min_freq: int = 3#
min_docs: int = 3#
min_users: int = 3#
g2_alpha: float = 0.001#
caps_thr: float = 0.6#
min_caps_obs: int = 20#
include_collocations: bool = True#
include_entities: bool = True#
resolve_overlap: bool = True#
prefer_overlap: str = 'evidence'#
write_html: bool = True#
html_max_colloc_rows: int = 60#
html_max_entity_rows: int = 60#
html_max_overlap_rows: int = 50#
doc_col: str = 'doc_id'#
user_col: str = 'user_id'#
sentence_col: str = 'sentence_id'#
token_id_col: str = 'token_id'#
token_col: str = 'token'#
lemma_col: str = 'lemma'#
upos_col: str = 'upos'#
surface_col: str = 'token'#
head_col: str = 'head_token_id'#
dep_rel_col: str = 'dep_rel'#
colloc_relations: set[str]#
false_entity_terms: set[str]#
stop_words: set[str]#
verbose: bool = True#
pyevoc.analysis.collocations_entities.normalise_annotation_columns(tokens, *, config=None)[source]#

Normalise token-table column names for compatibility.

The canonical annotation schema uses doc_id and token. This function also accepts older aliases such as text and creates the internal post_id alias required by this module.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.is_valid_term(value, stop_words=None)[source]#

Return True if a lexical item is eligible for collocation/NER extraction.

Parameters:
Return type:

bool

pyevoc.analysis.collocations_entities.prepare_tokens(tokens, *, config=None)[source]#

Prepare and validate a token-level annotation table.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.score_phrase_table(phrase_df, component_cols, *, term_col='term', min_freq=3, min_docs=3, min_users=3, alpha=0.001, source_label='candidate')[source]#

Score phrase candidates using the log-likelihood G² statistic.

Parameters:

  • phrase_df (DataFrame)

  • component_cols (list[str])

  • term_col (str)

  • min_freq (int)

  • min_docs (int)

  • min_users (int)

  • alpha (float)

  • source_label (str)

Return type:

DataFrame

pyevoc.analysis.collocations_entities.extract_dependency_collocations(tokens_df, *, config=None, min_freq=None, min_docs=None, min_users=None, alpha=None)[source]#

Extract dependency-based collocations.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.extract_named_entity_ngrams(tokens_df, *, config=None, entity_n=None, min_freq=None, min_docs=None, min_users=None, alpha=None)[source]#

Extract contiguous PROPN named-entity n-grams.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.compute_caps_evidence(tokens_df, terms, *, config=None)[source]#

Compute capitalisation evidence for overlapping collocation/NE terms.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.resolve_ne_colloc_overlap(collocations_df, entities_df, tokens_df, *, config=None, prefer=None, caps_thr=None, min_caps_obs=None)[source]#

Resolve overlaps between collocations and named entities.

Parameters:

  • collocations_df (DataFrame)

  • entities_df (DataFrame)

  • tokens_df (DataFrame)

  • config (CollocationEntityConfig | None)

  • prefer (str | None)

  • caps_thr (float | None)

  • min_caps_obs (int | None)

Return type:

tuple[DataFrame, DataFrame, DataFrame]

pyevoc.analysis.collocations_entities.write_html_table(df, title, output_file, max_rows)[source]#

Write a compact HTML table and return the output path.

Parameters:

  • df (DataFrame)

  • title (str)

  • output_file (str | Path)

  • max_rows (int)

Return type:

str

pyevoc.analysis.collocations_entities.extract_collocations_and_entities(tokens_df, *, config=None, output_dir=None, include_collocations=None, include_entities=None, entity_n=None, min_freq=None, min_docs=None, min_users=None, g2_alpha=None, resolve_overlap=None, prefer_overlap=None, write_html=None)[source]#

Extract collocations and/or named entities through one wrapper.

This is the recommended public API. The user can run only collocations, only named entities, or both.

Parameters:

  • tokens_df (DataFrame)

  • config (CollocationEntityConfig | None)

  • output_dir (str | Path | None)

  • include_collocations (bool | None)

  • include_entities (bool | None)

  • entity_n (int | None)

  • min_freq (int | None)

  • min_docs (int | None)

  • min_users (int | None)

  • g2_alpha (float | None)

  • resolve_overlap (bool | None)

  • prefer_overlap (str | None)

  • write_html (bool | None)

Return type:

dict[str, object]

pyevoc.analysis.collocations_entities.extract_collocations(tokens, *, config=None, **kwargs)[source]#

Backward-compatible wrapper returning only dependency collocations.

Parameters:
Return type:

DataFrame

pyevoc.analysis.collocations_entities.aggregate_named_entities(entities, text_col='text', type_col='type')[source]#

Backward-compatible utility for aggregating pre-extracted NER tables.

Parameters:

  • entities (DataFrame)

  • text_col (str)

  • type_col (str)

Return type:

DataFrame