pyevoc.features

Foregrounding, emoji assignment, unigram selection, term indices, concreteness, and emoji labelling.

PyEvoc feature-engineering layer.

This subpackage exposes foregrounding indicators, unigram selection, emoji assignment, concreteness enrichment, emoji labelling and term-level EVOC indices. Public objects are loaded lazily to keep documentation imports light.

class pyevoc.features.ConcretenessConfig(lexicon_file='concreteness_lexicon.csv', models_dir=None, term_col='term', upos_col='upos', absent_label='-', cut_abstract=2.5, cut_concrete=3.5, eligible_upos=('NOUN', 'ADJ'), emoji_upos='EMOJI', token_column_candidates=('word', 'term', 'lemma', 'token', 'surface', 'form', 'base_term', 'base'), score_column_candidates=('conc.m', 'conc_m', 'conc.mean', 'conc_mean', 'concreteness', 'concreteness_score', 'conc', 'score', 'rating', 'mean', 'value'), verbose=True)

Bases: object

Configuration for concreteness labelling.

Parameters:

• lexicon_file (str)

• models_dir (str | Path | None)

• term_col (str)

• upos_col (str)

• absent_label (str)

• cut_abstract (float)

• cut_concrete (float)

• eligible_upos (tuple[str, ...])

• emoji_upos (str)

• token_column_candidates (tuple[str, ...])

• score_column_candidates (tuple[str, ...])

• verbose (bool)

absent_label: str = '-'

cut_abstract: float = 2.5

cut_concrete: float = 3.5

eligible_upos: tuple[str, ...] = ('NOUN', 'ADJ')

emoji_upos: str = 'EMOJI'

lexicon_file: str = 'concreteness_lexicon.csv'

models_dir: str | Path | None = None

score_column_candidates: tuple[str, ...] = ('conc.m', 'conc_m', 'conc.mean', 'conc_mean', 'concreteness', 'concreteness_score', 'conc', 'score', 'rating', 'mean', 'value')

term_col: str = 'term'

token_column_candidates: tuple[str, ...] = ('word', 'term', 'lemma', 'token', 'surface', 'form', 'base_term', 'base')

upos_col: str = 'upos'

verbose: bool = True

class pyevoc.features.EmojiAssignmentConfig(token_col='text', lemma_col='lemma', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI', show_progress=True, verbose=True, top_n=10)

Bases: object

Configuration for emoji UPOS assignment and diagnostics.

Parameters:

• token_col (str)

• lemma_col (str)

• upos_col (str)

• doc_col (str)

• user_col (str)

• emoji_upos (str)

• show_progress (bool)

• verbose (bool)

• top_n (int)

doc_col: str = 'doc_id'

emoji_upos: str = 'EMOJI'

lemma_col: str = 'lemma'

show_progress: bool = True

token_col: str = 'text'

top_n: int = 10

upos_col: str = 'upos'

user_col: str = 'user_id'

verbose: bool = True

class pyevoc.features.EmojiLabellingConfig(lookup_file='emoji_lookup.csv', models_dir=None, term_col='term', upos_col='upos', emoji_upos='EMOJI', min_similarity=0.8, verbose=True)

Bases: object

Configuration for emoji description labelling.

Parameters:

• lookup_file (str)

• models_dir (str | Path | None)

• term_col (str)

• upos_col (str)

• emoji_upos (str)

• min_similarity (float)

• verbose (bool)

emoji_upos: str = 'EMOJI'

lookup_file: str = 'emoji_lookup.csv'

min_similarity: float = 0.8

models_dir: str | Path | None = None

term_col: str = 'term'

upos_col: str = 'upos'

verbose: bool = True

class pyevoc.features.ForegroundingConfig(doc_col='doc_id', user_col='user_id', time_col='time', source_col='source', token_col='text', sentence_col='sentence_id', position_col='position', opening_sentence_value=1, emphasis_pattern=re.compile('(?:\\*\\*|__|<b>|</b>|<strong>|</strong>|_{2,}|\\*{2,})', re.IGNORECASE), list_or_quote_pattern=re.compile('^\\s*(?:-|\\*|>|•|–|—|\\d+[\\.\\)]|[a-zA-Z][\\.\\)])'), intensification_pattern=re.compile('(?:[A-Z]{3,}|!{2,}|\\?{2,}|[!?]{3,})'), weights=ForegroundingWeights(opening_sentence=0.35, emphasis=0.25, list_or_quote=0.2, intensification=0.2), keep_component_scores=True, show_progress=True)

Bases: object

Configuration for token-level foregrounding indicators.

Parameters:

• doc_col (str)

• user_col (str)

• time_col (str)

• source_col (str)

• token_col (str)

• sentence_col (str)

• position_col (str)

• opening_sentence_value (int)

• emphasis_pattern (Pattern)

• list_or_quote_pattern (Pattern)

• intensification_pattern (Pattern)

• weights (ForegroundingWeights)

• keep_component_scores (bool)

• show_progress (bool)

doc_col: str = 'doc_id'

emphasis_pattern: Pattern = re.compile('(?:\\*\\*|__|<b>|</b>|<strong>|</strong>|_{2,}|\\*{2,})', re.IGNORECASE)

intensification_pattern: Pattern = re.compile('(?:[A-Z]{3,}|!{2,}|\\?{2,}|[!?]{3,})')

keep_component_scores: bool = True

list_or_quote_pattern: Pattern = re.compile('^\\s*(?:-|\\*|>|•|–|—|\\d+[\\.\\)]|[a-zA-Z][\\.\\)])')

opening_sentence_value: int = 1

position_col: str = 'position'

sentence_col: str = 'sentence_id'

show_progress: bool = True

source_col: str = 'source'

time_col: str = 'time'

token_col: str = 'text'

user_col: str = 'user_id'

weights: ForegroundingWeights = ForegroundingWeights(opening_sentence=0.35, emphasis=0.25, list_or_quote=0.2, intensification=0.2)

class pyevoc.features.ForegroundingWeights(opening_sentence=0.35, emphasis=0.25, list_or_quote=0.2, intensification=0.2)

Bases: object

Weights used to compute structural foregrounding salience.

Parameters:

• opening_sentence (float)

• emphasis (float)

• list_or_quote (float)

• intensification (float)

as_dict()

Return the weights as a plain dict keyed by indicator name.

Return type:

dict[str, float]

emphasis: float = 0.25

intensification: float = 0.2

list_or_quote: float = 0.2

normalised()

Return a copy of the weights scaled so that they sum to 1.

Raises:

ValueError – If the weights sum to zero or a negative value.

Return type:

ForegroundingWeights

opening_sentence: float = 0.35

class pyevoc.features.SalienceConfig(alpha=0.5, max_rank_value=5.0, use_users_for_frequency=True, focal_upos=<factory>)

Bases: object

Configuration for term-level salience and diffusion indicators.

Parameters:

• alpha (float)

• max_rank_value (float)

• use_users_for_frequency (bool)

• focal_upos (set[str])

alpha: float = 0.5

max_rank_value: float = 5.0

use_users_for_frequency: bool = True

focal_upos: set[str]

class pyevoc.features.UnigramSelectionConfig(focal_upos=<factory>, token_col='text', display_col=None, lemma_col='lemma', upos_col='upos', doc_col='doc_id', user_col='user_id', term_col='term', stop_words=<factory>, min_chars=2, min_docs_per_term=3, min_users_per_term=3, exclude_url_token=True, require_latin_terms=True, remove_numeric_terms=True, remove_punct_only=True, lowercase_non_emoji=True, return_diagnostics=False, verbose=True)

Bases: object

Configuration for unigram cleaning and selection.

Parameters:

• focal_upos (set[str])

• token_col (str)

• display_col (str | None)

• lemma_col (str)

• upos_col (str)

• doc_col (str)

• user_col (str)

• term_col (str)

• stop_words (set[str])

• min_chars (int)

• min_docs_per_term (int)

• min_users_per_term (int)

• exclude_url_token (bool)

• require_latin_terms (bool)

• remove_numeric_terms (bool)

• remove_punct_only (bool)

• lowercase_non_emoji (bool)

• return_diagnostics (bool)

• verbose (bool)

display_col: str | None = None

doc_col: str = 'doc_id'

exclude_url_token: bool = True

lemma_col: str = 'lemma'

lowercase_non_emoji: bool = True

min_chars: int = 2

min_docs_per_term: int = 3

min_users_per_term: int = 3

remove_numeric_terms: bool = True

remove_punct_only: bool = True

require_latin_terms: bool = True

return_diagnostics: bool = False

term_col: str = 'term'

token_col: str = 'text'

upos_col: str = 'upos'

user_col: str = 'user_id'

verbose: bool = True

focal_upos: set[str]

stop_words: set[str]

pyevoc.features.add_concreteness_labels(term_stats_df, *, lexicon=None, config=None, models_dir=None, lexicon_file=None, absent_label=None, cut_abstract=None, cut_concrete=None)

Add concreteness labels to a term-level dataframe.

Parameters:

• term_stats_df (DataFrame)

• lexicon (DataFrame | None)

• config (ConcretenessConfig | None)

• models_dir (str | Path | None)

• lexicon_file (str | None)

• absent_label (str | None)

• cut_abstract (float | None)

• cut_concrete (float | None)

Return type:

tuple[DataFrame, DataFrame]

pyevoc.features.add_emoji_descriptions(term_stats_df, *, lookup=None, config=None, models_dir=None, emoji_lookup_file=None, min_similarity=None)

Add emoji descriptions to a term-level dataframe.

Returns:

(out, coverage_df, method_counts_df, missing_df).

Return type:

tuple

Parameters:

• term_stats_df (DataFrame)

• lookup (DataFrame | None)

• config (EmojiLabellingConfig | None)

• models_dir (str | Path | None)

• emoji_lookup_file (str | None)

• min_similarity (float | None)

pyevoc.features.add_foregrounding_indicators(tokens, *, sentence_col='sentence_id', token_col='text', opening_sentence_value=1, emphasis_pattern=EMPHASIS_RE, list_or_quote_pattern=LIST_OR_QUOTE_RE, intensification_pattern=INTENSIFICATION_RE)

Add binary foregrounding indicators to a token table.

Parameters:

• tokens (DataFrame)

• sentence_col (str)

• token_col (str)

• opening_sentence_value (int)

• emphasis_pattern (Pattern)

• list_or_quote_pattern (Pattern)

• intensification_pattern (Pattern)

Return type:

DataFrame

pyevoc.features.add_positional_salience(tokens, doc_col='doc_id', position_col='position', *, output_col='r_pos', show_progress=True)

Compute positional salience r_pos within documents.

Parameters:

• tokens (DataFrame)

• doc_col (str)

• position_col (str)

• output_col (str)

• show_progress (bool)

Return type:

DataFrame

pyevoc.features.add_salience_indicators(tokens, *, config=None)

Add foregrounding, structural salience and positional salience.

Parameters:

• tokens (DataFrame)

• config (ForegroundingConfig | None)

Return type:

DataFrame

pyevoc.features.assign_emoji_upos(tokens, token_col='text', upos_col='upos', lemma_col='lemma', *, config=None, return_diagnostics=False)

Assign UPOS='EMOJI' to emoji tokens.

Parameters:

• tokens (DataFrame) – Token-level dataframe.

• token_col (str) – Column containing token text. Ignored when config is supplied.

• upos_col (str) – Column containing UPOS tags. Ignored when config is supplied.

• lemma_col (str) – Column containing lemma values. Ignored when config is supplied.

• config (EmojiAssignmentConfig | None) – Optional EmojiAssignmentConfig.

• return_diagnostics (bool) – If True, return (tokens, diagnostics, summary). If False, return only the modified token dataframe.

Returns:

Modified token table, and optionally diagnostic and summary tables.

Return type:

pandas.DataFrame or tuple[pandas.DataFrame, pandas.DataFrame, pandas.DataFrame]

pyevoc.features.available_concreteness_lexicon_paths(lexicon_file='concreteness_lexicon.csv')

Return a diagnostic table of detected lexicon locations.

Parameters:

lexicon_file (str)

Return type:

DataFrame

pyevoc.features.available_emoji_lookup_paths(lookup_file='emoji_lookup.csv')

Return a diagnostic table of detected emoji lookup locations.

Parameters:

lookup_file (str)

Return type:

DataFrame

pyevoc.features.clean_unigram_tokens(tokens, *, config=None)

Clean and select unigram tokens.

Parameters:

• tokens (DataFrame)

• config (UnigramSelectionConfig | None)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame]

pyevoc.features.compute_structural_salience(tokens, weights=ForegroundingWeights(), *, keep_component_scores=True, show_progress=True)

Compute weighted structural salience r_str.

Parameters:

• tokens (DataFrame)

• weights (ForegroundingWeights)

• keep_component_scores (bool)

• show_progress (bool)

Return type:

DataFrame

pyevoc.features.compute_term_indices(tokens, *, term_col='term', upos_col='upos', user_col='user_id', doc_col='doc_id', r_pos_col='r_pos', r_str_col='r_str', config=SalienceConfig(), return_thresholds=False, return_quadrants=False)

Backward-compatible wrapper for term-level index computation.

Parameters:

• tokens (DataFrame)

• term_col (str)

• upos_col (str)

• user_col (str)

• doc_col (str)

• r_pos_col (str)

• r_str_col (str)

• config (SalienceConfig)

• return_thresholds (bool)

• return_quadrants (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame]

pyevoc.features.compute_term_statistics(tokens, *, term_col='term', upos_col='upos', user_col='user_id', doc_col='doc_id', time_col='time', r_pos_col='r_pos', r_str_col='r_str', term_type_col='term_type', config=SalienceConfig(), human_readable_quadrants=False, show_progress=True)

Compute term-level statistics, thresholds and quadrant summaries.

Parameters:

• tokens (DataFrame)

• term_col (str)

• upos_col (str)

• user_col (str)

• doc_col (str)

• time_col (str)

• r_pos_col (str)

• r_str_col (str)

• term_type_col (str)

• config (SalienceConfig)

• human_readable_quadrants (bool)

• show_progress (bool)

Return type:

tuple[DataFrame, DataFrame, DataFrame]

pyevoc.features.emoji_assignment_diagnostics(tokens, *, token_col='text', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI')

Return one-row diagnostics for emoji assignment.

Parameters:

• tokens (DataFrame) – Token-level dataframe after emoji assignment.

• token_col (str) – Column containing token text.

• upos_col (str) – Column containing UPOS values.

• doc_col (str) – Column containing document identifiers.

• user_col (str) – Column containing user identifiers.

• emoji_upos (str) – UPOS value assigned to emoji tokens.

Returns:

One-row diagnostic table.

Return type:

pandas.DataFrame

pyevoc.features.emoji_summary(tokens, *, token_col='text', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI', top_n=None, only_assigned=True, show_progress=False)

Return a summary table of emoji usage.

Parameters:

• tokens (DataFrame) – Token-level dataframe.

• token_col (str) – Column containing the original token text.

• upos_col (str) – Column containing the UPOS tag.

• doc_col (str) – Column containing document identifiers. If absent, document counts are omitted.

• user_col (str) – Column containing user identifiers. If absent, user counts are omitted.

• emoji_upos (str) – UPOS value used for emoji tokens.

• top_n (int | None) – If provided, return only the top n emojis by frequency.

• only_assigned (bool) – If True, emojis are selected using upos_col == emoji_upos. If False, emoji detection is recomputed from token_col.

• show_progress (bool) – Whether to display a progress bar if emoji detection is recomputed.

Returns:

Emoji summary table with frequency and, when available, document and user coverage.

Return type:

pandas.DataFrame

pyevoc.features.foregrounding_diagnostics(tokens, *, doc_col='doc_id', user_col='user_id', time_col='time')

Return diagnostics for foregrounding and salience indicators.

Parameters:

• tokens (DataFrame)

• doc_col (str)

• user_col (str)

• time_col (str)

Return type:

DataFrame

pyevoc.features.label_concreteness(terms, *, lexicon=None, term_col='term', upos_col='upos', label_col='concreteness_label', score_col='concreteness_score', in_lexicon_col='concreteness_in_lexicon', absent_label='-', cut_abstract=2.5, cut_concrete=3.5, return_coverage=False)

Backward-compatible wrapper for concreteness labelling.

Parameters:

• terms (DataFrame)

• lexicon (DataFrame | None)

• term_col (str)

• upos_col (str)

• label_col (str)

• score_col (str)

• in_lexicon_col (str)

• absent_label (str)

• cut_abstract (float)

• cut_concrete (float)

• return_coverage (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame]

pyevoc.features.label_emojis(terms, *, lookup=None, term_col='term', upos_col='upos', description_col='emoji_description', return_diagnostics=False)

Backward-compatible wrapper for emoji labelling.

Parameters:

• terms (DataFrame)

• lookup (DataFrame | None)

• term_col (str)

• upos_col (str)

• description_col (str)

• return_diagnostics (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame, DataFrame]

pyevoc.features.load_concreteness_lexicon(path=None, *, models_dir=None, lexicon_file='concreteness_lexicon.csv', encoding='utf-8')

Load a concreteness lexicon.

Parameters:

• path (str | Path | None) – Explicit lexicon path. If omitted, PyEvoc searches package resources, local model folders, and the user model directory.

• models_dir (str | Path | None) – Optional model directory.

• lexicon_file (str) – Lexicon filename used when path is omitted.

• encoding (str) – File encoding.

Returns:

Raw concreteness lexicon.

Return type:

pandas.DataFrame

pyevoc.features.load_emoji_lookup(path=None, *, models_dir=None, lookup_file='emoji_lookup.csv', encoding='utf-8')

Load an emoji lookup table.

Parameters:

• path (str | Path | None)

• models_dir (str | Path | None)

• lookup_file (str)

• encoding (str)

Return type:

DataFrame

pyevoc.features.metadata_coverage(tokens, *, required_metadata=None)

Return a diagnostic table describing metadata availability.

Parameters:

• tokens (DataFrame)

• required_metadata (list[str] | None)

Return type:

DataFrame

pyevoc.features.resolve_concreteness_lexicon_path(lexicon_file='concreteness_lexicon.csv', *, models_dir=None)

Resolve the concreteness lexicon path.

Resolution order

1. lexicon_file if it already points to an existing file;

2. models_dir / lexicon_file if models_dir is supplied;

3. package-bundled resources and package-adjacent model folders;

4. user model path, usually ~/.pyevoc/models/lexicon_file.

Parameters:

• lexicon_file (str | Path)

• models_dir (str | Path | None)

Return type:

Path

pyevoc.features.select_unigrams(tokens, *, lemma_col='lemma', upos_col='upos', keep_upos=None, min_chars=2, lowercase=True, token_col='text', doc_col='doc_id', user_col='user_id', min_docs_per_term=3, min_users_per_term=3, return_diagnostics=False)

Backward-compatible wrapper for unigram selection.

Parameters:

• tokens (DataFrame)

• lemma_col (str)

• upos_col (str)

• keep_upos (set[str] | None)

• min_chars (int)

• lowercase (bool)

• token_col (str)

• doc_col (str)

• user_col (str)

• min_docs_per_term (int)

• min_users_per_term (int)

• return_diagnostics (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame, DataFrame]

pyevoc.features.term_statistics_summary(term_stats, *, upos_col='upos', quadrant_col='quadrant')

Return a compact summary of term-level statistics by UPOS and quadrant.

Parameters:

• term_stats (DataFrame)

• upos_col (str)

• quadrant_col (str)

Return type:

DataFrame

Modules

• pyevoc.features.concreteness
  • Lexicon lookup
  • ConcretenessConfig
  • resolve_concreteness_lexicon_path()
  • available_concreteness_lexicon_paths()
  • load_concreteness_lexicon()
  • infer_lexicon_columns()
  • prepare_concreteness_map()
  • add_concreteness_labels()
  • label_concreteness()
• pyevoc.features.emoji_assignment
  • Main functions
  • EmojiAssignmentConfig
  • is_emoji_token()
  • emoji_summary()
  • emoji_assignment_diagnostics()
  • assign_emoji_upos()
• pyevoc.features.emoji_labelling
  • Matching strategy
  • EmojiLabellingConfig
  • resolve_emoji_lookup_path()
  • available_emoji_lookup_paths()
  • load_emoji_lookup()
  • emoji_codepoints_hex()
  • normalise_emoji_glyph()
  • codepoint_tokens()
  • codepoint_jaccard()
  • prepare_emoji_lookup()
  • add_emoji_descriptions()
  • label_emojis()
• pyevoc.features.foregrounding
  • ForegroundingWeights
  • ForegroundingConfig
  • validate_token_metadata()
  • add_token_position()
  • add_foregrounding_indicators()
  • compute_structural_salience()
  • add_positional_salience()
  • add_salience_indicators()
  • foregrounding_diagnostics()
  • metadata_coverage()
  • assert_metadata_preserved()
• pyevoc.features.term_indices
  • SalienceConfig
  • compute_term_statistics()
  • compute_term_indices()
  • term_statistics_summary()
• pyevoc.features.unigram_selection
  • UnigramSelectionConfig
  • is_punct_only()
  • is_latin_term()
  • is_valid_emoji()
  • build_unigram_terms()
  • unigram_selection_diagnostics()
  • clean_unigram_tokens()
  • select_unigrams()