pyevoc.features.concreteness

Concreteness labelling for PyEvoc term-level tables.

This module enriches term-level EVOC outputs with lexical concreteness information. It is designed to operate after term-level statistics have been computed.

The expected input is a dataframe containing at least:

• term

• upos

The module adds:

• concreteness_score

• concreteness_label

• concreteness_in_lexicon

and returns a coverage diagnostic table.

Lexicon lookup

The concreteness lexicon is searched in the following order:

1. explicit path argument;

2. explicit lexicon_file if it is already a valid path;

3. models_dir / lexicon_file when models_dir is supplied;

4. package-bundled resource locations: - pyevoc.models - pyevoc.resources - pyevoc.data

5. local package-adjacent folders: - pyevoc/models - models - mdl

6. user model directory, usually ~/.pyevoc/models.

Emoji terms are explicitly retained but excluded from lexical concreteness labelling, because emojis are not standard lexical items in concreteness norms.

class pyevoc.features.concreteness.ConcretenessConfig(lexicon_file='concreteness_lexicon.csv', models_dir=None, term_col='term', upos_col='upos', absent_label='-', cut_abstract=2.5, cut_concrete=3.5, eligible_upos=('NOUN', 'ADJ'), emoji_upos='EMOJI', token_column_candidates=('word', 'term', 'lemma', 'token', 'surface', 'form', 'base_term', 'base'), score_column_candidates=('conc.m', 'conc_m', 'conc.mean', 'conc_mean', 'concreteness', 'concreteness_score', 'conc', 'score', 'rating', 'mean', 'value'), verbose=True)

Bases: object

Configuration for concreteness labelling.

Parameters:

• lexicon_file (str)

• models_dir (str | Path | None)

• term_col (str)

• upos_col (str)

• absent_label (str)

• cut_abstract (float)

• cut_concrete (float)

• eligible_upos (tuple[str, ...])

• emoji_upos (str)

• token_column_candidates (tuple[str, ...])

• score_column_candidates (tuple[str, ...])

• verbose (bool)

lexicon_file: str = 'concreteness_lexicon.csv'

models_dir: str | Path | None = None

term_col: str = 'term'

upos_col: str = 'upos'

absent_label: str = '-'

cut_abstract: float = 2.5

cut_concrete: float = 3.5

eligible_upos: tuple[str, ...] = ('NOUN', 'ADJ')

emoji_upos: str = 'EMOJI'

token_column_candidates: tuple[str, ...] = ('word', 'term', 'lemma', 'token', 'surface', 'form', 'base_term', 'base')

score_column_candidates: tuple[str, ...] = ('conc.m', 'conc_m', 'conc.mean', 'conc_mean', 'concreteness', 'concreteness_score', 'conc', 'score', 'rating', 'mean', 'value')

verbose: bool = True

pyevoc.features.concreteness.resolve_concreteness_lexicon_path(lexicon_file='concreteness_lexicon.csv', *, models_dir=None)

Resolve the concreteness lexicon path.

Resolution order

1. lexicon_file if it already points to an existing file;

2. models_dir / lexicon_file if models_dir is supplied;

3. package-bundled resources and package-adjacent model folders;

4. user model path, usually ~/.pyevoc/models/lexicon_file.

Parameters:

• lexicon_file (str | Path)

• models_dir (str | Path | None)

Return type:

Path

pyevoc.features.concreteness.available_concreteness_lexicon_paths(lexicon_file='concreteness_lexicon.csv')

Return a diagnostic table of detected lexicon locations.

Parameters:

lexicon_file (str)

Return type:

DataFrame

pyevoc.features.concreteness.load_concreteness_lexicon(path=None, *, models_dir=None, lexicon_file='concreteness_lexicon.csv', encoding='utf-8')

Load a concreteness lexicon.

Parameters:

• path (str | Path | None) – Explicit lexicon path. If omitted, PyEvoc searches package resources, local model folders, and the user model directory.

• models_dir (str | Path | None) – Optional model directory.

• lexicon_file (str) – Lexicon filename used when path is omitted.

• encoding (str) – File encoding.

Returns:

Raw concreteness lexicon.

Return type:

pandas.DataFrame

pyevoc.features.concreteness.infer_lexicon_columns(lexicon, *, token_candidates=TOKEN_COLUMN_CANDIDATES, score_candidates=SCORE_COLUMN_CANDIDATES)

Infer lexical-item and score columns from a concreteness lexicon.

Parameters:

• lexicon (DataFrame)

• token_candidates (tuple[str, ...])

• score_candidates (tuple[str, ...])

Return type:

tuple[str, str]

pyevoc.features.concreteness.prepare_concreteness_map(lexicon, *, config=None)

Prepare a cleaned concreteness lookup table.

Parameters:

• lexicon (DataFrame)

• config (ConcretenessConfig | None)

Return type:

tuple[DataFrame, str, str]

pyevoc.features.concreteness.add_concreteness_labels(term_stats_df, *, lexicon=None, config=None, models_dir=None, lexicon_file=None, absent_label=None, cut_abstract=None, cut_concrete=None)

Add concreteness labels to a term-level dataframe.

Parameters:

• term_stats_df (DataFrame)

• lexicon (DataFrame | None)

• config (ConcretenessConfig | None)

• models_dir (str | Path | None)

• lexicon_file (str | None)

• absent_label (str | None)

• cut_abstract (float | None)

• cut_concrete (float | None)

Return type:

tuple[DataFrame, DataFrame]

pyevoc.features.concreteness.label_concreteness(terms, *, lexicon=None, term_col='term', upos_col='upos', label_col='concreteness_label', score_col='concreteness_score', in_lexicon_col='concreteness_in_lexicon', absent_label='-', cut_abstract=2.5, cut_concrete=3.5, return_coverage=False)

Backward-compatible wrapper for concreteness labelling.

Parameters:

• terms (DataFrame)

• lexicon (DataFrame | None)

• term_col (str)

• upos_col (str)

• label_col (str)

• score_col (str)

• in_lexicon_col (str)

• absent_label (str)

• cut_abstract (float)

• cut_concrete (float)

• return_coverage (bool)

Return type:

DataFrame | tuple[DataFrame, DataFrame]