pyevoc.preprocessing.annotation#

Stanza UD annotation utilities for PyEvoc.

This module converts a document-level dataframe into a token-level Universal Dependencies table. The output intentionally follows the original notebook schema used by the EVOC pipeline, because dependency-based collocation extraction requires the original column names token, head_token_id and dep_rel.

Canonical output columns#

The core token-level columns are:

  • doc_id

  • paragraph_id

  • sentence_id

  • token_id

  • position

  • token

  • lemma

  • upos

  • xpos

  • feats

  • head_token_id

  • dep_rel

If present in the input dataframe, user_id and time are copied to every token row. No duplicate aliases such as text, head or deprel are created here. If other modules still expect those aliases, they should be updated to the canonical names above.

class pyevoc.preprocessing.annotation.AnnotationConfig(text_col='text_clean', doc_id_col='post_id', user_col='user_id', time_col='time', language='en', processors='tokenize,pos,lemma,depparse', tokenize_no_ssplit=False, use_gpu=False, batch_size=128, show_progress=True, save_parquet=False, output_dir='annotated_outputs', output_name=None, save_emoji_sample=True)[source]#

Bases: object

Configuration for Stanza-based UD annotation.

Parameters:

  • text_col (str)

  • doc_id_col (str)

  • user_col (str)

  • time_col (str)

  • language (str)

  • processors (str)

  • tokenize_no_ssplit (bool)

  • use_gpu (bool)

  • batch_size (int)

  • show_progress (bool)

  • save_parquet (bool)

  • output_dir (str | Path)

  • output_name (str | None)

  • save_emoji_sample (bool)

text_col: str = 'text_clean'#
doc_id_col: str = 'post_id'#
user_col: str = 'user_id'#
time_col: str = 'time'#
language: str = 'en'#
processors: str = 'tokenize,pos,lemma,depparse'#
tokenize_no_ssplit: bool = False#
use_gpu: bool = False#
batch_size: int = 128#
show_progress: bool = True#
save_parquet: bool = False#
output_dir: str | Path = 'annotated_outputs'#
output_name: str | None = None#
save_emoji_sample: bool = True#
pyevoc.preprocessing.annotation.annotate_with_stanza(df, config=AnnotationConfig())[source]#

Annotate a document-level dataframe with Stanza.

The returned dataframe is the correct input for dependency-based collocation extraction. In particular, it contains token, head_token_id and dep_rel.

Parameters:
Return type:

DataFrame

pyevoc.preprocessing.annotation.annotate_dataframe(df, df_name='df', *, doc_id_col='post_id', text_col='text_clean', user_col='user_id', time_col='time', batch_size=128, save_parquet=True, output_dir='annotated_outputs', show_progress=True, use_gpu=False)[source]#

Notebook-compatible wrapper around annotate_with_stanza().

This mirrors the original notebook call style:

anno_df = annotate_dataframe(df, "df")

and saves <df_name>_stanza_anno.parquet when save_parquet=True.

Parameters:

  • df (DataFrame)

  • df_name (str)

  • doc_id_col (str)

  • text_col (str)

  • user_col (str)

  • time_col (str)

  • batch_size (int)

  • save_parquet (bool)

  • output_dir (str | Path)

  • show_progress (bool)

  • use_gpu (bool)

Return type:

DataFrame

pyevoc.preprocessing.annotation.annotation_diagnostics(anno_df, *, doc_col='doc_id', user_col='user_id', time_col='time')[source]#

Return a compact diagnostic table for an annotated token dataframe.

Parameters:

  • anno_df (DataFrame)

  • doc_col (str)

  • user_col (str)

  • time_col (str)

Return type:

DataFrame

pyevoc.preprocessing.annotation.dependency_diagnostics(anno_df, *, dep_rel_col='dep_rel', head_col='head_token_id')[source]#

Return diagnostics for dependency columns used by collocations.

Parameters:

  • anno_df (DataFrame)

  • dep_rel_col (str)

  • head_col (str)

Return type:

DataFrame

pyevoc.preprocessing.annotation.contains_emoji_grapheme(value)[source]#

Return True if a string contains at least one emoji grapheme.

Parameters:

value (object)

Return type:

bool

pyevoc.preprocessing.annotation.count_emoji_documents(df, *, text_col='text_clean')[source]#

Count documents containing at least one emoji grapheme.

Parameters:

  • df (DataFrame)

  • text_col (str)

Return type:

int

pyevoc.preprocessing.annotation.count_emoji_tokens(anno_df, *, token_col='token')[source]#

Count annotated tokens containing at least one emoji grapheme.

Parameters:

  • anno_df (DataFrame)

  • token_col (str)

Return type:

int

pyevoc.preprocessing.annotation.emoji_token_sample(anno_df, *, n=30, token_col='token')[source]#

Return a sample of emoji-bearing annotated tokens.

Parameters:

  • anno_df (DataFrame)

  • n (int)

  • token_col (str)

Return type:

DataFrame

pyevoc.preprocessing.annotation.validate_annotation_input(df, config=AnnotationConfig())[source]#

Validate the input dataframe before annotation.

Parameters:
Return type:

None