pyevoc.features.emoji_assignment#

Emoji detection, token-level emoji assignment, and diagnostics.

This module provides utilities for identifying emoji tokens in an annotated token-level table and assigning them to the dedicated EMOJI UPOS category.

The module is intentionally conservative: a token is treated as an emoji if at least one Unicode character or grapheme cluster contained in the token is found in the emoji package metadata.

Main functions#

is_emoji_token

Detect whether a token contains emoji content.

assign_emoji_upos

Relabel emoji tokens as UPOS = “EMOJI” and set their lemma equal to the original emoji token.

emoji_summary

Produce a frequency/coverage table for emoji tokens.

emoji_assignment_diagnostics

Produce a compact one-row diagnostic table for the emoji assignment step.

class pyevoc.features.emoji_assignment.EmojiAssignmentConfig(token_col='text', lemma_col='lemma', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI', show_progress=True, verbose=True, top_n=10)[source]#

Bases: object

Configuration for emoji UPOS assignment and diagnostics.

Parameters:
token_col: str = 'text'#
lemma_col: str = 'lemma'#
upos_col: str = 'upos'#
doc_col: str = 'doc_id'#
user_col: str = 'user_id'#
emoji_upos: str = 'EMOJI'#
show_progress: bool = True#
verbose: bool = True#
top_n: int = 10#
pyevoc.features.emoji_assignment.is_emoji_token(value)[source]#

Return True if a token contains emoji content.

Parameters:

value (object) – Token value.

Returns:

True if the token contains at least one emoji recognised by the emoji package.

Return type:

bool

pyevoc.features.emoji_assignment.emoji_summary(tokens, *, token_col='text', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI', top_n=None, only_assigned=True, show_progress=False)[source]#

Return a summary table of emoji usage.

Parameters:

  • tokens (DataFrame) – Token-level dataframe.

  • token_col (str) – Column containing the original token text.

  • upos_col (str) – Column containing the UPOS tag.

  • doc_col (str) – Column containing document identifiers. If absent, document counts are omitted.

  • user_col (str) – Column containing user identifiers. If absent, user counts are omitted.

  • emoji_upos (str) – UPOS value used for emoji tokens.

  • top_n (int | None) – If provided, return only the top n emojis by frequency.

  • only_assigned (bool) – If True, emojis are selected using upos_col == emoji_upos. If False, emoji detection is recomputed from token_col.

  • show_progress (bool) – Whether to display a progress bar if emoji detection is recomputed.

Returns:

Emoji summary table with frequency and, when available, document and user coverage.

Return type:

pandas.DataFrame

pyevoc.features.emoji_assignment.emoji_assignment_diagnostics(tokens, *, token_col='text', upos_col='upos', doc_col='doc_id', user_col='user_id', emoji_upos='EMOJI')[source]#

Return one-row diagnostics for emoji assignment.

Parameters:

  • tokens (DataFrame) – Token-level dataframe after emoji assignment.

  • token_col (str) – Column containing token text.

  • upos_col (str) – Column containing UPOS values.

  • doc_col (str) – Column containing document identifiers.

  • user_col (str) – Column containing user identifiers.

  • emoji_upos (str) – UPOS value assigned to emoji tokens.

Returns:

One-row diagnostic table.

Return type:

pandas.DataFrame

pyevoc.features.emoji_assignment.assign_emoji_upos(tokens, token_col='text', upos_col='upos', lemma_col='lemma', *, config=None, return_diagnostics=False)[source]#

Assign UPOS='EMOJI' to emoji tokens.

Parameters:

  • tokens (DataFrame) – Token-level dataframe.

  • token_col (str) – Column containing token text. Ignored when config is supplied.

  • upos_col (str) – Column containing UPOS tags. Ignored when config is supplied.

  • lemma_col (str) – Column containing lemma values. Ignored when config is supplied.

  • config (EmojiAssignmentConfig | None) – Optional EmojiAssignmentConfig.

  • return_diagnostics (bool) – If True, return (tokens, diagnostics, summary). If False, return only the modified token dataframe.

Returns:

Modified token table, and optionally diagnostic and summary tables.

Return type:

pandas.DataFrame or tuple[pandas.DataFrame, pandas.DataFrame, pandas.DataFrame]