pyevoc.preprocessing.cleaning

Basic text cleaning for social-media corpora.

The module performs conservative, emoji-safe normalisation of raw textual data. It is designed for social-media corpora where URLs, contractions, emojis, hashtags, mentions, punctuation, and non-standard orthography may coexist.

The default behaviour follows the original PyEvoc analytical pipeline:

• decode HTML entities;

• normalise apostrophes;

• expand English contractions;

• replace URLs with a stable placeholder;

• preserve emojis as independent grapheme-level units;

• reduce exaggerated vowel repetitions;

• remove control characters;

• space punctuation without removing Unicode symbols;

• normalise whitespace.

class pyevoc.preprocessing.cleaning.CleaningConfig(lowercase=False, decode_html=True, replace_ampersand=True, normalise_apostrophes=True, expand_contractions=True, remove_urls=True, url_placeholder='URL', space_emojis=True, reduce_elongated_vowels=True, remove_control_chars=True, space_punctuation=True, normalise_whitespace=True, show_progress=True)

Bases: object

Configuration for basic text cleaning.

Parameters:

• lowercase (bool)

• decode_html (bool)

• replace_ampersand (bool)

• normalise_apostrophes (bool)

• expand_contractions (bool)

• remove_urls (bool)

• url_placeholder (str)

• space_emojis (bool)

• reduce_elongated_vowels (bool)

• remove_control_chars (bool)

• space_punctuation (bool)

• normalise_whitespace (bool)

• show_progress (bool)

lowercase: bool = False

decode_html: bool = True

replace_ampersand: bool = True

normalise_apostrophes: bool = True

expand_contractions: bool = True

remove_urls: bool = True

url_placeholder: str = 'URL'

space_emojis: bool = True

reduce_elongated_vowels: bool = True

remove_control_chars: bool = True

space_punctuation: bool = True

normalise_whitespace: bool = True

show_progress: bool = True

pyevoc.preprocessing.cleaning.is_emoji_grapheme(value)

Return True if a Unicode grapheme cluster is an emoji.

The function is robust to multi-codepoint emojis and modifiers.

Parameters:

value (object)

Return type:

bool

pyevoc.preprocessing.cleaning.separate_emojis(text)

Add spaces around emoji grapheme clusters without splitting them.

Examples

‘great😂’ -> ‘great 😂’

‘ok👍🏽’ -> ‘ok 👍🏽’

Parameters:

text (object)

Return type:

str

pyevoc.preprocessing.cleaning.clean_text(text, *, config=None, lowercase=None, remove_urls=None)

Clean a single text.

Parameters:

• text (object) – Raw textual input.

• config (CleaningConfig | None) – Optional cleaning configuration.

• lowercase (bool | None) – Optional override for config.lowercase.

• remove_urls (bool | None) – Optional override for config.remove_urls.

Returns:

Cleaned text.

Return type:

str

pyevoc.preprocessing.cleaning.clean_corpus(df, text_col='text', output_col='clean_text', *, config=None, lowercase=None, remove_urls=None)

Apply basic cleaning to a dataframe.

Parameters:

• df (DataFrame) – Input dataframe.

• text_col (str) – Name of the source text column.

• output_col (str) – Name of the output cleaned-text column.

• config (CleaningConfig | None) – Optional cleaning configuration.

• lowercase (bool | None) – Optional override for config.lowercase.

• remove_urls (bool | None) – Optional override for config.remove_urls.

Returns:

Copy of the input dataframe with an additional cleaned-text column.

Return type:

pandas.DataFrame