remove emojis
With over 600 million speakers worldwide, Hindi is one of the most widely spoken languages, especially in the Indian subcontinent. As more and more Hindi text data is generated online from sources like social media, news articles, blogs, and customer reviews, analyzing this data to derive meaningful insights has become increasingly important.
This is where natural language processing (NLP) comes in. NLP enables computers to understand, interpret, and manipulate human language. By applying NLP techniques to Hindi text, we can automatically extract information, understand sentiment and emotions, summarize content, detect topics, and much more.
In this in-depth, hands-on guide, we‘ll walk through the key steps involved in processing and analyzing Hindi text data using Python and NLP. Whether you‘re a budding data scientist interested in learning about NLP for Indic languages or a developer looking to add Hindi text processing capabilities to your application, this guide has you covered.
The Dataset
For this tutorial, we‘ll be working with a dataset of Hindi tweets available on Kaggle. The dataset contains over 16,000 tweets in Hindi, annotated as either sarcastic or non-sarcastic. While our goal isn‘t tweet classification, this dataset serves as a great example to demonstrate various Hindi text preprocessing techniques.
Cleaning and Preprocessing Hindi Text
Raw Hindi text data can be quite noisy, with unwanted characters, links, emojis, and inconsistencies. Before any analysis, it‘s crucial to clean and preprocess the text to make it compatible for NLP. Here are the key cleaning steps:
1. Load the Dataset
First, we‘ll load the Hindi tweets dataset into a Pandas DataFrame:
import pandas as pddf_sarcastic = pd.read_csv(‘Sarcasm_Hindi_Tweets-SARCASTIC.csv‘) df_non_sarcastic = pd.read_csv(‘Sarcasm_Hindi_Tweets-NON-SARCASTIC.csv‘)
df_sarcastic[‘label‘] = ‘sarcastic‘
df_non_sarcastic[‘label‘] = ‘non_sarcastic‘df = pd.concat([df_sarcastic, df_non_sarcastic], axis=0)
2. Remove Unwanted Attributes
The tweets dataset contains several columns that are irrelevant for text analysis, like user details, tweet timestamp, retweet count, etc. We can remove them:
columns_to_drop = [‘username‘,‘acctdesc‘,‘location‘,‘following‘,‘followers‘, ‘totaltweets‘, ‘usercreatedts‘, ‘tweetcreatedts‘, ‘retweetcount‘, ‘hashtags‘] df = df.drop(columns_to_drop, axis=1)
3. Remove Emojis and Unwanted Characters
Tweets often contain emojis, links, email addresses and other items that need to be removed before analysis. We can use regular expressions for this:
import reemoji_pattern = re.compile("[" u"U0001F600-U0001F64F" # emoticons u"U0001F300-U0001F5FF" # symbols & pictographs u"U0001F680-U0001F6FF" # transport & map symbols u"U0001F1E0-U0001F1FF" # flags u"U00002500-U00002BEF" # chinese char u"U00002702-U000027B0" u"U00002702-U000027B0" u"U000024C2-U0001F251" u"U0001f926-U0001f937" u"U00010000-U0010ffff" u"u2640-u2642" u"u2600-u2B55" u"u200d" u"u23cf" u"u23e9" u"u231a" u"ufe0f" # dingbats u"u3030" "]+", flags=re.UNICODE)
def clean_text(text): text = re.sub(emojipattern, ‘‘, text) text = re.sub(r‘http\S+‘, ‘‘, text) # remove links text = re.sub(r‘[a-zA-Z0-9+.-]+@[a-zA-Z0-9.-]+‘, ‘‘, text) # remove email text = re.sub(r‘[0-9]‘, ‘‘, text) # remove numbers return text
df[‘cleaned_text‘] = df[‘tweet‘].apply(clean_text)
4. Tokenize Text
Tokenization is the process of splitting text into smaller units called tokens, usually words. For Hindi tokenization, we can use the indic-nlp-library:
from indicnlp.tokenize import indic_tokenizedef tokenize(text): return indic_tokenize.trivial_tokenize(text)
df[‘tokens‘] = df[‘cleaned_text‘].apply(tokenize)
5. Remove Stopwords
Stopwords are commonly used words that add little meaning like "and", "the", "a", etc. We remove them to focus on more meaningful words:
from indicnlp.stopwords import hi_stopwordsdef remove_stopwords(tokens): return [tok for tok in tokens if tok not in hi_stopwords]
df[‘tokens‘] = df[‘tokens‘].apply(remove_stopwords)
In addition to removing standard Hindi stopwords, you may also want to remove any domain-specific stopwords that occur frequently in your dataset but aren‘t informative, like "Twitter" for a tweets dataset.
6. Remove Infrequent and Highly Frequent Words
Some words occur too rarely to be useful for analysis while others occur too frequently to be distinctive. We can filter them out based on a frequency threshold:
from collections import defaultdictfreq = defaultdict(int) for tokens in df[‘tokens‘]: for token in tokens: freq[token] += 1
min_freq, max_freq = 5, 1000 def filter_tokens(tokens): return [tok for tok in tokens if min_freq < freq[tok] < max_freq]
df[‘tokens‘] = df[‘tokens‘].apply(filter_tokens)
Analyzing the Cleaned Text
With the text data cleaned and preprocessed, let‘s do some exploratory analysis and visualization.
Plotting Word Frequencies
First, we can compare the frequency distribution of words before and after cleaning to gauge the impact of preprocessing:
from plotly import graph_objects as godef plot_word_freq(df, column): freq = defaultdict(int) for tokens in df[column]: for token in tokens: freq[token] += 1 freq = sorted(freq.items(), key=lambda x: x[1], reverse=True)
x, y = zip(*freq) fig = go.Figure(data=go.Bar(x=x[:100], y=y[:100])) fig.show()plot_word_freq(df, ‘tokens_before‘)
plot_word_freq(df, ‘tokens‘)We observe that the frequency distribution becomes less skewed and more evenly spread out after preprocessing, with rare and common words removed.
Generating a Word Cloud
Word clouds are a visual representation of text data where the size of each word indicates its frequency. Let‘s create a word cloud of the cleaned Hindi tweets:
from wordcloud import WordCloud from matplotlib import pyplot as plttext = ‘ ‘.join(df.cleaned_text.tolist())
font_path = ‘path/to/hindi/unicode/font.ttf‘ wordcloud = WordCloud(width=1600, height=800, font_path=font_path).generate(text)
plt.figure(figsize=(20,10)) plt.imshow(wordcloud, interpolation=‘bilinear‘) plt.axis(‘off‘) plt.show()
The word cloud gives a quick sense of the main topics and themes present in the tweets – with the appropriate Hindi Unicode font, the text is rendered correctly too.
Conclusion and Next Steps
In this article, we saw how to preprocess raw Hindi text data and make it suitable for NLP analysis. We covered techniques like tokenization, stopword removal, handling emojis/links/numbers, and filtering words by frequency that are essential for cleaning text in any language.
However, we also touched upon some aspects unique to Hindi NLP, like using a suitable tokenizer and specifying Unicode fonts for rendering. Hindi has some interesting linguistic characteristics like a free word order and rich morphology that can pose challenges for NLP compared to English.
The preprocessing steps shown here lay the foundation for more advanced Hindi NLP tasks like sentiment analysis, named entity recognition, text classification, machine translation, and so on. With the increasing availability of Hindi language models and datasets, it has become easier than ever to apply NLP to Hindi text.
Some helpful open-source libraries for Hindi NLP that you can explore are:
- IndoNLP: Has Hindi tokenizers, word embeddings and datasets
- iNLTK: Supports Hindi tokenization, sentiment analysis, POS-tagging, etc.
- HindiNLP: Collection of Hindi text corpora and models
- IndicNLP: Covers many Indic languages including Hindi
Hindi NLP also plays a key role in building conversational AI applications like chatbots and voice assistants that cater to the Indian user base. As India‘s Internet population grows exponentially, processing and analyzing Hindi and other Indic language data becomes imperative for businesses and organizations.
I hope this guide gave you a practical introduction to cleaning and analyzing Hindi text data using NLP. Experiment with the code shown here, try it out on your own Hindi datasets, and don‘t hesitate to share your experiences or feedback in the comments!
