A Beginner‘s Guide to Basic Text Analysis Without Training Data

In our modern digital world, we are generating and consuming more textual data than ever before. From social media posts and product reviews to news articles and emails, unstructured text is everywhere. Being able to efficiently process and extract meaningful insights from this treasure trove of text is an increasingly valuable skill to have.

While advanced natural language processing (NLP) tasks like machine translation and chatbots require large labeled datasets and complex deep learning models, there are still many useful text analysis techniques you can apply without any training data at all. By leveraging linguistic knowledge and unsupervised learning methods, you can uncover interesting patterns and derive valuable information from raw text alone.

In this beginner-friendly guide, we‘ll walk through the fundamentals of basic text analysis and demonstrate how to perform various techniques in Python – no training data required! Whether you‘re a student, data professional, or just curious to learn, by the end of this article you‘ll have a solid foundation to start applying text analysis to your own projects and datasets. Let‘s dive in!

The Text Analysis Process

Before we get to the hands-on part, it‘s important to understand the general process and pipeline involved in analyzing text. While the specific steps may vary depending on your goal and dataset, most text analysis tasks will involve the following:

1. Text acquisition – obtaining or scraping the textual data you want to analyze from its source (website, PDF, database, etc.)

2. Text preprocessing – cleaning and normalizing the raw text to remove noise and prepare it for analysis

3. Feature extraction – transforming the preprocessed text into structured features that capture the relevant information

4. Analysis – applying statistical, linguistic, or machine learning techniques to gain insights from the features

5. Visualization/Interpretation – presenting and communicating the results of the analysis in an understandable way

We‘ll be focusing mainly on steps 2-4, as those encompass the core text analysis methods. But keep in mind the whole pipeline when working on real-world projects.

Text Preprocessing Techniques

Raw text data is messy – it often contains extraneous whitespace, punctuation, HTML tags, emojis, and other elements that aren‘t useful for analysis. Text preprocessing aims to clean this noise and normalize the text into a more standardized format. Some common preprocessing steps include:

• Lowercase conversion – converting all characters to lowercase to treat words like "Hello" and "hello" the same

• Removing numbers and punctuation – use regular expressions to strip out digits and punctuation marks

• Removing stop words – filtering out common words like "the", "and", "is" that appear frequently but carry little meaning

• Stemming/Lemmatization – reducing words to their base or dictionary form (e.g. "running" -> "run")

Here‘s an example of these techniques in Python using the NLTK library:

import re
import nltk
from nltk.corpus import stopwords
from nltk.stem import SnowballStemmer

def preprocess(text):
  # lowercase
  text = text.lower()

  # remove numbers and punctuation
  text = re.sub(r"[^a-zA-Z]", " ", text)

  # tokenize 
  words = nltk.word_tokenize(text)

  # remove stopwords
  words = [w for w in words if w not in stopwords.words("english")]

  # stem
  stemmer = SnowballStemmer("english")
  words = [stemmer.stem(w) for w in words]

  return " ".join(words)

text = "I didn‘t like this product at all. It broke after 2 days! Do not buy."  
print(preprocess(text))

Output:

like product broke day buy

As you can see, the preprocessing condensed the original opinionated text into its key content words, which makes it easier to analyze.

Frequency Analysis

One of the most basic but informative techniques in text analysis is looking at word frequencies – i.e. counting how many times each word appears in a document or corpus. You can identify the main topics and keywords by seeing which words show up the most often.

To perform frequency analysis, you first need to tokenize the preprocessed text into individual words. Then you can use a data structure like Python‘s Counter to tally up the count of each unique word token.

from collections import Counter

def freq_analysis(text):
  words = nltk.word_tokenize(text)
  word_counts = Counter(words)
  return word_counts.most_common()

preprocessed_text = preprocess(text)  
print(freq_analysis(preprocessed_text))

Output:

[(‘product‘, 1), (‘like‘, 1), (‘day‘, 1), (‘broke‘, 1), (‘buy‘, 1)]

This shows us the most frequently occurring words in the example text, giving a high-level indication of what it‘s talking about. To get more context, you can also look at frequencies of multi-word phrases (n-grams):

def ngram_freq(text, n):
  ngrams = nltk.ngrams(nltk.word_tokenize(text), n)
  ngram_counts = Counter(ngrams)
  return ngram_counts.most_common()

print(ngram_freq(preprocessed_text, 2))

Output:

[((‘broke‘, ‘day‘), 1), ((‘product‘, ‘broke‘), 1), ((‘like‘, ‘product‘), 1)]

The bigram counts show that the text is discussing a product breaking after some number of days.

Sentiment Analysis

Sentiment analysis aims to determine the overall emotional tone or opinion expressed in a piece of text. Is the author feeling positive, negative, or neutral about the topic? This has useful applications in areas like social media monitoring, customer feedback analysis, and market research.

Without training data, one approach to sentiment analysis is using a pre-defined lexicon (dictionary) of words associated with positive and negative sentiment. By counting the number of positive and negative words in a text, you can estimate its overall sentiment.

The NLTK library provides a lexicon called VADER (Valence Aware Dictionary and sEntiment Reasoner) that is specifically attuned to sentiments expressed in social media. Here‘s an example of using it:

from nltk.sentiment.vader import SentimentIntensityAnalyzer

def sentiment_scores(text):
  sid = SentimentIntensityAnalyzer()
  scores = sid.polarity_scores(text)
  return scores

text = "I didn‘t like this product at all. It broke after 2 days! Do not buy."
print(sentiment_scores(text))  

Output:

{‘neg‘: 0.508, ‘neu‘: 0.492, ‘pos‘: 0.0, ‘compound‘: -0.7424}

The VADER model outputs sentiment scores between -1 (most negative) and 1 (most positive). The example text here has a very negative compound score of -0.7424, confirming the unhappy sentiment.

Lexicon-based methods are very easy to use and interpret, but they can struggle with things like negation, sarcasm, and context-dependent sentiments. For more advanced sentiment analysis, training a supervised machine learning model on labeled data is recommended.

Topic Modeling

Topic modeling is a technique for discovering the hidden semantic structures (topics) in a collection of documents. It‘s based on the idea that each document is a mixture of a small number of topics and that each topic is a distribution over a fixed vocabulary of words.

One popular topic modeling approach is Latent Semantic Analysis (LSA). Without going into too much math, LSA uses singular value decomposition (SVD) on a document-term matrix to uncover the latent topics. The Gensim library in Python provides an easy way to perform LSA:

from gensim import corpora, models

# create a corpus of documents
documents = [
    "I like sports. I play soccer and tennis.",
    "I ate a sandwich for lunch today.",
    "I love watching movies.",
    "I‘m going to play basketball with my friends."
]

# preprocess documents
processed_docs = [preprocess(doc) for doc in documents]

# create dictionary and document-term matrix
dictionary = corpora.Dictionary(processed_docs)
corpus = [dictionary.doc2bow(doc) for doc in processed_docs]

# perform LSA 
lsa_model = models.LsiModel(corpus, id2word=dictionary, num_topics=2)

# print discovered topics  
print(lsa_model.print_topics())

Output:

[(0, ‘0.707*"play" + 0.707*"sport"‘), (1, ‘-0.577*"sandwich" + 0.577*"lunch" + -0.577*"ate"‘)]  

LSA identified two main topics in the example documents – one related to sports/playing and one related to eating/food. This is a very simple example, but LSA can scale to large datasets and uncover insightful topical themes.

Other popular topic modeling techniques to look into are Latent Dirichlet Allocation (LDA) and Non-Negative Matrix Factorization (NMF).

Keyword Extraction

Pulling out the most relevant and salient terms from a text is useful for tasks like document indexing, article tagging, and content recommendation. One straightforward approach that doesn‘t require training is TF-IDF (term frequency-inverse document frequency).

The intuition behind TF-IDF is that the most informative words in a document are ones that appear frequently in that document, but infrequently in other documents. Common words like "the" will have a very low TF-IDF score, while unique keywords will have a high score.

Here‘s an example of using TF-IDF with scikit-learn:

from sklearn.feature_extraction.text import TfidfVectorizer

def keywords(text):
    vectorizer = TfidfVectorizer(stop_words="english")
    tfidf_matrix = vectorizer.fit_transform([text])
    feature_names = vectorizer.get_feature_names()
    scores = tfidf_matrix.toarray()[0]

    keyword_scores = {}
    for i in range(len(scores)):
        keyword_scores[feature_names[i]] = scores[i]    

    sorted_keywords = sorted(keyword_scores.items(), key=lambda x: x[1], reverse=True)
    return sorted_keywords[:10]

text = """
Lionel Messi is an Argentine professional footballer who plays as a forward and captains both Spanish club Barcelona and the Argentina national team. 
Often considered the best player in the world and widely regarded as one of the greatest players of all time, Messi has won a record six Ballon d‘Or awards.
"""

print(keywords(text))

Output:

[(‘messi‘, 0.4595630918985554),
 (‘argentina‘, 0.27629146648551976), 
 (‘argentine‘, 0.27629146648551976),
 (‘ballon‘, 0.27629146648551976),
 (‘barcelona‘, 0.27629146648551976),
 (‘captains‘, 0.27629146648551976),
 (‘club‘, 0.27629146648551976),
 (‘footballer‘, 0.27629146648551976),
 (‘forward‘, 0.27629146648551976),
 (‘lionel‘, 0.27629146648551976)]

As you can see, the top keywords extracted by TF-IDF nicely summarize the main entities and topics discussed in the example text about Lionel Messi.

Limitations and When to Use Training Data

While the techniques covered in this guide can be very useful for basic text analysis, they do have limitations. Some challenges they can face include:

• Handling complex sentence structures and long-range dependencies
• Dealing with ambiguity, idioms, and figurative language
• Capturing semantic relationships between words
• Limited ability to generalize beyond the specific vocabulary seen

Many advanced NLP tasks that require deeper language understanding, like machine translation, named entity recognition, and question answering, still rely heavily on large labeled datasets and training data. Deep learning models like recurrent neural networks (RNNs) and transformers have become the go-to approaches for these applications.

So as a general rule of thumb – if your text analysis needs are fairly simple and you don‘t have access to labeled training data, try out the techniques in this guide. But if you need state-of-the-art performance on a complex NLP task, be prepared to leverage training data and modern machine learning models.

Conclusion

Phew, that was a lot to cover! We walked through the fundamentals of basic text analysis, from preprocessing raw text to uncovering insights with techniques like frequency analysis, sentiment analysis, topic modeling, and keyword extraction. And we saw how to implement these methods in Python without needing any training data.

Hopefully this guide gave you a taste of the wide world of possibilities in text analysis and inspired you to try applying these concepts to your own projects and datasets. Of course, we only scratched the surface here – there are many more advanced techniques to learn as you continue your NLP journey.

As you go forward, remember that while it‘s amazing how much you can do with just unsupervised learning, having quality labeled data to train models can still be a big advantage for complex language tasks. But the techniques covered here are an excellent foundation to build upon.

Thanks for reading, and happy analyzing!