Building Intelligent Chatbots with Rasa Open Source: A Deep Dive

Chatbots have rapidly become a mainstream technology used by businesses to automate customer service, support, sales and marketing. By 2024, the global chatbot market size is expected to reach $9.4 billion, with a CAGR of 29.7% from 2019 to 2024. ((Source: https://www.businesswire.com/news/home/20200916005503/en/Global-9.4-Billion-Chatbot-Market-Insights-2020-2024))

While many proprietary chatbot building platforms exist, the open-source framework Rasa has emerged as a leading solution for creating highly capable AI assistants. Rasa adoption is quickly growing, with over 10k GitHub stars, 500+ contributors, 3 million+ downloads per month, and use cases from startups to Fortune 500s.

In this in-depth guide, we‘ll explore what makes Rasa unique and walk through how to build your own contextual AI chatbots with Rasa Open Source.

What is Rasa Open Source?

Rasa Open Source is a set of machine learning tools for building contextual AI assistants and chatbots. Rasa emerged in 2016 as one of the first frameworks to take a pure machine learning approach to conversational AI.

Some key features and benefits of Rasa Open Source include:

• Open source under the Apache 2.0 license
• Runs on-prem or in a private cloud for full data control
• Modular architecture with a choice of NLU/dialogue policies
• Supports text and voice conversations across popular messaging channels
• Production-ready and scalable to handle large volumes of conversations
• Extensible and customizable for a wide range of use cases
• Strong community of developers and adopters

Rasa‘s open source approach stands in contrast to cloud-based chatbot platforms like Google DialogFlow or IBM Watson. With Rasa, you own your models and data instead of handing it over to a third-party.

Rasa Architecture

Let‘s look more closely at Rasa‘s architecture:

(Source: https://rasa.com/docs/rasa/architecture/)

Rasa uses a model-driven approach, combining machine learning and probabilistic models to power the following key components:

• Natural Language Understanding (NLU): Rasa NLU handles intent classification and entity extraction from user messages using Transformers or other ML models. It maps free text to structured data.
• Dialogue Management: The dialogue manager predicts the next best action based on the conversation history and business logic. It uses policies like LSTM neural networks, transformers, and rule-based logic.
• Integrations: Rasa includes a rich set of built-in integrations to channels like Facebook Messenger, Slack, Twilio, and voice assistants. You can also build custom integrations.
• Training Data/Rules: Rasa learns from real conversations and developer-provided training data, including NLU examples, stories, and rules.
• Action Server: For more advanced integrations and dynamic behavior, custom actions and code can be executed on the action server.

Why Use Rasa Over Other Tools?

Compared to other open source chatbot frameworks like Botkit and Botpress, Rasa stands out for its:

• Purely machine learning-driven approach
• Extensive documentation and tutorials
• Vibrant developer community – 10k+ forum members, 60+ user groups globally
• Scalability to large production deployments
• Advanced contextual AI features like end-to-end training and transformers

A common question is how Rasa compares to commercial platforms like DialogFlow or Watson. While easier to get started with, these tools are a black box where you lack full control. They also tend to be more limited in terms of customization and integration options.

According to the 2021 State of Conversational AI survey, Rasa is the most popular open source framework, ahead of Botkit and Botpress:

(Source: https://research.aimultiple.com/conversational-ai-companies/)

Building Your First Rasa Assistant

Let‘s walk through the key steps in building a Rasa chatbot:

1. Install Rasa

First ensure you have Python 3.7 or 3.8 installed. Then install Rasa Open Source in a virtual environment:

python3 -m venv ./venv
source ./venv/bin/activate
pip install -U pip
pip install rasa

2. Create A New Project

To scaffold a new Rasa project, run:

rasa init

This will set up the recommended files and folders:

3. Define Domain and Intents

Next define your chatbot‘s knowledge using the domain.yml config file. This specifies the universe of intents, entities, slots, responses, forms, and actions your assistant should know about.

For example, here are some intents for a restaurant search bot:

intents:
  - greet
  - goodbye
  - affirm
  - deny
  - restaurant_search
  - inform

4. Prepare Training Data

Prepare training examples for your intents and entities. Rasa‘s training data format follows the YAML syntax. For each intent, provide a list of example user utterances:

nlu:
- intent: greet
  examples: |
    - hey
    - hello
    - hi
    - hello there
    - good morning
    - good evening

- intent: goodbye
  examples: |
    - bye
    - goodbye
    - see you around
    - see you later

- intent: restaurant_search
  examples: |
    - i‘m looking for a place to eat
    - I want to grab lunch
    - I am searching for a dinner spot
    - i‘m hungry

You‘ll want to start with 10-15 quality examples per intent, and expand over time.

5. Define Assistant Responses

To define your chatbot‘s responses, list utter templates in the domain.yml file.

responses:
  utter_greet:
  - text: "Hey there! How can I help?"

  utter_goodbye:
  - text: "Goodbye :("

  utter_ask_cuisine: 
  - text: "What kind of cuisine would you like?"

  utter_ask_location:
  - text: "Where are you looking to eat?"

Responses can include text, images, buttons, and other output.

6. Create Training Stories

Stories are example dialogs that show conversations between a user and your assistant. They use a format that includes NLU output like intents and entities, and action names:

stories:
- story: happy path
  steps:
  - intent: greet
  - action: utter_greet
  - intent: restaurant_search
  - action: restaurant_form
  - active_loop: restaurant_form
  - slot_was_set:
    - requested_slot: cuisine
  - slot_was_set:
    - cuisine: chinese
  - slot_was_set:
    - requested_slot: location  
  - slot_was_set:
    - location: "San Francisco"
  - active_loop: null
  - action: action_search_restaurants
  - action: utter_goodbye

Stories are used to train your dialogue management models.

7. Train NLU and Dialogue Models

To train a Rasa model:

rasa train

This will save trained models for NLU and dialogue in models/:

(venv) $ ls -1 models
20200910-132017.tar.gz

The model file is a zipped archive containing the trained NLU and dialogue models, plus configuration.

8. Chat with Your Assistant

To chat with your assistant on the command line, run:

rasa shell

You can then simulate conversations to test your chatbot‘s responses:

9. Deploy to a Messaging Channel

Rasa includes integrations for deploying your assistant to many popular messaging channels like Slack, Facebook Messenger, Microsoft Teams, Telegram, Twilio, and more.

To connect your Rasa chatbot to Slack:

1. Create a new Slack app
2. Enable Socket Mode, Event Subscriptions, and Bots
3. Get a Bot User OAuth Token
4. Update Rasa‘s credentials.yml with:

slack:
  slack_token: "xoxb-286426452756-safjasdf7sl38KLls"
  slack_channel: "#my-channel"

5. Start Rasa Open Source:

rasa run

6. Talk to your AI assistant in Slack!

Real-World Rasa Use Cases

Rasa is used by companies like Adobe, Airbus, Lemonade, T-Mobile, and more to power a wide variety of conversational AI use cases:

• Adobe‘s VA: Virtual assistant to handle IT tickets and requests
• Lemonade‘s Claims Bot: AI claims assistant to guide users through filing insurance claims
• Carbon Health: Healthcare bot for symptom checking and FAQ
• N26‘s Banking Bot: Handles banking queries and common requests
• Daisy Intelligence‘s HR Bot: Answers employee questions and supports HR workflows

(Source: https://www.lemonade.com/faq#service)

The Future of Conversational AI and Rasa

Conversational AI technologies like chatbots have matured rapidly in recent years due to advances in natural language processing, machine learning, and computing power.

Some key conversational AI trends to watch include:

• Deeper personalization based on user profiles and past conversations
• More human-like open-domain conversations
• Tighter integration with knowledge bases and external APIs
• Multimodal assistants that combine voice, vision, and language
• Low-code tools to expand access to non-developers

As an established open source leader, Rasa is well-positioned to stay at the forefront of these innovations. The Rasa team continues to invest heavily in research and development, with initiatives like Rasa X, a tool for improving AI assistants from real conversations, and regular new features in the core open source product.

(Source: https://rasa.com/rasa-x/)

By betting on open source, Rasa aims to become the Linux of conversational AI – a go-to framework with a huge ecosystem of developers, extensions, and applications.

Learning More

Hopefully this guide gave you a solid foundation for building chatbots with Rasa Open Source.
To continue your learning journey, I recommend:

• Rasa Masterclass video course: https://info.rasa.com/masterclass-yt-20
• Rasa Developer Certification: https://rasa.com/certification/
• Rasa blog tutorials: https://rasa.com/blog/
• Rasa YouTube tutorials: https://www.youtube.com/channel/UCJ0V6493mLvqdiVwOKWBODQ

For help and community, you can also:

• Join the Rasa Community Forum: https://forum.rasa.com
• Attend the weekly Rasa Developer Office Hours on Zoom
• Connect with local Rasa user groups: https://rasa.com/community/join-a-user-group/

With the amazing innovations happening in conversational AI, there‘s never been a better time to dive in and start building. I‘m excited to see what you‘ll create!