# Building an AI Storyteller with LangChain, OpenAI, and Hugging Face

- Canonical: https://33rdsquare.com/building-an-ai-storyteller-application-using-langchain-openai-and-hugging-face/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Artificial intelligence has made remarkable progress in understanding and generating human language in recent years. Large language models like OpenAI‘s GPT-3 can produce text that is often indistinguishable from human writing, while machine learning techniques allow computers to "see" and describe the contents of images.

By combining these capabilities, it‘s now possible to create AI applications that exhibit an impressive level of creativity and linguistic intelligence. In this post, we‘ll walk through how to build an "AI Storyteller" that can generate unique stories based on visual prompts, using the LangChain framework along with models from OpenAI and Hugging Face.

## The Rise of Language AI

The field of natural language processing (NLP) has undergone a paradigm shift in the past few years, moving from statistical methods based on local word patterns to large-scale neural language models that learn rich representations of meaning from vast amounts of text data.

The GPT (Generative Pre-trained Transformer) architecture developed by OpenAI is a prime example of this new wave of NLP technology. GPT-3, the largest model in the GPT family, contains 175 billion parameters and was trained on a dataset of half a trillion words, allowing it to understand and generate coherent text on almost any topic.

![GPT\-3 architecture diagram](https://33rdsquare.com/gpt3-architecture.png)

_The GPT-3 model architecture. Source: OpenAI_

Other notable language AI milestones include the BERT model from Google, which set new records on a wide range of NLP benchmarks in 2018, and the DALL-E image generation model from OpenAI, which can create strikingly realistic images from textual descriptions.

The economic potential of language AI is immense. The global NLP market size was valued at $11.1 billion in 2020 and is projected to reach $341.3 billion by 2030, growing at a CAGR of 40.9% from 2021 to 2030 (Source: Allied Market Research). Language AI is being applied across industries including healthcare, finance, e-commerce, and entertainment.

## The LangChain Framework

While language models like GPT-3 are extremely capable, building real-world applications with them requires chaining together multiple components and managing dataflows between them. This is where the LangChain framework comes in.

LangChain is an open-source Python library that helps developers build applications with large language models. It provides a standard interface for interacting with different language models (e.g. OpenAI, Cohere, Hugging Face), as well as utilities for processing text data, managing prompts, and combining components into multi-step workflows.

One of the core abstractions in LangChain is the `PromptTemplate`, which allows defining a standard template for the textual inputs passed to a language model. Prompt templates can include variables that are dynamically filled at runtime, making it easy to generate custom prompts programmatically.

For example, here‘s a simple prompt template for generating a story based on an image caption:

```
from langchain import PromptTemplate

template = """
You are an AI story generator.
Based on the context provided, write an engaging short story in a kid-friendly tone.
The story should be at least 100 words long.

Context: {caption}

Story:
"""

prompt = PromptTemplate(
    input_variables=["caption"],
    template=template
)
```

The `{caption}` placeholder will be replaced with the actual image caption at runtime.

LangChain also provides a standard interface for language models through the `LLM` class. Here‘s how we can use it with the OpenAI model API:

```
from langchain import OpenAI

llm = OpenAI(model_name="text-davinci-002", temperature=0.9)
```

The `temperature` parameter controls the randomness of the model‘s output, with higher values producing more varied results.

Chaining together prompts and models is a key feature of LangChain. For example, we can combine the story prompt template with the OpenAI model to generate stories:

```
from langchain import LLMChain

llm_chain = LLMChain(
    prompt=prompt,
    llm=llm
)

caption = "A boy riding his bicycle in the park on a sunny day"
story = llm_chain.run(caption)

print(story)
```

This will generate a story based on the provided caption, for example:

```
It was a beautiful day at the park and Johnny couldn‘t wait to ride his new bicycle. He put on his helmet, hopped on the bike, and started pedaling down the path. The sun was shining brightly and a gentle breeze blew through the trees.

As Johnny rode along, he waved hello to the other people enjoying the park - families having picnics, kids playing frisbee, and elderly couples out for a stroll. He felt a great sense of freedom and joy as he zoomed past the colorful flowers and sparkling fountain.

Suddenly, Johnny spotted an ice cream cart in the distance. He slowed to a stop and used his allowance money to buy a double scoop of his favorite flavors - chocolate and mint chip. Sitting on a nearby bench, he savored the cold treat while watching a squirrel scamper up a tree.

What a perfect day, Johnny thought to himself. He couldn‘t wait to ride his bike in the park again soon. But for now, it was time to head home for dinner. With a big smile on his face, Johnny hopped back on his bicycle and pedaled off, the warm sun on his back and the sweet taste of ice cream lingering on his tongue.
```

Of course, results will vary due to the randomness in the language model output. But this example demonstrates the power of chaining together simple components to produce impressive AI-generated stories.

## Hugging Face Integration

In addition to OpenAI, LangChain also integrates with the open-source models hosted on Hugging Face, a popular platform for sharing and discovering ML models.

Hugging Face provides a wide range of state-of-the-art NLP models that can be used for tasks like text classification, named entity recognition, question answering, and more. Many of these models follow the "transformer" architecture popularized by models like BERT and GPT.

For our AI storyteller application, we‘ll use two Hugging Face models in particular:

1. [Salesforce/blip-image-captioning-large](https://huggingface.co/Salesforce/blip-image-captioning-large) – An image captioning model trained on 129 million image-text pairs. It achieves state-of-the-art performance on the COCO Captions dataset with a BLEU score of 41.3.
2. [espnet/kan-bayashi_ljspeech_vits](https://huggingface.co/espnet/kan-bayashi_ljspeech_vits) – A high-quality text-to-speech model trained on the LJSpeech audiobook corpus. It can convert English text to natural-sounding speech in a female voice.

To use these models in LangChain, we first need to install the Hugging Face Hub client library:

```
pip install huggingface_hub
```

Then we can load the models using the `HuggingFacePipeline` class:

```
from langchain.llms import HuggingFacePipeline
from transformers import AutoTokenizer, AutoModelForSeq2SeqLM

tokenizer = AutoTokenizer.from_pretrained("Salesforce/blip-image-captioning-large")
model = AutoModelForSeq2SeqLM.from_pretrained("Salesforce/blip-image-captioning-large")

captioner = HuggingFacePipeline(
    pipeline=model.generate,
    tokenizer=tokenizer,
    task="image-to-text"
)
```

This loads the BLIP image captioning model and tokenizer, and wraps them in a LangChain pipeline object that we can use to generate captions for images.

For example, if we have an image file named `image.jpg`, we can caption it like this:

```
from langchain.docstore.document import Document
from PIL import Image

image = Image.open("image.jpg")

image_doc = Document(
    content=captioner.predict(image),
    metadata={"image": image}
)

caption = image_doc.page_content

print(caption)
```

This will output a caption for the image, such as:

```
a young boy riding a bicycle on a path in a park
```

We can then pass this caption through our LangChain storytelling pipeline to generate a full story.

The text-to-speech component works similarly:

```
tokenizer = AutoTokenizer.from_pretrained("espnet/kan-bayashi_ljspeech_vits")
model = AutoModelForSeq2SeqLM.from_pretrained("espnet/kan-bayashi_ljspeech_vits")

tts_pipeline = HuggingFacePipeline(
    pipeline=model.generate,
    tokenizer=tokenizer,
    task="text-to-speech"
)

story_text = "Once upon a time, in a land far, far away..."
audio_bytes = tts_pipeline(story_text)

with open("story.wav", "wb") as f:
    f.write(audio_bytes)
```

This converts the story text to an audio waveform and saves it as a WAV file that can be played back.

## User Interface

To make our AI storyteller accessible to end-users, we can create a simple web interface using the Streamlit library. Streamlit allows building interactive apps with Python, using a declarative syntax for defining UI components.

Here‘s the code for a basic Streamlit app that combines all the components we‘ve built:

```
import streamlit as st
from langchain.chains import SimpleSequentialChain
from langchain.text_splitter import CharacterTextSplitter

st.set_page_config(page_title="AI Storyteller")
st.title("🎉 AI Storyteller")

st.write("Upload an image and get an AI-generated story based on it!")
image_file = st.file_uploader("Choose an image", type=["jpg", "jpeg", "png"])
generate_button = st.button("Generate Story")

if generate_button and image_file is not None:
    # Save uploaded image to disk
    with open(image_file.name, "wb") as f:
        f.write(image_file.getbuffer())

    # Generate caption and story
    with st.spinner("Analyzing the image..."):
        caption = captioner.predict(Image.open(image_file.name))

    with st.spinner("Crafting a story..."):
        story = llm_chain.run(caption)

    with st.spinner("Converting to audio..."):
        audio_bytes = tts_pipeline(story)

    # Display results
    st.subheader("Your Generated Story")
    text_chunks = CharacterTextSplitter().split_text(story)
    for chunk in text_chunks:
        st.write(chunk)

    st.write("### Listen to the story")
    st.audio(audio_bytes, format=‘audio/wav‘)
```

Running this code (e.g. `streamlit run app.py`) starts a web server that presents the user with a file upload widget. After uploading an image and clicking the "Generate Story" button, the app displays the generated story text and an audio player to listen to it.

Here‘s a screenshot of the app in action:

![AI Storyteller app screenshot](https://33rdsquare.com/ai-storyteller-screenshot.png)

_Screenshot of the AI Storyteller Streamlit app._

## Challenges and Future Directions

While AI-generated stories can be entertaining and creative, there are some important limitations and potential issues to keep in mind:

- Lack of long-term coherence: While language models excel at producing locally fluent text, they often struggle to maintain consistency over longer passages. Generated stories may contain plot holes or contradictions when examined closely.
- Factual inaccuracies: Language models can easily produce false or misleading statements, as they have no inherent knowledge of what is true. Story details may not always align with real-world facts.
- Lack of emotion and nuance: Though language models can pick up on patterns of sentiment and emotion in their training data, they lack true empathy or emotional intelligence. Generated stories may feel shallow or formulaic compared to human-authored works.
- Potential for bias and toxicity: Models trained on broad, uncurated datasets from the internet can easily pick up on sexist, racist, or otherwise problematic language patterns. It‘s important to detect and filter potentially toxic model outputs.
- Computational expense: Large language models like GPT-3 are notoriously expensive to train and run, due to their size. Finding ways to make AI storytelling more efficient is an important challenge.

Despite these limitations, AI storytelling systems are rapidly improving and finding real-world applications. Some promising directions for future work include:

- Fine-tuning language models on high-quality story datasets, potentially in narrow domains, to improve coherence and style transfer
- Developing new decoding strategies and architectures to improve long-range dependencies and consistency in generated stories
- Incorporating explicit knowledge bases and fact-checking to enhance the factual grounding of generated text
- Imbuing language models with theory-of-mind and emotional intelligence, perhaps through multi-modal training paradigms
- Combining rule-based and neural approaches to get the best of human-specified domain knowledge and data-driven generalization

## Conclusion

By combining the language understanding and generation capabilities of large neural models with the compositional power of LangChain, it‘s possible to create engaging AI storytelling experiences with just a few lines of code.

While there are important technical and ethical challenges to overcome, AI storytelling systems are poised to become increasingly prevalent and powerful. As an AI practitioner, it‘s an exciting time to be exploring this space.

Some key takeaways and best practices covered in this post:

- Use prompt templating to provide structured context for language models
- Chain together models with different capabilities (e.g. captioning, generation, TTS) to create full-stack applications
- Integrate open-source models from platforms like Hugging Face to leverage state-of-the-art research
- Present AI model results to users with simple, intuitive interfaces built with tools like Streamlit
- Consider the limitations and potential issues of AI-generated content, and work to address them through improved modeling techniques and responsible deployment practices

I encourage you to experiment with the concepts and code samples presented here to build your own AI storytelling applications. Feel free to connect with me on Twitter ([@username](https://twitter.com/username)) or GitHub ([@username](https://github.com/username)) to share what you create!

## References and Further Reading

- [Language Models are Few-Shot Learners](https://arxiv.org/abs/2005.14165) (GPT-3 paper)
- [LangChain documentation](https://langchain.com/)
- [Hugging Face model hub](https://huggingface.co/models)
- [Streamlit documentation](https://docs.streamlit.io/)
- [Opportunities and Risks of Language Models](https://arxiv.org/abs/2302.04093)

---

Source: [Building an AI Storyteller with LangChain, OpenAI, and Hugging Face](https://33rdsquare.com/building-an-ai-storyteller-application-using-langchain-openai-and-hugging-face/)