Unveiling the Power of Chain of Verification: Implementing CoVe with LangChain and LLMs

Introduction

In the rapidly evolving world of artificial intelligence (AI), large language models (LLMs) have revolutionized the way we interact with and generate text. However, one significant challenge that persists in LLMs is the problem of hallucination – the generation of plausible-sounding but factually incorrect information. As we continue to rely on AI for various applications, it is crucial to address this issue and develop techniques to mitigate hallucination in LLMs.

One promising approach to reducing hallucination is the Chain of Verification (CoVe) technique, which combines prompting and consistency checks to create a self-verification system for LLMs. In this blog post, we will dive deep into the CoVe process, explore its implementation using the LangChain framework and LangChain Expression Language (LCEL), and discuss its potential for improving the reliability of LLMs.

Understanding the Chain of Verification (CoVe) Technique

The CoVe technique is designed to encourage LLMs to think critically about their responses and self-correct when necessary. It achieves this by breaking down the verification process into smaller, more manageable queries. The CoVe process consists of four main steps:

1. Generating the Baseline Response: Given a query, the LLM generates an initial response without any special prompting. This baseline response serves as the starting point for the CoVe process.

2. Planning Verifications: Based on the query and the baseline response, the LLM generates a list of verification questions that can help self-analyze the original response for potential mistakes or inconsistencies.

3. Executing Verifications: Each verification question is answered independently, and the answers are checked against the original response to identify any inconsistencies or errors.

4. Generating the Final Verified Response: Taking into account the discovered inconsistencies (if any), the LLM generates a revised response that incorporates the verification results.

By following this structured approach, the CoVe technique enables LLMs to critically examine their own outputs and make necessary corrections, resulting in more reliable and accurate responses.

Implementing CoVe with LangChain and LCEL

To demonstrate the implementation of CoVe, we will use the LangChain framework and LangChain Expression Language (LCEL). LangChain is a powerful tool that allows us to create custom chains and pipelines for various natural language processing tasks, making it an ideal choice for implementing CoVe.

Here‘s a step-by-step guide on how to implement CoVe using LangChain and LCEL:

Step 1: Install and Load Libraries

First, make sure you have the necessary libraries installed:

!pip install langchain duckduckgo-search

Then, import the required modules:

from langchain import PromptTemplate
from langchain.llms import GooglePalm
from langchain.schema.output_parser import StrOutputParser
from langchain.schema.runnable import RunnablePassthrough, RunnableLambda

Step 2: Create and Initialize the LLM Instance

In this example, we will use Google‘s Palm LLM, which is freely available through the LangChain package. Generate an API key for Google Palm and initialize the LLM instance:

API_KEY = ‘your_api_key‘
llm = GooglePalm(google_api_key=API_KEY)
llm.temperature = 0.4
llm.model_name = ‘models/text-bison-001‘
llm.max_output_tokens = 2048

Step 3: Generate the Baseline Response

Create a prompt template to generate the initial baseline response and use it to create the baseline response LLM chain:

BASELINE_PROMPT = """
Answer the below question which is asking for a concise factual answer. NO ADDITIONAL DETAILS.
Question: {query}
Answer:
"""

baseline_response_prompt_template = PromptTemplate.from_template(BASELINE_PROMPT)
baseline_response_chain = baseline_response_prompt_template | llm | StrOutputParser()

Step 4: Generate Verification Questions

Construct a verification question template and use it to generate verification questions based on the query and baseline response:

VERIFICATION_QUESTION_TEMPLATE = """
Your task is to create a verification question based on the below question provided.
...
Actual Question: {query}
Final Verification Question:
"""

verification_question_template_prompt_template = PromptTemplate.from_template(VERIFICATION_QUESTION_TEMPLATE)
verification_question_template_chain = verification_question_template_prompt_template | llm | StrOutputParser()

VERIFICATION_QUESTION_PROMPT = """
Your task is to create a series of verification questions based on the below question, the verification question template and baseline response.
...
Actual Question: {query}
Baseline Response: {base_response}
Verification Question Template: {verification_question_template}
Final Verification Questions:
"""

verification_question_generation_prompt_template = PromptTemplate.from_template(VERIFICATION_QUESTION_PROMPT)
verification_question_generation_chain = verification_question_generation_prompt_template | llm | StrOutputParser()

Step 5: Execute Verification Questions

Use an external search tool agent to execute the verification questions. In this example, we use the DuckDuckGo search module:

from langchain.agents import ConversationalChatAgent, AgentExecutor
from langchain.tools import DuckDuckGoSearchResults

search = DuckDuckGoSearchResults()
tools = [search]
custom_system_message = "Assistant assumes no knowledge & relies on internet search to answer user‘s queries."
max_agent_iterations = 5
max_execution_time = 10

chat_agent = ConversationalChatAgent.from_llm_and_tools(
    llm=llm, tools=tools, system_message=custom_system_message)
search_executor = AgentExecutor.from_agent_and_tools(
    agent=chat_agent,
    tools=tools,
    return_intermediate_steps=True,
    handle_parsing_errors=True,
    max_iterations=max_agent_iterations,
    max_execution_time=max_execution_time)

verification_chain = RunnablePassthrough.assign(
    split_questions=lambda x: x[‘verification_questions‘].split("\n"),
) | RunnablePassthrough.assign(
    answers=(lambda x: [{"input": q, "chat_history": []} for q in x[‘split_questions‘]])
) | search_executor.map() | (lambda x: "\n".join(["Question: {} Answer: {}\n".format(question, answer[‘output‘]) for question, answer in zip(x[‘split_questions‘], x[‘answers‘])]))

Step 6: Generate the Final Refined Response

Define a prompt template and LLM chain to generate the final refined answer:

FINAL_ANSWER_PROMPT = """
Given the below `Original Query` and `Baseline Answer`, analyze the `Verification Questions & Answers` to finally provide the refined answer.
Original Query: {query}
Baseline Answer: {base_response}
Verification Questions & Answer Pairs:
{verification_answers}
Final Refined Answer:
"""

final_answer_prompt_template = PromptTemplate.from_template(FINAL_ANSWER_PROMPT)
final_answer_chain = final_answer_prompt_template | llm | StrOutputParser()

Step 7: Put All the Chains Together

Combine all the chains defined earlier so that they run in sequence:

chain = RunnablePassthrough.assign(
    base_response=baseline_response_chain
) | RunnablePassthrough.assign(
    verification_question_template=verification_question_template_chain
) | RunnablePassthrough.assign(
    verification_questions=verification_question_generation_chain
) | RunnablePassthrough.assign(
    verification_answers=verification_chain
) | RunnablePassthrough.assign(
    final_answer=final_answer_chain)

response = chain.invoke({"query": "Who wrote the book ‘Economics of Small Things‘ ?"})
print(response)

Conclusion

The Chain of Verification (CoVe) technique is a promising approach to reducing hallucination in large language models. By breaking down the verification process into smaller, more manageable queries and preventing the model from reviewing its previous responses, CoVe encourages LLMs to think critically about their outputs and self-correct when necessary.

Implementing CoVe using the LangChain framework and LangChain Expression Language (LCEL) provides a streamlined and efficient way to create custom chains and pipelines for the verification process. As demonstrated in the step-by-step guide, LangChain‘s modular approach allows for easy integration of various components, such as prompt templates, LLMs, and external search tools, making it a powerful tool for building AI applications.

Looking ahead, there are several potential avenues for further improving the CoVe technique. One promising direction is to incorporate external tools or domain-specific data to enhance the accuracy and reliability of the verification process. For example, using retrieval techniques like Retrieval Augmented Generation (RAG) can enable LLMs to produce factually correct responses based on domain-specific data.

As the field of AI continues to evolve, techniques like CoVe will play a crucial role in addressing the challenges of hallucination and improving the reliability of LLMs. By combining innovative approaches with powerful tools like LangChain, we can unlock the full potential of AI and create more trustworthy and accurate language models.

Frequently Asked Questions (FAQ)

Q1: What are some other techniques for reducing hallucination in LLMs?
A1: Several techniques can be employed to reduce hallucination at different levels, such as:

• Prompt-level techniques: Tree of Thought, Chain of Thought
• Model-level techniques: DoLa Decoding by Contrasting Layers
• Self-check techniques: Chain of Verification (CoVe)

Q2: How can the CoVe process be further improved?
A2: The CoVe process can be enhanced by leveraging external search tools like Google Search API for more accurate verifications. Additionally, for domain-specific use cases, retrieval techniques such as Retrieval Augmented Generation (RAG) can be employed to ensure factually correct responses based on domain-specific data.

Q3: Are there any libraries or frameworks that support the CoVe verification mechanism?
A3: While there are no ready-to-use open-source tools specifically implementing the CoVe mechanism, it is possible to construct a custom solution using APIs like Serp API, Google Search, and LangChain, as demonstrated in this blog post.

Q4: What is Retrieval Augmented Generation (RAG), and how can it help in reducing hallucination?
A4: Retrieval Augmented Generation (RAG) is a technique used for domain-specific use cases, where an LLM produces factually correct responses based on retrieval from domain-specific data. By providing the LLM with relevant and accurate information, RAG can help reduce hallucination and improve the reliability of the generated responses.

Q5: How was the CoVe pipeline implemented in the original research paper?
A5: In the original research paper, the authors used the Llama 65B model as the LLM for the CoVe pipeline. They employed prompt engineering techniques, such as few-shot examples, to generate verification questions and provide guidance to the model throughout the verification process.

As the field of AI continues to advance, it is essential to stay updated with the latest developments and techniques. The information provided in this blog post is based on the state-of-the-art as of 2024, ensuring that readers have access to the most relevant and up-to-date knowledge on implementing chain of verification using LangChain and LLMs.