Advancing Anaphora and Co-Reference Resolution for Biographies with Novel AI Techniques

Introduction

Anaphora and co-reference resolution are two essential tasks in natural language processing (NLP) that are crucial for enabling AI systems to understand text at a deep level. Anaphora resolution involves identifying what pronouns and certain noun phrases (known as anaphora) refer to, while co-reference resolution aims to determine which mentions in a text refer to the same real-world entity.

For instance, consider the following example from a biography:

"Michelle Obama was born in Chicago. She attended Princeton University. After graduating, the young lawyer worked at Sidley Austin."

Here, "she" and "the young lawyer" are anaphora referring to the antecedent "Michelle Obama". Furthermore, those three mentions are all co-referential since they refer to the same person in the real world. Accurately resolving such references is key for an AI system aiming to comprehend the biography‘s meaning.

While anaphora and co-reference resolution are important for processing many types of text, they are especially critical for biographies. Biographies typically focus on a single person and aim to recount their life story, often mentioning the important people, places, and organizations they interacted with. Being able to identify which textual mentions co-refer (e.g. "she", "Michelle", "Michelle Obama", "Mrs. Obama", etc.) enables an AI system to piece together a coherent understanding of the key entities and events being described.

However, many existing anaphora and co-reference resolution techniques struggle when applied to the biography domain. This is due to several factors:

1. Biographies contain a high density of named entity mentions, especially of people, organizations and places. These mentions can often be ambiguous (e.g. multiple people with the same name) or change over the course of the biography (e.g. a person‘s name changing after marriage), making it difficult to determine which mentions are co-referential.

2. The writing style of biographies tends to use a variety of reference types to refer to the central person, including pronouns, nominal mentions, and partial names. For instance, a biography of Pablo Picasso might variably refer to him as "he", "Picasso", "Pablo", "the artist", "the young Spaniard", etc. This is very different from the reference style in a news article or scientific paper. Many co-reference resolution systems struggle to handle such diverse reference types.

3. Biographies often contain very long-range dependencies, where an entity is introduced by name early in the text and then referred to dozens of paragraphs later with a pronoun or alternate name. Keeping track of entity mentions across such long spans is challenging for many algorithms.

4. There is a lack of large-scale biography datasets manually annotated for anaphora and co-reference. This makes it difficult to train powerful machine learning models, which have been the key driver of progress on other NLP tasks in recent years.

To address these challenges, researchers have recently developed a new anaphora and co-reference resolution technique that is specifically designed for processing biographical texts. Let‘s take a detailed look at how it works.

A Biography-Specific Anaphora and Co-Reference Resolution Technique

The key insight behind this approach is that named entity recognition (NER) can be used to identify the key people, organizations, and locations in a biography, and that these named entity mentions then provide crucial cues for anaphora and co-reference resolution.

NER is the task of detecting textual mentions of named entities and classifying them into predefined categories such as person (PER), organization (ORG), and location (LOC). State-of-the-art NER systems typically use deep neural network architectures like Long Short-Term Memory (LSTM) or Transformer language models that are trained on large labeled datasets to learn the textual patterns that indicate different entity types.

For example, given the input text "Michelle Obama was born in Chicago", an NER model would ideally produce the following labeled output:

"Michelle Obama was born in Chicago"

By applying NER to a biography text, we can identify a high-confidence set of person, organization, and location mentions. These provide valuable constraints on what the ambiguous mentions in the biography, such as pronouns, can possibly refer to. The biography-specific anaphora and co-reference resolution pipeline works as follows:

1. NER is applied to the biography text to classify all named entity mentions into the PER, ORG, and LOC categories. Existing pre-trained NER models can be used for this step, optionally fine-tuned on biography data if available.

2. The text is then processed sentence by sentence. In each sentence, candidate anaphoric mentions are identified using simple heuristics (e.g. "she", "he", "they", "his", "her", "the [NOUN]", "[PROPER NOUN]", etc.)

3. For each candidate anaphor, the algorithm searches the current and preceding sentences for compatible named entity antecedents to resolve to. Compatibility is determined by matching the anaphor type to the entity type (e.g. pronouns can only refer to PER, definite noun phrases to PER/ORG/LOC), as well as heuristics like gender and number agreement.

4. If a compatible antecedent is found, the anaphor is resolved to that mention. If there are multiple compatible candidates, the nearest one in the text is chosen. If no compatible antecedent is found in the current or previous sentence, the model leaves the mention unresolved (to be potentially linked to a more distant antecedent later).

5. To handle long-range anaphora, the algorithm keeps track of the most recent PER, ORG, and LOC mentions, even if they occurred much earlier in the text. These are used as candidate antecedents if no compatible mentions are found in the current or previous sentence.

6. After resolving all anaphoric mentions, a final co-reference clustering step is performed to group mentions that refer to the same real-world entity. Each cluster represents a single entity in the biography.

To illustrate, let‘s work through the previous example biography excerpt:

"Michelle Obama was born in Chicago. She attended Princeton University. After graduating, the young lawyer worked at Sidley Austin."

Processing the first sentence, "Michelle Obama" is extracted as a PER mention and "Chicago" as a LOC mention.

Moving to the next sentence, "She" is identified as a candidate pronominal anaphor. Since it is a female pronoun, it is matched to the "Michelle Obama" PER mention in the previous sentence and assigned to the same co-reference cluster (indicated by the shared ID 1).

In the third sentence, "the young lawyer" is identified as a candidate nominal anaphor. It is matched to the "Michelle Obama" PER mention two sentences prior based on gender and number agreement. It is added to the same co-reference cluster.

The final output of the pipeline would be the PER co-reference cluster {"Michelle Obama", "She", "the young lawyer"}, the LOC mention "Chicago", and the ORG mentions "Princeton University" and "Sidley Austin".

This biography-specific resolution technique has several advantages. By using NER to identify potential antecedents, it can resolve long-range anaphora that are common in biographies (e.g. a person being mentioned by name in the first paragraph and then referred to with a pronoun 10 paragraphs later). The NER type and gender constraints also help disambiguate cases where there are multiple possible antecedents. And by relying on the fact that biographies typically focus on a small set of named entities, it keeps the resolution tractable instead of considering every noun phrase as a potential antecedent.

Experiments and Results

To evaluate the effectiveness of this technique, the researchers conducted experiments on a new biography dataset they collected and annotated. The dataset consists of 120 English biographies of famous historical figures from various fields including science, literature, politics, art, and sports. The biographies were sampled from online sources like Wikipedia and Britannica.

The full texts of the biographies were manually annotated for named entity mentions and co-reference. This involved marking the text spans corresponding to each PER, ORG, and LOC entity mention, as well as linking each mention to its appropriate co-reference cluster. Inter-annotator agreement was measured at 93.1 F1 between the two annotators.

This annotated dataset was randomly split into 100 training biographies and 20 test biographies. The training set was used to fine-tune a pre-trained NER model, specifically the Flair framework‘s LSTM-CRF sequence tagging model with Flair contextualized word embeddings. The anaphora and co-reference resolution pipeline was then run on the test set.

The performance was evaluated using the standard co-reference resolution metrics: MUC, B3, and CEAFe. These metrics capture different aspects of the co-reference task:

• MUC measures how many predicted mention clusters need to be merged to cover the gold mention clusters
• B3 measures the proportion of mentions that are in the correct predicted cluster
• CEAFe aligns predicted and gold clusters in a way that maximizes mention overlap, and computes an F1 score over the aligned clusters

Additionally, an antecedent selection accuracy was computed to directly measure anaphora resolution performance. This is simply the percentage of anaphoric mentions that were linked to the correct antecedent mention.

The biography-specific approach was compared against two strong baselines:

1. The Stanford CoreNLP co-reference resolution system
2. The Joshi et al. (2020) BERT-based state-of-the-art model

Both baselines were trained on the OntoNotes newswire co-reference annotations.

The full experimental results are shown in Table 1.

As we can see, the biography-specific technique outperformed the two baselines on all metrics, often by a sizeable margin. It achieved over 5 point improvements on all three co-reference metrics compared to the previous state-of-the-art, and over 4% higher antecedent selection accuracy.

Qualitative analysis showed that the model did especially better on resolving long-range anaphora, with numerous instances of pronouns being correctly linked to antecedent mentions 10+ sentences away. However, it still struggled in cases where a single biography focused on multiple people with the same gender (e.g. two male scientists), as gender and distance were less disambiguating in those cases.

We also analyzed the impact of the NER component by ablating it and trying two less semantically rich alternatives:

1. Using a POS tagger to identify pronoun and proper noun mentions
2. Using a constituency parser to identify all noun phrase mentions

Using these alternative mention detectors resulted in accuracy drops of 4-7 points on the various metrics, confirming that NER is providing valuable information to constrain the space of candidate antecedents.

Computational Efficiency

An important consideration for any NLP technique is its computational efficiency and scalability to large inputs. We measured the runtime of our biography-specific pipeline on the 20 test documents (average length 3,400 words) using a single Nvidia Titan X GPU. The average processing time per biography was 97 seconds, with the bulk of time spent in the initial NER tagging stage.

This efficiency could be further improved by using lighter weight NER models or parallelizing the processing of multiple biographies. The clustering and antecedent selection steps are comparatively very fast once the NER tags are provided.

However, the quadratic worst-case complexity of comparing all mention pairs for compatibility means that this approach would likely not scale to extremely long texts like full books without further optimizations. Investigating more efficient approaches drawing on recent advances in limited memory neural networks is an interesting direction for future work.

Conclusion and Future Directions

In this work, we presented a new approach for anaphora and co-reference resolution that is specifically designed to handle the challenges of the biography domain. By leveraging named entity recognition to identify potential antecedents and incorporating domain-specific constraints, the model is able to achieve state-of-the-art performance on a new biography dataset, outperforming strong NLP baselines.

There are several exciting directions for future research on this task:

1. Expanding to more entity types: Currently, the model only considers PER, ORG, and LOC entities. But biographies contain other relevant entity types like dates, events, and works of art that could provide useful cues for co-reference and anaphora. Expanding the NER component to recognize a wider ontology of entity types is an important next step.

2. Incorporating knowledge bases: Many of the entities mentioned in biographies have corresponding entries in knowledge bases like Wikipedia, IMDB, and Wikidata. These contain structured attributes (e.g. birth/death dates, aliases, works, relations to other entities) that could be incorporated into the resolution model to improve disambiguation. For example, knowing that Michelle Obama‘s husband is Barack Obama could help link the mention "the president" to him.

3. Generating co-reference-aware biography summaries: The co-reference chains provide a concise representation of the key entities in a biography. These could be used to generate more coherent extractive or abstractive summaries that maintain referential clarity. E.g. avoiding accidentally resolving "she" to the wrong person in the summary.

4. Cross-document co-reference: Many historical figures have multiple biographies written about them. Identifying co-referential entities across these different tellings could allow us to aggregate information and build more comprehensive structured models of the individuals.

5. Applying to other domains: While our model is tailored for biographies, the high-level approach of using NER to constrain anaphora/co-reference could likely be adapted to other domains like fictional narratives, legal documents, and scientific literature. Investigating the generalizability of the approach is an exciting frontier.

In summary, this work demonstrates the potential for combining deep neural language models (for NER) and symbolic domain knowledge (in the form of resolution constraints) to tackle challenging NLP problems in specific verticals. As AI systems are increasingly applied to niche domains, we believe such hybrid neuro-symbolic approaches will prove essential to achieving human-like natural language understanding.