# Word Sense Disambiguation: A Key Challenge for Natural Language Processing

- Canonical: https://33rdsquare.com/word-sense-disambiguation-importance-in-natural-language-processing/
- Published: 2024-09-03
- Author: Jordan Brown
- Categories: [Artificial Intelligence & Machine Learning & ChatGPT](https://33rdsquare.com/category/tech/ai/)

---

Word sense disambiguation (WSD) is a core challenge in natural language processing (NLP) that aims to identify the meaning of a word in a given context. WSD is essential for understanding the intended meaning of words and resolving lexical ambiguity. Despite being studied since the 1940s, WSD remains an active area of research with many open problems. In this post, we provide a comprehensive overview of WSD, covering its importance, main approaches, recent advances, and remaining challenges.

## The Challenge of Lexical Ambiguity

Lexical ambiguity is pervasive in natural language. Many words have multiple senses or meanings, depending on the context in which they appear. For example, the word "bank" can refer to a financial institution, the side of a river, or a collection of objects (e.g. "bank of switches"). The WordNet lexical database lists an average of 2.89 senses per word form for polysemous English nouns, 4.43 for adjectives, and 3.57 for verbs (Fellbaum, 1998). Some words like "run" have over 100 fine-grained senses.

This ambiguity poses a significant challenge for NLP systems. Incorrect or inconsistent sense disambiguation can drastically change the meaning of the text, leading to errors in downstream applications. For example, consider the following sentences:

1. I made a bank deposit before rushing to the bank to catch my flight.
2. The bat flew out of the cave.
3. The player gripped the bat tightly.

In (1), the first instance of "bank" refers to a financial institution, while the second refers to the shore of a river. In (2) and (3), "bat" refers to the animal and the sports equipment, respectively. A machine translation system must determine the correct sense of "bank" and "bat" to produce an appropriate translation into another language.

Disambiguation is also important for information retrieval, where different senses may be relevant for different queries (e.g. searching for documents about "jaguar" the animal vs. "jaguar" the car). In sentiment analysis, some senses of a word may be positive while others are negative (e.g. "the movie was terribly good"). Lexical ambiguity is one of the main sources of error for many NLP systems.

## Approaches to WSD

There are three main approaches to WSD: knowledge-based, supervised, and unsupervised.

### Knowledge-Based Approaches

Knowledge-based approaches leverage the information in lexical resources like dictionaries, thesauri, and ontologies. A pioneering example is the Lesk algorithm (Lesk, 1986), which disambiguates a word by comparing the dictionary definitions of each of its senses with the definitions of the surrounding words. The sense whose definition has the greatest overlap with the context words is selected.

Other knowledge-based approaches exploit the structure of semantic networks like WordNet (Miller, 1995). Graph-based methods represent words and senses as nodes in a graph, with edges encoding semantic relations like synonymy, hypernymy, and meronymy (Navigli & Lapata, 2010). Disambiguation is performed by propagating semantic similarity scores through the graph to select the most appropriate senses. Metalinguistic properties like domain, register, and collocation can also provide useful disambiguation cues.

However, knowledge-based methods are limited by the coverage and quality of underlying resources, which may be incomplete or inconsistent. They also struggle to capture contextual nuances and domain-specific sense distinctions.

### Supervised Approaches

Supervised approaches train machine learning models on datasets of manually sense-annotated examples. Early supervised methods used surface features like bag-of-words, collocations, and syntactic relations to train decision lists (Yarowsky, 1994), Naive Bayes classifiers (Mooney, 1996), and support vector machines (Lee et al., 2004). More recent work has explored richer linguistic features like word embeddings and syntax trees.

One of the most successful supervised approaches is It Makes Sense (IMS), an ensemble of linear classifiers trained on local and topical features (Zhong & Ng, 2010). IMS achieves up to 89% accuracy on the SemEval-2007 English all-words task. Other high-performing systems include latent logistic regression (Louhi & Schütze, 2014) and neural sequence models (Kågebäck & Salomonsson, 2016; Raganato et al., 2017).

The main limitation of supervised WSD is the knowledge acquisition bottleneck – the need for large amounts of costly hand-annotated training data. Existing sense-tagged corpora like SemCor (Miller et al., 1993) and OntoNotes (Pradhan et al., 2007) cover only a fraction of the English lexicon. Supervised models also struggle to generalize to new domains and unseen words.

### Unsupervised Approaches

Unsupervised WSD methods aim to induce word senses directly from raw text corpora, without relying on manual annotations. A common approach is to perform clustering of word occurrences based on co-occurrence statistics or distributional similarity (Pantel & Lin, 2002; Navigli, 2009). The induced clusters are then mapped to dictionary senses or used as targets for disambiguation.

Topic models like latent Dirichlet allocation (LDA) have also been used for WSD, by treating senses as latent variables (Cai et al., 2007). Sense-specific topic distributions are learned jointly with disambiguating target words. Graph-based methods like Personalized PageRank can be applied to derive sense-specific word embeddings from semantic networks (Pelevina et al., 2017).

While unsupervised methods offer greater flexibility and coverage, they typically achieve lower accuracy than supervised approaches. Evaluation is also challenging due to the lack of a gold standard sense inventory. Current state-of-the-art unsupervised WSD systems obtain around 70% F1-score on the SemEval benchmark (Pelevina et al., 2017).

## Recent Advances in Neural WSD

In recent years, the field of WSD has been transformed by the development of neural network models and deep learning. Neural language models can learn rich representations of words and contexts that capture both semantic and syntactic properties relevant to disambiguation.

Contextual word embeddings like ELMo (Peters et al., 2018) and BERT (Devlin et al., 2019) have achieved state-of-the-art results on WSD by pretraining on large unlabeled corpora. The contextual representations can then be fine-tuned on sense-annotated data for the target word inventory. Vial et al. (2019) use pretrained BERT embeddings as input to a multi-layer perceptron, obtaining 75.6% F1-score on the English all-words task.

Other approaches jointly learn context and sense embeddings using latent variable models. Loureiro et al. (2019) propose LMMS, a language model for word senses that learns to predict a dynamic sense distribution given the surrounding context. Scarlini et al. (2020) introduce SensEmBERT, which learns sense-specific contextualized embeddings on top of a pretrained BERT model.

Graph neural networks have also been applied to WSD, by encoding the structure of lexical knowledge bases. Bevilacqua & Navigli (2020) propose a system called EWISEwhich learns a synset embedding for each node in the WordNet graph hierarchy. Kumar et al. (2019) use graph convolutional networks to propagate disambiguation signals from seed examples.

Despite these advances, neural models still struggle with data sparsity, sense granularity, and reasoning about background knowledge. Integrating explicit knowledge representations with deep learning is an important direction for future research.

## Applications and Datasets

WSD is essential for many downstream NLP applications, including machine translation, information retrieval, text mining, and dialogue systems. Resolving lexical ambiguity enables more precise and coherent language understanding and generation.

Some key applications that have benefited from WSD include:

- Cross-lingual WSD for machine translation (Carpuat & Wu, 2007)
- Semantic indexing for information retrieval (Zhong & Ng, 2012)
- Aspect-based sentiment analysis (Schouten & Frasincar, 2018)
- Question answering (Ramakrishnan et al., 2003)
- Dialogue state tracking (Mrksic et al., 2017)
- Textual entailment and semantic similarity (Dagan et al., 2005)

To support WSD research, a number of benchmark datasets and shared tasks have been developed. The largest sense-annotated corpus for English WSD is SemCor (Miller et al., 1993), which contains over 200,000 words tagged with WordNet senses. The OMSTI corpus (Taghipour & Ng, 2015) provides an additional 30 million words of automatically sense-tagged text.

Other notable datasets include:

- Senseval/SemEval (Edmonds & Cotton, 2001; Snyder & Palmer, 2004; Pradhan et al., 2007)
- OntoNotes (Pradhan et al., 2007)
- Wikipedia sense annotations (Mihalcea, 2007)
- multilingual datasets like XL-WSD (Pasini & Navigli, 2020)

The SemEval/SenseEval shared tasks (Edmonds & Kilgarriff, 2002) have served as benchmarks for evaluating WSD systems across languages and domains. The all-words task evaluates disambiguation of all content words in running text, while the lexical sample task focuses on a pre-selected set of ambiguous words. Cross-lingual WSD (Lefever & Hoste, 2010) and domain-specific disambiguation (Agirre et al., 2010) have also been the targets of SemEval tasks.

## Current Challenges and Future Directions

Despite significant progress, WSD remains a challenging problem with many open research questions. One key issue is the sense granularity and subjectivity of manually-curated sense inventories like WordNet. Many fine-grained sense distinctions are difficult even for human annotators to consistently disambiguate (Erk et al., 2009). Integrating WSD systems into downstream applications often requires mapping to coarser-grained, domain-specific sense ontologies.

Another challenge is the limited coverage of existing sense-annotated corpora. Current datasets cover only a fraction of the lexicon and do not reflect the dynamic nature of word meanings, which can vary across domains, registers, and time periods. Developing methods for automatic sense discovery and adaptation is an important direction for future work (Lau et al., 2012).

WSD also requires deeper understanding and reasoning about commonsense knowledge. For example, disambiguating the senses of "bank" in "I made a bank deposit before rushing to the bank to catch my flight" requires knowledge about financial transactions, travel, and geography. Incorporating world knowledge and inference into WSD models, through knowledge bases, scripts, and frames, is a key challenge (Navigli & Velardi, 2005).

Finally, continued progress in WSD will require advances in learning from limited labeled data and transferring knowledge across languages and tasks. This includes few-shot learning (Brown et al., 2020), unsupervised cross-lingual alignment (Conneau et al., 2018), and multi-task architectures that jointly disambiguate words and learn semantic representations (Loureiro & Jorge, 2019).

## Conclusion

Word sense disambiguation is a fundamental challenge in natural language processing, with important implications for many downstream applications. WSD systems aim to identify the intended meaning of words in context, resolving lexical ambiguity and producing more precise semantic representations.

While WSD has been studied for many decades, recent advances in neural networks and deep learning have led to significant performance gains. Contextual word embeddings, graph neural networks, and joint semantic models have pushed the state-of-the-art on benchmark datasets.

However, many challenges remain, including data sparsity, sense granularity, and the integration of knowledge and reasoning. Ongoing research aims to develop more flexible, generalizable, and interpretable WSD models that can adapt to new domains and tasks.

As language technologies become more ubiquitous in our daily lives, the ability to accurately disambiguate and understand the meaning of words is more important than ever. With continued research and innovation, WSD will play a key role in building more intelligent, knowledgeable, and robust NLP systems to power the next generation of language-aware applications.

## References

- Agirre, E., & Edmonds, P. (2007). _Word Sense Disambiguation: Algorithms and Applications_. Springer.
- Bevilacqua, M., & Navigli, R. (2020). Breaking through the 80% glass ceiling: Raising the state of the art in Word Sense Disambiguation by incorporating knowledge graph information. _Proceedings of ACL 2020_.
- Brown, T. B., et al. (2020). Language Models are Few-Shot Learners. _arXiv preprint arXiv:2005.14165_.
- Conneau, A., et al. (2018). XNLI: Evaluating cross-lingual sentence representations. _Proceedings of EMNLP 2018_.
- Devlin, J., et al. (2019). BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. _Proceedings of NAACL 2019_.
- Edmonds, P., & Cotton, S. (2001). SENSEVAL-2: Overview. _Proceedings of SENSEVAL-2_.
- Erk, K., et al. (2009). Investigations on Word Senses and Word Usages. _Proceedings of ACL 2009_.
- Loureiro, D., & Jorge, A. (2019). Language modelling makes sense: Propagating representations through WordNet for full-coverage word sense disambiguation. _Proceedings of ACL 2019_.
- Miller, G. A. (1995). WordNet: A Lexical Database for English. _Communications of the ACM_.
- Navigli, R. (2009). Word Sense Disambiguation: A Survey. _ACM Computing Surveys_.
- Peters, M. E., et al. (2018). Deep Contextualized Word Representations. _Proceedings of NAACL 2018_.
- Raganato, A., et al. (2017). Neural sequence learning models for word sense disambiguation. _Proceedings of EMNLP 2017_.
- Scarlini, B., et al. (2020). SensEmBERT: Context-Enhanced Sense Embeddings for Multilingual Word Sense Disambiguation. _Proceedings of AAAI 2020_.
- Taghipour, K., & Ng, H. T. (2015). One million sense-tagged instances for word sense disambiguation and induction. _Proceedings of CoNLL 2015_.
- Zhong, Z., & Ng, H. T. (2010). It Makes Sense: A wide-coverage word sense disambiguation system for free text. _Proceedings of ACL 2010_.

---

Source: [Word Sense Disambiguation: A Key Challenge for Natural Language Processing](https://33rdsquare.com/word-sense-disambiguation-importance-in-natural-language-processing/)