WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representations

2018/08/28 by Mohammad Taher Pilehvar, Pilehvar, Mohammad Taher, José Camacho-Collados +1 · 25 citations
Computer Science · Social Sciences · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Misinformation and Its Impacts #Sentiment Analysis and Opinion Mining #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1808.09121

openalex publication_date 2018/08/28 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

By design, word embeddings are unable to model the dynamic nature of words' semantics, i.e., the property of words to correspond to potentially different meanings. To address this limitation, dozens of specialized meaning representation techniques such as sense or contextualized embeddings have been proposed. However, despite the popularity of research on this topic, very few evaluation benchmarks exist that specifically focus on the dynamic semantics of words. In this paper we show that existing models have surpassed the performance ceiling of the standard evaluation dataset for the purpose, i.e., Stanford Contextual Word Similarity, and highlight its shortcomings. To address the lack of a suitable benchmark, we put forward a large-scale Word in Context dataset, called WiC, based on annotations curated by experts, for generic evaluation of context-sensitive representations. WiC is released in https://pilehvar.github.io/wic/.

Citations

• Efficient Estimation of Word Representations in Vector Space
• BabelNet: The automatic construction, evaluation and application of a wide-coverage multilingual semantic network

Cited by

• MAGIC: Achieving Superior Model Merging via Magnitude Calibration
• PEFT-Factory: Unified Parameter-Efficient Fine-Tuning of Autoregressive Large Language Models
• PEFT-Bench: A Parameter-Efficient Fine-Tuning Methods Benchmark
• What Drives Cross-lingual Ranking? Retrieval Approaches with Multilingual Language Models
• MoodBench 1.0: An Evaluation Benchmark for Emotional Companionship Dialogue Systems
• ViConBERT: Context-Gloss Aligned Vietnamese Word Embedding for Polysemous and Sense-Aware Representations
• ConMeZO: Adaptive Descent-Direction Sampling for Gradient-Free Finetuning of Large Language Models
• Charting the European LLM Benchmarking Landscape: A New Taxonomy and a Set of Best Practices
• First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Training
• Flip-Flop Consistency: Unsupervised Training for Robustness to Prompt Perturbations in LLMs
• Learning to Look at the Other Side: A Semantic Probing Study of Word Embeddings in LLMs with Enabled Bidirectional Attention
• Dual-Head Reasoning Distillation: Improving Classifier Accuracy with Train-Time-Only Reasoning
• TASO: Task-Aligned Sparse Optimization for Parameter-Efficient Model Adaptation
• Can a Fruit Fly Learn Word Embeddings?
• Do Large Language Models Understand Word Senses?
• CTCC: A Robust and Stealthy Fingerprinting Framework for Large Language Models via Cross-Turn Contextual Correlation Backdoor
• Language Models are Few-Shot Learners
• EverTracer: Hunting Stolen Large Language Models via Stealthy and Robust Probabilistic Fingerprint
• Towards Fundamental Language Models: Does Linguistic Competence Scale with Model Size?
• MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper
• Unlocking the Effectiveness of LoRA-FP for Seamless Transfer Implantation of Fingerprints in Downstream Models
• Enabling MoE on the Edge via Importance-Driven Expert Scheduling
• Copyright Protection for Large Language Models: A Survey of Methods, Challenges, and Trends
• CALE : Concept-Aligned Embeddings for Both Within-Lemma and Inter-Lemma Sense Differentiation
• FAIR-Pruner: Leveraging Tolerance of Difference for Flexible Automatic Layer-Wise Neural Network Pruning
• FPEdit: Robust LLM Fingerprinting through Localized Parameter Editing

Related

• XL-WiC: A Multilingual Benchmark for Evaluating Semantic\n Contextualization
• Learning Word Embeddings from Intrinsic and Extrinsic Views
• WiC = TSV = WSD: On the Equivalence of Three Semantic Tasks
• Lost in Context? On the Sense-wise Variance of Contextualized Word Embeddings
• WiC-TSV: An Evaluation Benchmark for Target Sense Verification of Words in Context