Gamma Mixture Modeling for Cosine Similarity in Small Language Models

2025/10/06 by Player, Kevin
#FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.05309

Abstract

We study the cosine similarity of sentence transformer embeddings and observe that they are well modeled by gamma mixtures. From a fixed corpus, we measure similarities between all document embeddings and a reference query embedding. Empirically we find that these distributions are often well captured by a gamma distribution shifted and truncated to [-1,1], and in many cases, by a gamma mixture. We propose a heuristic model in which a hierarchical clustering of topics naturally leads to a gamma-mixture structure in the similarity scores. Finally, we outline an expectation-maximization algorithm for fitting shifted gamma mixtures, which provides a practical tool for modeling similarity distributions.

Citations

• Benchmarking Transcriptomics Foundation Models for Perturbation Analysis : one PCA still rules them all
• On the distribution of cosine similarity with application to biology
• Transparency Helps Reveal When Language Models Learn Meaning
• Sense Embeddings are also Biased--Evaluating Social Biases in Static and Contextualised Sense Embeddings
• BERTopic: Neural topic modeling with a class-based TF-IDF procedure
• MPNet: Masked and Permuted Pre-training for Language Understanding
• MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformers
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• On Measuring Social Biases in Sentence Encoders
• Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional\n Neural Networks for Extreme Summarization
• Character-level Convolutional Networks for Text Classification

Related

• Solving Cosine Similarity Underestimation between High Frequency Words by L2 Norm Discounting
• Problems with Cosine as a Measure of Embedding Similarity for High Frequency Words
• DOCS: Quantifying Weight Similarity for Deeper Insights into Large Language Models
• Comparing in context: Improving cosine similarity measures with a metric tensor
• All Bark and No Bite: Rogue Dimensions in Transformer Language Models Obscure Representational Quality