Text Segmentation based on Semantic Word Embeddings

2015/03/18 by Alexander A Alemi, Paul Ginsparg, Alemi, Alexander A +1 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Information Retrieval (cs.IR) #cs.CL #cs.IR

paper · pdf · doi:10.48550/arxiv.1503.05543

10 pages, 4 figures. KDD2015 submission

arxiv created 2015/03/18 · arxiv updated 2015/03/19

Abstract

We explore the use of semantic word embeddings in text segmentation algorithms, including the C99 segmentation algorithm and new algorithms inspired by the distributed word vector representation. By developing a general framework for discussing a class of segmentation objectives, we study the effectiveness of greedy versus exact optimization approaches and suggest a new iterative refinement technique for improving the performance of greedy strategies. We compare our results to known benchmarks, using known metrics. We demonstrate state-of-the-art performance for an untrained method with our Content Vector Segmentation (CVS) on the Choi test set. Finally, we apply the segmentation procedure to an in-the-wild dataset consisting of text extracted from scholarly articles in the arXiv.org database.

Cited by

• The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation
• SemChunk-C: Semantic Segmentation for C Code

Related

• Scaling up Multi-domain Semantic Segmentation with Sentence Embeddings
• Text Segmentation as a Supervised Learning Task
• A Survey of Semantic Segmentation
• Image-Text Co-Decomposition for Text-Supervised Semantic Segmentation
• Associating Spatially-Consistent Grouping with Text-supervised Semantic Segmentation