2020/05/01 by Bhargavi Paranjape, Mandar Joshi, Paranjape, Bhargavi +7 · 2 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques #Text Readability and Simplification #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2005.00652
openalex publication_date 2020/05/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Decisions of complex language understanding models can be rationalized by\nlimiting their inputs to a relevant subsequence of the original text. A\nrationale should be as concise as possible without significantly degrading task\nperformance, but this balance can be difficult to achieve in practice. In this\npaper, we show that it is possible to better manage this trade-off by\noptimizing a bound on the Information Bottleneck (IB) objective. Our fully\nunsupervised approach jointly learns an explainer that predicts sparse binary\nmasks over sentences, and an end-task predictor that considers only the\nextracted rationale. Using IB, we derive a learning objective that allows\ndirect control of mask sparsity levels through a tunable sparse prior.\nExperiments on ERASER benchmark tasks demonstrate significant gains over\nnorm-minimization techniques for both task performance and agreement with human\nrationales. Furthermore, we find that in the semi-supervised setting, a modest\namount of gold rationales (25% of training examples) closes the gap with a\nmodel that uses the full input.\n