Exploiting Domain Knowledge via Grouped Weight Sharing with Application to Text Categorization

2017/02/08 by Ye Zhang, Matthew Lease, Zhang, Ye +4
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Text and Document Classification Technologies #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.1702.02535

This paper is accepted by ACL 2017

openalex publication_date 2017/02/08 · arxiv created 2017/04/25 · arxiv updated 2017/04/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

A fundamental advantage of neural models for NLP is their ability to learn representations from scratch. However, in practice this often means ignoring existing external linguistic resources, e.g., WordNet or domain specific ontologies such as the Unified Medical Language System (UMLS). We propose a general, novel method for exploiting such resources via weight sharing. Prior work on weight sharing in neural networks has considered it largely as a means of model compression. In contrast, we treat weight sharing as a flexible mechanism for incorporating prior knowledge into neural models. We show that this approach consistently yields improved performance on classification tasks compared to baseline strategies that do not exploit weight sharing.

Citations

• ADADELTA: An Adaptive Learning Rate Method
• Convolutional Neural Networks for Sentence Classification
• Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

Related

• Beyond Weight Tying: Learning Joint Input-Output Embeddings for Neural Machine Translation
• Reusing Weights in Subword-aware Neural Language Models
• Weight Squeezing: Reparameterization for Knowledge Transfer and Model Compression
• Training and Generating Neural Networks in Compressed Weight Space
• Distilling Word Embeddings: An Encoding Approach