A Deep Network with Visual Text Composition Behavior

2017/07/05 by Hongyu Guo, Guo, Hongyu
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #FOS: Computer and information sciences #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Neural and Evolutionary Computing (cs.NE) #Topic Modeling

paper · pdf · doi:10.48550/arxiv.1707.01555

openalex publication_date 2017/07/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

While natural languages are compositional, how state-of-the-art neural models achieve compositionality is still unclear. We propose a deep network, which not only achieves competitive accuracy for text classification, but also exhibits compositional behavior. That is, while creating hierarchical representations of a piece of text, such as a sentence, the lower layers of the network distribute their layer-specific attention weights to individual words. In contrast, the higher layers compose meaningful phrases and clauses, whose lengths increase as the networks get deeper until fully composing the sentence.

Citations

• Visualizing and Understanding Neural Models in NLP
• Generating Text with Deep Reinforcement Learning
• Learning to Forget: Continual Prediction with LSTM
• Understanding Neural Networks through Representation Erasure

Related

• Deep neural networks and humans both benefit from compositional language structure
• From Frege to chatGPT: Compositionality in language, cognition, and deep neural networks
• Neural networks can understand compositional functions that humans do\n not, in the context of emergent communication
• Compositionality Through Language Transmission, using Artificial Neural Networks
• Break It Down: Evidence for Structural Compositionality in Neural Networks