2019/10/16 by David Donahue, David M. Donahue, Donahue, David +4 · 1 citation
Computer Science · Engineering · Mathematics · #Architecture #Art #Artificial intelligence #Computation #Computation and Language (cs.CL) #Computer science #Electrical engineering #Engineering #FOS: Computer and information sciences #Language model #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Multimodal Machine Learning Applications #Natural Language Processing Techniques #Natural language processing #Programming language #Topic Modeling #Transformer #Voltage #cs.CL #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.1910.10488
published in arXiv (Cornell University) (Cornell University) · 10 pages
openalex publication_date 2019/10/16 · arxiv created 2021/04/01 · arxiv updated 2021/04/05 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/05
The Transformer architecture has become increasingly popular over the past two years, owing to its impressive performance on a number of natural language processing (NLP) tasks. However, all Transformer computations occur at the level of word representations and therefore, it may be argued that Transformer models do not explicitly attempt to learn hierarchical structure which is widely assumed to be integral to language. In the present work, we introduce hierarchical processing into the Transformer model, taking inspiration from the U-Net architecture, popular in computer vision for its hierarchical view of natural images. We empirically demonstrate that the proposed architecture outperforms both the vanilla Transformer and some strong baselines in the domain of chit-chat dialogue.