LongT5: Efficient Text-To-Text Transformer for Long Sequences

2021/12/15 by Mandy Guo, Guo, Mandy, Joshua Ainslie +12 · 33 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling #cs.CL

paper · pdf · doi:10.48550/arxiv.2112.07916

Accepted in NAACL 2022

openalex publication_date 2021/12/15 · arxiv created 2022/05/03 · arxiv updated 2022/05/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call \em Transient Global (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

Cited by

• PAACE: A Plan-Aware Automated Agent Context Engineering Framework
• End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering
• Stress Testing Factual Consistency Metrics for Long-Document Summarization
• FlashEVA: Accelerating LLM inference via Efficient Attention
• Exploring the Utilities of the Rationales from Large Language Models to Enhance Automated Essay Scoring
• LTR-ICD: A Learning-to-Rank Approach for Automatic ICD Coding
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
• ResFormer: All-Time Reservoir Memory for Long Sequence Classification
• Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization
• CFDA & CLIP at TREC iKAT 2025: Enhancing Personalized Conversational Search via Query Reformulation and Rank Fusion
• HSGM: Hierarchical Segment-Graph Memory for Scalable Long-Text Semantics
• Reverse Browser: Vector-Image-to-Code Generator
• Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• DeCAL Tokenwise Compression
• Modality Agnostic Efficient Long Range Encoder
• KeyKnowledgeRAG (K²RAG): An Enhanced RAG method for improved LLM question-answering capabilities
• SAS: Simulated Attention Score
• Exploring LLM Capabilities in Extracting DCAT-Compatible Metadata for Data Cataloging
• Don't throw the baby out with the bathwater: How and why deep learning for ARC
• NexusSum: Hierarchical LLM Agents for Long-Form Narrative Summarization
• Beyond Chunking: Discourse-Aware Hierarchical Retrieval for Long Document Question Answering
• Requirements Coverage-Guided Minimization for Natural Language Test Cases
• SC4ANM: Identifying Optimal Section Combinations for Automated Novelty Prediction in Academic Papers
• VSA: Faster Video Diffusion with Trainable Sparse Attention
• Putting It All into Context: Simplifying Agents with LCLMs
• Signals from the Floods: AI-Driven Disaster Analysis through Multi-Source Data Fusion
• MateICL: Mitigating Attention Dispersion in Large-Scale In-Context Learning
• FlashAttention-4: Algorithm and Kernel Pipelining Co-Design for Asymmetric Hardware Scaling
• GLaMoR: Consistency Checking of OWL Ontologies using Graph Language Models
• Summarization of Multimodal Presentations with Vision-Language Models: Study of the Effect of Modalities and Structure
• Exploration of Plan-Guided Summarization for Narrative Texts: the Case of Small Language Models
• Single-Pass Document Scanning for Question Answering

Related

• Investigating Efficiently Extending Transformers for Long Input Summarization
• mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences
• CoLT5: Faster Long-Range Transformers with Conditional Computation
• LSG Attention: Extrapolation of pretrained Transformers to long sequences
• LongNet: Scaling Transformers to 1,000,000,000 Tokens