LongT5: Efficient Text-To-Text Transformer for Long Sequences

2021/12/15 by Mandy Guo, Joshua Ainslie, Guo, Mandy +11 · 16 citations
Computer Science · #Computation and Language (cs.CL) #FOS: Computer and information sciences #Natural Language Processing Techniques #Speech Recognition and Synthesis #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2112.07916

openalex publication_date 2021/12/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent work has shown that either (1) increasing the input length or (2) increasing model size can improve the performance of Transformer-based neural models. In this paper, we present a new model, called LongT5, with which we explore the effects of scaling both the input length and model size at the same time. Specifically, we integrated attention ideas from long-input transformers (ETC), and adopted pre-training strategies from summarization pre-training (PEGASUS) into the scalable T5 architecture. The result is a new attention mechanism we call \em Transient Global (TGlobal), which mimics ETC's local/global attention mechanism, but without requiring additional side-inputs. We are able to achieve state-of-the-art results on several summarization tasks and outperform the original T5 models on question answering tasks.

Cited by

• PAACE: A Plan-Aware Automated Agent Context Engineering Framework
• End-to-end Contrastive Language-Speech Pretraining Model For Long-form Spoken Question Answering
• Stress Testing Factual Consistency Metrics for Long-Document Summarization
• FlashEVA: Accelerating LLM inference via Efficient Attention
• Exploring the Utilities of the Rationales from Large Language Models to Enhance Automated Essay Scoring
• LTR-ICD: A Learning-to-Rank Approach for Automatic ICD Coding
• CAM: A Constructivist View of Agentic Memory for LLM-Based Reading Comprehension
• ResFormer: All-Time Reservoir Memory for Long Sequence Classification
• Learning to Summarize by Learning to Quiz: Adversarial Agentic Collaboration for Long Document Summarization
• CFDA & CLIP at TREC iKAT 2025: Enhancing Personalized Conversational Search via Query Reformulation and Rank Fusion
• HSGM: Hierarchical Segment-Graph Memory for Scalable Long-Text Semantics
• Reverse Browser: Vector-Image-to-Code Generator
• Privacy-Preserving Reasoning with Knowledge-Distilled Parametric Retrieval Augmented Generation
• Bidirectional Sparse Attention for Faster Video Diffusion Training
• DeCAL Tokenwise Compression
• Modality Agnostic Efficient Long Range Encoder

Related

• Investigating Efficiently Extending Transformers for Long Input Summarization
• mLongT5: A Multilingual and Efficient Text-To-Text Transformer for Longer Sequences
• CoLT5: Faster Long-Range Transformers with Conditional Computation
• LSG Attention: Extrapolation of pretrained Transformers to long sequences
• LongNet: Scaling Transformers to 1,000,000,000 Tokens