2021/12/10 by Marjan Ghazvininejad, Vladimir Karpukhin, Ghazvininejad, Marjan +6 · 4 citations
Computer Science · Mathematics · #Artificial intelligence #Artificial neural network #Blocking (statistics) #Computer science #Function (biology) #Linguistics #Natural Language Processing Techniques #Natural language processing #Prefix #Set (abstract data type) #Softmax function #Speech Recognition and Synthesis #Text generation #Topic Modeling #cs.CL #cs.LG #stat.ML
paper · pdf · doi:10.48550/arxiv.2112.05717
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2021/12/10 · arxiv created 2022/05/23 · arxiv updated 2022/05/24 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Current efficient fine-tuning methods (e.g., adapters, prefix-tuning, etc.) have optimized conditional text generation via training a small set of extra parameters of the neural language model, while freezing the rest for efficiency. While showing strong performance on some generation tasks, they don't generalize across all generation tasks. We show that soft-prompt based conditional text generation can be improved with simple and efficient methods that simulate modeling the discourse structure of human written text. We investigate two design choices: First, we apply hierarchical blocking on the prefix parameters to simulate a higher-level discourse structure of human written text. Second, we apply attention sparsity on the prefix parameters at different layers of the network and learn sparse transformations on the softmax-function. We show that structured design of prefix parameters yields more coherent, faithful and relevant generations than the baseline prefix-tuning on all generation tasks.