SALSA: Single-pass Autoregressive LLM Structured Classification

2025/10/26 by Berdichevsky, Ruslan, Nahum-Gefen, Shai, Zaken, Elad Ben
#Computation and Language (cs.CL) #FOS: Computer and information sciences #Machine Learning (cs.LG)

paper · doi:10.48550/arxiv.2510.22691

Abstract

Despite their impressive generalization capabilities, instruction-tuned Large Language Models often underperform on text classification benchmarks. We introduce SALSA, a coherent pipeline that combines structured prompting, class-to-token mapping, and parameter-efficient fine-tuning, thereby avoiding cold-start training. Each class label is mapped to a distinct output token, and prompts are constructed to elicit a single-token response. During inference, the model's output is projected only onto the logits of the relevant class tokens, enabling efficient and accurate classification in a single forward pass. SALSA achieves state-of-the-art results across diverse benchmarks, demonstrating its robustness and scalability for LLM-based classification applications.

Citations

• Fine-Tuned 'Small' LLMs (Still) Significantly Outperform Zero-Shot Generative AI Models in Text Classification
• Capabilities of GPT-4 on Medical Challenge Problems
• LLaMA: Open and Efficient Foundation Language Models
• Bag of Tricks for Effective Language Model Pretraining and Downstream Adaptation: A Case Study on GLUE
• Leveraging Large Language Models for Multiple Choice Question Answering
• Finetuned Language Models Are Zero-Shot Learners
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models
• LoRA: Low-Rank Adaptation of Large Language Models
• Knowledgeable or Educated Guess? Revisiting Language Models as Knowledge Bases
• HateXplain: A Benchmark Dataset for Explainable Hate Speech Detection
• Language Models are Few-Shot Learners
• Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer
• ALBERT: A Lite BERT for Self-supervised Learning of Language Representations
• Language Models as Knowledge Bases?
• RoBERTa: A Robustly Optimized BERT Pretraining Approach
• XLNet: Generalized Autoregressive Pretraining for Language Understanding
• Lessons from Natural Language Inference in the Clinical Domain
• GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding
• Deep contextualized word representations
• Attention Is All You Need
• SQuAD: 100,000+ Questions for Machine Comprehension of Text
• Character-level Convolutional Networks for Text Classification
• A large annotated corpus for learning natural language inference
• Adam: A Method for Stochastic Optimization
• Convolutional Neural Networks for Sentence Classification
• Efficient Estimation of Word Representations in Vector Space
• BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding

Related

• Fine-Tuning Causal LLMs for Text Classification: Embedding-Based vs. Instruction-Based Approaches
• LLM-AutoDiff: Auto-Differentiate Any LLM Workflow
• SALSA: Speedy ASR-LLM Synchronous Aggregation
• Continuous Autoregressive Language Models
• Exploring the Hidden Capacity of LLMs for One-Step Text Generation