MachineLearningLM: Scaling Many-shot In-context Learning via Continued Pretraining

2025/09/08 by Haoyu Dong, Dong, Haoyu, Pengkun Zhang +6 · 3 citations
Computer Science · Medicine · #Artificial Intelligence (cs.AI) #Artificial Intelligence in Healthcare and Education #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Machine Learning in Healthcare

paper · pdf · doi:10.48550/arxiv.2509.06806

openalex publication_date 2025/09/08 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large language models (LLMs) possess broad world knowledge and strong general-purpose reasoning ability, yet they struggle to learn from many in-context examples on standard machine learning (ML) tasks, that is, to leverage many-shot demonstrations purely via in-context learning (ICL) without gradient descent. We introduce MachineLearningLM, a portable continued-pretraining framework that equips a general-purpose LLM with robust in-context ML capability while preserving its general knowledge and reasoning for broader chat workflows. Our pretraining procedure synthesizes ML tasks from millions of structural causal models (SCMs), spanning shot counts up to 1,024. We begin with a random-forest teacher, distilling tree-based decision strategies into the LLM to strengthen robustness in numerical modeling. All tasks are serialized with a token-efficient prompt, enabling 3x to 6x more examples per context window and delivering up to 50x amortized throughput via batch inference. Despite a modest setup (Qwen-2.5-7B-Instruct with LoRA rank 8), MachineLearningLM outperforms strong LLM baselines (e.g., GPT-5-mini) by an average of about 15% on out-of-distribution tabular classification across finance, physics, biology, and healthcare domains. It exhibits a striking many-shot scaling law: accuracy increases monotonically as in-context demonstrations grow from 8 to 1,024. Without any task-specific training, it attains random-forest-level accuracy across hundreds of shots. General chat capabilities, including knowledge and reasoning, are preserved: it achieves 75.4% on MMLU.

Citations

• SE-Agent: Self-Evolution Trajectory Optimization in Multi-Step Reasoning with LLM-Based Agents
• ML-Master: Towards AI-for-AI via Integration of Exploration and Reasoning
• Cache Me If You Can: How Many KVs Do You Need for Effective Long-Context LMs?
• Scaling Laws for Many-Shot In-Context Learning with Self-Generated Annotations
• TabICL: A Tabular Foundation Model for In-Context Learning on Large Data
• Scalable In-Context Learning on Tabular Data via Retrieval-Augmented Large Language Models
• DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning
• More is not always better? Enhancing Many-Shot In-Context Learning with Differentiated and Reweighting Objectives
• On Many-Shot In-Context Learning for Long-Context Evaluation
• TableGPT2: A Large Multimodal Model with Tabular Data Integration
• Why Does the Effective Context Length of LLMs Fall Short?
• Agent Workflow Memory
• SpreadsheetLLM: Encoding Spreadsheets for Large Language Models
• TALENT: A Tabular Analytics and Learning Toolbox
• A Closer Look at Deep Learning Methods on Tabular Datasets
• Large Scale Transfer Learning for Tabular Data via Language Modeling
• Many-Shot In-Context Learning in Multimodal Foundation Models
• Many-Shot In-Context Learning
• Elephants Never Forget: Memorization and Learning of Tabular Data in Large Language Models
• LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Models
• Tokenization counts: the impact of tokenization on arithmetic in frontier LLMs
• MLAgentBench: Evaluating Language Agents on Machine Learning Experimentation
• xVal: A Continuous Numerical Tokenization for Scientific Language Models
• Qwen Technical Report
• Effective Long-Context Scaling of Foundation Models
• BatchPrompt: Accomplish more with less
• Link-Context Learning for Multimodal LLMs
• Lost in the Middle: How Language Models Use Long Contexts
• Mitigating Label Biases for In-context Learning
• Voyager: An Open-Ended Embodied Agent with Large Language Models
• Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study
• What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
• Reflexion: Language Agents with Verbal Reinforcement Learning
• How Many Demonstrations Do You Need for In-context Learning?
• Batch Prompting: Efficient Inference with Large Language Model APIs
• Scaling Instruction-Finetuned Language Models
• TabLLM: Few-shot Classification of Tabular Data with Large Language Models
• TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second
• Flamingo: a Visual Language Model for Few-Shot Learning
• Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks
• Self-Consistency Improves Chain of Thought Reasoning in Language Models
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Benchmarking Multimodal AutoML for Tabular Data with Text Fields
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• What Makes Good In-Context Examples for GPT-3?
• Few-Shot Learning with Class Imbalance
• Language Models are Few-Shot Learners
• Do NLP Models Know Numbers? Probing Numeracy in Embeddings
• A Unified Approach to Interpreting Model Predictions
• "Why Should I Trust You?": Explaining the Predictions of Any Classifier
• Distilling the Knowledge in a Neural Network
• Interpreting Tree Ensembles with inTrees
• Towards Benchmarking Foundation Models for Tabular Data With Text
• Nearest neighbor pattern classification

Cited by

• Finch: Benchmarking Finance & Accounting across Spreadsheet-Centric Enterprise Workflows
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Talking Trees: Reasoning-Assisted Induction of Decision Trees for Tabular Data

Related

• Many-Shot In-Context Learning
• Will Pre-Training Ever End? A First Step Toward Next-Generation Foundation MLLMs via Self-Improving Systematic Cognition
• You Only Fine-tune Once: Many-Shot In-Context Fine-Tuning for Large Language Model
• LLM Post-Training: A Deep Dive into Reasoning Large Language Models
• MLLM-CTBench: A Benchmark for Continual Instruction Tuning with Reasoning Process Diagnosis