OmniPred: Language Models as Universal Regressors

2024/02/22 by Xingyou Song, Oscar Li, Song, Xingyou +11 · 11 citations
Computer Science · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Databases (cs.DB) #FOS: Computer and information sciences #Machine Learning (cs.LG) #Natural Language Processing Techniques

paper · pdf · doi:10.48550/arxiv.2402.14547

openalex publication_date 2024/02/22 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Regression is a powerful tool to accurately predict the outcome metric of a system given a set of parameters, but has traditionally been restricted to methods which are only applicable to a specific task. In this paper, we propose OmniPred, a framework for training language models as universal end-to-end regressors over (x,y) data from arbitrary formats. Using data sourced from Google Vizier, one of the largest proprietary blackbox optimization databases in the world, our extensive experiments demonstrate that language models are capable of very precise numerical regression using only textual representations of mathematical parameters and values, and if given the opportunity to train at scale over multiple tasks, can significantly outperform traditional regression models.

Cited by

• LASER: Language Model Regression for Semi-Structured Workflow Resource and Runtime Estimation
• Beyond Token-level Supervision: Unlocking the Potential of Decoding-based Regression via Reinforcement Learning
• Towards Unified Vision Language Models for Forest Ecological Analysis in Earth Observation
• Interpreting LLMs as Credit Risk Classifiers: Do Their Feature Explanations Align with Classical ML?
• ZeroShotOpt: Towards Zero-Shot Pretrained Models for Efficient Black-Box Optimization
• Exploring the Synergy of Quantitative Factors and Newsflow Representations from Large Language Models for Stock Return Prediction
• Regression Language Models for Code
• Look Before you Leap: Estimating LLM Benchmark Scores from Descriptions
• RELISH: LLM REgression with a Latent Iterative State Head
• FoMEMO: Towards Foundation Models for Expensive Multi-objective Optimization
• Reasoning-Intensive Regression

Related

• From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
• Regression Language Models for Code
• Predictable Scale: Part II, Farseer: A Refined Scaling Law in Large Language Models
• Symbolic Regression with Multimodal Large Language Models and Kolmogorov Arnold Networks
• ViSymRe: Vision-guided Multimodal Symbolic Regression