STEER: Assessing the Economic Rationality of Large Language Models

2024/02/14 by Narun Raman, Raman, Narun, Taylor Lundy +9 · 11 citations
Computer Science · Social Sciences · #Artificial Intelligence in Law #Computation and Language (cs.CL) #Computer science #Economics #Epistemology #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Economics and business #General Economics (econ.GN) #Mathematical economics #Philosophy #Positive economics #Rationality #Topic Modeling

paper · pdf · doi:10.48550/arxiv.2402.09552

published in arXiv (Cornell University) (Cornell University)

openalex publication_date 2024/02/14 · openalex created_date 2024/02/18 · openalex updated_date 2026/07/28

Abstract

There is increasing interest in using LLMs as decision-making "agents." Doing so includes many degrees of freedom: which model should be used; how should it be prompted; should it be asked to introspect, conduct chain-of-thought reasoning, etc? Settling these questions -- and more broadly, determining whether an LLM agent is reliable enough to be trusted -- requires a methodology for assessing such an agent's economic rationality. In this paper, we provide one. We begin by surveying the economic literature on rational decision making, taxonomizing a large set of fine-grained "elements" that an agent should exhibit, along with dependencies between them. We then propose a benchmark distribution that quantitatively scores an LLMs performance on these elements and, combined with a user-provided rubric, produces a "STEER report card." Finally, we describe the results of a large-scale empirical experiment with 14 different LLMs, characterizing the both current state of the art and the impact of different model sizes on models' ability to exhibit rational behavior.

Cited by

• Magentic Marketplace: An Open-Source Environment for Studying Agentic Markets
• RLRF: Competitive Search Agent Design via Reinforcement Learning from Ranker Feedback
• ManagerBench: Evaluating the Safety-Pragmatism Trade-off in Autonomous LLMs
• Strategic Tradeoffs Between Humans and AI in Multi-Agent Bargaining
• Evaluating the Ability of Large Language Models to Reason about Cardinal Directions, Revisited
• Cash or Comfort? How LLMs Value Your Inconvenience
• AI Agent Behavioral Science
• Reasoning Like an Economist: Post-Training on Economic Problems Induces Strategic Generalization in LLMs
• Is Your LLM Overcharging You? Tokenization, Transparency, and Incentives
• Can LLMs Be CEOs? Benchmarking Strategic Resource Reallocation with Multi-Role Agent Simulation
• Revealed Rationality: Label-Free Evaluation and Regularization from Representation Theorems

Related

• STEER-ME: Assessing the Microeconomic Reasoning of Large Language Models
• LLM economicus? Mapping the Behavioral Biases of LLMs via Utility Theory
• Noise, Adaptation, and Strategy: Assessing LLM Fidelity in Decision-Making
• Rationality Check! Benchmarking the Rationality of Large Language Models
• Advances in LLMs with Focus on Reasoning, Adaptability, Efficiency and Ethics