TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks

2024/12/18 by Frank F. Xu, Xu, Frank F., Yufan Song +43 · 26 voices · 64 citations
Computer Science · #Blockchain Technology Applications and Security

paper · pdf · doi:10.48550/arxiv.2412.14161

Abstract

We interact with computers on an everyday basis, be it in everyday life or work, and many aspects of work can be done entirely with access to a computer and the Internet. At the same time, thanks to improvements in large language models (LLMs), there has also been a rapid development in AI agents that interact with and affect change in their surrounding environments. But how performant are AI agents at accelerating or even autonomously performing work-related tasks? The answer to this question has important implications both for industry looking to adopt AI into their workflows and for economic policy to understand the effects that adoption of AI may have on the labor market. To measure the progress of these LLM agents' performance on performing real-world professional tasks, in this paper we introduce TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and communicating with other coworkers. We build a self-contained environment with internal web sites and data that mimics a small software company environment, and create a variety of tasks that may be performed by workers in such a company. We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that the most competitive agent can complete 30% of tasks autonomously. This paints a nuanced picture on task automation with LM agents--in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more difficult long-horizon tasks are still beyond the reach of current systems. We release code, data, environment, and experiments on https://the-agent-company.com.

Cited by

• Copy-on-Write Scoring: Application-Specific Agent Evaluations
• CEO-Bench: Can Agents Play the Long Game?
• SAGA: Scene-Aware, Goal-Evolving Agents for Long-Horizon Strategy Game Planning
• Frontier AI performance across the business disciplines: a case-grounded benchmark of knowledge work and analytical reasoning
• ClawBench: Can AI Agents Complete Everyday Online Tasks?
• Just A Rather Very Intelligent Spoken Agent
• Telco-GAIA: Bilingual Benchmark for Agents in Telecom Domain
• LLMs Corrupt Your Documents When You Delegate
• CRMArena-Pro: Holistic Assessment of LLM Agents Across Diverse Business Scenarios and Interactions
• How malicious AI swarms can threaten democracy
• Prismata: Confining Cross-Site Prompt Injection in Web Agents
• ACM: Agentic Context Management for Long Horizon Tasks
• E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
• SQBench: A Benchmark for Evaluating Task Delivery by Language-Model Agents in Production-Oriented Workflows
• LLMs for Agentic Home Energy Management
• HANDBOOK.md: A Benchmark for Long-Context Agentic Instruction Following
• Synthetic Scenario Generation for Evaluation of Industry 4.0 Agents
• Your plan may succeed, but what about failure? Investigating how people use ChatGPT for long-term life task planning
• Towards a Science of Scaling Agent Systems
• Can AI autonomously build, operate, and use the entire data stack?
• How Well Does Agent Development Reflect Real-World Work?
• Rethinking AI Evaluation in Education: The TEACH-AI Framework and Benchmark for Generative AI Assistants
• A Rosetta Stone for AI Benchmarks
• OpenApps: Simulating Environment Variations to Measure UI-Agent Reliability
• UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliability
• TAMAS: Benchmarking Adversarial Risks in Multi-Agent LLM Systems
• OptiMA: A Transaction-Based Framework with Throughput Optimization for Very Complex Multi-Agent Systems
• What's the next frontier for Data-centric AI? Data Savvy Agents
• Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environments
• The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution
• WorkSurface-Bench: Benchmarking Enterprise Agents on Multi-Surface Knowledge Routing
• Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
• NELSSA: A GPU-PNM Heterogeneous System for Mixed-Length LLM Serving via Length-based Request Placement
• OrgForge: A Multi-Agent Simulation Framework for Verifiable Synthetic Corporate Corpora
• CustomerSim: Benchmarking and Aligning Multimodal Language Models as Retail User Simulators
• OrchVis: Hierarchical Multi-Agent Orchestration for Human Oversight
• How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupations
• TheMCPCompany: Creating General-purpose Agents with Task-specific Tools
• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
• Orchestrating Human-AI Teams: The Manager Agent as a Unifying Research Challenge
• ReUseIt: Synthesizing Reusable AI Agent Workflows for Web Automation
• From Craft to Constitution: A Governance-First Paradigm for Principled Agent Engineering
• How can we assess human-agent interactions? Case studies in software agent design
• Simulating Teams with LLM Agents: Interactive 2D Environments for Studying Human-AI Dynamics
• When Should Users Check? A Decision-Theoretic Model of Confirmation Frequency in Multi-Step AI Agent Tasks
• LH-Deception: Simulating and Understanding LLM Deceptive Behaviors in Long-Horizon Interactions
• MacroBench: A Novel Testbed for Web Automation Scripts via Large Language Models
• Demystifying deep search: a holistic evaluation with hint-free multi-hop questions and factorised metrics
• A Framework for Studying AI Agent Behavior: Evidence from Consumer Choice Experiments
• SafeSearch: Automated Red-Teaming for the Safety of LLM-Based Search Agents
• Regulating the Agency of LLM-based Agents
• An Empirical Study of Coordination Mode as the First-Class Citizen in From-Scratch Multi-Agent Coding
• ClawTrack: Towards Trace-Level Evaluation and Improvement of Real-World Autonomous Agents
• Offline Preference-Based Trajectory Evaluation
• Context Is What You Need: The Maximum Effective Context Window for Real World Limits of LLMs
• RecoWorld: Building Simulated Environments for Agentic Recommender Systems
• KubeIntellect: A Modular LLM-Orchestrated Agent Framework for End-to-End Kubernetes Management
• ASIC-Agent: An Autonomous Multi-Agent System for ASIC Design with Benchmark Evaluation
• Reliable Weak-to-Strong Monitoring of LLM Agents
• Building and Measuring Trust between Large Language Models
• Analyzing Information Sharing and Coordination in Multi-Agent Planning
• OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflows
• VeriGUI: Verifiable Long-Chain GUI Dataset
• Large Language Model-based Data Science Agent: A Survey

Discussions

• AI agents fail tasks 70% of the time [hn, 23 points, 8 comments]
• Paper: arxiv.org/abs/2412.14161 [bsky, 13 points, 0 comments]
• CMU TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks [hn, 7 points, 0 comments]
• Study of AIAgents-driven Company: at best 24% of the tasks could be completed [hn, 4 points, 0 comments]
• Here is the paper: arxiv.org/pdf/2412.14161 [bsky, 3 points, 0 comments]
• "We test baseline agents powered by both closed API-based and open-weights language models (LMs), and find that with the most competitive agent, 24% of the tasks can be completed autonomously." arxiv [bsky, 3 points, 0 comments]
• From the CMU study "Benchmarking LLM Agents on Consequential Real World Tasks" "We find in experiments that the best-performing model, Gemini 2.5 Pro, was able to autonomously perform 30.3 percent of [bsky, 2 points, 0 comments]
• Interesting paper: arxiv.org/abs/2412.14161 The benchmark simulates a software company to evaluate current LLMs across different tasks like software development, admin, finance, and HR. [bsky, 2 points, 1 comments]
• CMU research paper - "agents" were able to achieve only a 30% success rate when given common, real-world office tasks. arxiv.org/pdf/2412.14161 [bsky, 2 points, 2 comments]
• Benchmarking LLM Agents on Consequential Real World Tasks [hn, 2 points, 0 comments]
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks [hn, 2 points, 0 comments]
• “…find that with the most competitive agent, 24% of the tasks can be completed autonomously. (..) simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more di [bsky, 1 points, 0 comments]
• 기사 출처는 여기: arxiv.org/abs/2412.14161. 인간과 에이전트의 효율적인 협업 방식이 뭔지도 아직 찾아볼 시간이 없었는데, 전자동 에이전트나이나 AGI를 너무 쉽게 이야기하는 것 아닌가 하는 의문이 있다... [bsky, 1 points, 1 comments]
• I'm not sure I had seen the study arxiv.org/pdf/2412.14161 or the benchmark the-agent-company.com#/leaderboard. that's pretty fun my take on these things is that they always prove the opposite of what [bsky, 1 points, 0 comments]
• some researchers at CMU found that an "AI agent" would think it needs to talk to Alice, be unable to find her, so it'd go into Slack and rename Bob to be Alice instead. by that logic, I think your fir [bsky, 1 points, 0 comments]
• Benchmarking LLM Agents on Consequential Real World Tasks arxiv.org/abs/2412.14161 [bsky, 0 points, 0 comments]
• State of the art agents are not that impressive (solving only 24% of the tasks): “in a setting simulating a real workplace, a good portion of simpler tasks could be solved autonomously, but more diffi [bsky, 0 points, 0 comments]
• "TheAgentCompany, an extensible benchmark for evaluating AI agents that interact with the world in similar ways to those of a digital worker: by browsing the Web, writing code, running programs, and c [bsky, 0 points, 0 comments]
• Les IA agentiques qui remplacent les employés à l'intérieur d'une entreprise, une technique pas si efficace ? arxiv.org/abs/2412.14161 [bsky, 0 points, 0 comments]
• This paper is a must-read for anyone considering implementing AI agents in real use cases: arxiv.org/pdf/2412.14161 Despite benchmarks demonstrating LLMs performing well on PhD-level tasks, they still [bsky, 0 points, 0 comments]
• EU cookie banners: humanity's last defense against AI! 🍪 So, you've heard the hype: 2025 is the year AI agents take over our jobs. But a recent study suggests we might keep our desks a bit longer. [bsky, 0 points, 1 comments]
• We're safe. For now. Sources: arxiv.org/abs/2412.141... youtube.com/watch?v=j3eQ... #AI #LLM #AgenticAI #arxiv #ML 4/4 [bsky, 0 points, 0 comments]
• TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks arxiv.org/pdf/2412.14161 [bsky, 0 points, 0 comments]
• AI cannot be evaluated simply by tracing human methods. For example, AI doesn't need things like web browser operations or UIs. This is not an appropriate test for the future work of AI. arxiv.org/abs [bsky, 0 points, 0 comments]
• Carnegie-Mellon research seems to confirm: LLMs often fail at long-horizon reasoning, social interactions, and tasks requiring genuine understanding. They perform noticeably better when tasks align wi [bsky, 0 points, 1 comments]
• LLMs truly pushing boundaries in methodology for task failed successfully https://arxiv.org/pdf/2412.14161 [bsky, 0 points, 1 comments]

Related

• Empowering Real-World: A Survey on the Technology, Practice, and Evaluation of LLM-driven Industry Agents
• Assessing the Performance of Human-Capable LLMs -- Are LLMs Coming for Your Job?
• WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?
• Fundamentals of Building Autonomous LLM Agents
• LLM-Powered AI Agent Systems and Their Applications in Industry