SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks

2026/02/13 by Xiangyi Li, Yimin Liu, Wenbo Chen +75 · 27 voices · 24 citations
#cs.AI

paper · pdf

Abstract

Agent Skills are structured packages of procedural knowledge that augment large language model (LLM) agents at inference time. Despite rapid adoption, there is no standard way to measure whether they actually help. We present SkillsBench, a benchmark whose current inventory contains 87 tasks across 8 domains paired with curated Skills and deterministic verifiers. Our latest aggregate evaluation runs the 87-task benchmark under matched no-Skills and curated-Skills conditions for 18 model-harness configurations. Curated Skills raise the average pass rate from 33.9% to 50.5% (+16.6 percentage points; 25.5% normalized gain), with configuration-level gains ranging from +4.1 to +25.7 pp. Focused Skills with at most three modules outperform larger or exhaustive bundles, and smaller models with Skills can match larger models without them. SkillsBench establishes paired evaluation as the foundation for rigorous measurement of Skill efficacy on agentic, expertise-heavy work.

Citations

• ReplicationBench: Can AI Agents Replicate Astrophysics Research Papers?
• Language Agent Tree Search Unifies Reasoning Acting and Planning in Language Models
• Cognitive Architectures for Language Agents
• Tree of Thoughts: Deliberate Problem Solving with Large Language Models
• Toolformer: Language Models Can Teach Themselves to Use Tools
• Training language models to follow instructions with human feedback
• Program Synthesis with Large Language Models
• Language Models are Few-Shot Learners
• Between MDPs and semi-MDPs: A framework for temporal abstraction in reinforcement learning
• Interactive-engagement versus traditional methods: A six-thousand-student survey of mechanics test data for introductory physics courses

Cited by

• Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills
• AgentCompass: A Unified Evaluation Infrastructure for Agent Capabilities
• SkillSight: Calibrating Generic Content Bias for Skill Retrieval
• SkillRouter: Skill Routing for LLM Agents at Scale
• Pomona: Continuous Code Quality Improvement via Small, Agentic Pull Requests at Bloomberg
• The Tool Illusion: Rethinking Tool Use in Web Agents
• HydroAgent: Formalizing Forecaster Expertise into Skill-Orchestrated Flood Forecasting Workflows
• From Execution to Capability: Scientific Experience Consolidation via Procedural Knowledge Synthesis
• From Context to Skills: Can Language Models Learn from Context Skillfully?
• Agent Skills Matter: Inferring Proprietary Skills from Execution Trajectories
• Ratchet: A Minimal Hygiene Recipe for Self-Evolving LLM Agents
• Messier: A High-Resolution Corpus for Cross-Benchmark Agent Evaluation
• Living-Harness Is an Interactive-Agent Evolver
• SecRespond: Benchmarking AI Agents for Real-World Post-Compromise Incident Response
• GenDB: The Next Generation of Query Processing -- Synthesized, Not Engineered
• SkillCAT: Contrastive, Assessment-Augmented and Topology-AwareSkill Self-Evolution for LLM Agents
• Library Drift: Diagnosing and Fixing a Silent Failure Mode in Self-Evolving LLM Skill Libraries
• SkillOpt: Executive Strategy for Self-Evolving Agent Skills
• Diagon: A Programmable Testbed for AI-Agent Cognitive Labor Markets
• MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
• SkillOS: Learning Skill Curation for Self-Evolving Agents
• Who Grades the Grader? Co-Evolving Evaluation Metrics and Skills for Self-Improving LLM Agents
• VeriSkill: A Self-Evolution Framework for Program Verification Skills
• Skill Use or Skill Theater? Evaluating the Reasoning Backroom in Skill-Augmented Language Agents

Discussions

• SkillsBench: Benchmarking how well agent skills work across diverse tasks [hn, 364 points, 171 comments]
• arxiv.org/abs/2602.12670 This is why hexdocs.pm/usage_rules exists. Package authors need to be providing the context that will guide your agent to successfully use their package. As their library chan [bsky, 14 points, 2 comments]
• https://bsky.app/profile/ruoshuiresearch.bsky.social/post/3mg27twhxfc2n [bsky, 3 points, 1 comments]
• https://bsky.app/profile/tommis.fi/post/3mfixwvfyzc2a [bsky, 2 points, 0 comments]
• SkillsBench: Benchmarking How Well Agent #Skills Work Across Diverse Tasks https://arxiv.org/abs/26... #LLM #ClaudeCode [bsky, 1 points, 0 comments]
• Study: Self-generated Agent Skills are useless [bsky, 1 points, 0 comments]
• Skills benchmark and evals, main contributions are 1. skills benchmark 2. self-generated Skills are not so great arxiv.org/pdf/2602.12670 [bsky, 1 points, 1 comments]
• Study: Self-generated Agent Skills are useless #HackerNews https://arxiv.org/abs/2602.12670 [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 https://news.ycombinator.com/item?id=47040430 [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 https://news.ycombinator.com/item?id=47040430 [bsky, 0 points, 0 comments]
• SkillsBench: Benchmarking how well agent skills work across diverse tasks https://arxiv.org/abs/2602.12670 comments #arxiv.org [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 [bsky, 0 points, 0 comments]
• Similar take for skills arxiv.org/pdf/2602.12670 [bsky, 0 points, 0 comments]
• 📰 Study: Self-generated Agent Skills are useless 🔗 https://arxiv.org/abs/2602.12670 💬 Discuss on HN [bsky, 0 points, 0 comments]
• Paper: arxiv.org/pdf/2602.12670 [bsky, 0 points, 0 comments]
• ⚡ Hackernews Top story: Study: Self-generated Agent Skills are useless [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 (http://news.ycombinator.com/item?id=47040430) [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 (http://news.ycombinator.com/item?id=47040430) [bsky, 0 points, 0 comments]
• https://arxiv.org/abs/2602.12670 SkillsBenchは、AIエージェントの能力を測る新しいベンチマークです。 多様なタスクにおけるエージェントのスキル性能を評価します。 AIエージェントの汎用性を理解する上で重要な指標を提供します。 [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless view on hacker news [bsky, 0 points, 0 comments]
• SkillsBench: Benchmarking how well agent skills work across diverse tasks https:// arxiv.org/abs/2602.12670 # arxiv [mastodon, 0 points, 0 comments]
• SkillsBench: Benchmarking how well agent skills work across diverse tasks View Article | Join the HN Conversation Summary of HN discussion 🧵👇 [bsky, 0 points, 1 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 (https://news.ycombinator.com/item?id=47040430) [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 (https://news.ycombinator.com/item?id=47040430) [bsky, 0 points, 0 comments]
• Study: Self-generated Agent Skills are useless https://arxiv.org/abs/2602.12670 (https://news.ycombinator.com/item?id=47040430) [bsky, 0 points, 0 comments]
• https://bsky.app/profile/buzzing.cc.web.brid.gy/post/3mezbkpy4bil2 [bsky, 0 points, 0 comments]
• 1/2 Estudio: las habilidades auto-generadas por agentes no sirven Un paper de arXiv analiza si los agentes que se crean sus propias habilidades mejoran realmente el rendimiento. La discusión en Hacker [bsky, 0 points, 1 comments]

Related

• SkillCorpus: Consolidating and Evaluating the Open Skill Ecosystem for Real-World LLM Agents
• APTBench: Benchmarking Agentic Potential of Base LLMs During Pre-Training
• AgentBench: Evaluating LLMs as Agents
• MMAU: A Holistic Benchmark of Agent Capabilities Across Diverse Domains
• SkillOpt: Executive Strategy for Self-Evolving Agent Skills