Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?
2026/02/12 by Thibaud Gloaguen, Niels Mündler, Mark Müller +2 · 35 voices · 5 citations
#cs.SE #cs.AI
paper · pdf
Abstract
A widespread practice in software development is to tailor coding agents to repositories using context files, such as AGENTS.md. Although this practice is strongly encouraged by agent developers, there is currently no rigorous investigation into whether such context files are actually effective for real-world tasks. In this work, we study this question and evaluate coding agents' task completion performance in two complementary settings: established SWE-bench tasks from popular repositories, with LLM-generated context files, and a novel collection of issues from repositories containing developer-committed context files. Surprisingly, we find that providing context files does not generally improve task success rates, while increasing inference cost by over 20% on average. This observation holds across different LLMs, coding agents, and for both LLM-generated and developer-committed context files. Specifically, we find that while instructions in the context files are well followed by coding agents, repository overviews, although popular and recommended by model providers, are not helpful. We conclude that while context files are useful for specifying non-standard coding practices, any attempts to improve performance should be rigorously evaluated before deployment.
Citations
Cited by
Discussions
- Evaluating AGENTS.md: are they helpful for coding agents? [hn, 232 points, 161 comments]
- It turns out that AGENTS.md may be less than helpful. That's surprising. If you do provide them, write them by hand and make sure they're short and full of correct and critical information. [bsky, 36 points, 5 comments]
- Well this reads as quite the combo breaker, at least with agents powered by LLMs ~3 months ago, developer-submitted and LLM-created AGENT md files don't seem to improve task performance for python pro [bsky, 31 points, 4 comments]
- "Across multiple coding agents and LLMs, we find that context files tend to reduce task success rates compared to providing no repository context, while also increasing inference cost by over 20%" I'v [bsky, 10 points, 4 comments]
- lmao arxiv.org/abs/2602.11988 [bsky, 5 points, 1 comments]
- Surprising take in this paper: LLM-generated AGENTS md files hurt performance in most settings while increasing cost by 20%. arxiv.org/pdf/2602.11988 [bsky, 4 points, 1 comments]
- Files in repos like AGENTS dot md make results _worse_ and _more expensive_. Glad that me finding these files silly from the start is now supported by research showing they do more harm than good. arx [bsky, 4 points, 1 comments]
- (I asked Mike on LinkedIn.) Has anyone tried this, measured outcomes? Asking because: arxiv.org/abs/2602.11988 It found AGENTS.md “files tend to reduce task success rates compared to providing no rep [bsky, 3 points, 1 comments]
- Evaluating AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents?, by @veselinr.bsky.social and others: https://arxiv.org/abs/2602.11988 #studies #research #ai #aiagents #documentati [bsky, 3 points, 2 comments]
- Interesting: Many AGENTS files seem to be counterproductive and costly. arxiv.org/abs/2602.11988 [bsky, 3 points, 1 comments]
- You might be interested in this bit of research that found auto-generated files from init actually made the AI perform worse: [bsky, 2 points, 2 comments]
- arxiv.org/pdf/2602.11988 [bsky, 2 points, 1 comments]
- Lots of interesting findings about the effects of AGENTS.md files on coding agents in this paper. Surprisingly, there was very little lift from having one, and LLM-generated files significantly hurt p [bsky, 2 points, 1 comments]
- arxiv.org/abs/2602.11988 [bsky, 2 points, 0 comments]
- Actually funny timing as this got released some days ago: arxiv.org/pdf/2602.11988 [bsky, 1 points, 1 comments]
- Те, про що всі так багато говорять, але ви могли пропустити: Are Repository-Level Context Files Helpful for Coding #Agents? #llm [bsky, 1 points, 0 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? https://arxiv.org/abs/2602.11988 https://news.ycombinator.com/item?id=47034087 [bsky, 1 points, 0 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? https://arxiv.org/abs/2602.11988 [bsky, 1 points, 0 comments]
- ⚡ Hackernews Top story: Evaluating AGENTS.md: are they helpful for coding agents? [bsky, 1 points, 0 comments]
- Slightly related to the discussion, and interesting: arxiv.org/abs/2602.11988 [bsky, 1 points, 0 comments]
- apparently, llm-generated agent context files actually reduce task success rates. human-written ones have just marginal effect. ngl i never bothered to add these context files and did just fine. arxiv [bsky, 1 points, 0 comments]
- Evaluating #AGENTS.md: Are Repository-Level Context Files Helpful for Coding Agents? https://arxiv.org/abs/26... [bsky, 1 points, 0 comments]
- Wild. doi.org/10.48550/arX... [bsky, 0 points, 0 comments]
- ようやっとコンテキストファイルを作ろうとしていたのに、あんまり意味ないんでは？という話。しかし、せっかく仕組みがあるので書きたい気持も。先日VS Codeのプロンプトファイルを作っていたところ、良かれと思った1文が思わぬ方向に動いたような感じがあったので、確かにと思うところがある。 / arxiv.org/pdf/2602.11988 [bsky, 0 points, 0 comments]
- I built an open-source semantic router for Claude Code that loads only relevant rules per prompt instead of all of the [lemmy, 0 points, 0 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? https://arxiv.org/abs/2602.11988 (http://news.ycombinator.com/item?id=47034087) [bsky, 0 points, 0 comments]
- Notes on Evaluating AGENTS [dot] md: Are Repository-Level Context Files Helpful for Coding Agents? 🧵 The authors investigate AGENTS, CLAUDE, etc markdown files to attempt to measure their effectivene [bsky, 0 points, 1 comments]
- https://bsky.app/profile/hackernews.com.web.brid.gy/post/3mezzrw2kv6g2 [bsky, 0 points, 0 comments]
- A new research paper evaluating https://AGENTS.md files finds that they often fail to improve task success rates: https://arxiv.org/pdf/2602.11988 Discussion on HN: https://news.ycombinator.com/item? [bsky, 0 points, 1 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? https:// arxiv.org/abs/2602.11988 # arxiv [mastodon, 0 points, 0 comments]
- I came across this paper https://arxiv.org/abs/2602.11988 where authors evaluate coding agents ability to complete tasks with and without AGENTS.md files. The basic conclusion is that these files redu [bsky, 0 points, 1 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? View Article | Join the HN Conversation Summary of HN discussion 🧵👇 [bsky, 0 points, 1 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? https://arxiv.org/abs/2602.11988 https://news.ycombinator.com/item?id=47034087 [bsky, 0 points, 0 comments]
- Evaluating AGENTS.md: are they helpful for coding agents? #HackerNews https://arxiv.org/abs/2602.11988 [bsky, 0 points, 0 comments]
- I'd love to hear what @anthropic.com thinks about this preprint: arxiv.org/abs/2602.11988 [bsky, 0 points, 0 comments]
Related