Is Grep All You Need? How Agent Harnesses Reshape Agentic Search
2026/05/14 by Sahil Sen, Akhil Kasturi, Elias Lumer +2 · 15 voices · 2 citations
#cs.CL
paper · pdf
Abstract
Recent advances in Large Language Model (LLM) agents have enabled complex agentic workflows where models autonomously retrieve information, call tools, and reason over large corpora to complete tasks on behalf of users. Despite the growing adoption of retrieval-augmented generation (RAG) in agentic search systems, existing literature lacks a systematic comparison of how retrieval strategy choice interacts with agent architecture and tool-calling paradigm. Important practical dimensions, including how tool outputs are presented to the model and how performance changes when searches must cope with more irrelevant surrounding text, remain under-explored in agent loops. This paper reports an empirical study organized into two experiments. Experiment 1 compares grep and vector retrieval on a 116-question sample from LongMemEval, using a custom agent harness (Chronos) and provider-native CLI harnesses (Claude Code, Codex, and Gemini CLI), for both inline tool results and file-based tool results that the model reads separately. Experiment 2 compares grep-only and vector-only retrieval while progressively mixing in additional unrelated conversation history, so that each query is embedded in more distracting material alongside the passages that matter. Across Chronos and the provider CLIs, grep generally yields higher accuracy than vector retrieval in our comparisons in experiment 1; at the same time, overall scores still depend strongly on which harness and tool-calling style is used, even when the underlying conversation data are the same.
Citations
Cited by
Discussions
- Is grep all you need? How agent harnesses reshape agentic search [hn, 164 points, 64 comments]
- [23/30] 233 Upvotes, 52 Comments, 2 Posts, arXiv:2605.15184 🆕Is Grep All You Need? How Agent Harnesses Reshape Agentic Search Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah [bsky, 4 points, 4 comments]
- everyone spent two years building vector retrieval into coding agents. new pwc paper tested grep against it across claude code, codex and gemini cli. plain grep won every single matchup. https://arxiv [bsky, 2 points, 1 comments]
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search https://arxiv.org/abs/2605.15184 https://news.ycombinator.com/item?id=48460863 [bsky, 0 points, 0 comments]
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search https://arxiv.org/abs/2605.15184 [bsky, 0 points, 0 comments]
- 📰 Is Grep All You Need? How Agent Harnesses Reshape Agentic Search 🔗 https://arxiv.org/abs/2605.15184 💬 Discuss on HN [bsky, 0 points, 0 comments]
- [2605.15184] Is Grep All You Need? How Agent Harnesses Reshape Agentic Search — Useful look at how “agent harnesses” change the shape of agentic search—less about fancy models, more about the tooling [bsky, 0 points, 0 comments]
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search https://arxiv.org/abs/2605.15184 (https://news.ycombinator.com/item?id=48460863) [bsky, 0 points, 0 comments]
- When grep beats vector retrieval 😂 arxiv.org/abs/2605.15184 [bsky, 0 points, 0 comments]
- New paper 'Is Grep All You Need?': across harnesses like Claude Code and Codex, plain grep beats vector retrieval for finding facts in long histories. Exact matches on names, paths and error strings b [bsky, 0 points, 0 comments]
- 📰 Agent Harnesses Reshape Agentic Search with Grep as a foundational tool for reshaping search capabilities. 🔗 https://arxiv.org/abs/2605.15184 #Tech #Dev [bsky, 0 points, 0 comments]
- なんかおもろそうなのあった “Is Grep All You Need? How Agent Harnesses Reshape Agentic Search” arxiv.org/abs/2605.15184 [bsky, 0 points, 0 comments]
- Grep beat vector search on every agent-model pair in a PwC study (Claude Code, Codex, Gemini CLI). For code, exactness beats similarity in meaning: the agent looks for a literal token, a function name [bsky, 0 points, 1 comments]
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search https:// arxiv.org/abs/2605.15184 # arxiv [mastodon, 0 points, 0 comments]
- Is Grep All You Need? How Agent Harnesses Reshape Agentic Search [bsky, 0 points, 0 comments]
Related