BloomIntent: Automating Search Evaluation with LLM-Generated Fine-Grained User Intents

2025/09/23 by Yoonseo Choi, Eunhye Kim, Choi, Yoonseo +11 · 2 citations
Computer Science · Social Sciences · #Ethics and Social Impacts of AI #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Information Retrieval (cs.IR) #Information Retrieval and Search Behavior #Multimodal Machine Learning Applications

paper · pdf · doi:10.48550/arxiv.2509.18641

openalex publication_date 2025/09/23 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

If 100 people issue the same search query, they may have 100 different goals. While existing work on user-centric AI evaluation highlights the importance of aligning systems with fine-grained user intents, current search evaluation methods struggle to represent and assess this diversity. We introduce BloomIntent, a user-centric search evaluation method that uses user intents as the evaluation unit. BloomIntent first generates a set of plausible, fine-grained search intents grounded on taxonomies of user attributes and information-seeking intent types. Then, BloomIntent provides an automated evaluation of search results against each intent powered by large language models. To support practical analysis, BloomIntent clusters semantically similar intents and summarizes evaluation outcomes in a structured interface. With three technical evaluations, we showed that BloomIntent generated fine-grained, evaluable, and realistic intents and produced scalable assessments of intent-level satisfaction that achieved 72% agreement with expert evaluators. In a case study (N=4), we showed that BloomIntent supported search specialists in identifying intents for ambiguous queries, uncovering underserved user needs, and discovering actionable insights for improving search experiences. By shifting from query-level to intent-level evaluation, BloomIntent reimagines how search systems can be assessed -- not only for performance but for their ability to serve a multitude of user goals.

Citations

• VeriLA: A Human-Centered Evaluation Framework for Interpretable Verification of LLM Agent Failures
• LLM-Powered User Simulator for Recommender System
• LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methods
• CoPrompter: User-Centric Evaluation of LLM Instruction Alignment for Improved Prompt Engineering
• JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking
• Why am I seeing this: Democratizing End User Auditing for Online Content Recommendations
• AIME: AI System Optimization via Multiple LLM Evaluators
• Comparing Criteria Development Across Domain Experts, Lay Users, and Models in Large Language Model Evaluation
• Understanding the User: An Intent-Based Ranking Dataset
• Proxona: Supporting Creators' Sensemaking and Ideation with LLM-Powered Audience Personas
• When Search Engine Services meet Large Language Models: Visions and Challenges
• Large Language Models for Relevance Judgment in Product Search
• Who Validates the Validators? Aligning LLM-Assisted Evaluation of LLM Outputs with Human Preferences
• USimAgent: Large Language Models for Simulating Search Users
• Cognitive Personalized Search Integrating Large Language Models with an Efficient Memory Mechanism
• Multi-Intent Attribute-Aware Text Matching in Searching
• SPADE: Synthesizing Data Quality Assertions for Large Language Model Pipelines
• MILL: Mutual Verification with Large Language Models for Zero-Shot Query Expansion
• Large language models can accurately predict searcher preferences
• ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
• Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
• Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community
• Large Language Models are not Fair Evaluators
• Can Large Language Models Be an Alternative to Human Evaluations?
• G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment
• Query2doc: Query Expansion with Large Language Models
• Contrastive Decoding: Open-ended Text Generation as Optimization
• Explaining Documents' Relevance to Search Queries
• IntenT5: Search Result Diversification using Causal Language Models
• Towards Controllable Biases in Language Generation
• BERTScore: Evaluating Text Generation with BERT
• Contextualized Evaluations: Judging Language Model Responses to Underspecified Queries

Cited by

• Flash-Fusion: Enabling Expressive, Low-Latency Queries on IoT Sensor Streams with LLMs
• Reporting and Reviewing LLM-Integrated Systems in HCI: Challenges and Considerations

Related

• Using Large Language Models to Generate, Validate, and Apply User Intent Taxonomies
• Understanding the User: An Intent-Based Ranking Dataset
• Role-Augmented Intent-Driven Generative Search Engine Optimization
• PeopleSearchBench: A Multi-Dimensional Benchmark for Evaluating AI-Powered People Search Platforms
• Search Arena: Analyzing Search-Augmented LLMs