Source Coverage and Citation Bias in LLM-based vs. Traditional Search Engines

2025/12/10 by Peixian Zhang, Zhang, Peixian, Qiming Ye +7 · 1 citation
Computer Science · #Computation and Language (cs.CL) #Computers and Society (cs.CY) #Expert finding and Q&A systems #FOS: Computer and information sciences #Information Retrieval and Search Behavior #Web visibility and informetrics

paper · pdf · doi:10.48550/arxiv.2512.09483

openalex publication_date 2025/12/10 · openalex created_date 2025/12/12 · openalex updated_date 2026/07/28

Abstract

LLM-based Search Engines (LLM-SEs) introduces a new paradigm for information seeking. Unlike Traditional Search Engines (TSEs) (e.g., Google), these systems summarize results, often providing limited citation transparency. The implications of this shift remain largely unexplored, yet raises key questions regarding trust and transparency. In this paper, we present a large-scale empirical study of LLM-SEs, analyzing 55,936 queries and the corresponding search results across six LLM-SEs and two TSEs. We confirm that LLM-SEs cites domain resources with greater diversity than TSEs. Indeed, 37% of domains are unique to LLM-SEs. However, certain risks still persist: LLM-SEs do not outperform TSEs in credibility, political neutrality and safety metrics. Finally, to understand the selection criteria of LLM-SEs, we perform a feature-based analysis to identify key factors influencing source choice. Our findings provide actionable insights for end users, website owners, and developers.

Citations

• Social and Political Framing in Search Engine Results
• Unsafe LLM-Based Search: Quantitative Analysis and Mitigation of Safety Risks in AI Web Search
• Citations and Trust in LLM Generated Responses
• Decoding News Bias: Multi Bias Detection in News Articles
• Human and LLM Biases in Hate Speech Annotations: A Socio-Demographic Analysis of Annotators and Targets
• CrAM: Credibility-Aware Attention Modification in LLMs for Combating Misinformation in RAG
• Humans or LLMs as the Judge? A Study on Judgement Biases
• Navigating the Post-API Dilemma | Search Engine Results Pages Present a Biased View of Social Media Data
• Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios
• A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions
• DELPHI: Data for Evaluating LLMs' Performance in Handling Controversial Issues
• A Comprehensive Overview of Large Language Models
• Accuracy and Political Bias of News Source Credibility Ratings by Large Language Models
• Interleaving Retrieval with Chain-of-Thought Reasoning for Knowledge-Intensive Multi-Step Questions
• Measuring and Narrowing the Compositionality Gap in Language Models
• Distributing Accountability, Not Capability: Phase Separation and the LLM Workflow Quadrant in Autonomous AI Agent Architectures
• A Large Scale Study and Classification of VirusTotal Reports on Phishing and Malware URLs
• Least-to-Most Prompting Enables Complex Reasoning in Large Language Models
• Teaching language models to support answers with verified quotes
• Echo chambers and information transmission biases in homophilic and heterophilic networks
• WebGPT: Browser-assisted question-answering with human feedback
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks
• A Unified Approach to Interpreting Model Predictions
• Measuring Personalization of Web Search
• The Measurement of Observer Agreement for Categorical Data

Cited by

• Synthetic Sources?: Auditing Generative Search Engine Citations for Evidence of AI-Generated Sources

Related

• The Attribution Crisis in LLM Search Results
• The Extractive-Abstractive Spectrum: Uncovering Verifiability Trade-offs in LLM Generations
• Comparing Traditional and LLM-based Search for Consumer Choice: A Randomized Experiment
• Large Language Models vs. Search Engines: Evaluating User Preferences Across Varied Information Retrieval Scenarios
• Search Arena: Analyzing Search-Augmented LLMs