Evaluating LLM-Based Mobile App Recommendations: An Empirical Study

2025/10/21 by Quim Motger, Motger, Quim, Xavier Franch +5
Computer Science · Health Professions · #AI in Service Interactions #FOS: Computer and information sciences #Information Retrieval (cs.IR) #Mobile Health and mHealth Applications #Software Engineering (cs.SE) #Spreadsheets and End-User Computing

paper · pdf · doi:10.48550/arxiv.2510.18364

openalex publication_date 2025/10/21 · openalex created_date 2025/10/24 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) are increasingly used to recommend mobile applications through natural language prompts, offering a flexible alternative to keyword-based app store search. Yet, the reasoning behind these recommendations remains opaque, raising questions about their consistency, explainability, and alignment with traditional App Store Optimization (ASO) metrics. In this paper, we present an empirical analysis of how widely-used general purpose LLMs generate, justify, and rank mobile app recommendations. Our contributions are: (i) a taxonomy of 16 generalizable ranking criteria elicited from LLM outputs; (ii) a systematic evaluation framework to analyse recommendation consistency and responsiveness to explicit ranking instructions; and (iii) a replication package to support reproducibility and future research on AI-based recommendation systems. Our findings reveal that LLMs rely on a broad yet fragmented set of ranking criteria, only partially aligned with standard ASO metrics. While top-ranked apps tend to be consistent across runs, variability increases with ranking depth and search specificity. LLMs exhibit varying sensitivity to explicit ranking instructions - ranging from substantial adaptations to near-identical outputs - highlighting their complex reasoning dynamics in conversational app discovery. Our results aim to support end-users, app developers, and recommender-systems researchers in navigating the emerging landscape of conversational app discovery.

Citations

• INTERPOS: Interaction Rhythm Guided Positional Morphing for Mobile App Recommender Systems
• Contextualizing Recommendation Explanations with LLMs: A User Study
• Optimizing Novelty of Top-k Recommendations using Large Language Models and Reinforcement Learning
• Data-efficient Fine-tuning for LLM-based Recommendation
• Tapping the Potential of Large Language Models as Recommender Systems: A Comprehensive Framework and Empirical Analysis
• RecRanker: Instruction Tuning Large Language Model as Ranker for Top-k Recommendation
• Conversational Recommender System and Large Language Model Are Made for Each Other in E-commerce Pre-sales Dialogue
• Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting
• Large Language Models for Software Engineering: A Systematic Literature Review
• Large Language Models for Software Engineering: A Systematic Literature Review
• Enhancing Recommender Systems with Large Language Model Reasoning Graphs
• A Survey on Large Language Models for Recommendation
• Large Language Models are Zero-Shot Rankers for Recommender Systems
• A deep multi-embedding model for mobile application recommendation
• Modeling Scale-free Graphs with Hyperbolic Geometry for Knowledge-aware Recommendation
• Context-Aware Target Apps Selection and Recommendation for Enhancing Personal Mobile Assistants
• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks
• Deep Item-based Collaborative Filtering for Top-N Recommendation
• Beyond Google Play: A Large-Scale Comparative Study of Chinese Android App Markets
• Towards Release Strategy Optimization for Apps in Google Play
• A similarity measure for indefinite rankings

Related

• Exploring Requirements Elicitation from App Store User Reviews Using Large Language Models
• MobileConvRec: A Conversational Dataset for Mobile Apps Recommendations
• LaQual: A Novel Framework for Automated Evaluation of LLM App Quality
• Structural Analysis of User Choices for Mobile App Recommendation
• LLMRS: Unlocking Potentials of LLM-Based Recommender Systems for Software Purchase