Crowd-SFT: Crowdsourcing for LLM Alignment

2025/06/04 by Alex Sotiropoulos, Sotiropoulos, Alex, Sulyab Thottungal Valapu +7
Computer Science · Social Sciences · #Distributed #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC) #Machine Learning (cs.LG) #Mobile Crowdsensing and Crowdsourcing #Parallel #and Cluster Computing (cs.DC)

paper · pdf · doi:10.48550/arxiv.2506.04063

openalex publication_date 2025/06/04 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Large Language Models (LLMs) increasingly rely on Supervised Fine-Tuning (SFT) and Reinforcement Learning from Human Feedback (RLHF) to align model responses with human preferences. While RLHF employs a reinforcement learning approach with a separate reward model, SFT uses human-curated datasets for supervised learning. Both approaches traditionally depend on small, vetted groups of annotators, making them costly, prone to bias, and limited in scalability. We propose an open, crowd-sourced fine-tuning framework that addresses these limitations by enabling broader feedback collection for SFT without extensive annotator training. Our framework promotes incentive fairness via a point-based reward system correlated with Shapley values and guides model convergence through iterative model updates. Our multi-model selection framework demonstrates up to a 55% reduction in target distance over single-model selection, enabling subsequent experiments that validate our point-based reward mechanism's close alignment with Shapley values (a well-established method for attributing individual contributions) thereby supporting fair and scalable participation.

Citations

• Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference
• Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback
• Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback
• Training language models to follow instructions with human feedback
• Improving KernelSHAP: Practical Shapley Value Estimation via Linear Regression
• GoEmotions: A Dataset of Fine-Grained Emotions
• A Marketplace for Data: An Algorithmic Solution
• A Unified Approach to Interpreting Model Predictions
• Aligning Large Language Models with Human: A Survey

Related

• Aligning Large Language Models via Fine-grained Supervision
• Two Minds Better Than One: Collaborative Reward Modeling for LLM Alignment
• PluralLLM: Pluralistic Alignment in LLMs via Federated Learning
• RLTHF: Targeted Human Feedback for LLM Alignment
• Getting More Juice Out of the SFT Data: Reward Learning from Human Demonstration Improves SFT for LLM Alignment