PolicyPad: Collaborative Prototyping of LLM Policies
2025/09/24 by Feng, K. J. Kevin, Kuo, Tzu-Sheng, Ze, Quan +4
#Artificial Intelligence (cs.AI) #FOS: Computer and information sciences #Human-Computer Interaction (cs.HC)
paper · doi:10.48550/arxiv.2509.19680
Abstract
As LLMs gain adoption in high-stakes domains like mental health, domain experts are increasingly consulted to provide input into policies governing their behavior. From an observation of 19 policymaking workshops with 9 experts over 15 weeks, we identified opportunities to better support rapid experimentation, feedback, and iteration for collaborative policy design processes. We present PolicyPad, an interactive system that facilitates the emerging practice of LLM policy prototyping by drawing from established UX prototyping practices, including heuristic evaluation and storyboarding. Using PolicyPad, policy designers can collaborate on drafting a policy in real time while independently testing policy-informed model behavior with usage scenarios. We evaluate PolicyPad through workshops with 8 groups of 22 domain experts in mental health and law, finding that PolicyPad enhanced collaborative dynamics during policy design, enabled tight feedback loops, and led to novel policy contributions. Overall, our work paves participatory paths for advancing AI alignment and safety.
Citations
- PersonaTeaming: Exploring How Introducing Personas Can Improve Automated AI Red-Teaming
- Statutory Construction and Interpretation for Artificial Intelligence
- Neither Valid nor Reliable? Investigating the Use of LLMs as Judges
- Secondary Stakeholders in AI: Fighting for, Brokering, and Navigating Agency
- Scenarios in Computing Research: A Systematic Review of the Use of Scenario Methods for Exploring the Future of Computing Technologies in Society
- Expressing stigma and inappropriate responses prevents LLMs from safely replacing mental health providers.
- LLM Social Simulations Are a Promising Research Method
- AutoRedTeamer: Autonomous Red Teaming with Lifelong Attack Integration
- No Free Labels: Limitations of LLM-as-a-Judge Without Human Grounding
- s1: Simple test-time scaling
- "Ownership, Not Just Happy Talk": Co-Designing a Participatory Large Language Model for Journalism
- Envisioning Stakeholder-Action Pairs to Mitigate Negative Impacts of AI: A Participatory Approach to Inform Policy Making
- OpenAI's Approach to External Red Teaming for AI Models and Systems
- WeAudit: Scaffolding User Auditors and AI Practitioners in Auditing Generative AI
- AI and the Future of Digital Public Squares
- Democratic AI is Possible. The Democracy Levels Framework Shows How It Might Work
- Venire: A Machine Learning-Guided Panel Review System for Community Content Moderation
- Limitations of the LLM-as-a-Judge Approach for Evaluating LLM Outputs in Expert Knowledge Tasks
- IdeaSynth: Iterative Research Idea Development Through Evolving and Composing Idea Facets with Literature-Grounded Feedback
- Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviors
- Roleplay-doh: Enabling Domain-Experts to Create LLM-simulated Patients via Eliciting and Adhering to Principles
- Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judges
- MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning
- BLIP: Facilitating the Exploration of Undesirable Consequences of Digital Technologies
- (A)I Am Not a Lawyer, But...: Engaging Legal Experts towards Responsible LLM Policies for Legal Advice
- Red-Teaming for Generative AI: Silver Bullet or Security Theater?
- Canvil: Designerly Adaptation for LLM-Powered User Experiences
- Two Types of AI Existential Risk: Decisive and Accumulative
- Case Repositories: Towards Case-Based Reasoning for AI Alignment
- Benefits and Harms of Large Language Models in Digital Mental Health
- Democratic Policy Development using Collective Dialogues and AI
- ConstitutionMaker: Interactively Critiquing Large Language Models by Converting Feedback into Principles
- Interactive AI Alignment: Specification, Process, and Evaluation Alignment
- Case Law Grounding: Using Precedents to Align Decision-Making for Humans and AI
- The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practice
- ChainForge: A Visual Toolkit for Prompt Engineering and LLM Hypothesis Testing
- Frontier AI Regulation: Managing Emerging Risks to Public Safety
- Explore, Establish, Exploit: Red Teaming Language Models from Scratch
- Searching for inclusive artificial intelligence for social good: Participatory governance and policy recommendations for making <scp>AI</scp> more inclusive and benign for society
- Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experience
- Affective Coherence Monitoring for Transformer-Based Language Models
- Training language models to follow instructions with human feedback
- Red Teaming Language Models with Language Models
- Decentralizing Platform Power: A Design Space of Multi-level Governance in Online Social Platforms
- On the Opportunities and Risks of Foundation Models
- Engaging Teachers to Co-Design Integrated AI Curriculum for K-12 Classrooms
- Participation is not a Design Fix for Machine Learning
- Artificial Intelligence, Values, and Alignment
- Build it Break it Fix it for Dialogue Safety: Robustness from\n Adversarial Human Attack
- Policy experimentation: core concepts, political dynamics, governance and impacts
- The forthcoming Artificial Intelligence (AI) revolution: Its impact on society and firms
- Balanced Policy Evaluation and Learning
- Examining Citizen Participation: Local Participatory Policy Making and Democracy
- Institutional Ecology, `Translations' and Boundary Objects: Amateurs and Professionals in Berkeley's Museum of Vertebrate Zoology, 1907-39
Related