AI auditing: The Broken Bus on the Road to AI Accountability

2024/01/25 by Abeba Birhane, Ryan Steed, Birhane, Abeba +7 · 2 voices · 20 citations
Computer Science · Social Sciences · #Computers and Society (cs.CY) #Ethics and Social Impacts of AI #FOS: Computer and information sciences #cs.CY

paper · pdf · doi:10.48550/arxiv.2401.14462

openalex publication_date 2024/01/25 · arxiv published 2024/01/25 · arxiv updated 2024/01/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

One of the most concrete measures to take towards meaningful AI accountability is to consequentially assess and report the systems' performance and impact. However, the practical nature of the "AI audit" ecosystem is muddled and imprecise, making it difficult to work through various concepts and map out the stakeholders involved in the practice. First, we taxonomize current AI audit practices as completed by regulators, law firms, civil society, journalism, academia, consulting agencies. Next, we assess the impact of audits done by stakeholders within each domain. We find that only a subset of AI audit studies translate to desired accountability outcomes. We thus assess and isolate practices necessary for effective AI audit results, articulating the observed connections between AI audit design, methodology and institutional context on its effectiveness as a meaningful mechanism for accountability.

Cited by

• Generative Artificial Intelligence in Scientific Research: Individual Benefits, Collective Risks, and a Framework for Responsible Research with AI
• Prompt Governance? On Governing Technologies Governed by Natural Language
• Seeing Twice: How Side-by-Side T2I Comparison Changes Auditing Strategies
• Can AI be Accountable?
• Vipera: Blending Visual and LLM-Driven Guidance for Systematic Auditing of Text-to-Image Generative AI
• Accountability Capture: How Record-Keeping to Support AI Transparency and Accountability (Re)shapes Algorithmic Oversight
• Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lens
• AI Agent Architecture for Decentralized Trading of Alternative Assets
• The Singapore Consensus on Global AI Safety Research Priorities
• What Comes After Harm? Mapping Reparative Actions in AI through Justice Frameworks
• Robust ML Auditing using Prior Knowledge
• Say It Another Way: Auditing LLMs with a User-Grounded Automated Paraphrasing Framework
• The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating It
• The Accountability Paradox: How Platform API Restrictions Undermine AI Transparency Mandates
• MADP: A Multi-Agent Pipeline for Sustainable Document Processing with Human-in-the-Loop
• Who Controls the Conversation? User Perspectives On Generative AI (LLM) System Prompts
• All Eyes on the Ranker: Participatory Auditing to Surface Blind Spots in Ranked Search Results
• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• A Chain Is Only as Strong as Its Weakest Link: A Scoping Review of System Integration Audits in AI
• Audit Cards: Contextualizing AI Evaluations

Discussions

• AI audits are a mess arxiv.org/abs/2401.14462 [bsky, 1 points, 0 comments]
• AI Auditing: The Broken Bus on the Road to AI Accountability [hn, 1 points, 1 comments]

Related

• Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling
• Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
• Advancing AI Audits for Enhanced AI Governance
• Can AI be Auditable?
• The Necessity of AI Audit Standards Boards