Towards AI Accountability Infrastructure: Gaps and Opportunities in AI Audit Tooling

2025/04/25 by Victor Ojewale, Ryan Steed, Briana Vecchione +2 · 1 voice · 2 citations
Social Sciences · Computer Science · #Ethics and Social Impacts of AI #Explainable Artificial Intelligence (XAI) #Blockchain Technology Applications and Security

paper · pdf · doi:10.1145/3706598.3713301

openalex publication_date 2025/04/25 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/29

Abstract

Audits are critical mechanisms for identifying the risks and limitations of deployed artifcial intelligence (AI) systems.However, the efective execution of AI audits remains incredibly difcult, and practitioners often need to make use of various tools to support their eforts.Drawing on interviews with 35 AI audit practitioners and a landscape analysis of 435 tools, we compare the current ecosystem of AI audit tooling to practitioner needs.While many tools are designed to help set standards and evaluate AI systems, they often fall short in supporting accountability.We outline challenges practitioners faced in their eforts to use AI audit tools and highlight areas for future tool development beyond evaluationfrom harms discovery to advocacy.We conclude that the available resources do not currently support the full scope of AI audit practitioners' needs and recommend that the feld move beyond tools for just evaluation and towards more comprehensive infrastructure for AI accountability.

Citations

• The Ethnography of Infrastructure
• Model Cards for Model Reporting
• Fairness and Abstraction in Sociotechnical Systems
• On the Legal Compatibility of Fairness Definitions
• Closing the AI accountability gap
• "Public(s)-in-the-Loop": Facilitating Deliberation of Algorithmic Decisions in Contentious Public Policy Domains
• The Fallacy of AI Functionality
• Power to the People? Opportunities and Challenges for Participatory AI
• A Safe Harbor for AI Evaluation and Red Teaming
• Why ‘open’ AI systems are actually closed, and why this matters
• International AI Safety Report

Cited by

• Audits Under Resource, Data, and Access Constraints: Scaling Laws For Less Discriminatory Alternatives
• Impact Assessment Card: Communicating Risks and Benefits of AI Uses

Discussions

• And I would look here (and references therein) for insight into the actual practice of auditing, and with a much more informed and nuanced view than the NIST framework. Many deployed systems have neve [bsky, 0 points, 0 comments]

Related

• AI auditing: The Broken Bus on the Road to AI Accountability
• Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditing
• Advancing AI Audits for Enhanced AI Governance
• From Transparency to Accountability and Back: A Discussion of Access and Evidence in AI Auditing
• The Necessity of AI Audit Standards Boards