2025/01/26 by Gabriel Stanovsky, Renana Keydar, Stanovsky, Gabriel +5 · 3 voices · 2 citations
Computer Science · Social Sciences · #Ethics and Social Impacts of AI #Law, AI, and Intellectual Property #cs.AI #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2501.15693
openalex publication_date 2025/01/26 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
The performance of AI models on safety benchmarks does not indicate their real-world performance after deployment. This opaqueness of AI models impedes existing regulatory frameworks constituted on benchmark performance, leaving them incapable of mitigating ongoing real-world harm. The problem stems from a fundamental challenge in AI interpretability, which seems to be overlooked by regulators and decision makers. We propose a simple, realistic and readily usable regulatory framework which does not rely on benchmarks, and call for interdisciplinary collaboration to find new ways to address this crucial problem.