2025/01/02 by Dylan Sam, Marc Finzi, Sam, Dylan +3 · 3 citations
Computer Science · #Computation and Language (cs.CL) #Data Mining Algorithms and Applications #FOS: Computer and information sciences #Imbalanced Data Classification Techniques #Machine Learning (cs.LG)
paper · pdf · doi:10.48550/arxiv.2501.01558
openalex publication_date 2025/01/02 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/31
Reliably predicting the behavior of language models -- such as whether their outputs are correct or have been adversarially manipulated -- is a fundamentally challenging task. This is often made even more difficult as frontier language models are offered only through closed-source APIs, providing only black-box access. In this paper, we predict the behavior of black-box language models by asking follow-up questions and taking the probabilities of responses as representations to train reliable predictors. We first demonstrate that training a linear model on these responses reliably and accurately predicts model correctness on question-answering and reasoning benchmarks. Surprisingly, this can even outperform white-box linear predictors that operate over model internals or activations. Furthermore, we demonstrate that these follow-up question responses can reliably distinguish between a clean version of an LLM and one that has been adversarially influenced via a system prompt to answer questions incorrectly or to introduce bugs into generated code. Finally, we show that they can also be used to differentiate between black-box LLMs, enabling the detection of misrepresented models provided through an API. Overall, our work shows promise in monitoring black-box language model behavior, supporting their deployment in larger, autonomous systems.