Retrospective Learning from Interactions

2024/10/17 by Zizhao Chen, Mustafa Omer Gul, Chen, Zizhao +9 · 2 voices · 3 citations
Computer Science · #Semantic Web and Ontologies #Topic Modeling #cs.AI #cs.CL #cs.CV #cs.LG

paper · pdf · doi:10.48550/arxiv.2410.13852

openalex publication_date 2024/10/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Multi-turn interactions between large language models (LLMs) and users naturally include implicit feedback signals. If an LLM responds in an unexpected way to an instruction, the user is likely to signal it by rephrasing the request, expressing frustration, or pivoting to an alternative task. Such signals are task-independent and occupy a relatively constrained subspace of language, allowing the LLM to identify them even if it fails on the actual task. We introduce ReSpect, a method to learn from such signals in past interactions via retrospection without additional annotations. We deploy ReSpect in a new multimodal interaction scenario, where humans instruct a multimodal LLM to solve an abstract reasoning task with a combinatorial solution space. Through thousands of interactions with humans, we show how ReSpect gradually improves task completion rate from 31% to 82%, all without any external annotation.

Cited by

• Learning from 53.6K Real-World Developer Edits of AI-Generated Code
• Context informs pragmatic interpretation in vision-language models
• The Era of Real-World Human Interaction: RL from User Conversations

Discussions

• me: let’s start with a meme @yoavartzi.com: how about the paper’s fig1? 🙅 me: lesson learned. no memes 😭 A paper on continually learning from naturally occurring interaction signals, such as in the [bsky, 8 points, 2 comments]
• Maybe I am getting the specifics of what you mean wrong, cut we have a line of work on systems that adapt and improve over time. This is the latest: arxiv.org/abs/2410.13852 [bsky, 7 points, 1 comments]

Related

• Do LLMs Benefit From Their Own Words?
• In-Place Feedback: Reliable Refinement for Multi-Turn Expert-LLM Collaboration
• Beyond Single-Turn: A Survey on Multi-Turn Interactions with Large Language Models
• User Feedback in Human-LLM Dialogues: A Lens to Understand Users But Noisy as a Learning Signal
• Chain of Hindsight Aligns Language Models with Feedback