ICL CIPHERS: Quantifying "Learning" in In-Context Learning via Substitution Ciphers

2025/04/28 by Zhouxiang Fang, Fang, Zhouxiang, Aayush Mishra +7 · 2 voices · 2 citations
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Domain Adaptation and Few-Shot Learning #FOS: Computer and information sciences #User Authentication and Security Systems #cs.CL

paper · pdf · doi:10.48550/arxiv.2504.19395

openalex publication_date 2025/04/28 · arxiv published 2025/04/28 · arxiv updated 2025/08/27 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Recent works have suggested that In-Context Learning (ICL) operates in dual modes, i.e. task retrieval (remember learned patterns from pre-training) and task learning (inference-time ''learning'' from demonstrations). However, disentangling these the two modes remains a challenging goal. We introduce ICL CIPHERS, a class of task reformulations based on substitution ciphers borrowed from classic cryptography. In this approach, a subset of tokens in the in-context inputs are substituted with other (irrelevant) tokens, rendering English sentences less comprehensible to human eye. However, by design, there is a latent, fixed pattern to this substitution, making it reversible. This bijective (reversible) cipher ensures that the task remains a well-defined task in some abstract sense, despite the transformations. It is a curious question if LLMs can solve tasks reformulated by ICL CIPHERS with a BIJECTIVE mapping, which requires ''deciphering'' the latent cipher. We show that LLMs are better at solving tasks reformulated by ICL CIPHERS with BIJECTIVE mappings than the NON-BIJECTIVE (irreversible) baseline, providing a novel approach to quantify ''learning'' in ICL. While this gap is small, it is consistent across the board on four datasets and six models. Finally, we examine LLMs' internal representations and identify evidence in their ability to decode the ciphered inputs.

Citations

• Out-of-distribution generalization via composition: a lens through induction heads in Transformers
• Out-of-distribution generalization via composition: A lens through induction heads in Transformers
• Generalisation First, Memorisation Second? Memorisation Localisation for Natural Language Classification Tasks
• The Llama 3 Herd of Models
• Investigating the Pre-Training Dynamics of In-Context Learning: Task Recognition vs. Task Learning
• Dual Process Learning: Controlling Use of In-Context vs. In-Weights Strategies with Weight Forgetting
• From Words to Numbers: Your Large Language Model Is Secretly A Capable Regressor When Given In-Context Examples
• Bridging Language and Items for Retrieval and Recommendation
• Dual Operating Modes of In-Context Learning
• Transformers Learn Nonlinear Features In Context: Nonconvex Mean-field Dynamics on the Attention Landscape
• OLMo: Accelerating the Science of Language Models
• Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models
• The mechanistic basis of data dependence and abrupt learning in an in-context classification task
• In-context Learning Generalizes, But Not Always Robustly: The Case of Syntax
• In-Context Exemplars as Clues to Retrieving from Large Associative Memory
• Do pretrained Transformers Learn In-Context by Gradient Descent?
• GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher
• Trained Transformers Learn Linear Models In-Context
• What In-Context Learning "Learns" In-Context: Disentangling Task Recognition and Task Learning
• The Closeness of In-Context Learning and Weight Shifting for Softmax Regression
• A Theory of Emergent In-Context Learning as Implicit Structure Induction
• Self-Instruct: Aligning Language Models with Self-Generated Instructions
• What learning algorithm is in-context learning? Investigations with linear models
• What Can Transformers Learn In-Context? A Case Study of Simple Function Classes
• On the Effect of Pretraining Corpora on In-context Learning by a Large-scale Language Model
• Rethinking the Role of Demonstrations: What Makes In-Context Learning Work?
• Impact of Pretraining Term Frequencies on Few-Shot Reasoning
• An Explanation of In-context Learning as Implicit Bayesian Inference
• Reframing Instructional Prompts to GPTk's Language
• True Few-Shot Learning with Language Models
• Fantastically Ordered Prompts and Where to Find Them: Overcoming Few-Shot Prompt Order Sensitivity
• Calibrate Before Use: Improving Few-Shot Performance of Language Models
• Can Sequence-to-Sequence Models Crack Substitution Ciphers?
• Language Models are Few-Shot Learners
• HellaSwag: Can a Machine Really Finish Your Sentence?
• Note on the Sampling Error of the Difference Between Correlated Proportions or Percentages
• Learning vs Retrieval: The Role of In-Context Examples in Regression with Large Language Models
• Why Can GPT Learn In-Context? Language Models Implicitly Perform Gradient Descent as Meta-Optimizers
• When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories

Cited by

• Genomic Next-Token Predictors are In-Context Learners
• Controlling the Risk of Corrupted Contexts for Language Models via Early-Exiting

Discussions

• 🤔 To what extent can Large Language Models (LLMs) solve unseen tasks via In-context Learning (ICL) ? 💥 We propose a new general framework ✨ ICL ciphers ✨ to quantify ``learning'' in ICL via substitu [bsky, 1 points, 1 comments]
• **However,** the performance gap is small, suggesting that the extent of "task learning" may be more limited than previously thought. By Zhouxiang Fang @FocusV857 and Aayush Mishra @aamixsh ; with con [bsky, 0 points, 0 comments]

Related

• Can Transformers Break Encryption Schemes via In-Context Learning?
• ICLShield: Exploring and Mitigating In-Context Learning Backdoor Attacks
• Illusion or Algorithm? Investigating Memorization, Emergence, and Symbolic Processing in In-Context Learning
• ALICE: An Interpretable Neural Architecture for Generalization in Substitution Ciphers
• Is In-Context Learning Learning?