How is ChatGPT's behavior changing over time?
2023/07/18 by Lingjiao Chen, Matei Zaharia, Chen, Lingjiao +3 · 8 voices · 170 citations
Computer Science · Medicine · Psychology · #Artificial Intelligence in Healthcare and Education #Computer science #Psychology #Topic Modeling #cs.AI #cs.CL #cs.LG
paper · pdf · doi:10.48550/arxiv.2307.09009
published in arXiv (Cornell University) (Cornell University)
openalex publication_date 2023/07/18 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Abstract
GPT-3.5 and GPT-4 are the two most widely used large language model (LLM) services. However, when and how these models are updated over time is opaque. Here, we evaluate the March 2023 and June 2023 versions of GPT-3.5 and GPT-4 on several diverse tasks: 1) math problems, 2) sensitive/dangerous questions, 3) opinion surveys, 4) multi-hop knowledge-intensive questions, 5) generating code, 6) US Medical License tests, and 7) visual reasoning. We find that the performance and behavior of both GPT-3.5 and GPT-4 can vary greatly over time. For example, GPT-4 (March 2023) was reasonable at identifying prime vs. composite numbers (84% accuracy) but GPT-4 (June 2023) was poor on these same questions (51% accuracy). This is partly explained by a drop in GPT-4's amenity to follow chain-of-thought prompting. Interestingly, GPT-3.5 was much better in June than in March in this task. GPT-4 became less willing to answer sensitive questions and opinion survey questions in June than in March. GPT-4 performed better at multi-hop questions in June than in March, while GPT-3.5's performance dropped on this task. Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June than in March. We provide evidence that GPT-4's ability to follow user instructions has decreased over time, which is one common factor behind the many behavior drifts. Overall, our findings show that the behavior of the "same" LLM service can change substantially in a relatively short amount of time, highlighting the need for continuous monitoring of LLMs.
Cited by
- Mapping Political-Elite Networks in Europe with a Multilingual Joint Entity-Relation Extraction Pipeline
- Total Variation Distance Estimation in Autoregressive Models
- How Open Must Language Models be to Enable Reliable Scientific Inference?
- Generalization Bias in Large Language Model Summarization of Scientific Research
- The Cake that is Intelligence and Who Gets to Bake it: An AI Analogy and its Implications for Participation
- Securing the AI Supply Chain: What Can We Learn From Developer-Reported Security Issues and Solutions of AI Projects?
- Wrong and More Confident: A Field Experiment on Large Language Models Taking a Graduate Economics Exam
- Generative Artificial Intelligence for Software Engineering -- A Research Agenda
- The One-Word Census: Answer-Choice Conformity Across 44 Language Models
- A Structured Cyber Threat Intelligence Dataset Using STIX 2.1 Entities and MITRE ATT&CK Mappings
- ShareChat: A Dataset of Chatbot Conversations in the Wild
- Evaluating Code Reasoning Abilities of Large Language Models Under Real-World Settings
- The 2025 Foundation Model Transparency Index
- Log Probability Tracking of LLM APIs
- Small Models Achieve Large Language Model Performance: Evaluating Reasoning-Enabled AI for Secure Child Welfare Research
- When Safety Blocks Sense: Measuring Semantic Confusion in LLM Refusals
- Adaptive Diagnostic Reasoning Framework for Pathology with Multimodal Large Language Models
- Entrepreneurs of conflict: A descriptive analysis of when and how political elites use divisive rhetoric
- Pay for The Second-Best Service: A Game-Theoretic Approach Against Dishonest LLM Providers
- Reflections on the Reproducibility of Commercial LLM Performance in Empirical Software Engineering Studies
- Using customized, conversational AI agents in leadership and management research: Benefits, practical illustrations, and best practices
- Language-based assessments can predict psychological and subjective well-being
- Hybrid Instructor Ai Assessment In Academic Projects: Efficiency, Equity, And Methodological Lessons
- Can ChatGPT Evaluate Plans?
- HoneyBee: Data Recipes for Vision-Language Reasoners
- From Craft to Constitution: A Governance-First Paradigm for Principled Agent Engineering
- NurseLLM: The First Specialized Language Model for Nursing
- A RESEARCH NOTE ON RETHINKING SOFTWARE DOCUMENTATION: CREATING INCLUSIVE COMPUTATIONAL TOOLS FOR SOCIAL SCIENCES
- Ethical opportunities and risks of using ChatGPT for open science: evidence from a three-year mixed-methods study
- Aging as a Loss of Goal‐Directedness: An Evolutionary Simulation and Analysis Unifying Regeneration with Anatomical Rejuvenation
- Programmable Cognitive Bias in Social Agents
- Testing theory of mind in large language models and humans
- ChatGPT for Code Refactoring: Analyzing Topics, Interaction, and Effective Prompts
- Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
- The impact of fine tuning in LLaMA on hallucinations for named entity extraction in legal documentation
- CounselBench: A Large-Scale Expert Evaluation and Adversarial Benchmarking of Large Language Models in Mental Health Question Answering
- A validity-guided workflow for robust large language model research in psychology
- The Formalism Trap: Are LLM-as-a-Judge Evaluators Blinded by Consensus Mimicry under Social Load?
- Our Coding Adventure: Using LLMs to Personalise the Narrative of a Tangible Programming Robot for Preschoolers
- Quantifying Fairness in LLMs Beyond Tokens: A Semantic and Statistical Perspective
- Fanfiction in the Age of AI: Community Perspectives on Creativity, Authenticity and Adoption
- Keeping Medical AI Healthy and Trustworthy: A Review of Detection and Correction Methods for System Degradation
- Auditing Black-Box LLM APIs with a Rank-Based Uniformity Test
- The Lock-in Hypothesis: Stagnation by Algorithm
- Winning at All Cost: A Small Environment for Eliciting Specification Gaming Behaviors in Large Language Models
- Bye-bye, Bluebook? Automating Legal Procedure with Large Language Models
- Program of Equations Thoughts to Solve Algebra Word Problems
- Improving Model Alignment Through Collective Intelligence of Open-Source LLMS
- What Prompts Don't Say: Understanding and Managing Underspecification in LLM Prompts
- Automated Text Scoring in the Age of Generative AI for the GPU-poor
- Telco-oRAG: Optimizing Retrieval-augmented Generation for Telecom Queries via Hybrid Retrieval and Neural Routing
- Freemium Is All You Need
- The Hitchhikers Guide to Production-ready Trustworthy Foundation Model powered Software (FMware)
- Token-Efficient Change Detection in LLM APIs
- JudgeArena: A Unified Framework for Reproducible LLM-Judge Evaluation
- Large Databases Need Small, Open-Weight Language Models
- What LLMs Think When You Don't Tell Them What to Think About?
- Memorization and Knowledge Injection in Gated LLMs
- LLM-Evaluation Tropes: Perspectives on the Validity of LLM-Evaluations
- AgentSPEX: An Agent SPecification and EXecution Language
- Improving Language Model Personas via Rationalization with Psychological Scaffolds
- Testing LLMs' Capabilities in Annotating Translations Based on an Error Typology Designed for LSP Translation: First Experiments with ChatGPT
- You've Changed: Detecting Modification of Black-Box Large Language Models
- Should you use LLMs to simulate opinions? Quality checks for early-stage deliberation
- Assessing how hyperparameters impact Large Language Models' sarcasm detection performance
- Stock Market Forecasting: From Traditional Predictive Models to Large Language Models
Discussions
- How is ChatGPT's behavior changing over time? [hn, 289 points, 178 comments]
- [Other] OpenAI's Models performance on March 2023 vs June 2023 [lemmy, 24 points, 1 comments]
- very very plausible. "Both GPT-4 and GPT-3.5 had more formatting mistakes in code generation in June [2023] than in March. We provide evidence that GPT-4's ability to follow user instructions has decr [bsky, 17 points, 1 comments]
- You may be underestimating how quickly we’re approaching model collapse, some models lost up to 90% accuracy in a very short time. As the AI shit-eating-shit machine grows we’ll only see this get wors [bsky, 2 points, 0 comments]
- How Is ChatGPT’s Behavior Changing over Time? https://arxiv.org/pdf/2307.09009.pdf #chatgpt [bsky, 1 points, 0 comments]
- How Is ChatGPT’s Behavior Changing over Time? https://github.com/lchen001/LLMDrift https://arxiv.org/abs/2307.09009 This states that GPT-4's performance has decreased from March to June 2023. For exa [bsky, 1 points, 1 comments]
- https://arxiv.org/abs/2307.09009 ChatGPT is changing quickly and constantly, but how the LLMs are being updated are completely opaque. There's a mad rush to integrate LLMs into our workflows with zero [bsky, 1 points, 0 comments]
- My post was a very short summary, of course, and there's plenty of other things going on in the paper. I guess my particular point is best illustrated by the right-hand side of Fig. 4. But it's also j [bsky, 1 points, 1 comments]
Related