2023/08/15 by Yugeng Liu, Tianshuo Cong, Liu, Yugeng +9
Computer Science · Medicine · #Adversarial Robustness in Machine Learning #Artificial Intelligence in Healthcare and Education #Cryptography and Security (cs.CR) #FOS: Computer and information sciences #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2308.07847
openalex publication_date 2023/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Large Language Models (LLMs) undergo continuous updates to improve user experience. However, prior research on the security and safety implications of LLMs has primarily focused on their specific versions, overlooking the impact of successive LLM updates. This prompts the need for a holistic understanding of the risks in these different versions of LLMs. To fill this gap, in this paper, we conduct a longitudinal study to examine the adversarial robustness -- specifically misclassification, jailbreak, and hallucination -- of three prominent LLM families: GPT, Llama, and Qwen. Our study reveals that LLM updates do not consistently improve adversarial robustness as expected. For instance, a later version of GPT-3.5 degrades regarding misclassification and hallucination despite its improved resilience against jailbreaks. GPT-4 and GPT-4o demonstrate (incrementally) higher robustness overall. Larger Llama and Qwen models do not uniformly exhibit improved robustness across all three aspects studied. In addition, larger model sizes do not necessarily yield improved robustness. Minor updates lacking substantial robustness improvements can exacerbate existing issues rather than resolve them. We hope our study can offer valuable insights into navigating model updates and informed decisions in model development and usage.