A tutorial on open-source large language models for behavioral science

2024/08/15 by Zak Hussain, Marcel Binz, Rui Mata +1 · 1 voice · 64 citations
Computer Science · Psychology · Social Sciences · #Artificial intelligence #Behavioral modeling #Computational and Text Analysis Methods #Computer science #Computer security #Conceptualization #Data science #Executable #Interpretability #Mental Health via Writing #Open science #Programming language #Topic Modeling #Transparency (behavior)

paper · pdf · doi:10.3758/s13428-024-02455-8

published in Behavior Research Methods 56(8), 8214-8237 (Springer Science+Business Media)

openalex publication_date 2024/08/15 · openalex created_date 2025/10/10 · openalex updated_date 2026/08/06

Abstract

Large language models (LLMs) have the potential to revolutionize behavioral science by accelerating and improving the research cycle, from conceptualization to data analysis. Unlike closed-source solutions, open-source frameworks for LLMs can enable transparency, reproducibility, and adherence to data protection standards, which gives them a crucial advantage for use in behavioral science. To help researchers harness the promise of LLMs, this tutorial offers a primer on the open-source Hugging Face ecosystem and demonstrates several applications that advance conceptual and empirical work in behavioral science, including feature extraction, fine-tuning of models for prediction, and generation of behavioral responses. Executable code is made available at github.com/Zak-Hussain/LLM4BeSci.git . Finally, the tutorial discusses challenges faced by research with (open-source) LLMs related to interpretability and safety and offers a perspective on future research at the intersection of language modeling and behavioral science.

Citations

• I.—COMPUTING MACHINERY AND INTELLIGENCE
• On the Dangers of Stochastic Parrots
• ChatGPT outperforms crowd workers for text-annotation tasks

Cited by

• Automating the Scientist? Perspektiven der Fachgruppe Methoden und Evaluation auf die Nutzung von KI in der psychologischen Forschung
• Escaping the Jingle-Jangle Jungle: Increasing Conceptual Clarity in Psychology Using Large Language Models
• Sustainable Development Goals in Psychology: A Century of Progress in Publications
• Empirical Prompt Engineering for Construct Identification with Large Language Models
• Addressing Longstanding Challenges in Cognitive Science with Language Models
• Digital Prompting in Education: A Design Framework and Bibliometric Analysis
• A tentative roadmap for consensus building processes
• The Mixed Subjects Design: Treating Large Language Models as Potentially Informative Observations
• Crucible: Quantifying the Potential of Control Algorithms through LLM Agents
• Open-Source Large Language Models in Education: A Narrative Review of Evidence, Pedagogical Roles, and Learning Outcomes
• Adding LLMs to the psycholinguistic norming toolbox: A practical guide to getting the most out of human ratings
• Beyond Interpretability: Exploring the Comprehensibility of Adaptive Video Streaming through Large Language Models
• The Agent Behavior: Model, Governance and Challenges in the AI Digital Age
• A validity-guided workflow for robust large language model research in psychology
• Large Language Models as Psychological Simulators: A Methodological Guide
• Properties of Person Descriptors in the Natural German Language: A Preregistered Replication and Extension
• Mapping Mental Representations With Free Associations: A Tutorial Using the R Package associatoR. [europepmc]
• AI can outperform humans in predicting correlations between personality items. [europepmc]
• SEMbeddings: how to evaluate model misfit before data collection using large-language models. [europepmc]
• Semantic embeddings reveal and address taxonomic incommensurability in psychological measurement. [europepmc]
• Examining Chat GPT with nonwords and machine psycholinguistic techniques. [europepmc]
• Crowdsourced and AI-generated age-of-acquisition (AoA) norms for vocabulary in print: Extending the Kuperman et al. (2012) norms. [europepmc]
• Evaluating Web Retrieval-Assisted Large Language Models With and Without Whitelisting for Evidence-Based Neurology: Comparative Study. [europepmc]
• Can Large Language Models Generate Useful Linguistic Corpora?: A Case Study of the Word Frequency Effect in Young German Readers. [europepmc]
• Updating the German Psycholinguistic Word Toolbox with AI-Generated Estimates of Concreteness, Valence, Arousal, Age of Acquisition, and Familiarity. [europepmc]
• Escaping the Jingle-Jangle Jungle: Increasing Conceptual Clarity in Psychology Using Large Language Models. [europepmc]
• A review for navigating the trade-offs: evaluating open-source and proprietary large language models for clinical and biomedical information extraction. [europepmc]
• Snake Oil or Panacea? How to Misuse AI in Scientific Inquiries of the Human Mind [europepmc]

Discussions

• 🚨 New article 🚨 Open LLMs are powerful and reproducible alternatives to closed models like GPT or Gemini. This tutorial shows how LLMs work and how to apply open LLMs in behavioral science resear [bsky, 24 points, 0 comments]

Related

• CogBench: a large language model walks into a psychology lab
• Large language models for behavioral modeling: A literature survey
• Finetuning LLMs for Human Behavior Prediction in Social Science Experiments
• Behavioral Fingerprinting of Large Language Models
• MacBehaviour: An R package for behavioural experimentation on large language models