Using Nonnaive Participants Can Reduce Effect Sizes

2015/06/10 by Jesse Chandler, Gabriele Paolacci, Eyal Peer +2 · 1 voice · 6 citations
Computer Science · Decision Sciences · Social Sciences · #Innovation, Sustainability, Human-Machine Systems #Mobile Crowdsensing and Crowdsourcing #Personal Information Management and User Behavior

paper · doi:10.1177/0956797615585115

openalex publication_date 2015/06/10 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28

Abstract

Although researchers often assume their participants are naive to experimental materials, this is not always the case. We investigated how prior exposure to a task affects subsequent experimental results. Participants in this study completed the same set of 12 experimental tasks at two points in time, first as a part of the Many Labs replication project and again a few days, a week, or a month later. Effect sizes were markedly lower in the second wave than in the first. The reduction was most pronounced when participants were assigned to a different condition in the second wave. We discuss the methodological implications of these findings.

Citations

• Subject effects in laboratory research: An examination of subject roles, demand characteristics, and valid inference.
• Response Scales: Effects of Category Range on Reported Behavior and Comparative Judgments
• Inside the Turk
• Experimental methods: Between-subject and within-subject design

Cited by

• Data quality of platforms and panels for online behavioral research
• Exposing omitted moderators: Explaining why effect sizes differ in the social sciences
• Amazon Mechanical Turk: The Human Sciences’ Labor Problem
• Demand Effects in Survey Experiments: An Empirical Assessment
• Understanding and managing the threat of common method bias: Detection, prevention and control
• Attention by design: Using attention checks to detect inattentive respondents and improve data quality

Discussions

• Is there a librarian who can tell me why a DOI of a paper (specifically doi.org/10.1177/0956...) in Psychological Science would point to this weird version of the website: sage.cnpereading.com/paragra [bsky, 5 points, 2 comments]

Related

• Addressing two blind spots of commonly used experimental designs: The Highly‐Repeated Within‐Person approach
• Instruction-based Replication Studies Raise Challenging Questions for Psychological Science
• More than meets the ITT: A guide for anticipating and investigating nonsignificant results in survey experiments
• How Conditioning on Posttreatment Variables Can Ruin Your Experiment and What to Do about It
• Valid Replications Require Valid Methods: Suggestions for Best Methodological Practices with Lab Experiments