Does Research Design Affect Study Outcomes in Criminal Justice?

2001/11/01 by David Weisburd, Cynthia M. Lum, Cynthia Lum +1 · 335 citations
Decision Sciences · Medicine · Psychology · Social Sciences · #Affect (linguistics) #Clinical study design #Clinical trial #Crime Patterns and Interventions #Criminal Justice and Corrections Analysis #Criminal justice #Criminology #Economic Justice #Experimental research #External validity #Internal validity #Law #Medicine #Meta-analysis and systematic reviews #Political science #Psychiatry #Psychological intervention #Psychology #Randomized controlled trial #Randomized experiment #Research design #Social psychology #Social science #Sociology

paper · doi:10.1177/000271620157800104

published in The Annals of the American Academy of Political and Social Science 578(1), 50-70 (SAGE Publishing)

openalex publication_date 2001/11/01 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/25

Abstract

Does the type of research design used in a crime and justice study influence its conclusions? Scholars agree in theory that randomized experimental studies have higher internal validity than do nonrandomized studies. But there is not consensus regarding the costs of using nonrandomized studies in coming to conclusions regarding criminal justice interventions. To examine these issues, the authors look at the relationship between research design and study outcomes in a broad review of research evidence on crime and justice commissioned by the National Institute of Justice. Their findings suggest that design does have a systematic effect on outcomes in criminal justice studies. The weaker a design, indicated by internal validity, the more likely a study is to report a result in favor of treatment and the less likely it is to report a harmful effect of treatment. Even when comparing randomized studies with strong quasi-experimental research designs, systematic and statistically significant differences are observed.

Cited by

• What Can Police Do to Reduce Crime, Disorder, and Fear?
• Evidence‐based intervention against bullying and cyberbullying: Evaluation of the NoTrap! program in two independent trials
• The Process-Based Model of Policing Revisited: Examining the Reciprocal Relationship Between Procedural Justice and Police Legitimacy
• The evidence-based policing matrix at 14 years
• The Effect of Youth Diversion Programs on Recidivism
• Focused deterrence can reduce crime: A systematic review of randomized controlled trials and quasi‐experiments
• The Effects of Focused Deterrence Strategies on Crime
• Deterring Gang-Involved Gun Violence: Measuring the Impact of Boston’s Operation Ceasefire on Street Gang Behavior
• The effects of judicial supervision on recidivism of offenders in Australia and New Zealand: a systematic review and meta-analysis
• Correctional policy for offenders with mental illness: Creating a new paradigm for recidivism reduction.
• The Effect of Police Body-Worn Cameras on Use of Force and Citizens’ Complaints Against the Police: A Randomized Controlled Trial
• Can Policing Disorder Reduce Crime? A Systematic Review and Meta-analysis
• Randomized experiments in criminology: What have we learned in the last two decades?
• Evaluating offending behaviour programmes
• Why do evaluation researchers in crime and justice choose non-experimental methods?
• The Long-Term Effects of the Youth Crime Prevention Program "New Perspectives" on Delinquency and Recidivism. [europepmc]
• Multidimensional family therapy in adolescents with a cannabis use disorder: long-term effects on delinquency in a randomized controlled trial. [europepmc]
• The effects of judicial supervision on recidivism of offenders in Australia and New Zealand: a systematic review and meta-analysis. [europepmc]
• Police stops to reduce crime: A systematic review and meta-analysis. [europepmc]
• Effects of second responder programs on repeat incidents of family abuse: An updated systematic review and meta-analysis. [europepmc]
• PROTOCOL: Police stops to reduce crime: A systematic review. [europepmc]
• Effectiveness of school-based programs to reduce bullying perpetration and victimization: An updated systematic review and meta-analysis. [europepmc]
• Commentary: Prioritising Randomized Trials; David P. Farrington's Legacy for Criminology in the 21st Century: As Criminological Research Tilts Towards Ideology, Randomized Experiments Are Key for Reinforcing True Science. [europepmc]
• Letting Work What Works-Effectively Preventing Juvenile Delinquency in the Netherlands: A Meta-Analysis of the Evidence. [europepmc]

Related

• Are Most Published Criminological Research Findings Wrong? Taking Stock of Criminological Research Using a Bayesian Simulation Approach
• On the Validity of Laboratory Research in the Political and Social Sciences: The Example of Crime and Punishment
• Single group, pre- and post-test research designs: Some methodological concerns
• The Prominence of Qualitative Research in Criminology and Criminal Justice Scholarship
• Restorative justice and the evidence hierarchy: implications for policy and practice