From peer review nuances to best practices

2026/07/12 by Sheng Lu
#cs.DL #cs.CL

paper · pdf

Abstract

This report studies three nuances in peer review data: paper version, score version, and input format. We characterize how the variants differ, and measure their impact on downstream tasks. Based on our findings, we offer best practices for both data providers and data users.

Citations

• ReviewerToo: Should AI Join The Program Committee? A Look At The Future of Peer Review
• Automatic Reviewers Fail to Detect Faulty Reasoning in Research Papers: A New Counterfactual Evaluation Framework
• ReviewRL: Towards Automated Scientific Review with RL
• gpt-oss-120b & gpt-oss-20b Model Card
• Re²: A Consistency-ensured Dataset for Full-stage Peer Review and Multi-turn Rebuttal Discussions
• ReviewAgents: Bridging the Gap Between Human and AI-Generated Paper Reviews
• DeepReview: Improving LLM-based Paper Review with Human-like Deep Thinking Process
• OpenReviewer: A Specialized Large Language Model for Generating Critical Scientific Paper Reviews
• Are We There Yet? Revealing the Risks of Utilizing Large Language Models in Scholarly Peer Review
• Automated Peer Reviewing in Paper SEA: Standardization, Evaluation, and Analysis
• AgentReview: Exploring Peer Review Dynamics with LLM Agents
• What Can Natural Language Processing Do for Peer Review?
• NLPeer: A Unified Resource for the Computational Study of Peer Review
• A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications

Related

• What Constitutes Peer Review of Data: A survey of published peer review guidelines
• A Dataset of Peer Reviews (PeerRead): Collection, Insights and NLP Applications
• What Factors Should Paper-Reviewer Assignments Rely On? Community Perspectives on Issues and Ideals in Conference Peer-Review
• What Drives Paper Acceptance? A Process-Centric Analysis of Modern Peer Review
• Assisting Decision Making in Scholarly Peer Review: A Preference Learning Perspective