StoryScope: Investigating idiosyncrasies in AI fiction

2026/04/03 by Jenna Russell, Rishanth Rajendhran, Chau Minh Pham +2 · 30 voices · 2 citations
#cs.CL

paper · pdf

Abstract

As AI-generated fiction becomes increasingly prevalent, questions of authorship and originality are becoming central to how written work is evaluated. While most existing work in this space focuses on identifying surface-level signatures of AI writing, we ask instead whether AI-generated stories can be distinguished from human ones without relying on stylistic signals, focusing on discourse-level narrative choices such as character agency and chronological discontinuity. We propose StoryScope, a pipeline that automatically induces a fine-grained, interpretable feature space of discourse-level narrative features across 10 dimensions. We apply StoryScope to a parallel corpus of 10,272 writing prompts, each written by a human author and five LLMs, yielding 61,608 stories, each ~5,000 words, and 304 extracted features per story. Narrative features alone achieve 93.2% macro-F1 for human vs. AI detection and 68.4% macro-F1 for six-way authorship attribution, retaining over 97% of the performance of models that include stylistic cues. A compact set of 30 core narrative features captures much of this signal: AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. Per-model fingerprint features enable six-way attribution: for example, Claude produces notably flat event escalation, GPT over-indexes on dream sequences, and Gemini defaults to external character description. We find that AI-generated stories cluster in a shared region of narrative space, while human-authored stories exhibit greater diversity. More broadly, these results suggest that differences in underlying narrative construction, not just writing style, can be used to separate human-written original works from AI-generated fiction.

Citations

• Alignment Whack-a-Mole : Finetuning Activates Verbatim Recall of Copyrighted Books in Large Language Models
• EditLens: Quantifying the Extent of AI Editing in Text
• Idiosyncrasies in Large Language Models
• People who frequently use ChatGPT for writing tasks are accurate and robust detectors of AI-generated text
• A Comparative Study of Clinical ModernBERT and BioMedical ModernBERT on the DDXPlus Dataset
• Delving into LLM-assisted writing in biomedical publications through excess vocabulary
• Technical Report on the Pangram AI-Generated Text Classifier
• Spotting LLMs With Binoculars: Zero-Shot Detection of Machine-Generated Text
• Art or Artifice? Large Language Models and the False Promise of Creativity
• Language Models are Few-Shot Learners

Cited by

• Generative AI floods and dilutes the market for books
• Argument Collapse: LLMs Flatten Long-Form Public Debate

Discussions

• There is a lot being written about the stylistic tells of AI writing (em-dashes, etc.) but this paper looks at AI narrative tells instead. Fascinating differences between AI & human narrative, and ask [bsky, 219 points, 6 comments]
• arxiv.org/pdf/2604.03136 Lots of people are skeptical of the possibility of automatically detecting use of AI in writing, and they're right in the sense that many attempts at this have been pretty bad [bsky, 42 points, 2 comments]
• Turns out, AI fiction does not look like human fiction, but unfortunately, very little of the common wisdom about em dashes matches up with it, and neither do the algorithms AI detectors use: arxiv.or [bsky, 15 points, 2 comments]
• Great research by @jennarussell.bsky.social et al. She also had a preprint this month on stylistic patterns in AI fiction (incl. that "GPT over-indexes on dream sequences") that's completely fascinati [bsky, 13 points, 1 comments]
• Ai generated stories resemble poop Source: arxiv.org/pdf/2604.03136 [bsky, 10 points, 1 comments]
• StoryScope: Investigating Idiosyncrasies in AI Fiction [hn, 4 points, 1 comments]
• Investigating idiosyncrasies in AI fiction [lobsters, 4 points, 2 comments]
• Have not read the whole paper yet but the link is here: [bsky, 3 points, 1 comments]
• StoryScope: Investigating Idiosyncrasies in AI Fiction [hn, 3 points, 0 comments]
• I only found this: arxiv.org/abs/2604.03136 Curious what the English dept would say about "transform[ing] each story into a template-based representation that preserves narrative elements while abstra [bsky, 2 points, 1 comments]
• Wasn’t the whole point of the study that they weren’t just looking at syntax? arxiv.org/abs/2604.03136 [bsky, 2 points, 2 comments]
• "AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. ... AI stories clust [bsky, 2 points, 0 comments]
• Another study out of University of Maryland showed that there are structural tells in story construction that can reliably identify LLM authors even without the stylistic tells that frequent users can [bsky, 2 points, 0 comments]
• Pretty fascinating stuff: it's possible to detect AI generated writing based purely on narrative structure without examination of idiosyncratic word choice or style. Predictably: human writing tends t [bsky, 2 points, 0 comments]
• StoryScope: Investigating Idiosyncracies in AI Fiction [hn, 2 points, 0 comments]
• Looks like there was a paper about this arxiv.org/pdf/2604.03136 [bsky, 1 points, 0 comments]
• StoryScope: Investigating Idiosyncrasies in AI Fiction [hn, 1 points, 0 comments]
• Since the 404 article didn't have it, and folks like @mkirschenbaum.bsky.social were looking for it, here's the preprint of the study. arxiv.org/pdf/2604.03136 [bsky, 1 points, 1 comments]
• For those of you interested in the differences between Human writing and AI writing, here is an interesting technical paper on the topic, which could perhaps assist also in detecting nefarious use of [bsky, 1 points, 0 comments]
• folks are starting to do deeper dives into parsing exactly how AI fiction differs from human-authored fiction and it's pretty fascinating, actually arxiv.org/pdf/2604.03136 [bsky, 0 points, 0 comments]
• Investigating idiosyncrasies in AI fiction arxiv.org/abs/2604.03136 [bsky, 0 points, 1 comments]
• Beyond em-dashes and "it's not X, it's Y": #ai #fiction also clusters together in plot space, regardless of model. Human writing, on the contrary, spreads all over. #LLMs converge toward probable patt [bsky, 0 points, 0 comments]
• “StoryScope: Investigating idiosyncrasies in AI fiction” arxiv.org/pdf/2604.03136 [bsky, 0 points, 0 comments]
• interesting study on the differences of human and AI writing. AI is a) more direct with its themes b) more linear in telling the story c) focusing on body and senses, d) less likely to use real world [bsky, 0 points, 1 comments]
• arxiv.org/pdf/2604.03136 [bsky, 0 points, 0 comments]
• AI stories over-explain themes and favor tidy, single-track plots while human stories frame protagonist' choices as more morally ambiguous and have increased temporal complexity. arxiv.org/abs/2604.03 [bsky, 0 points, 0 comments]
• How an AI writing fiction differs from humans: "AI pays closer attention to physical environment and characters’ inner mental states, showing emotion through bodies and environments rather than naming [bsky, 0 points, 1 comments]
• Investigating idiosyncrasies in AI fiction https://lobste.rs/s/hjuopb #science #ai [bsky, 0 points, 0 comments]
• Interesting arxiv.org/pdf/2604.03136 #ai #writing [bsky, 0 points, 0 comments]
• Here is a link to the paper - arxiv.org/pdf/2604.03136 [bsky, 0 points, 1 comments]

Related

• Experimental narratives: A comparison of human crowdsourced storytelling and AI storytelling
• The AI Fiction Paradox
• Agents' Room: Narrative Generation through Multi-step Collaboration
• AI Fiction in the Wild
• Stylometric comparisons of human versus AI-generated creative writing