2020/05/20 by Nils Köbis, Köbis, Nils, Luca Mossink +1
Computer Science · #AI in Service Interactions #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Explainable Artificial Intelligence (XAI) #FOS: Computer and information sciences #FOS: Economics and business #General Economics (econ.GN) #Topic Modeling
paper · pdf · doi:10.48550/arxiv.2005.09980
openalex publication_date 2020/05/20 · openalex created_date 2022/07/23 · openalex updated_date 2026/07/28
The release of openly available, robust natural language generation\nalgorithms (NLG) has spurred much public attention and debate. One reason lies\nin the algorithms' purported ability to generate human-like text across various\ndomains. Empirical evidence using incentivized tasks to assess whether people\n(a) can distinguish and (b) prefer algorithm-generated versus human-written\ntext is lacking. We conducted two experiments assessing behavioral reactions to\nthe state-of-the-art Natural Language Generation algorithm GPT-2 (Ntotal =\n830). Using the identical starting lines of human poems, GPT-2 produced samples\nof poems. From these samples, either a random poem was chosen\n(Human-out-of-the-loop) or the best one was selected (Human-in-the-loop) and in\nturn matched with a human-written poem. In a new incentivized version of the\nTuring Test, participants failed to reliably detect the\nalgorithmically-generated poems in the Human-in-the-loop treatment, yet\nsucceeded in the Human-out-of-the-loop treatment. Further, people reveal a\nslight aversion to algorithm-generated poetry, independent on whether\nparticipants were informed about the algorithmic origin of the poem\n(Transparency) or not (Opacity). We discuss what these results convey about the\nperformance of NLG algorithms to produce human-like text and propose\nmethodologies to study such learning algorithms in human-agent experimental\nsettings.\n