2019/05/17 by Siavash Haghiri, Haghiri, Siavash, Patricia Rubisch +7
Agricultural and Biological Sciences · Neuroscience · Psychology · #FOS: Computer and information sciences #Functional Brain Connectivity Studies #Machine Learning (cs.LG) #Machine Learning (stat.ML) #Mental Health Research Topics #Sensory Analysis and Statistical Methods
paper · pdf · doi:10.48550/arxiv.1905.07234
openalex publication_date 2019/05/17 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
Traditionally, psychophysical experiments are conducted by repeated measurements on a few well-trained participants under well-controlled conditions, often resulting in, if done properly, high quality data. In recent years, however, crowdsourcing platforms are becoming increasingly popular means of data collection, measuring many participants at the potential cost of obtaining data of worse quality. In this paper we study whether the use of comparison-based (ordinal) data, combined with machine learning algorithms, can boost the reliability of crowdsourcing studies for psychophysics, such that they can achieve performance close to a lab experiment. To this end, we compare three setups: simulations, a psychophysics lab experiment, and the same experiment on Amazon Mechanical Turk. All these experiments are conducted in a comparison-based setting where participants have to answer triplet questions of the form "is object x closer to y or to z?". We then use machine learning to solve the triplet prediction problem: given a subset of triplet questions with corresponding answers, we predict the answer to the remaining questions. Considering the limitations and noise on MTurk, we find that the accuracy of triplet prediction is surprisingly close---but not equal---to our lab study.