2020/06/29 by Elizabeth Reichert, Reichert, Elizabeth, Helen Qiu +3
Computer Science · #Adversarial Robustness in Machine Learning #Computation and Language (cs.CL) #Computers and Society (cs.CY) #FOS: Computer and information sciences #Hate Speech and Cyberbullying Detection
paper · pdf · doi:10.48550/arxiv.2006.16402
openalex publication_date 2020/06/29 · openalex created_date 2020/07/10 · openalex updated_date 2026/07/28
The censorship of toxic comments is often left to the judgment of imperfect models. Perspective API, a creation of Google technology incubator Jigsaw, is perhaps the most widely used toxicity classifier in industry; the model is employed by several online communities including The New York Times to identify and filter out toxic comments with the goal of preserving online safety. Unfortunately, Google's model tends to unfairly assign higher toxicity scores to comments containing words referring to the identities of commonly targeted groups (e.g., "woman,'' "gay,'' etc.) because these identities are frequently referenced in a disrespectful manner in the training data. As a result, comments generated by marginalized groups referencing their identities are often mistakenly censored. It is important to be cognizant of this unintended bias and strive to mitigate its effects. To address this issue, we have constructed several toxicity classifiers with the intention of reducing unintended bias while maintaining strong classification performance.