2023/05/17 by Zuhao Yang, Yang, Zuhao, Yingfang Yuan +9 · 2 citations
Social Sciences · Computer Science · #Language and cultural evolution #Natural Language Processing Techniques #Speech and dialogue systems
paper · pdf · doi:10.48550/arxiv.2305.10307
Measuring the distance between machine-produced and human language is a critical open problem. Inspired by empirical findings from psycholinguistics on the periodicity of entropy in language, we propose FACE, a set of metrics based on Fourier Analysis of the estimated Cross-Entropy of language, for measuring the similarity between model-generated and human-written languages. Based on an open-ended generation task and the experimental data from previous studies, we find that FACE can effectively identify the human-model gap, scales with model size, reflects the outcomes of different sampling methods for decoding, correlates well with other evaluation metrics and with human judgment scores.