2022/02/22 by Caluã de Lacerda Pataca, Paula Dornhofer Paro Costa · 3 voices
Arts and Humanities · Psychology · #Art #Communication #Computer science #Computer vision #Direct speech #Language, Metaphor, and Cognition #Linguistics #Loudness #Multisensory perception and integration #Natural language processing #Paralanguage #Prosody #Psychology #Representation (politics) #Speech recognition #Subtitles and Audiovisual Media #Typography
paper · pdf · doi:10.1109/taffc.2022.3174721
openalex publication_date 2022/05/12 · openalex created_date 2022/07/14 · openalex updated_date 2026/07/29
Whether a word was bawled, whispered, or yelped, captions will typically represent it in the same way. If they are your only way to access what is being said, subjective nuances expressed in the voice will be lost. Since so much of communication is carried by these nuances, we posit that if captions are to be used as an accurate representation of speech, embedding visual representations of paralinguistic qualities into captions could help readers use them to better understand speech beyond its mere textual content. This paper presents a model for processing vocal prosody (its loudness, pitch, and duration) and mapping it into visual dimensions of typography (respectively, font-weight, baseline shift, and letter-spacing), creating a visual representation of these lost vocal subtleties that can be embedded directly into the typographical form of text. An evaluation was carried out where participants were exposed to thisspeech-modulated typographyand asked to match it to its originating audio, presented between similar alternatives. Participants (n=117) were able to correctly identify the original audios with an average accuracy of 65%, with no significant difference when showing them modulations as animated or static text. Additionally, participants’ comments showed their mental models of speech-modulated typography varied widely.