2020/01/30 by Tuomo Hiippala, Hiippala, Tuomo, John Bateman +1
Computer Science · Social Sciences · #Artificial Intelligence (cs.AI) #Computation and Language (cs.CL) #Data Visualization and Analytics #FOS: Computer and information sciences #Multimedia Communication and Technology #Software Engineering Research
paper · pdf · doi:10.48550/arxiv.2001.11224
openalex publication_date 2020/01/30 · openalex created_date 2025/10/10 · openalex updated_date 2026/07/28
As the use and diversity of diagrams across many disciplines grows, there is an increasing interest in the diagrams research community concerning how such diversity might be documented and explained. In this article, we argue that one way of achieving increased reliability, coverage, and utility for a general classification of diagrams is to draw on recently developed semiotic principles developed within the field of multimodality. To this end, we sketch out the internal details of what may tentatively be termed the diagrammatic semiotic mode. This provides a natural account of how diagrammatic representations may integrate natural language, various forms of graphics, diagrammatic elements such as arrows, lines and other expressive resources into coherent organisations, while still respecting the crucial diagrammatic contributions of visual organisation. We illustrate the proposed approach using two recent diagram corpora and show how a multimodal approach supports the empirical analysis of diagrammatic representations, especially in identifying diagrammatic constituents and describing their interrelations in a manner that may be generalised across diagram types and be used to characterise distinct kinds of functionality.