Editorial: Multimodal coherence across media and genres

Editorial on the Research Topic
Editorial: Multimodal coherence across media and genres

The present Research Topic reflects the central position coherence ought to occupy in the study of multimodal artifacts. While the exact nature and concrete realization of coherent mode relations often elude us, they are in no way trivial. On the contrary, any act of comprehending a multimodal interaction or text requires tracing and piecing together the plausible links between such diverse expressive resources as speech, writing, image, sound or gesture.

Our feeling when we initiated the RT was that multimodal coherence was still under-researched despite its significance and that an explicit call for submissions orienting to the area could help multiply efforts. Our conviction that multimodal coherence constitutes a vital research challenge that must now be engaged with seriously for multimodality and its study to progress is strengthened by the work represented here.

The notion of “coherence” within linguistically-oriented work is a legacy of 1960s text linguistics, which posited that between parts of a text there are regular meaningful links that constitute something that Werlich (1976) called a “text grammar”. Whereas, as Halliday and Hasan (1976) discuss, early linguistic studies would often conflate notions such as “coherence” or “cohesion”, text linguists began to distinguish them rather sharply (e.g., Halliday and Hasan, 1976; De Beaugrande and Dresser, 1981), commonly drawing a difference between “text-internal” cohesive properties and “text-external” properties of coherence.

Seen multimodally then, multimodal cohesion refers to links between modes that actually materialize in the structure of a multimodal text (something that came to be called a cross-modal cohesive tie, e.g., “this rogue politician” building a tie to a press photo of V. Putin). In contrast, multimodal coherence relates to the sense or discourse continuities between modes that may or may not be indicated by properties of mode usage (e.g., the image of an oil-polluted stretch of coastline with dead seabirds serving as a motivation to call for action against marine pollution in a slogan). In much recent work, multimodality researchers have then started to adopt a focus on inter-mode connections rather than mode-internal connections. Indeed, multimodality has been defined as “textual combinations of different modes and their integration in terms of structure, discourse semantics, and rhetorical function” (Stöckl, 2019, p. 50). It is, therefore, fair to argue that the very essence of multimodality resides in inter-modal coherence.

As a consequence, the present RT works from the assumption that coherence is the more comprehensive concept with the greater explanatory power and reach, while still acknowledging the rather intricate and often unresolved relations between cohesion and coherence. Multimodal coherence has been given various telling names. The notion of “inter-semiotic complementarity” (Royce, 1998) may be understood as highlighting the equal weight that modes bring to the meaning making, while also marking a general reciprocity of the cognitive processes involved. Jewitt's idea of an “interplay between modes” (Jewitt, 2014, p. 27) is indicative of the active construal required to generate multi-/intermodal meaning. And, “intermodal harmony” (Norris and Maier, 2014, p. 390) appears to emphasize the fit between the modes needed to achieve some kind of balance.

For a review of different approaches to the study of multimodal coherence, we may conveniently rely on Bateman's seminal theory review (Bateman, 2014). Abstracting from the fine historically adequate detail, we can essentially glean three large and persistent paradigms. First, coherence was captured by looking at the cohesive ties between modes and the ways these can be classified. Second, a number of approaches have looked at multimodal coherence as deriving from discourse relations, i.e., logico-semantic relations between parts of text realized in different modes. Third, coherence may also be seen to materialize in or follow multimodal rhetorical figures, such as metaphor, metonymy, antonymy and analogy, where the rhetorical operation only works when two or more different modes connect.

The contributions in the present RT draw on and develop further several perspectives on coherence in multimodality showing more of the general applicability of the concept and the role of the phenomenon in multimodal communication. They are consequently quite varied and serve to give a good account of the different interests in, and current approaches to, multimodal coherence as a developing area of study.

Messner explores musical instruction in music lessons and shows how such interactions rely not just on speech but marshal a number of different semiotic resources, which must be coherently aligned. The essential modes mobilized here are gesture, gaze, and vocalizing, i.e., the singing or humming of tunes and rhythms. The study convincingly demonstrates that the discourse semantics of the modes involved must link to realize higher-level actions and to meaningfully sequence them in instruction.

Wildfeuer and Coffie turn their attention to the communicative domain of advertising and use a Heineken commercial campaign to show how filmic montage works toward coherence. These intermodally coherent structures in turn promote the rhetorical task of the beer campaign to foster social responsibility and wellbeing during the COVID pandemic. Methodologically, the paper shows the strengths of consistent annotations for various relevant features of multimodal structure, which are based on solid theoretical modeling.

Thurlow and Haudenschild study the communicative practices and trans-modal interplay at work in the business class airline meal, which is both a semiotic event and a multimodal performance. They subject this semiotic assemblage to a critical discourse analysis of the underlying ideologies. Overall, the study argues that in order to decipher the social/ideological meanings conveyed, we must carefully recover the multimodal coherence that underlies the multi-semiotic representation of the meal.

Meier raises awareness of the simple fact that producing and recovering multimodal coherence is a necessary competence to be developed in teaching on various levels. Following Jäger (2002), he regards coherence as a cognitive process of transcribing meaning from one text, medium, genre to another. The paper posits that learning which modes work well in what medium and what can be expressed how in the different modes is key to developing a multimodal competence.

Finally, Stöckl and Pflaeging enquire into the ways of annotating a larger corpus of avant-garde, visually complex print advertisements for critical features in multimodal coherence. Based on a review of relevant theory-building, the paper assembles a multi-layer annotation framework, which comprises six interconnecting levels. Exploring a smaller corpus of 50 ads, the authors test the available coding systems and their workability on each level of the framework and extend/modify them as necessary.

We take the opportunity to thank the authors for their varied contributions. Likewise, we are grateful to all the reviewers for their constructive critique of the papers. We hope that this collection of articles will promote the study of multimodal coherence, providing ideas on the theoretical foundations as well as for the design of empirical studies. In order to move forward, we need multi-level frameworks and the recognition that the specifics of multimodal coherence are shaped by medium and genre.

Author contributions

HS drafted and wrote up the text of the editorial. JB added to it and edited the piece. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Bateman, J. A. (2014). “Multimodal coherence research and its applications,” in The Pragmatics of Discourse Coherence, Gruber, H., and Redeker, G., (eds.). (Amsterdam: Benjamins) p. 145–177.

Google Scholar

De Beaugrande, R., and Dresser, R. (1981). Introduction to Text Linguistics. London: Longman.

Google Scholar

Halliday, M. A. K., and Hasan, R. (1976). Cohesion in English. London: Pearson.

Google Scholar

Jäger, L. (2002). “Transkriptivität. Zur medialen Logik der kulturellen Semantik,” in: Transkribieren. Medien/Lektüre, Jäger, L., and Stanitzek, G., (eds.). (München: Fink) p. 19–41.

Google Scholar

Jewitt, C. (2014). “An introduction to multimodality,” in The Routledge Handbook of Multimodal Analysis, Jewitt, C., (ed.). (London/New York: Routledge) p. 15–30.

Google Scholar

Norris, S., and Maier, C. D. (2014). Interactions, Images, and Text: A Reader in Multimodality. London, New York, NY: Routledge. doi: 10.1515/9781614511175

CrossRef Full Text | Google Scholar

Royce, T. D. (1998). Synergy on the page: exploring intersemiotic complementarity in page-based multimodal text. JASFL Occas. Pap. 1,25–48.

Google Scholar

Stöckl, H. (2019). “Linguistic multimodality – multimodal linguistics: a state-of- the-art sketch,” in Multimodality: Disciplinary Thoughts and the Challenge of Diversity, Wildfeuer, J., Pflaeging, J., Bateman, J. A., Seizov, O., and Tseng, C-I., (eds.). (Berlin: De Gruyter) p. 41–68.

Google Scholar

Werlich, E. (1976). A Text Grammar of English. Heidelberg: Quelle and Meyer.

Google Scholar