Editorial: Drawing multimodality's bigger picture: metalanguages and corpora for multimodal analyses - in lieu of a Festschrift for John A. Bateman

Editorial on the Research Topic
Drawing multimodality's bigger picture: metalanguages and corpora for multimodal analyses - in lieu of a Festschrift for John A. Bateman

1 Drawing multimodality's bigger picture

The present Research Topic is dedicated to the work and achievements of one of the key figures in multimodality research, Professor John A. Bateman. John officially retired from his professorship in September 2023, after nearly 25 years of service at the University of Bremen, Germany, and more than 35 years of academic work at several universities and research centers around the world. Those who know John are aware that this retirement is desirably only a formality and that he will hopefully remain engaged in academic work for many years to come.

In good old German academic tradition, professorial retirements are often accompanied by a so-called Festschrift, a “celebratory writing,” i.e. a book honoring the academic and their work, with contributions from colleagues, friends, and PhD students. However, such a Festschrift is something that John himself did not allow to happen due to “too much cult of the individual” (quote from a personal email conversation in 2023). Normally, as long-time colleagues and mentees, we follow John's judgment and accept his decision, but through his training, we also learned to scrutinize and challenge some of his statements, only for his own good, of course.

So here we are with this Research Topic, which is indeed a collection of articles by John's colleagues, research associates, friends, and PhD students, in a journal that is edited by him and that is dedicated to one of his main research fields: multimodal communication. The majority of the articles in this Research Topic are papers from two conference panels we organized for the 11th International Conference on Multimodality (ICOM-11) in London in September 2023; some others are contributions from even more colleagues and friends from all over the world. We thank everyone cordially for their input, support, interest, and contribution to this Research Topic—and we thank John for making it possible. After all, it was John who officially approved our plans to organize the conference panels and publish this Research Topic as something “almost reasonable” (another quote from a personal email conversation in 2023). Admittedly and naturally, John did not have much say in the matter after his approval, nor was he involved in the review process. Of course, his spirit is present (or alive) in every contribution and the research behind it—and this is exactly what we were aiming for: we wanted to bring together scholars from a variety of disciplines interested in multimodality research to review, explore, and advance the contributions that John has made both to theory- and method-building and to the advancement of multimodal empirical and corpus analyses.

2 Metalanguages and corpora for multimodal analysis

Our main starting points for this Research Topic were twofold, building on discussion points recently raised by John himself: On the one hand, after 30 years of development, mainly in the humanities, and after having been evaluated in many different ways (see for an overview Wildfeuer et al., 2019), multimodality should no longer be seen as a research field or discipline, but rather as a “stage of development within a field,” a stage that every discipline goes through (Bateman, 2022a, p. 49). This means that many different fields and disciplines (not only in the humanities) have already entered, are currently entering, or will soon enter their own multimodal phase with a renewed interest in multimodal phenomena. With this comes a particular commitment to theory and method development, with each discipline or field bringing in its own principles and tools. This leads not only to an immense breadth of potential objects of analysis and points of discussion, but also and more importantly to the need to bridge differences and incompatibilities in favor of what John calls a “meta-methodology”:

“We need to find ways of ‘combining' insights from the variously imported theoretical and methodological backgrounds brought along by previous non-multimodal stages of any contributing disciplines.” (Bateman, 2022a, p. 49)

On the other hand, this search for a meta-methodology to guide multimodal analysis has recently been driven by more empirical approaches and the development and use of larger multimodal corpora, which also require theoretical and methodological refinement.

“We need to develop ways of strengthening claims with robustly applicable methods which nevertheless remain firmly anchored theoretically.” (Bateman, 2022b, p. 64)[SIC]

Making available these large-scale corpora and providing broader and more complex empirical and experimental setups aim to reconceptualize the practice of multimodal analysis and fully implement the “move from theory to data” (see Pflaeging et al., 2021). Following Bateman (2022a), for a productive treatment of these issues, disciplinary triangulation and the development of a “common language” or metalanguage (Maton and Chen, 2016) for an “integrationist interdisciplinarity” (Van Leeuwen, 2005) are the greatest challenges in contemporary multimodality research. It is precisely these challenges that we productively defined as the main aims of this Research Topic and as “a multimodal task from the ground up” (Bateman, 2022b, p. 64). We explicitly called for works that critically addressed John's theoretical and methodological advancements, that tested and reviewed the many approaches that he has developed for the analysis of multimodal artifacts, and that expanded on or even rejected some of the ideas and insights provided in his work.

True to John's research, the resulting contributions show theoretical and methodological concerns on the one hand, and data-driven analyses and approaches to a variety of multimodal artifacts on the other. Similar to the breadth and depth of his own work in more than 350 publications since 1983, the contributions to the present Research Topic are diversely rich and broad, ranging from brief research reports to a mini review to expanded research articles, all of which make a significant contribution to the field of multimodality research. Several articles challenge the theoretical and methodological concepts that were originally discussed and/or further developed by John, such as the notion of discourse semantics and a multimodal metalanguage (Martin), the concept of semiotic mode (Castaldi), the use of Segmented Discourse Representation Theory for multimodal artifacts (Kim and Calway), or the idea of a comprehensive semiotics for multimodal (corpus) analysis (Wildgen; Hiippala). Some papers show the breadth and reach of these theoretical and methodological concepts to provide an application-oriented approach to specific sub-disciplines of multimodality research, including diachronic multimodality studies (Pflaeging), multimodal argumentation studies (Stöckl), or multimodal corpus analysis (Hiippala). Several other articles provide results and evidence from empirical multimodality research with annotation systems and/or larger corpora (Maiorani; Hiippala; Thiele et al.), computational and (semi-)automatic tools (Wilson et al.; Mattei), or experimental studies such as eyetracking, surveys and interviews, or motion detection (Thiele et al.; Markhabayeva and Tseng; Lehmann; Maiorani). Together, these contributions provide insights into a wide range of communicative situations and media, including face-to-face interactions (Lehmann; Wilson et al.), foodscaping (Kim and Calway), film and audiovisual media (Wildgen; Thiele et al.; Markhabayeva and Tseng; Schmidt), websites and social media (Mattei), dance (Maiorani), and diagrams (Hiippala).

John can and should be present in all contributions—certainly not as a dominant sovereign (of which he was initially afraid; cf. “the cult of the individual” in Durkheim, 1964). Instead, we believe that each contribution developed its own voice and standpoint as part of the bigger picture of multimodality research. This voice may have been trained, educated, and influenced by John, through his writings, his comments and reviews, or his famous discussion practice, but it is certainly also presented with a particular independent stance, be it critical or affirmative, bringing out new and sometimes challenging ideas, reasonably.

Following the suggestion by Hiippala (2024), we label this Research Topic a “not-a-Festschrift Research Topic,” because it is, indeed, not simply a way of honoring John's scholarly achievements in a retrospective. Rather, this Research Topic intends to foster theory- and method-building in multimodality research with a prospective, future-oriented, outlook. Very much in the spirit of John's work as a mentor, supervisor, colleague and friend, we see the papers as examples of intellectual positions that can and should be discussed and challenged. We also see them as calls for future work, for the advancement of the field of multimodality research, something that John has always striven for with admirable curiosity, open-mindedness, and exceptional innovation and commitment.

Author contributions

JW: Conceptualization, Project administration, Resources, Writing – original draft, Writing – review & editing. CL: Conceptualization, Project administration, Resources, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Acknowledgments

We thank Tamara Drummond for the initial support and setup of this Research Topic and the accompanying conference panels. We also thank all contributors for their work in and for the Research Topic. A particular thank you to Damiano La Manna at Frontiers for his continuous support throughout the process of creating, editing, and finalizing this Research Topic.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The author(s) declared that they were an editorial board member of Frontiers, at the time of submission. This had no impact on the peer review process and the final decision.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Bateman, J. A. (2022a). Multimodality, where next? – some meta-methodological considerations. Multimodality Soc. 2, 41–63. doi: 10.1177/26349795211073043

Crossref Full Text | Google Scholar

Bateman, J. A. (2022b). Growing theory for practice: empirical multimodality beyond the case study. Multimodal Commun. 11, 63–74. doi: 10.1515/mc-2021-0006

Crossref Full Text | Google Scholar

Durkheim, E. (1964). The Division of Labor in Society (Transl. by G. Simpson). New York, NY: The Free Press.

Google Scholar

Hiippala, T. (2024). I have a weird new article out in @FrontComm that discusses multimodal corpora from the perspective of Peircean semiotics. It's also a part of John Bateman's not-a-Festschrift special issue edited by @neous et al. #multimodality #digitalhumanities. Twitter/X. Available online at: https://twitter.com/tuomo_h/status/1756999198292017546 (accessed June 24, 2024).

Google Scholar

Maton, K., and Chen, R. T.-H. (2016). “LCT in qualitative research: creating a translation device for studying constructivist pedagogy,” in Knowledge-Building: Educational Studies in Legitimation Code Theory, eds. K. Maton, S. Hood, and S. Shay (London: Routledge), 27–48.

Google Scholar

Pflaeging, J., Bateman, J. A., and Wildfeuer, J. (2021). “Empirical multimodality research: the state of play,” in Empirical Multimodality Research: Methods, Evaluations, Implications, eds. J. Pflaeging, J. Wildfeuer, and J. A. Bateman (Berlin: De Gruyter), 1–32. doi: 10.1515/9783110725001-001

Crossref Full Text | Google Scholar

Van Leeuwen, T. (2005). “Three models of interdisciplinarity,” in A New Agenda in (critical) Discourse Analysis: Theory, Methodology and Interdisciplinarity, eds. R. Wodak, and P. Chilton (Amsterdam: John Benjamins), 3–18. doi: 10.1075/dapsac.13.04lee

PubMed Abstract | Crossref Full Text | Google Scholar

Wildfeuer, J., Pflaeging, J., Bateman, J. A., Seizov, O., and Tseng, C. (2019). “Multimodality. disciplinary thoughts and the challenge of diversity – introduction,” in Multimodality. Disciplinary Thoughts and the Challenge of Diversity, eds. J. Wildfeuer, J. Pflaeging, J. A. Bateman, O. Seizov, and C. Tseng (Berlin: de Gruyter), 3–38. doi: 10.1515/9783110608694

Crossref Full Text | Google Scholar