The materiality key: how work on empirical data can improve analytical models and theoretical frameworks for multimodal discourse analysis

This article is a critical reflection on the way the notion of materiality informed the project and the development of The Kinesemiotic Body project carried out by a UK and German research team and of the model of analysis it adopted, the Functional Grammar of Dance. It starts with an excursus of some of the most interesting developments in other discipline that turned to the investigation of materiality as an epistemological perspective, and it shows how the same type of focus has impacted multimodal discourse analysis focusing on movement-based communication. The overarching theme that characterises this multidisciplinary attention to materiality is its anchoring function to the temporal and spatial coordinates in which social phenomena are contextualised, which is taken as the fundamental condition for shaping our perception and understanding of the world in all areas of experience and knowledge. A more specific example of how the notion of materiality impacted the development of movement-based discourse analysis will be provided by an example of analysis of rich movement data captured live from professional dancers from the English National Ballet.

Introduction

This paper provides a critical reflection on the role played by the notion of materiality in the development of movement-based discourse analysis within the wider area of Multimodality studies. It is positioned within an even wider area of multidisciplinary research that focused on this notion in the last few decades and that foregrounded some very interesting points for reflection and development across disciplines. Through examples drawn from a recent research project in movement-based communication, it will demonstrate how in order to incorporate effectively the awareness and understanding of materiality in a communicative environment, it is essential to turn to the analysis of empirical data, which in turn provides solid evidence to strengthen and/or advance theoretical frameworks. The project in question is The Kinesemiotic Body, funded by the Arts and Humanities Research Council (AHRC) in the UK and the German Research Foundation (DFG) in Germany. The fact that the project focused on movement-based analysis (specifically on dance choreography) carried out by scholars from very different disciplines, where the importance of incorporating the materiality of the human body in interaction with a performance environment was considered through different approaches (Multimodal Discourse Analysis, Engineering, Computer Science, etc.) makes the examples of empirical data analysis proposed here particularly appropriate to the consideration of the notion of materiality as an interdisciplinary one and provides a clear connection with John Bateman’s discussion of materiality in relation to the development of Multimodality as a practice that encompasses borders between disciplines and research areas (Bateman et al., 2017; Bateman, 2019, 2022). This article will also show how the consideration of the materiality of dance allowed in primis for the further development of the Functional Grammar of Dance (Maiorani, 2021; Maiorani et al., 2022; Maiorani and Liu, 2023), which is now a more comprehensive and even more flexible tool that scholars have started to use for analysing movement-based communication in dance performances other than ballet or even outside the domain of dance altogether (see Mouard Ruiz, 2021; Bolens, 2022; Meissl et al., 2022; Prové, 2022; Sindoni, 2022; Vidal Claramonte, 2022; Wu, 2022; Elyamany, 2023). The examples of empirical data analysis will be preceded by a presentation of how the Functional Grammar of Dance is implemented in ELAN, a widespread commercial, free-to-use software traditionally used for annotating conversations or verbal interactions, for which we created a complete set of interdependent tiers and controlled vocabulary. By including spatial annotation categories and the distinction of internal discourse structures, our annotation offers quite innovative insights into the way movement-based communication can be annotated and analysed.

In order to describe the impact of the concept of materiality on The Kinesemiotic Body project—and especially the way materiality was foregrounded by Bateman’s work in multimodal discourse analysis—I need to take a series of steps backwards, to the time when Kinesemiotics, a new interdisciplinary research area, was developed at Loughborough University. Kinesemiotics started with an interdisciplinary team of researchers created at Loughborugh University in 2016, where we covered Linguistics, Semiotics, Multimodality, sensor Engineering, and Computer Science. After receiving funding from the Loughborough University CALIBRE programme in 2017 to work in collaboration with the English National on the investigation of movement-based discourse analysis by capturing a small amount of dance movement data, we joined forces with John Bateman for a joint grant application to the Arts and Humanities Research Council (AHRC) and the German Research Foundation (Deutsche Forschungsgemeinschaft, DFG) and we were funded for the collaborative international project called The Kinesemiotic Body.¹ The project aim was to advance the understanding of movement-based communication starting from choreographed movement in a worldwide renown movement-based form of performance: ballet. This choice was driven by the team’s specific expertise as well as by the pre-existing collaboration with the English National Ballet and the status of ballet as a form of performance based on movement with a tradition long recognised and established at international level, a tradition that has had an enormous impact on the elaboration and the development of many other forms of movement-based performances. Our intention was to evaluate whether and how we could apply a linguistically-motivated model for the analysis of verbal discourse to the study of how the body communicates by interacting with the space within the context of a performance setup. This would not only allow to deepen our understanding of the specific form of performance on which we were focusing, but also to develop a new approach to non-verbal communication with a more finely elaborated notion of movement-based discourse structure.

To contextualise the results of the project within a much more comprehensive scientific overview, this article will start by considering how the same notion has been approached by different disciplines in recent years looking at some significant examples of literature, thus showing how this concept actually taps into the very foundations of a multidisciplinary idea of knowledge.

Materiality across research areas and disciplines

In several research areas, materiality seems to be considered as a contextualised configuration of the spatial and temporal location of multimodal communication, an architecture that anchors theory to real-life situations and allows for the encounter and cross-fertilisation of diverse fields of study. Contemporary ontological philosophy puts materiality at the centre of social life and interaction (Schatzki, 2010), positing social phenomena as configurations of practices and material arrangements, thus recognising materiality itself as a component of social phenomena that combines with technology and practices. In this way, the relationship between practices (including meaning-making practices) and the material arrangements in which these practices take place spatially and temporally – the fundamental socio-cultural coordinates – becomes the focus of contemporary social ontology. Schatzki (2010, p. 125) also points out that materiality is not merely physicality: it is rather to be defined as ‘composition’, the ‘stuff’ of which social life is made. The issue at stake is therefore to find a way to describe it systematically. Schatzki (2010, p. 129) also defines practices as ‘organized spatial–temporal manifolds of human activity. Examples are cooking practices, political practices, manufacturing practices, football practices, dating practices, and horse breeding practices’. The material arrangements that form nexi with practices to generate social phenomena are ‘sets of interconnected material entities’ (Schatzki, 2010) that can be human beings, artifacts, organisms, and other natural items. The materiality of a social phenomenon can therefore be extremely complex, and the problem is finding a systematic way to pick up the elements that compose it.

In the area of semiotic studies, and essentially drawing on Peirce, the complexity and centrality of the notion of materiality has already been foregrounded by Petrilli (2008) through the specification of two types of materiality that inhere the sign itself: ‘In a global semiotic perspective, it would seem that the first claim to be made is that the existence of biological material is the initial condition for sign material or semiosic material to exist. It goes without saying that no less necessary for the existence of biological material is the existence of chemicophysical material. Therefore, we could begin by stating that the materiality of signs presents itself on various levels, upon which basis we may propose a typology of semiosic materiality’ (Petrilli, 2008, p. 139). Understanding semiosic materiality in this respect involves the recognition of a clear distinction between physical and biological materiality, the latter generating a further distinction between living and non-living organic materiality. Materiality is therefore seen as being at the origin of human experience of communication.

One of the most interesting examples of the analysis and use of materiality as a foundational epistemological concept comes from energy studies, where energy is conceptualised in its materiality to understand how its perception impacts on daily practices and transactions worldwide. A whole trend of energy studies has been working for decades on the reconceptualisation of the very notion of ‘energy’ by drawing on multidisciplinary and interdisciplinary approaches that include also theories from geography, politics, history, anthropology, etc. All these approaches focus on the effort to define the materiality of energy. Balmaceda et al. (2019) pose four fundamental questions to open a fruitful dialogue and exchange amongst different research areas; their queries are about the location of energy materiality, its users and the way they use it, its relational characteristics with context in terms of spatial and temporal scales, and the analytical role of energy materiality in the different epistemological areas. These questions are meant to anchor a theoretical enquiry on this fundamental notion to specific contexts in real life, such as the way energy materiality determines constraints in agency that will then impact on infrastructures and politics (i.e., energy consumption, supply chains, etc.). These questions also highlight the historical relationship between energy materiality and the evolution of technology (Leonardi and Barley, 2010), which is also a factor that impacts in a fundamental way social semiotics practices across time and space. Dance discourse—the meaning produced and shaped by choreographed movement in dance performances—is movement-based and movement involves the flow of kinetic energy. The way we experience and capture this flow for various purposes (archiving, documenting, visualising, etc.) is also impacted by the development of technology and of the devices that allow us to anchor to a specific time and place performances that would otherwise be lost once they have taken place. To understand how these questions may be of considerable relevance even when studying the development and perception of dance discourse, it suffices to think of the way an audience perception of live dance performances has changed considerably during and after COVID 19 lockdowns, when the perception of temporal and spatial location of performances worldwide was dramatically changed by the impossibility of actually attending a live performance in theatres. It was the audience’s perception of these coordinates that technological affordances successfully managed to change when a number of theatre and ballet companies survived thanks to the broadcast of performances originally recorded for live streaming in cinemas and then turned into ‘live pre-recorded events’ packaged for home entertainment (Maiorani, 2020).

The importance of how the flow of movement is anchored to a spatially and temporally located context also emerges in trans-contextual analysis, a branch of social semiotics that looks at how materiality is perceived in different contexts through mobility. Kell (2015, p. 425) proposes the concept of ‘meaning-making trajectories which are made up of recontextualizing and resemiotising moves’. This concept is meant to incorporate the flow of meaning movement and transformation within contextualised communication through language, and it is linked to the materiality of communicative contexts moving across time and space; it also resonates with that of trajectory in Minimal Ballet Sequences, a unit of dance discourse analysis that I will explain below and that provides the description of dance discourse with a connective thread that incorporates the flow of movement and allows for the understanding of the different functions of orientation and direction in movement-based discourse. Meaning-making practices in trans-contextual analysis do not only take into consideration movement across contexts but also the role of material objects that interact with the ‘text-artefacts’ (Kell, 2015, p. 426), thus advocating for multimodality as a more comprehensive approach to the analysis of communication.

The connection between materiality, flow of experience and energy and embodiment is also at the centre of several cutting-edge theoretical approaches to knowledge understanding in the humanities. Whilst creative writing practices and cultural anthropology interrogate the relationship between identity and the materiality of the semiotic forms (Wilf, 2011), experimental literature focuses on the notion of materiality when trying to provide a flow of multimodal experiences to its readers (Lee, 2014). The consideration of materiality becomes particularly crucial in translation practices, where the materiality of the text emerges in all its complexity, ranging from its physical features to the way the written word conveys auditory, tactile, visual, and other sensorially-perceived (in other words, multimodal) meanings. In this case and drawing on Gibbons’s (2012) idea of reading as an activity involving multisensory perception, the embodiment of a text materiality is once more conceived as the anchoring of the reader’s meaning perception of multimodal, multisensorial meaning to a specifically located spatial and temporal context. In this way, the reader’s body and its physical environment, its way of perceiving the world through the senses, becomes the nexus, the filter through which the very act of reading, of perceiving the materiality of a text turns into its embodiment.

Whilst experimental literature focuses on the nexus between narrated spaces and topographies and the way these are perceived through reading, recent studies on national mobility and infrastructures also pay attention to the materiality of the environment and how it influences the emergence and understanding of nationalisms and national identities (Merriman and Jones, 2017). Also in this case, materiality is theorised as the constellation of materials of diverse nature that anchors the flow of multimodal discourse—one of nation and identity—to specific temporal and spatial locations or to the process of crossing them. This perspective was generated by a wider context of studies on the relationship between discourse and materiality and its impact on management and organisational theory (Putnam, 2014), which has at its centre the dimensions of time and space and sees communication as the locus of the interplay between human agency and discourse.

Educational contexts have also turned to the study of the materiality in the context of traditional teaching and learning activities to develop more updated and effective pedagogical strategies within the perspective of multimodality. The materiality of multimodal forms of feedback has been studied to improve and update current forms of teaching and learning practices (Tyrer, 2021), whilst lectures have been considered as a form of ‘multimodal, sociomaterial performance’ (Lacković and Popova, 2021) that has the human body and movement-based communication at its centre. This reconceptualisation of lectures as a multimodal, movement-based practice draws on the concept of sociomateriality (Gherardi, 2017), which is grounded in post-humanist studies and essentially describes the interplay between social structures and material contexts made of bodies and items interacting in space in which every day meaning-making practices are habitually carried out. This new epistemological approach to knowledge shuns from human-centred approaches to learning and considers human experiences through materially contextualised phenomena. One of its central areas of research is embodied work practices, which posits the human body as epistemological focus.

In the more specific area of science education, the educational environment is seen as a synthesis of semiotic agents that interact to produce meaning (Pantidos et al., 2010). The teaching of physics is particularly seen as an activity that involves creating connections amongst different signifying items and anchoring them to specific spatial and temporal contexts to explain theories. This activity generates narratives that make use of verbal language, gestures, objects, graphs, body movement, etc., a specific teaching practice whose general features can be observed in all types of science teaching. In this respect, science teaching is very similar to theatre practice, and its materiality is very similar to the materiality of theatre, where narrative spaces are characterised by referents whose meanings define a specific semiotic landscape anchored to a specific time and space (which is more or less what happens with the set-up of a dance performance space). The materiality of these narrative spaces is also similar to those used to teach robots when providing them with exemplary situations: thus, concepts are taught and learnt by anchoring them to the spatial and temporal materiality of a real-life context, to the materiality of everyday semiosis that is shaped into meaning through discourse (Björkvall and Karlsson, 2011). Björkvall and Karlsson draw strongly on social semiotics and anchor the specificity of contextual materiality into culture: according to them, materiality offers a meaning potential (Kress, 2010) that is then shaped through meaning making practices grounded in specific cultures. The shaping activity of cultures also involves choices amongst affordances that will be selected to become semiotic resources for communication. As a matter of fact, in a specific context within a specific culture, not all material affordances will become semiotic affordances. It is therefore the regularity and recurrence of configurations of semiotic affordances that allows us to identify modes (Bateman et al., 2017) within specific temporally and spatially located cultures: ‘for an affordance to be turned into a semiotic resource, it needs to be picked up by a culture or by a social group and be continuously worked upon in activity types of various kinds. In other words, the affordance needs to be shaped by culture to become what we call a semiotic resource. From this it follows that even if affordances are material resources for humans to perceive when they act in their environment, they are not necessarily semiotic resources. However, also affordances that are not defined as semiotic resources can have meaning potential’. (Björkvall and Karlsson, 2011, p. 147).

As it will be demonstrated below, the challenge of understanding which affordances in dance are regularly and consistently used as semiotic resources was one that was faced by The Kinesemiotic Body project and one that benefited from the consideration of the notion of materiality as a an external language for description, a language that applies to the analysis of rich, live-captured movement data by taking into consideration the specific configurations of materials that are shaped into semiotic resources in the meaning-making practice of dance performances.

A theoretical framework to anchor the flow of dance to its materiality

The theoretical framework of The Kinesemiotic Body project was strongly based in linguistic theory and multimodal analysis; besides the Functional Grammar of Dance (FGD, Maiorani, 2017, 2021), our work also drew on segmented discourse representation theory (Asher and Lascarides, 2003; Bateman and Wildfeuer, 2014) as well as on recent developments in corpora analysis, live movement data collection and data visualisation. Working both on video materials and on movement data collected live from professional dancers of the English National Ballet (ENB)—who performed whole ballet sequences both as single performers and in couple—we developed a method of multimodal annotation using the annotation software ELAN that allowed us to annotate and analyse not just how movement is structurally carried out along a temporal line and within a specific space, but also how through structured movement sequences, dancers communicate by projecting their body parts towards meaningful portions of space, thus creating semantic connections that guide the audience’s interpretation. In this way, we created a method for annotating dance sequences that incorporates both movement structures and meaning structures in a flow of data. To show the effect of the research carried out through the analysis of empirical data within The Kinesemiotic Body project, I will first describe the original version of the Functional Grammar of Dance model and then I will introduce the updated version with all the relevant additions.

The first model of the Functional Grammar of Dance

The first model of the Functional Grammar of Dance was published in 2017 and it clearly drew on Halliday’s Functional Grammar for verbal language. The model was already completely different from traditional dance notation systems as it was created and used for the manual analysis of dance discourse (movement structures and corresponding meaning), not for the notation of deconstructed movements and their physical qualities. With respect to the current and updated FGD model, it was simpler but it already incorporated first and foremost the dancer’s point of view as the starting point of movement, even if at the time it had only been applied to manual analysis of video clips. The FGD posits that movement-based communication, like verbal communication, always happens in a specific Context of Situation whose variables, Field (what is happening), Tenor (who is taking part), and Mode (how communication is being carried out by the participants to the communicative event), activate as many meanings (respectively Experiential, Interpersonal, and Textual) that will then be realised by different linguistic structures. In movement-based communication, these structures are also movement-based and they are called Choreographic Affordances, namely all possible body-structure combinations performed by dancers whilst moving, structures that vary according to the dance style that is being adopted and the possibilities and limitations of human bodies. Figure 1 shows the first version of the FGD.

Figure 1

Figure 1. The 2017 version of the Functional Grammar of Dance.

In order to generate meaning, choreographic affordances allow for the creation of structured projections of body parts towards meaningful portions of the performance space. The first fundamental difference between the Hallidayan Functional Grammar and the FGD is the distinction between physical space and contextual space. Space is a fundamental dimension of the FGD: “A body is a spatial construct. It exists and functions through its relationship with space. Space itself is defined by the presence of bodies of any kind: without bodies, we call it ‘void’”(Maiorani, 2021, p. 1). From its first version, the FGD posited that whereas dance movement can be instantiated during training classes by dancers carrying out choreographic combinations in a studio’s physical space just for the purpose of training, meaning is only created through the interaction between body structures and the performance space, which is populated by contextually relevant objects, people, props, etc., and it is therefore designed for this purpose. Whilst dancing a choreography, dancers extend in various manners their body parts towards meaningful spots in the performance space, thus creating interactions between their dancing bodies and people, or objects, or props, or light effects, and these interactions will provide the audience with cues to follow a narrative, to understand who is interacting with whom or what, and to enjoy the choreographed sequences as a whole. The visualisations of these interactions are called Projections: the narrative ones indicate action (i.e., going to, coming from, locating, connecting, addressing, engaging, etc.), the interactive ones indicate interaction either with the audience (AU) or with participants on stage (POS). Only in the contextual space—whether actually built or just imagined during rehearsals it does not matter, provided that there is a shared awareness of it—can dance discourse actually be realised.

Interestingly, the elaboration of the FGD also allowed for a more in-depth discussion of the discussion of instantiation as a foundational concept of Systemic Functional Linguistic theory, leading to its definition as a dynamic relationship and to a further elaboration of the theory of Context (see Maiorani and Wegener, 2022). However, the first FGD model in Figure 1 shows how some areas of analysis could not be fully developed without the use of a larger amount of data collected live from dancers: the whole model is based on the development of its theoretical foundations and on manual analysis performed on small scale video data and drawing on a solid knowledge of the range offered by choreographic affordances, especially in terms of ballet. The lack of analysis of richer data sourced from different dancers performing different roles shows particularly in the area of Textual meanings, which was still developed on merely theoretical assumptions that needed to be tested empirically. The work carried out through The Kinesemiotic Body project on a corpus of live-dance captured data provided exactly this opportunity to test the FGD application empirically and to develop an analytical method that could be implemented in a widely commercially available software for annotation.

The updated model of the Functional Grammar of Dance and our annotation system

The updated version of the Functional Grammar of Dance was elaborated whilst annotating rich live-captured movement data with the ELAN software. The annotation system we have developed is not an alternative to traditional notation systems like Labanotation or Benesh notation, which involve intensive training in using specific scores and symbols and provide a notation of the physical characteristics and qualities of unstructured movement along the music score. These systems are movement notation systems. The FGD annotation we implemented using ELAN is a dance discourse annotation that always puts the dancer’s point of view at the centre of each movement and provides information both on movement structures and on discursive structures using labels that make no use of specialistic terminology. The FGD annotation is a dance annotation method, which implies that dance is not considered only as physical movement but as a meaningful and contextualised movement-based performance (Maiorani, 2021; Maiorani et al., 2022). Our annotation system in ELAN develops on different levels: the lower level of Move, which is the basic unit of analysis of the FGD, and the level of Minimal Ballet Sequence (MBS), which is the smallest discursive unit and comprises two consecutive Moves. The annotation is based on the work of the body articulators: head, torso, arms and hands, legs and feet. The Move marks the minimum movement across space performed by a dancer and is delimited by a starting set of projections and an arrival set of projections. The two consecutive Moves in an MBS are the smallest discursive unit that provides a trajectory in direction: if the Move direction is the same for both consecutive Moves, the MBS trajectory is defined as continuous; it the direction changes, the MBS trajectory is defined as varied. When the choreography requires it, we also annotate at the level of Elaborations: these are extra arrival sets of projections that mark a change in position of the body articulators at the end of a Move that does not involve any movement across space.

The use of sensors and the related software to capture live movement data from dancers immediately showed us that we had to deal with a complexity of movement parameters that needed to be ordered and put into clear functional relationships in the annotation. As soon as we started working with ELAN, we realised that we had to make three dimensions of annotation visibly distinguished and integrated at the same time: the level of physical movement, which accounts for the way each body articulator moves in relation to the surrounding space coordinates (i.e., inwards/outwards, up/down, backwards/forwards, etc.), the level of structure, which accounts for the way the different body articulators are positioned with respect to the Move direction, and the level of projections, which accounts for the narrative and interactive values of body parts projections towards meaningful portions of the performance space. These distinctions were necessary to show the complexity of the movement-based discourse enacted by dance, where the meaning created by projections is determined also by the position of articulators with respect to the immediate space references and the direction that a whole Move has taken. This complexity of relationships became visible when we started capturing live-data from dancers and had to take into consideration all the elements of movement we had to measure in order to account for all the factors that determined choreographic choices. The tiers related to each dimension are separated and colour coded but they are all at the same level, thus allowing for the visualisation of the complexity of factors all contributing at the same time to the realisation of projections within the performance space. After segmenting the flow of data into Moves, the first tier we annotate is always that of physical movement, which provides us with a picture of where every articulator is at the moment of annotation with respect to the immediate spatial references as they are perceived by the dancer; then we annotate the structures, which incorporate the direction the dancer takes when moving and the respective positioning in space of all the articulators with respect to movement; finally, we incorporate the discursive dimension by annotating narrative and interactive projections, which shows what type of actions and interactions the choices made in terms of physical movement and structure determine. The version of the FGD we used within ELAN is the most recent one, which we started developing after a preliminary work of live-movement capture data with the English National Ballet in 2017 and then kept on elaborating during The Kinesemiotic Body project. The impact of this work carried out on empirical data is reflected in a more detailed distinction of units of analysis (Choreoraphic units) specifically devised for empirical data segmentation and in the inclusion of narrathletic enhancers (showcasing dancers’ athletic qualities) and modal values of projections (highlighting concentrations of projections in one direction) that reinforce the integration of physical and semantic description of the collected data. Figure 2 shows the current, updated model of the FGD (Maiorani, 2021, p. 30).

Figure 2

Figure 2. The updated version of the FGD model (Maiorani, 2021, p. 30).

The improved work on the role of Move direction carried out on empirical data also allowed us to understand the discursive role of trajectories, designed by two consecutive Moves, thus highlighting the importance of segmenting MBSs. Figure 3 outlines the annotation framework we have developed.

Figure 3

Figure 3. An FGD-derived framework for the analysis of dance discourse (Maiorani and Liu, 2023, p. 9).

When we transfer this annotation framework to ELAN, we create different tiers to annotate the Moves and MBSs. The highest level of description is that of the MBS tier, under which we annotate the Move tier (second description level) and Elaboration tier (third description level). The tiers with the descriptions of physical movement, structures, narrative projection, interactive projections, narrathletic enhancers and modal values of projections are all dependent on the Move tier and on the Elaboration tier when this occurs.

Every dance sequence is segmented according to the three levels of Move and couples of consecutive Moves are then grouped into MBSs. Therefore, the tiers depending on the Move (and Elaboration when present) align with Move (and Elaboration) segmentation. The end point of each Move aligns with the start of the subsequent one, and the same happens with the MBS segmentation, thus incorporating the flow of movement into the annotation. Figure 4 shows the FGD annotation framework implemented in ELAN.

Figure 4

Figure 4. FGD annotation framework implemented in ELAN (Maiorani and Liu, 2023, p. 10).

The annotation tiers are linked to a controlled vocabulary divided into menus that provides specific options for each type of information annotated (i.e., physical movement, structures, projections, etc.) and draws on the FGD. The vocabulary is generated into a drop-down list in the ELAN annotation template, from which the annotator can select the most appropriate choice. The vocabulary does not contain any technical term and it is therefore very user friendly and open to non-specialist users. For this reason, it can also be easily adapted to the annotation of movements other than ballet. Figure 5 shows an example of the drop-down controlled vocabulary list with options provided by labels that do not contain specialistic language.

Figure 5

Figure 5. Example of drop-down controlled vocabulary (Maiorani and Liu, 2023, p. 11).

The annotation template includes a window where the dance sequence that is being annotated is visualised and this can be reduced or enlarged in size according to necessity. Figure 6 provides an example of annotation made on data collected live from a dancer from the English National Ballet whose body was synthesised into an avatar. The figure does not offer the whole annotation but just a screenshot of a section as for the whole script it is necessary to scroll the text down. The video window has been reduced in size to provide a larger view of the annotation.

Figure 6

Figure 6. Example of annotation with visualisation of a dancer’s avatar.

Figure 7 offers an example of annotation that highlights the segmentation into Moves and MBSs. The visualisation of the sequence that is being annotated is a video taken in a rehearsal studio at the English National Ballet headquarters in London.

Figure 7

Figure 7. Example of annotation showing the segmentation into Moves and MBSs.

The various tiers in which the annotation is organised is evidence in itself of the complexity of the materiality of dance that we were capable of capturing when working empirically and with live-captured movement data. The empirical work we carried out within The Kinesemiotic Body project allowed us to capture not only the relationships between movement structures and projections at various levels but it also made us realise that there are different levels of meaning carried out at different levels of discourse segmentation, and that direction and orientation have different and complementary roles in the perception of dance discourse, as will be discussed in the following section.

Results of working with empirical data of a ballet sequence corpus

The FGD model was elaborated further when we started capturing live data from the dancers in a real rehearsal studio in preliminary work carried out in 2017 and then implemented in the analysis of the dance data corpus carried out with The Kinesemiotic Body project. When having to organise and annotate the data we recorded from dancers in rehearsal studios, the research activity based on data analysis had to face two main challenges. The first challenge was posed by the complexity of data which involved not only the dancers’ movements but also the space set-up and the use of direction and orientation. Unlike what we had to take into consideration in the first examples of analysis performed manually with the FGD, where selected movement structures and projections were analysed on the basis of the systemic functional theoretical framework, a much greater amount of features and levels of communication deployment was suddenly available for analysis through the corpus of dance sequences collected with the English National Ballet. The second, consequential challenge was that the organisation of all these new features and levels that had not yet been captured or addressed by manual analysis had to be systematised in a consistent and replicable framework for annotation to be used for all items of the corpus. The initial manual analysis with the FGD had paved the road for a systematic investigation of dance discourse as movement-based communication in context but had not yet benefited from the amount of information provided by live-captured data. It lacked empirical application and was therefore limited in its scope and capability. Bateman (2022, p. 42) highlights this problem with reference to work carried out before empirical data collection and analysis by stating that ‘many multimodal analyses were overly impressionistic, and that analyses tended in any case to be restricted to small-scale studies rarely capable of producing the degree of empirical robustness that would be necessary to improve on impressionistic categories; even when the intuitions underlying such categories are generally sound, it is unlikely that they offer the last word on the precise treatments required’. The FGD application was therefore still restricted to a small case study, a first step that needed to be developed through more empirical work and larger, more complex data analysis.

The first development we achieved when we started working on the rich data provided by the live movement-capture sessions with the English National Ballet was the distinction between more local levels of annotation and more discursive ones. By implementing the FGD in ELAN for annotating our data, we found out that Moves and MBSs create different types of meaning at different levels that are then integrated through the movement flow: the same features that at the more local level of the Move have a specific structural function, at the level of MBS acquire a more discursive one. The analysis carried out on Moves through our annotation method reveals that at this level meaning is created more locally. The annotation of Moves provides three important sets of information that this minimal semantic unit delivers: two sets (starting and arrival) of narrative and interactive projections that the dancer realises for the viewer to interpret within the context of the performance; the positions in which the dancer moves their articulators in relation to each other across space; the flow of relationships between direction and orientation, which not only determines the possible values to be attached to projections for the viewer (i.e., moving towards VS moving away from, going forwards VS going backwards, etc.) but also connects the more local meanings realised at the Move level to the syntactic choices observable at MBS level, where the direction of two consecutive Moves determines the type of MBS trajectory. These findings allowed us to gain an important insight into the mechanism of movement-based communication realised through choreography that the manual analysis simply based on the application of theory and tools did not allow us to uncover; the annotation of the rich data we collected through live movement-capture sessions put us in front of a multi-level discursive complexity that we would not have captured otherwise. The meanings expressed at the more local level of Move acquire a discursive flow in a more complex relation with the performance space at the level of MBS, which provides them with trajectories and highlights discursive patterns where bodies and space are integrated. Thus, whereas in previous manual analyses the focus had been predominantly on reading narrative and interactive projections on the basis of the theoretical framework underlying the FGD, thanks to empirical work our annotation had to take into consideration the analysis of physical movement as a separate but integrated part of the analysis, foregrounding the importance of annotating the positions of the articulators with respect to each other and with respect to the physical space and to the dimensions of direction and orientation. The result was a systematic integration of the physical and the semantic data that are described in integration through the annotation within the same model and according to the same theoretical principles. The consideration of the materiality of dance, which involves also the integration of spatial features, direction, and orientation, led us to a discourse description that anchors the theoretically grounded visualisations of projections to the time and space of the physical dance performance phenomenon.

I will illustrate this point with a simple example of annotation taken from the English National Ballet’s most recent production of Raymonda, a very traditional ballet from the classical repertoire first choreographed by Marius Petipa in 1987. Even in the most recent version, this traditional piece is based on an intricate and equally traditional love-triangle story against a romanticised historical setting, and it is therefore not too difficult for the audience to follow the flow of its scenes and the relationships amongst the characters based on the synopsis presented in the programme, which is supported by a very classical choreography. However, the extracts we annotated reserved us some surprises. Amongst the extracts we chose, there is one from a solo danced by Abdur, one of the three protagonists: in love with Raymonda, who is already engaged with his best friend, in the extract we analysed he finds himself alone with her and declares his feelings by dancing a solo variation. The annotation we carried out is exemplified in Figure 8, where a particular pattern is showing. The annotation includes four Moves and two MBSs and the figure shows annotations both of some physical movement structures and of some narrative and interactive projections. The more local annotation of the four Moves shows a considerable amount of repetition of specific physical movements that corresponds to an equally repetitive series of meanings: through narrative projections, the dancer interpreting Abdur forms repeated connections with Raymonda, who is sitting in the corner in the stage setup with which he is also repeatedly addressing and engaging. Interactive projections show that his interactions are entirely devoted to her and the stage space around her. However, the physical data annotation shows that the same types of narrative and interactive projections are being repeated alternatively in opposite directions and maintaining the same orientation, thus indicating that Abdur is moving back and forth, towards and away from Raymonda, which impacts on the way narrative and interactive projections are perceived by the audience. The annotation itself offers a visualisation of this discursive pattern that develops across two MBSs, which we named ‘mirrored pattern’. It also shows how the more local meanings at Move level are incorporated and shaped into a discursive strategy at the higher level of MBS. These patterns also made us realise that whereas movement orientation is important to capture the local value of narrative and interactive projections within Moves because it determines the perception of the narrative and interactive meanings realised by each set of movement structures, movement direction has a more discursive value because it incorporates those more locally determined meanings within a discursive flow that shows how those meanings can change in relation to the perception of the whole performance space surrounding the dancer. Whereas Abdur’s unchanging Move orientation repetitively shows his focus towards Raymonda, his alternatively changing direction at MBS level shows the conflictual situation in which that focus is experienced by the character.

Figure 8

Figure 8. Example of annotation from Abdur’s sequence in ENB’s version of Raymonda.

Projecting conclusions

I have started this article with an overview of the way the notion of materiality is understood and used as a nexus for connecting the different components of complex phenomena approached by a variety of disciplines and research areas. The pattern that emerged from such an overview highlights how materiality is actually used, as foregrounded by Bateman (2022), as an external language for description that can be applied to several contexts where human experience manifests itself in and is carried out through multimodal meaning-making socio-semiotic practices. The overview also showed that the notion of materiality helps anchoring theoretical advances to phenomenological studies, thus highlighting the importance of empirical data in any analytical activity across disciplines. The developments observed in other research areas were echoed by the developments evidenced in multimodal discourse analysis through an excursus of the work carried out within The Kinesemiotic Body project, which focused on movement-based discourse. In this specific case, the application of the Functional Grammar of Dance in the annotation and testing of empirical data led not only to the further development of this analytical model and its theoretical framework of reference, but also to a much better understanding of the complex structures that underlie movement-based discourse and their interaction with the contextual space in which communication happens, thus providing much stronger foundations for the extension of this type of analysis to forms of movement-based communication other than ballet and dance in general.

Focusing on materiality really means looking at the complexity of human experience and the processes through which it is shaped into semiotic constellations where configurations of modes work in interplay. Stage performances offer great examples of this complexity, involving music, dance, sung or recited text, movement, lighting, settings, costumes, all in need of more empirical investigation. When working on movement-based communication and in particular on movement-based performance, the ‘materiality key’ has opened the door to the integrated work of linguists, computer scientists, semioticians and engineers as it has provided a common ground for collecting, processing and analysing movement data under mutually understood and shared theoretical principles, and also for creating a common language for defining fundamental concepts. It has also highlighted the complex relationships occurring amongst the different factors that enable this type of communication where human bodies interact with space and its perception. Eventually, the project led to the creation of more effective ways of collecting, annotating and understanding movement data. Our work is still ongoing: one of the project’s results was the creation of short videos where live-captured movement data is turned into avatars which can be inserted in virtual stage set-ups that can be modelled ad-hoc for experiments on perception of how the body interacts with contextual space in communication. The avatars represent both female and male dancers and can now be visualised as carrying out projections when dancing across the virtual stage as the software is now capable of reading automatically the FGD annotations in ELAN. We can even select which types of projections to visualise and how to distinguish one type from the other. These visualisations are still undergoing some level of refinement but there is great potential for future applications in dance education for both dance students, professionals and general audience and for different forms of performance studies and movement-based communication analysis. The same principles of visualisation are currently being applied to the study of potential gender bias in the representation of avatars’ movement in popular fighting games, thus extending the work started with The Kinesemiotic Body project to EDI issues related to the gaming world and relevant communities. These extensions of the work carried out by The Kinesemiotic Body project are possible precisely because the advances we made both in analysis and theory benefited from the focus on materiality as a descriptive language for unpacking and understanding the complexity of semiotic resources that work in interplay to produce dance discourse.

Data availability statement

The datasets presented in this study can be found in online repositories. The names of the repository/repositories and accession number(s) can be found at: https://doi.org/10.17028/rd.lboro.c.6230502.v1, Loughborough University Research Repository.

Ethics statement

The studies involving humans were approved by Loughborough University Ethics Review Sub-Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

AM: Funding acquisition, Methodology, Writing – original draft, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. Funding received from Loughborough University through the UKRI.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^The collaboration with John Bateman and the University of Bremen had actually started with John’s fellowship at Loughborough University funded by the Institute of Advanced Studies and aimed at fostering our interdisciplinary collaboration in 2017. The rest of the research team was made by Massimiliano Zecca, Russell Lock, Chun Liu from Loughborough University and Dayana Markhabayeva from the University of Bremen.

References

Asher, N., and Lascarides, A. (2003). Logics of conversation. Cambridge: Cambridge University Press.

Google Scholar

Balmaceda, M., Högselius, P., Johnson, C., Pleines, H., Rogers, D., and Tynkkynen, V. P. (2019). Energy materiality: a conceptual review of multi-disciplinary approaches. Energy Res. Soc. Sci. 56:101220. doi: 10.1016/j.erss.2019.101220

Crossref Full Text | Google Scholar

Bateman, J. A. (2019). Multimodality and materiality: the interplay of textuality and Texturality in the aesthetics of film. Poetics Today 40, 235–268. doi: 10.1215/03335372-7298536

Crossref Full Text | Google Scholar

Bateman, J. A. (2022). Multimodality, where next? Some meta-methodological considerations. Multimodal Soc 2, 41–63. doi: 10.1177/26349795211073043

Crossref Full Text | Google Scholar

Bateman, J. A., and Wildfeuer, J. (2014). A multimodal discourse theory of visual narrative. J. Pragmat. 74, 180–208. doi: 10.1016/j.pragma.2014.10.001

Crossref Full Text | Google Scholar

Bateman, J., Wildfeuer, J., and Hiippala, T. (2017). Multimodality: foundations, research and analysis. A problem-orented introduction. Berlin, Boston: De Gruyter.

Google Scholar

Björkvall, A., and Karlsson, A. M. (2011). The materiality of discourses and the semiotics of materials: a social perspective on the meaning potentials of written texts and furniture. Semiotica 2011, 187–1/4. doi: 10.1515/semi.2011.068

Crossref Full Text | Google Scholar

Bolens, G. (2022). Embodied cognition, Kinaesthetic knowledge, and Kinesic imagination in literature and visual arts. Front Commun 7:232. doi: 10.3389/fcomm.2022.926232

Crossref Full Text | Google Scholar

Elyamany, N. (2023). A chronotopic approach to identity performance in musical numbers: a choreo-musical case study of ‘rewrite the stars’ and ‘this is me’. Vis. Commun. 22, 278–296. doi: 10.1177/1470357220974069

Crossref Full Text | Google Scholar

Gherardi, S. (2017). “Sociomateriality in posthuman practice theory” in The Nexus of practices: Connections, constellations, and practitioners. eds. S. Hui, E. Shove, and T. Schatzki (London-New York: Routledge), 38–51.

Google Scholar

Gibbons, A. (2012). Multimodality, cognition, and experimental literature. London: Routledge.

Google Scholar

Kell, C. (2015). “Making people happen”: materiality and movement in meaning-making trajectories. Soc. Semiot. 25, 423–445. doi: 10.1080/10350330.2015.1060666

Crossref Full Text | Google Scholar

Kress, G. (2010). Multimodality: A social semiotic approach to contemporary communication. London-New York: Routledge.

Google Scholar

Lacković, N., and Popova, B. (2021). Multimodality and sociomateriality of lectures in global universities’ media: accounting for bodies and things. Learn. Media Technol. 46, 531–549. doi: 10.1080/17439884.2021.1928694

Crossref Full Text | Google Scholar

Lee, T.-K. (2014). Translation, materiality, intersemioticity: excursions in experimental literature. Semiotica 202, 345–364. doi: 10.1515/sem-2014-0044

Crossref Full Text | Google Scholar

Leonardi, P. M., and Barley, S. R. (2010). What’s under construction Here? Social action, materiality, and power in constructivist studies of technology and organizing. Acad. Manag. Ann. 4, 1–51. doi: 10.5465/19416521003654160

Crossref Full Text | Google Scholar

Maiorani, A. (2017). “Making meaning through movement: a functional grammar of dance movement” in Mapping multimodal performance studies. eds. M. G. Sindoni, J. Wildfeuer, and K. L. O'Halloran (London/New York: Routledge), 39–60.

Google Scholar

Maiorani, A. (2020). Selling the past and the present alike: streaming ballet for live audiences during lockdown. J Int Cult Arts 1, 1–9. doi: 10.46506/jica.2020.1.2.001

Crossref Full Text | Google Scholar

Maiorani, A. (2021). Kinesemiotics: Modelling how choreographed movement means in space. London-New York: Routledge.

Google Scholar

Maiorani, A., Bateman, J. A., Liu, C., Markhabayeva, D., Lock, R., and Zecca, M. (2022). Towards semiotically driven empirical studies of ballet as a communicative form. Human Soc Sci Commun 9:429. doi: 10.1057/s41599-022-01399-8

Crossref Full Text | Google Scholar

Maiorani, A., and Liu, C. (2023). The functional grammar of dance applied to ELAN annotation: meaning beyond the naked eye. J World Lang. doi: 10.1515/jwl-2023-0050

Crossref Full Text | Google Scholar

Maiorani, A., and Wegener, R. (2022). “Challenging instantiation in modelling movement-based multimodal communication” in Empirical evidences and theoretical assumptions in functional linguistics. eds. E. Asp and M. Aldridge (London – New York: Routledge), 151–169.

Google Scholar

Meissl, K., Sambre, P., and Feyaerts, K. (2022). Mapping musical dynamics in space. A qualitative analysis of conductors' movements in orchestra rehearsals. Front Commun 7:733. doi: 10.3389/fcomm.2022.986733

Crossref Full Text | Google Scholar

Merriman, P., and Jones, R. (2017). Nations, materialities and affects. Prog. Hum. Geogr. 41, 600–617. doi: 10.1177/0309132516649453

Crossref Full Text | Google Scholar

Mouard Ruiz, E. (2021). En movimiento: audiodescripción del acuario del arrecife de coral. Universidad de Granada. Departamento de Traducción e Interpretación. Available at: http://hdl.handle.net/10481/69420

Google Scholar

Pantidos, P., Valakas, K., Vitoratos, E., and Ravanis, K. (2010). The materiality of narrative spaces: a theatre semiotics perspective into the teaching of physics. Semiotica 2010, 305–325. doi: 10.1515/semi.2010.062

Crossref Full Text | Google Scholar

Petrilli, S. (2008). Bodies and signs: for a typology of semiosic materiality. Am J Semiotics 24, 137–158. doi: 10.5840/ajs200824422

Crossref Full Text | Google Scholar

Prové, V. (2022). Measuring embodied conceptualizations of pitch in singing performances: insights from an OpenPose study. Front Commun 7:987. doi: 10.3389/fcomm.2022.957987

Crossref Full Text | Google Scholar

Putnam, L. (2014). Unpacking the dialectic: alternative views on the discourse–materiality relationship. J. Manag. Stud. 52:115. doi: 10.1111/joms.12115

Crossref Full Text | Google Scholar

Schatzki, T. (2010). Materiality and social life. Nat Cult 5, 123–149. doi: 10.3167/nc.2010.050202

Crossref Full Text | Google Scholar

Sindoni, M. G. (2022). Traiettorie della multimodalità: gli snodi teorici e i modelli applicativi. Ital LinguaDue 14:3597. doi: 10.54103/2037-3597/2

Crossref Full Text | Google Scholar

Tyrer, C. (2021). The voice, text, and the visual as semiotic companions: an analysis of the materiality and meaning potential of multimodal screen feedback. Educ. Inf. Technol. 26, 4241–4260. doi: 10.1007/s10639-021-10455-w

Crossref Full Text | Google Scholar

Vidal Claramonte, M. Á. (2022). Translation and contemporary art: transdisciplinary encounters. New York: Routledge.

Google Scholar

Wilf, E. (2011). Sincerity versus self-expression: modern creative agency and the materiality of semiotic forms. Cult. Anthropol. 26, 462–484. doi: 10.1111/j.1548-1360.2011.01107.x

Crossref Full Text | Google Scholar

Wu, X. (2022). Space and practice: a multifaceted understanding of the designs and uses of “active learning classrooms”. UNSW: Sydney.

Google Scholar

Software

ELAN (Version 6.2) [Computer software] . (2021). Nijmegen: Max Planck Institute for Psycholinguistics, The Language Archive. Available at: https://archive.mpi.nl/tla/elan.

Google Scholar