Intradisciplinarity: can one theory do it all?

Martin, J. R.

doi:10.3389/fcomm.2023.1310001

ORIGINAL RESEARCH article

Front. Commun., 05 January 2024

Sec. Multimodality of Communication

Volume 8 - 2023 | https://doi.org/10.3389/fcomm.2023.1310001

This article is part of the Research Topic Drawing Multimodality's Bigger Picture: Metalanguages and Corpora for Multimodal Analyses View all 15 articles

Intradisciplinarity: can one theory do it all?

$\r\nJ. R. Martin$ J. R. Martin^*

Department of Linguistics, University of Sydney, Sydney, NSW, Australia

This position paper draws on Bernstein and Maton's sociology of knowledge to explore Systemic Functional Linguistics (SFL) and Systemic Functional Semiotics (SFS), alongside their relation to Bateman's vision for empirical multimodality research. The paper suggests that SFL/SFS's internal grammar is by and large compatible with Bateman's vision, even if its external grammar falls short insofar as extant descriptions of one semiotic system or another are concerned. The paper closes with the suggestion that SFS and Bateman's multimodality can learn most from one another in research projects embracing a dialectic of theory, description, and ideologically committed practice.

1 Disciplinarity

In recent papers, Bateman (2020a,b); Bateman (2021, 2022a,b) explores his vision for multimodality as an empirical discipline. In doing so, he draws on sociological studies of knowledge structure, including the work by Bernstein (2000), Maton (2011, 2014, 2016), Maton and Chen (2016), Maton and Howard (2016), and Maton et al. (2016). As part of this projection, he warns against falling foul of “various flavors and variations of Saussure's well-known proposal of language (or any other system) as a ‘master template' for semiotics as such” (Bateman, 2022a, p. 47) and what he calls “linguistic imperialism” (Bateman, 2022b, p. 63). In addition, he notes that “predatory” interdisciplinarity “will be rejected from the start” (Bateman, 2021, p. 308).

Read in tandem with Kress's many declarations of a new age of meaning making called “Multimodality” (e.g., Kress, 2003, 2010, 2015), superseding language and the discipline of linguistics, serious questions have to be raised about the work on multimodality informed by a theory of language such as Systemic Functional Linguistics (SFL)—work evolving into something we might call Systemic Functional Semiotics (SFS) via publications such as Kress and van Leeuwen's Reading Images (Kress and van Leeuwen, 1990 and subsequent editions), Caple (2013) Photojournalism, Doran (2018) The Discourse of Physics, Painter et al.'s (2013) Reading Visual Narratives, He (2021) “Toward a stratified metafunctional model of animation,” Ngo et al. (2022) Modeling Paralanguage using Systemic Functional Semiotics, Martin and Unsworth (2024) Reading Images for Knowledge Building, Zappavigna and Logi (2024) Emoji and Social Media Paralanguage, and Yu (forthcoming) Multimodal Knowledge Building in Secondary School Chemistry Textbooks.

Accordingly, in this paper, I will draw on the sociological studies referred to above to explore the nature of SFL and SFS as knowledge structures, compare them with the model of empirical multimodality envisioned by Bateman, and make some suggestions about how his ambitions for the field might be most effectively accommodated. I write as an SFL linguist (discourse analyst in particular), who has been drawn into work on multimodality by research students and colleagues over the past two and a half decades. As such, given the misgivings about the contribution of linguistics noted above, I should perhaps request readers' indulgence—as I suggest that an SFL/SFS perspective need not be read as the foul and predatory one that some of the more logophobic multimodalists apparently fear.

In discussions of this kind, it is important to distinguish multimodality as a field of research and multimodality as its object of study. Multimodalists (like psychologists) unfortunately tend to use the same term for both phenomena (cf., language and linguistics for linguists). Where confusion might arise, I will refer to the field of research as Multimodal Studies below.

2 Knowledge structure

By way of framing the discussion, let us begin with Bernstein's (1996, p. 23) distinction between singulars and regions. For Bernstein, a singular is “a discourse which has appropriated a space to give itself a unique name,” for example, “physics, chemistry, sociology, psychology” and which “created the field of the production of knowledge.” These he contrasts with regions, “a recontextualising of singulars,” for example, “medicine, architecture, engineering, information science,” noting that “any regionalisation of knowledge implies a recontextualising principle: which singulars are to be selected, what knowledge within the singular is to be introduced and related.” Importantly, he goes on to comment that “regions are the interface between the field of the production of knowledge and any field of practice.” Had Bernstein's vision extended into the 21st century, he might well have added multimodality as an emerging region to his list, with media and communication as its field of practice.

Seen in these terms, SFL is a canonical singular (Martin, 2014, 2016) and contrasts with its regionalisation in the Sydney School's well-known genre-based literacy programmes (Rose and Martin, 2012)—which tend to draw on a range of relevant singulars (including, for example, Bernstein and Maton's sociology of knowledge, neo/Vygotskyan social psychology, and strands of critical discourse analysis). One possible reading of Bateman's vision would entail, via design and/or evolution, the transformation of Multimodal Studies into a singular—with its own distinctive knowledge structure deploying an empirical methodology grounding theory and description.

Bernstein's perspective is further elaborated in the distinction he draws between horizontal and vertical discourse (an opposition between what he earlier referred to as common and uncommon sense). A horizontal discourse involves “a set of strategies which are local, segmentally organized, context specific and dependent, for maximizing encounters with persons and habitats.... This form has a group of well-known features: it is likely to be oral, local, context dependent and specific, tacit, multi-layered and contradictory across but not within contexts” (Bernstein, 2000, p. 157). A vertical discourse on the other hand “takes the form of a coherent, explicit and systematically principled structure, hierarchically organized as in the sciences, or it takes the form of a series of specialized languages with specialized modes of interrogation and specialized criteria for the production and circulation of texts as in the social sciences and humanities” (Bernstein, 2000, p. 157).

In addition, two forms of vertical discourse are distinguished—hierarchical knowledge structures vs. horizontal ones. A hierarchical knowledge structure is “a coherent, explicit and systematically principled structure, hierarchically organized” which “attempts to create very general propositions and theories, which integrate knowledge at lower levels, and in this way shows underlying uniformities across an expanding range of apparently different phenomena” (Bernstein, 1999, p. 161–162)—e.g., physics, chemistry, or biology. A horizontal knowledge structure, on the other hand, is defined as “a series of specialized languages with specialized modes of interrogation and criteria for the construction and circulation of texts” (Bernstein, 1999, p. 162)—e.g., linguistic theories which position themselves as functional, arguably West Coast Functionalism, Lexical Functional Grammar, Functional Grammar, Discourse Functional Grammar, Role and Reference Grammar or Systemic Functional Linguistics. Bernstein uses a triangle to symbolize hierarchical knowledge structures since they attempt to create ever more general propositions which account for an expanding range of phenomena (e.g., Newtonian physics, superseded by Einstein's relativity, and superseded by string theory). Horizontal knowledge structures, on the other hand, are visualized by a succession of “Ls” since what counts as development is the introduction of a new perspective, typically by junior speakers who challenge the power and legitimacy of more senior ones (e.g., Marxist history, feminist history, and post-colonial history). A synoptic overview of these distinctions is offered in Figure 1.

Figure 1

Figure 1. Discourse and knowledge structure (after Bernstein, 1999, 2000).

As exemplified above, in Bernstein's terms, SFL is a canonical member of a horizontal knowledge structure comprising many different theories. Bateman's vision for Multimodal Studies is perhaps a more ambitious one, leaning toward the design and evolution of a hierarchical knowledge structure. This is a trajectory that linguistic theories have embraced, without success, since the modern discipline was founded by Saussure (1916/1959).

Wignell (2007a,b) examines the history of social science, focussing on the emergence of economics, political economy, and sociology as “a hybrid of the language of the physical sciences and the language of the humanities” (Wignell, 2007a, p. 202)—suggesting that the stronger the boundaries around one of these disciplines, the more it will evolve the characteristics of a hierarchical knowledge structure. In his 2004 conference presentation of Wignell (2007a), he in fact refers to social science knowledge structures as “warring triangles,” since they, in general, aspire to be recognized as hierarchical knowledge structures (viz., linguists' claims for their discipline as the “science of language”). What happens in practice, however, is that one or another linguistic theory gains institutional rather than intellectual control of the discipline, for a specific period of time, in a specific place (e.g., Chomskyan linguistics' supremacist control of American linguistics and its intellectual dominions in the 1960s, waning not long thereafter). Seen in these terms, Bateman's vision involves strengthening boundaries around what counts as empirical Multimodal Studies, thereby fostering its development as a hierarchical knowledge structure—occluding more “weakly bounded” competing triangles as it does so and enjoying globalized longevity.

Bernstein (2000, p. 132–134) probes more deeply into the characteristics of hierarchical and horizontal knowledge structures in his recognition of internal and external languages of description (which he labels L¹ and L², respectively). L¹ “refers to the syntax whereby a conceptual language is created” or how constituent concepts of a theory are interrelated, and L² “refers to the syntax whereby the internal language can describe something other than itself” (2000, p. 132) or how a theory's concepts are related to referents. Knowledge structures with a strong internal grammar (L¹) have concepts that are tightly interrelated; in hierarchical knowledge structures, this facilitates the deployment of a strong external grammar (L²) whereby concepts are related to data in relatively unambiguous ways. Muller (2007) elaborates on these ideas, focussing on how knowledge structures progress (Muller, 2000, 2011; Moore and Muller, 2002). He introduces the term “verticality” to focus on how internal grammar develops—via ever more general propositions accounting for a broader range of data (more verticality) or the addition of new incommensurable languages of description (less verticality). He introduces the term “grammaticality” to focus on how knowledge structures manage data—via testable hypotheses about a restricted set of referents (strong grammaticality) or via readings of a less restricted set of referents that are hard to disconfirm (weak grammaticality). An outline sketch of these ideas is presented in Figure 2, including a rough positioning of canonical knowledge structures along a hierarchical/horizontal knowledge structure cline. Seen in these terms, Bateman's ambitions for Multimodal Studies involve strengthening internal and external grammars of description so that the field can progress via what Bateman (2020a, p. 71) refers to as “explanatory sophistication” based on “worldly corroboration.”¹

Figure 2

Figure 2. Knowledge structures (vertical discourse).

As far as grammaticality is concerned, Bateman (2021, p. 302–303) draws attention to Maton and Chen's (2016) discussion and exemplification of mediating languages of description and external ones (termed L^1.5 and L², respectively). Mediating languages are designed to be more general and less data-specific than external languages. In SFL, for example, mediating languages comprise what are generally referred to as “descriptive motifs and generalizations” (Matthiessen, 2004)—i.e., general categories such as transitivity, modality, or tense (often presented as complementarities such as transitivity/ergativity, modality/assessment, or tense/aspect). These help a linguist approach the description of the grammar of a language with relatively “soft eyes” before locking into a more specific description of the data to hand. What ends up counting as L¹, L^1.5, and L² is itself a process (Martin et al., 2020a, 2023), unfolding over time, as L^1.5 motifs and generalizations are promoted to L¹ status or L¹ concepts are demoted to mediating L^1.5 language status (or perhaps relegated to L² external grammar). We focus more specifically on this process when we consider the evolution of SFS from SFL below.

3 SFL and SFS (internal and external grammars)

SFL itself comprises a number of different languages of description, as reflected in the Routledge and Cambridge handbooks (Bartlett and O'Grady, 2017; Thompson et al., 2019). Here, we will assume the model developed by Martin (1992, 2010, 2014), which is the one that has most strongly influenced Bateman (e.g., Bateman, 1998, 2008, 2020b)—hereafter referred to simply as SFL. In relation to other social sciences, SFL has a strong internal grammar. Following Saussure (1916/1959), it treats language as a system of signs. Following Firth (1957), it takes the complementarity of paradigmatic and syntagmatic relations as fundamental. Following Halliday (1966, 1992), it skews this complementarity, privileging system over structure. This axial orientation underpins all language description, resulting in external grammar which formalizes valeur in networks of options realized in structure (Martin et al., 2013). Over time, SFL's internal grammar has expanded to include the notion of hierarchy—i.e., realization (levels of abstraction), instantiation (a cline of sub-potentialisation/generalization), and individuation (a scale of allocation/affiliation). Of these, realization has the strongest grammar as systems in system networks bundle together in relation to the size of the structural unit realizing options (rank), the ideational, interpersonal, or textual meaning and corresponding types of particulate, prosodic, or periodic structure involved (metafunction) and the level of abstraction (phonology/graphology/signology, lexicogrammar, and discourse semantics). A synoptic overview of these dimensions (following Martin, 2010) is presented in Figure 3 (using an English MOOD system to represent axis). Of these, both instantiation and individuation are underarticulated compared with realization and constitute major challenges for future research.

Figure 3

Figure 3. SFL internal grammar (L¹). From an axial perspective, for some languages, there is no need to distinguish word and morpheme ranks (since there is no word structure—i.e., no words consisting of more than one morpheme); parentheses make room for this variability at the bottom of the rank scale in this figure.

To this compilation, I will add five elaborations that bear on the discussion. First, form vs. substance. As clarified by Martin et al. (2013), the register of SFL at stake here follows Saussure (1916/1959) and Hjelmslev (1961) in treating language as form, not substance. This means that phonetics is not treated as a stratum of language in its own right. Rather it is a region in Bernstein's terms (interfacing with practices such as speech recognition or speech pathology), drawing on physics (acoustic phonetics) and neuro/biology (articulatory phonetics)—and thus deploying a set of internal and external grammars very different to those employed by linguistics proper (stronger grammars in fact). This is not to deny the relevance of phonetics and phonology to one another (linguistics students are generally trained in both) but simply to acknowledge the very different knowledge structures involved in the description of form as opposed to substance.²

Second, structure and syntagm. As far as the description of grammatical “form” is concerned, SFL does not restrict its description to what Whorf (1945) called phenotypes—i.e., single or multi-segment syntagms consisting (for grammar) of classes of morpheme, word, group/phrase, or clause. To develop rich meaning-making grammatical descriptions, SFL is also inspired by Whorf's notion of cryptotypes. Halliday and Matthiessen (2014) distinction between Epithet and Classifier in English can be used to illustrate this point here. From the perspective of system, English nominal groups make a distinction between describing and classifying. Both types of nominal groups can be realized by the same syntagm (i.e., determiner ^∧ adjective ^∧ noun), but a covert distinction can be uncovered by asking whether the adjective in the relevant syntagm is gradable or not. Describing adjectives are gradable (a really lovely film), whereas classifying adjectives are not (^*a very Korean film).³ Accordingly, the same syntagm is assigned different structures, as in examples (1) and (2) below. SFL grammar descriptions in other words are not simply a catalog of syntagms; they build function structures on top of syntagms to reflect the meaning-making valeur at stake.

(1)

(2)

This means, for example, in relation to an SFS description of some horizontally polarized images that an optional information value system can be set up realized by the function structure Given ^∧ New, without making the claim that all horizontally polarized imagic syntagms in fact realize this system. Relevant options are extended from Kress and van Leeuwen (2021, p. 216–217) in Figure 4. The name of the system is INFORMATION VALUE; it is an optional system; if the feature [newsy] is selected, then the structural functions Given and New are present, in the sequence Given followed by New (with Given realized to the left and New to the right). This formalization makes no claims about all horizontally polarized systems; it simply positions [newsy] ones as having a Given ∧ New structure realized by a horizontally polarized imagic syntagm. Note in passing that this is perhaps too generous a reading of Kress and van Leeuwen's often criticized account of information structure in polarized images, but my point here is that SFS need not fall foul of their apparent overgeneralisations.

Figure 4

Figure 4. INFORMATION VALUE (system and structure). The downward slanting arrow in this diagram specifies the structural consequences of the feature [newsy]—namely, insert the function Given, insert the function New, sequence Given before New, realize Given through a left imagic block, and realize New through a right imagic block. The fourth edition of Reading Images (Kress and van Leeuwen, 2021, p. 217) in fact uses images to specify the realization of imagic functions.

This approach to axis (i.e., paradigmatic and syntagmatic relations) lies at the heart of SFL/SFS descriptions of semiosis; Martin et al. (2013) provide a basic introduction. As in Figures 5, 6, it privileges the formalization of paradigmatic relations as the basic organizing principle of descriptions and derives structural realizations from choices for meaning. The approach leaves open the question of whether systems need to be set up to generalize the syntagms available for realizing function structures in a given semiotic system—with reference to Figure 4, for example, opposing all polarized images to non-polarized ones, and if polarized, all horizontally opposed images to all vertically opposed ones. This could be important if polarized images are used to realize different function structures (and thus different meanings) for a given semiotic system. Arrows (as opposed to lines) and grids (as opposed to vertical or horizontal alignment) are good examples of imagic syntagms that arguably need generalization in such terms—since arrows are used to realize motion or links, for example (not to mention the system network specific uses of arrows in Figure 4), and grids can realize cross-classification (as in linguists' paradigms) or momented activity (as in comics), for example (not to mention culturally specific arrangements such as that organizing Shirley Purdie's remarkable artwork Goowoolem Gijam “Gija plants” which features at the Museum of Contemporary Art in Sydney).⁴

Figure 5

Figure 5. MACRO-GROUPING systems (Martin and Unsworth, 2024, p. 107).

Figure 6

Figure 6. English MOOD systems and delicacy. In the realization statements headed by arrows in this network, “+” indicates the presence of a function, “/” indicates conflation of functions, and “∧” sequences functions, with respect to one another or first position (marked by “#”); distinct realization statements are separated by “;” (Martin et al., 2013).

Martin and Unsworth (2024) take this step in their work on secondary school science infographics, drawing directly on work by Hiippala et al. (2021), Hiippala (2023). Martin and Unsworth's network for MACRO-GROUPING is presented in Figure 5. Therein, a square bracket means “or” (as shown in Figures 2, 3), a slanted square bracket indicates a cline, a brace means “and,” and a combination of brace and square bracket means “and/or.” So for the CO-TEXT systems, we have the option of including a text block or not, and if we choose to do so, we can include a caption or an interpolation or both. To follow one path in the DESIGN system, if we choose line, it can be more or less vertical or horizontal or both (in the latter case we end up with a grid).

If this syntagm oriented step is taken, then an analysis dedicated to such regularities of form can be established (e.g., Caple, 2013, 2022 on BALANCE systems for images), and some kind of stratification of “meaning” and “form” can potentially be brought into the description (as in He's, 2021 work on animations). There is nothing in the knowledge structure of SFS, as informed by SFL, blocking stratified generalizations of this kind.

A related point about knowledge structure and SFS can be made in relation to “etics” and materiality. As van Leeuwen (1999, 2011) shows through his work on parametric systems for sound and color, axis can be used to formalize descriptions that cover the material oppositions which afford traces of function structures and syntagms of the kind introduced above. I would hesitate to refer to these systems as a stratum of language or any other semiotic system since clearly something other than semiotic internal and external grammars inform their description (the binary scaled simultaneous nature of “parametric” systems reflects exactly this point); in Hjelmslev's terms, we are dealing with substance, not form. As emphasized above in relation to phonetics and phonology, this is not to suggest that work on materiality is not relevant to semiosis. It is simply to restrict stratification to cases where we have bundles of interdependent systems at different levels of abstraction (in a pattern of patterns relationship referred to by Lemke, 1984 as metaredundancy). The relation of “emics” to “etics” is not strictly speaking a pattern of this kind.

Third, axis. In SFL, other dimensions of internal grammar are all articulated in relation to axis (Martin et al., 2013). The concepts of rank, metafunction, and strata, in other words, are all based on bundles of interdependent features (organized with respect to constituency, type of meaning, or level of abstraction, respectively). Instantiation has to do with the manifestation of system in text and the generalization of instances as system, over time—as texts unfold (logogenesis), as speakers mature (ontogenesis), or as languages evolve (phylogenesis). Individuation has to do with the allocation of systems to members of a culture and their use of those systems to affiliate in social groups—once again, over time. Take away axis (e.g., O'Toole, 1994) and you may arguably be left with one kind of functional theory or another but not Systemic Functional Linguistics or Systemic Functional Semiotics as the disciplines are being construed here.

As suggested by Martin (2011a), in multimodal studies which take axis as fundamental (i.e., SFS), it is critical not to make a priori assumptions about how systems will enter into interdependency relations with one another. Depending on the semiotic system in question, constituency (rank), kind of meaning and type of structure (metafunction), and level of abstraction (stratification) may shape external grammar (L²) but may not. Looking across the SFS studies surveyed in Table 1, the constant L¹ notion is axis. Accordingly in SFS rank, metafunction and strata are clearly better positioned as mediating L^1.5 notions—possibly shaping the description (L²), possibly not.

Table 1

Table 1. SFS studies in relation to axis, rank, metafunction, and strata.

Fourth, delicacy. Recognition of mediating languages of description (L^1.5s) carries with it the idea that the relation between L¹ and L² can be treated as a cline. SFL's approach to axis is well adapted to a conception of this kind since systems are arranged from more general to more specific along a cline referred to as delicacy. Thus, in Figure 6, for example, indicative clauses are more general than interrogative ones, which are in turn more general than wh ones and so on. This makes it possible to be more and less specific about what gets treated as L¹, L^1.5, and L²–perhaps treating the [indicative/imperative] opposition as L^1.5, but remaining agnostic about more delicate options, pending construction of L² (i.e., a specific language's grammar of MOOD). It is important to keep in mind in relation to this point that positioning more or less general systems as L^1.5 can be done without making any claims at all about how such systems are realized in structure. Commitment to structural realizations of mood options needs to be withheld for L², since structures realizing mood vary considerably across languages (Martin, 2018; Martin et al., 2021).

Fifth, context. In SFL, context is treated as form (Figure 7), not substance (as connotative semiotics in Hjelmslev's terms). In the model of SFL assumed here, context is stratified as register and genre (Martin, 1992; Martin and Rose, 2008), and register is modeled metafunctionally in proportion to its realization in language—i.e., field is to ideational meaning, as tenor is to interpersonal meaning, and as mode is to textual meaning. This move allows context strata to be treated as resources for making meaning and modeled axially, comparably to language ones (Doran and Martin, 2021; Doran et al., 2024; Martin and Doran, in press). These context dimensions of SFL's L¹ are less well articulated and more controversial than language ones—probably because of their level of abstraction and concomitant realization as patterns of language patterns in addition to a lack of clarity in argumentation as far as the distinction between realization and instantiation is concerned.

Figure 7

Figure 7. Stratified model of language and context.

We are now in a position to compare SFS with Bateman's proposals for the basic L¹ of what he refers to as a semiotic mode (a semiotic system in SFS).⁵ In doing so, we need to keep in mind the potential of SFS's L¹ in relation to extant descriptions of semiotic modes (L²). External grammar (L²) can be critiqued and revised in ways that do not call for renovations or reconstructions of L¹ or L^1.5. Some shortcomings, in other words, are more serious than others—as we shall flag below.

4 Bateman's model of multimodality

SFL's stratified model of language and context, as presented in Figure 7, provides a useful point of departure for comparison with Bateman's diagrammatic illustrations of his modeling of semiotic modes. Bateman et al. (2017, p. 117) introduce the diagram in Figure 8, consisting of three strata—“the material substrate or dimension, the technical features organized along several axes of descriptions (abbreviated as ‘form'), and the level of discourse semantics.” Compared to SFS, this treats the “etics” of materiality as a stratum proper and groups it together with the stratum of form—the two together realizing discourse semantics.

Figure 8

Figure 8. Bateman et al. (2017) “abstract definition of a semiotic mode.”

Bateman's approach to discourse semantics is a dynamic one, whereby what he calls form is imbued with meaning as texts unfold. As outlined in Bateman (2022b, p. 69), forms are not treated as already themselves contextually meaningful. He continues:

“…meanings are only mediated by the application of the discourse semantics of the semiotic mode… Thus, as an example, whereas the often used classification of graphical resources set out in Kress and van Leeuwen (2006 [1996]: 59–68) might classify graphical ‘arrows' as ‘narrative processes' (by virtue of their directionality as vectors), from the perspective of the approach adopted here this conflates two semiotic strata of description: the formal level at which visual properties of connection and directedness properly reside, and the discourse semantic level at which, under certain circumstances, it may be possible to abduce that the graphical connective is serving a ‘narrative purpose' (but then, in other circumstances, it may not be). … meaning-making using semiotic modes is best characterised as discourse ‘unfolding' and it is this that offers a higher ‘unity' to any material regularities exhibited.”

This characterization of meaning making in semiotic modes is helpfully reviewed in Bateman (2020b) in relation to Martin (1992) model of discourse semantics. Seen in SFS terms, Bateman's discourse semantics is strongly focussed on instantiation—logogenesis in particular. From this perspective, meaning is only ever something that can be abduced in relation to co-text and context as texts build meaning—rolling out a snowball of semiosis as they unfold. So what Bateman treats as a stratum called discourse semantics, SFS would interpret from the hierarchy of instantiation, not realization. SFL's discourse semantic stratum, along with lexicogrammar and phonology, would all be treated as form in Bateman's modeling (i.e., “technical features organized along several axes of descriptions”; Bateman et al., 2017, p. 117). Of these, Martin's notion of covariate structure (e.g., Martin, 2015) comes closest to Bateman's conception of discourse semantics—since covariate structures instantiate discourse semantic systems through a process of abducing relations of indefinite extent as texts unfold. Martin's stratum of discourse semantics would have to be interpreted in Bateman's modeling as proposals for the systems of relations that can be so abduced.⁶

In at least one articulation of Bateman's model (Bateman and Schmidt, 2012, p. 81), discourse semantics as well as form is presented as involving paradigmatic systems of choice and syntagmatic organization imposing structure. This modeling informs Bateman (2007) work on semantic relations between shots in film (his “grande paradigmatique”) and Tseng and Bateman (2011) description of filmic identification—with systems formalizing relations to be abduced. In later work, perhaps because of reservations about the synoptic, non-dynamic nature of system networks,⁷ axis seems to be reserved for the stratum of form. In its place, at the level of discourse semantics, lists of relations, elaborated from Mann and Thompson (1986) Rhetorical Structure Theory (RST), regularly function as reservoirs of meaning to be abduced (e.g., Bateman, 2008; Hiippala, 2015).

As acknowledged in Section 3, compared to realization, instantiation is to date a still developing hierarchy in SFL. Suggestive explorations include Martin (2006) on intralingual re-instantiation, de Souza (2013) on interlingual re-instantiation, Martin (2010) on coupling and commitment, Painter et al. (2013) on intermodal convergence (concurrence, resonance, and synchronicity), Martin and Matruglio (2013) on presence, Martin (2017) on mass, Martin and Doran (in press) on context as realization vs. instantiation, and Martin and Unsworth (2024) on syndromes of instantiation referred to as mass, presence (and association in (Martin and Doran, in press). None of these approaches comes anywhere near the level of explicitness and detail underpinning work by Bateman and his colleagues (e.g., Bateman and Rhonhuis, 1997; Wildfeuer, 2021), inspired as it is by Asher and Lascarides' work on the logic of abduction (e.g. Lascarides and Asher, 1991, 1993; Asher and Lascarides, 2003). This conceptual shortcoming is clearly one area where SFL and SFS are certain to be positively influenced by Bateman's modeling of multimodality (discourse semantics in particular).

In addition to the three strata outlined in Figure 8, alternative L¹ imaging of Bateman's model includes a more abstract stratum comprising “context, social norms, and values” (Bateman and Schmidt, 2012, p. 81) or “context/register/situation” (Bateman, 2022b, p. 69), and in related work (e.g., Bateman, 2008, 2016), the notion of genre is brought into the picture. This work resonates with SFL work on modeling context as register and genre, and there is clearly room for ongoing collaboration in this area. That said, one weakness of some SFS work on semiotic systems such as that noted in Table 1 is the lack comprehensive treatments of register and genre—especially for descriptions clearly inspired by Kress and van Leeuwen (1990 and subsequent editions) grammar of images. Further work, modeled on the attention paid to field and genre in Doran's (2018) work on physics and mathematics or to field in Yu (forthcoming) work on chemistry, is clearly in order.

5 Knowledge and knowers

As foregrounded in Maton's Legitimation Code Theory (LCT), disciplinarity needs to be considered from the complementary perspectives of knowledge and knowers (Maton, 2014). Accordingly, LCT's legitimation code referred to as specialization takes into account both epistemic relations (between knowledge and what it describes) and social relations (between knowledge and who is producing it). Based on the relative strength of epistemic and social relations, LCT establishes a topology⁸ of legitimation codes, with four principal “modalities”—knowledge codes (ER+, SR–) for which legitimacy depends on what you know; knower codes (ER–, SR+) where what matters is who you are; elite codes (ER+, SR+) where it matters both what you know and who you are; and relativist codes (ER–, SR–) through which everyone's voice and knowledge is equally valid (Maton, 2014, p. 30–33). This framework is exemplified in Figure 9 with reference to speaking rights in 19th century debates about evolution. The key players were Watson (knowledge code), a middle class biologist who made a living by selling specimens to private collectors and museums, Darwin (elite code), an wealthy biologist who married into the Wedgewood family pottery fortune, and Bishop Wilberforce (knower code), a gifted public speaker and high ranking clergyman. They are positioned according to specialization in Figure 9 below. Tellingly, it is Darwin (not Watson) who comes readily to mind when we think of evolution, along perhaps with Bishop Wilberforce who played the role of Huxley's protagonist in the famous debate at a meeting of the British Science Association in 1860. To complete the picture, I have imagined a know-it-all nameless blogger (relativist code), opining about evolution in the 21st century.

Figure 9

Figure 9. Illustration of LCT's specialization codes; adapted from Maton (2014, Figure 2.1).

Turning to multimodality, we might position Bateman as legitimized by a knowledge code, van Leeuwen (a professional jazz musician and filmmaker) by an elite code, someone such as Lima⁹ by a knower code, and our blogger (still blogging) by a relativist code.¹⁰ Kress's positioning would have to be a more transitional one, beginning ER+/SR– with the publication of Reading Images (1990), but sliding toward ER–/+SR thereafter—viz., publications such as Literacy in the New Media Age (2003) and Multimodality: a social semiotic approach to contemporary communication (2010) which typify his later work, rarely draw on any analysis at all from Reading Images and popularize his thinking. With his passing in 2019, the field of multimodality lost its most influential knower. This is significant because turning a region into a singular with a strong internal and external grammar requires a range of voices (including popularizing knowers), even if legitimation via a knowledge code is what protagonists such as Bateman ultimately have in mind.

Digging deeper into epistemic relations, Maton (2014, p. 175–177) distinguishes relations between knowledge practices and the part of the world they are oriented to (ontic relations, OR) and relations between knowledge practices (discursive relations, DR). Stronger ontic and discursive relations (OR+/DR+) establish a purist code, which emphasizes both the object of study and how it is studied. Weaker ontic relations but strong discursive relations (OR–/DR+) establish a doctrinal code, which legitimate a multiplicity of objects of study but foregrounds a particular way of studying them. Stronger ontic but weaker discursive relations (OR+/DR–) allow for a situational code, whereby a specific object of study is in focus but it can be approached from multiple points of view (or as Maton allows, no clear knowledge code at all). Weaker ontic and discursive relations (OR–/DR–) legitimate an approach, which is unlimited with respect to both what is studied and how. As suggested in Figure 10, a framework of this kind would position SFL as purist (since it studies language from one theoretical perspective), SFS as doctrinal (since it studies any semiotic system but always from the perspective of SFL informed theory), social semiotics (a la Kress and van Leeuwen, 2005) as DR–/OR– (since it encourages the study of multimodality from different points of view), and something such as information visualization as DR–/OR+ (since it focusses on graphic representations of complex data by whatever means afford a clear “synoptic” overview). Seen in these terms, one of Bateman's objectives is to re-orient the current trajectory of Multimodal Studies, which at some conferences seems to sprawl toward ever weaker discursive and ontic relations; he wants to shift its trajectory toward stronger discursive relations whatever its object of study (without, we might reiterate here, falling foul of the linguistic imperialism and predatory interdisciplinarity that SFS's doctrinal stance might be accused of).

Figure 10

Figure 10. Multimodal discourse analysis and Maton's epistemic relations (insight); adapted from Maton (2014, Figure 9.3).

The main message to take from Maton's work on specialization and epistemic relations is that disciplines involve both knowledge and knowers. Prescribing strong internal and external grammars for Multimodality Studies is not enough; a given field needs knowers as well. In this regard, SFS has the advantage of being able to recruit both knowledge and knowers from SFL, since in practice SFL informed discourse analysis cannot avoid bumping into multimodal texts and the technicality of SFL's internal and external grammar is already in play. Given Batemans' vision for Multimodal Studies, recruiting knowers from social semiotics is perhaps more of a challenge since its relatively weak internal and external grammar and its multidisciplinary stance make stronger grammar a harder sell. Work that draws on Lascarides and Asher (1991, 1993) to formalize the complexity of discourse semantic abduction seems certain to frighten large numbers of OR–/DR– multimodalists well away.

6 Fair play

In this paper, I have drawn on Bernstein and Maton's sociology of knowledge to explore SFL and SFS in relation to Bateman's vision for empirical multimodality research. There is of course much more to survey. In closing, let me just highlight three main points here.

First, there is the question of which theoretical dimension is privileged as fundamental. For SFL/SFS, this is axis; all other dimensions of the theory, including stratification, depend on a specific conception of paradigmatic relations underpinning and underpinned by syntagmatic ones. For Bateman, the fundamental dimension is stratification, further specified as materiality, form, and discourse semantics. So where SFL/SFS derives strata from axis depending on the interdependency of systems according to levels of abstraction in a particular semiotic system, Bateman's vision assumes three strata and in recent work uses axis to characterize just one of these (i.e., form). Related to this point is Bateman's treatment of materiality as a stratum, whereas in the model of SFL/SFS assumed here, it would be treated as “etic” substance and explored through knowledge structures that have evolved for the study of physical and biological reality (as opposed to those which have evolved for exploring semiotic reality, i.e., systems of meaning).

Second, and perhaps most crucially from Bateman's perspective, discourse semantics is approached from a dynamic perspective by Bateman—with form imbued with meaning through a process of abduction as texts unfold. This is compatible with SFL/SFS's approach to instantiation (logogenesis in particular) and its conception of covariate structure, but as noted above, SFL/SFS description (L²) has not caught up with theory (L¹) as far as instantiation is concerned. Bateman's misgivings about SFL/SFS's many promissory notes in this regard are right on target.

Third, in SFL/SFS, key concepts are deployed across strata. One dimension, axis, is fractal; all strata, ranks, and metafunctions are explored axially, and systems of choice shape SFL/SFS's conception of hierarchy (realization, instantiation, and individuation). This axial orientation grounds decisions for a specific semiotic system—with respect to how many strata, how many ranks, and which metafunctions (if any) are presumed as L¹, suggested as L^1.5 or described as L² (Martin et al., 2013). Bateman's model is more modular in design, with distinct internal and external grammars proposed for each stratum (and for context, it would appear, once we move beyond his semiotic modes). The accessibility of work on materiality by Bateman et al. (i.e., their slices of canvas) contrasts markedly with the technicality of their adoption of Lascarides and Asher's formalization of the logic of abduction. To be frank, it is clear to me that such an approach potentially formalizes the complexity of what is going on ideationally in logogenesis as far as the snowballing of meaning is concerned, but I am much less clear about how it manages this complexity for descriptive or applied purposes (especially once we scale up and move beyond the fragments of exemplificatory discourse used as illustrations of the approach). This may simply be a matter of unfamiliar technicality and the challenge it imposes on outsiders (such as myself), but it may be more than that. As I often tell my research students when they are feeling overwhelmed by the phenomena they are describing, there is a difference between documenting complexity and managing it. The job of internal and external grammars in any discipline is to manage complexity, not simply catalog it. I will leave the much needed discussion of this instantiation modeling crisis for another time (to another generation perhaps, who can come to our rescue in this regard).

Overall, my comment would be that SFL/SFS's internal grammar is by and large compatible with Bateman's vision, even if its external grammar falls short insofar as extant descriptions of one semiotic system or another are concerned. This is hardly surprising since SFL has had a considerable influence on Bateman's thinking. Where differences arise, I think that by and large the models can learn from each other—provided suitable contexts for working together are formed.

Drawing on one knowledge structure to position two others as I have done is a challenging task, but it has the salutary advantage of drawing attention to the demands of coming to grips with the technicality of incommensurable L¹s—which can be forbidding given the every-worsening time constraints of academic life and the challenge of practical applications which demand solutions yesterday for what really needs to happen tomorrow. This takes me to my final point, appliability, which in my experience bears critically on what it takes to come to grips with an unfamiliar L¹ and the L² descriptions it affords.

Halliday (2008, p. 7) coins the term “appliable linguistics” for the dialectic of theory, description, and practice informing his linguistic work. An outline of this problem-oriented perspective is outlined in Figure 11 (with multiple Ls allowing for the probability of a number of singulars influencing a given region of practice). This orientation to linguistics was an unusual one in the 20th century, the closest parallel being Pike (1988) tagmemics (viz., Pike's tagmemics and his “maxim” from 1988: “I wanted a theory that would allow one to live outside the office with the same philosophy one uses inside it”).¹¹

Figure 11

Figure 11. Dialectic of theory, description, and practice.

The challenge for SFS and Bateman's Multimodal Studies, as I see it, lies precisely in finding contexts of application (in educational semiotics, clinical semiotics, forensic semiotics, etc.), which foster a dialectic of theory, description, and practice. It is in these contexts, especially if knowledge workers share a politics in relation to a specific problem, that the challenge of incommensurable technicality can be overcome. This dialectic works best when theory and description provide complementary perspectives on the same data. I am not in other words talking about interdisciplinarity (i.e., “you do your bit, I'll do mine”) but rather about transdisciplinarity (i.e., “this is how I see it, how about you?”). By approaching the same data from a different point of view, with a practical politically charged challenge in mind, theories can learn from one another—if they can make explicit what another theory is interested in and/or draw attention to regularities another theory/description has missed (Martin, 2011b; Maton et al., 2016; Martin et al., 2020b).¹²

It is for this reason that I have done what I can to encourage the evolution of SFS out of SFL—so it can “trespass” into conversations of this kind. Dialogue between SFS and Bateman's vision for an empirical multimodality is likely to be far more productive than one involving SFL rather than SFS, precisely because SFS and Multimodal Studies can focus on the same data from complementary points of view. For me, then it is important to avoid dialogue in which linguists focus on language and multimodalists deal with everything else. Multimodal discourse needs to be the focus of all parties in the conversation.

In saying this, I hope I am allaying fears about linguistic imperialism and have presented a less-than-predatory vision of SFS and SFL. One abiding concern I have is the “logophobia” generated by multimodality “knowers” through their by now rather dated proclamations of a new multimodality age and the striking absence of any language analysis to speak of in key multimodal conferences and publications. Language needs to be part of the picture, if multimodalists are serious about what is going on.

The title of my paper is of course a provocative one—which I propose to suggest that we need to be careful about how we weigh up the advantages and disadvantages of one large encompassing theory such as SFS (my “intradisciplinarity”) in relation to the advantages and disadvantages of the more generally celebrated interdisciplinary approach. This provocation is licensed in this context I hope by the incisive provocations marking Bateman's interventions over many decades—calling out what needs calling out and challenging thinkers to think some more. Whenever my students embark on multimodal research, I warn them, “Read Bateman; he's someone that is truly serious about what is going on.” For all this, John, my sincere thanks, many times over; and best wishes for the next phase of your brilliant career.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author/s.

Author contributions

JM: Writing – original draft.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^It is important to acknowledge that this modeling presents a “deficit” view of the humanities, a point which needs to be redressed but is unfortunately beyond the scope of this paper.

2. ^Note that in more recent work, Halliday and Matthiessen (2014, p. 26) adopt a position similar to Bateman's, treating phonetics as a stratum of language. I will not pursue an argument with them here.

3. ^Unless we are in fact using Korean as an Epithet, describing characteristics of the genre.

4. ^This grid features 72 45 cm by 45 cm paintings of Kimberley flora, in four rows of 18 panels each, arranged top down as rows of taller plants and trees, then smaller plants and shrubs, and then plants from in or around water and ground dwelling plants—an Indigenous arrangement which could only be abduced by viewers very familiar with Gija culture.

5. ^As noted above, the term mode is used for a register variable in SFL, so the term semiotic system is preferred to semiotic mode in SFS.

6. ^Tellingly, in Martin (1992), realization statements are not provided for covariate structures realizing discourse semantic options—precisely because such structures have to be abduced as discourse semantic systems are instantiated in texts.

7. ^This problem is elaborated in relation to exchange structure and genre analysis in Martin (1985). Bateman (2020b) comments on the lack of progress in SFL as far as transcending axis and developing meaning building approaches to unfolding discourse are concerned.

8. ^The horizontal and vertical axes in the LCT derived topologies in Figures 9, 10 represent clines, as imaged by the double-headed arrows.

9. ^Lima is the only “multimodalist,” as far as I am aware, who has been called on to give a TED talk. His best-known publications (Lima, 2011, 2014, 2017) are popularisations.

10. ^I should perhaps emphazise here that codes do not ascribe value to speakers in different ways; they simply characterize the factors that legitimize a given voice.

11. ^From “Kenneth L. Pike Maxims”; https://www.sil.org/about/klp/pike-maxims. Pike's Christian motivations were of course very different from Halliday's Marxist ones (Halliday, 2015).

12. ^Note in this regard that I have strong reservations about the viability of what van Leeuwen (2005) refers to as an integrationist model of interdisciplinarity involving several knowledge structures which develop a 'common metalanguage'. Singulars are incommensurable, and we have to always be on guard against borrowing terms or concepts from one knowledge structure into another and assuming they mean the same thing; they will not. What I am suggesting rather is that knowledge structures provoke one another and react by developing theory and description in their own terms. This is not of course to foreclose the possibility of a new Multimodal Studies singular emerging out of interdisciplinary or transdisciplinary work, which is on my reading what Bateman has in mind.

References

Asher, N., and Lascarides, A. (2003). Logics of Conversation. Cambridge: Cambridge University Press.

Google Scholar

Bartlett, T., and O'Grady, G., (eds) (2017). The Routledge Handbook of Systemic Functional Linguistics. London: Routledge. doi: 10.4324/9781315413891