Interactive Digital Narratives as Complex Expressive Means

Bellini, Mattia

doi:10.3389/frvir.2022.854960

ORIGINAL RESEARCH article

Front. Virtual Real., 26 May 2022

Sec. Virtual Reality and Human Behaviour

Volume 3 - 2022 | https://doi.org/10.3389/frvir.2022.854960

This article is part of the Research Topic Interactive Digital Narratives Representing Complexity View all 7 articles

Interactive Digital Narratives as Complex Expressive Means

Mattia Bellini*

Research Group on Narrative, Culture, and Cognition, Institute of Cultural Research, University of Tartu, Tartu, Estonia

Is our way of expressing meanings through digital interactive artifacts simple? How does our sensemaking work when we try to understand Interactive Digital Narratives? To answer these and other questions, the present article discusses a complex-systemic understanding of the expressive mechanisms of Interactive Digital Narratives, to argue the expressive complexity of these artifacts. Interactors of Interactive Digital Narratives necessarily base their hermeneutic processes mainly on what is conveyed in the artifact itself; yet the question of how meaning is expressed in (and sense-making is guided by) Interactive Digital Narratives remains significantly open. I contend that sense-making in such artifacts works by synthetizing the knowledge coming from a number of layers of information, which are intercurrent, interdependent and interoperating, and which concurrently participate in the creation of an overall meaning of a higher order. According to complex systems theory, these layers are therefore elements of a complex system: this justifies the understanding of Interactive Digital Narratives as complex expressive means. Even though largely unexplored, this understanding may help advance our knowledge of the representational capabilities and affordances of Interactive Digital Narrarives, not least in representing multifaceted worlds and complex phenomena. A complex-systemic view can also improve our comprehension of the interpretative processes involved in the sense-making of Interactive Digital Narratives. Furthermore, the awareness gained through this understanding could be useful to get a better sense of the impact of the narratives featured in these artifacts, and ultimately to create more engaging and more powerful experiences that can help foster the societal impact of Interactive Digital Narratives.

1 Introduction (or What to Expect)

The expressive power of Interactive Digital Narratives (IDNs) and video games¹ is by now unquestioned: as Roth and Koenitz maintain, “IDN is a form of narrative expression in the digital interactive medium” (Roth and Koenitz, 2016)². This is also proved by the number of studies discussing the entertainment-related aspects of such artifacts (e.g., among the others, Sheldon, 2004; Koster, 2005; Salen and Zimmerman, 2005; Fullerton, 2008; Schell, 2014; Battey, 2017; Miller, 2019; Egenfeldt-Nielsen et al., 2020), but also–and in particular–their aesthetic qualities (e.g. Goggin, 2006; Bogost, 2007; Tavinor, 2009; Knoller, 2012; Domsch, 2013; Elson et al., 2014a; Ciccoricco, 2015; Anable, 2018; Backe, 2020; Karhulahti, 2020). One of the main assumptions on which all these studies are based is that IDNs are different compared to more traditional and contemplative media. Their participatory nature and the interactivity they afford produce significant differences in the hermeneutic mechanisms conducive to their understanding. The strategies for interpreting IDNs have been thoroughly studied in recent years, and several discussions developed around the effort to describe such functions (Karhulahti, 2012; Koenitz et al., 2015; Roth et al., 2018; Knoller, 2019). However, the ways in which these interpretative modes are set in motion received much less attention (on this, cf. Elson et al., 2014b).

In this article, I propose to understand meaning expression in IDNs as coming from different layers of information (Grishakova and Poulaki, 2019), which interact and mutually inform each other in a loop, forming a system that could be considered complex. First, I will trace a route of some of the main theories that has been proposed to explain IDN comprehension. Moving forward, I will explain what I mean with “layer of information,” and I will show why a systemic view of the expressive mechanisms working in IDNs is justified. I will analyze three of these layers separately, highlighting the feedback loops between them, and I will show the reasons why I believe they could be deemed to exhibit features of a complex system. Then, I will discuss a practical example of the functioning of these layers by looking closely at a very short scene of the game Detroit: Become Human (Quantic Dream, 2018). Thanks to this case study, I will explain why IDNs could be regarded as complex expressive means. Finally, I will show how this definition can elucidate the capability of IDNs to represent complex topics, something which has also been demonstrated by the many contributions to this special issue.

To give a better sense of what is the focus of the present investigation, it may be useful to draw a comparison with the triad usually employed in literary criticism, which includes author, text, and reader: if we were to see IDNs as texts, then what I am going to investigate is the text-reader relationship. Therefore, in more specific terms, my aim is to look at how these texts express a meaning that is understood by human readers. Or, leaving aside the terminology of literary studies, to look at how IDNs convey certain meanings to their human interactors.

2 Interactive Digital Narratives Comprehension

In order for interactors to understand the meaning of an IDN, such encoded meaning needs to be expressed through the IDN itself. This assumption might seem fairly obvious and still, the question of how meaning is expressed in (and its comprehension is guided by) such artifacts remains open for the most part. The effort of answering this question is currently pursued by several researchers, who are trying to parse and understand the functioning of the various elements forming video games and, with them, IDNs. Expectably, narrative was among the elements receiving the largest attention. In this field, one of the most famous theories is that of “Narrative Architectures,” by Henry Jenkins, (2003). In his article, Jenkins defines his term referring to urban design practices, and suggests creating in video games a complex structure of intercurrent, interdependent and interoperating bits of narrative that are presented sparsely and sometimes in indirect ways to the audience. Jenkins’ understanding accounts both for the multimodality of narrative presentation in such artifacts, and for their sensorimotor nature. Much more recently, Kristian Hjaltson and colleagues showed empirical evidence linking the sensorimotor experience of video games to mnemonic recollection even in games with multimodal systems made neutral, i.e., deprived of characterizing features (Hjaltason et al., 2015). To the best of my knowledge, except for rare cases that will be touched below, most of the studies investigating similar issues focus only on specific elements or clusters of elements forming IDNs or video games, including, among others, studies on the lights and lighting effects (e.g. Seif El-Nasr et al., 2006; Knez and Niedenthal, 2008), on colors and color palettes (e.g. Geslin et al., 2016; GomezRomero-Borquez and Del-Valle-Soto, 2020) and on graphics in general (e.g., Lee L. et al., 2016; McLaughlin et al., 2010; Clarke and Mitchell, 2007), on audiovisual style (Järvinen, 2002), on music (e.g. Zehnder and Lipscomb, 2006; Munday, 2007), and on game mechanics (e.g. Fiadotau, 2015). However, this approach seems to be partial and partly arbitrary, like analyzing the functioning of verbs in a novel without taking in consideration their being part of sentences, paragraphs and chapters: possibly insightful, but extremely partial.

Kuvich and Perlovsky, (2013) maintain that the mind uses a system of interconnected cues at all times, and not just in specific cases like dealing with IDNs. Similarly, Maitlis and Christianson’s definition of sensemaking refers to an “attending and bracketing cues in the environment, creating intersubjective meaning through cycles of interpretation and action, and thereby enacting a more ordered environment from which further cues can be drawn” (Maitlis and Christianson, 2014 cf. also Eysenck and Keane, 2005). As it has been empirically showed, these cues come, among other sources, from perceptual processes, motor memories and prediction models (Kuvich, 2005; Perlovsky and Ilin, 2010). In addition to the more discursive explanation of the previous paragraph, this shows why one could not consider a unique source of information when approaching the expression of meaning in IDNs. Rather, IDNs should be regarded as systems of cues, and the nature of the connections between these cues should be taken into account, as was already argued in literature: “understanding […] mind processes requires understanding of the entire system context” (Kuvich and Perlovsky, 2013).

Elson and colleagues advance similar considerations: in two articles (Elson et al., 2014a; Elson et al., 2014b), they mention the paucity of research on the expressive elements of video games, and, in an attempt to address the same gap in critical discourse, they propose to adopt a view analogous to the present one, through what they call “the IMP³ framework”. This framework identifies three main elements of video games that contribute to determine the playing experience, namely game narrative, mechanics, and context dimension. In their understanding, too, the three elements are interacting and mutually informing: “the IMP provides a general model describing a feedback loop of uses, experiences, and effects in specific contexts with a focus on the relationships between game characteristics and experiential variables in the actual playing phase” (Elson et al., 2014a). However, unlike my current proposal, the IMP framework presents quite a different scope and perspective: it discusses existing empirical approaches usable to analyze the components that form the overall game experience (Elson et al., 2014b). Due to this objective, they only look at the interactions of their three elements to outline differences in recorded experiences of video games players. They argue that “the meaningful experiences of players are shaped by an interplay of game narrative, mechanics, and context dimensions” (Elson et al., 2014a), but without further investigating what this meaningfulness is based on, namely how “game narratives” and “mechanics” are expressed. With the view to achieving a more all-encompassing understanding of the sense-making⁴ of IDNs, I suggest to adopt the approach proposed by Grishakova and Poulaki, (2019) for narrative comprehension and adapt it to the digital interactive environment for general sense-making.

3 Layers of Information in IDNs

In the introduction to their edited volume on Narrative Complexity, Grishakova and Poulaki maintain that “narrative comprehension involves integration of different layers of information–rich percepts, sensorimotor experiences, attentional structuring, retrieval of memory images, and complex meaningful contexts stitched into pattern” (Grishakova and Poulaki, 2019). I believe that, in a similar fashion, the overall meaning of an IDN results from the synthesis of cues coming from a number of layers of information, which are mostly shared by both narrative comprehension and ludonarrative understanding, with minor adjustments and specification.

For example, the “rich percepts” in the scholarly discourse about IDNs and video games could be thought of as multimodality (a), i.e., the compresence of many semiotic modes and media, typical of the digital environment, that generates a richer set of perceptually-available “signs”—intended in a semiotic sense. In addition, other sources for the overall comprehension of video games and IDNs could be, among others (following Grishakova and Poulaki): (b) scaffolding of attentional focus: e.g., by teaching interactors which elements are important and which are not; (c) epistemic contextualization through mnemonic recollection: due to the dissemination of bits of information, each piece of knowledge is to be contextualized by inserting it in a specific “place” of the storyworld. Information may come also from inter- and transmediality, and from genre-dictated customs; (d) patternization for meaning-making and meaningful interaction: e.g., by associating the specific artifact or portion of artifact to a genre, or by creating interaction-reaction patterns that could be used to predict the outcomes of an action. Last but not least, IDNs and digital games afford interactivity, which explains why they could be conceived not only as sensorimotor in nature (e), as Knoller maintains (Knoller, 2019), but with a very high engagement of the sensorimotor system of its interactors, as they require actual physical action.

One could debate that other identifiable layers are missing from the list mentioned above–emotional engagement, to cite one (cf. Herman, 2007; and the further extension made in; Arnavas, 2021). This is because the lists of layers presented in this article are not meant to be exclusive, nor prescriptive, nor at all exhaustive. The intention of this work is neither to identify and indicate all possible layers of information, for it would be a titanic effort, nor to present a definite model of product design, but rather to discuss a complex-systemic perspective on the cognitive processes involved in understanding a certain kind of cultural (digital) objects. This understanding could inform future authors, but showing how and to what extent this could happen is far beyond the scope of the current work, which is intended to lay the conceptual ground and which is, therefore, primarily of theoretical nature.

The very idea of “layers” is aimed at emphasizing the stratification of the different information sources, and the possibility of designing them partly independently. However, this stratification does not imply a hierarchical organization, but rather a “contiguity in a fleshly sense [with] information transactions occurring across membranes, involuted and convoluted surfaces, and multiple volumetric entities interacting with many conspecifics simultaneously” (cf. the concept of ‘cognitive assemblage’ in Hayles, 2016). To be noted is also that the meanings these layers convey are not to be seen as separated from an abstract overall sense-making of interactors: all layers are information sources that in the mind of the interactor are tied together in an integrated understanding of the IDN and of its story.

Indeed, these layers of information are not only and not simply co-present in IDNs: they are interconnected, interoperating and interdependent, and they give rise to a whole of a higher order by mutually informing each other. In brief, they present feedback loops: the context of the IDN, cognitively constructed on the basis of the multimodal system, informs the structuring of attention, it guides the sensorimotor experiences on the basis of memories, and it affects the storyworld and gameworld, which in turn form a meaningful context, and so on, in a cycle.

Thus, using video games as a simpler example, when players understand the context of a video game, and comprehend, for instance, that they are playing a horror game with a chasing entity—which is made perceivable through a set of semiotic resources (audio, graphics, etc.)—they will probably look for what their past experiences catalogue as hiding places. These places will therefore become attention attractors. Players will move in the fictional world according to their understanding of the narrative and to the arrangement of these attractors, shifting between new places and already-known locations: this will cause the game- and storyworld, and their presentation, to adapt accordingly, by changing at each moment or even by progressing in the story as a consequence to the current state of the engine governing the game. This adaptability further builds the context of future developments and helps foster its meaningfulness, it triggers further movements that could be informed by different memories, and so on, in a circular and continuously looping cognitive mechanism whose hermeneutic modes have been discussed by many scholars in recent times (Karhulahti, 2012; Koenitz et al., 2015; Roth et al., 2018; Knoller, 2019).

Perhaps unsurprisingly, this dynamic is often described through the shape of a circle (e.g. in Karhulahti, 2012) or a spiral (as proposed in Knoller, 2019), de facto acknowledging a similar understanding of the matter as a continuous loop. All these layers of information, presenting feedback loops and multiple mutual interactions, form a complex system that is presented to the player as a single and unified whole, which we call Interactive Digital Narrative (see Koenitz and Eladhari, 2021 for similar considerations).

Developers and designers model these layers to obtain specific expressive outcomes from the unified whole. Exactly for this reason, I believe we can–and should, as I will argue towards the end of the article–conceive IDNs as complex expressive means (cf. for a similar position also D. Lee, 2000; as also reported in Traninger, 2012). It should be noted that with expressive means I intend any means used to make an idea explicit (on this use, cf. Adami, 2017; Bosco et al., 2013; and Leach et al., 2000, among others). I will also align with Adami in using the term “meaning” in the rather general sense of “idea” or “concept,” without referring to–and entering in dialogue with–semiotic discourses on the meaning of “meaning”.

The complex-systemic understanding I offer here is not built on pure speculations, nor on entirely new ideas. Rather, all the aforementioned layers of information which I believe compose the system, as well as the interactions they present, have been thoroughly discussed in literature, and my understanding of them is grounded in a number of widely accepted theories, coming not only from game studies, but also from media studies, computer sciences, cognitive humanities, biology, psycholinguistics, literary theory and neurosciences (cf. the concept of “theoretical triangulation” as explained in Denzin, 2017). Even the systemic view of digital artifacts is not new: Lindley proposed the idea of a “game-play gestalt” already in 2002. Lindley intended this systemic understanding as “a particular way of thinking about the game state from the perspective of a player, together with a pattern of repetitive perceptual, cognitive, and motor operations” (Lindley, 2002). What I propose here is not a completely different position on these shared conceptions, but rather a slight readjustment of perspective on how these elements are tied together in a systemic, and in particular in a complex, whole. In supporting this claim, my understanding of complex system is grounded in the definition provided by Susan Stepney: “a complex system exhibits strong interactions between components, feedback between levels, emergence, self-organization, openness, adaptation, growth, and change” (Stepney, 2018). Throughout the article I will show the reasons why this description works also to portray the system of layers of information involved in meaning expression in IDNs.

To keep the discussion not only productive and informative but also manageable, I intend to discuss here only three of the many layers of information, namely multimodality and the multimodal systems–which parallel Grishakova and Poulaki’s “rich percepts”—the sensorimotor experiences, and the mnemonic recollection–or “recollection of memory images” (Grishakova and Poulaki, 2019). I believe discussing these three layers and how they interact with one another should suffice to give a sense of the higher order of complexity of IDNs as expressive means. A similar triad is also the one proposed by Lindley (Lindley, 2002).

Throughout the text, I will expand on why understanding IDNs as complex expressive means can be beneficial not only to our comprehension of IDNs as “simple” expressive means, but also to our understanding of them as representations of complexity. Indeed, as this special issue is showing, IDNs are powerful tools to represent complex topics and to address complexity as a societal challenge. I believe a portion of this capacity of representing complexity comes from the fact that they are complex in their very nature.

Multimodality and Multimodal Systems

With a very limited set of exceptions, IDNs feature the compresence of a number of different semiotic modes and media, among which are text, still and moving images, speech, music, sound effects, and haptic percepts (cf. Danielsson, 2016). Not least for this reason, video games are one of the most all-encompassing multimodal means of communication (as noted also by Toh, 2018; and Zagalo, 2019) to the point of being possibly considered examples of Gesamtkunstwerk, due to the synergy of different arts they often exhibit (cf. Backe, 2020; and in some regards also by; Smith, 2007). However, as reported by Dunne (2014), Zagalo (2019) and, with a different terminology, Backe (2020), studies on the multimodal aspects of video games and on how these aspects impact the expressive functions of digital games are generally scarce and mostly resorting to theories of multimodality borrowed from distantly related disciplines, such as film studies (cf. e.g. Burn, 2016). In this section, I will deal with a systemic understanding of multimodality which, for the reasons that will be shown shortly, is better suited to discuss multimodality in IDNs.

Multimodality offers the possibility to represent a wealth of details, to a degree sometimes impossible for a single mode. This is closely related to the encyclopedic capacity of digital media as defined by Murray (2017). Magliano and colleagues highlight the multiplicity of perceptions involved in the understanding of digital media, which is linked to their encyclopedic capacity and their reliance on multimodal systems: “text-based (and spoken) narratives are entirely consumed through language processing, but visual narratives (comics and film) are typically conveyed in a multimedia format that involves both language and visual content” (Magliano et al., 2019). Additionally, multimodal media are more eager to generate immersion: Steuer, (1992) talks of a “breadth of information,” a stream of sensory percepts simultaneously presented to the audience, which functions as a great facilitator of immersion. Similarly to Steuer, Ryan states that this breath of information is achieved through “the collaboration of multiple media: image, sound, olfactory signals [and haptic devices]” (Ryan, 1999). Furthermore, as also Adami maintains, in a multimodal communication it is practically impossible to identify a “main” semiotic resource of which the other modes are accompaniments or support, but rather “each [mode] concur with a specific functional load to the meaning made by the overall text” (Adami, 2017). She also argues that:

(1) all communication is multimodal; (2) analyses focused solely or primarily on language cannot adequately account for meaning; (3) each mode has specific affordances arising from its materiality and from its social histories, which shape its resources to fulfill given communicative needs; and (4) modes concur together, each with a specialized role, to meaning-making; hence relations among modes are key to understand every instance of communication (Adami, 2017).

Drawing on this definition of multimodality, and on Ryan’s claims regarding the collaboration of modes in digital media, it might be argued that multimodal communication–and therefore all communications, as per Adami, and also Kress, (2010)—employs a system of semiotic modes, in which each mode participates in the creation of the overall meaning⁵. This is particularly evident in IDNs and video games, where the same message is often delivered redundantly through a number of semiotic resources. Indeed, the player is presented with multiple and sometimes repetitive messages coming from, among others, graphics and visual appearance, soundtracks, sound effects, text, spoken dialogues and even haptic feedbacks like vibrations, adaptive triggers (as those in the PlayStation 5 DualSense controllers⁶), and the adaptive sensory feature of the recent PlayStation VR2 headset, defined an “intelligent tactile element”⁷. The same idea is also maintained by Backe, who opens to the view of video games as an interplay of artistic practices “combined in a totalizing whole” (2020). Reporting Marquard’s claim, Backe also states that to create this totalizing effect, there is a “need for a careful systemic integration of elements” (Backe, 2020; cf.; Marquard, 1983) [emphasis added].

This systemic perspective that encourages to see the overall meaning as coming from an integration of parts can explain examples such as that made by Faulstich in the introduction of his Grundkurs Filmanalyse (Faulstich, 2013, where the same videoclip employing two different pieces of background music arguably conveys to the viewer two different meanings, or at least two different moods. In that case, these meanings emerge from the interplay of the audio and visual elements, which only together give rise to a final, unified message of higher order. I believe this is precisely the reason why it can be useful to conceptualize multimodal systems as systems exhibiting a certain degree of complexity: they are systems of bits of information, partially autonomous both at the objective (music and sound effects do not need each other to express their very own meaning) and at the subjective level (audiences do not need music to understand sound effects, and vice versa), oriented towards the transmission of a unified message of a higher order, which emerges from their intercurrent, interdependent and interoperating occurrence. This same “unified message of higher level” that emerges in movies in more-than-combinatorial ways and sometimes outside the direct control of the authors (cf. the uncertainty of system design as discussed in Koenitz and Eladhari, 2021) also justifies my view of IDNs as based on complex multimodal systems: immersive totalities experienced through a synergic compresence of modes, rather than through their simple co-occurrence.

Therefore, if multimodality is the compresence of a number of semiotic modes of resources used for making meaning, multimodality in IDNs is better conceived as a multimodal system in which each semiotic resource is in a complex relationship with the others. This constitutes one of the layers of information of IDNs.

In a similar fashion, Zagalo, (2019) talks of video games as complex audio-visual objects that are variable and infinite, for it is always possible to obtain a slightly different “product” (Koenitz, 2015) by interacting with them. Interactivity is precisely the element on which I shall focus in the next section. Indeed, when compared with contemplative audio-visual media, video games and Interactive Digital Narratives are characterized by an interactive dimension that constitutes an additional element enhancing their complexity. More relevantly to our discussion, interactivity is also what enables them to feature sensorimotor experiences.

Sensorimotor Experiences

In the onset of his Action in Perception, Noë claimed that “the world makes itself available to the perceiver through physical movement and interaction” (Nöe, 2004). Noë is one of the main supporters of the enactivist paradigm in the study of human cognition. Enactivism proposes that cognition does not happen only in our heads but is situated in an environment and co-constitutive with the external world, though action. Polvinen explains the status of the “action” according to enactive theory in a way that aligns particularly well with the point of view of IDN comprehension: “the action from which enactive theory takes its name is thus not just action that has an impact in the world, but instead the action is constitutive: it “brings forth” the world” (Polvinen, 2021).

Indeed, if this is true on a primary real-world level, where we interact with the world in order to access it through our senses, in video games and IDNs it is necessary also on the secondary and fictional-world level. In such artifacts we, as interactors, must inhabit the environment, and it is necessary for us to actually act in order to make the world be–in the very practical sense that acting is required by the computing device to generate a sensorially perceivable representation of the fictional environment. This is why I believe IDNs offer sensorial experiences that are made available through motion. In the real world in which the interactor is situated, this motion ranges from the movement of a finger (using a controller) to the involvement of the entire body (when using a Virtual Reality gear and/or motion capture devices⁸), but it usually translates into a movement with a much wider range and scope (cf. the notion of “amplification of inputs” in Ciccoricco, 2010). Murray defined this characteristic of digital artifacts as their “spatial” property: their capacity to present “space that we can move through” (Murray, 2017).

A similar enactive perspective on computer-mediated artifacts has been discussed already by Laurel in her influential work on the juxtaposition of theatre and computers (Laurel, 2013). However, her view of performativity in digital media as a sort of self-staging, even though grounded in the historical moments it originated (1991, when neither online gaming nor video streaming were available), limits the scope of the notion, which in modern and contemporary digital products is in fact much more substantial. More recently, this perspective has been thoroughly examined by Knoller, (2019) who maintains that interacting with an IDN has a sensorimotor nature. He also argues that when a story accompanies such artifacts, this sensorimotor quality has a sense-making potential. Lahti argued for an even stronger bodily involvement, supporting the idea of corporealized pleasures elicited by digital artifacts (Lahti, 2004).

In IDNs, sensorimotion and enaction are crucial parts of the generation of the artifact itself, and not only of its understanding, whereas this is not true for on-paper literature: the narrative of a book exists in abstracto identical to every reader, while in an IDN it is instantiated in a specific way (among the often countless possible ones) only through interactions with the game- (and therefore story-) world. An IDN changes and evolves internally in response to intrasystemic dynamics, whereas in books these are extrasystemic, and do not impact the objective shape of the artifact.

Contrarily to contemplative media, video games and IDNs are not pre-determined, but rather they are generally constituted by a set of “possibility spaces” (Bogost, 2007) contained in “protostories” (Roth et al., 2018). Authors and designers do not create an artifact that exists per se in a unique, immutable form: on the contrary, IDNs are based on an interactive engine, which requires exchanges between the player(s) and the computing unit, where the computer actively responds to the physical inputs coming from the player and the player actively responds to the sensory inputs coming from the computer. This sort of sensorimotor interaction is necessary to extract a unilinear experience from the artifact, and it is therefore a defining feature of the objects of this analysis, and a further layer of information for IDN comprehension.

Along similar lines, Goggin observes that in video games “the subject does not only observe what s/he lives out, the subject lives out what s/he observes” (Goggin, 2006; cf. on this; Lehto, 2009; and the double hermeneutics defined in; Karhulahti, 2012; and expanded by; Roth et al., 2018). Similarly, in a more recent reflection, Milesi claimed that accessing video games narratives is in a way “paralleling quantum physics, where the fact of observing changes the state of what is being observed” (Milesi, 2019): the very fact of observing the video game–which can be done only through interaction, as we have seen–changes the game system itself. Even more importantly, given that players are urged to act and feel through a mediated bodily presence, the kinesthetic interaction becomes part of a unified aesthetic experience (cf. on this Perron, 2009).

The enactive perspective highlights an additional point of discussion: as Caracciolo and Kukkonen argue, “cognition is an organism’s activity of working out the relevance–or significance–of external features of the environment” (Caracciolo and Kukkonen, 2021). As also Maitlis and Christianson maintain in the abovementioned article (Maitlis and Christianson, 2014), the world one inhabits is a source of information and, therefore, in case of a digital world, the multimodal system through which the world itself is built is the primary source of information, and is made meaningful by piecing together cues that can be understood only through action. However, enactivism also supports the idea that the subject and the world are in a dynamic relationship of mutual shaping: “the world is inseparable from the subject, but from a subject who is nothing but a project of the world; and the subject is inseparable from the world, but from a world that it itself projects” (Merleau-Ponty and Landes, 2012; reported also in; Polvinen, 2021; cf. also; Di Paolo, 2018). This claim shows once more that the shape of the world inhabited by a subject is likely to impact on the subjects’ cognition, and therefore on their understanding of the world itself. Being multimodality the perceptually available trace of the fictional world of an IDN, its design is thus likely to impact on its comprehension. Laurel, too, argued that multimodality is the basis on which these sensorimotor experiences are offered by computer-mediated media (Laurel, 2013). Given that “to perceive you must be in possession of sensorimotor bodily skill” (Nöe, 2004), the complex multimodal system of IDNs appears to be strictly linked to the sensorimotor experience provided by such artifacts, or at least to its comprehension. Knoller adds that multimodality and kinaesthesis are in a position of mutual reinforcement, and he states that embodied responses to external stimuli “can serve as a locus of additional feedback loops that are part of the narrative experience” (Knoller, 2019). This opens to a view of the sensorimotor experiences afforded and required by IDNs as being an integrated aspect of narrative understanding.

In a similar fashion, also Shibolet maintains that kinaesthesis has a role in narrative understanding of IDNs due to its informative power on the sequentialization of events (Shibolet, 2018); this means that a mutual reinforcement between narrative understanding and sensorimotor experience is in place when it comes to understanding IDNs. But feedback loops can be identified between sensorimotor experiences and all kinds of sense-making. Cheng, (2007) provides an example of this, found in the game King Kong (Ubisoft Montpellier, 2005): in a scene in which the player character is tied at a stake, the interaction afforded to the player is limited to a single button instead of the usual ten-twelve. In this situation, Chang maintains, one would expect the player to feel constrained by the game mechanics, but the exact opposite occurs instead. The scholar justifies it saying that “this occurs because the limitations imposed upon the player are not due to any arbitrary reason of game mechanics, but arises [sic] from a situation that makes logical sense in the game world” (Cheng, 2007). This means that the situation is justified by the narrative and, more importantly, by the sensorimotor experience elicited by the narrative, in a mechanic made narratively relevant through kinaesthesis: being the player character tied to a stake, players experience the bodily restraints through interactivity limitations. Additional feedback loops, aptly made explicit in the reported example, are therefore realized in this interplay of multimodal system, narrative, game mechanics, and sensorimotor experiences.

To summarize, Interactive Digital Narratives offer sensorial experiences that are made available and understandable through motion, which therefore constitutes an aspect strongly integrated in the narrative understanding of such artifacts. There is a double sensorimotor involvement in IDNs: the practical hand or bodily movement of the interactor and the amplified translation of it in the fictional world. Both are required to prompt the dynamic generation of the fictional worlds featured in these artifacts. However, several studies agree that bodily skills are also necessary to understand the world in which we are inserted. In IDNs, where the world is formed through a system of semiotic modes, sensorimotor experiences and multimodal systems are in a relation of mutual reinforcement.

However, there is another layer of information that I ought to consider looking at interactors’ comprehension of IDNs, resulting from the fact that our understanding of the world is always grounded in our background knowledge. Indeed, enactivism offers a further suggestion that could be insightful in the current analysis: the assumed structural resemblance between real, perceptual experiences and imaginative, fictional ones (cf. Caracciolo, 2014). Mnemonic recollection, or the retrieval of memory images, is the cognitive mechanism thanks to which we make sense of the current situation through background knowledge (cf. Herman, 2002).

Mnemonic Recollection

Caracciolo offered a major contribution in forwarding the idea of the experiential nature of narrative sense-making in his The Experientiality of Narrative. In this book, he states that “readers respond to narrative on the basis of their experiential background” (Caracciolo, 2014). A discussion of the experiential grounding of narratives lies beyond the scope of the current analysis, but the points raised by Caracciolo suggest the existence of a further layer of information in IDNs’ understanding, or at least part of it.

In Caracciolo’s theory, the retrieval of memory images is one of the leading psychological mechanisms in the process of readers’ response to narrative. This mechanisms is carried through “experiential traces” (Zwaan, 2008), i.e., through memories that guide one’s evocation of past experiences. Experiential traces appear to be a source of information involved in all kinds of sense-making, including but not limited to narrative comprehension: indeed, I believe this view can equally apply to non-narrative understanding. For instance, players need no explanation about gravity: they expect gravity to be present, i.e., they expect Super Mario (Nintendo, 1985) to fall back on the ground after a jump, without any need for further justification. This automatic assumption is grounded in our experiential background as living beings on Earth, and informs our sensorimotor interaction with an artifact: as Herman argue, memory is a “organization of prior experience into patterns of expectations for current experiences” (Herman, 2002). Players’ motion in Super Mario Bros. is dictated not only by the multimodal system through which the gameworld is made perceptually available, but also in response to players’ experiential knowledge of gravity: jumping is perceived through colors and moving images, but it is automatically understood as a movement through the recollection of the real-world experience of pushing oneself off a surface and into the air by using the muscles in one’s legs and feet. Therefore, only through mutual reinforcement between the multimodal system and the experiential traces the sensorimotor interaction is made meaningful and, one might argue, even understandable.

However, mnemonic recollection does not only refer to experiential traces, but also to memory images intended in the more general sense: as Beltrami highlights, background knowledge includes “all kind of information not explicitly provided by the text, derived both by other texts (intertextual knowledge) or by general everyday (extra-textual) experience” (Beltrami, 2021). Specifically referring to narrative understanding, Walsh maintains that it “can mediate between explicit propositional knowledge, knowing that, and experiential knowledge by acquaintance, knowing of”: this is “the core of narrative logic, grounded as it must be in embodiment, in experience” (Walsh, 2018).

Similarly to experiential traces, I believe also this kind of mnemonic recollection works in analogous ways for narrative and non-narrative comprehension. This can be seen by looking at how easily one understands different instances of the same “operational logic” (Wardrip-Fruin, 2009). Wardrip-Fruin speaks of operational logics as of patterns in the interplay between data, process, user experience, interaction, author and audience. For instance, video games players understand the functioning of an operational logic like collision detection without major difficulties across different games, both two- and three-dimensional: this understanding is based on previous real-world sensorimotor experiences, but, as Walsh (2018) suggests, it also requires recollection and retrieval of memories of other instances of collision detection in other games, to foresee what will happen after a collision. Even less grounded in experience, but not less rooted in mnemonic recollection, are the logics of quests or that of assertive artificial intelligences (which are other operational logics identified in Wardrip-Fruin, 2009).

In concert with the serialization of events in the virtual world through sensorimotor experiences mentioned above (Shibolet, 2018), mnemonic recollection, too, participates in the contextualization of games and of all sorts of events happening therein. Magliano and colleagues supports the idea that comprehension works thanks to a relational network between expressed information and background knowledge, and is therefore based on mnemonic recollection of the previously-built understanding of the world: “the network is constructed through inferences that establish relationships between propositions (i.e., bridging inferences) and knowledge-based inferences that elaborate on the propositions” (Magliano et al., 2019). Together with Grishakova, I believe that these inferences are often times of the abductive kind (on the difference between inductive and abductive reasoning, cf. Grishakova, 2022).

Inferences are in turn further contextualized by relying on a series of scripts and schemata dictated by our social, cultural and personal background (cf. among others Schank and Abelson, 1977; Rumelhart, 1980; Stockwell, 2002). In their influential work on cognitive psychology, Schank and Abelson define script as a “standardized generalized episode” (Schank and Abelson, 1977), or, in other words, behavioral pattern shared between a socio-cultural groups and evoked through more-or-less conscious mnemonic recollection. On the other hand, Rumelhart (Rumelhart, 1980) defines schemata as data structures representing our knowledge about the world that enable us not only to understand, but to perceive, and, eventually, even to act (cf. on this Douglas and Hargadon, 2000). Being low-level structures, schemata are equally somehow shared between large groups of people. For this reason, one can expect interactors of an IDN to mostly comply with shared scripts and schemata. As such, the recollection of this patterns can affect the interactors’ understanding of the multimodal system and, as mentioned, can elicit actions and therefore influence the sensorimotor experiences. Specifically talking about video games, Ciccoricco similarly suggests that scripts and schemata evoke contexts that elicit different experiences, but also that different kinesthetic experiences may prompt the recollection of–and the contextualization through–different scripts and schemata (Ciccoricco, 2010). At the same time, however, interactions with IDNs can give rise to new scripts and schemata, both personal and shared: the community of assiduous players of the Grand Theft Auto series (Rockstar North, Digital Eclipse, Rockstar Leeds, and Rockstar Canada, 2021) share a behavioral script about what to do when chased by the police; similarly, the community of Pokémon (Game Freak and ILCA, 2021) players share a schema linking colors to elemental affinity: blue creatures are water types, red monsters are fire types, etc. These examples show, once more, that the three layers of information here discussed are in a position of interoperation, interdependence and interoccurrence, where each one informs and is informed by the others, in a loop.

The relationship between memory images (and experiential traces) and the comprehension of an IDN is two-ways also regarding interactors’ background knowledge. Indeed, Caracciolo supports the existence of feedback loops between our experiential background and a specific artifact:

“like experiential machines, stories need experiential input, but also produce some output, since they can bring about a restructuring of each reader’s experiential background by generating new “story-driven” experiences. […] Engaging with a narrative can leave a mark on readers at the level of their more self-conscious— and culturally mediated—judgments about the world” (Caracciolo, 2014)

In these claims a sort of two-way movement can be identified between the interactors’ background and narrative; even though Caracciolo talks about stories here, we have seen that this understanding can be expanded also to other aspects. Indeed, as it has been empirically proved to happen for pure imagination which is itself linked to memory (Pearson, 2019), sensory inputs are the basis on which we build our memory, and memory is the basis on which we imagine and therefore act. Research in cognitive neuroscience showed evidence supporting the correlation of memory and sensory imagery (Schacter et al., 2012; Pearson, 2019), thus reasserting the idea that not only our memory is formed on the basis of sensory inputs, but also that we usually perform mnemonic tasks by relying partly on cues from the external world. For instance, when walking in an unknown city one could tend to remember the position of one’s hotel in relation with a particularly colorful building, rather than memorizing the names of the surrounding roads. In video games and IDNs, the external world and therefore the sensory inputs come from the multimodal system with which they have been built. Comprehension of the multimodal system and of the sensorimotor experiences are therefore partly based on these mnemonic recollections, but at each moment these two other layers of information provide elements that feed the interactors’ memory, which will be used for further understanding, and so on, cyclically.

In brief: to situate ourselves in the world, and therefore to understand it, we contextualize the situation in which we are presently involved through a comparison with previous experiences and with a background knowledge rooted in memory. But at the same time we continuously build additional memories from the events that we live, from the multimodal system we perceive while accessing an IDN and from the interactions we experience. These memories are then used to process the world–which for IDNs is multimodal and interactively constructed–in a looping circle. In the light of what expounded so far, it is therefore possible to consider the mnemonic recollection of memory images (of experiences, previous knowledge, scripts, and schemata) as a third layer of information participating in the overall IDN comprehension.

What has been conducted until this point is a discussion of three layers of information that have a role in the comprehension of Interactive Digital Narratives. We have seen that IDNs are made perceivable through a (1) multimodal system in which each semiotic resource is in a complex relationship with the others. They offer (2) sensorial experiences that are made available through motion, and they rely on (3) mnemonic recollection of memory images to be understandable. As they mutually inform each other, these three layers present relationships of the complex kind, such that each one is interdependent from, and intercurrent and interoperating with the others, to give rise to a whole of a higher order. This explains why human comprehension of IDNs could be considered complex, but in order to support the idea of IDNs as complex expressive means, I need to show how this complex understanding is exploited by game design practitioners to convey, through complex expressions, an emergent meaning. In the next section, I will do so by looking at a very short scene of the video game Detroit: Become Human (Quantic Dream, 2018) and by analyzing how the scene works, particularly in relation to the design and interactions of the three layers of information described above.

4 Case Study: The Fish Scene in Detroit: Become Human

Detroit: Become Human is a narrative-driven adventure game particularly appreciated by players and critics for its narrative design and for the IDN it embeds. It is strongly focused on ethics of technology and philosophy of artificial intelligence, and it has been largely discussed in relation to moral values and the ethical dimension (cf. Holl, 2019; Craig et al., 2020; Pallavicini et al., 2020; Meier and Bellini, 2021). The game is set in a futuristic Detroit in which extremely smart anthropomorphic androids have been developed and are employed to carry out all sorts of jobs, in a quasi-slavery condition. The players switch control between three androids that developed self-awareness and personal conscience, discovering and directing their journeys. The first of these characters is Connor⁹, a detective android employed to hunt “deviants,” i.e., self-aware androids.

In the very first level of the game, players control Connor on a crime scene: after a short cutscene showing Connor on an elevator, the doors open, and interactors are free to walk around the entrance of a messy apartment. By getting closer to the end of the corridor, another cutscene is triggered. Before (or after) the cutscene, players can notice in the corridor a broken aquarium embedded in the wall on the left, and a fish floundering on the floor. The fish is signaled by an icon for interaction above it. This is what I call here “the fish scene”. It is a very short scene, which lasts just a few moments, and it has a relatively simple framing. However, this scene has been artfully constructed by using the three layers of information presented here, and modelling their interactions. Indeed, even in these few moments and despite its simple framing, I believe it is possible to identify here why IDNs can be conceived as complex expressive means.

Just as most video games, Detroit: Become Human is based on a multimodal system: the sensorially-perceivable part of the game is constructed through the use of a number of concurrent media, like images, text and audio. In particular, we find in the fish scene¹⁰:

• Moving images (e.g., the fish floundering): also used to capture the players attention when Connor is in the corridor depicted in Figure 1)

• Still images (the exemplary image of the fish brought up by Connor’s computational unit–Figure 2);

• Music (in the background);

• Sound effects, with both mimetic and symbolic intents (the splashing of water provoked by the fish and the pulsation when Connor’s computing unit is turned on–which also signals when players have a choice to make–respectively);

• Text, in an homodiegetic (the “ANDROID” writing on the back of Connor’s jacket–Figure 1), heterodiegetic (the indications to players on how to interact, and the output of each interaction–Figure 1, Figure 3), and blurred position (the text with information about the fish, which is unclear whether is seen also by Connor or not–Figure 2) (cf. about UI position Bellini, 2018);

• Haptic feedbacks, with a mimetic intent (if players decide to save the fish, while putting it back in the tank they will feel three mild vibrations paralleled by three movements of the fish in Connor’s hands. Contrarily, no vibrations will be triggered if the fish is left on the floor).

FIGURE 1

FIGURE 1. Connor stands in the corridor where the fish scene takes place. On its back it is possible to read “ANDROID”. The broken fish tank is visible on the wall on the left. On the floor lies the fish. An overlayed interface signals the possibility to kneel down near the fish by moving the right analogue stick towards the player’s body (down). Screenshot from Detroit: Become Human (Quantic Dream 2018) for PlayStation 4.

FIGURE 2

FIGURE 2. Connor’s computational unit shows information about the fish in an overlayed interface. Both text and still images are visible. Screenshot from Detroit: Become Human (Quantic Dream 2018) for PlayStation 4.

FIGURE 3

FIGURE 3. An overlayed interface shows players how to perform one of the two available interactions. By moving the right analogue stick away from their body (up), players can decide to leave the scene immediately without saving the fish. By moving the stick to the right and then perform quarter a clockwise circle, players can save the fish by putting it pack in the tank. Screenshot from Detroit: Become Human (Quantic Dream 2018) for PlayStation 4.

The systemic understanding of the scene emerges from the interplay of these elements, producing a carefully crafted whole. But in order to understand this multimodal system, we as players of the game and interactors of the IDN need to perform, we need to trigger some sort of sensorimotor experience, otherwise the game will remain stuck forever right at the beginning of the level. By moving our hands, we explore the fictional environment and by moving around it we get to the fish scene. Once there, we are required to act two further times:

1) in order to initiate the fish scene, we need to crouch and observe the fish closely. This is suggested by the interaction icon appearing above the fish itself, which reads “KNEEL DOWN,” with an indication to move the right analogue stick (otherwise used to orient the camera) towards the player’s body (down) (Figure 1);

2) to interact with the fish scene, players are given two options (Figure 3):

a) to leave and let the fish die, by moving the stick away from the player’s body (up), or

b) to save the fish, by moving the stick on the right, and then perform quarter a clockwise circle, moving therefore from 3 to 6 h.

Depending on the performance of either action, the game responds differently and therefore adapts the multimodal system accordingly: if the fish is left to die, Connor simply stands up, the fish remains on the floor and the player can proceed moving around immediately; if the fish is saved, Connor picks up the fish and after looking at it, he puts it back in the aquarium, seeing it swimming away. This latter situation also triggers an event in the user interface, showing a text that reads “software instability,” and an upward arrow symbolizing an increase.

In this simple example, we can already see the looping between the sensorimotor experiences made relevant and required by the multimodal system, and the responses required to the multimodal system by the inputs of players-through their movements in the real world, which triggers an amplified movement in the fictional world, and which is taken as inputs by the engine running the game.

The reader might have already detected also the informativeness of the mnemonic recollection: interactors comprehend the fish scene, created through the multimodal system, due to previously-built knowledge and abductive inferential thinking. One needs experience of fishes, of fish tanks, and of fish tanks made of glass, to understand that the fish fell off the aquarium which was somehow broken and is dying on the floor due to absence of water to breathe through. One can also understand only thanks to previous experience and/or background knowledge that the fish is going to die if left on the floor, or it is saved by putting it back in its tank, even if half empty. The very existence of the in-wall fish tank contextualizes the whole scene in an expensive apartment, context that is further enhanced by the exotic fish species¹¹.

All of these contextualizations are grounded in interactors’ background knowledge. But even the exemplary image and the text description of the fish brought up by Connor’s computational unit (Figure 2) needs grounding in experience of encyclopedias and of multi-layered image composition. Similarly, the mimetic sounds are mimetic as soon as they resemble the real world–or at least the interactors’ expectation of the real world –, by definition. The symbolic use of graphics and sounds, employed to signal the availability of an interaction, are taught to players and equally retrieved whenever necessary. Lastly, though perhaps a bit redundantly, also reading a text necessarily require background knowledge.

In addition, the sensorimotor experiences are based on mnemonic recollection. In this scene, this is particularly visible, as well as it is particularly foregrounded how they are experientially grounded. Indeed, thanks to a third-person camera free to revolve around the characters’ head, interactors have the feeling of manipulating with the right analogue stick of their controller the head and eyes of the character, and with the left analogue stick its body, with a forward direction always oriented away from the interactor’s body (up). For as complicated as it might sound in writing, this movement is very familiar and intuitive even for interactors with minimal experience. In addition, the engine requires a specific movement to instantiate the fish scene, namely moving the right analogue stick towards the interactor’s body (Figure 1): this movement, which normally would cause the head of the character to tilt towards the floor, is here metaphorically associated with bringing into focus something that is laying on the floor. Indeed, Connor crouches and the multimodal system shows the fish scene. Similarly, in order to decide whether to save or not the fish, interactors can 1) bring forward Connor’s hand and pick up the fish, by moving the right analogue stick to the right and rotate it, with a movement similar to what is going to be Connor’s hand gesture to pick up the fish; or 2) stand back up, by directing Connor’s attention upwards, moving the right analogue stick away from their body (up) (Figure 3). In these cases, too, the interaction modes are fluid and require very little explanation because they are grounded in human experiential knowledge. Designers and developers made use of this shared experiential knowledge to improve easiness of interaction and of comprehension of the scene. While it might be argued that these interactions highlight also the exploitation of conceptual metaphors (Lakoff and Johnson, 2008), this discussion will be left for future elaboration.

5 Layers of Information and Complex Systems Theory

The brief analysis here proposed shows how the theoretical reflections advanced in the first half of the article have very practical confirmations in real video games and IDNs. As we have seen, IDN comprehension comes from the synthesis of the three layers of information here discussed, at the very least.

IDNs authors rely on these layers to express a meaning, to convey a certain idea through an artifact, by designing a desirable possibility space for interpretation (Bogost, 2007) through modelling these layers. Designers embed them more or less knowingly, integrating this complex system of layers of information into a single artifact, which, resulting in a unified product, is therefore of the complex kind. Taking back Stepney’s definition of complex systems (Stepney, 2018), by relying on a number of layers of information, IDNs are complex because they exhibit:

• Strong interaction between components: an IDN multimodal system, the sensorimotor experiences it requires and affords, and the mnemonic recollection it demands are inseparable from each other as they are mutually informing. Without one of the three, the artifact as a whole would simply not work: it would either not be perceivable, not be instantiable, or not be understandable. Even inside the multimodal system the elements often work in concert, with parallels between audio, video, text and haptic sensations;

• Feedback between levels, and growth: as I have shown, the three layers of information trigger many feedback loops thanks to which each of the three informs and at the same time is informed by the others. In this cycle, the three layers reinforce each other and provide further information. On the basis of these additional information, each layer works: interactors act on the basis of the information coming from the multimodal system, which adapt to the action, and so on. This causes the system itself to grow spontaneously due to internal dynamics;

• Emergence: the compresence of different layers of information, together with feedback loops, produces the IDN as a unified artifact which is more than the sum of its single layers taken separately. Pressing buttons on a remote controller while looking at a Leonardo painting cannot be deemed an IDN, because there is no relationship between the perceivable elements and the interaction modes;

• Self-organization: a number of works in literature tend to regard the multiplicity of professional figures often participating in the creation of IDNs as sufficient to consider the resulting artifacts as being self-organized (e.g. Knudsen and Olesen, 2018; and, in some regards, Abbott, 2008). An additional point to consider is that highlighted by Koenitz and Eladhari, according to whom there is a certain degree of lack of control in all video games featuring large player populations, user-generated contents, procedural generation or even complex combinatorics (Koenitz and Eladhari, 2021). However, given an undeniable coordination underpinning IDN design, I acknowledge that the status of self-organization is the most problematic in this theorization. Improved self-organization could germinate from procedurally generated games like Minecraft (Mojang, 2011) or No Man’s Sky (Hello Games, 2016), but true self-organization could be achieved probably only once projects like Realspawn¹² will be fully integrated in widely available games;

• Openness,adaptation, and change: the list of layers of information discussed here is not exclusive, nor prescriptive. As I mentioned, the ones discussed here are only some of the layers that constitute the complex expressive means we call Interactive Digital Narrative. Other layers, or other elements of the layers, could be present in specific artifacts without preventing the working of the IDN itself. This is because the initial and basic system is open to external influences and can readjust itself to include further sources, changing internally without being disrupted. Additionally, Interactive Digital Narratives as coherent wholes can be adapted for employment in a wide variety of fields and with many functions, as showed by the multifarious perspectives presented by this special issue.

6 Conclusion

In this article I presented theoretical evidence supporting the idea of understanding IDNs as complex expressive means.

Building on Grishakova and Poulaki’s view of narrative comprehension as coming from the synthesis of different layers of information, I showed how we can see three of these layers as present also in IDNs, how they interact with each other and how they can therefore be conceived as forming a complex system. The layers treated in this article are: the multimodal system, which comprise all perceptually-available elements forming the IDN; the sensorimotor experiences, afforded by the artifact through its interactivity and required for the narrative to develop; and the mnemonic recollection, that is, the retrieval of past experiences and background knowledge of the interactors, also in terms of personal and shared scripts and schemata.

I then showed, through a practical example, how developers and designers exploit these layers by modelling them in the creation of the unified whole we call Interactive Digital Narrative. Relying on Stepney’s definition of complex systems, I explained why IDNs can therefore be regarded as complex expressive means. Following up on this article, a test to empirically validate these claims is in the early-stage of development, together with a discussion of other identifiable layers of information.

Understanding IDNs as complex expressive means allows to explain their capacity to represent complex topics and to address complexity as a societal challenge, by making it more accessible to the wider audience (cf. on this Koenitz et al., 2021). Due to their reliance on different layers of information, authors can lower the required cognitive load of interactors and therefore make more easily understandable even complex topics that constitute contemporary social challenges, like climate change and the migration crises (as somehow suggested also by Morton, 2013). Thanks to the multimodal systems on which they are built, IDNs can represent a wealth of details and can portray large pieces of representation, due to the multiplicity of perceptions they involve synchronically. Because they rely on sensorimotor experiences, they can adapt and respond in a coherent manner to interactors’ inputs, which enables a deeper and more all-encompassing understanding of the subject matter. This also permits IDNs to afford replayability, and to present different outcomes from multiple interactions with the same artifact, which further enhance comprehension—as discussed by Knoller, (2019). Furthermore, thanks to mnemonic recollection, they can rely on previous knowledge of complex topics to aid their comprehension inside the artifacts themselves—even though this is not exclusive of IDNs. All these layers of information participate in an overall integrated understanding of the artifact and of its narrative and are not to be seen as separated from each other.

IDNs can achieve their societal impact not only through direct representations of complex subjects: they can be used to educate audiences in a systemic understanding of complexity in many diverse contexts even beyond the individual represented topic. Interactors of IDNs like Detroit: Become Human could get a grasp of the functioning of oppressive ideologies and regulations well beyond the fictional androids of the game, they can understand complex issues related to technological advances, and even realize the difficulty of adequately representing such complex matters through traditional narratives. In addition, due to the ways sense-making work in IDNs, interactors could be prompted to shift behavioral scripts in regard to complex matters, for example when dealing with oppression or when deciding on their political position. Systemic understanding and shifts to behavioral scripts are two important benefits of the use of Interactive Digital Narratives to address complexity as a societal challenge.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

The author confirms being the sole contributor of this work and has approved it for publication.

Funding

The research for this article was supported by the University of Tartu (Basic Funding PHVKU21921, 2021-2022).

Conflict of Interest

The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

The handling editor declared a shared committee and past collaboration with the author at time of review.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

The author thanks all members and guests of the Narrative, Culture, and Cognition research group for the invaluable suggestions while developing the ideas here presented, and the participants to the Doctoral Consortium of ICIDS 2021 for the insightful discussion. The author also thanks the COST Action 18230—Interactive Narrative Design for Complexity Representation (INDCOR) and its members, for the many thought-provoking debates on the topic of interactive digital narratives, complexity, and representation.

Footnotes

¹In my understanding, video games and IDNs are two distinct categories only partly overlapping: just as IDNs, also video games are interactive and digital, but not all video games are IDNs and not all IDNs are video games. Digital games might be seen as IDNs, but not always: non-narrative games do not present an IDN by definition, even though they might afford re-telling (Eladhari, 2018). However, due to their many shared features and to the still limited literature discussing IDNs specifically, I will sometimes resort on literature in game studies. Examples of IDNs that are not video games are interactive movies like Zena, an interactive VR film (Reyes, 2017) or Black Mirror: Bandersnatch (Slade, 2018).

²For an expanded definition of IDNs, the reader can refer to the introduction to this special issue.

³IMP stands for “Integrated Model for Player Experience” (cf. Elson et al., 2014a).

⁴In the current article sense-making will be adopted to refer to the cognitive mechanism of understanding, decoding a meaning, i.e. the operation of the reader–or interactor, in this case. On the other hand, meaning-making will refer to the encoding of meaning, i.e. the operation of the author.

⁵Grishakova pushes this understanding even further, maintaining that even human cognition is multimodal (Grishakova, 2010).

⁶cf.https://www.youtube.com/watch?v=7gJLVQy-bvM

⁷cf.https://blog.playstation.com/2022/01/04/playstation-vr2-and-playstation-vr2-sense-controller-the-next-generation-of-vr-gaming-on-ps5/.

⁸Some of which can even capture gaze direction, that allow enhanced exploit by designers of sensorimotor responses only partly conscious, like eye motion. An example is HTC VIVE Pro Eye: https://www.vive.com/us/product/vive-pro-eye/overview/.

⁹Connor will be here referred to as a male due to its apparent masculine look, but it is not completely clear in the game whether androids have a defined sex or gender.

¹⁰A video version of the fish scene is available here: https://www.youtube.com/watch?v=qJCt_TNjH24.

¹¹An additional level of mnemonic recollection could also be detected at the end of the scene if the fish is saved, when players with a reasonable knowledge of science fiction tropes would be able to predict possible meanings of the increasing software instability of Connor on the basis of their past experiences with genre-specific customs. However, I will not touch here upon the conception of genres as complex entities (cf. on this Sinding, 2012), nor on prospective and retrospective narrative comprehension (see e.g. Walsh, 2018), as they would drive me too far away from the scope of the current work.

¹²Realspawn project uses generative interaction theory to design persistent player narratives in digital game worlds, basically transforming players’ interactions into integrated game content, cf. http://www.realspawn.com.

References

Abbott, H. P. (2008). Narrative and Emergent Behavior. Poet. Today 29 (2), 227–244. doi:10.1215/03335372-2007-024

CrossRef Full Text | Google Scholar

Adami, E. (2017). “Multimodality,” in The Oxford Handbook of Language and Society. Editors O. García, N. Flores, and M. Spotti (Oxford University Press).