Ágnes Karolina Bakk- Innovation Center of Moholy-Nagy University of Art and Design, Budapest, Hungary
In this paper, I present various strategies adopted by creators of artistic interactive virtual reality (VR) experiences to represent mental health problems and their contexts. The strategies can foster dialogues about these problems, as they present their complexities by embedding them into various narrative and non-narrative elements. In discussing the representational complexity of these works, I emphasize design strategies that tackle stereotypes and their ethical representations and which create a specific type of suspense for the experiencer to bypass the representation of suffering that documentary VR works often use. These productions approach mental health issues as dynamic systems and represent them through linearity or non-linearity (see Stepney 2018). This approach offers limited agency to the interactor, in the sense that the user has a sense of embodiment (Kilteni et al., 2012) and is scripted as an interactor in specific cases (Murray 1997). In this paper, I argue that the VR medium’s characteristic of expressing non-linearity via multimodality (see Elleström 2019) and sense of embodiment makes it suitable for such productions.
1 Introduction
Virtual reality (VR) technology is already widely used in treating psychological and psychiatric illnesses. The technology is considered suitable for those with such problems, thanks to the medium’s characteristic of enabling a high degree of immersion. VR can also provide experiential learning opportunities for those who live with people suffering from mental illnesses and for those, such as doctors, who would like to achieve a deeper level of understanding about specific illnesses. This change is enabled by the fact that mental health is now less of a socio-cultural taboo, and artists have begun to provide a larger role in facilitating this process of understanding.
In this paper, I analyze the representation of mental health in three artistic VR productions. The case studies, Manic VR by Kalina Bertin (2018), Goliath by Anagram (2021), and Welcome to Respite! by Ferryman Collective and CoAct (2021), offer the user an entry point to experience the systems that overlap and interact, resulting in mental health issues. Current mental health research shows that “studying one component is insufficient to understand[ing] the cause of a mental disorder, [and] studying one system alone is also unlikely to be sufficient” (Fried and Robinaugh 2020). I have specifically chosen artistic VR productions as case studies, as they present an easily identifiable visual style and narrative structure that is not based on simulation (unlike other, more science-oriented VR applications). Further, these works pay more attention to creating artistically detailed and unique environments, which help experiencers to gain a better glimpse of the issues in question. In this paper, I use the term “experiencer” instead of “user” to highlight the experiential impact potential of the above-mentioned productions.
When discussing the representational complexity of these works, I emphasize design strategies that tackle stereotypes and the ethical representation of mental health issues and which create a specific type of suspense for the experiencer in order to bypass the kind of representation of suffering that documentary VR works often use. I argue that the VR medium’s characteristic of expressing non-linearity via its multimodality (see Elleström 2019) and sense of embodiment makes it suitable for such productions.
While Manic VR and Goliath are single-user VR experiences, Welcome to Respite! is a multiuser experience in which participants can choose between being an invisible witness and being the main protagonist, who suffers from dissociative identity disorder. The three productions rely on different design strategies to offer a sense of embodiment, but not all of them offer the experiencer the dramaturgical role of the interactor (e.g., Welcome to Respite! uses mechanisms of theatrical events and live-action role playing).
I also discuss how these VR applications address empathy, while reflecting on the ambiguity of this term. Among the above-mentioned VR applications, only Manic VR is tagged as a documentary because its creator, Bertin, used voice recordings of her siblings, who are diagnosed with bipolar disorder; it therefore offers an experiential view of how she interprets their problems (Welcome to Respite! does not claim a documentarian aspect, though the story is based on the experiences of the narrator, Jon, with paranoid schizophrenia, which is also presented in Goliath).
In the closing section, I present Michael Kirby’s taxonomy of acting strategies and develop a new, related taxonomy of various participation possibilities and design strategies for VR. I conclude by expanding on Fisher’s taxonomy of ethics (Fisher, 2017) by arguing that the designers of these applications aim to create empathy toward the unimaginable and in this way provide critical information framed with artistic concepts, which can help bystanders. As Bown and Goltsov (2018) claim, clinicians may thus also understand more fully the complexity of mental health issues.
2 Materials and methods
In this paper, I present various strategies adopted by creators of artistic interactive VR experiences to represent mental health problems and their contexts. Such works can foster dialogues about these problems, as they present their complexity by embedding them into various narrative and non-narrative elements. These productions are situated between science communication and the cultural manifestation of complexity in narrative (as described by Walsh and Stepney 2018), and they produce an illusion of first-hand, experiential, and embodied acquaintance with mental health conditions, which nevertheless helps experiencers to acquire some knowledge about them (Nash, 2021, 106).
By presenting three case studies—Manic VR by Kalina Bertin (2018), Goliath by Anagram (2021), and Welcome to Respite! by Ferryman Collective and CoAct (2021)—I highlight the importance of the designers’ having deep knowledge about the subject matter and its context in offering coherent interactive digital narrative experiences that represent varied mental disorders. Importantly, in many cases, the designers also have personal connections to those who actually suffer from these diseases.
3 Case study
3.1 Goliath (2021)
Goliath was created by the UK-based company Anagram, and it premiered in 2021. The creators frame the nature of the experience they offer as one that “explores the limits of reality in this true story of so-called ‘schizophrenia’ and the power of gaming communities” (see their website: https://weareanagram.co.uk/project/goliath). The 25-min long VR experience is available for free in the Oculus Store. When starting the experience, the experiencer sees a black environment and later an introduction about schizophrenia. The room-scale experience can be watched in standing or seated mode, and in some scenes, the experiencer can only look around or interact or play with the controller. The narrator (Tilda Swinton’s voice) presents an onboarding experience, getting the viewer accustomed to the dark and building up the world by stating that “the mind is a mysterious place” and that our realities come together when our mind “joins the dots between what you feel and see.” The experience starts with a mental exercise about how embodiment actually works in VR (the experiencers have to focus on their hand representation in VR). Experiencers then must play a game in which they bring a ball into the middle of a small panel that can be moved with the controllers, a design strategy which aligns with the narrator’s statement that our actions and reactions make us aware of being really here. While, according to the narrator, some stories are not true, this one is. The protagonist, Goliath, appears after the onboarding. The protagonist narrates his life story: he has had a burdened family background, and in order to escape, he often visited game arcades. When he grew up, he was more interested in techno parties, where he consumed various types of drugs. One time these caused a heavy psychosis, and he was brought to a mental institution in which he had to spend a couple of years after being diagnosed with schizophrenia. According to the narrator, this is a result of the protagonist’s moving between realities. When he was released from the institution, he says, he could not find his place and people always looked strangely at him. But later, his experiences in online video game communities turned out to be fun and very supportive for him. Goliath is living between various layers of reality, and this story itself is a product of his mind. The narrator, meanwhile, claims that “all realities are imagined, but the one we share endures.”
The experience is more like a “collection of scenes” (Vitillio 2021) that is realized with the use of various immersive technologies, and here the connecting element is the main protagonist’s story.
3.2 Case study 2: Manic Virtual Reality (2019)
While the previous experience is not tagged as a documentary production, even though the narrator states that it is, Kalina Bertin’s production Manic VR is based on actual voice recordings of patients diagnosed with bipolar disorder. The 11-min-long experience, screened only at festivals, is based on Bertin’s voicemail recordings in a voice diary kept for three years by her brother Francois and her sister Felicia to capture their state. While listening first to Francois, the experiencer is in a bedroom, where blurry spots appear in her vision and small light dots also start to fly and can be interacted with through sonoric feedback. When Felicia speaks about the process of getting into the worst phases, the ceiling above the experiencer opens, and the point of view is lifted into the starry universe, while the voice discusses how various natural obstacles, such as volcanos, are in her way. When Francois’ monologue is heard again—sharing how he believes in destiny—the hands of the experiencer become visible, and they are connected with lines that look like strings, which can be further stretched by hand movements. Meanwhile Felicia shares Shakespeare’s famous quote that “all the world is a stage.” Then the experiencer arrives at another room, where a white ghost-shaped figure appears, manifesting Francois’ sentences about ghosts. Then the bed evaporates and instead we see tropical plants occupied by buzzing light dots, which form a bathtub. After this, knives appear in the air, depicting the narrator’s sentences. “Half of it is true, half of it is in my mind,” says Francois. The director of the movie is also directly addressed by her sister. An isolation room is also discussed and depicted. Francois reveals his desire for suicide when a teddy bear in the isolation room starts to float, and the image suddenly starts to look as if the experiencer would be underwater, floating together with the teddy. Later, the experiencer is immersed in a black universe and can walk back and forth between windows that offer a glimpse into previous scenes. While in a sunny room, both siblings talk about how they were diagnosed and accepted the diagnosis.
3.3 Case study 3: The severance theory: Welcome to Respite!
The Welcome to Respite! theatrical VR performance premiered in 2021 and was developed by CoAct Productions and Ferryman Collective. This production is also hosted by the VR Chat platform. It is the first part of a four-chapter-long production and represents mental disorders such as dissociative identity disorder. The audience members, when stepping into VR Chat, are greeted in a space filled with childhood memory objects, most of which are oversized compared to the size of the avatar. The actor greeting the audience offers a long onboarding: audience members can interact with the actor and can sing songs together while they receive a technical onboarding. Here the audience members are separated, with one member ultimately becoming the protagonist of the performance and the other participants receiving extra technical guidance. The rest of the participants become invisible, and they can move very fast (creating some nausea for many audience members).
The invisible audience members arrive at the terrace of a house, where they meet Alex, the protagonist (played by the audience member), a young boy waiting for his parents. The mother, and later the father, also arrive home. While there is no narrative arc (the family is cooking dinner, the father and son play in the attic, and then the parents bring their son to sleep), small comments about the father’s drinking habits and the son’s possible mental health issues create suspense. There are two interactive moments when the invisible audience members can interact: in both moments, the interactors must try to save Alex (in a metaphorical way, by pushing the controller’s buttons) from amorphous evil forces. These are represented as artistic colorful forms, in order to not to attach any concrete meaning to these representations. After the performance, the audience members can have an offboarding in another VR Chat room, where there is further information about this mental disorder as well as about the production.
4 Cinema vs. virtual reality in representing mental health issues
4.1 Mental disorder and complex narratives
In the last decade, researchers in film studies have delved deeper into the representation of mental health issues in film. For example, Erin Heath, in her book Mental Disorders in Popular Film: How Hollywood Uses, Shames, and Obscures Mental Diversity states that “mental disorder does not generally appear in ways that demonstrate real people’s lived experiences. Films […] intentionally make psychological breakdowns immediate and narratively convenient to provide audiences with an entertaining emotional struggle.” Heath uses examples from Fight Club and Black Swan. One could argue that these films’ representations of mental disorders do not totally comply with how they are described in The Diagnostic and Statistical Manual of Mental Disorders, leading to very shallow and instrumentalized depictions of mental health disorders. As Kreitler states, many American movies fail to be medically accurate, and they are “deemed to perpetuate mental illness stigma” (Kreitler, n.d.). This perpetuation means that mental disorders are represented in stereotypical ways and are often associated with immoral choices that characters who have such disorders must make. Robb and Stone state that “[t]hree of the most common stereotypes have included beliefs that people with mental illness are dangerous, incompetent, and responsible for the onset and offset of their condition” (Robb and Stone 2016, 4).
These stereotypes result in anxiety, fear, and misunderstandings; therefore, they are of concern for VR experiencers. In cinema studies, many relevant scientific research outputs have already discussed this subject, but the phenomenological effects of cinema on viewers are entirely a matter of speculation. Meanwhile VR can offer—through its special affordances—a stronger sense of presence and embodiment, which influences creators to be more careful when creating these kinds of strong immersive experiences and to study and represent various types of mental health disorders in more multilayered and complex ways.
According to Suedfeld and Pennbaker (1997), narrative complexity mirrors the characteristics of an individual’s cognitive processing during writing. The complexity depends on how many dimensions of a situation can be recognized in a narrative and on how many differentiated perspectives are integrated into the complexity. One important design strategy in presenting conflicting narratives is to include conflicting narratives of memories. According to Brewin (2001), stressful memories are disorganized, incomplete, and incoherent compared to less stressful, autobiographical memories. Stressful or traumatic events are initially disconnected representations of a single event. Foa and Riggs (1994) state that recovering from a stressful event requires organizing and streamlining the traumatic memory. According to Klein (2003), disorganized memories (i.e., intrusive memories) are hyper-accessible; their recollection can be triggered easily by internal or external stimuli. These intrusive memories are present in many psychiatric disorders, and they can be recalled by, for instance, seeing a person who resembles a deceased friend (external) or by thinking of certain activities (internal). Narrative constructions that embed traumatic or stressful events can weaken the “accessibility of these bad experiences and [lessen] the likelihood that internal or external stimuli will activate them” (Klein, 2001, 65). For researchers of autobiographical memory as well as for psychologists, the question of how negative experiences can be transformed into representations that do not derail cognition is significant (Klein, 2003, 65). These memories are not deliberate. Therefore, they can be understood better in a medium that offers the possibility of embodiment rather than as flashbacks in cinematic media.
In my analysis of the three case studies, the possibility of the occurrence of these memories is a key factor in why I consider VR to be a suitable medium to present the complexity of these mental disorders from a subjective perspective. Intrusive memories can signpost a turning point in a story by offering a different trajectory (though the branching narrative form is not possible in these VR projects), and they also create a sense of suspense that urges the recipient to continue exploring the experience.
4.2 Why virtual reality for mental disorder representation?
VR can be a promising medium to represent mental disorders, as it offers a strong sense of presence with the help of an illusion of space and a sense of embodiment that can enable visual, auditory, and somatic inputs to be transposed into a bodily experience. In the following, I briefly describe how the features of VR technology enable an illusion of place, and I present how VR can offer a sense of embodiment in such places. Through this sense of embodiment, the experiencer can have a more direct experience of the narrative, and the medium, through its unique affordances, can extend experiencers’ previous embodied experiences.
The immersive characteristics of VR devices and recent advancements in VR technology have made VR productions even more attractive for scholars. With their promise to fully immerse their consumers, VR productions have been the subject of many speculative theoretical writings, some of which relate them to, for example, André Bazin’s concept of total cinema (Bornkamm et al., 2018). Other studies include more data-oriented approaches that study perceptual and representational aspects of VR. According to Biocca and Delaney, VR can be defined as “the sum of the hardware and software systems that seek to perfect an all-inclusive, sensory illusion of being present in another environment” (Biocca and Delaney, 1995). VR’s core characteristics are immersion, presence, and interactivity (Walsh et al., 2002). They lead back to the grounding features of the technology that are, according to Frederick Brooks, three real features: 1) real-time rendering with viewpoint changes brought about by the user moving her head, 2) real space, namely concrete or abstract 3D virtual environments, and 3) real interaction, that is, direct manipulation of virtual objects (Brooks 1999).
These features can enable the user to experience new illusory encounters such as having a 1) sense-of-place illusion, in which the user experiences the VR space as a real space. In media psychology, “spatial presence” is a key term to express the sense of presence, a sense of being there. This feature has two core aspects: “a simulated spatial environment in which the user feels located” and “the mediated environment offering perceivable options for activity” (Hameed and Perkis, 2018).
The intensity of the place illusion is correlated with the intensity of the 2) plausibility illusion, that is, the illusion that the depicted scenario is actually occurring (Slater 2009). This sense can occur even though the interface is not yet transparent and the experiencer is aware that the given production is a simulation (Steinicke 2016). This simulation situation offers the experiencer a sense of immersion, which manifests also in the production and in the engagement. While the production, technically speaking, satisfies the above-listed categories regarding the experiencer, it is important to mention its embodying effect and how this shapes the relationship with the experiencer.
The combination of these two aspects, the place illusion and the plausibility illusion, can result in a stronger experience of physicality. Having the illusion of being at a virtual spot and of having agency in a virtual event exposes the subject (or the protagonist) to new viewpoints and perspectives. While 360° videos watched on VR headsets cannot enable the experiencer to have a feeling of embodiment in the presented environment, interactive and game-engine-based VR productions can enable such an experience.
“Embodiment” is a term that creates much confusion. Therefore, Kilteni et al. developed a working definition to address the ambiguity (Kilteni et al., 2012). Embodiment has three components: sense of self-location, sense of agency, and sense of body ownership. A VR production that offers this sense of embodiment by being present allows the user to have a deeper understanding of perspectives other than their own (De la Peña 2010). Although VR is sometimes called “the empathy machine,” and many VR creators do aim at increasing empathy, the type and direction of empathy can be questionable in certain cases. However, I claim that VR does have the potential to create stronger experiences of empathy due to its multimodality, as described in more detail in the following sections.
4.3 Nonlinearity as a tool for creating complexity in narratives: A specific approach to interactive virtual reality experiences
Lars Elleström states that those “represented events that are temporally interrelated in a meaningful way” (Elleström 2019, 37) constitute the narrative, and that how these events are scaffolded constitute the story. In his discussion of transmedial narrative, he also emphasizes that “stories are embedded in narratives and they may also, to a certain extent, be realized by dissimilar media” (ibid., 38) and that different media products present narration in various degrees (ibid., 46). In this paper, I do not discuss transmedia storytelling products, but I find his categories helpful in discussing VR, as they foreground the specificities of this media in relation to cinema.
Thus, the fact that media types can narrate but “cannot do it [to] the same degree” (ibid.) is influenced by the general context as well as the cognitive schemata of the experiencer. However, this does not explain “why narration is realized differently in dissimilar media types” (ibid.). Although Elleström does not discuss the medium of VR (especially interactive VR experiences that provide six degrees of freedom), I consider a medium that exploits the highest number of modality capacities (“related basic and universal media features,” ibid. 48). VR is a medium that can offer embodied experiences of various narrations and stories, and the seams between these modalities represent its special type of non-linearity. Three of these modalities are presemiotic: material modality (e.g., solid vs non-solid, organic or inorganic), spatiotemporal modality (temporality, two or three-dimensional spaces), and sensorial modality (visual, auditory, tactile). These features “do not transfer any cognitive import until the perceiver’s mind comprehends them as signs” (ibid., 49). The fourth modality is the semiotic one, which consists of iconicity, indexicality, and symbolicity—usually in a mixed way. Together, these help create cognitive import, even though they also depend heavily on the physical aspect, and therefore always must be combined with the presemiotic aspects. This combination supports meaning-making, and “the combination of modes partly determine what kinds of cognitive import can be transferred from the producer’s mind to the perceiver’s mind” (ibid., 51). Due to these various layers of modalities and the possibilities they offer in imbuing the content with meaning, the experiencer is made to focus on multiple features in a way that derails her from what would otherwise be a straight path, enabling a feeling of non-linearity. The attention of the viewer can be guided by the creators and by traditional dramaturgical elements, such as movement or voice, and by specific user-interface elements such as color. However, as the experiences are usually short, and the experiencer is not accustomed to what can be expected in the production, her senses are more sharpened.
Based on these features, we can define an interactive 6DOF VR production’s modalities that influence the story according to the following: VR’s material modality is solid, it is easy to place on the head, and it offers a glasslike screen when placed. Because the screen is situated close to the eyes, and because of the videogame-like design procedures, VR of this kind offers a feeling of three-dimensional space, while its temporality partially depends on the user (in some cases, the story can only scaffold when the experiencer interacts with the graphical environment). In terms of sensorial modality, VR offers visual, auditory, tactile, and even somatic senses that offer a multilayered experience. The semiotic modality, which relies on all of the presemiotic ones, also depends on the sensorial modality, and the senses evoked in VR productions can actually enrich the narrative. VR also aligns with what Elleström calls “multimodality,” a concept that is more layered than it initially seems: it is possible to differentiate between several types of multimodality, including multimateriality, multisensoriality, and multisemiocity (Elleström 2019, 58). The semiotic modality in these productions enables the creation of signs that refer not only to our everyday perceptions of reality but also to less figurative visuals that can be interpreted in many ways.
All three case studies embed these less-figurative elements (visuals combined with auditory, tactile, and somatic-enabling elements, such as urging the experiencer to interact) into “everyday aesthetic” semiotic elements. These rely on the general context and the experiencer’s cognitive schemata to offer a novel glimpse into the protagonists’ states of mind, especially their memories, and VR represents it in a unique non-linear way.
Walsh and Stepney 2018 describe two types of non-linearity: the narrative and the mathematic. While in the narrative sense, non-linearity represents mainly temporal non-linearity, mathematic non-linearity reveals “how one property of a system (e.g., weight) depends on another property (e.g., height)” (2018, 30). The extension of non-linearity that I propose results from various types of participation strategies that are enabled by the multimodality of the medium. This particular non-linearity makes these productions more viable to represent and allows experiencers to embody the witnessing aspect of these disorders. Bodily movements are necessary to contribute to the unfolding of these states of psychosis or mental disorder. Even though it is often expected that VR can represent non-linearity, in the case of these three productions, linear does not mean that “the events that happen can be fully ordered in a line representing increasing time” (ibid., 31). For example, Welcome to Respite! and Goliath do follow a certain increase in time, even if they are embedded into a narrative framing (onboarding). And while we can observe a temporal non-linearity in Manic VR, its structure is based more on associative constructions. Welcome to Respite! presents a snippet of time that is divided by a different temporality and spatiality. When we encounter the imagination of the main protagonist and the experiencer together with the invisible witnesses, we can experience and simultaneously “fight” the inner monsters induced by his mental disorder. In the case of Goliath, its non-linearity is a characteristic of how the experiencer can unfold the story of Goliath and how the experiencers let themselves play, according to the narrator’s rules and guidance. Based on these productions, I propose to extend the sense of non-linearity to VR productions that rely heavily on their multimodal characters, even if their narratives maintain a linear characteristic. In the case studies, the protagonist’s state of mind is presented by being embedded into the unfolding narrative via various methods, and to unfold these, the active participation of the experiencer is also sometimes needed.
This non-linearity bears the promise of suspense due to its characteristic to manifest seams. According to Walsh (2018, 53), suspense “precedes narrative resolution; it is affectively constituted by an unresolved tension between two perspectives.” It is exactly the audience’s expectation of an ambiguous outcome, nurtured by the entrepreneurial attitude, “which itself assumes a disparity of knowledge between the current uncertain prospect and a future retrospect of what will have happened” (ibid.) This suspense is created by the sharpening of the senses that I have outlined, and all the three case studies use this particular design guideline to create a state of sharpened senses. The experiencers, not knowing the precise rules of the experience or the limitations of the agency, encounter the memories but also the current state of the protagonist, embodying these in a parallel way (Bakk, 2020, 328). This condition is expressed by the multimodal characteristics of the VR production and can be interpreted as a non-linear narrative of the virtual self.
4.4 Perspective-taking—Empathy
With the proliferation of immersive journalism, especially in the case of documentary VR productions, the question of empathy and its use by creators of VR has become more important. Stephen Harper’s book Madness, Power and the Media, based on his extensive study of mainly British media, makes the important point that the conflation of psychosis and psychopathy implies that people with mental disorders are “ruthlessly immoral.” He also discusses the lack of correlation between actual lived mental disorders and their cinematic representation, which occurs because the typical mainstream movie reproduces “social norms, punishing certain kinds of outliers, and oppressing—sometimes overtly, sometimes subtly—the Other” (Heath 2019, 6).
In VR, the question of morality is different due to the intimate situations that VR can enable and its transparency as a medium: we are imaginarily transported, and we also have a risk of “improper distance” in which the “other” whom we are witnessing in VR becomes “indistinguishable from ourselves” (Silverstone 2007, 172). In the case of journalistic VR productions, creators can contextualize their work as a design of a “unique aspect of their reality” (ibid.) in which the user’s empathy depends on that designed representation, which may result in an experience with a limited basis in lived reality (ibid.). I propose to use Peter’s concept of “witnessing,” which refers to the creation of first-person perspectives that can offer “a uniquely different level of understanding contrasted to reading the printed page or passively watching audiovisual material” (de la Pena 2010, 2 99). Witnessing is “the discursive act of stating one’s experience for the benefit of an audience that was not present at the event and yet must make some kind of judgment about it” (Peters 2001, 709). Witnessing also presumes inconsistency between one’s ignorance and someone else’s awareness and knowledge. Jones (2017, 180) identifies two types of witnessing: a reporter-led approach in which the experiencer works with the author, and a first-person approach in which the experiencer is the victim of the events. These two types of witnessing can be observed in the three VR productions that I analyze here.
I identify these productions as documentary works, as they all have a documentary aspect, following Gaudenzi’s definition of interactive documentaries: “[a]ny project that starts with an intention to document the “real” and does so by using digital interactive technology [is to be] considered an interactive documentary” (Gaudenzi 2013, 31–32). The represented mental illnesses do exist, their characteristics are presented in accordance with reality, and the only fictional elements are the characters, with whom the experiencer can witness and empathize.
Manic VR and Goliath offer a reporter-led approach. Welcome to Respite! also offers a reporter-led approach for its invisible audience members, but it almost literally creates the situation for the experiencer to have the first-person view in case of the protagonist. Manic VR uses a twist in the story: the phone recordings offer clear signals that help the experiencer to identify their nature, which at first offers the sense of being a voyeur. Later, when the speakers mention the name of Kalina (referring to Kalina Bertin, the director), the experiencer receives authorization and understands the clear references which affirm that these are not fictional recordings but real ones. The variety and mixture of fictional spaces (that lack any physicality and offer surreal visuals) with realistically arranged room environments, together with the smooth, fluidly changing scenes, soften the experience of the subject matter. The colors used in this experience are also more natural-looking: the visuals are generally soft, not harsh, and rather slowly changing shapes gently appear. The experiencer has no agency in the events but can experience some elements in a multisensorial way, such as by vibrating controllers. On the other hand, the experiencer has a low feeling of embodiment, as the body is not visualized besides the hand movements when manipulating the strings.
This is similar to the experience that Goliath offers, where the narrator states that this is a real story and Jo, the person suffering from the mental illness, talks directly to the experiencer. The narrator creates the onboarding situation, gives guidance to the new arrivals, and reflects on the experiencers’ embodiment process (even secretly manipulating their body awareness). The main protagonist, Goliath, does not interact or talk directly to the experiencers. He elucidates his life story, and the VR experience shows in a very subtle way how his reality is constructed between virtual and real worlds. First, experiencers become accustomed to the virtual space, and then they see the protagonist, first as his avatar in videogames (as a big transformer robot) before arriving at his various layers. The experiencers slide from more realistic scenes into a computer game that continues to illustrate Goliath’s life story. This continues, for example, by positioning the experiencers at a music mixing board where they can regulate the rhythm. The experiencers can also participate in an easy shooter game, as Goliath, in order to gain a better understanding of how videogames can be so engaging. At the end of the experience, the viewer is positioned into the 3D-scanned room of Goliath. From this room, the experiencer can also see the videogame space where they previously were; in this way, the production offers a blended view of realistic-looking and videogame-like spaces. Also, the experiencers here do not have space to manifest their agency. For example, when playing the videogame that illustrates a part of the life of the protagonist (or this is how one interprets it), one can feel that the storyline will stop if the buttons are not moved, even though the experience is actually designed to move forward after a while. The colors used in this experience express a more vivid, stressed environment. The gray color used in the experience—when showing Goliath in the mental asylum—expresses hopelessness, while the colors used in the gameplay scenes at the end are vivid and happier. All these design elements are used to offer a multisensorial message of the unspeakable: schizophrenia’s various phases. Here one can encounter symptoms that are not verbally expressible, which this VR experience tries to convey by addressing the various senses of the experiencer not by embodying the person who has the mental disorder but by rather closely witnessing his state. The experiencers can understand the causal relations better—and still the snippets when he or she they could interact as Goliath—offering fragments for embodying a person who has a psychosis.
In Welcome to Respite!, the experiencer embodies the main character. Here the experiencer receives onboarding material about dissociative identity disorder before joining the VR chat world. The participant who embodies the main character receives no direct “tag” about the person whom they must embody, and therefore the character is in constant suspense. The character also has a very high level of agency, although the actors who embody his parents continuously guide his verbal manifestations and actions according to their prescripted storyline. While the experience takes place in an everyday, homelike environment, the experiencer (and also the other audience members) experience snippets that depict the unknown. The creators include amorphous fluid figures surrounded by dark to offer a sense of how this disorder can feel. The audience member(s) try to fight these figures and sounds using their controller, but the system does not clearly show whether the actions are successful; the figures simply disappear or cease. Welcome to Respite! is part of a longer series, so the proposed narrative is not unfolded and the experiencer is left rather with a feeling of suspense.
However, almost none of these experiences puts the experiencers in the shoes of the victim of the events. Rather, they offer “the reporter-led approach” mentioned in Jones’s taxonomy. Experiencing mental health diseases as the person suffering from them can be challenging, and except for one specific role in Welcome to Respite!, none of the productions offers the right kind of onboarding. All of these VR experiences use the reporter-led approach in a very particular way: the experiencers encounter a storytelling approach that allows them to step into the world of the protagonist for some fragments of time, but the audience members are always reassured that they are outsiders or voyeurs who can only have these glimpses via the visual, auditory, tactile, and somatic modalities. In Welcome to Respite!, one experiencer has the chance to be in the shoes of the protagonist, but even in that case, that person does not receive the background information of the protagonist’s biography.
Even though perspective-taking is possible only for a very limited time, it nevertheless sharpens the senses of the participant, helping them achieve a “sense of impossibility,” as defined in [reference removed for review] as an experiencer feeling present in a world while knowing that the world is unreal, an element that strengthens the immersivity of a production. According to a study by Wiseman and Watt (2022), such experiences can play causal roles in enhancing creativity (Wiseman and Watt, 2022), and a more creative attitude on the side of the experiencer can lead to a better understanding of mental health issues.
4.5 Virtual reality productions and the ethical aspects of their design vs. complexity
All these productions deal with the topic of mental disorders, which requires a very careful and ethically considerate approach by the creators. In this section, I draft a possible ethical approach, relying on a framework by Madary and Metzinger, by which one can assess the ethical approach of the creators. After this, I inquire into how the engagement with the protagonist’s character in VR can help with developing an ethical approach.
In 2016, Michael Madary and Thomas K. Metzinger published one of the first articles on VR ethics, introducing the idea of a possible code of conduct for good scientific practices in the field of VR and for consumers using VR technologies (Madary et al., 2016). They emphasize that the main motivation behind their investigation is VR’s ability to enable the illusion of embodiment. Madary and Metzinger argue that this can have a manipulative effect on deep behavior, especially if “illusions of embodiment are misused” (ibid.), and they pose general questions about research ethics, such as the limitations of the code of conduct for research and the limits of the overall experimental environments. They also point to the importance of informing users about the lasting psychological effects of VR and “the possibility of using results of VR research for malicious purposes” (ibid.). Madary and Metzinger emphasize that those who use VR excessively can develop a condition in which they “experience the real world and their real bodies as unreal, effectively shifting their sense of reality exclusively to the virtual environment” (ibid.). While in these three case studies, none of the productions changes behavior in a manipulative way, neither the designers of Goliath nor of Manic VR intend to misuse embodiment, while Welcome to Respite! relies on embodiment and its effect on the experiencers.
In his book Engaging Characters (1995), Murray Smith 1995 argues that the term “engagement” has a broader application than does “identification.” Fictional narrations elicit three levels of imaginative engagement with characters, and these comprise the “structure of sympathy” (Smith, 75). He suggests using the term “engagement” to fight the misconception that the term “identification” can elicit, as it can mean that the spectator may become one with the character on the screen. This relationship can arise from a structure of sympathy, which includes recognition, alignment, and allegiance (Heath, 43). These steps can arise if the characters are engaging and if they are based on the “person schema” (as Smith describes it) that can help the creators construct an engaging character. This schema encompasses seven types of elements that a character on the screen should have: a body; actions and self-awareness; intents, beliefs, and desires; emotions; language; self-impelled actions and self-interpretation; and traits. Many of these can be addressed by VR applications, and it is important to keep them in mind as a ground for understanding the model of complex participation which I introduce later.
The described VR applications present two very different standpoints from the character-engagement point of view. Goliath and Welcome to Respite! both address their audiences with particular strategies. In Goliath, the narrator is the guiding agent; this offers the audience the role of voyeur, and the experiencers have to interact with her or the given interactive assets while being able to encounter some representations of the mental state included in the narrative. In Welcome to Respite!, most audience members can have the exact role of the invisible audience member, and besides this, they must step out and act with the other (invisible) audience members, without being able to learn about the actual effects of their interactions or about the other members. The experiencers who embody the main protagonist also have very little possibility of manifesting their agency, mainly due to the improvisational skills of the actor—even though it still requires complex focus. On the other hand, the experiencers can start to sympathize with their own character, as they receive continuous feedback and background information from the parents, and they can slowly develop an imaginative engagement and learn the emotions and the traits of their character. In Goliath, the experiencers can also develop this allegiance, both because the character presents his dramatically constructed life story and because he presents his playful character.
In the case of Manic VR, the viewers themselves can be considered the protagonist in some sense, as they are alone in the experience and no protagonist is guiding it—so here, no engagement could be developed with the protagonist. The phone texts mention the name of the director several times, which works to alienate the experiencer from embodiment and from feeling empathy, and it breaks the suspension of disbelief: the experiencers become aware of the documentary aspect. This enables them to have a more critical approach to the depicted events. This technique used by the creators goes against a suggestion of Brecht, who states that the Verfremdungs-effect creates the opportunity for the audience to lose “the illusion of being the unseen spectator at an event which is really taking place” (Brecht 1964, 91). This technique goes against the expectations of interactive VR applications and creates a strong sense of voyeurism in the experiencer.
The methods used in these productions, which offer the status of a voyeur for the participants or let them occupy the position of the protagonist only briefly, align with the ethical guidelines suggested by Madary and Metzinger. They provide a certain degree of alienation for the participants, which can prevent them from being too overwhelmed by the VR experience. This distance not only prevents them from the potential harmful effects of an intense VR experience involving mental illness but also reminds them that what they are experiencing is only a weak substitute for reality: they may be informed about mental illnesses but only to a limited extent.
5 Results
5.1 Participatory strategies designed to experience mental health disorders in interactive digital narratives
It is also worth comparing these VR productions from the aspect of experiencer participation activity. I will rely on a scale suggested by Michael Kirby to taxonomize this aspect. In his 1972 article titled “Acting and Not-Acting,” Kirby draws up a continuum of acting types. Here the scale shows the involved acting of the performer, meaning how much complexity the acting style represents or, as Kirby states, how much the performer is concerned with the “amount of acting.” On the two sides of the continuum are “not-acting” and “acting.”
For “not-acting,” he gives examples such as the stage attendants in Kabuki performances. They move props, help with on-stage costume changes, and bring tea. On the next stage of the continuum is found the performer who does not act but whose costume represents something or someone. Kirby calls this a “symbolized matrix,” as the performer bears referential elements while not acting as a performer. The next stage is “received acting.” This is described by the following example: if we imagine a setting in a bar where men are playing cards, and they do not have to react in any way with the main characters of the storyline, then according to him, these men do not act but merely play cards, “and yet we also see them as characters, however minor, in the story, and we say that they, too, are acting. We do not distinguish them from the other actors” (Kirby 1972, 5). According to the author, this is not “true acting.” One step closer to “true acting” is what is called “simple acting,” which can be observed in the case of public speakers or actors who are telling their “personal” truth while talking to the audience. Kirby states that acting appears “at the point at which the emotions are ‘pushed’ for the sake of the spectators. This does not mean that such speakers or actors are faking or that they do not believe what they are saying. It merely means that they are selecting and projecting an element of character (i.e., emotion) to the audience” (Kirby 1972, 7). The ultimate stage of acting on this scale is “complex acting,” which relates to the creation of several layers of the character. It also relates to the technical skills and abilities of the performers: “anyone can act, but not everyone in a complex way” (Kirby 1972, 8). Kirby’s scale measures the amounts of representation, simulation, and impersonation that can be helpful tools in drawing a clear line in audience behavior and level of acting.
I propose a repurposed scale, shown below, by rethinking Kirby’s scale with a focus on audience activity while considering what an interactive digital narrative (IDN) work requires. “Not-participating” simply refers to the type of audience who does not have to effectuate any interaction with the IDN. “Non-matrixed” audience members have a symbolic character. This is especially important as IDNs need the active participation of the experiencer in order to unfold. In “received participation,” IDNs offer a role for the experiencer, but this role is not crucial for understanding the IDN’s story framework, as it also enables meaning-making for which the status of the experiencer does not matter. In “simple participation,” participants are aware that their participation is important in unfolding the storyline, but their performative skills are not important for creating a meaningful experience. “Complex participation” happens when the participating audience member can have local (or even global) agency and can also use their verbal capacities to express their thoughts.
This last type of participation—complex participation—is what many IDNs offer their experiencers. In the particular examples above, received participation and complex participation are involved, and a curiosity to experience a novel embodied experience motivates participants to take the risk of trying out the VR experience. Through their multimodality, VR experiences allow these various types of participation; the presemiotic modalities of the VR technological devices nudge the experiencer in the direction of simple or complex participation. Meanwhile, the semiotic modalities offer seams for nonlinearity Figure 1. For example, when semiotic modalities change, temporal linearity also changes, and experiencers can find themselves in a new environment.
FIGURE 1. The participation scale developed by the author based on Kirby’s taxonomy (Kirby 1972).
The design strategies of Goliath offer the simple participation type to experiencers, while in Welcome to Respite! participants must adopt the form of complex participation in order to unfold the story. Therefore, the design must enable them to develop a certain type of sympathy. In the case of Manic VR, the participation can be considered non-matrixed, as experiencers must be there to unfold the storyline. In Goliath, the moments when the experiencers can instantiate the narrative system are those when, for instance, the environment or dimension is changing between the everyday and the virtual. In Welcome to Respite!, the experiencers’ expectations are managed such that they know that those moments will come when they interact with the environment, and the visuals guide them in doing this. In Manic VR, the narrative itself is non-linear and therefore cannot offer such prospects of meaningful interaction to the experiencer, whereas a coherent narrative could do so.
In a discussion of immersive theater, Alston develops the concept of “entrepreneurial participation,” a phenomenon that relates to the neoliberal attitude of our society (Alston 2013, 129) in which the experiencer is “bolstered by receiving the fruits of one’s own participatory effort as well as the efforts of others” (Alston 2013, 129). This is a common trait found in all three participation types discussed above. Even though Alston offers this term in the context of theater, elsewhere I have proposed that it should be extended to immersive digital media productions in general (see Bakk, 2021). Entrepreneurial participation relies on design strategies that create suspense for the experiencer. Mental health disorders, partially due to their taboo-ish nature and partially due to existing stereotypes, are an issue that many audience members are interested in experiencing, and they may pursue an element of suspense in these experiences.
6 Discussion and conclusion
The productions that I analyzed in this paper are careful to not create an experience of dark tourism, unlike some documentary VR experiences which were made to recreate physical historical locations of suffering or even death (see Fisher and Schoemann, 2018, for examples). Also, they offer neither a moral response nor a stereotype of the represented mental disorders. However, the creators carefully designed the roles such that the experiencer can find suspense in the seams of the non-linearity of the experience. Two of them, Goliath and Welcome to Respite!, do offer a certain type of sympathy toward the characters but do not conclude with a moral standpoint. The participation is created to offer a complex experience in which the audience members can experience non-linearity as a source for their inspiration to continue the experience provided by the multimodal character of interactive VR technology. A certain type of seamed character can also be observed, a character that permits the identification of a media-specific non-linearity in the case of interactive 6DOF VR experiences. All three productions approach mental health issues as dynamic systems and represent them through linearity or non-linearity (see Stepney 2018), while also offering limited agency to the interactor. The limitation is achieved in the sense that the experiencer has a sense of embodiment (Kilteni et al., 2012) and is scripted as an interactor in specific cases (Murray 1997).
The above-presented VR productions offer a different understanding of non-linearity by relying on the specificities of the VR medium. VR as a multi-modal medium can offer design possibilities that allow creators to offer a layered meaning-making with the help of visual, auditory, somatic, and tactile clues. Multi-modality also poses ethical risks, which can be avoided by scripting the interactor in particular ways. For instance, creators may achieve this by offering the witness role to the interactor or by providing live attendance by creators or orchestrators (as in Welcome to Respite!, when the interactor is put in the shoes of the protagonist). These various participatory strategies demonstrate how to present experiences that are not accessible in everyday life, as they are very complex and challenging to represent.
While complex subjects, including mental health disorders, have been presented throughout the history of literature and the arts, in our current era VR can offer a very unique experience to recipients. Many artworks have offered various ways of experiencing the subject, but VR offers a closer encounter by engaging the experiencer more intensely at the tactile and somatic levels. The complexity of this subject can be presented with the help of one of the most important specificities of the VR medium: a sense of embodiment, which offers a path for scenarios that are less linguistic and descriptive, even though, as I have outlined, this state can sometimes create fuzziness in the unfolding narrative. The productions I have provided as case studies have solved this by offering glimpses into the visualizations of the mental states and feelings of the protagonists while maintaining the outsider’s view of the participant (with the only exception being the main participant role in Welcome to Respite!).
Based on these case studies, I conclude that artists must adopt a complex approach when representing various mental health issues in order to present as many aspects of mental illnesses as possible. One way to approach these illnesses, such as depression and mood disorders, is through IDN productions that capture the complexity of these conditions and present them in multiple perspectives and in complex multisensorial ways. Understanding biological, psychological, and social levels of analysis and “both the individual components and the complex interactions among them” will enable designers and experiencers “to understand the systems from which psychopathology emerges” (Fried and Robinaugh 2020). These representations provide, in subjective and heuristic ways, the interpretation of a specific state of mind in relation to its context and environment. The representations can subsequently be grasped by VR experiencers more effectively as compared to other mediums, such as data visualization. Further research is needed to carefully analyze how VR productions with strong artistic approaches can deepen the sense of impossibility (as defined in Section 4) for participants and therefore also deepen their sense of immersion, enhancing or modifying their engagement or even the empathy they feel toward a particular protagonist.
Data availability statement
The raw data supporting the conclusion of this article will be made available by the authors, without undue reservation.
Author contributions
AB authored the entire article.
Funding
This research was supported by the Innovation Center of Moholy–Nagy University of Art and Design, Budapest (Innovation Center), and by COST Action no. 18230 Interactive Digital Narratives for Complexity Representations. The publication costs are entirely supported by the Innovation Center of Moholy–Nagy University of Art and Design, Budapest (Innovation Center).
Acknowledgments
The author is grateful for the careful notes of Işık Sarıhan.
Conflict of interest
The author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Alston, A. (2013). Audience participation and neoliberal value: Risk, agency, and responsibility in immersive theatre. Perform. Res. 18, 128–138. doi:10.1080/13528165.2013.807177
Bakk, Á. K. (2021). Curate It Yourself! Game mechanics and personalized experience in the immersive performance installation. Well Play. J. 10 (2), 116–134. doi:10.1184/R1/14919645.v4
Bakk, ÁA. K. (2020). “Magic and immersion in VR,” in Proceedings of the international conference on interactive digital storytelling. Editors A.-G. Bosser, D. E. Millard, and C. Hargood (Cham: Springer), 327–331. doi:10.1007/978-3-030-62516-0_29
Biocca, F., and Delaney, B. (1995). “Immersive virtual reality technology” in Communication in the age of virtual reality. Editors F. Biocca, and M. R. Levy (Hillsdale, NJ, US: Lawrence Erlbaum Associates), 57–124.
Bornkamm, H., Köhler, K., and Petraitis, M. (2018). Virtual and augmented realities – A conversation with william uricchio. Cinema 63, 38.
Bown, J., and Goltsov, A. (2018). “Narratives for drug design,” in Narrating complexity. Editors R. Walsh, and S. Stepney (Cham, Berlin, Germany: Springer), 253–268. doi:10.1007/978-3-319-64714-2_18
Brewin, C. R. (2001). A cognitive neuroscience account of posttraumatic stress disorder and its treatment. Behav. Res. Ther. 39, 373–393. doi:10.1016/s0005-7967(00)00087-5
Brooks, F. P. (1999). What's real about virtual reality? IEEE Comput. Graph. Appl. 19, 16–27. doi:10.1109/38.799723
Fisher, J. A. (2017) “Empathic actualities: Toward a taxonomy of empathy in virtual reality,” in Proceedings of the Conference on Interactive Digital Storytelling ICIDS 2017. Editors N. Nunes, I. Oakley, and V. Nisi (Funchal, Madeira, Portugal: Springer), 233.
Fisher, J. A., and Schoemann, S. (2018). “Toward an ethics of interactive storytelling at dark tourism sites in virtual reality,” in Interactive storytelling. ICIDS 2018. Lecture notes in computer science. Editors R. Rouse, H. Koenitz, and M. Haar (Cham,Berlin, Germany: Springer), 11318. doi:10.1007/978-3-030-04028-4_68
Foa, E. B., and Riggs, D. S. (1994). “Posttraumatic stress disorder and rape,” in Post- traumatic stress disorder: A clinical review. Editor R. S. Pynoos (Baltimore, MD: The Sidran Press), 133–163.
Fried, E. I., and Robinaugh, D. J. (2020). Systems all the way down: Embracing complexity in mental health research. BMC Med. 18, 205. doi:10.1186/s12916-020-01668-w
Gaudenzi, S. (2013). The living documentary: From representing reality to Co-creating reality in digital interactive documentary. PhD Thesis (London, UK: Goldsmiths - Centre for Cultural Studies, University of London).
Hameed, A., and Perkis, A. (2018). “Spatial storytelling: Finding interdisciplinary immersion,” in Proceedings of the International Conference on Interactive Digital Storytelling ICIDS 2018. Lecture Notes in Computer Science. Editors R. Rouse, H. Koenitz, and M. Haahr (Cham: Springer), 11318. doi:10.1007/978-3-030-04028-4_35
Jones, S. (2017). Disrupting the narrative: Immersive journalism in virtual reality. J. Media Pract. 18 (2–3), 171–185. doi:10.1080/14682753.2017.1374677
Kilteni, K., Groten, R., and Slater, M. (2012). The sense of embodiment in virtual reality. Presence. (Camb). 21 (4), 373–387. doi:10.1162/PRES_a_00124
Klein., K. (2003). “Narrative construction, cognitive processing, and health,” in Narrative theory and the cognitive sciences by david herman (Stanford, CA, USA: CSLI Publications), 56
Kreitler, M. “Disinformation in discourse: A mental note on the role of the DSM in movie analysis.Disinformation Cultures. Available at: https://disinformationcultures.net/kreitler
Madary, M., and Metzinger, T. (2016). Recommendations for good scientific practice and the consumers of VR-technology. Front. Robot. AI 3, 3. doi:10.3389/frobt.2016.00003
Murray, J. H. (1997). Hamlet on the holodeck: The future of narrative in cyberspace. NY, USA: Free Press.
Peters, C. (2011). Emotion aside or emotional side? Crafting an ‘experience of involve- ment’ in the news. Journalism 12 (3), 297–316. doi:10.1177/1464884910388224
Robb, J., and Stone, J. (2016). Implicit bias toward people with mental illness: A systematic literature review. J. Rehabilitation 82 (4), 3
Silverstone, R. (2007). Media and morality: On the rise of the mediapolis. Malden MA and Cambridge UK: Polity.
Silberman, M., Giles, S., and Kuhn, T. (Editors) (2014). Brecht on theatre (London, UK: Bloomsbury).
Slater, M. (2009). Place illusion and plausibility can lead to realistic behaviour in immersive virtual environments. Phil. Trans. R. Soc. B 364 (1535), 3549–3557. doi:10.1098/rstb.2009.0138
Stepney, S. (2018). “Complex systems for narrative theorists,” in Narrating complexity. Editors R. Walsh, and S. Stepney (Berlin. Germany: Springer), 253–268. doi:10.1007/978-3-319-64714-2_18
Suedfeld, P., and Pennebaker, J. W. (1997). Health outcomes and cognitive aspects of recalled negative life events. Psychosom. Med. 59, 172–177.
Vitillio, T. (2021). Goliath review: A deep experience about psychosis. Skarredghost blog. Available at: https://skarredghost.com/2021/09/09/goliath-review-vr/.
Walsh, K. R., and Pawlowski, S. D. (2002). Virtual reality: A technology in need of IS research. Commun. Assoc. Inf. Syst. 8 (1), 20. doi:10.17705/1cais.00820
Keywords: virtual reality, complexity, narrative, participation, mental disorder
Citation: Bakk ÁK (2023) Representing mental disorders with virtual reality applications: Designing for multimodality and complex participation. Front. Virtual Real. 3:881766. doi: 10.3389/frvir.2022.881766
Received: 23 February 2022; Accepted: 24 October 2022;
Published: 04 January 2023.
Edited by:
Lissa Holloway-Attaway, University of Skövde, SwedenReviewed by:
Thomas Hughes-Roberts, University of Derby, United KingdomJamie Fawcus, University of Skövde, Sweden
Copyright © 2023 Bakk. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ágnes Karolina Bakk, bakk@mome.hu