A Best-Fit Framework and Systematic Review of Asymmetric Gameplay in Multiplayer Virtual Reality Games

Increasingly, virtual reality (VR) design and research leverages gameplay asymmetries, flattening discrepancies of interface, abilities, information or other aspects between players. A common goal is to induce social interactions that draw players without head-mounted displays into a shared game world. Exploring these asymmetries resulted in many artifacts, creating an innovative yet disparate research landscape that showcases points for improvement in coverage of the field and theoretical underpinnings. In this article, we present a literature review of asymmetry in multiplayer VR games, using a framework synthesis method to assess the field through a lens of existing literature on asymmetries in gameplay. We provide an overview of this emerging subfield and identify gaps and opportunities for future research. Moreover, we discuss how research artifacts address prior theoretical work and present a “best fit” framework of asymmetric multiplayer VR games for the community to build upon.

1 Introduction

Virtual reality (VR) technology has received immense interest from researchers, developers, designers, and the games industry alike in recent years. Proponents praise VR’s ability to facilitate novel and immersive experiences. Nevertheless, the immersive quality of VR technologies, in particular head-mounted displays (HMDs), has also received criticism for its potentially isolating characteristics, both technologically and socially (Boland and McGill, 2015). This isolation risk has inspired game designs that enhance the VR experience by including bystanders as co-players. These game designs can leverage the benefits of multiplayer game experiences (e.g., supporting social connectedness; Woods, 2009; Vella et al., 2019) without requiring additional HMDs. Examples include giving bystanders insight into the virtual environments (VEs) inhabited by the HMD player (e.g., via a monitor displaying their camera perspective; Jeong et al., 2020), or an active role in the game (e.g., giving them essential information that the HMD player requires, thus enforcing communication; Liszio and Masuch, 2016; Smilovitch and Lachman, 2019).

These interactions of HMD players with non-HMD players can be considered a type of asymmetry: a difference in the interface with which users interact with the VE. However, while asymmetry of player interfaces is increasingly common in VR experiences (rooted in cross-reality paradigms), there are many other asymmetry types that can be designed (Harris et al., 2016). Asymmetric games explicitly incorporate and design for differences between players in how they interact with the game, and for the abilities and information the players possess within the game world. With the influx of research in this context, asymmetric multiplayer games are emerging as a subfield of VR games research.

The VR artifacts¹ being produced—and related findings on how asymmetric game designs affect player experience (PX)—are novel and innovative because this is an emerging research field. Nevertheless, the artifacts are also disparate in their focus. In particular, few designers or researchers deeply engage with prior theories on designing for and catering to gameplay asymmetries in general (outside of VR). We argue that a systematic approach and close integration of theory-driven games research is essential to leverage the full potential of asymmetric multiplayer VR games. As a first step, we aim to gain an overview of how asymmetries are being incorporated in multiplayer VR games, investigating what asymmetries multiplayer VR games contain and how multiplayer VR asymmetries affect player experience.

To map out this research area, we conducted a literature review on asymmetric multiplayer VR games. Beginning with 481 identified records from The ACM Guide to Computing Literature and Scopus covering the majority of the human-computer interaction (HCI) literature, we followed a systematic screening and snowballing approach (via PRISMA; Moher et al., 2009) to arrive at a corpus of 25 relevant papers on asymmetric multiplayer VR games.

Given the novelty of asymmetric multiplayer VR, we employed a method based on framework synthesis (Carroll et al., 2011; Dixon-Woods, 2011) to guide our analysis of these papers against the backdrop of prior literature. Papers were synthesized within a “best fit” framework based on theoretical work on asymmetry in games, gameplay aesthetics, social aspects of PX, and shared control. This methodology works well for research fields in which there is no clear “best fit” theory: it applies multiple theories or models to reduce limitations of a single one. This allows researchers to “engage with theory but not be constrained by it” (Carroll et al., 2013). Further, as it is open to improvements on the a priori framework, it turns the review into an opportunity to reflect on and refine existing theory.

The goal of the review was to explore the following questions: 1) what kinds of asymmetry are being explored in research on multiplayer VR games, and 2) how asymmetry in these games affects PX. For this purpose, we sought out papers that touched on the design and/or evaluation (for any participant group, with any intervention, comparison, or outcome) of any kind of asymmetric gameplay within a multiplayer game experience featuring at least one HMD-VR player.

The contributions of this work are twofold: 1) Our work provides an overview of this emerging subfield of VR games research, shows how it engages with prior theoretical research on asymmetry, and identifies gaps in the literature to guide future research. Specifically, we point to the following as opportunities for future work: multiplayer games with more than two players, alternative interfaces to monitor variants, mirrored and unidirectional interdependence between players, remote play, shared control within the game world are rare in the design of asymmetric multiplayer VR games. Additionally, effects of player skill or familiarity with interface and partner are rarely considered. Explicitly shared physical spaces, embodied physical interaction, and the presence of a human co-player emerged as the key drivers of positive PX in asymmetric multiplayer VR. These key findings are summarized in Table 3. 2) Furthermore, we present, apply, and refine the “best fit” framework for the asymmetric VR games field to employ and build upon (see Figures 2, 3). While time will bear out the use of the refined framework, we suggest that it has generative, structural, and analytical potential: it could inform the design of asymmetric multiplayer VR games to systematically explore the overall design space (generative), scaffold reporting and description of such games (structural), and guide both future evaluations and literature reviews of such games (analytical).

2 Background

Progress in VR has long been entangled with “other” forms of realities resulting in mixed realities (MR; Milgram and Kishino, 1994), which enrich the real world with virtual content as in augmented reality (AR) or augmented virtuality (AV), which “mutually reflect, influence, and merge into each other” (Lifton, 2007). Such mutual interactions between realities is often described as cross-reality (XR; Want, 2009) or blended reality (Schmidt et al., 2019), and increasingly refers to any exchange of information between realities “to […] a meaningful and discernable effect” (Coleman, 2009). A close and bidirectional connection between VR and the real world is necessary for collaboration in scenarios that see one user located in VR and one in the real world (Coleman, 2009; Reilly et al., 2010). The general increase in VR usage in recent years is also driving a trend toward XR designs and research (Efstratios et al., 2018; George et al., 2019; Nakagawa and Sonobe, 2019; Kiourt et al., 2020; Simeone et al., 2020; Wang et al., 2020). In particular, the social aspects of cross-reality VR research are becoming popular (Yassien et al., 2020). For example, VR researchers are exploring the questions of understanding how different environments (e.g., public, private) and familiarity among users can affect XR experiences (O’Hagan et al., 2020), how they can provide VR users with an awareness of the presence of those around them (McGill et al., 2015; O’Hagan and Williamson, 2020), and whether bystanders can understand the experiences of VR users by observing them (George et al., 2019).

VR is increasingly available to consumers. However, the usage of multiple HMDs at the same time remains rare because of the cost, required space, and potential collisions between users. VR’s most appealing feature—its immersiveness—can also engender isolation (Boland and McGill, 2015; Mütterlein and Hess, 2017; Rogers et al., 2019). Thus, cross-reality consisting of the inclusion of non-VR users into the VR experience (asymmetry of interface) has been explored as a way to increase social interaction for both the HMD user and bystanders (e.g., Gugenheimer et al., 2017a; Zhou et al., 2019; Lee et al., 2020), with promising results for enjoyment and social connection. Furthermore, it may offer an alternative to users suffering from VR sickness (Jerald, 2015; Peck et al., 2020).

Commercial VR games have also begun employing asymmetry of interface between an HMD player and one or multiple non-HMD players, such as Keep Talking And Nobody Explodes (Steel Crate Games, 2015), Acron: Attack of the Squirrels (Resolution Games, 2019), Carly and the Reaperman—Escape from the Underworld (Odd Raven Studios, 2019) and Panoptic (Team Panoptes, 2019). Yet games offer a much broader design space for asymmetries, including aspects beyond differences in interface. Multiplayer games enable social environments wherein players can engage, interact, and develop trust toward each other (Depping et al., 2016). They can foster relatedness between players, which supports well-being (Ryan and Deci, 2000; Deci and Ryan, 2011). However, designing for multiplayer engagement holds additional challenges, as multiple players may require the game to accommodate different abilities, preferences, or technical equipment. By integrating asymmetries in their mechanics, dynamics, and aesthetics (Harris et al., 2016), games can engage many different players.

Previous works (Manninen and Korva, 2005; Zagal et al., 2006; Beznosyk et al., 2012) have addressed points of asymmetric game design (such as distribution of information, goals, and varying levels of responsibility) in the design of collaborative games; similar aspects are reflected in game balancing to address differences in player skill (Cechanowicz et al., 2014; Vicencio-Moreira et al., 2014; Abuhamdeh et al., 2015). Harris et al. (2016) introduced a first conceptual framework of asymmetric games, using the widely adopted (Mechanics-Dynamics-Aesthetics) MDA framework (Hunicke et al., 2004) as an analytical lens for asymmetric games. In their framework, they formulated themes specific to that context (i.e., mechanics of asymmetry, dynamics of asymmetry, and aesthetics of asymmetry), which we will discuss in detail below. Further, they showed that asymmetric gameplay can increase social presence, connectedness, and immersion compared to a symmetric game (Harris and Hancock, 2019). Their findings highlight the resulting increased interdependence between players—their degree of reliance on each other (Beznosyk et al., 2012; Harris and Hancock, 2019)—as fostering positive effects on PX (e.g., higher social presence and connectedness). A recent example in VR found similar effects (Hansen et al., 2020).

In summary, we observe asymmetry in VR (particularly—but not only—of interface) as an emerging trend in XR. In the context of games, the inclusion of non-VR players in VR games has the potential to combat isolation, enhance engagement, and cater to social interaction motivations, thus prompting this review. We address further theoretical related work in more detail below as part of our systematic review of this research field.

3 Methods

To gain a deeper understanding of this emerging research field, we conducted a systematic review of the literature. We build on the research gap and motivation outlined in the previous section: the combination of VR and non-VR users is a promising way to increase immersion and reduce potential isolation. This led us to articulate our research questions and inclusion criteria for this systematic review to focus on artifacts with at least one VR user. Our two research questions were: $R{Q}_1$:“What kinds of asymmetry are being designed in multiplayer VR games?” and $R{Q}_2$: “How does VR asymmetry in multiplayer VR games affect PX?”

3.1 Approach

We followed the PRISMA protocol for systematic reviews (Moher et al., 2009) to identify and screen records; our PRISMA protocol (Shamseer et al., 2015) is provided in the Supplementary Materials. We employed a technique based on framework synthesis (Carroll et al., 2011; Dixon-Woods, 2011) to thematically analyze relevant results through a theoretical lens of existing literature on asymmetric player experience.

Figure 1’s PRISMA flowchart (Moher et al., 2009) details the stages (and the corresponding number of records) prior to the synthesis of this review: initial identification (N = 481 records), removal of duplicates (→N = 399), rigorous screening based on inclusion criteria (→N = 21), a snowball approach to identify additional papers (n = 6) based on the initial screening selection (→N = 27), and full-text screening for final inclusion (→N = 25). This final corpus of N = 25 records (of which n = 11 were included in quality assessment) underwent the “best fit” framework synthesis analysis.

FIGURE 1

FIGURE 1. PRISMA flowchart detailing the identification, screening, and eligibility stages prior to the framework synthesis.

3.1.1 Identification: Data Sources and Queries

We used two online databases to identify potential papers: The ACM Guide to Computing Literature published by the Association for Computing Machinery (ACM) and Scopus², Elsevier’s abstract and citation database. With ACM focusing on computing sciences and Scopus providing a larger but more general coverage, these two databases together offer comprehensive access to research on VR, games, and human-computer interaction (HCI). While this necessitates two different search strings, the use of more than one database is a mark of quality and comprehensive coverage in reviews.

Starting with our research questions, we phrased and tested search queries to target asymmetry in VR games via the keywords asymmetr*, VR/virtual reality, and game*. Additionally, in a recent study exploring inter-dependencies between players within a custom asymmetric VR game, agency emerged as an important aspect in the asymmetric gameplay experience (Karaosmanoglu et al., 2021). In this study, we followed the definition put forth by Murray (2017) as “the satisfying power to take meaningful action and see the results of our decisions and choices.” This PX component is particularly relevant in asymmetric VR games: creating interdependence between players can easily cause one player to have a more active or dominant role, which manifests as a stronger ability to take meaningful action. This is potentially amplified in VR when users are separated from each other by the VR headset. Karaosmanoglu et al. (2021) have shown that low/high agency can be felt keenly by players of an asymmetric VR game (and also that low agency is not inherently something to be avoided, nor is high agency always something to be sought). Because of this, and in the expectation some research in this context might refer to “asymmetry” in roles, mechanics, or interfaces as a difference in agency, we included the term agency as well. As this was added via an “OR” operator the additional search term does not exclude any relevant papers. We applied a filter to all keywords except game* to check for occurrence in the abstract, to exclude papers that do not focus on VR and asymmetric interaction. Thus, we aimed for papers that mention the term “game” at least somewhere in the paper (as a minimum requirement), but that are focused closely enough on VR and asymmetry that they mention both in the abstract.

Table 1 shows the exact queries, and the number of papers identified thereby in the ACM and Scopus databases (prior to duplicate removal).

TABLE 1

TABLE 1. Data sources, queries, dates, numbers and publication dates of results.

Beyond requiring the involvement of (any kind of) asymmetry in a multiplayer game with at least one VR player, we did not exclude papers based on study characteristics (e.g., participant groups). We otherwise considered any English-language, peer-reviewed original research article. Merging of the database exports and removal of duplicate³, non-English, and ineligible-type⁴ papers resulted in an initial pool of 399 papers to enter the screening stage.

3.1.2 Initial Screening and Inclusion Criteria

We performed our screening of the 399 records identified after duplicate removal based on their Abstract and Title, using the following inclusion criteria:

1. Does it address an HMD-based VR gameful experience?

2. Does it make a statement about any type(s) of asymmetric gameplay in VR games?

3. Does it make a statement about how players experience any type(s) of asymmetric gameplay in VR games?

To be included, papers had to fulfill item 1, as well as either item 2 and/or item 3. Rating based on these inclusion criteria was conducted by three of the authors, with each coder rating two-thirds of the papers (n = 266). Each paper received inclusion/exclusion votes by two authors. In case of disagreement, the third coder was asked to break the tie⁵. As a result, a total of 21 papers were identified in this stage as included for further steps; 378 papers were excluded. Of these, n = 42 papers were excluded because they did not involve a VR game experience; n = 366 papers involved no asymmetric VR game experience. Several excluded papers involved asymmetric collaborative scenarios (VR or otherwise), but were not games (e.g., Piumsomboon et al., 2018). While we note that such papers may certainly be interesting to reflect on collaboration in asymmetric scenarios, for the scope of this paper we wanted to focus on games or gameful experiences in particular, as we expect these kinds of experiences to leverage higher intrinsic motivation among users/players. How the findings in asymmetric games can be generalized to collaborative work scenarios (or vice versa) is a question for future work. Additionally, we note that the ACM database is loose in matching keywords, and so a portion of our excluded papers were ones that did not actually mention asymmetr* or agency, but rather provided results for subsets of the keywords, or did match the keywords but only in metadata outside the abstract field (despite the ABS parameter⁶). Overall, however, the queries may have caught more than strictly relevant, and were subsequently filtered in the screening process via human coders. We considered this a safer approach compared to more restrictive queries that might miss relevant papers, and it is common in systematic literature reviews.

We calculated inter-rater reliability (IRR) between the three coders for the paper screening (included/excluded) via Cohen’s Kappa test (Gamer et al., 2012). Results indicated high IRR between the first and second (97.7%; κ = 0.901, z = 10.4, p < 0.0001), between second and third (100%; κ = 1, z = 11.5, p < 0.0001), and between the first and the third (100%; κ = 1, z = 11.5, p < 0.0001) authors.

3.1.3 Snowball Approach

To avoid being too dependent on the specifics of our search terms, and to catch papers that describe asymmetric gameplay while not using that terminology, we applied a backwards snowball approach (Wohlin, 2014) to the 21 initially identified papers. This snowball approach consisted of the following steps: 1) For each initially included paper, we extracted references that appeared relevant based on title. We expanded this list based on reading each of the initially included papers’ introduction and related work/background sections. 2) Potential additions were checked for not being duplicates. We then applied our review’s inclusion criteria to their abstract, and decided based on consensus from two authors. Resulting from this procedure, six more articles were identified for our review (see Table 2), yielding a total of 27 papers for the next steps of the review.

TABLE 2

TABLE 2. Quality assessment scores of the identified papers in our literature review. The range of overall quantitative and qualitative scores were 0–1; 1 represents the highest possible point (Kmet et al., 2004). For appropriateness scores (Connolly et al., 2012), 6 constitutes the highest possible score while 2 equates to the lowest score. In the synthesis, we refer to papers by ID.

3.1.4 Full-Text Eligibility

We checked the full text of all identified papers for eligibility. During this process, two articles were removed from the pool of included papers entirely⁷. Removing these two papers resulted in the final selection of 25 papers (the final corpus; see Table 2).

3.1.5 Quality Assessment

As recommended for literature reviews, we conducted a quality assessment (QA) stage (Aromataris and Munn, 2020). Given the early stage of the research field, however, we consider even informal playtesting to hold potentially relevant information, and so we did not aim to exclude papers with only informal empirical findings based on the QA procedure. Instead, we use this stage to gain insight into the applied methodology and the quality of empirical work in this field. Further, some papers did not contain studies at all, instead consisting of theoretical work or system descriptions. While these are still useful for the framework synthesis stage, there is—to our knowledge—no checklist for assessing quality of that kind of work. The QA scoring was thus only applied to papers with a clearly identifiable quantitative and/or qualitative user study; papers without this were excluded from the QA process (see Table 2 for an overview).

The first nine papers (36%) were assessed based on their full text by the first two authors together (double data extraction) to ensure consensus in rating, while the remaining papers were split to be assessed separately. The JBI Manual for Evidence Synthesis (Aromataris and Munn, 2020) suggests that two-reviewer quality QA meets best practices for systematic reviews. Further, we conducted double data extraction for 36% of papers, which further reduces error (Buscemi et al., 2006), and ensures reliability of coding for the QA process (McDonald et al., 2019).

In total, 11 papers were found to fulfill the prerequisites for the QA stage. To these, we applied Kmet et al. (2004)’s QA criteria checklist for quantitative (n = 14 checklist items) and qualitative (n = 10 items) studies. For example, these verify that papers describe their research questions and study designs appropriately, but equally focus on reporting of results, methodology, data collection methods, and reliability and validity of the measurements. It is rated on a scale of Yes = 2, Partial = 1, No = 0, Not applicable (n/a) = -2. Based on Kmet et al. (2004)’s instructions, the papers’ QA scores were calculated by dividing the sum score by the total possible score (quantitative: 28; qualitative: 20). This scoring is punitive to more informal studies (as n/a ratings lead to minus points). Additionally, all empirical studies comparing an HMD experience to a non-HMD experience cannot realistically blind participants and investigators to the study conditions⁸. Low scores for quantitative studies are the result. We thus excluded two questions to also calculate an adjusted QA score for the quantitative papers.

Additionally, we used two items based on Connolly et al. (2012)’s checklist to determine the relevance of each paper to our review⁹. We applied this rating based on a scale of high = 3, medium = 2, or low = 1 appropriateness for this review; the overall appropriateness score was calculated by summing up the two item ratings.

Table 2 illustrates which papers were included in the QA, the mean QA score for empirical studies (qualitative, quantitative, and adjusted quantitative), and each paper’s appropriateness scores to be included for review.

3.1.6 Framework Synthesis

For the synthesis stage of our literature review, we drew on framework synthesis as a methodology, which is based on framework analysis (Pope, 2000; Ritchie and Spencer, 2002; Dixon-Woods, 2011), a qualitative method for data analysis. This method consists of five stages (Pope, 2000): 1. familiarization (becoming acquainted with the raw data), 2. identifying a thematic framework (development of our a priori framework), 3. indexing (applying the framework to the data), 4. charting (extracting and summarizing data into a framework overview), and 5. mapping and interpretation (synthesizing the themes and findings from the framework overview). When used for a literature review, this is called framework synthesis (Carroll et al., 2011; Dixon-Woods, 2011).

In particular, we used a variant of framework synthesis that employs a “best fit” strategy (Carroll et al., 2011; Dixon-Woods, 2011; Carroll et al., 2013): Researchers use a theoretical framework as a guiding lens through which they explore relevant literature. With the “best fit” strategy, the lens can be an existing framework, or alternatively an extension or combination of (an) existing framework(s). The benefit is that the lens can provide a frame through which the literature review can be explored and synthesized systematically, without first creating a comprehensive framework through time-consuming inductive methods. For this reason, it has found uptake in health science for policy-making (Dixon-Woods, 2011); it is considered a reliable synthesis method and allows “testing [of] existing potentially generalisable theories and models within a specific context” (Carroll et al., 2011).

Given the novelty of the field, we consider it a useful approach here as well, because there is no seminal, confirmed model of asymmetric game experiences. Applying framework synthesis enabled us to explore both how the current papers in this emerging field engage with prior theoretical work, and how well suited the a priori framework (based on prior theoretical work) is to classify asymmetric VR games. We consider now to be a crucial time to conduct this systematic review, because there are a few candidates for theoretical frameworks to draw from (see next section), and enough relevant papers to review to generate a spectrum of the emerging field in our synthesis. Yet it is also “early” enough in the subfield’s emergence for our findings to yield a thorough foundation for future work.

After developing our “best fit” a priori framework for asymmetric multiplayer VR games (see next section), we conducted an informal thematic analysis (Braun and Clarke, 2006; Nowell et al., 2017; Braun and Clarke, 2020), extracting text relating to and describing how each column or facet of the framework applied to each paper. We consider our approach to be a hybrid orientation of both codebook and reflexive thematic analysis (Braun and Clarke, 2020): we employed the framework as our codebook, and eschewed inter-rater reliability, but we nevertheless valued consensus between coders as a means to drive discussion surrounding the coding of papers in the framework, and the validity of the framework itself. We followed a consensus coding approach between two coders for nine papers (36%), and then coded the remaining papers separately. Disagreements and arising uncertainties in coding were discussed and resolved (thus making a second inter-rater reliability calculation moot) in recurring meetings throughout the synthesis stage¹⁰.

3.2 Developing the A Priori Framework

The “best fit” a priori framework (see Figure 2) began with the work by Harris et al. (2016). Based on several game design iterations and a user study (Harris et al., 2014; Harris et al., 2016; Harris and Hancock, 2019), their work has resulted in a conceptual design framework for asymmetric games. In turn, their framework builds upon the mechanics, dynamics, and aesthetics (MDA) framework (Hunicke et al., 2004), which describes games through the designed rules and logic (mechanics), the resulting gameplay based on players’ input (dynamics), as well as players’ emotional and immersive experience while interacting with the game (aesthetics). Harris et al. (2016) provide an overview of dimensions of asymmetry across the MDA categories. In game mechanics, they describe asymmetry of ability, challenge, interface, information, investment and goal/responsibility. For dynamics, they classify the type/direction of interdependence between players (mirrored, unidirectional, or bidirectional dependence) and temporal aspects thereof (asynchronous, sequential, expectant, concurrent, coincident). We included these categories to systematize our coding and synthesis of the described VR games.

FIGURE 2

FIGURE 2. Our a priori “best fit” framework for asymmetric VR games.

A thematic analysis of PX in an asymmetric game reported in the same paper also generated “salient themes most relevant to the design of asymmetric games” (Harris et al., 2016). These themes were posited based on their analysis of asymmetric games and their own design work (creating and evaluating an asymmetric game using mixed methods) — under application of the MDA framework (Hunicke et al., 2004) (including aesthetics) as an analytical lens. We thus use the Harris et al.’s aesthetics of asymmetry in our paper to build on this prior work and its specific application within asymmetric games. These categories of aesthetics of asymmetry consist of: leadership and primacy, effects of player skill, familiarity with interface or partner, interdependence and necessity, and coordination. Because the aesthetics relate to players’ actual experience, we applied these categories to reports of empirical user studies or in formal playtesting, as well as speculation about expected PX.

The MDA framework on which Harris et al. (2016)’s conceptual work is based further discusses several dimensions of players’ aesthetic experience: sensation, fantasy, narrative, challenge, fellowship, discovery, expression, and submission (Hunicke et al., 2004). Because the aesthetics categories described by Harris et al. (2016) are specific to the game stimuli they tested in their study, we included the MDA aesthetics in our a priori “best fit” framework, to enable us to capture a more extensive understanding of PX in asymmetric VR games.

Further, we added two more exploratory framework parts, in the spirit of the “best fit” nature of this methodological approach: one designed to explore social asymmetry such as differences in shared space, age, and abilities (custom categories) and factors put forth by Kaye (2016) regarding teamwork and communication in games. The dimension of shared space emerged from the initial full-text eligibility check of the selected papers when we noticed that some papers allow for or address remote play, meaning that there can be a difference in location or space (These options of remote vs. co-located were later extended during the review by the observation that differences also occur within co-located setups in how the space and proximity been players are framed and utilised.). We included the categories of age and abilities with the expectation that differences between players could frame players of different ages (e.g., young children and parents or grandparents), or different cognitive or physical abilities (e.g., for rehabilitative training games in clinical settings). Further, we include work by Kaye (2016) which has suggested factors that facilitate an experience of flow within social play in digital games: communication, teamwork, and knowledge of others (e.g., in terms of skills relevant to the task). While not relating specifically to asymmetry (e.g., of communication), we saw this as an opportunity to both look out for such factors, and capture relevant factors for social multiplayer gaming experiences in general.

The final aspect of the framework consisted of patterns of shared control suggested by Sykownik et al. (2017). This distinguishes between different ways that players have control over points or entities in a digital game, which are termed loci of manipulation. They distinguish between distinct loci of manipulation (players control different points/entities in the game), and mutual locus of manipulation (players share control of the same points/entities in the game). We added this because shared control is described as another type of interdependence between players.

The full a priori “best fit” framework is illustrated in Figure 2. Its categories make up the variables for which data was collected by looking at each paper’s full text.

4 Results

This section presents the synthesis for the 25 papers in our corpus, derived from the PRISMA procedure described above and illustrated in Figure 1. We begin with an overview of the empirical papers (a detailed overview of study characteristics is provided in the Supplementary Materials), then discuss all papers through the lens of our a priori framework. We refer to papers by their index in the corpus (i.e., #x for the paper with ID x, see Table 2). When referring to numbers of papers, we count #11 and #5 as one paper given their very close similarity, as well as #13 and #9 due to the former being a demonstration of the latter. #21is an earlier iteration of #18, yet these were counted separately as they do vary in content.

A summarized overview of our key takeaways is also provided in Table 3. Further, based on our synthesis process and findings, we propose changes to our “best fit” framework to develop a posthoc framework (addressed in detail within the discussion of this paper). How our findings fit within the posthoc framework is shown in Tables 4,5.

TABLE 3

TABLE 3. Summarized overview of takeaways and research gaps from the review synthesis, as well as meta takeaways that led to the adjustment of the a priori framework to develop the posthoc framework.

TABLE 4

TABLE 4. An overview of how the final corpus of surveyed papers fits into the (Harris et al. part of the) posthoc framework. Emptier columns indicate existing research gaps in the field.

TABLE 5

TABLE 5. An overview of how the final corpus of surveyed papers fits into the (MDA, social asymmetry and shared control parts of the) posthoc framework. Emptier columns indicate existing research gaps in the field. Codings are subjective and relate to the paper as a whole (i.e., covering multiple applications).

4.1 Summary of Empirical Research

A total of 11 papers were identified as containing user studies of some kind (n = 4 mixed-design, n = 6 quantitative, n = 1 qualitative). All featured one or multiple custom game(s); commercial asymmetric VR games were only used in one of the papers (#10): AudioShield (Fitterer, 2016) and Keep Talking Then Nobody Explodes (Steel Crate Games, 2015), in addition to a custom game.

Across all studies, the corpus reports on the experience of a total of 289 participants [an average of 26.27 (SD = 23.35) participants per paper]. Gender was not always clearly reported for all study participants. A few papers reported no gender information at all (#4, #6), or only reported it partially (e.g., reporting only the number of female participants, but not specifying the remainder of participants’ gender as in #18 or #24). Additionally, it was sometimes unclear how gender was assessed (e.g., which answer options were presented). We summarized gender based on reported distributions: of n = 221 participants with reported gender, n = 81 were female (37%), while n = 140 male (63%); non-binary participants are not reported. This pattern of unclear reporting repeated for the age of participants; while some papers reported both average age/age range and standard deviation (#9, #16, #18), others only presented the age range or no age at all. Overall, PX was tested with fairly young participants (average = 22.43). The context of recruitment was often also unclear but can be assumed to have been a university/lab setting in many papers. A notable exception are the public museum settings in Serubugo et al.’s work (#6, #7), where detailed demographics are omitted in favor of in-the-wild empirical playtesting. Almost all papers focused on asymmetry of interface as the main independent variable in their study: namely an HMD player, a (or multiple) non-HMD player(s), and in some cases additional spectators. A few studies instead explored varying viewpoints of the non-HMD player (#1, #4) as their independent variable: first-person point, third-person point of view, or seeing both (#4)/switching between the two (#1). One paper used the type of social entities in VR (human vs. agent) and their interaction with the HMD player (interactive vs. non-interactive) as independent variables (#16).

For quantitative studies, we observed similarities in how studies assessed PX of participants; the majority of these papers utilized questionnaires. For general PX, many employed the Game Experience Questionnaire (GEQ; IJsselsteijn et al., 2013; core and post-game modules, or specifically the positive experience modules therein); a few only used custom items (#24, #6). Emotions were assessed via the self-assessment manikin (SAM; Bradley and Lang, 1994) (#9, #10, #18). Approximately half of the empirical quantitative papers measured presence. This was assessed with one of three different questionnaires: Witmer et al. (2005)’s presence questionnaire (#1, #4); the Igroup Presence Questionnaire (IPQ; Schubert et al., 2001) (#16); and Slater-Usoh-Steed (SUS; Slater et al., 1994) (#9, #18) questionnaires. #16 employed the immersion subscale of the GEQ, yet otherwise none used dedicated immersion questionnaires. Only one paper used a measure for simulator sickness: the Simulator Sickness Questionnaire (SSQ; Kennedy et al., 1993) (#16). The same paper (#16) was also the only one to measure individual differences in participants’ inherent traits, such as their immersive tendencies (Immersive Tendencies Questionnaire, ITQ; Witmer and Singer, 1998), and their attitude toward loneliness (Preference for Solitude Scale; Nestler et al., 2011).

Surprisingly few papers used questionnaires to measure social factors of PX. While a variety of social experience questionnaires were in use, the majority of these were employed in a single paper (#16). The questionnaires used for this aspect of PX were: the GEQ’s social presence submodule (#1, #4, #18; and the submodule’s behavioral involvement subscale in #9), the Social Presence in Gaming Questionnaire (SPGQ; De Kort et al., 2007) (#16, #20) and the Cooperative Social Presence scale (CSP; Hudson and Cairns, 2014) (#16), the relatedness score within the Player Experience of Needs Satisfaction (PENS; Rigby and Ryan, 2007) questionnaire (#16), an adapted version of UCLA Loneliness Scale (Russell et al., 1978) (#16), and the Inclusion of Other in the Self Scale (Aron et al., 1992) (#10).

Qualitative studies largely employed interviews and video analysis to assess PX, although one paper used cooperative performance metrics (Seif El-Nasr et al., 2010) to analyze their observations (#20). Two papers employed some form of thematic analysis for the interview and video analysis (#10, #18). The qualitative papers largely did not clarify the type of thematic analysis used (e.g., codebook vs. reflexive; Braun and Clarke, 2020) nor did they provide a detailed description of the process they followed.

Overall, much of the empirical research reported in the reviewed papers was of an informal nature; only few consisted of formal experiments. Fourteen papers contained either user testing too informal to be included in the QA process, or technical descriptions of their system/implementation. This highlights the potential for more comprehensive empirical work in this subfield, and we look forward to future work that builds upon these early explorations.

4.2 Mechanics, Dynamics, and Aesthetics—Harris et al. (2016)

4.2.1 Mechanics of Asymmetry in Game Design

All papers explored an asymmetry of more than one mechanic specified by Harris et al. (2016); most commonly, this consisted of asymmetry of the interface (all papers), as well as ability/challenge, information, and responsibility. In terms of interface, the majority of papers (19 total) described games designed for two players, one of whom wore an HMD. The non-HMD players then usually interacted with the game via an interactive display medium (PC, smartphone, tablet device, projection, or Sifteo Cubes). In several cases, non-HMD player involvement was markedly subtle: they only viewed the virtual world (mostly via a monitor) without interaction, assisting the HMD player through verbal communication. Two papers employed additional props or devices: one game featured an upright visual screen projection plus a tangible robot avatar of the HMD player (#17). Both game variants in #9 offered the non-HMD playera monitor-attached hand-held controller, once with a lightsaber prop. In one game variant in #8, the non-HMD player(s) had no visual cues at all, and instead reacted only to auditory cues or based on the HMD player’s movement in the physical world (In their other game variant, all players had an HMD, but one of the player roles additionally held a nerf gun controller.). Further, in one notable example across two papers, the non-HMD player still interacts with the HMD while it is worn by the HMD player, by pressing on attached touchscreens (#18, #21).

Four papers explored game designs with more than two players. The breadth of variation here shows a wide array of options to explore: two HMD players while the non-HMD player views a projector; one HMD player while two non-HMD player use tablet devices; one HMD player, one co-located player in an AR environment, and a remote player with a tablet device. The gameful experience in #24 employed people “actuators”; they were instructed on how to carry and move the HMD player via smartphones attached to the HMD player.

Almost all papers featured games with differing ability assigned to specific players, making it a common type of asymmetry. For example, the HMD player is often tasked with navigating and interacting with the virtual environment directly, while the non-HMD player may be presented with a top-down view and perform a guiding or assisting task. In one artifact these roles were reversed: #23’s games put the HMD player in the guiding/assisting role, while the non-HMD players interacted with the game world (CAVE system) more directly. It was difficult to distinguish ability from challenge, as abilities are generally directly bound to specific challenges. We included this in our framework only when an ability was explicitly discussed in terms of challenge in the paper. Only two papers addressed challenge in their game design at all (#16, #24); none discussed challenge in terms of a difference between players.

In many papers, player abilities also involved the asymmetric distribution of information. It was not always clear which player had what kind of information. Commonly, however, the non-HMD player would have information that the HMD player relied upon to either complete their goal at all or to complete it faster. Generally, this was designed to give the non-HMD player something to do or to encourage communication. Only #15 featured an informational imbalance in both directions: “To ensure interdependence, players receive the instructions for the other player” (Smilovitch and Lachman, 2019).

Asymmetry was never discussed in terms of time investment required. However, in discussing the papers, we noted that this category could have been useful to clarify and classify degrees of involvement. For example, several papers featured game designs in which the HMD player is not entirely reliant on non-HMD player but can simply achieve their goal faster or easier with minor involvement or assistance from the non-HMD player.

In discussing the papers in terms of goal/responsibility, we found that the category may need to be refined in two ways. First, for the majority of collaborative games, players commonly had the same goal, but different responsibilities. For games with lower degrees of investment on behalf of the non-HMD player, the goal was difficult to define: is their goal merely to support the HMD player (different goals and different responsibilities), or is their shared goal to win the game (same goal and different responsibilities)? The category may therefore need to be split into two separate ones (goal vs. responsibility). Second, the category is difficult to apply to competitive games, as here players may be considered to have the same goal (i.e., winning) but in opposition to each other.

4.2.2 Dynamics of Asymmetry in Game Design

We report the dynamics designed or speculated to arise from asymmetry for directional dependence categories (mirrored, unidirectional, and bidirectional dependence), and synchronicity and timing between player actions (Harris et al., 2016). For mirrored dependence, the nature of each player’s reliance on the other(s) is the same (Harris et al., 2016). While the nature of reliance was sometimes difficult to discern, we observed this kind of dependence in only three papers (#1, #15, #21). For example, both non-HMDand HMD player need to work together by performing slicing actions to cut fruits to win (Gugenheimer et al., 2017b)’s collaborative game variant (#21). For #1, only the first-person view point variant of the game provides mirrored dependence. #15 constitutes a stronger mirrored interdependence: here instructions for each player are conveyed to the co-player; they rely on each other to pass on the information. For unidirectional dependence, we identified four papers (#5, #14, #19, #24). In general, these contained games that could be completed without the help of other players. #19 featured a maze game: the HMD player completed it faster with the non-HMD player’s guidance, however, the outcome of the game was achievable without this help.

The majority of identified papers (n = 15) featured bi-directional dependence; players relied on (the) other player(s) but in different ways. The degree of this type of dependence between players varied, placing one player in a more active role or less reliant role. However, we note that in many cases it was not clear what the non-HMD player’s goal was, making it difficult to determine the nature of reliance between the players: for example, in a maze game navigated by the HMD player, but guided by the non-HMD player, does the non-HMD player rely on the HMD player to physically navigate the virtual world in their stead? This was mainly due to the role definition of the non-HMD player; we often had difficulty determining whether this player was a facilitator or an essential element of the game.

Four papers (#3, #4, #9, #25) featured competitive games, which proved difficult to code as a category of directional dependence, as here players generally do not rely on each other as in the collaborative games—except in the sense that a positive gameplay experience requires a certain degree of motivation and effort from the other players.

Synchronicity and timing was also difficult to apply, as we were discussing and coding entire games, as opposed to specific game mechanics or segments. We observed multiple categories of synchronicity and timing for different player actions in the games. Further, these player actions were often not clearly described in terms of temporal occurrence or dependence, exacerbating the coding process. For instance, in one game (#6), one player had the role of navigator while the other held the role of the explorer (Serubugo et al., 2018b). Yet the game’s description did not clarify whether players performed their own respective actions regardless of the others’ (asynchronous timing) or whether the explorer requires information from the navigator to start to perform their action (sequential timing).

4.2.3 Emergent/Expected Aesthetics of Asymmetry in Game Design

Here we coded empirically reported PX (or speculations thereon), based on the aesthetics components proposed by Harris et al. (2016) (based on analytical work employing, but not identical to MDA aesthetics). Leadership and primacy, interdependence and necessity, and coordination were addressed most commonly by the papers. For leadership and primacy, this was often only a very implicit coding of the theme (e.g., based on a player role being described as “assistant” to another), but nevertheless some papers point toward how this kind of asymmetry was induced, e.g., through informational asymmetry (#2). A few papers explicitly highlighted trust as an important factor in asymmetric VR gameplay (#12, #21). Gugenheimer et al.’s works (#9, #18) especially touch on power and skill imbalances, as does Kerure and Freeman (2018) (#3) and Zhou et al. (2019) (#10). Gugenheimer et al. report that the difference in “power level” (e.g., through asymmetry in information, ability, and interface) drives enjoyment and need not necessarily be equally balanced, as players restrained themselves (although they also knew they would switch roles). Generally their players were aware of the non-HMD player’s “more dominant role,” which therefore required trust and a more “more responsible” playing style, given the potential for abuse of this power. Conversely, when an early iteration of #3’s game found an imbalance in favor of the non-HMD player due to an easier game mechanic (voice commands as input); they adjusted this to increase difficulty (voice commands based on pitch as input) and reduce this asymmetry. Zhou et al. (2019) also highlight a balance in responsibilities as a positive factor increasing enjoyment, with an appealing “give and take’ interaction style.”

The categories interdependence and necessity and coordination were closely related: many games fostered interdependence through game mechanics that required communication and coordination (e.g., one player waits for instructions from another to continue). #22 emphasized that mutual dependencies “force players either to collaborate or compete” and result in complex and unique PX. They also speculate player choice over their role increases involvement. Coordination was often only verbal in the reviewed games, or not explicitly described. #15 is a notable example for interdependence explicitly enforced through a communication constraint: at some point during the game, audio chat functionality is removed, and players have to develop their own communication system through gestures to succeed (Smilovitch and Lachman, 2019). #10is another interesting example: here, interdependence had a strong physical nature; “physical social contact” became a key coordination strategy to win the game (Zhou et al., 2019).

The other aspects in the framework were rarely addressed. #18 and #20 mention effects of player skill as contributing to blame attribution and frustration, respectively. The latter also mention this as a potential adaptation or customization factor in #20. Familiarity with the interface was barely discussed, even though most studies stem from a time when many participants likely were experiencing VR via HMDs for the first time. Similarly, although the most common motivation for asymmetric VR was including non-HMD players in the HMD player’s social experience, only few papers discussed familiarity with the partner as a factor in PX. #16 enforced gameplay between strangers by placing one of the examiners as the co-player, but did not report how this affected PX (Liszio et al., 2017). #24’s participants expressed greater acceptance to play with friends or family as opposed to the public. #18 report that couples displayed higher degrees of intimacy in interaction; it also affected players’ acceptance of physical contact (Gugenheimer et al., 2018b). Zhou et al. (2019) discuss this aspect in the most detail, describing the social connection between players as a strong factor in high social and affective experience ratings. It was reported to enhance relationships and support communication and synchronization.

4.3 MDA Aesthetics in Gameplay

We applied the MDA framework of aesthetics (Hunicke et al., 2004) to obtain more comprehensive information about PX in asymmetric VR, therefore coding study results for PX. Sensation and fellowship were addressed by the majority of papers, but PX relating to narrative, discovery, expression, and submission was not reported by any papers.

The sensation aspect of aesthetics was strongly present in the data, consisting of various PX factors (e.g., enjoyment, positive/negative affect, and engagement). Surprisingly, several papers did not collect any measure of fun or enjoyment (#12, #14, #17, #2, #21), but most papers did report some measure of multiplayer engagement for their participants. Several papers discuss which elements of the game provided or induced these positive experiences. A few of them linked players’ sense of game enjoyment to player abilities featured by game roles (#1, #4). For example, comparable levels of presence and social interaction were attributed to different view capabilities (#1, #4). The specifics of game design likely strongly influence PX results; for example, Gugenheimer et al. (2017a) reported higher valence scores for their ShareVR (#9) system, which fosters explicitly shared physical space with a projection space and controller-mounted props compared to a gamepad-and-TV condition. Similarly, Zhou et al. (2019) (#10) attributed the high enjoyment for their HMD player and non-HMD player to the richness of the highly embodied, physical interaction. Interestingly, they point out some enjoyment was derived from the peripheral interaction with spectators, who also reported high enjoyment and engagement (“It was really funny […] I was screaming for you to get the one on the right”; Zhou et al. (2019)). Further, Liszio et al. (2017) indicated PX improved due to playing with a human co-player instead of an agent/bot co-player (#16).

Unsurprisingly, fellowship was a commonly coded aesthetic of asymmetric game play; here we coded aspects relating to social interaction, relatedness, or perceived loneliness. In terms of questionnaires, interviews, and behavioral observations, the papers here indicate that asymmetry successfully involved non-HMD players and created interesting and positive social experiences between the players (e.g., “by providing various roles to the non-HMD users and accordingly expanding their interactions, the social relationship with the HMD users increased”; Jeong et al., 2020). Cheng et al. (2014) suggest that existing social bonds could be leveraged to improve player satisfaction: playing with family and friends as opposed to strangers (#24). Yet many of the reviewed papers do not discuss whether or how well participants knew each other.

In #9, this positive social experience was shown to rate “significantly more socially engaging” (Gugenheimer et al., 2017a) compared to a less physically enmeshed asymmetric setup. A few papers report that the social interaction reported between players was enabled through physical social proximity (#10, #9, #13, #18, #21). In particular, Zhou et al. (2019) highlight social touch, increasing interpersonal relationships, and enabled social affordances between the HMD player, the non-HMD player, and spectators as key contributions to PX (#10). Similarly, Gugenheimer et al. (2018b) report emergent positive social interaction between both players via enabling close proximity (#18) and players’ self-regulation of imbalances in perceived “power level.” Their collaborative game led to higher empathy and less negative feelings compared to their competitive one; perhaps due to a reliance on the higher-power player (non-HMD player) to self-regulate and avoid abusing their power, which could negatively affect PX. The paper comparing an asymmetric VR game with a human co-player to a version with a virtual agent and a single-player variant (#16) is particularly interesting for the fellowship lens. Here, players with an interactive human co-player reported significantly less loneliness than players whose human co-player did not interact with them, or than single-players.

As fantasy, we coded aspects like enjoyment of role-playing and immersion. Surprisingly, this was only reported once; Liszio et al. (2017) (#16) reported higher immersion for a single player game compared to game play with an agent and interactive co-player. Challenge (as aesthetics/during gameplay, as opposed to the games’ mechanics/dynamics) was addressed by four papers (#1, #16, #20, #25). For instance, two papers described both the HMD player and non-HMD player reporting similar degrees of challenge as a design goal; implicitly (#1) or explicitly (#20).

4.4 Social Asymmetry

The category of shared spaces emerged as a useful distinction. The large majority of games were designed as co-located, meaning players are located in the same room (A few do not address this directly, but can be inferred to be co-located from context or images.). Only very few address a remote game design (i.e., asymmetry of space): #5 has two co-located players, but the tablet player is remote. The games in #23 are not explicitly remote, but also not quite co-located; players are within hearing distance but “separated by distance and projection curtains” (Schmitz et al., 2015). While #8’s evaluation is co-located, it is described as capable of remote play—likely, most of the games are capable of or easily adaptable to remote play.

A further interesting distinction within co-located games emerged: some games encourage or enforce closely co-located games, inviting direct physical contact between players (#18, #21, some game variants in #9), #24, and #10). Herein, players were in close proximity to each; they touched the other player’s HMD, “hit” them with an inflatable prop, physically carried or held another player’s body, or danced together. This gameplay almost makes conventional co-located settings appear to demonstrate asymmetry of space, and resulted in engaging and dynamic social experiences. We term this co-located and explicitly shared physical space.

Differences in age or abilities (cognitive/physical) were not explored by any of the papers in this review. Nevertheless, we note that this is a gap within the field worth exploring, as we address in more detail in the discussion.

4.5 Shared Control

This part of the framework was based on Sykownik et al. (2017)’s patterns of shared control: distinct loci of manipulation vs. mutual locus of manipulation. Discerning the VR games’s locus of manipulation was often difficult when no game entity is controlled, complicating coding in more specific sub-categories.

Distinct loci of manipulation—both players control different entities in the game—was the most commonly observed category of shared control between players. The games generally featured a specific game element or entity that the players were responsible to control; for example, in #9’s Be My Light, the HMD player controls a sword to damage monsters, while the non-HMD player controls a light to locate them. We note that Sykownik et al. (2017) also mention a sub-type of distinct shared control wherein the loci of manipulation “establish a coherent entity” (e.g., one player controls a game entity’s legs for locomotion, while the other controls the same entity’s hands during object manipulation). This did not occur in the reviewed games.

Instead, we noted other patterns of distinct shared control not described by Sykownik et al. (2017): we noted several cases in which one player controls a game entity while the other player controls the game environment. We consider this a subtype of distinct loci of manipulation, but one that may hold greater asymmetry than when players control distinct game entities. For example, in Smilovitch and Lachman (2019)’s BirdQuestVR, the HMD player physically interacts as an entity in the spaceship environment, while one of the non-HMD players only controls parts of the ship’s systems through an interface on their tablet device, thereby affecting the overall game environment without controlling an in-game entity. Interestingly, there was only a single paper that did not feature in-game entity control for any players (#25).

In a new pattern of shared control that emerged in our review, only one of the players has a distinct loci of manipulation. While the other player may have additional information that the first does not, they do not directly control any game entity: they have no loci of manipulation. Instead, they generally provide only verbal guidance to the player capable of affecting the game world. In most papers that displayed this pattern of indirect control/none, the non-HMD player engaged in verbal communication to guide the HMD player (e.g., #6, #7, #19). Twice this was reversed: Schmitz et al. (2015)’s (#23) Coral Rift featured a non-HMD player within a CAVE system and an HMD player (VR). The HMD player observes the sea and warns the non-HMD players about obstacles, but only the non-HMD player actively balanced and navigated the ship. Similarly, Sajjadi et al. (2014)’s HMD player provided guidance to the non-HMD player who manipulated the actual game pieces. Alternatively, sometimes the non-HMD player’s control within the virtual world was indirect instead of none: for Sra et al. (2017), the non-HMD player could trigger a galvanic vestibular response to impact the HMD player’s navigation.

Mutual locus of manipulation (through control alternation or an input processing function) did not occur in the reviewed games. We note that it could be argued that a game like’s FaceDisplay might represent this pattern. In this game’s cooperative variant, both players use HMD-attached touch screens (front vs. side) to “slice” fruit projectiles. If one considers the locus of manipulation to be the virtual screen on which players slice to interact with the world, then this could be considered a mutual locus of manipulation pattern. However we argue that the players have separate loci of control wherever they slice on the virtual screen, thus creating distinct loci (even though they can at times overlap).

5 Discussion

We separate our discussion of the synthesis into implications of the systematic review for the field of asymmetric multiplayer VR games, and reflections on the framework and how it can benefit the VR research community. Takeaways from both aspects of the discussion are summarized in our overview in Table 3.

5.1 Implications and Opportunities for Asymmetric VR Games

The review showcases a vibrant emerging research field, yet also traces several gaps in the literature that represent opportunities for future research.

5.1.1 Commonalities

The classification based on Harris et al. (2016)’s mechanics and dynamics dimensions indicates that most of the existing asymmetric VR games consist of two players, one of which is the HMD player, and provide information to an non-HMD player through a secondary screen of some sort. In addition to asymmetry of interface, asymmetries of ability/challenge, information, and responsibility are most common. Player roles are frequently designed to contain bidirectional interdependence and generally expected to affect leadership, interdependence and coordination between players (see Table 4).

5.1.2 Research Gaps

Alternative interfaces [e.g., physiological input (Karaosmanoglu et al., 2021)] are rare, as are games that accommodate more than two players (although the few artifacts in this space are promising). Gaps in the literature are visible in terms of effects of player skill, familiarity with the interface, and familiarity with the co-player (s) (see Table 4). A lot of valuable research in games and HCI could inform these factors in design considerations, expectations for PX, and study design implications, for example, attribution theory (“how people assign causes to effects”; Depping and Mandryk, 2017), novelty effects (Wells et al., 2010), and pre-existing relationships in social gameplay (Eklund, 2015; Perry et al., 2018). Differences in age and abilities also represent a large gap in the literature, even though asymmetries of age are of course already being explored in non-VR games: there is a long history of inter-generational gameplay (Othlinghaus et al., 2011; Voida and Greenberg, 2012; Osmanovic and Pecchioni, 2016). Increasingly, asymmetries in terms of cognitive or physical ability are also being explored in games research (Cairns et al., 2019). For example, Gerling et al. (2014) investigated the use of game balancing approaches to accommodate players with different physical abilities (e.g., players with or without a wheelchair), to avoid reducing self-esteem and relatedness. A more recent example by Graf et al. (2019) featured a projection-based AR exergame, which enabled children with and without a wheelchair to play together. We emphasize that the context of VR may hold additional challenges for asymmetries in player age or ability (e.g., to design suitable inter-dependencies between players with dementia and their relatives), or make certain constellations entirely inappropriate (e.g., the HMD may constitute a safety hazard for people with high injury risk). However, perhaps the immersive and socially engaging qualities of asymmetric VR can be transferred to games with such asymmetries while still addressing these challenges.

We also point out that the focus on immersive and social qualities of asymmetric VR that is prevalent in the corpus (and thus also this review) may in itself represent a research gap. Player behavior and performance were rarely explored in detail in the papers we surveyed. Performance is likely disregarded in favor of focusing on aspects more closely aligned with common goals of asymmetric mechanics: increased communication/coordination and social connection. However, at the line of distinction between competitive and collaborative games, performance may become a more relevant factor. Player behavior was largely discussed within the context of socially motivated behavior in our corpus, yet other aspects of player behavior may also be interesting (and also showcase differences between competitive and collaborative game designs). For example, the use of F formations (Ciolek, 1983; Kendon, 2010) may serve to inform explorations of player behavior in terms of orientation [see Marquardt et al. (2012) for an exploration of how proxemics apply to people and devices in cross-device contexts].

Furthermore, we note that physicality and explicitly shared physical spaces are a particularly interesting dynamic in asymmetric VR games. This was particularly noticeable in FaceDisplay (#21; Gugenheimer et al., 2017b) and AstaireVR (#10; Zhou et al., 2019). We assume that this is a valuable factor for future research; not only do players have a need for social interaction and relatedness (Ryan et al., 2006), there is also strong evidence for an appreciation of or need for embodied interaction and tangible as well as kinaesthetic experiences (Hall, 1966; Dourish, 2004; Hornecker, 2005; Kim and Schneider, 2020). Prior work has discussed phenomena such as gestural excess (Simon, 2009; Harper and Mentis, 2013) and embarrassing or uncomfortable interactions (Benford et al., 2012; Deterding et al., 2015) as a valuable tool and affordance within (body-focused) game design. This is likely also part of the appeal of asymmetric VR games, yet VR users are also inherently (technologically and immersively) isolated from the real world to a degree (Boland and McGill, 2015; Mütterlein and Hess, 2017). Benford et al. (2012) in particular described isolation and giving up control to other people as examples for uncomfortable interactions in HCI—without negative connotation, necessarily. We argue that their research is (implicitly) built upon in asymmetric VR games research. While discomfort is often a negative aspect of user experience, it can be less so in gameful contexts (Bopp et al., 2016). Yet we also point out that Benford et al. (2012) have discussed this kind of design in terms of ethical considerations, suggesting that issues of justification, informed consent, risk management, as well as rights to withdrawal, privacy, and anonymity require special focus in this context. Given the rather tangential and cursory engagement with prior theoretical work displayed by many of the reviewed papers (which we address in more detail below), we find it particularly important to highlight this connection.

5.1.3 Methodology

Conducting this review revealed that a lot of the games and systems were only roughly described (e.g., missing information, information scattered throughout the sections). We suggest that our post-hoc framework, and in particular the dimensions based on Harris et al. (2016)’s mechanics and dynamics, can scaffold game design reporting in a way that would make it easier to understand the design and resulting PX. For example, explicitly addressing players’ different goals would make classification and comparison of games a lot easier. It would also increase our understanding of the results in terms of potential effects of the framing with which each role was presented (Further, it may also be worth assessing how players actually understand their role, as this could differ.). This could also prove useful partially for the reporting of non-game mixed-reality applications.

With regards to measurement, our review shows that there were surprisingly few quantitative psychometrics for enjoyment: this consisted largely of the GEQ (or a submodule thereof) and SAM, as well as some interview questions. The GEQ’s factor structure has been called into question in recent years (Johnson et al., 2018; Law et al., 2018), making this a potentially problematic choice depending on the submodule used (we suggest using alternative questionnaires such as the Player Experience Inventory by Abeele et al., 2020 or The Player Experience of Need Satisfaction by Ryan et al., 2006). We further propose that a wider range of participants is worth exploring: in age and abilities but also in terms of pre-existing social relationships and gender (Burtscher and Spiel, 2020). Especially when considering interdependency relating to power imbalances, isolation, and embarrassing interactions, a broader range of demographics would be highly beneficial. It is also surprising how few papers reported measures for players’ perception of their social relationship with each other.

Existing questionnaires may not be well suited to investigate many aspects of PX that are prevalent in asymmetric VR games (e.g., experienced leadership). In light of this, we find it surprising that only five papers in our review employed qualitative methods, which may be more flexible in this regard. When qualititative methods were applied, their reporting could also be improved (e.g., what specific kind of thematic analysis was used; Braun and Clarke, 2020) and would increase methodological rigor in this research field.

5.1.4 Connection to Theory

Our review indicates that a lot of research on asymmetric VR games does not deeply engage with existing theoretical work on asymmetry in games. For example, although several papers cite Harris et al. (2016)’s work, very few actively incorporate aspects thereof in their design. Reeves et al. (2005)’s taxonomy for the design of spectator experiences of public interfaces is similarly relevant but was addressed by only a single paper. The taxonomy classifies whether users’ manipulations and/or the effects of their manipulation are visible to or hidden from spectators (ranging from “secretive” to “expressive”; Reeves et al., 2005). Other recent work on users’ experience of performing interaction in front of other people (Martínez-Ruiz et al., 2019) and a general spectrum of the degree of interactivity provided to an audience (Striner et al., 2019) may be useful for future research in this area as well. This tenuous theoretical foundation is likely a side effect of the laudably innovative nature of the designed VR artifacts and the youth of the field. Yet a closer connection to the theory in terms of asymmetric game design and social factors in digital games—especially at this early stage—would allow this field to grow more systematically and gain better coverage across relevant dimensions in game design and aesthetic PX.

5.2 Reflections on the Framework

Overall, the “best fit” a priori framework was an extensively useful guiding lens for our synthesis of asymmetric VR games, giving us both a high-level overview of the field’s thematic focus points, and dimensions along which to more deeply consider results and implementations explored so far. However it also showcased ways in which the a priori framework can be improved: we detail these aspects here, and provide the suggested posthoc framework in Figure 3.

FIGURE 3

FIGURE 3. Suggested post-hoc “best fit” framework, with dashed lines and more transparent color for categories that are under-represented in existing asymmetric multiplayer VR games research so far.

Dimensions of Harris et al. (2016)’s conceptual framework for the design of asymmetric games were largely applicable to the reviewed asymmetric VR games, allowing systematic descriptions and categorisations of the different kinds of asymmetry in the reviewed games. However, as we pointed out in the results, some categories proved difficult to apply. In mechanics of asymmetry, we found it difficult to distinguish between ability and challenge; generally challenge was not explicitly addressed at all, and in our discussions we saw this as so closely tied to differences in ability that a distinction may not be necessary. However we note that asymmetry of interface can sometimes result in asymmetry of challenge—even in cases when abilities remain the same (e.g., targeting abilities may be easier when controlled by head tracking than via mouse, even though the ability). In those cases, asymmetry of ability and challenge might have to be separate again.

The category of goal/responsibility yielded many interesting observations. First, it was often difficult to determine players’ goals. This difficulty partially resulted from game descriptions—which were often unclear or across several sections—but was especially prominent for competitive games. In competitive games, it is more difficult to determine individual players’ goals, because these can be subject to interpretation. In a collaborative game, players seek to achieve a shared goal. In competitive games, opponents’ goals can be framed as the same or shared as well (e.g., “win X points”). But it could also be seen as either asymmetric, or symmetric but opposed (player A: “win X points more than B”; player B: “win X points more than A”). Second, as described in more detail above, we found that in many cases players had the same goal, but different responsibilities. We thus suggest that this category of goal/responsibility may need to be split up in two. Further, its use should be carefully considered for competitive asymmetric games, because it may not be directly applicable. An additional way in which competitive games were different relates to dependence: in competing against one another, opponents generally do not depend on one another. Nevertheless, there may be cases of players embracing a voluntary or emergent dependence that could be considered a kind of mirrored dependence: For example, in a competitive racing game, players might rely on other players’ unspoken adherence to not purposefully causing collisions [what Salen and Zimmerman (2004) refer to as the implicit rules of the game that will cause a game to function beyond its operational and constituative rules]. However such voluntary dependence is easily circumvented by accident (e.g., due to lack of skill) or wilful noncompliance (e.g., purposeful crashing).

For dynamics, directional dependence was suitable for defining interdependencies between players, but clarifications of the dependence degree can avoid confusion between unidirectional and bidirectional dependence. While some papers displayed a mirrored dependence (#1, #15, #21), it was hard to say that “the nature of each player’s reliance on each other is identical” (Harris et al., 2016) because of their interface asymmetry. Further, synchronicity and timing was challenging to apply to games as a whole; even determining which types of synchronicity occur over the course of the game was often arduous because of unclear descriptions in the reviewed game designs. Overall, we thus suggest to merge ability and challenge, to split goal and responsibility, and to eliminate the synchronicity category in future iterations of this framework, as displayed in Figure 3.

Regarding dependence, we note that this classification is the subject to the researchers’ understanding of dependence. This was also shaped by many sessions of discussions, and re-coding upon assessing new items in the corpus. This understanding might have to be adjusted further in the future. A more formalized definition of dependence (and subtypes thereof) would be useful for future work. In empirical work, assessment tools like the perceived behavioral and affective interdependence subscales in Networked Minds Social Scale (Harms and Biocca, 2004) might be of use. We further also refer to prior work that has categorized degrees of interdependence in collaborative settings [e.g., tightly vs. loosely coupled, as also touched upon by Harris and Hancock (2019)]. For example, Sigitov et al. (2019) have observed different user roles for tightly coupled collaborative work when interacting with a shared large display using mobile phones for interaction purposes. Similarly, different types of interdependence have been addressed for shared PC monitor (Tse et al., 2004), and co-located digital tabletops (Scott et al., 2003; Morris et al., 2004). How well such degrees of interdependence (designed or observed) can be applied to asymmetric VR settings remains to be explored in the future, but we note the long history of research on trade-offs between mechanics for individual system use vs. for collaborative system use (Gutwin and Greenberg, 1998).

The MDA aesthetics part of the framework worked well for sensation, fantasy, and fellowship; however, the other aspects like narrative, challenge, or discovery were not addressed by the papers. As we do believe these have potential application in asymmetric VR, we suggest to keep this in the framework for future use. However, we also suggest that more nuanced alternatives to address different or more specific kinds of engagement, immersion, or social experience may be a useful addition or replacement in the future.

For social asymmetry, shared spaces were the most addressed category. Based on our review, we suggest that this category would be particularly useful if further distinguished as co-located, remote, and co-located and explicitly shared physical space. These different shared space setups fostered different kinds of highly engaging asymmetries of social and physical interaction, pointing toward new potential categories for a future iteration of the framework. Further, while age or abilities were not explored as asymmetry by any of the reviewed games, we acknowledge that these may exist as papers that do not use the keywords we employed in our review (see limitations, below). As discussed, asymmetries in these categories could constitute a valuable design space to engage underexplored (combinations of) demographics, such as young children and their parents, or caregivers/physical therapists and older adults with cognitive impairments, or users with varying visual abilities (Gonçalves et al., 2021).

Kaye (2016)’s theoretical factors for social group play were surprisingly under-represented in our review, barring communication—which itself was reported, but often not clearly. We do believe that—with clearer descriptions of expected and observed player communication, this could be a useful part of the framework for future reviews, as well as research and design work. However to fully embrace the social factors involved in asymmetric and interdependent VR games, this part of the framework may need expanding or partially replacing by more detailed theories of social communication in games. We note that while there is research on how players communicate in games (Klimmt and Hartmann, 2008; Walther et al., 2015; Leavitt et al., 2016), much less is known about communication between players in VR games (Rubio-Tamayo et al., 2017). Yet asymmetry and resulting interdependency are likely to heighten experiences of group or team flow (Borderie and Michinov, 2017) in VR, as well.

The shared control patterns were sometimes challenging to apply in our identified papers. We found it difficult to determine what constituted a loci of manipulation, especially when there was no distinct game entity through which the player acted (e.g., one player controls the game environment, or when there is no visible game entity representation). One pattern that emerged saw some players without control over the game world: neither distinct loci, nor mutual loci, but a player with no orindirect loci of control (e.g., only providing verbal support, or impacting the HMD player’s virtual-world navigation through real-world actuation). Comparing this kind of non-HMD player experience to a more active one may be an interesting starting point for future work. The other pattern that emerged, giving (usually) the non-HMD player control over the game environment, is one way to create a more active role. We thus suggest a distinction between players having direct control over distinct loci of control (either entities or the environment), sharing locus of control, or having only indirect or no control (Figure 3).

In current asymmetric games, distinct loci of manipulation are more prevalent, yet mutual locus of manipulation could reveal interesting social dynamics through highly interdependent games.

Based on our findings resulting from the application of the a priori framework, we believe that it is largely well suited for this field, albeit with the adjustments described in this section. We therefore suggest a post-hoc “best fit” framework in Figure 3 that incorporates these changes. We believe that future systematic reviews will be able to re-use and build upon this framework as the field of research and designed artifacts grows. Further, we highlight those categories that are strong potential factors for PX in asymmetric VR games yet are thus far underexplored (indicated via dashed lines and greater transparency in Figure 3).

In Tables 4,5, we illustrate which papers in our corpus apply the framework’s specific categories—based on the post-hoc framework—to also highlight existing gaps in the research. The research gaps evident in Tables 4,5 complement the takeaways of this work described in Table 3. We note that our framework claims no completeness with regards to research gaps: Other aspects beyond those uncovered by our framework and review may need to be added in the future. For example, the papers in our corpus rarely explored performance as a metric. We assume that this is because performance is rarely the goal for including asymmetries in game design, although that does not mean it could not be used for it.

Regardless of the focus of future work designed for this field, the post-hoc framework provides a lens through which to more clearly describe and design future asymmetric VR games, more clearly delineate desired, expected, and observed results for PX, and offers dimensions for comparison. We further refer readers to related work by Márquez Segura et al. (2021) for their suggested lenses of designed, expected, and observed to describe and analyze player behavior; this could be an interesting addition to our framework in terms of analytical application and methodology.

Finally, we note that adjacent research areas of co-located collaborative systems may hold interesting relevant findings for (collaborative) asymmetric VR games, and thus for future extensions to the framework. We here refer the reader to Brudy et al. (2019)’s taxonomy of cross-device designs and Olin et al. (2020)’s design considerations for cross-device collaboration that include VR. Pinelle et al. (2003)’s mechanics of collaboration framework may be another option for extending our framework. Further, Ouverson and Gilbert (2021) very recently published similar research to our own, positing a framework for asymmetric VR—which they define more narrowly as only asymmetry of interface, opposed to the broader asymmetric VR design space covered in our framework. We believe that our framework can stand concurrently to their five dimensions of asymmetry in VR (spatial co-presence, transportation, informational richness, team interdependence, and balance of power) in future practice. Moreover, given that our chosen methodology has a strong theoretical grounding in multiple theories, our own posthoc framework and our findings with regards to existing research gaps (illustrated in Tables 3–5, as well as Figure 3) may be able to augment their framework. While our work is derived from the context of games, some aspects may also transfer well to the non-game space, and thus enrich the application of both frameworks in parallel. Future work will have to explore this in more detail.

5.2.1 Limitations

As all methodologies, the “best fit” strategy in framework synthesis has limitations. A framework carefully created through comprehensive inductive methods would be preferable to a patchwork-style framework. However, there is no definitive theoretical framework for asymmetric multiplayer VR yet. Perhaps this work can provide a stepping stone toward this. The youth of the field is precisely why this systematic review—using “best fit” framework synthesis—is important at this time.

Further limitations must be acknowledged in terms of review execution. Relevant work that uses different vocabulary could have been missed. As the field (hopefully) moves toward more consistent terminology, this factor should be reduced in future reviews. Yet our added snowball approach mitigates this limitation. Further, although two authors conducted the synthesis together in close communication and over many extensive discussions, bias is still likely. Following recommendations for qualitative methods, we provide a statement on reflexivity (Newton et al., 2012; Berger, 2015): the authors conducting the synthesis have a Computer Science and Cognitive Systems background, respectively, and both have previously conducted VR and VR games research.

We also note that our search query no longer results in the same number of documents as it did at the time of initial data collection. The ACM digital library presents irregularities in terms of its database query results (with lower numbers for our query over time). While we are in correspondence with ACM and the company that built their search engine, at this stage it is unclear why this is the case. However, the initial search resulting in the corpus of this review yielded the highest number of results in our (re-)sampling over the past year. This suggests that our analysis simply screened a larger pool of publications. Yet we point out that even if a few papers were missed due to using different terms, this would not limit the validity of the framework and the review’s synthesis.

Our review is based on 25 papers, relating 30 VR artifacts (plus design variants) and 17 studies of sort (plus reports of informal playtesting). This of course only provides a glimpse into the potential overall design space, so while we did speculate to a degree based on non-VR research (e.g., suggesting the design of asymmetry in age or abilities), we cannot claim completeness, only a first step toward it. In future work, we will explore commercial asymmetric VR games as well. Additionally, limitations of our thematic findings are bound to the limitations of the reviewed papers—for example, as also found in general VR research (Peck et al., 2020; MacArthur et al., 2021), this subfield also has predominantly investigated PX for male participants, which may have introduced bias into results.

6 Conclusion

This paper presents the first literature review on how asymmetric game design is thus far being leveraged in multiplayer VR. Based on “best fit” framework synthesis, we draw on existing theoretical games research (within and outside of VR research) to gain insight into the state of the field, identify opportunities for more complete coverage of the design space, and point out where the field can improve in methodological rigor. Our final suggestions for a post-hoc framework can also be used by future papers to describe asymmetric VR games, as well extended in future systematic reviews as the field grows. The results showcase a novel field with great promise in including non-HMD players, facilitating multiplayer engagement, minimizing VR isolation, and providing room to research a complex range of social dynamics. We hope this review can both spark discussion and orient the field in the journey toward achieving these aims.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.

Author Contributions

KR and SK contributed to the development of the protocol, including search strategy, selection criteria, risk of bias assessment strategy and data extraction criteria. LN provided feedback on this process. KR conducted the search and removal of duplicates; KR and SK conducted the screening phases; DW acted as third reviewer (tie-breaker). KR and SK developed the a priori framework and applied the framework synthesis methodology. KR led the drafting of the manuscript, in collaboration with SK; LN and FS assisted with phrasing and framing. All authors read, provided feedback and approved the final manuscript.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank our reviewers, and are especially glad for the thoughtful examples of asymmetry that improved this manuscript.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2021.694660/full#supplementary-material

Footnotes

¹We use the term artifact to refer to games, prototypes, or technical systems.

²For detailed information on Scopus’s indexed resources (e.g., Institute of Electrical and Electronics Engineers; IEEE), please refer to the website: https://www.scopus.com/sources and the Scopus Content Coverage Guide: https://www.elsevier.com/?a=69451.

³Based on title regardless of capitalization and year, as databases were not consistent in their use of DOI, ID, and URL fields.

⁴Document type Conference Review. Not based on DOI because databases are not consistent in their use of DOI, ID, and URL fields.

⁵Twelve papers were initially flagged as undecided based on only the Abstract and Title. For these papers, we accessed the papers’ full text to make an informed decision, yielding their exclusion.

⁶We have been in touch with the ACM digital library team about such irregularities but have so far not received a definitive answer.

⁷Streck et al. (2019b) was identified as the reprinted version of a paper already in our list Streck et al. (2019a) and Woodworth et al. (2017) was identified as not game related. We further note that #24 was almost excluded because although it describes a game, it does not seem to consider any other involved people except the HMD playeras “players.” Nevertheless, the non-HMD “actuators” are incentivized to help the HMD player, they have a goal, a physical challenge, and they reportedly enjoyed it; we thus kept this paper included as a gameful multiplayer experience.

⁸Items six and seven in Kmet et al. (2004)’s quantitative checklist.

⁹1) “How relevant is the particular focus of the study (including conceptual focus, context, sample and measures) [, paper, or system description] for addressing the question or sub-questions of this review?” 2) “To what extent can the study findings [or paper contributions] be trusted in answering the study question(s) [or paper objective(s)]?”

¹⁰The final framework coding is presented in Supplementary Materials.

References

Abeele, V. V., Spiel, K., Nacke, L., Johnson, D., and Gerling, K. (2020). Development and Validation of the Player Experience Inventory: A Scale to Measure Player Experiences at the Level of Functional and Psychosocial Consequences. Int. J. Human-Computer Stud. 135, 102370. doi:10.1016/j.ijhcs.2019.102370

CrossRef Full Text | Google Scholar

Abuhamdeh, S., Csikszentmihalyi, M., and Jalal, B. (2015). Enjoying the Possibility of Defeat: Outcome Uncertainty, Suspense, and Intrinsic Motivation. Motiv. Emot. 39, 1–10. doi:10.1007/s11031-014-9425-2

CrossRef Full Text | Google Scholar

Aromataris, E., and Munn, Z. (2020). JBI Manual for Evidence Synthesis. Available at: https://synthesismanual.jbi.global.doi:10.46658/JBIMES-20-01

CrossRef Full Text | Google Scholar

Aron, A., Aron, E. N., and Smollan, D. (1992). Inclusion of Other in the Self Scale and the Structure of Interpersonal Closeness. J. Personal. Soc. Psychol. 63, 596–612. doi:10.1037/0022-3514.63.4.596

CrossRef Full Text | Google Scholar

Benford, S., Greenhalgh, C., Giannachi, G., Walker, B., Marshall, J., and Rodden, T. (2012). “Uncomfortable Interactions,” in CHI ’12: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, May 5–10, 2012 (New York, NY, USA: Association for Computing Machinery), 2005–2014. doi:10.1145/2207676.2208347

CrossRef Full Text | Google Scholar

Berger, R. (2015). Now I See it, Now I Don't: Researcher's Position and Reflexivity in Qualitative Research. Qual. Res. 15, 219–234. doi:10.1177/1468794112468475

CrossRef Full Text | Google Scholar

Beznosyk, A., Quax, P., Lamotte, W., and Coninx, K. (2012). “The Effect of Closely-Coupled Interaction on Player Experience in Casual Games,” in 11th International Confernece on Entertainment Computing (ICEC), Bremen, Germany, September 2012 (Berlin/Heidelberg, Germany: Springer), 243–255. doi:10.1007/978-3-642-33542-6_21

CrossRef Full Text | Google Scholar

Boland, D., and McGill, M. (2015). Lost in the Rift. Xrds 22, 40–45. doi:10.1145/2810046

CrossRef Full Text | Google Scholar

Bopp, J. A., Mekler, E. D., and Opwis, K. (2016). “Negative Emotion, Positive Experience?,” in Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems, May 7–12, 2016 (New York, NY, USAAssociation for Computing Machinery), 2996–3006. doi:10.1145/2858036.2858227

CrossRef Full Text | Google Scholar

Borderie, J., and Michinov, N. (2017). Identifying Social Forms of Flow in Multiuser Video Games. New York: Routledge, 32–64. doi:10.4324/9781315629308-3

CrossRef Full Text

Bortolaso, C., Bourdiol, J., and Graham, T. C. N. (2019). “Enhancing Communication and Awareness in Asymmetric Games,” in Joint International Conference on Entertainment Computing and Serious Games, Arequipa, Peru, November 11–15, 2019 (Cham, Switzerland: Springer), 250–262. doi:10.1007/978-3-030-34644-7_20

CrossRef Full Text | Google Scholar

Bradley, M. M., and Lang, P. J. (1994). Measuring Emotion: the Self-Assessment Manikin and the Semantic Differential. J. Behav. Ther. Exp. Psychiatry 25, 49–59. doi:10.1016/0005-7916(94)90063-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2020). One Size Fits All? what Counts as Quality Practice in (Reflexive) Thematic Analysis? Qual. Res. Psychol. 1, 1–25. doi:10.1080/14780887.2020.1769238

CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using Thematic Analysis in Psychology. Qual. Res. Psychol. 3, 77–101. doi:10.1191/1478088706qp063oa

CrossRef Full Text | Google Scholar

Brudy, F., Holz, C., Rädle, R., Wu, C.-J., Houben, S., Klokmose, C. N., et al. (2019). Cross-Device Taxonomy: Survey, Opportunities and Challenges of Interactions Spanning across Multiple Devices. New York, NY, USA: Association for Computing Machinery, 1–28.

Burtscher, S., and Spiel, K. (2020). “But where Would I Even Start?,” in MuC ’20: Mensch und Computer 2020 — Tagungsband (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3404983.3405510

CrossRef Full Text | Google Scholar

Buscemi, N., Hartling, L., Vandermeer, B., Tjosvold, L., and Klassen, T. P. (2006). Single Data Extraction Generated More Errors Than Double Data Extraction in Systematic Reviews. J. Clin. Epidemiol. 59, 697–703. doi:10.1016/j.jclinepi.2005.11.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Cairns, P., Power, C., Barlet, M., Haynes, G., Kaufman, C., and Beeston, J. (2019). Enabled Players: The Value of Accessible Digital Games. Games Cult. 16, 262–282. doi:10.1177/1555412019893877

CrossRef Full Text | Google Scholar

Carroll, C., Booth, A., and Cooper, K. (2011). A Worked Example of "best Fit" Framework Synthesis: A Systematic Review of Views Concerning the Taking of Some Potential Chemopreventive Agents. BMC Med. Res. Methodol. 11, 1–9. doi:10.1186/1471-2288-11-29

PubMed Abstract | CrossRef Full Text | Google Scholar

Carroll, C., Booth, A., Leaviss, J., and Rick, J. (2013). “best Fit” Framework Synthesis: Refining the Method. BMC Med. Res. Methodol. 13, 37. doi:10.1186/1471-2288-13-37

PubMed Abstract | CrossRef Full Text | Google Scholar

Cechanowicz, J. E., Gutwin, C., Bateman, S., Mandryk, R., and Stavness, I. (2014). “Improving Player Balancing in Racing Games,” in CHI PLAY ’14: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play, October 19–21, 2014 (New York, NY, USA: Association for Computing Machinery), 47–56. doi:10.1145/2658537.2658701

CrossRef Full Text | Google Scholar

Cheng, L.-P., Lühne, P., Lopes, P., Sterz, C., and Baudisch, P. (2014). “Haptic Turk,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 3463–3472. doi:10.1145/2556288.2557101

CrossRef Full Text | Google Scholar

Ciolek, T. M. (1983). The Proxemics Lexicon: A First Approximation. J. Nonverbal Behav. 8, 55–79. doi:10.1007/bf00986330

CrossRef Full Text | Google Scholar

Coleman, B. (2009). Using Sensor Inputs to Affect Virtual and Real Environments. IEEE Pervasive Comput. 8, 16–23. doi:10.1109/MPRV.2009.60

CrossRef Full Text | Google Scholar

Connolly, T. M., Boyle, E. A., MacArthur, E., Hainey, T., and Boyle, J. M. (2012). A Systematic Literature Review of Empirical Evidence on Computer Games and Serious Games. Comput. Educ. 59, 661–686. doi:10.1016/j.compedu.2012.03.004

CrossRef Full Text | Google Scholar

De Kort, Y. A., IJsselsteijn, W. A., and Poels, K. (2007). “Digital Games as Social Presence Technology: Development of the Social Presence in Gaming Questionnaire (SPGQ),” in PRESENCE 2007. Vol. 195203 (Stockholm, Sweden: Starlab), 195–203.

Google Scholar

Deci, E. L., and Ryan, R. M. (2011). Self-determination Theory. Handbook theories Soc. Psychol. 1, 416–437. doi:10.4135/9781446249215.n21

CrossRef Full Text | Google Scholar

Depping, A. E., Mandryk, R. L., Johanson, C., Bowey, J. T., and Thomson, S. C. (2016). “Trust Me,” in CHI PLAY ’16: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, October 16–19, 2016 (New York, NY, USA: Association for Computing Machinery), 116–129. doi:10.1145/2967934.2968097

CrossRef Full Text | Google Scholar

Depping, A. E., and Mandryk, R. L. (2017). “Why Is This Happening to Me?,” in CHI ’17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, May 6–11, 2017 (New York, NY, USA: Association for Computing Machinery), 1040–1052. doi:10.1145/3025453.3025648

CrossRef Full Text | Google Scholar

Deterding, S., Lucero, A., Holopainen, J., Min, C., Cheok, A., Waern, A., and Walz, S. (2015). “Embarrassing Interactions,” in CHI EA ’15: Proceedings of the 33rd Annual ACM Conference Extended Abstracts on Human Factors in Computing Systems, April 18–23, 2015 (New York, NY, USA: Association for Computing Machinery), 2365–2368. doi:10.1145/2702613.2702647

CrossRef Full Text | Google Scholar

Dixon-Woods, M. (2011). Using Framework-Based Synthesis for Conducting Reviews of Qualitative Studies. BMC Med. 9, 39. doi:10.1186/1741-7015-9-39

PubMed Abstract | CrossRef Full Text | Google Scholar

Dourish, P. (2004). Where the Action Is: The Foundations of Embodied Interaction. Cambridge, MA, USA: MIT press.

Efstratios, G., Michael, T., Stephanie, B., Athanasios, L., Paul, Z., and George, P. (2018). “New Cross/augmented Reality Experiences for the Virtual Museums of the Future,” in Digital Heritage. Progress in Cultural Heritage: Documentation, Preservation, and Protection, Nicosia, Cyprus, October 29–November 3, 2018 (Cham, Switzerland: Springer International Publishing), 518–527. doi:10.1007/978-3-030-01762-0_45

CrossRef Full Text | Google Scholar

Eklund, L. (2015). Playing Video Games Together with Others: Differences in Gaming with Family, Friends and Strangers. journal gaming virtual worlds 7, 259–277. doi:10.1386/jgvw.7.3.259_1

CrossRef Full Text | Google Scholar

Fitterer, D. (2016). Audioshield. Vashon, WA, United States: Game [VR, PC]. Audiosurf LLC. https://store.steampowered.com/app/412740/Audioshield/.

Furukawa, T., Yamamoto, D., Sugawa, M., Peiris, R., and Minamizawa, K. (2019). “TeleSight.” in SIGGRAPH ’19: ACM SIGGRAPH 2019 Emerging Technologies, July 28–August 1, 2019 (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3305367.3335040

CrossRef Full Text | Google Scholar

Gamer, M., Lemon, J., Gamer, M. M., Robinson, A., and Kendall’s, W. (2012). Package ‘irr’. Various Coefficients of Interrater Reliability and Agreement. https://cran.r-project.org/web/packages/irr/irr.pdf.

George, C., Janssen, P., Heuss, D., and Alt, F. (2019). “Should I Interrupt or Not?,” in DIS ’19: Proceedings of the 2019 on Designing Interactive Systems Conference, June 23–28, 2019 (New York, NY, USA:Association for Computing Machinery), 497–510. doi:10.1145/3322276.3322363

CrossRef Full Text | Google Scholar

Gerling, K. M., Miller, M., Mandryk, R. L., Birk, M. V., and Smeddinck, J. D. (2014). Effects of Balancing for Physical Abilities on Player Performance, Experience and Self-Esteem in Exergames. New York, NY, USA: Association for Computing Machinery, 2201–2210. doi:10.1145/2556288.2556963

CrossRef Full Text

Gonçalves, D., Rodrigues, A., Richardson, M. L., de Sousa, A. A., Proulx, M. J., and Guerreiro, T. (2021). “Exploring Asymmetric Roles in Mixed-Ability Gaming,” in Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY: Association for Computing Machinery). doi:10.1145/3411764.3445494

Google Scholar

Graf, R., Benawri, P., Whitesall, A. E., Carichner, D., Li, Z., Nebeling, M., and Kim, H. S. (2019). “iGYM,” in CHI PLAY ’19: Proceedings of the Annual Symposium on Computer-Human Interaction in Play, October 22–25, 2019 (New York, NY, USA:Association for Computing Machinery), 31–43. doi:10.1145/3311350.3347161

CrossRef Full Text | Google Scholar

Gugenheimer, J., Stemasov, E., Frommel, J., and Rukzio, E. (2018a). “A Demonstration of Sharevr: Co-located Experiences for Virtual Reality between Hmd and Non-hmd Users,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR), Reutlingen, Germany, March 18–22, 2018 (Piscataway, NJ: IEEE), 755–756. doi:10.1109/VR.2018.8446551

CrossRef Full Text | Google Scholar

Gugenheimer, J., Stemasov, E., Frommel, J., and Rukzio, E. (2017a). “ShareVR,” in CHI ’17: Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems, May 6–11, 2017 (New York, NY, USA:Association for Computing Machinery), 4021–4033. doi:10.1145/3025453.3025683

CrossRef Full Text | Google Scholar

Gugenheimer, J., Stemasov, E., Sareen, H., and Rukzio, E. (2017b). “FaceDisplay,” in CHI EA ’17: Proceedings of the 2017 CHI Conference Extended Abstracts on Human Factors in Computing Systems, May 6–11, 2017 (New York, NY, USA: Association for Computing Machinery), 369–372. doi:10.1145/3027063.3052962

CrossRef Full Text | Google Scholar

Gugenheimer, J., Stemasov, E., Sareen, H., and Rukzio, E. (2018b). “FaceDisplay,” in CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems, April 21–26, 2018 (New York, NY, USA:Association for Computing Machinery), 1–13. doi:10.1145/3173574.3173628

CrossRef Full Text | Google Scholar

Gutwin, C., and Greenberg, S. (1998). “Design for Individuals, Design for Groups,” in CSCW ’98: Proceedings of the 1998 ACM Conference on Computer Supported Cooperative Work, November 14–18, 1998 (New York, NY, USA:Association for Computing Machinery), 207–216. doi:10.1145/289444.289495

CrossRef Full Text | Google Scholar

Hall, E. T. (1966). The Hidden Dimension. 1966. New York, NY, USA: New York: Doubleday.

Hansen, A., Larsen, K. B., Nielsen, H. H., Sokolov, M. K., and Kraus, M. (2020). “Asymmetrical Multiplayer versus Single Player: Effects on Game Experience in a Virtual Reality Edutainment Game,” in Augmented Reality, Virtual Reality and Computer Graphics (AVR 2020), Lecce, Italy, September 7–10, 2020 (Cham, Switzerland: Springer International Publishing), 22–33. doi:10.1007/978-3-030-58465-8_2

CrossRef Full Text | Google Scholar

Harms, C., and Biocca, F. (2004). “Internal Consistency and Reliability of the Networked Minds Measure of Social Presence,” in Seventh Annual International Workshop: Presence 2004. Editors M. Alcaniz, and B. Rey (Valencia: Universidad Politecnica de Valencia).

Google Scholar

Harper, R., and Mentis, H. (2013). “The Mocking Gaze,” in CSCW ’13: Proceedings of the 2013 Conference on Computer Supported Cooperative Work (New York, NY, USA:ACM), 167–180. doi:10.1145/2441776.2441797

CrossRef Full Text | Google Scholar

Harris, J., Hancock, M., and Scott, S. (2014). “"beam Me 'round, Scotty!",” in CHI PLAY ’14: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA:Association for Computing Machinery), 417–418. doi:10.1145/2658537.2661311

CrossRef Full Text | Google Scholar

Harris, J., Hancock, M., and Scott, S. D. (2016). “Leveraging Asymmetries in Multiplayer Games,” in CHI PLAY ’16: Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA:Association for Computing Machinery), 350–361. doi:10.1145/2967934.2968113

CrossRef Full Text | Google Scholar

Harris, J., and Hancock, M. (2019). “To Asymmetry and beyond!,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York, NY, USA:ACM), 9. doi:10.1145/3290605.3300239

CrossRef Full Text | Google Scholar

Hornecker, E. (2005). “A Design Theme for Tangible Interaction: Embodied Facilitation,” in ECSCW 2005, Paris, France, September 18–22, 2005 (Dordrecht, Switzerland: Springer), 23–43. doi:10.1007/1-4020-4023-7_2

CrossRef Full Text | Google Scholar

Hudson, M., and Cairns, P. (2014). “6 Measuring Social Presence in Team-Based Digital Games,” in Interacting with Presence: HCI and the Sense of Presence in Computer-mediated Environments, 83. doi:10.2478/9783110409697.6

CrossRef Full Text | Google Scholar

Hunicke, R., LeBlanc, M., and Zubek, R. (2004). “MDA: A Formal Approach to Game Design and Game Research,” in Proceedings of the AAAI Workshop on Challenges in Game AI. Vol. 4, 1722.

Google Scholar

IJsselsteijn, W. A., de Kort, Y. A., and Poels, K. (2013). The Game Experience Questionnaire. Eindhoven: Technische Universiteit Eindhoven, 3–9.

Jeong, K., Kim, J., Kim, M., Lee, J., and Kim, C. (2020). Asymmetric Interface: User Interface of Asymmetric Virtual Reality for New Presence and Experience. Symmetry 12, 53. doi:10.3390/sym12010053

CrossRef Full Text | Google Scholar

Jerald, J. (2015). The VR Book: Human-Centered Design for Virtual Reality. San Rafael, CA: Association for Computing Machinery and Morgan & Claypool.

Johnson, D., Gardner, M. J., and Perry, R. (2018). Validation of Two Game Experience Scales: The Player Experience of Need Satisfaction (PENS) and Game Experience Questionnaire (GEQ). Int. J. Human-Computer Stud. 118, 38–46. doi:10.1016/j.ijhcs.2018.05.003

CrossRef Full Text | Google Scholar

Karaosmanoglu, S., Rogers, K., Wolf, D., Rukzio, E., Steinicke, F., and Nacke, L. E. (2021). “Feels like Team Spirit: Biometric and Strategic Interdependence in Asymmetric Multiplayer Vr Games,” in CHI ’21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3411764.3445492

CrossRef Full Text | Google Scholar

Kaye, L. K. (2016). Exploring Flow Experiences in Cooperative Digital Gaming Contexts. Comput. Hum. Behav. 55, 286–291. doi:10.1016/j.chb.2015.09.023

CrossRef Full Text | Google Scholar

Kendon, A. (2010). “Spacing and Orientation in Co-present Interaction,” in Development of Multimodal Interfaces: Active Listening and Synchrony (Berlin/Heidelberg, Germany: Springer), 1–15. doi:10.1007/978-3-642-12397-9_1

CrossRef Full Text | Google Scholar

Kennedy, R. S., Lane, N. E., Berbaum, K. S., and Lilienthal, M. G. (1993). Simulator Sickness Questionnaire: An Enhanced Method for Quantifying Simulator Sickness. Int. J. aviation Psychol. 3, 203–220. doi:10.1207/s15327108ijap0303_3

CrossRef Full Text | Google Scholar

Kerure, A. A., and Freeman, J. (2018). “Audio Source Localization as an Input to Virtual Reality Environments,” in Audio Engineering Society Convention, Milan, Italy, May 23–26, 2018 (New York, NY: Audio Engineering Society), 144.

Google Scholar

Kim, E., and Schneider, O. (2020). “Defining Haptic Experience: Foundations for Understanding, Communicating, and Evaluating Hx,” in CHI ’20: Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, April 25–30, 2020 (New York, NY, USA:Association for Computing Machinery), 1–13. doi:10.1145/3313831.3376280

CrossRef Full Text | Google Scholar

Kiourt, C., Theodoropoulou, H. G., Koutsoudis, A., Ioannakis, J. A., Pavlidis, G., and Kalles, D. (2020). “Exploiting Cross-Reality Technologies for Cultural Heritage Dissemination,” in Applying Innovative Technologies in Heritage Science (Hershey, PA: IGI Global), 85–108. doi:10.4018/978-1-7998-2871-6.ch005

CrossRef Full Text | Google Scholar

Klimmt, C., and Hartmann, T. (2008). Mediated Interpersonal Communication in Multiplayer Video Games: Implications for Entertainment and Relationship Management. Mediated interpersonal Commun. 309, 323–344. doi:10.4324/9780203926864-26

CrossRef Full Text | Google Scholar

Kmet, L. M., Cook, L. S., and Lee, R. C. (2004). Standard Quality Assessment Criteria for Evaluating Primary Research Papers from a Variety of fields. Canada: Institute of Health Economies, IHE, University of Alberta. doi:10.7939/R37M04F16

CrossRef Full Text

Knierim, P., Funk, M., Kosch, T., Fedosov, A., Müller, T., Schopf, B., Weise, M., and Schmidt, A. (2016). “UbiBeam++,” in NordiCHI ’16: Proceedings of the 9th Nordic Conference on Human-Computer Interaction, October 23–27, 2016 (New York, NY, USA:Association for Computing Machinery). doi:10.1145/2971485.2996747

CrossRef Full Text | Google Scholar

Law, E. L.-C., Brühlmann, F., and Mekler, E. D. (2018). in CHI PLAY ’18: Systematic review and validation of the game experience questionnaire (geq) - implications for citation and reporting practice (New York, NY, USA:Association for Computing Machinery), 257–270. doi:10.1145/3242671.3242683

CrossRef Full Text

Leavitt, A., Keegan, B. C., and Clark, J. (2016). “Ping to Win?,” in CHI ’16: Proceedings of the 2016 CHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 4337–4350. doi:10.1145/2858036.2858132

CrossRef Full Text | Google Scholar

Lee, J., Kim, M., and Kim, J. (2020). Rolevr: Multi-Experience in Immersive Virtual Reality between Co-located Hmd and Non-hmd Users. Multimed Tools Appl. 79, 979–1005. doi:10.1007/s11042-019-08220-w

CrossRef Full Text | Google Scholar

Li, J., Deng, H., and Michalatos, P. (2017a). “CatEscape,” in CHI PLAY ’17: Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA:Association for Computing Machinery), 585–590. doi:10.1145/3130859.3130860

CrossRef Full Text | Google Scholar

Li, J., Deng, H., and Michalatos, P. (2017b). “MagicTorch,” in CHI PLAY ’17: Extended Abstracts Publication of the Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA: Association for Computing Machinery), 431–436. doi:10.1145/3130859.3131341

CrossRef Full Text | Google Scholar

Lifton, J. H. (2007). Dual Reality: An Emerging Medium. PhD thesis. (Massachusetts, MA: Massachusetts Institute of Technology). http://hdl.handle.net/1721.1/41707.

Liszio, S., Emmerich, K., and Masuch, M. (2017). “The Influence of Social Entities in Virtual Reality Games on Player Experience and Immersion,” in FDG ’17: Proceedings of the 12th International Conference on the Foundations of Digital Games (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3102071.3102086

CrossRef Full Text | Google Scholar

Liszio, S., and Masuch, M. (2016). “Designing Shared Virtual Reality Gaming Experiences in Local Multi-Platform Games,” in International Conference on Entertainment Computing (Springer), 235–240. doi:10.1007/978-3-319-46100-7_23

CrossRef Full Text | Google Scholar

MacArthur, C., Grinberg, A., Harley, D., and Hancock, M. (2021). “You're Making Me Sick,” in CHI ’21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3411764.3445701

CrossRef Full Text | Google Scholar

Manninen, T., and Korva, T. (2005). “Designing Puzzles for Collaborative Gaming Experience - Case: Escape,” in DiGRA Conference: Selected Papers Proceedings of Digital Games Research Association’s Second International Conference, 233–247.

Google Scholar

Marquardt, N., Hinckley, K., and Greenberg, S. (2012). “Cross-device Interaction via Micro-mobility and F-Formations,” in UIST ’12: Proceedings of the 25th Annual ACM Symposium on User Interface Software and Technology (New York, NY, USA: Association for Computing Machinery), 13–22. doi:10.1145/2380116.2380121

CrossRef Full Text | Google Scholar

Márquez Segura, E., Rogers, K., Martin-Niedecken, A. L., Niedecken, S., and Vidal, L. T. (2021). “Exploring the Design Space of Immersive Social Fitness Games: The Imsofit Games Model,” in CHI ’21: Proceedings of the 2021 CHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery). doi:10.1145/3411764.3445592

CrossRef Full Text | Google Scholar

Martínez-Ruiz, F. J., Rauh, S. F., and Meixner, G. (2019). “Understanding Peripheral Audiences: From Subtle to Full Body Gestures,” in Advances in Intelligent Systems and Computing (Springer International Publishing), 489–495. doi:10.1007/978-3-030-25629-6_76

CrossRef Full Text | Google Scholar

McDonald, N., Schoenebeck, S., and Forte, A. (2019). Reliability and Inter-rater Reliability in Qualitative Research. Proc. ACM Hum.-Comput. Interact. 3, 1–23. doi:10.1145/3359174

CrossRef Full Text | Google Scholar

McGill, M., Boland, D., Murray-Smith, R., and Brewster, S. (2015). “A Dose of Reality,” in CHI ’15: Proceedings of the 33rd Annual ACM Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 2143–2152. doi:10.1145/2702123.2702382

CrossRef Full Text | Google Scholar

Milgram, P., and Kishino, F. (1994). A Taxonomy of Mixed Reality Visual Displays. IEICE Trans. Inf. Syst. E77-D (12), 1321–1329.

Google Scholar

Moher, D., Liberati, A., Tetzlaff, J., and Altman, D. G. Prisma Group (2009). Preferred Reporting Items for Systematic Reviews and Meta-Analyses: the Prisma Statement. PLoS Med. 6 (7), e1000097. doi:10.1371/journal.pmed.1000097

Morris, M. R., Ryall, K., Shen, C., Forlines, C., and Vernier, F. (2004). “Beyond "social Protocols",” in CSCW ’04: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (New York, NY, USA:Association for Computing Machinery), 262–265. doi:10.1145/1031607.1031648

CrossRef Full Text | Google Scholar

Murray, J. H. (2017). Hamlet on the Holodeck: The Future of Narrative in Cyberspace. Cambridge, MA, USA: MIT press.

Mütterlein, J., and Hess, T. (2017). “Immersion, Presence, Interactivity: towards a Joint Understanding of Factors Influencing Virtual Reality Acceptance and Use,” in Twenty-third Americas Conference on Information Systems (AMCIS) AMCIS2017 Proceedings: Adoption and Diffusion of Information Technology (SIGADIT).

Google Scholar

Nakagawa, R., and Sonobe, K. (2019). “Encounters,” in SA ’19: SIGGRAPH Asia 2019 XR (New York, NY, USA:Association for Computing Machinery), 6–8. doi:10.1145/3355355.3361886

CrossRef Full Text | Google Scholar

Nestler, S., Back, M. D., and Egloff, B. (2011). Psychometrische Eigenschaften zweier Skalen zur Erfassung interindividueller Unterschiede in der Präferenz zum Alleinsein. Diagnostica 57, 57–67. doi:10.1026/0012-1924/a000032

CrossRef Full Text | Google Scholar

Newton, B. J., Rothlingova, Z., Gutteridge, R., LeMarchand, K., and Raphael, J. H. (2012). No Room for Reflexivity? Critical Reflections Following a Systematic Review of Qualitative Research. J. Health Psychol. 17, 866–885. doi:10.1177/1359105311427615

PubMed Abstract | CrossRef Full Text | Google Scholar

Nowell, L. S., Norris, J. M., White, D. E., and Moules, N. J. (2017). Thematic Analysis. Int. J. Qual. Methods 16, 160940691773384. doi:10.1177/1609406917733847

CrossRef Full Text | Google Scholar

O'Hagan, J., Williamson, J. R., and Khamis, M. (2020). “Bystander Interruption of Vr Users,” in PerDis ’20: Proceedings of the 9TH ACM International Symposium on Pervasive Displays (New York, NY, USA: Association for Computing Machinery), 19–27. doi:10.1145/3393712.3395339

CrossRef Full Text | Google Scholar

O'Hagan, J., and Williamson, J. R. (2020). “Reality Aware Vr Headsets,” in PerDis ’20: Proceedings of the 9TH ACM International Symposium on Pervasive Displays (New York, NY, USA: Association for Computing Machinery), 9–17. doi:10.1145/3393712.3395334

CrossRef Full Text | Google Scholar

Odd Raven Studios (2019). Carly and the Reaperman—Escape from the Underworld. Stockholm, Sweden: Game [HTC Vive, Windows]. Odd Raven Studios.

Olin, P. A., Issa, A. M., Feuchtner, T., and Grønbæk, K. (2020). “Designing for Heterogeneous Cross-Device Collaboration and Social Interaction in Virtual Reality,” in OzCHI ’20: 32nd Australian Conference on Human-Computer Interaction (New York, NY, USA: Association for Computing Machinery), 112–127. doi:10.1145/3441000.3441070

CrossRef Full Text | Google Scholar

Osmanovic, S., and Pecchioni, L. (2016). Beyond Entertainment. Games Cult. 11, 130–149. doi:10.1177/1555412015602819

CrossRef Full Text | Google Scholar

Othlinghaus, J., Gerling, K. M., and Masuch, M. (2011). “Intergenerational Play: Exploring the Needs of Children and Elderly,” in Workshop-Proceedings of Mensch & Computer 2011 (Chemnitz, Germany: Universitätsverlag der Technischen Universität Chemnitz).

Google Scholar

Ouverson, K. M., and Gilbert, S. B. (2021). A Composite Framework of Co-located Asymmetric Virtual Reality. Proc. ACM Hum.-Comput. Interact. 5, 1–20. doi:10.1145/3449079

CrossRef Full Text | Google Scholar

Peck, T. C., Sockol, L. E., and Hancock, S. M. (2020). Mind the gap: The Underrepresentation of Female Participants and Authors in Virtual Reality Research. IEEE Trans. Vis. Comput. Graphics 26, 1945–1954. doi:10.1109/TVCG.2020.2973498

CrossRef Full Text | Google Scholar

Perry, R., Drachen, A., Kearney, A., Kriglstein, S., Nacke, L. E., Sifa, R., et al. (2018). Online-only Friends, Real-Life Friends or Strangers? Differential Associations with Passion and Social Capital in Video Game Play. Comput. Hum. Behav. 79, 202–210. doi:10.1016/j.chb.2017.10.032

CrossRef Full Text | Google Scholar

Pinelle, D., Gutwin, C., and Greenberg, S. (2003). Task Analysis for Groupware Usability Evaluation. ACM Trans. Comput.-Hum. Interact. 10, 281–311. doi:10.1145/966930.966932

CrossRef Full Text | Google Scholar

Piumsomboon, T., Lee, G. A., Hart, J. D., Ens, B., Lindeman, R. W., Thomas, B. H., and Billinghurst, M. (2018). “Mini-Me,” in CHI ’18: Proceedings of the 2018 CHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 1–13. doi:10.1145/3173574.3173620

CrossRef Full Text | Google Scholar

Pope, C. (2000). Qualitative Research in Health Care: Analysing Qualitative Data. BMJ 320, 114–116. doi:10.1136/bmj.320.7227.114

PubMed Abstract | CrossRef Full Text | Google Scholar

Reeves, S., Benford, S., O'Malley, C., and Fraser, M. (2005). “Designing the Spectator Experience,” in CHI ’05: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 741–750. doi:10.1145/1054972.1055074

CrossRef Full Text | Google Scholar

Reilly, D. F., Rouzati, H., Wu, A., Hwang, J. Y., Brudvik, J., and Edwards, W. K. (2010). “TwinSpace,” in UIST ’10: Proceedings of the 23nd Annual ACM Symposium on User Interface Software and Technology (New York, NY, USA: Association for Computing Machinery), 119–128. doi:10.1145/1866029.1866050

CrossRef Full Text | Google Scholar

Resolution Games (2019). Acron: Attack of the Squirrels. Stockholm, Sweden: Game [Oculus Quest, iOS/Android]. Resolution Games. https://www.resolutiongames.com/acron.

Rigby, S., and Ryan, R. (2007). The Player Experience of Need Satisfaction (Pens): An Applied Model and Methodology for Understanding Key Components of the Player Experience.

Ritchie, J., and Spencer, L. (2002). Qualitative Data Analysis for Applied Policy Research. The Qual. researcher’s companion 573, 305–329. doi:10.4135/9781412986274.n12

Google Scholar

Rogers, K., Funke, J., Frommel, J., Stamm, S., and Weber, M. (2019). “Exploring Interaction Fidelity in Virtual Reality,” in CHI ’19: Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems (New York, NY, USA: Association for Computing Machinery), 1–14. doi:10.1145/3290605.3300644

CrossRef Full Text | Google Scholar

Rubio-Tamayo, J., Gertrudix Barrio, M., and García García, F. (2017). Immersive Environments and Virtual Reality: Systematic Review and Advances in Communication, Interaction and Simulation. Multimodal Technol. Interact. 1, 21. doi:10.3390/mti1040021

CrossRef Full Text | Google Scholar

Russell, D., Peplau, L. A., and Ferguson, M. L. (1978). Developing a Measure of Loneliness. J. Personal. Assess. 42, 290–294. doi:10.1207/s15327752jpa4203_11

CrossRef Full Text | Google Scholar

Ryan, R. M., and Deci, E. L. (2000). Self-determination Theory and the Facilitation of Intrinsic Motivation, Social Development, and Well-Being. Am. Psychol. 55, 68–78. doi:10.1037/0003-066x.55.1.68

PubMed Abstract | CrossRef Full Text | Google Scholar

Ryan, R. M., Rigby, C. S., and Przybylski, A. (2006). The Motivational Pull of Video Games: A Self-Determination Theory Approach. Motiv. Emot. 30, 344–360. doi:10.1007/s11031-006-9051-8

CrossRef Full Text | Google Scholar

Sajjadi, P., Cebolledo Gutierrez, E. O., Trullemans, S., and De Troyer, O. (2014). “Maze Commander,” in CHI PLAY ’14: Proceedings of the First ACM SIGCHI Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA:Association for Computing Machinery), 227–236. doi:10.1145/2658537.2658690

CrossRef Full Text | Google Scholar

Salen, K., and Zimmerman, E. (2004). Rules of Play: Game Design Fundamentals. Cambridge, MA, USA: MIT press.

Schmidt, S., Nunez, O. J. A., and Steinicke, F. (2019). “Blended Agents: Manipulation of Physical Objects within Mixed Reality Environments and beyond,” in Symposium on Spatial User Interaction, SUI 2019, New Orleans, LA, October 19-20, 2019. Editors C. W. Borst, A. K. Kulshreshth, G. Bruder, S. Serafin, C. Sandor, K. Johnsenet al. (New York, NY: Association for Computing Machinery), 6:1–6:10. doi:10.1145/3357251.3357591

CrossRef Full Text | Google Scholar

Schmitz, M., Akbal, M., and Zehle, S. (2015). “SpielRaum,” in CHI PLAY ’15: Proceedings of the 2015 Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA:Association for Computing Machinery), 427–432. doi:10.1145/2793107.2810306

CrossRef Full Text | Google Scholar

Schubert, T., Friedmann, F., and Regenbrecht, H. (2001). The Experience of Presence: Factor Analytic Insights. Presence: Teleoperators & Virtual Environments 10, 266–281. doi:10.1162/105474601300343603

CrossRef Full Text | Google Scholar

Scott, S. D., Grant, K. D., and Mandryk, R. L. (2003). “System Guidelines for Co-located, Collaborative Work on a Tabletop Display,” in Proceedings of the Eighth Conference on European Conference on Computer Supported Cooperative Work - ECSCW’03 (Springer), 159–178. doi:10.1007/978-94-010-0068-0_9

CrossRef Full Text | Google Scholar

Seif El-Nasr, M., Aghabeigi, B., Milam, D., Erfani, M., Lameman, B., Maygoli, H., and Mah, S. (2010). “Understanding and Evaluating Cooperative Games,” in CHI ’10: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 253–262. doi:10.1145/1753326.1753363

CrossRef Full Text | Google Scholar

Serubugo, S., Skantarova, D., Evers, N., and Kraus, M. (2018a). “Facilitating Asymmetric Collaborative Navigation in Room-Scale Virtual Reality for Public Spaces,” in Interactivity, Game Creation, Design, Learning, and Innovation, Heraklion, Greece, October 30–31, 2018 (Cham, Switzerland: Springer), 64–73. doi:10.1007/978-3-319-76908-0_7

CrossRef Full Text | Google Scholar

Serubugo, S., Skantarova, D., Evers, N., and Kraus, M. (2018b). “Self-overlapping Maze and Map Design for Asymmetric Collaboration in Room-Scale Virtual Reality for Public Spaces,” in Interactivity, Game Creation, Design, Learning, and Innovation, Heraklion, Greece, October 30–31, 2018 (Cham, Switzerland: Springer), 194–203. doi:10.1007/978-3-319-76908-0_19

CrossRef Full Text | Google Scholar

Serubugo, S., Skantarova, D., Evers, N., and Kraus, M. (2017). “Walkable Self-Overlapping Virtual Reality Maze and Map Visualization Demo,” in VRST ’17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3139131.3141774

CrossRef Full Text | Google Scholar

Shamseer, L., Moher, D., Clarke, M., Ghersi, D., Liberati, A., Petticrew, M., et al. (2015). Preferred Reporting Items for Systematic Review and Meta-Analysis Protocols (PRISMA-P) 2015: Elaboration and Explanation. BMJ 349, g7647. doi:10.1136/bmj.g7647

CrossRef Full Text | Google Scholar

Sigitov, A., Hinkenjann, A., Kruijff, E., and Staadt, O. (2019). Task Dependent Group Coupling and Territorial Behavior on Large Tiled Displays. Front. Robot. AI 6, 128. doi:10.3389/frobt.2019.00128

PubMed Abstract | CrossRef Full Text | Google Scholar

Simeone, A., Khamis, M., Kljun, M., Isokoski, P., Esteves, A., Daiber, F., et al. (2020). “International Workshop on Cross-Reality (XR) Interaction,” in Companion Proceedings of the 2020 Conference on Interactive Surfaces and Spaces, Virtual Event, Portugal (New York, NY: Association for Computing Machinery), 111–114. doi:10.1145/3380867.3424551

Simon, B. (2009). Wii Are Out of Control: Bodies, Game Screens and the Production of Gestural Excess. SSRN J. doi:10.2139/ssrn.1354043

CrossRef Full Text | Google Scholar

Slater, M., Usoh, M., and Steed, A. (1994). Depth of Presence in Virtual Environments. Presence: Teleoperators & Virtual Environments 3, 130–144. doi:10.1162/pres.1994.3.2.130

CrossRef Full Text | Google Scholar

Smilovitch, M., and Lachman, R. (2019). “Birdquestvr: A Cross-Platform Asymmetric Communication Game,” in CHI PLAY ’19: Extended Abstracts of the Annual Symposium on Computer-Human Interaction in Play Companion Extended Abstracts (New York, NY, USA:Association for Computing Machinery), 307–313. doi:10.1145/3341215.3358246

CrossRef Full Text | Google Scholar

Sra, M., Jain, D., Caetano, A. P., Calvo, A., Hilton, E., and Schmandt, C. (2016). “Resolving Spatial Variation and Allowing Spectator Participation in Multiplayer Vr,” in UIST ’16 Adjunct: Proceedings of the 29th Annual Symposium on User Interface Software and Technology (New York, NY, USA:Association for Computing Machinery), 221–222. doi:10.1145/2984751.2984779

CrossRef Full Text | Google Scholar

Sra, M., Xu, X., and Maes, P. (2017). “GalVR,” in VRST ’17: Proceedings of the 23rd ACM Symposium on Virtual Reality Software and Technology (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3139131.3141219

CrossRef Full Text | Google Scholar

Steel Crate Games (2015). Keep Talking and Nobody Explodes. Ottawa, Canada: Game [VR, PC]. Steel Crate Games. https://keeptalkinggame.com/.

Streck, A., Stepnicka, P., Klaubert, J., and Wolbers, T. (2019b). “Neomento - towards Building a Universal Solution for Virtual Reality Exposure Psychotherapy,” in 2019 IEEE Conference on Games (CoG), London, United Kingdom, August 20–23, 2019 (Piscataway, NJ: IEEE), 1–2. doi:10.1109/cig.2019.8848110

CrossRef Full Text | Google Scholar

Streck, A., Stepnicka, P., Klaubert, J., and Wolbers, T. (2019a). “Neomento SAD - VR Treatment for Social Anxiety,” in 2019 IEEE International Conference on Artificial Intelligence and Virtual Reality (AIVR), San Diego, CA, December 9–11, 2019 (Piscataway, NJ: IEEE). doi:10.1109/aivr46125.2019.00054

CrossRef Full Text | Google Scholar

Striner, A., Azad, S., and Martens, C. (2019). “A Spectrum of Audience Interactivity for Entertainment Domains,” in Interactive Storytelling, Salt Lake City, UT, November 19–22, 2019 (Cham, Switzerland: Springer International Publishing), 214–232. doi:10.1007/978-3-030-33894-7_23

CrossRef Full Text | Google Scholar

Sykownik, P., Emmerich, K., and Masuch, M. (2017). “Exploring Patterns of Shared Control in Digital Multiplayer Games,” in International Conference on Advances in Computer Entertainment, London, United Kingdom, December 14–16, 2017 (Cham, Switzerland: Springer), 847–867. doi:10.1007/978-3-319-76270-8_57

CrossRef Full Text | Google Scholar

Team Panoptes (2019). Panoptic. Brussels, Belgium: Game [VR, PC]. Team Panoptes. https://panopticgame.com/.

Tse, E., Histon, J., Scott, S. D., and Greenberg, S. (2004). “Avoiding Interference,” in CSCW ’04: Proceedings of the 2004 ACM Conference on Computer Supported Cooperative Work (New York, NY, USA:Association for Computing Machinery), 252–261. doi:10.1145/1031607.1031647

CrossRef Full Text | Google Scholar

Vella, K., Klarkowski, M., Turkay, S., and Johnson, D. (2019). Making Friends in Online Games: Gender Differences and Designing for Greater Social Connectedness. Behav. Inf. Technol. 39, 917–934. doi:10.1080/0144929x.2019.1625442

CrossRef Full Text | Google Scholar

Vicencio-Moreira, R., Mandryk, R. L., Gutwin, C., and Bateman, S. (2014). “The Effectiveness (Or Lack Thereof) of Aim-Assist Techniques in First-Person Shooter Games,” in CHI ’14: Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (New York, NY, USA:Association for Computing Machinery), 937–946. doi:10.1145/2556288.2557308

CrossRef Full Text | Google Scholar

Voida, A., and Greenberg, S. (2012). Console Gaming across Generations: Exploring Intergenerational Interactions in Collocated Console Gaming. Univ. Access Inf. Soc. 11, 45–56. doi:10.1007/s10209-011-0232-1

CrossRef Full Text | Google Scholar

Walther, J. B., Van Der Heide, B., Ramirez, A., Burgoon, J. K., and Peña, J. (2015). Interpersonal and Hyperpersonal Dimensions of Computer-Mediated Communication. (West Sussex, United Kingdom: John Wiley & Sons), 1–22. doi:10.1002/9781118426456.ch1

CrossRef Full Text

Wang, M., Ryoo, J., and Winkelmann, K. (2020). Preface to the Special Issue on Cross Reality (XR) and Immersive Learning Environments (ILE) in Education. Interactive Learn. Environments 28, 539–542. doi:10.1080/10494820.2019.1696845

CrossRef Full Text | Google Scholar

Want, R. (2009). Through Tinted Eyeglasses. IEEE Pervasive Comput. 8, 2–4. doi:10.1109/mprv.2009.58

CrossRef Full Text | Google Scholar

Wells, J. D., Campbell, D. E., Valacich, J. S., and Featherman, M. (2010). The Effect of Perceived novelty on the Adoption of Information Technology Innovations: A Risk/reward Perspective. Decis. Sci. 41, 813–843. doi:10.1111/j.1540-5915.2010.00292.x

CrossRef Full Text | Google Scholar

Witmer, B. G., Jerome, C. J., and Singer, M. J. (2005). The Factor Structure of the Presence Questionnaire. Presence: Teleoperators & Virtual Environments 14, 298–312. doi:10.1162/105474605323384654

CrossRef Full Text | Google Scholar

Witmer, B. G., and Singer, M. J. (1998). Measuring Presence in Virtual Environments: A Presence Questionnaire. Presence 7, 225–240. doi:10.1162/105474698565686

CrossRef Full Text | Google Scholar

Wohlin, C. (2014). “Guidelines for Snowballing in Systematic Literature Studies and a Replication in Software Engineering,” in Proceedings of the 18th international conference on evaluation and assessment in software engineering, 1–10. doi:10.1145/2601248.2601268

CrossRef Full Text | Google Scholar

Woods, S. J. (2009). (play) Ground Rules: The Social Contract and the Magic circle. Observatorio (OBS*) 3, 204–222. doi:10.15847/obsOBS312009243

CrossRef Full Text | Google Scholar

Woodworth, J. W., Ekong, S., and Borst, C. W. (2017). “Virtual Field Trips with Networked Depth-Camera-Based Teacher, Heterogeneous Displays, and Example Energy center Application,” in 2017 IEEE Virtual Reality (VR) (IEEE), 471–472. doi:10.1109/VR.2017.7892384

CrossRef Full Text | Google Scholar

Yassien, A., ElAgroudy, P., Makled, E., and Abdennadher, S. (2020). “A Design Space for Social Presence in Vr,” in NordiCHI ’20: Proceedings of the 11th Nordic Conference on Human-Computer Interaction: Shaping Experiences, Shaping Society (New York, NY, USA:Association for Computing Machinery). doi:10.1145/3419249.3420112

CrossRef Full Text | Google Scholar

Zagal, J. P., Rick, J., and Hsi, I. (2006). Collaborative Games: Lessons Learned from Board Games. Simulation & Gaming 37, 24–40. doi:10.1177/1046878105282279

CrossRef Full Text | Google Scholar

Zhou, Z., Márquez Segura, E., Duval, J., John, M., and Isbister, K. (2019). “Astaire.” in CHI PLAY ’19: Proceedings of the Annual Symposium on Computer-Human Interaction in Play (New York, NY, USA: Association for Computing Machinery), 5–18. doi:10.1145/3311350.3347152

CrossRef Full Text | Google Scholar