The Use of Brain–Computer Interfaces in Games Is Not Ready for the General Public

Cattan, Grégoire

doi:10.3389/fcomp.2021.628773

OPINION article

Front. Comput. Sci., 24 March 2021

Sec. Human-Media Interaction

Volume 3 - 2021 | https://doi.org/10.3389/fcomp.2021.628773

This article is part of the Research Topic Brain-Computer Interfaces for Non-clinical (Home, Sports, Art, Entertainment, Education, Well-being) Applications View all 12 articles

The Use of Brain–Computer Interfaces in Games Is Not Ready for the General Public

$\nGrgoire Cattan$ Grégoire Cattan^*

IBM, Krakow, Poland

Introduction

The use of brain–computer interfaces (BCIs) based on electroencephalography (EEG) in video games has been widely investigated. Research in adaptive training, single-trial classification (Congedo, 2013; Barachant and Congedo, 2014) and the creation of affordable EEG acquisition devices (Vos et al., 2014; Yohanandan et al., 2018) has paved the way for the development of a ubiquitous BCI technology. For example, Congedo (2013) developed “Brain Invaders,” a BCI game inspired by the famous vintage game Space Invaders (Taito, Tokyo, Japan) and based on the so-called visual P300—an electromagnetic potential produced by the brain about 300 ms after a visual stimulation. Brain Invaders uses an adaptive algorithm that allows the player to plug the material and play without the need for calibration (Barachant and Congedo, 2014), while still achieving a high accuracy rate (Barachant et al., 2012). The game also demonstrates a good understanding of game design by naturally incorporating visual stimulations in the virtual environment. In this respect, Kaplan et al. (2013), Cattan et al. (2018b), and Rashid et al. (2020) provided a set of guidelines to adapt game implementation for BCI games, such as the use of turn-based games with a slow gameplay. Although Brain Invaders use a research-grade amplifier, the feasibility of using a low-cost EEG acquisition system for BCI has been demonstrated by Vos et al. (2014) and Yohanandan et al. (2018). These affordable headsets are comparable with research-grade amplifiers. In addition, Lotte et al. (2008) and Debener et al. (2012) demonstrated promising results when BCIs were used out of the lab for BCIs based on visual stimulation and movement imagination. The feasibility of using BCIs outside the lab has also been demonstrated in different context and at events. For instance, in the BCI game developed by Mentalista (Paris, France) for the 2016 European Football Championship, two players were asked to score against each other by moving a ball toward the opposite player's cage by concentrating¹.

Although a positive step forwards, these achievements have led to the false opinion that BCIs are ready for entertainment – a belief that is supported by enthusiastic visions claiming, for example, that brains will be connected to the internet through USB². This opinion is rather qualified in the scientific community, which reported that BCIs suffer from (1) a low transfer rate, (2) a lack of market-ready, affordable, and user-friendly research-grade EEG acquisition devices, and (3) a gap between the game design and graphics of video games available on the market vs. in laboratories. Priorities of these limitations for video game development are discussed in the literature (Nijholt et al., 2009; Ferreira et al., 2013; Marshall et al., 2013; van de Laar et al., 2013; Ahn et al., 2014; Cattan et al., 2018b; Kerous et al., 2018; Vasiljevic and Miranda, 2019; Pierce et al., 2020), and in general preponderance of quantitative over qualitative aspects is criticized (Nijholt et al., 2009; Vasiljevic and Miranda, 2019). This article supports the claim that BCIs are not ready for general public use, based on other aspects than performance. Limitations are further detailed in section Limitations of BCI Games of the present study, and the obstacles to public use are analyzed. The conclusion is presented in section Discussion and Conclusion.

Limitations of BCI Games

Low Transfer Rate

The transfer rate (in bits per second) is a computed measure for communication devices that reflects the speed and accuracy of a device. This measure is derived from information theory and was adapted for BCIs (Wolpaw and Wolpaw, 2012) to compare different implementations. Following the development of Riemannian Geometry during the last decade (e.g., Barachant et al., 2012), the transfer rate of BCIs has considerably increased to reach, for example in the case of a P300-based BCI, a 90% accuracy within a couple of seconds (Cattan et al., 2018b). For comparison, the Guinness World Records reports a record of stenotype writing of around 360 words per minutes with 97% accuracy³. In practice, this means that BCIs are unusable in traditional inputs, such as in keyboards or mice. Another concern is that the algorithm complexity [in terms of mathematics and set-up (e.g., Cattan et al., 2018a; Andreev et al., 2019)] behind BCIs might be an obstacle for game developers. This is true for VR games especially, as the VR market is mostly represented by small, independent companies with limited resources⁴^,⁵.

Cost and Encumberment of Materials

Until the early 2000s, research-grade amplifiers were medical and were made of expensive materials and could mostly only be afforded by institutions or consortia (e.g., g.USBAmp, g.tec, Schiedlberg, Austria). Emotiv (San Francisco, US) was among the first manufacturers to release a commercial EEG cap for individual customers in 2009. However, low-cost versions of the Emotiv headsets omit a proper electrode location for P300-based BCIs, while research-grade versions still practice dissuasive pricing. Similar concerns apply to most customer-grade EEGs (Ahn et al., 2014; Vasiljevic and Miranda, 2019). That is, cheap materials only include a few and unmovable channels which are inappropriate for BCI based on visual stimulation, whereas medical-grade EEG caps are expensive. In 2013, OpenBCI (New York, US) finished a successful fundraising campaign, with the aim of providing a high-quality EEG acquisition system for <1,000 euros. Nevertheless, concerns were raised about the usability of the technology, such as its association with unstable wireless communication, non-standardized sampling rate and use of gel electrodes (e.g., Chabance et al., 2019). Some researchers (e.g., Yohanandan et al., 2018) demonstrated that similar performance could be obtained with an in-house, and thus cheaper, EEG headset. A Huffington Post (New York, US) publication presented an EEG headset that was ergonomic (i.e., with dry electrodes and wireless) for <500 euros⁶^,⁷. However, such headsets are not available for public use, in the sense that, even though some are open-source, developers cannot build them in practice.

Another consideration is that any development is material-dependant, as all are based on different hardware, drivers and protocols (e.g., Ahn et al., 2014; Pierce et al., 2020). Despite the lack of standards, commercial brands such as Emotiv (San Francisco, the US) or NeuroSky (San Rose, the US) have developed plugins for game development. For instance, Rosca and Leba (2019) developed a pool game by integrating the Emotiv SDK with Unity 3D (San Francisco, the US), a notorious game engine. A review of existing BCI software can be found in Pierce et al. (2020). That said, the lack of middleware supported by a large and independent community creates maintenance and portability issues, as long as plugins are constructor-dependent or rely on the willingness of a research team.

Lack of Game Design and Graphics

Graphics and game design are a key concern and expectation for games on the market, from AAA to indie games. This is why video game studios hire developers as well as graphic designers, art work designers, concept artists and game designers. However, graphics and game design have been underestimated in the development of BCI games in laboratories. For instance, the number of frames per second (FPS) is a major concern for stimulation-based BCI, as they require the exact onset of the stimulation with a precision of around ±2 ms (Andreev et al., 2016). Indeed, if the tagging of such stimulation on the ongoing EEG does not happen all the time in the same frame in which the stimulation is displayed, a jitter is observable, which varies as an inverse function of the frame rate. For example, an FPS of around 50 Hz will output a frame every 20 ms and generate a jitter of a similar amplitude in the worst cases. This is particularly true for VR games where low FPS can result in a higher jitter (Cattan et al., 2018a). However, to our knowledge, the impact of graphical quality in FPS resulting in jitters is poorly understood in the context of BCI entertainment.

Ahn et al. (2014) conducted an opinion survey on the importance of BCI games elements. One aspect of this study was to outline the difference of perspective between developers and researcher communities regarding BCI development. For example, easiness of playing was one of the most important elements for 58% of the developers but only 19% of the researchers. However, the importance of graphics was minimized in the study when it should be a major concern for video games. In fact, the authors reported a developer opinion (later confirmed by the opinion survey for the two communities) that aesthetic is rather not considered as one of the most important factors for BCI games. However, we believe this only means that graphics are a basic requirement for video games. In Schell (2014), a pillar reference for game design, the author said about one of his work experiences in virtual reality:

“We had to make things look beautiful. […]. We used high-end graphics hardware and rich textures and models […].”

In general, laboratory BCI games demonstrate good design, being turn-based and having slow gameplay—in this aspect following the guidelines stated by the scientific community (e.g., Nijholt et al., 2009; Marshall et al., 2013). For example, Lotte et al. (2008) and Andreev et al. (2016) presented the BCI game Use the force!, which consisted of lifting up a vessel with motor imagery. Meanwhile, in the game Alphawow (van de Laar et al., 2013), the avatar's character changes its behavior according to the player's state of relaxation. These two games consider the low-transfer rate of the BCI by mapping it to a feature that is expected to fail from time to time (it can be agreed that using the mind to move objects is difficult to realize) and to not compete with traditional inputs (a keyboard cannot determine a person's relaxation state). Nevertheless, the use of BCI was restricted to a unique aspect of these games. On one hand, if this aspect is a side aspect (e.g., Alphawow), the use of BCI is not valuable because of its cost. On the other hand, if this aspect is the main aspect of the game, it means that the player's ability to finish the game depends on an unreliable input and thus leads to frustration. A subjective study on the use of BCI for gaming (Cattan et al., 2019) also reported a lack of feedback for error quantification from participants. This couples with the fact that around 20% of the users are not proficient using a typical BCI. In fact, BCI illiteracy is an issue which is also well-established in the existing literature (e.g., Nijholt et al., 2009; Allison and Neuper, 2010; Marshall et al., 2013; Vasiljevic and Miranda, 2019), although the idea that physiological traits are responsible for BCI illiteracy is controverted (Thompson, 2019; Riquelme-Ros et al., 2020). These limitations impact replay and the difficulty of games, as players who do not succeed in a BCI task will become stuck in the game without knowing how to improve. In fact, except for a proof of concept or a contest, such as the game created by Mentalista (Paris, France), these games are not suitable for public.

Discussion and Conclusion

This section discusses BCI limitations and explains why overcoming these limitations to develop a ubiquitous BCI game is challenging. These difficulties include a plateau in performance compared with mechanical inputs, technical and algorithmic complexity behind BCI, a lack of middleware for BCI development and an underestimation of graphics and design complexity compared with games in the market.

Transfer rate is the most common limitation discussed in the literature. For example, Rashid et al. (2020) has argued that “most BCI games demonstrate very low accuracy and speed as compared to conventional interfaces, suggesting that there are issues that must be addressed to facilitate the acceptance of BCI games.” According to Cattan et al. (2018b) and Rashid et al. (2020), this is particularly true for games requiring movements, such as VR games, as muscular artifacts interfere with the detection of brain signals. In this regard, the complexity of signal detection and classification is, to our belief, a key obstacle to creating effective BCIs for use in games.

From our perspective, recent developments in non-metric (e.g., Quemy, 2019) or quantum classification (Grant et al., 2018; Havenstein et al., 2019) might lead to significant improvements in BCI acceptance and performance. Indeed, non-metric classification reduces the need for data pre-processing and engineering, while quantum classification takes advantage of quantum physics to improve the speed and accuracy of classification. The emerging field of quantum machine learning has become increasingly mature thanks to the availability of open-source toolkits (e.g., Abraham et al., 2019) and cloud-based quantum machines (such as the IBM quantum experience by IBM, Armonk, US). Havenstein et al. (2019) showed the advantages of using a quantum vs. classical support vector machine for multi-class classification. Further interesting developments are expected in this field in the next decade.

Nevertheless, in our opinion, the impact of a low-transfer rate on BCI games is overestimated. In practice, if BCIs are considered an interesting yet dispensable device for video games, this is mostly due to design issues because BCIs are either used as an ancillary feature of games (despite requiring expensive materials) or as a means of competing with traditional inputs (e.g., keyboards and mice) to achieve the same task faster and with less concentration. In other words, BCIs in games should be limited to a set of aspects that cannot be achieved by traditional inputs, but at the same time should create sufficient value to justify the cost and encumberment of the material. However, despite some of the positive features previously enumerated (e.g., the use of BCIs for naturally imprecise behavior), a complete game concept that can be sold for concrete video game entertainment is lacking. Further, design reflection for BCI games is still in its infancy and is close to the prototypal-use cases created in the early 2000s (e.g., Bayliss and Ballard, 2000). Similar concerns were broached in recent studies, such as Vasiljevic and Miranda (2019) which reported that only a few studies focused on qualitative aspects of the interaction with BCI games.

In short, developers should above all be concerned with game design and game portability. In practice, BCI games cannot be downloaded and run independently of the EEG acquisition system, which is an impediment for both researchers and game developers. In this respect, the work achieved by platforms such as OpenVibe (Renard et al., 2010) or open-source initiatives that rely on a standard protocol, such as Lab Streaming Layer (Stenner et al., 2015), should be emphasized (see text footnote 7). Nevertheless, there is no support for developers (at any level, technologic, mathematical, usability) to integrate BCI in concrete game production. To our knowledge, practical obstacles such as compilation and integration of LSL DLL into game engines, along with the lack of command-line support or synchronization solutions between OpenVibe acquisition server and recreational applications, are rarely mentioned in the existing literature.

Author Contributions

The author confirms being the sole contributor of this work and has approved it for publication.

Conflict of Interest

GC was employed by the company IBM.

Footnotes

1. ^https://mentalista.fr/foot

2. ^https://www.thequint.com/explainers/what-is-neuralink-and-how-does-it-work-explained

3. ^https://www.guinnessworldrecords.com/world-records/fastest-realtime-court-reporter-(stenotype-writing)/

4. ^https://www.vrfocus.com/2020/05/why-now-is-the-time-for-aaa-studios-to-consider-vr/

5. ^https://labusinessjournal.com/news/2015/jun/17/independent-virtual-reality-studios-benefit-early-/

6. ^https://www.youtube.com/watch?v=GgKEOlcX9R8

7. ^http://alexandre.barachant.org/eeg.io/

References

Abraham, H., AduOffei Agarwal, R., Akhalwaya, I. Y., Aleksandrowicz, G., Alexander, T., et al. (2019). Qiskit: An Open-source Framework for Quantum Computing. doi: 10.5281/zenodo.2562110