“Enheduanna—A Manifesto of Falling” Live Brain-Computer Cinema Performance: Performer and Audience Participation, Cognition and Emotional Engagement Using Multi-Brain BCI Interaction

The fields of neural prosthetic technologies and Brain-Computer Interfaces (BCIs) have witnessed in the past 15 years an unprecedented development, bringing together theories and methods from different scientific fields, digital media, and the arts. More in particular, artists have been amongst the pioneers of the design of relevant applications since their emergence in the 1960s, pushing the boundaries of applications in real-life contexts. With the new research, advancements, and since 2007, the new low-cost commercial-grade wireless devices, there is a new increasing number of computer games, interactive installations, and performances that involve the use of these interfaces, combining scientific, and creative methodologies. The vast majority of these works use the brain-activity of a single participant. However, earlier, as well as recent examples, involve the simultaneous interaction of more than one participants or performers with the use of Electroencephalography (EEG)-based multi-brain BCIs. In this frame, we discuss and evaluate “Enheduanna—A Manifesto of Falling,” a live brain-computer cinema performance that enables for the first time the simultaneous real-time multi-brain interaction of more than two participants, including a performer and members of the audience, using a passive EEG-based BCI system in the context of a mixed-media performance. The performance was realised as a neuroscientific study conducted in a real-life setting. The raw EEG data of seven participants, one performer and two different members of the audience for each performance, were simultaneously recorded during three live events. The results reveal that the majority of the participants were able to successfully identify whether their brain-activity was interacting with the live video projections or not. A correlation has been found between their answers to the questionnaires, the elements of the performance that they identified as most special, and the audience's indicators of attention and emotional engagement. Also, the results obtained from the performer's data analysis are consistent with the recall of working memory representations and the increase of cognitive load. Thus, these results prove the efficiency of the interaction design, as well as the importance of the directing strategy, dramaturgy and narrative structure on the audience's perception, cognitive state, and engagement.

Introduction

The fields of neural prosthetic technologies and Brain-Computer Interfaces (BCIs) have witnessed in the past 15 years an unprecedented development (Wolpaw and Wolpaw, 2012; He et al., 2013), bringing together theories and methods from the areas of signal processing, machine learning (pattern recognition), computational intelligence, neuroscience, statistics, linear algebra but also digital media and the arts. Accordingly, the interdisciplinary applications of BCIs span across health purposes for patients, purposes for healthy populations in their working environment, as well as entertainment and the arts (He et al., 2013; Bamdad et al., 2015). However, the latter are often viewed with scepticism. Historically, ideas such as that art is useful in order to disseminate scientific theories, but not to be used itself as a methodological approach in the pursuit of scientific evidence, or that the creative process cannot be examined scientifically, have been rather disseminated (Belk, 1966). Nevertheless, more recently the cross-fertilisation of different disciplines, working together in the frame of interdisciplinary studies and new movements, is leading the use of scientific theories and methods in the creative process, and creative approaches in the scientific explorations. In the frame of BCIs, artists have been amongst the pioneers of the design of BCI applications since their emergence in the 1960s (Nijholt, 2015), pushing the boundaries of applications in real-life contexts. One year after the first demonstration of an EEG-based BCI, Alvin Lucier presented the Music For Solo Performer (1965), which is considered the first performance using the EEG technology. Other pioneering composers, artists, and performers soon followed. With the new research, advancements, and since 2007, the new low-cost commercial-grade wireless devices, there is a new increasing number of interdisciplinary practices, like computer games, interactive installations, and performances that involve the use of these interfaces, combining scientific, and creative methodologies. The vast majority of these works use the brain-activity of a single participant. However, earlier, as well as recent examples, involve the simultaneous interaction of more than one participants or performers with the use of EEG-based multi-brain BCIs (Nijholt, 2015; Zioga et al., 2016). This trend coincides with the increasing interest in the fields of neuroscience and experimental psychology “in studying the mechanisms, dynamics, and processes of the interaction between multiple subjects and their brain-activity” (Zioga et al., 2015) and even more in a real-life context, away from the lab.

In this frame, we discuss and evaluate Enheduanna—A Manifesto of Falling, a live brain-computer cinema performance—that is an interactive performative work that combines live, mediatised representations, more specifically live cinema, and the use of BCIs (Zioga et al., 2015, 2016). The performance was realised as a complete combination of scientific and creative practice-based methodologies and solutions, addressing the challenges of the design and implementation of multi-brain BCIs in mixed-media performances with audience participation. It enables for the first time the simultaneous real-time multi-brain interaction of more than two participants, including a performer and members of the audience, using a passive EEG-based BCI system in the context of a mixed-media performance. In passive BCI systems the outputs are derived from the “arbitrary brain activity” taking place “without the purpose of voluntary control”; in contrast to “active” BCIs that can be controlled by the user consciously i.e. with Motor Imagery; and the “reactive” BCIs that derive their outputs from “brain activity arising in reaction to external stimulation,” such as the P300 BCIs (Zander et al., 2008). The project involves a large multidisciplinary team of professionals, performers and public audiences. The premiere took place at the Theatre space of CCA: Centre for Contemporary Arts Glasgow, UK, in July 2015 (Zioga et al., 2016). The performance was also realised as a neuroscientific study in a real-life setting. The first demonstrations and the implementation of the passive EEG-based BCI system within the real-life conditions of a public event enabled the collection of valuable data from seven participants, one performer and two different members of the audience for each performance: qualitative through the completion of questionnaires and quantitative through the recording of their EEG data. The results reveal that the majority of the audience participants, and the performer participant across the majority of the events, were able to successfully identify whether their brain-activity was interacting with the live video projections or not. Additionally, a correlation has been found between their answers to the questionnaires, the elements of the performance that they identified as most special, and the audience's indicators of attention and emotional engagement. Also, the results obtained from the performer's data analysis are consistent with the recall of working memory representations and the increase of cognitive load.

Thus, these results prove the efficiency of the interaction design, as well as the importance of the directing strategy, dramaturgy, and narrative structure on the audience's perception, cognitive state and engagement. The article presents the materials and methods used (section Materials and Methods); followed by the analysis of the participants' behavioural data, and the analysis of the participant's EEG data (section Results). The aforementioned lead and inform a critical discussion of the results (section Discussion), together with additional observations that were made during the live events, and in comparison with important studies and dominant positions on the cognitive experience and engagement of spectators during live performative works and free viewing of films.

Materials and Methods

Participants

The participants of the study were recruited directly by the first author in the close geographical proximity of the event and included one performer and six members of the audience, two for each performance. The study was approved by the Glasgow School of Art Research Ethics Committee and all participants gave written informed consent. The inclusion criteria were female and male general adult population, aged 18–65 years old, not suffering from a neurological deficit, nor receiving psychiatric or other neurological medication (Zioga et al., 2016). Alongside their demographic data, the participants were asked to complete the “Edinburgh Handedness Inventory” (Oldfield, 1971), in order to determine whether they are right- or left-handed, as this can indicate differences in the underlying mechanisms of the neural processes. More specifically, the performer participant was an ambidextrous, 36 years old female, whose mother tongue is Greek. The six audience participants included (Table 1): three right-handed females, with a mean age of 32.67 years and range 25–43 years; and three right-handed males, with a mean age of 29.67 years and range 28–33 years. Regarding their mother tongues, one audience participant has stated Danish, two English, one Romanian and two Spanish. Whereas no audience participants stated French and Greek as their mother tongues, which were two out of the three languages in which the work was performed, while the third one was English.

TABLE 1

Table 1. Overall demographic data for the audience participants.

All participants were asked not to consume high caffeine drinks, cigarettes and alcohol and avoid the recreational use of drugs for at least 12 h prior to the study, as these substances can affect the brain's electrical activity (brain waves) and therefore the EEG recordings. The audience participants arrived at the space ~60 min prior to the start of the event, when the venue was closed for the public, in order to answer a preliminary brief questionnaire and start the preparation. When this was completed, the space opened to the general public, they were asked to watch the performance like any other spectator and complete in the end a final brief questionnaire. A similar process was also followed for the actress (Zioga et al., 2016).

EEG Data Acquisition

The system used during the performance consists of the participants, the performer and the audience members; the data acquisition; the real-time EEG data processing and feature extraction; and the MAX MSP Jitter programming (Figure 1). The participants' raw EEG data are acquired simultaneously with the MyndPlay Brain-BandXL EEG Headset, which has two dry sensors located in the prefrontal cortex, with one passive (Fpz), one active (Fp1) and a grounding electrode on the ear lobe. The data are transmitted wirelessly to the main computer via Bluetooth (Zioga et al., 2015, 2016). The real-time digital recording and processing is performed with the OpenViBE software (Renard et al., 2010) using algorithms that follow the frequency analysis method and a custom-based feature extraction. The methodology focuses on the oscillatory processes of the brain activity, which occur when large neural populations are not engaged with a specific task and synchronise following an oscillatory pattern. While, with the use of temporal band-pass filters, the 4–40 Hz frequencies are processed and sent to the MAX MSP Jitter software. There, the processed EEG values of each participant are scaled to RGB colour values. The blue colour is controlled by the theta frequency band (4–8 Hz); the green by the alpha band (8–13 Hz); and the red colour by the beta (13–25 Hz) and lower gamma frequency (25–40 Hz) bands. The resulting values from the different participants during different parts of the event feed into a “swatch” that provides 2-dimensional RGB colour selection and display, creating a constantly changing single colour, which is then applied as a filter to pre-rendered black and white video files reproduced in real-time. This way, the transition of the participants from relaxed to more alert cognitive states is visualised in the colour scale of the live video-projections as a shift from colder to warmer tints, setting the atmosphere of the visuals but also of the theatrical stage, within the interactive storytelling and the narrative structure (Zioga et al., 2016).

FIGURE 1

Figure 1. The passive multi-brain EEG-based BCI system. (Images of human heads originally designed by Freepik). Reproduced with permission from Zioga et al. (2016).

Experimental Conditions and Creative Methods

The thematic idea of the performance explores the life and work of the historical figure of Enheduanna (ca. 2285–2250 B.C.E.), an Akkadian Princess, the first documented High Priestess of the deity of the Moon Nanna in the city of Ur (present-day Iraq), who is regarded as possibly the first known author and poet in the history of human civilisation, regardless of gender. The concept of the performance is based on the first author's aesthetic, visual, and dramaturgical vision as the director of the work; the thematic idea and the inter-text devised by the actress; and the design and implementation of a new passive multi-brain EEG-based BCI system (Zioga et al., 2016).

More specifically, the performance consists of two parts, introduced to the audience with vignettes projected during the live visuals. During part 1, titled “Me,” we are introduced and immersed in the story of Enheduanna, with the interaction based solely on the actress' brain-activity. This is further divided in scene 1 “Me Transmitting Signals” with an approximate duration of 13 min 14 s, scene 2 “Me Rising” with an approximate duration of 8 min 42 s, and scene 3 “Me Falling” with approximate duration of 10 min 14 s. During part 2, titled “You/We,” the interaction is based first on either of the two audience members' brain-activity (scene 4 “You Measuring the F-Scale” with approximate duration of 5 min 00 s). Whereas, in the second half (scene 5 ‘We’ with approximate duration of 9 min and 43 s), the real-time brain-interaction of the actress and the audience member gradually merges and their averaged values control the colour filter applied to the live video stream. As a result, the narrative and dramaturgical structure brings together the thematic idea of the performance with the use of the interaction technology in a coherent and comprehensive manner and by this it also serves as an evidence of liveness (Zioga et al., 2016).

Results

Participants' Behavioral Data Analysis

The collected data from the pre- and post-performance questionnaires aimed to reveal:

(1) whether the participants had a prior knowledge or experience using a Brain-Computer Interface device (Table 2) that could possibly influence their participation in the study;

(2) whether the participants were able to identify when and how their brain-activity was controlling the live video projections (Table 2); and

(3) what were the most special elements of the performance (Table 3).

More specifically, in the frame of the pre-performance preliminary questionnaires the participants were asked: Do you have a prior knowledge or experience of using a Brain-Computer Interface device? If yes, please give more information i.e. the type/model or device used, the mental tasks performed etc. The majority of the audience participants (83.33%) replied that they had no prior experience. One audience participant (16.67%) replied that he previously tried out for a few minutes a similar device, as part of a demo presented in a conference, but he didn't know anything about the device. Whereas, the performer participant replied that her only prior experience has been wearing the BCI device during the rehearsals of the currently discussed study (Table 2). Therefore, it is concluded that the participants' prior knowledge did not influence their participation in the study and their responses to the post-performance experience questionnaires.

TABLE 2

Table 2. Quantitative analysis of participants' answers to pre- and post-performance questions.

TABLE 3

Table 3. Descriptive analysis of participants' answers to post-performance questions.

Additionally, the participants were asked after the performance: Did you think/understand that your brain-activity was interacting with the audio and videos during the performance? If yes, when? And how? As described in section EEG Data Acquisition, in each one of the three events, the brain-activity of both the audience participants was being recorded and processed, but one of them was manually selected in order to generate the RGB colour filter during scene 4 and scene 5. Additionally, the brain-activity of the performer participant was generating the RGB colour filter during scene 1–3 and then jointly with the audience during scene 5. Based on this, the answers of the participants can be categorised as true positive, true negative, false positive, and false negative. As demonstrated more specifically in Table 2, the majority of the audience participants (66.67%) were able to successfully identify whether their brain-activity was interacting with the live visuals (true positive answers) or not (true negative answers). Whereas, one audience participant gave a false positive answer (16.67%) and one was not able to determine (16.67%). The performer participant was 66.67% successful in identifying that her brain-activity was interacting with the live visuals (true positive answers), compared to 33.33% of false positive answers.

In the frame of the post-performance questionnaires the participants were asked what they did during the study, in order to verify that they followed the instructions for participation in the study (Table 3). All the audience participants replied that they watched the performance and the actress replied that she performed, as expected. Additionally, two audience participants reported that they were concentrating on reading the supertitles. The participants were also asked whether they used any specific strategy related to mental tasks, in order to identify any particular factor that might have influenced their cognitive state during the study (Table 3). The majority, the actress in all three events and four out of the six audience participants, did not report any specific strategy. Interestingly, two of the audience participants mentioned that they were consciously staying engaged, one that was focusing especially in part 2 “You” and one that was imagining pleasant activities. None of these though are considered to have any particular influence in their cognitive states during the study.

Another post-performance question that offered valuable insight and feedback was whether there was something in the performance that made a special impression on the participants. Apart from the experience as a whole for both the actress, as well as the audience, the most highlighted elements also include the live visuals and the colours, part 2 “You/We” of the performance, the use of different languages and the “moving” texts. Other elements include the feeling of being “connected” through the colours and the sounds and the physical control of the performer.

Additionally, the participants that were able to successfully identify that their brain-activity was interacting with the live video projections (Table 2) in the questions If yes, when? And How?, they highlighted as main factors the changing colours of the visuals, part 2 “You/We” and the explanatory vignettes. Other factors include the use of lights, whereas the actress reported that she was able to understand that her brain-activity was interacting with the video projections when she was not focusing on acting and was able to pay more attention to the environment.

Participants' EEG Data Analysis and Statistical Methods

The participants' raw EEG data were also collected, the quantitative and statistical analysis of which most interestingly revealed a correlation with their answers to the questionnaires. The raw EEG data from all participants were processed offline in the MATLAB R2016b software (MathWorks, 2016) with the EEGLAB 13.6.5b interactive toolbox (Swartz Center of Computational Neuroscience University of California San Diego, 2016) and the IBM SPSS software [IBM (no date)]. During the process a series of challenges and issues have been encountered, mainly due to the particularly long recorded datasets and their resulting increased size. The first challenge was the required increased computational power in order to perform the analysis. In order to address this, the datasets were down-sampled from 512 to 256 Hz rate. Nevertheless, this did not have an effect on the quality of the analysis, since according to Nyquist theorem a sufficient sampling rate is equal to 2 times of the highest frequency of the data (Baraniuk, 2007), which in the case of the currently discussed study is 40 Hz. Following the down-sampling, the datasets were filtered using a windowed sinc Finite Impulse Response (FIR) filter, with cutoff frequency 4–40 Hz, bandpass filter type blackman window type and transition bandwidth 1 (Smith, 1997). Additionally, all the recorded data from all three events included a time-period at the beginning prior to the performance. These initial data were removed and saved, then used in the analysis as the participants' brain-activity baseline, in order to compare it with their brain-activity during the performance.

Following the processing of the datasets, group studies were created in order to plot and analyse in comparison:

(1) the 4–40 Hz Signal Potential (μV) of each audience participant in the Time domain (ms) during the baseline, the overall performance (Figure 2), part 1 “Me” (scenes 1–3), part 2 “You/We” scene 4, and part 2 “You/We” scene 5;

(2) the Power Spectral Density, which describes the distribution of power, in the 4–40 Hz frequency range of each audience participant during the baseline, the overall performance, part 1 “Me” (scenes 1–3), part 2 “You/We” scene 4, and part 2 “You/We” scene 5 (Figures 3–5);

(3) the Power Spectral Density for the 4–40 Hz frequency range of the performer participant during the baseline and the overall performance (Figure 6). The recorded EEG data from the first and third performance had to be excluded, due to the repetitive disconnections that occurred in the transmission of the data from the BCI device to the computer. For similar reasons the scenes 1–3, 4, and 5 were not examined individually from the overall performance, as in the case of the audience participants.

FIGURE 2

Figure 2. The plotted 4–40 Hz Signal Potential (μV) in the Time Domain (ms) of the audience participants during the overall performance of each performance.

FIGURE 3

Figure 3. The plotted Power Spectral Density for the 4–40 Hz frequency range of the audience participants 1 and 2 during the baseline, the overall performance, scenes 1–3, scene 4, and scene 5 of the first performance.

FIGURE 4

Figure 4. The plotted Power Spectral Density for the 4–40 Hz frequency range of the audience participants 3 and 4 during the baseline, the overall performance, scenes 1–3, scene 4, and scene 5 of the second performance.

FIGURE 5

Figure 5. The plotted Power Spectral Density for the 4–40 Hz frequency range of the audience participants 5 and 6 during the baseline, the overall performance, scenes 1–3, scene 4, and scene 5 of the third performance.

FIGURE 6

Figure 6. The Power Spectral Density for the 4–40 Hz frequency range of the performer participant during the baseline and the overall performance of one rehearsal and the second performance.

Audience Participants' Power Spectral Density Intra-Subject Variability

Although the EEGLAB interactive toolbox offers automatic functions for calculating a range of statistics for the plotted data, these did not seem to reliably work in the case of the currently discussed study and for this reason manual calculations were performed. More specifically, the datapoints' values from each individual plot were extracted and the statistical analysis was performed in combination with the Microsoft Excel, MATLAB R2016b and IBM SPSS software.

The analysis, and more specifically the intra-subject variability, of the Power Spectral Density of the audience participants across all performances has shown that their Alpha power (8–13 Hz) tended in most cases to decrease during the performance compared to the baseline (pre-performance time-period) and mostly during the scene 4 of part 2 “You/We.” This is when the actress leaves the stage, two soft spots above the two audience participants are turned on and their brain-activities are interacting with the video projections. The decrease of the Alpha power has been associated by previous research (O'Connell et al., 2009) with the increase of the cognitive function of attention. Therefore, the results of the audience participants indicate that not only were they more attentive, as expected, during the performance compared to the baseline period, but particularly during part 2 “You/We.” Whereas later on (scene 5) the actress reappears on stage, addressing directly the audience, while their real-time brain-interaction gradually merges and their averaged values are controlling the colour filter applied to the live visuals (Zioga et al., 2016).

The intra-subject variability of the Power Spectral Density of the audience participants across all performances has also shown that their Lower Gamma power (25–40 Hz) tended in most cases to increase during scene 5 of part 2 “You/We” of the performance, especially compared to the overall performance. The increase of the Lower Gamma power has been associated by previous research with increased emotional engagement (Müller et al., 1999) and emotional facial information processing (Balconi and Lucchiari, 2008). Therefore, the results of the audience participants indicate that they were more emotionally engaged during scene 5 of part 2 “You/We,” when the performer was addressing directly the audience citing “moving” fragments from On Being Ill by Virginia Woolf (2002), while their brain-activity was jointly interacting with the video projections.

Furthermore, a one-way Analysis of Variance (ANOVA) with repeated measures and Bonferroni post-hoc test was performed, in order to compare the mean variability of each frequency band between the baseline period, the overall performance, scenes 1–3, scene 4, and scene 5, for each audience participant. A one-way ANOVA is the statistical method of choice, when the aim is to compare groups of three or more means at the same time and identify whether they differ significantly (Lowry, 1999). Whereas, post-hoc tests and pairwise comparisons are used to determine which means differ the most. The analysis confirmed that there were significant between the means differences in the Alpha Power Spectral Density [F_{(4, 20)} = 3.422 and p < 0.05], with a greater decrease occurring during scene 4 compared to the baseline period. The analysis also confirmed that there were significant between the means differences in the Lower Gamma Power Spectral Density [F_{(4, 20)} = 2.899 and p < 0.05] with a significant increase during scene 5 compared to the overall performance.

Audience Participants' Inter-subject Time-Frequency Correlation Analysis

Additionally, an inter-subject 4–40 Hz time-frequency correlation analysis was performed between the audience participants of each performance. As also explained at the beginning of this section, during the processing and analysis of the data a series of challenges have been encountered. In the case of the correlation analysis the process was not possible to be completed inside the IBM SPSS software due to particular large amount—up to 725,505—datapoints. In order to overcome this problem, the datapoints' values from each individual plot were extracted and the MATLAB R2016b software was used in order to calculate the Spearman's rank correlation coefficient. The results showed that in all three performances, the correlation, ρ (rho) value, between the audience participants was greater and significant (p < 0.05) or highly significant (p < 0.001) during scene 4 of part 2 “You/We” when also their attention was increased (Table 4).

TABLE 4

Table 4. Inter-subject time-frequency correlation analysis for the audience participants' 4–40 Hz.

Performer Participant's Power Spectral Density Intra-Subject Variability

As in the case of the audience participants' data processing and analysis, the available automatic functions of the EEGLAB interactive toolbox for calculating a range of statistics did not seem to reliably work also with the performer participant's plotted data and for this reason manual calculations were performed. Similarly, the datapoints' values from each individual plot were extracted and the statistical analysis was performed in combination with the Microsoft Excel, MATLAB R2016b and IBM SPSS software. Also as mentioned previously, the EEG data from the first and third performance had to be excluded, due to the repetitive disconnections that occurred in the transmission of the data from the BCI device to the computer. For similar reasons the scenes 1–3, 4, and 5 were not examined individually from the overall performance, as in the case of the audience participants. However, recordings were obtained during the performer's rehearsals and these were used in the analysis.

The analysis of the intra-subject variability of the 4–40 Hz Power Spectral Density of the performer participant during one rehearsal and the second performance has shown that the Theta power (4–8 Hz) and Lower Gamma power (25–40 Hz) tended to increase significantly during the overall performance compared to the baseline period. The increase of the Theta power, and more specifically over frontal brain areas, has been associated by previous research with memory encoding (Klimesch, 1999), the active maintenance and recall of working memory representations, and the increase of “memory load in a working memory task” (Jensen and Tesche, 2002). At the same time, Gamma power has also been associated by previous research with directed attention and maintenance of working memory (Howard et al., 2003; Jensen et al., 2007).

Discussion

Summary and Interpretation of Findings

The analysis of the participants' data and the comparison of the results reveal a correlation between their answers to the questionnaires and the EEG data. More specifically, the majority of the audience participants and the performer participant across the majority of the events were able to successfully identify whether their brain-activity was interacting with the live visuals or not, and highlighted as main factors the changing colours of the visuals, part 2 “You/We” of the performance, the explanatory vignettes and the dramaturgical use of lights. At the same time, apart from the experience as a whole, the most highlighted elements that made a special impression on them include again the live visuals and the colours, part 2 “You/We” of the performance, the use of different languages and also the “moving” texts.

These results are further reinforced in comparison with the intra-subject variability of the EEG Power Spectral Density of the audience participants across all performances, confirmed also by a one-way ANOVA analysis. One the one hand, it has shown that their Alpha power (8–13 Hz) tended to decrease and therefore were more attentive during scene 4 of part 2 “You/We,” when their brain-activity was interacting with the video projections without the presence of the actress. On the other hand, it has also shown that their Lower Gamma power (25–40 Hz) tended to increase during scene 5 of part 2 “You/We,” which means that they were more emotionally engaged, while they were processing the actress' emotional facial information, when she was directly addressing them and citing “moving” texts. Additionally, the inter-subject 4–40 Hz time-frequency correlation analysis showed that the correlation between the audience participants was greater and significant or highly significant during scene 4 of part 2 “You/We.”

Furthermore, the evidenced relationship between the participants' BCI interaction awareness, the elements of special impression on them and their cognitive state during scene 4 and 5 of part 2 “You/We” can be compared to findings from studies that investigate the effect of films on the spectators' brain activity, searching for similarities in their spatiotemporal responses (Hasson et al., 2008). For example, in a study by Dmochowski et al. (2012), the results revealed peak inter-subject correlation of neural activity during arousing moments of a film, which according to the authors “reflects attention- and emotion-modulated cortical processing.” In a similar investigation in a real-life context by Jola et al. (2011), the audience participants' cortical excitability was measured while watching “a dress rehearsal of a commercial production of Sleeping Beauty, lasting 2.5 h, performed by the Scottish Ballet.” By cortical excitability the authors refer to the motor empathy of the participant's while watching the performance of the dancers. As they observed, the participants' responses “were strongly individual” and their “cortical excitability decreased with time,” which could be due to different reasons, such as the long duration of the play, but nevertheless highlights the importance of the use of time in a dance performance. Relevant observations, such as “fatigue and distraction” (Jin et al., 2011), are also commonly observed in BCI studies with long duration, while researchers are currently investigating new experimental paradigms that will help address these issues. Following this thread, we argue that the results of the currently discussed study, more in particular the increase of the factors associated with the audience participants' attention, emotional engagement and facial processing during the last two scenes of the performance, serve as a strong evidence of the importance of the directing strategy, dramaturgy, and narrative structure. These methods and strategies can make effective use of the performance time and lead the audience's perception and cognitive state during the different stages of the work.

Last but not least, the analysis of the intra-subject variability of the 4–40 Hz Power Spectral Density of the performer participant has shown that the Theta power (4–8 Hz) and Lower Gamma power (25–40 Hz) tended to increase significantly during the overall performance compared to the baseline period, which is consistent with the recall of working memory representations and the increase of cognitive load.

Limitations

One of the important limitations in “Enheduanna—A Manifesto of Falling” Live Brain-Computer Cinema Performance, as in the case of other similar studies (Jola et al., 2011; Eaton et al., 2015), has been the data collection in terms of quantity. Due to the demanding logistics of the events (allocation of space, costs etc.), it has not been made possible to repeat the performance several times nor include a large number of audience participants in each event. With six audience participants and one performer participant one can argue that the sample of the data is non-representative. While at the same time, the comparison between participants of different events is difficult, since the duration of live events, no matter how well coordinated they are, can be even slightly different due to possible delays or differences in the transition from scene to scene. However, the results not only provide strong indications and insight on the perception, cognition, and engagement of spectators and performers during a live mixed-media performance, but they can also inform future development and encourage similar studies in real-life settings.

The Live Brain-Computer Cinema Performance as a Neuroscientific Experiment in a Real-Life Context and Future Work

Enheduanna—A Manifesto of Falling (Zioga and Katsinavaki, 2015), the currently discussed live brain-computer cinema performance, is a new format of interactive work that combines live cinema and the use of BCIs, but also a neuroscientific experiment in a real-life context. As mentioned in section Introduction, in recent years in the fields of neuroscience and experimental psychology has emerged a new and increasing interest in studying the mechanisms, dynamics and processes of the interaction between multiple subjects and their brain-activity and even more in a real-life setting, away from the lab. Although until recently the majority of the scientific research was realised by presenting to the participants small fragments of audio-visual material and employing event-related designs, it is now acknowledged that in natural settings the brain-activity is rather continuous and transient (Dmochowski et al., 2012). While, recent studies are already using new calculation methods, such as the “practical bit rate” vs. the “raw bit rate,” in order to “estimate the speed” of BCI systems in “a reasonable world setting” (Jin et al., 2011). For this reason, the need of studies in real-life contexts has also been identified (Jin et al., 2014), including free viewing of films and live performances. As in the study by Jola et al. (2011), the currently discussed live brain-computer cinema performance makes a claim about live experiences and experiments outside the laboratory and contributes on new hypotheses about the effects of the length of time, but also the role of the directing strategy, dramaturgy and narrative structure on the audience's perception, cognitive state, and engagement. At the same time, it informs other relevant discussions of the use of BCIs as query tools in the frame of Practice as Research (PaR) (Aparicio and Cadiz, 2017).

The results of the EEG data analysis can also lead to further investigations. More specifically, new studies with increased number of audience participants could provide the initial results with statistical significance. Whereas, additional studies could look into the effect of real-life conditions, like in scene 5, when the performer by directly addressing the audience participants probed the increase of their emotional engagement and emotional facial information processing. For the above reasons, we believe that this study is a step toward gaining greater understanding of neuromodulation in performers and audiences alike and its potential applications; but it also serves as evidence that interdisciplinary research not only can contribute to the advancement of the different fields involved, but can also result in new observations, not possible to be made in isolation.

Author Contributions

PZ: conceptualised and conducted the study, acquired the data and wrote the manuscript; PZ, FP, PC, and MM: contributed to the methodological design of the study; PZ, KS, and FP: carried out the data analysis and the interpretation of the results. All authors contributed to the critical revision and editing of the manuscript.

Funding

This work was supported by the NEON Organization under the 2014–2015 Grant for Performance Production; and the Glasgow School of Art under the Global Excellence Initiative Fund Ph.D. Studentship.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This article is issued from the Ph.D. Thesis of PZ (Zioga, 2017). Enheduanna—A Manifesto of Falling Live Brain-Computer Cinema Performance has been developed in the School of Simulation and Visualisation (formerly known as Digital Design Studio) of the Glasgow School of Art (GSA). The data analysis was performed in the Perception Action Cognition (PACo) Lab at the School of Psychology of the University of Glasgow. The project has been awarded with the NEON Organization 2014–2015 Grant for Performance Production and is supported by MyndPlay. We are indebted to Dr. Francis McKee, who warmly provided the support of CCA: Centre for Contemporary Arts Glasgow, while the premiere presentations were made possible also with the support of the GSA Students' Association. We would like to heartily thank all the collaborators of the project; the members of the audience who volunteered to participate in the study; Salim Al-Wasity for his valuable advice on the EEG data analysis; and Emeritus Prof Basil N. Ziogas for his continued support. We would also like to thank the reviewers and the editor for their valuable feedback and comments.

References

Aparicio, A., and Cadiz, R. F. (2017). “Wheels within wheels: brain-computer interfaces as tools for artistic practice as research,” in Augmented Cognition. Enhancing Cognition and Behavior in Complex Human Environments: 11th International Conference, AC 2017, Held as Part of HCI International 2017, July 9–14, 2017, Proceedings, Part II, eds D. D. Schmorrow and C. M. Fidopiastis, (Vancouver, BC: Springer International Publishing), 266–281.

Google Scholar

Balconi, M., and Lucchiari, C. (2008). Consciousness and arousal effects on emotional face processing as revealed by brain oscillations. A gamma band analysis. Int. J. Psychophysiol. 67, 41–46. doi: 10.1016/j.ijpsycho.2007.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Bamdad, M., Zarshenas, H., and Auais, M. A. (2015). Application of BCI systems in neurorehabilitation: a scoping review. Disabil. Rehabil. Assist. Technol. 10, 355–364. doi: 10.3109/17483107.2014.961569

PubMed Abstract | CrossRef Full Text | Google Scholar

Baraniuk, R. G. (2007). Compressive sensing. IEEE Sign. Proc. Mag. 24, 118–121. doi: 10.1109/MSP.2007.4286571

CrossRef Full Text | Google Scholar

Belk, R. W. (1966). “Art versus science as ways of generating knowledge about materialism,” in Methodological Innovations in Consumer Research, 1986, eds D. Brinberg and R. Lutz (New York, NY: Springer-Verlag), 3-36.

Google Scholar

Dmochowski, J. P., Sajda, P., Dias, J., and Parra, L. C. (2012). Correlated components of ongoing EEG point to emotionally laden attention – a possible marker of engagement? Front. Hum. Neurosci. 6:112. doi: 10.3389/fnhum.2012.00112

PubMed Abstract | CrossRef Full Text | Google Scholar

Eaton, J., Williams, D., and Miranda, E. (2015). The space between us: evaluating a multi-user affective brain-computer music interface. Brain Comput. Interface 2, 103–116. doi: 10.1080/2326263X.2015.1101922

CrossRef Full Text | Google Scholar

Hasson, U., Landesman, O., Knappmeyer, B., Vallines, I., Rubin, N., and Heeger, D. (2008). Neurocinematics: the neuroscience of film. Projections 2, 1–26. doi: 10.3167/proj.2008.020102

CrossRef Full Text | Google Scholar

He, B., Gao, S., Yuan, H., and Wolpaw, J. R. (2013). “Brain-computer interfaces,” in Neural Engineering, ed B. He (New York, NY: Springer Science+Business Media), 87-151.

Google Scholar

Howard, M. W., Rizzuto, D. S., Caplan, J. B., Madsen, J. R., Lisman, J., Aschenbrenner-Scheibe, R., et al. (2003). Gamma oscillations correlate with working memory load in humans. Cereb. Cortex 13, 1369–1374. doi: 10.1093/cercor/bhg084

PubMed Abstract | CrossRef Full Text | Google Scholar

IBM (no date). IBM SPSS Software. Available online at: http://www.ibm.com/analytics/us/en/technology/spss (Accessed December 14, 2016).,

Jensen, O., Kaiser, J., and Lachaux, J. P. (2007). Human gamma-frequenct oscillations associated with attention and memory. Trends Neurosci. 30, 317–324. doi: 10.1016/j.tins.2007.05.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Jensen, O., and Tesche, C. D. (2002). Frontal theta activity in humans increases with memory load in a working memory task. Euro. J. Neurosci. 15, 1395–1399. doi: 10.1046/j.1460-9568.2002.01975.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Jin, J., Allison, B. Z., Sellers, E. W., Brunner, C., Horki, P., Wang, X., et al. (2011). An adaptive P300-based control system. J. Neural Eng. 8:036006. doi: 10.1088/1741-2560/8/3/036006

PubMed Abstract | CrossRef Full Text | Google Scholar

Jin, J., Allison, B. Z., Zhang, Y., Wang, X., and Cichocki, A. (2014). An ERP-based BCI using an oddball paradigm with different faces and reduced errors in critical functions. Int. J. Neural Syst. 24:1450027. doi: 10.1142/S0129065714500270

CrossRef Full Text | Google Scholar

Jola, C., Pollick, F. E., and Grosbras, M.-H. (2011). Arousal decrease in sleeping beauty: audiences' neurophysiological correlates to watching a narrative dance performance of two-and-a-half hours. Dance Res. 29, 378–403. doi: 10.3366/drs.2011.0025

CrossRef Full Text | Google Scholar

Klimesch, W. (1999). EEG alpha and theta oscillations reflect cognitive and memory performance: a review and analysis. Brain Res. Rev. 29, 169–195. doi: 10.1016/S0165-0173(98)00056-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Lowry, R. (1999). Chapter 13. Conceptual Introduction to the Analysis of Variance. Available online at: http://vassarstats.net/textbook/index.html (Accessed February 12, 2017).

MathWorks (2016). MATLAB. Available online at: https://uk.mathworks.com/products/matlab.html?s_tid=srchtitle (Accessed December 11, 2016)

Müller, M. M., Keil, A., Gruber, T., and Elbert, T. (1999). Processing of affective pictures modulates right-hemispheric gamma band EEG activity. Clin. Neurophysiol. 110, 1913–1920.

PubMed Abstract | Google Scholar

Nijholt, A. (2015). “Competing and collaborating brains: multi-brain computer interfacing,” in Brain-Computer Interfaces: Current trends and Applications, Intelligent Systems Reference Library Series, 74, eds A. E. Hassanien and A. T. Azar (Cham: Springer International Publishing), 313–335. doi: 10.1007/978-3-319-10978-7_12

CrossRef Full Text | Google Scholar

O'Connell, R. G., Dockree, P. M., Robertson, I. H., Bellgrove, M. A., Foxe, J. J., and Kelly, S. P. (2009). Uncovering the neural signature of lapsing attention: electrophysiological signals predict errors up to 20 s before they occur. J. Neurosci. 29, 8604–8611. doi: 10.1523/JNEUROSCI.5967-08.2009

PubMed Abstract | CrossRef Full Text | Google Scholar

Oldfield, R. C. (1971). The assessment and analysis of handedness: the Edinburgh inventory. Neuropsychololgia 9, 97–113. doi: 10.1016/0028-3932(71)90067-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Renard, Y., Lotte, F., Gibert, G., Congedo, M., Maby, E., Delannoy, B., et al. (2010). OpenViBE: an open-source software platform to design, test and use brain-computer interfaces in real and virtual environments. Presence Teleop. Virt. Environ. 19, 35–53. doi: 10.1162/pres.19.1.35

CrossRef Full Text | Google Scholar

Smith, S. W. (1997). The Scientist and Engineer's Guide to Digital Signal Processing. San Diego, CA: Technical Publishing.

Google Scholar

Swartz Center of Computational Neuroscience University of California San Diego (2016). What is EEGLAB? Available online at: https://sccn.ucsd.edu/eeglab (Accessed December 11, 2016).

Wolpaw, J. R., and Wolpaw, E. W. (2012). “Brain-computer interfaces: something new under the sun,” in Brain-Computer Interfaces: Principles and Practice eds J. R. Wolpaw and E. W. Wolpaw (Oxford: Oxford University Press), 3–15.

Google Scholar

Woolf, V. (2002). University Press On Being Ill. Ashfield: Paris Press.

Zander, T. O., Kothe, C., Welke, S., and Roetting, M. (2008). “Enhancing human-machine systems with secondary input from passive brain-computer interfaces,” in Proceedings of the 4th International BCI Workshop & Training Course, Graz, Austria (Graz: Graz University of Technology Publishing House).

Zioga, P. (2017). Live Brain-Computer Cinema Performance. Ph.D. thesis, The Glasgow School of Art, Glasgow.

Zioga, P., Chapman, P., Ma, M., and Pollick, F. (2015). “A hypothesis of brain-to-brain coupling in interactive new media art and games using brain-computer interfaces,” in 2015 Serious Games First Joint International Conference, JCSG 2015, Huddersfield, UK, June 3–4, 2015, Proceedings. LNCS 9090, eds S. Göbel, M. Ma, J. Baalsrud Hauge, M. F. Oliveira, J. Wiemeyer, and V. Wendel (Berlin; Heidelberg: Springer-Verlag), 103–113.

Google Scholar

Zioga, P., Chapman, P., Ma, M., and Pollick, F. (2016). Enheduanna – a manifesto of falling: first demonstration of a live brain-computer cinema performance with multi-brain bci interaction for one performer and two audience members. Digit. Creativ. 28, 103–122. doi: 10.1080/14626268.2016.1260593

CrossRef Full Text | Google Scholar

Zioga, P., and Katsinavaki, A. (2015). ‘Enheduanna–A Manifesto of Falling’ Live Brain-Computer Cinema Performance. Polina Zioga. dir. Glasgow: CCA: Centre for Contemporary Arts. (Accessed July 29, 2015).