Mental Effort When Playing, Listening, and Imagining Music in One Pianist’s Eyes and Brain

Endestad, Tor; Godøy, Rolf Inge; Sneve, Markus Handal; Hagen, Thomas; Bochynska, Agata; Laeng, Bruno

doi:10.3389/fnhum.2020.576888

ORIGINAL RESEARCH article

Front. Hum. Neurosci., 15 October 2020

Sec. Cognitive Neuroscience

Volume 14 - 2020 | https://doi.org/10.3389/fnhum.2020.576888

This article is part of the Research Topic The Effects of Music on Cognition and Action View all 36 articles

Mental Effort When Playing, Listening, and Imagining Music in One Pianist’s Eyes and Brain

$\r\nTor Endestad,,$ Tor Endestad^1,2,3

Rolf Inge Godøy²

Markus Handal Sneve¹

Thomas Hagen¹

Agata Bochynska^1,4

Bruno Laeng^1,2*

¹Department of Psychology, University of Oslo, Oslo, Norway
²RITMO Centre for Interdisciplinary Studies in Rhythm, Time and Motion, University of Oslo, Oslo, Norway
³Helgelandssykehuset, Mosjøen, Norway
⁴Department of Psychology, New York University, New York, NY, United States

We investigated “musical effort” with an internationally renowned, classical, pianist while playing, listening, and imagining music. We used pupillometry as an objective measure of mental effort and fMRI as an exploratory method of effort with the same musical pieces. We also compared a group of non-professional pianists and non-musicians by the use of pupillometry and a small group of non-musicians with fMRI. This combined approach of psychophysiology and neuroimaging revealed the cognitive work during different musical activities. We found that pupil diameters were largest when “playing” (regardless of whether there was sound produced or not) compared to conditions with no movement (i.e., “listening” and “imagery”). We found positive correlations between pupil diameters of the professional pianist during different conditions with the same piano piece (i.e., normal playing, silenced playing, listen, imagining), which might indicate similar degrees of load on cognitive resources as well as an intimate link between the motor imagery of sound-producing body motions and gestures. We also confirmed that musical imagery had a strong commonality with music listening in both pianists and musically naïve individuals. Neuroimaging provided evidence for a relationship between noradrenergic (NE) activity and mental workload or attentional intensity within the domain of music cognition. We found effort related activity in the superior part of the locus coeruleus (LC) and, similarly to the pupil, the listening and imagery engaged less the LC–NE network than the motor condition. The pianists attended more intensively to the most difficult piece than the non-musicians since they showed larger pupils for the most difficult piece. Non-musicians were the most engaged by the music listening task, suggesting that the amount of attention allocated for the same task may follow a hierarchy of expertise demanding less attentional effort in expert or performers than in novices. In the professional pianist, we found only weak evidence for a commonality between subjective effort (as rated measure-by-measure) and the objective effort gauged with pupil diameter during listening. We suggest that psychophysiological methods like pupillometry can index mental effort in a manner that is not available to subjective awareness or introspection.

Introduction

Musical Effort

The piano is not an easy instrument to master. It requires a lifetime of extended practice and early (possibly during childhood) onset of training to achieve a high level of performance. Well-known musical pieces in either classical music or jazz are technically particularly challenging, and they constitute a paradigmatic example of intense practice of cognitive executive functions (Montero, 2016). There is the need of dividing attention over the control of complex, synchronized, sequences of finger positions and movements (Mikumo, 1994) and paying attention to the obtained auditory stimuli. Such a fine and precise motor control of hands and fingers require cognitive and motor control, guiding constant adjustments of bodily actions for both the execution and preparation of the following movements during the different passages of a musical piece, also depending on the level of cognitive and motoric demands in performing the movements. Hence, we assume that playing the piano—or any musical instrument professionally—requires mental resources or “mental effort,” as Kahneman (1973) labeled it originally. Such a type of neurocognitive effort is distinguishable from physical effort, though it has clear analogies with it.

Surprisingly, research on “musical effort” or the broad process of allocation of attentional resources during music performance or listening (Keller, 2001; Shenhav et al., 2017) has been so far an understudied aspect of music cognition. In cognitive psychology, pupillometry or the measurement of pupil dilations during a task has been considered the best psychophysiological measure (Kahneman et al., 1969) of the “intensive aspect of attention” or cognitive workload (e.g., Hess and Polt, 1964; Beatty, 1982; Just and Carpenter, 1993). Changes in pupil diameter are not simply evoked by changes in light stimulating the eye, and changes related to mental processing, albeit tiny compared to those provoked by light, are separable. Most importantly, these pupillary changes provide a reliable, “honest” (i.e., difficult to affect voluntarily; see Laeng and Sulutvedt, 2014), and valid measure of the overall aggregate of resource demand and capacity utilization by the brain (Just et al., 2003). A wide variety of pupillometry studies confirm a tight relationship between pupillary dilation and the allocation of attention and load on cognitive resources (for reviews see Laeng et al., 2012; Laeng and Alnæs, 2019). Recently, neurophysiological studies with monkeys (e.g., Joshi et al., 2016) and neuroimaging in humans have indicated the involvement of the noradrenergic (NE) brainstem structure called the locus coeruleus (LC) in the control of pupil size during cognitive work (e.g., Alnæs et al., 2014; Murphy et al., 2014; Mäki-Marttunen et al., 2019). Both pupil diameter and the LC have been known as a physiological measure and structure involved in both cognitive and affective “arousal” (e.g., Nunnally et al., 1967; Libby et al., 1973; Nassar et al., 2012; Sara and Bouret, 2012).

Notably, even just music listening triggers high arousal (e.g., Gingras et al., 2015; Laeng et al., 2016; Weiss et al., 2016; Bowling et al., 2019). To our knowledge, no studies have applied eye pupillometry to the study of “musical effort” during the performance. Instead, there have been a few pupillometry studies on mental effort during the perceptual processing of auditory stimuli like speech (e.g., Zekveld et al., 2014) and music (e.g., Kang and Wheatley, 2015; O’Shea and Moran, 2016; Liao et al., 2018). Moreover, it remains unclear whether the cognitive mechanisms engaged during music listening are equally engaged during musical auditory images (e.g., when a musician “plays in her head” a musical piece) than with music perception or when listening to the same musical piece played on a real instrument. Research within cognitive psychology had mainly focused on visual imagery (e.g., Kosslyn, 1980, 1994), which has overall confirmed a strong overlap of the cognitive mechanisms engaged by both imagery and perception.

Musical Imagery

Musical imagery consists of actively evoking and maintaining sound “images” of music in our minds, like mental imagery in other specific modalities (Godøy and Jørgensen, 2001). However, musical imagery would seem to be a process at the very heart of not only music-making but also listening, as an ongoing anticipatory activity (e.g., Janata, 2001; Leaver et al., 2009; Gracyk, 2019). An interest in the topic has been rekindled within the last 20 years by several neuroimaging studies investigating the effects of music perception and imagery in the brains of both musicians and non-musicians (e.g., Zatorre, 1999; Lotze et al., 2003; Herholz et al., 2008). Recently, most references to musical imagery have focused on “involuntary musical imagery” (IMI, sometimes called “earworms,” “sticky music,” “catchy tunes,” or simply “hooks”; e.g., Sacks, 2007; Farrugia et al., 2015; Williams, 2015; Moseley et al., 2018). However, in this article, we will focus on the “volitional” or active type of imagery of musical sounds and report a psychophysiological plus neuroimaging study on imagery, listening, and performance by a professional, internationally renowned, pianist.

Musical Imagery as a Multimodal Experience

Importantly, the phenomenology of music-making and listening is not just related to sounds or finger movements but it is a multisensory or multimodal experience (Vines et al., 2006; Godøy, 2010b; Zimmerman and Lahav, 2012; Fine et al., 2015). The musical imagery of musicians seems particularly embedded in complex action plans, co-articulated executive programs, and predictions of the effects of actions both within the body and in the environment (e.g., Reybrouck, 2001; Wolpert and Flanagan, 2001). Musicians seem capable of constructing “sonic images” when playing, based on timbral and timing features (Wöllner and Williamon, 2007), allowing them to anticipate sonic actions and even perform without auditory feedback (e.g., with sound is switched off). However, also non-musicians appear to have an intuitive and coarsely correct understanding of the visual features linked to playing a particular musical instrument and of its related specific gestures or “gestural affordances” (Godøy and Leman, 2010; Godøy, 2010a); as also testified by the amusing ability of non-musicians in “playing” virtual instruments (e.g., “air-guitar”; see Godøy, 2001; Godøy et al., 2006). Just the sight of the musicians’ gestures (with no sound) can influence our understanding and evaluation of music (e.g., Platz and Kopiez, 2012; Tsay, 2013) and it also triggers activity in the auditory cortex of the observers’ brain (Haslinger et al., 2005), probably indicating that the sight of music-making evokes in turn spontaneous musical (auditory) imagery.

Although we can in principle study in isolation each modality-specific type, given the underlying modularity of the sensory and motor system, there is likely to be a great interconnectedness between processes and structures of the brain for musical imagery. The current picture is that the auditory type of imagery is active together with the motor imagery of the sound-producing body motions. Moreover, the motor imagery seems imbued with somesthetic images, characteristic of the kinesthetic feedback that such bodily motions would typically produce (Betts, 1909; Hubbard and Stoeckig, 1988; Reisberg, 1992; Hodges, 2009; Hubbard, 2010, 2013, 2017, 2018; Vlek et al., 2011; Vuvan and Schmuckler, 2011). Indeed, perception, in general, is “active” and makes use of the motor system to achieve perceptual categorization or, at least, to facilitate it (e.g., Liberman et al., 1967; Liberman and Mattingly, 1985; Smith et al., 1995; Gallese and Metzinger, 2003; Gallese and Lakoff, 2005; Galantucci et al., 2006; Glenberg and Gallese, 2012). Embodied cognition accounts give prominence to action and behavior for perceptual processes of all kinds (e.g., Varela et al., 1991; Gallagher, 2005), including especially music (e.g., Cox, 2016). These accounts have influenced music psychology (Leman, 2008; Peñalba, 2011; Schiavio et al., 2014; Korsakova-Kreyn, 2018; Bailes, 2019). We believe that studying volitional musical imagery is important because it could be the gateway to the more systematic exploitation of musical imagery in practical tasks such as composition, improvisation, and performance (Christensen, 2019).

Musical “Audiation”

A remarkable phenomenon of musical imagery is the so-called notational “audiation” often reported by professional musicians; that is, the simple act of reading a musical score evokes auditory imagery of the music in reading’s real-time (e.g., Schürmann et al., 2002; Battisti, 2007; Brodsky et al., 2003, 2008). Remarkably, the electrophysiological activity from the brain of trained musicians during note reading or the actual perception of notes is undistinguishable (Simoens and Tervaniemi, 2013). The ability to “hear” with the “mind”s ear’ or “thinking in sounds” (Combarieu, 1907) and “replay” virtual music with the “inner voice,” could assist the making of creative compositions. A well-known case of the power of musical imagery in creating music is Ludwig van Beethoven who composed many of is most praised compositions (e.g., the last piano sonatas and string quartets, the Missa Solemnis, and the Ninth Symphony) while he was practically deaf due to an inner ear problem. Presumably, he was perfectly able to compose music because he could “hear the music in his head” (Jourdain, 1997; Zatorre and Halpern, 2005) despite his spared auditory cortex was unable to be stimulated by the actual sounds of musical instruments.

Hearing music in the head appears to be a ubiquitous experience (Cotter, 2019) but the role that mental control or effort plays in these experiences has not been addressed in current research in any thorough manner. That is, both professional musicians and music writers, as well as musically naïve individuals, have the power to start, stop, shape, and maintain in their head musical images unfolding in time (e.g., Zatorre, 1999; Janata, 2001; Cotter, 2019). However, all these mental “actions” are a form of cognitive work and require focused attention; they are likely to draw on mental resources and engage brain systems the control cognitive arousal (Alnæs et al., 2014). Indeed, musical “mental practice” (Coffman, 1990) is well known as an effective tool for enhancing memorization of music and refining performance (e.g., Driskell et al., 1994; Halpern et al., 2004; Highben and Palmer, 2004; Holmes, 2005; Lotze and Halsband, 2006; Cahn, 2008; Gregg et al., 2008; Keller, 2012; Halpern and Overy, 2019). Deliberately imagining music, both the sound-producing actions (e.g., finger movements in pianists) and the resulting musical sounds (Davidson-Kelly et al., 2015), seems almost as common as actually listening to music (respectively 32 and 44% of times when querying musicians; Bailes, 2006, 2015). According to Fine et al. (2015), about 70% of the classical music performers they surveyed state that they use regularly mental practice, and among these about 90% also experience musical imagery in the form of “audiation” (i.e., “hearing music in the head” while reading a score away from the instrument; Bishop et al., 2013). A well-known case of extensive use of the mental practice is that of the classical pianist Glenn Gould who mentally rehearsed a performance, without touching the piano for prolonged periods, leaving the actual testing of the “mechanics” of the finger movements on the piano to just the last period of preparation (Mesaros, 2008). In a seminal study, Repp (1999) compared the timing profiles of six pianists during a live performance and the “imagined performance” and found that the temporal fluctuations occurred in a very similar manner (i.e., they were positively correlated). Other experiments in “mental chronometry” of imagined music have yielded similar results (Wöllner and Williamon, 2007; Clark and Williamon, 2011; Clark et al., 2012).

The Present Study

Because a piano piece can present cognitive challenges that rapidly vary during the music’s stream, we expect that changes in pupillary diameter in the eyes of the pianist mirror the level of required effort as the music unfolds. We expect that the complete trace of pupillary changes will provide a continuous physiological measure of changes in control processes (executive and attentional) occurring in the brain of a high-level professional pianist as she focusses on the piece over time. We also expect that, to some extent, independent listeners would react to the perceived effort inherent in the music, likely more in musicians than in non-musicians. Hence, we also monitored the eye pupil in a “control” group of pianists and non-musicians, while they listened to the same piano renditions of the musical pieces by our professional pianist.

Importantly, the pupillometry method is currently considered not only a reliable gauge of cognitive workload but also as a window into the activity of the brain’s NE arousal system and the involvement of brainstem structures like the LC, involved in the NE control of pupil size (e.g., in other mammals; Joshi et al., 2016). Very few human neuroimaging studies have explored the role of the brainstem’s NE structures and pupillary activity (Alnæs et al., 2014; Murphy et al., 2014; Mäki-Marttunen et al., 2019) and none have related the role of these brainstem’s NE structures to music listening performance, and imagery. Hence, we further explored music listening, playing, and imagery, in the same professional pianist, during these tasks with functional MRI, seeking converging evidence to the engagement of mental effort by brainstem structures. Since musical imagery is a quintessential multisensory experience (auditory, kinesthetic, and visuospatial at least), playing music in one’s head or imagery should reveal the strongest relationship with actual piano playing, since all of the components of music-making that takes place in the mind (brain) would be re-instated, despite the lack of behavioral enactment. Importantly, we expect to reveal converging evidence between the findings obtained with pupillometry (Experiment 1) and those obtained with neuroimaging (Experiment 2). Specifically, since we posit that the pupil index effort-related cognitive arousal, we expect that pupillary changes across conditions should also appear as changes in activity in the LC or the NE center of the brainstem (Alnæs et al., 2014). Moreover, processes that are similar in terms of pupillary activity and highly correlated should also appear similar in terms of cortical activity.

In sum, we hope to offer initial answers about some fundamental questions in music cognition: (a) Can we measure musical effort through the eye pupil similarly to reading out the cognitive workload and arousal in other domains? (b) Does musical imagery engage sensory and motor areas of the brain concerning the mental effort required by the complexity of the structure and execution of the imagined music? (c) Is musical imagery more similar to music listening in non-musicians in terms of neural networks than in musicians where it engages more motor aspects (i.e., it is multimodal)? (d) Can we in general deduce the degree of functional similarity between playing, imagining, and listening by comparing the activity in the NE system (as indexed by the pupil) and/or the overlap between sensory and motor neural networks in the whole brain? and (e) Specifically, can we reveal “audiation” in an expert musician by the similarity in which the pupil changes during musical listening and imagery?

Pupillometry of Playing, Listening, and Imagining Music

Pupillometry measures “objectively” mental effort, but an experienced musician may be able to estimate “subjectively” how effortful a moment can be during the piano execution. Hence, we introduce a distinction between “subjective effort” and “objective effort”. With the former, we mean what a participant judges to be the processing load based on her private experience, even when estimated via ordinal scales (e.g., the NASA-TLX; e.g., Chaffin, 2009). Instead, physiologically driven changes in pupil diameter are an objective measure of mental effort (Kahneman, 1973) since—differently from verbal reports and ratings—they are not under volitional control (Loewenfeld, 1999; Laeng and Sulutvedt, 2014). Given our assumption that musical imagery is intimately linked with the motor imagery of sound-producing body motions, one straightforward expectation is that there will be a close affinity in load on cognitive resources between standard playing and silent playing. Both involve the planning and execution of complex coordinated body motions and both result in actual movement, muscular deployment, and related metabolic expenditure. This would also be consistent with much evidence from pupillometry research indicating that a simple, single, keypress (used in many paradigms to indicate the detection of a target stimulus or a discriminatory choice) results in measurable dilation of the eye pupil (e.g., Simpson, 1969; Simpson and Climan, 1969). However, overt behaviors and muscles’ contractions are not necessary for pupillary dilations to occur and there is overwhelming evidence in the literature that dilations can index the presence and degree of internal (cognitive or affective) processes without the need for overt responses (e.g., Kahneman and Beatty, 1966; Laeng et al., 2016).

In the present study, we opted for correlational analyses of the measure-by-measure changes in pupil diameter across conditions within each musical piece. We expect that the pupil sizes will co-vary positively across the different phases of a musical piece as a support to the hypothesis of functional overlap. Similarly, there should be a chronometric correspondence between the time required to perform a musical piece and that required for its execution, but this does not imply that the absolute times cannot differ, even when there is strong functional overlap. Based on the theory and findings of the original “mental scanning” experiments (e.g., Kosslyn et al., 1978; Borst and Kosslyn, 2008), the time to scan increasing distances increases at comparable rates in perception and visual imagery. However, when generating mental images from long-term memory, participants could scan more slowly in the mental image condition. Hence, we expect the time course of imaging to be slower for musical imagery than in listening or playing. This seems also to be the case for the time taken to imagine an action and its actual execution in athletes (e.g., Reed, 2002).

Previous studies with the pupillometry method have shown its ability to measure the moment-by-moment changes in cognitive and affective arousal during music listening (e.g., Laeng et al., 2016; Bowling et al., 2019). To our knowledge, there are no previous pupillometry studies during a musical performance. The present use of pupillometry as a gauge of mental work in music seems novel and potentially fruitful.

Experiment 1A: Pupillometry

If similar processes and mechanisms underlie perception and imagery, executed and imagined movements, then we reason that this functional overlap should be reflected in the level of allocation of cognitive resources and, in turn, in changes in the pupil diameter. Moreover, the degree of effort required during a musical piece may be visible not only in the eyes of a performer but also in those of the listeners. The experience could modulate this response, being stronger in musicians with experience with the same instrument than in the non-musicians. Hence, in this study, we explore how the pianist’s eye pupil can index effectively the level of cognitive resources required in a performance by using differentially challenging musical piano pieces and annotating the momentary difficulty, at different points in time during a musical performance. Also, we monitor the pupils of listeners, either pianists or non-musicians, while listening to the pianistic performances of the professional musician.

We expect positive correlations between pupil diameter changes of the professional pianist during different “executions” (i.e., normal, silenced, listened, and imagined) of the same piano piece, which would indicate the presence of similar task demands and load on resources and the engagement of similar cognitive and execution mechanisms. Conversely, a lack of correlation between pupil diameter changes in the different “executions” of the same piano piece (i.e., normal, silenced, listened, and imagined) might indicate the absence of common mechanisms but the strength of the correlations should give a hint to the degree of functional overlap or equivalence.

Method

We use the pupillometry method based on infrared eye-tracking, which allows the precise tracking of the size of the pupil in both eyes simultaneously in a non-invasive manner and with no restrictions on eye movements, at a sampling rate equal or superior to standard film or television (e.g., PAL is 50 Hz).

Participants

This first experiment involved the single case of a professional pianist (PP hereafter) who is a music teacher in Oslo, Norway, and an internationally renowned performer (e.g., at Carnegie Hall), with expertise in 19th century piano music and technique. PP is a 41 years old female and she has played the piano since the age of six. Based on her responses to The Goldsmiths Musical Sophistication Index (v1.0), she has a top score of 36 in musical training and a General Sophistication score = 88. PP plays five or more hours per day on her primary instrument (piano), but she can play four other additional instruments. She is highly active in perfecting her technique and has lectured and published on piano techniques from previous centuries. Hence, PP qualifies as exceptional and a true expert in music performance according to the criteria of Montero (2016) and Høffding (2019) and most definitions of expertise regardless of the domain (e.g., for elite sports; Swann et al., 2015). The Department of Psychology’s IRB at the University of Oslo approved the present study (Reference number: 3568281) and PP as well as the other participants (in Experiments 1B and 2B) received a consent form before testing and were treated according to the Declaration of Helsinki.

Musical Stimuli

PP provided us with three piano pieces of different technical and interpretive difficulty: an easy piece, a middle-level one, and an advanced one. She chose two compositions of Edvard Grieg: Wächterlied, from Lyrical Pieces, Op.12 No.3, and Holberg Suite, Op. 40, Praeludium (Allegro vivace); these are considered as an easy and a difficult piano piece, respectively. The medium difficulty composition was Robert Schumann’s Träumerei (from Kinderszenen or “Scenes from Childhood,” Op. 15). Each piano piece was performed twice on an electronic keyboard set to “grand piano,” once normally, “standard playing” (hereafter) and once with no sound (by turning off the audio). Also, PP listened to her normal renditions of each piece (as registered in a MIDI file) as well as in a condition where there was neither auditory nor kinesthetic feedback from finger movements, i.e., the “imagery” (hereafter) condition. In all the four conditions, we showed the first two pages of the musical score on the computer screen of the eye-tracking device.

Apparatus

We used a Yamaha electronic or digital piano (P-140) set to “grand piano.” The P-140 keyboard has 88 graded-hammer keys with sounds based on Yamaha’s AWM sampling technology, with the convincing similarity of sound to a real acoustic instrument. We interfaced with a MIDI Unit (MOTU UltraLight Hybrid MK3) which recorded the two performances (with sound and without), plus it allowed the playback of the performance with sound during the listening-only condition. We positioned the piano keyboard on an adjustable desk at a comfortable height for the pianist. We positioned the eye-tracking computer’s screen on the same adjustable desk, behind and above the keyboard for optimal visibility. We attached the infrared camera of the ET unit to the lower edge of the computer screen, a flat DELL LCD monitor, with a screen resolution of 1,680 × 1,050. We presented the first two pages of each of the three pieces’ musical scores, after digitalizing them at a high-resolution, so that the pianist could read the score at a comfortable distance of about 50 cm. To facilitate playing and making the session more natural we did not use a chinrest, which is not problematic for the SMI eye-tracking equipment since it automatically corrects for changes in head position and rotation. Hence, in such testing conditions, it is possible to obtain reliable gaze data and mapped pupil diameter (in mm) that are free of artifacts due to head movements or changes in distance between the eyes and the screen.

A R.E.D. 250 SMI infrared eye and iView X Hi-Speed Software (SMI; Berlin, Germany) recorded eye positions at a sampling rate of 250 Hz. The RED can operate at a distance of 0.5–1.5 m. This device has two sources of infrared light from an infrared light-sensitive video camera, placed under the monitor frame. According to SMI, the RED system can detect changes as small as 0.004 mm. During the experiment, PP looked directly into the screen to the musical scores. We used BeGaze software from SMI to extract the gaze and pupil data, and Microsoft^® Excel, JASP, and Statview software for the statistical analyses.

Procedure

A 4-point calibration procedure preceded each experimental session. The pianist always looked to the music score while playing as well as in the separate conditions, where PP imagined performing the same pieces “in her head.” Also, the pianist listened to her own playing of the same piano pieces (as recorded by MIDI from the keyboard and played back with headphones). One condition consisted of playing the piece while looking at the score on the screen with the muted electronic keyboard so that the hands/fingers’ movements produced no sound. At the end of all experimental sessions, the pianist rated levels of technical difficulty or expressivity and harmonic intensity on the musical score (by use of a 7-step Likert scale), measure-by-measure (see Figure 1 for an example).

FIGURE 1

Figure 1. The score of Grieg’s Wächterlied for piano, with annotations by the professional pianist of its technical difficulty (measure by measure on a 7-step Likert scale).

To estimate subjective effort, at a later date after the pupillometry session, we asked the professional pianist to estimate with a 7-step rating scale each piece, measure by measure (roughly following the musical metric framework; see Keller, 2001), along with three different parameters of “difficulty”. One type we label “technical” (i.e., of the motor-related challenges of playing the notes as indicated in the score); then, the “expressive” (i.e., expressivity-related difficulties, like shaping the performance as intended); and finally, the “harmonic tension” (i.e., subjectively experienced harmonic tension and release).

Results and Discussion

We first computed descriptive statistics for the ratings given to each piece, measure by measure (see Figure 2). The pianist (PP) judged on Likert 7-step scales the technical difficulty, the expressivity, and the harmonic tension. Three separate ANOVAs with each of three ratings as dependent variables showed that all pieces differed from one another. Specifically: (a) technical difficulty, F₍₂₎ = 194.1, p < 0.001, η² = 0.86 (post hoc tests: 2.8 < t < 18.6; 0.02 < p < 0.001); (b) expression, F₍₂₎ = 45.5, p < 0.001, η² = 0.59 (post hoc tests: 4.5 < t < 9.5; all p < 0.001); (c) harmonic tension, F₍₂₎ = 38.3, p < 0.001, η² = 0.55 (post hoc tests: 2.9 < t < 8.7; 0.01 < p < 0.001). These mean ratings essentially confirm PP’s selection of three pieces in terms of pianistic challenges or performance demands, since consistently she rated the Holberg Suite highest on all three measures, Wächterlied was rated the lowest, while Träumerei was placed in between.

FIGURE 2

Figure 2. Violin boxplots of the ratings by the Professional Pianist of the three types of difficulty (from top to bottom: technical, expressive, and harmonic tension) for each of the three piano pieces (Holberg Suite, Träumerei, and Wächterlied).

Interestingly, a multiple regression analysis with “technical difficulty” as the dependent variable and “expression” and “harmonic tension” as the independent variables revealed a strong positive relationship between the three measures, F₍₂₎ = 85.5, p < 0.0001, r = 0.855. At the same time, these measures tap, as intended, on different aspects of the subjective effort in performance. Specifically, “technical difficulty” appeared to be more closely related to “harmonic tension” (Regression Coefficient = 0.865, t = 6.2, p < 0.0001) than to “expression” (Regression Coefficient = 0.280, t = 2.1, p = 0.04).

Our measure of objective (mental) effort was the pupil diameter during each condition. Since the pianist was looking at the same score in all conditions, we assume that the light stimulation to her eyes remained constant across musical measures and across conditions. In Figure 5 we show the waveforms of PP’s pupil diameter (as color lines) along time (in seconds) for each of the four conditions and for the three pieces (split in panels). These waveforms reveal several interesting aspects. First, the pupillary waveforms when “playing” (i.e., moving the fingers “with” or “without” sound) are consistently above the other two conditions where there is no movement (i.e., “listening” and “imagery”).

FIGURE 3

Figure 3. Pupillary waveforms of professional pianist (PP) during each of the three piano excerpts (Holberg Suite, Traäumerei and Wächterlied) and for each of the four conditions (“Playing with sound”—red line, or “Playing with no sound”—green line, “listening”—blue dotted line, and “imagery”—orange dotted line).

FIGURE 4

Figure 4. Mean pupil diameters (in mm) of the Professional Pianist for the four conditions. Bars represent 95% confidence intervals.

FIGURE 5

Figure 5. Violin boxplots of the mean pupil diameters (in mm) of the Professional Pianist for the three piano pieces (Holberg Suite, Träumerei, and Wächterlied) split by the four conditions (from top left panel: Playing with sound; Playing with no sound; Listening; and Imagery).

Second, it is clear that while the duration of each piece when playing either with or without sound differs only a few seconds from one another (and from listening), the imagery condition was—as seen in previous studies (e.g., Janata and Paroo, 2006)—longer than the other conditions. Traäumerei was imagined for about 40 s longer (i.e., a 40% lengthening) than when listening or performing. Similarly, Wächterlied was imagined for about 22 s longer (i.e., a 16% lengthening) than when listening or performing. PP performed the Holberg Suite at a faster rate than the other two pieces (in about 35 s) but, when imagined, its length stretched of about 5 s (i.e., a 14% lengthening).

We also computed descriptive statistics for mean pupil responses. Variations in pupil diameter approximate a normally distributed parameter (e.g., Mathôt et al., 2018). Moreover, F-tests remain robust also when data are not entirely normal (Blanca et al., 2017). Hence, we applied a repeated-measures ANOVA with the mean pupil diameter within each measure as the random factor and Conditions (Playing with sound; Playing with no sound; Listening; Imagery) as the within factor. This analysis revealed a significant effect of Conditions, F₍₃₎ = 320.6, p < 0.001, η² = 0.83. Post hoc (Bonferroni) tests confirmed that the four conditions differed significantly from one another (p < 0.001; Cohen’s d range: 0.4–2.68).

As visible in Figure 4, the most effortful condition—according to the pupil diameter—was “Playing with no sound,” followed by “Playing with sound”. The condition of “Listening” and “Imagery” were clearly less effortful (Cohen’s d > 2) compared to the previous “motoric” conditions. Looking at the score and imagining the music was the least demanding of all conditions.

Also, we run an ANOVA on the mean pupil diameters for the three musical pieces. This analysis revealed a significant effect of the factor of Musical Piece, F₍₂₎ = 18.9, p < 0.001, η² = 0.38. Post hoc (Bonferroni) tests showed that both the Holberg Suite (mean pupil diameter = 3.432; SD = 0.13) and Träumerei (mean pupil diameter = 3.429; SD = 0.07) differed significantly (p < 0.001) from Wächterlied (mean pupil diameter = 3.296; SD = 0.05). However, they did not differ from each other (see Figure 5 illustrating in boxplots the pupil diameter for each piano piece and each condition in separate panels).

A multiple regression analysis explored the relationship between the four conditions. One multiple regression used “imagery” as the dependent variable and the other three conditions as independent variables, which revealed a highly significant relationship and a moderate positive relationship, F₍₃₎ = 17.5, p < 0.0001, r = 0.68. Specifically, “imagery” was highly significantly related to “listening” (Regression Coefficient = 0.65, t = 4.85, p < 0.0001). Imagery was also significantly related to “Playing with no sound” (Regression Coefficient = 0.224, t = 2.1, p = 0.04), but failed to reach significance with the standard performance condition or “Playing with sound” (Regression Coefficient = 0.12, t = 1.05, p = 0.30).

Figure 6 illustrates in detail how the mean pupil diameters (in each musical measure or bar) are related to each other in each condition. Specifically, we subdivided the pupil time series by the number of bars in the score, to standardize the pupil data between conditions differing in length. The results confirmed our expectation, based on the idea that each of the conditions would draw resources or demand mental effort, that the pupil changed similarly during the same moments (or “chunks,” i.e., measures) of a musical piece. Indeed, the two motoric conditions (Playing with sound and Playing with no sound), both requiring actual finger movements, showed the strongest relationship (Figure 6, top left panel), F_(1,64) = 160.6, p < 0.0001, r = 0.85. Most interestingly, the second strongest relationship was between “listening” and “imagery” (i.e., the two conditions without explicit motoric involvement), F_(1,64) = 39.1, p < 0.0001, r = 0.62. This positive relationship might be attributed to hearing the music, not only when listening, but also “in the mind”s ear’ when imagining.

FIGURE 6

Figure 6. Bivariate plots and simple regression lines for PP’s mean pupil diameters (in mm) for each measure of all three pieces together. Top left panel: dependent variable = “Playing with no sound”; independent variable = “Playing with sound”; Top right panel: dependent variable = “Imagery”; independent variable = “Listening”; Bottom left panel: dependent variable = “Listening”; independent variable = “Playing with sound”; Bottom right panel: dependent variable = “Imagery”; independent variable = “Playing with sound”.

The two next strongest relationships (displayed in the bottom panels of Figure 6) showed only moderate correlations, though both statistically significant. “Playing with sound” was positively related to “Listening” (bottom left panel), F_(1,64) = 20.9, p < 0.0001, r = 0.49. Remarkably, “Playing with no sound” (bottom right panel) was positively related to “Imagery,” F_(1,64) = 21.6, p < 0.0001, r = 0.50. All other simple regression had still positive correlations, but with Spearman’s coefficients r below 0.5, accounting for less than a quarter of the variance and are not displayed here.

In sum, our objective measure of effort, the pupil size, showed that pupil diameters were largest (Figure 4) when “playing” (regardless whether there was sound produced or not) and above the conditions where there is no movement (i.e., “listening” and “imagery”). This suggests that programming and executing motion is considerably more demanding than situations in which action is not required. This could also be partly because mental and physical efforts are meshed during the action and both could show up in the measure of attentive workload.

Although “musical effort” in general has been an understudied aspect of music cognition (see Keller, 2001), a study by O’Shea and Moran (2016, in Study II) examined explicitly how the eye pupil adjusted when pianists performed a piece but also while they simply imagined the performance. They reported no difference between their musicians’ pupils during the actual performance and its imagery, which seems consistent with an equivalent deployment of arousal for the same processes in the two conditions. However, they based such a conclusion on the results of an analysis of variance that barely missed the 0.5% cut-off (p = 0.053), which might not constitute conclusive evidence for no difference.

We confirmed that subjective effort is related but not identical to the objective effort, since the ordering of the two differed slightly (Figures 2, 5), especially for “listening” and “imagery.” Most interestingly, changes in the mental effort as the musical piece evolved (measure-by-measure) were all positively related (Figure 6) and their relationship was strong for the two playing conditions and moderate for the two non-playing conditions, revealing that regardless music is performed or only listened (with the physical ear or mind’s ear) the pupils’ diameter co-vary at each point in time. This can be interpreted as good evidence that all conditions share the same cognitive mechanisms and a similar workload along with the musical piece.

Finally, we replicated the finding that imagery can stretch the timeline. It is possible that when playing music in one’s head, there is the luxury to pause or dwell on a particular moment. Because imagery involves additional mechanisms of generation and maintenance as well executive processes (Glover et al., 2020), this could lengthen processing, without apparently increasing the cognitive workload compared to actually listening (Figure 4).

Experiment 1B

In the following experiment, we sought to provide evidence that also listeners would react to the perceived “musical effort” inherent in the music and likely more in musicians than in non-musicians, while they listened to the piano renditions by our professional pianist. The spontaneous pupillary responses can be compared for similarity across groups and with those of the performer.