Mixed reality strategies for piano education

Head-mounted mixed reality interfaces, in which a user perceives a seamless blend of real, virtual or remote content, have great potential in a wide range of educational contexts. In this paper, we explore the use of mixed reality (MR) in piano education. We begin with a review of previous examples of virtual and mixed reality for piano teaching and learning, identifying four major categories of functionality: remote teaching, learning to read musical notation, providing alternative notation systems to Western musical notation, and enhancing users’ understanding and experience of music. Following this review, we present an application designed to demonstrate the use of MR for facilitating remote student-teacher piano lessons. Additionally, hand-tracking enables real-time, bi-directional visualization of both the student’s and teacher’s hand movements while playing, which is a crucial communication channel in piano instruction. We also present the Piano Theory Hub, an interactive area for independent learning and practice designed to compliment remote instructions. The Piano Theory Hub uses MR to provide spatial in situ explanations and exercises on notes, intervals, scales, and chords. User studies were conducted to evaluate both remote and solo aspects of the application. The first study with 10 participants revealed a strong sense of immersion and co-presence with the teacher during remote lessons. Trials also revealed that, in addition to virtual hands, visual highlighting of key-presses was found to improve visualisation of the remote users’ play. The second study, with 15 participants found the Piano Theory Hub to be helpful for beginners and some intermediate players, with feedback suggesting improvements in user experience and highlighting the limitations of replacing traditional piano teachers. Our experiments demonstrate that MR can be successfully employed for the following strategies for piano education: hand interaction coaching, flexible virtual hand and piano visibility, augmented feedback including key highlighting, gamified learning elements, and flexible teacher positioning options. Overall, the findings suggest that mixed reality holds promise as an effective tool for remote piano learning and music education, offering immersive and engaging learning experiences.

1 Introduction

Mixed Reality (MR) technologies, where users see a seamless blend of real-world and virtual content, have the potential to enhance educational experiences across a wide range of disciplines. In this paper, we explore the use of head-mounted mixed reality interfaces for piano education.

The most common way of learning to play the piano involves a combination of piano lessons with an instructor and solo practice. Interactive computing has been used for improving piano education since 1990 (Dannenberg et al., 1990). As such, in this paper, we explore the use of MR in both student teacher lessons and in self-learning exercises.

For student-teacher lessons, we focus on the benefits that MR might bring to remote lessons. Whether due to inaccessibility of a trained piano teacher or because of a global pandemic, there is a rising demand for remote piano lessons (Wang, 2020). Video conferencing platforms are still the predominant tool for remote music instruction, but especially for practical lessons, delivering them effectively over typical video conferencing tools is challenging (Okan, 2022). Elements such as correcting hand positioning and body posture, non-verbal cues and communication, playing duets or accompaniment are difficult to realise through typical video conferencing systems (Duffy and Healey, 2017). For these reasons, use of augmented reality has been proposed as one possible solution for overcoming these challenges (Tan and Lim, 2018).

In the context of self-learning, MR provides novel ways of presenting teaching materials (Li, 2018; Das et al., 2017). Information can be presented in situ, directly on the keys of the piano, for example,. Users can receive constant and direct audio and visual feedback, improving their visual, auditory, and motor skills simultaneously (Mei and Yang, 2021). MR also allows traditional exercises and materials to be transformed into dynamic, interactive experiences. This opens the door to reactive and gamified learning, helping to keep students motivated. This is particularly important when learning music theory, which can often appear abstract and inaccessible for beginners. Knowledge of music theory can help students memorize pieces more quickly and enhance reading skills by recognizing patterns in the music (Shockley, 2001). Sha (2019) suggests that there is an urgent need to find new teaching methods because traditional methods and lack of pedagogical variety can diminish students’ motivation and interest. MR can also be used to provide instantaneous feedback when no teacher is available, detecting and highlighting mistakes or deviations in timing. However, Piano learning extends beyond hitting the correct key at the right moment; expressive timing and dynamics (Oore et al., 2017), as well as the ability to convey musical meaning are key aspects for well-rounded musical understanding. This can be facilitated by remote lessons in multi-user MR.

Despite its promise, using HMD-based MR technologies in piano education remains a relatively unexplored territory. This work examines how MR can be used in remote piano learning and for self-study between piano lessons. By developing a prototype application, we aim to demonstrate the potential of MR in this context and reveal some of the learning activities that might be well served by MR.

2 Related work

We briefly review previous use of immersive media technologies to support piano learning. We restrict our review to use virtual or mixed reality or in situ projection to facilitate piano instruction. 16 examples were selected and analyzed according to the interface technology and pedagogical content (see Table 1).

Table 1

Table 1. Literature overview of Piano Learning Applications using Immersive Media Technologies.

When grouped according to their primary pedagogical goal, four main groups emerged:

$•$ Group 1: facilitating learning of musical notation sight-reading,

$•$ Group 2: replace Western musical notation with a more intuitive notation system,

$•$ Group 3: giving learners a “feel” for music; e.g., by teaching improvisation,

$•$ Group 4: remote piano lessons with an instructor.

2.1 Group 1: teaching sight-reading

Learning to read sheet music is often a long process for beginners (Tan et al., 2009). As such, Group 1 applications explore ways of making the process of learning to read sheet music easier, more intuitive, and fun. The applications by Chow et al. (2013) and Rigby et al. (2020) concentrate on teaching to play specific songs and KeynVision focuses on playfully teaching particular parts of music theory, namely, “octave scales, chords, and arpeggios” (Birhanu and Rank, 2017).

2.2 Group 2: replacing sight-reading

Western musical notation can be difficult for beginners (Tan et al., 2009). The mapping between notes in sheet music and the key to be played on the piano is abstract and somewhat counter-intuitive (Rogers et al., 2014) and, for new students, requires a high cognitive load (Molloy et al., 2019). The considerable time required to learn to read and play musical notation can lead to frustration and discourage beginners (Kuo and Chuang, 2013).

To overcome these problems related to Western musical notation, several of the presented applications propose the use of piano roll notation. In piano roll notation, often implemented with AR, information about which notes to play next is projected directly onto the piano keys. Each note is represented by a block, and instead of sheet music, a stream of blocks is shown moving toward the piano. The blocks indicate when and for how long a key should be pressed and held (Molloy et al., 2019). The players can thus anticipate the next notes and the blocks can be directly mapped on the respective piano keys, reducing the cognitive load (Rogers et al., 2014).

Applications of Group 2 use the piano roll notation as a way to teach piano pieces without the need for sight-reading skills. They focus on teaching specific songs and not on teaching music theory like scales or chords.

2.3 Group 3: getting a ‘feel’ for music

Improvisation allows for self-expression, the development of musicianship, and the building of confidence (Chyu, 2004). According to Jaques-Dalcroze and Rothwell (1932), learning how to improvise facilitates the learning of piano because it requires a good basic musical understanding of harmonies and a feel for rhythm (Jaques-Dalcroze and Rothwell, 1932). Therefore, group 3 applications focus on teaching improvisation as well as on getting a feel for the instrument, rhythm, and harmonies.

Music Everywhere (Das et al., 2017) allows users to play and improvise with a virtual band. The application of Sandnes and Eika (2019) aims at enabling its users to construct their own chord voicings by teaching jazz chord structures. Andante (Xiao et al., 2014) and Andantino (Xiao et al., 2016) use a projector to visualize animated characters walking over the piano keys, changing shape, emotions, and walking style depending on the music played. The approach of Rhodes et al. (2023) to improve and understand musical performances is to capture and analyze biometric information during the performances to create a virtual music instructor giving real-time feedback. Similarly, Labrou et al. (2023) created a multimodal performance dataset from expert pianists’ hand gestures, used in their Pianoverse MR application to convey correct posture and assist piano learning.

2.4 Group 4: remote piano lessons with an instructor

Applications of Group 4 focus on improving remote piano lessons with an instructor. For example, HoloKeys (Stanbury et al., 2021) allows piano instructors to use an AR interface to do live demonstrations on the piano of their students using an Internet of Things (IoT) system that can be clipped onto the keys of the student’s piano. PPVR (Wang, 2020) is a tool designed for asynchronous as well as synchronous communication between piano teachers and their students. MirrorFugue by Xiao and Ishii (2010) allows users to see a visualization of the remote user’s hand movements in different viewpoint configurations.

2.5 Research opportunity

Serafin et al. (2016) argue that insufficient attention is given to immersive virtual musical instruments with simulated visual components. Although valuable efforts exist to address piano learning, they lack specific solutions for remote lessons and shared virtual spaces for interactive piano education. Additionally, they prioritize individual songs over broader music theory and often rely on complex setups. Notably, none offer real-time feedback during self-study music theory exercises. This research proposes an MR piano learning application that addresses these limitations by providing a shared virtual space for interactive lessons and a music theory hub for practicing with real-time feedback.

3 Design

We propose a piano application prototype consisting of two parts. The first part allows for piano lessons in a shared virtual room with a remote instructor (Figure 1). The second part is designed for practising basic piano theory concepts in between lessons while receiving real-time feedback.

Figure 1

Figure 1. For remote piano lessons, the teacher and student connect to a shared virtual room using a digital piano and an HMD. The teacher/student is visible as a virtual avatar for the remote user.

3.1 Setup

A Yamaha P-45 B piano was used throughout the development phase and the user studies for testing purposes. To identify the pitch value and velocity of the pressed keys, MIDI data from the piano is required. To obtain this data, a digital piano must be connected to the Oculus Quest using a USB-C to USB-B cable (Figure 2A). The Oculus Passthrough API is used to see the physical world in the headset. To facilitate data transfer in remote lessons and to enable multiplayer networking, the Normcore framework is used.

Figure 2

Figure 2. (A) A participant of the user study on the music theory hub uses the application. The setup requires an HMD, a digital piano, and a cable to connect the HMD to the piano. (B) Within the Hand Coach component, users practice using the hand menu, triggered when the left-hand palm faces upward. This menu facilitates quick settings adjustments and enables teleportation throughout the virtual space. (C) To calibrate the virtual piano to the physical piano first the lowest and then the highest key of the physical piano need to be pressed. (D) The position of the virtual piano can be fine-tuned using the adjustment buttons.

The application consists of two main parts: one for remote piano lessons with an instructor and a practice hub for music theory exercises for in-between lessons. It can be operated with hands alone or by using the right-hand controller. An introductory hand coach menu (Figure 2B) is displayed upon application initiation to familiarize users with hand interactions in the virtual space.

To calibrate the virtual piano, users need to press the lowest key of the physical piano with their left index finger and the highest key with their right index finger (Figure 2C). The positional data derived from these two contact points is then used to calculate a transformation that aligns the virtual piano with the physical piano. Following the initial alignment, users can fine-tune the position and rotation through arrow buttons (Figure 2D).

Text and note names can be shown on the virtual piano keys for displaying fingering and feedback. Keys pressed by a student light up in blue, while those pressed by teachers light up in purple. The remote user’s virtual piano is always shown, while the local virtual piano’s visibility can be toggled. Instead of a virtual keyboard (Figures 3A, C), subtle lines indicate the alignment of the virtual piano with the physical keyboard (Figures 3B, D).

Figure 3

Figure 3. Users can toggle virtual hand and piano visibility based on their preferences. (A) Both virtual hands and the virtual piano are visible. (B) Only the virtual hands are displayed. (C) Only the virtual piano is displayed. (D) Neither the virtual hands nor the virtual piano is displayed.

3.2 Part 1: piano lessons with a remote instructor

For remote piano lessons, a student and a teacher both need to connect to the virtual room. Within this shared space, users can see and interact with virtual representations of each other’s heads and hands (positions and rotations), allowing for interactions, including non-verbal communication. eThe shared virtual classroom enables the teacher to guide and observe the student, demonstrate musical concepts, and offer real-time feedback. To allow for different teaching scenarios, the users have the freedom to teleport within the virtual environment (as described in the next section, Section 3.4). Moreover, the virtual room includes interactive elements like interval and scale games, along with an interactive theory area (compare Section 3.2.1).

3.2.1 Multiplayer music theory games

$•$ Interval Game: The teacher initiates the game by playing an interval, which is then transmitted to the student. On the student’s screen, a set of four buttons displaying different interval names becomes visible (Figure 4A). The student’s challenge lies in identifying and selecting the correct interval from the options provided. The teacher receives real-time feedback on the interval the student has chosen, and the student automatically receives points for correct answers.

$•$ Scale Game: Within the scale game, the teacher controls the parameters of the scale that the student is supposed to play (Figure 4B): The teacher selects the scale’s starting note, designates the hand for play (left or right), and specifies whether a major or minor scale is to be practised. This configuration is then transmitted to the student who tries to play the scale accurately.

Figure 4

Figure 4. (A) The Interval Game prompts the student to identify intervals played by the teacher. (B) In the Scale Game, the teacher assigns a scale for the student to play, with feedback provided both automatically and by the teacher.

3.2.2 Interactive theory area

The Theory Area provides piano teachers with gamified tools for teaching basic concepts (Figure 5). It allows for the explanation of basic principles and enables teachers to create custom piano theory games. The Theory Area features a table with a staff notation, where students can switch between bass and treble clef. Note heads and cubes to display note names can be moved around on the staff notation collaboratively (Figures 5B, C). The interactivity aims for students to understand the relationship between note nomenclature and their corresponding positions on the staff. Stems and bars in note heads are intentionally omitted for simplicity. While interacting with virtual three-dimensional objects, the approach without stems and bar lines comes in handy, as layering and placing conflicts are reduced. Especially for beginners color-coding the cubes based on pitch enhances the learning visually (Demirel, 2022). For advanced beginners and intermediates note lengths, bars, key and time signature, as well as a second notation system should be added in future work. In addition, note labels for different octaves should be integrated (i.e., A, a, a’, a”).

Figure 5

Figure 5. (A) The Theory Area can be used by the teacher for interactive music theory-related exercises and explanations. (B, C) Teachers and students can interact collaboratively with the objects.

3.3 Part 2: independent piano practice

The Piano Theory Hub is designed to be used by piano students in between lessons, to apply and practice concepts explained by the teacher with real-time feedback. It consists of four consecutive modules that structure the learning process: Notes and Keys, Intervals, Scales, and Chords. Each module contains a short theoretical introduction explaining the fundamental musical concept and a series of exercises with real-time feedback.

Feedback concerning correct or incorrect input is provided in several ways:

$•$ Incorrectly played keys are highlighted in red, while correctly played notes are highlighted in green.

$•$ A distinctive sound is played to indicate whether the input is correct or incorrect.

$•$ Feedback and explanations are displayed on the screen.

The number of correct or incorrect inputs is tracked in test exercises. These exercises require playing a set number of musical concepts from prior practice sessions. Progress is visually depicted by a progress bar at the top of the screen. An analysis of the number of correct plays is provided at the end of each exercise.

In all exercises, hints can be enabled that assist in understanding the musical concept.

Additionally, a Song module was implemented allowing the users to play their first melodies by following the key highlighting and fingering displayed on the virtual piano..

3.3.1 Exercises–notes and keys

$•$ Piano Words: An arbitrary word (a random sequence of letters) is displayed containing only letters corresponding to the note names (Figure 6). The sequence does not have to form a valid English word. The user has to play the words by pressing the keys with the corresponding name. As an aid, beginners can display the note name over the keys of the virtual keyboard.

$•$ Note Name: The user has to play 20 different notes in sequence. A progress bar at the top of the screen shows how many of the 20 notes have been played. At the end, the user is shown the number of correctly played notes.

Figure 6

Figure 6. Within the Piano Words game of the Piano Theory Hub, users can practice recognizing the piano keys given by their note names. Optionally, the note names can be displayed directly over the virtual keyboard while practicing. (A) Correct key presses illuminate the key in green, accompanied by a positive sound effect. (B) Incorrect key presses result in red illumination on both the key and displayed note name, accompanied by a negative sound effect. Additionally, the screen provides feedback on the played note.

3.3.2 Exercises–intervals

The interval exercise requires users to play a specified interval by name. Additionally, an interval hearing exercise was implemented where users have to play the heard interval given the start key. As assistance, the number of semitones between the start note and the corresponding second note of the interval can be displayed on the screen and the keys (Figure 7A).

Figure 7

Figure 7. (A) In the interval exercise users play an Interval given by its name. When hints are enabled, the distance in semitones between the two notes gets displayed on the screen and the keys. (B) In the scale exercise users play a scale given by its name. When hints are enabled, the next key to be pressed is highlighted in yellow and the correct fingering is displayed on the keys.

3.3.3 Exercises–scales

The scale module contains two exercises to practice major scales and minor scales (Figure 7B). In those practice exercises, hints are enabled that highlight the key to be played next in yellow. The “All scales” option allows users to test their skills on a variety of scales. This option involves playing several randomly selected major and minor scales for beginners. In the end, the number of scales played correctly is displayed. For advanced players, more scale types can be added.

3.3.4 Exercises–chords

The chord module covers four chord types: major, minor, diminished, and augmented. More chord types and inversions can be added for advanced players. To understand how chords follow patterns of semitone spacing, hints can be enabled that display the distance of semitones to the previous note. Furthermore, the keys of a possible correct chord are highlighted (Figure 8). The module also includes an exercise that involves playing several random chords containing these four chord types. A final display shows the number of chords played correctly. In addition, the module contains an exercise that requires the user to construct the chords of a major scale. In a subsequent exercise, users are asked to play common chord progressions based on scale degrees.

Figure 8

Figure 8. Chord Exercise. (A) The number of semitones between the notes of the chord is indicated on the keys to facilitate comprehension of how chord types follow specific patterns of semitone distance. (B) When a chord is played correctly the keys light up in green. Otherwise, the keys light up in red, and explanations of the chord structure are displayed on the screen.

3.4 User interface (UI) and teleportation

Players can decide whether they want to see the virtual hands, which appear in light gray, or only the video feed of their own hands (Figure 3). During piano lessons, the hands of the remote avatar are displayed in dark gray to the local user (Figure 9) and are always visible.

Figure 9

Figure 9. (A, D) The teacher is placed next to the student and can provide feedback, e.g., on hand positioning, fingering, and dynamics, while the student plays the piano.(B, C) The teacher is positioned opposite the student and plays a melody that the student tries to replicate by observing the highlighting of the keys. (E) The positioning opposite of each other facilitates conversation and allows for the use of non-verbal communication and gesture, e.g., to explain theoretical concepts.

For navigating and interacting within the virtual piano application two menu systems are provided:

1. Within the main menu users can select games, have access to the calibration screen, and to settings such as toggling the visibility of virtual hands and the virtual piano.

2. The hand menu (Figure 2B) is activated by rotating the palm of the left hand upwards. It provides quick access to the teleportation options and to settings such as toggling the main menu or the display of virtual hands and exiting the application.

Teachers can teleport opposite to the student during demonstrations and explanations to facilitate non-verbal communication (Figures 9A, C). When the teacher assumes the role of an observer, he or she can teleport to positions to the left or right of the student (Figure 9B) to emulate the traditional teacher positioning in on-site piano lessons. Similarly, the student can teleport relative to the teacher’s position. Additionally, users can teleport to the theory area for theoretical discussions and games.

4 User study

Two user studies were conducted to evaluate the application. The first user study with 15 participants focused on the Piano Theory Hub and used a Meta Quest 2¹ HMD for testing. The second user study with 10 participants focused on piano lessons with a remote instructor and a Meta Quest Pro² HMD. In both studies, the think-aloud method was used, where participants are asked to speak their thoughts aloud on how they thought about what is happening.

4.1 User study setup - piano theory hub

Three music teachers, six piano beginners, and six people with prior piano experience tested the Piano Theory Hub. Except for one beginner and two intermediate players, all participants had never used an HMD.

The user tests followed the structure shown in Figure 10A. To be able to guide the testers, the screen of the headset was mirrored on a laptop. After both the pre-exposure and the post-exposure test the users were asked to rate their confidence in pressing the correct keys on a Likert scale ranging from not confident at all (1) to completely confident (5). An overview of the results of the pre-exposure and post-exposure tests can be seen in Figure 11.

Figure 10

Figure 10. Structure of the user studies. (A) user study on the self-study music theory learning part of the application with 15 participants. (B) user study on the remote piano lessons part of the application with 10 participants who assumed the role of the piano students.

Figure 11

Figure 11. Results of the pre-exposure and post-exposure of all users (U) conducted for the user study on the music theory hub. Piano beginners had to press a key given its name. Intermediate players were asked to play chords based on the given chord name. The number of correctly played keys and chords was evaluated automatically by the application.

4.2 User study setup - remote piano lessons with teacher

Six piano beginners and four people with prior piano experience tested the remote lesson component (Figure 10B). During the study, the participants assumed the role of the piano student.

5 Results

5.1 Results - self-study music theory learning

5.1.1 Piano beginners

In the pre-exposure test, all participants, except for Participant 2, had no prior knowledge of the mapping between note names and piano keys. The purpose of the test was to establish a baseline understanding and motivate improvement. The six piano beginners had an average of 3.667 out of 20 notes played correctly, indicating chance-based performance with an average Likert scale score of 1.167 indicating minimal confidence.

In the post-exposure test, an improvement was observed. The mean score for correctly played notes increased to 18.667. Confidence levels also increased, with an average Likert scale score of 4.5, indicating a fairly confident attitude.

A paired t-test was conducted to compare the number of correctly played notes before and after the practice session. There was a significant increase in the number of correctly played notes post-exposure (mean = 18.667, standard deviation = 1.51) compared to pre-exposure (mean = 3.667, standard deviation = 2.50), t (5) = 11.18, p $<$ 0.001. This suggests that the practice session had a statistically significant positive effect on the beginners’ ability to identify and play piano notes correctly.

All of the beginners stated that they found it motivating to see their progress during the 10-min practice session. Participant 1 expressed that she felt proud when she saw how much she had improved. In addition, participants expressed that they enjoyed the practice, found it beginner-friendly, and found it a beneficial way to acquire basic piano skills.

5.1.2 Intermediate piano players

Following the pre-exposure test, most of the six intermediate pianists expressed slight confidence in pressing the correct key and had a basic understanding of major and minor chords. However, all were unfamiliar with diminished and augmented chords. Participants played an average of 4.5 chords correctly, with an average confidence level of 2.167 on the Likert scale, indicating slight confidence.

Post-exposure, a vast improvement in correctly played chords was observed. The post-exposure test showed an average of 15.833 correctly played chords per participant. According to the Likert scale results, participants, on average, felt fairly confident.

A paired t-test was conducted to compare the number of correctly played chords before and after the practice session. There was a significant increase in the number of correctly played chords post-exposure (mean = 15.833, standard deviation = 4.71) compared to pre-exposure (mean = 4.5, standard deviation = 1.87), t (5) = 4.86, p = 0.0046. This indicates that the practice session had a statistically significant positive effect on the intermediate players’ ability to correctly identify and play chords.

Participant 9 did not experience an increase in confidence before and after practising. The participant learned piano the traditional way with a teacher and did not find VR the best way to learn theory. This suggests that the application may not be appropriate for all types of learners. However, all other intermediate players noted that practising with the application helped them understand chord formation patterns. Most participants counted semitones to find the next higher note in a chord, and as they practised, they became more efficient at finding the correct chord.

Participant 7 stated that the feedback he received made him feel like he was constantly being guided by a teacher, which motivated him to continue learning. During the practice phase, participants reported having fun and feeling like they were playing a game. Other participants were motivated by wanting to beat their high score: “I’m quite competitive, so knowing I only played 3 out of 20 chords correctly in the first test motivated me a lot to have a better high score at the end. This kind of kept me focused while practising”.

5.1.3 Feedback of piano beginners and intermediate players

Overall, the participants approved of the way that the help and feedback was provided, describing it as “intuitive”, “helpful”, “fun”, “motivating”, and “not too much and not too little”. As part of the post-exposure interview, participants were asked whether they found it more helpful to receive feedback directly on the keys or to receive feedback on the display screen. All but one participant found the feedback on the keys most helpful during practice between the pre-and post-exposure test: “I rarely looked at the screen when I made a mistake and rather what was actually happening on the piano itself. The colour of the keys helped the most and the feedback with the sound also made me aware of mistakes”. However, one piano beginner noted that he did not notice the keys turning red or green and relied solely on feedback from the display. This illustrates the importance of having different methods of getting hints simultaneously because different types of learners require different types of feedback.

In addition, several participants pointed out that they liked that when making a mistake, they were required to play the correct note to continue practising: “It was really nice to have some feedback right away when I played the wrong chord because it’s quite easy to just ignore errors without noticing when you learn by yourself.”

It was reported by all piano beginners that they found the display of note names directly over the corresponding piano keys to be very helpful. The visual clues provided through the overlay helped several intermediate players remember the pattern behind the different chord types and helped them remember where to press: “I really like the assistance of the app for the chords that leads you to see the patterns in the music directly on the keys. I play the guitar and you mostly learn the patterns and then you repeat them. When I went to the piano it was way harder to see the pattern when I tried to modulate to a different key. On the guitar, you can easily see the semitones because it is always one thread. On the piano I find that it’s quite hard, so I think with the VR assistance telling you where it is until you memorize it is really good to understand the patterns”. Another participant noted: “As a beginner you are not necessarily able to understand the principles of, for example, intervals. And now with the app, you get visual clues. Being able to visually grasp intervals is pretty good”. In addition, an intermediate pianist stated that the feedback on the keys helps to get a feel for the distance between the keys and that it is faster to learn something than to consult a book or search for information online.

Overall, participants found the HMD to be heavy and uncomfortable to wear for an extended period. Furthermore, the alignment of the virtual piano and the physical piano was not always precise and sometimes depended on the angle of the head. As a result, there was a difference between what could be seen and what could be felt.

Most participants found the explanation texts helpful. One piano teacher suggested using more colours and bold text for better structuring. All but one beginner and intermediate player agreed on the helpfulness of the texts. While one beginner preferred longer explanations, others appreciated the current length, with one stating, “I probably would not have read them if they would have been longer”. Furthermore, several users and two of the piano teachers mentioned that the acoustic feedback was helpful and that the sound, when something was correct, was motivating and “felt like a reward”. Several participants expressed that they liked the responsiveness of the virtual piano, which responded to pressing a physical piano key by turning blue and rotating.

5.1.4 Feedback of piano teachers

The Piano Theory Hub was tested by three music teachers who have each taught the piano for at least 10 years: Teacher 1, a music teacher at a secondary school giving regular music classes and band classes to students of 5th to 10th grade, and teachers 2 and 3 that teach all styles of piano from classical to jazz to students aged five to over 50 years.

Teachers 1 and 3 could imagine using the technologies with their students under the conditions that enough VR headsets are available, and the video quality improves. Teacher 3 mentioned that the feedback shown on the keys might be the most motivating factor for adults. However, the teacher suggested changing the position of the display, so there would be no need to look up and down to look from the keys to the screen. This could lead to more participants noticing and using the feedback and hints on the display.

Teacher 2 felt overwhelmed by the MR technology and would not consider using an MR application, preferring “to be in the real world”. Although teacher 2 acknowledged the usefulness of MR overlays for beginner challenges like finger placement and note identification, the teacher believes traditional methods suffice for piano instruction.

Teacher 1 notes that many beginners struggle with finger coordination, especially when playing chords, which initially requires considerable control. The teacher believes that the application could provide this control during home practice and serve as a motivational tool. However, teacher 1 emphasizes that an application cannot completely replace a teacher, as the emotional aspects and different variations of interpreting music are too complex and vary too much from case to case to be explained by an application.

5.2 Results - remote piano lessons with teacher

All results of the survey conducted with the participants of the user test can be found in Figure 12.

Figure 12

Figure 12. Results of 16 statements (S) included in the survey conducted for the user study on the remote piano lessons with a teacher.

The different levels of experience and interest in VR (compare S1 and S2 in Figure 12), influence how people react to, and engage with the application: Most users found the initial hand coach adequate (see S3 in Figure 12), except for two with no prior HMD or hand tracking experience. Post-exposure, they expressed a desire for additional brief exercises and multiple examples within the hand coach. This shows the importance of tailoring instructional elements to be inclusive for users with varying levels of familiarity with both the technology and the specific interaction mechanics.

The virtual piano calibration process was perceived as easy by most participants (S4, Figure 12). One participant expressed a neutral stance as he would have preferred an automated calibration method that would have placed the piano in the correct position without requiring manual input by pressing the lowest and highest keys and the need for recalibration.

Most participants found interaction within the theory area easy (S5, Figure 12), except for two who initially desired more hand interaction introduction and faced challenges moving the cubes. For one of them, this initial hurdle translated into a decreased level of enjoyment due to a sense of frustration. Yet, the majority showed no difficulties with the interaction mechanics and found the capability to manipulate the notes and cubes to be fun (S6, Figure 12). Overall, most users considered the interaction natural and immersive (S7, Figure 12), although a subset of three remained neutral, explaining that the physics applied to the cubes and notes felt different from real-world expectations. Nonetheless, participants noted that they especially liked how it felt to hand a cube to a teacher and to work collaboratively in the virtual room.

Eight participants agreed or strongly agreed that the interaction with the teacher felt natural and immersive, while the remaining two felt neutral (S8, Figure 12). This was because they would have liked to observe the virtual teacher’s facial expressions and due to occasional virtual hand inaccuracies. In contrast, all participants shared a feeling of co-presence with the teacher within the shared virtual space (S9, Figure 12). Moreover, all users attested to have enjoyed using the application (S10, Figure 12). The responses to statements 8, 9, and 10 confirm that users perceive the application positively in terms of immersion, co-presence, and enjoyment. These consistent positive responses indicate the application’s effectiveness in creating an engaging and authentic learning environment.

In the remote piano lessons user study with the Meta Quest Pro, preferences for virtual hand visibility varied based on participants’ piano proficiency. Intermediate players preferred deactivating virtual hands (S11, Figure 12), while some beginners preferred them, likely due to different levels of confidence and reliance on visual cues. The improved video quality of the Meta Quest Pro influenced this preference compared to the Meta Quest 2 used in the Piano Theory Hub study, where the majority of testers found playing with the virtual hands activated to be easier than just seeing the video feed. They mentioned that without the virtual hands, they could barely see their own hands due to the video quality of the passthrough mode. A piano teacher found the virtual hands confusing because they were slightly shorter than the real hands and the hand tracking was not always perfectly accurate. This created a sense of dissonance between what was seen and what was felt. All participants preferred viewing their virtual hands for UI selections (S12, Figure 12). This was expected since the UI overlays onto the video feed, rendering real hands invisible. When replicating melodies played by the teacher, 9 out of 10 participants leaned toward relying on the visual highlighting of keys on their virtual pianos (S15, Figure 12). In post-exposure interviews, users mentioned that the hand tracking was too imprecise to use it as the only means to observe the teacher’s playing. As hand tracking becomes more precise and reliable, the role of observing hand movements could gain prominence.

The results of the survey show a preference for positioning the teacher across from the student when explaining concepts (S16, Figure 12). Participants articulated that this positioning facilitated more effective observation of the teacher’s hand gestures and created an organic conversational atmosphere, allowing them to observe nonverbal communication cues. However, two intermediate players preferred the teacher beside them, and two others had no preference. This reflects comfort and familiarity with traditional arrangements, highlighting the need to balance familiarity with effective learning dynamics.

6 Discussion

The results of the user studies suggest that the developed MR application can provide remote piano lessons and convey knowledge of piano music theory. The results show notable improvements for both piano beginners and intermediate piano players after engaging with the application, although a bigger study with statistic evaluation needs to be conducted. For piano beginners, there was a notable increase in correctly played notes, accompanied by an improvement in confidence levels. Similarly, intermediate players showed an enhancement in playing chords correctly, coupled with an increase in confidence levels. Moreover, users provided positive feedback regarding the application’s usability, intuitiveness, and motivational aspects. The additional feedback and hints provided using the virtual piano overlay were perceived as helpful and motivating. The display of feedback, fingering, and note names directly above the piano keys made it easy to associate piano keys with note names.

The MR application also allows for remote piano lessons with a teacher, fostering engagement and interaction between students and instructors. Participants felt a sense of co-presence with the teacher within the shared virtual space. The study on remote piano lessons revealed that the platform effectively establishes an engaging and immersive learning environment that participants enjoy using. The application’s success in creating positive user perceptions regarding immersion, co-presence, and enjoyment demonstrates its potential to enhance the remote piano education experience.

We identify that the following MR strategies could be employed for piano education.

$•$ Hand interaction coaching: Providing a hand interaction coaching session is important, especially for users unfamiliar with MR technology.

$•$ Virtual hand visibility: Users should have the option to toggle virtual hands. Most users preferred to play the piano without seeing the virtual hands, in part due to occasional imprecise hand tracking causing misalignment. However, all participants preferred to have the virtual hands visible during UI interactions to avoid being distracted by UI overlays on the real hands.

$•$ Key highlighting: Key highlighting should be implemented as a supporting feature. It can serve as practical feedback for correct or incorrect play and aid users in visually identifying the pressed keys by the remote user.

$•$ Multiple forms of feedback: The feedback displayed on the virtual piano proved the most helpful and utilized by participants in the user study. However, to suit different learning styles, feedback should be provided through several means such as visual cues on the piano keys, on-screen text, auditory signals, and potentially haptic feedback in the future.

$•$ Gamified learning: Gamification elements, such as interactive games within the Piano Theory Hub, should be integrated for controlled self-study but also as a tool to be used by piano teachers during piano lessons. Users found these games to be fun and engaging. In addition, aspects of structural gamification can motivate students to learn the fundamentals of music theory: All participants in the self-study user test felt motivated to improve and beat their high scores by seeing the number of correct notes played after an exercise. Students also felt motivated and rewarded by auditory feedback.

$•$ Balancing familiarity and optimal learning dynamics: A balance between providing familiar experiences for intermediate players accustomed to traditional, on-site lessons and optimizing the learning dynamics offered by the MR environment should be achieved.

$•$ Teacher positioning: Flexible positioning of the instructor, whether facing or beside the student, allows for a wider range of teaching scenarios to meet different needs. Most participants preferred to have the teacher face them during concept explanations, allowing for nonverbal communication. However, preferences for teacher positioning can vary based on user comfort and familiarity. Some users may prefer the traditional on-site piano lesson arrangement, where the teacher is adjacent to the student. In the context of replicating melodies guided by the teacher, the specific positioning of the teacher appeared to carry less significance. Participants leaned toward relying on the visual highlighting of keys on their virtual pianos. As hand tracking becomes more precise and reliable, the role of observing hand movements could gain prominence, potentially influencing optimal positioning strategies.

6.1 Limitations and future work

Currently, the application currently only supports pianos with 88 keys and MIDI output. For MIDI keyboards, an adapter is required to connect to both the HMD and sound output device via cable. Bluetooth could be an option, but its slow transfer rate makes it unsuitable for real-time evaluation due to increased latency. Moreover, hand tracking was occasionally inaccurate, and participants often found the headset uncomfortable for prolonged wear. Additionally, using the application requires openness to newer teaching methods from both teachers and learners. While the user study highlights the application’s effectiveness in improving piano theory skills, notably in chord and note accuracy, it is important to note that the study does not make direct comparisons with other learning methods nor asserts superiority over alternative approaches. Future research should consider isolating the MR component to better understand its unique contributions to the learning process.

The application could be extended to include machine learning to assess the user’s performance, identify areas for improvement, and recommend exercises based on the user’s learning needs. This could also expand to an AI companion that is an animated character that can demonstrate exercises, show correct body posture when playing the piano, or analyze optimal fingerings for songs. Moreover, multi-user games could be integrated to increase motivation and engagement. In addition, although a user study showed the effectiveness of the application in teaching music theory, further research is needed to compare it with other self-study/teaching methods such as apps, videos, or books. An extended user study should be conducted to track user progress and usability over time.

6.2 Conclusion

Mastering the piano takes time and patience. A solid foundation in music theory is essential for long-term success. Thereby, this paper includes features for both multi-user remote piano lessons and self-study using the proposed Piano Theory Hub. For remote piano lessons a networked piano lesson setup, as well as two multi-user musical games and a theory area, were developed. Additionally, the Piano Theory Hub introduces four music theory modules for Notes and Keys, Intervals, Scales, and Chords, each consisting of theory and practice components. Integrated structural gamification elements provide users with feedback and track their progress in completing exercises. Improvements were observed for both piano beginners and intermediate players, accompanied by increased confidence levels and positive feedback regarding the application’s usability and motivational aspects.

Main strategies for creating an interactive piano learning experience using MR technology were identified, including implementing hand interaction coaching, offering toggling options for virtual hand and piano visibility, integrating key highlighting for feedback, providing multiple forms of feedback, incorporating gamified learning elements, balancing familiarity with optimal learning dynamics, and allowing flexible teacher positioning to leverage different preferences and needs. These findings highlight the potential of MR technology to enhance piano education, offering personalized and engaging learning experiences.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

Ethical approval was not required for the studies involving humans because the proposed paper and study involves a low-risk MR interface, such as basic usability testing and exploration of interaction techniques, and does not pose a significant risk to participants. During the study, non-sensitive data is collected anonymously (e.g., tracking user interactions in an MR environment without personal identifiers). Standards for transparency in data handling and informed consent have been respected”. The studies were conducted in accordance with the local legislation and institutional requirements. Written informed consent for participation was not required from the participants or the participants’ legal guardians/next of kin in accordance with the national legislation and institutional requirements because Recorded verbal informed consent was obtained from the participants. Written informed consent was obtained from the individual(s) for the publication of any potentially identifiable images or data included in this article.

Author contributions

VA: Writing–original draft, Writing–review and editing. KC: Writing–review and editing. LE: Writing–review and editing. MM: Writing–review and editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. The publication of the paper is financed by the DFG-sponsored publication fund of the TU Dresden/SLUB.

Acknowledgments

The authors thank the participants of the user studies. The piano model provided in Unity-MIDI-Piano under the MIT License was used as a basis for the 3D model of the piano. The USB Midi Android Plugin available on the Unity Asset Store manages the MIDI input. The remote setup was based on the Mixed Reality Toolkit (MRTK) example available on GitHub.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/frvir.2024.1397154/full#supplementary-material

Supplementary Video | A brief demonstration of key features, including visibility controls, real-time feedback for piano practice, and the theory area.

Footnotes

¹https://www.meta.com/de/quest/products/quest-2/

²https://www.meta.com/de/quest/quest-pro/

References

Banquiero, M., Valdeolivas, G., Trincado, S., García, N., and Juan, M. C. (2023). Passthrough mixed reality with oculus quest 2: a case study on learning piano. IEEE Multimed. 30, 60–69. doi:10.1109/mmul.2022.3232892

CrossRef Full Text | Google Scholar

Birhanu, A., and Rank, S. (2017). “Keynvision: exploring piano pedagogy in mixed reality,” in Extended abstracts publication of the annual symposium on computer-human interaction in play, 299–304.

CrossRef Full Text | Google Scholar

Chow, J., Feng, H., Amor, R., and Wünsche, B. C. (2013). Music education using augmented reality with a head mounted display. In Proc. Fourteenth Australas. User Interface Conference, 139. 73–79. doi:10.5555/2525493.2525501

CrossRef Full Text | Google Scholar

Chyu, Y. E. (2004). Teaching improvisation to piano students of elementary to intermediate levels. Columbus, OH: The Ohio State University.

Google Scholar

Dannenberg, R. B., Sanchez, M., Joseph, A., Capell, P., Joseph, R., and Saul, R. (1990). “An expert system for teaching piano to novices,” in Icmc.

Google Scholar

Das, S., Glickman, S., Hsiao, F. Y., and Lee, B. (2017). “Music everywhere–augmented reality piano improvisation learning system,” in Proceedings of the international conference on new interfaces for musical expression (NIME) (Copenhagen, Denmark: Aalborg University Copenhagen), 511–512.

Google Scholar

Demirel, S. (2022). A research on the design and use of colored notes for children in music education. Shanlax Int. J. Educ. 10, 11–20. doi:10.34293/education.v10is1-aug.5181

CrossRef Full Text | Google Scholar

Duffy, S., and Healey, P. (2017). A new medium for remote music tuition. J. Music, Technol. and Educ. 10, 5–29. doi:10.1386/jmte.10.1.5_1

CrossRef Full Text | Google Scholar

Guo, R., Cui, J., Zhao, W., Li, S., and Hao, A. (2021). “Hand-by-hand mentor: an ar based training system for piano performance,” in 2021 IEEE conference on virtual reality and 3D user interfaces abstracts and workshops (VRW) (IEEE), 436–437.

CrossRef Full Text | Google Scholar

Hackl, D., and Anthes, C. (2017). “Holokeys-an augmented reality application for learning the piano,” in Forum media technology, 140–144.

Google Scholar

Jaques-Dalcroze, E., and Rothwell, F. (1932). Rhythmics and pianoforte improvisation. Music and Lett., 371–380. doi:10.1093/ml/xiii.4.371

CrossRef Full Text | Google Scholar

Kuo, Y.-T., and Chuang, M.-C. (2013). A proposal of a color music notation system on a single melody for music beginners. Int. J. Music Educ. 31, 394–412. doi:10.1177/0255761413489082

CrossRef Full Text | Google Scholar

Labrou, K., Zaman, C. H., Turkyasar, A., and Davis, R. (2023). Following the master’s hands: Capturing piano performances for mixed reality piano learning applications. New York, NY, USA: Association for Computing Machinery.

Google Scholar

Li, L. (2018). Application of augmented reality technology in piano teaching system design. Educ. Sci. Theory and Pract. 18. doi:10.12738/estp.2018.5.070

CrossRef Full Text | Google Scholar

Mei, B., and Yang, S. (2021). Chinese pre-service music teachers’ perceptions of augmented reality-assisted musical instrument learning. Front. Psychol. 12, 609028. doi:10.3389/fpsyg.2021.609028

PubMed Abstract | CrossRef Full Text | Google Scholar

Molero, D., Schez-Sobrino, S., Vallejo, D., Glez-Morcillo, C., and Albusac, J. (2021). A novel approach to learning music and piano based on mixed reality and gamification. Multimedia Tools Appl. 80, 165–186. doi:10.1007/s11042-020-09678-9

CrossRef Full Text | Google Scholar

Molloy, W., Huang, E., and Wünsche, B. C. (2019). “Mixed reality piano tutor: a gamified piano practice environment,” in 2019 international conference on electronics, information, and communication (ICEIC) (IEEE), 1–7.

CrossRef Full Text | Google Scholar

Okan, H. (2022). Conservatory voice department students’ opinions on piano and accompaniment courses in The Covid-19. Gazi Üniversitesi Gazi Eğitim Fakültesi Dergisi (Gazi University) 42, 1791–1820. doi:10.17152/gefad.1078466

CrossRef Full Text | Google Scholar

Oore, S., Simon, I., Dieleman, S., and Eck, D. (2017). “Learning to create piano performances,” in NIPS 2017 workshop on machine learning for creativity and design.

Google Scholar

Rhodes, C., Allmendinger, R., Jay, C., and Climent, R. (2023). Towards developing a virtual guitar instructor through biometrics informed human-computer interaction. Ext. Abstr. 2023 CHI Conf. Hum. Factors Comput. Syst., 1–8. doi:10.1145/3544549.3582738

CrossRef Full Text | Google Scholar

Rigby, L., Wünsche, B. C., and Shaw, A. (2020). “piarno-an augmented reality piano tutor,” in Proceedings of the 32nd Australian conference on human-computer interaction, 481–491.

CrossRef Full Text | Google Scholar

Rogers, K., Röhlig, A., Weing, M., Gugenheimer, J., Könings, B., Klepsch, M., et al. (2014). “Piano: faster piano learning with interactive projection,” in Proceedings of the Ninth ACM International Conference on Interactive Tabletops and Surfaces, 149–158.

CrossRef Full Text | Google Scholar

Sandnes, F. E., and Eika, E. (2019). “Enhanced learning of jazz chords with a projector based piano keyboard augmentation,” in International conference on innovative technologies and learning (Springer), 194–203.

CrossRef Full Text | Google Scholar

Serafin, S., Erkut, C., Kojs, J., Nilsson, N. C., and Nordahl, R. (2016). Virtual reality musical instruments: state of the art, design principles, and future directions. Comput. Music J. 40, 22–40. doi:10.1162/comj_a_00372

CrossRef Full Text | Google Scholar

Sha, L. (2019). Analysis of problems and reform methods in modern piano teaching.

Google Scholar

Shockley, R. P. (2001). Mapping music: for faster learning and secure memory AR Editions, Inc.

Google Scholar

Stanbury, A. J., Said, I., and Kang, H. J. (2021). “Holokeys: interactive piano education using augmented reality and iot,” in Proceedings of the 27th ACM Symposium on Virtual Reality Software and Technology, 1–3.

CrossRef Full Text | Google Scholar

Tan, K. L., and Lim, C. K. (2018). Development of traditional musical instruments using augmented reality (AR) through mobile learning. AIP Conf. Proc. 2016, 020140. doi:10.1063/1.5055542

CrossRef Full Text | Google Scholar

Tan, S.-L., Wakefield, E. M., and Jeffries, P. W. (2009). Musically untrained college students’ interpretations of musical notation: sound, silence, loudness, duration, and temporal order. Psychol. Music 37, 5–24. doi:10.1177/0305735608090845

CrossRef Full Text | Google Scholar

Wang, X. (2020). Ppvr: teaching and learning to play the piano in virtual reality.

Google Scholar

Xiao, X., and Ishii, H. (2010). Mirrorfugue: communicating hand gesture in remote piano collaboration. New York, NY, USA: Association for Computing Machinery.

Google Scholar

Xiao, X., Puentes, P., Ackermann, E., and Ishii, H. (2016). “Andantino: teaching children piano with projected animated characters,” in Proceedings of the the 15th international conference on interaction design and children, 37–45.

CrossRef Full Text | Google Scholar

Xiao, X., Tome, B., and Ishii, H. (2014). “Andante: walking figures on the piano keyboard to visualize musical motion,” in Nime (Cambridge, MA), 629–632.

Google Scholar