Collaborative music-making: special educational needs school assistants as facilitators in performances with accessible digital musical instruments

The field of research dedicated to Accessible Digital Musical Instruments (ADMIs) is growing and there is an increased interest in promoting diversity and inclusion in music-making. We have designed a novel system built into previously tested ADMIs that aims at involving assistants, students with Profound and Multiple Learning Disabilities (PMLD), and a professional musician in playing music together. In this study the system is evaluated in a workshop setting using quantitative as well as qualitative methods. One of the main findings was that the sounds from the ADMIs added to the musical context without making errors that impacted the music negatively even when the assistants mentioned experiencing a split between attending to different tasks, and a feeling of insecurity toward their musical contribution. We discuss the results in terms of how we perceive them as drivers or barriers toward reaching our overarching goal of organizing a joint concert that brings together students from the SEN school with students from a music school with a specific focus on traditional orchestral instruments. Our study highlights how a system of networked and synchronized ADMIs could be conceptualized to include assistants more actively in collaborative music-making, as well as design considerations that support them as facilitators.

1. Introduction

The field of research dedicated to Accessible Digital Musical Instruments (ADMIs), i.e., accessible musical control interfaces used in electronic music, inclusive music practice, and music therapy settings (see Frid, 2019a), is growing. With an increased interest in exploring how different accessible music technologies can be used to promote diversity and inclusion in music-making and musicking (Small, 1998) comes a need to understand, design, and possibly also evaluate such systems. Although attempts have been made to introduce design principles and classification methods based on different use cases (Frid, 2019b; Davanzo and Avanzini, 2020; Harrison, 2020) and to investigate the potential of existing ecological frameworks in design and evaluation of ADMIs (e.g. the Human Activity Assistive Technology and the Matching Person and Technology frameworks deployed by Lucas et al. 2021), there is still no commonly accepted or established evaluation methodology for ADMIs. The topic of evaluation has been widely debated in the fields of New Interfaces for Musical Expression (NIME) and Digital Musical Instruments (DMI) over the years (see e.g. Greenberg and Buxton, 2008; Johnston, 2011; Barbosa et al., 2015). Lately, researchers have emphasized the importance of considering the sociocultural context of the musical interaction when trying to understand what makes “a good musical instrument” (see e.g. Jack et al., 2020; Rodger et al., 2020). In this paper, we describe the context-specific design and evaluation of a master controller instrument developed for assistants at a Special Educational Needs (SEN) School. More specifically, we explore how such an interface should be designed to support the assistants' roles as facilitators in a musical performance with a group of students with Profound and Multiple Learning Disabilities (PMLD)^1,2 who are playing ADMIs together with a professional musician.

The field of Digital Musical Instruments (DMIs) has historically largely focused on borrowing tools and frameworks from Human-Computer Interaction (HCI) (see notable work on evaluation by Wanderley and Orio, 2002, for example). Such approaches often build on the idea that DMIs can be described as “devices” with certain properties that can be evaluated from a “usability” and “accessibility” perspective. Employing HCI concepts directly to musical instruments is, however, something that should be done with care, since this might lead to an oversimplification of the complexity of musical interactions. While the framework proposed by Wanderley and Orio (2002) indeed is useful in certain contexts, it also has drawbacks. The reduction of musical interaction to simple tasks may compromise the authenticity of the interaction; since musical interactions involve creative and affective aspects, they cannot simply be described as tasks with completion rates (Stowell et al., 2009). Frameworks that emphasize the different perspectives in instrument design, such as that of the listener, the performer, vs. the instrument constructor, among others (Kvifte and Jensenius, 2006, see also O'Modhrain, 2011) have been proposed to address (some of) these issues. The need to move away from task-based views of musical performance (see for example El-Shimy and Cooperstock, 2016) and instead take a more holistic view of music activities has been stressed. Rodger et al. (2020) argue that the functional properties of an instrument can only be meaningfully understood relative to the capabilities of a specific musician at a specific period in their musical development. In other words, it is difficult to fully comprehend what a musician does with an instrument if disconnected from the immediate and extended sociocultural context. Building on this notion, they propose to view instruments as “processes” rather than “devices”, and musicians as “agents” in “musical ecologies”, rather than “users”.³ Jack et al. (2020) propose, based on the idea of “performance eco-systems” discussed by Waters (2007), to view DMIs as situated and ecologically valid artifacts which should be evaluated using qualitative and reflective processes focusing on sociocultural phenomena rather than first-wave HCI techniques. Similarly, Lucas et al. (2021) explored ecological perspectives of human activity in the use of DMIs and assistive technology. To conclude, numerous researchers have stressed the importance of shifting focus from the technical aspects of DMIs to the sociocultural contexts in which musical interactions are taking place.

In the current paper we describe an exploratory study focused on a specific scenario in which a group of students with Profound and Multiple Learning Disabilities (PMLD) at a Special Educational Needs (SEN) school in Sweden⁴ interacted with a set of ADMIs – the Funki instruments – in collaborative music-making sessions together with their assistants and a professional musician (an accordionist improvising over a four-chord progression). The Funki instruments are tangible ADMIs that allow for different modes of interaction to create sound by physically engaging with sliders, buttons, and touch pads, embedded in wooden boxes (Figure 1). In this paper, we use the term “assistant” to include any personnel attending these sessions, such as teachers, teaching assistants, or carers. Our work is part of a longer study with the overall goal of organizing a joint concert that brings together students from the SEN school with students without disability enrolled in a music school focused on traditional orchestral instruments. The music-making aspects that we focus on in this study are primarily those of a goal-oriented composition with a relatively well-defined structure. Our research is largely inspired by the “Able Orchestra”,⁵ a project based on the principle of enabling people to create and perform music on equal terms, regardless of their physical dexterity or musical experience. The Able Orchestra allows musicians using technology to join acoustic instrumentalists and orchestral players to create new work in an ensemble (Orchestras Live, 2023).

FIGURE 1

Figure 1. The three Funki instruments used in the study, with different controllers. From the left: two touch-sensitive pads, two momentary buttons, and two sliders. The black area is the backside of the loudspeaker.

To reach the above-discussed goal, multiple challenges need to be addressed. Besides aspects such as instrument design and the sociocultural setting of a joint concert, one challenge is that the music to be performed, i.e., the composition, will have musical boundaries in terms of tonality and chord changes. The students from the music school, which is focused on traditional orchestral instruments, read scores and are trained to follow a shared tempo. However, neither the students nor the assistants at the SEN school have this type of musical training. The students at the SEN school also have multifunctional physical and intellectual challenges which make it difficult for them to play notes at pre-defined fixed timings. Overall, it should be noted that the musical activities held at the two schools differ a lot in their focus; music sessions at the school dedicated to traditional orchestral instruments generally put emphasis on performance aspects, while music sessions at the SEN school focus more on aspects such as active participation, personal development, and social aspects of music-making. To explore how we could merge these two musical practices into one musical performance context, we designed a dedicated system. The design of the system was based on allowing an assistant to control certain aspects of the music, i.e., the chord changes, and to communicate with a professional musician who improvised over a pre-defined chord progression. We organized music-making sessions at the SEN school to explore how a master controller instrument developed specifically for the assistants should be designed to support their role as facilitators in a performance context. The aim of the music-making sessions was to explore the interplay between assistants and the professional musician that may arise in this context, and what is required for such a session to be successful.

Although student assistants have existed as a support function for students in the Swedish school system since the 1960s, the profession has not received much attention in the Scandinavian educational literature, and research on this role is scarce, with very few official statistics describing the group (Östlund, 2017). Overall, literature exploring the advantages vs. disadvantages of including assistants in SEN settings appears to be somewhat inconclusive, with work pointing toward both potential benefits and risks. For example, a study on how help is provided for pupils with physical disabilities and how school assistants influence their participation published by Hemmingsson et al. (2003) suggested that the assistants could both facilitate and hinder student participation. More specifically, the study highlighted the dilemma of closeness and distance; an overly distant approach to the pupil can result in an experience of alienation, but a too-close relationship may result in the student not participating in the challenges faced by other pupils, leading to less independence. Recommendations on how to enhance the efficacy and practices of teaching assistants are discussed in the review of studies on inclusive classrooms presented by Sharma and Salend (2016).

The ADMIs used in this study, the Funki Instruments, have previously been used in experiments in the same SEN school (without a professional musician) for a total of eight half-hour music-making sessions (see Svahn et al., 2021). Findings from this work suggested that the instruments provided a foundation that allowed the students to play together in a band; different instruments allowed students with different abilities and needs to take part, and each student could find some instrument that they liked. However, the results also indicated that musical interactions and group dynamics were highly dependent on the participation of session assistants. In other words, it is important to consider the role of assistants in the music-making taking place at the school, and further research is needed to understand how the assistants' roles as facilitators can be supported by providing tools that may enable more active participation. It should be noted that the assistants at the specific SEN school described in this paper have little or no music training. They also have limited experience with music technology overall. The assistants often have a lot of responsibilities in everyday situations, and therefore they usually have little headroom to learn or take on new tasks. In the previous study, Svahn et al. (2021) concluded that it is important that the assistants understand both the student's needs as well as their interaction with the musical instrument, in order to be able to successfully engage the students in musical activities. The added master instrument described in the current paper should not only be easy to play and allow the assistant to support the students in their music-making; it should also complement the other instruments with sounds that contribute to the musical context. By designing a master instrument according to these constraints, we aim to support the assistants to become more actively involved in the music-making process, thereby also facilitating the students' performances on their instruments.

The current work expands on the previous study by Svahn et al. (2021) by embedding WebAudioXML (Lindetorp and Falkenberg, 2022) into the previous versions of the Funki instruments. This allows for expanded communication between the instruments to let interactions from multiple users control sounds according to mapping rules. WebAudioXML can through these mappings be used to connect the Funki instruments to a master instrument controlling musical aspects such as tonality, rhythmic density, and structure of the composition. In the configuration used for this particular study, we focused especially on tonality. The master instrument intended to be used by the assistants could both trigger sounds in itself and control the pitches of the students' instruments. The metaphor that inspired us during this design process was that of a child strumming guitar strings, with a parent simultaneously changing the chords by moving the fingers on the fretboard. Building on this metaphor, the master instrument acted as a harmony controller for the three student instruments; they all changed pitches according to the rules set by the assistant.

With the above-described setup, the students played along with a professional musician in a performance repeated three times, each with a new assistant. This setting allowed us to explore aspects of collaboration in the music-making taking place between the participants. The sessions were video recorded and the assistants' controller events were tracked and analyzed. This was followed by semi-structured interviews with the assistants and the musician, focusing on how the setup supported inclusive music-making in the group and the assistant's role in this context. Our findings highlight how ADMI systems could be conceptualized and designed to include assistants more actively in collaborative music-making involving a professional musician, and design considerations that may support the assistants' facilitator role in such contexts.

2. Related work

As previously mentioned, this paper builds on work by Svahn et al. (2021), in which a set of four collaborative AMDIs for students with Profound and Multiple Learning Disabilities (PMLD) were developed and tested with a group of students at a Swedish SEN school. Students with PMLD may express themselves using several different communication methods, depending on what is most efficient for them at the time. Pre-verbal students who do not have verbal communication skills yet can use bodily gestures, nonverbal sounds, pointing, and facial expressions to express themselves. To support communication with students with PMLD, alternative methods for augmented communication, such as PODD (Pragmatic Organization Dynamic Display) (Light, 1988; Porter and Cafiero, 2009) can be used. In a related previous study, we invited a group of students from the same SEN school to take part in a 1.5-year-long project focused on musical interfaces and musical haptics (Frid et al., 2022). In this work, we assessed how the students could be involved in the customization and evaluation of the design of a multisensory music experience intended for a large-scale ADMI using a Participatory Design with Proxies (PDwP) methodology. The proxies in this context were teaching assistants and a teacher working with the students, as well as parents. The PDwP method enabled the inclusion of input from different stakeholders providing valuable insights and feedback to augment direct input from the students. Findings highlighted accessibility limitations of the musical interface as well as the importance of using a multifaceted variety of methods to arrive at more informed conclusions when applying a PDwP methodology with pre-verbal students. In the current study, we explore the role of the teaching assistants further, using similar techniques as the ones employed in the previous study, e.g., stimulated recall of videos collected during music-making sessions with the students (see Calderhead, 1981).

To the authors knowledge, little previous work has focused specifically on design considerations for ADMIs that support assistants when musically engaging in ensemble play together with students with PMLD. Despite a long research tradition focused on collaborative music-making using DMIs and NIMEs (see e.g. “Multi-User Instruments” by Jordà, 2005; “Orchestras of Digital Musical Instruments” by Berthaut and Dahl, 2015; and the dimension space for evaluating collaborative musical performance systems by Hattwick and Wanderley, 2012) findings from the systematic review of ADMIs published by Frid (2019a) revealed that few of the surveyed instruments were designed for ensemble settings. Relevant work in this context includes, for example, an exploration of the context and design of collaborative musical experiences for novices published by Blaine and Fels (2003). A discussion of accessibility challenges that people with disabilities face when making music in groups was presented in work by Steinmeier et al. (2022), together with design suggestions. 18 design considerations for DMIs used in SEN settings were proposed by Ward et al. (2017) and a set of design considerations based on interviews with SEN teachers in Germany was recently published by Förster (2022). Access barriers in SEN settings have also been discussed (Davis et al., 2019). Farrimond et al. (2011) mention three access barriers for music technology in special educational and disabled music settings in the UK: a need for specialist training (this was also mentioned by Welch et al., 2016 and Davis et al., 2019), resources, and a fear and dislike or indifference to technology. For example, Davis described issues when asking non-musically trained staff to come up with and deliver music-based activities.

A study on facilitator involvement in a music technology project where participants with complex disabilities⁶ use technology to assist them in music performance was described in Dickens et al. (2018). In this paper, the term “facilitated performance” was used to describe the practice of musical performance involving performers with complex disabilities who are supported by musical experts and other facilitators. Findings suggested that including facilitators in the design of DMIs could allow for improved accessibility for users with complex disabilities, highlighting the importance of the facilitator role in adapting and supporting the use of technology in such contexts. The work also revealed that the social relationships between performer and facilitator were paramount for the success of the project, and that participatory design is a strong design methodology for facilitated performance. The authors describe that facilitation can take many forms throughout the process of supporting interactions with DMIs. This includes prompting, in which participants are encouraged to play and structure performances, for example using tools to inform about the current section of the piece or queuing gestures like pointing at performers when it is their time to play; demonstration, in which demonstrative behavior (including mimetic gestures) is used repeatedly to reinforce understanding and provide reminders; and assisting, for example involving holding equipment in place and making sure that the equipment is positioned correctly. Interfaces developed for the facilitator should focus on providing a simplified baseline level that can be built on, to expand the interface for those with greater technical knowledge. Dickens et al. (2018) suggested that facilitators can be considered gatekeepers to musical activities for performers with complex disabilities since they possess a multitude of knowledge around music performance and technologies and thus are most equipped to communicate this knowledge to the performer. In the current work, we explore the assistants' roles as gatekeepers in musical activities for students with PMLD. However, the assistants described in our work are somewhat different from the facilitators in Dickens et al. (2018), in the sense that the assistants have little to no prior background in musical performance.

3. Materials and methods

The current study focuses on the design and evaluation of a system developed to support the assistants roles as facilitators in music-making sessions with students with PMLD (playing ADMIs) and a professional musician (playing the accordion). In this section, we describe the technical setup, how the instruments communicate with each other, the musical preconditions, and how the user interactions affect the sound.

3.1. Hardware

The setup consisted of four ADMIs: three existing Funki instruments and one new master instrument to be used by an assistant. The prototyping, development, and affordances of the earlier versions of the student instruments are further described in Svahn et al. (2021) and have in accordance with a continued participatory design approach been updated for the purpose of this study, based on findings from long-term use in the SEN school's daily practice. The student instruments are made of wood (30 × 20 × 20 cm) and can be placed on a table, on a wheelchair, or on your lap. Each instrument has an integrated loudspeaker, a microcomputer, and two interactive controllers of the same type (10 cm in size), which can be used to trigger sounds. The controllers are of three different types: trim potentiometer sliders, discrete momentary push-buttons, and touch-sensitive pads made using conductive paint. The wooden boxes are painted in white color, and the controllers are made with different distinct colors (Figure 1). The master instrument differs from the student instruments in several ways. Most importantly, it is not embedded into a wooden box, but designed as a Graphical User Interface (GUI) that runs on a computer. As such, it can be played using the space-bar on the keyboard to trigger a sound, and to control the selection of sounds for the other instruments.

3.2. Software

All instruments are connected to the master instrument, which has several important functions: it stores the samples and audio configurations for the different instruments, it receives the control data from all instruments, it determines how the assistant can control the composition, and finally, it sends out the appropriate sound to each integrated loudspeaker (Figure 2). The master instrument computer runs a server built with Node.js, Socket.IO and Express.js, which gets the control data from all connected instruments through Open Sound Control (OSC)⁷. The instruments send OSC messages continuously to update the system about their current state and the master instrument responds by generating and returning the sounds to the loudspeakers in the corresponding instruments. The logic that controls how the sounds shall respond to the interactions is built using web technologies and is configured using WebAudioXML.

FIGURE 2

Figure 2. The system with three instruments sending control data (OSC) to a Node.js-server. The data is forwarded to the master instrument (in a web client). The instrument data is combined with the master instrument interactions to determine the audio output sent to the loudspeakers of respective instruments.

3.3. Composition

The sounds generated by the instruments are sampled elements of a musical composition. The first author (HL) produced a collection of 42 audio files to respond to various combinations of interactions with the instruments. The overall goal of the sound design was to give the students a natural response to their actions and also to limit the way the sounds could be combined, to ensure that the result would always stay within a defined tonality. This was achieved by creating a collection of sounds for each instrument that could be combined in predefined ways. The master instrument triggers a strummed harp cord, cycling through a chord progression in E major (E, C#m7, F#m7, B11). At the same time, the master instrument will also set the sound of the other instruments to match the chord that is currently playing. The instruments trigger sounds that are composed and arranged to complement each other in terms of pitch and timbre and are selected according to the actions on the instrument, the current chord, and a random factor that adds variation to the performance. A detailed description of the composition represented as a score is displayed in Figure 3.

FIGURE 3

Figure 3. All components of the composition represented as a score. Each green box represents a chord controlled by the master instrument, which plays one of two variations for each chord. The red and yellow boxes indicate the different notes for the two controllers on each instrument. The guitar has the same sequence of notes for the first two and the second two chords, respectively.

3.4. Mapping strategies

The instruments use different gestures to trigger sounds. The final mapping was informed by previous experiments with the Funki instruments conducted with the same user group (Svahn et al., 2021). To make the interactions intuitive, we tried to match the sounds with the actions as closely as possible to acoustic instruments. For example, the instrument with two sliders uses the metaphor of a bow and a string to make the instrument sound only when the user is actively moving the slider. It maps the speed of the slider movement to control the volume of the sound (no movement results in silence). For this instrument, we used a collection of six samples (three for each slider). The second and the fourth chord used the same samples. One slider plays notes in a high octave and the other plays notes one octave higher. The instrument with two buttons triggers samples with a decay similar to the response of depressing a piano key or plucking a guitar string. For this instrument, we use a collection of 20 acoustic guitar samples; each button iterates through a sample set with five different pitches, one set for the E and C#m7 chords, and one set for the F#m7 and B11 chords. One button triggers notes in a medium octave and the other button triggers notes one octave higher. Finally, the instrument with two touch-sensitive pads turns on and off the volume of its sound in an organ-like manner. It is built using eight loops playing samples of synth bass notes. The pads have one set of samples each, with one sample for each chord. One pad triggers pitches in a low octave, and the other pad triggers pitches one octave below. Several testing sessions were devoted to fine-tuning the interactions/gestures to the sounds and deciding how the instruments should respond to the control data from the master instrument.

3.5. Workshop

The evaluation of the system was performed in the form of a workshop with three students (2F, ages=10 & 16 yrs and 1M, age=11 yrs) with Profound and Multiple Learning Disabilities. The students who participated are pre-verbal, have multifunctional physical challenges, varying motor skills, moderate to severe intellectual challenges, and use wheelchairs. We repeated the music-making sessions three times, with the same students, but with three different assistants (3F, ages=25, 47, 52 yrs). The students kept the same instrument in all three sessions. One of the authors (MS) was assisting two of the students while the assistant played the master instrument. Each assistant participated once, i.e., in one session each. Apart from managing the master instrument, the assistants also assisted one of the students. A professional musician (M, age=37 yrs) participated in all three sessions, playing the accordion together with the students and the assistant. The following sections describe these sessions, as well as the data collected, in detail. The first author (HL) was present during all sessions, to observe the interplay in the room, and to provide help with the technical setup if required.

3.5.1. Procedure

Each assistant spent 15 min with the group (three students, one musician, as well as the first and second author). The assistants first got a short introduction to the system by the first author. This introduction focused on the functionality of the master instrument. The assistant and the musician were instructed to communicate, with or without words, about when the assistant should press the key on the master instrument in order to move the composition forward, see Sections 4.1.1, 4.1.2, 4.1.3. The musician then indicated that the music would start. This was followed by an improvised session shaped by the creative possibilities and limitations of the composition outlined in Section 3.3. When the assistant pressed the key on the master instrument, the music advanced to the next chord. This also triggered a sample with the sound of a harp arpeggio and updated the selection of possible samples for the other instruments to harmonize with the currently selected chord. The musician was asked to identify strategies for communicating his intentions to the assistant. He was free to test different approaches during the three different sessions. Video and audio were captured during the three sessions.

3.5.2. Logging of data

The videos were manually tagged by the first author, who is a professional musician and lecturer in music, to identify master instrument onsets. The events were tracked using a MIDI keyboard and a Digital Audio Workstation with video support (we used Logic Pro⁸). The events were exported as a standard MIDI file and converted to JSON data using Tone.js.⁹

3.5.3. Interviews

In a second meeting organized after the workshop, two of the authors (HL, KF) performed a semi-structured interview with the assistants and the musician. These interviews were based on a stimulated recall methodology (Calderhead, 1981), meaning that the assistants were shown replays of video clips from the sessions to stimulate a commentary upon the thought process at that time. The clips were selected by the first author with a focus on when the music started or stopped, if the tempo changed, or if anything special happened that affected the communication between the participants or the music. Each interview session included 2–3 video clips showing instances of communication for a total duration of a couple of minutes. The assistants only watched clips from their own sessions. They were then asked about what happened in the video at that specific time. In addition, the assistants were also asked questions about (1) their musical background, (2) how they communicated with the musician, and (3) how they understood when to press the keyboard on the master instrument. We also asked all participants about suggested improvements. This discussion reflected ideas both for the instruments, the composition, and the way the sessions were structured and organized. All interviews were held in Swedish and the average duration was 18 min. They were recorded and automatically transcribed using the built-in transcription feature in Microsoft Word.

3.5.4. Analysis

The quantitative data (recorded MIDI onsets) was analyzed to calculate chord durations and the deviation of the time between onsets. Deviation here refers to the difference between subsequent chord duration values; this is a measure of how much the assistant deviated from a steady tempo for the chord changes. The MIDI onsets were visualized with positions and durations using the “arrange view” of the Digital Audio Workstation and the graphs showing the deviation for each onset were produced to align the values with the corresponding MIDI onset. Statistical analysis was performed to identify significant differences between assistants.

A thematic analysis of the material collected during the interview sessions was performed, following the procedure outlined by Braun and Clarke (2006). The transcription was saved in a spreadsheet format for easier processing. First, one of the authors (KF) watched the recordings and read the transcriptions, generating a set of initial codes for tagging the responses and collating these into themes. Then, two other authors (HL, EF) systematically tagged the material, adding new codes when relevant, collated the codes into themes, and then reviewed and revised the themes. In the final step, all three authors involved in the tagging searched the material for recurrent themes, discussed the material, and revised and refined the themes to a set that was coherent with the study's objective.

3.5.5. Ethics statement

The research procedure described in this paper was reviewed by the Swedish Ethical Review Authority (application No. 2021-06307-01). The study was carried out in accordance with the declaration of Helsinki. We followed informed consent rules and guidelines for ethical research practices.¹⁰ All parents gave written informed consent before participation and agreed to the data being collected as described in the consent form. It was important to make sure that all students gave consent to participate at all times; this was ensured through direct communication between students and the assistants. The assistants and the musician filled out a consent form and gave written informed consent prior to participation. Management of datasets that include personal information of study participants was compliant with the General Data Protection Regulation (GDPR). Procedures for registration and storage of personal data (including sensitive personal data, Swedish: känsliga personuppgifter) were reviewed and approved by KTHs data protection officer (dataskyddsombud@kth.se) and KTHs Research Data Team. None of the video material is published as Supplementary Material. The full Ethics Approval can be obtained from the Swedish Ethical Review Authority or by emailing the first author. The Ethics Approval report includes all consent forms and information for research subjects (Swedish: personuppgiftsinformation), as well as the Data Management Plan (DMP) approved by KTH officials.

4. Results

The data from the study is both quantitative (tracked events in the form of MIDI data) and qualitative (transcriptions of the interviews). The data are presented separately in the sections below.

4.1. Tracked events

Boxplots of the chord durations for respective assistants are displayed in Figure 4. Since neither the chord duration nor the absolute deviation data met the assumptions required for a One Way Between Groups ANOVA, Kruskal-Wallis Tests were used as omnibus tests to explore potential differences between assistants. The Kruskal-Wallis test revealed a statistically significant difference in chord durations across the three assistants, χ²(2) = 131.01, p < 0.001^***. Post-hoc comparisons using Mann-Whitney U tests revealed significant differences in duration between Assistant 1 (A1) and Assistant 3 (A3), with higher mean rank¹¹ for A1 than for A3 (MR_A1 = 229.9, MR_A3 = 117.5, U = 8786, p < 0.001^***); and between Assistant 2 (A2) and A3, with higher mean rank for A2 than for A3 (MR_A2 = 258.5, MR_A3 = 117.5, U = 9342, p < 0.001^***). No significant difference between A1 and A2 could be observed.

FIGURE 4

Figure 4. Boxplots of chord durations for assistants A1-A3. Whiskers extend to the most extreme data point which is no more than 1.5 times the interquartile range from the box.

A Kruskal-Wallis test revealed a statistically significant difference in absolute deviation (the difference between subsequent chord duration values, i.e., how much the assistant deviated from a steady tempo for the chord change) across the three assistants, χ²(2) = 118.73, p < 0.001^***. Post-hoc comparisons using Mann-Whitney U tests revealed a significant difference in absolute deviation between A1 and A3, with higher median for A1 than for A3 (Mdn_A1 = 1.50s, Mdn_A3 = 0.14s, U = 8775, p < 0.001^***); and between A2 and A3, with higher median for A2 than for A3 (Mdn_A2 = 1.34s, Mdn_A3 = 0.14s, U = 8617, p < 0.001^***). No significant difference between A1 and A2 could be observed.

Overall, the tracked event data suggested that although the setup and overall procedure were equivalent for the three assistants (sessions), some tendencies toward different musical strategies could be observed. The following sections highlight the unique qualities of the respective session.

4.1.1. Assistant 1

Before the music started, Assistant 1 (A1) was instructed that “you and the musician will talk and he will show you when to press the key”. During the first 2 min, A1 was pressing the key in a manner that resulted in different chord durations and the musician was following the chord changes when they happened. After two minutes, the musician started playing a steady pulse, but this did not prompt A1 to synchronize with the pulse. The music stopped after 5 min and 12 s when the musician made eye contact with the assistant to signal that the music had come to an end.

Analysis of tracked chord onsets revealed that A1 pressed the key 48 times, playing rubato without a steady pulse throughout the session. After one minute the chord durations increased (Figure 5). This corresponds to when A1 started assisting the student more actively. The mean ranks of chord duration, as well as the median of the absolute deviation, were significantly higher for A1 compared to those for A3. Moreover, 12 chords were longer than 10 s, corresponding to instances when the assistant focused completely on the student.

FIGURE 5

Figure 5. Tracked key events for assistant A1. The X-axis represents time and the Y-axis represents, top (white): the deviation (in seconds) between the onsets for each chord, compared to the previous chord, and bottom (green): the duration of each chord in the sequence.

4.1.2. Assistant 2

Before the music started, Assistant 2 (A2) was given the following instruction “You and the musician will agree on when you shall press the key”. After that, she waited for the musician to nod before pressing the key, typically one second after the nod. After 30 s, the musician started playing at a steady pulse, but this did not make A2 synchronize with the beat. The music stopped after 4 min and 30 s, when the musician indicated an ending through eye contact, playing slower and slower.

Analysis of tracked onsets revealed that A2 pressed the key 44 times and played rubato without a steady pulse throughout the session. The mean ranks of chord duration, as well as the median of the absolute deviation, were significantly higher for A2 compared to those of A3. Only one chord duration was longer than 10 s (around 2:35–2:45) (Figure 6). This event correspond to when A2 was assisting the student more actively.

FIGURE 6

Figure 6. Tracked key events for assistant A2. The X-axis represents time and the Y-axis represents, top (white): the deviation (in s) between the onsets for each chord, compared to the previous chord, and bottom (green): the duration of each chord in the sequence.

4.1.3. Assistant 3

Before the music started, Assistant 3 (A3) was instructed that “The two of you will find a way of playing together so it sounds like music” and then musician told her to start. During the first minute, A3 took the lead and the musician followed. Then the musician started playing at a steady pulse, but this did not make A3 synchronize with the beat. After two minutes, A3 started assisting the student, who had put the instrument cable in her mouth. This caused the chord duration to increase for a while. After 3 minutes and 30 s, A3 and the musician made a pause. The musician then instructed A3 to follow his tempo, indicated by counting to four. A3 responded by pressing the key on every beat. The musician then instructed A3 to only press the key on the first beat in every bar. This caused A3 to immediately increase the chord duration to the length of four beats. The music stopped after 7 min and 35 s when the musician indicated an ending using eye contact, and by playing slower and slower.

The tracking of the onsets revealed that A3 pressed the key 217 times and kept a relatively steady pulse throughout the session, except for the instances described above. After that the musician had given a clear tempo, A3 kept the tempo even during periods when she needed to assist the student. The outliers in the recorded interval data are caused by external factors, like a double trigger and the pause when the musician gave the second instruction (Figure 7). The mean ranks of chord duration, as well as the median of the absolute deviation, were significantly lower for A3 compared to those of A1 and A2.

FIGURE 7

Figure 7. Tracked key events for assistant A3. The X-axis represents time and the Y-axis represents, top (white): the deviation (in s) between the onsets for each chord, compared to the previous chord, and bottom (green): the duration of each chord in the sequence.

4.2. Interviews

Themes deemed to be coherent with the study's objective, identified by all three authors, are presented and described in the following sections.

4.2.1. Positive feedback

In general, the assistants all provided positive feedback during the interviews. They said that “it was a fun activity” (A1) and that the “musical result was nice” (A3). The assistants also described the students' experience saying that they “were engaged and enjoyed the workshop” (A1), “did really well” (A2), “were relaxed and focused” (A1), and “liked the instruments” (A1). One of the assistants also expressed that “a live musician brings an atmosphere and a musical flow that the group can join”.

4.2.2. The assistants' musical background

With “musical background” we refer to any prior experience of musical activities like singing, dancing, or playing an instrument. All three assistants were asked about their musical background and the answers indicate a range of different backgrounds, spanning from no musical training (apart from music classes at elementary school) (A1), to “I sang in a choir and have always been very interested in music even if I have never given it a chance.” (A3). The musician was asked about which one of the assistants he thought had the most musical training and he guessed (correctly) that it was A3.

4.2.3. The physical interface

With “physical interface” we refer to all hardware used in the sessions. The assistants made a few comments about the instruments, the most common one being that “the students were used to the instruments” (A1, A2). These comments also revealed that there are other aspects of the instruments that might attract more attention from the students, apart from their ability to create sounds. For example, one student showed interest in the instrument by placing the instrument cable in her mouth.

4.2.4. Sound and music

With “sound and music” we refer to all aspects of the sound of the instruments and the composition as a whole. The assistants made comments about the master instrument, for example suggesting that it “adds a layer of structure to the music” (A1). A1 said that the sound of the instruments does not necessarily affect the students' playing experience, but A2 thought that a new version with new sounds would probably stimulate curiosity among the students. The interview also revealed that it was hard for the assistants to identify and remember the different sounds of the different instruments. The musician, on the other hand, commented on the harmonic structure of the composition and that it required a steady pulse and a fixed chord length to make sense. He also mentioned that the volume of the instruments was too weak to match the sound of the accordion and that it would be interesting to explore how the instruments could work with other genres of music (for example, he mentioned that the gestures could affect a soundscape sound in various ways).

4.2.5. Musical interplay/collaboration

With “musical interplay and collaboration” we refer to any aspects related to playing music together, using the Funki instruments. The assistants were generally very positive about participating in the music-making sessions and playing together with a professional musician. A1 said that “It is a very nice feeling to have a setting like a band that plays with an external musician that plays continuously” and A3 commented that “My instrument made me more engaged and gave my student more space to play on her instrument, but yet it felt like we were doing it together”. A comment that highlighted the need for focusing on more than one thing (on both the student, the musician, and on the master instrument) was made by A2, who said: “My student needs eye contact and the confirmation that we're doing this together”. A3 also expressed that “He [the musician] told me how to count and then he followed me really well” indicating that she felt she was in control of the progression of the music. The musician thought “it was hard when it felt like the assistant (A3) was not comfortable with me” but also said that “When I and the assistant play well together, it starts becoming fun for me and I started to get impulses from what the students did”. He expressed that the composition was dependent on a steady beat and that he “would have wanted to play the drums instead [of the accordion] to be able to control the pulse”.

4.2.6. The role of the musician

A1 and A2 expected the musician to lead and expressed that “we had good eye contact” (A1) and that “the musician was very clear” (A2). Even if the video shows that it was the musician that made a count-in to indicate the tempo, A3 said that “we counted to four, I believe, and he followed me”. A2 expressed that she thought the students “listened a lot to the musician”. A3 speculated about the potential advantages of playing with a musician, commenting: “If the students would notice when a musician imitates them, I think it will encourage them to keep on playing”. The musician commented on his role that “It is a quite demanding task for me as a musician to choose a strategy [depending on the assistant]”.

4.2.7. The role of the assistant

The assistants generally describe their role as helping or encouraging the student to play and commented that “I use my hand to guide her to play on the instrument” but A1 also reflected on how she guided the student saying that “I should maybe have let her take more initiatives on her own”. Both A1 and A3 pointed out that imitation is important and A3 said that “When she [my student] saw that I did my thing, she also started doing her thing”.

4.2.8. The role of the students

The assistants described both the actions of listening to the music and playing music when talking about the students. For example, A2 said: “I can see that she is very focused and interested. She likes to explore.” A1 also described: “I think they listened a lot to the musician”. A3 mentioned: “She was active and pressed the buttons. She once unplugged the instrument cable but we got it sorted.” The assistants also pointed out that the situation with a new person and a new instrument in the room, compared to earlier music-making sessions with the Funki instruments when no strangers had participated, requires a lot of effort from the students. A1 commented “I think she was so fascinated by the instrument cables and the accordion so she forgot to play on her instrument”. She also pointed out that “repetition is very important for our students”. One factor that was mentioned in the interviews was that the students' current condition and energy levels can significantly affect their participation and experience overall.

4.2.9. Timing

The composition could be played freely or with a steady pulse. A3 expressed that “It's fun and there is a reward when you keep the beat” and that “It feels like my student also got more active when we started to play in tempo”. A1 pointed out that “it is easier to get into the rhythm when you are relaxed” and A3 said that “it was quite simple to keep the beat”.

4.2.10. Nonverbal communication

With nonverbal communication we refer to any form of communication using eye contact, nodding the head, using body gestures, or changing the way the musician played on his instrument (through musical gestures). When the assistants were commenting on the nonverbal communication, they focused on the communication between the musician and themselves. For example, A3 described that “he [the musician] sought eye contact”. A2 described that “it felt like we had good communication”. The assistants also commented on how they interpreted the students' engagement through their body language. A3 described: “I think I saw that they were active when they moved their hand or arm”. Although the assistants overall made positive comments about the musicians' non-verbal communication, the musician himself described that “It was obvious when I saw the video with A1 that my body language is very unclear”.

4.2.11. Uncomfortable/confused

This theme refers to comments that are related to the problem of not being confident or comfortable with the situation and musical task. For example, A1 and the musician reflected on the need to be relaxed to perform well in the session. A1 said that “I was a bit nervous in the beginning but [then] I understood what to do”. She continued: “I [then] became more relaxed and could get more into the rhythm”. The musician emphasized the importance of the assistant being comfortable with the situation, in order for the communication to work.

4.2.12. Suggested improvements

All respondents emphasized that it is important that the instruments get a more separated and well-defined sound. The musician also mentioned that it would be interesting to try out “different genres and letting the instruments control different aspects of the composition”. The assistants pointed out the importance of having repeated sessions with the same musician, to create a familiar situation for the students. They also suggested that they could have a similar instrument to the student instrument, to better use imitation as a strategy to guide the students. Finally, the musician reflected on how he gave instructions to the assistants and suggested that he should “lead more clearly” by “setting a tempo by counting in”.

5. Discussion

The results presented in Section 4 shed light on many aspects that are important to consider when it comes to the assistants' role as facilitators in music-making sessions with students with PMLD involving a professional musician. The quantitative data from the tracked events highlight how the three assistants understand the task differently, and also how different strategies used by the musician, and perhaps also prior musical training, might affect the results. The qualitative data from the interviews add another perspective that further describes how the assistants and the musician experienced the sessions, with hints about aspects that may require improvement. In this section, we discuss the results in terms of how we perceive them as drivers or barriers toward reaching our overarching goal of organizing a joint concert that brings together students from the SEN school with students from a music school with a specific focus on traditional orchestral instruments.

5.1. Drivers

A common theme that all assistants and the musician emphasize is that the activity to play together using the different instruments, as well as the roles assigned in the particular musical interaction explored in this study, was fun and engaging for everyone involved. This is arguably an important quality of both the setting and the system that shall be valued and fostered in future projects. In the current study, the main contributing factors to this positive experience were likely that the students were familiar with the instruments since before, that the assistants could relax (at least after a couple of minutes of training), and the fact that they all got to play together with a live musician. That being said, it is worth evaluating how the different aspects of the setup affect the participants differently and how the roles, as well as the musical responsibilities, can be shared between the participants.

The tracked event data overlaps well with what could be observed in the video from the respective session, as well as the comments from the interviews about the communication between the assistants and the musician. The assistants generally thought that the musician was clear in his body language and that they understood what to do, after a while. A3 differs by having a much more stable pulse than the others, both when playing freely and when the musician instructed her to follow his tempo. It would be interesting to further investigate what factors contribute to this and to what degree prior musical training plays an important role. The musician's reflection that he would probably count-in more directly if he was to repeat the session suggests that he wanted the music to have a steady pulse. This presence of a steady beat was something that also was appreciated by A3 (once it happened).

When asked about improvements, the assistants mentioned “repetition” as an important driver for the students. This is well-known when it comes to learning to play a traditional musical instrument, but it might easily be overseen in a research setting. Having repeated rehearsals would require more logistics and put certain pressure on the personnel at the school, but it would most certainly also greatly affect the outcome, compared to a session that happens only once (without the chance to practice or improve over time).

Another suggestion from the assistants that remains unexplored is to provide instruments in pairs, so that the assistant can demonstrate to the student how to play, using the same gestures and the same output sounds as the students. Such an approach would make sense given the educational setting; the assistants often demonstrate actions using their hands, moving the students' hands to teach them how to perform specific tasks, thereby encouraging interaction.

Computational systems of musical expression always involve the establishment of a stratum that provides certain affordances to the musician, while simultaneously posing constraints (Magnusson, 2010). In this study focused on the Funki instruments, affordances have to do more with usability,¹² whereas constraints define the limits of the musical expression; the mapping can be viewed as a compositional process that engenders a structure of constraints. The Funki instruments allow for music-making with specific stylistic constraints, within the limits of a specific genre (Pearce and Wiggins, 2002). There are also internal constraints imposed by the system, affecting the logical possibilities of how the music can progress. Finally, there are external constraints, i.e., the need to be sure that the instruments are physically possible to play for the students and the teaching assistant. The composition used in this study was based on an E major pentatonic scale and a matching four-chord progression. In other words, the configuration of the Funki instruments did not allow you to play the “wrong note” or notes out of tune; there are fixed notes and there is no possibility to explore tones in between. There are advantages but also potential pitfalls with such a design (see e.g., the discussion about allowing for maximum participation within defined bounds vs. granting greater individual choice – with the risk of increased frustration—presented in Wright and Dooley, 2019). All elements in the composition worked well together even when the assistant and the musician did not plan the performance in advance or synchronize while playing. This compositional approach made it easy for the musician to improvise alongside the sounds from the other instruments, which in turn might have had an impact on the positive responses from the assistants. A strategy that includes a more predefined and score-based composition would likely introduce a new set of drivers and barriers for the participants. Future studies on different compositional approaches (improvisational vs. goal-oriented) and different strategies used for mapping interactions to sounds would be required to identify how different strategies could impact the music-making sessions and how this—in turn—would affect the level of engagement of all participants, in particular the students with PMLD.

5.2. Barriers

A lot of comments from the assistants focused on barriers to a successful session related to the role of the assistant and the students. Particularly interesting in this context is, arguably, that factors that can be drivers may at the same time become barriers. For example, the presence of a professional musician (playing live) was generally appreciated, but at the same time, this new person captured the students' attention so they forgot to play themselves. The assistants also mentioned that the instruments had properties (apart from the sounds) that made them attractive to the students, e.g., one of the students was at times more occupied by the instrument cables than the act of playing the instrument (using button presses). It would maybe be a good idea to reduce the number of distracting factors, for example by making the instruments wireless, in order to encourage the focus on creating sounds. However, considering that the cables were very interesting to some students, removing the cables completely might have the opposite effect, making the students less interested in the activity overall.

We could conclude that organizing repeated rehearsals probably would contribute to reducing various barriers. This could, for example, result in the students becoming even more familiar with the instruments and the sounds they produce, and allow them to become more used to the professional musician (as well as the acoustic instrument played by the musician). Repeated rehearsal and a sense of recognition of the music-making setting could in this sense perhaps help create a more safe setting for the students, which in turn would allow them to more comfortably engage in musical interactions.

Both the assistants and the musician agreed that unclear body language and lack of instructions create an insecure setting, resulting in the assistant easily getting nervous. This introduces barriers for all participants. These barriers, in combination with the driver of having a familiar context for the students, will be important factors to consider when planning the procedure for future workshops and new iterations of the study described in this paper.

One barrier that was mentioned, which could also be seen in the quantitative data, was that the assistants were somewhat split between the task to assist the student and to control the chord progression by hitting the space key on the master instrument. The main reason for this was that this musical interaction required the assistants to look at the musician while simultaneously maintaining eye contact with the student. A3 managed to synchronize with the musician without looking at him, but such an act requires some musical training. The system should ideally not require that the assistants have musical training to work successfully. A different approach that would be interesting to explore would be to let the assistant focus entirely on helping and encouraging the student to play an instrument, and to introduce another facilitator role; yet another trained musician that controls the master instrument and focuses on the synchronization with the professional musician.

Finally, one aspect that became apparent from this study was that the sound design of the instruments needs to be somewhat adjusted when combined with an acoustic instrument. The musician expressed that the volume on the instruments was too low to match his accordion and a general response was that it was hard to identify the sounds of the different instruments. One possible solution to this could be to use an external speaker that matches the sound of the particular instrument. If the role of the instrument is to play a bass line, the speaker needs to be able to produce low frequencies with an amplitude matching that of an acoustic bass instrument, and if the sound contains a rich spectrum of frequencies, the speaker needs to be able to reproduce all of them with good separation.

5.3. Limitations

We acknowledge that the interactions taking place in the collaborative music-making that is described in this paper are complex to describe, with many different perspectives and factors that may influence one another. In this study, we did not gather data or analyze the interaction between students and their instruments in particular. Instead, the analysis presented in this paper is focusing on the communication between assistants and the professional musician; not on interactions between students and the musician. We have explored the student perspective more in detail in previous work (see e.g., Svahn et al., 2021; Frid et al., 2022). In addition, the Funki ADMIs have been previously validated, and their designs are not a topic of detailed examination in this paper. Although our current paper builds on several previous studies and a multi-year collaboration with the SEN school, this manuscript presents a small-scale study with 3 assistants in total. As such, the results should be interpreted with care, since the small sample size and the long-term collaboration with the assistants affects both generalizability and potential risk for positive bias.

6. Conclusion

This study aimed to explore how a system that was designed for collaborative music-making involving students with Profound and Multiple Learning Disabilities (PMLD) can support assistants in their facilitation of music performance with ADMIs. We carried out three workshop sessions with different assistants together with SEN school students and a professional musician to evaluate the setup using quantitative and qualitative methods. We could identify drivers and barriers for the assistants in using the system, as well as design considerations for similar and continued work focused on collaborative music-making with ADMIs in SEN schools.

One of the drivers that the assistants participating in the study mentioned was how they appreciated contributing to the actual music that was produced during the session. Considering that assistants may have little or no musical training or experience, a system for collaborative playing should thus allow for triggering musical events or changing the performance of the composition without making errors that impact the music. Furthermore, music composed for this setup should allow for unsynchronized, unplanned, and irregular actions from the assistant; else, if the music is more complex or have a goal-oriented structure, it might be better to let a musically trained person be responsible for synchronizing and aligning to a composition or plan.

Another driver mentioned was the possibilities given to allow students to learn by imitating assistants. The assistants stated that the students showed interest in the situation, which may have further positive chain effects such as motivation for repeated rehearsals and increased confidence in participating. To stimulate this, the ADMI design for the assistant should not only be similar to the ones used by the students – thereby promoting learning and enabling them to imitate gestures – but the sounds produced by the ADMIs also need to match each other in terms of volume, timbre, and frequency range, while simultaneously being easy to distinguish from one another, to support the identification of the different sound sources. This is especially important when an acoustic instrument is participating. The hands of the assistant should be in close proximity to the student's own hands during the musical interactions since this allows the assistant to guide the student and encourage participation, demonstrate gestures, and help out when the student gets distracted.

The seemingly biggest barrier for the assistants was an expressed concern about having to split attention between contributing to the music and attending to the student. ADMIs (as well as the composition, and the setup with the musician in the room) should be configured in a way that the assistants do not need to focus visually on the musician or the instrument's interface and thereby lose concentration on the student's gestures and actions. Ideally, the setup should allow the assistant to listen to the sound but maintain visual attention to the student. The main responsibility of the assistant naturally lies with the student and the system should not rely on the assistant's actions to produce a valuable musical output.

Using ADMIs together with a live professional musician and acoustic instruments also results in certain challenges in terms of sound design and composition. We believe that with the new possibilities for collaborations and meetings between different types of players, a system such as the one presented will inspire all participants as long as the resulting musical quality is not compromised. Within this format, even progress toward better artistic inclusion is imaginable.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by Swedish Ethical Review Authority. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin. Written informed consent was obtained from the individual(s), and minor(s)' legal guardian/next of kin, for the publication of any potentially identifiable images or data included in this article.

Author contributions

MS, JH, and HL contributed to the conception and design of the hardware and software. HL contributed to the music composition, audio production, as well as tagging of audio data. MS, EF, HL, and KF contributed to the conception and design of the study. EF contributed to the literature study and performed the statistical analysis. EF, HL, and KF performed the thematic analysis and wrote sections of the manuscript. MS contributed to writing the manuscript. All authors contributed to the manuscript revision, read, and approved the submitted version.

Funding

This research was supported by a joint doctoral program between KMH and KTH and partly funded by the Swedish Research Council (Vetenskapsrådet), grant number 2020-00343. Funki Instruments are funded by KTH Innovation's Incubator program.

Acknowledgments

The authors wish to thank all participating students and personnel at Dibber Rullen in Stockholm, the accordion player Leif Ottosson, and all who have been involved in designing and testing the instruments.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^In this paper, we use Person/People First Language (PFL) when writing about disability, as opposed to Identity First Language (IFL) (see Ferrigon and Tucker, 2019).

2. ^PMLD is commonly used to describe a person with severe learning disabilities who most likely has other complex disabilities and health conditions (Bellamy et al., 2010), although there is no single universally agreed definition of the term, and research has highlighted that no definition can fully articulate the complexities associated with it (Bellamy et al., 2010).

3. ^An ecology in this context corresponds to a system compromising an agent and environment.

4. ^Dibber Rullen, see https://dibber.se/skola/rullens-sarskola/om-oss/.

5. ^See for example https://youtu.be/pf8k3uA3dxM.

6. ^Here defined as “conditions that affect both cognitive and motor abilities of an individual”.

7. ^See: Node.js (https://nodejs.org/), Socket.IO (https://socket.io), Express.js (https://expressjs.com/) and Open Sound Control (https://opensoundcontrol.stanford.edu/).

8. ^“Apple Logic Pro”, see https://www.apple.com/logic-pro/, accessed May 31, 2023.

9. ^“Parse a MIDI file into a Tone.js-friendly JSON format”, see http://tonejs.github.io/Midi/, accessed January 31, 2023.

10. ^APA Ethical Principles of Psychologists and Code of Conduct: https://www.apa.org/ethics/code, CODEX: https://www.codex.uu.se/?languageId=1, ALLEA: https://allea.org/code-of-conduct/, SATORI: https://satoriproject.eu/framework/section-1/.

11. ^We report mean ranks instead of medians since the assumption of equal distribution shape was not met for chord duration; the Mann Whitney U thus explores differences in distributions, not differences in medians (which is the case for the absolute deviation parameter discussed below).

12. ^The affordances and usability of the Funki instruments have been extensively explored in previous experiments (see e.g. Svahn et al., 2021).

References

Barbosa, J., Malloch, J., Wanderley, M. M., and Huot, S. (2015). What does evaluation mean for the NIME community? in Proceedings of the International Conference on New Interfaces for Musical Expression (NIME) (Baton Rouge, LA), 156–161.

Google Scholar

Bellamy, G., Croot, L., Bush, A., Berry, H., and Smith, A. (2010). A study to define: profound and multiple learning disabilities (PMLD). J. Intellect. Disabil. 14, 221–235. doi: 10.1177/1744629510386290

PubMed Abstract | CrossRef Full Text | Google Scholar

Berthaut, F., and Dahl, L. (2015). BOEUF: a unified framework for modeling and designing digital orchestras, in International Symposium on Computer Music Multidisciplinary Research. Berlin: Springer, 153–166. doi: 10.1007/978-3-319-46282-0_10

CrossRef Full Text | Google Scholar

Blaine, T., and Fels, S. (2003). Collaborative musical experiences for novices. J. New Music Res. 32, 411–428. doi: 10.1076/jnmr.32.4.411.18850

CrossRef Full Text | Google Scholar

Braun, V., and Clarke, V. (2006). Using thematic analysis in psychology. Qual. Res. Psychol. 3, 77–101. doi: 10.1191/1478088706qp063oa

CrossRef Full Text | Google Scholar

Calderhead, J. (1981). Stimulated recall: A method for research on teaching. Br. J. Educ. Psychol. 51, 211–217. doi: 10.1111/j.2044-8279.1981.tb02474.x

CrossRef Full Text | Google Scholar

Davanzo, N., and Avanzini, F. (2020). A dimension space for the evaluation of accessible digital musical instruments, in International Conference on New Interfaces for Musical Expression. Birmingham: Birmingham City University, 214–220. doi: 10.5220/0009816106200628

CrossRef Full Text | Google Scholar

Davis, T., Pierson, D., and Bevan, A. (2019). Increasing access to music in SEN settings, in Proceedings of the 16th Sound and Music Computing Conference (SMC 2019) (Málaga), 281–286.

Google Scholar

Dickens, A., Greenhalgh, C., and Koleva, B. (2018). Facilitating accessibility in performance: Participatory design for digital musical instruments. J. Audio Eng. Soc. 66, 211–219. doi: 10.17743/jaes.2018.0010

CrossRef Full Text | Google Scholar

El-Shimy, D., and Cooperstock, J. R. (2016). User-driven techniques for the design and evaluation of new musical interfaces. Comp. Music J. 40, 35–46. doi: 10.1162/COMJ_a_00357

CrossRef Full Text | Google Scholar

Farrimond, B., Gillard, D., Bott, D., and Lonie, D. (2011). Engagement with technology in special educational & disabled music settings, in Youth Music Report (London: National Foundation for Youth Music), 1–40.

Google Scholar

Ferrigon, P., and Tucker, K. (2019). Person-first language vs. identity-first language: an examination of the gains and drawbacks of disability language in society. J. Teach. Disabil. Stud. (New York, NY) 70, 1.

Google Scholar

Förster, A. (2022). Accessible digital musical instruments in special educational needs schools-design considerations based on 16 qualitative interviews with music teachers, in International Journal of Human-Computer Interaction (London: Taylor & Francis), 1–11.

Google Scholar

Frid, E. (2019a). Accessible digital musical instrumentsa review of musical interfaces in inclusive music practice. Multimodal. Technol. 3, 57. doi: 10.3390/mti3030057

CrossRef Full Text | Google Scholar

Frid, E. (2019b). Diverse Sounds: Enabling Inclusive Sonic Interaction. PhD thesis. Stockholm: KTH Royal Institute of Technology.

Google Scholar

Frid, E., Panariello, C., and Nez-Pacheco, C. (2022). Customizing and evaluating accessible multisensory music experiences with pre-verbal children; a case study on the perception of musical haptics using participatory design with proxies. Multimodal Technol. Interact. 6, 55. doi: 10.3390/mti6070055

CrossRef Full Text | Google Scholar

Greenberg, S., and Buxton, B. (2008). Usability evaluation considered harmful (some of the time), in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems, 111–120. doi: 10.1145/1357054.1357074

CrossRef Full Text | Google Scholar

Harrison, J. (2020). Instruments and Access: The Role of Instruments in Music and Disability. PhD thesis. London: Queen Mary University of London.

Google Scholar

Hattwick, I., and Wanderley, M. M. (2012). A dimension space for evaluating collaborative musical performance systems. in Proceedings of the international conference on New Interfaces for Musical Expression (Ann Arbor). doi: 10.5281/zenodo.1178281

CrossRef Full Text | Google Scholar

Hemmingsson, H., Borell, L., and Gustavsson, A. (2003). Participation in school: School assistants creating opportunities and obstacles for pupils with disabilities. OTJR. 23, 88–98. doi: 10.1177/153944920302300302

CrossRef Full Text | Google Scholar

Jack, R., Harrison, J., and McPherson, A. (2020). Digital Musical Instruments as Research Products, in Proceedings of the International Conference on New Interfaces for Musical Expression (Birmingham), 446–451.

Google Scholar

Johnston, A. J. (2011). Beyond evaluation: linking practice and theory in new musical interface design in Proceedings of the International New Interfaces for Musical Expression Conference (NIME). Oslo.

Google Scholar

Jordà, S. (2005). Multi-user instruments: models, examples and promises, in Proceedings of the 2005 Conference on New Interfaces for Musical Expression, 23–26.

Google Scholar

Kvifte, T., and Jensenius, A. R. (2006). Towards a Coherent Terminology and Model of Instrument Description and Design. Paris.

Google Scholar

Light, J. (1988). Interaction involving individuals using augmentative and alternative communication systems: State of the art and future directions. Augment. Alternat. Communicat. 4, 66–82. doi: 10.1080/07434618812331274657

CrossRef Full Text | Google Scholar

Lindetorp, H., and Falkenberg, K. (2022). Evaluating Web Audio for learning, accessibility, and distribution. JAES. 70, 951–961. doi: 10.17743/jaes.2022.0031

CrossRef Full Text | Google Scholar

Lucas, A., Harrison, J., Schroeder, F., and Ortiz, M. (2021). Cross-pollinating ecological perspectives in ADMI design and evaluation, in Proceedings of the International Conference on New Interfaces for Musical Expression, Shanghai, China.

Google Scholar

Magnusson, T. (2010). Designing constraints: composing and performing with digital musical systems. Comp. Music J. 34, 62–73. doi: 10.1162/COMJ_a_00026

CrossRef Full Text | Google Scholar

O'Modhrain, S. (2011). A framework for the evaluation of digital musical instruments. Comp. Music J. 35, 28–42. doi: 10.1162/COMJ_a_00038

PubMed Abstract | CrossRef Full Text | Google Scholar

Orchestras Live (2023). Able Orchestra.

Google Scholar

Östlund, D. (2017). Elevassistenter: En möjlighet eller ett hinder för elevers inkludering och delaktighet. (Kristianstad) 37, 106–117.

Google Scholar

Pearce, M., and Wiggins, G. A. (2002). Aspects of a cognitive theory of creativity in musical composition, in Proceedings of the ECAI02 Workshop on Creative Systems. Lyon.

Google Scholar

Porter, G., and Cafiero, J. M. (2009). Pragmatic organization dynamic display (PODD) communication books: a promising practice for individuals with autism spectrum disorders. Perspect. Augment. Altern. Commun. 18, 121–129. doi: 10.1044/aac18.4.121

CrossRef Full Text | Google Scholar

Rodger, M., Stapleton, P., Van Walstijn, M., Ortiz, M., and Pardue, L. (2020). What makes a good musical instrument? a matter of processes, ecologies and specificities, in Proceedings of the International Conference on New Interfaces for Musical Expression. Birmingham, UK: Birmingham City University, 405–410.

Google Scholar

Sharma, U., and Salend, S. J. (2016). Teaching assistants in inclusive classrooms: A systematic analysis of the international research. Aust. J. Teach. Educ. (Online). 41, 118–134. doi: 10.14221/ajte.2016v41n8.7

CrossRef Full Text | Google Scholar

Small, C. (1998). Musicking: The Meanings of Performing and Listening. Middletown, CT: Wesleyan University Press.

Google Scholar

Steinmeier, C., Becking, D., and Kanders, M. (2022). The perfect musical instrument does not exist: Experience reports for the development of accessible nimes, in Proceedings of the 15th International Conference on PErvasive Technologies Related to Assistive Environments (Corfu), 507–513.

Google Scholar

Stowell, D., Robertson, A., Bryan-Kinns, N., and Plumbley, M. D. (2009). Evaluation of live human-computer music-making: quantitative and qualitative approaches. Int. J. Hum. Comput. Stud. 67, 960–975. doi: 10.1016/j.ijhcs.2009.05.007

CrossRef Full Text | Google Scholar

Svahn, M., Hölling, J., Curtsson, F., and Nokelainen, N. (2021). The Rullen band, in NIME 2021. in Proceedings of the International Conference on New Interfaces for Musical Expression, Shanghai, China.

PubMed Abstract | Google Scholar

Wanderley, M. M., and Orio, N. (2002). Evaluation of input devices for musical expression: Borrowing tools from HCI. Comp. Music J. 26, 62–76. doi: 10.1162/014892602320582981

CrossRef Full Text | Google Scholar

Ward, A., Woodbury, L., and Davis, T. (2017). Design Considerations for Instruments for Users with Complex Needs in SEN Settings (Copenhagen).

Google Scholar

Waters, S. (2007). Performance ecosystems: ecological approaches to musical interaction, in EMS: Electroacoustic Music Studies Network (Leicester), 1–20.

Google Scholar

Welch, G. F., Ockelford, A., Zimmermann, S.-A., Himonides, E., and Wilde, E. (2016). The provision of music in special education (PROMISE) 2015, in Procedings of the International Perspectives on Research in Music Education. Reading: International Music Education Research Centre (iMerc) Press.

Google Scholar

Wright, J., and Dooley, J. (2019). On the inclusivity of constraint: Creative appropriation in instruments for neurodiverse children and young people, in Proceedings of the New Interfaces for Musical Expression Conference (NIME).

Google Scholar