Victor Zappi
Andrew McPherson- 1Department of Advanced Robotics, Istituto Italiano di Tecnologia, Genoa, Italy
- 2Centre for Digital Music, Queen Mary University of London, London, United Kingdom
This paper investigates the appropriation of digital musical instruments, wherein the performer develops a personal working relationship with an instrument that may differ from the designer's intent. Two studies are presented which explore different facets of appropriation. First, a highly restrictive instrument was designed to assess the effects of constraint on unexpected creative use. Second, a digital instrument was created which initially shared several constraints and interaction modalities with the first instrument, but which could be rewired by the performer to discover sounds not directly anticipated by the designers. Each instrument was studied with 10 musicians working individually to prepare public performances on the instrument. The results suggest that constrained musical interactions can promote the discovery of unusual and idiosyncratic playing techniques, and that tighter constraints may paradoxically lead to a richer performer experience. The diversity of ways in which the rewirable instrument was modified and used indicates that its design is open to interpretation by the performer, who may discover interaction modalities that were not anticipated by the designers.
1. Introduction
It is a natural human activity to adapt the objects around us to fit our desired ends. Commonly, the use of an object can differ from its original intended purpose: a screwdriver can open a can of paint; a stack of books can prop up a monitor; email can be sent to oneself to set reminders or transfer files between devices.
Human-computer interaction research has long grappled with the design implications of appropriation, which Dourish (2003) defines as “the way in which technologies are adopted, adapted and incorporated into working practice.” He elaborates that appropriation might involve explicit customisation of a technology, or it might “simply involve making use of the technology for purposes beyond those for which it was originally designed, or to serve new ends.” Though the appropriation of a technology is apparently driven by the user working at odds to the designer, design decisions have a strong influence on appropriation and can encourage or discourage unexpected use (March et al., 2005; Dix, 2007).
This paper investigates the design implications of appropriation in creative contexts, focusing on the design of digital musical instruments (DMIs). The history of musical instruments is tightly bound up with unexpected creative use: from the saxophone to the electric guitar to the turntable, the playing techniques we associate with many iconic instruments were not those originally envisaged by the designer. The complex interdependence of music technology and culture is explored in more detail in Bijsterveld and Peters (2010).
This paper presents two studies involving new DMIs, each exploring a different aspect of appropriation. The first study, briefly summarized here and presented in more detail in Zappi and McPherson (2014b), explores unexpected creative use of a deliberately highly constrained instrument. The second study, presented in detail here, concerns modification and hacking using an instrument which could be rewired by the performer in an open-ended way, enabling them to elicit patterns of behavior that were not explicitly envisioned by the designers.
1.1. Success and Failure of Creative Interfaces
The traditional view of technology design holds that objects or systems are designed for specific well-defined purposes, and they can then be evaluated on how well they achieve those purposes. Within the domain of digital musical instruments, first-wave HCI notions of control and accuracy have been applied to instrument evaluation (Wanderley and Orio, 2002), while other perspectives on evaluation consider complexity and learning curve (Jordà, 2004) or broader groups of stakeholders beyond the performer (O'Modhrain, 2011). In general, robustly measuring the effectiveness of creative tools remains a challenge though general-purpose metrics have been developed such as the Creativity Support Index (Cherry and Latulipe, 2014).
A particular challenge in evaluating the effectiveness of creative tools is that the technological failures can be more interesting than the ostensible successes (Cascone, 2000; Kim et al., 2015). In some cases, while the original design goal of an instrument may not be fully realized, unexpected new behaviors and playing techniques may be discovered that prove to be more interesting than the original goal (McPherson and Kim, 2013).
A useful concept for understanding the interaction between player and instrument is the affordance, which originates from ecological psychology and has been variously used by Gibson (1966), Norman (1988), Gaver (1991) and others to refer to the ways in which an object suggests or allows possible actions by a human user. Magnusson (2010) notes the importance of affordances in musical interfaces, while observing that performers learning an unfamiliar instrument tend to explore its constraints rather than systematically engaging with the affordances of the instrument. The process of constraint exploration may be closely coupled to the appropriation of an instrument, as the performer discovers techniques and outcomes that may not be intended by the designer.
Kirsh (2014) notes the importance that serendipitous discovery can play an important role in the uptake of creative interfaces. Tangibility may facilitate this discovery process. Kirsh (2013), citing Gibson (1966), writes: “the by now classic position of embodied cognition is that the more actions you can perform the more affordances you register (e.g., if you can juggle you can see an object as affording juggling).” The importance of discovery has been noted in several tangible and reconfigurable musical interfaces (Newton and Marshall, 2011; Xambó et al., 2013; Zamborlin, 2015).
1.2. Customization, Modification and Openness
Not all tools are fixed in their purpose or design. Customization and personalization are important factors in tool design (Buechley et al., 2009; Marathe and Sundar, 2011; Zamborlin, 2015). However, appropriation does not require the designer to explicitly make a tool customisable (Dourish, 2003). Even the simplest, most constrained tool can be used in idiosyncratic ways, without modification. The practice of circuit bending (Ghazala, 2005; Collins, 2008), or exploratory rewiring of commercial electronic devices, shows how resourceful “users” can modify or hack even apparently closed systems. Nonetheless, just as Dix (2007) makes the case that design can encourage or discourage appropriation, Galloway et al. (2004) argue that design decisions can influence a tool's suitability for hacking: “Hackability implies more than simple customization or adaptation—it calls for redefinition. Design for hackability involves creating spaces for play where people are never forced to adapt to technology.”
We might ask whether tools which are more open-ended are better suited to appropriation and hacking. Höök (2006) argues for open-ended tools as drivers of appropriation: “Our aim is to discuss how we can create designs that, within certain limits, allow users to appropriate the system once it is built.…This does not mean that we design tools that are empty to start with and where the whole content is given by the user. Instead, we try to design ambiguous, abstract representation that open up for some kind of familiarity–in recognition of the representations from our daily social, emotional and bodily interactions with the world.” Gaver et al. (2003) and Sengers and Gaver (2006) likewise highlight the values of openness and ambiguity in design, while Seok et al. (2014), drawing on the non-finito technique in visual arts, argue for “unfinished” products as a way of engaging user creativity.
The term user is itself coming into question in HCI discourse. Redström (2006) critiques the common practice of user-centred design as risking “becoming a kind of user design.…We, as designers, turn people into users by means of our designs, by presenting a thing to be used. By making the desired interpretation of the objects obvious and impossible to resist, we aim to design not only the object itself but also the perception, and even the experience, of it.” Roedl et al. (2015) examine the shift in terminology from user toward terms such as maker, crafter or hacker that acknowledge the fuzzy boundary between creators and operators of technology.
1.3. Research Aims
This paper explores to what extent it is possible to design digital musical instruments with the goal of being appropriated and hacked by the performer. It examines the performer's actions and experiences in relation to two different design approaches: a highly restrictive and limited instrument, and an instrument whose internal circuitry can be rewired in an open-ended way. Following the presentation of two studies examining these approaches, we discuss to what extent the performer's actions should be considered appropriation and hacking, and we reflect on how our designs and outcomes differ from more conventional digital or software instruments.
2. The Cube Instrument Study
This section briefly presents a study of the performer reaction to a restrictive and limited instrument, further details of which can be found in Zappi and McPherson (2014b). Constraints can be a powerful creative motivator, and this study seeks to assess whether a highly constrained musical situation will encourage appropriation of the instrument.
Our work extends Gurevich et al. (2010), who found that performers of a simple one-button instrument developed a wide range of stylistic variations and techniques based on unexpected usage of the instrument. In this study, we sought to validate those results on a different constrained instrument. We also investigated whether the dimensionality of the instrument – how many control dimensions it offers – affects the performer's usage patterns and potential appropriation.
2.1. Instrument Design
We created a self-contained digital musical instrument, here referred to as the Cube Instrument, which consisted of a 15 cm wooden cube containing a speaker, a touch sensor, a battery and a BeagleBone Black1 (BBB) (Figure 1). The touch sensor consists of a 2x5cm capacitive touch pad sensing the position of a finger on the surface, and is placed atop a force-sensing resistor (FSR) measuring the pressure on the sensor. The BBB is a 1GHz ARM single-board computer which reads data from the touch sensor and handles the audio synthesis. Software running on the BBB generates two overlapping sounds in response to a touch event: a percussive filtered noise burst at the onset of a touch, and a pitched drone which sustains for the length of the touch.
Figure 1. The right side of the figure shows a picture of the Cube Instrument, a self-contained musical device equipped with a single sensor, a touch strip mounted on its top. On the left side of the figure a view from the bottom of the arrangement of its inner components (inside of box not accessible to the performer).
Where Gurevich et al. (2010) study a single instrument design, we are interested in the relationship between instrument complexity and performer usage. Therefore, we created two different versions of the touch-to-sound mapping. In the first version, the pressure reading controls the amplitude of the percussive sound and the cut-off frequency of a lowpass filter applied to the drone, which allows continuous modulation of the sound within a limited range. We call this the 1 degree of freedom (1DoF) mapping, as it responds to a single sensor dimension (pressure).
A second version of the mapping also adds limited control of pitch using touch location along the long axis of the sensor. The pitch control affects the center frequency of the filtered noise burst and the frequency of the drone, the latter within a range of approximately 3 semitones. We call this the 2 degree of freedom (2DoF) mapping since it responds to both pressure and touch location.
2.2. User Study
10 Cube Instruments were built which were identical in hardware. 5 of the instruments were configured with the 1DoF mapping and 5 with the 2DoF mapping. Each instrument was given to one of 10 performers recruited through an open call, which included the only requisite of being interested in experimenting with a new musical instrument. Participants were given 2 weeks to prepare an unaccompanied performance between 1 and 3 minutes long, to be played for the investigators. Participants were then asked to return after a further period of approximately 2 weeks to deliver a second performance for a public audience. Each of the two groups (i.e., 1DoF and 2DoF) were composed of 2 classical instrumentalists, 2 experimental musicians and 1 electronic composer, all unaware that two versions of the instrument existed; in the performance with audience, participants were not allowed to hear other performances than their own, and they were requested not to discuss the instrument with other participants.
At each performance, the usage of the instrument was annotated in terms of affordances, musical variations, postures and interaction techniques. Participants also completed Likert-scale questionnaires about their experience with the instrument. Structured interviews were conducted at the conclusion of the first session, addressing topics similar to those found in Gurevich et al. (2010).
2.3. Results
A full discussion of results, including the self-reported experiences of the performers, can be found in Zappi and McPherson (2014b). Here we highlight the most important results in relation to appropriation of the instrument, centered around the ways in which they used the instrument.
Table 1 shows the behaviors annotated from video recordings of the 10 performances, which are divided into four categories: affordances, listing the actions employed by the performers to produce sound; musical variations, listing the musical outcomes of how the affordances were employed; interaction techniques, a higher-level view of how the performer engaged with the affordances of the instrument; and postures, indicating bodily positions and ways of holding the instrument. These categories support many-to-many relationships. For example, the same affordance could produce several musical variations, and conversely several affordances might be used to produce the same musical variation.
Table 1. Style, technique and affordances on the Cube Instrument.
2.3.1. Use of Hidden Affordances
Gaver (1991) defines a hidden affordance as one for which no information is available, and which must be inferred from other evidence. In our study, in line with other usage of the term in the digital musical instrument community, we consider hidden affordances to be those which either are not intentionally built into the instrument, or are not obvious from its interface.
A striking result is the number of hidden or non-obvious affordances that the players use in performance. Of 14 observed affordances, 11 of these go beyond the basic, obvious operation of the instrument. 6 of the 11 involve making musical use of other parts of the physical box, including the wooden sides and the speaker cone. Others involve exploiting apparent defects in the instrument. One performer discovered that the capacitive touch sensor was sensitive to moisture. By licking the sensor during his performance, he produced a sustained drone which freed both hands to be used for other percussive techniques.
The basic affordances of the instrument were nonetheless capable of a wide range of musical variations, particularly in the varieties and combinations of rhythmic and drone material. 8 of 17 musical variations derived specifically from the hidden affordances, several of which concerned variations of timbre created by mechanical means: filtering by hand or by table, and rattling objects on the speaker.
2.3.2. Comparing Degrees of Freedom
A comparison of actions, survey and interview responses between the 1DoF group and the 2DoF group produced a noteworthy set of results. First, the players in the 1DoF group collectively explored substantially more hidden affordances than the players in the 2DoF group. 5 of the 11 hidden affordances were used only by the 1DoF group, while the 2DoF group used only a single hidden affordance (touching speaker) which wasn't also used by the 1DoF group. Hidden affordances also contributed to 3 musical variations which were found only in the 1DoF group, while only one variation related to the hidden affordances was found exclusively in the 2DoF group. As expected, the additional dimension of pitch in the 2DoF produced two musical techniques, glide and vibrato, not found in the 1DoF group.
This finding suggests that the addition of an extra degree of freedom reduced the propensity for performers to seek out hidden affordances from the instrument. With a sample of only 10 players, drawing robust conclusions is difficult. However, this finding is reinforced by the results of the questionnaires and interviews, which indicated that the 2DoF group found the instrument to be more constrained than the 1DoF did. In particular, players in the 2DoF became preoccupied with the limited pitch range. One 2DoF performer commented: “Even though I had a couple of notes, I think I still found it a quietly frustrating experience just to have those notes instead of a couple of octaves”; another commented: “as soon as I realized it had a pitch…I concentrated on that.” When asked for suggestions to improve the instrument, 4 of 5 2DoF performers asked for greater pitch range, while only one 1DoF performer mentioned having more than a single pitch.
2.3.3. Analysis
The Cube Instrument study and its predecessor (Gurevich et al., 2010) suggest that highly constrained instruments encourage the discovery and exploration of hidden affordances, or unconventional ways of playing the instrument, at least amongst the experimentally-inclined musicians who took part in these studies. Our comparison between 1DoF and 2DoF groups shows that adding an extra dimension of control may paradoxically reduce the perceived capability of the instrument, leading to both fewer distinctive playing techniques and a greater frustration with the instrument's limitations. Whether this result holds for any pair of control dimensions or is a specific response to controlling pitch is discussed further in Section 4.3, but would also be an interesting basis for a specific further study.
In our next study, we instead turned to a different facet of appropriation: the tendency for users to modify their tools to suit their needs. Building on the results of the Cube Instrument study, we sought to create an instrument which would initially appear to the performer to be highly constrained, but with the important difference that the user could rewire it to discover and create sonic capabilities which were not part of the original design.
3. The D-Box Study
3.1. Instrument Design
The D-Box (Figure 2) is an instrument deliberately designed for modification and hacking by the performer. In its outward appearance, it is similar to the earlier Cube Instrument: a 15cm wooden cube containing a loudspeaker, a battery and a BeagleBone Black for audio processing. Some changes are subtle: in place of the Cube Instrument's one small touch sensor, the D-Box has two 10cm capacitive touch strips, each of which senses touch location along a single axis. One of the sensors (the one furthest from the speaker) also senses pressure via two parallel FSRs underneath.
Figure 2. The D-Box, a digital musical instrument designed to be modified by the performer. Opening the side panel reveals a breadboard whose circuits can be altered in an exploratory manner to change the behavior of the instrument.
Other changes are infrastructural: where the Cube Instrument was based on the ALSA2 audio framework, the D-Box was the genesis of a new hardware-software architecture (McPherson and Zappi, 2015a) which was later released as the open source Bela3 platform. This architecture features stereo audio I/O and 8 channels each of 16-bit analog I/O, running with round-trip latencies as low as 1ms. The D-Box software stores several short sounds (by default 7 sounds selected by the designers, but user-replaceable), rendering them not through conventional sampling but through additive synthesis with dozens or even hundreds of oscillators varying over time in frequency and amplitude.
The most consequential feature of the D-Box is the matrix: an electronic breadboard which is revealed by removing one of the wooden side panels. The matrix contains a network of analog circuits which govern important aspects of the instrument's behavior, including aspects related to the pitch, amplitude, timing, and timbre of the sounds it creates. The matrix is not modelled after a conventional modular synth paradigm: rather, the circuits form idiosyncratic feedback loops between hardware and software such that when they are modified, the results are often surprising or even chaotic, even though there is no explicit element of randomness in the instrument. Further detail on the operation of the matrix can be found in Zappi and McPherson (2014a) and McPherson and Zappi (2015b).
The purpose of the D-Box design is to present an initially simple and constrained interface to the player. With the box closed, one touch sensor controls the pitch of a bassoon-like sound, with pressure controlling its volume. This control resembles the 2DoF Cube Instrument, but with a longer sensor and greater pitch range. A second touch sensor applies a bandpass filter to the sound whose frequency depends on touch position. However, the matrix allows the D-Box to diverge significantly from the Cube Instrument: by rewiring the matrix, the sounds and behaviors of the instrument can be transformed, with possibilities ranging from drones to percussion, glitchy and noisy sounds, and even semi-automated behaviors. The matrix is designed so that any two points can be connected without damaging or silencing the instrument. As a further design principle, we did not seek to circumscribe or even necessarily understand the space of possible outcomes of rewiring the matrix; rather, the sounds emerge as a byproduct of the disruption and cross-wiring of the feedback loops.
3.2. User Study
To understand the role that hacking may have in the process of musical appropriation, we conducted a longitudinal user study during which musicians were asked to compose and perform with the D-Box. 14 identical copies of the instrument were built and handed out to 14 volunteers (1 female), recruited through an open call. Similarly to the Cube Instrument study, the participants varied in skills and background, ranging from experienced circuit benders to classical instrumentalists with no knowledge of electronics. Due to personal reasons, four participants dropped out of the study before its conclusion, leaving a final roster of ten musicians, all males yet still constituting a broad musical and technical variety.
Each participant was met separately and received a D-Box in its original configuration; the device was introduced as a “hackable musical instrument” and it was accompanied by a bag of spare electronic components, including: resistors, capacitors, push-buttons, trimmers, jumper wires and light sensors. Details about the other enrolled musicians were not made known, with the aim of preventing mutual influences that might have affected the usage and the perception of the instrument.
The study was divided in two consecutive sections. In the first one, the participants were given from two to five weeks to prepare a 1-3 minute unaccompanied performance and showcase it in a small black-box theater in front of the authors only. Complete freedom was given in terms of how to play and use the instrument, highlighting that its modification (hacking) was allowed but not strictly required. Participants were also warned that, although the D-Box was specifically designed to allow for the arbitrary alteration of its structure and components, the hacking of the onboard BBB was not recommended (yet not forbidden), as it may cause electronic failures and the irreversible breakage of the whole instrument. This information was reported in a small manual that accompanied the instrument's delivery; the manual also described the procedure to follow when loading custom audio samples on the onboard memory card and a picture/schematics of the original layout of the matrix. At the end of each performance, an interview was scheduled, covering the following topics in the form of open-ended questions: musical content of the performance (e.g., structure, range of material), physical interaction with the instrument (e.g., posture, playing techniques), impressions of the instrument (e.g., expectations, features and limits) and the rehearsal experience (e.g., duration, skill assessment). Moreover, any spontaneous comments diverting from this list were encouraged.
In the second section of the study, the participants had to prepare a further performance, this time staged in a venue hosting a live audience of more than 80 people. The same guidelines defined for the first part of the study were applied, except for the fact that the participants were allowed to include other musical instruments in the piece. The rehearsal period ranged from five days to four weeks, according to the date they first performed. Unfortunately, two of the ten participants (P5 and P13) were abroad on the day of the concert, so they were asked to send a video recording of their new piece to be screened during the event. At the end of the concert, the participants were given an open-ended written questionnaire covering similar topics to the interview after the first performance. The study was concluded with a group discussion during which the seven participants who could stay after the performance exchanged reflections on their experiences under the moderation of the authors. All the performances, the interviews and the group discussions were audio and video recorded. The video at the following URL contains some of the highlights of the pieces presented during the study: https://youtu.be/JOAO-EUtrGQ.
Höök et al. (2003) note the challenge of evaluating interactive artwork in an HCI context, in that “arists are interested in the richness and complexity of unique, individual users, cultural contexts, and resulting variety of interpretations and experiences of their system” and “statistical averaging and laboratory simplifications necessary for reliable scientific statements may wash out all the details that interest them.” Accordingly, our analytical approach to the D-Box seeks to capture the diversity of outcomes and creative decisions that the same instrument inspired in the 10 musicians, presenting trends where they are suggested, highlighting specific case studies where notable, but without claiming that every group of musicians would produce the same set of results.
3.3. Results
A wide variety of D-Box hacks4 were observed throughout the study, accompanied by different targets and motivations. These data are summarized in Table 2. Out of the 10 participants 8 decided to modify their instrument, while two maintained its original configuration. As expected, most of the hacks capitalized on the rewiring of the matrix (electronic hacks) and included additional electronic components, often not coming from the set delivered with the instrument. The most common observed approach consisted of adding extra controls/sensors to some of the built-in circuits, in particular potentiometers and trimmers used to modify in real-time some of the parameters used in the synthesis software; except for P3 and P8, all the performers who experimented with electronic hacking went down this road, using the added features as both expressive musical controls to use while playing live (P2, P7, P10) and tools for the fine tuning of the synthesis parameters before the performance (P5, P6, P13). The range of added features also includes discrete controls: P5 and P7 employed push-buttons and switches to activate/deactivate specific portions of the matrix and enable diverse musical behaviors, while P10 achieved a similar result via closing and opening connections within the circuits by means of crocodile clips. To extend the control capabilities afforded by the matrix, one of the performers (P13) even made use of an Arduino5 Uno. The board was programmed as a voltage selector driven by the peaks detected on the signal of one of the two piezo microphones, and was used to switch between two samples every time the sensor plate was hit.
Table 2. A summary of the diverse hacks and technical as well as artistic choices made by each participant throughout the study.
Two other electronic hacking approaches were found to be recurrent among the participants: the rewiring of the touch sensors and the interconnection of unrelated circuits of the matrix. P7 rerouted the outputs of the two touch sensors and used them for the continuous control of different circuits (other than the default pitch selection), while P5 experimented with the cross-wiring of the matrix's columns to unlock bizarre yet controllable behaviors. P8 and P10 combined both the techniques to introduce new feedback loops within the circuitry and control their activation by tapping and sliding their fingers on the touch sensors. Furthermore, during the interviews and the group discussion, several performers mentioned that in the first phase of exploration of the instrument they also consistently experimented with hacks based on floating wires and open connections. These can be touched and moved around to inject electrical interference into the matrix and generate glitches. Despite the initial interest, this approach eventually had a very limited uptake and when discussed triggered mixed feelings among the group. In the first round of performances only P3 and P10 showcased designs supporting such hacks, and while P10's final piece still relied on the handling of floating wires, P3 preferred to go back to a fixed-circuit configuration for the public concert. The effects of floating wires were even detrimental to the composition of P8. In the group discussion he stated that “[In the piece] I am taking one of the wires on the pitch circuit and if I keep that floating out for too long, it's just gonna flutter randomly and it sounds really awkward.”
Other interesting hacks regard the physical alteration of the instrument, as six D-Boxes also had their external enclosure heavily modified. Some of the circuits that underwent alterations were pulled out of the matrix and placed on external supports, to have more space and more freedom of action (P8, P10, and P13); in two cases the buttons and the knobs included in the hacks were mounted on the instrument's body, by means of an additional breadboard glued to one of its sides (P5) or by sticking the components directly through holes drilled in the wood (P7). P3 and P5 changed the position of the built-in sensors (the touch strips and the piezo microphones) to facilitate the execution of the playing techniques their performances relied on. To achieve the same aim, P2 completely disassembled the box; he removed the speaker and rearranged the sensors and the inner components in a custom slim enclosure, that allowed him to secure the instrument to a music stand and play it while standing. Additionally, he designed an extra structure that surrounded the new enclosure and only served as a decoration, in line with the aesthetics of his piece and of his stage outfit.
3.3.1. Different Instruments, Different Music
Throughout the study the participants produced a wide variety of modifcations of the D-Box, all deriving from its original electronic and physical configuration. This raises the question whether the newly created designs should be simply considered variations of the D-Box or may be classified as completely new musical instruments.
As previously discussed, the diverse circuit configurations conceived by the performers made the inner workings of their D-Boxes quite different from one another, and the combination of matrix hacks and physical modifications produced recognizable visual features that set them easily apart (Figure 3). However, the most distinctive features of the designed instruments were their sound signatures and the music it was possible to play with them. Except for P3, whose first piece was entirely based on the processing of the seven default D-Box audio files, all the participants that modified their instruments used hacking always in combination with the customization of the set of playback samples loaded into the on-board memory card. This made it easy to distinguish the sound of each of these instruments, even when used in circuit configurations that left the output samples relatively unmodified. The files used to replace the original sounds included: drones, rhythmic patterns/sequences, pitched sounds, ambient recordings, as well as atonal/arhythmic/noise-like samples.
Figure 3. Other than informing diverse playing techniques, the electronic and physical hacks carried out by the participants gave to their D-Boxes a quite distinctive look. From left to right, in the top row P7's instrument (matrix), P3's, P6's (matrix), P15's; in the bottom row P2's instrument, P10's, P7's (enclosure), P8's (matrix).
Moreover, the range of material played by the performers was not strictly defined by the chosen sounds. While some hacks were conceived to facilitate interaction techniques and musical variations (e.g., P5's electronic and physical layout, designed to play a specific scale with two hands), or to switch between samples and play modes (e.g., P13's tap-based sound selector), in other cases the instruments were modified to support the extensive manipulation/processing of the loaded samples. This allowed for the exploration of sounds that went beyond the original sonic and musical features of the loaded files, remarkably enlarging the variety of the material played during the performances. Some interesting examples of the different attitudes we observed toward hacking in relation to sound file control come from P2 and P10, as well as from P6 and P7. Both P2 and P10 loaded drone samples on their instruments, but while P2's performances were clearly characterized by static or slowly evolving sustained sounds (i.e., drones), P10 made use of classic circuit bending techniques to combine every sample into harsh noise, making the original files unrecognizable. P6 and P7 instead achieved very similar arpeggiator-like sounds, though they used two opposite approaches. P7 built an arpeggiator; he rerouted the multi-touch output of the second sensor strip to the pitch circuit, to create looped pitch-steps on sustained samples. P6 instead preferred to directly load a pre-recorded sequence and simply trigger it. A further approach is the one from P0, who did not hack his D-Box and even discarded the idea of loading custom samples, to focus on the first default sound (which “actually sounds nice!”). He explained his choice by saying that “to make a nice performance and to make nice music you don't actually need to hack the instrument. It just has to have one sound that you like and then you have to put some effort on the piece you're gonna perform.”
3.3.2. Hacking: Why and When
Although every participant had a specific personal reason to turn his D-Box into a unique instrument, the data gathered from the questionnaires and throughout the group discussion highlight two main trends. Three performers were particularly interested in the possibility to modify the D-Box to unlock new unconventional musical controls that are hard to find in traditional musical devices: P3 worked on audio feedback-loops and designed a layout for controlling their intensity, while P8's and P13's instruments allowed them to change mappings by tapping and hitting the side and top plates of the enclosure. During the interviews, these participants clearly referred to hacking as a means to explore new musical ideas, including sounds and interaction techniques.
On the other hand, four participants modified their instruments to incorporate musical features they were acquainted with already, with the aim of making the D-Box a more familiar instrument to play. This is the case of P5, who is a cellist and crafted a D-Box that plays much like a stringed instrument (with sounds reminiscent of sustained/bowed notes as well as of pizzicatos), and of P7, who likes to compose with sequences and, as previously introduced, managed to create a circuit configuration that behaves similarly to an arpeggiator; then we have P6 and P2, whose habit to play with samples in combination with effect rigs pushed them to turn their D-Boxes into samplers with built-in sound processing features.
A further important note about hacking concerns its role in the creative process. P10 bent the circuits of his instrument during his performances, using hacking as a form of live musical interaction. Such an approach was only shared by two other participants, P6 and P8, who experimented with much simpler live-hacks.6 P3 also experimented with live-hacking for his first piece, but then he preferred to work on a fixed setup in the final concert. The other 5 participants who modified their instrument used hacking strictly as a design tool, to prepare the instrument during composition and rehearsals. However, the emergence of some examples of live rewiring of the matrix is an important result. From a technical point of view, it confirmed the robustness of the design of the D-Box, which was originally conceived as an explorative platform resilient to crashing or breaking, even when undergoing severe electronic or physical mistreatments. Moreover, the participants who hacked the instrument in front of the audience exhibited a strong trust in the instrument, probably promoted by the fact that, with only one known exception (a specific wiring choice by P10), none of the matrix hacks throughout the study led the instrument to switch off or fall silent.
Regardless of the underlying motivations and the specific approach followed, the possibility to modify the instrument results in the strongest source of artistic inspiration among the participants of the study. The combination of hacking (electronic and physical) with sound file customization produced versions of the D-Box extremely different from its original design as well as from one another, in terms of sound, look and afforded interactions. A detailed analysis of some of the most interesting instruments created during the study as well as of the related playing techniques can be found in McPherson and Zappi (2015b) and Zappi and McPherson (2015).
Not all the participants channeled their creativity through hacking. P0 kept the matrix of his D-Box in its original configuration and experimented neither with physical hacks nor with custom audio files. In a similar fashion, P15 did not even open the case of his instrument to reach the memory card or the matrix, not even to switch the sound loaded by default via the built-in control. As a result, their clean and sealed D-Boxes look and sound identical, but they are extremely easy to recognize among the bizarre creations shown in Figure 3. Despite the apparent similarities between the approaches of these two participants, the motivations behind their choices and the related musical outcome of their performances are divergent. When interviewed, P0 stated that before finding inspiration in the basic sound of the D-Box (see Section 3.3.1) he was initially attracted by the idea of modifying how the instrument works, but he did not manage to get any satisfying result and soon gave up. He then explained “it's not like the instrument didn't allow me! Basically I don't have a lot hardware knowledge.” By contrast, P15 has a background in electronics and musical instrument design, but since the beginning disregarded the inner workings of his D-Box and gave precedence to the understanding of its basic metaphor (“I wanted to respect its original configuration, I wanted more to get into the interaction with the current sensors”). During the group discussion that concluded the study he blamed time, as he got his D-Box only two weeks before the final concert7, then adding “I would [hack it] now!.”
Although P0's avoidance of hacking might relate to his lack of electronics experience, overall we did not see any clear dependency between the extent of modifications made by each performer and his background as a hacker, or his knowledge of electronics (summarized in the second column of Table 2). P5, P6 and P8 had almost no experience with electronic components and had never previously engaged with circuit bending, but they extensively hacked the matrices on their D-Boxes. On the other hand, people with a strong background in engineering or electronics like P2 and P15 either did not explore much of the matrix or were not able to turn their hacks into musical features, and discarded them. Finally, we have more predictable cases, like P0 previously described, and P10, who leveraged his expertise as circuit bender to disrupt his instrument.
3.3.3. Style Diversity and Performance Techniques
The design of the D-Box fostered very personal interpretations of the instrument, allowing the 10 participants to musically engage with a wide variety of affordances. The diverse hacks, intentions and musical performances resulting from the study can help understand more in depth the effect that hackable design had on this process, by focusing in particular on the development of style among the group.
In line with the analysis carried out during the Cube Instrument study (Section 2.3), we can frame style diversity in terms of affordances, interaction techniques, postures and musical variations explored by the participants. The first column of Table 3 lists all the affordances the performers capitalized on to compose and play their pieces. As the D-Box shares a consistent number of design features with the Cube Instrument, several entries overlap with the list displayed in Table 1, in particular most of the affordances concerning the usage of the touch sensors. Overall, the number of actions afforded by the D-Box is quite high and comparable with the case of the Cube Instrument, and includes both main and hidden affordances (the starred bold entries). However, some of the entries did not belong to the original design of the D-Box and stemmed from hacking, in terms of both rewiring of the matrix and modification of the physical structure of the instrument. These entries are shown with a gray background and cover a third of the total number of affordances displayed during the study.
Table 3. Diversity of style and technique observed during the D-Box study.
The possibility to hack the D-Box seems to have had an even stronger impact on the assortment of interaction techniques and musical variations showcased by the participants. The number of entries in the second column of Table 3 (interaction techniques) doubled up with respect to the case of the Cube Instrument, and all the new techniques reported can be attributed to electronic or physical hacking. These include direct interaction with the matrix (e.g., moving components and wires with the hands), but also the usage of external tools, like screwdrivers or crocodile clips. Furthermore, as introduced in Section 3.3.1, matrix hacking was largely explored with the aim to produce specific synthesis effects (e.g., polyphony, envelope, timbre), as well as to allow for the executions of rhythms, tunes and for the control of dynamics. The combination of sounds and different ways of controlling them makes the number of musical variations employed by the 10 participants too large to be exhaustively listed as a column in Table 3. Across the two performances, the participants showcased all the musical variations reported during the Cube Instrument study, except for the placement of rattling objects inside of the cone of the instrument's built-in speaker8. In addition, they employed original variations like: the rhythmic and arhythmic damping of the sound using the filter, the modulation of the length of the playback portion of the file, the coloring or the detuning of the samples, the shaping of the resonances triggered by the feedback between the microphones and the speaker, just to name a few.
The third column of Table 3 shows a certain variety in posture too. Participants stood or sat while playing, some pointed the speaker toward the audience, while others preferred less obvious and more idiosyncratic orientations that better complied with the adopted playing techniques. This scenario may be an issue if playing a piece conceived to use the speaker as the only acoustic radiator (as opposed to connecting the line out to an audio system), as the sound can get quite attenuated. To overcome this limitation, most performers simply asked to place a microphone in front of the speaker. Others took another direction though, and specifically modified the morphology of the electronic hacks and the physical structure of the D-Box to be able to keep the standard orientation (speaker forward) even while executing their personal playing techniques. Differently from the case of the Cube Instrument, once chosen, the orientation was never changed during the performance. A further difference pertains to how the instrument was held, as all the participants decided to rest their D-Box on a surface and no one placed it on the laps or between the legs.
Overall Table 3 helps quantify the diversity of style that characterized the performances. The resulting stylistic range is extremely high, even higher than what observed during the Cube Instrument concerts. An interesting note concerns the usage of the affordances, as a consistent subset of afforded actions were explored by more than one participant. These include the usage of the most obvious instrument's controls, i.e., the touch sensors (featured in all performances except for P3's second piece), and more prominently all the affordances made available by hacking (each shared between at least 3 participants). When compared to the specificity of musical variations, which tend to characterize the style of individual participants, this scenario suggests the role that hacking assumed in the process of appropriation of the instrument. The hackable design of the D-Box allowed the majority of the performers to interpret the instrument's affordances in extremely different ways, and strongly channel personal style and musical aesthetics through them (same affordances, different musical outcome). The result is a fruitful exploration of the instrument, composed of several paths through its affordances that led to concrete musical milestones.
3.4. Analysis
Both HCI and DMI design research highlight how strongly the behavior of a user is influenced by the features of the device in use (Dix, 2007; Magnusson, 2010). In this section we analyse the design of the D-Box through the actions and feedback of the participants. Our aim is to understand what elements of its construction directed interaction the most, how they were interpreted and if the result is in line with the original purpose of the employed design rationale.
3.4.1. Perception of the Instrument: Limited or Unconstrained?
The results from the Cube Instrument study as well as previous literature (Magnusson, 2010; Gurevich et al., 2012) suggest that the development of style is fostered, to a great extent, by the discovery of limitations in the instrument's metaphor. In the case of the Cube Instrument, the simplicity of the chosen design raised a homogeneous awareness of the constrained nature of its metaphor. On the other hand, the design of the D-Box, once the matrix is exposed, cannot be labeled as simple and the perception of its features and limits varied widely amongst performers.
P3 and P13 stated they found nearly no constraints in the D-Box, as they both managed to hack the instrument to reach their musical targets. When interviewed, P3 said “there's little limitations…but I can possibly re-configure them down here [on the matrix]”; P13 was even more direct, saying that “you can modify whatever and…do pretty much anything you think you can do,” but this makes the instrument “difficult to pick up and play straight away.” In this regard, hacking was seen as a complex degree of freedom, powerful enough to overshadow most if not all the constraints of the instrument at the expense of its intuitiveness. On the opposite end of the spectrum we can find P0 and P15, who did not modify the original configuration of the D-Box and perceived it as a very constrained instrument. In line with the experience of the performers of the Cube Instrument study, both these participants enjoyed exploring and challenging the constraints they encountered. P0 found inspiration in the simple sound and the basic interaction technique afforded by the instrument to prepare a performance that differed from his usual repertoire, mixing music and poetry (“this was something I always wanted to do, I really wanted to tell a story and have sound together with it and never did it”). Similarly, P15 stated that he decided to “play with the limitations of this little box,” then adding that he “kept discovering things…and way to do things easier.”
The rest of the participants had an intermediate approach, and shared their attention between the limitations and the degrees of freedom found in the instrument's design. P7 clearly saw numerous constraints in the D-Box, yet he also sensed the freedom of action granted by hacking: “The initial setup kind of left you to want a lot more, but it kind of was set up in a way you think…wow! This can really do a lot.” His experience hinged on the breakage of constraints, to reach for the potential affordances concealed underneath the simplicity of the default configuration; P7 also underlined the level of challenge linked to this process (“I kept thinking arpeggiator or step sequencer […], it took me a while to figure out how to get that happening”). Likewise, P6 recognized several limitations in the D-Box, going from its non-polyphonic nature to the lack of frets or other physical references to help play in tune, and employed hacking as a way to cope with them. However, his hack did not target any of these limitations; rather, P6 modified his D-Box to introduce alternative affordances and interaction techniques that would still allow him to play his music in a comfortable way. In particular, he forwent control over melody and harmony to focus on the creation and on the concatenation of rhythms, which he found to better resonate with the overall design of the D-Box. P5 showcased an analogous behavior, as he felt constrained by the lack of polyphony but counterbalanced it with the introduction of extended controls over timbre. A further approach is the one from P2, who started from the design of its limitations to obtain a familiar device to explore and further modify (“it's a simple configuration […], it allows me to change things, experiment and add”).
Overall, participants' comments and actions suggest that the D-Box was perceived as a constrained device, reinforcing the theory that musicians tend to focus on the limits of a design (Magnusson, 2010), which in turn fosters style diversity and musical appropriation (Gurevich et al., 2012). Almost all the participants in the D-Box study worked on the limitations of the instrument, hacked constraints or because of constraints, and in some cases deliberately added them to their personal designs. In line with the results from the Cube Instrument study, the limitations of the diverse configurations pushed the development of skills, through hacking or via the generic exploration of the main and hidden affordances of the D-Box.
3.4.2. Hacking as a Degree of Freedom
A great part of the complexity of the D-Box is ascribable to its hackable nature and to the diverse interpretations it triggered among the participants. As introduced in the previous section, the possibility to rewire the matrix was perceived as the most powerful degree of freedom of the instrument, or as a feature capable of unlocking additional ones. Although the potential recognized in this practice led some participants to see the D-Box as an almost unconstrained device, there is evidence that some limitations in the hackable design of the instrument were encountered during the study.
The analysis of the interviews and of the group discussion highlights mentions of playing techniques that appealed the participants, but whose realization was not possible via the rewiring of the matrix. Some examples are P6, who noticed it is not possible to automate the filter, nor to disable it, and P5 who could not figure out how to play two different samples at the same time. Furthermore, P10 and even P13 (who stated in the interviews that hacking would allow him to do whatever he might imagine) looked for a way to trigger the sample playback on the matrix without having to touch the sensor on the plate, a solution not achievable except via the modification of the source code. As designers of the instrument we are aware of these and other limitations on what can be done through hacking. However, the study suggests that the number of possibilities overshadows the limitations that constraint the modification of the instrument, and even us designers were surprised by the structure and the outcome of certain hacks (e.g., P3, P5, P8). Hence, while the D-Box is clearly not an unconstrained instrument, it is safe to assume that its hackable nature is the most conspicuous and appealing of its degrees of freedom.
The freedom of action and expression entailed by the hacking of the matrix was also linked to the perception of a certain level of challenge. Overall, mental and manual skills are required to translate a musical idea into a new D-Box configuration capable of supporting it, and several participants described the hurdles they faced in the modification process. Some performers managed to deal with the complexity of this degree of freedom. As reported in the previous section, P3 and P13 crafted most of the hacks they had in mind, while P7 had to put time and effort to create and control an arpeggiator, yet he expressed much satisfaction while talking about his final result. Another successful approach is the one showcased by P5 and P10, who leveraged serendipity to shape some of their hacks. Conversely, other participants experienced much more frustration during the preparation of their instruments. P0 and P15 (the two participants who did not hack the instrument) generically pointed out the very high level of challenge entailed by hacking, which was overwhelming considering the skills or the time they had at their disposal. A similar yet less extreme example is P6, who had to drop some hacks too complicated to concretize but succeeded with several simpler ones. Furthermore, as argued in the group discussion, the complexity of hacking as a degree of freedom may resolve into a challenge in an indirect way too; hacking may have unwanted consequences, and some of the instrument's behaviors that derive from the ability to completely disassemble the matrix may indeed hamper interaction. This is for example the case of P8, who made use of the flexibility and stability of the matrix to expressively play with live hacking, but struggled with the effects of floating wires temporarily removed from the circuits (see Section 3.3).
In the literature challenge is described as a generally positive design feature (McDermott et al., 2013), however in the case of the D-Box its effects on appropriation consistently varied among the participants. The ability or inability of each musician to turn a musical idea into a hack (i.e., the perceived level of challenge) affected not only the final electronic and physical design of the instrument, but also the exploration of its main and hidden affordances. Some extreme cases are P0, whose inability to properly hack the instrument pushed him to focus only on its main affordances (the control of the default sound), and P3, who got so absorbed into hacking as to completely discard any of the basic features of the D-Box, including the usage of the touch sensors. In between we can find a whole range of approaches: P15, who could not hack but enjoyed the exploration of as many hidden affordances as he could find; P2, who modified his D-Box, engaged with its main affordances but did not showcase any unexpected ways of making music with it; then P5, P7 and P8, who managed to combine the usage of main and hidden affordances of the idiosyncratic configurations they designed. Interestingly, these different approaches do not seem to be directly connected to the specific background of each participant. Similarly to the absence of a clear connection between individual experience with electronics and the amount of modifications made to the matrix discussed in Section 3.3.2, in these examples we can find skilled hackers with completely different attitudes toward the exploration of affordances (P15 vs. P3 vs. P2), and coders displaying diverse abilities to hack and interpret the instrument (P0 vs. P5 and P8). The only consistent connection between behavior and background detected in the study concerns the musical outcome of the compositions. As shown in Table 2, all the participants except for P0 presented musical material in line with their artistic background (e.g., instrumentalists enjoyed rhythms and melodies, experimental artists preferred noise and abstract sounds).
4. Discussion: In Search of Appropriation
For both the highly constrained Cube Instrument and the modifiable D-Box, a central question is whether the actions of the performers constituted appropriation of the instrument. If so, then we should reflect on how this appropriation took place and whether the Cube Instrument and the D-Box give rise to different patterns of appropriation.
One potential proxy for appropriation is the use of hidden affordances, or non-obvious modes of interaction (Zappi and McPherson, 2014b) which are discovered or invented by the player. In the Cube Instrument study, 10 players collectively discovered 19 different hidden affordances. Players of the more-constrained 1DoF version used 17 of these 19 hidden affordances, while players of the 2DoF version only used 11 of the 19. The difference suggests that extremely tight constraints may encourage the exploration of non-obvious modes of interaction. An alternative explanation is that the ability to control pitch in the 2DoF version put those players into a different mindset focused on melodic playing, and that this musical mindset, rather than any property of the instrument itself, discouraged the exploration of hidden affordances.
Hidden affordance use, excluding modification of the instrument, was also observed in the D-Box performances (Table 3), though in lesser proportion to the overall number of techniques employed. It may be that the creative impulses that led to discovery of hidden affordances on the Cube Instrument were instead directed toward modification of the D-Box.
Regardless of the reason, though, hidden affordance use is an incomplete and imperfect proxy for appropriation. Hildebrand Marques Lopes et al. (2017, p. 341) critique the very notion of appropriation in musical instruments: “In common engineering terminology, things have uses they are designed for, and then people discover other possibilities for them, which some may consider misuse, appropriation and the like, but in fact they are simply unforeseen affordances as soon as someone notices them.” This comment raises interesting questions: is there a categorical difference in using an instrument in the way a designer foresaw, vs. using it in other ways? Specifically with regard to the D-Box, since we intended the instrument to be rewired, does rewiring it constitute a hacking process which is different than any other form of customization? These questions and other aspects of personalization and musical interaction are explored in the following sections.
4.1. Personalization and Modification
Personalization is an important part of the appropriation process, but what it means to personalize an instrument is unclear. In the Cube Instrument study, we saw some performers make pencil marks on the instrument surface to guide their playing. In other cases, without any visible modification at all, the performers became attached to the subtle idiosyncrasies of their own instrument, such that swapping it for an apparently identical unit was an uncomfortable experience (Zappi and McPherson, 2014b). These observations suggest that personalization may be a two-way process: performers can make alterations to an instrument, but likewise the process of playing the instrument yields an attachment to the particular form and behavior of that object as originally designed.
On the D-Box, more extensive modification of the instrument is possible, and modification becomes an explicit focus of the performer study. Although Dourish (2003) identifies customization as one possible avenue for appropriation, not all customization is appropriation. If it was, then every person who twiddles the controls on a synth and everyone who chooses from amongst pre-made templates in page layout software would be appropriating the interface. In looking for deeper evidence of appropriation, two characteristics stand out. The first is ownership (Dix, 2007), the feeling of being in control. The second is resourcefulness, which Wakkary and Maestri (2007) identify as a key creative element of appropriation, often seen in the repair of broken objects (Maestri and Wakkary, 2011).
We might ask, then, whether the D-Box modifications show evidence of ownership and resourcefulness. Ownership may be implied in the divergent identity of the instruments. As discussed in Section 3.3.1, the general opinion amongst performers seemed to be that the different modified D-Boxes were no longer the same instrument, in that they were no longer capable of playing the same music. This observation feeds into a persistent debate with the digital musical instrument community about the boundaries between instrument design and composition (Murray-Browne et al., 2011): if a piece can only be played on a single instrument, and conversely an instrument is only capable of playing that single piece, then are the instrument and the piece distinct artistic creations?
Several of the D-Box modifications suggest a sense of ownership in other ways. P2's physically rebuilt the instrument into a form (a large bee) that matched the theatricality of his performance practice (conducted in a beekeeper suit). P5, P7 and P13 all added adjustable controls to the box which carried meaning for their particular musical practice. In each of these cases, a generic D-Box would not be as well adapted to their practice, nor would their particular modifications necessarily suit the preferences of other musicians.
Identifying resourceful adaptation of the instrument is more challenging due to the parameters of the study. Performers were given an instrument and tasked with preparing a performance with it, so we would not expect to see the Cube Instrument and D-Box repurposed in their lives for different ends. However, the nature of the performer-instrument relationship during the performance merits further scrutiny, as discussed in the following section.
4.2. Modes of Interaction in Performance
Sengers and Gaver (2006) advocate for designers to create objects open to multiple interpretations. In the context of multiple or ambiguous meanings, evaluation should then reflect on the process of meaning-making by the users. In an earlier paper (McPherson et al., 2016), we conducted workshops to examine how D-Box users made sense of the matrix as they rewired the instrument, finding a process of exploratory play in which free experimentation was balanced by the caution of not having an easy undo feature in a hardware context. In this paper, our focus is instead on what takes place during performance.
The performer-instrument relationship could be characterized along two axes: embodied vs. symbolic (Magnusson, 2009) and communication-oriented vs. material-oriented (Mudd, forthcoming). Traditional acoustic instruments are characterized by an embodied relationship between performer and instrument, drawing on sensorimotor learning and pre-conceptual understanding, whereas software instruments (including live coding languages) afford an interaction on a more symbolic or conceptual level. Magnusson (2009), while noting that all instruments encode certain musical values, highlights the “black box” nature of software instruments: “The act of formalizing is therefore always an act of fossilization. As opposed to the acoustic instrument maker, the designer of the composed digital instrument frames affordances through symbolic design, thereby creating a snapshot of musical theory, freezing musical culture in time. The digital instrument is thus more likely to contain an expressive closure as contrasted with the explorative openings of the acoustic instrument.”
Viewed along the embodied-symbolic axis, the Cube Instrument appears to more closely follow the traditional embodied interaction paradigm, as evidenced by playing techniques (particularly those which incorporate the physicality of the box into the performance) and the kinds of music made with it. The D-Box, however, does not neatly fit either an embodied or symbolic mould. In particular, it does not appear to represent the same black-box or fossilized qualities as many software instruments. It is not merely that the instrument can be rewired; more importantly, the meaning of the musical results is not predetermined according to any particular musical theory, or even anticipated by the designers at all. It is instead left open to interpretation by the performer.
Evidence that the D-Box is open to interpretation by the user can be found in the ways it is used in performance. It is not simply that each performer has a different musical style, but that different performers have a completely different relationship to the D-Box. Mudd (forthcoming) contrasts communication-oriented musical interaction, where the instrument is idealized as a transparent medium through which to transmit a message, to the material-oriented approach, where instruments are viewed “more as instigators and collaborators in the formation of creative outputs. The bidirectional nature of the interaction is foregrounded: the material ‘kicks back'.”
According to this dichotomy, some D-Box performances might be communication-oriented, including the melodic-rhythmic approaches of P5, P7, P8 and P13. Performance P0 was narrative rather than musical in nature, but nonetheless sculpted a message from the sonic qualities of the D-Box. Other performers devoted their attention to the material and behavioral aspects of the D-Box, particularly P3 (Mudd himself), who explored feedback between speaker and pickups, and P10, who engaged in live-rewiring of the instrument. Particularly for performances involving live-rewiring, the musical relationship between performer and instrument is exploratory and bidirectional.
To summarize, then, one of the strongest indicators that the D-Box is open to multiple interpretations is that the fundamental relationship between performer and instrument changes from one performer to the next, and from one wiring configuration to the next. Unlike other digital instruments highlighted by Magnusson (2009), the D-Box does not freeze a particular set of musical values, but leaves it open for the performer to take ownership through adapting the instrument to their needs.
This is not to say that the D-Box is completely open-ended, but only that the performer's exploration and modification of the D-Box appears to represent a different process than turning knobs on a conventional synth or piece of music software.
4.3. The Role of Pitch
In Section 2.3.3 we noted that performers of the 2DoF Cube Instrument found the instrument to be more limited, and explored fewer techniques, than performers who received the 1DoF instrument. The difference between the two versions was the ability to vary pitch across a small range (approximately 4 semitones) on the 2DoF instrument. A lingering question was whether the difference in perceived limitations was a general feature of adding more degrees of freedom, or whether it was specifically the addition of pitch control which changed how performers viewed the instrument.
The use of pitch on the more complex D-Box offers a point of comparison. In its unmodified form, the pitch control on the D-Box is essentially what several performers on the 2DoF Cube Instrument requested: a longer touch sensor which covers a larger pitch range (approximately 2 octaves). A quote from P8 seems to support the hypothesis that pitch control tended to trigger a very specific response in the participants of both studies, related to common musical conventions: “It's a continuous pitch controller and you kind of expect to play something in tune”. This opinion found strong agreement during the group discussion, and none of the other control dimensions featured in the instrument raised similar concerns or expectations. However, P8 then adds: “…but…you're not supposed to play tonally, unless you really want to. But then if you play it really out of tune […] it sounds really awkward”.
P8's quote underlines how the hard constraint on the Cube Instrument of a limited pitch range is replaced by a softer, but no less palpable, constraint on pitch control through musical expectations of intonation. Notably, of the 10 D-Box performers, only one (P5) used melody as a principal musical element of their piece; this decision required P5 to develop high dexterity through practice, a long and difficult process as clearly reported in the interview. Where the touch sensor controlling pitch was used, it was more commonly operated in an approximate way to control distinctive percussive sounds (P8, P13) than to articulate intervals of a traditional Western scale. In other cases, prerecorded sounds and melodies were triggered with a single steady touch (P6, P7), sometimes applying subtle pitch bending by sliding the finger (P2). Of the two performers who did not modify the instrument (P0 and P15), neither employed melodic techniques on the pitch sensor: P0 slowly glided pitch to resemble the buzzing of an insect; where P15, alike P8 and P13, used discrete pitches to characterize different short-attack sounds.
Though the musical backgrounds of some participants, particularly the predilection toward experimental and noise music (cf. Table 2, second column), may contribute to the relative lack of melodic material, the result may equally stem from the perceived challenge of in-tune melodic playing or from the perception that this style of playing was less interesting than the alternatives. In any case, the availability of a wide pitch range did not appear to change the perception of whether or not the D-Box was a limited instrument: P0 and P15, who did not modify their instrument, found it to be limited, while participants who found the D-Box to be a complex instrument cited the capacity for hacking as the principal reason. Moreover, some of the analysed hacks included custom constraints on the behavior of the pitch circuit (P5, P7), which in some cases even completely disabled it (P3, P8). This brings back a highly limited control scenario.
In both studies, pitch appears to be a sensitive choice in terms of control dimensions, especially when coupled with constraints. In the Cube Instrument study, participants with the 2DoF instrument felt high levels of frustration for being unable to fulfil the specific expectation of playing melodic music, which was tied to the design of the pitch controller—continuous yet extremely limited. This misalignment between expectation (what the performer should/wants to do) and freedom of action (what the instrument allows them to do) likely informed the perception of an overall constrained device, whose limitations appear capable of overshadowing its musical features. On the other hand, the D-Box performers did share the same expectation, but their instrument afforded the ability to fully explore pitch control and even modify the rules governing its implementation; this allowed for a shift of focus from freedom of action to skill self-assessment (what the musician is capable of doing with the instrument). As a result, in the second study the melodic approach is generally dropped as being too difficult or not worth the effort, in favor of the development of alternative styles. Interestingly this fruitful process, which marked the path toward the appropriation of the D-Box, capitalized on self-imposed limitations, suggesting once again the dominant creative role of constraints over degrees of freedom.
4.4. Is It Really Hacking?
We often describe the D-Box as a “hackable” musical instrument. However, the term hacking carries a subversive connotation. To describe circuit bending of consumer electronics (Collins, 2008) as hacking is relatively uncontroversial. But since the D-Box is designed to be modified, should rewiring it be considered hacking?
Following the D-Box performances, we held a group discussion with several of the performers where we asked this question. Most performers seemed to feel that some, but not all, modifications could be considered hacks. P8 suggested that non-obvious or unconventional changes might be considered hacking (leaving open the question of what should be considered obvious). P2 offered that circuit bending should only describe alteration of finished designs. In his view, because the matrix is designed to modified, changing it would not be considered a hack, but altering the fixed or soldered circuits would be. P10 pointed to one of his modifications involving a feedback loop as “maybe a hack, because I don't think most people necessarily do that.” P3 summarized his approach as: “I'm going to break it, I'm going to learn it, and I'm going to break it better.”
These performer comments suggest a certain tension between hacking and open or unfinished designs, but they also align with the comments of Galloway et al. (2004): “Hackability implies more than simple customization or adaptation—it calls for redefinition. Design for hackability involves creating spaces for play where people are never forced to adapt to technology.” Fundamentally, each performer seems to argue for a personal, distinctive approach that has not been pre-scripted by the designer nor made so obvious that anyone else would discover it. These qualities are also those of the appropriation of technology.
5. Conclusion
The appropriation of technology (Dourish, 2003) is a process which involves considerable creativity and resourcefulness. While appropriation is often highly personal, the decisions of the designer can nonetheless encourage or discourage appropriation or push the user in particular directions (Redström, 2006; Dix, 2007). Designers of digital musical instruments should carefully consider appropriation in their designs, not only because the history of musical instruments is one of performers creatively adapting the tools around them, but because the most artistically successful uses of a new instrument may be those that the designer has not foreseen.
In this paper, we considered two approaches to designing musical instruments for appropriation. The first focused on extreme constraint as a driver of the exploration of hidden affordances. In this minimalist context, we found that paradoxically, tighter constraints appear to yield a richer set of interaction techniques. The other approach gave the performer the ability to freely rewire the instrument, placing very few constraints on the accepted or expected patterns of wiring. By leaving the meaning of the sounds and behaviors open to interpretation (Sengers and Gaver, 2006), the D-Box gave rise to a striking diversity of musical styles and interaction modalities amongst the 10 performers we studied. Ultimately, whether any particular performer's action should be labeled as “appropriation” or “hacking” is less important than creating a space of possibilities which allows for creative exploration and reaction to constraints without the designer seeking to script the entire space of musical outcomes.
Ethics Statement
The studies with musical performers were carried out in accordance with the recommendations of the Queen Mary University of London Research Ethics Committee. The research was approved by the committee chair under the fast-track review procedure for studies deemed to be extremely low-risk. All subjects gave written informed consent in accordance with the Declaration of Helsinki.
Author Contributions
VZ and AM jointly created the instruments used in this paper. VZ led and analyzed the user studies, and VZ and AM jointly wrote up the results.
Funding
This work is supported by the UK Engineering and Physical Sciences Research Council under grant EP/K032046/1 and EP/N005112/1.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Thank you to London Music Hackspace and to all the performers for making both studies possible.
Footnotes
2. ^Advanced Linux Sound Architecture
4. ^In this section we use the term “hack” to describe modifications to the instrument. Section 4.4 returns to the question of which modifications do and do not constitute hacks.
6. ^With live-hacking we refer to actions like removing/adding components or pulling wires as part of the performance, a process completely different from the usage of knobs and switches discussed at the beginning of Section 3.3.
7. ^P15 is the performer who had the D-Box for the shortest time among the whole group, as he took the place of a former participant who unexpectedly dropped out after presenting his first piece and not long before the final concert.
8. ^During the group discussion P7 mentioned this as an early musical idea he had, but which he never got to test.
References
Bijsterveld, K., and Peters, P. F. (2010). Composing claims on musical instrument development: A science and technology studies' contribution. Interdisc. Sci. Rev. 35, 106–121. doi: 10.1179/030801810X12723585301039
Buechley, L., Rosner, D. K., Paulos, E., and Williams, A. (2009). “DIY for CHI: methods, communities, and values of reuse and customization,” in CHI'09 Extended Abstracts on Human Factors in Computing Systems (Boston, MA: ACM), 4823–4826.
Cascone, K. (2000). The aesthetics of failure: “post-digital” tendencies in contemporary computer music. Comput. Music J. 24, 12–18. doi: 10.1162/014892600559489
Cherry, E., and Latulipe, C. (2014). Quantifying the creativity support of digital tools through the creativity support index. ACM Trans. Comput. Hum. Inter. 21:21. doi: 10.1145/2617588
Collins, N. (2008). A solder's tale: putting the “lead” back in “lead users”. IEEE Pervasive Comput. 7, 32–38. doi: 10.1109/MPRV.2008.42
Dix, A. (2007). “Designing for appropriation,” in Proceeding British HCI Group Conference on People and Computers (Lancaster).
Dourish, P. (2003). The appropriation of interactive technologies: some lessons from placeless documents. Comput. Support. Cooperat. Work 12, 465–490. doi: 10.1023/A:1026149119426
Galloway, A., Brucker-Cohen, J., Gaye, L., Goodman, E., and Hill, D. (2004). “Design for hackability,” in Proceedings of the 5th conference on Designing Interactive Systems (ACM), 363–366.
Gaver, W. W. (1991). “Technology affordances,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM), 79–84.
Gaver, W. W., Beaver, J., and Benford, S. (2003). “Ambiguity as a resource for design,” in Proceedings of the SIGCHI conference on Human Factors in Computing Systems (Ft. Lauderdale, FL: ACM), 233–240.
Ghazala, R. (2005). Circuit-Bending: Build Your Own Alien Instruments, Vol. 15. John Wiley and Sons.
Gibson, J. J. (1966). The Senses Considered as Perceptual Systems. Oxford: Houghton Mifflin. doi: 10.1080/10407413.2017.1297680
Gurevich, M., Marquez-Borbon, A., and Stapleton, P. (2012). Playing with constraints: stylistic variation with a simple electronic instrument. Comput. Music J. 36, 23–41. doi: 10.1162/COMJ_a_00103
Gurevich, M., Stapleton, P., and Marquez-Borbon, A. (2010). “Style and constraint in electronic musical instruments,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Sydney, NSW).
Hildebrand Marques Lopes, D., Hoelzl, H., and de Campo, A. (2017). “Three flavors of post-instrumentalities: the musical practices of, and a many-festo by Trio Brachiale,” in Musical Instruments in the 21st Century, eds T. Bovermann, A. de Campo, H. Egermann, S.-I. Hardjowirogo, and S. Weinzierl (Singapore: Springer), 335–360. doi: 10.1007/978-981-10-2951-6_22
Höök, K. (2006). “Designing familiar open surfaces,” in Proceedings of the 4th Nordic Conference on Human-Computer Interaction: Changing Roles (Oslo: ACM), 242–251.
Höök, K., Sengers, P., and Andersson, G. (2003). “Sense and sensibility: evaluation and interactive art,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (Ft. Lauderdale, FL: ACM), 241–248.
Jordà, S. (2004). Instruments and players: some thoughts on digital lutherie. J. New Music Res. 33, 321–341. doi: 10.1080/0929821042000317886
Kim, J., Bagla, A., and Bernstein, M. S. (2015). “Designing creativity support tools for failure,” in Proceedings of the ACM SIGCHI Conference on Creativity and Cognition (Glasgow: ACM), 157–160.
Kirsh, D. (2013). Embodied cognition and the magical future of interaction design. ACM Trans. Comput. Hum. Interact. 20:3. doi: 10.1145/2442106.2442109
Kirsh, D. (2014). The importance of chance and interactivity in creativity. Pragmat. Cogn. 22, 5–26. doi: 10.1075/pc.22.1.01kir
Maestri, L., and Wakkary, R. (2011). “Understanding repair as a creative process of everyday design,” in Proceedings of the 8th ACM conference on Creativity and Cognition, (Atlanta, GA: ACM) 81–90.
Magnusson, T. (2009). Of epistemic tools: musical instruments as cognitive extensions. Organ. Sound 14, 168–176. doi: 10.1017/S1355771809000272
Magnusson, T. (2010). Designing constraints: composing and performing with digital musical systems. Comput. Music J. 34, 62–73. doi: 10.1162/COMJ_a_00026
Marathe, S., and Sundar, S. S. (2011). “What drives customization?: control or identity?” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM), 781–790.
March, W., Jacobs, M., and Salvador, T. (2005). “Designing technology for community appropriation,” in CHI'05 Extended Abstracts on Human Factors in Computing Systems (ACM). 2126–2127.
McDermott, J., Gifford, T., Bouwer, A., and Wagy, M. (2013). “Should music interaction be easy?,” in Music and Human-Computer Interaction (London: Springer) 29–47. doi: 10.1007/978-1-4471-2990-5_2
McPherson, A., and Zappi, V. (2015a). “An environment for submillisecond-latency audio and sensor processing on BeagleBone Black,” in Proceedings of the Audio Engineering Society 138th Convention (Warsaw).
McPherson, A., and Zappi, V. (2015b). “Exposing the scaffolding of digital instruments with hardware-software feedback loops,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Baton Rouge, LA), 162–167.
McPherson, A. P., Chamberlain, A., Hazzard, A., McGrath, S., and Benford, S. (2016). “Designing for exploratory play with a hackable digital musical instrument,” in Proceedings of the ACM Conference on Designing Interactive Systems (New York, NY: DIS '16), 1233–1245.
McPherson, A. P., and Kim, Y. E. (2013). The problem of the second performer: building a community around an augmented piano. Comput. Music J. 36, 10–27. doi: 10.1162/COMJ_a_00149
Mudd, T. (forthcoming). “Material-oriented musical interactions,” in New Directions in Music and HCI, eds S. Holland, T. Mudd, K. Wilkie, M. Wanderley, and A. McPherson (Springer).
Murray-Browne, T., Mainstone, D., Bryan-Kinns, N., and Plumbley, M. D. (2011). “The medium is the message: Composing instruments and performing mappings,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Oslo), 56–59.
Newton, D., and Marshall, M. T. (2011). “Examining how musicians create augmented musical instruments,” in Proceedings of the International Conference on New Interfaces for Musical Expression (Oslo).
O'Modhrain, S. (2011). A Framework for the Evaluation of Digital Musical Instruments. Comput. Music J. 35, 28–42. doi: 10.1162/COMJ_a_00038
Redström, J. (2006). Towards user design? on the shift from object to user as the subject of design. Design Stud. 27, 123–139. doi: 10.1016/j.destud.2005.06.001
Roedl, D., Bardzell, S., and Bardzell, J. (2015). Sustainable making? balancing optimism and criticism in HCI discourse. ACM Trans. Comput. Hum. Interac. 22:15. doi: 10.1145/2699742
Sengers, P., and Gaver, B. (2006). “Staying open to interpretation: engaging multiple meanings in design and evaluation,” in Proceedings of the 6th Conference on Designing Interactive Systems (ACM), 99–108.
Seok, J.-M., Woo, J.-B., and Lim, Y.-K. (2014). “Non-finito products: a new design space of user creativity for personal user experience,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems (ACM), 693–702.
Wakkary, R., and Maestri, L. (2007). “The resourcefulness of everyday design,” in Proceedings of the 6th ACM SIGCHI conference on Creativity and Cognition (ACM), 163–172.
Wanderley, M. M., and Orio, N. (2002). Evaluation of input devices for musical expression: borrowing tools from HCI. Comput. Music J. 26, 62–76. doi: 10.1162/014892602320582981
Xambó, A., Hornecker, E., Marshall, P., Jordà, S., Dobbyn, C., and Laney, R. (2013). Let's jam the Reactable: peer learning during musical improvisation with a tabletop tangible interface. ACM Trans. Comput. Hum. Interac. 20:36. doi: 10.1145/2530541
Zamborlin, B. (2015). Studies on Customisation-Driven digital Music Instruments. PhD thesis, University of London, Goldsmiths College.
Zappi, V., and McPherson, A. (2014a). “Design and use of a hackable digital instrument,” in Proceedings of the International Conference on Live Interfaces (Lisbon).
Keywords: appropriation, affordance, constraint, digital musical instrument, style, hacking
Citation: Zappi V and McPherson A (2018) Hackable Instruments: Supporting Appropriation and Modification in Digital Musical Interaction. Front. ICT 5:26. doi: 10.3389/fict.2018.00026
Received: 01 March 2018; Accepted: 12 September 2018;
Published: 04 October 2018.
Edited by:
Robin Lee Bargar, Columbia College Chicago, United StatesReviewed by:
Roger B. Dannenberg, Carnegie Mellon University, United StatesMitsunori Ogihara, University of Miami, United States
Copyright © 2018 Zappi and McPherson. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Victor Zappi, victor.zappi@gmail.com