Nik Sawe
Chris Chafe
Jeffrey Treviño3- 1Emmett Interdisciplinary Program in Environment and Resources, Stanford University, Stanford, CA, United States
- 2Center for Computer Research in Music and Acoustics, Stanford University, Stanford, CA, United States
- 3Department of Music & Performing Arts, California State University, Monterey Bay, Seaside, CA, United States
Sharing the complex narratives within scientific data in an intuitive fashion has proven difficult, especially for communicators endeavoring to reach a wide audience comprised of individuals with differing levels of scientific knowledge and mathematical ability. We discuss the application of data sonification—the process of translating data into sound, sometimes in a musical context—as a method of overcoming barriers to science communication. Data sonification can convey large datasets with many dimensions in an efficient and engaging way that reduces scientific literacy and numeracy barriers to understanding the underlying scientific data. This method is particularly beneficial for its ability to portray scientific data to those with visual impairments, who are often unable to engage with traditional data visualizations. We explore the applications of data sonification for science communicators and researchers alike, as well as considerations for making sonified data accessible and engaging to broad audiences with diverse levels of expertise.
Introduction
Conveying complex scientific narratives to a broad audience has been an ever-present challenge for science communicators and educators. The magnitude of this challenge has grown as studies in the sciences and social sciences have become increasingly more interdisciplinary in their exploration of systems and interactions (Klein, 2004), requiring both depth and breadth of knowledge across multiple fields to appropriately characterize the scope and impact of phenomena, such as climate change. Richer, more multidimensional datasets present new challenges: a three-dimensional plot, for example, reduces interpretability in comparison with a two-dimensional one (Amini et al., 2015).
In public communication, lack of scientific literacy and numeracy compound this problem. We suggest that a change in modality, from graphical representations to auditory ones using a process called data sonification, can reduce these barriers by creating an alternate way to engage with complex scientific data. This experience can be enriched by, but does not require, prior scientific expertise. Sonified data has also been theorized to require less time in training compared to visual data (Hegg et al., 2018). Data sonification's ability to convey a number of dimensions at once, as well as its potential to highlight local interactions between variables, makes it a powerful tool for data exploration for not only educators, but also scientific researchers. We will explore both of these applications, as well as sonification's unique potential to convey scientific data for the visually impaired, for whom graphical representations present greater challenges.
We regard any mapping between data and sound as a data sonification (Lodha et al., 1997; Dunn and Clark, 1999; Vickers, 2016). Such a mapping may exist in a scientific context, as an auditory graph, or in a musical context, as a work of data-driven music (Scaletti, 2018). To illustrate evaluative criteria of both these contexts, we focus on a project undertaken as both a scientific data sonification and a work of data-driven music.
Data Sonification: A Primer and Case Study
Reporting on data sonification for the National Science Foundation, Kramer et al. (2010) summarized the method clearly as the “transformation of data relations into…an acoustic signal,” with sonar and the Geiger counter among the early notable examples; the field progressed significantly during the 1980s and 1990s (Frysinger, 2005). The nature of that transformational process may directly map data to sound or may apply more creative and open-ended mappings. The results can be as diverse as a stream of numbers changing a sine wave's frequency, in the case of an auditory graph, or, in a musical context, an orchestral composition that renders the data through conventions of music theory.
For science communication purposes, the translation from data into audio reveals changing variables to the listener through changes in sonic dimensions, such as frequency, pitch, amplitude, and location in the stereo field. In musical contexts, data can map to these sonic dimensions, as well as higher-order musical dimensions, such as tempo, form, and timbre. It is relatively easy for us to attend to changes in each of these elements simultaneously, as many aspects of hearing are intrinsically multidimensional (Hermann et al., 2011). As a mental exercise, think of a song you like, and speed it up or change the instruments it features. Because a sonification must play out over time, many sonification examples represent time series data, such as salmon migration patterns (Hegg et al., 2018), and brain wave fluctuations (Parvizi et al., 2018).
When undertaken in a musical context, data sonification may facilitate or augment the learning process. A range of studies (reviewed in Rickard et al., 2005) have found that passive and active music listening improve performance on a range of cognitive tasks including reading comprehension, mathematical and general IQ test performance, visual-spatial tasks, and learning and memory.
For a more detailed case and thought experiment, we can walk through our sonification of an ecological study (Oakes et al., 2014) on the effects of climate change on the Alaskan yellow cedar tree. The audio can be found here - http://stanford.edu/~sawe/alaskanyellowcedarsonification.wav. Oakes et al. painstakingly surveyed thousands of trees across 50 vegetation plots, including five conifer species with over 30 documented variables per tree. While there was not an explicit time series element to the data, geographic latitude became a proxy for time in what is called a chronosequence: climate change had longer to impact the southern range of the forests, and so effectively, north to south told the temporal story of climate change's impacts on the forest composition. As Oakes et al. traveled south along the Alaskan coastline, the yellow cedar died off, replaced by western hemlock. This sonification maps the data to several sonic parameters, with the twin goals of rendering audible patterns in the data and creating an aesthetically satisfying musical experience that tells the story of the Alaskan forests.
To sonify the data, we chose Western orchestral instruments to represent each of the five conifer species. The yellow cedar, the central figure in the narrative, played the piano; the western hemlock was the flute. The Sitka spruce, with its wood often used to create stringed instruments, played the cello, the mountain hemlock played the violin, and the shore pine played the clarinet. Every tree was represented by a note, and the note's characteristics reflected those of the tree: the height of the tree was mapped to pitch, and its diameter was mapped to velocity (the force with which the note was struck). The fullness of the tree's crown was reflected in the note's duration. If a tree was dead—as many of the yellow cedars were in the southern plots—it was instead represented by a musical rest (silence). The form of the sonification maps direction to time: beginning with the northern-most plot and ending at the southern-most plot, it traverses the experiment's fifty tree plots from north to south, devoting an equal amount of time to each.
Explaining these mappings takes less than 30 s and anchors the listeners with a concrete understanding of what transpires within the data. When the yellow cedar's piano grows increasingly sporadic and quiet as the sonification proceeds, and the western hemlock's flute rises to prominence, listeners have the potential to grasp the study's core narrative at a visceral and intuitive level. Ability to comprehend a graph or regression table is unnecessary. And because the ecological variables were mapped to palpable musical parameters, such as loudness and rhythmic event density, listeners are able to directly infer individual tree characteristics and localized forest species compositions, details which are otherwise obscured when aggregated in the study's journal figures, and inaccessible in their complexity when viewed in raw data tables. The accessibility of the sonification led to widespread coverage by media outlets (Kahn, 2016; Nijhuis, 2016; Rassler, 2016).
Learning and the Senses: Teaching Modes and Visual Impairments
The idea that individuals have different holistic “learning styles” determined by predominant reliance on one of the senses (visual, auditory, and kinesthetic) has been widely mythologized; yet experimental evidence does not support such claims (Pashler et al., 2009; Riener and Willingham, 2010). However, presentation modes that leverage differing aspects of senses can still aid in the understanding of data, with differing receptivity across individuals. Statistical learning improves when presented through an auditory modality rather than vision or touch (Conway and Christiansen, 2005). Similarly, the ability to recognize patterns in information is improved in auditory over visual modalities (Rubinstein and Gruenberg, 1971), a result not altogether unsurprising given our frequent engagement with musical rhythm and meter.
Combining visual and auditory presentation modes is also likely beneficial. According to the modality effect, presenting some information in visual format and other elements in audio can effectively expand working memory capacity, reducing cognitive load while facilitating the integration (and hopefully retention) of information (Mayer, 2014). Sound has been shown to facilitate visual learning, arguing for multisensory training for new skills (Seitz et al., 2006), and to augment visual interface tasks (Brewster, 1997). Adding visual monitoring to an auditory monitoring task has been shown to impair performance transiently, with performance returning to normal relatively quickly (e.g., ~25 task trials) (Peres and Lane, 2005). However, in data sonification experiments, combining the modalities increased response time in listeners attempting to comprehend modeled ecological data, and the majority of listeners reported that the visuals were unhelpful or even detrimental to interpretation, competing for their attention (Hegg et al., 2018). Further research may identify the optimal ways to combine both methods for data interpretation and retention.
An obvious benefit of data sonification is its interpretability for those with visual impairments who may not be able to readily obtain analogs of traditional data visualizations. Informal learning environments (ILEs), such as museums, zoos, and aquariums, where data sonification could complement existing methods of instruction, rarely have accessible exhibits for the visually impaired. In a national survey of ILEs, 51% of respondents reported that fewer than a quarter of their exhibits were accessible to the visually impaired (Tokar, 2004). This has led visually impaired individuals to avoid ILEs, stating that there are not sufficient activities for them to engage with (Landau et al., 2005), a finding that led a group of Georgia Tech researchers to create tools for sonification and auditory displays in ILEs (Walker et al., 2006). Data sonification would also obviously benefit education across age groups within more formal learning environments, providing an additional, more tangible tool for the visually impaired to interpret textbook studies.
However, most instances of sonification research for the visually impaired are for assistance with daily life and navigation (Velázquez, 2010; Mascetti et al., 2016), and studies of engagement with the sonification of geographic or scientific (e.g., gas particle models) data show promising results but are often in the exploratory or small-sample-size stages (Delogu et al., 2010; Levy and Lahav, 2012; Weir et al., 2012). While further research into data sonification can help to quantify the learning benefits for both sighted and visually impaired individuals, the modality certainly offers engaging ways for the visually impaired to interact with informal learning environments and scientific textbook studies that would otherwise be inaccessible to them.
Challenges With Scientific and Graphical Literacy
Exacerbated by demographic and socioeconomic factors, deficits in public science literacy, graphical literacy, and numeracy impede scientific understanding (Allum et al., 2018). There is also a large degree of heterogeneity in whether individuals prefer graphically or numerically represented data, and which they find more accessible and intuitive (Politi et al., 2011), due perhaps to differing capacities for visual literacy (Avgerinou and Ericson, 1997).
Individuals with low science knowledge may feel improperly equipped to parse the scientific content they encounter, leading to disengagement or feelings that it is “too difficult” to grasp. In a survey by Pew Research Center and the Smithsonian, a representative sample of over 1,000 US adults were asked the key reason that young people avoid careers in math and science. The most common answer (46%) was that science and math were “too hard” (Monmaney, 2013). Scientists must create ways to overcome this perception and make STEM material more accessible and relatable without additional science literacy requirements, to engage individuals who do not feel qualified or empowered to navigate that material.
In a 2019 Pew Research Center report, 29% of the US respondents studied were categorized as possessing low scientific knowledge, scoring 0 to 4 correct answers out of 11 test questions (Kennedy and Hefferon, 2019). The International Literacy Survey places approximately half of Americans without the minimal numeracy skills required to utilize numbers in printed materials (Kirsch et al., 2002). This negatively impacts the ability to make decisions about their own health, finances, and other everyday decisions, compromising the ability to grasp risk magnitudes, percentages and proportions, and probabilities (Hibbard et al., 2007; Peters et al., 2007). Experiments have shown that presentation formats which reduce the required cognitive effort and improve ease of interpretation aid in complex decision-making, particularly for those with lower numeracy skills (Gurmankin et al., 2004).
As science and social science data complexity has grown over time (Klein, 2004), an understanding of multivariate datasets has also become increasingly relevant for parsing current events and assertions from news sources (Engel, 2017). Educators advocate for earlier education in data science and statistical literacy in order to foster an engaged and informed citizenry (Engel, 2017). If an engaging, holistic, and even emotional grasp of trends in multivariate datasets can be obtained through sonification, however, this creates a cognitive shortcut to understanding for individuals whose educational institutions may not yet provide such data science training. Mathematics education professor Joachim Engel asserts:
Making sense of multivariate data does not necessarily involve advanced sophisticated multivariate statistical procedures as often applied in social science research (e.g., factor analysis or logistic regression). Rather, it involves understanding multivariate phenomena and is based on developing sound heuristics, including awareness of biases and fallacies (Engel, 2017).
Data sonification can provide one of these sound heuristics by revealing the structure of underlying data and reshaping prior misperceptions.
In doing so, sonification creates an alternative or complement to the graphical representations otherwise necessary to understand large-scale data, which require different interpretive skills than those traditionally taught in current curricula that focus on smaller sample sizes (Engel, 2017). Graph literacy is a skill that correlates highly with numeracy, and in the US, with education (Galesic and Garcia-Retamero, 2011). However, about a third of low-numeracy individuals are helped greatly in data comprehension by the presence of graphs (Galesic and Garcia-Retamero, 2011), indicating that varied presentation methods can help surmount deficits in understanding of core data concepts. Graphical literacy is highly subject to individual differences (Politi et al., 2011), and so supplementary graphs are unlikely to be a cure-all for data comprehension. The degree to which data sonification can aid in data comprehension for those with low numeracy or low graphical literacy, and the proportion of the population whose data comprehension would benefit from this modality, remain open questions.
While individual differences may inform which methods of data presentation are most beneficial to understanding, baseline comprehension of science and math is unfortunately not distributed equally. With structural, geographic, social, and economic factors that combine to compromise much of the US population's interactions with scientific data, science communicators should be vigilant in searching for accessible ways of conveying that data that depend as little as possible on prior experience and preconceptions, while still offering the potential to dive deeper and more substantively into the data for those who so desire. Below, we offer a way of thinking about how to sonify data for accessibility, so that science communicators can best determine how the method might share their knowledge with a wider audience.
Sonification Mapping Design Considerations
The nature of the data itself constrains possible sonic mappings: categorical variables (differences of kind) that do not change in time can only map to discrete parameter choices that do not vary during the sonification, such as instrumental timbre, while continuous variables (differences of scale) may translate into frequency or tempo, which may change continuously in time (Walker and Nees, 2011). Hegg et al.'s data sonification experiments found that participants were most sensitive to transitions in pitch and timbre, yielding the recommendation that the most important elements of the data should be mapped to these elements given their primacy (Hegg et al., 2018). Psychoacoustics literature can provide an empirical roadmap—for example, showing how we cue to pitch perception over space perception (Deutsch, 1975)—but a great deal of flexibility remains in the sonification process. Accessibility concerns must also be considered; for example, sighted and congenitally blind listeners experience pitch height differently (Eitan et al., 2012). Science communicators therefore need to strike a balance in the sonification of their data between four key but interrelated elements: fidelity to the data, level of complexity, aesthetics, and accessibility.
Data Fidelity
How closely does the sonification represent the original scientific data? Some cases draw a clear relationship between data and sound: in a time series mapping of brain activity to pitch to hear seizures in epileptic patients, a simple power law relationship traces the microvolt amplitude of brain activity using pitch height (Parvizi et al., 2018). However, in instances where there is no clear variable to map to pitch, given how strongly we cue to this element of sound (Hegg et al., 2018), one might imagine a sonification composed of chord progressions, the structure of which is defined by higher-order moments in the data (e.g., slope, skewness, kurtosis) or weighted averages of multiple variables. While such mappings might provide a clear holistic picture of the shape of the scientific data, or draw attention to specific aspects, the relationship between the raw data and the audio becomes increasingly abstracted, potentially increasing the difficulty in conveying the science behind the sound. Sonification mappings range from the direct to the symbolic or metaphorical, with varying outcomes in ease of interpretability and learning (Keller and Stevens, 2004).
When scientific data remains unintuitive or requires too high a knowledge level, some aspects of scientific data may need to be eschewed entirely. Confidence intervals and compounding uncertainties are examples (Jones, 2000); we respond non-linearly in the subjective weightings that we assign to probabilities and expected rewards (Hsu et al., 2009; Winman et al., 2014), and we are strongly influenced by uncertainty (Wu and Gonzalez, 1999), which has presented challenges for the Intergovernmental Panel on Climate Change. In some cases, science communicators may need to consider aspects of the way this information is traditionally processed, and assess whether to counteract or reinforce these perceptual and cognitive biases.
Complexity
While sonification offers a favorable medium for conveying multivariate datasets, science communicators must still critically assess the degree of complexity in the narrative they are relating. How many dimensions are represented? What is the scale of the dataset? Are there ways to simplify, or sample from a subset of the data, while still accurately representing the whole? Can trends be used in place of individual data points? How do documented perceptual thresholds and boundaries of sound and music cognition constrain the narratives that one might construct?
The linear, time-dependent presentation of sonification data mitigates this complexity and keeps it in a digestible format: by slowing down the rate at which data is translated into audio, science communicators can provide listeners with the necessary time to process and interpret that data. Here, too, lies a trade-off, as one should not assume a captive audience, or one with the luxury of time. Thought should be put into the delivery method and audience surroundings. How many people can access the sonification at the same time? What are the attentional and cognitive demands on the audience? These pragmatic constraints will lend insight into how ambitious the data complexity should be for a given setting. For example, the IceCoreWalk project (Chafe, 2019) is a narrated sonification that traverses 800,000 years' worth of CO2 and temperature data in the time it takes to walk the length of the ice cores that provided the data (3 km). Self-directed pacing keeps the data manageable: the listener can walk with a group or solo, can move continuously or make stops, and can take in their surroundings while absorbing the data. Additionally, voiceover narration can help draw the listener's attention to key changes in sonic parameters.
Aesthetics
With so many sound design choices available, thinking carefully about aesthetic decisions is crucial. Are acoustic or synthesized sounds used? Are traditional musical instruments employed, and if so, from what culture and genre? Do the authors intend the sonification to be used as an auditory graph, or to be experienced as a musical composition, gallery installation, or soundwalk?
Data sonification need not necessarily be musical in nature, and many scientifically-useful auditory graphs are not particularly musical, or even pleasant to listen to. There are some rationales for abstracting the sonification (as we'll see in our discussion of musical choices below), and abstraction can bring some interesting choices to the communicator. For instance, audio connected to the dataset itself (e.g., whale sounds) can tie the listener more directly to the content. Likewise, sonifications may employ sounds that are “signal-referent” but indirectly related, such as using the sound of a striking match to represent a fire (Keller and Stevens, 2004). These indirect representations rely more strongly on the listener's associative memory to draw connections between data context and sonification (Keller and Stevens, 2004). A range of creative virtual studio technologies (VSTs) plugins and sound libraries offer a wealth of opportunities for such mappings, such as Soniccouture's Geosonics, which morphs field recordings of glaciers calving and frogs chirping into playable instruments. Searchable online databases of public domain audio files, such as freesound.org and BBC Sound Effects Beta, enable scientists to efficiently find, listen to, and utilize iconic domain-specific audio in their projects. The abstraction of less familiar sounds may come at a cost, however, decreasing the interpretability or relatability of the piece or prompting disengagement by the listener before they have fully understood what the sonification was trying to communicate. On the other hand, synthesized sounds can change in more subtle and precisely mapped ways than many sample-based western orchestral sounds can, due to the difference between pre-recorded and synthesized sound.
Imposing norms and structures informed by music theory onto the data has both benefits and disadvantages. The choices of scale, range, instruments, tempo, and so forth will heavily influence the interpretation of the data by the listener. For instance, we utilized the d minor scale in our sonification of the yellow cedar data, knowing that the underlying dataset characterized the decline of an iconic species, and that minor scales correlate with the emotion of sadness (Juslin and Laukka, 2004). Yet the yellow cedar's decline is counterbalanced by the rise of the western hemlock as the piece continues; viewed through this lens, one could easily frame the story as one of emergence and change, and instead play the piece in a key meant to evoke opposite emotions. Science communicators need to be cautious in such decisions. Because music evokes affective states, deciding how strongly to connect the data to emotional characteristics of the sonification, and in what ways, must be a conscious choice and responsibility. The affective states that data sonification may elicit—even while faithfully representing a dataset via a systematic and direct mapping rubric—hold the potential to exacerbate issues of science communication as advocacy or even, in the extreme, manipulation, all without using a single word. This opens up the possibility for both more ambiguous and more misleading implications, depending on the way the data is represented.
Aesthetic data interpretation choices may also influence the extent to which the data sonification facilitates improvements in cognition. Spatial-temporal task performance in the presence of music was found to depend on the tempo and mode of the music (which relate to psychological arousal and mood, respectively), with a preference for fast, major mode pieces (Husain et al., 2002). Thus, particular mapping approaches may better facilitate different types of learning goals.
Peer group determines musical preferences that echo from our formative years into adulthood (North and Hargreaves, 1999; Creed and Scully, 2011). A number of studies have analyzed those broad preferences to identify the separate factors that help determine those preferences (Colley, 2008; Delsing et al., 2008; Rentfrow et al., 2011). Individual differences in musical preferences determine what types of musical representations best suit a particular audience, and whether genre choice may be alienating. In interactive sonification contexts, such as museum exhibits, it may be possible to compose multiple translations of the same dataset to convey the same information across a range of musical genres, styles, moods, and instrumentations, leaving the choice to the listener. However, this might place undue emphasis on these musical aspects over the data itself, distracting from the narrative of the scientific data.
Benefits for Science Exploration
Data sonification can help tell the story at the heart of the data to not only the general public, but scientific experts as well. The auditory system has been theorized to be especially well-suited to trend identification (Walker and Nees, 2011), with similarities between trend and melodic contour (the abstracted shape of a succession of sonic frequencies in time). This capacity of the auditory system for pattern detection, as well as its excellent temporal resolution, can facilitate data exploration (Walker and Nees, 2011). Our hearing is well-suited to identify and contrast periodic and aperiodic events, as well as detect small changes in frequency within continuous signals, enabling us to extract complex data that might be embedded deeply within both static and noisy signals (Kramer et al., 2010).
The capacity of data sonification for simultaneous representation of many data dimensions is one of its greatest strengths for data exploration. Various traditional ways of engaging with data, such as regression analyses, can encounter problems with collinearity that compromise the ability to include the full array of variables from a rich data set. Similarly, standard graphing techniques represent only a few dimensions at once. These shortcomings can make it more likely for researchers to miss complex interactions between several variables, especially if they were not posited a priori, or if they occur only under certain conditions, such as within certain time windows. Data sonification allows researchers to stumble upon new patterns and questions when exploring their data. In this way, data sonification performs much the same function for scientific experts as it does for the general public: sonification clarifies the data's narrative and suggests a path forward for inquiry.
Conclusion
As interdisciplinary explorations of rich datasets in the sciences and social sciences uncover vast interconnections between many variables that explain the systems we observe in the world around us, the challenge for science communicators attempting to balance data complexity, fidelity, and comprehensibility is more difficult than ever. The scientific narratives that result from such exploration need to be conveyed clearly and accurately, in ways that faithfully represent the underlying data while still remaining engaging the general public. Because scientific knowledge, numeracy, and graph literacy are not equitably distributed across the population, traditional visualization methods may require skills and knowledge that present a barrier to engagement for many individuals who science communicators desire to reach.
Data sonification offers a unique tool in the toolkit of science communicators that can surmount some of these challenges, thanks to the unique ways in which our auditory system processes information and detects patterns, as well as the medium's creative and aesthetic opportunities for facilitating engagement (e.g., through musical renditions of data sets). It also enables those for whom traditional visualization methods are inaccessible, such as the visually impaired, to engage meaningfully with rich data sets. It is data sonification's potential for more accessible science communication on a variety of fronts, while enabling exciting new opportunities for data exploration, which warrants its application in a wide array of science communication contexts, from articles to classrooms to informal learning environments. There are many ways to tell the stories underlying scientific data, and as science communicators, we should endeavor to ensure that those stories reach as many ears as possible.
Author Contributions
The manuscript was written by NS, with additions from CC and JT, and revised by NS and JT. The Alaskan yellow cedar data sonification project used as an example employed an algorithm by NS to transform data to music, this was rendered into a MIDI music data file by CC, and into musical notation by JT for a previous publication.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
We would like to thank Julia Beth Goolsby for research assistance, and Adina Abeles and Arren Allegretti for feedback on the manuscript. We would also like to thank Alison Rush for coding assistance on the example Alaskan yellow cedar data sonification project.
References
Allum, N., Besley, J., Gomez, L., and Brunton-Smith, I. (2018). Disparities in science literacy. Science 360, 861–862. doi: 10.1126/science.aar8480
Amini, F., Rufiange, S., Hossain, Z., Ventura, Q., Irani, P., and McGuffin, M. J. (2015). The impact of interactivity on comprehending 2D and 3D visualizations of movement data. IEEE Trans. Vis. Comput. Graph. 21, 122–135. doi: 10.1109/TVCG.2014.2329308
Avgerinou, M., and Ericson, J. (1997). A review of the concept of visual literacy. Br. J. Educ. Technol. 28, 280–291. doi: 10.1111/1467-8535.00035
Brewster, S. A. (1997). Using non-speech sound to overcome information overload. Displays 17, 179–189. doi: 10.1016/S0141-9382(96)01034-7
Chafe, C. (2019). Ice Core Walk Project. Available online at: http://icecorewalk.org/ (accessed May 3, 2020).
Colley, A. (2008). Young people's musical taste: relationship with gender and gender-related traits. J. Appl. Soc. Psychol. 38, 2039–2055. doi: 10.1111/j.1559-1816.2008.00379.x
Conway, C. M., and Christiansen, M. H. (2005). Modality-constrained statistical learning of tactile, visual, and auditory sequences. J. Exp. Psychol. Learn. Mem. Cogn. 31, 24–39. doi: 10.1037/0278-7393.31.1.24
Creed, W. E. D., and Scully, M. A. (2011). Songs of ourselves: employees' deployment of social identity in workplace encounters. J. Manag. Inq. 20, 408–429. doi: 10.1177/1056492611432810
Delogu, F., Palmiero, M., Federici, S., Plaisant, C., Zhao, H., and Belardinelli, O. (2010). Non-visual exploration of geographic maps: does sonification help? Disabil. Rehabil. Assist. Technol. 5, 164–174. doi: 10.3109/17483100903100277
Delsing, M. J. M. H., Ter Bogt, T. F. M., Engels, R. C. M. E., and Meeus, W. H. J. (2008). Adolescents' music preferences and personality characteristics. Eur. J. Pers. 22, 109–130. doi: 10.1002/per.665
Deutsch, D. (1975). Two-channel listening to musical scales. J. Acoust. Soc. Am. 57, 1156–1160. doi: 10.1121/1.380573
Dunn, J., and Clark, M. A. (1999). Life music: the sonification of proteins. Leonardo 32, 25–32. doi: 10.1162/002409499552966
Eitan, Z., Ornoy, E., and Granot, R. Y. (2012). Listening in the dark: congenital and early blindness and cross-domain mappings in music. Psychomusicol. Music Mind Brain 22, 33–45. doi: 10.1037/a0028939
Engel, J. (2017). Statistical literacy for active citizenship: a call for data science education. Stat. Educ. Res. J. 16, 44–49.
Frysinger, S. P. (2005). “A brief history of auditory data representation to the 1980s,” in Proceedings of the International Conference on Auditory Display (Limerick: ICAD).
Galesic, M., and Garcia-Retamero, R. (2011). Graph literacy a cross-cultural comparison. Med. Decis. Mak. 31, 444–57. doi: 10.1177/0272989X10373805
Gurmankin, A. D., Baron, J., and Armstrong, K. (2004). The effect of numerical statements of risk on trust and comfort with hypothetical physician risk communication. Med. Decis. Mak. 24, 265–271. doi: 10.1177/0272989X04265482
Hegg, J. C., Middleton, J., Robertson, B. L., and Kennedy, B. P. (2018). The sound of migration: exploring data sonification as a means of interpreting multivariate salmon movement datasets. Heliyon 4:e00532. doi: 10.1016/j.heliyon.2018.e00532
Hermann, T., Hunt, A., Neuhoff, J. G., Dombois, F., and Eckel, G. (2011). The Sonification Handbook. Berlin: Logos Publishing House.
Hibbard, J. H., Peters, E., Dixon, A., and Tusler, M. (2007). Consumer competencies and the use of comaparative quality information. Med. Care Res. Rev. 64, 379–394. doi: 10.1177/1077558707301630
Hsu, M., Krajbich, I., Zhao, C., and Camerer, C. F. (2009). Neural response to reward anticipation under risk is nonlinear in probabilities. J. Neurosci. 29, 2231–2237. doi: 10.1523/JNEUROSCI.5296-08.2009
Husain, G., Thompson, W. F., and Schellenberg, E. G. (2002). Effects of musical tempo and mode on arousal, mood, and spatial abilities. Music Percept. 20, 151–171. doi: 10.1525/mp.2002.20.2.151
Jones, R. N. (2000). Managing uncertainty in climate change projections–issues for impact assessment. Clim. Change. 45, 403–419. doi: 10.1023/A:1005551626280
Juslin, P. N., and Laukka, P. (2004). Expression, perception, and induction of musical emotions: a review and a questionnaire study of everyday listening. J. New Music Res. 33, 217–238. doi: 10.1080/0929821042000317813
Kahn, B. (2016). Tree Loss Is Put to Music. Scientific American. Available online at: https://www-scientificamerican-com.stanford.idm.oclc.org/article/tree-loss-is-put-to-music-audio/ (accessed May 3, 2020).
Keller, P., and Stevens, C. (2004). Meaning from environmental sounds: types of signal-referent relations and their effect on recognizing auditory icons. J. Exp. Psychol. Appl. 10, 3–12. doi: 10.1037/1076-898X.10.1.3
Kennedy, B., and Hefferon, M. (2019). What Americans Know About Science. Available online at: https://www.pewresearch.org/science/wp-content/uploads/sites/16/2019/03/PS_2019.03.28_science-knowledge_FINAL.pdf
Kirsch, I., Jungeblut, A., Jenkins, L., and Kolstad, A. (2002). Adult Literacy in America: A First Look at the Findings of the National Adult Literacy Survey. Washington, DC. Available online at: https://nces.ed.gov/pubs93/93275.pdf
Klein, J. T. (2004). Interdisciplinarity and complexity: an evolving relationship. ECO Emerg. Complex. Organ. 6, 2–10.
Kramer, G., Walker, B., Bonebright, T. L., Cook, P. R., Flowers, J. H., Miner, N., et al. (2010). Sonification Report: Status of the Field and Research Agenda. Lincoln city, OR: Faculty Publications, Department of Psychology.
Landau, S., Wiener, W., Naghshineh, K., and Giusti, E. (2005). Creating accessible science museums with user-activated environmental audio beacons (ping!). Assist. Technol. 17, 133–143. doi: 10.1080/10400435.2005.10132103
Levy, S. T., and Lahav, O. (2012). Enabling people who are blind to experience science inquiry learning through sound-based mediation. J. Comput. Assist. Learn. 28, 499–513. doi: 10.1111/j.1365-2729.2011.00457.x
Lodha, S. K., Beahan, J., Heppe, T., Joseph, A., and Zane-ulman, B. (1997). MUSE: a musical data sonification toolkit. Environment 97. Availabe online at: https://smartech.gatech.edu/bitstream/handle/1853/50750/LodhaBeahan1997.pdf?sequence=1&isAllowed=y
Mascetti, S., Picinali, L., Gerino, A., Ahmetovic, D., and Bernareggi, C. (2016). Sonification of guidance data during road crossing for people with visual impairments or blindness. Int. J. Hum. Comput. Stud. 85, 16–26. doi: 10.1016/j.ijhcs.2015.08.003
Mayer, R. E. (2014). The Cambridge Handbook of Multimedia Learning. 2nd Edn. Cambridge: Cambridge University Press.
Monmaney, T. (2013). Educating Americans for the 21st Century. Smithson. Mag. Available online at: https://www.smithsonianmag.com/innovation/how-much-do-americans-know-about-science-27747364/ (accessed May 3, 2020).
Nijhuis, M. (2016). The Sound of Climate Change. Atl. Available online at: https://www.theatlantic.com/science/archive/2016/09/this-is-the-sound-of-a-forest-changing/499802/ (accessed May 3, 2020).
North, A. C., and Hargreaves, D. J. (1999). Music and adolescent identity. Music Educ. Res. 1, 75–92. doi: 10.1080/1461380990010107
Oakes, L. E., Hennon, P. E., O'Hara, K. L., and Dirzo, R. (2014). Long-term vegetation changes in a temperate forest. Ecosphere 5, 1–28. doi: 10.1890/ES14-00225.1
Parvizi, J., Gururangan, K., Razavi, B., and Chafe, C. (2018). Detecting silent seizures by their sound. Epilepsia 59, 877–884. doi: 10.1111/epi.14043
Pashler, H., Mcdaniel, M., Rohrer, D., and Bjork, R. (2009). Learning styles: concepts and evidence. Psychol. Sci. Public Interest 9, 105–119. doi: 10.1111/j.1539-6053.2009.01038.x
Peres, S. C., and Lane, D. M. (2005). “Auditory graphs: the effects of redundant dimensions and divided attention,” in Proceedings of the International Conference on Auditory Display (Limerick).
Peters, E., Hibbard, J., Slovic, P., and Dieckmann, N. (2007). Numeracy skill and the communication, comprehension, and use of risk-benefit information. Health Aff. 26, 741–748. doi: 10.1377/hlthaff.26.3.741
Politi, M. C., Skopec, D., Broschinski, S., Müller, A.-S., Gaissmaier, W., and Wegwarth, O. (2011). Numbers can be worth a thousand pictures: individual differences in understanding graphical and numerical representations of health-related information. Heal. Psychol. 31, 286–296. doi: 10.1037/a0024850
Rassler, B. (2016). The Art of Turning Climate Change Science Into Music. Outs. Mag. Online. Available online at: https://www.outsideonline.com/2109116/art-turning-climate-change-science-music (accessed May 3, 2020).
Rentfrow, P. J., Goldberg, L. R., and Levitin, D. J. (2011). The structure of musical preferences: a five-factor model. J. Pers. Soc. Psychol. 100, 1139–1157. doi: 10.1037/a0022406
Rickard, N. S., Toukhsati, S. R., and Field, S. E. (2005). The effect of music on cognitive performance: insight from neurobiological and animal studies. Behav. Cogn. Neurosci. Rev. 4, 235–261. doi: 10.1177/1534582305285869
Riener, C., and Willingham, D. (2010). The myth of learning styles. Chang. Mag. High. Learn. 42, 32–35. doi: 10.1080/00091383.2010.503139
Rubinstein, L., and Gruenberg, E. M. (1971). Intramodal and crossmodal sensory transfer of visual and auditory temporal patterns. Percept. Psychophys. 9, 385–390. doi: 10.3758/BF03210235
Scaletti, C. (2018). “Sonification 6= music,” in The Oxford Handbook of Algorithmic Music, eds A. McClean and R. T. Dean (Oxford: Oxford University Press), 363–386. doi: 10.1093/oxfordhb/9780190226992.013.9
Seitz, A. R., Kim, R., and Shams, L. (2006). Sound facilitates visual learning. Curr. Biol. 16, 1422–1427. doi: 10.1016/j.cub.2006.05.048
Tokar, S. (2004). Universal design in North American museums with hands-on science exhibits. Visit. Stud. Today 7, 6–10.
Velázquez, R. (2010) “Wearable assistive devices for the blind,” in Wearable and Autonomous Biomedical Devices and Systems for Smart Environment. Lecture Notes in Electrical Engineering, Vol. 75, eds A. Lay-Ekuakille and S. C. Mukhopadhyay (Berlin; Heidelberg: Springer).
Vickers, P. (2016). “Sonification and music, music and sonification,” in The Routledge Companion to Sounding Art, eds M. Cobussen, V. Meelberg, and B. Truax (London: Taylor & Francis), 135–144.
Walker, B. N., Godfrey, M. T., Orlosky, J. E., Bruce, C. M., and Sanford, J. (2006). “Aquarium sonification: soundscapes for accessible dynamic informal learning environments,” in Proceedings of the 12th International Conference on Auditory Display (ICAD 2006) (London), 238–239.
Walker, B. N., and Nees, M. A. (2011). “Theory of sonification,” in The Sonification Handbook, eds T. Hermann, A. Hunt, J. G. Neuhoff, F. Dombois, and G. Eckel (Berlin: Logos Publishing House), 9–39.
Weir, R., Sizemore, B., Henderson, H., Chakraborty, S., and Lazar, J. (2012) “Development evaluation of sonified weather maps for blind users,” in Designing Inclusive Systems eds P. Langdon, J. Clarkson, P. Robinson, J. Lazar, A. Heylighen (London: Springer).
Winman, A., Juslin, P., Lindskog, M., Nilsson, H., and Kerimi, N. (2014). The role of ANS acuity and numeracy for the calibration and the coherence of subjective probability judgments. Front. Psychol. 5:851. doi: 10.3389/fpsyg.2014.00851
Keywords: data sonification, science communication, science education, visual impairment, science literacy, numeracy, data visualization, multidimensional data
Citation: Sawe N, Chafe C and Treviño J (2020) Using Data Sonification to Overcome Science Literacy, Numeracy, and Visualization Barriers in Science Communication. Front. Commun. 5:46. doi: 10.3389/fcomm.2020.00046
Received: 27 November 2019; Accepted: 09 June 2020;
Published: 29 July 2020.
Edited by:
Erika Check Hayden, University of California, Santa Cruz, United StatesReviewed by:
Jon Bellona, University of Oregon, United StatesBailey Hilgren, University of Oregon, United States, in collaboration with reviewer JB
Copyright © 2020 Sawe, Chafe and Treviño. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Nik Sawe, c2F3ZSYjeDAwMDQwO3N0YW5mb3JkLmVkdQ==