Editorial: Perspectives on Multisensory Human-Food Interaction

Editorial on the Research Topic
Perspectives on Multisensory Human-Food Interaction

Introduction

Eating and drinking are undoubtedly amongst life’s most multisensory experiences. Take, for instance, the enjoyment of flavor, which is one of the most important elements of such experiences, resulting from the integration of gustatory, (retronasal) olfactory, and possibly also trigeminal/oral-somatosensory cues (Prescott, 2015). Nevertheless, researchers have suggested that all our senses can influence the way in which we perceive flavor, not to mention our eating and drinking experiences. For instance, the color and shape of the food, the background sonic/noise cues in our eating environments, and/or the sounds associated with mastication can all influence our perception and enjoyment of our eating and drinking experiences (Spence, 2020).

Human-Food Interaction (HFI) research has been growing steadily in recent years (e.g., Deng et al., 2021). Research into multisensory interactions designed to create, modify, and/or enhance our food-related experiences is one of the core areas of HFI (Multisensory HFI or MHFI, Altarriba Bertran et al., 2019; Velasco and Obrist, 2020). The aim being to further our understanding of the principles that govern the systematic connections between the senses in the context of HFI.

In this Research Topic, we called for investigations and applications of systems that create new, or enhance already existing, multisensory eating and drinking experiences (what can be considered the “hacking” of food experiences) in the context of HFI. Moreover, we were also interested in those works that focus on or are based on the principles governing the systematic connections that exist between the senses. HFI also involves the experiencing of food interactions digitally in remote locations. Therefore, we were also interested in sensing and actuation interfaces, new communication mediums, and persisting and retrieving technologies for human food interactions. Enhancing social interactions to augment the eating experience is another issue we wanted to see addressed here, what has been referred to as “digital commensality” (Spence et al., 2019).

Contributions

In total, 12 papers were eventually accepted in this Research Topic. They consist of a mixture of basic research papers, perspective pieces, and review articles. Below are short summaries of these papers, following the contribution of Velasco et al. who present a reflection on the state of MHFI research. They identify different themes in which studies in this area can be classified and which may be considered to connect basic research on food and the senses, through the development of technology, to consumer-related applications. These themes are approached from a multidisciplinary perspective and involved: 1) psychological mechanisms; 2) data collection and analyses, 3) design studies and frameworks, 4) interfaces and augmentation, 5) applications. The papers included in the present Research Topic can be classified in all but category 2).

Psychological Mechanisms

In one review paper, Spence argues that the coffee-drinking experience should be conceptualized as engaging all of the senses rather than focusing solely on the chemical senses of smell and taste. The research now shows that everything from the sound of coffee preparation to the multisensory properties of the cup in which that coffee is served, and the multisensory attributes of the environment are among the product-extrinsic influences that affect the tasting experience. The paper reviews the crossmodal influences of sonic seasoning on the coffee-drinking experience. In a second review paper, Spence explores the convoluted history of blue coloring as it relates to food and drink. With a focus on the possibly apocryphal story of the blue steak meal that made people ill, the broader research on the changing meanings of color in relation to the consumer’s experience of food and drink are discussed. The role of digital technology in the rise of blue foods and drinks on sites such as Instagram’s The Art of Plating is also highlighted.

Motoki et al. investigate the topic of crossmodal correspondences between taste and temperature with a focus on food-extrinsic warmth. Experiments included the investigation of the association of temperature words with taste attributes such as sweet, sour, salty, bitter. They also address the question of how physical warmth influences sensory/hedonic ratings. The authors’ research demonstrates the existence of taste-temperature correspondences.

Kokaji and Nakatani summarize the results of an exploratory study on the role of the peripheral visual presentation of different kinds of garnish for a main dish. It was shown that the garnish placed on the plates strongly contributes to the arousal of appetite. Suggested uses of their results include images of dishes on menus, food model displays, and the use of augmented reality to place specific garnishes on, or near, real foods in order to influence the multisensory dining experience.

Design Studies and Frameworks

In order to provide a general framework and first point of contact for researchers on multisensory experiences, Velasco and Obrist present a primer on the topic, as well as its relationship with MHFI. The authors conceptualize multisensory experiences as impressions shaped by specific events, such as stages of interaction, whose sensory elements have been constructed by someone, following different principles of multisensory perception. In the context of HFI, multisensory experiences refer to impressions shaped by specific food-related events, whose sensory elements (e.g., intrinsic, and extrinsic to the food, have been constructed by someone.

Velasco et al. introduce their reality-impossibility model to guide the design of experiences in extended reality. The model encompasses two continua, that is, the reality-fantasy nature of objects and environments (described in the form of nouns), and the degree to which they follow the laws of physics or other laws (described in the form of verbs). The authors suggest that by considering these dimensions, it allows one to approach impossible experiences, that is, experiences that would not be possible without extended reality technologies.

Interfaces and Augmentation

Wang et al. discuss how research into food color perception can benefit from Virtual Reality (VR) technology, by allowing for the easy alteration of the visual appearance of stimuli. They demonstrate how VR can be used to alter the appearance of cold brew coffee. Intriguingly, they show that coffee that was made to look milkier in VR (i.e., a lighter brown color) was rated as tasting creamier than coffee having a darker appearance, even though the coffee itself did not contain milk. That said, no effects on perceived sweetness or liking of the beverages were observed. The authors highlight how their use of VR can enable ways of changing the visual appearance of stimuli that would otherwise not be possible (e.g., changing visual appearance without the need for additives).

Stäger et al. not only consider how the visual appearance of a beverage can influence flavor perception, but also how flavor might influence color perception. Using a mixed-reality (MR) setup, they had their participants taste beverages having different colors and flavors. The participants rated the flavor and color of several beverages associated with different pairings of color and flavor. Color was found to crossmodally influence flavor perception, but no effects were observed in the other direction.

While the participants in the experiments reported in the last two papers had to wear head-mounted devices in order to see the manipulated colors of beverages, in the research reported by Suzuki et al. show how a boiling motion texture was actually projected directly onto the food itself. The experiment investigated whether this projection would enhance perception (e.g., of sweetness, sourness, saltiness, spiciness, or temperature) and/or value judgments (relating to the price of the food, or appetite). The study revealed the potential of the technology to increase the expectations of the consumers and promote their purchasing intentions.

Applications

Andersen et al. present a holistic overview on how the 1) viewing, 2) creating, and 3) online sharing of digital food photography can influence consumer eating behaviour. The authors highlight multiple gaps and insufficiencies in current state-of-the-art research. Through identifying those gaps, the authors provide valuable guidance on future research opportunities, such as if and how digital food photography can support an obesity-preventative lifestyle, and how we could shape the future human-food relationships in digital, analogue, and mixed reality worlds.

Barbosa Escobar et al. present a study designed to assess how utilizing different media associated with the origin of specialty coffee would influence consumers’ expectations and experiences of premium coffee. Their results suggested that online images broadly associated with the origin of coffee (i.e., a farm) could influence premiumness expectations of coffee and that using a VR environment that depicted this general origin (vs. a control but not a city atmosphere) could enhance the perception of coffee premiumness for non-expert consumers and the enjoyment of the experience for coffee professionals.

Conclusion

Taken together, the articles brought together here therefore help to highlight the dynamic state of MHFI research currently as a growing number of opportunities are emerging for the merging of digital technologies to enhance and modify multisensory eating and drinking experiences.

Author Contributions

All authors listed have made a substantial, direct, and intellectual contribution to the work and approved it for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Altarriba Bertran, F., Jhaveri, S., Lutz, R., Isbister, K., and Wilde, D. (2019). “Making Sense of Human-Food Interaction,” in Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 1–13.

CrossRef Full Text | Google Scholar

Deng, J., Wang, Y., Velasco, C., Altarriba Bertran, F., Comber, R., Obrist, M., et al. (2021). “The Future of Human-Food Interaction,” in Extended Abstracts of the 2021 CHI Conference on Human Factors in Computing Systems, 1–6.

CrossRef Full Text | Google Scholar

Prescott, J. (2015). Multisensory Processes in Flavour Perception and Their Influence on Food Choice. Curr. Opin. Food Sci. 3, 47–52. doi:10.1016/j.cofs.2015.02.007

CrossRef Full Text | Google Scholar

Spence, C., Mancini, M., and Huisman, G. (2019). Digital Commensality: Eating and Drinking in the Company of Technology. Front. Psychol. 10, 2252. doi:10.3389/fpsyg.2019.02252

PubMed Abstract | CrossRef Full Text | Google Scholar

Spence, C. (2020). Multisensory Flavour Perception: Blending, Mixing, Fusion, and Pairing Within and Between the Senses. Foods 9 (4), 407. doi:10.3390/foods9040407

PubMed Abstract | CrossRef Full Text | Google Scholar

Velasco, C., and Obrist, M. (2020). Multisensory Experiences: Where the Senses Meet Technology. Oxford: Oxford University Press.

Google Scholar