Carlos Velasco
Qian Janice Wang
Marianna Obrist
Anton Nijholt- 1Centre for Multisensory Marketing, Department of Marketing, BI Norwegian Business School, Oslo, Norway
- 2Department of Food Science, Faculty of Technical Sciences, Aarhus University, Aarhus, Denmark
- 3Department of Computer Science, University College London, London, United Kingdom
- 4Human Media Interaction, University of Twente, Enschede, Netherlands
We present a perspective article on the state of multisensory human–food interaction (MHFI) research and lay out some reflections for research and development in this area of inquiry, based on a revision of the different spaces that we have co-created with researchers in this space. We begin by conceptualizing and defining MHFI, before moving onto presenting some of its major themes, as well as possible ways in which such themes can guide future research in the area. This article provides key definitions and foundations for the area of MHFI, as well as a first point of contact for those interested in it.
Introduction
The emerging field of Human–Food Interaction (HFI) is thought to be an area of Human–Computer Interaction (HCI) research that addresses our interactions with food (Comber et al., 2014). HFI specifically focuses on the role of technology in supporting and enriching food practices throughout the food chain, that is, from growing, through experiencing, to disposing (Khot and Mueller, 2019).
A Google Scholar search of “Human–Food Interaction” up to 2010 results in 18 research articles; however, the same search between 2010 and 2020 results in 508 (up to September 22, 2021). This, of course, is not an exhaustive index of the size of the field (perhaps some works might not talk directly about HFI, yet still be part of the field), considering that before researchers started to use the term HFI, research was already being conducted within HCI on human–food interaction (Grimes and Harper, 2008). Yet, this is initial evidence that HFI has grown significantly over the last decade. HFI has given rise to three communities of researchers, namely, Food CHI focusing on food and interaction design; another group of researchers focusing on artificial intelligence (AI) approaches to HFI; and Multisensory Human–Food Interaction (MHFI), which highlights the multisensory aspects of HFI (see Bertran et al., 2019, for an exhaustive survey of the field of HFI).
In the present article, we present our perspective on MHFI based on the work of multiple workshops and a special issue that we have co-created. While this area of research is still in its infancy, we have seen an increasing interest in fields as diverse as HCI, psychology, sensory science, and marketing, as well as an emerging effort to study the intersection between the senses, food, and technology (e.g., Velasco et al., 2018; Crofton et al., 2019; Petit et al., 2019). Last year, the specialized ACM International Conference on Multimodal Interaction (ICMI) workshop on MHFI was conducted for the fourth time (Velasco et al., 2020), yielding a total of 32 research articles across the four workshops. In addition, one special issue has been presented (with another one currently in course), together with the journals Frontiers in Psychology, Computer Science, and Nutrition, resulting in a total of nine articles. Importantly, contributions to MHFI have not been limited to this workshops and special issues.
There have been other initiatives in which research and ideas on the senses, food, and technology have been presented and discussed, such as three Data Engineering meets Intelligent Food and Cooking Recipe (DECOR, http://research.nii.ac.jp/decor/decor2020.html) workshops (2018–2020), EAT—The ICMI 2018 Eating Analysis and Tracking Challenge (https://icmi.acm.org/2018/index.php?id=challenges#eat), ACM Conference on Human Factors in Computing Systems (CHI) Play 2019 Session on Gustatory and Other Sensations, CHI Play 2019 Workshop: Chasing Play Potentials in Food Culture to Inspire Technology Design, the 1st (https://sites.google.com/dis.uniroma1.it/avi2018/co-located-events/satellite-events/the-future-of-computing-food) and 2nd (https://sites.google.com/view/fcfws2/) workshops on the Future of Computing and Food, Internet of Food (https://ieee-iotj.org/special-issues/), IoT4Food (https://www.iof2020.eu/latest/events/2020/03/iot4food), and the Play Food Children workshop at Interaction Design and Children 2020 (https://playandculture.food.blog/). In addition, this year, there was a CHI workshop entitled “The Future of Human–Food Interaction” (https://www.humanfoodinteraction.org/), which also has a dedicated special issue in the International Journal of Gastronomy and Food Science.
For all the aforesaid reasons, we thought it was time to present an article to reflect on the state of the MHFI research area from our perspective, and to contribute to the discussion on current advances and future directions, by considering where it comes from and where it is heading to. To this end, this perspective article focuses on the following aims: (1) define MHFI and present its current state, (2) reflect on the opportunities and challenges to develop this interdisciplinary area of inquiry, (3) reflect on the way forward to maximize the transfer of MHFI research into practice, and (4) present some general conclusions and takeaways. In addressing these aims, we hope to provide a starting point for reflection and discussion on the future developments of the MHFI area for those interested in this emerging topic.
The State of MHFI Research
Defining MHFI
As suggested by Velasco and colleagues (2018), the growing interest in HFI to capitalize on multisensory processes to create, modify, and enhance our food-related experiences may be explained, at least in part, by two observations. First, eating and drinking are among the most multisensory events in our everyday lives. Indeed, we interface with food through most, if not all our senses (Prescott, 2015; Spence, 2017). Second, technology is ubiquitous and there are growing efforts toward developing multisensory technologies, that is, technologies that are designed to stimulate the human senses beyond audition and vision allow researchers and practitioners to precisely control sensory quality, quantity, and delivery (e.g., haptic stimulation in mid-air, digitally-controlled smell delivery, electric taste devices; Cornelio et al., 2021; Covaci et al., 2018; Obrist et al., 2017; Velasco and Obrist, 2020). This context paved the way for MHFI as an area of inquiry. As such, it was conceived to focus mainly on the understanding of the multisensory process associated with our interaction with food (mostly eating) and on capitalizing on them when designing novel technologies and food interaction systems (Nijholt et al., 2016).
It is clear, however, that the initial scope of MHFI need to be broadened as the area progressed. First, our interactions with food are not limited to eating. Indeed, research from different fields have characterized the food interaction journey as consisting of different stages such as growing/purchasing, cooking, eating, and disposing (Choi et al., 2014; Schifferstein, 2016). Secondly, the world is currently facing multiple challenges concerning food including, but not limited to, unsustainable food practices, food and climate change, food waste, obesity, (mal)nutrition, and hunger (FAO, 2018). With this in mind, MHFI can contribute beyond eating (e.g., nudging, expectations development, and disposing), to tackle other interaction stages (pre-eating and post-eating), as well as important food-related challenges that humanity faces (e.g., associated with health and sustainability). In fact, MHFI can connect research on the senses, food, and multisensory technologies to design any kind of food interaction and experience.
So how should we define MHFI? We follow the definition of Choi et al. (2014) of HFI as the interrelationship between self and food, though we include others (a social element) as part of our definition, considering that HFI can also involve food interactions between selves, mediated by technology (e.g., commensality, see Spence et al., 2019). Therefore, we define MHFI as a research area that studies the role of the senses in the interrelationship between self, others, and food, and that capitalizes on such understanding to modify existing and/or create new self–others–food interrelationships through technology. Note that the fact that food experiences are multisensory in nature makes HFI multisensory per se; however, the term MHFI involves the word “multisensory” as it places the senses at the center of, and emphasizes their role in HFI research and practice. While MHIF is a multidisciplinary area of study as defined above, it is worth mentioning that there is extensive research on multisensory perception and its relationship to food in fields such as psychology and sensory science (e.g., Prescott, 2015; Spence, 2017) and thus MHFI can build on such research to modify existing and/or create new self–others–food interrelationships through technology.
Here, it is important to mention that, in HCI, interaction is defined as any communication between a user and a computer, be it direct or indirect (Hornbæk and Oulasvirta, 2017). As such, MHFI also involves communication as part of it. Importantly, however, MHFI can also be about designing interactive interfaces between humans and digital technology and about designing interactive experiences (Spence et al., 2019). It is perhaps useful to think about MHFI as multisensory experiences in HFI. In other words, “In the context of HFI, multisensory experiences refer to impressions formed by specific food-related events, whose sensory elements (e.g., intrinsic and extrinsic to the food, see, for example, Wang et al., 2019) have been carefully crafted by someone for a given receiver (e.g., diners). For instance, to create the impression of a taste, say “sweet”, colors, textures, and specific smells can be considered in a specific event.” (Velasco and Obrist, 2021, p. 3). Given that experiences are only one part of MHFI, this definition may be broadened as follows: MHFI refers to self–food interrelationships formed by specific food-related events, whose sensory elements have been carefully designed by somebody for a given receiver or group of receivers.
Major Themes in MHFI
To formulate our perspective on the current state of MHFI research, we revisited the various contributions to our four ICMI workshops on MHFI, as well as one research topic in Frontiers (n = 41). We identified key themes, associated with the research and development process of the area, that emerged from the articles (see Supplementary Appendix SA1, for the titles of the articles and themes identified). Although the number of articles is relatively small, it is representative if one considers that MHFI focuses only on a subset of research of HFI (see Introduction). Importantly, because this article is intended to reflect upon our perspective in MHFI, the sample of articles serves that purpose.
After reviewing these articles, we identified, through a series of iterations, five major themes. Below, we present a summary of such themes and the proportions of articles in each theme (see also Supplementary Appendix SA1).
1) Data collection and analyses: Articles in which a system for data collection and/or analyses are presented. Number of articles = 5 (12.3%).
2) Psychological mechanisms: Articles presenting studies designed to better understand psychological mechanisms associated with MHFI (e.g., crossmodal effects on flavor perception, such as the influence of, say, auditory stimuli on perceived taste intensity). Number of articles = 13 (31.7%).
3) Design studies: Articles studying design approaches and frameworks. Number of articles = 5 (12.2%).
4) Augmentation and interfaces: Articles focused on food augmentation processes and interfaces. Number of articles = 7 (17.1%).
5) Applications—Commensality, education, entertainment, and/or health: Articles studying interactions specifically targeting one of these areas. Number of articles = 11 (26.8%).
The majority of articles (about a third) have focused on psychological mechanisms, followed by specific applications in commensality, education, entertainment, and/or health, followed by augmentation and interfaces, design studies, and data collection and analyses. Here, it is worth mentioning that research and development in MHFI can capitalize on existing research from other fields (e.g., research on the neuroscience of flavor perception to develop new systems; Prescott, 2015).
Foundations of MHFI: Connecting Fields, Research, and Practice
Following our analyses of the articles and the key themes that we identified, we see that there is possibility for guiding research in MHFI in such a way that it starts from an understanding of MHFI psychological processes, which result in applications (Figure 1).
FIGURE 1. Possible themes associated with research and development in MHFI.
In Figure 1, we present the themes associated with progress in MHFI. The first and second themes consist of elucidating psychological mechanisms and data collection and analytical methods, both of which support the human understanding foundations of MHFI. The third theme consists of design studies to develop frameworks and the fourth theme on the development of specific interfaces and augmentation technologies, both of which support the user interaction foundations. These technological interfaces can then be used to target specific applications in various areas such as commensality, education, entertainment, and/or health, which constitutes the fifth theme. Note that it is possible that specific interfaces are already designed with applications in mind, though.
It is perhaps worth illustrating with a now classic example of MHFI, namely, the Chewing Jockey, which is a system that monitors mastication and synchronizes sound-delivery to it (Koizumi et al., 2011). This technology capitalized (1. Psychological mechanisms) on previous studies developed to understand and document the role of auditory cues on modulating texture and taste perception (Zampini and Spence, 2004). Said studies have suggested, for instance, that the crispiness of potato chips can be enhanced by chewing sounds or white noise with a high-pass filter (see also Spence, 2015, for a review). Based on this idea and aiming to redesign the eating experience (3. Design studies and frameworks), Koizumi et al. (2011) developed the chewing jockey technology (4. Interfaces and augmentation). Once this interface was designed, the authors moved on to specific applications (5. Applications). Here, the authors suggested at least two applications. The first consisted of using the system to enhance texture perception for the elderly (health), and the second, to design novel fun interactions (entertainment), such as mapping the sounds of screaming sounds to gummy bear chewing. While this study did not fully capitalize on available design frameworks for sound augmentation (3), it follows general experience design guidelines. In addition, while data collection (2) was not a part of it, given that the system uses a chewing tracking system, it is possible to collect data, as well (see also Lin et al., 2020, FoodFab and Narumi et al., 2011 Metacookie, for other examples).
All in all, the possible themes associated with research and development in MHFI allow researchers and practitioners to think of how to connect everything from basic research on multisensory influences on self–others–food interrelationships all the way to possible applications. It is important to mention here that one of the key characteristics of this area of research and practice is its interdisciplinary nature, involving fields such as, though not limited to, psychology and neuroscience, sensory science, HCI, and marketing. This interdisciplinary work can guarantee the strong conceptual and practical foundations of every step of the research and development process (Figure 1).
Research Directions and Conclusions
MHFI is a nascent area of research and, as such, there are multiple unanswered questions and directions for research that need to be addressed. For example:
• What human problems can MHFI design help with? (e.g., help children to enjoy and engage with food in new interactive ways, help enhance flavor experiences for the elderly often suffering of reduced sensory abilities, augment food experiences in outer space where food could be perceived as bland).
• How do we move from lab-based research explorations to real-world deployments of MHFI applications/technologies to improve people’s lives?
• What are some key ethical reflections and responsibilities around MHFI design?
Considering those questions, we reflect below upon some key areas for future research and development in the area of MHFI, especially based on the discussions in our workshops and special issues. We also reflect on how to approach MHFI research and put it into practice.
Areas of Future Development
Direct Interaction With Food: Designing Experiences That Enhance the Eating Experience
Designing technology around the ingestion process, we can use the senses to highlight flavor as well as influence appetite. For instance, for the elderly (Doets and Kremer, 2016), work in MHFI may contribute to making up for losses in smell and taste perception as well as promote desire to eat. MHFI could also solve challenges around eating in extreme situations (e.g., space exploration) where technology and psychological understanding are needed to create new ways of eating (Obrist et al., 2019). Referring to our framework (Figure 1), this area relies on making use of existing knowledge in psychological mechanisms to develop design frameworks, interfaces, and applications to support people’s eating experiences.
Social Aspect: Designing Interaction With Others Around Food
Food is a means for socialization and sharing (Niewiadomski et al., 2019). The social aspects around food can include food growing, producing, purchasing, preparation, eating, sharing, and disposing (Velasco and Obrist, 2021). Remote commensality is a special area of interest (e.g., Ceccaldi et al., 2020), which can be enhanced by considering the multisensory processes associated with social dining (Spence et al., 2019).
Change Attitude Towards Food: Designing Food Interaction to Nudge People
Beyond direct ingestion, how can we use the senses to change people’s mindset about specific foods (de Vries et al., 2020; Zhao et al., 2016)? This could be influencing their food-based decision-making—including attitude towards certain foods, purchase intentions, and disposal habits (Cadario and Chandon, 2020; Hollands et al., 2017). For example, the integration of augmented reality and other visually enabling technologies in the process of food purchases can influence the way in which people develop purchase intentions (Petit et al., 2021; Velasco et al., 2018; Toet et al., 2017). MHFI has already seen several theoretical developments, but what is missing is the integrated use of novel technologies and data collection mechanisms that can measure large quantities of real-world data.
Digital Augmentation: Technology Enabled Food Interaction Experiences
An increasing number of digital technologies are being developed to stimulate our chemical senses and thus create new MHFIs and/or to study multisensory processes (Cornelio et al., 2021). Indeed, there is increasing interest in the way in which technologies can be used to create and/or augment eating and drinking experiences digitally (Spence et al., 2017; Vi et al., 2017). Note, however, that augmentation is not limited to the chemical senses. Indeed, a wide range of research directions and applications (e.g., context enhancement, food structure and texture, and sensory augmentation) have been forwarded for augmented reality in food interactions (see Narumi, 2016; Crofton et al., 2019). With respect to the pipeline framework, this is an exciting area with many new interface developments. However, what is less clear are how such interactions can spread outside the lab and be deployed in the real world.
Ethical Considerations: Responsible Innovation Around Food Interaction
Recently, Velasco and Obrist (2021) indicated that, as there is scope for development in MHFI, there are also key responsibilities, and thus, we need to consider the ethical implications of this area of research. This is particularly important when moving from lab-based explorations into real-world deployments. What are key ethical reflections in MHFI? Consider the abilities of digital technologies like food 3D printers and virtual and augmented (VR/AR) reality. Those technologies enable us to create/design realities that are not matching the physical world. For example, we can now change the appearance of food to make it look more appetizing, or we can change the infill structure of 3D printed food to affect people’s feeling of satiety (as in FoodFab by Lin et al., 2020). We can create food perception illusions that can deceive people, but in this case benefit the person (eating less, which in light of a global challenge of obesity can be considered a desirable intervention). However, who decides about the beneficiaries and when it is ok to create such experiences? While this will require ongoing discussion on the topic, it is key to treat receivers of MHFIs fairly by considering their differences and similarities, avoiding biases, and ensuring accessibility of technology (see also Choi et al., 2014). This is an overarching concern that touches upon all elements of the pipeline framework.
Open a Dialogue on How to Approach MHFI in Research and Practice
While the themes identified in Figure 1 are all important, the question remains how research should be conducted going forward. Should researchers target one specific step, or should all five be considered in order for a study to be considered MHFI? Moreover, how should researchers from different disciplines approach a potential MHFI research topic? Should it grow organically from the ground up, where people should focus on first developing psychological mechanisms? Or should researchers identify problem areas that are needed, then look up research on psychological/neurological mechanisms upon which to develop the technological application? Developments from the four MHFI workshops have shown that in 2016, the focus was on augmentation and interfaces, but with time, the focus has shifted towards the two ends of the research pipeline: Either towards psychological mechanisms or final applications (Supplementary Appendix SA1).
The list of questions raised is evidence of the idea that we are only at the beginning of understanding and exploring the areas and themes around MHFI, and while we wish to provide answers, it is more our intention to open up a dialogue with the community, and as part of that, in effect, continue our past efforts (workshops, special issues), with this perspective article.
MHFI applications are often developed in the laboratory. Brands have recently focused on AR mobile applications to highlight the sensory aspects of their products (Jacobsen et al., 2021). How can the area move research and development from just workshop demos to wider adoptability? Should we engage companies as collaborators, or encourage entrepreneurship among MHFI researchers? Should people work with relevant stakeholders (e.g., hospitals and schools) with an interest in putting research into practice? These questions deserve discussion. Notably, our position is that, in order to make MHFI research relevant in both basic and applied research, in theory development and practical implications, having relevant stakeholders involved will be critical, from consumers, through researchers, to firms and/or other applied contexts. What is more, as the area develops, it will be important to develop both qualitative reviews and meta-analyses that help shape the foundations of MHFI beyond our perspective.
Conclusion
We presented here our perspective on the state of MHFI research. We started by placing it in the broader context of HCI and more particularly HFI, and then defining it. Building on the four ICMI workshops on MHFI, as well as one research topic in Frontiers, we identified five key themes of research in this area, namely, (1) data collection and analyses, (2) psychological mechanisms, (3) design studies, (4) augmentation and interfaces, and (5) applications—commensality, education, entertainment, and/or health. These themes can constitute a compass for the interdisciplinary development of this area, from basic research to practice.
In addition to these themes, we described some key areas of research we believe will be crucial in the development of MHFI, which include: (1) designing experiences that enhance the eating experience, (2) interaction with others around food, (3) changing mindsets and attitudes, (4) interfaces and technologies, and (5) ethics.
We believe that research in MFHI should be approached in a way that connects basic and applied research, and which results, in the end, in applications, potentially co-developed with stakeholders in the applied world.
Data Availability Statement
The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author.
Author Contributions
All the authors listed made a substantial, direct, and intellectual contribution to the work and approved it for publication.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary Material
The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fcomp.2021.694691/full#supplementary-material
References
Bertran, F. A., Jhaveri, S., Lutz, R., Isbister, K., and Wilde, D. (2019). Making Sense of Human-Food Interaction. In Proceedings of the 2019 CHI Conference on Human Factors in Computing Systems, 678. doi:10.1145/3290605.3300908
Cadario, R., and Chandon, P. (2020). Which Healthy Eating Nudges Work Best? A Meta-Analysis of Field Experiments. Marketing Sci. 39 (3), 465–486. doi:10.1287/mksc.2018.1128
Ceccaldi, E., Huisman, G., Volpe, G., and Mancini, M. (2020). Guess Who's Coming to Dinner? Surveying Digital Commensality during Covid-19 Outbreak. In Companion Publication of the 2020 International Conference on Multimodal Interaction (pp. 317–321).
Choi, J. H.-J., Foth, M., and Hearn, G. (2014). Eat, cook, Grow: Mixing Human-Computer Interactions with Human-Food Interactions. Cambridge, MA: MIT Press.
Comber, R., Choi, J. H.-j., Hoonhout, J., and O'Hara, K. (2014). Designing for Human-Food Interaction: An Introduction to the Special Issue on 'food and Interaction Design'. Int. J. Human-Computer Stud. 72 (2), 181–184. doi:10.1016/j.ijhcs.2013.09.001
Cornelio, P., Velasco, C., and Obrist, M. (2021). Multisensory Integration as Per Technological Advances: A Review. Front. Neurosci. 15, 652611. doi:10.3389/fnins.2021.652611
Covaci, A., Zou, L., Tal, I., Muntean, G. M., and Ghinea, G. (2018). Is Multimedia Multisensorial?-A Review of Mulsemedia Systems. ACM Comput. Surv. (Csur) 51 (5), 1–35.
Crofton, E. C., Botinestean, C., Fenelon, M., and Gallagher, E. (2019). Potential Applications for Virtual and Augmented Reality Technologies in Sensory Science. Innovative Food Sci. Emerging Tech. 56, 102178. doi:10.1016/j.ifset.2019.102178
de Vries, R. A., Keizers, G. H., van Arum, S. R., Haarman, J. A., Klaassen, R., van Delden, R. W., et al. (2020). Multimodal Interactive Dining with the Sensory Interactive Table: Two Use Cases. In Companion Publication of the 2020 International Conference on Multimodal Interaction (pp. 332–340). doi:10.1145/3395035.3425654
Doets, E. L., and Kremer, S. (2016). The Silver Sensory Experience - A Review of Senior Consumers' Food Perception, Liking and Intake. Food Qual. Preference 48, 316–332. doi:10.1016/j.foodqual.2015.08.010
FAO (2018). The Future of Food and Agriculture – Alternative Pathways to 2050. Rome, 224. Licence:CC BY-NC-SA 3.0 IGO.
Grimes, A., and Harper, R. (2008). Celebratory Technology: New Directions for Food Research in HCI. In Proceedings of the SIGCHI conference on human factors in computing systems, Florence, Italy, April 5 - 10, 2008 (pp. 467–476).
Hollands, G. J., Bignardi, G., Johnston, M., Kelly, M. P., Ogilvie, D., Petticrew, M., et al. (2017). The TIPPME Intervention Typology for Changing Environments to Change Behaviour. Nat. Hum. Behav. 1 (8), 1–9. doi:10.1038/s41562-017-0140
Hornbæk, K., and Oulasvirta, A. (2017). What Is Interaction. In Proceedings of the 2017 CHI Conference on Human Factors in Computing Systems (CHI '17), Denver, Colorado, USA, May 6 - 11, 2017. New York, NY, USA: ACM, 5040–5052. doi:10.1145/3025453.3025765
Jacobsen, L. F., Stancu, V., Wang, Q. J., Aschemann-Witzel, J., and Lähteenmäki, L. (2021). Connecting Food Consumers to Organisations, Peers, and Technical Devices: The Potential of Interactive Communication Technology to Support Consumers' Value Creation. Trends Food Sci. Techn. 109, 622–631. doi:10.1016/j.tifs.2021.01.063
Khot, R. A., Mueller, F., and Young, D. (2019). Human-food Interaction. FNT in Human-Computer Interaction 12 (4), 238–415. doi:10.1561/1100000074
Koizumi, N., Tanaka, H., Uema, Y., and Inami, M. (2011). Chewing Jockey: Augmented Food Texture by Using Sound Based on the Cross-Modal Effect. In Proceedings of the 8th international conference on advances in computer entertainment technology, Lisbon, Portugal, November 8 - 11, 2011 (pp. 1–4).
Lin, Y. J., Punpongsanon, P., Wen, X., Iwai, D., Sato, K., Obrist, M., et al. (2020). FoodFab: Creating Food Perception Illusions Using Food 3D Printing. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, Honolulu, HI, USA April 25 - 30, 2020 (pp. 1–13). doi:10.1145/3313831.3376421
Narumi, T., Nishizaka, S., Kajinami, T., Tanikawa, T., and Hirose, M. (2011). MetaCookie+. In 2011 IEEE Virtual Reality Conference, Singapore, 19-23 March 2011. IEEE, 265–266.
Narumi, T. (2016, November). Multi-Sensorial Virtual Reality and Augmented Human Food Interaction. In Proceedings of the 1st Workshop on Multi-sensorial Approaches to Human-Food Interaction, Tokyo, Japan, 16 November 2016 (pp. 1–6).
Niewiadomski, R., Ceccaldi, E., Huisman, G., Volpe, G., and Mancini, M. (2019). Computational Commensality: from Theories to Computational Models for Social Food Preparation and Consumption in HCI. Front. Robot. AI 6, 119. doi:10.3389/frobt.2019.00119
Nijholt, A., Velasco, C., Karunanayaka, K., and Huisman, G. (2016, October). 1st International Workshop on Multi-Sensorial Approaches to Human-Food Interaction (Workshop Summary). In Proceedings of the 18th ACM International Conference on Multimodal Interaction, Tokyo, Japan, 16 November 2016 (pp. 601–603).
Obrist, M., Gatti, E., Maggioni, E., Vi, C. T., and Velasco, C. (2017). Multisensory Experiences in HCI. IEEE MultiMedia 24 (2), 9–13. doi:10.1109/mmul.2017.33
Obrist, M., Tu, Y., Yao, L., and Velasco, C. (2019). Space Food Experiences: Designing Passenger's Eating Experiences for Future Space Travel Scenarios. Front. Comput. Sci. 1, 3. doi:10.3389/fcomp.2019.00003
Petit, O., Javornik, A., and Velasco, C. (2021). We Eat First with Our (Digital) Eyes: Enhancing Simulation of Eating through Visual-Enabling Technologies. J. Retailing. Available at: https://www.sciencedirect.com/science/article/abs/pii/S0022435921000336
Petit, O., Velasco, C., and Spence, C. (2019). Digital Sensory Marketing: Integrating New Technologies into Multisensory Online Experience. J. Interactive Marketing 45, 42–61. doi:10.1016/j.intmar.2018.07.004
Prescott, J. (2015). Multisensory Processes in Flavour Perception and Their Influence on Food Choice. Curr. Opin. Food Sci. 3, 47–52. doi:10.1016/j.cofs.2015.02.007
Schifferstein, H. N. J. (2016). “The Roles of the Senses in Different Stages of Consumers’ Interactions with Food Products,” in Multisensory Flavor Perception: From Fundamental Nueroscience through to the Marketplace. Editors P. F. Betina, and C. Spence (Sawston: Woodhead Publishing). doi:10.1016/b978-0-08-100350-3.00015-8
Spence, C. (2015). Eating with Our Ears: Assessing the Importance of the Sounds of Consumption on Our Perception and Enjoyment of Multisensory Flavour Experiences. Flavour 4, 1. doi:10.1186/2044-7248-4-3
Spence, C., Mancini, M., and Huisman, G. (2019). Digital Commensality: Eating and Drinking in the Company of Technology. Front. Psychol. 10, 2252. doi:10.3389/fpsyg.2019.02252
Spence, C., Obrist, M., Velasco, C., and Ranasinghe, N. (2017). Digitizing the Chemical Senses: Possibilities & Pitfalls. Int. J. Human-Computer Stud. 107, 62–74. doi:10.1016/j.ijhcs.2017.06.003
Toet, A., Schaik, M. G. V., Kaneko, D., and Erp, J. B. V. (2017). Are Food Cinemagraphs More Yummy Than Stills. In Proceedings of the 2nd ACM SIGCHI International Workshop on Multisensory Approaches to Human-Food Interaction, Glasgow UK, 13 November 2017 (pp. 1–4).
Velasco, C., Karunanayaka, K., and Nijholt, A. (2018). Editorial: Multisensory Human-Food Interaction. Front. Psychol. 9, 796. doi:10.3389/fpsyg.2018.00796
Velasco, C., Nijholt, A., Spence, C., Narumi, T., Motoki, K., Huisman, G., et al. (2020). Multisensory Approaches to Human-Food Interaction. In Proceedings of 22nd ACM International Conference on Multimodal Interaction (ICMI’20), Virtual Event Netherlands October 25 - 29, 2020. Utrecht, the Netherlands, New York, NY, USA: ACM, 3. doi:10.1145/3382507.3419749
Velasco, C., and Obrist, M. (2021). Multisensory Experiences: A Primer. Front. Comput. Sci. 3, 614524. doi:10.3389/fcomp.2021.614524
Velasco, C., and Obrist, M. (2020). Multisensory Experiences: Where the Senses Meet Technology. Oxford: Oxford University Press.
Velasco, C., Obrist, M., Petit, O., and Spence, C. (2018). Multisensory Technology for Flavor Augmentation: a Mini Review. Front. Psychol. 9, 26. doi:10.3389/fpsyg.2018.00026
Vi, C. T., Ablart, D., Arthur, D., and Obrist, M. (2017). Gustatory Interface: the Challenges of ‘how’to Stimulate the Sense of Taste. In Proceedings of the 2nd ACM SIGCHI International Workshop on Multisensory Approaches to Human-Food Interaction, Glasgow UK, 13 November 2017 (pp. 29–33).
Wang, Q. J., Mielby, L. A., Junge, J. Y., Bertelsen, A. S., Kidmose, U., Spence, C., et al. (2019). The Role of Intrinsic and Extrinsic Sensory Factors in Sweetness Perception of Food and Beverages: A Review. Foods 8, 211. doi:10.3390/foods8060211
Zampini, M., and Spence, C. (2004). The Role of Auditory Cues in Modulating the Perceived Crispness and Staleness of Potato Chips. J. Sensory Stud. 19, 347–363. doi:10.1111/j.1745-459x.2004.080403.x
Keywords: multisensory, human–food interaction, technology, senses, foundations
Citation: Velasco C, Wang QJ, Obrist M and Nijholt A (2021) A Reflection on the State of Multisensory Human–Food Interaction Research. Front. Comput. Sci. 3:694691. doi: 10.3389/fcomp.2021.694691
Received: 13 April 2021; Accepted: 29 October 2021;
Published: 01 December 2021.
Edited by:
Marko Tkalcic, University of Primorska, SloveniaReviewed by:
Jettie Hoonhout, Philips, NetherlandsHanna Jasmin Hauptmann, University of Konstanz, Germany
Copyright © 2021 Velasco, Wang, Obrist and Nijholt. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Carlos Velasco, Y2FybG9zLnZlbGFzY29AYmkubm8=