Method for modifying haptic feedback by displaying onomatopoeia

Haptic feedback, which enhances operability and realism, is extensively employed in smartphones and controllers. One notable tactile presentation method involves the use of visual stimuli to evoke tactile sensations, exemplified by the concept of pseudo-haptics. In these methodologies, resistance and force are simulated by modulating the velocity of the avatar’s finger or the operation pointer. Currently, there exists a discrepancy between the user’s inherent sensory perception and the visual information presented. In this study, we propose a novel approach to modifying the perception of tactile stimulation by concurrently presenting an onomatopoeic word with the tactile stimulus. For the experiment, we developed a smartphone application that, upon tapping a displayed button, triggers both a vibration stimulus and the presentation of an onomatopoeic word that conveys a sense of touch and sound. We employed six switches with varying tactile sensations to evaluate whether the user’s perception would be influenced by the type of onomatopoeia displayed in the application. The experimental results demonstrated that five onomatopoeic words elicited distinct tactile sensations. Additionally, we observed that four of these words enhanced the perceived realism of the button-press sensation. This method diverges from existing techniques by altering the tactile perception through visually presented linguistic information. Although this approach is constrained to scenarios where the haptic object is within the visual field, it is straightforward to implement and can be readily applied to existing smartphones and virtual reality devices.

1 Introduction

Cross-modal haptic feedback, such as pseudo-haptics and the rubber-hand illusion, has garnered significant attention in graphical user interface (GUI) operations on both computers and smartphones.

Pseudo-haptics (Lecuyer et al., 2000) refers to a phenomenon in which visual stimuli, such as a self-projected mouse cursor or an avatar’s arm, alter their speed and shape, thereby creating the illusion of haptic feedback. The rubber-hand illusion (Armel and Ramachandran, 2003) is another intriguing phenomenon in which individuals feel as though they are touched on their real arm when a rubber arm, perceived as part of their body, is touched. These phenomena underscore the strong interaction between vision and touch.

Tactile presentation through visual stimuli is cost-effective because it removes the need for dedicated tactile presentation devices. However, relying solely on visual stimuli for haptic presentation is impractical. Contemporary devices, such as smartphones and game controllers, are typically equipped with haptic feedback mechanisms. Moreover, in entertainment media, including films, immersive experiences are often enhanced through the combined use of auditory, tactile, and visual stimuli.

In this study, we do not discount the value of tactile presentation methods that rely solely on visual perception. However, we posit that a higher degree of immersion and realism can be achieved by integrating visual stimuli with widely available vibrotactile feedback. To this end, we propose inducing sensory changes by simultaneously presenting onomatopoeic words—terms that evoke changes in state and tactile sensations—rather than employing visual motion changes as the presentation method.

In this study, we conducted our experiment in Japanese, which is the author’s first language and allowed us to recruit participants more readily in our setting. Japanese is particularly known for having a large number of onomatopoeic expressions: one earlier study identified around 2,000 such words (Ivanova, 2006), and some commercially available dictionaries list as many as 4,500. Consequently, a wide range of sensations can be succinctly expressed in Japanese.

Whereas English sometimes relies on verbs to convey sounds or sensations, Japanese features many distinctive onomatopoeic terms that symbolically represent sounds or tactile impressions in written form, thus enriching expressions in manga, anime, and everyday communication. We hypothesize that presenting onomatopoeic words that strongly convey tactile impressions—concurrently with the physical sensation—could alter users’ perceptions of that sensation.

Recent studies have increasingly highlighted the significant influence of onomatopoeia on tactile perception. In a study by Sakamoto and Watanabe (2018), researchers analyzed the onomatopoeic expressions used by Japanese speakers when touching various materials and demonstrated a systematic relationship between phonetic features and tactile properties. For example, voiced consonants (e.g.,/dz/,/g/) are associated with a “rough” texture, whereas voiceless consonants (e.g.,/s/,/ts/) correspond to a “smooth” texture.

Additionally, Nalbantoğlu et al. (2024) demonstrated that presenting onomatopoeia congruent with tactile stimuli alters perceptions of an object’s softness or hardness, suggesting that onomatopoeia can influence the subjective experience of tactile perception. Furthermore, a study by Lo et al. (2017). Lo et al. (2017) confirmed that, for English speakers, fricatives evoke a “rough” sensation, while plosives elicit a “smooth” sensation. This finding points to the possibility that the relationship between onomatopoeia and tactile perception may be universal across languages.

These insights suggest that, much like vision and audition, visually presented onomatopoeia can modulate tactile perception, strongly supporting the notion that onomatopoeia functions as an integral part of tactile information processing.

This approach represents an extension of pseudo-haptics, utilizing visual stimuli to elicit changes in tactile perception. However, the precise nature of the tactile perception changes induced by onomatopoeia remains unexplored. This paper aims to investigate the effects of onomatopoeia on tactile perception.

We propose a practical method for altering tactile impressions by displaying onomatopoeia, thereby enhancing tactile realism and immersion through the combination of visual and tactile stimuli. Onomatopoeia, which can express changes in state, touch, and movement with minimal characters, effectively conjures vivid scenes in comics and novels. Research by Oh and Fukusato et al. has shown that onomatopoeia can enhance the reality and presence of sound by displaying it on the screen of audiovisual works (Oh and Kim, 2018; Fukusato and Morishima, 2014; Wang et al., 2017).

This study explores the application of the immersive and realistic effects of onomatopoeic display to tactile presentation. By using simple vibration feedback as a catalyst and presenting onomatopoeia simultaneously, we aim to alter the user’s tactile impression and enhance the sense of immersion. If effective, this method could provide a variety of tactile feedback without requiring complex visual changes or additional mechanisms. It holds particular promise for delivering diverse tactile sensations in entertainment contexts.

To evaluate the proposed method’s efficacy, we developed a smartphone application that simultaneously provides vibration feedback and displays onomatopoeia when the user taps a button on the screen. We then investigated the resulting changes in tactile impressions, as well as variations in perceived reality and immersion, caused by displaying different onomatopoeic words.

2 Related works

2.1 Application of onomatopoeia to sensory presentation

Various methods have been proposed to enhance the realism of images, the reality of audio, and immersion in VR by displaying onomatopoeia. Fukusato et al. proposed a method of adding auto-generated onomatopoeia to movies and animations. In their study, they displayed and moved onomatopoeic captions appropriate for the situation and sound (Fukusato and Morishima, 2014). Questionnaire results indicated that this approach increased participants’ immersion in the movies.

Choi et al. (2018) demonstrated that adding onomatopoeia to sound feedback could enhance both the presence and reality of the sound. It has also been suggested that combining incomplete or inadequate sounds with onomatopoeia can offer sufficient realism, comparable to real sounds.

Oh and Kim (2018) conducted an experiment in which onomatopoeia was displayed simultaneously with sound feedback in a VR environment. Their findings indicated that presenting onomatopoeia and sounds together could draw users’ attention to the sounds and enhance their presence and immersion in the experience.

As these studies illustrate, displaying onomatopoeia can enhance immersion and increase the realism of concurrently presented stimuli. However, most research on onomatopoeia-based sensory presentation has focused primarily on visual and auditory modalities. Consequently, the effect of presenting onomatopoeia on tactile perception has not been thoroughly investigated.

2.2 Haptic feedback methods

Haptic feedback is a useful means of improving both usability and immersion (Koskinen et al., 2008; Chen et al., 2011; Fukumoto and Sugimura, 2001). Accordingly, various devices incorporate haptic feedback and are used in system operations and entertainment applications, such as games. Multiple methods exist for presenting haptic feedback, including the reproduction of vibrations that occur when a hand or tool touches an object, electrical stimulation to activate nerves, and skin deformation.

Some tactile presentation methods focus on reproducing vibrations, such as pen-type interfaces studied by Cho et al. (2016). These interfaces enhance realism by presenting vibrations that correspond to the frequency at which the pen, serving as a medium, rubs against a surface like paper. Regarding tactile presentation methods involving skin deformation, examples include pin array devices, such as those of Ota et al. (2021), which reproduce shapes to convey tactile sensations, and research by Schorr and Okamura (2017), where shapes are presented by attaching a device that generates shear deformation.

With respect to tactile presentation methods using electrical stimuli, Yem et al. (2016) demonstrated that one can perceive hardness and stickiness by applying electrical stimuli to the fingertips. Additionally, studies by Yem et al. (2018) and Mizuhara et al. (2019) have shown that combining electrical stimuli with mechanical stimuli can increase the resolution of tactile sensations. It has also been suggested that reproducing vibrations and the shape of the object being touched is effective in presenting a highly realistic tactile sensation. Although electrical stimulation is currently limited because it directly stimulates the organs and nerves involved in touch, the level of realism and variety of stimuli that can be presented is expected to increase substantially in the future.

However, pin array devices are typically large, pen-type devices are needed to produce reproduced vibration stimuli, and dedicated wearable equipment—such as fingertip shear deformation devices or electrodes—must be attached. Consequently, it is challenging to integrate these methods into small controllers or devices that can be used in various environments, such as touch panel devices with vibration feedback. In principle, electrical stimulation can directly activate the nerves and tissues involved in tactile sensation, making it the least restrictive means of providing tactile feedback. However, it remains difficult to separate pain from tactile sensation or to selectively stimulate only specific receptors with current electrical stimulation techniques.

Methods based on reproducing vibrations require prior recording of contact vibrations and the use of specialized equipment. Vibro-tactile feedback is one of the most widely used tactile presentation methods because it often employs relatively inexpensive piezoelectric devices, voice coil motors, or DC motors, all of which are easy to implement. Our proposed method extends vibro-tactile feedback by leveraging onomatopoeia. This approach makes it possible to implement a system capable of presenting complex haptic feedback using current hardware.

2.3 Pseudo-haptics

Pseudo-haptics is a technique that exploits visual information to create a tactile illusion. In pseudo-haptics, modifying the background or pointer movement can induce the illusion of resistance and tactile sensation. Lécuyer (2009) reported that the hardness of virtually displayed springs and weights could be perceived as though they were real.

SoftAR, proposed by Punpongsanon et al., demonstrated that the perception of an object’s softness could be controlled by altering the object’s shape deformation and the color of the user’s hand superimposed in an AR environment (Punpongsanon et al., 2015). Additionally, Hachisu et al. (2011) proposed a method to enhance pseudo-haptic presentation and the perceived stiffness of objects by combining pseudo-haptics with vibrotactile feedback. Consequently, there is a growing number of proposals that integrate pseudo-haptic presentation with other sensory modalities.

One challenge in pseudo-haptics is the discrepancy between visual movements and actual bodily movements. Additionally, this method is difficult to apply to small-scale movements.

This study aims to present multiple sensory experiences by combining onomatopoeic presentation with tactile feedback. In this manner, users can experience sensations without any discrepancy between their actual movements and the visual presentation. Moreover, onomatopoeia can be implemented in a straightforward manner and is easier to use than object deformation or color changes through superimposition.

3 Method and experiment application

3.1 Method

The method proposed in this paper for modifying tactile feedback is straightforward. As shown in Figure 1, a smartphone held in the user’s hand provides simple vibrotactile stimulation to the hand and fingertips (see also Figure 2), while onomatopoeic words are displayed on the screen. The goal is to alter the tactile sensation felt by the user according to the onomatopoeia presented. Text display and vibration presentation are executed simultaneously in the program, ensuring virtually no delay. In this study, we used a physical switch to examine whether the user’s impression of the tactile stimulus changes based on the meaning or sound symbolism of the displayed onomatopoeia.

Figure 1

Figure 1. Application.

Figure 2

Figure 2. Operation and screen transitions of the application and the onomatopoeia word set.

3.2 Experiment application

To evaluate the effectiveness of this method, we developed a smartphone application that displays onomatopoeia on the screen simultaneously with haptic feedback. We used an Apple Inc. iPhone 6 s Plus with a 5.5-inch screen running iOS 13.1.1 as the experimental device. Figure 2 shows the application’s screen and how it operates when the user taps the on-screen button. The application presents a single button on the smartphone screen. When tapped, the button appears to be pressed down, as shown on the right side of Figure 2. At the same time, onomatopoeia is displayed at the top-left portion of the screen with a speech balloon, accompanied by vibration feedback. The button image changes from an unpressed to a pressed state upon detecting a tap. As soon as the tap was recognized, the onomatopoeic word was displayed and a vibration was delivered. The onomatopoeic word remained on-screen until the button was returned to its normal state by tapping it again. We intentionally omitted animation effects such as fade-in and fade-out to focus solely on the impact of onomatopoeic presentation, without introducing pseudo-haptic effects. The vibration stimulus was delivered by playing the iOS system sound ID 1003 (e.g., used for messaging notifications) for about 0.1 s in iOS 13.1.

Table 1 and the right side of Figure 2 illustrate the onomatopoeic words and their meanings used in this application. These words were selected through a voting process in which users chose the onomatopoeia they felt best represented the act of pressing an on-screen button. The pool of candidates came from a Japanese onomatopoeia dictionary containing 4,500 entries. One student in the laboratory and the author then refined the selection to words suitable for switches. As a control, the author separately added words related to tactile sensations (e.g., “gunyu,” “nyuru”) that are not typically used for switches. Additionally, we included several onomatopoeic expressions unrelated to buttons or switches, as well as a condition with no onomatopoeia at all. Within this set, “kachi” and “pochi” are commonly used for buttons, “bachi,” “tan,” and “pi” are sometimes used, whereas “nyuru” and “gunyu” are almost never used in button contexts.

Table 1

Table 1. Meaning and pronunciation of the onomatopoeia set.

4 Experiment

Onomatopoeia and vibrotactile feedback were presented simultaneously to investigate whether the impression of tactile feedback could be altered.

As shown on the left side of Figure 3, we evaluated the impression of vibrotactile feedback using keyboard switches. We employed six physical switches, consisting of three types of tactile feedback (linear, tactile, and clicky) and two different weights (heavy and light). These types and weights are listed in Table 2. Figure 4 shows the measured force curves of the six switches used in the experiment. We obtained these curves using load cells (USL06-H5-50N from Tec Gihan Co., Ltd.) and linear actuators. The measured values deviated from the maximum resilience force specified by the manufacturer for each switch. Such discrepancies may stem from measurement errors, including distortion caused by insufficient rigidity in the push-in component or inaccuracies from relying solely on the Z-axis. Nevertheless, these deviations do not undermine the validity of the experiment, as each switch still provides sufficiently distinct tactile stimuli across the different types. The repulsive force for each switch, as indicated by product specifications, is also inherently different.

Figure 3

Figure 3. (A): Key switch for comparison and (B): experiment posture.

Table 2

Table 2. Weight and tactile characteristics of three switch types.

Figure 4

Figure 4. Key Switches Force Curve. (a) Linear, (b) Tactile, (c) Clicky.

In the experiment, after participants tapped the on-screen button accompanied by onomatopoeia, they were asked to select the physical switch whose tactile sensation most closely matched the one they perceived. If the tactile impression changes depending on the onomatopoeia, participants’ switch selections should also change accordingly.

Throughout the experiment, participants wore headphones playing white noise, so they could not hear any switch sounds or environmental noise. This arrangement minimized the influence of auditory cues on identifying the type of key switch. The physical order of the switches was randomized for each trial.

Before choosing the switch that most resembled the application’s tactile feedback, participants were instructed to press all six switches. They were also asked to fix their wrists, as shown on the right side of Figure 3, and to press each switch from the same position. Keeping the pressing posture consistent ensured that the tactile sensation remained the same each time a specific switch was pressed.

As an additional experiment, we examined the tactile impressions of each switch. After pressing all six switches, participants selected from the word set in Table 1 the onomatopoeia that best represented each switch’s tactile characteristics.

The experimental procedure consisted of five steps, as outlined in Figure 5. The participants included seven men, two women, and one other, with a mean age of 22.9 years. The experiment was conducted after confirming that none of the participants had health issues or tactile impairments in their hands.

1. Sit in a chair and put on noise-canceling headphones.

2. Hold the smartphone in the non-dominant hand, and tap the on-screen button five times with the index finger of the dominant hand. (Each tap triggers both vibration and one of the onomatopoeic words from the word set).

3. Press all six physical switches, each with a different tactile characteristic, and select the one whose sensation most closely matches that of the application.

4. Complete a 7-point Likert scale questionnaire on the statement: “When you tap the button on the screen, did it feel as if you were actually pressing a physical button?”

5. Repeat Steps (2) through (4) three times for each of the seven onomatopoeia patterns and for the no-onomatopoeia condition.

Figure 5

Figure 5. Experiment procedure [(A) Step 2, (B) Step 3].

This experiment was conducted with the approval of the University of Electro-Communications Ethics Review Committee (No. 19055).

5 Result and discussion

5.1 Switch selection analysis (chi-square test)

Table 3 presents the frequency of each selected key switch for every onomatopoeia displayed. Because these questionnaire responses are categorical, we first conducted a chi-square test to assess overall goodness of fit. With 35 degrees of freedom, the chi-square test yielded a p-value of 0.00008 and a Cramer’s V of 0.251, indicating a moderate effect size. This result confirms a significant difference in the distribution of selected switches when onomatopoeia is displayed versus when it is not.

Table 3

Table 3. Experimental questionnaire responses.

To further explore differences among the displayed onomatopoeia, we performed seven additional chi-square tests (each with 5 degrees of freedom), using the non-displayed condition as the theoretical reference. We applied the Benjamini–Hochberg method for multiple-comparison correction. Table 4 shows the results of these chi-square tests, along with the significant differences at q-values of 0.05 and 0.1.

Table 4

Table 4. Results comparing each onomatopoeia and no labeling with a chi-square test with 5 degrees of freedom.

At q = 0.05, “kachi” and “bachi” demonstrated significant differences compared with the no-onomatopoeia condition. At q = 0.1, there was a significant trend for “pochi,” “nyuru,” and “tan.” The effect sizes for these comparisons exceeded 0.3, indicating a moderate level of practical significance. Specifically, participants most frequently chose the 45 g clicky switch for “kachi,” the 72 g tactile switch for both “bachi” and “pochi,” the 35 g linear switch for “nyuru,” and the 28 g tactile switch for “tan.” These findings underscore that displaying onomatopoeia alongside simple tactile stimuli can indeed alter users’ tactile impressions and that the specific impression varies according to the onomatopoeia presented.

5.1.1 Cell residual analysis

To identify which specific cell (Onomatopoeia $\times$ Switch) contributed most to the overall chi-square result, we conducted a cell residual analysis using adjusted residuals. We first calculated the observed frequencies $O_{ij}$ and the expected frequencies $E_{ij}$ for each cell, then derived the adjusted residuals ${\mathrm{A}\mathrm{R}}_{ij}$ as described by Haberman (1973):

${\mathrm{A}\mathrm{R}}_{ij}=\frac{O_{ij}-E_{ij}}{\sqrt{E_{ij}\left(1-{\mathrm{R}\mathrm{o}\mathrm{w}\mathrm{S}\mathrm{u}\mathrm{m}}_{i}N\right)\left(1-{\mathrm{C}\mathrm{o}\mathrm{l}\mathrm{S}\mathrm{u}\mathrm{m}}_{j}N\right)}}$

Table 5 shows only the cells where $|{\mathrm{A}\mathrm{R}}_{ij}|\ge 2.0$, suggesting a significant deviation from expected frequencies at approximately the 5% level. For instance, “nyuru” was chosen notably more often with the 60 g and 35 g linear switches and notably less with the 72 g and 28 g tactile switches. Similarly, “kachi” was frequently paired with the 45 g clicky switch, whereas the 60 g linear switch was rarely selected for “kachi.” These patterns indicate that onomatopoeic nuances—such as the crispness implied by “kachi” or the smoothness implied by “nyuru”—may prime participants’ perception of mechanical feedback.

Table 5

Table 5. Results of cell residual analysis with $|\mathrm{A}\mathrm{R}|\ge 2.0$.

Beyond “nyuru” and “kachi,” “gunyu” was associated with heavier linear switches, while “bachi” and “pochi” tended to avoid certain linear switches based on perceived weight or click characteristics. Overall, these results imply that an appropriate onomatopoeic label can significantly shape users’ haptic impressions even when the underlying vibration or mechanical properties are unchanged. In other words, simple vibrotactile feedback combined with carefully selected onomatopoeias could enhance perceived realism or guide users toward a specific tactile interpretation, thereby reducing the need for hardware modifications.

5.2 Realistic sensation ratings (likert scale)

Figure 6 illustrates the distribution of responses to a 7-point Likert scale questionnaire asking participants whether they felt as if they were “actually pressing a switch” for each onomatopoeic presentation. Table 6 summarizes the mean, median, and standard deviation for each onomatopoeia.

Figure 6

Figure 6. Tukey’s test results for the questionnaire “When you tap a button on the screen, did you feel like you actually pressed the button?” with and without each onomatopoeia.

Table 6

Table 6. Result of average, median and SD in questionnaire about push feel.

To compare each onomatopoeic condition with the no-onomatopoeia baseline, we conducted multiple comparisons using Tukey’s test. Table 7 shows these results. For “kachi,” “pochi,” “bachi,” and “tan,” p-values were below 0.01, and their 95.

Table 7

Table 7. Tukey’s test statistics for the questionnaire “When you tap a button on the screen, did you feel like you actually pressed the button?” with and without each onomatopoeia.

Moreover, the mean and median scores for these four words ranged from 0.7 to 1.03 (median = 1), whereas the mean and median for the non-displayed case were −0.3 and 0, respectively. Thus, presenting these onomatopoeias significantly enhanced users’ perceived tactile realism. By contrast, “nyuru,” an onomatopoeic term not commonly associated with switches, had a negative mean and median relative to the no-onomatopoeia condition, suggesting that a mismatched onomatopoeic image may reduce the sense of realism.

5.3 Tactile representation of each switch

Table 8 shows how participants selected the onomatopoeia that best represented each switch’s tactile impression after pressing it. If the onomatopoeic tendencies evoked by the switch’s tactile sensations aligned perfectly with the questionnaire results, we would expect a direct correspondence between the switch and its chosen onomatopoeia.

Table 8

Table 8. Result of questionnaire, which onomatopoeia closest to the tactile representation of each keyswitch

However, the data reveal that the actual matches between each switch and the chosen onomatopoeia were minimal. This finding implies that the user’s impression is formed not merely by the tactile stimulus and vibration feedback alone, but rather by the interaction of these factors with the specific onomatopoeia presented.

5.4 Sound symbolism

We further examined the effects of onomatopoeic sound symbolism. Hamano et al. have reported that, in Japanese, specific initial consonants, vowels, and final consonants can evoke distinct semantic or sensory impressions. Table 9 summarizes the onomatopoeic words used in this experiment, along with their presumed sound-symbolic attributes.

Table 9

Table 9. Sound symbolism of the onomatopoeia used in the experiment.

Notably, several of these onomatopoeias imply a “linear” sensation (e.g., “bachi,” “pochi,” “pi”) based on their initial consonants or vowels, whereas others suggest hardness (e.g., “kachi,” “gunyu”) or other qualities. We also considered the concept of weight, as certain consonants can imply heavier or lighter sensations. Table 10 compares the symbolic weight suggested by each onomatopoeia to the actual weight of the selected switches.

Table 10

Table 10. Comparison of sound symbolism and questionnaire responses on weight.

For example, “bachi” and “pochi” share equal frequencies in some cases, while “nyuru” lacks a clear weight-related image. Nonetheless, for most words, the observed switch selections align reasonably well with their sound-symbolic tendencies. These findings imply that weight perception can be modulated by sound-symbolic cues. Moreover, in many instances, participants’ tactile impressions appear to rely on the semantic meaning of the presented onomatopoeia, rather than on purely mechanical feedback.

5.5 Limitations

5.5.1 Screen space constraints

The proposed method effectively enhances tactile realism on a smartphone but occupies screen space. This may reduce efficiency in applications where the primary goal of tactile feedback is functional rather than immersive.

5.5.2 Field of view dependence

Because this approach relies on visually presented onomatopoeia, it is challenging to apply to tactile stimuli beyond the user’s field of view, such as behind the user or in otherwise unseen areas.

5.5.3 Language dependency

This study was conducted exclusively in Japanese, leaving its applicability to other languages unverified. However, previous studies (Lo et al., 2017; Nalbantoğlu et al., 2024) indicate that sound symbolism could influence tactile perception in other languages as well. Further cross-linguistic research is needed to validate the generalizability of our method.

5.5.4 Limitations in stimulus conditions

Our experiment used simple vibration feedback, basic button images, and single onomatopoeic words to investigate their effects. While we found that semantic and phonetic alignment between the tactile stimulus and onomatopoeia can enhance perceived realism, mismatched cases decreased it. This suggests that even more advanced haptic feedback—such as electrical stimulation or complex vibration patterns—could benefit from carefully chosen onomatopoeias. Future research should explore how this method integrates with more sophisticated haptic interactions.

6 Conclusion and future works

In this paper, we propose a method to modify the impression of haptic feedback by displaying onomatopoeia alongside simple haptic stimuli. We developed a smartphone application that provides both simple vibration feedback and onomatopoeic words when the user taps an on-screen switch, and we conducted evaluation experiments to validate the proposed approach.

Six physical switches with different tactile sensations were used to evaluate participants’ perceptions of the smartphone application. Compared with the no-onomatopoeia condition, two onomatopoeic words produced statistically significant differences, and three showed a trend toward significance. For these words, participants selected different switches, indicating that the impression of the tactile stimulus could be altered by presenting onomatopoeia simultaneously with the vibration. A questionnaire also assessed whether users felt as if they were pressing a real button on the screen. Although no strong trend emerged, the results suggest that displaying unrelated onomatopoeic words may reduce perceived realism. Overall, these findings indicate that simple tactile stimuli can be effectively enhanced by presenting onomatopoeia at the same time.

Previous research has investigated non-verbal visual presentation methods such as pseudo-haptics, but the present study demonstrates that it is also possible to influence haptic perception using verbal cues in the form of onomatopoeia. Because displaying textual information on the screen is straightforward, this method can be readily integrated into current devices such as smartphones.

One limitation of this study is that, although it is useful for providing more realistic tactile stimulation, it may reduce efficiency in applications focused on functionality or productivity, as it requires a portion of the screen. Furthermore, this approach is only applicable to areas within the user’s field of view and is difficult to implement for sensations outside that range, such as on the user’s back.

Future work will involve investigating the method’s applicability to other languages and verifying its effectiveness when combined with more advanced forms of tactile presentation. In this study, only native Japanese speakers were recruited, so the method’s effects in other languages remain unverified. Previous research suggests that English and other languages may exhibit sound-symbolic phenomena similar to those in Japanese, indicating potential cross-linguistic utility. Nevertheless, further verification is required. Additionally, our experiments were limited to a smartphone environment, and the method’s effectiveness on other devices has yet to be examined. Given that this approach could be highly beneficial for the increasingly popular XR platforms, we plan to conduct future experiments in AR and VR environments. Moreover, this study focused exclusively on button-based interactions, leaving the applicability to other objects and non-tap gestures (such as swiping) unexamined. Future research should investigate a wider range of objects and interaction methods to develop a more generalized method.

In this study, we successfully altered the perceived sense of touch into five distinct types by simultaneously presenting onomatopoeic words with simple stimuli. As a next step, it will be important to investigate how the meaning and sound symbolism of onomatopoeic words interact with more advanced and realistic tactile feedback.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving humans were approved by the University of Electro-Communications Ethics Review Committee. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

IM: Writing–original draft, Writing–review and editing. SS: Writing–review and editing. KH: Writing–review and editing. TN: Writing–review and editing.

Funding

The author(s) declare that no financial support was received for the research, authorship, and/or publication of this article.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Generative AI statement

The author(s) declare that no Generative AI was used in the creation of this manuscript.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

Armel, K. C., and Ramachandran, V. S. (2003). Projecting sensations to external objects: evidence from skin conductance response. Proc. R. Soc. B Biol. Sci. 270, 1499–1506. doi:10.1098/rspb.2003.2364

PubMed Abstract | CrossRef Full Text | Google Scholar

Association, I. P. (2020). Full ipa chart. Available online at: https://www.internationalphoneticassociation.org/content/full-ipa-chart.

Google Scholar

Chen, H. Y., Park, J., Tan, H. Z., and Dai, S. (2011). Design and evaluation of identifiable key-click signals for mobile devices. IEEE Trans. Haptics 4, 229–241. doi:10.1109/TOH.2011.21

PubMed Abstract | CrossRef Full Text | Google Scholar

Cho, Y., Bianchi, A., Marquardt, N., and Bianchi-Berthouze, N. (2016). “RealPen: providing realism in handwriting tasks on touch surfaces using auditory-tactile feedback,” in Proceedings of the 29th annual symposium on user interface software and technology. doi:10.1145/2984511

CrossRef Full Text | Google Scholar

Choi, H., Oh, J., Chang, M., and Kim, G. J. (2018). “Effect of accompanying onomatopoeia to interaction sound for altering user perception in virtual reality,” in Proceedings of the ACM symposium on virtual reality software and technology (Tokyo, Japan: VRST (Association for Computing Machinery). doi:10.1145/3281505.3281614

CrossRef Full Text | Google Scholar

Fukumoto, M., and Sugimura, T. (2001). “Active click: tactile feedback for touch panels,” in CHI ’01 extended abstracts on human factors in computing systems (New York, NY, USA), 121–122. doi:10.1145/634067.634141

CrossRef Full Text | Google Scholar

Fukusato, T., and Morishima, S. (2014). “Automatic depiction of onomatopoeia in animation considering physical phenomena,” in Proceedings - motion in games 2014, MIG 2014 (Playa Vista, California: Association for Computing Machinery, Inc), 161–169. doi:10.1145/2668064.2668096

CrossRef Full Text | Google Scholar

Haberman, S. J. (1973). The analysis of residuals in cross-classified tables. Biometrics 29, 205. doi:10.2307/2529686

CrossRef Full Text | Google Scholar

Hachisu, T., Cirio, G., Marchal, M., Lécuyer, A., and Kajimoto, H. (2011). “Pseudo-haptic feedback augmented with visual and tactile vibrations,” in Isvri 2011 - IEEE international symposium on virtual reality innovations 2011 (Singapore: Proceedings), 327–328. doi:10.1109/ISVRI.2011.5759662

CrossRef Full Text | Google Scholar

Ivanova, G. (2006). “Sound-symbolic approach to Japanese mimetic words,” in Toronto working papers in linguistics 26.

Google Scholar

Koskinen, E., Kaaresoja, T., and Laitinen, P. (2008). “Feel-good touch: finding the most pleasant tactile feedback for a mobile touch screen button,” in ICMI’08: proceedings of the 10th international conference on multimodal interfaces, 297–304. doi:10.1145/1452392.1452453

CrossRef Full Text | Google Scholar

Lécuyer, A. (2009). Simulating haptic feedback using vision: a survey of research and applications of pseudo-haptic. feedback 18, 39–53. doi:10.1162/pres.18.1.39

CrossRef Full Text | Google Scholar

Lecuyer, A., Coquillart, S., Kheddar, A., Richard, P., and Coiffet, P. (2000). “Pseudo-haptic feedback: can isometric input devices simulate force feedback,” in Proceedings - Virtual Reality Annual International Symposium, 83–90. doi:10.1109/vr.2000.840369

CrossRef Full Text | Google Scholar

Lo, L.-Y., Luk, H. M., and Thompson, N. (2017). An investigation of sound-symbolism in the context of tactile feeling. J. General Psychol. 144, 157–167. doi:10.1080/00221309.2016.1276047

PubMed Abstract | CrossRef Full Text | Google Scholar

Mizuhara, R., Takahashi, A., and Kajimoto, H. (2019). “Combination of mechanical and electrical stimulation for an intense and realistic tactile sensation,” in Proceedings - VRCAI 2019: 17th ACM SIGGRAPH International Conference on Virtual-Reality Continuum and its Applications in Industry, 1–5. doi:10.1145/3359997.3365714

CrossRef Full Text | Google Scholar

Nalbantoğlu, H., Hazır, B. M., and Dövencioğlu, D. N. (2024). Selectively manipulating softness perception of materials through sound symbolism. Front. Psychol. 14, 1323873. doi:10.3389/fpsyg.2023.1323873

PubMed Abstract | CrossRef Full Text | Google Scholar

Oh, J., and Kim, G. J. (2018). “Efect of accompanying onomatopoeia with sound feedback toward presence and user experience in virtual reality,” in Proceedings of the ACM symposium on virtual reality software and technology (Tokyo, Japan: VRST (Association for Computing Machinery). doi:10.1145/3281505.3283401

CrossRef Full Text | Google Scholar

Ota, Y., Ujitoko, Y., Sakurai, S., Nojima, T., and Hirota, K. (2021). Inside touch: presentation of tactile feeling inside virtual object using finger-mounted pin-array display. IEEE Access 9, 75150–75157. doi:10.1109/ACCESS.2021.3082100

CrossRef Full Text | Google Scholar

Punpongsanon, P., Iwai, D., and Sato, K. (2015). SoftAR: visually manipulating haptic softness perception in spatial augmented reality. IEEE Trans. Vis. Comput. Graph. 21, 1279–1288. doi:10.1109/TVCG.2015.2459792

PubMed Abstract | CrossRef Full Text | Google Scholar

Sakamoto, M., and Watanabe, J. (2018). Bouba/kiki in touch: associations between tactile perceptual qualities and Japanese phonemes. Front. Psychol. 9, 295. doi:10.3389/fpsyg.2018.00295

PubMed Abstract | CrossRef Full Text | Google Scholar

Schorr, S. B., and Okamura, A. M. (2017). Fingertip tactile devices for virtual object manipulation and exploration. Proc. 2017 CHI Conf. Hum. Factors Comput. Syst., 3115–3119. doi:10.1145/3025453.3025744

CrossRef Full Text | Google Scholar

Wang, F., Nagano, H., Kashino, K., and Igarashi, T. (2017). Visualizing video sounds with sound word animation to enrich user experience. IEEE Trans. Multimedia 19, 418–429. doi:10.1109/TMM.2016.2613641

CrossRef Full Text | Google Scholar

Yem, V., Okazaki, R., and Kajimoto, H. (2016). FinGAR: combination of electrical and mechanical stimulation for high-fidelity tactile presentation. ACM SIGGRAPH 2016 Emerg. Technol. SIGGRAPH 2016. doi:10.1145/2929464.2929474

CrossRef Full Text | Google Scholar

Yem, V., Vu, K., Kon, Y., and Kajimoto, H. (2018). “Effect of electrical stimulation haptic feedback on perceptions of softness-hardness and stickiness while touching a virtual object,” in 25th IEEE Conference on Virtual Reality and 3D User Interfaces, VR 2018 - Proceedings, 89–96. doi:10.1109/VR.2018.8446403

CrossRef Full Text | Google Scholar