The influence of a color themed HMI on trust and take-over performance in automated vehicles

Introduction: SAE Level 3 is known as conditional driving automation. As long as certain conditions are met, there is no need to supervise the technology and the driver can engage in non-driving related tasks (NDRTs). However, a human driver must be present and alert to take over when the automation is facing its system limits. When such an emergency takes place, the automation uses the human machine interface (HMI) to send a take-over request (TOR) to the driver.

Methods: We investigated the influence of a color themed HMI on the trust and take-over performance in automated vehicles. Using a driving simulator, we tested 45 participants divided in three groups with a baseline auditory HMI and two advanced color themed HMIs consisting of a display and ambient lighting with the colors red and blue. Trust in automation was assessed using questionnaires while take-over performance was assessed through response time and success rate.

Results: Compared to the baseline HMI, the color themed HMI is more trustworthy, and participants understood their driving tasks better. Results show that the color themed HMI is perceived as more pleasant compared to the baseline HMI and leads to shorter reaction times. Red ambient lighting is seen as more urging than blue, but HMI color did not significantly affect the general HMI perception and TOR performance.

Discussion: Further research can explore the use of color and other modalities to express varying urgency levels and validate findings in complex on road driving conditions.

1. Introduction

Development of Automated Vehicles (AV) has been the main focus of car manufacturers as well as tech companies. It is envisioned that AV could replace human driving, which could reduce traffic accidents and thus increase road safety. The implementation of AV could also benefit the “driver” by increasing comfort and enabling use of the driving time for work or leisure (Kyriakidis et al., 2015). The adoption of AV in day-to-day life is not only dependent on the performance of the automation. The ultimate goal for AV is to be broadly adopted in society. For this to happen, AV have to be trusted and accepted by the public.

The main focus of this study will be on SAE Level 3 automation, which is defined as “The sustained and operational design domain (ODD) specific performance by an automated driving system (ADS) of the entire dynamic driving task (DDT) with the expectation that the DDT fallback-ready user is receptive to ADS-issued requests to intervene, as well as to DDT performance-relevant system failures in other vehicle systems, and will respond appropriately” (SAE International, 2021). This means that when the automation is activated, the driver can take the eyes off the road and perform non-driving related tasks (NDRTs). However, this requires high levels of trust defined as “the attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vulnerability” (Lee and See, 2004). When the automation reaches its limits, the automation requests the driver to take over control of the vehicle. This request initiated by the vehicle is called a take-over request (TOR; Lu et al., 2016). The TOR is conveyed by a Human Machine Interface (HMI) which relays information from a machine or system to a person (Stouffer et al., 2011).

With the increasing advancements and availability of automated vehicles, the number of studies related to HMI in automated vehicles has been rising. A review by Ayoub et al. (2019) showed that take-over, acceptance and trust, interaction with other road users, and design methodology were studied extensively in the last decade. Several studies demonstrated effects of TOR HMI modality and information on performance (Politis et al., 2015; Petermeijer et al., 2017; Zhang et al., 2019; Naujoks et al., 2021). Simple sounds and lights and ambient lighting can alert the driver and are often sufficient to elicit safe take-overs. Zhang et al. (2019) and Kim et al. (2021) found that TOR presented only through the visual modality yielded the longest reaction time, whereas adding the auditory modality reduced the reaction time. A range of studies on various HMI elements demonstrated that advanced HMI informing users of the automation operation, and desired driver actions through images, written text or spoken text, outperform basic HMI in terms of trust as well as performance (Kim et al., 2021). Trust was enhanced adding speech based maneuver information (Forster et al., 2017) and vibrotactile spatial information related to traffic objects (Sonoda and Wada, 2017). Trust in automation and take-over performance were enhanced showing the TOR on a display (Löcken et al., 2020) and situation awareness was increased through a LED bar indicating the direction of conflicts and the vehicle trajectory (Yang et al., 2018). Ambient lighting provides information without demanding much cognitive workload and does not distract users' from their primary tasks (Matthews et al., 2004). It has a positive influence on, amongst others, perceived safety and interior functionality (Caberletti et al., 2010). Mirnig et al. (2023) showed that ambient light interfaces can improve mode awareness and reduce unsafe NDRT engagement in Level 3 automation. Several studies showed effects of various design aspect of ambient lighting in automated vehicles such as ambient lighting position (Feierle et al., 2020) or patterns of ambient lighting (Borojeni et al., 2016). Some literature addresses symbol color (Campbell et al., 2018) and color contrast with background (Naujoks et al., 2019). However, to the best of our knowledge, the effect of the ambient lighting color has not been studied for HMI in automated driving.

In general, a red color, which has a large wavelength, stimulates physical performance (Hill and Barton, 2005). Red can convey the urgency and safety critical aspect of the TOR, while colors as blue would suggest a lower urgency (Friedrich and Vollrath, 2022). A higher urgency can promote a rapid TOR response (Politis et al., 2015) but may also induce high stress levels hampering trust, acceptance and take-over performance. This indicates that red would be beneficial for the reaction time. On the other hand, red undermines cognitive performance and blue is perceived as more calming and trustworthy (Elliot et al., 2007). This indicates that blue colors, which have a smaller wavelength, let the driver be more comfortable and relaxed, which would lead to a more calm and safe take-over.

As outlined above, ample studies report effects of HMI on take-over reaction time but few report the effect on perceived safety and trust in automation. In most cases the color of the HMI display and ambient lighting is assigned rather than researched. Hence this paper investigates the following main hypotheses:

• A color themed informative HMI with ambient lighting will increase the trust in AV (Hypothesis 1a) and improve the take-over reaction time and quality (Hypothesis 1b).

• A red themed HMI will result in a higher urgency and therefore a shorter reaction time as compared to a blue-themed HMI (Hypothesis 2).

In order to validate these hypotheses, a color themed HMI including ambient lighting and an informative display was designed. The influence of the presence and color of the HMI was evaluated considering trust and take-over performance in a driving simulator experiment.

2. Methods

A color themed HMI was designed including a display at the midconsole, ambient lighting and sound whereas a basic HMI, which only uses sound, was used as baseline. The choices that led to the design of the HMI and the experiment are given below.

2.1. Participants

Forty-five participants were recruited via various channels to take part in this experiment, of which 14 were female and 31 were male. All had at least 2 years of driving experience and were between 18 and 28 years old. The participants were evenly divided in three test groups based on Gender, Driving Experience and results from the pre-trust in automation questionnaire. The group that was tested with a basic HMI that is not color themed was called “Baseline”. The group that was tested with the red themed HMI was called “Red”. The last group that was tested with a blue themed HMI was called “Blue”.

2.2. Apparatus

The experiment took place at the intelligent vehicles group at the Delft University of Technology, and was conducted using the Delft Advanced Vehicle Simulator (DAVSi) (Figure 1; Khusro et al., 2020). The mock-up vehicle placed in the simulator, a 2013 Toyota Yaris, includes a driver and passenger seat. The instrument cluster is functioning and shows the speed and the RPM of the vehicle. The projection screen that shows the driving environment is placed in front of the mock-up vehicle and shows a 200° field of view. IPG CarMaker 8.0.1 was used to generate the environment and traffic of the scenario. IPG movie was used to display the graphics on the screen. The color themed HMIs used a 10.9-inch display in the center console and ambient lighting to show the TOR request (Figures 2A, B). A smartphone on the left side of the steering wheel was used for the NDRT (Figures 2A, B). The ambient lighting consisted of multiple led strips controlled by an Arduino receiving commands from DAVSi via a series port. The IPG driver model was used to realize automated driving. During take-over drivers were expected to control speed by braking.

FIGURE 1

Figure 1. DAVSi driving simulator.

FIGURE 2

Figure 2. Two color-themed HMIs with ambient lighting, HMI display at the midconsole, and NDRT left of the steer. (A) Red themed HMI. (B) Blue themed HMI.

2.3. HMIs

Three different HMIs were designed and used in the experiment. The HMI of the baseline group consisted of only an auditory interface, in which no ambient lighting or display was shown during a TOR. The HMI of the color themed groups consisted of the same auditory and a visual interface as shown in Table 1 and Figures 2A, B.

TABLE 1

Table 1. Overview of the different HMIs.

2.3.1. Auditory interface

There were two transition-related sounds in the vehicle (see Supplementary material). When the AV issued a TOR, a short sound of 1.5 s was repeated until the driver intervened. This sound is called “TOR-sound.” As the driver was engaging in an NDRT and was visually distracted, the TOR sound was expected to be the first TOR sign that would be noticed by the participant. When the AV resumed automated driving after the TOR, a short sound was played for 1.5 s. This sound is called “automation-start-sound.” After hearing this sound the driver was expected to continue with the NDRT as the automation took back control. The two sounds were designed for informing AV activation and requesting take-over. Alarm levels, usefulness, and appropriateness with situations of the two sounds were validated in our previous study (Kim et al., 2022). Results have shown that the two sounds were appropriately designed for their driving context.

2.3.2. Visual interface

2.3.2.1. Color

The three different HMIs could be distinguished by the presence and color of a visual HMI and ambient lighting. The basic HMI used sound only. The other two HMIs, used the colors red and blue in both the HMI display and as ambient lighting.

2.3.2.2. Display

For the color themed HMIs, the visual TOR was shown on the mid-console display which was placed just below the window. A drawing of a brake pedal together with the text “Brake” indicated that the participant had to take over control by braking.

2.3.2.3. Lighting

For the color themed HMIs, ambient lighting was added on several points in the interior, below eyesight, so that the driver could notice the lights without getting distracted or blinded. The red themed HMI contained red ambient lighting and the blue themed HMI contained blue ambient lighting (Figures 2A, B).

Both the display and the ambient lighting were activated at the start of each TOR and remained active until the automation took back control. When seeing the lighting, the driver was expected to check the HMI display and take over control by braking.

2.4. Scenario

The driving scenario had eight separate situations which took place on a highway. Four situations were handled by the automation, whereas four others included a TOR. The TOR were designed to create a new experience in each TOR requiring an original action rather than triggering a preprogrammed response. The drive was initiated with SAE Level 3 automation, starting from standstill and accelerating to the speed limit which is 100 km/h in most parts and 130 km/h in others (Figure 3). The 8 situations can be subdivided into two categories, system maneuvers and TORs. The system maneuvers were handled by the automation. The TORs were to be handled by the participant, where in all TOR braking was the appropriate intervention. However, when participants did not brake in time, the automation intervened if the time to collision (TTC) became < 2 s, and prevented collisions. To ensure that the participants would not be confused by the situations, a distinction between system maneuvers and TORs has been made. Take-over situations only included braking (speed control) while system maneuvers included steering. During the whole scenario, there was foggy weather with a visibility of 300 m. This created a somewhat more critical situation to test trust in automation. However, the 300 m always enabled timely visual detection of other vehicles well before the TOR.

FIGURE 3

Figure 3. Road map of the scenario.

The first phase of the scenario was designed to make the participants acquainted with the simulator, the automation and the TOR. This phase consisted of two system maneuvers. Following this phase, the first and second TOR took place. After these TORs, the third system maneuver, the third TOR, the fourth system maneuver, and the fourth TOR took place, respectively (Figure 3). In the next sections, the system maneuvers and TORs will be explained in detail. To minimize the influence of one situation on the other, there was a minimum of 3 min in between every traffic situation. Whenever, the AV was in automated state, the participant had to engage in an NDRT, which was playing Angry bird, a visual-motor task without sound, on a smart phone mounted left of the steering wheel (Figure 2). The NDRT was self-paced and interruptible, so participants could pause it whenever necessary to check the surrounding (Naujoks et al., 2017).

2.4.1. System maneuvers

There were three types of system maneuvers.

• The ego vehicle overtakes a vehicle driving slower on the right lane in System maneuver 1 and 4 (Figure 4A).

• The ego vehicle overtakes a vehicle suddenly merging in front from the insertion lane in System maneuver 2 (Figure 4B).

• Steering of the ego vehicle to make it follow the curvature of the road in System Manouvre 3.

FIGURE 4

Figure 4. System maneuvers and take-over requests. (A) System maneuver 1 and 4 [ego car (blue) is overtaking a slower moving traffic car (orange)]. (B) System maneuver 2 [ego car (blue) is changing lanes to create space for a merging traffic car (orange)]. (C) TOR 1 [Traffic car (red) brakes hard, ego car (blue) cannot change to the left lane due to the traffic cars (orange), so a TOR is initiated. The participant is expected to brake.]. (D) TOR 2 [Traffic car (orange) cuts off the red traffic car, the traffic car (red) brakes which leads to a TOR for the driver which is represented by the ego car (blue)]. (E) TOR 3 [Traffic jam on a one-lane road caused by stationary traffic cars (orange), traffic car (red) brakes which results in a TOR for the ego car (blue)]. (F) TOR 4 [Traffic jam on a two-lane road caused by stationary traffic cars (traffic), traffic car (red) brakes which results in a TOR for the ego car (blue)].

2.4.2. TORs

In total there were four TORs where a sudden change of events caused the AV to request a TOR.

• The first TOR (TOR 1 in Figure 3) took place on a two-lane road. The ego car had a constant speed of 100 km/h for 2 km before the automation suddenly noticed that the traffic car in front of the ego car was braking. The distance to the traffic car was around 100 m and even though there was a left lane, an overtake was no option because of multiple cars passing on the left lane with a speed of 110 km/h (Figure 4C). Therefore, the ego-car driver had to take over control and achieve hard braking. After braking, the traffic car in front of the ego car had a speed of 40 km/h which resulted in the ego car eventually overtaking the traffic car.

• The second TOR (TOR 2 in Figure 3) took place on a piece of road where two lanes merged into one (Figure 4D). The ego car was driving on the right lane and there was a traffic car driving in front of it. As the end of the left lane approached, a traffic car on the left lane tried to overtake the traffic car in front of the ego car (Figure 4D). The traffic car was forced to brake suddenly to a speed of 50 km/h and the automation requested a TOR. The distance to the traffic car in front was around 80 m, and an overtake was no option because of the merging left lane. The ego-car driver had to achieve hard braking to prevent an accident. After the TOR, the traffic car accelerated to a speed of 110 km/h.

• The third TOR (TOR 3 in Figure 3) took place on a one-lane road. The ego car encountered a traffic jam (Figure 4E). The distance to the traffic car in front was around 150 m, and an overtake was not possible as there was no left lane. The only safe outcome was achieved by hard braking. After the TOR, the traffic cars accelerated to a speed of 115 km/h, leaving the ego car behind.

• The fourth and last TOR (TOR 4 in Figure 3) took place on a two-lane road. Just like the previous TOR, the ego car encountered a traffic jam (Figure 4F). The distance to the traffic car in front was around 150 m. An overtake was not possible as there was a traffic jam on the left lane too. The ego-car driver had to achieve hard-braking. After the TOR, the traffic cars accelerated to a speed of 115 km/h, leaving the ego car behind.

2.5. Procedure

Each participant had to read an instruction with information about the steps that would take place before, during and after the simulator drive. After this instruction, the participants were asked to fill in a pre-drive questionnaire, which contained an explanation about SAE Level 3 automation (Table 2). The pre-drive questionnaire was used to test the familiarity and trust of the participant regarding SAE Level 3 automation before taking part in the research, and was used to create balanced participant groups for the 3 HMIs. Filling in the pre-drive questionnaire took about 10 min. After finishing the pre-drive questionnaire the participants took place in the simulator. The participants went through the designed scenario which took around 30 min to finish. During the simulator drive, two questions were asked after each TOR. After finishing the drive, participants were asked to fill in a post-drive questionnaire which took about 15 min to finish (Table 3). The post-drive questionnaire was used to measure the trust in the automation after having experienced the automation. The whole process took around 50–60 min per participant.

TABLE 2

Table 2. Pre-drive questionnaire divided in three groups.

TABLE 3

Table 3. Post-drive questionnaire divided in groups with their corresponding references.

2.6. Measurements

Objective measurements of driver performance were gained during the simulator drive while subjective measurements were obtained through pre-drive and post-drive questionnaires, and questions asked during the drive after each TOR.

2.6.1. Objective measurement

The objective data can be divided into two groups, the take-over time and take-over quality. The take-over time was measured by the reaction time, between the initiation of the TOR and the first brake or gas input from the participant (Gold et al., 2013). The take-over quality was measured by the percentage of successful TORs. A TOR was considered successful if the automation did not intervene by braking (automation intervened if the TTC became < 2 s).

2.6.2. Subjective measurement

The pre-drive questionnaire (Table 2) was used to assess participants' earlier experiences with AV (G1), their initial thoughts on trust and acceptance in AV (G2) and willingness to perform NDRT in AV (G3) and was based on earlier studies (e.g., Nordhoff et al., 2021, 2023) where they proved to be effective to assess automation experience.

During the simulator drive, the participant were asked two questions after each TOR:

G11—How stressful was this TOR on a scale from 1 to 5?

G12—How urgent was this TOR on a scale from 1 to 5?

The data gathered from these questions were used to assess the take-over perception of the participants. The questions were asked directly after each TOR so the participants had a clear memory of their perception.

The post-drive questionnaire filled in after the simulator drive has been used to obtain the participants' thoughts on the automation as experienced. Questions were derived from a wide range of literature sources (e.g., Nordhoff et al., 2021, 2023) and complemented with two new items as detailed in Table 3. The advantage of having pre- and post-drive questionnaires is that apart from drawing conclusions on the absolute trust in automation, conclusions about the increase in trust can be drawn.

3. Results

All participants completed the full experiment, responded adequately to most TORs, and did not intervene in the automated system maneuvers. Results with the three HMIs are presented in Tables 4–6 and Figures 5–8.

TABLE 4

Table 4. Summary of the pre-drive and post-drive questionnaire data.

TABLE 5

Table 5. Summary of the questionnaire responses on the take-over perception asked after each TOR.

TABLE 6

Table 6. Percentage of successful TOR results for every HMI.

FIGURE 5

Figure 5. Trust increase visualized by the pre-drive trust in automation G2, post-drive trust G5, trust process G6, and trust purpose G7 (mean ratings with standard deviations on a Likert scale from 1 to 5).

FIGURE 6

Figure 6. HMI driving task G8 and HMI feeling G9 (mean ratings with standard deviations on a Likert scale from 1 to 5).

FIGURE 7

Figure 7. TOR perception of urgency G11 and stress G12 (mean ratings with standard deviations on a Likert scale from 1 to 5).

FIGURE 8

Figure 8. TOR reaction times G17 for every TOR (means with standard deviations).

Results for question groups are presented as an average over items in each question group (see Tables 2, 3). Before averaging, items representing negative perceptions were reverse coded (“How dangerous are SAE Level 3 automated vehicles,” “The system is deceptive,” “The user interface is complex”).

Results show clear positive effects of the presence of the color themed visual HMIs with few effects of HMI color. The significance of these effects was evaluated using two p-values. The p_BC value, calculated with a two-sample t-test, compares the baseline group with the two color themed HMIs combined, to evaluate the difference between the simple auditory and the more advanced HMIs. The p_RB value, also calculated with a two-sample t-test, compares the red group with the blue group, to assess the effect of HMI color.

Questions G2 and G3 from the pre-drive questionnaire were used to create balanced participant groups for the three HMIs. This was successful as differences between groups for G2 and G3 were insignificant (see Table 4).

3.1. Trust and HMI experience

Table 4 shows pre-drive and post-drive questionnaire responses for the three HMIs. The post-drive questionnaire shows substantial benefits of the color themed HMIs for G4-G9. These benefits are significant for all question groups except for G7. However in G4–G9 no significant differences are found between the blue themed and red themed HMI. The only significant effect of HMI color is the ambient color perception (G10) where, as expected, red was perceived as much more urging (3.92) compared to blue (2.07) on a scale from 1 to 5. However, this higher urgency associated with ambient color did not have negative or positive effects on the overall perception of trust and HMI (G4–G9). This demonstrates that both color themed HMIs significantly enhanced the automation experience and trust and that these HMIs were positively evaluated. However, these subjective benefits of the color themed HMIs did not differ between the red and blue themed HMIs.

3.2. Take-over perception

After each TOR, every participant answered questions about their perception of the TOR. Since the urgency and stress per TOR is not relevant for this paper, the results of the four TORs averaged per HMI group are reported in Table 5. The color themed HMIs induced a significantly higher urgency and stress (p_BC = 0.042). The p_RB values however are very high (>0.6) which indicates that differences between red and blue are insignificant. Thus, both color themed HMIs create a similar TOR urgency and stress in G11 and G12 even though the ambient color red was perceived as more urging in G10.

3.3. Take-over performance

Table 7 shows the TOR reaction time. TOR data of four participants were not saved. With the first and fourth TOR, two participants did not react and thus no reaction time was collected. The second and third TOR had three participants which did not react. This missing data was not taken into account in any calculations of reaction time. The color themed HMIs reduced the TOR reaction time in TOR1–TOR4 and this effect was significant for TOR1, TOR3, and the average over TOR1–TOR4. The red themed HMI reduced the reaction time with an average of 0.42 s against baseline whereas the blue themed HMI reduced the reaction time with only 0.15 s. However this difference between red and blue themed HMIs was not significant (p_RB = 0.156). The quality of the take-over is measured by the success rate. This is a percentage that shows how many take-overs were completed without any intervention of the automation. This success rate is shown in Table 6 and indicates that the automation intervened in a majority of TORs. Apparently the time-window offered to react was rather short while the perceived urgency to intervene was modest (the urgency was around 3 on a scale from 1 to 5: see G11 in Table 5). This only emerged analyzing all results. The majority of the participants thought they were handling the take-over themselves, while in reality it was the automation that intervened. This issue was most apparent with the baseline HMI, whereas the transition between manual and automated driving was more clear for the color themed HMIs since the ambient lighting turned off when the automation took over. The red themed HMI led to most successful TORs but effects of HMI on TOR quality were not significant.

TABLE 7

Table 7. Summary of the reaction time (in seconds) for every HMI.

4. Discussion

This study investigated the influence of a color themed HMI on trust and take-over performance in an automated vehicle. Forty-five participants were divided in three groups, each with a different HMI. Every participant had to perform four TORs in a 30 min drive, during which their performance was measured. Before, during and after the drive, participants filled in questionnaires to assess their increase in trust.

4.1. Trust and HMI experience

With the color themed HMIs the TORs were perceived as more urgent and more stressful as compared to the baseline HMI (Figure 7). The higher urgency can be seen as positive and the higher stress is not alarming with a mean around 2.7 on a scale from 1 to 5. However, the higher stress could be associated with the additional information to be processed with the advanced HMIs.

With all three HMIs trust in automation was higher after the experiment (Figure 5) and this trust increase was significantly higher with the color themed HMIs. While the trust of the baseline group only increased with 0.35, the trust of the red and blue themed HMI groups increased with 0.73 and 0.74, respectively. Furthermore, the color themed HMIs scored better on “Trust Process (G6),” “Trust purpose (G7),” “HMI Driving task (G8),” and “HMI Feeling (G9)” and this effect was significant for G6, G8, and G9.

These positive effects of the ambient lighting and the informative visual HMI on trust strongly support Hypothesis 1a. Lee and See (2004) defined trust as “the attitude that an agent will help achieve an individual's goals in a situation characterized by uncertainty and vulnerability.” The positive effects of HMIs can be jointly attributed to the ambient lighting attracting the attention when the TOR is issued and the HMI display instructing the driver to brake. Apparently the non-intrusive ambient lighting combined with the specific instructions on the display are well appreciated. These positive effects of the informative visual HMI and ambient lighting align with (Yang et al., 2018; Zhang et al., 2019; Löcken et al., 2020) who have shown how visual HMI using ambient lighting increase trust (Löcken et al., 2020) and support faster driver reactions (Yang et al., 2018; Zhang et al., 2019). Furthermore, our study advances the existing literature by demonstrating the combined effects of ambient lighting and informative visual HMIs, which was not addressed in previous studies such as Yang et al. (2018) and Löcken et al. (2020).

4.2. Take-over performance

Regarding the take-over performance (G13, G14, G15, G16, G17) in Figure 8, two results stand out. The reaction times in the first TOR are significantly higher in comparison to the following TORs with all three HMIs. This indicates that participants were learning how to react to the TORs, which is a common finding (Gold et al., 2018). The red themed HMI elicited the shortest reaction times in all four TORs. This reaction time benefit was significant comparing the color themed HMIs to baseline which supports Hypothesis 1b and can (similar to the effect on trust) be jointly attributed to the ambient lighting attracting the attention when the TOR is issued and the HMI display instructing the driver to brake. Our findings on the benefits of the red-themed HMI are consistent with previous research on color and urgency perception. Friedrich and Vollrath (2022) found that red leads to a significantly shorter reaction time than yellow in situations requiring quick reactions with unmanned aircraft systems so it was suggested to use red for safety critical indications. The shorter reaction time with the red-themed HMI is in line with Hypothesis 2 but the difference between red and blue was not significant. In line with Hypothesis 2, the red themed HMI also led to most successful TORs but effects of HMI on TOR quality were also not significant. This calls for the collection of more data including more participants and varying the actual urgency of the events. The automation performance can also be enhanced. The percentage of successful TORs (G18) indicates that the automation intervened in a majority of cases. This could have been alleviated through an even earlier TOR which however may not always be technically feasible. This could also be alleviated through a later but more aggressive intervention by the automation but this would presumably reduce perceived safety, trust, and motion comfort. In our study the automation intervened if the TTC became < 2 s. However this is overly conservative as TCC ignores accelerations. This could be resolved using the enhanced time to collision (ETTC) to control the intervention. ETTC takes in consideration the decelelration of both vehicles, which is relevant in our scenarios (Happee et al., 2017).

$\begin{array}{ll}TTC=\frac{dx}{v}& (1)\end{array}$

$\begin{array}{ll}ETTC=\{\begin{array}{cc}\frac{dx}{v},& \text{if }a=0\\ \frac{v-\sqrt{v^2+2a·dx}}{-a},& \text{if }a\ne 0\end{array}& (2)\end{array}$

4.3. Marginal effects of HMI color

The difference between the baseline and the color themed HMIs is substantial across a majority of results which demonstrates the ability of the selected experimental conditions and measures to assess the benefits of HMIs to enhance human performance and trust in automated driving. The differences between the red and blue themed HMIs, however, are small. The only significant difference between red and blue regards the ambient color perception (G10; Table 4). While red is perceived as urging, blue tends to be a more calming color (e.g., Elliot et al., 2007). Benefits of the red interface were insignificant in the current data but showed the expected trend (Hypothesis 2) with shorter reaction times (G13–G17) and more successful TORs (G18). This suggests that HMI color affects the perception of urgency rather than (or more strongly than) the actual TOR response.

4.4. Limitations and future work

We tested 3 groups of participants between 18 and 28 years old with at least 2 years of driving experience. Driving experience is a key factor in both trust and safe driving performance (Jin et al., 2020) and age affects risk perception and driving skill. The limited sample size may explain the insignificance of several effects of HMI color. Testing a larger and broader population sample is recommended to further establish benefits of color themed HMIs. Here also other colors and combinations of multimodal TOR HMIs can be explored including mixed reality (Riegler et al., 2020). The range of scenarios and urgency levels can be expanded, and users can develop their interpretation of HMI color over prolonged exposure. Effects of color could become significant testing more events but the current data shows limited effect sizes comparing red or blue HMIs. Potentially more promising would be the use of varying colors within HMIs using gradually changing colors with red signaling the highest urgency. TOR timing could be enhanced using ETTC rather than TTC as outlined in Section 4.2. Additionally, studying the effects of long exposure to different HMI colors could provide valuable insights into how users adapt and interpret HMI color over time. This would allow for a more comprehensive understanding of the impact of color and HMI design on trust and performance in automated driving scenarios. Finally, HMIs can be tested in actual vehicles on tracks and public roads and measures can be extended to further probe the understanding of HMI information and AV operation in realistic complex driving conditions.

5. Conclusions

This driving simulator study demonstrates significant benefits of a color themed HMI:

• The advanced color themed HMI with informative display and ambient lighting contributes more to the trust in automated vehicles in comparison to the baseline auditory HMI. Participants found the color themed HMI more trustworthy, they understood their driving tasks better and they found the HMI more pleasant to work with.

• The advanced color themed HMI outperforms the baseline HMI in case of TOR. Participants who experienced the color themed HMI found that the TORs were more urgent and stressful, which led to a reduced reaction time.

• The red ambient lighting is perceived as more urging while blue is perceived as calming, but effects of color on TOR performance and trust were not significant.

The obtained results could contribute to acceptance of automated vehicles. An advanced color themed HMI can improve trust in automated vehicles and contribute to safety in TOR.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by HREC Committee of Delft University of Technology. The patients/participants provided their written informed consent to participate in this study.

Author contributions

AJ, AS, HP, and OY: conceptualization, data curation, formal analysis, methodology, validation, visualization, and writing—original draft. XH, SK, and RH: supervision, conceptualization, methodology, and writing—review and editing. All authors contributed to the article and approved the submitted version.

Funding

SK and RH were supported by the HADRIAN European Union's Horizon 2020 research and innovation program (grant agreement 875597) and XH was supported by the SHAPE-IT European Union's Horizon 2020 research and innovation program (Marie Skłodowska-Curie grant agreement 860410).

Acknowledgments

We want to thank Drs. Barys Shyrokau and Joris Giltay for their support using the DAVSi driving simulator.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Author disclaimer

The contents of this publication are the sole responsibility of the authors and do not necessarily reflect the opinion of the European Union.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpsyg.2023.1128285/full#supplementary-material

Two sounds representing the auditory cues and one video recording the whole experiment process.

References

Ayoub, J., Zhou, F., Bao, S., and Yang, X. J. (2019). “From manual driving to automated driving: a review of 10 years of Autoui,” in Proceedings of the 11th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Utrecht), 70–90. doi: 10.1145/3342197.3344529

CrossRef Full Text | Google Scholar

Borojeni, S. S., Chuang, L., Heuten, W., and Boll, S. (2016). “Assisting drivers with ambient take-over requests in highly automated driving,” in Proceedings of the 8th International Conference on Automotive User Interfaces and Interactive Vehicular Applications (Ann Arbor, MI), 237–244. doi: 10.1145/3003715.3005409

CrossRef Full Text | Google Scholar

Caberletti, L., Elfmann, K., Kummel, M., and Schierz, C. (2010). Influence of ambient lighting in a vehicle interior on the drivers perceptions. Light. Res. Technol. 42, 297–311. doi: 10.1177/1477153510370554

CrossRef Full Text | Google Scholar

Campbell, J. L., Brown, J. L., Graving, J. S., Richard, C. M., Lichty, M. G., Bacon, L. P., et al. (2018). Human Factors Design Guidance for Level 2 and Level 3 Automated Driving Concepts. NHTSA (National Highway Traffic Safety Administration), Technical report.

Google Scholar

Elliot, A. J., Maier, M. A., Moller, A. C., Friedman, R., and Meinhardt, J. (2007). Color and psychological functioning: the effect of red on performance attainment. J. Exp. Psychol. 136, 154–168. doi: 10.1037/0096-3445.136.1.154

PubMed Abstract | CrossRef Full Text | Google Scholar

Feierle, A., Holderied, M., and Bengler, K. (2020). “Evaluation of ambient light displays for requests to intervene and minimal risk maneuvers in highly automated urban driving,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC), 1–8. doi: 10.1109/ITSC45102.2020.9294645

CrossRef Full Text | Google Scholar

Forster, Y., Naujoks, F., and Neukum, A. (2017). “Increasing anthropomorphism and trust in automated driving functions by adding speech output,” in IEEE Intelligent Vehicles Symposium (Los Angeles, CA). doi: 10.1109/IVS.2017.7995746

CrossRef Full Text | Google Scholar

Friedrich, M., and Vollrath, M. (2022). Urgency-based color coding to support visual search in displays for supervisory control of multiple unmanned aircraft systems. Displays 74:102185. doi: 10.1016/j.displa.2022.102185

CrossRef Full Text | Google Scholar

Gold, C., Damböck, D., Lorenz, L., and Bengler, K. (2013). “Take over! How long does it take to get the driver back into the loop?,” in Proceedings of the Human Factors and Ergonomics Society (Washington, DC). doi: 10.1177/1541931213571433

CrossRef Full Text | Google Scholar

Gold, C., Happee, R., and Bengler, K. (2018). Modeling take-over performance in level 3 conditionally automated vehicles. Accid. Anal. Prevent. 116, 3–13. doi: 10.1016/j.aap.2017.11.009

PubMed Abstract | CrossRef Full Text | Google Scholar

Happee, R., Gold, C., Radlmayr, J., Hergeth, S., and Bengler, K. (2017). Take-over performance in evasive manoeuvres. Accid. Anal. Prevent. 106, 211–222. doi: 10.1016/j.aap.2017.04.017

PubMed Abstract | CrossRef Full Text | Google Scholar

Hill, R. A., and Barton, R. A. (2005). Red enhances human performance in contests. Nature 435:293. doi: 10.1038/435293a

PubMed Abstract | CrossRef Full Text | Google Scholar

Jian, J.-Y., Bisantz, A. M., and Drury, C. G. (2000). Foundations for an empirically determined scale of trust in automated systems. Int. J. Cogn. Ergon. 4, 53–71. doi: 10.1207/S15327566IJCE0401_04

PubMed Abstract | CrossRef Full Text | Google Scholar

Jin, M., Lu, G., Chen, F., and Shi, X. (2020). “How driving experience affect trust in automation from level 3 automated vehicles? An Experimental Analysis,” in 2020 IEEE 23rd International Conference on Intelligent Transportation Systems, ITSC 2020 (Rhodes). doi: 10.1109/ITSC45102.2020.9294247

CrossRef Full Text | Google Scholar

Khusro, Y. R., Zheng, Y., Grottoli, M., and Shyrokau, B. (2020). MPC-based motion-cueing algorithm for a 6-dof driving simulator with actuator constraints. Vehicles 2, 625–647. doi: 10.3390/vehicles2040036

CrossRef Full Text | Google Scholar

Kim, S., Kabbani, T., Serbes, D., Happee, R., Hartavi, A. E., and van Egmond, R. (2022). “A new approach to sound design in automated vehicles,” in Proceedings of the Human Factors and Ergonomics Society Europe Chapter 2022 Annual Conference, 149–159.

Google Scholar

Kim, S., van Egmond, R., and Happee, R. (2021). Effects of user interfaces on take-over performance: a review of the empirical evidence. Information 12:162. doi: 10.3390/info12040162

CrossRef Full Text | Google Scholar

Kyriakidis, M., Happee, R., and de Winter, J. C. (2015). Public opinion on automated driving: Results of an international questionnaire among 5000 respondents. Transport. Res. Part F 32, 127–140. doi: 10.1016/j.trf.2015.04.014

CrossRef Full Text | Google Scholar

Lee, J. D., and See, K. A. (2004). Trust in automation: designing for appropriate reliance. Hum. Fact. 46, 50–80. doi: 10.1518/hfes.46.1.50.30392

PubMed Abstract | CrossRef Full Text | Google Scholar

Löcken, A., Frison, A. K., Fahn, V., Kreppold, D., Götz, M., and Riener, A. (2020). “Increasing user experience and trust in automated vehicles via an ambient light display,” in Conference Proceedings - 22nd International Conference on Human-Computer Interaction with Mobile Devices and Services: Expanding the Horizon of Mobile Interaction, MobileHCI 2020 (Oldenburg). doi: 10.1145/3379503.3403567

CrossRef Full Text | Google Scholar

Lu, Z., Happee, R., Cabrall, C. D., Kyriakidis, M., and de Winter, J. C. (2016). Human factors of transitions in automated driving: a general framework and literature survey. Transport. Res. Part F 43, 183–198. doi: 10.1016/j.trf.2016.10.007

CrossRef Full Text | Google Scholar

Matthews, T., Dey, A. K., Mankoff, J., Carter, S., and Rattenbury, T. (2004). “A toolkit for managing user attention in peripheral displays,” in Proceedings of the 17th Annual ACM Symposium on User Interface Software and Technology (Santa Fe, NM), 247–256. doi: 10.1145/1029632.1029676

CrossRef Full Text | Google Scholar

Mirnig, A. G., Gärtner, M., Wallner, V., Demir, C., Özkan, D. Y., Sypniewski1, J., and Alexander, M. (2023). Personal and Ubiquitous Computing - Theme Issue on Engaging with Automation: Understanding and Designing for Operation, Appropriation, and Behavior Change (Springer).

Google Scholar

Naujoks, F., Befelein, D., Wiedemann, K., and Neukum, A. (2017). “A review of non-driving-related tasks used in studies on automated driving,” in International Conference on Applied Human Factors and Ergonomics (Los Angeles, CA: Springer), 525–537. doi: 10.1007/978-3-319-60441-1_52

CrossRef Full Text | Google Scholar

Naujoks, F., Mai, C., and Neukum, A. (2021). “The effect of urgency of take-over requests during highly automated driving under distraction conditions,” in Advances in Human Aspects of Transportation: Part I, Vol. 15 (New York, NY). doi: 10.54941/ahfe100646

CrossRef Full Text | Google Scholar

Naujoks, F., Wiedemann, K., Schömig, N., Hergeth, S., and Keinath, A. (2019). Towards guidelines and verification methods for automated vehicle HMIS. Transp. Res. Part F 60, 121–136. doi: 10.1016/j.trf.2018.10.012

CrossRef Full Text | Google Scholar

Nordhoff, S., de Winter, J., Madigan, R., Merat, N., van Arem, B., and Happee, R. (2018). User acceptance of automated shuttles in Berlin-Schöneberg: a questionnaire study. Transp. Res. Part F 58, 843–854. doi: 10.1016/j.trf.2018.06.024

CrossRef Full Text | Google Scholar

Nordhoff, S., Kim, S., He, X., Gentner, A., and Happee, R. (2023). Understanding human machine interfaces of SAE Level 2 automated cars.

Google Scholar

Nordhoff, S., Stapel, J., He, X., Gentner, A., and Happee, R. (2021). Perceived safety and trust in SAE Level 2 partially automated cars: results from an online questionnaire. PLoS ONE 16:e0260953. doi: 10.1371/journal.pone.0260953

PubMed Abstract | CrossRef Full Text | Google Scholar

Petermeijer, S., Doubek, F., and de Winter, J. (2017). “Driver response times to auditory, visual, and tactile take-over requests: a simulator study with 101 participants,” in 2017 IEEE International Conference on Systems, Man, and Cybernetics (SMC), 1505–1510. doi: 10.1109/SMC.2017.8122827

CrossRef Full Text | Google Scholar

Politis, I., Brewster, S., and Pollick, F. (2015). “Language-based multimodal displays for the handover of control in autonomous cars,” in Adjunct Proceedings of the 7th International Conference on Automotive User Interfaces and Interactive Vehicular Applications, AutomotiveUI 2015 (Nottingham). doi: 10.1145/2799250.2799262

CrossRef Full Text | Google Scholar

Riegler, A., Riener, A., and Holzmann, C. (2020). “A research agenda for mixed reality in automated vehicles,” in Proceedings of the 19th International Conference on Mobile and Ubiquitous Multimedia, MUM '20 (New York, NY: Association for Computing Machinery), 119–131. doi: 10.1145/3428361.3428390

CrossRef Full Text | Google Scholar

SAE International (2021). SAE J3016: Levels of Driving Automation. Society of Automotive Engineers.

Google Scholar

Sonoda, K., and Wada, T. (2017). Displaying system situation awareness increases driver trust in automated driving. IEEE Trans. Intell. Vehicles 2, 185–193. doi: 10.1109/TIV.2017.2749178

PubMed Abstract | CrossRef Full Text | Google Scholar

Stouffer, K., Falco, J., and Scarfone, K. (2011). “GUIDE to industrial control systems (ICS) security,” in The Stuxnet Computer Worm and Industrial Control System Security.

Google Scholar

Verberne, F. M., Ham, J., and Midden, C. J. (2012). Trust in smart systems: sharing driving goals and giving information to increase trustworthiness and acceptability of smart systems in cars. Hum. Fact. 54, 799–810. doi: 10.1177/0018720812443825

PubMed Abstract | CrossRef Full Text | Google Scholar

Waytz, A., Heafner, J., and Epley, N. (2014). The mind in the machine: Anthropomorphism increases trust in an autonomous vehicle. J. Exp. Soc. Psychol. 52, 113–117. doi: 10.1016/j.jesp.2014.01.005

CrossRef Full Text | Google Scholar

Yang, Y., Karakaya, B., Dominioni, G. C., Kawabe, K., and Bengler, K. (2018). “An HMI concept to improve driver's visual behavior and situation awareness in automated vehicle,” in IEEE Conference on Intelligent Transportation Systems, Proceedings, ITSC (Maui, HI). doi: 10.1109/ITSC.2018.8569986

CrossRef Full Text | Google Scholar

Zhang, B., de Winter, J., Varotto, S., Happee, R., and Martens, M. (2019). Determinants of take-over time from automated driving: a meta-analysis of 129 studies. Transp. Res. Part F 64, 285–307. doi: 10.1016/j.trf.2019.04.020

CrossRef Full Text | Google Scholar