Validation of Affect-tag Affective and Cognitive Indicators

The Affect-tag solution measures physiological signals to deliver indicators derived from cognitive science. To provide the most accurate and effective results, a database of electrodermal activity (EDA) signals acquired using the Affect-tag A1 band was created. An experimental paradigm was designed to measure action-taking, autonomic regulation, cognitive load (CL), emotions, and stress, affects, and social stress. The Affect-tag emotional power (EP), emotional density (ED), and CL affective and cognitive indicators were refined based on the physiological responses of 48 participants during these tasks. Statistical significance was obtained for all indicators in tasks they were designed to measure, resulting in a total accuracy score of 89% for the combined indicators. Data obtained during this study will be further analyzed to define emotional and affective states.

Introduction

NEOTROPE is a design and creation laboratory made up of hardware, software, and cognitive engineers dedicated to providing scientifically based products to the global market. Specializing in the design, development, integration, and publishing of electronic biometric measurement and software solutions for the processing of physiological data, NEOTROPE makes available intra-and interpersonal affective and cognitive indicators based on neuroscientific and cognitive principles for affective researchers in various fields.

Using principles of embodied cognitive and emotional functioning that can be objectively measured, NEOTROPE has created the Affect-tag solution. Where questionnaires and self-reporting on emotional and cognitive behaviors are subjective and are often difficult to interpret, Affect-tag captures the spontaneous corporal changes that cannot be detected by the naked eye. These minute physiological changes reveal information about mental functions, affective states, and arousal. The relationship between physiological readings and mental functioning is well known and has been used for many years in academic research. Thanks to miniaturization and progress in signal processing algorithms, it is now possible to measure these mobile indicators with only a connected wristband device.

Affect-tag includes a wristband measuring heart rate (HR) and electrodermal activity (EDA) as well as a computerized solution for real-time calculation of cognitive and affective indicators. This article presents an experiment allowing to validate, in a first step, the electrodermal indicators thanks to different tasks affecting differently the physiological responses. So, we want to create a database of individual physiological responses in standardized cognitive and affective tasks. To carry out this sampling and to ensure that it complies as closely as possible with current scientific standards, NEOTROPE collaborated with the teams of the SCALab laboratory (Centre National de la Recherche Scientifique and the University of Lille) for their methodological expertise, recruitment, and procurement of samples, and rigorous criteria ofmeasurements.

Theoretical Framework

Interpreting Physiological Signals

The body’s electrodermal and cardiac responses can be transformed into indicators that provide information about the functioning of the peripheral nervous system (PNS). The acting arm of the Central Nervous System, the PNS [which includes the somatic and autonomic nervous systems (ANSs)] is able to control the functioning of the peripheral organs and ensure the internal balance of the body independently of any conscious control.

Consequently, HR and EDA are under the influence of both central commands (cognitive and/or conscious) and purely autonomous homeostatic processes (unconscious and triggered by subcortical centers). Among these, the limbic system acts as an interface between cognition and the autonomic division of the PNS: it controls many emotional behaviors, like defense behaviors (stress), aggressiveness, fear, and pleasure. The limbic system is directly linked to the sympathetic branch of the ANS.

In order to distinguish between the different branches and possible origins of the measured signal, a number of precautions must be taken when interpreting them. The most common solution used to control variability is to intentionally induce specific peripheral reactions and sample the signal immediately after stimulation. By comparing the responses obtained during this time window with those collected during other phases (such as those without any induced stimulation), it is possible to evaluate the effects of the stimulation on peripheral responses.

Physiological Response to Stimuli

The ANS manages the physiology of the body by balancing internal physical and chemical conditions (homeostasis). In the event of a provocation, the body’s resources are mobilized resulting in observable physiological changes. Therefore, if an individual is stimulated or stressed, a momentary physiological imbalance and return to balance will be observed (allostasis, McEwen et al., 2015) and tools exist to measure these direct changes in EDA and HR specifically.

A stressor representing danger or threat is accompanied by a particular configuration of brain electrical activity. This autonomic defensive reaction, commonly known as the fight or flight response, has a significant impact on the body (Cannon, 1987) which can be simulated in order to extract patterns of EDA and HR.

In humans, cardiovascular reactions to mental tasks generally result in increased blood pressure, faster HR, renal vasoconstriction, skeletal muscle vasodilation, and decreased (or steady) vascular resistance (Callister et al., 1992). These responses are caused by an activation of the sympathetic branch of the ANS (Heidbreder et al., 1982). Since this branch also controls the activity of the sweat glands, it is therefore not surprising that mental activities also cause electrodermal reactions (Boucsein, 2012).

The EDA can be separated into its phasic and tonic components that yield information about specific processes. The phasic component, or Skin Conductance Response (SCR), measures rapid transient changes and is related to cognitive and emotional processes and arousal. The tonic component, or Skin Conductance Level (SCL), describes the background activity of the ANS and measures general activation and changes in cognitive processes (Dawson et al., 2007).

Although it is possible to objectively measure cognitive and emotional reactions using these physiological indicators, the majority of scientific studies do not distinguish between the different functions involved in the tasks used or do not systematically study several physiological responses simultaneously in the same individual. This study was therefore carried out in order to allow cardiac and electrodermal measurements to be made by a single measuring device, the Affect-tag A1 band, and for the same individual confronted with different standardized tasks involving cognitive and affective functions.

Measuring Cognition and Affect

Cognition is the set of mental processes that involve memory, language, reasoning, learning, intelligence, problem-solving, decision making, perception, and attention. All these processes enrich knowledge. The intensity with which these different functions are implemented can be measured and is referred to as cognitive load (CL). The study of cognition is carried out using various scientifically validated and standardized experimental tasks.

The definition of affective functions is much more complex. Emotions are commonly referred to as emotions when they represent only a small part of the various emotional behaviors. Given the complexity of the field, it is necessary to rely on solid and recognized scientific foundations to implement this concept, such as the work of Scherer (2005), which is precise in defining the various concepts. According to him, there are five types of affect: emotions (anger, sadness, joy…), moods (being cheerful, dark, depressed…), interpersonal stances (being distant, warm, caring…), attitudes (sympathetic, affectionate, hateful…), and affect dispositions (nervous, anxious, thoughtless…). These different affects are complex to manipulate and control: emotions are intense and difficult to reproduce experimentally, while moods, postures and attitudes are personality traits that are stable over time but dependent on individuals and therefore difficult to control. On the other hand, affect dispositions are more accessible because they reflect tendencies to experience certain moods and types of emotions, even when the stimulation is low (Scherer, 2005). This definition opens the door to experimental and controlled manipulations because it is possible to create transient (sad, happy…) and moderate moods to objectively measure cognitive and affective functions using different experimental tasks. The results will produce distinct impacts on the physiological functioning of the body that can be measured and interpreted using electrodermal and cardiac indicators.

As mentioned above, these peripheral responses may come from a central control (cognitive or affective) but also from subcortical or autonomic commands. It is therefore necessary to add a task that specifically activates the ANS without the intervention of cognition or other central commands.

Design of Experimental Procedure

As mentioned above, researchers at the SCALab laboratory designed and conducted a study to collect physiological data using standard tasks from a representative sample of the population. These data, recorded using the Affect-tag band, were used to correlate specific configurations of physiological responses (electrodermal and cardiac activity) with specific psychological states induced by controlled experimental situations.

The psychological states studied are as follows:

• Decision-making and action

• CL:

° Engagement of cognition in a task requiring large mental resources and involving memory, concentration, and mental calculation

• Affective reactions:

° Emotional: situations with positive and negative valence, reactivity to unpleasant memories

° Social: surprise reactions to a situation of social pressure

In addition to these main states, a provoked reflex reaction was also measured through breathing exercises to study autonomous physiological responses.

Decision Making and Reactivity

To measure cognitive engagement, decision making, and reaction, the participant was prepared (engaged) to detect the appearance of a stimulus and then react by pressing a key on the keyboard. Responses to acute stress are characterized by a sympathetic excitation (arousal) associated with a reciprocal decrease of the parasympathetic activity (vagal withdrawal) but moderate psychological stress, such as that caused by a simple reaction time task, does not imply a reciprocal decrease of the parasympathetic activity (Berntson et al., 1994; McEwen and Wingfield, 2003). The slope of tonic activation of EDA (i.e., electrodermal response rise time) is an indicator of the ability to mobilize cognitive resources (Turpin and Siddle, 1979; Boucsein, 2012). Behavioral response (reaction time) is also a relevant indicator (Powell et al., 1980; Bagherli et al., 2011) regarding EDA interpretation (Wilson, 1987).

Cognitive Load

Al-Shargie et al. (2016) showed that a mental calculation task based on addition and subtraction using three digits (i.e., 7–3 + 1) from the Montreal Imaging Stress Task (Montreal Imaging Stress Task, Dedovic et al., 2005) provided typical hormonal, physiological and electroencephalographic responses to cognitive stress, particularly when the participant is under time pressure (Al-Shargie et al., 2016). For this task, participants performed 25 increasingly difficult arithmetic operations and wrote the result on a sheet of paper. Compared to the previous task, there is a similar motor component but with a much higher working memory load. In addition to noticeable and progressive sympathetic activation, inhibition of the parasympathetic system should also be observed, particularly on frequency indicators of cardiac variability. The accuracy of the responses can also be correlated with physiological indicators.

Emotional Tasks

Emotional Stress

A video showing the harmful consequences of dangerous and irresponsible driving has the ability to provoke emotional reactions in a systematic way. Sympathetic activation, especially in the phasic component of EDA and the low-frequency HRV, and a pronounced inhibition of the parasympathetic system is expected (Kreibig, 2010). Because of the immersive nature of videos in general, compared to static images, we expect physiological responses that are close to emotional stress. Since events with a strong emotional charge are better retained, three images presenting the faces of the actors in the video are presented at the end of the affective test. This allows testing the effect of the video on the felt emotions: if the participant experiences an emotion, the probability of obtaining physiological responses for these three faces will then be higher.

Affective Dispositions

Many studies have examined the effects of emotional stress on peripheral responses (see Kreibig, 2010, for a review). Different types of paradigms have been implemented (presentation of videos, standardized images, sounds, etc.) but the effects of these paradigms on physiological responses are quite modest and variable from one individual to another. However, under certain conditions, the responses can be amplified. For example, when emotional inducers are images, EDA responses are more pronounced for large stimuli (Codispoti and De Cesarei, 2007) and when presented for more than 6 s (Codispoti et al., 2001). Since EDA is also correlated with certain personality variables and anxiety can influence both SCL and SCR (Naveteur and Baque, 1987), the level of anxiety of each participant must be assessed. This was done using the Positive and Negative Affect Schedule (PANAS,Watson et al., 1988) questionnaire.

Social Stress

In their article, Kirschbaum et al. (1993) defined a form of public speaking as a trigger for social stress (Trier Social Stress Test, TSST). This situation causes significant hormonal release (ACTH, prolactin, and cortisol) and physiological changes (increase in HR) and are therefore comparable to emotional stress. It also provokes specific responses to phasic characteristics of EDA (mainly amplitude, but also frequency). However, the TSST requires conditions that are complicated to implement (presence of a jury of three people and audio equipment) and requires that the participants be standing and active, which causes motor activation that interferes with purely emotional responses. We used a test similar to the TSST and Sing-a-Song-Stress Test (SSST, Brouwer and Hogervorst, 2014) that is simpler to implement and gives rise to the intended emotional response. In the modified SSST used in this experiment, the participant was asked to sing a song in the presence of the investigator. There was a countdown timer (5 s) for the activity during which social stress increases. However, as the counter reaches zero, the participants were told they did not have to sing after all. This situation is expected to cause sympathetic activation coupled with marked inhibition of the parasympathetic system.

Independent Tasks

Breathing exercises included asking the participant to rapidly inhale and slowly exhale. This pattern of breathing leads to parasympathetic activation and decrease sympathetic activation, as observed on EDA measurements (Cappo and Holmes, 1984). This breathing cycle specifically activates the parasympathetic system (Hirsch and Bishop, 1981; Moser et al., 1994; Strauss–Blasche et al., 2000) because respiratory sinus arrhythmia (RSA) is influenced by breathing and is under the exclusive control of the parasympathetic system.

The Affect-tag Solution

Wristbands are available for measuring physiological parameters but some of them can exhibit some limitations like an external light sensibility (Ollander et al., 2016). Neotrope designed the Affect-tag Band A1 to overcome this kind of limitation, for example, the shape of the band was made to minimize the external light impact on the HR sensor and to optimize the contact between the skin and the EDA sensors. Additionally, Neotrope designed their own wristband so they could control the signal processing chain which is pretty opaque in the case of other wristbands.

After 5 years of research and development, NEOTROPE has created the complete Affect-tag system, a mobile solution for measuring affective and cognitive reactions using physiological data allowing scientists as well as non-scientists to benefit from the complete study of the physiological response of the measurement to the results. Every component of the solution has been designed and developed by the team: the Band A1 to capture physiological data, the signal processing algorithms to extract information, mobile applications to synchronize context, and a cloud architecture for visualization of automatically processed results.

Hardware and Raw Signal Validation

The Affect-tag A1 band, worn on the wrist like a watch, continuously records EDA, HR, and Acceleration (ACC). Hardware components are medical-grade sensors and materials, including 4 mm Ag/Cl electrodes for precise EDA measurement and a silicon encasing for comfort and portability. The sampling rates are 20 Hz and 50 Hz for EDA and HR, respectively. The band went through a series of in-house and external comparative tests to validate the physiological data acquisition precision using laboratory-grade equipment. The wristband is connected to the Affect-tag mobile application via Bluetooth and transmits filtered EDA and HR which is displayed on the smart device in real-time.

Electrodermal Activity

One device for measuring EDA is the BIOPAC Systems, Inc. Values for this device are precise and respond to quick changes due to sensors placed on fingertips and the addition of a gel to amplify the sensitivity of the measurements. The downside of this device is that it is cumbersome, is not mobile, and is expensive. A preliminary test was conducted with SCALab to verify the raw EDA data capture between BioPac and Affect-tag using a series of repeatable protocols to control for expected outcomes.

Visual inspection of the raw data acquisition shows comparable results between Affect-tag and BIOPAC (see Figure 1). The lower values registered with Affect-tag are expected since the use of a conducting gel is not used.

FIGURE 1

Figure 1. Raw EDA signals for a single participant simultaneously measured using BIOPAC and Affect-tag.

Correlation results, obtained from a panel of 10 participants, from the comparison tests yielded the following results (Table 1).

TABLE 1

Table 1. Average correlation between Affect-tag and BIOPAC signals during specific tasks.

Based on these results we concluded that Affect-tag is suitable as a tool for measuring raw Electrodermal responses.

Affective and Cognitive Indicators

Neotrope has developed five affective and cognitive indicators to provide clear, straightforward results that represent the emotional experience. These indicators have been created and refined using data obtained in various experimental situations, including laboratory settings (e.g., product testing) and real-world settings (e.g., driving a car).

The indicators are:

• Emotional Power (EP): from EDA, this indicator reflects the strength of emotional reactions and is calculated using the amplitudes of the EDA phasic peaks in a given time frame. Studies have highlighted the specific link between the amplitude of SCRs and the sympathetic activation strength (Routtenberg, 1971; Lockhart, 1972; Pribram and McGuinness, 1975; Fowles, 1980; Gray, 1982; DeLong et al., 1983; Boucsein, 2012). Furthermore, the scientific literature has demonstrated the existence of several distinct activation systems (Routtenberg, 1971; Pribram and McGuinness, 1975; Fowles, 1980; Gray, 1982; DeLong et al., 1983; Boucsein, 2012): one of these systems is involved in affective activation or effort and is characterized by short phasic responses with a high amplitude approaching reflex responses and causing inhibition of subcortical structures involved in motivation and cognitive tasks, it is a fight/flight type response.

• Emotional Density (ED): from EDA, this indicator reflects the frequency of emotional reactions and is calculated using the number of EDA phasic peaks in a given time frame. An activated organism is characterized by a lowering of perceptual thresholds and an increase in the activity of central alert-related structures, resulting in bursts of sympathetic activity and thus SCRs. Following the lowering of sensory thresholds (confirmed by the increase in the activity of the somatomotor areas) and the need for reactivity (neuronal bursts of sympathetic activation), the frequency of occurrence of peaks increases, without affecting the amplitude, when the body is in a state of prolonged activation (Munro et al., 1987). This variation in the frequency of peak appearance, called lability, is related to emotional factors and the regulation of affect (Crider, 2008; Sarchiapone et al., 2018).

• CL: from EDA, this indicator reflects the amount of mental effort exerted and is calculated from the slow-moving tonic components of EDA in a given time frame. The tonic level is related to the general activation of the body, this indicator varies slowly over time. An increase in tonicity is observed when the situation leads to sustained engagement and attention (thus the opposite of a fight/flight type reaction, short and powerful; Boucsein, 2012; Saitis et al., 2018).

This approach takes into account the variation of objective measurements in various tasks already in use in the scientific community and the Affect-tag indicators are based only on those and not on subjective measurements. The creation of this database and the purpose of this testing is to further refine specific parameters and minimize errors for each Affect-tag indicator.

Methods

Participants

The study included 50 adults (25 males, 25 females) between the ages of 20 years and 55 years. All testing took place between May 2 and July 15, 2019. Recruitment was carried out in different locations in the north of France to diversify recruitment and all participants were paid 10€ for their involvement.

Procedure

Testing took place at the Imaginarium in Tourcoing, France, and other local laboratories. Upon arrival, participants signed a consent form, the Affect-tag band was placed on the left wrist, and they were seated in front of a laptop computer. The participants completed the PANAS questionnaire to correct anxiety levels and their influence on physiological signals. All instructions and tasks were displayed using E-Prime without any operator intervention. A marker was used to align the physiological data acquisition from Affect-tag with the start of the E-Prime program and for each task.

The tasks were as follows:

• Reaction time task

• Breathing exercises

• Mental calculation

• A video sequence, with negative valence

• Several images, which may be pleasant or unpleasant

• Affective memory task

• Social stress test (preparation to sing out loud)

• A final calibration phase.

A 60-s decompression phase occurred between each task. Table 2 outlines the progression of the tasks.

TABLE 2

Table 2. Progression of emotional and cognitive tasks tested.

All data is anonymous and no participant information is retained except for age and gender.

Stimuli

Affects

The International Affective Picture System (IAPS, Lang et al., 2008) database was used for the video and image sequences based on two factors: valence (positive and negative) and activation (low and high). Each block of images had a mean valence score and a mean arousal score. These values (along with their standard deviations) are presented in Table 3.

TABLE 3

Table 3. Mean valence and arousal scores for the images taken from the International Affective Picture System (IAPS) used during the passive testing phases.

Five images were presented for each block for a total of 75 s per block. Between each block, two neutral images were presented to neutralize the emotion.

The affective memory task entailed showing images related to the video with negative valence shown just before. Three images, which included characters from the video, designed to evoke an emotional response were shown to participants to test the presence of affective memory.

Cognitive Load and Autonomic Regulation Tasks

The CL tasks required participants to perform 25 arithmetic operations and record the result for each on a response sheet provided. This test allows for variation in mental load level by adjusting the complexity of the operations. The first 10 operations are very simple to perform (e.g., 2 + 1 + 4) while the final 15 are more complex (e.g., 3–9 + 7).

The breathing tasks involved a short period of inhalation, holding the breath, and a long period of exhalation because this respiratory time ratio increases cardiac parasympathetic tone. This method was adapted from Strauss–Blasche et al. (2000).

Analysis

Data Preparation

Data from the first two participants were excluded due to procedure adjustments. Data for 24 males and 24 females were analyzed.

The precise timing for responses and execution of each task was imported from E-Prime to synchronize it with the physiological data from Affect-tag. Data for everyone was then stored into phases by task, with the image tasks being separated by each block and the mental calculation phases separated into “easy” and “difficult”. To remove the contamination of the previous block, the instruction phases were used as the baseline.

Method Validation

To ensure the validity of each presented task, statistical significance was performed on the results. The following sections explain the validation method.

Anxiety Levels

The participants were asked to take the PANAS questionnaire to verify their capability to balance positive and negative affects. Analysis revealed that the distribution of scores on the positive scale is Gaussian and centered on a value slightly higher than the theoretical average. The distribution on the negative scale, however, is rectangular and centered on the minimum values of the scale. It can therefore be concluded that the population tested has rather positive feelings (mean = 27.39) without the intrusion of negative affects (mean = 13.06). This population, therefore, leans to a more positive affect.

Reaction Time

Analysis of the reaction times revealed that four participants did not complete the task as expected, the reaction times were too long for the type of task, indicating the response was not detected by the computer.

In this quite simple task (pressing a key on the keyboard), reaction times are usually very short (less than 2 s). But for four participants, this time was more than 4 s, which is very suspicious. We believe that the participants did not press the keyboard key hard enough for the computer to detect the response.

After removing these four participants from the global analysis, the distribution of results is asymmetric Gaussian. Therefore, the median was used as the central index rather than the average (Figure 2).

FIGURE 2

Figure 2. Number of participants vs. reaction time intervals during the reaction time tests showing an asymmetric Gaussian distribution.

Calculation Task

The 25 arithmetic operations were separated into weak and strong levels of mental load. The success rate of the arithmetic problems for the weak mental load (mean = 84%) was statistically significant and lower than the success rate of the strong mental load problems (mean = 61, t₍₄₈₎ = −7.96, p-value = 0). The items chosen for this experiment, therefore, make it possible to create two very different levels of difficulty.

Affect-tag Cogtech

The Affect-tag Cognitive Technology (CogTech) is first applied using the electronic components built into the wristband. The EDA is measured and transformed into readable data that are displayed in real-time on the Affect-tag mobile application.

For offline analysis, a “sanitation” algorithm was applied to the raw EDA signal to determine signal quality and reconstruct data identified as artifacts. This algorithm includes correction of gaps or duplicate data entries and an application of a median filter to reconstruct artifacts identified as noise or movement artifacts. A Signal Quality Index (SQI) score was calculated for each data set for the length of the signal (mean = 95%) and for each separated phase of the data. A threshold is set for an acceptable level of correction; if there are too many artifact corrections in the signal and the SQI is too low, that phase of the data will not be included in the analysis. For optimization of indicators, the threshold for acceptable signal quality was a variable that was tested.

Affect-tag tonic algorithms extract the tonic component of the measured EDA signal by using a series of specially designed signal filters. The Ledalab Continuous Decomposition Analysis (CDA) method was used as the gold standard for EDA signal decomposition (Benedek and Kaernbach, 2010; Brouwer et al., 2017) for the development and refinement of the Affect-tag algorithms. From the Ledalab documentation: “this method extracts the phasic (driver) information underlying EDA and aims at retrieving the signal characteristics of the underlying sudomotor nerve activity (SNA).” When correlated with the Ledalab CDA algorithms, the Affect-tag algorithms achieved a 98% correlation over 30 tests from data sets in multiple contexts. Before being displayed or recorded, an artifact detection algorithm is passed to remove artifacts in the tonic component based on signal shape and physiological feasibility.

The fast-changing phasic component peaks are detected using a set threshold level, time minimum between detected peaks, and achievable slope. Normalization is applied to both the tonic and phasic components to perform interindividual analysis and compare results across participants.

Accuracy Metrics

Before calculating the Affect-tag emotional indicators for each phase of the test, the expected outcomes by indicator were defined. For example, we expect to see a significant positive difference between the reaction task and baseline for EP but the same comparison is not of interest for CL where a much longer time is needed to measure an effect. A summary of the expected outcomes for each task and the baseline to which they were compared is shown in Table 4.

TABLE 4

Table 4. Comparisons by task: the phases used as the baseline are listed in the first column and the tasks are listed in the second column.

Some phases, such as Baseline 2, were omitted from the analysis due to possible contamination with data from other testing windows. This decision was made after the testing procedure was carried out and to ensure the slow EDA changes have had ample time to return to baseline in between tasks.

To refine the algorithm that calculates the emotional indicators, certain parameters were altered to attain results that were congruent with hypotheses and significantly different. The following parameters were modified until an acceptable error emerged:

• SQI threshold

• Minimum peak detection threshold

• Amplitude correction

• Stimulus window

• Minimum acceptable time between successive peaks

• Normalization method

Results

Validity of Passive Tasks

The analysis of the PANAS results revealed that the population tested was not balanced between positive and negative affect. Results for emotional measurements could therefore be skewed, with values for EP and ED not reflecting the general population. The test phases were therefore separated into “active” tasks and “passive” tasks, with the passive tasks being those using the valence and arousal media and the active tasks being the phases where participants were invited to perform an action. Since the analysis revealed that the analysis of the passive tasks was not representative of the population, they have been ruled inconclusive.

Statistical Significance Tests

A two-sided Student’s t-test (t₍₄₆₎ = 2.013) was performed for every test case listed in Table 4. The associated mean differences in indicators’ values for each test case are listed in Table 5.

TABLE 5

Table 5. Difference by task: the phases used as the baseline are listed in the first column and the tasks are listed in the second column.

The Emotional Power indicator reached significant differences for the reaction time task, breathing task, and all social stress situations (p-value <0.05). There were no significant differences for the cognitive charge tasks or the arousal images; however, the results were consistent with expectations. The affective memory and negative valence tasks produced no effect.

The ED indicator reached significant differences for the reaction time task and cognitive charge tasks. There were no significant differences for the arousal images showed and social stress tasks but, however, the countdown part of the social stress tasks exhibited a trend towards significant (p-value = 0.059).

The CL indicator reached significant differences for the social stress tasks and comparisons using high cognitive charge (p-value <0.001). Only the comparison between baseline and low cognitive charge had a non-significant difference yet it exhibited a trend in the expected direction.

Accuracy Representation

To go further, for every test case, an experimental score was assigned for each indicator based on the significance of the expected results. This permitted calculation of a weighted accuracy for each indicator in the active tasks for which they are essential. In Table 6 below are the results for each indicator.

TABLE 6

Table 6. Accuracy for each Affect-tag Emotional Indicator based on test cases with statistically significant results.

Discussion

To provide scientifically validated affective and cognitive indicators, it was important to create a database of expected physiological behavior. This database served as a basis for the refinement of the Affect-tag indicators which resulted in viable representations of emotional intensity, frequency, and CL.

The database for the affective tasks remains inconclusive. Affective memory depends on the ability to call on the memory of details from a similarly emotional situation. To have a database that can be generalized to the population, a normal distribution of affective responses is required on the PANAS questionnaire. An acceptable database can be obtained by measuring additional participants that result in a normal distribution of affective sentiments.

Perhaps a redesign of the paradigm to measure affect is required to ensure an accurate database. The IAPS, for example, was created for measuring certain pathologies vs. control groups. This database consists of only a control group with no specific pathology to measure against.

The hardware used in our study is a device that was entirely imagined and built by a start-up company. This required the construction of a wristband containing the sensors and embedded electronics, programming the firmware and the development of an application to control the device and conduct experiments using an adapted interface, to proceed with automated online data analyses, and finally to produce a final online report on a web application.

A native solution based on the recommendations of the Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures (2012) was developed and, at this time, the deconvolution method proposed by Benedek and Kaernbach (2010) seemed to be the most promising one (Boucsein, 2012). However, new methods have recently appeared and are currently being studied for implementation in our systems. Although the results of the study are encouraging, the fact that only one type of algorithm for signal decomposition was tested in this study is a limitation. Further studies are therefore necessary in order to optimize signal processing using the latest techniques. For example, the convex optimization approach (cvxEDA, Greco et al., 2015) and the sparse deconvolution approach (sparsEDA, Hernando-Gallego et al., 2017) are relatively fast techniques that can be implemented in wearable devices for a lower computational cost. Moreover, the convex optimization approach can be better to address inter-and intra-subject variability for a specific subject or condition and sparseEDA is a solution that allows for the fully automated extraction of phasic component from large and small EDA (Posada-Quintero and Chon, 2020). More interestingly, a new method based on spectral analysis of the electrodermal signal has recently been proposed (Posada-Quintero et al., 2016). The authors propose a new measure (termed TVSymp), based on the low frequency power spectra of the EDA. They found that this measure is highly sensitive to cognitive and physical stress, even exhibiting a higher between-subject consistency than did other measures of EDA. These methods became more prominent and widely accepted after the development of the Affect-tag EDA analysis algorithms using CDA, but they have not yet been used for this application. However, these techniques will be implemented in Affect-Tag for HRV analysis and therefore, it will become possible to combine spectral analyses of heartbeat and EDA, which will ensure more consistency between measurements and will make it possible to study new phenomena acting on both of these two measures in different ways and thus, through the study of these multiple signals, further open up possible case studies and interpretations.

This study shows that the Affect-tag sensor can measure physiological states accurately enough to detect changes in responses caused by tasks involving cognitive, emotional, and social functions. With this type of mobile system, it is possible to study people’s reactions in everyday situations (for example, stress detection in daily life, Can et al., 2019) or professional situations (for example, to study construction workers’ perceived risk, Choi et al., 2019), and above all, with a scientific rigor comparable to that of laboratories. The quality of the measurement is not the only factor to be considered here, the quality of the interpretation of this measurement is also important. It is therefore essential to have at one’s disposal a reference database constructed from measurements carried out under standardized conditions.

Data Availability Statement

The datasets generated for this study are available on request to the corresponding author.

Ethics Statement

The studies involving human participants were reviewed and approved by Comité d’Éthique Lille—Université de Lille. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

LS developed the theory. LS conceived and planned the experiment with the support of HS. A research engineer was employed by SCAlab to carry out the experiment. HS directed the study, processed the experimental data, and performed the analysis. LV aided in the analysis and drafted the manuscript. OJ supervised the project. All authors discussed the results and contributed to the final manuscript. All authors contributed to the article and approved the submitted version.

Funding

This work was a collaborative research project between Neotrope and SCAlab. Neotrope provided funding exclusively for the remuneration of the participants and the salary of the experimenter. This study was in part supported by grants from the Programme d’Investissements d’Avenir (grant number ANR-11-EQPX-0023, grant number FEDER SCV-IRDIVE).

Conflict of Interest

HS, LV and OJ were employed by the company NEOTROPE.

The remaining author declares that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We wish to thank Antoine Deplancke for his help in software development and technical expertise.

References

Al-Shargie, F., Kiguchi, M., Badruddin, N., Dass, S. C., Hani, A. F. M., Tang, T. B., et al. (2016). Mental stress assessment using simultaneous measurement of EEG and fNIRS. Biomed. Opt. Express 7, 3882–3898. doi: 10.1364/BOE.7.003882

PubMed Abstract | CrossRef Full Text | Google Scholar

Bagherli, J., Vaez-Mousavi, S. M., and Mokhtari, P. (2011). Effects of arousal and activation on simple and discriminative reaction time in a stimulated arousal state. World App. Sci. J. 12, 1877–1882.

Google Scholar

Benedek, M., and Kaernbach, C. (2010). A continuous measure of phasic electrodermal activity. J. Neurosci. Methods 190, 80–91. doi: 10.1016/j.jneumeth.2010.04.028

PubMed Abstract | CrossRef Full Text | Google Scholar

Berntson, G. G., Cacioppo, J. T., Binkley, P. F., Uchino, B. N., Quigley, K. S., Fieldstone, A., et al. (1994). Autonomic cardiac control. III. Psychological stress and cardiac response in autonomic space as revealed by pharmacological blockades. Psychophysiology 31, 599–608. doi: 10.1111/j.1469-8986.1994.tb02352.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Boucsein, W. (2012). Electrodermal Activity. New York, NY: Springer Science+Business Media, LLC.doi: 10.1007/978-1-4614-1126-0

CrossRef Full Text | Google Scholar

Brouwer, A. M., and Hogervorst, M. A. (2014). A new paradigm to induce mental stress: the sing-a-song stress test (SSST). Front. Neurosci. 8:224. doi: 10.3389/fnins.2014.00224

PubMed Abstract | CrossRef Full Text | Google Scholar

Brouwer, A. M., van Beurden, M., Nijboer, L., Derikx, L., Binsch, O., Gjaltema, C., et al. (2017). “A comparison of different electrodermal variables in response to an acute social stressor,” in International Workshop on Symbiotic Interaction (Cham: Springer), 7–17.

Google Scholar

Callister, R. O. B. I. N., Suwarno, N. O., and Seals, D. R. (1992). Sympathetic activity is influenced by task difficulty and stress perception during mental challenge in humans. J. Physiol. 454, 373–387. doi: 10.1113/jphysiol.1992.sp019269

PubMed Abstract | CrossRef Full Text | Google Scholar

Can, Y. S., Arnrich, B., and Ersoy, C. (2019). Stress detection in daily life scenarios using smart phones and wearable sensors: a survey. J. Biomed. Inform. 92:103139. doi: 10.1016/j.jbi.2019.103139

PubMed Abstract | CrossRef Full Text | Google Scholar

Cannon, W. B. (1987). The James-Lange theory of emotions: a critical examination and an alternative theory. Am. J. Psychol. 39, 106–124. doi: 10.2307/1422695

PubMed Abstract | CrossRef Full Text | Google Scholar

Cappo, B. M., and Holmes, D. S. (1984). The utility of prolonged respiratory exhalation for reducing physiological and psychological arousal in non-threatening and threatening situations. J. Psychosm. Res. 28, 265–273. doi: 10.1016/0022-3999(84)90048-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Choi, B., Jebelli, H., and Lee, S. (2019). Feasibility analysis of electrodermal activity (EDA) acquired from wearable sensors to assess construction workers’ perceived risk. Safety Sci. 115, 110–120. doi: 10.1016/j.ssci.2019.01.022

CrossRef Full Text | Google Scholar

Codispoti, M., Bradley, M. M., and Lang, P. J. (2001). Affective reactions to briefly presented pictures. Psychophysiology 38, 474–478. doi: 10.1111/1469-8986.3830474

PubMed Abstract | CrossRef Full Text | Google Scholar

Codispoti, M., and De Cesarei, A. (2007). Arousal and attention: picture size and emotional reactions. Psychophysiology 44, 680–686. doi: 10.1111/j.1469-8986.2007.00545.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Crider, A. (2008). Personality and electrodermal response lability: an interpretation. Appl. Psychophysiol. Biofeedback 33:141. doi: 10.1007/s10484-008-9057-y

PubMed Abstract | CrossRef Full Text | Google Scholar

Dawson, M. E., Schell, A. M., and Filion, D. L. (2007). “The electrodermal system,” in Handbook of Psychophysiology, 3rd Edn., eds J. T. Cacioppo, L. G. Tassinary and G. G. Berntson (New York, NY: Cambridge University Press), 159–181.

Google Scholar

Dedovic, K., Renwick, R., Mahani, N. K., Engert, V., Lupien, S. J., and Pruessner, J. C. (2005). The montreal imaging stress task: using functional imaging to investigate the effects of perceiving and processing psychosocial stress in the human brain. J. Psychiatry Neurosci. 30:319.

PubMed Abstract | Google Scholar

DeLong, M. R., Georgopoulos, A. P., and Crutcher, M. D. (1983). Cortico-basal ganglia relations and coding of motor performance. Experiment. Brain Res. 7, 30–40.

Google Scholar

Fowles, D. C. (1980). The three arousal model: implications of Gray’s two-factor learning theory for heart rate, electrodermal activity and psychopathy. Psychophysiology 17, 87–104. doi: 10.1111/j.1469-8986.1980.tb00117.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Gray, J. A. (1982). The neuropsychology of anxiety: an inquiry into the functions of the septohippocampal system. Br. J. Psychiatry 141:634. doi: 10.1017/S000712500011373X

CrossRef Full Text | Google Scholar

Greco, A., Valenza, G., Lanata, A., Scilingo, E. P., and Citi, L. (2015). cvxEDA: a convex optimization approach to electrodermal activity processing. IEEE Trans. Biomed. Eng. 63, 797–804. doi: 10.1109/TBME.2015.2474131

PubMed Abstract | CrossRef Full Text | Google Scholar

Heidbreder, E., Ziegler, A., Heidland, A., Kirsten, R., and Grüninger, W. (1982). Circulatory changes during mental stress in tetraplegic and paraplegic man. Klin. Wochenschr. 60, 795–801. doi: 10.1007/BF01721144

PubMed Abstract | CrossRef Full Text | Google Scholar

Hernando-Gallego, F., Luengo, D., and Artes-Rodriguez, A. (2017). Feature extraction of galvanic skin responses by non-negative sparse deconvolution. IEEE J. Biomed. Health Inform. 22, 1385–1394. doi: 10.1109/JBHI.2017.2780252

PubMed Abstract | CrossRef Full Text | Google Scholar

Hirsch, J. A., and Bishop, B. (1981). Respiratory sinus arrhythmia in humans: how breathing pattern modulates heart rate. Am. J. Physiol. 241, H620–H629. doi: 10.1152/ajpheart.1981.241.4.H620

PubMed Abstract | CrossRef Full Text | Google Scholar

Kirschbaum, C., Pirke, K. M., and Hellhammer, D. H. (1993). The ‘Trier Social Stress Test’-a tool for investigating psychobiological stress responses in a laboratory setting. Neuropsychobiology 28, 76–81. doi: 10.1159/000119004

PubMed Abstract | CrossRef Full Text | Google Scholar

Kreibig, S. D. (2010). Autonomic nervous system activity in emotion: a review. Biol. Psychol. 84, 394–421. doi: 10.1016/j.biopsycho.2010.03.010

PubMed Abstract | CrossRef Full Text | Google Scholar

Lang, P. J., Bradley, M. M., and Cuthbert, B. N. (2008). International affective picture system (IAPS): affective ratings of pictures and instruction manual. Technical Report A-8. Gainesville, FL: University of Florida.

Google Scholar

Lockhart, R. A. (1972). Interrelations between amplitude, latency, rise time and the Edelberg recovery measure of the galvanic skin response. Psychophysiology 9, 437–442. doi: 10.1111/j.1469-8986.1972.tb01791.x

PubMed Abstract | CrossRef Full Text | Google Scholar

McEwen, B. S., Bowles, N. P., Gray, J. D., Hill, M. N., Hunter, R. G., Karatsoreos, I. N., et al. (2015). Mechanisms of stress in the brain. Nat. Neurosci. 18:1353. doi: 10.1038/nn.4086

CrossRef Full Text | Google Scholar

McEwen, B. S., and Wingfield, J. C. (2003). The concept of allostasis in biology and biomedicine. Horm. Behav. 43, 2–15. doi: 10.1016/s0018-506x(02)00024-7

PubMed Abstract | CrossRef Full Text | Google Scholar

Moser, M., Lehofer, M., and Sedminek, A. (1994). Heart rate variability as a prognostic tool in cardiology. Circulation 90, 1078–1082. doi: 10.1161/01.cir.90.2.1078

PubMed Abstract | CrossRef Full Text | Google Scholar

Munro, L. L., Dawson, M. E., Schell, A. M., and Sakai, L. M. (1987). Electrodermal lability and rapid vigilance decrement in a degraded stimulus continuous performance task. J. Psychophysiol. 1, 249–257.

Naveteur, J., and Baque, E. F. I. (1987). Individual differences in electrodermal activity as a function of subjects’ anxiety. Pers. Individual Differ. 8, 615–626. doi: 10.1016/0191-8869(87)90059-6

CrossRef Full Text | Google Scholar

Ollander, S., Godin, C., Campagne, A., and Charbonnier, S. (2016). “A comparison of wearable and stationary sensors for stress detection,” in 2016 IEEE International Conference on Systems, Man, and Cybernetics (SMC) (Budapest, Hungary), 004362–004366. doi: 10.1109/SMC.2016.7844917

CrossRef Full Text | Google Scholar

Posada-Quintero, H. F., and Chon, K. H. (2020). Innovations in electrodermal activity data collection and signal processing: a systematic review. Sensors 20:479. doi: 10.3390/s20020479

PubMed Abstract | CrossRef Full Text | Google Scholar

Posada-Quintero, H. F., Florian, J. P., Orjuela-Cañón, A. D., Aljama-Corrales, T., Charleston-Villalobos, S., Chon, K. H., et al. (2016). Power spectral density analysis of electrodermal activity for sympathetic function assessment. Ann. Biomed. Eng. 44, 3124–3135. doi: 10.1007/s10439-016-1606-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Powell, D. A., Milligan, W. L., and Furchtgott, E. (1980). Peripheral autonomic changes accompanying learning and reaction time performance in older people. J. Gerontol. 35, 57–65. doi: 10.1093/geronj/35.1.57

PubMed Abstract | CrossRef Full Text | Google Scholar

Pribram, K. H., and McGuinness, D. (1975). Arousal, activation and effort in the control of attention. Psychol. Rev. 82, 116–149. doi: 10.1037/h0076780

PubMed Abstract | CrossRef Full Text | Google Scholar

Routtenberg, A. (1971). Stimulus processing and response execution: a neurobehavioral theory. Physiol. Behav. 6, 589–596. doi: 10.1016/0031-9384(71)90210-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Saitis, C., Parvez, M. Z., and Kalimeri, K. (2018). Cognitive load assessment from EEG and peripheral biosignals for the design of visually impaired mobility aids. Wireless Commun. Mobile Comput. 2018:8971206. doi: 10.1155/2018/8971206

CrossRef Full Text | Google Scholar

Sarchiapone, M., Gramaglia, C., Iosue, M., Carli, V., Mandelli, L., Serretti, A., et al. (2018). The association between electrodermal activity (EDA), depression and suicidal behaviour: a systematic review and narrative synthesis. BMC Psychiatry 18:22. doi: 10.1186/s12888-017-1551-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Scherer, K. R. (2005). What are emotions? And how can they be measured?. Soc. Sci. Inform. 44, 695–729. doi: 10.1177/0539018405058216

CrossRef Full Text | Google Scholar

Society for Psychophysiological Research Ad Hoc Committee on Electrodermal Measures, Boucsein, W., Fowles, D. C., Grimnes, S., Ben-Shakhar, G., Roth, W. T., et al. (2012). Publication recommendations for electrodermal measurements. Psychophysiology 49, 1017–1034. doi: 10.1111/j.1469-8986.2012.01384.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Strauss–Blasche, G., Moser, M., Voica, M., McLeod, D. R., Klammer, N., Marktl, W., et al. (2000). Relative timing of inspiration and expiration affects respiratory sinus arrhythmia. Clin. Exp. Pharmacol. Physiol. 27, 601–606. doi: 10.1046/j.1440-1681.2000.03306.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Turpin, G. and Siddle, D. A. (1979). Effects of stimulus intensity on electrodermal activity. Psychophysiology 16, 582–591.

PubMed Abstract | Google Scholar

Watson, D., Clark, L. A., and Tellegen, A. (1988). Development and validation of brief measures of positive and negative affect: the PANAS scales. J. Pers. Soc. Psychol. 54:1063. doi: 10.1037/0022-3514.54.6.1063

PubMed Abstract | CrossRef Full Text | Google Scholar

Wilson, K. G. (1987). Electrodermal lability and simple reaction time. Biol. Psychol. 24, 275–289. doi: 10.1016/0301-0511(87)90008-1

PubMed Abstract | CrossRef Full Text | Google Scholar