Abstract
The race model inequality has become an important testing tool for the analysis of redundant signals tasks. In crossmodal reaction time experiments, the strength of violation of the inequality is taken as measure of multisensory integration occurring beyond probability summation. Here we extend previous results on trimodal race model inequalities and specify the underlying context invariance assumptions required for their validity. Some simulation results comparing the race model and the superposition model for Erlang distributed random variables illustrate the trimodal inequalities.
1. Introduction
When stimulus information, perceived via several sensory modalities, indicates the occurrence of some event, an observer is faster detecting and responding to the stimulus compared to receiving only unimodal information, given a background of noisy signals. As a daily-life example consider the warning lights and siren of an ambulance in a traffic environment, a common audio-visual signal that allows, e.g., a driver to initiate an adequate reaction like giving way faster than if only acoustic or only visual information was available. Since the pioneering study by Todd (1912), this redundant signals effect (RSE) has frequently been replicated under laboratory conditions for crossmodal redundant signals combining different modalities, for both manual and saccadic reaction times (RT), and under different experimental conditions (e.g., divided vs. focused attention) (e.g., Miller, 1982; Gielen et al., 1983; Diederich and Colonius, 1987; Corneil et al., 2002).
A number of different models for the mechanisms underlying the RSE have been suggested. Raab (1962) proposed that race models could explain the speedup of responses. Race models assume that (a) each individual stimulus elicits a modality-specific process performed in parallel with the others, and (b) the winners time determines the observable RT, which will also consist of other components like motor execution time. This model implies that the RSE is generated by statistical facilitation, or probability summation: If latencies are interpreted as (non-negative) random variables, the time to respond to the first of several redundant signals is faster, on average, than the response time to a single signal. More generally, Miller (1982) observed that for the race model with stimuli x and y, the following inequality should hold:
for all non-negative time points t, with Fxy, Fx, and Fy denoting the distribution function for the redundant-signals condition and the single-stimulus conditions, respectively. Literature on this race model inequality (RMI) test involving different sensory modalities is huge (for a recent review, see Gondan and Minakata, 2016), likely due to the following reason: a statistically significant violation of the inequality for some value of t suggests that the observed speedup of the response cannot be accounted for entirely by probability summation and some, additional or alternative, coactivation mechanism has to be postulated.
Numerous modeling approaches for a coactivation mechanism have been proposed (Diederich, 1995) (see also Diederich and Colonius, 2012, for an overview). In contrast to the race model, to our knowledge they all are based on assuming some specific probability distribution or stochastic process. The one adopted for the simulations described below is introduced in Section 3.
Sometimes, instead of inequality (1), the race model is tested using inequality
However, this is not generally recommended since it is more restrictive than (1) by assuming stochastic independence between the random latencies. While this assumption is not required by the general race model, there is another, essential assumption hidden in any version of the model, known as “context independence” or “context invariance”: the processing of a stimulus of a given modality does not depend on which and how many stimuli from other modalities are presented concurrently (e.g., Luce, 1986, pp. 128–129; Colonius, 1990; Colonius and Vorberg, 1994; Townsend and Wenger, 2004; Gondan and Minakata, 2016). For a more general discussion using coupling theory, see Colonius (2016). A formal definition of this assumption is presented in the next section.
In the majority of RT studies on the RSE, only the bimodal case has been tested. Exceptions are, without claiming exhaustiveness, (Diederich, 1992a, 1995; Diederich and Colonius, 2004; Hecht et al., 2008; Oskarsson et al., 2012; Wang et al., 2012, 2013; Pomper et al., 2014; Hagmann and Russo, 2016), but it seems that no systematic investigation of all possible bimodal and trimodal RMIs and their interdependencies has been performed so far. Here we first specify the context invariance assumptions underlying the inequalities and then discuss various types of trimodal RMIs that can be tested assuming different patterns of (non-) violation of the inequalities. This is illustrated with a first set of simulations comparing the race model with a coactivation model.
2. Generalized race model inequalities
In most cases, the stimuli being tested for multisensory integration are from the visual, auditory, or somatosensory modality. Many notable studies also involve other modalities (e.g., Gu et al., 2008; Hoechenberger et al., 2015; Kaliuzhna et al., 2016), like vestibular and olfactory stimuli but, for simplicity, we refer to the first three here only. We write for the unimodal visual, for the bimodal visual-auditory, and for the trimodal visual-auditory-somatosensory condition, with the remaining obvious uni- and bimodal cases denoted accordingly.
2.1. The “context invariance” assumption of the race model
Let A, V, S be random latencies corresponding to stimulus modalities , and , respectively. FA, FV, and FS, denote the distribution function for the unimodal conditions , and respectively. For bimodal condition , HAV is the bivariate distribution function of random vector (A, V), and for trimodal condition , HAVS stands for the trivariate distribution function of (A, V, S). Thus, for example,
for all s, t, u ≥ 0. Moreover, for marginal distributions we write, e.g.,
The formal definition of “context invariance” is as follows:
DEFINITION 1. For all s, t, u ≥ 0, necessary and sufficient conditions for (trimodal) complete context invariance are
and
In other words, complete context invariance holds in the trimodal case if the distributions in the unimodal and bimodal conditions are identical to the corresponding univariate and bivariate marginal distributions of the trivariate distribution. As recently argued in Miller (2016), context invariance is an essential part of the race model concept.
2.2. Proving race model inequalities
The proof of Inequality 1 relies on a simple probability inequality. Rewrite (1) for the condition as
For any t ≥ 0, we define events
Because of context invariance, FAV(t) = P(Bt ∪ Ct), and the inequality follows from
where the last probability inequality is known as a special case of “Boole's inequality” (e.g., Diederich, 1992b). It is important to recognize the role of the context invariance assumption here: it guarantees that events Bt and Ct are defined on the same probability space for all t or, more generally, that there exists a bivariate distribution HAV(s, t) with marginals equal to FA(s) and FV(t), respectively: HAV(s, ∞) = FA(s) and HAV(∞, t) = FV(t).
2.3. Generalized race model inequalities: complete context invariance
In the following, we assume that trimodal context invariance holds unless indicated otherwise. In order to avoid trivial upper bounds larger than 1, the right-hand side of all inequalities presented may be replaced by, e.g., min{FA(t) + FV(t), 1}, etc. For simplicity, we do not state this explicitly.
For further reference, let us start with a listing of all possible bimodal RMIs:
A straightforward generalization to the trimodal case is
which follows again as special case of Boole's inequality.
As shown in Diederich (1992b), Bonferroni-type probability inequalities (Worsley, 1982) can be used to derive further trimodal RMIs:
A sharper1 bound for FAVS(t) results by taking the minimum (at each value of t) across all three bounds in 7–9:
Given the bimodal inequalities 3–5 hold, the trimodal inequalities 7–9 are sharper than Inequality 6. For example, if Inequalities 3 and 4 are satisfied, Inequality 9 implies Inequality 6:
2.4. Generalized race model inequalities with restricted context invariance
Next we consider a situation where, for example, data from conditions , , and are available but none from or . Responses under the bimodal condition can then be conceived as representing reaction times to a “combined” visual-auditory modality formally equivalent to a unimodal condition. The underlying distribution function is bivariate,
for all w, z ≥ 0, with AV denoting the RT in condition . Context invariance is restricted to the bivariate case, that is:
The race model then implies
Obviously, with adding the following two, there are three inequalities in total:
with the corresponding, mutually incompatible, restricted context invariance assumptions.
An alternative situation for considering Inequalities 13–15 is when all three univariate distributions are available but violations occur for some of the univariate pairs. For example, there may be one or more values t′ such that
While the race model for condition would be ruled out in this case, the joint processing of AV and S may still be consistent with a race.
3. An illustration with simulated data sets
In this section, we (i) illustrate a possible simulation approach and (ii) point to a specific aspect of dependency occurring for trimodal race models.
3.1. Erlang distribution simulation
Simulating the race model and comparing it with a coactivation model requires specifying some RT distributions. Here we select distribution functions derived from the most basic stochastic counting process, i.e., the Poisson process. The time between two randomly occurring events follows an exponential distribution with intensity rate λ. Stimulus processing time is defined as the waiting time of the Poisson process for c-th event. Empirically, criterion c may be influenced by the experimental condition. For example, rewarding high detection accuracy would increase the threshold (Luce, 1986) and higher values of c will result in longer detection time, denoted D. The distribution of D is known as Erlang distribution, a special case of the gamma distribution with an integer-valued shape parameter c and rate λ:
A rate parameter λx, x ∈ {A, V, S}, has to be specified for each single stimulus condition, auditory (A), visual (V), and somato-sensory (S), respectively, while threshold parameter c is assumed to be constant across the modalities. For simplicity, we neglect residual processes, like motor time, and assume that D equals the observed reaction time.
For x, y, z pairwise different modalities chosen from {A, V, S}, detection time in the race model for trimodal redundant stimulation, Dxyz, or Dxy for bimodal stimulation, is defined as the minimum of the corresponding single stimulus detection times Dx, Dy, Dz :
For a coactivation model, we choose the superposition model proposed in Schwarz (1989) (see also Diederich and Colonius, 1991). While a number of alternative coactivation models are available, choosing one based on the Poisson process has the advantage that, with one and the same set of values for the parameters, detection time D can be simulated either for the race model or for the superposition model. For the latter, detection time for redundant stimuli follows again an Erlang/Gamma distribution with the intensity rate given by the sum of the single stimulus intensity rates λx, λy, λz:
The prediction of the race model, an exponential distribution of RTs, is of course not consistent with typical data. It is taken here just for illustration; for fitting empirical data, it would be easy to add a Gaussian component, resulting in an ex-Gaussian distribution. For an empirical evaluation of race and superposition models we refer to Diederich (1992a).
Figures 1, 2 depict empirical distribution functions obtained from simulations of race and superposition models with parameter values c = 2, λA = λV = λS = 0.01 with sample size n = 2,000. In both figures the trimodal distribution was obtained from the superposition model, i.e., a Gamma (2, 0.03) distribution. Figure 1 compares it to the bounds described by Inequality 6 and 10 (denoted as “sharp RMI”). Both bounds are violated for a large range of percentiles, with the latter bound being violated even for percentiles beyond 80%. Figure 2 compares the trimodal distribution of the superposition model with data from simulating the bounds described by Inequalities 6 and 13 for the race model. Again, there are large-range violations of the bounds. FAV in bound 13 was obtained from the race model but, in principle, it could be made arbitrarily close to 1 by choosing some coactivation model for condition .
Figure 1
It shows the empirical distribution function of the simulated data for condition and the bounds described by Inequality 6 and 10 (denoted as “sharp RMI”). RTs for condition were simulated according to the superposition model, whereas RTs for all other conditions were generated according to the race model (n = 2,000, c = 2, λA = λV = λS = 0.01).
Figure 2
It shows the empirical distribution function of the simulated data for condition and the bounds described by Inequalities 6 and 13. RTs for condition were simulated according to a superposition model, whereas RTs for condition were generated according to the race model (n = 2,000, c = 2, λA = λV = λS = 0.01).
3.2. Trimodal dependency: adjusting correlations
One characteristic of the race model is the possibility to increase the size of the redundant signals effect by tweaking the correlations between the involved detection processes. Given a race between two detection processes, consider their random processing times, X and Y, varying from trial to trial. By definition, the shortest processing time min(X, Y) determines the detection time for a trial. Negative correlation means that, when one process is fast, the other tends to be slow in a given trial. Thus, there would only be few trials where both processes are rather slow. This results in decreased average detection times, since long times for one process are replaced by shorter times of the other process. One can show that a race model with two processes and maximal negative correlation yields the maximum redundant signals effect (Miller, 1982; Colonius, 1990, 2016).
However, a situation with three “competing” processes X, Y, Z, e.g., audio, visual, and somato-sensory, is more complicated. First, instead of one there are three correlation coefficients rxy, rxz, ryz. For a fixed set of two-sample Neyman–Pearson correlation coefficients rxy and rxz, the third coefficient ryz can not vary freely between −1 and +1 but is restricted to a narrower range (Stanley and Wang, 1969):
Second, values for the correlations generating maximal facilitation are not as trivial to find as in a two-process situation. Limited by the above mentioned constraints, it is not possible to construct a correlation matrix with coefficients rxy = rxz = ryz = −1. Our simulations with a multivariate gamma distribution (not presented here) suggest, for example, that setting rxy = rxz = ryz = −0.5 yields a relatively large redundant signals effect when processing times X, Y, Z have an identical underlying distribution.
Conclusion and outlook
We have shown that the race model inequality extends naturally from the bimodal to the trimodal case, as long as essential assumptions about context invariance are specified. Moreover, the trimodal case permits “mixed models,” that is, models (i) where the race assumption is only valid for certain modality combinations but not for others, and (ii) where not all unimodal distributions may be available.
For an application of the generalized race model inequalities presented here, the next step is to extend the current statistical tests developed for the bimodal case to the different trimodal cases (for a recent overview, see Gondan and Minakata, 2016). This will also require extensive simulation work as in Kiesel et al. (2007) including an extension to introduce intersubject variability. Finally, it is well-known that the upper bound of the bimodal race model inequality corresponds to maximal negative dependency between the two processes (Miller, 1982; Colonius, 1990, 2016); a particularly challenging task for future study is to characterize the new race model inequalities with respect to the trivariate statistical dependencies underlying their bounds.
Statements
Author contributions
HC, FW, and AD conceived of the analysis. FW performed the simulations. HC and FW wrote the paper.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Footnotes
1.^By “sharper” we mean “strictly smaller or equal.”
References
1
ColoniusH. (1990). Possibly dependent probability summation of reaction time. J. Math. Psychol.34, 253–275.
2
ColoniusH. (2016). An invitation to coupling and copulas, with applications to multisensory modeling. J. Math. Psychol.74, 2–10. 10.1016/j.jmp.2016.02.004
3
ColoniusH.VorbergD. (1994). Distribution inequalities for parallel models with unlimited capacity. J. Math. Psychol.38, 35–58.
4
CorneilB.van WanrooijM.MunozD.van OpstalA. (2002). Auditory-visual interactions subserving goal-directed saccades in a complex scene. J. Neurophsiol.88, 438–2002.
5
DiederichA. (1992a). Intersensory Facilitation: Race, Superposition, and Diffusion Models for Reaction Time with Multiple Stimuli.Frankfurt: Peter Lang.
6
DiederichA. (1992b). Probability inequalities for testing separate activation models of divided attention. Percept. Psychophys.52, 714–716.
7
DiederichA. (1995). Intersensory facilitation of reaction time: evaluation of counter and diffusion coactivation models. J. Math. Psychol.39, 197–215.
8
DiederichA.ColoniusH. (1987). Intersensory facilitation in the motor component?Psychol. Res.49, 23–29.
9
DiederichA.ColoniusH. (1991). A further test of the superposition model for the redundant-signals effect in bimodal detection. Percept. Psychophys.50, 83–86.
10
DiederichA.ColoniusH. (2004). Bimodal and trimodal multisensory enhancement: effects of stimulus onset and intensity on reaction time. Percept. Psychophys.66, 1388–1404. 10.3758/BF03195006
11
DiederichA.ColoniusH. (2012). Modeling multisensory processes in saccadic responses, in The Neural Bases of Multisensory Processes, eds MurrayM. M.WallaceM. T. (Boca Raton, FL: CRC Press), 253–276.
12
GielenS.SchmidtR.van den HeuvelP. (1983). On the nature of intersensory facilitation of reaction time. Percept. Psychophys.34, 161–168.
13
GondanM.MinakataK. (2016). A tutorial on testing the race model inequality. Atten. Percept. Psychophys. 78, 723–735. 10.3758/s13414-015-1018-y
14
GuY.AngelakiD.DeAngelisG. (2008). Neural correlates of multisensory cue integration in macaque MSTd. Nat. Neurosci.11, 1201–1210. 10.1038/nn.2191
15
HagmannC.RussoN. (2016). Multisensory integration of redundant trisensory stimulation. Attent. Percept. Psychophys. 78, 2558–2568. 10.3758/s13414-016-1192-6
16
HechtD.ReinerM.KarniA. (2008). Multisensory enhancement: gains in choice and in simple response times. Exp. Brain Res.189, 133–143. 10.1007/s00221-008-1410-0
17
HoechenbergerR.BuschN.OhlaK. (2015). Nonlinear response speedup in bimodal visual-olfactory object identification. Front. Psychol.6:1477. 10.3389/fpsyg.2015.01477
18
KaliuzhnaM.FerréE.HerbelinB.BlankeO.HaggardP. (2016). Multisensory effects on somatosensation: a trimodal visuo-vestibular-tactile interaction. Sci. Rep.6:26301. 10.1038/srep26301
19
KieselA.MillerJ.UlrichR. (2007). Systematic biases and type i error accumulation in tests of the race model inequality. Behav. Res. Methods39, 539–551. 10.3758/BF03193024
20
LuceR. (1986). Response Times: Their Role in Inferring Elementary Mental Organization. New York, NY: Oxford University Press.
21
MillerJ. (1982). Divided attention: evidence for coactivation with redundant signals. Cogn. Psychol.14, 247–279.
22
MillerJ. O. (2016). Statistical facilitation and the redundant signals effect: what are race and coactivation models?Atten. Percept. Psychophys.78, 516–519. 10.3758/s13414-015-1017-z
23
OskarssonP.ErikssonL.CarlanderO. (2012). Enhanced perception and performance by multimodal threat cueing in simulated combat vehicle. Hum. Factors54, 122–137. 10.1177/0018720811424895
24
PomperU.BrinckerJ.HarwoodJ.PrikhodkoI.SenkowskiD. (2014). Taking a call is facilitated by the multisensory processing of smartphone vibrations, sounds, and flashes. PLoS ONE9:e103238. 10.1371/journal.pone.0103238
25
RaabD. (1962). Statistical facilitation of simple reaction time. Trans. N.Y. Acad. Sci.24, 574–590.
26
SchwarzW. (1989). A new model to explain the redundant-signals effect. Percept. Psychophys.46, 498–500.
27
StanleyJ.WangM. (1969). Restriction on the possible values of r12, given values of r13 and r23. Educ. Psychol. Meas.29, 579–581.
28
ToddJ. (1912). Reaction to multiple stimuli. Arch. Psychol.3, 1–65.
29
TownsendJ.WengerM. (2004). A theory of interactive parallel processing: new capacity measures and predictions for a response time inequality series. Psychol. Rev.111, 1003–1035. 10.1037/0033-295X.111.4.1003
30
WangW.HuL.CuiH.XieX.HuY. (2013). Spatio-temporal measures of electrophysiological correlates for behavioral multisensory enhancement during visual, auditory and somatosensory stimulation: a behavioral and ERP study. Neurosci. Bull.29, 715–724. 10.1007/s12264-013-1386-z
31
WangW.HuL.ValentiniE.XieX.CuiH.HuY. (2012). Dynamic characteristics of multisensory facilitation and inhibition. Cogn. Neurodyn.6, 409–419. 10.1007/s11571-012-9197-x
32
WorsleyK. (1982). An improved bonferroni inequality and applications. Biometrika69, 297–302.
Summary
Keywords
multisensory integration, race model inequality, context invariance, trimodal case, redundant signals effect, superposition model, statistical facilitation, probability summation
Citation
Colonius H, Wolff FH and Diederich A (2017) Trimodal Race Model Inequalities in Multisensory Integration: I. Basics. Front. Psychol. 8:1141. doi: 10.3389/fpsyg.2017.01141
Received
23 February 2017
Accepted
22 June 2017
Published
11 July 2017
Volume
8 - 2017
Edited by
Martin Lages, University of Glasgow, United Kingdom
Reviewed by
Juergen Heller, University of Tübingen, Germany; Edgar Erdfelder, University of Mannheim, Germany
Updates
Copyright
© 2017 Colonius, Wolff and Diederich.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Hans Colonius hans.colonius@uol.de
This article was submitted to Quantitative Psychology and Measurement, a section of the journal Frontiers in Psychology
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.