Piotr Winkielman
Michał Ziembowicz
Andrzej Nowak4- 1Psychology Department, University of California, San Diego, San Diego, CA, USA
- 2Behavioural Science Group, Warwick Business School, University of Warwick, Coventry, UK
- 3Department of Psychology, University of Social Sciences and Humanities, Warsaw, Poland
- 4Center for Complex Systems and New Technologies, Institute for Social Studies, University of Warsaw, Warsaw, Poland
Many philosophical approaches hypothesize that one function of consciousness is the creation of a unified subjective experience (Baars, 2005; Bayne, 2010). Such unified experience links different processing streams, originating in separate perceptual modules, thus enabling common access and generation of integrated decisions. All of this presumably occurs via a mechanism that blends information from different modalities into a single, multidimensional representation. But what exactly is unified in conscious experience? Prevailing explanations focus on integration of specific stimulus features at a perceptual or decisional level. In this opinion piece we discuss a simple but underappreciated explanation that focuses on processing dynamics. Specifically, we propose that cross-modal integration is facilitated by different modalities having a similar effect on the global subjective experience of processing quality. This integrated experience can then enter into decisional processes concerned with its source and relevance for the current behavior. As such, our account combines “experiential” and “decisional” process. Below we place this argument in the context of research on cross-modal integration and processing experiences, and discuss some implications.
Traditionally, research on multisensory experiences focuses on integration of information from different perceptual and conceptual cues. Some classic examples of such phenomena include the McGurk effect (changes in audition as function of vision; McGurk and MacDonald, 1976) and the double-flash illusion (changes in vision as a function of audition; e.g., Shams et al., 2000). Other classic examples of low-level cross-modal interactions include influences between pitch and brightness, loudness and size, or pitch and elevation. On a more conceptual level, cross-modal influences include shape or sound symbolism, such as the “bouba/kiki” effect (Ramachandran and Hubbard, 2001) and semantically-driven cases of synesthesia (Mroczko-Wąsowicz and Nikolic, 2014). A lively debate concerns when individual perceptual components from one modality are mandatorily modified by another modality, undergoing a low-level fusion that produces a single integrated percept, or when they are separate and integrated in high-level, post-perceptual stages via decisional processes (Spence, 2011; for an empirical example, see Hillis et al., 2002). Importantly, what such studies investigate are cross-modal influences on the representational content related to specific stimulus features.
Here we propose that cross-modal influences can also occur via processes that care less about the specific representational content but more about general representational quality, yielding global processing experiences. This proposal is grounded in several theoretical and empirical considerations.
Historically, the basic idea of processing experiences goes back to William James (1890) who spoke of “fringe consciousness” as experience that communicates a vague, unarticulated sense of peripheral contents relevant to the main task. Some “fringe experiences” include the feelings of familiarity and knowing, tip-of-the-tongue phenomena, and the sense of ease, rightness or coherence. Initially neglected by cognitive science, processing experiences and global “quality signals” are now of interest as a computationally efficient way of representing rich relational information (Mangan, 1993; Reber et al., 2002).
Empirically, the initial evidence for processing experiences came from research on fluency and familiarity (Whittlesea, 2002). For example, a pioneering study observed that people judge variable background noise as less loud when they hear a target word that was previously studied (Jacoby et al., 1988). Apparently, the ease (fluency) of target processing, deriving from previous exposure, gets misattributed to the loudness judgment. A related study reported misattributions of previous exposure to visual blur judgments (Whittlesea et al., 1990). Subsequent memory research documented similar influences using changes in perceptual format between stimuli appearing in the study and test phase of the experiment, including crossing words and pictures (e.g., Fazendeiro et al., 2005). Critically for the present argument, similar effects can occur for changes in modality, such as crossing auditory and visual stimulus presentation at study and test (e.g., Curran and Dien, 2003; Miller et al., 2008). These studies suggest that subjective experiences such as “fluency” and “familiarity” can be amodal and reflect joint influences from separate modalities. As a result, people cannot easily separate the processing quality associated with the target stimulus from contextual influences.
An important inspiration for our proposal are findings that a similar subjective experience can derive from processing facilitation at different processing stages. For example, factors that objectively facilitate visual detection and visual identification have similar effects on feelings of processing ease (Reber et al., 2004; Wurtz et al., 2008; but see Reber et al., 2014). As such, our proposal basically adds that experiential integration into a unified subjective feeling can occur even when the sources of processing experiences originate in different sensory modalities (e.g., quality signals from auditory processes can combine with quality signals from visual processes).
Importantly, our proposal assumes that experiential signals of processing quality can originate in processing of abstract, conceptual material, and extend beyond fluency (sense of ease) and familiarity (sense of oldness) to “structural experiences,” such as a sense of coherence, integrity, or rightness (Whittlesea, 2002). One example comes from research using artificial grammars and shows that decisions about grammaticality in one modality are influenced by previously learned grammatical rules in another modality, and that this influence involves non-analytical processes (Dienes et al., 2011). Importantly, this effect may not involve a feeling of fluency or familiarity, but rather a sense of structural coherence (Scott and Dienes, 2010). Stressing the breadth of such effects, our recent research shows that decisions about patterns in one modality can be influenced by the coherence of completely unrelated patterns from another modality (Ziembowicz et al., 2013). Let us elaborate as this research illustrates our core argument. In three experiments participants judged targets in one sensory modality while being incidentally exposed to regular or irregular background stimuli from a different modality. For example, targets were auditory melodies and backgrounds were visual figures, or vice versa. Critically, the specific regularity of targets and backgrounds was unrelated—auditory regularity was tone sequence grammar, visual regularity was 3D realizability. We explored the effect of cross-modal coherence with different types of subjective judgments: “regularity” (Experiment 1), “familiarity” (Experiment 2), and “possibility” (Experiment 3). All three experiments showed similar results: the coherence of the background stimulus influenced the target judgment, regardless of judgment type and target modality. That is, visual and auditory targets were judged as more “regular,” “familiar,” and “possible” when the incidental cross-modal backgrounds were coherent.
What are the implications of such findings? As mentioned, the standard explanation of cross-modal phenomena assumes changes in representation of stimulus features, whether driven by perceptual processes or decisional processes that integrate cues from different modalities. In contrast, we argue that cross-modal influences also reflect integration at the level of processing experiences. We admit the need for direct evidence that the just discussed cross-modal studies (including Ziembowicz et al., 2013) involve changes in subjective experiences and that their integration is causally responsible for the obtained behavioral effects. However, there is good evidence that related phenomena do involve “experiences”–i.e., cognitive or affective feelings. First, participants in many (though not all) experiments actually report changes in the feeling of “ease,” “effort,” “familiarity” or “regularity” associated with processing (Schwarz, 2015). Second, various physiological measures pick up indicators of changes in experience, such as positivity associated with fluent processing (e.g., Winkielman et al., 2003, 2012). Third, many experiments show that “bleed-over” or “misattribution” effects vanish once a person is provided with an explanation targeting subjective experience, not unlike classic studies on misattribution and discounting of affect or arousal (e.g., Dutton and Aron, 1974). For example in the previously mentioned cross-format study of Fazendeiro et al. (2005), participants were asked to recognize (old/new) words and pictures, some of which appeared earlier as related cross-format stimuli (essentially serving as semantic primes). During this recognition task, background music was played, which for some participants was explained as influencing their “sense of familiarity.” In this condition, participants showed reduced false recognition judgments for the cross-format stimuli, presumably reflecting their discounting of familiarity experience. Additional evidence for the notion that participants consciously experience changes in processing quality comes from research on hidden semantic coherence and the intuitive basis of such judgments (Topolinski and Strack, 2009a,b). Interestingly, this work shows that participants cannot report and re-attribute changing levels of fluency (facilitation due to semantic coherence) but are only aware of affective (hedonic) consequences of changed fluency. This suggests that what specifically is “felt” about objective processing quality varies depending on the details of the task. Finally, the just mentioned studies again highlight that the integration at the level of subjective experience interacts with high-level decisional processes. That is, the exact impact of experience on stimulus judgments depends on the perceiver's beliefs about the sources and relevance of the experience for the task at hand (Schwarz, 2015).
Neuroscientifically, our “joint quality signal” explanation for cross-modal integration matches evidence for global conflict signals or global prediction error (e.g., Fernandez-Duque et al., 2000; Friston, 2010; Shackman et al., 2011; Botvinick and Braver, 2015). Computationally, our account fits with connectionist models using global signals of processing quality (Lewenstein and Nowak, 1989; Norman and O'Reilly, 2003; Cleeremans and Dienes, 2008). Critically, these signals are non-specific, with different sources of coherence, ease, or familiarity generating a similar signal. Further, these signals are free-floating—not tightly bound to the original representation, and thus transferable across contents. Still, the signals are useful. They highlight abstract correspondences across patterns (e.g., regularity). They also regulate the network's own behavior, terminating the recognition process (preventing pattern discovery) when coherence is low and letting recognition continue when coherence is high (Rychwalska et al., 2005). The specifics of the mechanisms can be illustrated using a model by Lewenstein and Nowak (1989). It is a Hopfield type neural network enhanced with a mechanism allowing the network to control its own processing dynamics. The controlling system is implemented as a feedback loop that draws on one of a set of parameters: coherence, volatility, signal strength, etc. Based on the momentary values of this “order parameter,” the system can distinguish between known and unknown stimuli, but also react differently to primed, prototypical, regular, coherent, and distorted material. This model applies well to behavioral data, as seen in simulations of the mere exposure effect, which involves changes in fluency (Drogosz and Nowak, 2006). Consistent with behavioral data, the network reproduces the asymmetrical effect of “mere-exposed” stimuli on non-analytic, implicit, fluency-dependent judgments (preferences, familiarity) and analytic, explicit memory judgments. That is, the network results show that implicit measures of recognition (using the dynamic order parameter) can be faster than explicit measures, recreating a paradoxical phenomenon of somehow “knowing” the valence or familiarity of a stimulus before actually recognizing it.
In sum, we propose that some cross-modal phenomena involve integration via common experiences, including fluency, familiarity, and coherence, grounded in global signals about network dynamics. As a result, even when the modal origins of such signals differ, individuals experience integrated feelings of processing quality. Such feelings can then enter meta-cognitive processes and inform fundamental cognitive and social judgments (Winkielman and Schooler, 2011; Schwarz, 2015). Future research may explore cross-modal influences on experience-based judgments (risk, frequency, truth, fame, beauty, etc.). It should also determine when such effects are pre- and post-decisional. One question in this regard concerns the level at which processing signals are combined. It could be pre-experiential (e.g., fluency signals could blend before any experience) or experiential (e.g., with one already blended, or two blendable feeling signals appearing in the experience). A related question is whether experiences from different sources are genuinely fused (i.e., their origin information is lost) or potentially separable. Research should also explore the specificity of experiences. That is, sometimes experiences act broadly, allowing for conflation of drastically different inputs such as physical arousal with familiarity (Goldinger and Hansen, 2005) or physical effort with retrieval difficulty (Stepper and Strack, 1993). But, individual processing experiences are also unique in subjective quality (e.g., feelings of coherence differ from familiarity or ease, not unlike different emotions). This should constrain possible experiential fusion (genuine blending) and judgmental misattributions (source errors).
In conclusion, it appears that the creation of a unified consciousness is facilitated by an experiential mechanism that combines signals of processing quality. This mechanism links diverse contents in the mind and allows people to experience the multi-modal world as integrated—though also sometimes as more (or less) unified than it actually is.
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Acknowledgments
Andrzej Nowak was partially supported by Polish National Science Centre (project no. DEC-2011/02/A/HS6/00231). Piotr Winkielman was supported by the UCSD Academic Senate Bridge Grant. We thank Evan Carr, Liam Kavanagh, Robert St. Louis, and Shlomi Sher for helpful comments.
References
Baars, B. J. (2005). Global workspace theory of consciousness: toward a cognitive neuroscience of human experience. Prog. Brain Res. 150, 45–53. doi: 10.1016/S0079-6123(05)50004-9
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Botvinick, M., and Braver, T. (2015). Motivation and cognitive control: from behavior to neural mechanisms. Annu. Rev. Psychol. 66, 83–113. doi: 10.1146/annurev-psych-010814-015044
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Cleeremans, A., and Dienes, Z. (2008). “Computational models of implicit learning,” in Handbook of Computational Cognitive Modeling, ed R. Sun (Cambridge: University Press), 396–421.
Curran, T., and Dien, J. (2003). Differentiating amodal familiarity from modality-specific memory processes: an ERP study. Psychophysiology 40, 979–988. doi: 10.1111/1469-8986.00116
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Dienes, Z., Scott, R. B., and Wan, L. L. (2011). “The role of familiarity in implicit learning,” in Constructions of Remembering and Metacognition: Essays in honour of Bruce Whittlesea, eds P. Higham and J. Leboe (New York, NY: Algrave Macmillan), 51–62.
Drogosz, M., and Nowak, A. (2006). A neural model of mere exposure: the EXAC mechanism. Polish Psychol. Bull. 37, 7–15.
Dutton, D. G., and Aron, A. P. (1974). Some evidence for heightened sexual attraction under conditions of high anxiety. J. Pers. Soc. Psychol. 30, 510–517. doi: 10.1037/h0037031
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Fazendeiro, T., Winkielman, P., Luo, C., and Lorah, C. (2005). False recognition across meaning, language, and stimulus format: conceptual relatedness and the feeling of familiarity. Mem. Cogn. 33, 249–260. doi: 10.3758/BF03195314
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Fernandez-Duque, D., Baird, J. A., and Posner, M. I. (2000). Executive attention and metacognitive regulation. Conscious. Cogn. 9, 288–307. doi: 10.1006/ccog.2000.0447
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Friston, K. (2010). The free-energy principle: a unified brain theory? Nat. Rev. Neurosci. 11, 127–138. doi: 10.1038/nrn2787
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Goldinger, S. D., and Hansen, W. A. (2005). Remembering by the seat of your pants. Psychol. Sci. 16, 525–529. doi: 10.1111/j.0956-7976.2005.01569.x
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Hillis, J. M., Ernst, M. O., Banks, M. S., and Landy, M. S. (2002). Combining sensory information: mandatory fusion within, but not between, senses. Science 298, 627–1630. doi: 10.1126/science.1075396
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Jacoby, L. L., Allan, L. G., Collins, J. C., and Larwill, L. K. (1988). Memory influences subjective experience: noise judgments. J. Exp. Psychol. Learn. Mem. Cogn. 14, 240–247. doi: 10.1037/0278-7393.14.2.240
Lewenstein, M., and Nowak, A. (1989). Recognition with self-control in neural networks. Phys. Rev. 40, 4652–4664. doi: 10.1103/PhysRevA.40.4652
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Mangan, B. (1993). Taking phenomenology seriously: the “fringe” and its implications for cognitive research. Conscious. Cogn. 2, 89–108. doi: 10.1006/ccog.1993.1008
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
McGurk, H., and MacDonald, J. (1976). Hearing lips and seeing voices. Nature 264, 746–748. doi: 10.1038/264746a0
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Miller, J. K., Lloyd, M. E., and Westerman, D. L. (2008). When does modality matter? Perceptual versus conceptual fluency-based illusions in memory. J. Mem. Lang. 58, 1080–1094. doi: 10.1016/j.jml.2007.12.006
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Mroczko-Wąsowicz, A., and Nikolic, D. (2014). Semantic mechanisms may be responsible for developing synesthesia. Front. Hum. Neurosci. 8:509. doi: 10.3389/fnhum.2014.00509
Norman, K. A., and O'Reilly, R. C. (2003). Modeling hippocampal and neocortical contributions to recognition memory: a complementary learning systems approach. Psychol. Rev. 110, 611–646. doi: 10.1037/0033-295X.110.4.611
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Ramachandran, V. S., and Hubbard, E. M. (2001). Synaesthesia—A window into perception, thought and language. J. Conscious. Stud. 8, 3–34.
Reber, R., Christensen, B., and Meier, B. (2014). Effects of meaning and symmetry on judgments of size: an amount of information account. Front. Psychol. 5:1270. doi: 10.3389/fpsyg.2014.01270
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Reber, R., Fazendeiro, T. A., and Winkielman, P. (2002). Processing fluency as the source of experiences at the fringe of consciousness: commentary on Mangan. Psyche 8.
Reber, R., Wurtz, P., and Zimmermann, T. D. (2004). Exploring “fringe” consciousness: The subjective experience of perceptual fluency and its objective bases. Conscious. Cogn. 13, 47–60. doi: 10.1016/S1053-8100(03)00049-7
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Rychwalska, A., JabJabłski, P., Zochowski, M., and Nowak, A. (2005). Novelty based feedback regulation in artificial neural networks. Acta Neurobiol. Exp. 65, 453–463.
Schwarz, N. (2015). “Metacognition,” in APA Handbook of Personality and Social Psychology: Attitudes and Social Cognition, eds E. Borgida and J. A. Bargh (Washington, DC: APA), 203–229.
Scott, R. B., and Dienes, Z. (2010). Fluency does not express implicit knowledge of artificial grammars. Cognition 114, 372–388. doi: 10.1016/j.cognition.2009.10.010
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Shackman, A. J., Salomons, T. V., Slagter, H. A., Fox, A. S., Winter, J. J., and Davidson, R. J. (2011). The integration of negative affect, pain and cognitive control in the cingulate cortex. Nat. Rev. Neurosci. 12, 154–167. doi: 10.1038/nrn2994
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Shams, L., Kamitani, Y., and Shimojo, S. (2000). Illusions. What you see is what you hear. Nature 408, 788. doi: 10.1038/35048669
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Spence, C. (2011). Crossmodal correspondences: a tutorial review. Atten. Percept. Psychophys. 73, 971–995. doi: 10.3758/s13414-010-0073-7
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Stepper, S., and Strack, F. (1993). Proprioceptive determinants of emotional and nonemotional feelings. J. Pers. Soc. Psychol. 64, 211–220 doi: 10.1037/0022-3514.64.2.211
Topolinski, S., and Strack, F. (2009a). Scanning the “fringe” of consciousness: what is felt and what is not felt in intuitions about semantic coherence. Conscious. Cogn. 18, 608–618. doi: 10.1016/j.concog.2008.06.002
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Topolinski, S., and Strack, F. (2009b). The analysis of intuition: processing fluency and affect in judgements of semantic coherence. Cogn. Emot. 23, 1465–1503. doi: 10.1080/02699930802420745
Whittlesea, B. W. A., Jacoby, L. L., and Girard, K. (1990). Illusions of immediate memory: Evidence of an attributional basis for feelings of familiarity and perceptual quality. J. Mem. Lang. 29, 716–732. doi: 10.1016/0749-596X(90)90045-2
Whittlesea, B. W. A. (2002). “On the construction of behavior and subjective experience: the production and evaluation of performance,” in Rethinking Implicit Memory, eds J. Bowers and C. Marsolek (Oxford: Oxford University Press), 239–260.
Winkielman, P., Huber, D. E., Kavanagh, L., and Schwarz, N. (2012). “Fluency of consistency: when thoughts fit nicely and flow smoothly” in Cognitive consistency: A Fundamental Principle in Social Cognition, eds B. Gawronski and F. Strack (New York, NY: Guilford Press), 89–111
Winkielman, P., and Schooler, J. W. (2011). Splitting consciousness: unconscious, conscious, and metaconscious processes in social cognition. Eur. Rev. Soc. Psychol. 22, 1–35. doi: 10.1080/10463283.2011.576580
Winkielman, P., Schwarz, N., Fazendeiro, T., and Reber, R. (2003). “The hedonic marking of processing fluency: implications for evaluative judgment,” in The Psychology of Evaluation: Affective Processes in Cognition and Emotion, eds J. Musch and K. C. Klauer (Mahwah, NJ: Lawrence Erlbaum), 189–217.
Wurtz, P., Reber, R., and Zimmermann, T. D. (2008). The feeling of fluent perception: a single experience from multiple asynchronous sources. Conscious. Cogn. 17, 171–184. doi: 10.1016/j.concog.2007.07.001
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Ziembowicz, M., Nowak, A., and Winkielman, P. (2013). When sounds look right and images sound correct: cross-modal coherence enhances claims of pattern presence. Cognition 2, 273–278. doi: 10.1016/j.cognition.2013.07.010
Pubmed Abstract | Pubmed Full Text | CrossRef Full Text | Google Scholar
Keywords: consciousness, coherence, fluency, subjective experience, cross-modal perception
Citation: Winkielman P, Ziembowicz M and Nowak A (2015) The coherent and fluent mind: how unified consciousness is constructed from cross-modal inputs via integrated processing experiences. Front. Psychol. 6:83. doi: 10.3389/fpsyg.2015.00083
Received: 01 October 2014; Accepted: 15 January 2015;
Published online: 09 February 2015.
Edited by:
Aleksandra Mroczko-Wasowicz, National Yang Ming University, TaiwanReviewed by:
Sascha Topolinski, University of Cologne, GermanyTeresa Garcia-Marques, Instituto Superior de Psicologia Aplicada-Instituto Universitário, Portugal
Copyright © 2015 Winkielman, Ziembowicz and Nowak. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: pwinkiel@ucsd.edu