1
Department of Psychology and Interdisciplinary Center for Neural Computation, The Hebrew University of Jerusalem, Jerusalem, Israel
2
Department of Psychology and Helen Wills Neuroscience Institute, University of California, Berkeley, CA, USA
3
Department of Psychology, University of California, San Diego, CA, USA
4
Loewenstein Rehabilitation Hospital and Sackler faculty of Medicine, Tel Aviv University, Tel Aviv, Israel
The formation of coherent percepts requires grouping together spatio-temporally disparate sensory inputs. Two major questions arise: (1) is awareness necessary for this process; and (2) can non-conscious elements of the sensory input be grouped into a conscious percept? To address this question, we tested two patients suffering from severe left auditory extinction following right hemisphere damage. In extinction, patients are unaware of the presence of left side stimuli when they are presented simultaneously with right side stimuli. We used the quotidnscale illusionquotidn to test whether extinguished tones on the left can be incorporated into the content of conscious awareness. In the scale illusion, healthy listeners obtain the illusion of distinct melodies, which are the result of grouping of information from both ears into illusory auditory streams. We show that the two patients were susceptible to the scale illusion while being consciously unaware of the stimuli presented on their left. This suggests that awareness is not necessary for auditory grouping and non-conscious elements can be incorporated into a conscious percept.
The syndrome of unilateral neglect (UN) is a frequent consequence of right hemisphere damage. At the core of UN is the failure of patients to be consciously aware of sensory events (real or imagined) on their left. Patients fail to consistently orient towards, act upon or report these events. The presence of UN is one of the major impediments to the rehabilitation of patients after right hemisphere stroke (Katz et al., 1999
). Extinction is a related condition in which this failure to detect events occurs when the left side stimuli are presented simultaneously with right sided stimuli. From the point of view of human cognitive neuroscience, UN and extinction have been central to the investigation of conscious awareness and attention. Since UN and extinction have been dissociated from primary sensory deficits like hemianopia (Halligan et al., 1990
), they open a window onto the role of conscious awareness in cognitive processes (Deouell, 2002
; Driver and Vuilleumier, 2001
).
A pivotal question in the study of UN and extinction is where in the processing chain neglected or extinguished stimuli are blocked and denied access to conscious awareness, and why this happens. Since the seminal work of Volpe and colleagues (1979)
a multitude of experimental paradigms have been used to show that visual stimuli, to which the patient is unaware, are nevertheless processed to a considerable extent, and influence behaviour (for a comprehensive review see Driver and Vuilleumier, 2001
). In general, the processing of the extinguished stimulus was demonstrated by 3 methods. First, performance (e.g., reaction time) related to the non-extinguished right sided stimuli, or to a subsequent central stimulus, could be influenced by the information conveyed by the extinguished stimulus. For example, the categorization of a right side object was faster if it was accompanied with a same-category than a different category object on the left, even if the latter was extinguished (Berti and Rizzolatti, 1992
; Morein-Zamir et al., 2005
). Similarly, extinguished visual stimuli induced semantic priming in a subsequent lexical decision on central stimuli (McGlinchey-Beroth et al., 1993
). Second, the information conveyed by the contralesional stimulus could affect the probability of its own extinction. For example, the chances of two simple stimuli like line gratings or geometric shapes to be both detected depends on whether the two stimuli are co-linear, or have the same shape or color (Gilchrist et al., 1996
; Pavlovskaya et al., 1997
), effects that were associated with grouping. At a higher level of object processing, salient stimuli like faces or spiders have higher chances than neutral stimuli to be detected (Vuilleumier and Sagiv, 2001
; Vuilleumier and Schwartz, 2001
). Third, extinguished stimuli have been shown to elicit a response at specific brain circuits, suggesting processing without awareness (Vuilleumier et al., 2001
).
Whereas UN and extinction are most commonly investigated in the visual modality, they may be manifested in other modalities as well (for recent reviews see Brozzoli et al., 2006
; Pavani et al., 2003
, 2004
), although not necessarily to the same extent (e.g., Sinnett et al., 2007
). In audition, patients may have difficulty to localize detected stimuli accurately in the left space (e.g., Bisiach et al., 1984
; Pavani et al., 2002
; cf. Deouell et al., 2000
) and they may have problems detecting or identifying stimuli presented on their left either singly or in an extinction situation (Soroker et al., 1997
; see Pavani et al., 2003
, 2004
, for reviews). However, in contrast to the many demonstrations of implicit processing of extinguished visual stimuli, and some in the tactile modality (Berti et al., 1999
; Maravita, 1997
) little is known on the fate of extinguished auditory stimuli. In one study, spoken syllables presented on the left (otherwise extinguished) side merged with lip movement of incongruent syllables which were presented visually on the right (McGurk illusion, McGurk and MacDonald, 1976
), creating illusory blends at the same rate as in normal subjects (Soroker et al., 1995a
). Here, we directly examined the fate of extinguished stimuli in two patients using simple tones, exploiting a dramatic auditory illusion, known as the ‘Scale Illusion’ (Deutsch, 1975
). We show that tones from the extinguished side were incorporated into an auditory stream of which the patients were aware.
Patients
The 2 patients reported here were selected from a larger cohort of UN patients, because they had severe enough auditory extinction, so that when the illusion-inducing sequence was presented to them several times, they reported only right ear stimulation for several consecutive trials starting from the first trial
[1
.
JF is a 71-year-old U.S. navy veteran who suffered a right hemisphere infarction 5 months prior to the tests described here. He was admitted for rehabilitation at the Center for Rehabilitation and Extended Care at Martinez, California, where testing took place. A whole brain three dimensional T1 and T2 weighted MRI scans performed at the chronic stage revealed a lesion in the right inferior and part of the superior parietal lobe, extending partially into the frontal lobe. The temporal lobe, including the superior temporal plane, was spared (Figure 1
). On neurological examination he was fully orientated and cooperative. He had left side motor weakness and somatosensory impairment. His visual fields were intact. Pure tone audiometry revealed bilaterally symmetrical high frequency reduction, well beyond the frequencies used in our test (261–523 Hz). On bed-side examination, JF detected most unilateral visual and auditory stimuli, but failed to detect most auditory and visual stimuli presented to his left when those were presented simultaneously with a stimulus on his right (extinction). Neuropsychological testing including cancellation tasks, line bisection and figure drawing revealed left visuo-spatial neglect.
Figure 1. Left: Representative CT slices of patient MA (top row) and T1 MRI slices of patient JF (bottom row). The right side of the brain is on the right. The affected area is marked by a red line. Right: A lateral view of the cortical lesion projected onto a normal brain. For this reconstruction, the lesions were drawn manually using MRIcrosoftware (http://www.mricro.com) on slices of a T1-weighted single subject template MRI scan from the Montreal Neurological Institute (www.bic.mni.mcgill.ca/cgi/icbm_view), distributed with MRIcro. In JF, the right parietal and posterior frontal cortex is affected, but the superior temporal plane is spared (arrow pointing to transverse temporal gyri). A small white matter extension is seen in the right frontal lobe. In MA a more extensive right MCA lesion affected the inferior parietal lobe, superior and middle temporal gyri, and inferior frontal regions, as well subcortical structures.
MA is a 75-year-old retired Israeli pharmacist, who suffered an ischemic stroke 2 months prior to the testing. He was hospitalized for rehabilitation at the Loewenstein Rehabilitation Hospital in Israel where testing took place. CT scans taken ∼6 weeks following stroke revealed an extensive stroke in the territory of the middle cerebral artery (Figure 1
). On examination, he was fully oriented and cooperative. He had severe left side hemiparesis and hemihypoesthesia. There was no visual field defect on confrontation, but left visual extinction was present during bilateral simultaneous stimulation. Standard neuropsychological assessment for visual neglect including the Behavioral Inattention Test (BIT; Wilson et al., 1987
), incorporating cancellation tests, line bisection, figure drawing and copying, revealed severe deficits in his performance on the left relative to the right (BIT score – 50/146, cutoff score for normal performance 129). MA missed the left stimulus on most bilateral simultaneous auditory stimulation trials (>80% for auditory clicks). He also occasionally missed unilateral left side clicks (30%), but detected all right side stimuli.
In the case of JF, the procedures were approved by the institutional review board of the University of California, Berkeley, and VA Northern California Health Care System. Prior to the beginning of the tests, JF gave his informed consent to participate, after the nature of the study was explained. He also gave his written consent to publish his pictures and video movie for scientific and educational purposes. In the case of MA, testing was approved by the local Helsinki committee for the protection of human subjects at the Loewenstein Rehabilitation Hospital. The patient signed an informed consent after the nature of the tests was explained and agreed to be videotaped on the condition that this videotape will not become public.
Procedure
Scale illusion. In this sequence (Figure 2
), a dichotic tonal sequence consisting of the repetitive presentation of the C major scale is presented with successive tones alternating between ears. The scale is presented simultaneously in both ascending and descending form, such that when a component of the ascending scale is in one ear, a component of the descending scale is in the other, and vice versa. Thus, when each channel is presented alone no scale pattern is perceived (Figure 2
A; Audio S2, S3 in supplementary material at http://pissaro.soc.huji.ac.il/∼leon/Scale/). Nevertheless, when both channels are presented together, healthy listeners obtain the illusion of distinct melodies, consisting of the higher and lower halves of a musical scale, ascending and descending in pitch (Figure 2
B). This percept requires grouping of information from both ears into illusory auditory streams (Bregman, 1990
; Deutsch, 1975
). The stimuli used here were pure tones, matching the notes in Figure 2
. Each tone was 250 ms long, with no gap between tones. Sequences were saved as WAV files with sampling rate of 44.1 KHz/16 bit resolution, and were presented using a laptop computer, through circumaural headphones, at an intensity of ∼53 Db SPL. This low intensity eliminated the possibility of audible cross talk between channels. The demonstrations included with the supplementary material were down-sampled to reduce file size. For the spontaneous report (see below) the sequences were 20 seconds long. For the multiple-choice test, the sequence was presented for about 10 seconds, after which the samples were presented in random order. Each sample was 4 seconds long.
Figure 2. The scale illusion. (A) Repeating sequences of tones are presented to the right and left ears simultaneously. Dashed lines with open arrowheads connect notes which belong to an ascending scale, whereas dotted lines with closed arrowheads connect notes which belong to the descending form of the same scale. The two scales alternate between the two ears. (B) Normal listeners group notes from the left (blue) and right (red) ear to perceive one or two illusory ‘melodies’, each consisting of the higher or lower half of the scale; they typically ascribe each melody to a different ear. (C) Movie frames from Movie S1 (supplementary material), showing the patient indicating with his finger the perceived pitch of each tone as he listens to the sequences shown in (A). The frames were taken at points where the finger paused. The sequence resembles the illusory percept shown in (B).
Due to the nature of the test, and the need to obtain an assessment of the patient’s subjective perception, a somewhat different scheme was used for each patient, tailored to the patient’s ability to reproduce his percept. The number of trials also depended on the patient’s ability to perform the task repeatedly. Note that even in healthy individuals it is not possible to repeat the test too many times, as after certain amount of exposure, the illusion often breaks down (Diana Deutsch, unpublished observation). Each patient was initially presented with the sequence normally producing the scale illusion as described above and asked to freely report what he heard, and whether he heard tones on his left, right, or on both sides. Since both patients had difficulty singing, we sought other means for deducing the patient’s percept, and encouraged gesturing with the hands. In the case of patient MA, we also asked him to draw a line based on his percept. Next, we used a matching procedure to try and assess the patient’s percept. To this end, we presented the dichotic sequence producing the illusion, and followed it by 4 shorter diotic (i.e., same stimulus in both ears) sequences, which reflected one of 4 possible percepts: (a) the input to the left ear, (b) the input to the right ear, (c) illusory high pitch melody (Figure 2
B) and (d) the illusory low pitch melody. Several such trials were presented. The order of these presentations was altered from trial to trial. The patient had to report which sequence most closely matched his percept. The patients showed no tendency to select a particular sample position (e.g., the first or the last sample).
For the χ2 test used to analyze patient’s JF data, we computed the expected frequency of illusory perception based on the null hypothesis that the illusory percept depends on consciously perceiving both channels. Since the patient was aware of the presence of stimuli in both channels in at most 4/14 bilateral (he was sure only once), the expected illusion rate was at most 3 out of 10. Yates correction was used.
Patient JF
The test started with two presentations of the illusion sequence, in which JF was asked to report the percept as best he could. He was encouraged to hum or gesture. On two of two of these initial presentations JF reported hearing sounds only in his right ear (i.e., demonstrating left extinction), and at the same time his responses (humming and gesturing; Figure 2
C and Movie S1 in v) clearly indicated that he was perceiving the ‘illusory percept’ of half a scale ascending and descending in pitch as shown in Figure 2
B, a normal percept which requires grouping of tones from the two ears. In an attempt to get a better assessment of what JF was hearing, we asked him to match it, in a multiple-choice test, to samples of the 4 musical scores depicted in Figures 2
A and 2
B, each presented in turn to both ears in unison (diotically), without asking for localization. JF selected an illusory scale on 5 out of 5 trials, although on the last trial he stated that his perception could also match the sequence presented originally to the right ear. Thus, JF experienced the illusion in all cases. Next, we assessed JF’s extinction level with the same stimuli, using unilateral and bilateral presentations, asking only for localization of sounds as right, left, or both. He localized correctly 8 of 8 unilateral presentations, extinguished the left side in 5 out of 9 bilateral presentations, the right side in 1 of 9, and was unsure on 3 bilateral trials. Last, we presented the dichotic stimulus again, and asked JF to localize the sounds, as well as to match his perception to one of the samples as before. JF selected the illusory percept in 3 of 5 trials. In 2 of these, he reported hearing sounds only on his right. All tests considered, JF perceived the illusion in 8 out of 10 trials, significantly more than expected by his rate of extinction (10 out of 14 trials; χ2 = 12.54, p < 0.001). JF was therefore susceptible to the scale illusion, requiring the integration of stimuli from the left and the right ear into a cohesive auditory stream, while being unaware of the stimuli presented to his left ear.
Patient MA
We first presented the illusion 4 times and asked the patient to report, as best he could, what he heard. His singing was quite poor, but gesturing was suggestive of a tune going up and down in all cases. To further assess this impression, he was asked to draw the melodic contour as best he could during the last of these presentations (Figure 3
). In all 4 repetitions, the patient indicated that he heard sounds on his left only. On the next 4 trials the multiple choice test was given. The patient chose twice the low illusion and twice the high illusion diotic simulation, and in all cases indicated that he heard sounds on his right only. At this point the test had to be stopped due to patient fatigue. In summary, the patient extinguished the left side on 8 out of 8 consecutive trials, starting from trial 1, and in all cases reported precepts matching the scale illusion. Like JF, he seems therefore susceptible to the scale illusion.
Figure 3. MA’s depiction of his perception upon presentation of repetitions of the illusion inducing sequence. The patient depicted a repeating smooth contour, consistent with the judgments he made on multiple choice testing.
Whereas numerous studies provided evidence for implicit processing of extinguished visual stimuli, equivalent reports from the auditory modality are rare (but see Deouell and Soroker, 2000
; Soroker et al., 1995a
). We reported here the observation of two patients with left auditory extinction, who were susceptible to a normal illusion which results from grouping and integration of sounds from the left and right ears. The two patients were chosen for having significant auditory extinction, so that upon the first trials in which the illusion-inducing sequence was presented, they demonstrated left extinction, yet showed perception of the illusory tune. Thus, the patients’ perception of the illusion in these initial trials could not have been biased by the perception of the illusion in trials in which no extinction was demonstrated. These results suggest that acoustic information presented on the spatially extinguished side can nonetheless find its way into conscious awareness.
To make sense of the auditory input and allow goal-directed action towards auditory sources, the auditory system has to segregate sounds into separate streams in the process of auditory scene analysis (Bregman, 1990
). The patients’ perception of the scale illusion provides unique direct support to the claim (Alain and Arnott, 2000
; Bregman, 1990
; Macken et al., 2003
) that streaming processes in audition are pre-attentive, and independent of conscious awareness (but see Carlyon et al., 2001
[2
). Moreover, it shows that when stimuli from an extinguished spatial location are segregated into one stream with non-extinguished stimuli, they may be explicitly reported. Previously it has been shown in patients with visual extinction that conditions facilitating perceptual grouping of stimuli presented simultaneously on the right and on the left of patients helps patients consciously detect elements on the left side, which would otherwise be extinguished. In these studies, grouping (in the general sense of the term) has been achieved by diverse mechanisms such as connecting stimuli on the right and on the left with a line (Humphreys, 1999
), using Gestalt principals such as collinearity (Gilchrist et al., 1996
; Humphreys, 1999
; Pavlovskaya et al., 1997
), symmetry (Ward et al., 1994
) and contrast similarity (Gilchrist et al., 1996
; Pavlovskaya et al., 2000
), by induction of subjective modal completion (e.g., Kaniza-type figures) and amodal completion (e.g., when the middle part of a bar seems to be hidden behind an occluder) between left and right stimuli (Mattingley et al., 1997
), or even by priming two abstract stimuli to be perceived as eyes in a face (Vuilleumier and Sagiv, 2001
). While some of these effects may rely on low level mechanisms such as preserved long-range lateral interactions within primary visual cortex (Pavlovskaya et al., 1997
, 2000
;Polat, 1999
; Polat and Sagi, 1993
) or on the integrity of visual neurons which are responsive to illusory contours (Driver and Mattingley, 1998
; Hirsch et al., 1995
; von der Heydt et al., 1984
), the diversity of manipulations leading to this effect suggest a more general mechanism. The present finding, from another modality, demonstrates the generality of this effect, by showing that it is not limited to the visual domain.
However, important differences exist between the grouping processes in vision and in audition. In vision, grouping processes link separate elements into a construct which is extended in space. In contrast, a perceived auditory stream, especially of the type studied here, is continuous in time, and tends to be collapsed onto a single spatial source, in effect overriding the veridical spatial sources of the sounds (Bregman, 1990
; Deutsch, 1975
). This may explain why in vision, presenting two stimuli in a form that enhances grouping reduces the rate of extinction (i.e., the patients become aware of the left sided stimulus being on the left), while JF and MA remained unaware of the spatial source of the information, despite the fact that information from the left channel was incorporated into a conscious percept by mixing it with right sided information.
Spatial localization is more tenuous in audition than in vision. In audition, space is not mapped directly on the receptor surface but has to be computed through inter-aural comparisons; spatial information is complicated by environmentally dependent reverberations; and in humans the pinnae cannot be separately moved to home in on a sound source. In addition, in audition, the information from both ears is transmitted to both hemispheres (although with contralateral predilection; Rosenzweig, 1951
). This may explain why translocation of the acoustic information to a new location in space is more likely in auditory than in visual perception (as in the case of ventriloquism for example; Bertelson and Aschersleben, 1998
). It may also explain why alloacusis, the pathological translocation of auditory information from the left towards the right, is one of the most ubiquitous finding in auditory neglect, when patients are tested with unilateral stimuli (Bisiach et al., 1984
; Soroker et al., 1997
; see Pavani et al., 2004
for review). Superior identification and detection than localization can be seen in perceptual tests of visual extinction as well (e.g., Baylis et al., 2001
; Vuilleumier and Rafal, 1999
), but whether these involve overt allochiria has not been established. Translocation of visual information from left to right (visual allochiria) has been described (Lepore, 2003
; Manly et al., 2002
; Toraldo, 2005
; Vallar, 2006
) but apparently these reports are mostly related to production (output) errors, such as drawing all the numerals of the clock face on the right hand, and repeated marking of right side stimuli on cancellation tasks. This makes it hard to disentangle intentional (motor) effects from perceptual effects (e.g., Lepore, 2003
; Manly et al., 2002
; Toraldo, 2005
; Vallar, 2006
).
The specific difficulty in localizing an event to the left is congruent with a previous report on auditory extinction (Deouell and Soroker, 2000
). When patients were tested in free field and were required to localize and report the identity of two syllables presented simultaneously one on their left and one on right, they were frequently unaware of the presence of a left side stimulus (reporting ‘right’ instead of ‘both sides’ in the localization task). Yet, in a significant number of trials in which this happened, they nevertheless reported the identity of the ‘extinguished’ stimulus, ascribing it to the right side. Thus, nominally extinguished information was overtly reported. We suggested that the location of left-sided events may fail to register, but the content of the information can reach awareness, provided that it can be linked to a location ‘tag’ on the right (Deouell, 2002
; Deouell and Soroker, 2000
). In the case of the scale illusion, dominance of pitch cues in auditory streaming encourages a misattribution of source location even in healthy individuals (Deutsch, 1975
), and in patients, rescues left side stimuli from oblivion.
The effect of spatial misattribution on auditory neglect was dramatically demonstrated previously by creating conditions which promote a ‘ventriloquism illusion’ (Bertelson and Aschersleben, 1998
): the presence of a visually salient dummy speaker on the right of the patients improved the identification rate of auditory stimuli (recorded syllables) that were emitted from a hidden left-sided real speaker (Calamaro et al., 1995
; Soroker et al., 1995b
). The problem with spatial localization on the left may be tracked to deficient pre-attentive encoding of spatial information, as demonstrated by the lack of the mismatch negativity (MMN), a 100–200 ms signature of non-intentional change detection, in response change in the location of sounds on the left (Deouell et al., 2000
).
In conclusion, the occurrence of the scale illusion in the presence of left extinction is a unique demonstration of processing of extinguished information in audition. The finding supports the evidence suggesting that a core deficit in unilateral neglect and extinction lies in the encoding of spatial attributes of sensory and imaginary events (Berti and Rizzolatti, 1992
; Bisiach and Berti, 1987
; Bisiach and Luzatti, 1978
). Without such encoding, awareness of the stimuli is compromised (Deouell, 2002
; Driver and Vuilleumier, 2001
).
The research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
This study was supported by Grant grant G-815-8.4/2004 from the Germany-Israel Foundation (GIF) to LYD, and NINDS grant NS21135 to RTK. We thank Shani Shalgi and Tamara B. Harisson for their her help in examining the patients.
Movie S1 and audio S2 and S3 can be found at http://pissaro.soc.huji.ac.il/∼leon/Scale/).
Footnotes
- ^ The rational was that once the patient reported a bilateral stimulation and obtained the illusion, it was harder to say whether the following responses, during extinction, were not biased in some way by this normal experience of the illusion.
- ^ Carlyon et al. (2001) reported that in patients with left neglect, streaming of sounds within the left ear stimuli, based on pitch, is deficient relative to streaming of right side stimuli, and suggested this as evidence that streaming requires attention. As reviewed by Macken et al., 2003, this result is not consistent with most of the literature. It is not clear why in Carlyon et al.’s patients streaming was lacking and in ours it was normal, but we note that the paradigms were very different. In their case, sounds in the two streams were sequential and appeared all in one ear. In the present case, sounds in the two streams appeared simultaneously at the two ears. It remains to be seen whether these factors or patient differences were responsible for the different results.