The Comprehension of Counterfactual Conditionals: Evidence From Eye-Tracking in the Visual World Paradigm

Orenes, Isabel; García-Madruga, Juan A.; Gómez-Veiga, Isabel; Espino, Orlando; Byrne, Ruth M. J.

doi:10.3389/fpsyg.2019.01172

ORIGINAL RESEARCH article

Front. Psychol., 14 June 2019

Sec. Psychology of Language

Volume 10 - 2019 | https://doi.org/10.3389/fpsyg.2019.01172

This article is part of the Research Topic The Role of Alternatives in Language View all 16 articles

The Comprehension of Counterfactual Conditionals: Evidence From Eye-Tracking in the Visual World Paradigm

$\r\nIsabel Orenes*$ Isabel Orenes^1*

Juan A. García-Madruga²

Isabel Gómez-Veiga²

Orlando Espino³

Ruth M. J. Byrne⁴

¹Department of Basic Psychology I, Universidad Nacional de Educación a Distancia (UNED), Madrid, Spain
²Department of Developmental and Educational Psychology (UNED), Madrid, Spain
³Department of Psychology, Universidad de La Laguna, Santa Cruz de Tenerife, Spain
⁴School of Psychology and Institute of Neuroscience, Trinity College Dublin, University of Dublin, Dublin, Ireland

Three experiments tracked participants’ eye-movements to examine the time course of comprehension of the dual meaning of counterfactuals, such as “if there had been oranges then there would have been pears.” Participants listened to conditionals while looking at images in the visual world paradigm, including an image of oranges and pears that corresponds to the counterfactual’s conjecture, and one of no oranges and no pears that corresponds to its presumed facts, to establish at what point in time they consider each one. The results revealed striking individual differences: some participants looked at the negative image and the affirmative one, and some only at the affirmative image. The first experiment showed that participants who looked at the negative image increased their fixation on it within half a second. The second experiment showed they do so even without explicit instructions, and the third showed they do so even for printed words.

Introduction

People often create counterfactual alternatives to reality in their everyday thoughts when they think “if only…” or “what if…” and imagine how a situation could have turned out differently (e.g., Kahneman and Tversky, 1982; Byrne, 2005). When people understand a counterfactual, such as “if there had been oranges then there would have been pears,” they appear to envisage two possibilities, the imagined alternative to reality that corresponds to the counterfactual’s conjecture, “there were oranges and pears” and the known or presumed facts that correspond to actual reality, “there were no oranges and no pears” (for a review see Byrne, 2016). In contrast, for a conditional in the indicative mood, such as “if there were oranges, then there were pears,” they tend to envisage just a single possibility at the outset, “there were oranges and pears” (e.g., Johnson-Laird and Byrne, 2002; Khemlani et al., 2018). Our aim is to examine the mental representations and cognitive processes that underpin the comprehension of counterfactuals.

Our starting point is the extensive evidence for the dual meaning of counterfactuals. To fully understand the meaning of a counterfactual, people must not only simulate the imagined alternative to reality that is conjectured in a counterfactual, they must also recover the presumed reality. But little is known about the cognitive processes that they rely on to do so (e.g., Johnson-Laird and Byrne, 1991; Byrne, 2005; Espino and Byrne, 2018). The accessibility of an imagined “possible world” from a representation of the real world poses difficulties (e.g., Stalnaker, 1968; Lewis, 1973), and what constitutes a “minimal change” is a slippery notion (e.g., Williamson, 2007; Kratzer, 2012). Nonetheless, people appear to readily recover the known or presumed facts when they understand a counterfactual. For example, participants tend to misremember a counterfactual, “if there had been oranges, then there would have been pears” and believe they were told instead, “there were no oranges and no pears” (Fillenbaum, 1974). They believe that someone uttering the counterfactual meant to imply this situation, and they judge that the items that best fit the description include this situation (e.g., Byrne and Tasso, 1999; Thompson and Byrne, 2002). Hence, the evidence indicates that they envisage the known or presumed facts, relying on their knowledge or on the cues of the subjunctive mood or content to do so.

People make more inferences that require access to “there were no oranges and no pears” from the counterfactual compared to the factual indicative conditional, such as modus tollens (from “there were no pears” to “therefore there were no oranges”). But they also make the same frequency of inferences that require access to “there were oranges and pears” from both conditionals, such as modus ponens – from “there were oranges” to “therefore there were pears” (e.g., Byrne and Tasso, 1999; Thompson and Byrne, 2002; see also Moreno-Ríos et al., 2008; Egan et al., 2009). They do so for various different sorts of content (e.g., Frosch and Byrne, 2012; see also Quelhas and Byrne, 2003; Egan and Byrne, 2012). Moreover, participants are primed to read, “there were no oranges and no pears” when they have first read the counterfactual, and they do so more quickly than when they have first read the factual conditional. But they also read, “there were oranges and pears” equally quickly from both sorts of conditional (e.g., Santamaría et al., 2005, see also Gómez-Veiga et al., 2010). Hence, the evidence indicates that people envisage both the imagined alternative to reality conjectured by the counterfactual, “there were oranges and there were pears,” and the actual reality, known or presumed by the counterfactual, “there were no oranges and there were no pears.” They keep track of their epistemic status as corresponding to real or imagined situations (Johnson-Laird and Byrne, 1991). The essence of the dual meaning of a counterfactual lies in this comparison of reality to an imagined alternative (e.g., Beck et al., 2006; Espino and Byrne, 2018). The question we wish to address is, at what point in their comprehension of a counterfactual do people detect the two messages of a counterfactual, that is, at what point do they envisage the conjecture, and at what point do they recover the presumed facts? The three experiments we report aim to advance knowledge of the comprehension of counterfactuals by establishing the point at which participants envisage each of the possibilities, during the temporal course of processing a counterfactual.

The question of when people envisage the situation corresponding to a counterfactual’s conjecture and when they envisage the presumed facts is important, first because some theories dispute that people consider both possibilities, and second, because the time at which people consider each possibility can provide a clue about the cognitive processes that they rely on to do so. Some online comprehension studies have been interpreted to support the idea that people represent both the conjecture and the presumed facts, and others have been interpreted to suggest that they represent only the conjecture. On the one hand, in eye-tracking studies it has been found that participants looked at a target word more quickly when it was presented in a context that was consistent with the real world rather than a counterfactual world. The result indicates an early and fleeting reading-time penalty that appears to reflect the construction of two representations (e.g., Ferguson and Sanford, 2008), although it may be sensitive to methodological factors (e.g., Ferguson et al., 2008, 2010; Ferguson, 2012; see also Stewart et al., 2009). Similarly, counterfactuals elicit greater brain activation, compared to factual conditionals, in areas related to conflict detection (e.g., Kulakova et al., 2013; see also Urrutia et al., 2012b). The results suggest that people represent both possibilities. But on the other hand, false counterfactuals elicit more brain activity than true ones, which may indicate the activation of only the conjecture (e.g., Nieuwland and Martin, 2012; see also Nieuwland, 2013; Kulakova and Nieuwland, 2016a,b). Accordingly, some theorists have proposed that people understand a counterfactual by considering their belief only in the imagined alternative to reality and they do so by simulating only the conjecture (e.g., Evans and Over, 2004; Evans, 2007). Hence, one view is that only the conjecture about an imagined alternative to reality is represented; another view is that both the conjecture and the presumed facts of reality are represented.

Even among theorists who propose that people consider both possibilities, there are disagreements. One theory is that the counterfactual conjecture, “there were oranges and pears” is more highly activated than the presumed facts, “there were no oranges and no pears” (e.g., Ferguson, 2012; Ferguson and Cane, 2015). An alternative theory is that when people understand a counterfactual, they first represent the presumed facts, “there were no oranges and no pears,” and the conjecture “there were oranges and pears,” although activated, does not contribute to discourse updating, is not semantically integrated, and does not remain in focus following a delay (e.g., De Vega et al., 2007; De Vega and Urrutia, 2012; Urrutia et al., 2012a). Hence, one view is that the conjecture is the more highly activated of the two possibilities, whereas another view is that the presumed facts are more highly activated. Our aim is to contribute to the resolution of these conflicting ideas.

We address a novel and nuanced question in our three experiments: if people envisage both the counterfactual’s conjecture about an imagined alternative to reality and its presumed facts, when do they do so? Our question is, at what point in the temporal process of comprehension do people consider each possibility? An answer to this question has the potential not only to distinguish between alternative theories of the comprehension of counterfactuals but also to shed light on the, at times, conflicting results of previous experiments, which have not been uniform in their choice of times at which to measure comprehension. Most importantly an answer to this question can contribute to understanding the nature of the cognitive processes by which people recover the presumed reality when they entertain the imagined alternative to reality conjectured by the counterfactual.

The three experiments we report rely on eye-tracking in the visual world paradigm to attempt to establish the time course during which people construct each possibility to understand a counterfactual. The visual world paradigm allows us to study the unfolding process of comprehending a counterfactual. In a typical visual world task, verbal and visual inputs are presented simultaneously while the participants’ eye movements are recorded to provide an index of real-time processing, sensitive to subtle aspects of language, attention, and memory (e.g., Allopenna et al., 1998; Rayner, 1998; Duñabeitia et al., 2009; Orenes et al., 2014, 2015). The logic of the method is that when something is heard, it is processed and attended to automatically; at the same time, if the corresponding object is visible, the eyes begin to move toward it (e.g., Cooper, 1974; Tanenhaus et al., 1995; for a review see Huettig et al., 2011a). It follows that when people listen to counterfactuals in a visual world paradigm, they will look more frequently at the most active information in working memory.

In the three experiments we report, participants heard short stories that contained an indicative factual conditional, e.g., “if there are oranges, then there are pears,” or a counterfactual, e.g., “if there had been oranges, then there would have been pears” (and we used a wide range of different contents, see the Supplementary Material A). In Experiment 1 and 2, four visual images related to the conditionals were shown on the screen, i.e., an image corresponding to the counterfactual’s conjecture, e.g., of oranges and pears, and an image corresponding to the presumed facts, e.g., of no oranges and no pears, as well as “distractor” images, i.e., an image corresponding to other sorts of fruit, such as apples and strawberries, and an image corresponding to no apples and no strawberries, as Figure 1 illustrates. The first two experiments differed in the instructions participants were given. In Experiment 1 participants were explicitly instructed to look at the objects on the screen that matched the meaning of the stories that they heard, in line with typical tasks in eye-tracking experiments, which encourage controlled, top-down processing. In Experiment 2 participants did not receive explicit instructions to look at the objects that matched the meaning of the stories, so that we could examine the processes underlying spontaneous counterfactual processing. In both experiments, we hypothesized that participants would look at the affirmative image, e.g., of oranges and pears, for indicative conditionals, whereas they would look at both the affirmative image and the negative image, e.g., of no oranges and no pears, for counterfactual conditionals. We anticipated that we would observe the same results with and without explicit instructions, notwithstanding an anticipated acceleration of processing of the counterfactual given explicit instructions. In Experiment 3, the same technique was used except that printed words were used instead of visual images, e.g., “oranges and pears.” We again hypothesized that participants would look at the affirmative words for indicative conditionals, whereas they would look at both the affirmative and negative words for counterfactual conditionals. We anticipated that we would observe the same results for printed words, notwithstanding once again an anticipated acceleration of processing of the counterfactual given printed words, since images can impede the comprehension of negation (e.g., Orenes and Santamaría, 2014). Our key predictions concern the temporal course of looking at the affirmative and negative images as a participant hears a counterfactual. If the recovery of the presumed reality is essential to understanding the meaning of a counterfactual, then we expect to observe a rapid increase in looking at the negative image as soon as participants detect – for example through the cues of the subjunctive mood – that the conditional conveys an imagined alternative to reality. In addition, if the essence of the dual meaning of a counterfactual is the comparison of reality to an imagined alternative, then we expect to observe that participants will maintain their gaze on both the negative and the affirmative image throughout the period of time measured.

FIGURE 1

Figure 1. Examples of the pictorial-based visual display used in Experiments 1 and 2 (on the left), and the word-based visual display used in Experiment 3 (on the right), for the counterfactual conditional “if there had been oranges then there would have been pears.”

We analyzed participants’ eye gaze at every 50 ms interval for a period of 4000 ms. One possibility is that a participant will look at only one image for a counterfactual during this time period, e.g., the affirmative image, and they will not look at the other three images. If so, the probabilities of fixations on the affirmative image will approximate 1, at every 50 ms “snapshot.” We anticipate this outcome to be the case for indicative conditionals. An important possibility is that participants may look at two images, the affirmative one and the negative one. Even if they rapidly and constantly switch their eye gaze from the affirmative image to the negative image and back, our snapshot of fixations every 50 ms will capture their gaze on one image or the other at that precise time. Hence if participants look entirely equally at both images on every trial for each counterfactual, moving their gaze from one image to the other, the probabilities of fixations on the affirmative image and on the negative image will each approximate 0.5. We anticipate this outcome to be the case for counterfactual conditionals. Of course, the same is true if a participant looks only at the affirmative image for a counterfactual on one trial, and only at the negative image for a different counterfactual on another trial. Hence, we examine not only group data but also individual data in our experiments. If participants look entirely equally at all 4 images, including the distractors, the probabilities of fixations on each one of them will each approximate 0.25, although this outcome is unlikely given that listeners tend to fixate objects that are mentioned or expected.

Experiment 1

The aim of the experiment was to study the temporal course of the comprehension of counterfactuals. The question we wished to ask was, when people understand a counterfactual conditional, such as, “if there had been oranges then there would have been pears,” at what point in the temporal course of processing do they focus on an image corresponding to the conjecture, e.g., “there were oranges and pears” and at what point do they focus on an image corresponding to the presumed facts, e.g., “there were no oranges and no pears.” We expect that participants will begin by looking at the affirmative image for both counterfactual and indicative conditionals, since the items in the affirmative image match what is mentioned in the conditionals, but we predict that participants will exhibit a rapid increase in looking at the negative image as soon as they detect that the conditional conveys an imagined alternative to reality. We also predict that participants will continue to look at both the negative and the affirmative image throughout the measured period of time.

Methods

Participants

The participants were 24 volunteers who were students at the University of La Laguna, Tenerife, Spain, and they participated in the experiment in exchange for course credits. There were 21 women and 3 men, and their average age was 20 years, with a range from 18 to 26 years. The participants were native Spanish speakers and they all reported normal vision or wore soft contact lenses or glasses.

Materials and Design

The design was a within-participants one and participants received vignettes in each of two conditions: indicative or counterfactual conditionals. They heard 36 vignettes about simple events (adapted from Santamaría et al., 2005), 18 trials in each of the two conditions, and the order of the trials was randomized.

The vignettes were presented to the participants in their native Spanish and started with an opening scene, e.g., “Maria went to the fruit shop to buy fruit to make a cake for Valentine’s Day. While she was waiting in the queue, she heard some clients who said” (“María fue a la frutería para comprar fruta que necesitaba para hacer un pastel por San Valentín. Mientras estaba esperando en la cola escuchó a unos clientes que decían”). The next sentence contained a conditional, either an indicative conditional, e.g., “if there are oranges, then there are pears” (“si hay naranjas, entonces hay peras”) or a counterfactual conditional, e.g., “if there had been oranges, then there would have been pears” (“si hubiera habido naranjas, entonces habría habido peras”). The following sentence contained a conjunction, either an affirmative conjunction, e.g., “María realized that there were oranges and there were pears” (“María se dio cuenta que había naranjas y había peras”) or a negative conjunction, e.g., “María realized that there were no oranges and there were no pears” (“María se dio cuenta que no había naranjas y no había peras”). The vignette ended with a closing-scene, e.g., “Finally, María also bought chocolate” (“Finalmente, María también compró chocolate”). The full set of 36 contents and their associated images is in the Supplementary Material A.

Each sentence was prerecorded and presented via speakers while four images were shown on a computer screen: two target images, e.g., an image of an orange and a pear, and an image of an orange and a pear with a cross through it, as well two distractor images, e.g., an image of other fruit such as an apple and a strawberry, and an image of an apple and a strawberry with a cross through it (see the four images on the left side of Figure 1). The position of each image (top left, top right, bottom left, bottom right quadrant) was counterbalanced across conditions. We constructed 8 versions of each vignette (e.g., oranges and pears) that varied in the conditional (indicative or counterfactual), the conjunction (affirmative or negative), and the reference to the objects in the set (e.g., oranges and pears, or apples and strawberries), as illustrated in Table 1. Each participant received only one of the 8 possible versions of each content in the set of 36 trials and the contents were assigned to the trials in a counterbalanced manner.

TABLE 1

Table 1. Examples of the 8 versions of the verbal description of each content, illustrated for the oranges and pears/apples and strawberries content, for the visual display in Figure 1.

Apparatus and Procedure

Participants listened to the 36 stories over speakers while looking at a computer screen and at the end of each story they answered a simple question about it. Participants were instructed that they should listen to the sentences carefully and that they should not take their eyes off the screen throughout the experiment. They were explicitly instructed to look at the object or objects on the screen that matched the meaning of the stories that they heard. These explicit instructions were based on the usual information provided to participants in eye-tracking experiments, which typically specify how to interact with the display, e.g., by touching, clicking, or moving objects. Their eye movements were recorded at a rate of 500 Hz using an SR Research EyeLink II head-mounted eye-tracker connected to a 21 color CRT for visual stimulus presentation. Procedures were implemented in SR Research Experiment Builder. Calibration and validation procedures were carried out at the beginning of the experiment and were repeated several times per session. Trials started with the presentation of a central fixation dot for drift correction while participants listened to the opening scene sentence. Next, a display with four images appeared for 2 s. Then the story began, and the images remained on screen for the entire time while the remainder of the story was heard over speakers. The trial concluded with the appearance of a simple question on the screen, e.g., “Did María go to the fruit shop?” (“¿Fue María a la frutería?”)¹, which participants answered by pressing either a “yes” or a “no” button on a game-pad, by pressing with their right index finger for yes and their left index finger for no. There was a practice block of four trials before the experiment proper started. The experiment lasted approximately 30–40 min and each participant was tested individually.

Results and Discussion

The data for the three experiments is available at https://reasoningandimagination.com/data-archive/ and on OSF at https://osf.io/n6hk3/. Prior to any data analysis one participant was eliminated from the analysis because her eye-movements explored the screen continuously without any systematic fixations on any point.

Eye-Tracking Data Coding

The eye-movement data generated by the EyeLink system were analyzed as follows. First, bitmap templates were created for identifying regions of interest in each display (the four pictures of the screen, e.g., oranges and pears, no oranges and no pears, apples and strawberries, and no apples and no strawberries). The object regions were defined in terms of rectangles containing the relevant objects, fixations landing within the perimeters of these rectangles were coded as fixations on the relevant objects. The output of the eye-tracker included the x- and y-coordinates of participant fixations, which were converted into region of interest codes using the templates.

The analysis of fixations was time-locked to the onset of the first object in the conditional, e.g., the onset of “oranges” in “if there are/had been oranges” and continued to 4000 ms after that word, which included listening to the rest of the conditional, “then there are/would have been pears” followed by a silent period. The periods were divided into 50 ms time slots and for each time slot, the number of fixations on each rectangle quadrant of the image was counted and converted into fixation probabilities².

To avoid problems inherent in proportional data, participant averages were arcsin-transformed prior to t-test comparisons. Given that 180–200 ms are usually assumed to account for saccade programming (Martin et al., 1993), the mean of the first time-region (0–100 ms) was considered to be the baseline and was used to conduct statistical comparisons against means on each time points at 50 ms intervals until 4000 ms later (for a similar method, see Huettig and Altmann, 2011). This correction to baseline allowed us to control for any bias in the pattern of fixations on images caused by the type of context. A false discovery rate (FDR) thresholding procedure, referred to as pFDR-corr, was used as an alpha correction to control for Type 1 errors due to multiple comparisons (81 for each condition; see Genovese et al., 2002).

T-Tests Against Baseline

The results reveal that participants looked at very different parts of the visual images on screen when they heard an indicative conditional compared to when they heard a counterfactual conditional, as Figure 2 shows. For indicative conditionals, at the onset of the target word (e.g., oranges), participants were focused on the affirmative image (oranges and pears) and the affirmative distractor (apples and strawberries) about equally frequently, with probabilities of fixation of about 0.4, as Figure 2A shows. This starting point may merely reflect a tendency to look at what is present rather than what is not present. What is revealing is that very early on in the process, 450 ms after the target word onset, the probabilities of fixation on the affirmative image started to increase (pFDR-corr = 0.002); fixations decreased on all other images, including the negative image (from 350 ms, pFDR corr = 0.034) (see the Supplementary Material B for details of the comparisons for the distractor images). The results show that for an ordinary indicative conditional, participants increase their fixation on the affirmative image very early indeed in the temporal course of processing, and fixations on the other three images decrease rapidly, as Figure 2A shows. Their fixation on the affirmative image continued throughout the period of measurement to 4000 ms.

FIGURE 2

Figure 2. Probabilities of fixations for indicative conditionals, e.g., “if there are oranges, then there are pears” (A) and counterfactual conditionals, e.g., “if there had been oranges, then there would have been pears” (B) in Experiment 1 on the affirmative and negative images, and the two distractor images, time-locked to the onset of the first object word, e.g., oranges. The differences in the probabilities of fixations (on the affirmative image minus the negative image) for indicative and counterfactual conditionals emerge at 850 ms, as (C) shows. Error bars are 95% confidence intervals within participants (see Morey, 2008; O’Brien and Cousineau, 2014).

A very different pattern emerges for counterfactual conditionals, e.g., “if there had been oranges, then there would have been pears.” At the onset of the target word, e.g., “oranges,” participants were focused on the affirmative image (oranges and pears) and the affirmative distractor (apples and strawberries) equally frequently with probabilities of fixations of about 0.4, as Figure 2B shows. Most revealingly, from very early on, 300 ms after the target word onset, the probabilities of fixation on the affirmative image started to increase (pFDR-corr = 0.039), and fixation on the negative image also increased (from 650 ms, pFDR-corr = 0.031). The results show that early in the temporal course of processing, within about half a second, participants increase their fixation not only on the affirmative image but also on the negative image; fixations on the two distractor images decrease rapidly, as Figure 2B shows (see the Supplementary Material B for details about the distractor images). Equally importantly, their fixation on both images continues throughout the period of measurement to 4000 ms.

Growth-Curve Analysis

We carried out a growth-curve analysis (Mirman, 2014; see the Supplementary Material B for details) which showed that people looked at the affirmative image more for the indicative conditional than the counterfactual, and they looked at the negative image more for the counterfactual than the indicative conditional. The increase of fixations on the affirmative image occurred more quickly for the indicative than the counterfactual conditional, and the opposite was the case for the negative image.

Analysis by Items

As a check that each of the 36 contents was interpreted in essentially the same way, we also carried out a similar analysis to compare indicative and counterfactual conditionals with t-tests against the baseline, but this time by items rather than by participants. It showed the same results. For indicative conditionals, at the onset of the target word (e.g., oranges), participants’ focus was on the affirmative image and the affirmative distractor equally frequently. From 450 ms after the target word onset, the probabilities of fixation on the affirmative image started to increase (pFDR-corr = 0.003); fixations on the other images decreased, including for the negative image (from 1050 ms, pFDR-corr = 0.034) (see the Supplementary Material B for details about the distractor images).

For counterfactual conditionals, at the onset of the target word, the focus was on the affirmative image and the affirmative distractor equally frequently. From 300 ms after the target word onset, the probabilities of fixation on the affirmative image started to increase (pFDR-corr = 0.013) and so did fixation on the negative image (from 500 ms, pFDR-corr = 0.017), as Figure 3 shows. It is noteworthy that the 95% confidence interval error is very much reduced for the counterfactuals in the by-item analysis compared to the by-participants one, as a glance at Figures 2B and 3B shows, which suggests that the variance originates in differences between participants rather than differences between items. Accordingly, we also carried out an analysis of individual differences.

FIGURE 3

Figure 3. Item-analysis probabilities of fixations for indicative conditionals, e.g., “if there are oranges, then there are pears” (A) and counterfactual conditionals, e.g., “if there had been oranges, then there would have been pears” (B) in Experiment 1. Error bars are 95% confidence intervals within items.

Individual Differences Analysis

We plotted individual graphs for each of the 23 participants, which are provided in the Supplementary Material C. As these graphs show, about half of the participants (n = 11) tended to look at the affirmative image only when they heard the counterfactual, just as they did for the indicative conditional; the other half of the participants (n = 12) tended to look at the negative image only (seven participants), or at both the affirmative and the negative image (five participants) for the counterfactual. We combined participants who looked at the negative image only and those who looked at the affirmative and negative image into a single sub-set group because consideration of the negative image (corresponding to the presumed facts of a counterfactual) indicates that individuals have reached a counterfactual interpretation of the conditional (and there are in any case too few participants to create three separate groups for reliable statistical analysis). These two sub-set groups of participants, affirmative only versus negative or negative-plus-affirmative, exhibited very different fixation patterns on the affirmative and negative image, as Figure 4 shows.

FIGURE 4

Figure 4. Individual differences probabilities of fixations for indicative and counterfactual sentences for one subset group of 11 participants who looked at the affirmative image for counterfactuals (A), and a second subset group of 12 participants who looked at the negative image only or at the affirmative and negative image for counterfactuals (B) in Experiment 1. Error bars are 95% confidence intervals within participants.

Both groups showed similar patterns for the indicative conditional, the probabilities of fixation on the affirmative image started to increase early on (group 1 from 450 ms, pFDR-corr = 0.025; group 2 from 450 ms, pFDR-corr = 0.020), and to decrease on the other three images, including the negative image (group 1 from 800 ms, pFDR-corr = 0.037; group 2 from 1200 ms, pFDR-corr = 0.034) (see the Supplementary Material B for details about the distractor images). However, the two groups showed different patterns for the counterfactual. Group 1’s pattern was the same as for the indicative: probabilities of fixation on the affirmative image started to increase early on (from 350 ms, pFDR-corr = 0.030), and probabilities of fixations on the other three images decreased, including for the negative image (from 1450 ms, pFDR-corr = 0.023). Group 2’s pattern was different: probabilities of fixation for the affirmative image showed no significant changes to the baseline, but they increased for the negative image (from 400 ms, pFDR-corr = 0.039).

The analysis shows that when people understand the indicative conditional, they look at the affirmative image from very early (450 ms) and decrease their fixations on the negative image quite some time later (800 ms in Group 1; 1200 ms in Group 2). When they understand the counterfactual, one subset of participants do the same thing as for the indicative, they look at the affirmative image from very early (350 ms) and decrease their fixations on the negative image quite some time later (1450 ms); however, the other subset of participants look at the affirmative image early and continue to do so at the same rate as at the baseline throughout, but these participants look increasingly at the negative image and from very early indeed (400 ms).

The experiment provides information on the points in the temporal course of processing a counterfactual, e.g., “if there had been oranges, then there would have been pears,” when people focus on an image corresponding to the conjecture, e.g., “there were oranges and pears” and on an image corresponding to the presumed facts, e.g., “there were no oranges and no pears.” The results show that when people understand an indicative conditional, e.g., “if there are oranges, then there are pears,” shortly after they hear the word “oranges,” their focus increases on the affirmative image, and their focus on the other three images decreases fairly rapidly. The overall group results for indicative conditionals are reflected also in the results for each individual. The results show a different pattern for counterfactuals. The results averaged over the whole group of participants show that when they understand the counterfactual, e.g., “if there had been oranges, then there would have been pears,” very early in the temporal course of processing, they increase their focus not only on the affirmative image but also on the negative image, and this focus on both images continues throughout the period of measurement. However, there are pronounced individual differences. About half of the participants appear to understand the counterfactual just as they do the indicative conditional, and they focus on the affirmative image only. The other half of the participants understand the counterfactual differently from the indicative conditional – they continue to look at the affirmative image as much as they do at the outset, but they increase their focus on the negative image.

One possible explanation for the individual differences is that the instruction to look at what the stories mean may be interpreted by some participants to look at what is explicitly mentioned in the counterfactual, e.g., oranges and pears, whereas it may be interpreted by others to look at what is presumed by the counterfactual, e.g., no oranges and no pears. To rule out this possibility, we carried out a second experiment with the aim of testing whether the results are replicated when participants are not given this explicit instruction.

Experiment 2

The aim of the experiment was to test whether the results of the previous experiment are replicated in an implicit task, that is, when participants are not given an explicit instruction to look at the object or objects on the screen that matched the meaning of the stories that they heard. In this way we aimed to examine further the spontaneous or automatic processes underlying the comprehension of counterfactuals.