Visual search and the inhibitions of return

Klein, Raymond M.; Redden, Ralph S.; Hilchey, Matthew D.

doi:10.3389/fcogn.2023.1146511

REVIEW article

Front. Cognit., 12 May 2023

Sec. Attention

Volume 2 - 2023 | https://doi.org/10.3389/fcogn.2023.1146511

This article is part of the Research Topic Insights in Attention: 2022 View all 15 articles

Visual search and the inhibitions of return

$\r\nRaymond M. Klein$ Raymond M. Klein¹^*

Ralph S. Redden^1,2

Matthew D. Hilchey³

¹Department of Psychology and Neuroscience, Dalhousie University, Halifax, NS, Canada
²Department of Psychology, University of Alberta, Edmonton, AB, Canada
³Rotman School of Management, University of Toronto, Toronto, ON, Canada

In the early 1980's independent research streams were launched by two of the 20th century's leading attention researchers. Anne Treisman's research program is best-known for distinguishing empirically between serial and pop-out search and for proposing feature integration theory and the idea of an attentional operator, that sequentially inspects items or groups of items when search is difficult. Among his many contributions to psychological science, Michael Posner is well-known for pioneering simple model tasks that made the allocation of visual attention in space amenable to scientific inquiry. When one version of the Posner cuing paradigm was used to explore visuospatial orienting it was serendipitously discovered that an “inhibitory” effect occurred in the aftermath of events that captured visuospatial attention involuntarily. This “inhibitory” phenomenon became known as Inhibition of Return (IOR), and, as implied by its name, the underlying mechanisms were thought to bias attention away from previously explored places. These two research programs were linked in 1988 when Raymond Klein exploited the distinction between pop-out and serial search to test and verify Posner's proposal that this inhibition might be a novelty seeking mechanism that could improve search efficiency. Subsequent research has identified at least two different inhibitory mechanisms that bias attention toward novelty. We present evidence using several diagnostics (central vs. peripheral targets, joint consideration of speed and accuracy, and the locus of slack logic embedded in the psychological refractory period effect) to illustrate the dual natures of IOR. The input form operates on a salience map that influences what will capture our attention, while the output form operates on a priority map that influences what behaviors (including orienting) are likely to be executed. The input form is generated when the reflexive oculomotor system is suppressed while the output form is generated when this system is not suppressed. We believe that both forms of IOR can serve the novelty seeking (and search facilitating) function proposed by Posner and others. Yet, many questions remain unanswered. Some of the many remaining puzzles will be described and we hope that our review will stimulate research to solve them.

Introduction

This story begins in the early 1980s when two of the late 20th century's leading attention researchers, Michael Posner and Anne Treisman, made seminal empirical discoveries for which they offered exciting interpretations, aspects of which remain topical today ~40 years later. This review is not intended to provide comprehensive coverage of all the seminal contributions of these luminaries. Rather we will focus on our thoughts about the concept of inhibition of return (IOR) and how IOR provides a bridge between Treisman's “glue” and Posner's “beam.”

Treisman and Gelade (1980) distinguished empirically between serial and pop-out search. Reaction time to find the target in serial search increases roughly linearly with the number of items in the display whereas search reaction time to find the target in pop-out search is relatively unaffected by the number of distractors (see Figure 1A). To help explain these findings and to provide one method for minimizing the possibility of illusory conjunctions, Treisman proposed feature integration theory, wherein spatial attention was the metaphorical “glue” that integrated otherwise free-floating features that fell within the metaphorical “beam” of attention (see Figure 1B). Whereas Treisman (e.g., Treisman and Schmidt, 1982; for a review see Treisman, 1988) supported this theory via converging operations, later, various aspects of the theory were discarded (e.g., Wolfe et al., 1989; Treisman, 1993; see Humphreys, 2016 for a review). Nevertheless, the idea of an attentional operator, that sequentially inspects items or groups of items when search is difficult, has endured (highlighted in the inset of Figure 1B).

FIGURE 1

Figure 1. (A) Search reaction time as a function of set size for a difficult search task (solid lines) and an easy (or pop-out) search task (dashed lines). Target absent trials are plotted as open circles; target present trials as solid circles (redrawn from Treisman and Souther, 1985). (B) A schematic illustration of Feature Integration Theory with an inset used to highlight the “spotlight” of attention.

Our focus on Treisman's work is not dependent on the correctness of feature integration theory, but rather is rooted in how the metaphorical spotlight behaves when search is difficult, as illustrated in solid lines of Figure 1A. The functions are not only nearly linear, but the slope of the target present trials is ~1/2 of that for target absent trials. It is generally agreed that the kind of difficult search task illustrated here is characterized by a serial, self-terminating inspection of items or small sets of items in the array. The efficiency of such a sequential inspection strategy would be improved if there were a mechanism to discourage reinspections of already inspected items or regions in the array. Whereas, such a serial inspection might depend on eye movements, as are necessary for some search tasks (such as searching for Waldo/Wally, in the series of picture books by Martin Handford), using targets defined by the absence of a feature (inspired by Treisman and Souther, 1985) as illustrated in Figure 2A, Klein and Farrell (1989) demonstrated that when search is covert (because eye movements were not permitted), the pattern of results was more or less the same as when eye movements were permitted (Figure 2B). Thus, with stimuli like these, the inspections can be attributed to covert shifts of the attentional spotlight highlighted in Treisman's model [inset in panel (B) of Figure 1].

FIGURE 2

Figure 2. Sample target present stimuli (A) and results (B) from Klein and Farrell (1989, Exp. 2).

In the late 1970's, in a series of presentations at meetings of the Psychonomic Society, Posner and colleagues developed and exploited two model tasks, often referred to as “Posner cuing tasks,” for exploring the endogenous and exogenous allocation of the “beam” of attention in space (see also, Posner et al., 1978). This work was punctuated by two papers published in 1980 (the same year as Treisman and Gelade): Posner (1980) and Posner et al. (1980). While somewhat lacking in ecological validity, such model tasks are useful because they enable isolation of mental operations for investigation. In the Posner cuing paradigms the onset of a cue precedes the target, often by an interval (the cue-target onset asynchrony, or CTOA) that might be varied. Usually targets are presented at two alternative peripheral locations. In one variant, used to explore endogenous control of spatial attention (see Figure 3A), the cue is presented centrally and is informative about the upcoming target's location. Using the model task for exploring exogenous control of attention (see Figure 3B), wherein cues are uninformative and presented in the periphery, Posner and Cohen (1984) discovered that an initial facilitation at a cued location was followed by inhibition there (see Figure 3C), an effect that has come to be known as inhibition of return (IOR; Posner et al., 1985).

FIGURE 3

Figure 3. (A) Paradigm pioneered by Posner for exploring the endogenous control of attention in visual space. (B) Paradigm pioneered by Posner for exploring the exogenous control of attention in visual space. (C) From the exogenous cuing paradigm the pattern of response times is shown as a function the cue-target onset asynchrony (CTOA) (redrawn from Posner and Cohen, 1984).

These streams of research, pioneered by Treisman and Posner, were linked in 1988 when Klein (1988) tested and verified Posner's proposal that this inhibition might be a novelty seeking mechanism that could improve search efficiency. Two related themes will be explored in this paper. Firstly, as noted by Hilchey et al. (2014b) the causes and effects of the inhibitions identified in the aforementioned papers from Posner's lab (Posner and Cohen, 1984; Posner et al., 1985) were different, suggesting that there might be (at least) two forms of inhibition¹. In the next section, we will review converging evidence for this possibility and we will show that one form of inhibition operates early in the processing stream to affect the salience of possible targets while another form operates late in the processing stream to bias responding (including eye movements) away from previously attended objects and locations. Secondly, Posner et al. (1985), for whom IOR was a bias against re-orienting toward a previously inspected location, explicitly suggested that “such a bias would have obvious advantages in scanning for visual targets” presumably because IOR could operate to discourage reinspections. In the following section we will review research demonstrating that IOR operates during search and we will propose that both forms of inhibition could serve the novelty seeking function attributed to it/them in Posner's seminal papers. Finally, there are many interesting and unanswered questions about IOR and IOR in visual search, some of which will be introduced in the final section of our review.

Converging evidence for two inhibitory after-effects

The possibility of two inhibitory aftereffects of orienting can be found in the two seminal papers from Posner's lab. As described in Hilchey et al. (2014b), whereas Posner and Cohen (1984) proposed that repeated stimulation was the cause and the effect was to slow the detection of the target, Posner et al. (1985) proposed that oculomotor activation was the cause and the effect was a response bias. Unfortunately, this dramatic difference in cause and effect was not signaled by Posner et al. (1985) who named the(ir) inhibitory aftereffect, inhibition of return (IOR) while presenting their work as a continuation of Posner and Cohen's. As we will show in the next sections we strongly agree that there are two different inhibitory phenomena with different effects upon processing. Repeated stimulation of an input pathway does generate a relatively short-lived sensory adaptation effect (e.g., Boehnke et al., 2011) that can be difficult to distinguish from the IORs described below, and might be considered to be a third inhibitory aftereffect. Three diagnostics that permit researchers to distinguish whether an inhibitory effect is operating early or late in the processing of targets will be described briefly below. These are more thoroughly described in Klein and Redden (2018), Redden et al. (2021) and in the various papers wherein the original research was reported. It is noteworthy that all of the evidence described in the next three sections was obtained using versions of Posner's model task for exploring exogenous orienting.

Measuring aftereffects with spatial responses to peripheral targets vs. central arrows

The idea to compare responses toward inhibited locations using peripheral targets vs. central arrows was pioneered by Rafal et al. (1994) and Abrams and Dobkin (1994). The logic is straightforward. If the effect is a response bias away from the inhibited location then it shouldn't matter if localization responses are made to centrally presented arrows or peripheral targets. On the other hand if the inhibition is about the encoding of information presented at the originally cued location, the effect should be present with peripheral targets and absent with central targets.

The most thorough application of this diagnostic was rationalized and presented in Taylor's dissertation (as reported in Taylor and Klein, 1998, 2000)². The methods and results from Taylor's 24 variants on Posner's model task are presented in Figure 4. Two dramatically different effects can be seen here: When eye movements were made either to the first or second stimulus (conditions outlined in green), measured IOR, if observed with peripheral targets was also observed with central targets suggesting an effect that is a straightforward response bias. In striking contrast, when no eye movement was executed to either stimulus (conditions outlined in red) inhibition was only observed if the target was presented in the periphery. This “input” form of inhibition is consistent with Posner and Cohen's (1984) hypothesis that “inhibition” decreases the quality of the target's input signal. When eye movements were made, however, the evidence is consistent with Posner et al. (1985)'s hypothesis that “inhibition,” thereafter called inhibition of return, biases responses against locations to which eye movements were made or primed (viz the “output” form).

FIGURE 4

Figure 4. Methods and results from Taylor and Klein (2000). In a counterbalanced order each of 18 participants experienced, in sessions on different days, the conditions represented in the six boxes which differed according to the task used to generate IOR (S1) and the task used to measure it (S2). The iconograms that mark the small rows and columns within each box represent the two signal types: peripheral onset and central arrow. Solid circles depict statistically significant inhibitory effects. The conditions purported to elicit output-based effects are identified by the green highlighting (solid black circles) and input-based effects are identified by the red highlighting (solid gray circles).

From this pattern, Taylor and Klein (2000) inferred that which form of IOR would be observed depended on whether eye movements were made (an inference later endorsed with converging evidence by, among others: Fischer et al., 2003; Hunt and Kingstone, 2003; Pratt and Neggers, 2008). Based, in part, on the input pattern being observed when anti-saccades are made, Klein and Hilchey (2011) later proposed that it is not response modality per se that is the “switch” determining which form of IOR would be generated; rather it is the activation state of the reflexive oculomotor system. The input form is generated when this system is suppressed; otherwise, the output form is generated. Along with some of the studies presented in this section, a series of empirical papers confirmed this proposal (Satel et al., 2013; Hilchey et al., 2014a,b, 2016; Eng et al., 2017).

Measuring aftereffects in speed-accuracy space using a non-spatial discrimination task

When participants report a non-spatial property of a target stimulus (e.g., color or shape) with keypress responses, response times are faster when there is congruence between the target's location and the location of the responding effector. Discovered by Dick Simon (Simon, 1969; for a review, see Simon, 1990) this compatibility effect was later labeled the Simon effect (Hedge and Marsh, 1975). With a focus on the Simon effect and without connecting their ideas to Taylor and Klein's (2000) two forms, Ivanoff et al. (2002) proposed that the RT delay that ubiquitously characterizes IOR could arise for two quite different reasons as illustrated by the arrows and speed-accuracy tradeoff (SAT) functions in Figure 5: IOR delays or slows the accumulation of information from the inhibited location or IOR is simply a response bias against the inhibited location. Although their meta-analysis of the literature revealed a significant interaction between IOR and the Simon effect (with a larger Simon effect at the inhibited location) this interaction was consistent with either explanation of IOR's slowing of RTs (as illustrated above in Figures 5A, B) depending on one's view of how the inhibition might affect the task-irrelevant location code activated by the target.

FIGURE 5

Figure 5. In all three panels in which performance is represented in SAT-space, accuracy of responses is represented on the Y-axis and response time is represented on the X-axis. (A, B) Two ways IOR might slow reaction times when a non-spatial discrimination is required to a target. (A) IOR might delay (as illustrated here) or slow the accumulation of task-relevant evidence about the target's identity (this would be represented as a shallower slope of the accumulation function). (B) IOR might have no effect on the accumulation of evidence about the target but instead is a bias against responding to targets at the inhibited location (represented here as in increase in the amount of evidence required to make a response to inhibited targets). (C) The slope of the IOR effect in SAT-space can be used as a diagnostic for determining whether the effect is of the input form (red arrows) or likely not (green arrow).

Importantly, the direction of the cuing effect in SAT space (as in Figure 5C) can be used as a diagnostic for the form of IOR one has generated in an experiment with a non-spatial discrimination task. When RT is delayed by the inhibition and accuracy is either unaffected or decreases, the input form was generated. Otherwise, the output form was generated. Using and supporting this framework, Chica et al. (2010) reported IOR findings from a non-spatial discrimination task: when observers were instructed to ignore the cue the input form was generated, whereas when observers were instructed to make a pro-saccade to the cue and back to fixation, the output form was generated. Redden et al. (2016) built upon this result, and, using an anti-saccade condition rather than “ignore” condition in Chica et al. (2010), explicitly tested the hypothesis posited by Klein and Hilchey (2011) that the nature of IOR is contingent not on whether an overt orienting response is or is not made, but rather on the activation state of the reflexive oculomotor system. If any overt orienting response was to generate the output form of IOR, then both the anti-saccade and pro saccade conditions ought to generate an SAT. However, if the state of the reflexive oculomotor system is the determining factor, then the anti-saccade condition ought to generate an input effect akin to the “ignore” condition. This is based on the proposal (Forbes and Klein, 1996) that an observer must suppress the reflexive oculomotor system in order to correctly perform an anti-saccade (Everling et al., 1999; Ignashchenkova et al., 2004). The striking results are presented in Figure 6 where it can be seen, using the SAT diagnostic, that the input form of IOR was generated by the cue when an anti-saccade was executed (Figure 6A) while the output form was generated by in the pro-saccade condition (Figure 6B). It is noteworthy that when diffusion modeling was applied to these data (Redden et al., 2021) the parameters that best explained these findings were a slower accumulation of evidence in the anti-saccade condition and an increase in the threshold evidence for triggering a response in the pro-saccade condition.

FIGURE 6

Figure 6. Reaction time (X-axis) and accuracy (Y-axis) as a function of whether the target requiring a non-spatial discrimination was presented at the previously cue or uncued location following either (A) an anti-saccade or (B) a pro-saccade made in response to the cue. Redrawn from Redden et al. (2016).

Linking back to the Simon effect, Redden et al. (2016) also found that IOR and Simon interact in opposite directions depending on which form was generated: when the input form was generated, the Simon effect was enhanced at the cued relative to the uncued location—a pattern consistent with an increased tendency toward the prepotent response when target signal quality is reduced, whereas when the output form was generated, the Simon effect was reduced at the cued relative to the uncued location—a pattern consistent with a reluctance to make responses in the direction of the cue.

Recently, using pro-saccades to elicit the output form of IOR, Redden et al. (2023) combined the central arrow diagnostic with the SAT diagnostic by intermixing peripheral and central arrow targets. Performance following peripheral targets replicated the pattern found by Redden et al. (2016), with slower but more accurate responses to targets presented at the location of the original cue and a reduced Simon effect for these targets. Responses to central arrows calling for responses toward the cued location were also delayed. These three findings converge on the conclusion that the output form of IOR is a bias against responses in the direction of the cue.

Measuring aftereffects using the psychological refractory period paradigm

When two targets requiring speeded responses by different responding effectors are presented in close succession to different sensory modalities, the response to the second target suffers a delay that typically decreases linearly with increases in the interval separating the targets. The delay, which is thought to reflect a relatively “central” bottleneck stage of processing that can only be allocated to one task at a time, has come to be called the “Psychological Refractory Period.”

When the task associated with the second target is delayed by some factor, such as IOR, the “locus of slack” logic (as described in Pashler, 1998, p. 275–287) can be used to determine whether the effect is operating relatively early in the processing sequence (before the bottleneck) or relatively late (at the bottleneck stage or after the bottleneck). At short intervals between the two targets, when the IOR effect is operating early, no effect is seen in reaction time because the extended processing during a pre-bottleneck stage is absorbed into the period of slack. In contrast, if the effect is operating at or after the bottleneck, it will be the same regardless of the interval between targets. As can be seen in Figure 7 we found that IOR when generated while the reflexive oculomotor system was suppressed was operating early (Kavyani et al., 2017) whereas when IOR was generated while the reflexive oculomotor system was active (generated by a saccade to center after the drift-correction in the periphery) the effect was operating later (Klein et al., 2020).

FIGURE 7

Figure 7. Cuing effects (positive values represent inhibition of return) in reaction time as a function of the intervals between the onsets of the targets for Task 1 and Task 2. In two experiments Task 1 required an auditory discrimination whereas Task 2 required indicating whether a visual target appeared to the left or right of the current fixation. (A) Eye movements were forbidden and IOR was generated by an uninformative peripheral cue presented (before both targets) at a position to the left or right of a central fixation stimulus (from Kavyani et al., 2017). (B) The initial fixation (drift-correction) stimulus (cue) was presented either to the left or right of the central stimulus and IOR was generated at the initial fixation location when an eye movement to the central stimulus was made (from Klein et al., 2020). The two tasks were presented after this saccade.

Summary

We believe that the evidence described above converges inexorably on the conclusion that there are at least two different inhibitory effects and that what determines which effect will be generated depends on the state of the reflexive oculomotor system around the time the effect is generated. As originally proposed and demonstrated by Posner et al. (1985) we also believe that the cause of the output effect is activation of the oculomotor system. For a variety of reasons, we are less confident about the cause of the input form, a question to which we will return later.

From the simple model task to visual search

Several paradigms have been used to converge on the conclusion that IOR operates during search by encouraging orienting to novel items or discouraging inspections of already inspected items. In the sections that follow some of these paradigms and the evidence they have provided will be described.

IOR after a covert-search episode

To test Posner's proposal that the inhibitory tags left in the aftermath of orienting might encourage novelty-seeking and play a constructive role in some visual search tasks, Klein (1988) combined search arrays inspired by Treisman's work (see Figure 8, top panel) with post-search probes inspired by Posner's cuing paradigm (see Figure 9). The search arrays were designed to generate pop-out or serial search (as was obtained, see bottom panel of Figure 8) and the post-search probes called for a simple, speeded detection response. Klein reasoned that if inhibitory tags were left behind whenever attention visits a location (display item), then (particularly on target absent trials) there should be inhibition at the locations of distractors (relative to empty locations) following serial but not popout search which is precisely what was found. After some early failures to replicate (Wolfe and Pokorny, 1990; Klein and Taylor, 1994) it was demonstrated (Müller and von Mühlenen, 2000; Takeda and Yagi, 2000) that the inhibitory tags are likely to be removed if the search array is removed before the probe is presented—as if the tags are in the brain's mental representation of the scene (for a review see, Wang and Klein, 2010).

FIGURE 8

Figure 8. Top: Search displays used in the two experiments from Klein (1988) are exemplified (not to scale) using set size of 6. Bottom: Search performance (combined across the gap and line stimuli) is shown as reaction time as a function of set size. Popout search is plotted with dashed lines; serial search with solid lines. Open symbols are for target absent trials; solid symbols for target present trials.

FIGURE 9

Figure 9. Methods and probe results from Klein (1988). A search array designed to generate either popout or serial search was presented. Klein reasoned that if inhibitory tags are left behind whenever attention visits a location, then there should be such tags at the locations of distractors on target absent trials following serial but not popout search. The different shades of gray are meant to reflect differential decay (older tags being weaker). Occasionally a probe (calling for a simple detection response) was presented AFTER the search response and removal of the search display. According to the proposal, the time to detect ON probes following serial search should be delayed by IOR. RTs to probes presented after search responses on target absent trials are presented for each of the four key conditions.

One limitation of this paradigm is that the shifts of attention are not directly observable. They are, instead, hypothetically implied by the notion of the covert (see Klein and Farrell, 1989) sequential inspection by attention of the items or groups of items in a display when search is difficult. Relatedly, the notion of serial self-terminating search makes a focus on target absent trials particularly useful, because after such trials it is assumed that attention had examined each of the non-targets (distractors) in the search array.

IOR during or after an overt search episode

Klein and MacInnes (1999) sought to evaluate the manifestation of IOR under more ecologically valid conditions by exploring it using a search task with images from Martin Handford's, Where's Wally (or Waldo—in North America) series of books. While not really real-world searching, looking for Wally is considerably more real than looking for the absence of a feature in a display of circles with lines (one of the conditions in Klein, 1988). Importantly, when the eyes are allowed to move, as they must to find Wally, their path can be recorded and thus an objective indication of which locations had been fixated—overtly attended—can be generated.

By monitoring participant's eye movements it was possible to probe locations that were the same distance from the current fixation but were either at an old (previously fixated) location or at a new location at varying angular distances from the old one. If IOR were present saccades to an old location (0 deg) would be slow and if there were a gradient of inhibition, there would be a monotonic effect of distance from this location. This is what was found when the search array was maintained; but (in agreement with Müller and von Mühlenen, 2000; Takeda and Yagi, 2000) there was no inhibitory gradient when the probe was presented after removal of the search array (see Figure 10A).

FIGURE 10

Figure 10. Saccadic probe reaction time as a function of distance from a previous fixation. (A) From the 2-back condition of Klein and MacInnes (1999). (B) From MacInnes and Klein (2003) who only tested 1-back. See text for further explanation.

Why–despite being unaffected by the distance from the previous fixation–were participants so slow in the removed condition? Here targets would not be embedded in the complex Wally scene but instead would be single objects in an otherwise empty field. Perhaps the sudden removal of the scene at the same time the next search saccade was being planned may have disrupted saccade programming to the unexpected probes. This idea was tested (MacInnes and Klein, 2003) by changing the search task while keeping the stimuli the same: Instead of asking participants to find Wally, they were asked to find something interesting and then stop there. When a pause in eye movements of 500 ms was detected it was assumed that participants had stopped at something interesting and would not be planning their next saccade. At this point the probing and scene removal methods described above (Klein and MacInnes, 1999) were applied. As before, a gradient of IOR was observed when the scene remained present; but now saccades in the removed condition were, as expected, faster and unaffected by distance from the previously fixated locations (See Figure 10B).

A substantial literature looking for IOR during and after visual search was stimulated by these two seminal studies. The studies (about 15) published in the 20 years or so after the publication of Klein (1988) were reviewed in 2010 by Wang and Klein (2010) (see also Klein and Hilchey, 2011). IOR during search has since been explored in the monkey (e.g., Torbaghan et al., 2012; Westerberg et al., 2020) and using ERPs (Pierce et al., 2017). The relative ubiquity of oculomotor IOR was recently demonstrated by Murziakova et al. (2022) who explored five tasks: static and dynamic visual search, foraging (akin to search but with multiple targets), memorization and change detection. Their use of the probe paradigm revealed IOR in all of these tasks except for change detection and, consistent with the novelty-seeking proposal for IOR, across the five tasks there was a relatively strong negative relation between these IOR scores and the probability of return saccades.

Several interesting findings about IOR during and after search have been reported by Höfler and colleagues. Höfler et al. (2011) had their participants search the same array twice in succession for different targets. Using the probe during search method they found that IOR was present in each of the successive searches. Interestingly, when they tested for IOR after the first target was found and immediately after the presentation of the second target, there was no evidence for IOR at the locations previously fixated during the first search. This resetting of the inhibitory tags suggests a flexibility that is consistent with IOR's sensitivity to task (e.g., Dodd et al., 2009). When the second target was presented before the first target was found, IOR from the first search was maintained (that is resetting was abolished). When considered together with the studies reviewed earlier in which IOR was probed after the search response, it was suggested that the resetting of IOR might depend on both the completion of the first search and the start of a new one. Regardless, the resetting discovered by Höfler et al. (2011) likely ensures that the inhibition from a prior search will not interfere with a subsequent one. Recently, Höfler and Kieslinger (2022) demonstrated that the number of items being held in spatial working memory had no effect on IOR when measured using a probe during search method similar to that of Klein and MacInnes (1999).

Summary

In the context of visual search, the need for and benefit of inhibitory tags at previously inspected locations in a scene was highlighted in Itti and Koch's (2001) computational model of attention. Their model hypothesizes that the sequence of inspections is guided by a “winner-take-all” algorithm that operates on a saliency map. This map is generated primarily by bottom-up processing of the features in the visual scene, processing which can be biased by top down mechanisms and prior experience. Importantly, inhibition of return is hypothesized as necessary to suppress “the last attended location from the saliency map, so that attention can focus onto the next most salient location” (p. 196). A similar role for IOR was previously proposed by Klein (1988) who asked: “How does the serial search mechanism keep track of where attention has been, so that it does not return there again?” and answered: “Inhibition of return might help perform this function” (p. 430).

It is our view that either of the two forms of IOR described in the earlier sections of this paper can perform the novelty-seeking, search-facilitating function of minimizing unnecessary return inspections. The output form does so by operating on a priority map which represents the likelihood (perhaps likely value) of different possible next behaviors. The input form does so, as described by Itti and Koch, by operating on a saliency map.

Some outstanding puzzles/questions

There are many puzzles/questions in the literature about IOR and visual search and it is not our intention to be comprehensive in this section. What we have covered below are some of the puzzles/questions that have intrigued us and for which we believe the solutions/answers will be illuminating for scholars interested in attention and visual search. We believe readers will find here suggestions for exciting research.

The measurement of IOR is not straightforward

In their seminal paper Posner and Cohen (1984) suggested that “a peripheral visual stimulus both summons attention and serves to inhibit the processing of further information at that position in space. These two effects appear to be independent and may cancel each other out.” That facilitation due to attention (whether under endogenous or exogenous control) and inhibition can be present simultaneously is now well-established³ (see e.g., Berlucchi et al., 2000; Berger et al., 2005; Chica et al., 2006; Farrell et al., 2010). Thus obtaining precise measurement of IOR is challenged by the possibility that these two factors are potentially contributing whenever we are measuring RT in the cue-target paradigm. Relatedly, the time course of the transition from cue-elicited facilitation to IOR can be highly variable, and is influenced by task demands (Lupiáñez et al., 1997) and attentional control settings (Klein, 2000). Behavioral IOR manifests sooner (i.e., at shorter CTOA) when the target calls for a simple detection response than when it requires a non-spatial discrimination. Moreover, other non-IOR negative effects, such as sensory adaptation, may be contributing to an amalgam of context-dependent inhibitory consequences (Hilchey et al., 2014b).

Individual differences have been shown to influence the measurement of IOR (for a review of some of these, see Klein et al., 2005). IOR has been shown to be quite unreliable within individuals across multiple sessions (Berger et al., 2005), however the use of a central cue-back prior to target onset seemed to partially correct this lack of consistency. Practiced individuals have shown less IOR (Weaver et al., 1998; but see also, Pratt and McAuliffe, 1999), but it is unknown whether the cognitive mechanism(s) underlying IOR attenuate due to practice, or whether observers simply become more efficient at ignoring the task-irrelevant cues.

Finally, in some paradigms IOR might be present but contaminated by binding effects which are covered in the next section.

IOR may be masked by ”binding” effects

Whereas, input- and output-based forms of IOR are robustly observed in model tasks that require easily-formed localization and detection responses (Huffman et al., 2018, for review), IOR may be conspicuously absent in tasks that require participants to engage deeply with one or more stimulus dimensions (e.g., space, color, shape, etc.) in order to form correct responses (Hilchey et al., 2017, for review). This is at least partly because tasks that require greater engagement with stimuli in order to form correct responses often give rise to so-called integration or binding effects (e.g., Hommel, 1998).

In action control theories (Hommel and Colzato, 2004; Hommel, 2009; Frings et al., 2020), integration effects occur when one property of a stimulus (say color) becomes transiently associated with another property of a stimulus (say shape, or the response that was made to it) in episodic memory. Once this binary event representation is formed, stimuli that resemble it will trigger its retrieval if the task demands processing of a stimulus property that is in common with the event representation. When there is a partial mismatch between the retrieved event representation and the imperative stimulus, response times tend to be slowest. When there is a full match between the retrieved event representation and the imperative stimulus or if retrieval does not occur because there is no match, response times tend to be fastest. Notably, in some situations, binding effects co-occur with IOR, which can make it enormously difficult to derive pure estimates of IOR and binding effects. In other situations, binding effects are not a concern because they simply do not occur or occur very weakly.

More specifically, binding effects are rarely a concern in simple model tasks without distractors and with localization responses to each stimulus (Huffman et al., 2018). This is because it is usually not necessary to process non-spatial information in order to form correct responses and each stimulus location is usually associated with a unique response. Binding effects are also usually absent in simple model tasks without distractors and with detection responses to each stimulus, but they can be induced by increasing the amount of spatial processing that is needed to form correct responses (Hilchey et al., 2020). Furthermore, whereas some form of binding may occur in cue-target visual discrimination tasks (Klein et al., 2015), the findings from such tasks appear inconsistent with action control theories. Habituation theories (e.g., Dukewich, 2009) and Lupiáñez's (2010) 3-process account provide better fits.

Binding effects are most robust in tasks that impose significant non-spatial or spatial processing burdens in order to form responses and in which responses are required to successively presented stimuli (i.e., target-target paradigms). They are so robust in standard target-target, 2-alternative forced visual discrimination tasks that there is usually no behavioral trace of IOR in these tasks (e.g., Terry et al., 1994; Hilchey et al., 2017, for review). The same can be said for the visual search tasks used by the spatial negative priming (Frings et al., 2015) and priming-of-popout literatures (Hilchey et al., 2018a, 2019b). Nevertheless, our current view is that the evidence strongly suggests that some form of IOR is in effect in all such cases and is simply overshadowed by episodic retrieval processes that occur after a stimulus has been oriented to (Milliken et al., 2000; Christie and Klein, 2001; Hilchey et al., 2018b, 2019a). However, whether IOR is absent or merely overshadowed by other effects in all cases where processing requirements of a task increase or change (e.g., Smith and Henderson, 2011; Talcott and Gaspelin, 2020; Talcott et al., 2022) remains unclear.

On the relation between measured IOR and search efficiency

In 2006, one of the authors (Klein and Dukewich, 2006) suggested a positive relation between IOR and search efficiency: “If IOR plays a role in serial search by discouraging reinspections, then factors that interfere with IOR should result in decreased serial search efficiency; and, conversely, factors that enhance IOR should increase serial search efficiency” (p. 663). Whereas, these predictions still seem valid consideration of the converse (what happens to IOR when search efficiency is manipulated?) demonstrates that the relation is not so simple. If search is improved by making search less dependent on the inspection of individual items (less serial), then measured IOR during search might be reduced (for a recent example, see Li et al., 2022).

Are the inhibitions encoded retinotopically, environmentally, on objects and/or in scenes?

Posner and Cohen (1984) and Maylor and Hockey (1985) demonstrated that IOR was coded at locations in the environment rather than retinotopically (or, oculocentrically). When ERPs were collected in this paradigm by Satel et al. (2012), targets presented at the retinotopically cued location showed substantial reductions of the sensory-related P1 ERP component and significant, but very little inhibition (~6 ms) while targets presented at the environmental location of the original cue show substantial IOR (~19 ms) and no reduction of the P1 component. This pattern provides converging evidence for what the phenomenon itself demonstrates: That repeated stimulation of an input pathway is not necessary for generating the inhibition. Importantly, in all of these studies the retino/spatio dissociation was effected by interposing one eye movement (Maylor and Hockey, 1985; Satel et al., 2012) or several (Posner and Cohen, 1984) between the cue and target. Because in these experiments it is unclear what would have been the state of the reflexive oculomotor system, we can't be sure whether this environmental coding characterizes the input or the output form of IOR.

Later, Tipper et al. (1991) discovered that inhibition of return could be tagged to an object by cueing an object before it moved predictably in space (see also: Weaver et al., 1998; Tipper et al., 1999; Theeuwes et al., 2014; Smith et al., 2016). Moreover, object-based inhibition of return survived occlusion of the cued object (Yi et al., 2003) and was observed when the objects in the scene moved in random and unpredictable directions (Ogawa et al., 2002). While these studies measured performance with a manual response, Abrams and Dobkin (1994) found object-based IOR effects when measured with a saccadic response in the moving boxes paradigm, as did Tas et al. (2012). However, several others have failed to replicate this object-based saccadic IOR finding, either conceptually (Souto and Kerzel, 2009; Sentürk et al., 2016) or directly (Redden et al., 2018). If the output form of IOR is not tagged to moving objects, then could it possibly be useful in real-world situations such a looking for your child in a busy daycare or at a soccer game?

As noted earlier, studies exploring inhibition of return in the aftermath of a visual search task have demonstrated that the inhibitory tags depend on the persistence of the scene. The importance of the scene (or its removal) has also been observed in the model Posner cue-target paradigm (Redden et al., 2017). These findings led to the idea that removal of the scene would inexorably lead to removal of the inhibitory tags. Importantly, however, even when the scene is removed the inhibition may remain so long as the observer expects the search to continue (Thomas and Lleras, 2009).

What is the cause of the input form?

As established by Posner et al. (1985) when they named the inhibition: inhibition of return, there is little doubt that the cause of the output form of IOR is activation of the oculomotor system. An unresolved question is, “what is the cause of the input form?” Posner and Cohen (1984) identified a form of inhibition that slowed responses to targets which were “handled less efficiently” than targets at uninhibited locations. In all but two of the experiments in that seminal paper eye movements were discouraged and EOG and experimenter feedback were used to reinforce this instruction. Therefore, with the exception of the studies with intentional eye movements, we surmise that the form of IOR generated here was the input form. We also agree with Posner and Cohen's inference that the processing of visual information at the previously cued location in these experiments was made less efficient by the inhibition there. Converging evidence for this belief comes from Smith et al. (2012) who demonstrated that IOR decreased the probability of TMS-induced phosphenes in a paradigm wherein the authors proposed that participants were spontaneously suppress eye movements.

Posner and Cohen proposed that the cause of this effect could be due to the “… inhibitory effect on individual neural cells that occurs with the presentation of a second signal in the visual field occupied by a target” and that the “...inhibition effect is sensory, not attentional, in origin…” (p. 522). Two primary observations led to these conclusions. First, when generated by simultaneous cues at the two possible target locations they observed that the magnitude of the inhibition was as great as compared to following just one cue. This double cue finding was critical because Posner and Cohen thought that the attentional spotlight could not be split between two non-contiguous regions, and therefore the putative inhibition could not have been caused by attentional orienting. Second, there was no inhibition at a location that had previously been attended endogenously and covertly by way of an arrow at fixation.

With the benefit of hindsight, we know now that stimulation of the visual periphery is neither necessary nor sufficient to generate some forms of IOR (e.g., see Figure 4) and the allocation of attention might still be important so long as it is controlled exogenously (see Klein, 2009, for a review of evidence that the forms of covert orienting when controlled endogenously and exogenously are not the same). Consider, for example, that the “inhibition” in Posner and Cohen's double cue condition was estimated from a flawed baseline that confounded the mental state of the participant before the target was presented (i.e., following either a single or double cue). Klein et al. (2005) overcame this flaw by cueing 1–4 of eight equi-eccentric locations prior to presenting a target at one of the eight locations. They found little to no inhibition when the cue array (of 2 or 4 elements) was balanced around fixation and would thus be unlikely to activate a shift of attention or an eye movement. Moreover, following multiple cues whose center of gravity was some distance from fixation, IOR was maximal at unstimulated locations that were in the general direction of this center of gravity (see also, Langley et al., 2011, who replicated many patterns from Klein et al., 2005).

Others believe that repeat stimulation of an input pathway is at least partly responsible for slower responses to targets at cued as compared to uncued locations. For example, habituation theories (e.g., Dukewich, 2009; Klein et al., 2015) and the detection cost theory of IOR (e.g., Lupiáñez, 2010; Lupiáñez et al., 2013) suggest that the magnitude of IOR should be positively related to the physical resemblance between stimuli (cue and target) that are separated in time but not in space. At the crux of habituation theories is the belief that stimulus features that are irrelevant to a response become less likely to generate an orienting response upon repeated presentation because the neurons representing them cease to fire as vigorously. At the crux of the detection cost theory is the belief that the nervous system strives to integrate information that looks like other information over time and space, with increased resemblance between the cue and target increasing the probability that a cognitively demanding integration process will even occur in tasks for which integration is not useful (e.g., stimulus detection tasks).

The behavioral evidence supporting the prediction that the theories have in common is mixed. There is virtually no evidence to suggest that the magnitude of IOR gets bigger with feature matching when detection or localization responses are required to sequentially presented signals in the visual periphery and their onsets are separated by >500 ms (e.g., Hilchey et al., 2018a; Huffman et al., 2018, for review). In the cue-target analogs, where a response is required to the second but not to the first of two signals, earlier studies showed that the magnitude of IOR was similar or only weakly affected by whether cue features repeated as target features (e.g., Fox and de Fockert, 2001; Pratt et al., 2001; Riggio et al., 2004; Taylor and Ivanoff, 2005). More recent studies have shown that IOR may be significantly reduced, if not abolished, when cue features do not repeat as the target (e.g., Hu et al., 2011; Klein et al., 2015). Significant reductions in the magnitude of IOR when non-spatial features switch is a sign that repeat stimulation of input pathways can matter in critical ways, but it remains unclear when or why this occurs and thus what the boundaries are on these theories. Moreover, it is important to keep in mind that even if there were an input-based form in the visual periphery that depended critically on feature matching, this finding alone would not be sufficient to rule out whether a covert, exogenous shift of attention were also necessary to generate the effect.

Whereas the cause of the input form remains undetermined, we suggest that it could be the same as the cause of the output form: activation of the oculomotor system. Critically, when this activation occurs while the reflexive oculomotor system is suppressed, the form generated is the input form. Although we do not think there are any data that directly contradict this proposal Sumner et al. (2004) might provide an indirect challenge. They were able to generate IOR with manual responding using S-cone stimuli as cues while with saccades these same cues did not generate IOR. Because S-cone stimuli were thought to be invisible to the superior colliculus (but see, Hall and Colby, 2014) this finding could be consistent with a cortical locus for the cause of the input form. Regardless, it would be exciting if some readers would devise a direct test of our suggestion.

How are the two forms of IOR implemented neurally?

It was first proposed by Posner et al. (1985) that subcortical circuits played an important role in the generation of IOR. This proposal is supported by the presence of IOR in newborns (Valenza et al., 1994) for whom subcortical but not cortical circuits are relatively mature and by larger IOR in the temporal hemifield field (e.g., Rafal et al., 1989) which is more richly represented (than the nasal hemifield) in the superior colliculus. More directly confirming this proposal, subsequent studies of individual patients with damage to the superior colliculus (Sapir et al., 1999; Sereno et al., 2006) demonstrated that IOR was absent in the direction(s) affected by the lesions. Converging evidence for the importance of sub-cortical circuitry was provided by Gabay et al. (2013) who discovered IOR in the archer fish, a species with limited cortical circuitry.

In two split-brain patients, Tipper et al. (1997) demonstrated that an intact corpus callosum was required for inter-hemispheric transfer of object-based inhibition of return when a cued object crossed the vertical meridian. In patients with damage to the right intra-parietal sulcus, Sapir et al. (2004) demonstrated that IOR following an eye movement was present at the retinotopic location but not at the environmental location (where, as noted above, it was found in normal controls). Converging evidence for the importance of right parietal cortex was later provided by Van Koningsbruggen et al. (2010) who used dual-pulse TMS to disrupt neural circuits in the right and left anterior intraparietal cortices. Spatiotopic coding of IOR was totally disrupted when TMS was delivered to the right but not the left parietal lobe. Thus, whereas the generation of IOR requires an intact superior colliculus, once generated its preservation in the context of object motion and its coding in environmental coordinates seems to depend on cortical circuits.

Single unit recording can be a particularly revealing neuroscientific tool for understanding how a behavioral phenomenon like IOR, might be implemented neurally. By demonstrating IOR in the rhesus monkey Dorris et al. (1999) opened the door to exploring IOR using this tool and later Dorris et al. (2002) discovered a strong relation between the sensory responses of neurons in the superior colliculus and delayed saccades to targets presented at a previously cue location. Importantly, prior to presentation of the target the firing rate of neurons in the receptive field of the cue was higher than baseline and electrical micro-stimulation of these neurons generated faster not slower saccades. These finding suggest that these neurons were not inhibited but rather were receiving signals that were already reduced by some form of inhibition. Whereas, further studies from Munoz's lab (Fecteau et al., 2004; Fecteau and Munoz, 2005) have identified input-based inhibitory effects in the visual and visuomotor neurons of the superior colliculus, we believe that these effects do not last long enough to represent IOR and cannot explain instances of IOR that do not depend on repeat stimulation of an input-pathway as described earlier.

From the point of view of the role of IOR in visual search, studies from Bisley's lab are particularly pertinent. Mirpour et al. (2009) explored the responses of neurons in the lateral intraparietal (LIP) cortex while the monkey performed a visual foraging task. It was found that the responses of LIP neurons were reduced when a previously fixated (as compared to a new) distractor entered the neuron's receptive field. Later, Mirpour et al. (2019) discovered neurons in the frontal eye fields (FEF) whose activity was maintained throughout a trial once the location that these neurons represented had been fixated. It was proposed that these neurons keep track of fixated stimuli and transmit this information to priority maps in parietal cortex. In a recent review of the neuroscience of IOR, Satel et al. (2019) proposed that “such priority maps in parietal cortex, driven by FEF signals, are a likely locus for the inhibitory tags leading to the output form of IOR.”

With only a few exceptions (e.g., Bourgeois et al., 2012, 2013; Satel et al., 2013, 2014) neuroscientific studies of IOR have generally ignored the possibility of the two forms of IOR that we have so clearly distinguished by their different effects. Indeed, despite reflecting a broad range of behavioral and neuroscientific findings, two relatively comprehensive theories presented by Malkinson and Bartolomeo (2018) and by Tian et al. (2011) suffer, in our view, from a similar ignoration. Finally, to our knowledge, no single unit recording studies have been set up to dissociate different forms of IOR. That is, all of these studies measured behavioral IOR using eye movements and in none was an effort made to suppress the reflexive oculomotor machinery. Monkeys can certainly be trained to make manual responses and encouraged to refrain from making unwanted eye movements (e.g., Bowman et al., 1993). Important, missing pieces of the puzzle about the neural implementation of IOR, might be provided if an effort were made to generate neuroscientific data about the input form of IOR, particularly about its cause.

Which form of IOR is operating during visual search?

We believe this is perhaps the most important and interesting puzzle that arises from our review. When search is not explicitly overt, eye movements may still be executed. As such, it is heretofore inconclusive as to whether the form of IOR that is generated in many (all?) search studies is input-based or output-based. Moreover, to our knowledge, with the exception of several experiments reported in Klein and Taylor (1994) (which suffer from the fact that the search scene was removed before the probes were delivered) none of these studies employed eye monitoring to determine (or control) the extent to which eye movements were made. Perhaps the design from Klein (1988) should be repeated with the “keep the display on” caveat while eye movements are monitored and observers are alerted when they make any (as in Klein and Farrell, 1989).

Moreover, even when search is mediated by eye movements (as in the Waldo studies), it is inconclusive as to whether the input or output form is generated and left behind to influence subsequent orienting. Logic would suggest it could be the output form because eye movements are being made and it would seem the reflexive oculomotor system might not be suppressed; but also there is reason to believe it could be the input form because eye movements are not generated so much by peripheral stimuli as by a(n endogenous) plan to find the target.

Ultimately, we believe it is likely that the activation state of the reflexive oculomotor system fluctuates during complex real-world search. When the target/goal for the search episode is robustly encoded, or when there is prior intuition regarding probable target location(s), the reflexive oculomotor system might be tonically suppressed to help avoid unwanted distractions from scene components that are either salient or share properties with the target.

Author contributions

This review is based on a presentation at the August 2022 meeting of the European Conference on Visual Perception made by RK at a symposium entitled Inhibition of Return and Visual Search. The authors of that presentation are the authors of this review. The first draft of the review was generated by RK and missing materials and many editorial improvements were made by RR and MH.

Funding

Most of the research in the KleinLab has been funded by NSERC's Discovery Grants Program.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Footnotes

1. ^Some readers might be uncomfortable calling both forms of inhibition “inhibition of return” as we do in this review. One reason for such discomfort can be found in Hilchey et al. (2014b). After dissociating “late” input and output forms of inhibition, they recommended: “that this late ‘inhibitory' cueing effect (ICE) be distinguished from IOR because it lacks the cause (oculomotor activation) and effect (response bias) attributed to IOR when it was named by Posner et al. (1985)”.

2. ^Prior to this study, IOR had been reported or inferred to be present in 10 of the 24 conditions tested by Taylor. It is noteworthy, particularly in the context of the replication crisis, that Taylor's dissertation found IOR in each of these 10 conditions. Although there are no published replications of Taylor's design, studies published since 2000 on individual or sets of conditions represented in Figure 4, largely confirm what is reported here.

3. ^Indeed, inhibition can even be present at fixation following a stimulus there (Ivanoff and Klein, 2001; Rafal et al., 2006).

References

Abrams, R. A., and Dobkin, R. S. (1994). Inhibition of return: effects of attentional cuing on eye movement latencies. J. Exp. Psychol. Hum. Percept. Perform. 20, 467–477.