How do we see art: an eye-tracker study

We describe the pattern of fixations of subjects looking at figurative and abstract paintings from different artists (Molina, Mondrian, Rembrandt, della Francesca) and at modified versions in which different aspects of these art pieces were altered with simple digital manipulations. We show that the fixations of the subjects followed some general common principles (e.g., being attracted to saliency regions) but with a large variability for the figurative paintings, according to the subject’s personal appreciation and knowledge. In particular, we found different gazing patterns depending on whether the subject saw the original or the modified version of the painting first. We conclude that the study of gazing patterns obtained by using the eye-tracker technology gives a useful approach to quantify how subjects observe art.

Introduction

What principles use our brain to create an exquisite representation of the external world, the one that allow us to recognize a face, reach to an object or appreciate a piece of art? Scientists have been dealing with this type of questions for quite some time, but the root of the query on how we perceive goes back to ancient philosophy. In fact, it was more than two milleniums ago that Aristotle, the brilliant Greek philosopher, noted that our minds create images, internal representations of the external world, which we use for our thought (Aristotle, De Anima; 431_a, 431_b). Later on, in the nineteenth century, Hermann von Helmholtz developed further this idea and argued that perception involves unconscious inferences from the incomplete information we get from the different senses (von Helmholtz, 1866; Gregory, 1997, 1998). The process of seeing is, therefore, far from a reproduction of the images impinging the retina. It is rather the result of our unique interpretation of ambiguous sensory information. The most notable confirmation of this statement is given by the existence of visual illusions, where assumptions made by our brain can trick or bias our perception, even if we are fully aware that this is happening (Gregory, 1997, 1998; Eagleman, 2001).

Visual information is converted into neural firing patterns in the retina and further processed by the cerebral cortex (Kandel et al., 2000). In cortex, the process of visual perception – i.e., extracting and inferring relevant features of what we see – starts in the primary visual area (V1), in the back of the head, and continues along the ventral visual pathway, going up to the infero-temporal cortex (IT; Logothetis and Sheinberg, 1996; Tanaka, 1996). Although much remains to be understood, after decades of research with single cell recordings in monkeys and fMRI studies in humans, converging evidence have shown that segregated areas in this processing pathway represent different aspects of the visual stimulus, from the encoding of local orientations in V1 (Hubel and Wiesel, 1962), to a more complex representation of faces in IT (Gross et al., 1969), and even concepts in higher, memory related areas (Quian Quiroga et al., 2005). Both using fMRI and single cell recordings, the subjectivity of perception has been clearly shown in different studies using ambiguous or alternating percepts. These studies linked the neuronal firing to the conscious perception by the subjects, which is triggered by the external stimulus but it is dominated by the subjects’ own internal representations (Logothetis, 1998; Leopold and Logothetis, 1999; Kreiman et al., 2002; Quian Quiroga et al., 2008).

One of the most subjective perceptual experiences is given by arts and it is, perhaps, this unique and highly variable personal experience what makes art so attractive. But in spite of this subjectivity, the perception of, for example, a self-portrait of van Gogh is ruled by similar visual perception principles as those involved in recognizing a familiar face, a cat or an apple. We may then ask what processes in our brain make art so especial and, most importantly, how we can scientifically start addressing this question. Such studies require the interaction between art and science, two fields that, with few notable exceptions, as in the case of the genial Leonardo da Vinci, have grown in parallel with only counted interactions. In spite of the impact that the scientific study of art could have, it is somehow understandable that such enterprise is only starting to take off (for pioneering studies see Zeki and Lam, 1994; Gregory et al., 1995; Ramachandran and Hirstein, 1999; Zeki, 1999; Livingstone, 2002; Cavanagh, 2005). On the one hand, art perception is too subjective and challenging for rigorous scientific exploration. On the other hand, artists may fear that scientists could bring a misleading reductionism that would oversimplify all the aspects involved in the appreciation of art.

In an attempt to bridge these two fields, i.e., using scientific methods to study art, in this study we used eye-tracking technology to quantify how subjects look at different art pieces. Far from a reductionist approach, we try to give further insights on how subjects “see art” according to their particular gazing patterns and, besides finding general common principles, we also observed a large variability according to the subjects’ previous knowledge and preferences.

Materials and Methods

Subjects and Stimuli

Eye-tracking recordings were done in 10 subjects (six male, four female; age 23–34) sitting comfortably in front of a 24 inches computer screen, at a distance of about 70 cm, in which different images were shown for 1 min each. Images were digital reproductions (size 1024 × 768 pixels) of original and modified art pieces from Piero della Francesca (Baptism of Christ), Piet Mondrian (Composition No. 8, and Composition II in red, blue, and yellow), Rembrandt van Rijn (Philosopher in meditation), and contemporary artist Mariano Molina (center of gaze). Modified versions of the paintings were done with Photoshop. Subjects were asked to freely view at the images, which were shown in a pseudo-randomized sequence. To balance and evaluate possible learning effects, i.e., seeing the original or the modified version of a painting first, half of the subjects saw the sequence of images in the forward order and the other half in the reverse order.

Recordings and Data Processing

The eye movement data was collected with an EyeLink II system (SR Research) in remote monocular mode with a sampling rate of 1 kHz. Before each stimulus presentation a quick calibration was done using a grid of five data points: one in the center and four displaced 150 mm in the four cardinal directions.

For each image presentation we obtained the spatial coordinates of each fixation. To quantify results, for each figure we defined different regions of interest (as described in the Results section) and compared the relative number of fixations in these regions, between the original and modified versions of the paintings, using T-tests.

Results

Figure 1 shows the eye gazing pattern of a subject looking at the painting “center of gaze” by Mariano Molina, a contemporary artist from Argentina. In this painting the artist blurred most of the image, focalizing the attention of the viewer in a central area. Clearly, the saccades made by this and nine other subjects (see Figure A1 in Appendix) were attracted to this “center of gaze.” In particular, the number of fixations in this area during the 60 s of free-screening of the painting was significantly larger than those in the rest of the painting (right hand side of Figure 1; T-test: p < 10⁻³), in spite of the fact that the area of this region of interest was less than a third of the area of the rest of the canvas. The eye-tracker study in this case confirmed and quantified the degree to which the artist achieved the expected effect in his creation.

FIGURE 1

Figure 1. “Center of gaze” by Mariano Molina with the saccades and fixations of one subject during a free viewing of the painting for 1 min. Note the larger number of fixations to the area in focus in the middle left. The right hand side plot shows the average relative number of fixations for 10 subjects in this region of interest (ROI1, marked in blue in the painting) compared to the rest of the canvas. The lines on top of the bars denote SEM.

The dutch artist Piet Mondrian (1872–1944) is unanimously acclaimed for the mastery of balance and harmony in this paintings, created with vertical and horizontal lines, and an exquisite use of primary colors (Gombrich, 1995). The upper left plot of Figure 2 shows Mondrian’s Composition II in red, blue, and yellow and the saccade patterns of a subject looking at this painting. The balance in this canvas is in part achieved by the use of a small square of blue, with a high contrast with the surrounding white rectangles, and a much larger red patch, but with a lesser contrast with the neighboring rectangles. For this painting we defined four regions of interest, as marked in the figure, and observed that the subject’s gazing was attracted to these regions. The figure in the right, displays a modified version of the painting, in which we altered the balance (high contrast small square vs. low contrast large square) by swapping the blue and the red color patches. Compared to the original, the modified painting looks awkward, out of balance. The eye-tracking pattern of the same subject confirmed this impression, as the fixations were basically constrained to the high contrast area of the large blue square. In fact, it looks as if the fixations were somehow trapped in this area. Although with some variability, there was an overall similar pattern of fixations for the other subjects (Figure A2 in Appendix). To quantify this observation, we calculated the number of fixations during the 60 s presentations in the four regions of interest for all subjects (Figure 2 bottom plots) and statistically compared the number of fixations in large area minus the ones in the other three areas (i.e., ROI1 − [ROI2 + ROI3 + ROI4]) for the original and the modified Mondrian. The difference between the original and the modified version of the painting was significant (T-test: p < 0.05), in agreement with the observation that in the modified version the large blue rectangle attracts more fixations than the rest of the canvas.

FIGURE 2

Figure 2. Original (left) and modified version (right) of Mondrian’s “Composition II in red, blue, and yellow,” with the fixation patterns of a subject superimposed. For both versions of the painting we defined four regions of interest (ROI1 in blue, ROI2 in cyan, ROI3 in yellow, and ROI4 in red) and the average relative number of fixations in each region is shown on the bottom. In the modified version, the large blue patch attracts most fixations, thus breaking down the balance of the original painting.

To further study harmony and balance, as quantified by the number of fixations obtained with the eye-tracker, we used another famous Mondrian painting, the “Composition No. 8” (Figure 3, left hand side). In this case, the balance is obtained with a more complex combination of color (e.g., the red rectangle on the top-left corner) and the density of lines and line crossings (toward the center and bottom right of the painting). In the figure we show superimposed the saccades of a subject, who scrolled most areas of the painting and went back and forth from the top-left red rectangle to the center right line crossings and the salient colors of the very bottom. The right hand side figure shows a modified version of the painting, in which the red rectangle was moved to the bottom right. In this case the subject clearly biased his attention, resulting in a larger number of fixations in the lower right corner. As in the previous case, after a simple change the balance of the painting is completely lost. With some variability, similar results were seen for the other subjects (Figure A3 in Appendix). For this painting we defined three regions of interest, as shown in the Figure, and we calculated the number of fixations in each of these regions. The bottom plots of Figure 3 show the average relative number of fixations in each region of interest for all subjects. To quantify the bias toward the bottom right corner, we statistically compared the number of fixations in the bottom right (ROI3) with the ones in the other two areas (i.e., ROI3 − [ROI1 + ROI2]) for the original and the modified Mondrian paintings. This comparison showed a tendency toward larger number of fixations for the most salient region (ROI3) in the modified picture (T-test: p = 0.069), which did not reach significance due to a large variability in the number of fixations to the region ROI2. However, considering only ROI3–ROI1, the difference between the original and modified painting became significant (T-test: p < 0.05).

FIGURE 3

Figure 3. Original (left) and modified version (right) of Mondrian’s “Composition No. 8,” with the fixations of a subject superimposed. In the original version the subject explored most of the canvas and in the modified version, the gaze is attracted to the bottom right corner. The color rectangles show three regions of interest (ROI1 in blue, ROI2 in green, and ROI3 in red). The relative average number of fixations for all subjects are shown in the bottom plots. There were a larger number of fixations to ROI3 for the modified version.

Next we studied two examples of figurative art. Rembrandt van Rijn (1606–1669) is considered one of the masters of the chiaroscuro, i.e., the use of strong contrasts between light and darkness (Gombrich, 1995). One fabulous example of his technique is given by “Philosopher in meditation” (left painting in Figure 4), where the black background on the left hand side increases the contrast of the white/yellow color of the window, thus giving a formidable reinforcement to the brightness of the sunlight coming through (for a detailed analysis of the use of color and contrast in this painting see Livingstone, 2002). The high contrast used by Rembrandt drives the attention to the figure of the philosopher and the rest of the scene remains somehow in the dark. This produces an extraordinary effect in the painting, as the character in the bottom right appears to be relegated to a secondary role and the spiral stair seems to be leading to a mysterious dark room upstairs. Such descriptions are of course very subjective and hard to be quantified or tested under rigorous scientific experimentation. However, we hypothesized that the removal of the black background on the left and bottom (Figure 4 right hand side), should diminish the saliency of the philosopher and should make the picture look more homogeneous, somehow creating the perception of the picture “opening up,” as if the overall illumination had been increased. The gazing pattern of a subject is shown superimposed to the original and modified versions of the painting. As expected, in the original version there were a large number of fixations to the philosopher and fewer fixations to the character in the right or the rest of the scene, whereas in the modified version the distribution of fixations was more homogeneous. To quantify this observation we defined two regions of interest, around the philosopher (ROI1) and around the other character (ROI2), and calculated the difference between the fixations in these two regions (ROI1–ROI2) both for the original and the modified versions. The bottom panels show the average relative number of fixations for all subjects, where we observe that the difference in the number of fixations between both regions was larger for the original version, with a larger number of fixations to the philosopher. However, this difference did not reach statistical significance (T-test: p = 0.33). The lack of statistical significance was due to a large variability from subject-to-subject, as for this painting the particular and subjective interest of each of the subjects (e.g., an interest in the philosopher, the person in the right, an exploration of the background, previous knowledge of the painting, etc.) completely modified the obtained gaze pattern (see Figure A4 in Appendix). It is also very interesting to note that a different result was obtained for the subjects that first saw the original version, compared to those that first saw the modified version. In the former group, a larger number of fixations to the philosopher was observed both in the original and modified versions (right upper plots in Figure A4 in Appendix). This could be attributed to the fact that, in the modified version the subjects remembered and paid attention to the salient character of the philosopher they saw in the original version. On the contrary, for those subjects who first saw the modified version, the philosopher and the person on the right were at first equally salient and a larger number of fixations to the philosopher were obtained only in the original version (right bottom plots in Figure A4 in Appendix).

FIGURE 4

Figure 4. Original (left) and modified version (right) of Rembrandt’s “Philosopher in meditation,” with the fixations of a subject superimposed. Although the modified version is just a cropping of the original, it appears to be more illuminated and the subject explored the whole painting, whereas in the original version he was more focused on the philosopher. The bottom plots show the relative average number of fixations in two regions of interest (ROI1 in blue and ROI2 in red).

Finally we studied a masterpiece from the Renaissance, “The baptism of Christ” by the Italian master Piero della Francesca (1415–1492). The left hand side of Figure 5 shows the original painting and the right hand side a modified version in which we changed the position of the dove (representing the Holy Spirit). In the original painting there is a vertical symmetry axis formed by the body and hands of Christ, his face and the position of the dove. The superimposed eye patterns of the subject shown in the figure clearly reflect the saliency of this axis, which is not present in the modified version. In fact, in the modified version the eye pattern of the subject seems more erratic, going from one face to the other, and the harmony of the vertical symmetric axis, the key feature of this composition, disappeared. For this painting we defined four regions of interest, as shown in the figure. The average number of fixations for all subjects (Figure 5, bottom plots) shows a larger number of fixations to the dove (ROI3) for the original painting compared to the modified one (T-test: p < 0.05). In contrast, there were no significant differences in the number of fixations to the dove in the modified position (ROI4). This stresses the fact that the dove is more salient when it is part of the abovementioned symmetry axis, but as in the case of the Rembrandt, we also found a completely different pattern of fixations depending on whether the subject looked first at the original or the modified version of the painting. For the subjects that first looked at the modified version (Figure A5 in Appendix, bottom two rows), the dove was not salient (i.e., they were few fixations to it) and when later looking at the original, they spent time going back and forth the actual and previous position of the dove. In other words, they noted the change and ended up looking more times at region ROI4 in the original (even if this was the empty space where the dove was before), compared to the number of fixations in this region when the dove was actually there in the modified version. The subjects that looked at the original first tended to follow the abovementioned vertical axis and when later looking at the modified version, they also focused on the altered position of the dove (Figure A5 in Appendix top two rows). This example clearly shows how the gazing patterns, which we use as a proxy to quantify and study the subjects’ appreciation of art, can be completely modified by the subjects’ previous knowledge; i.e., seeing the original or the modified version of the painting first.

FIGURE 5

Figure 5. Original (left) and modified version (right) of Piero della Francesca’s “Baptism of Christ” with the fixations of a subject superimposed. In the modified version the position of the dove was changed, eliminating the vertical symmetry axis of the original version, something that is clearly observed in the saccade patterns done by the subject. The bottom plot shows the average relative number of fixations in four regions of interest (ROI1 in blue, ROI2 in cyan, ROI3 in yellow, and ROI4 in red).

Discussion

In this study we used eye-tracker technology to quantify how subjects look at art pieces. We found that in spite of the inevitable subject-to-subject variability, there were common basic patterns of fixations. In the painting by Molina, “The center of gaze,” the attention of all the subjects was inevitably attracted to the area in the painting with a sharp resolution. Therefore, the manipulation implemented by the artist acted as a low level visual cue that forced a common pattern of exploration of the painting. We then studied the concept of balance in two Mondrian paintings and showed how simple alterations could destroy the delicate harmony of the composition, biasing the eye toward determined sections of the canvas. Although there was a similar pattern of response for the subjects, the manipulation was more complex than the one used in the “center of gaze” and therefore there was a much larger variability across subjects. These results are in line with previous studies showing changes in the perception of Mondrian’s paintings when varying their orientation, proportional relations and original colors (McManus et al., 1993; Latto et al., 2000; Locher et al., 2005).

In the painting of Rembrandt we varied the contrast and, although there was a tendency in agreement with what was expected, we found that there was a large influence of the subjects’ knowledge, depending on whether they first saw the original or the modified version of the painting. A similar effect was also present for the painting of Piero della Francesca. These examples show how the perception of art is a very complex process conditioned by factors at different levels. On the one hand, there are basic visual principles, such as contrast and saliency, which introduce some uniformity in the gazing pattern of different subjects by driving the attention to particular areas and, on the other hand, there are also more complex cognitive factors, such as previous experience and knowledge, which introduce a large variability across subjects.

In summary, we found some common principles in the way people look at art and a large variability depending on the subjects’ own interests, artistic appreciation, and knowledge. This large subject-to-subject variability makes the scientific study of art very challenging. It is indeed very difficult to find common principles but this lack of uniformity and objectivity is perhaps one of the reasons that make art so unique, personal and fascinating.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

This work was funded by the MRC and the AHRC Beyond Text Program.

References

Cavanagh, P. (2005). The artist as a neuroscientist. Nature 434, 301–307.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Eagleman, D. M. (2001). Visual illusions and neurobiology. Nat. Rev. Neurosci. 2, 920–926.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Gombrich, E. H. (1995). The Story of Art. Phaidon: Oxford.

Gregory, R. L. (1997). Knowledge in perception and illusion. Philos. Trans. R. Soc. Lond. B 352, 1121–1128.

CrossRef Full Text

Gregory, R. L. (1998). Eye and Brain: The Psychology of Seeing. Oxford: Oxford University Press.

Gregory, R. L., Harris, J., Heard, P., and Rose, D. (1995). The Artful Eye. Oxford: Oxford University Press.

Gross, C. G., Bender, D. B., and Rocha-Miranda, C. E. (1969). Visual receptive fields of neurons in inferotemporal cortex of the monkey. Science 166, 1303–1306.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Hubel, D. H., and Wiesel, T. N. (1962). Receptive fields, binocular interaction and functional architecture in the cat’s visual cortex. J. Physiol. (Lond.) 160, 106–154.

Pubmed Abstract | Pubmed Full Text

Kandel, E. R., Schwartz, J. H., and Jessell, T. M. (2000). Principles of Neural Science. New York, NY: McGraw Hill.

Kreiman, G., Fried, I., and Koch, C. (2002). Single neuron correlates of subjective vision in the human medial temporal lobe. Proc. Natl. Acad. Sci. U.S.A. 99, 8378–8383.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Latto, R., Brain, D., and Kelly, B. (2000). An oblique effect in aesthetics: homage to Mondrian (1872–1944). Perception 29, 981–987.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Leopold, D. A., and Logothetis, N. K. (1999). Multistable phenomena: changing views in perception. Trends Cogn. Sci (Regul. Ed.) 3, 254–264.

CrossRef Full Text

Livingstone, M. (2002). Vision and Art: The Biology of Seeing. New York, NY: Abrams.

Locher, P., Overbeeke, K., and Stappers, P. J. (2005). Spatial balance of color triads in the abstract art of Piet Mondrian. Perception 34, 169–189.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Logothetis, N. K. (1998). Single units, and conscious vision. Philos. Trans. R. Soc. Lond. B Biol. Sci. 353, 1801–1818.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Logothetis, N. K., and Sheinberg, D. L. (1996). Visual object recognition. Annu. Rev. Neurosci. 19, 577–621.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

McManus, I. C., Cheema, B., and Stocker, J. (1993). The aesthetics of composition: a study of Mondrian. Empir. Stud. Arts 11, 83–94.

Quian Quiroga, R., Mukamel, R., Isham, E. A., Malach, R., and Fried, I. (2008). Human single-neuron responses at the threshold of conscious recognition. Proc. Natl. Acad. Sci. U.S.A. 105, 3599–3604.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Quian Quiroga, R., Reddy, L., Kreiman, G., Koch, C., and Fried, I. (2005). Invariant visual representation by single-neurons in the human brain. Nature 435, 1102–1107.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

Ramachandran, V. S., and Hirstein, W. (1999). The science of art: a neurological theory of aesthetic experience. J. Cons. Studies 6, 15–51.

Tanaka, K. (1996). Inferotemporal cortex and object vision. Annu. Rev. Neurosci. 19, 109–139.

Pubmed Abstract | Pubmed Full Text | CrossRef Full Text

von Helmholtz, H. (1866). “Concerning the perceptions in general,” in Tratise of Physiological Optics, Vol. III, 3rd Edn (New York, NY: Dover).

Zeki, S. (1999). Inner Vision. Oxford: Oxford University Press.

Zeki, S., and Lam, M. (1994). The neurology of kinetic art. Brain 117, 607–636.

Appendix

FIGURE A1

Figure A1. Saccade patterns for all 10 subjects looking at “center of gaze,” by Mariano Molina.

FIGURE A2

Figure A2. Saccade patterns for all 10 subjects looking at the original and modified versions of “Composition II in red, blue, and yellow” by Mondrian. The upper (lower) two rows show the subjects that looked at the original (modified) version first. The graphs on the right of each row show the average relative number of fixations in each region of interest for the five subjects.

FIGURE A3

Figure A3. Saccade patterns for all 10 subjects looking at the original and modified versions of “Composition No. 8” by Mondrian. The upper (lower) two rows show the subjects that looked at the original (modified) version first. The graphs on the right of each row show the average relative number of fixations in each region of interest for the five subjects.

FIGURE A4

Figure A4. Saccade patterns for all 10 subjects looking at the original and modified versions of “Philosopher in meditation” by Rembrandt. The upper (lower) two rows show the subjects that looked at the original (modified) version first. The graphs on the right of each row show the average relative number of fixations in each region of interest for the five subjects.

FIGURE A5

Figure A5. Saccade patterns for all 10 subjects looking at the original and modified versions of “Baptism of Christ” by Piero della Francesca. The upper (lower) two rows show the subjects that looked at the original (modified) version first. The graphs on the right of each row show the average relative number of fixations in each region of interest for the five subjects.