Makoto Mizunami
Yukihisa Matsumoto- 1Faculty of Science, Hokkaido University, Sapporo, Japan
- 2College of Liberal Arts and Science, Tokyo Medical and Dental University, Ichikawa, Japan
Revealing neural systems that mediate appetite and aversive signals in associative learning is critical for understanding the brain mechanisms controlling adaptive behavior in animals. In mammals, it has been shown that some classes of dopamine neurons in the midbrain mediate prediction error signals that govern the learning process, whereas other classes of dopamine neurons control execution of learned actions. In this review, based on the results of our studies on Pavlovian conditioning in the cricket Gryllus bimaculatus and by referring to the findings in honey bees and fruit-flies, we argue that comparable aminergic systems exist in the insect brain. We found that administrations of octopamine (the invertebrate counterpart of noradrenaline) and dopamine receptor antagonists impair conditioning to associate an olfactory or visual conditioned stimulus (CS) with water or sodium chloride solution (appetitive or aversive unconditioned stimulus, US), respectively, suggesting that specific octopamine and dopamine neurons mediate appetitive and aversive signals, respectively, in conditioning in crickets. These findings differ from findings in fruit-flies. In fruit-flies, appetitive and aversive signals are mediated by different dopamine neuron subsets, suggesting diversity in neurotransmitters mediating appetitive signals in insects. We also found evidences of “blocking” and “auto-blocking” phenomena, which suggested that the prediction error, the discrepancy between actual US and predicted US, governs the conditioning in crickets and that octopamine neurons mediate prediction error signals for appetitive US. Our studies also showed that activations of octopamine and dopamine neurons are needed for the execution of an appetitive conditioned response (CR) and an aversive CR, respectively, and we, thus, proposed that these neurons mediate US prediction signals that drive appetitive and aversive CRs. Our findings suggest that the basic principles of functioning of aminergic systems in associative learning, i.e., to transmit prediction error signals for conditioning and to convey US prediction signals for execution of CR, are conserved among insects and mammals, on account of the fact that the organization of the insect brain is much simpler than that of the mammalian brain. Further investigation of aminergic systems that govern associative learning in insects should lead to a better understanding of commonalities and diversities of computational rules underlying associative learning in animals.
Introduction
Elucidation of neural systems that mediate appetite and aversive signals in associative learning is an important subject in neuroscience. By associative learning, animals can acquire knowledge in their environments, which allow them, for example, to find suitable food, avoid toxic compounds, and escape from predators. Efforts have been made to elucidate neural systems mediating appetitive and aversive signals in associative learning in many animals, including mammals (Schultz, 2013, 2015), insects (Hammer and Menzel, 1998; Schwaerzel et al., 2003; Mizunami and Matsumoto, 2010; Waddell, 2013), and mollusks (Hawkins and Byrne, 2015). Prediction error, i.e., the discrepancy, or error, between the actual unconditioned stimulus (US) and the predicted US, represents a key determinant for whether a US-paired stimulus is learned (Rescorla and Wagner, 1972; Schultz, 2013, 2015). There is evidence that some classes of midbrain dopamine neurons mediate prediction error signals for appetitive events (Schultz, 2013, 2015), and some researchers have suggested that other classes of midbrain dopamine neurons mediate prediction error signals for aversive events (Matsumoto and Hikosaka, 2009; Matsumoto H. et al., 2016). Other classes of midbrain dopamine neurons control the execution of both appetitively and aversively learned actions (Berridge et al., 2009; Bromberg-Martin et al., 2010).
This review deals with results of our studies on the roles of biogenic amines in appetitive and aversive learning in crickets. Crickets are useful insects for the study of neurotransmitter mechanisms of learning and memory. First, they have excellent capabilities of olfactory and visual learning. For example, they exhibit lifetime olfactory memory (Matsumoto and Mizunami, 2002a), simultaneous memorization of seven pairs of odors (Matsumoto and Mizunami, 2006), context-dependent discriminatory learning (Matsumoto and Mizunami, 2004), and higher-order associative learning such as second-order conditioning (Mizunami et al., 2009) and sensory preconditioning (Matsumoto et al., 2013a). They also exhibit excellent capability to learn color and pattern of visual targets (Unoki et al., 2006; Nakatani et al., 2009; Matsumoto et al., 2013b). Second, applications of pharmacological studies (Unoki et al., 2005, 2006; Matsumoto et al., 2006, 2016; Matsumoto Y. et al., 2009; Mizunami et al., 2014; Sugimachi et al., 2016), gene knockdown by RNA interference (RNAi; Takahashi et al., 2009; Awata et al., 2016), and genome editing by the CRISPR/Cas9 system (Awata et al., 2015) are feasible, thereby greatly facilitating the analysis of molecular basis of learning and memory. Indeed, it can be stated that crickets are one of the best insect models for pharmacological analysis of learning and memory (Mizunami et al., 2013). Third, much information on the brain and behavior of crickets has been obtained as crickets have been used in diverse neuroethological studies (Stevenson and Schildberger, 2013; Hedwig, 2016). We first deal with the recent debate about whether appetite and aversive signals are conveyed by octopamine and dopamine neurons, respectively, as has been suggested in honey bees and crickets, or whether both appetitive and aversive signals are mediated by dopamine neurons, as has been suggested in fruit-flies. Next, we discuss the results of our studies suggesting (1) that activations of octopamine neurons and activation of dopamine neurons are needed for responding to an appetitive conditioned stimulus (CS) and an aversive CS, respectively, and (2) that conditioning is governed by US prediction error and that octopamine neurons mediate the prediction error signals for appetitive learning.
Conditioning Procedures
We have established four different conditioning procedures for crickets (Matsumoto and Mizunami, 2000, 2002a,b; Matsumoto et al., 2015). Among them, we used a “classical conditioning and operant testing procedure” (Matsumoto and Mizunami, 2002b; Matsumoto et al., 2003), which is based on the transfer of memory formed during classical conditioning to an operant testing situation. Crickets were individually placed in a beaker and deprived of drinking water for 3 days to enhance motivation to uptake water. For conditioning of an odor (CS) with water US, a filter paper soaked with an odor was presented to the antennae of the cricket for 3 s, and then a drop of water was applied to the mouth. For conditioning of an odor with sodium chloride US, an odor was presented to the antennae and then a drop of 20% sodium chloride solution was applied to the mouth. Crickets were eager to drink water when it was applied to the mouth, whereas they immediately retracted from sodium chloride solution, indicating that the former serves as an appetitive stimulus and that the latter serves as an aversive stimulus. Odor preferences of individual crickets were tested before and after conditioning. In the test, crickets were individually placed in a test chamber and allowed to freely visit two odor sources, a conditioned odor and a control odor, for 4 min. The time that the cricket spent exploring each odor source with its mouth or palpi was recorded for evaluation of the relative odor preference of each cricket.
For conditioning of a visual pattern, presentation of water or sodium chloride solution to the mouth was paired with either a black-center and white-surround pattern or with its reverse pattern (Unoki et al., 2006). In the pattern preference test, the two patterns were simultaneously presented on the wall of the test chamber, and the time that the cricket spent touching each of the patterns was recorded for evaluating the relative preference between the two patterns. For color conditioning, crickets were presented with purple and green disks paired with water or sodium chloride US (Nakatani et al., 2009), and the two disks were presented simultaneously on the wall of the test chamber for the color preference test.
We also used conditioning of maxillary palpi extension response (MER) with odor CS and water or sodium chloride US, which allowed us to investigate the memory acquisition process (Matsumoto et al., 2015). Crickets often extend their maxillary palpi and then vigorously swing them when a drop of water is applied to their antenna or to the mouth, and we refer to this behavior as the MER. Crickets often exhibited MER to some odors such as vanilla and maple odors, whereas they rarely exhibited MER to other odors such as peppermint and apple odors. We showed that the MER to peppermint or apple odor is increased by pairing the odor with water US (Matsumoto et al., 2015). MER conditioning is analogous to the conditioning of proboscis extension responses (PERs) with odor CS and sucrose US in honey bees (Menzel and Giurfa, 2006; Giurfa and Sandoz, 2012). Moreover, we also observed that the MER to vanilla or maple odor is decreased by pairing an odor with sodium chloride. Therefore, MER conditioning allows appetitive conditioning and aversive conditioning to be achieved in a similar experimental situation, as in the case of a classical conditioning and operant testing procedure.
Roles of Octopamine and Dopamine in Appetitive and Aversive Learning
Previous studies done in the honey bee Apis mellifera (Hammer and Menzel, 1998) and the fruit-fly Drosophila melanogaster (Schwaerzel et al., 2003) suggested that octopamine and dopamine neurons play critical roles in appetitive and aversive olfactory conditioning, respectively (for alternative view, see later section). We first investigated whether this was the case in crickets using a classical conditioning and operant testing procedure (Unoki et al., 2005). Crickets were injected with an octopamine receptor antagonist (epinastine or mianserin) into the hemolymph prior to the conditioning of an odor with water. In a post-training retention test, they did not exhibit an increase of preference for the odor conditioned with water. However, crickets showed normal scores of aversive conditioning with sodium chloride, and the scores being as high as those for control crickets that had been injected with cricket's saline solution. The latter observation indicates that octopamine receptor antagonists do not impair sensory function, motor function, or motivation necessary for learning. We also observed that crickets injected with a dopamine receptor antagonist (fluphenazine, chlorpromazine, spiperone, or flupentixol: Different dopamine receptor types are not discriminated by these drugs, see Mustard et al., 2005) exhibited no aversive learning with sodium chloride US, but appetitive learning with water US was unaffected. Sensory function, motor function, or motivation necessary for learning is not affected by dopamine receptor antagonists. Similar results were obtained in a recent study using olfactory conditioning of MER (Matsumoto et al., 2015). We, thus, suggest that octopamine codes for appetitive signals, and that dopamine neurons transmit aversive signals in two different forms of Pavlovian conditioning in crickets. Notably, crickets that were injected with octopamine or dopamine receptor antagonist exhibited a normal appetitive or aversive response, respectively, when water or sodium chloride solution was applied to the mouth. Hence, these neurotransmitters are not involved in the execution of a behavioral response to appetitive or aversive US.
We also investigated whether the blockade of octopaminergic and dopaminergic transmissions impairs appetitive and aversive conditioning, respectively, of a visual pattern (Unoki et al., 2006) and a color cue (Nakatani et al., 2009). For conditioning of a visual pattern, we observed that crickets injected with an octopamine receptor antagonist (epinastine or mianserin) exhibited no appetitive learning with water, but they exhibited normal aversive learning with sodium chloride solution. In contrast, crickets injected with a dopamine receptor antagonist (spiperone, chlorpromazine, or fluphenazine) exhibited no aversive learning, but appetitive learning was unaffected (Unoki et al., 2006). In color conditioning, crickets injected with an octopamine receptor antagonist (epinastine or mianserin) exhibited impaired appetitive color learning, but aversive color learning was unaffected. In contrast, crickets injected with a dopamine receptor antagonist (flupentixol, fluphenazine, or chlorpromazine) exhibited impaired aversive color learning, whereas appetitive color learning was unaffected (Nakatani et al., 2009). The results indicate that octopamine and dopamine neurons convey signals about an appetitive vs. an aversive US, regardless of the specific paradigm used, thereby suggesting the action of separate neurotransmitter systems to mediate appetitive and aversive signals, respectively, in associative learning in crickets.
Roles of Octopamine and Dopamine in Appetite and Aversive Learning Confirmed by RNAi and Transgenic Crickets
Recent studies on neurotransmitters mediating appetitive and aversive signals for Pavlovian conditioning in the fruit-fly, have yielded conclusions that differ from those obtained in crickets (Burke et al., 2012; Liu et al., 2012). In the fruit-fly, different sets of dopamine neurons mediate appetitive and aversive signals, such as sucrose and electric shock signals, respectively, to intrinsic neurons (Kenyon cells) of the mushroom body (MB), via the type 1 dopamine receptor Dop1, in the MB lobes (Kim et al., 2007; Burke et al., 2012; Liu et al., 2012; Perry and Barron, 2013; Waddell, 2013; Ichinose et al., 2015). Octopamine neurons in the subesophageal ganglion receive sweet taste signals from sugar receptor neurons and relay the signals to dopamine neurons in the protocerebrum that project to the MB lobes (Burke et al., 2012). Therefore, octopamine neurons have a peripheral role for relaying sweet taste signals (Huetteroth et al., 2015), whereas dopamine neurons transmit appetitive US signals to the MB to associate them with an olfactory CS (Burke et al., 2012). Considering that octopamine neurons play roles in mediating appetitive signals in flies, a critical difference between flies and crickets is that dopamine neurons mediate appetitive signals in flies but not in crickets. We considered three possible reasons for this difference, and we investigated them in crickets. The first possible reason is the use of different methods to inhibit dopaminergic signaling: while the use of transgenic techniques in flies allows a sophisticated way to silence dopamine or octopamine signaling, efficacies and specificities of antagonists used in the cricket may not be perfect. For example, a recent study in honey bees suggested that epinastine and mianserin antagonize not only OA1 octopamine receptors but also Dop2 dopamine receptors (Beggs et al., 2011), which raises the possibility that impairment of appetitive learning by epinastine and mianserin might be mediated via blockade of Dop2 receptors, instead of or in addition to OA1 receptors. The second possible reason is the use of different kinds of appetitive US for conditioning. We used water as US in our studies on crickets, whereas sucrose was used in studies on flies except for two studies using water (Lin et al., 2014; Shyu et al., 2017). We, thus, considered the possibility that dopamine conveys sucrose US but not water US in crickets. The third possible reason is that neurotransmitters mediating appetitive signals are not the same in flies and crickets.
For clarifying the issues discussed above, we prepared transgenic crickets with Dop1 gene knockout using the CRISPR/Cas9 system [clustered regularly interspaced short palindromic repeats (CRISPR)/CRISPR-associated protein 9 (Cas9) system; Awata et al., 2015]. Dop1 is known to be highly enriched in the MB in fruit-flies (Kim et al., 2007) and honey bees (Mustard et al., 2005). Dop1 knockout crickets exhibited no obvious abnormality in behavior and external morphology. Our conditioning experiments showed that Dop1 knockout crickets exhibited no aversive learning with sodium chloride US but exhibited normal appetitive learning with water US or sucrose US (Awata et al., 2015). The latter finding indicates that the impairment of aversive learning was not due to the impairment of sensory or motor functions or motivation necessary for learning and for responding to the conditioned odor in the post-training test. The results suggest that Dop1 participates in aversive learning with sodium chloride but not in appetitive learning with water or sucrose in crickets. This differ from the findings in flies in which Dop1 is required for both appetitive learning with water or sugar US and aversive learning with electric shock (Kim et al., 2007; Burke et al., 2012; Liu et al., 2012).
It could be argued, however, that knockout of Dop1 might have caused an abnormality in the development of neural circuits in the brain necessary for aversive learning, not that Dop1 has acute roles in learning in adults. For further clarification of this issue, we investigated the effects of silencing the expression of genes that code the OA1 octopamine receptor and the Dop1 and Dop2 dopamine receptors by RNAi in adult crickets (Awata et al., 2016). In those studies, we used olfactory conditioning of MER to investigate the effect of gene silencing on the acquisition process. Crickets were injected with dsRNA-targeting OA1, Dop1, or Dop2 into the hemolymph and subjected 2 days later to conditioning trials to associate an odor with water or sodium chloride. Studies with quantitative real-time PCR (qPCR) confirmed a significant reduction in the mRNA level of each gene 2 days after dsRNA injection. OA1-silenced crickets exhibited no appetitive learning, but they exhibited normal scores in aversive learning. In contrast, Dop1-silenced crickets exhibited no aversive learning but exhibited normal scores in appetitive learning. Dop2-silenced crickets, as well as control crickets injected with DsRed dsRNA, showed normal scores in both appetitive learning and aversive learning. We, thus, conclude that octopamine mediates appetitive signals via OA1 receptors, whereas dopamine mediates aversive signals via Dop1 receptors in crickets. The perfect agreements of the results of pharmacological, transgenic, and RNAi studies provide decisive evidence that neurotransmitters and receptors that mediate appetitive signals indeed differ in crickets and flies. Our findings in crickets are in accordance with the findings in honey bees, where it has been suggested that appetitive learning is mediated by octopamine neurons via OA1 receptors (Hammer, 1993; Hammer and Menzel, 1998; Farooqui et al., 2003) and that aversive learning is mediated by dopamine neurons (Vergoz et al., 2007; the types of dopamine receptors involved are not known). Neurotransmitters involved in appetitive and aversive learning in other species of insects, however, remain elusive. More studies on various species of insects are needed to elucidate the diversity and evolutionary history of the neurotransmitters in mediating appetite and aversive signals in insects.
In associative learning in mammals, there is evidence that some classes of midbrain dopamine neurons convey signals about appetitive events (Schultz, 2013, 2015), whereas other classes may convey signals about aversive events (Matsumoto and Hikosaka, 2009; Matsumoto H. et al., 2016) (for more details, see Discussion in a later section). Hence, biogenic amines mediating appetitive signals are not the same between crickets and mammals, although the roles of dopamine in mediating aversive signals may be conserved between them. Dopamine has been reported to mediate appetitive signals in the mollusk Aplysia (Brembs et al., 2002). The origin of octopamine signaling for mediating appetite signals in crickets remains to be studied.
Roles of Octopamine and Dopamine in Execution of Appetitive and Aversive Conditioned Responses
We next investigated whether administration of octopamine and dopamine receptor antagonists affects the performance of conditioned responses (CRs; or memory retrieval) after appetitive or aversive conditioning. Crickets were subjected to appetitive or aversive olfactory conditioning and then they received an injection of either octopamine or dopamine receptor antagonist before a retention test (Mizunami et al., 2009). Crickets injected with an octopamine receptor antagonist (epinastine) exhibited no CR to the odor associated with water, whereas they exhibited normal CR to the odor associated with sodium chloride. The latter indicates that epinastine had no effect on sensory and motor functions as well as the motivation necessary to perform a CR. This is in contrast to the finding that crickets injected with a dopamine receptor antagonist (flupentixol) exhibited no CR to the odor conditioned with aversive US but that they showed a normal CR to the odor conditioned with appetitive US. The latter finding indicates that flupentixol had no effect on sensory and motor functions as well as the motivation necessary to perform a CR. After recovery from the effect of the antagonists, crickets exhibited normal CRs. These observations are in accordance with the evidence from honey bees in which a disruption of antennal lobe (i.e., the primary olfactory center) octopaminergic transmission by either the octopamine receptor antagonist mianserin or RNAi of the OA1 gene, disrupted the execution of an appetitive CR (or of appetitive memory retrieval; Farooqui et al., 2003). Moreover, visual pattern conditioning for appetitive or aversive CRs was impaired by injections of an octopamine or dopamine receptor antagonist, respectively (Mizunami et al., 2009). Therefore, we conclude that the execution of appetitive and aversive CRs for olfactory and visual cues requires intact octopaminergic or dopaminergic transmission, respectively.
Our findings were not in accordance with a neural model of classical conditioning proposed by Schwaerzel et al. (2003) (Figure 1A), which was designed to account for the roles of intrinsic neurons (Kenyon cells) and extrinsic (output) neurons of the MB lobes in conditioning of an odor with sugar or electric shock US in the fruit-fly. The model assumed that (1) “CS” neurons (Kenyon cells) carry CS signals and make synaptic connections with dendrites of “CR” neurons (output neurons of the lobes), activation of which leads to a CR, (2) these synaptic connections are silent or very weak prior to conditioning, (3) octopamine and dopamine neurons projecting to the lobes (“OA/DA” neurons) convey signals for appetitive and aversive US, respectively, and make synaptic connections with axon terminals of “CS” neurons (in recent models of fruit-flies, “OA/DA” neurons have been replaced with different sets of DA neurons. See Burke et al., 2012; Liu et al., 2012.), and (4) coincident activation of “CS” neurons and “OA/DA” neurons in conditioning strengthens the efficacy of synaptic transmission from “CS” neurons to “CR” neurons.
Figure 1. Models of classical conditioning in flies and crickets. (A) A model proposed to account for the roles of intrinsic and extrinsic (output) neurons of the mushroom body (MB) in olfactory conditioning in fruit-flies (Schwaerzel et al., 2003). Octopamine neurons and dopamine neurons (“OA/DA” neurons) convey signals for appetitive and aversive US, respectively (In recent models in fruit-flies, “OA/DA” neurons have been replaced to different sets of DA neurons,). “CS” neurons, which convey signals for olfactory CS, make synaptic connections with “CR” neurons that induce a CR, which mimics an unconditioned response (UR). “OA/DA” neurons make synaptic connections with axon terminals of the “CS” neurons. The efficacy of the “CS-CR” synaptic connection is strengthened by coincident activation of “CS” neurons and “OA/DA” neurons by conditioning. For recent elaborations of the model in fruit-flies, see Hige (2017). (B) Our model of classical conditioning proposed for crickets (Mizunami et al., 2009). The model assumes that (1) efficacy of synaptic transmission from “CS” neurons to “OA/DA” neurons is strengthened by conditioning and that (2) coincident activation of “OA/DA” neurons and “CS” neurons is needed to activate “CR” neurons (AND gate) and to produce a CR. Synapses for which the efficacies are modifiable by CS-US pairings are shown as open triangles and marked “modifiable”. Following the terminology of learning theories in mammals, the model in flies is characterized as an S-R model assuming formation of CS-CR connections, while our model is characterized as an S-R and S-S hybrid model assuming formation of CS-CR and CS-US connections. Modified from Mizunami et al. (2009).
We proposed a novel neural model of classical conditioning for the cricket (Mizunami et al., 2009) with minimal modifications of the model proposed for the fruit-fly by Schwaerzel et al. (2003). In our model (Figure 1B), it is assumed that (1) coincident activation of “CS” neurons and “OA/DA” neurons is required for activating “CR” neurons (AND gate) and producing a CR after conditioning and (2) simultaneous activation of “CS” and “OA/DA” neurons from CS/US pairing strengthens the synaptic connection between “CS” and “OA/DA” neurons. Following conventional learning theory, the model proposed for the fly is termed as S-R (or CS-CR) model, as it assumes the formation of stimulus-response (CS-CR) sensorimotor pathways by conditioning, whereas our model is termed as S-R and S-S (or CS-US) hybrid model, which assumes the formation of S-R connections and CS-US connections; the latter of which enables the CS to activate internal representation of US (for details, see Mizunami et al., 2009; Mizunami and Matsumoto, 2010). In our model, the extent by which the CS activates “OA/DA” neurons represents the extent by which the CS predicts the US, and the requirement of activated “OA/DA” neurons for execution of a CR indicates that US prediction guides the execution of the CR, as assumed in S-S learning theory (see Mizunami et al., 2009; Mizunami and Matsumoto, 2010). This is analogous to the findings that some classes of midbrain dopamine neurons govern the execution of learned actions in Pavlovian conditioning in mammals (Balleine et al., 2007; Bromberg-Martin et al., 2010). Following the terminology of human psychology, it has been stated that dopamine neurons confer a “wanting” attribute to the CS to drive actions to seek a US (Berridge et al., 2009). Our findings suggest that motivational mechanisms that govern the execution of a CR in insects are analogous to those in mammals.
Roles of Octopamine in Mediating Prediction Error for Appetite US
Finally, we address the question of what computational rules govern the learning process in crickets. In mammals, a discrepancy, or an error, between the actual US and the predicted US facilitates the classical conditioning for a stimulus paired with the US (Schultz, 2013, 2015). This theory emerged from the finding of “blocking” in rats (Kamin, 1969), in which pairing of stimulus X with US, and subsequent pairing of a compound of stimulus X and another stimulus Y with the US, blocked the learning of stimulus Y. Kamin (1969) argued that blocking requires surprise for learning, whereas learning does not occur when the animal fully predicts the occurrence of the US, and this argument was formulated into the prediction error theory of the Rescorla–Wagner model (Rescorla and Wagner, 1972). Activation of dopamine neurons in the mammalian ventral tegmental area is thought to mediate the prediction error signals for rewarding events in Pavlovian and instrumental conditioning (Waelti et al., 2001; Schultz, 2013, 2015). However, blocking can also be accounted for theories other than the prediction error theory, such as attentional theory (Mackintosh, 1975; Pearce and Hall, 1980) and retrieval theory (Miller and Matzel, 1988), and decisive evidence to discriminate prediction error theory from competitive theories has not been obtained in any learning systems of animals (Miller et al., 1995; Pearce, 2008; Mazur, 2013). Therefore, unambiguous demonstration of the validity of the prediction error theory remains to be achieved.
We performed experiments to investigate whether blocking occurs in classical conditioning in crickets (Terao et al., 2015). No convincing evidence of blocking has yet been obtained in any species of insects. In honey bees, for example, it has been concluded that blocking is not a robust phenomenon (Guerrieri et al., 2005; Blaser et al., 2006, 2008). We first investigated whether blocking of learning of an odor occurs. One group of crickets (blocking group) was subjected to pairing of a visual pattern (X) with a water US (reward) (X+ training) and then subjected to pairing of a pattern (X)-odor (Y) compound with water (XY+ training). An unpaired group received unpaired presentations of a visual pattern (X) and reward and then XY+ training. The blocking group exhibited no learning of the odor (Y), whereas the unpaired group exhibited normal learning of the odor (Y). We found that blocking of visual pattern learning also occurs (Terao et al., 2015). In a test of the prediction error theory, 1-trial XY+ conditioning should be successful, whereas in attentional theory, it should not be successful. We observed successful 1-trial XY+ conditioning, which matches with the prediction error theory but not with the attentional theory (Terao et al., 2015).
We revised our previous model (Figure 1B; Terao et al., 2015) for Pavlovian conditioning, thereby matching the prediction error theory (Figure 2A). How this model accounts for blocking is shown in Figure 2B. We noticed that the model predicted that the application of an octopamine receptor antagonist (epinastine) before Y+ training impairs the learning of Y but does not disrupt the formation of reward prediction by Y (see legend of Figure 2). Therefore, the model predicts that crickets that received Y+ training under the condition of application of epinastine and then Y+ training after recovery from the effect of epinastine exhibit no learning of Y. Indeed, crickets that received such training exhibited no learning of Y (Terao et al., 2015). The “auto-blocking” phenomenon can be easily accounted for by the prediction error theory. However, it cannot be accounted for by any of the competitive theories, as these theories assume cue competition to account for blocking, but it does not occur in an auto-blocking experiment (Terao et al., 2015). The occurrence of blocking and auto-blocking in the same learning system of the same species provides rigorous evidence for validity of the prediction error theory. Moreover, our observation that injection of an octopamine receptor antagonist leads to auto-blocking suggests that reward prediction error signals in crickets are mediated by octopamine neurons. Further neuroanatomical and electrophysiological studies of dopamine neurons are needed to elucidate neural circuit mechanisms for computation of the prediction error in crickets. Investigation is also needed to determine whether dopamine neurons mediate prediction error for aversive US.
Figure 2. Our updated model of the roles of octopamine neurons in appetitive conditioning. (A) Our updated model of classical conditioning established by modifying our previous model (Figure 1B) to match the prediction error theory (Terao et al., 2015). The model retains basic feature of the S-R and S-S hybrid model but assumes the presence of two classes of octopamine neurons, namely, “OA1” neurons that govern enhancement of “CS-CR” synapses (but not execution of a CR) and “OA2” neurons that govern execution of a CR or memory retrieval (but not a conditioning process). OA2 neurons, but not OA1 neurons, govern the “AND gate”. In this figure, we focus on the roles of “OA1” neurons: “OA2” neurons are not illustrated for simplicity. We assume that “OA1” neurons receive no or very weak inhibitory synaptic input from “CS” neurons before training and that the efficacy of the inhibitory synapses is strengthened by CS-US pairing in training. During training, “OA1” neurons receive excitatory synaptic input (triangle) representing actual US and inhibitory input (rectangle) from “CS” neurons representing “predicted US” from the CS. Thus, activities of “OA1” neurons represent US prediction errors. Synapses for which the efficacies are modifiable by CS-US pairings are shown as open rectangles or open triangles and are marked “modifiable”. (B) The model accounts for blocking and auto-blocking. In the figure, “OA2” neurons are omitted to focus on the roles of “OA1” neurons. The model assumes that pairing of a stimulus (CS1) with appetitive US leads to (1) enhancement of inhibitory pathways from “CS1” neurons to “OA1” neurons and (2) enhancement of excitatory synapses from “CS1” neurons to “CR” neurons. During pairing of a compound of CS1 and CS2 with US after sufficient repetition of CS1-US pairing trials, “OA1” neurons are inhibited by activation of “CS1” neurons and thus activation of “OA1” neurons in response to US presentation is inhibited. As a result, enhancement of “CS2-OA1” synapses and “CS2-CR”synapses, in which “CS2” neurons mediate CS2, does not occur. Therefore, no learning of CS2 occurs. The model also predicts that injection of an octopamine receptor antagonist before CS1-US conditioning trials impairs enhancement of “CS1-CR” synapses but not “CS-OA1” synapses. Therefore, no enhancement of “CS1-CR” synapses should occur in subsequent training even after recovery from the effect of the drug. We refer to this phenomenon as auto-blocking. Modified from Terao et al. (2015).
Conclusions
Most animals possess neural mechanisms that allow modification of their behavior for receiving appetitive stimuli and avoiding aversive stimuli. We showed that some octopamine and dopamine neurons play critical roles in appetitive and aversive learning, respectively, and more specifically, the octopamine neurons mediate reward prediction error in appetitive learning in crickets. Moreover, we suggested that some octopamine and dopamine neurons mediate signals about the extent by which the CS predicts the US and such signals drive appetitive and aversive CRs, respectively. Those roles of aminergic neurons in crickets match the S-S learning theory (see Mizunami et al., 2009) and are analogous to the roles of midbrain dopaminergic neurons in the execution of learned actions in mammals (Balleine et al., 2007; Berridge et al., 2009; Bromberg-Martin et al., 2010). We propose that the basic principles of information processing in associative learning are conserved among insects and mammals, on account of the fact that the organization of the insect brain is much simpler than that of the mammalian brain (Mizunami et al., 1999, 2004; Menzel and Giurfa, 2006, Menzel, 2012). Further studies on insect Pavlovian conditioning should pave the way for elucidating the diversity and evolution of associative learning mechanisms in animals.
In addition, our studies have demonstrated that crickets are one of most suitable animals for pharmacological analysis of learning and memory, and crickets may, thus, also be efficient model animals for screening drugs that affect motivational states of animals, and such screening may contribute to therapeutic applications in humans.
Author Contributions
MM and YM wrote the manuscript and approved the final version.
Funding
This study was supported by Grants-in-Aid for Scientific Research from the Ministry of Education, Science, Culture, Sports, and Technology of Japan to MM (No. 16H04814 and 16K18586) and to YM (No. 16K07434).
Conflict of Interest Statement
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
Awata, H., Wakuda, R., Ishimaru, Y., Matsuoka, Y., Terao, K., Katata, S., et al. (2016). Roles of OA1 octopamine receptor and Dop1 dopamine receptor in mediating appetitive and aversive reinforcement revealed by RNAi studies. Sci. Rep. 6:29696. doi: 10.1038/srep29696
Awata, H., Watanabe, T., Hamanaka, Y., Mito, T., Noji, S., and Mizunami, M. (2015). Knockout crickets for the study of learning and memory: dopamine receptor Dop1 mediates aversive but not appetitive reinforcement in crickets. Sci. Rep. 5:15885. doi: 10.1038/srep15885
Balleine, B. W., Delgado, M. R., and Hikosaka, O. (2007). The role of the dorsal striatum in reward and decision-making. J. Neurosci. 27, 8161–8165. doi: 10.1523/JNEUROSCI.1554-07.2007
Beggs, K. T., Tyndall, J. D., and Mercer, A. R. (2011). Honey bee dopamine and octopamine receptors linked to intracellular calcium signaling have a close phylogenetic and pharmacological relationship. PLoS ONE 6:e26809. doi: 10.1371/journal.pone.0026809
Berridge, K. C., Robinson, T. E., and Aldridge, J. W. (2009). Dissecting components of reward: ‘liking’, ‘wanting’, and learning. Curr. Opin. Pharmacol. 9, 65–73. doi: 10.1016/j.coph.2008.12.014
Blaser, R. E., Couvillon, P. A., and Bitterman, M. E. (2006). Blocking and pseudoblocking: new control experiments with honeybees. Q. J. Exp. Psychol. 59, 68–76. doi: 10.1080/17470210500242938
Blaser, R. E., Couvillon, P. A., and Bitterman, M. E. (2008). Within-subjects experiments on blocking and facilitation in honeybees (Apis mellifera). J. Comp. Psychol. 122, 373–378. doi: 10.1037/a0012623
Brembs, B., Lorenzetti, F. D., Reyes, F. D., Baxter, D. A., and Byrne, J. H. (2002). Operant reward learning in Aplysia: neuronal correlates and mechanisms. Science 296, 1706–1709. doi: 10.1126/science.1069434
Bromberg-Martin, E. S., Matsumoto, M., and Hikosaka, O. (2010). Dopamine in motivational control: rewarding, aversive, and alerting. Neuron 68, 815–834. doi: 10.1016/j.neuron.2010.11.022
Burke, C. J., Huetteroth, W., Owald, D., Perisse, E., Krashes, M. J., Das, G., et al. (2012). Layered reward signalling through octopamine and dopamine in Drosophila. Nature 492, 433–437. doi: 10.1038/nature11614
Farooqui, T., Robinson, K., Vaessin, H., and Smith, B. H. (2003). Modulation of early olfactory processing by an octopaminergic reinforcement pathway in the honeybee. J. Neurosci. 23, 5370–5380.
Giurfa, M., and Sandoz, J. C. (2012). Invertebrate learning and memory: fifty years of olfactory conditioning of the proboscis extension response in honeybees. Learn. Mem. 19, 54–66. doi: 10.1101/lm.024711.111
Guerrieri, F., Lachnit, H., Gerber, B., and Giurfa, M. (2005). Olfactory blocking and odorant similarity in the honeybee. Learn. Mem. 12, 86–95. doi: 10.1101/lm.79305
Hammer, M. (1993). An identified neuron mediates the unconditioned stimulus in associative olfactory learning in honeybees. Nature 366, 59–63. doi: 10.1038/366059a0
Hammer, M., and Menzel, R. (1998). Multiple sites of associative odor learning as revealed by local brain microinjections of octopamine in honeybees. Learn. Mem. 5, 146–156.
Hawkins, R. D., and Byrne, J. H. (2015). Associative learning in invertebrates. Cold Spring Harb. Perspect. Biol. 7:a021709. doi: 10.1101/cshperspect.a021709
Hedwig, B. G. (2016). Sequential filtering processes shape feature detection in crickets: a framework for song pattern recognition. Front. Physiol. 7:46. doi: 10.3389/fphys.2016.00046
Hige, T. (2017). What can tiny mushrooms in fruit flies tell us about learning and memory? Neurosci. Res. doi: 10.1016/j.neures.2017.05.002. [Epub ahead of print].
Huetteroth, W., Perisse, E., Lin, S., Klappenbach, M., Burke, C., and Waddell, S. (2015). Sweet taste and nutrient value subdivide rewarding dopaminergic neurons in Drosophila. Curr. Biol. 25, 751–758. doi: 10.1016/j.cub.2015.01.036
Ichinose, T., Aso, Y., Yamagata, N., Abe, A., Rubin, G. M., and Tanimoto, H. (2015). Reward signal in a recurrent circuit drives appetitive long-term memory formation. Elife 4:e10719. doi: 10.7554/eLife.10719
Kamin, L. (1969). “Predictability, surprise, attention and conditioning,” in Punishment and Aversive Behavior, eds B. A. Campbell and R. M. Church (New York, NY: Appleton-Century-Crofts), 279–298.
Kim, Y. C., Lee, H. G., and Han, K. A. (2007). D1 dopamine receptor dDA1 is required in the mushroom body neurons for aversive and appetitive learning in Drosophila. J. Neurosci. 27, 7640–7647. doi: 10.1523/JNEUROSCI.1167-07.2007
Lin, S., Owald, D., Chandra, V., Talbot, C., Huetteroth, W., and Waddell, S. (2014). Neural correlates of water reward in thirsty Drosophila. Nat. Neurosci. 17, 1536–1542. doi: 10.1038/nn.3827
Liu, C., Plaçais, P. Y., Yamagata, N., Pfeiffer, B. D., Aso, Y., Friedrich, A. B., et al. (2012). A subset of dopamine neurons signals reward for odour memory in Drosophila. Nature 488, 512–516. doi: 10.1038/nature11304
Mackintosh, N. J. (1975). A theory of attention: variations in the associability of stimuli with reinforcement. Psychol. Rev. 82, 276–298. doi: 10.1037/h0076778
Matsumoto, H., Tian, J., Uchida, N., and Watabe-Uchida, M. (2016). Midbrain dopamine neurons signal aversion in a reward-context dependent manner. Elife 5:e17328. doi: 10.7554/eLife.17328
Matsumoto, M., and Hikosaka, O. (2009). Two types of dopamine neuron distinctly convey positive and negative motivational signals. Nature 459, 837–841. doi: 10.1038/nature08028
Matsumoto, Y., Hatano, A., Unoki, S., and Mizunami, M. (2009). Stimulation of the cAMP system by the nitric oxide-cGMP system underlying the formation of long-term memory in an insect. Neurosci. Lett. 467, 81–85. doi: 10.1016/j.neulet.2009.10.008
Matsumoto, Y., Hirashima, D., and Mizunami, M. (2013a). Analysis and modeling of neural processes underlying sensory preconditioning. Neurobiol. Learn. Mem. 101, 103–113. doi: 10.1016/j.nlm.2013.01.008
Matsumoto, Y., Hirashima, D., Terao, K., and Mizunami, M. (2013b). Roles of NO signaling in long-term memory formation in visual learning in an insect. PLoS ONE 8:e68538. doi: 10.1371/journal.pone.0068538
Matsumoto, Y., Matsumoto, C. S., Takahashi, T., and Mizunami, M. (2016). Activation of NO-cGMP signaling rescues age-related memory impairment in an insect. Front. Behav. Neurosci. 10:166. doi: 10.3389/fnbeh.2016.00166
Matsumoto, Y., Matsumoto, C. S., Wakuda, R., Ichihara, S., and Mizunami, M. (2015). Roles of octopamine and dopamine in appetitive and aversive memory acquisition studied in olfactory conditioning of maxillary palpi extension response in crickets. Front. Behav. Neurosci. 9:230. doi: 10.3389/fnbeh.2015.00230
Matsumoto, Y., and Mizunami, M. (2000). Olfactory learning in the cricket Gryllus bimaculatus. J. Exp. Biol. 203, 2581–2588.
Matsumoto, Y., and Mizunami, M. (2002a). Lifetime olfactory memory in the cricket Gryllus bimaculatus. J. Comp. Physiol. A Neuroethol. Sens. Neural Behav. Physiol. 188, 295–299. doi: 10.1007/s00359-002-0303-0
Matsumoto, Y., and Mizunami, M. (2002b). Temporal determinants of olfactory long-term retention in the cricket Gryllus bimaculatus. J. Exp. Biol. 205, 1429–1437.
Matsumoto, Y., and Mizunami, M. (2004). Context-dependent olfactory learning in an insect. Learn. Mem. 11, 288–293. doi: 10.1101/lm.72504
Matsumoto, Y., and Mizunami, M. (2006). Olfactory memory capacity of the cricket Gryllus bimaculatus. Biol. Lett. 2, 608–610. doi: 10.1098/rsbl.2006.0540
Matsumoto, Y., Noji, S., and Mizunami, M. (2003). Time course of protein synthesis-dependent phase of olfactory memory in the cricket Gryllus bimaculatus. Zool. Sci. 20, 409–416. doi: 10.2108/zsj.20.409
Matsumoto, Y., Unoki, S., Aonuma, H., and Mizunami, M. (2006). Critical roles of the nitric oxide-cGMP cascade in the formation of cAMP-dependent long-term memory. Learn. Mem. 13, 35–44. doi: 10.1101/lm.130506
Mazur, J. E. (2013). “Chapter 4: Theories and research on classical conditioning,” in Learning and Behavior (Boston, MA: Pearson education), 75–100.
Menzel, R. (2012). The honeybee as a model for understanding the basis of cognition. Nat. Rev. Neurosci. 13, 758–768. doi: 10.1038/nrn3357
Menzel, R., and Giurfa, M. (2006). Dimensions of cognition in an insect, the honeybee. Behav. Cogn. Neurosci. Rev. 5, 24–40. doi: 10.1177/1534582306289522
Miller, R. R., Barnet, R. C., and Grahame, N. J. (1995). Assessment of the Rescorla-Wagner model. Psychol. Bull. 117, 363–386. doi: 10.1037/0033-2909.117.3.363
Miller, R. R., and Matzel, L. D. (1988). The comparator hypothesis: a response rule for the expression of associations. Psychol. Learn. Motiv. 22, 51–92. doi: 10.1016/S0079-7421(08)60038-9
Mizunami, M., and Matsumoto, Y. (2010). Roles of aminergic neurons in formation and recall of associative memory in crickets. Front. Behav. Neurosci. 4:172. doi: 10.3389/fnbeh.2010.00172
Mizunami, M., Matsumoto, Y., Watanabe, H., and Nishino, H. (2013). “Chapter 41: Olfactory and visual learning in cockroaches and crickets,” in Invertebrate Learning and Memory, eds R. Menzel and P. R. Benjamin (Berlin: Springer-verlag), 547–558.
Mizunami, M., Nemoto, Y., Terao, K., Hamanaka, Y., and Matsumoto, Y. (2014). Roles of calcium/calmodulin-dependent kinase II in long-term memory formation in crickets. PLoS ONE 9:e107442. doi: 10.1371/journal.pone.0107442
Mizunami, M., Unoki, S., Mori, Y., Hirashima, D., Hatano, A., and Matsumoto, Y. (2009). Roles of octopaminergic and dopaminergic neurons in appetitive and aversive memory recall in an insect. BMC Biol. 7:46. doi: 10.1186/1741-7007-7-46
Mizunami, M., Yokohari, F., and Takahata, M. (1999). Exploration into the adaptive design of the arthropod “microbrain”. Zool. Sci. 16, 703–709.
Mizunami, M., Yokohari, F., and Takahata, M. (2004). Further exploration into the adaptive design of the arthropod “microbrain”: I. Sensory and memory-processing systems. Zool. Sci. 21, 1141–1151. doi: 10.2108/zsj.21.1141
Mustard, J. A., Beggs, K. T., and Mercer, A. R. (2005). Molecular biology of the invertebrate dopamine receptors. Arch. Insect Biochem. Physiol. 59, 103–117. doi: 10.1002/arch.20065
Nakatani, Y., Matsumoto, Y., Mori, Y., Hirashima, D., Nishino, H., Arikawa, K., et al. (2009). Why the carrot is more effective than the stick: different dynamics of punishment memory and reward memory and its possible biological basis. Neurobiol. Learn. Mem. 92, 370–380. doi: 10.1016/j.nlm.2009.05.003
Pearce, J. M. (2008). “Associative learning,” in Animal Learning and Cognition (New York, NY: Psychology Press), 35–91.
Pearce, J. M., and Hall, G. (1980). A model for pavlovian learning: variations in the effectiveness of conditioned but not of unconditioned stimuli. Psychol. Rev. 87, 532–552. doi: 10.1037/0033-295X.87.6.532
Perry, C. J., and Barron, A. B. (2013). Neural mechanisms of reward in insects. Annu. Rev. Entomol. 58, 543–562. doi: 10.1146/annurev-ento-120811-153631
Rescorla, R. A., and Wagner, A. R. (1972). “A theory of Pavlovian conditioning: variations in the effectiveness of reinforcement and nonreinforcement,” in Classical Conditioning II, eds A. Black and W. R. Prokasy (New York, NY: Academic Press), 64–99.
Schultz, W. (2013). Updating dopamine reward signals. Curr. Opin. Neurobiol. 23, 229–238. doi: 10.1016/j.conb.2012.11.012
Schultz, W. (2015). Neuronal reward and decision signals: from theories to data. Physiol. Rev. 95, 853–951. doi: 10.1152/physrev.00023.2014
Schwaerzel, M., Monastirioti, M., Scholz, H., Friggi-Grelin, F., Birman, S., and Heisenberg, M. (2003). Dopamine and octopamine differentiate between aversive and appetitive olfactory memories in Drosophila. J. Neurosci. 23, 10495–10502.
Shyu, W. H., Chiu, T. H., Chiang, M. H., Cheng, Y. C., Tsai, Y. L., Fu, T. F., et al. (2017). Neural circuits for long-term water-reward memory processing in thirsty Drosophila. Nat. Commun. 8:15230. doi: 10.1038/ncomms15230
Stevenson, P. A., and Schildberger, K. (2013). Mechanisms of experience dependent control of aggression in crickets. Curr. Opin. Neurobiol. 23, 318–323. doi: 10.1016/j.conb.2013.03.002
Sugimachi, S., Matsumoto, Y., Mizunami, M., and Okada, J. (2016). Effects of caffeine on olfactory learning in crickets. Zool. Sci. 33, 513–519. doi: 10.2108/zs150209
Takahashi, T., Hamada, A., Miyawaki, K., Matsumoto, Y., Mito, T., Noji, S., et al. (2009). Systemic RNA interference for the study of learning and memory in an insect. J. Neurosci. Methods 179, 9–15. doi: 10.1016/j.jneumeth.2009.01.002
Terao, K., Matsumoto, Y., and Mizunami, M. (2015). Critical evidence for the prediction error theory in associative learning. Sci. Rep. 5:8929. doi: 10.1038/srep08929
Unoki, S., Matsumoto, Y., and Mizunami, M. (2005). Participation of octopaminergic reward system and dopaminergic punishment system in insect olfactory learning revealed by pharmacological study. Eur. J. Neurosci. 22, 1409–1416. doi: 10.1111/j.1460-9568.2005.04318.x
Unoki, S., Matsumoto, Y., and Mizunami, M. (2006). Roles of octopaminergic and dopaminergic neurons in mediating reward and punishment signals in insect visual learning. Eur. J. Neurosci. 24, 2031–2038. doi: 10.1111/j.1460-9568.2006.05099.x
Vergoz, V., Roussel, E., Sandoz, J. C., and Giurfa, M. (2007). Aversive learning in honeybees revealed by the olfactory conditioning of the sting extension reflex. PLoS ONE 2:e288. doi: 10.1371/journal.pone.0000288
Waddell, S. (2013). Reinforcement signalling in Drosophila; dopamine does it all after all. Curr. Opin. Neurobiol. 23, 324–329. doi: 10.1016/j.conb.2013.01.005
Keywords: octopamine, dopamine, appetitive learning, aversive learning, insects, classical conditioning
Citation: Mizunami M and Matsumoto Y (2017) Roles of Octopamine and Dopamine Neurons for Mediating Appetitive and Aversive Signals in Pavlovian Conditioning in Crickets. Front. Physiol. 8:1027. doi: 10.3389/fphys.2017.01027
Received: 30 September 2017; Accepted: 27 November 2017;
Published: 12 December 2017.
Edited by:
Robert Huber, Bowling Green State University, United StatesReviewed by:
Toshiharu Ichinose, Niigata University, JapanRalf Heinrich, University of Göttingen, Germany
Copyright © 2017 Mizunami and Matsumoto. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) or licensor are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Makoto Mizunami, bWl6dW5hbWlAc2NpLmhva3VkYWkuYWMuanA=