A Hippocampal Model for Behavioral Time Acquisition and Fast Bidirectional Replay of Spatio-Temporal Memory Sequences

The hippocampus is known to play a crucial role in the formation of long-term memory. For this, fast replays of previously experienced activities during sleep or after reward experiences are believed to be crucial. But how such replays are generated is still completely unclear. In this paper we propose a possible mechanism for this: we present a model that can store experienced trajectories on a behavioral timescale after a single run, and can subsequently bidirectionally replay such trajectories, thereby omitting any specifics of the previous behavior like speed, etc, but allowing repetitions of events, even with different subsequent events. Our solution builds on well-known concepts, one-shot learning and synfire chains, enhancing them by additional mechanisms using global inhibition and disinhibition. For replays our approach relies on dendritic spikes and cholinergic modulation, as supported by experimental data. We also hypothesize a functional role of disinhibition as a pacemaker during behavioral time.

1. Introduction

An animal's ability to acquire and retrieve episodic memories is essential to guide its decision making and how it interacts with the environment. The hippocampus plays an important role in episodic memory encoding (Scoville and Milner, 1957; O'Keefe and Nadel, 1978; Squire and Zola-Morgan, 1991; Eichenbaum and Cohen, 2004), and in episodic memory relay into neocortex via long-term memory consolidation processes (Squire, 1982; Eichenbaum, 1993; Rudy and Sutherland, 1995). During active behavior (encoding phase), hippocampal neurons display asynchronous firing (Vanderwolf, 1969; Buzsáki, 2002), while during quiescence (sleep or quiet wakefulness), the hippocampus exhibits synchronous bursts of activity (sharp-wave ripples) (Buzsáki, 1986, 1989, 2015), and previously stored memories are replayed on a faster timescale (Buzsáki, 1986; Wilson and McNaughton, 1994; Diekelmann and Born, 2010). It is believed that these replays are essential in reorganizing and strengthening memory traces (Girardeau and Zugaro, 2011; Middleton et al., 2018). Replays in sharp-wave ripples have also been observed to occur during wakefulness after reward experiences (Wilson and McNaughton, 1994; Lee and Wilson, 2002; Carr et al., 2011; Foster, 2017); in this case, replays are mostly played in the reverse chronological order (Foster and Wilson, 2006; Csicsvari et al., 2007; Ambrose et al., 2016; Foster, 2017).

While there are evidences (Girardeau et al., 2009; Ego-Stengel and Wilson, 2010; Middleton et al., 2018; Schapiro et al., 2018) suggesting that forward and reverse replays of memories during sharp-wave ripples play an essential role in memory consolidation, it is much less clear how such replays can actually be formed. Memory sequences that happen on the timescale of behavior (order of minutes) have to be stored in an online and one-shot fashion in such a way that subsequent replay is possible on a much faster timescale (order of a couple of hundred milliseconds). In addition, it is not enough to link each item in the memory sequence with its successor, as some items may occur several times in the sequence, but with different successors. Even further, different memory sequences which share many items would introduce severe corruptions in memories stored by linking items with their successors. As the nature of replays is best understood in the context of navigation, we next illustrate these challenges in more detail in this context.

It is known that in the hippocampus there are place cells which fire whenever the animal is at a specific spatial location in a given environment (O'Keefe and Dostrovsky, 1971; O'Keefe, 1976; Harvey et al., 2009). An animal which navigates in an environment, will do so varying specifics of its motion, such as its speed, the stopping locations and the duration of these stops. Spontaneous replays of such events in sharp-wave ripples, however, are independent of such speeds, stopping times or locations of the animal (Wilson and McNaughton, 1994; Nádasdy et al., 1999; Geisler et al., 2007; Davidson et al., 2009; Yamamoto and Tonegawa, 2017), as well during sleep (Wilson and McNaughton, 1994; Lee and Wilson, 2002) as in quiet wakefulness (Foster and Wilson, 2006; Karlsson and Frank, 2009; Carr et al., 2011; Foster, 2017; Pfeiffer, 2017; Yamamoto and Tonegawa, 2017). In addition, if the animal goes through the same location twice, turning left the first time and right the second time, the hippocampus is still able to activate the correct order of the place cells during replay (e.g., left first then right) (Wu and Foster, 2014).

In this work, we propose a possible mechanism that addresses and resolves all challenges mentioned above. We present a model that can store experienced trajectories on a behavioral timescale after a single run, and can subsequently replay (forward and backwards) such trajectories, thereby omitting any specifics as speed, stopping times, etc. Our solution builds on two well-known concepts: (i) one-shot learning (Amit and Fusi, 1994) and (ii) synfire chains (Abeles, 1982). More specifically, we make use of the fact that random connectivity is enough to learn associations between ensembles of cells in a one-shot fashion using standard forms of Hebbian plasticity. Synfire chains, on the other hand, are a well known mechanism that activates neurons in a pre-specified order. Our main contribution is an enhancement (inspired by biological observations) of the standard synfire chain approach that allows the network to alternate between an asynchronous mode, used during encoding of memories, and a synchronous mode, used during replays. We term this enhancement a sequence encoding module. Cells in the sequence encoding modules are called sequence cells. Unlike place cells, which fire at a particular location, sequence cells fire in a predefined position of the sequence, independent of the environment. A sequence ensemble consists of all sequence cells that fire at, say, the fourth position of the sequence; i.e., that correspond to the fourth layer of a synfire chain. We explain in more detail how the sequence encoding modules function in what follows. See also the video which we provide in the Supplementary Material.

Before continuing the discussion on sequence cells, we would like to argue why the recurrent connectivity among place cells alone can hardly reproduce the observed replay phenomena. First, replays of trajectories containing loops become inherently corrupted, as there is no way of locally deciding whether it is the first or second visit to a particular location in the absence of a context signal. In particular, Wu and Foster (2014) indicate that the hippocampal replays are able to correctly decide the direction in a forked environment, where the animal visits the same location multiple times. Second, during multiple runs of the animal, many possible trajectories will be stored for replay. Even if each stored trajectory contains no loops, together each location will be visited many times. If replays used solely the recurrent connectivity among place cells, over time, the produced replays would resemble much more a random walk over the space than a previous experience. While it has been reported that some replays do not reflect previously traversed trajectories (Gupta et al., 2010), the same work reports that a significant fraction of replays does.

For clarity we present our approach in the context of place cell replays, even though it can be used to store any form of spatio-temporal memory sequence. Note that we require no assumptions on the geometry of the environment and on the specific trajectory taken (such as whether it contains loops). The sequence encoding modules operate in two distinct modes: an encoding mode, corresponding to the asynchronous firing hippocampal (theta) state observed during navigation and a replay mode, corresponding to sharp-wave ripples observed during sleep and quiet wakefulness. In the encoding mode, the synapses from sequence cells to the corresponding place cells are learned by standard Hebbian plasticity rules. For this to work, we do not need any specific (learned) connectivity between sequence cells and place cells: random connectivity with low synaptic strength between all sequence cells and place cells is enough. Associations can be learned in an online and one-shot manner between coactive cells by standard Hebbian plasticity. We use a global inhibitory population to ensure that the same sequence ensemble remains active, regardless of how much time the animal spends at a particular location. Once the animal moves to a new location, we postulate that a locomotion signal activates a second group of inhibitory neurons that temporarily inhibits the global inhibitory population, thereby allowing the sequence module to move its active state to the next ensemble. We note that we do not require any specific connectivity for the inhibitory neurons (random connectivity to the sequence cells is enough). The use of disinhibitory input to shape activity patterns so they match behavioral navigation times is motivated by the experimental observation that glutamatergic interneurons in the medial septum effectively control speed-correlated firing of hippocampal neurons (Fuhrmann et al., 2015; Justus et al., 2017). The replay mode, unlike the encoding mode, exhibits extremely synchronous activity as observed in sharp-wave ripples (Buzsáki, 1986, 2015). This synchrony of activity induces dendritic spikes, which produce a fast forward and/or reverse replay of the sequence cells. In replay mode, our sequence ensembles trigger, due to the learned connectivity between sequence ensembles and place cells in the encoding phase, a forward and/or reverse replay of the place cell ensembles in the correct order. Consistent with previous theoretical work (Jahnke et al., 2015) and experimental observations (Hasselmo, 1999; Buzsáki, 2015), our model assumes that replays, dendritic spikes, sharp-wave ripples and cholinergic modulation are tightly interconnected. See the section 3 for a more complete discussion on the assumptions of the model and how they relate to what has been observed in experiments.

2. Results

2.1. Encoding of Place Cell Sequential Activity

We demonstrate that the sequence encoding modules can account for the robust forward and reverse replays observed in the hippocampus. This is done through the simulation of a virtual mouse exploring a predefined environment. A spiking neural network of exponential integrate-and-fire neurons with physiological parameters, emulating the behavior of place and sequence cells, is associated to the mouse.

As the mouse walks through the environment, a sequence of place cells in the spiking neural network will fire, each place cell corresponding to a particular place in the environment (in the model, we do not speculate how place cells could correspond to a particular place in an environment. Instead, when the mouse is at a certain location, we enforce that the corresponding place cell ensemble fires at the correct rate, as is experimentally observed, O'Keefe, 1976). In parallel, the sequence cells of the spiking neural network also fire, but they do so along a predefined sequential activity pattern, which is independent of the environment and the route taken by the mouse. Crucially, after exploration of the environment by the mouse, the sequence cell activity is capable of triggering replays of place cell activity, due to the plasticity of synapses between sequence and place cells. These sequence cells can be used to replay many different sequences of place cell activity. We defer the detailed explanation of the mechanism that allows sequence cells to produce the desired behavior to the next subsection. For ease of understanding we first provide some general intuition relating sequence cell dynamics to the place cell activity.

Consider as an example, a virtual mouse walking on a track, as depicted in Figure 1A. Each place cell corresponds to a specific location on the track. The place cells fire with ≈50 Hz when the mouse is at the corresponding location, and are otherwise silent. As the mouse moves along the track, a sequence of place cell ensembles is activated. The place cells are also targeted by sequence cells and plastic synaptic connections will encode the sequence of place cell activity produced by the motion of the mouse, as shown in Figure 1B. The sequence cells fire following a predefined activation pattern that possesses two modes of operation: the encoding mode and the replay mode. These correspond to the experimentally observed different modes of operation at the hippocampus during exploration and slow-wave sleep (or quiet wakefulness), respectively. In the encoding mode, the next ensemble in the sequence is only activated when the mouse moves to another location. The synapses from sequence cells to the corresponding place cells are learned and, thereafter, sequence cells can activate the corresponding place cell ensembles without requiring external input from the environment, as illustrated in Figure 1C. The applied learning rule is a simple Hebbian plasticity mechanism (a detailed description of the applied learning rule is given in the section 4.2.5).

FIGURE 1

Figure 1. Robust replay of place cell activity. (A) An animal walks along a path, containing a loop in a 2-dimensional environment. (B) Locations in this path are associated to specific populations of CA3 place cells which fire at high rates if the animal is at said location. In this particular scenario, each position is associated with an ensemble of 100 place cells. Concomitantly, the sequence cells fire following a predefined sequential activity pattern that is explained in detail in section 2.2. Hebbian-like synaptic plasticity connects the sequence cells to the corresponding place cell ensembles. (C) The sequence cell dynamics allow for place cell activity replay on a much shorter timescale and independent of the animal's stopping times and locations. (D) 15 s simulation of place cell activity as the animal walks through the track in (A); a rate plot of the place cell ensembles corresponding to each position the animal walks through is shown; the x axis shows time in seconds, the left y axis shows the track positions corresponding to the place cell populations and the right y axis shows the firing rate of each place cell ensemble corresponding to a particular track position. Note that in addition to the loop in the path, the animal occasionally stays arbitrarily longer at particular locations. (E) Fast replay of the path taken by the animal in less than 100ms; a raster plot of place cell activity is shown; the x axis shows time in milliseconds, the left y axis shows the track positions corresponding to the place cell populations. (F) Reverse replay of the path taken by the animal; a raster plot of place cell is shown; the axis are the same as in (E). Note that each weight in the spiking network is multiplied by a gaussian multiplicative noise.

Thus, in the encoding mode, the place cell ensembles are activated in sequence as the animal moves along the track, as shown in Figure 1D. In the replay mode, the sequence cell ensembles are triggered one after the other, in a small time window. Through the synaptic connections that were learned in the encoding mode, the sequence cells trigger a forward replay of the place cells. A raster plot of place cell activity during the replay mode is shown in Figure 1E. Furthermore, the sequence cell ensembles can also be activated in the opposite order, and this produces reverse replay of place cell activities. A raster plot of a reverse replay of place cell activity is shown in Figure 1F. It may be noted that both forward and reverse replay are much faster than the original encoding time, that they are independent of the animal's speed, stopping times and locations, and that the majority of place cells fire in the right order, driven by the activity of the sequence cells. Moreover, these replays can be produced after a single traversal of the path by the mouse.

We emphasize that our approach does not require any particular assumption on the geometry of the environment, the connectivity among the place cells and particularities of the path (such as loops). Sequence cells enable our model to produce replays in such a general setting, and this is discussed in the next subsection.

2.2. Sequence Encoding Modules of Sequence Cells

Sequence cells are organized in a specific structure that resembles a synfire chain. This structure allows the sequence cells to produce the replays observed in place cells in Figure 1. Sequence cells might coexist in the CA3 with the place cells (or might be localized somewhere else in the hippocampus). Figure 2A shows the general structure of sequence cell connectivity, showing an example of what we term a sequence encoding module. The sequence cells can, in principle, compose many such modules that could store many parallel sequences. Each sequence encoding module consists of ℓ ensembles of neurons B₁, …, B_ℓ. Forward connections from B_i to B_i+1 are present. Backward connections, which allow reverse replays, are also present and are generally weaker than the forward connections. These ensembles share two different sources of inhibition: the gating inhibition I_G and the competitive inhibition I_C. The rate dynamics of individual ensembles B_i have two stable fixed points, one at low firing rate (≈0 Hz) and one at high firing rate (≈50 Hz). Henceforth, we term the ensembles B_i bistable units, and refer to B_i as active at a particular time if its instantaneous rate is at the 50 Hz stable point, and inactive if it is at the silent stable point. Furthermore, the gating inhibition I_G is also either active (if its inhibitory neurons fire at a high rate) or inactive (if its inhibitory neurons are silent).

FIGURE 2

Figure 2. Sequence encoding modules. (A) Illustration of a sequence encoding module architecture consisting of 3 bistable units B_i, B_i+1, B_i+2, the competitive inhibitory population I_C and the gating inhibitory input I_G. The inhibitory unit I_C prevents two bistable units B_j from having high rate at the same time, whereas the gating inhibition I_G is the ”break” that prevents activity from propagating from B_j to B_j+1. This break is externally controlled, by some mechanism that recognizes when the animal changes its location. (B) Evolution of the rates of 2 abstract rate units B_i, I_C and I_G, disposed according to the sequence encoding architecture in (A). (C) General illustration of the states the network goes through over time. (i,vi) correspond to stable states in the network while I_G is activated. Inactivating I_G induces a dynamical process that brings the network from state (ii) into state (v). This dynamical process is depicted in further detail below with (iii,iv). In (iii), due to inactivation of I_G, the activity in B_i+1 picks up, driven from the activity B_i. As shown in (iv), once both B_i and B_i+1 are active, they together activate I_C, which gives strong inhibitory input to B_i and B_i+1 and (in v) forces B_i to become inactive, while the recurrent activity permits B_i+1 to recover and stay active. (D) Spiking network of sequence cells. Above, the x-axis shows 1.2 s of simulated activity. The left y-axis shows the indices of the cells in each sequence ensemble, and a raster plot is given. The right y-axis shows the rate of each sequence ensemble. Below, the period where I_G is inactivated is emphasized. During this period, the stability of the system is broken and the dynamics described above take place, leading ultimately to B_i+1 overtaking B_i as the active sequence ensemble. (E) Analogous to (D), but we now purposedly enforce a longer time out of the global inhibitory mechanism. Observe that the system still functions much like in (D), so the period where I_G is turned off is flexible.

As noted in the previous subsection, these sequence encoding modules possess two modes of operation: the encoding mode and the replay mode. In the encoding mode, only one bistable unit is active at a particular point in time, and this bistable unit remains active for as long as necessary, only triggering activity in the next bistable unit and turning itself inactive when the mouse moves to another location. In the replay mode, the bistable units are triggered one after the other in a very brief period of time.

The discussion of the dynamics of sequence cells is split between this subsection and the section 2.3. This is done to distinguish the essential building blocks of our model, presented in this subsection, from specific implementation decisions that are not essential.

2.2.1. Encoding Mode

The encoding mode of the sequence encoding modules relies on a specific, complex rate dynamics while the animal is exploring an environment. To describe them, and to extract the fundamental ingredients that allow a spiking neural network to produce the desired behavior, we first adopt a mean-field approach and collapse the bistable units B_i, the gating inhibition I_G and the competitive inhibition I_C into abstract rate units. This approach is inspired by Litwin-Kumar and Doiron (2014); Zenke et al. (2015). A detailed description of this rate-based system is given in the section 4.1.

At all times, the competitive inhibition I_C guarantees that only one bistable unit B_i can be active at any particular point in time, by realizing essentially a winner-takes-all mechanism. The gating inhibition I_G is responsible for controlling the amount of time B_i remains active. While I_G is active, the currently active bistable unit remains active. If the mouse location changes, I_G turns inactive, which implements a disinhibitory mechanism that allows progression of activity in the bistable units as follows (see the section 3 for experimental evidence relating disinhibition and behavior). As soon as I_G turns inactive, the active bistable unit B_i drives the rate of the inactive bistable unit B_i+1. Eventually B_i+1 turns active and triggers the competitive inhibition I_C. As B_i+1 receives more input than any other bistable unit (if we set the connections from B_i to B_i+1 stronger than the ones from B_i+1 to B_i), it will be the only one active at the end due to the winner-take-all dynamics imposed by I_C. Once B_i+1 has taken over, the disinhibitory signal is removed and I_G turns active again and prevents B_i+1 from activating B_i+2. Figure 2B shows this evolution of activity in a rate-based system that reproduces the desired rate dynamics.

Figure 2C illustrates the most notable states of activity the network goes through. In (i) while I_G is active, B_i remains active; (ii) once I_G is inactive; (iii) the rate of B_i+1 is driven up by B_i; (iv) this, in turn, triggers an increase in the rate of I_C, which (v) forces a competition among B_i and B_i+1; eventually, B_i+1 emerges as the winner and (vi) I_G is again active, preventing B_i+1 from activating B_i+2, while B_i+1 stays active.

Figures 2D,E show the spiking network of sequence cells transitioning between two consecutive patterns and exhibiting the aforementioned rate dynamics. Note that the length of the period in which the gating inhibition I_G is inactive is different in both. Thus, the length of the period in which the disinhibitory signal is given to I_G does not need to be strictly timed. The most important requirement is that it gives enough time for the dynamics happening between (ii) and (v) to reach its stable point.

Furthermore, as will be seen in the next subsection, when the spiking network obeying these rate dynamics is presented in detail, the encoding mode has two defining characteristics other than the combined gating and competitive inhibitory processing mechanisms. They are: (1) neurons in bistable units fire asynchronously; and (2) transition time between one bistable unit and the next is in the order of 50 ms to 100 ms. The first property emerges from a balance of excitation and inhibition in a spiking network of neurons (Van Vreeswijk and Sompolinsky, 1996; Renart et al., 2010) and is crucial in guaranteeing the stable rate dynamics we expect from the bistable units. The latter property is a consequence of having a spiking neural network produce the complex rate dynamics of the sequence encoding modules. Roughly speaking, for the gating inhibition I_G to prevent activation of the next bistable unit, the weights between bistable units must be generally weaker than those inside each bistable unit. Since too high stable rates for a bistable unit B_i are not reasonable, these weights between bistable units units will be weak. Finally, weak weights imply that some time is necessary to increase the membrane potential of the neurons beyond the spiking threshold.

In essence, in the encoding mode, the sequence encoding module activates bistable units in a sequence, with the crucial property that the sequence can be stopped, for an arbitrarily long period, at any particular point in time. This allows it to capture the essential elements of the path explored by the animal, as the sequence transitions to the next element only when a new location is reached. It may be noted that such an ability to produce compressed representations is potentially useful in other contexts. We discuss further applicability in the section 3 of the paper.

2.2.2. Replay Mode

The main distinguishing feature of the encoding and replay modes is the synchrony of activity. In the replay mode, activity progresses through the bistable units in synchronous waves, reminiscent of a synfire chain (Abeles, 1982), in a manner depicted in Figure 3A. This synchrony of activity in a bistable unit yields fast dendritic sodium spikes in the subsequent bistable unit. Dendritic sodium spikes are observed whenever, in a period of a couple milliseconds, the membrane potential at a particular branch of the dendrite of a neuron receives excitatory input above some threshold (Ariav et al., 2003; Gasparini et al., 2004; Polsky et al., 2004; Gasparini and Magee, 2006). Therefore, we assume that the neurons in bistable unit B_i have the axons projecting on close branches of the dendrites of neurons in bistable unit B_i+1. We assume a similar structure for the backward connections, as reverse replays will also be driven by dendritic spikes (in the model, we follow the same approach as Memmesheimer (2010), Jahnke et al. (2012), and Jahnke et al. (2015) and assume that given enough synchronous input dendritic spikes will occur). Whenever B_i is synchronously activated, some branch of the dendritic tree of a neuron in B_i+1 receives strong excitatory input in a few milliseconds, producing a dendritic spike that has a large depolarizing effect on the soma. Figure 3B shows the effect on the soma of one such dendritic spike, in comparison to the effect that would be produced if dendritic nonlinearities were not present.

FIGURE 3

Figure 3. Replay mode of the sequence encoding module. (A) Synfire chain like synchronous activation of three consecutive bistable units B₁, B₂, B₃. Synchrony triggers dendritic spikes in the next bistable unit, which in turn synchronously activate it. Activity moves in a single direction due to neurons entering a refractory period. (B) Nonlinear effect of a dendritic spike on the membrane potential at the soma of a neuron. There is a latency between the moment the dendrite triggers the dendritic spike and its effect on the soma. All dendritic spikes produce the same stereotypical current pulse effect in the soma. (C) Raster plot of the sequence encoding module in the replay mode, without any dendritic spikes, and showing different levels of cholinergic effect (for ACh factor of 5, only the first five spikes are shown). Without any cholinergic modulation (red), or with the cholinergic factor used in the rest of the paper (black), the sequence encoding module does produce a replay of the path, but on a much slower timescale. Each transition, in the same way as during encoding, takes between 50 and 100 ms. If one increases the cholinergic effect beyond what is reported in Hasselmo and Schnell (1994) and Hasselmo et al. (1995), then as shown in the plot (blue), the transitions between consecutive ensembles is around 15 ms in the upper range of what is often reported in the literature (Lee and Wilson, 2002; Diba and Buzsáki, 2007). We emphasize that only the first five spikes are shown in the blue plot. Due to the fivefold increase in excitatory weights and no counterbalance by inhibition, the network becomes too excitable and the neurons spike at very high rates. Still, the blue plot indicates that it would be possible to produce fast replays if the cholinergic effect would be two times or more of what has been reported in the literature, even without using dendritic spikes. On the other hand, reverse replays rely crucially on dendritic spikes and cholinergic modulatory effect and do not happen without either. (D) Raster plot of the sequence encoding module in the replay mode, now with dendritic spikes. Dendritic spikes increase the speed of replays by two orders of magnitude, making it in line with what is observed during sharp-wave ripples.

There is experimental support for the connection between synchronous events and replays, as replays are observed typically during sharp-wave ripple events, the most synchronous pattern of activity in the brain (Buzsaki, 2006; Buzsáki, 2015; Diba and Buzsáki, 2007). In our model, as shown in red in Figure 3C, the sequence cells can also be played asynchronously forward, without the appearance of dendritic spikes due to the lack of synchrony. In fact, this asynchronous slow replay of sequence cells produces replays that resemble what is experimentally observed during REM sleep (Louie and Wilson, 2001). In the model, the forward replays without synchrony are not significantly faster than the original encoding time, with transition between consecutive bistable units is between 50 ms to 100 ms. Such slow transition times are not in line with the speed of forward replays observed during sharp-wave ripples, but they are comparable to the speed of replays observed during REM sleep where sharp-wave ripples are rare and theta activity is predominant (Buzsáki, 2002). Furthermore, without using synchrony and cholinergic modulation, it is not possible to significantly reduce such transition times (see Figure 3C in red). If asynchronous firing of a bistable unit B_i, in a couple of milliseconds, sends too large input to B_i+1, then a simple form of inhibitory gating of activity may not prevent B_i+1 from turning active. Even more, the non-linear effect of dendritic spikes is also crucial if the neurons in B_i are synchronously activated. This is because, without the non-linear effect of dendritic spikes, it is difficult to distinguish high asynchronous firing rates, from synchronous events. Only a couple milliseconds are required for a bistable unit firing at 50 Hz to send forward the same input that it would send if it fired synchronously (lowering the stable rate at which a bistable unit is active would also have undesirable consequences as one needs to be able to clearly distinguish between the low and high firing rate stable points of the bistable units. This will be further discussed in the next subsection).

Figure 3D shows a fast replay of sequence cells produced by the spiking network. Note that the transition time drops to around 5 ms, that the activity in each bistable unit is very synchronous and that most neurons spike once and in the right order.

The second distinguishing feature between the encoding and replay modes is a different cholinergic environment. In the replay mode, we assume lower levels of Acetylcholine (ACh) in the network. This is motivated by the experimental observation that replays typically occur in sharp-wave ripples during slow-wave sleep and quiet wakefulness, where lower ACh levels are documented (Hasselmo, 1999, 2006). It is known that a lower ACh level makes synapses overall stronger and the network more excitable (Hasselmo et al., 1995). In our model, higher excitability is not necessary for forward replays, but it is strictly necessary for reverse replays. Furthermore, higher excitability facilitates faster transition times between bistable units, as well as the occurrence of dendritic spikes. In principle, if there would not be any dendritic spikes, this higher excitability of the network could decrease the transition times between bistable units, even when the bistable units fire in an asynchronous manner. Hence, the higher network excitability would allow for the bistable units to be replayed asynchronously faster than shown in red in Figure 3C, which would in turn allow for replays even without a synchronous mode containing dendritic spikes. However, in order for the replays to be as fast as what is observed in the literature (Lee and Wilson, 2002; Diba and Buzsáki, 2007), the magnitude of the cholinergic effect on the network would have to be much higher than what is reported in Hasselmo et al. (1995) (as shown in Figure 3C in black and blue). Dendritic spikes are therefore necessary for fast replays of activity, in line with what is reported in the literature.

In short, due to slow transition times, the encoding mode of the sequence encoding module cannot account for the fast robust replays observed in the hippocampus. The module needs another mode of activity that has faster transition times. We remark that this different mode of activity is consistent with biology, as these replays are observed during quiet wakefulness and sleep, where the hippocampus is known to be operating in a different regime. Two essential observed properties of the hippocampus in quiet wakefulness and sleep motivate the replay mode: sharp wave ripples and a different neuromodulatory environment, especially, lower levels of Acetylcholine (ACh). Evidence concerning sharp wave ripples indicate that the sequence encoding module in the replay mode might have its bistable units activated in an almost synchronous manner (Buzsáki, 2015). Lower ACh levels, as is observed during sleep and quiet wakefulness (Hasselmo, 1999, 2006), induce higher excitatory synaptic weights which facilitate faster transitions between bistable units.

2.2.3. Reverse Replays

In our model, as mentioned before, the reverse replays rely crucially on the higher network excitability produced by lower levels of ACh, whereas forward replays do not. This is because the rate dynamics of the encoding mode lead to a weight asymmetry condition. The backward connections among bistable units that enable reverse replays must necessarily be weaker than the forward connections among bistable units. In particular, this asymmetry gives a potential function for the (experimentally observed) increased excitatory weights induced by lower ACh levels (Hasselmo, 1999, 2006). For the backward connections to produce synchronous activation of the bistable units in reverse order, they need to be strong enough to produce dendritic spikes. However, as they are weaker than the forward connections, in the encoding mode, asynchronous activation of a bistable unit should not produce dendritic spikes in the next one. If the excitatory synaptic weights are not generally increased in the replay mode it is difficult to have, at the same time, dendritic spikes at the backward connections in the replay mode and a fully functioning encoding mode. One either gets instability in the encoding mode, or not enough dendritic spikes in the replay mode. Therefore, reverse replays are made possible by the difference in the neuromodulatory environment, in addition to sharp-wave like synchronous activity triggering dendritic spikes.

Observe, additionally, that dendritic spikes have a refractory period of about 5 ms (Memmesheimer, 2010; Jahnke et al., 2012, 2015) and that within this refractory period the respective dendrites do not transmit any spikes. It is this refractory period that forces the synchronous wave to travel in a single direction. This is also shown in Figure 3A. In this case, if bistable unit B_i is synchronously activated by dendritic spikes from B_i−1 (from B_i+1), it will produce dendritic spikes only in B_i+1 (in B_i−1), as the neurons of B_i−1 (of B_i+1) are in the refractory period, respectively. Moreover, Figure 3D shows a reverse replay of activity produced by the spiking network.

In essence, the asynchronous encoding mode has a preferred direction for the activity to travel, as the forward connections between bistable units B_i and B_i+1 are stronger than the backward connections between B_i and B_i−1. Dendritic spikes, cholinergic modulation and the refractory period of neurons allow the sequence cells to be activated in both directions, depending on which bistable unit the synchronous volley starts.

While the previous discussion is constrained to the setting of our model, in general, there are other possible ways of enabling the sequences to be traversed only forward during encoding but in both directions during replays. For example, replays and reverse replays can be generated by symmetric connection weights between sequence modules as done in Chenkov et al. (2017) and unidirectional sequence propagation during encoding could be enforced by short-term synaptic depression (Stevens and Wang, 1995; Markram and Tsodyks, 1996; Abbott et al., 1997).

2.3. Sequence Encoding Modules—Spiking Network

In the previous subsection, the essential building blocks of sequence encoding modules were discussed on a conceptual level. In this subsection, we describe more detailed conditions that are necessary so the dynamical system comprising our spiking network model behaves as intended.

2.3.1. Bistable Units

In the spiking network, each bistable unit consists of a recurrently connected population of excitatory and inhibitory cells as depicted in Figure 4A. To produce bistable units where the rate dynamics maintain two stable fixed points, one around 0 Hz and one around 50 Hz, we analyse the gain function (Gerstner et al., 2014) of excitatory and inhibitory neurons as shown in the left of Figure 4B. There, the rate of excitation (red) and inhibition (teal) is shown for varying recurrent excitatory (inhibitory) and compared with the identity (black). Formally, to obtain the stable points of activity one has to solve the system of equations: r_E = F(r_E, r_I) and r_I = G(r_E, r_I), where F and G represent the evolution of the rate of excitatory and inhibitory neurons when receiving excitatory rate r_E and inhibitory rate r_I. For the plot, the implicit solution r_I(x) = G(x, r_I(x)) is approximated for all x. For the excitatory gain function, the function F(x, r_I(x)) (in red) is plotted and compared to the identity (in black). An analogous procedure is done to plot the inhibitory gain function G(r_E(x), x), where we now define r_E(x) implicitly via the equation r_E(x) = F(r_E(x), x).

FIGURE 4

Figure 4. Spiking network. (A) Illustration of a bistable unit consisting of both an excitatory and an inhibitory population. (B) On the left, the gain function of excitatory (red) and inhibitory(teal) neurons inside a bistable unit B_i is depicted. As shown in the center, if I_G is turned on, then the shape of the gain function produces two stable states, one at 0Hz and another at approximately 50Hz. The stable states are shown on the right. (C) Analogous to (B), but now with I_G turned off. As seen on the left, the gain function shape changes when receiving input from the previous pattern and the stable state at 0Hz ceases to exist. The network always settles for the only stable point around 50 Hz, a fact which is shown on the right.

Note that the excitatory gain function is below the identity for low rates in Figure 4B but not in Figure 4C. This implies that there is a stable point at 0 Hz while the gating inhibition I_G is active, even when receiving input from the previous bistable unit, but this stable point disappears when I_G is turned inactive. This allows the activity to move forward. On the other hand, in both Figures 4B,C, the excitatory gain function is above and then below the identity around the corresponding stable points. This characterizes the stable point as well as the stable region around it.

2.3.2. Encoding Mode

The spiking network operates in the encoding mode if the initial excitatory input that activates the first bistable unit does so in an asynchronous manner. If the bistable units exhibit the correct rate dynamics, then the gating and competitive inhibitory mechanisms guarantee that it behaves as intended, again, by a through analysis of the gain functions of excitatory and inhibitory neurons. From the conditions given by the rate dynamics we have: (1) if the gating inhibition I_G is inactive, then the input sent by B_i to B_i+1 should be enough to activate B_i+1; (2) if I_G is active, then B_i remains active; and (3) while I_G is active, B_i+1 remains silent. Figures 4B,C explain the functioning of the gating inhibitory mechanism. In Figure 4B observe that if I_G is activated then the bistable unit B_i maintains both high and low rate stable points, even when receiving input from the previous bistable unit.

Now, we explain in a little more detail how to choose I_G based on the conditions in the previous paragraph. Essentially, condition (2) imposes an upper bound on the weight between I_G and the ensembles B_i (which depends only on the weight parameters inside an ensemble B_i); condition (3) imposes a lower bound on the weight between I_G and the ensembles B_i (which depends only on the weight of synapses between two consecutive ensembles). The upper and lower bounds for the weight of I_G have to be consistent, meaning the parameters have to be chosen in such a way that the lower bound is smaller than the upper bound. Condition (1) itself only influences I_G indirectly by imposing a lower bound on the weight of the connection between two consecutive ensembles (which depends only on the weight parameters inside the ensembles). Typically, for our network to behave as intended, one chooses the weight between consecutive ensembles close to the lower bound imposed by condition (1). Then, one checks the upper bound on the weight from I_G to B_i imposed by condition (2) and the lower bound on the same quantity imposed by condition (3).

The behavior of the network changes once I_G is inactivated. If bistable unit B_i receives input from the previous bistable unit then the stable point at 0 Hz disappears and only the 50 Hz stable point remains. This is shown in Figure 4C. Observe that if the gain functions behave as shown in Figures 4B,C then the spiking network will satisfy conditions (1), (2), and (3) from above.

We observe a few additional points. Increasing the rate of I_G decreases the gain functions of excitatory and inhibitory neurons. This enlarges the stable region around 0 Hz, but shrinks the stable region around 50 Hz, and also shifting to a slightly smaller rate value. Thus, satisfying the three conditions, corresponds to not shrinking the stable region around 50 Hz by too much in (2) and enlarging the stable region around 0 Hz enough to satisfy (3). This observation also implies that it is not feasible to have the high firing rate stable point at too small firing rates, as satisfying all three conditions is difficult.

Again, from the rate dynamics we see that: once I_G is inactive, and B_i+1 gets active, the competitive inhibition I_C will spike and guarantee that only B_i+1 remains active. The competitive inhibition I_C functions here as a coincidence detector of two active bistable units. It remains inactive if only a single bistable unit is active, but once two or more are active, its neurons receive enough excitatory input to spike. Such behavior is a simple consequence of the thresholding of the membrane potential of neurons to produce an action potential. With such a one spike volley, I_C gives strong enough inhibitory input to quickly inactivate B_i, but B_i+1 recovers and keeps active as it gets excitatory input from B_i and from itself. As soon as the activity B_i+1 stabilizes, the gating inhibition I_G turns active and prevents activity from going onwards to activate B_i+2. In particular, these observations indicate how the weight parameters between I_C and the ensembles B_i should be chosen. Essentially, the weight from B_i to I_C should be such that if a single B_i is active, I_C is not active. However, if two or more B_i are active then I_C neurons should start spiking. This condition is pretty easy to guarantee due to the threshold of the membrane potential. In addition, the weight from I_C to B_i is chosen large enough to guarantee that if I_C is active then its activity is strong enough to inactivate the patterns which are active.

The last observation we make about the spiking network in the encoding mode is that one should decide when to turn I_G active again. To simplify things, we turn I_G active again as soon as the rate in the next ensemble in the sequence stabilizes. In our simulations, we did not observe that it was necessary to be too strict about the moment that I_G is turned active again (as illustrated by Figures 2D,E), as long as I_G was turned active after the activity in the next ensemble in the sequence had stabilized.

2.3.3. Replay Mode

The spiking network is in the replay mode if the first bistable unit is activated in a synchronous manner. This synchronous activation of the neurons in a bistable unit triggers dendritic spikes in neurons in the next bistable unit, which in turn, trigger synchronous spikes at the corresponding neurons and the activity can continue in a synchronous wave, much like in a synfire chain (Abeles, 1982). Such activity travels only forward because the dendrites have a refractory period of 5 ms after transmitting a dendritic spike to the soma.

2.3.4. Reverse Replay

The replay mode of the spiking network can produce reverse replays if the last bistable unit is activated synchronously. As in the forward replays, the activity proceeds synchronously through dendritic spikes. Again, it has a single direction of motion, due to the refractory period of the dendritic compartments that produce dendritic spikes. We emphasize that the increased synaptic weights in the replay mode, due to lower ACh levels, are essential if one wants the encoding and reverse replay modes to reliably behave as intended.

2.4. Robustness to Noise and Parameter Variations

We want to point out that the chosen parameters are biologically plausible and do not need to be fine-tuned. To this end, we produce Figure 5, showing that the choice network ensemble sizes and the choice of network weights are not crucial for the spiking network to behave as intended. In Figures 5A,B we double the network ensemble sizes, while halving all the weight parameters (alternatively, one could halve the connection probability). The encoding and replay modes function the same way as before. In Figures 5C,D we change all the used parameters following the guide of the gain functions and show another set of parameters where the encoding and replay modes behave as intended. We observe that while the conditions of the dynamical system imply one cannot take a single parameter and change it significantly, one can change the parameters if a certain set of reasonable conditions is ensured (in other words, one has to respect the coupling among different parameters).

FIGURE 5

Figure 5. Robustness to parameter variations. (A) Raster plot of the encoding mode of the sequence encoding module with 6 patterns. The size of ensembles B_i and I_C are doubled, while the connection probability is halved (alternatively, one can half all corresponding weight parameters). This shows that the size of each ensemble is not important for the model to work. (B) Analogous to (A) but now the replay mode is shown. Again each ensemble size is doubled. (C) Raster plot of the encoding mode of the sequence encoding module, with an alternative parameter set (see Table 6). This shows that the choice of parameters we used is not unique. Instead, as long as the conditions on the gain functions are satisfied, there is a large range of weight parameters where the model performs as expected. (D) Raster plot of the replay mode using the alternative parameter set (see Table 6).

As a final note, we emphasize that the plots in Figure 1 were produced using a gaussian multiplicative noise on each individual weight. It is not crucial for the spiking network that each individual weight is as defined, only that the total average weight inside a particular ensemble or between ensembles is close to the desired values.

3. Discussion

3.1. Summary

In this work, we show a model which can store in behavioral time place cell activity traces corresponding to paths run by an animal and produce fast bidirectional replays of these place cell activity traces during sleep and quiet wakefulness, even after a single run of the path. Furthermore, our approach works independently of the geometry of the environment and of particularities of the path taken (such as loops present in it). As observed in the literature, the replays produced by our model have constant speed and are not influenced by the potentially varying speed of the mouse during the run. As the primary conclusion of our model, we predict the existence of a population of sequence cells, dedicated to the task of encoding of sequential activity, which enables the model to properly deal with loops in the path of the mouse. This is an experimentally verifiable hypothesis, as sequence cell firing should be distinguishable from place cell firing, as well as from head direction cells, border cells and so on [time cells, (Eichenbaum, 2014) could pose a challenge as sequence cells also measure time in a certain sense]. Sequence cells enable behavioral time storage of paths traversed and subsequent spontaneous fast replay of place cell activities as observed in rodents during slow-wave sleep and quiet wakefulness. The sequence cells are organized in sequence encoding modules where, crucially, the activity progresses through stereotypical sequences of ensembles. The sequence encoding modules possess two modes of operation one used during encoding and another during replay. The encoding mode relies on a disinhibitory mechanism to control the speed of asynchronous activity progression in the ensembles. The replay mode relies on synchronous activity and resulting dendritic spikes to trigger fast replay of the sequence encoding modules. Lower levels of ACh, leading to higher excitability, allow the sequence encoding modules to be played in reverse—which would also account for the observed reverse replays of place cell activity in rodents. In addition, reverse replays happen mostly after rewards, which in our model, permit the sequence to be played in reverse to find the causes that produced the reward. In particular, a very long sequence encoding module can be used to store multiple paths, by just using the reverse replays to mark the relevant starting and ending positions in the sequence. Effectively, this allows the sequence to be split up in many subsequences, each corresponding to a relevant memory.

In the upcoming subsections, we discuss the bioplausibility of the properties we used, such as how the sequence encoding modules can be formed, the form of inhibitory processing, the dendritic spikes and the different cholinergic environment in movement and quiescence.

3.2. Formation of Sequence Encoding Modules

The sequence encoding modules proposed here rely on complex dynamics and a specific connectivity structure. Their structure can either be hardwired, as a product of evolution, or perhaps more likely, emerge as a consequence of the interaction of the many plasticity rules present in the hippocampus. For example, sequences of ensembles forming long synfire chains can spontaneously emerge as a product of spike-timing dependent plasticity (STDP) and axon remodeling (Jun and Jin, 2007), or a three factor variant of STDP (Frémaux and Gerstner, 2015) which is modulated by global population activity (Weissenberger et al., 2017). Such long sequences of ensembles form the basis of the sequence encoding modules in here.

Furthermore, while the sequence encoding modules presented in this paper require a defined connectivity structure, in the brain they would likely emerge as a consequence of plasticity (or due to evolution). Forming the sequence encoding modules through learning should produce sequence cells that are more heterogenous and whose ensembles overlap (as for example in Weissenberger et al., 2017). In particular, in Weissenberger et al. (2017), it has been shown that such heterogeneity and ensemble overlap poses no problem, as through proper learning rules the ensemble sequences can be learned to exhibit the desired behavior. Our sequence encoding modules can be described with very few parameters, so using learning rules to induce the correct behavior is (at least theoretically) relatively simple.

In addition, in Steemers et al. (2016), experimental evidence is presented for attractor dynamics in the hippocampus. Their data shows that linear changes in the environment context induce dynamic, non-linear changes in hippocampal activity patterns. Moreover, Pfeiffer and Foster (2015) present evidence for autoassociative and heteroassociative dynamics in the hippocampus. Densely connected populations representing attractor states (autoassociative dynamics) are sparsely connected among themselves (heteroassociative dynamics), a characteristic which permits sequential processing of information. Their data reveals a fundamental discretization in the memory retrieval processes of the hippocampus, supporting the idea that sequences of attractor states are sparsely connected, as is the case in the sequence encoding modules. Finally, Grosmark and Buzsáki (2016) report two different firing dynamics in place cells, one of rigid and fast-firing pyramidal neurons, and another of plastic and slow-firing neurons. Our work provides an interesting interpretation of the reasons for such difference in firing dynamics, with the rigid, fast-firing neurons essentially functioning as our sequence cells and driving replays of the plastic neurons.

3.3. Inhibitory Neural Processing

Crucial to the proper functioning of the encoding mode of the sequence encoding modules are the two forms of inhibitory neural processing, the gating inhibition and the competitive inhibition. Speed-correlated firing of hippocampal neurons has been reported and is controlled by glutamatergic interneurons in the medial septum (Fuhrmann et al., 2015; Justus et al., 2017). The firing rates of these glutamatergic interneurons are proportional to the speed of locomotion and control the theta oscillations during locomotion, being crucial to the initiation of theta oscillations. These glutamatergic septo-hippocampal projections perform disinhibitory control of the activity in areas CA3 and CA1, which is in line with our proposal of a gating inhibitory mechanism that is switched off during movement, possibly via some disinhibitory mechanism.

Furthermore, a specific type of inhibitory interneurons, called VIP interneurons, are known to have a disinhibitory function and seem to be ubiquitous in the brain (Letzkus et al., 2011; Jiang et al., 2013; Pi et al., 2013). These VIP interneurons perform disinhibitory control in the brain which is activated under specific behavioral conditions (Pi et al., 2013), a fact which is consistent with our hypothesis that the gating inhibition is switched off when the animal changes its location.

The winner-take-all dynamics implemented using the competitive inhibition are commonplace in the theoretical neuroscience literature (Maass, 2000; Binas et al., 2014). Furthermore, anatomical studies have shown that cortical networks contain essential features which can be implemented via winner-take-all circuits (Douglas and Martin, 2004; Carandini and Heeger, 2012).

As the above examples show, there exists a large diversity of inhibitory interneuron subtypes each potentially serving a different purpose (Petilla Interneuron Nomenclature Group et al., 2008). Such diversity of interneuron types and functions is a strong indicator that the simple forms of inhibitory control proposed here can be realized in the brain.

3.4. Sharp-Wave Ripples, Dendritic Spikes and Replays

Previous works (Jahnke et al., 2015) also suggest the relation between dendritic spikes, sharp-wave ripples and replays. Our work goes along similar lines, suggesting that replays are a product of synfire chain like behavior, driven by dendritic spikes, of special populations of neurons, that fire in a stereotypical manner, independent of instantial specifics of an animal's movement. Sharp-wave ripples are the most synchronous events reported in the hippocampus (Ylinen et al., 1995). The replay of place cell activities is known to be concurrent with ripple events (Nádasdy et al., 1999; Lee and Wilson, 2002). Such synchronous events favor the occurrence of dendritic spikes, which, in turn, favor synchronous excitatory activity along a feedforward structure (Jahnke et al., 2015; Haga and Fukai, 2018).

3.5. ACh Levels and Neuromodulators

It is known that Acetylcholine levels are significantly reduced in slow-wave sleep and quiet wakefulness, in comparison to an awake state (Kametani and Kawamura, 1990; Marrosu et al., 1995; Hasselmo, 1999). It is further believed that such change in the cholinergic modulation is crucial for the process of acquiring and consolidating new memories (Hasselmo and Bower, 1993; Hasselmo, 1999, 2006). In accordance with this work, Hasselmo (2006) proposes that different levels of Acetylcholine during exploration, slow-wave sleep and quiet wakefulness set the appropriate dynamics of the network for encoding and consolidating memories (replays). In fact, septal cholinergic activation to CA3 pyramidal neurons has been experimentally shown to enhance theta oscillations and effectively suppress sharp-wave ripples in the CA3 network (Hasselmo, 1999, 2006; Vandecasteele et al., 2014). The nicotinic receptors either on the soma, pre-synaptic boutons, or post-synaptic spines of CA3 pyramidal cells can effectively inhibit firing of CA3 pyramidal neurons. Another potential inhibitory mechanism could be through the GABA-B receptors on CA3 pyramidal neurons, which can serve a similar role, although the origin of inputs to GABA-B receptors are currently unknown.

Lastly, a recent study (Singer and Frank, 2009) indicates that replays, in both directions, are more likely to occur after rewards are obtained. In particular, their results suggest that reverse replays could be a mechanism to reinforce experiences that produced rewards. This connects nicely to our work, as replays (and especially reverse replays) in our model rely on higher excitability of the network which does not need to happen exclusively due to lower levels of ACh. Other neuromodulators, such as dopamine, have been reported to possess increased levels in the presence of rewards (Arias-Carrión et al., 2010), and these increased levels typically induce higher excitability of neurons (Henze et al., 2000; Kobayashi and Suzuki, 2007). However, there are reports indicating inhibitory effects of dopamine (Stanzione et al., 1984; Smiałowski and Bijak, 1987), and, to the best of our knowledge, it is currently unknown whether the total neuromodulatory effect in the presence of rewards is excitatory or inhibitory in the hippocampus.

3.6. Other Models of Sequence Replay

There are alternative models that achieve partially similar results to the ones reported here. These previous works fail to produce replays which cope with the irregularity of behavior and need specific assumptions about the geometry of the environment or about the trajectory taken (e.g., that it does not contain loops). In Romani and Tsodyks (2015), a short-term synaptic depression mechanism is shown to produce two different regimes of activity of place cell rate dynamics, that can account for replays observed in the literature (Wilson and McNaughton, 1994; Lee and Wilson, 2002) and preplays (Dragoi and Tonegawa, 2011; Ólafsdóttir et al., 2015; Grosmark and Buzsáki, 2016), at least under the assumption that a map of the environment is stored in the synaptic connections among the place cell rate units. In Theodoni et al. (2018), it is studied how standard plasticity rules (Markram et al., 1997; Bi and Poo, 1998) shape the patterns of activity of place cell rate units and it is found that the theta rhythm speeds up encoding of newly explored environments. The mechanism for generating the replays is the same as the one used in Romani and Tsodyks (2015). In Chenkov et al. (2017), a spiking network consisting of Hebbian assemblies that are weak and sparsely connected uses pre-existing recurrent connections to quickly amplify weak signals. As in our work, the assemblies are activated like a synfire chain during replays and network states are altered by globally changing neuronal excitability (say, by cholinergic modulation). The authors of Jahnke et al. (2015), in line with our work, propose that sharp-wave ripples and replays are interconnected and that nonlinear dendritic computation, in the form of dendritic sodium spikes, accounts for the observed replays in the rodent hippocampus. As in this work, the dendritic spikes distinguish between an asynchronous state present while the animal is awake and a synchronous activity state, related to sharp-wave ripples, occurring during slow-wave sleep and quiet wakefulness. In Haga and Fukai (2018), dendritic processing is used in the generation of replays and preplays of activity. In an hypothesis similar to ours, CA3 possesses innate sequential structure that gives rise to spontaneous firing sequences and can be used to one-shot learn place fields. Still, we note that our sequence encoding modules are the first to account for a compressed representation of place cell activity to produce replays, both forward and reverse, which are independent of the instantial specifics of the animal's movement and environment.

3.7. Compressing Sequence Representations in Other Contexts

In this work, we restrict sequence processing to a navigation task because the hippocampus is extensively studied in this regard and there are many reported phenomena (sharp-wave ripples, cholinergic modulation, forward and reverse replays, dendritic spikes) which fit well into our framework of sequence encoding modules.

Still, the ability to divide complex tasks into simpler, compounding, sequential subtasks is essential to understand and perform the original more complex task. Identifying these subtasks and their interconnectedness, compressing useful sequences into the most relevant parts and factoring out the non-relevant ones, is an important characteristic if one wants to learn ever more complex behavioral patterns. It is a given that animals perform this form of sequence processing operation on a high-level. In this work, it is further argued that this sequence processing is such an essential operation, that the brain might develop special modules of cells capable of extracting the relevant subparts contained in a more complex task.

3.8. Limitations and Outlook

While our model produces near perfect replica of replays and reverse replays of previous trajectories, doing so after a single traversal of the path, there are many avenues which are still left to be explored and which go beyond the scope of this work.

First, our model does not include the observed theta phase precession (O'Keefe and Recce, 1993) and theta sequences (Foster and Wilson, 2007). Theta phase precession corresponds to the fact that place cells fire progressively earlier in the theta phase as the animal traverses the place field. As a result, theta sequences, that is sequences of place cells firing corresponding to the movement of the animal, naturally emerge. This is currently not considered in our model. In principle, it is possible to incorporate theta waves and phase precession by increasing the level of detail in place cell modeling. However, this goes beyond the scope of our work as the replay phenomena is already reproduced even with a simple model of place cell activity. This thus preclude our model from offering insight into how theta waves interplay with the replay/ripple phenomena. However, it also shows that theoretically theta waves are not strictly necessary to obtain robust bidirectional replays in a one shot fashion, which indicates that the presence of theta waves in conjunction with ripples cannot be fully understood exclusively from the point of view of replays.

Second, the firing rates occurring in our simulations are typically higher and more uniform than what has been reported in the literature (Mizuseki and Buzsáki, 2013). However, the high firing rates of our model could be significantly decreased by combining it with some mechanism which induces excitability decay during immobility (e.g., short-term synaptic depression Stevens and Wang, 1995; Markram and Tsodyks, 1996; Abbott et al., 1997). The increase in excitability during movement would greatly facilitate the task of transitioning from one sequence ensemble to the next (while staying at the current ensemble during immobility), and, as a consequence, the firing rates of active sequence ensembles would not need to be as high. In particular, there is indirect experimental support for such a mechanism (Kay et al., 2016), as hippocampal neurons have been reported to possess significantly reduced firing rates during immobility. Furthermore, in our model, place cells may have lower firing rates than sequence cells, as the only condition is that the combination of place cell firing and plasticity rule used must be adequate to associate the place cell ensembles to the currently active sequence ensemble. In particular, lower firing rates of place cells as opposed to sequence cells is consistent with the results reported in Grosmark and Buzsáki (2016). Finally, the sub-populations we simulate are precisely those which represent the current behavior of the mouse and, thus, must have naturally higher firing rates than the rest of the network. Therefore, the entire network naturally has more realistic firing rate profile.

Third, we restrict our discussion and analysis to the initial learning of a sequence in a single environment. There are some possibilities about how the sequence cells would behave during the second run of an already traversed path. For example, each new run, a new sequence encoding module may be selected and the place cells are associated to it (independent of whether this path had been traversed before). Alternatively, the same sequence of place cells may be associated to the corresponding sequence encoding module used in the first run of this path (for example, by associating the place cells to the corresponding sequence cells). The latter case is less likely though, as over multiple runs, the place cells would tend to be associated to many sequences which leads again to the problem that the context cannot be decided during replays (as was the case when replays are produced exclusively by the place cell connectivity). However, there is a whole range of intermediate cases where the sequence encoding module used would not be neither completely independent of the path it is associated to, nor completely determined by this path. This is an interesting matter because if sequence cells are associated to a specific path, their firing pattern would match closely that of place cells, whereas if they would be independent their firing would not appear as that of place cells. In particular, as place cells are known to remap across different environments (Alme et al., 2014), this indicates that sequence encoding modules are rarely reused. Consequently, sequence cells might remap depending on the environment, just like place cells. Observe that, as shown in Weissenberger et al. (2017), it is possible to pack many such sequence encoding modules in a population of neurons.

Lastly, we recognize that our model fails to shed light on how certain replays seem to correspond to novel virtual experiences, seemingly combining prior experiences to create a novel one (Gupta et al., 2010; Pfeiffer and Foster, 2013). However, this phenomena should rely on the ability to produce coherent internal models of the environment, which can thus be used to produce coherent virtual experiences. Therefore, comprehending such phenomena is clearly out of the scope of our model.

4. Materials and Methods

4.1. Reduced Rate Encoding Model

To understand the dynamics of the spiking network, we consider a reduced model where we collapse ensembles into abstract rate units. As before, we denote by B₁, …, B_ℓ the ℓ bistable units that form the sequence, by I_C the competitive inhibitory population to all ℓ ensembles and by I_G the gating inhibitory input. The activity of unit X at time t is given by λ_X(t) which corresponds to the rate of the ensemble. The units are connected into a network which dictates their interaction. We denote the interaction from unit X to itself by w_X, and the interaction strength from unit Y to X by w_XY. The connectivity is as in Figure 2A. We assume that interactions with an inhibitory (excitatory) source negatively (positively) affect the target's rate. Moreover, we assume that the activity λ_IG(t) alternates between high and low rate states as depicted in Figures 2B,C. The rate of other units evolves according to the following set of differential equations:

$\begin{array}{rc}\frac{d{\lambda }_{B_i}(t)}{dt}=f(I_{B_i}(t))-{\lambda }_{B_i}(t)& (1)\end{array}$

for the rate of unit B_i and

$\begin{array}{rc}\frac{d{\lambda }_{I_C}(t)}{dt}=s*(g(I_{I_C}(t))-{\lambda }_{I_C}(t))& (2)\end{array}$

for the rate of unit I_C, where I_Bi(t) is:

$\begin{array}{l}{I}_{B_i}(t)=w_{B_i}{\lambda }_{B_i}(t)+w_{B_i{B}_{i-1}}{\lambda }_{B_{i-1}}(t)+w_{B_i{B}_{i+1}}{\lambda }_{B_{i+1}}(t)\\ +w_{B_i{I}_C}{\lambda }_{I_C}(t)+w_{B_i{I}_G}{\lambda }_{I_G}(t)& (3)\end{array}$

and I_IC(t) is:

$\begin{array}{cc}{I}_{I_C}(t)={\displaystyle \sum _{i=1}^{\ell }{w}_{I_C{B}_i}}{\lambda }_{B_i}(t),& (4)\end{array}$

the functions f and g are generally chosen as saturating functions such as the sigmoid function $\frac{e^x}{1+e^x}$ and the factor s multiplying the right-hand side of Equation (2) controls the speed of inhibitory dynamics relative to excitatory dynamics in the reduced rate model. These equations represent how the rate of the populations in the spiking network should evolve, with f and g being gain functions of neurons and I_Bi(t) and I_IC(t) being the total input received by the respective neurons. The goal is to set the parameters of the system such that the activity evolves as in Figure 2C.

To produce the dynamics of Figures 2B,C, it is enough to study this rate model with two bistable units B_i and B_i+1, with the additional constraint that weights from and to B_i are equal to those from and to B_i+1. This allows for a simple extension to ℓ bistable units, as the symmetry of the weights guarantees that the system will have the same rate dynamics independent of which bistable unit is currently active. In particular, with only two bistable units, it is possible, under the assumption that f and g are piecewise linear, to write a simple dynamical system of equations that can be fully solved and understood.

Moreover, it is crucial that f is chosen to guarantee bistable rate dynamics of each B_i, that is, specifically that close to zero or negative input produces silent (zero rate) responses and that large enough inputs produce a maximal rate response. In addition, from g it is required that it guarantees the behavior we expect from I_C, namely, that it serves as a detector for two bistable units being active at the same time, producing maximal response if it receives input from two bistable units close to the maximal response, and being silent if it receives not much more than what a single maximally active bistable unit can produce.

Given this, one can state simple requirements that are essential: (1) that the system is at a stable point while B₁ is at its maximal rate, B₂ silent and I_G is at the high rate state. (2) that B₁ can activate B₂ to its maximal rate once I_G is at the low rate state; (3) that the rate of I_C increases when both bistable units are jointly active; (4) I_C being active is strong enough to turn silent both B₁ and B₂; (5) the speed of the inhibitory I_C dynamics is faster than that of B₁ and B₂, which can force B₁ below the point where it can reactivate itself, while B₂ remains above said point. In particular, the speed of inhibitory I_C dynamics (controlled by s) is crucial to guarantee that the rate system can produce the desired rate dynamics with symmetric weights.

These requirements of the rate-based system, therefore, imply that one of three possible outcomes can happen, where only the third is desired: (1) B_i and B_i+1 turn both inactive; (2) the rates of B_i, B_i+1 and I_C oscillate; (3) B_i turns inactive, but B_i+1 stays active, driven by activity in B_i and by its own dynamics.

4.1.1. Overview of Parameters—Rate Model

Table 1, we summarize the notation and parameters used in the rate reduced model.

TABLE 1

Table 1. Summary of parameters for the rate model.

4.2. Spiking Network

4.2.1. Neuron Models

In the spiking version of the model each ensemble unit is composed of an excitatory (E_x) population with N^E units and an inhibitory (I_x) population with N^I units where the lower-case subscript x corresponds to the index of the pattern. We follow the convention of denoting the populations by superscript and the neuron index by subscripts. The membrane potential dynamics of neuron i in population X follow:

$\begin{array}{l}\frac{d}{dt}{V}_i^X(t)=\frac{1}{{\tau }^X}\left(E_L^X-V_i^X(t)+{\Delta }_T^X\exp \left(\frac{V_i^X(t)-V_{T,i}^X(t)}{{\Delta }_T^X}\right)\right)\\ +\frac{g_i^{XE}(t)}{C}(E^E-V_i^X(t))+\frac{g_i^{XI}(t)}{C}-\frac{w_i^X(t)}{C}.\end{array}$

The parameter values are summarized in Table 2 and the dynamics are consistent with (Clopath et al., 2010; Litwin-Kumar and Doiron, 2014). The main difference between the excitatory and inhibitory units is that the excitatory neurons have an adaptation current and an adaptive threshold (Brette and Gerstner, 2005; Badel et al., 2008) whereas the inhibitory neurons are non-adapting. The adaptive threshold follows:

$\frac{d}{dt}{V}_{T,i}^{E_x}=\frac{1}{{\tau }_T}\left(V_T-V_{T,i}^{E_x}(t)\right).$

We record a spike when the voltage exceeds 20mV and we reset it to and hold it at V_re for the refractory period duration τ_abs. For an adaptive threshold we set it to V_T + A_T after a spike. The adaptation current for a neuron in population E_x follows:

$\frac{d}{dt}{w}_i^{E_x}(t)=\frac{1}{{\tau }_w}\left(a_w\left(V_i^{E_x}(t)-E_L^{E_x}\right)-w_i^{E_x}(t)\right)$

and it is increased by b_w when neuron i spikes.

TABLE 2

Table 2. Summary of parameters for membrane dynamics.

4.2.2. Synapse Dynamics

Connection probability between population X and Y is given by p^XY and the strength of a connection between unit i ∈ X and unit j ∈ Y is given by $w_{ij}^{XY}$. The total conductance of neuron i in population X from a source of type Y (excitatory or inhibitory) follows:

$g_i^{XY}(t)=F^Y(t)*\left(w_{\text{ext}}^{XY}{s}_{i,\text{ext}}^{XY}(t)+{\displaystyle \sum _{Y_k\in Y}{\displaystyle \sum _j{w}_{ij}^{X{Y}_k}}}{s}_j^{Y_k}(t)\right),$

where $F^Y(t)=\frac{1}{{\tau }_d^Y-{\tau }_r^Y}(e^{-t/{\tau }_d^Y}-e^{-t/{\tau }_r^Y})$ is the synaptic kernel for input from neurons of type Y, Y_k corresponds to subpopulations of type Y, * is the convolution operator and $s_j^{Y_k}(t)={\displaystyle {\sum }_{\ell =1}^n\delta (t-t_{i,\ell }^{Y_k})}$ denotes the spike train of neuron j in population Y_k. Additionally, each neuron receives external input given by the spike train $s_{i,\mathrm{\text{ext}}}^{XY}(t)$ which follows a homogeneous Poisson process with rate $r_{\mathrm{\text{ext}}}^{XY}$ and weight $w_{\mathrm{\text{ext}}}^{XY}$ for neurons in population X where Y can either be an excitatory (E) or an inhibitory (I) source. The resolution of the simulation is 0.1 ms and the coupling parameters can be found in Table 3.

TABLE 3

Table 3. Summary of parameters for neuronal coupling.

4.2.3. Dendritic Spikes

Dendritic spikes are triggered by excitatory spikes arriving in a short time span at a particular position in the dendrite of a neuron. We incorporate them by setting the synaptic conductance of the afferent input to produce a stereotypical current pulse, using parameters observed in single-neuron experiments (Ariav et al., 2003; Gasparini et al., 2004; Polsky et al., 2004; Gasparini and Magee, 2006) and similar to the one used in recent spiking network models (Memmesheimer, 2010; Jahnke et al., 2012, 2015). Whenever the excitatory input to a (dendritic branch of) a neuron changes the conductance of the neuron by a threshold θ_dspike in a time window of 2 ms, a dendritic spike is initiated. This produces a stereotypical current pulse at the soma whose effect is triggered with a time lag of τ_DS after the dendritic threshold is crossed. This time lag is on the same order as the one observed in single neuron experiments. The stereotypical current pulse, like that of Jahnke et al. (2015), is described by:

$\begin{array}{l}{I}_{ds}=\Theta (t-{\tau }^{DS})\\ \left[-A{\exp }^{\left(-\frac{t-{\tau }^{DS}}{{\tau }^{ds,1}}\right)}+B{\exp }^{\left(-\frac{t-{\tau }^{DS}}{{\tau }^{ds,2}}\right)}-C{\exp }^{\left(-\frac{t-{\tau }^{DS}}{{\tau }^{ds,3}}\right)}\right]\end{array}$

where the factors A, B, C and the decay time constants τ^ds,1, τ^ds,2, τ^ds,3 are chosen so that the depolarizing effect on the soma fits experimental data (Ariav et al., 2003; Gasparini et al., 2004; Polsky et al., 2004; Gasparini and Magee, 2006), and whose values are shown in Table 4. Moreover, we add a refractory period t^ref,ds = 5 ms, as in Memmesheimer (2010), Jahnke et al. (2012), and Jahnke et al. (2015), to these dendritic branches after the stereotypical current pulse has influenced the soma, in accordance to the saturating effect of somatic depolarization of dendritic spikes reported in the literature (Ariav et al., 2003).

TABLE 4

Table 4. Summary of parameters for dendritic spikes.

4.2.4. Cholinergic Modulation

In accordance with (Hasselmo and Bower, 1993; Hasselmo, 1999, 2006), we use the fact that, during exploration, high levels of cholinergic modulation are present and the excitatory feedback is thus in part suppressed; during sleep and quiet wakefulness, the ACh levels drop and this suppression is reduced, thus leading to generally stronger synaptic weights. We use a simple 2.5 factor to increase the strength of excitatory synapses during the replay mode in comparison with the encoding mode. Though not too much data is available on the cholinergic effect in vivo, this factor is in line with the one obtained from hippocampal slices of areas CA3 and CA1 in rats (Hasselmo and Schnell, 1994; Hasselmo et al., 1995).

4.2.5. Hebbian Plasticity From Sequence to Place Cells

To connect a bistable unit in the sequence encoding module with the corresponding population of place cells, we use a Hebbian like plasticity mechanism. Initially, all synapses from sequence to place cells are silent, and they are potentiated and turned active if the corresponding pre and post cells fire at high rates in a short period. The exact implementation was done by a voltage-based STDP plasticity rule (Clopath et al., 2010) which tends to increase synaptic weight if both pre and post neurons have a high firing rate. As the process of activating a silent synapse is equivalent to LTP (Isaac et al., 1995; Kerchner and Nicoll, 2008), we speed up the dynamics to make it consistent with the timescale of LTP. In accordance with (Clopath et al., 2010; Litwin-Kumar and Doiron, 2014), the dynamics of the synaptic strength w_ij from sequence cell j to place cell i are given as:

$\begin{array}{l}\frac{d}{dt}{w}_{ij}(t)=-A_{LTD}{s}_j(t)R\left(u_i(t)-{\theta }_{LTD}\right)\\ +A_{LTP}{x}_j(t)R\left(V_i(t)-{\theta }_{LTP}\right)\left(v_i(t)-{\theta }_{LTP}\right)\end{array}$

where R(x) is a linear-rectifying function, that is, R(x) = 0 if x ≤ 0 and R(x) = x otherwise; u_i and v_i represent the membrane voltage V_i low-pass filtered with time constants τ_u and τ_v, respectively; and x_j represents the spike train s_j low-pass filtered with time constant τ_x. The constants A_LTD, A_LTP, θ_LTD, θ_LTP, τ_u, τ_v and τ_x are given in Table 5 and are the same as Litwin-Kumar and Doiron (2014), with the exception of A_LTP, which is increased to speed up the dynamics and bring the timescale of activating a synapse into the hundred milliseconds range. Furthermore, the maximum value of w_ij is 5pF.

TABLE 5

Table 5. Summary of plasticity parameters.

4.3. Overview of Parameters

Here we summarize the parameters of the spiking neural network simulation of exponential integrate-and-fire neurons. Tables 2, 3 contain the parameters regarding the membrane potential dynamics of a neuron, and the parameters regarding the neuronal coupling, respectively. Table 4 contains the dendritic spikes parameters that were used, and Table 5 the Hebbian plasticity ones.

For Figure 1, we use a multiplicative gaussian noise ${N}$(1, 0.2) for each individual synaptic weight (shown in Table 3) in the spiking network. For Figures 5C,D, we use a different set of synaptic weights which we show in Table 6.

TABLE 6

Table 6. Summary of alternative parameters for neuronal coupling.

In addition, code implementing the model can be found at GitHub.

Author's Note

A preprint version of this paper can be found at BioRxiv (Gauy et al., 2018).

Author Contributions

The project idea came from AS and the analysis was performed in collaboration by all the authors. MM did all simulations, prepared the figures and the video and wrote a first draft of the manuscript. All authors helped revise the manuscript and everyone approved the final version of it.

Funding

MM was supported by CNPq grant no. 248952/2013-7.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fnins.2018.00961/full#supplementary-material

References

Abbott, L. F., Varela, J., Sen, K., and Nelson, S. (1997). Synaptic depression and cortical gain control. Science 275, 221–224. doi: 10.1126/science.275.5297.221

PubMed Abstract | CrossRef Full Text | Google Scholar

Abeles, M. (1982). Local Cortical Circuits. Berlin: Springer.

Google Scholar

Alme, C. B., Miao, C., Jezek, K., Treves, A., Moser, E. I., and Moser, M.-B. (2014). Place cells in the hippocampus: eleven maps for eleven rooms. Proc. Natl. Acad. Sci. U.S.A. 111, 18428–18435. doi: 10.1073/pnas.1421056111

PubMed Abstract | CrossRef Full Text | Google Scholar

Ambrose, R. E., Pfeiffer, B. E., and Foster, D. J. (2016). Reverse replay of hippocampal place cells is uniquely modulated by changing reward. Neuron 91, 1124–1136. doi: 10.1016/j.neuron.2016.07.047

PubMed Abstract | CrossRef Full Text | Google Scholar

Amit, D. J., and Fusi, S. (1994). Learning in neural networks with material synapses. Neural Comput. 6, 957–982. doi: 10.1162/neco.1994.6.5.957

CrossRef Full Text | Google Scholar

Arias-Carrión, O., Stamelou, M., Murillo-Rodríguez, E., Menéndez-González, M., and Pöppel, E. (2010). Dopaminergic reward system: a short integrative review. Int. Arch. Med. 3:24. doi: 10.1186/1755-7682-3-24

PubMed Abstract | CrossRef Full Text | Google Scholar

Ariav, G., Polsky, A., and Schiller, J. (2003). Submillisecond precision of the input-output transformation function mediated by fast sodium dendritic spikes in basal dendrites of ca1 pyramidal neurons. J. Neurosci. 23, 7750–7758. doi: 10.1523/JNEUROSCI.23-21-07750.2003

PubMed Abstract | CrossRef Full Text | Google Scholar

Badel, L., Lefort, S., Brette, R., Petersen, C. C. H., Gerstner, W., and Richardson, M. J. E. (2008). Dynamic I-V curves are reliable predictors of naturalistic pyramidal-neuron voltage traces. J. Neurophysiol. 99, 656–666. doi: 10.1152/jn.01107.2007

PubMed Abstract | CrossRef Full Text | Google Scholar

Bi, G.-Q., and Poo, M.-m. (1998). Synaptic modifications in cultured hippocampal neurons: dependence on spike timing, synaptic strength, and postsynaptic cell type. J. Neurosci. 18, 10464–10472. doi: 10.1523/JNEUROSCI.18-24-10464.1998

PubMed Abstract | CrossRef Full Text | Google Scholar

Binas, J., Rutishauser, U., Indiveri, G., and Pfeiffer, M. (2014). Learning and stabilization of winner-take-all dynamics through interacting excitatory and inhibitory plasticity. Front. Comput. Neurosci. 8:68. doi: 10.3389/fncom.2014.00068

PubMed Abstract | CrossRef Full Text | Google Scholar

Brette, R., and Gerstner, W. (2005). Adaptive exponential integrate-and-fire model as an effective description of neuronal activity. J. Neurophysiol. 94, 3637–3642. doi: 10.1152/jn.00686.2005

PubMed Abstract | CrossRef Full Text | Google Scholar

Buzsáki, G. (1986). Hippocampal sharp waves: their origin and significance. Brain Res. 398, 242–252. doi: 10.1016/0006-8993(86)91483-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Buzsáki, G. (1989). Two-stage model of memory trace formation: a role for “noisy” brain states. Neuroscience 31, 551–570. doi: 10.1016/0306-4522(89)90423-5

PubMed Abstract | CrossRef Full Text | Google Scholar

Buzsáki, G. (2002). Theta oscillations in the hippocampus. Neuron 33, 325–340. doi: 10.1016/S0896-6273(02)00586-X

PubMed Abstract | CrossRef Full Text | Google Scholar

Buzsaki, G. (2006). Rhythms of the Brain. Oxford: Oxford University Press.

Google Scholar

Buzsáki, G. (2015). Hippocampal sharp wave-ripple: a cognitive biomarker for episodic memory and planning. Hippocampus 25, 1073–1188. doi: 10.1002/hipo.22488

PubMed Abstract | CrossRef Full Text | Google Scholar

Carandini, M., and Heeger, D. J. (2012). Normalization as a canonical neural computation. Nat. Rev. Neurosci. 13:51. doi: 10.1038/nrn3136

PubMed Abstract | CrossRef Full Text | Google Scholar

Carr, M. F., Jadhav, S. P., and Frank, L. M. (2011). Hippocampal replay in the awake state: a potential substrate for memory consolidation and retrieval. Nat. Neurosci. 14, 147–153. doi: 10.1038/nn.2732

PubMed Abstract | CrossRef Full Text | Google Scholar

Chenkov, N., Sprekeler, H., and Kempter, R. (2017). Memory replay in balanced recurrent networks. PLoS Comput. Biol. 13:e1005359. doi: 10.1371/journal.pcbi.1005359

PubMed Abstract | CrossRef Full Text | Google Scholar

Clopath, C., Büsing, L., Vasilaki, E., and Gerstner, W. (2010). Connectivity reflects coding: a model of voltage-based STDP with homeostasis. Nat. Neurosci. 13, 344–352. doi: 10.1038/nn.2479

PubMed Abstract | CrossRef Full Text | Google Scholar

Csicsvari, J., O'neill, J., Allen, K., and Senior, T. (2007). Place-selective firing contributes to the reverse-order reactivation of ca1 pyramidal cells during sharp waves in open-field exploration. Eur. J. Neurosci. 26, 704–716. doi: 10.1111/j.1460-9568.2007.05684.x

PubMed Abstract | CrossRef Full Text | Google Scholar

Davidson, T. J., Kloosterman, F., and Wilson, M. A. (2009). Hippocampal replay of extended experience. Neuron 63, 497–507. doi: 10.1016/j.neuron.2009.07.027

PubMed Abstract | CrossRef Full Text | Google Scholar

Diba, K., and Buzsáki, G. (2007). Forward and reverse hippocampal place-cell sequences during ripples. Nat. Neurosci. 10:1241. doi: 10.1038/nn1961

PubMed Abstract | CrossRef Full Text | Google Scholar

Diekelmann, S., and Born, J. (2010). Slow-wave sleep takes the leading role in memory reorganization. Nat. Rev. Neurosci. 11:218. doi: 10.1038/nrn2762-c2

CrossRef Full Text | Google Scholar

Douglas, R. J., and Martin, K. A. (2004). Neuronal circuits of the neocortex. Annu. Rev. Neurosci. 27, 419–451. doi: 10.1146/annurev.neuro.27.070203.144152

PubMed Abstract | CrossRef Full Text | Google Scholar

Dragoi, G., and Tonegawa, S. (2011). Preplay of future place cell sequences by hippocampal cellular assemblies. Nature 469:397. doi: 10.1038/nature09633

PubMed Abstract | CrossRef Full Text | Google Scholar

Ego-Stengel, V., and Wilson, M. A. (2010). Disruption of ripple-associated hippocampal activity during rest impairs spatial learning in the rat. Hippocampus 20, 1–10. doi: 10.1002/hipo.20707

PubMed Abstract | CrossRef Full Text | Google Scholar

Eichenbaum, H. (1993). Memory, Amnesia, and the Hippocampal System. Cambridge: MIT Press.

Google Scholar

Eichenbaum, H. (2014). Time cells in the hippocampus: a new dimension for mapping memories. Nat. Rev. Neurosci. 15:732. doi: 10.1038/nrn3827

PubMed Abstract | CrossRef Full Text | Google Scholar

Eichenbaum, H., and Cohen, N. J. (2004). From Conditioning to Conscious Recollection: Memory Systems of the Brain. Number 35. Oxford: Oxford University Press on Demand.

Google Scholar

Foster, D. J. (2017). Replay comes of age. Annu. Rev. Neurosci. 40, 581–602. doi: 10.1146/annurev-neuro-072116-031538

PubMed Abstract | CrossRef Full Text | Google Scholar

Foster, D. J., and Wilson, M. A. (2006). Reverse replay of behavioural sequences in hippocampal place cells during the awake state. Nature 440:680. doi: 10.1038/nature04587

PubMed Abstract | CrossRef Full Text | Google Scholar

Foster, D. J., and Wilson, M. A. (2007). Hippocampal theta sequences. Hippocampus 17, 1093–1099. doi: 10.1002/hipo.20345

PubMed Abstract | CrossRef Full Text | Google Scholar

Frémaux, N., and Gerstner, W. (2015). Neuromodulated spike-timing-dependent plasticity, and theory of three-factor learning rules. Front. Neural Circuits 9:85. doi: 10.3389/fncir.2015.00085

PubMed Abstract | CrossRef Full Text | Google Scholar

Fuhrmann, F., Justus, D., Sosulina, L., Kaneko, H., Beutel, T., Friedrichs, D., et al. (2015). Locomotion, theta oscillations, and the speed-correlated firing of hippocampal neurons are controlled by a medial septal glutamatergic circuit. Neuron 86, 1253–1264. doi: 10.1016/j.neuron.2015.05.001

PubMed Abstract | CrossRef Full Text | Google Scholar

Gasparini, S., and Magee, J. C. (2006). State-dependent dendritic computation in hippocampal ca1 pyramidal neurons. J. Neurosci. 26, 2088–2100. doi: 10.1523/JNEUROSCI.4428-05.2006

PubMed Abstract | CrossRef Full Text | Google Scholar

Gasparini, S., Migliore, M., and Magee, J. C. (2004). On the initiation and propagation of dendritic spikes in ca1 pyramidal neurons. J. Neurosci. 24, 11046–11056. doi: 10.1523/JNEUROSCI.2520-04.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

Gauy, M. M., Lengler, J., Einarsson, H., Meier, F., Weissenberger, F., Yanik, M. F., et al. (2018). A hippocampal model for behavioral time acquisition and fast bidirectional replay of spatio-temporal memory sequences. bioRxiv 343988.

Google Scholar

Geisler, C., Robbe, D., Zugaro, M., Sirota, A., and Buzsáki, G. (2007). Hippocampal place cell assemblies are speed-controlled oscillators. Proc. Natl. Acad. Sci. U.S.A. 104, 8149–8154. doi: 10.1073/pnas.0610121104

PubMed Abstract | CrossRef Full Text | Google Scholar

Gerstner, W., Kistler, W. M., Naud, R., and Paninski, L. (2014). Neuronal Dynamics: From Single Neurons to Networks and Models of Cognition. Cambridge: Cambridge University Press.

Google Scholar

Girardeau, G., Benchenane, K., Wiener, S. I., Buzsáki, G., and Zugaro, M. B. (2009). Selective suppression of hippocampal ripples impairs spatial memory. Nat. Neurosci. 12:1222. doi: 10.1038/nn.2384

PubMed Abstract | CrossRef Full Text | Google Scholar

Girardeau, G., and Zugaro, M. (2011). Hippocampal ripples and memory consolidation. Curr. Opin. Neurobiol. 21, 452–459. doi: 10.1016/j.conb.2011.02.005

PubMed Abstract | CrossRef Full Text | Google Scholar

Grosmark, A. D., and Buzsáki, G. (2016). Diversity in neural firing dynamics supports both rigid and learned hippocampal sequences. Science 351, 1440–1443. doi: 10.1126/science.aad1935

PubMed Abstract | CrossRef Full Text | Google Scholar

Gupta, A. S., van der Meer, M. A., Touretzky, D. S., and Redish, A. D. (2010). Hippocampal replay is not a simple function of experience. Neuron 65, 695–705. doi: 10.1016/j.neuron.2010.01.034

CrossRef Full Text | Google Scholar

Haga, T., and Fukai, T. (2018). Dendritic processing of spontaneous neuronal sequences for single-trial learning. Sci. Rep. 8:15166. doi: 10.1038/s41598-018-33513-9

PubMed Abstract | CrossRef Full Text | Google Scholar

Harvey, C. D., Collman, F., Dombeck, D. A., and Tank, D. W. (2009). Intracellular dynamics of hippocampal place cells during virtual navigation. Nature 461:941. doi: 10.1038/nature08499

PubMed Abstract | CrossRef Full Text | Google Scholar

Hasselmo, M. E. (1999). Neuromodulation: acetylcholine and memory consolidation. Trends Cogn. Sci. 3, 351–359. doi: 10.1016/S1364-6613(99)01365-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Hasselmo, M. E. (2006). The role of acetylcholine in learning and memory. Curr. Opin. Neurobiol. 16, 710–715. doi: 10.1016/j.conb.2006.09.002

PubMed Abstract | CrossRef Full Text | Google Scholar

Hasselmo, M. E., and Bower, J. M. (1993). Acetylcholine and memory. Trends Neurosci. 16, 218–222. doi: 10.1016/0166-2236(93)90159-J

CrossRef Full Text | Google Scholar

Hasselmo, M. E., and Schnell, E. (1994). Laminar selectivity of the cholinergic suppression of synaptic transmission in rat hippocampal region ca1: computational modeling and brain slice physiology. J. Neurosci. 14, 3898–3914. doi: 10.1523/JNEUROSCI.14-06-03898.1994

PubMed Abstract | CrossRef Full Text | Google Scholar

Hasselmo, M. E., Schnell, E., and Barkai, E. (1995). Dynamics of learning and recall at excitatory recurrent synapses and cholinergic modulation in rat hippocampal region ca3. J. Neurosci. 15, 5249–5262. doi: 10.1523/JNEUROSCI.15-07-05249.1995

PubMed Abstract | CrossRef Full Text | Google Scholar

Henze, D. A., González-Burgos, G. R., Urban, N. N., Lewis, D. A., and Barrionuevo, G. (2000). Dopamine increases excitability of pyramidal neurons in primate prefrontal cortex. J. Neurophysiol. 84, 2799–2809. doi: 10.1152/jn.2000.84.6.2799

PubMed Abstract | CrossRef Full Text | Google Scholar

Isaac, J. T., Nicoll, R. A., and Malenka, R. C. (1995). Evidence for silent synapses: implications for the expression of ltp. Neuron 15, 427–434. doi: 10.1016/0896-6273(95)90046-2

PubMed Abstract | CrossRef Full Text | Google Scholar

Jahnke, S., Timme, M., and Memmesheimer, R.-M. (2012). Guiding synchrony through random networks. Phys. Rev. X 2:041016. doi: 10.1103/PhysRevX.2.041016

CrossRef Full Text | Google Scholar

Jahnke, S., Timme, M., and Memmesheimer, R.-M. (2015). A unified dynamic model for learning, replay, and sharp-wave/ripples. J. Neurosci. 35, 16236–16258. doi: 10.1523/JNEUROSCI.3977-14.2015

PubMed Abstract | CrossRef Full Text | Google Scholar

Jiang, X., Wang, G., Lee, A. J., Stornetta, R. L., and Zhu, J. J. (2013). The organization of two new cortical interneuronal circuits. Nat. Neurosci. 16, 210–218. doi: 10.1038/nn.3305

PubMed Abstract | CrossRef Full Text | Google Scholar

Jun, J. K., and Jin, D. Z. (2007). Development of neural circuitry for precise temporal sequences through spontaneous activity, axon remodeling, and synaptic plasticity. PLoS ONE 2:e723. doi: 10.1371/journal.pone.0000723

PubMed Abstract | CrossRef Full Text | Google Scholar

Justus, D., Dalügge, D., Bothe, S., Fuhrmann, F., Hannes, C., Kaneko, H., et al. (2017). Glutamatergic synaptic integration of locomotion speed via septoentorhinal projections. Nat. Neurosci. 20, 16–19. doi: 10.1038/nn.4447

PubMed Abstract | CrossRef Full Text | Google Scholar

Kametani, H., and Kawamura, H. (1990). Alterations in acetylcholine release in the rat hippocampus during sleep-wakefulness detected by intracerebral dialysis. Life Sci. 47, 421–426. doi: 10.1016/0024-3205(90)90300-G

PubMed Abstract | CrossRef Full Text | Google Scholar

Karlsson, M. P., and Frank, L. M. (2009). Awake replay of remote experiences in the hippocampus. Nat. Neurosci. 12, 913–918. doi: 10.1038/nn.2344

PubMed Abstract | CrossRef Full Text | Google Scholar

Kay, K., Sosa, M., Chung, J. E., Karlsson, M. P., Larkin, M. C., and Frank, L. M. (2016). A hippocampal network for spatial coding during immobility and sleep. Nature 531:185. doi: 10.1038/nature17144

PubMed Abstract | CrossRef Full Text | Google Scholar

Kerchner, G. A., and Nicoll, R. A. (2008). Silent synapses and the emergence of a postsynaptic mechanism for ltp. Nat. Rev. Neurosci. 9:813. doi: 10.1038/nrn2501

PubMed Abstract | CrossRef Full Text | Google Scholar

Kobayashi, K., and Suzuki, H. (2007). Dopamine selectively potentiates hippocampal mossy fiber to ca3 synaptic transmission. Neuropharmacology 52, 552–561. doi: 10.1016/j.neuropharm.2006.08.026

PubMed Abstract | CrossRef Full Text | Google Scholar

Lee, A. K., and Wilson, M. A. (2002). Memory of sequential experience in the hippocampus during slow wave sleep. Neuron 36, 1183–1194. doi: 10.1016/S0896-6273(02)01096-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Letzkus, J. J., Wolff, S. B., Meyer, E. M., Tovote, P., Courtin, J., Herry, C., et al. (2011). A disinhibitory microcircuit for associative fear learning in the auditory cortex. Nature 480:331. doi: 10.1038/nature10674

PubMed Abstract | CrossRef Full Text | Google Scholar

Litwin-Kumar, A., and Doiron, B. (2014). Formation and maintenance of neuronal assemblies through synaptic plasticity. Nat. Commun. 5:5319. doi: 10.1038/ncomms6319

PubMed Abstract | CrossRef Full Text | Google Scholar

Louie, K., and Wilson, M. A. (2001). Temporally structured replay of awake hippocampal ensemble activity during rapid eye movement sleep. Neuron 29, 145–156. doi: 10.1016/S0896-6273(01)00186-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Maass, W. (2000). On the computational power of winner-take-all. Neural Comput. 12, 2519–2535. doi: 10.1162/089976600300014827

PubMed Abstract | CrossRef Full Text | Google Scholar

Markram, H., Lübke, J., Frotscher, M., and Sakmann, B. (1997). Regulation of synaptic efficacy by coincidence of postsynaptic aps and epsps. Science 275, 213–215. doi: 10.1126/science.275.5297.213

PubMed Abstract | CrossRef Full Text | Google Scholar

Markram, H., and Tsodyks, M. (1996). Redistribution of synaptic efficacy between neocortical pyramidal neurons. Nature 382:807. doi: 10.1038/382807a0

PubMed Abstract | CrossRef Full Text | Google Scholar

Marrosu, F., Portas, C., Mascia, M. S., Casu, M. A., Fà, M., Giagheddu, M., et al. (1995). Microdialysis measurement of cortical and hippocampal acetylcholine release during sleep-wake cycle in freely moving cats. Brain Res. 671, 329–332. doi: 10.1016/0006-8993(94)01399-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Memmesheimer, R.-M. (2010). Quantitative prediction of intermittent high-frequency oscillations in neural networks with supralinear dendritic interactions. Proc. Natl. Acad. Sci. U.S.A. 107, 11092–11097. doi: 10.1073/pnas.0909615107

PubMed Abstract | CrossRef Full Text | Google Scholar

Middleton, S. J., Kneller, E. M., Chen, S., Ogiwara, I., Montal, M., Yamakawa, K., et al. (2018). Altered hippocampal replay is associated with memory impairment in mice heterozygous for the scn2a gene. Nat. Neurosci. 21, 996–1003. doi: 10.1038/s41593-018-0163-8

PubMed Abstract | CrossRef Full Text | Google Scholar

Mizuseki, K., and Buzsáki, G. (2013). Preconfigured, skewed distribution of firing rates in the hippocampus and entorhinal cortex. Cell Rep. 4, 1010–1021. doi: 10.1016/j.celrep.2013.07.039

PubMed Abstract | CrossRef Full Text | Google Scholar

Nádasdy, Z., Hirase, H., Czurkó, A., Csicsvari, J., and Buzsáki, G. (1999). Replay and time compression of recurring spike sequences in the hippocampus. J. Neurosci. 19, 9497–9507. doi: 10.1523/JNEUROSCI.19-21-09497.1999

PubMed Abstract | CrossRef Full Text | Google Scholar

O'Keefe, J. (1976). Place units in the hippocampus of the freely moving rat. Exp. Neurol. 51, 78–109. doi: 10.1016/0014-4886(76)90055-8

PubMed Abstract | CrossRef Full Text | Google Scholar

O'Keefe, J., and Dostrovsky, J. (1971). The hippocampus as a spatial map. preliminary evidence from unit activity in the freely-moving rat. Brain Res. 34, 171–175. doi: 10.1016/0006-8993(71)90358-1

PubMed Abstract | CrossRef Full Text | Google Scholar

O'Keefe, J., and Nadel, L. (1978). The Hippocampus as a Cognitive Map. Oxford: Clarendon Press.

Google Scholar

O'Keefe, J., and Recce, M. L. (1993). Phase relationship between hippocampal place units and the eeg theta rhythm. Hippocampus 3, 317–330. doi: 10.1002/hipo.450030307

PubMed Abstract | CrossRef Full Text | Google Scholar

Ólafsdóttir, H. F., Barry, C., Saleem, A. B., Hassabis, D., and Spiers, H. J. (2015). Hippocampal place cells construct reward related sequences through unexplored space. Elife 4:e06063. doi: 10.7554/eLife.06063

PubMed Abstract | CrossRef Full Text | Google Scholar

Ascoli, G. A., Alonso-Nanclares, L., Anderson, S. A., Barrionuevo, G., Benavides-Piccione, R., et al. (2008). Petilla terminology: nomenclature of features of gabaergic interneurons of the cerebral cortex. Nat. Rev. Neurosci. 9:557. doi: 10.1038/nrn2402

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfeiffer, B. E. (2017). The content of hippocampal replay. Hippocampus. doi: 10.1002/hipo.22824. [Epub ahead of print].

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfeiffer, B. E., and Foster, D. J. (2013). Hippocampal place-cell sequences depict future paths to remembered goals. Nature 497:74. doi: 10.1038/nature12112

PubMed Abstract | CrossRef Full Text | Google Scholar

Pfeiffer, B. E., and Foster, D. J. (2015). Autoassociative dynamics in the generation of sequences of hippocampal place cells. Science 349, 180–183. doi: 10.1126/science.aaa9633

PubMed Abstract | CrossRef Full Text | Google Scholar

Pi, H.-J., Hangya, B., Kvitsiani, D., Sanders, J. I., Huang, Z. J., and Kepecs, A. (2013). Cortical interneurons that specialize in disinhibitory control. Nature 503:521. doi: 10.1038/nature12676

PubMed Abstract | CrossRef Full Text | Google Scholar

Polsky, A., Mel, B. W., and Schiller, J. (2004). Computational subunits in thin dendrites of pyramidal cells. Nat. Neurosci. 7:621. doi: 10.1038/nn1253

PubMed Abstract | CrossRef Full Text | Google Scholar

Renart, A., de la Rocha, J., Bartho, P., Hollender, L., Parga, N., Reyes, A., et al. (2010). The asynchronous state in cortical circuits. science 327, 587–590. doi: 10.1126/science.1179850

PubMed Abstract | CrossRef Full Text | Google Scholar

Romani, S., and Tsodyks, M. (2015). Short-term plasticity based network model of place cells dynamics. Hippocampus 25, 94–105. doi: 10.1002/hipo.22355

PubMed Abstract | CrossRef Full Text | Google Scholar

Rudy, J. W., and Sutherland, R. J. (1995). Configural association theory and the hippocampal formation: an appraisal and reconfiguration. Hippocampus 5, 375–389. doi: 10.1002/hipo.450050502

PubMed Abstract | CrossRef Full Text | Google Scholar

Schapiro, A. C., McDevitt, E. A., Rogers, T. T., Mednick, S. C., and Norman, K. A. (2018). Human hippocampal replay during rest prioritizes weakly learned information and predicts memory performance. Nat. Commun. 9:3920. doi: 10.1038/s41467-018-06213-1

PubMed Abstract | CrossRef Full Text | Google Scholar

Scoville, W. B., and Milner, B. (1957). Loss of recent memory after bilateral hippocampal lesions. J. Neurol. Neurosurg. Psychiatry 20:11. doi: 10.1136/jnnp.20.1.11

PubMed Abstract | CrossRef Full Text | Google Scholar

Singer, A. C., and Frank, L. M. (2009). Rewarded outcomes enhance reactivation of experience in the hippocampus. Neuron 64, 910–921. doi: 10.1016/j.neuron.2009.11.016

PubMed Abstract | CrossRef Full Text | Google Scholar

Smiałowski, A., and Bijak, M. (1987). Excitatory and inhibitory action of dopamine on hippocampal neurons in vitro. Involvement of d2 and d1 receptors. Neuroscience 23, 95–101. doi: 10.1016/0306-4522(87)90274-0

PubMed Abstract | CrossRef Full Text | Google Scholar

Squire, L. R. (1982). The neuropsychology of human memory. Annu. Rev. Neurosci. 5, 241–273. doi: 10.1146/annurev.ne.05.030182.001325

PubMed Abstract | CrossRef Full Text | Google Scholar

Squire, L. R., and Zola-Morgan, S. (1991). The medial temporal lobe memory system. Science 253:1380. doi: 10.1126/science.1896849

PubMed Abstract | CrossRef Full Text | Google Scholar

Stanzione, P., Calabresi, P., Mercuri, N., and Bernardi, G. (1984). Dopamine modulates ca1 hippocampal neurons by elevating the threshold for spike generation: an in vitro study. Neuroscience 13, 1105–1116. doi: 10.1016/0306-4522(84)90291-4

PubMed Abstract | CrossRef Full Text | Google Scholar

Steemers, B., Vicente-Grabovetsky, A., Barry, C., Smulders, P., Schröder, T. N., Burgess, N., et al. (2016). Hippocampal attractor dynamics predict memory-based decision making. Curr. Biol. 26, 1750–1757. doi: 10.1016/j.cub.2016.04.063

PubMed Abstract | CrossRef Full Text | Google Scholar

Stevens, C. F., and Wang, Y. (1995). Facilitation and depression at single central synapses. Neuron 14, 795–802. doi: 10.1016/0896-6273(95)90223-6

PubMed Abstract | CrossRef Full Text | Google Scholar

Theodoni, P., Rovira, B., Wang, Y., and Roxin, A. (2018). Theta-modulation drives the emergence of connectivity patterns underlying replay in a network model of place cells. eLife 7:e37388. doi: 10.7554/eLife.37388

PubMed Abstract | CrossRef Full Text | Google Scholar

Van Vreeswijk, C., and Sompolinsky, H. (1996). Chaos in neuronal networks with balanced excitatory and inhibitory activity. Science 274, 1724–1726. doi: 10.1126/science.274.5293.1724

PubMed Abstract | CrossRef Full Text | Google Scholar

Vandecasteele, M., Varga, V., Berényi, A., Papp, E., Barthó, P., Venance, L., et al. (2014). Optogenetic activation of septal cholinergic neurons suppresses sharp wave ripples and enhances theta oscillations in the hippocampus. Proc. Natl. Acad. Sci. U.S.A. 111, 13535–13540. doi: 10.1073/pnas.1411233111

PubMed Abstract | CrossRef Full Text | Google Scholar

Vanderwolf, C. H. (1969). Hippocampal electrical activity and voluntary movement in the rat. Electroencephalogr. Clin. Neurophysiol. 26, 407–418. doi: 10.1016/0013-4694(69)90092-3

PubMed Abstract | CrossRef Full Text | Google Scholar

Weissenberger, F., Meier, F., Lengler, J., Einarsson, H., and Steger, A. (2017). Long synfire chains emerge by spike-timing dependent plasticity modulated by population activity. Int. J. Neural Syst. 27:1750044. doi: 10.1142/S0129065717500447

PubMed Abstract | CrossRef Full Text | Google Scholar

Wilson, M. A., and McNaughton, B. L. (1994). Reactivation of hippocampal ensemble memories during sleep. Science 265, 676–679. doi: 10.1126/science.8036517

PubMed Abstract | CrossRef Full Text | Google Scholar

Wu, X., and Foster, D. J. (2014). Hippocampal replay captures the unique topological structure of a novel environment. J. Neurosci. 34, 6459–6469. doi: 10.1523/JNEUROSCI.3414-13.2014

PubMed Abstract | CrossRef Full Text | Google Scholar

Yamamoto, J., and Tonegawa, S. (2017). Direct medial entorhinal cortex input to hippocampal ca1 is crucial for extended quiet awake replay. Neuron 96, 217–227. doi: 10.1016/j.neuron.2017.09.017

PubMed Abstract | CrossRef Full Text | Google Scholar

Ylinen, A., Bragin, A., Nádasdy, Z., Jandó, G., Szabo, I., Sik, A., et al. (1995). Sharp wave-associated high-frequency oscillation (200 hz) in the intact hippocampus: network and intracellular mechanisms. J. Neurosci. 15, 30–46. doi: 10.1523/JNEUROSCI.15-01-00030.1995

PubMed Abstract | CrossRef Full Text | Google Scholar

Zenke, F., Agnes, E. J., and Gerstner, W. (2015). Diverse synaptic plasticity mechanisms orchestrated to form and retrieve memories in spiking neural networks. Nat. Commun. 6:6922. doi: 10.1038/ncomms7922

PubMed Abstract | CrossRef Full Text | Google Scholar