More Than Spikes: On the Added Value of Non-linear Intracranial EEG Analysis for Surgery Planning in Temporal Lobe Epilepsy

Epilepsy surgery can be a very effective therapy in medication refractory patients. During patient evaluation intracranial EEG is analyzed by clinical experts to identify the brain tissue generating epileptiform events. Quantitative EEG analysis increasingly complements this approach in research settings, but not yet in clinical routine. We investigate the correspondence between epileptiform events and a specific quantitative EEG marker. We analyzed 99 preictal epochs of multichannel intracranial EEG of 40 patients with mixed etiologies. Time and channel of occurrence of epileptiform events (spikes, slow waves, sharp waves, fast oscillations) were annotated by a human expert and non-linear excess interrelations were calculated as a quantitative EEG marker. We assessed whether the visually identified preictal events predicted channels that belonged to the seizure onset zone, that were later resected or that showed strong non-linear interrelations. We also investigated whether the seizure onset zone or the resection were predicted by channels with strong non-linear interrelations. In patients with temporal lobe epilepsy (32 of 40), epileptic spikes and the seizure onset zone predicted the resected brain tissue much better in patients with favorable seizure control after surgery than in unfavorable outcomes. Beyond that, our analysis did not reveal any significant associations with epileptiform EEG events. Specifically, none of the epileptiform event types did predict non-linear interrelations. In contrast, channels with strong non-linear excess EEG interrelations predicted the resected channels better in patients with temporal lobe epilepsy and favorable outcome. Also in the small number of patients with seizure onset in the frontal and parietal lobes, no association between epileptiform events and channels with strong non-linear excess EEG interrelations was detectable. In contrast to patients with temporal seizure onset, EEG channels with strong non-linear excess interrelations did neither predict the seizure onset zone nor the resection of these patients or allow separation between patients with favorable and unfavorable seizure control. Our study indicates that non-linear excess EEG interrelations are not strictly associated with epileptiform events, which are one key concept of current clinical EEG assessment. Rather, they may provide information relevant for surgery planning in temporal lobe epilepsy. Our study suggests to incorporate quantitative EEG analysis in the workup of clinical cases. We make the EEG epochs and expert annotations publicly available in anonymized form to foster similar analyses for other quantitative EEG methods.

1. Introduction

Surgical removal of seizure-generating brain tissue is an established and often beneficial treatment option for patients with drug-resistant epilepsy. Identification of the tissue necessary and sufficient to cease seizure activity [the “epileptogenic zone,” EZ, (1, 2)] is essential but challenging, especially if there is no obvious anatomical correlate (as e.g., a brain lesion or tumor).

The decision on which area to resect and how this will putatively influence epileptic activity is individually determined for each patient, taking various diagnostic information sources into consideration (including scalp EEG, structural and functional MRI, psychological assessments and intracranial EEG if necessary). Although these assessments usually follow established concepts like the importance of the “seizure onset zone” (SOZ), which is used as a proxy for the EZ, the procedure suffers from a considerable amount of subjectivity. The limitations of the current approaches regarding reliable prediction of the patients' benefit from epilepsy surgery are apparent, since only about half of all patients undergoing surgery become permanently seizure free, a rate that has practically not improved over decades (3–10). This issue is strongly associated with the heterogeneity of the disorder. It is now generally accepted that epilepsy needs to be considered a network-based disease, characterized by the interaction of multiple brain regions (11–15). Due to the lack of methodologies allowing to record brain activity with simultaneously high spatial and temporal resolution and full coverage, seizure dynamics are still not understood in full detail.

Intracranial EEG (iEEG) is currently the gold standard in this regard, providing excellent temporal and spatial resolution with the drawback of limited spatial coverage. However, purely visual analysis is limited due to the abundance of potential interrelations between dozens of iEEG channels (16), exacerbated by the ever-growing amount of data due to increasing spatial resolution (17) and recording time (18, 19). Quantitative EEG (qEEG) methods, capable of capturing a variety of signal properties (including very complex, visually undetectable ones) and processing large amounts of data have been presented over the last decades. They hold great promise to discern and provide additional information and ultimately increase the success rate of epilepsy surgery. High frequency oscillations (80–500 Hz) have long been considered a promising marker of the EZ (20–22). Along with the network conception of epilepsy, multivariate methods, quantifying signal dependencies, gained increasing attention [see (23, 24) for reviews]. In congruence with the strong evidence for high non-linearity of epileptic activity (25–29), segregated non-linear signal dependencies have been demonstrated to contain relevant information (27, 30).

Despite a large variety of qEEG measures have shown to capture some disease-related properties, none is applied in clinical routine to date. Besides undefined standardization of methods and interpretation and a lack of implementation in certified software, one reason might be the suspicion that qEEG markers could only be sensitive to signal features that are similarly detectable by visual expert analysis, like e.g., frequent interictal spikes (31). Thus, besides an extensive evaluation of qEEG measures, a better understanding to what extent these are related to traditional markers of epilepsy and what they might reflect beyond, will be very helpful in the effort to gain clinical acceptance.

Strictly speaking, the epileptogenicity of brain tissue (i.e., its belonging to the EZ, a theoretical concept) is not accessible by iEEG or any other current mapping technique. This implies that when aiming to identify markers that are closely associated with the brain tissue's epileptogenicity, one is confronted with the problem of a missing ground truth. The SOZ can be visually determined by experts from the transition from preictal to ictal EEG before surgery (i.e., agnostic of post-surgical seizure control) but is known to be only an approximation of the EZ (1). In addition, the extent of its observation depends on electrode placement. On the other hand, the resected brain tissue (RBT) is available only in patients who undergo surgery, often larger than minimally required and (by definition) fully contains the EZ only in patients who became seizure free. From this we hypothesized that the ability of any marker of epileptogenicity to determine the RBT should be larger in patients who became seizure free after surgery than in patients with unfavorable outcome. In contrast, the agreement between the SOZ and any marker of epileptogenicity might depend on post-surgical seizure control only indirectly.

In our previous study (32), we have demonstrated for patients with mesiotemporal implantation of depth electrodes that surrogate corrected non-linear interrelations between iEEG signals were associated with the individual pathology. In addition, the spatial overlap of salient non-linear interrelations with the RBT was associated with post-surgical seizure control. Here, we investigated the relationship between the occurrence of salient non-linear interrelations and traditional markers of epileptogenicity (33), namely spikes, slow waves, sharp waves, and fast oscillations, as identified by a human expert. In addition, we determined the relationship with the SOZ and where applicable with the RBT. We manually annotated and analyzed 99 preictal epochs of multi-channel iEEG from 40 patients including several types of epilepsy syndromes and etiologies as well as electrode implantation schemes beyond mesio-temporal depth electrodes. To reduce data heterogeneity, we limited our main analyses to patients with seizure onset in the temporal lobe, which were the vast majority in our dataset (32/40). Explorative examination of patients with a non-temporal seizure onset (six frontal, two parietal) are provided in the Supplementary Materials. To assess the association of epileptogenic brain tissue with EEG markers (be they visual or quantitative), we contrasted accuracy quantifiers on the channel level between patients with favorable and unfavorable post-surgical outcomes.

To foster similar analyses, we make the full EEG recordings and expert annotations used in this paper publicly available in anonymized form via GitHub together with a custom EEG reader (github.com/SCAN-NRAD/scanEEGviewer).

2. Methods

2.1. Patients and Data

We included data of 40 patients with drug-resistant epilepsies, who were considered for epilepsy surgery at the Inselspital Bern (58% female, median age 35 years, IQR 19 years, range 9−66 years). More detailed patient information is provided in Table 1. To characterize success of the intervention, we used the first available outcome assessment at least three months after surgery according to the Engel scale. The first assessment is most representative of the direct effects of the surgery, not influenced by subsequent effects like neuronal plasticity and changes in patient compliance, which might change the long-term outcome but are hardly predictable. In total, 19 patients became completely seizure free (Engel class I), 4 patients became almost seizure free (Engel class II), 4 patients had worthwhile improvement (Engel class III), and 7 patients had no improvement (Engel class IV). In the group analyses presented in the main text we only included patients with seizure onset in the temporal lobe to preserve data homogeneity. The results for the remaining patients are compiled in the Supplementary Materials. At the same time, to increase sample size, we dichotomized outcomes into “favorable” (Engel classes I & II) and “unfavorable” (Engel classes III & IV). Using coregistration of a post-implantation CT and a postsurgical MRI, the patient-specific RBT and thereby the iEEG channels recording from this tissue were determined [see (34) for a detailed description of this procedure].

TABLE 1

Table 1. Patients included in this study.

This study was approved by the Ethics Committee of the Kanton of Bern (approval number 2017-00697). All decisions regarding the actual treatment of the patients (especially implantation and resection) were made solely on clinical grounds prior to this retrospective study and all patients gave written and informed consent that EEG and imaging data may be used for research purposes.

2.2. EEG Data, Epoch Selection, and Manual Annotation

EEG data was recorded using a NicoletOneTM recording system with a C64 amplifier (VIASYS Healthcore Inc., Madison, Wisconsin, USA) and intracranial depth, strip, and grid electrodes (AD-TECH, Wisconsin, USA). The sampling rate was 512 or 1,024 Hz, depending on whether more or less than 64 channels were used. Signals were referenced to an extracranial electrode (localized between 10–20 positions Fz and Cz) during recording and later re-referenced against the median of all artifact free channels. In addition, signals were band-pass filtered between 0.5 and 150 Hz using a fourth-order Butterworth filter.

Since the extensive manual annotation of iEEG data, typically comprising between 50 and 60 channels, is very time-consuming, it was impossible to analyze the entire long-term EEG recordings. In addition, the calculation of the qEEG measure used in this study (see below) is computationally expensive and prohibits long-term analysis beyond the range of minutes. In consequence, a compromise between epoch duration included per patient and the number of patients was inevitable. To minimize bias toward patients with many seizures we restricted the number of iEEG epochs per patient to at most three. All epochs had a duration between 110 and 200 s and ended at seizure onset. Permanently artifact corrupted channels (according to visual analysis by experts) were excluded from detailed visual or quantitative analysis (<5% of channels).

Several studies have shown that epilepsy dynamics underlie oscillations on various timescales, from circadian to multidien rhytms (35–39). Correspondingly, network measures calculated from iEEG data exhibit large circadian variations (40). To confine the arbitrariness of temporal data selection, we chose in each patient segments directly preceding the earliest artifact-free seizures recorded after implantation of the intracranial electrodes. This period serves as a relevant baseline for visual EEG analysis in clinical routine, and in contrast to ictal data, avoids artifacts that might be caused by seizure manifestation.

A clinical expert (M.D.) visually inspected all included iEEG epochs and manually annotated the extent of all epileptiform events (33) regarding time of occurrence and affected channels, corresponding to at least one of the following types: (1) spikes, (2) slow waves, (3) sharp waves, (4) fast oscillations. Channels were scored in a custom EEG-reader in referential mode. We scored pre-ictal transients as typical for epilepsy based on its sharp configuration and compared to the background activity in the same channel, looking either for high amplitudes or a disruption of ongoing rhythms. We distinguished spikes (duration <70 ms) and sharp waves (duration 70−200 ms). Slow wave activity was scored based on either a marked focal slowing of background activity, or the presence of slow waves with a high amplitude compared to the background activity. Fast oscillations were identified as episodes of focal activity with a frequency above 30 Hz. In addition, the channels comprising the seizure-specific SOZ were identified based on the presence of low-amplitude fast activity at seizure onset.

2.3. Non-linear Signal Dependence and Identification of Core Channels

The qEEG measure used in this study was introduced by Rummel et al. (34, 41) and has been applied in (32, 42, 43). A similar measure was also used in (16, 27, 30). In brief, non-linear interrelation matrices were determined by calculating mutual information of signal pairs over segments of 8 s duration, which were shifted over the entire epoch by 1-s steps. Mutual information quantifies the amount of information one signal provides about the other. Since it is sensitive to both linear an non-linear dependences alike, we used multivariate iterative amplitude adjusted Fourier transform (IAAFT) surrogate time series with conserved Pearson correlation matrix (44) to account for linear interrelation effects. Non-zero elements of the resulting interaction matrices had significantly stronger mutual information than the surrogate time series with conserved Pearson correlation. Hence, the matrices describe the non-linear excess interrelations, i.e., the interrelation that is not measurable by linear measures. To condense information, we averaged the resulting matrices over time. Since patient-wise contrasting of separate averages over segments with and without epileptiform events (45) were not consistently possible due to too dense or too sparse event occurrence in some patients, we averaged the interrelation matrices over the entire preictal epochs. From the resulting mean interrelation matrix, we calculated the normalized “node strength” (i.e., the mean interrelation of a channel with the remainder). This single value per channel is confined to the range [0, 1] and indicates how strongly it is connected with all others. Based on the channels' connection strength, we automatically separated the most strongly connected channels by sorting all channels by their node strengths and identifying the largest difference between two adjacent values on the linear and the logarithmic scale (32, 46). We call these epoch-specific channel collections the “core” and based on our previous findings (32, 34) hypothesized them to be indicative of pathological epileptic activity.

2.4. Statistical Analyses

To rule out the possibility of systematic differences, we compared the following quantifiers between patients of different outcome groups: total number of channels implanted, total number and relative portion of channels containing events, epoch-wise average number of channels per event, average event duration, total epoch duration. Likewise, we compared the total number and relative portion of resected channels, core channels, and channels constituting the SOZ and RBT. Moreover, we tested for different proportions of event types depending on the patients' post-surgical seizure control.

We used non-parametric testing throughout this study because sample sizes were small and distributions potentially skewed. Since patients who did not undergo surgery are likely a mixture of the favorable and unfavorable outcome groups with respect to surgery independent quantifiers, we excluded them from all outcome-dependent statistical comparisons, which enabled Mann-Whitney U-tests (MWU) between only two groups. Nevertheless, we display these data in our figures to document that this patient group did not behave systematically different. To compare event proportions between different groups we used Chi-squared tests.

The main objective of this study was to investigate the association between the channel-wise occurence of epileptiform events identified by expert EEG reading and sets of iEEG channels defined by the SOZ, the RBT, and the core channels of non-linear excess interrelation (see Figure S1 of the Supplementary Materials for an illustration), which either require information aggregation, surgical intervention or quantitative analysis. This was done by studying the degree to which one of these channel sets predicted another. Besides, we also examined dependences between these sets. Specifically, we defined a predictive set of iEEG channels, and a target set of channels. We then labeled all channels according to whether they were part of both sets (true positives, TP), only part of the predictive set (false positives, FP), only part of the target set (false negatives, FN), or not part of any set (true negatives, TN). Whenever epileptiform events were used as predictors, we performed this analysis separately for the four event types as well as for all types in aggregation. For every epoch and patient, we then pooled true/false positives/negatives over all events (same type and aggregated). Among the other channel sets, we determined for each epoch the predictive power of the presurgically defined SOZ for the RBT, which becomes available only after surgery. Similarly, as a specific example of qEEG analysis, we have assessed the predictive power of the core of non-linear excess interrelations for both the SOZ and the RBT.

Since events and the aforementioned sets of channels typically only comprised a minority of all iEEG channels, the number of TN by far exceeded those of the other categories in virtually all cases, heavily biasing all dependent accuracy measures. To avoid such bias, we report our results in terms of the TN-independent quantities precision and recall. As an overall accuracy quantifier we used their harmonic mean, the F1-score (see Supplementary Materials for details). All these quantifiers range in the interval [0, 1]. The precision (also called positive predictive value) specifies how indicative the predictive set is for the target set. Low values indicate that channels in the predictive set are often not part of the target set (many FP). Recall (also called sensitivity) specifies to what degree the target set is determined by the predictive set. Low values indicate that channels of the target set are often missed by the predictive set (many FN).

For all statistics we used an uncorrected significance level α = 0.01. Values p < 0.05 were interpreted as trends. Since each accuracy quantifier (precision, recall, F1-score) was tested for six different combinations of predictive and target sets, we applied Bonferroni correction when comparing predictions. The significance level was adjusted to α_Bonf = 0.0017 and values p < 0.0083 were interpreted as trends.

3. Results

In total, our data set contained 99 epochs of intracranial EEG. We found no outcome-dependent differences in the absolute numbers of artifact-free EEG channels, epoch duration, number of channels constituting the SOZ, RBT, or core channels of non-linear excess interrelation between iEEG channels (all p>0.09, MWU). Figure 1 illustrates the relation between iEEG waveforms, visually detectable epileptiform events (here slow and sharp waves, many of them outside the RBT) and the non-linear excess interrelations at the example of patient p10. The selected segment is representative for the total interrelation pattern and shows high precision as well as low recall regarding prediction of the RBT by channels of the core of non-linear excess interrelations or epileptiform events.

FIGURE 1

Figure 1. Example display of iEEG signals and corresponding non-linear excess interrelations. Shown are 8 s of preictal iEEG signals (panel 1) with various annotations of epileptiform events (green boxes). The non-linear excess interrelation matrices are shown for the selected 8-s segment (panel 2) and as average over the entire epoch of 180 s duration (panel 3). The core channels of both matrices are indicated by arrows on the respective y-axes and the selected segment's core channel TAR3 is in addition plotted in red in the EEG display. The RBT is indicated by arrows on the x-axes of the matrices and typeset in boldface in the EEG display. The similarity between all segment-wise matrices and their epoch-wise average was measured by the Pearson correlation coefficient between their elements (panel 4). High precision and low recall of the core of the selected segment to predict the RBT are representative for the entire epoch (panel 5).

In total, 15,070 preictal epileptiform events have been included: 5,693 spikes (37.8%), 5,226 slow waves (34.7%), 3,369 sharp waves (22.4%), and 782 fast oscillations (5.2%). We found no outcome-dependent difference in average event duration, total number of channels containing events, average number of channels per event, and number of events per minute and channel (all p > 0.5, MWU). Likewise, we found no outcome-dependent difference in the relative portion of channels being part of the SOZ, the RBT, or the core of our qEEG analysis (all p > 0.2, MWU). However, there was a trend toward a higher portion of channels containing visually detectable epileptiform events in the favorable outcome group (p = 0.016, MWU).

The number of epileptiform events identified before seizure onset largely varied between different patients and epochs (see Figure 2). Whereas spikes, slow waves, and sharp waves occurred in all patients, fast oscillations were present only in some. The relative partition of event types clearly differed between patients and was roughly patient-specific. In addition, a highly significant outcome-dependent difference in the relative frequencies of event-types was found (spikes, slow waves, sharp waves, fast oscillations), indicating reduced proportion of sharp waves in patients with favorable outcome (p = 0, Chi-Square).

FIGURE 2

Figure 2. Epoch-wise frequency of preictal epileptiform events. To compare the number of events across different epoch lengths and different implantation schemes (i.e. different number of iEEG channels), we normalized to the epoch duration and total number of channels that comprised events. We did not normalize to the total number of channels implanted, because the portion of channels recording from tissue able to produce epileptiform events varied between patients. Epochs are grouped patient-wise. Patients with a favorable post-surgical outcome appear in the upper panel. The dashed vertical line in the lower panel separates patients with an unfavorable outcome (left) resp. without surgery (right). IDs of patients with temporal lobe epilepsy are plotted in bold face.

After removal of patients with a non-temporal seizure onset from the main analysis (see Table 1), 44 epochs from patients with favorable outcome, 21 from patients with unfavorable outcome, and 14 from patients without surgery were used in our group-wise comparisons.

3.1. Do Preictal Epileptiform Events Predict SOZ, RBT or Core Channels?

Figure 3 shows the precision for prediction of the SOZ, the RBT and core channels of the qEEG marker by visually detectable epileptiform events of patients with temporal lobe epilepsy regardless their type. Accuracy quantifiers are summarized in Table 2. Similar results for the small patient group with extra-temporal seizure onset are compiled in section 5 of the Supplementary Materials. General prediction power of epileptiform events for any of the channel sets was low (F1-scores below 0.37 in more than 75% of epochs). No difference was found between patients with favorable and unfavorable post-surgical seizure control for any channel set or measure (all p > 0.01, MWU).

FIGURE 3

Figure 3. Precision of preictal epileptiform events to predict various channel sets. Results are grouped by post-surgical outcome (favorable/unfavorable) resp. those without surgery (n/a). In all panels the circled dot indicates the median of the distribution, the first (q1) and third quartile (q3) are indicated by the bottom and top edges of the box and the whiskers comprise all data points in the range q1−1.5*(q3−q1) to q3+1.5*(q3−q1). Values beyond this range are displayed as dots. The p-values for differences between the favorable and unfavorable outcome groups is indicated at the top. Similar figures for recall and F1-score can be found in the Supplementary Materials.

TABLE 2

Table 2. Distribution of epoch-wise accuracy quantifiers for predictions.

Precision was higher for the prediction of the RBT than of the SOZ (p = 0.0002, MWU), whereas for recall the opposite was found (p = 0.0022, MWU). For prediction of the RBT the precision was higher than the recall (p < 10⁻⁸, MWU), indicating that we found more FN than FP. For prediction of core channels a significant difference in the opposite direction was observed (p < 10⁻³, MWU) and no difference was found for prediction of the SOZ.

Results for separate analysis of all four types of epileptiform events in patients with temporal seizure onset are presented in Figure 4. The precision for the prediction of the RBT by epileptic spikes was higher in the favorable than in the unfavorable outcome group. Apart from this exception, the observations made for event sub-types separately were not different from the pooled analysis. Specifically, none of the event subtypes was associated with the core channels of non-linear excess interrelations.

FIGURE 4

Figure 4. Breakdown of the prediction of channel sets by specific event types. Shown is the precision grouped by the post-surgical outcome.

3.2. Can SOZ or RBT Be Predicted by Quantitative EEG Analysis?

For patients with temporal seizure onset, the precision for prediction of the RBT by the SOZ was high in the favorable outcome group (see Figure 5A and Table 2), and the group difference was significant (p = 0.0004, MWU). For recall and F1-score trends for higher values in the favorable group were observed (see Figure S6 of the Supplementary Materials for a compilation of box plots). Precision was higher than recall in the favorable outcome group (p < 10⁻⁷, MWU), again indicating that prediction of the RBT by the SOZ yielded many more FN than FP.

FIGURE 5

Figure 5. Association among various channel sets. Group-wise precision of the SOZ to predict the RBT resp. of the core to predict either of them.

Figures 5B,C show precision for the prediction of the SOZ and the RBT by the core channels of the qEEG marker. In both cases the median of all accuracy measures (precision, recall and F1-score, see Figures S7, S8 of the Supplementary Materials for box plots) was zero in the unfavorable group but finite if outcome was favorable. In more than 80% of cases with favorable outcome the precision for prediction of the RBT was one, whereas in the unfavorable group it was smaller than 0.7 in 75% of cases. The group difference was significant (p < 10⁻⁵, MWU). In addition, significantly higher recall and F1-score were found in the favorable group (both p < 10⁻⁴, MWU).

For prediction of the SOZ there were outcome-dependent trends for higher precision and recall in the favorable group (p = 0.0023 resp. p = 0.0021, MWU) and a significant difference in the F1-score (p = 0.0014, MWU). In the favorable outcome group precision for prediction of the RBT was higher than recall (p < 10⁻⁹, MWU), again indicating that many more FN than FP were generated. No such difference was found for prediction of the SOZ. The median precision for prediction of the SOZ by core channels was also 1 in the favorable group, but the IQR was broader.

Our preliminary results in section 5 of the Supplementary Materials indicate generally lower associations among epileptiform events, the qEEG marker, the SOZ and the RBT in patients with extra-temporal seizure onset.

3.3. Focus on Patients With Favorable Outcome After Surgery

Reasons for unfavorable seizure control after surgery can be manifold. Since we only know with certainty that the EZ was included in the RBT if seizure freedom was reached, we analyzed the favorable outcome group in more detail. Here, the median precision of 1 for core channels of non-linear interrelation to predict the RBT was significantly higher than the value 0.47 of the epileptiform events (p < 10⁻⁸, MWU, see Figures 5B,C). Recall and F1-score were not different, though (p > 0.45). For prediction of the SOZ we did not find a performance difference between events and core channels (p > 0.025 for all accuracy quantifiers).

3.4. Patients With Bilateral Seizure Onset and Unilateral Resection

Two of the included patients had (unilateral) resections despite bilateral seizure onsets during presurgical evaluation. In these, no epochs preceding seizures with onset contralateral to the resection were included in the previous analyses because they occurred after the first three recorded seizures (which was a selection criterion). However, since we consider these cases as especially elucidating, we analyzed also the seizures with contralateral onset and discuss them separately.

Patient p1 had a right-sided temporal pole-ectomy and selective amygdala-hippocampectomy and became free of disabling seizures after surgery (Engel class I). Epoch-wise averages of the non-linear excess interrelation matrices preceding the first three seizures with left-sided onset are displayed in panels 1 to 3 of Figure 6. Epileptiform events were similarly observable in the RBT and the SOZ located in different brain hemispheres. Strong non-linear excess interrelations were present in the right hemisphere and especially in the RBT but not in any of the channels recording from the left hemisphere (electrodes TE1TL, FML, FPL, and FBL). According to our hypothesis that non-linear excess interrelations could be associated with epileptogenic tissue, this suggests favorable post-surgical seizure control after a right-sided resection. This is indeed in agreement with the observed outcome.

FIGURE 6

Figure 6. Epoch-wise averaged non-linear excess interrelation matrices of two patients with presurgical bilateral seizure onset. Shown are the epochs preceding the seizures with onset contralateral to the resection. Above the matrices, the SOZ and the RBT are indicated by white bars. Below the matrices, the node strength (NS) and the channel-wise number of events (EVT, normalized to the respective color scale) are displayed.

Patient p34 was already discussed in detail in our previous study (32), see Figure 8 and associated paragraphs in section 3.4 of that publication. This patient had a right-sided selective amygdala-hippocampectomy without any subsequent worthwhile improvement (Engel class IV). In the epoch preceding the only available seizure with onset in the left hemisphere, virtually all channels recording from this hemisphere (electrodes AL and HL) show strong non-linear excess interrelations (panel 4 in Figure 6). Epileptiform events were equally dominant in the hippocampus of both hemispheres. Based on our hypothesis one would expect that these widespread non-linear excess interrelations contradict seizure freedom after surgery. Again, this matches the observed outcome.

4. Discussion

4.1. Summary

The main goal of this work was to investigate the relation between our qEEG marker non-linear excess interrelation (32, 34, 41) and preictal epileptiform events detected by a human expert in visual EEG assessment. Since the large majority of our patients had seizure onset in the temporal lobe (32/40), we restricted our analyses to these cases to increase data homogeneity and investigated patients with extra-temporal seizure onset only exploratively, see section 5 of the Supplementary Materials. We did not find a close relation between both; precision for the prediction of core channels of non-linear excess interrelation by preictal events was generally low and so were recall and F1-score (see Table 2, Figure 3C, right column of Figure 4 and Figure S5 of the Supplementary Materials). No significant separation between the favorable and unfavorable outcome groups was observed.

Our analysis revealed a significant outcome dependence of the association between the qEEG marker and the RBT (see Figure 5C) as well as an association with the SOZ (see Figure 5B and Figure S8B of the Supplementary Materials). The main effect was higher precision to predict the RBT in the favorable group (p < 10⁻⁵), confirming an observation made already in (32) based on patients with standardized mesiotemporal electrode implantations. In the present work we refined this analysis and extended to various electrode implantation schemes with depth, strip and grid electrodes placed in the temporal, frontal and parietal lobes, from which only the by far largest subgroup of patients with temporal lobe epilepsy allowed detailed analysis.

Our findings are in line with recent results of a simulation study by (45). When introducing sporadic synthetic spike-and-wave discharges into scalp EEG of healthy controls with physiologically plausible amplitudes they could not observe a relevant alteration of the network structure or strength as measured by (linear) finite-lag cross-correlation. In contrast, when comparing functional connectivity patterns between patients with infantile spasms and frequent spikes to those of healthy controls, they did find patient-specific differences. We view our own results as consistent with these findings in the sense that iEEG recorded from seizure generating brain tissue is identifyable by its altered non-linear excess interrelation (ability to predict the resection in patients with favorable outcome) but individual epileptiform events do not directly cause this interrelation pattern (no prediction of core channels by epileptiform events).

Regarding prediction of the RBT by any of (i) the epileptiform events, (ii) the SOZ, or (iii) the core channels of non-linear excess interrelation we observed a significantly smaller number of FP than FN in patients of the favorable outcome group. This implies that the predictions mainly fall inside resections that help to render patients seizure free but do not fill them entirely. This is plausible, since the RBT is known to be often larger than minimally required for surgical reasons.

For the association with the SOZ, the outcome-dependent group difference was significant for the F1-score (see Figure S7B of the Supplementary Materials) and trends were observed for precision and recall (see Figure 5B and Figure S7A of the Supplementary Materials).

Taken together, the independence of visual and qEEG markers and the better prediction of the resection in patients with favorable post-surgical seizure control provide evidence that our quantitative iEEG analysis may provide additional information about signals recorded from epileptogenic brain tissue that is not accessible to visual inspection. The more detailed examination of two patients with bilateral seizure onsets additionally support our hypotheses (see Figure 6).

The relatively weak association between SOZ and RBT (F1-score < 0.55 in more than 75% of cases, see Table 2 and Figure S6B of the Supplementary Materials) requires explanation, since the resection is usually tailored to remove the SOZ. Precision was high in the majority of cases with favorable outcome (small number of FP), whereas recall was only moderate or even small (considerable number of FN, see Table 2 and Figure 5A). This observation is consistent with the fact that despite the crucial role of the SOZ in surgery planning, the actual resection is typically more extensive for surgical reasons.

Agreement of preictal epileptiform events with the SOZ and the RBT was in general low (see Figures 3A,B, 4). Spikes were the only event sub-type that had differential predictive values in the favorable and unfavorable outcome groups, whereas neither any of the other event sub-types nor all events in conjunction did. Separation between the outcome groups was larger for prediction of the RBT than for prediction of the SOZ. This observation is remarkable, since the SOZ is determined by visual EEG assessment. However, it is crucial to note that the SOZ was defined based on the first ictal signal alterations, whereas the epileptiform events studied here were preictal. It is known that the mechanisms behind both are not necessarily identical (1, 47–49).

4.2. Limitations

Our study has limitations. First, our indirect argument based on contrasting the favorable (Engel classes I and II) and the unfavorable outcome groups (Engel classes III and IV) may be regarded sub-optimal, because there might be reasons for patients to experience ongoing seizures other than incomplete resection of the EZ (e.g., scarring or hypothetical generation of a new EZ). However, this does not affect the data points in the favorable outcome group and our main observations remain valid: The occurrence of preictal epileptiform events does in general not predict the SOZ, the RBT or the core channels of non-linear excess interrelations (see Figure 3). At the same time, the ability of core channels to predict the RBT has a median precision of 1 in the favorable outcome group (see Figure 5C), a value significantly higher than for epileptiform events.

Second, since seizure onset in our patient group was temporal in the vast majority of cases, our data did not allow to investigate a potential confounding influence of etiology. Instead, we restricted our main group analyses to temporal onset cases. Robust evaluation of patients with extra-temporal seizure onset will require collection of more such cases and remains the scope of future work.

Third, we did not explore the impact of disease duration.

Fourth, EEGs of patients in the favorable outcome group showed a trend toward a higher proportion of channels with visually detectable epileptiform events. A possible explanation is that in “easier patients” the location of the epileptogenic brain tissue was clearer a priori. Thus, also the implantation scheme and the resection were better defined. The relevance of spatial sampling for qEEG results has recently also been highlighted by (50).

Finally, we had expert annotations available only from a single rater since detailed annotating is very time consuming. Publicly available EEG data of epilepsy patients could not be used to enhance our study size for different reasons. These data are either restricted to scalp EEG [(51); isip.piconepress.com/projects/tuh_eeg/], lack at least one essential piece of information about surgery extent, outcome or detailed EEG annotations (ieeg.org, openneuro.org/datasets/ds003029/versions/1.0.2) or require payment of usage fees (epilepsy-database.eu/).

4.3. Conclusion

In conclusion, we herein demonstrated the potential of non-linear excess interrelations between preictal iEEG signals to provide clinically useful additional information for the planning of resective surgery in temporal lobe epilepsy. Importantly, we have shown additionally that this qEEG marker is largely independent from visually detectable preictal epileptiform events and has a higher precision for predicting the RBT than the SOZ in cases with favorable outcome. Hence, our quantitative iEEG analysis is not directly associated with established visually detectable markers of epileptogenic brain tissue, which are regarded as locally restricted phenomena. It rather captures potentially far-reaching non-linear dependencies between brain regions that seem to reflect pathological activity. This further underlines the benefit and importance to incorporate the network concept of epilepsy into presurgical patient evaluation. To foster similar research also for other qEEG makers, we made the iEEG recordings and human annotations used in this paper publicly available. Furthermore, we encourage the scientific community to provide independent expert annotations of these recordings using our custom EEG reader to enable comparison also with inter-rater agreement.

Data Availability Statement

The EEG reader and the raw data supporting the conclusions of this article are available via GitHub: https://github.com/SCAN-NRAD/scanEEGviewer.

Ethics Statement

The studies involving human participants were reviewed and approved by Ethics Committee of the Canton of Bern. Written informed consent to participate in this study was provided by the participants' legal guardian/next of kin.

Author Contributions

CR: study design. MD: data annotation. MM and CR: data analysis and writing of manuscript. KS and CR: data curation. MM: writing of software. MM, MD, KS, RW, and CR: final review of manuscript. All authors contributed to the article and approved the submitted version.

Funding

This study was supported by the Swiss National Science Foundation (SNSF) (grant no: CRSK-3_190817/1).

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank Stefan Jonas for the extensive testing of our EEG reader.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fneur.2021.741450/full#supplementary-material

References

1. Rosenow F, Lüders H. Presurgical evaluation of epilepsy. Brain. (2001) 124:1683–700. doi: 10.1093/brain/124.9.1683

CrossRef Full Text | Google Scholar

2. Lüders HO, Najm I, Nair D, Widdess-Walsh P, Bingman W. The epileptogenic zone: general principles. Epileptic Disord. (2006) 8:1–9.

Google Scholar

3. Téllez-Zenteno JF, Dhar R, Wiebe S. Long-term seizure outcomes following epilepsy surgery: a systematic review and meta-analysis. Brain. (2005) 128:1188–98. doi: 10.1093/brain/awh449

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Spencer S, Huh L. Outcomes of epilepsy surgery in adults and children. Lancet Neurol. (2008) 7:525–37. doi: 10.1016/S1474-4422(08)70109-1

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Aull-Watschinger S, Pataraia E, Czech T, Baumgartner C. Outcome predictors for surgical treatment of temporal lobe epilepsy with hippocampal sclerosis. Epilepsia. (2008) 49:1308–16. doi: 10.1111/j.1528-1167.2008.01732.x

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Tisi JD, Bell GS, Peacock JL, McEvoy AW, Harkness WF, Sander JW, et al. The long-term outcome of adult epilepsy surgery, patterns of seizure remission, and relapse: a cohort study. Lancet. (2011) 378:1388–95. doi: 10.1016/S0140-6736(11)60890-8

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Bulacio JC, Jehi L, Wong C, Gonzalez-Martinez J, Kotagal P, Nair D, et al. Long-term seizure outcome after resective surgery in patients evaluated with intracranial electrodes. Epilepsia. (2012) 53:1722–30. doi: 10.1111/j.1528-1167.2012.03633.x

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Edelvik A, Rydenhag B, Olsson I, Flink R, Kumlien E, Källén K, et al. Long-term outcomes of epilepsy surgery in Sweden: a national prospective and longitudinal study. Neurology. (2013) 81:1244–51. doi: 10.1212/WNL.0b013e3182a6ca7b

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Fauser S, Essang C, Altenmüller DM, Staack AM, Steinhoff BJ, Strobl K, et al. Long-term seizure outcome in 211 patients with focal cortical dysplasia. Epilepsia. (2015) 56:66–76. doi: 10.1111/epi.12876

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Mohan M, Keller S, Nicolson A, Biswas S, Smith D, Farah JO, et al. The long-term outcomes of epilepsy surgery. PLoS ONE. (2018) 13:e0196274. doi: 10.1371/journal.pone.0196274

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Engel J, Thompson PM, Stern JM, Staba RJ, Bragin A, Mody I. Connectomics and epilepsy. Curr Opin Neurol. (2013) 26:186–94. doi: 10.1097/WCO.0b013e32835ee5b8

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Kramer MA, Cash SS. Epilepsy as a disorder of cortical network organization. Neuroscientist. (2012) 18:360–72. doi: 10.1177/1073858411422754

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Scott RC, de la Prida LM, Mahoney JM, Kobow K, Sankar R, de Curtis M. WONOEP APPRAISAL: the many facets of epilepsy networks. Epilepsia. (2018) 59:1475–83. doi: 10.1111/epi.14503

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Shih JJ. It's all about the networks. Epilepsy Curr. (2019) 19:165–7. doi: 10.1177/1535759719843301

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Diessen EV, Diederen SJH, Braun KPJ, Jansen FE, Stam CJ. Functional and structural brain networks in epilepsy: what have we learned? Epilepsia. (2013) 54:1855–65. doi: 10.1111/epi.12350

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Andrzejak RG, David O, Gnatkovsky V, Wendling F, Bartolomei F, Francione S, et al. Localization of epileptogenic zone on pre-surgical intracranial EEG recordings: toward a validation of quantitative signal analysis approaches. Brain Topogr. (2015) 28:832–7. doi: 10.1007/s10548-014-0380-8

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Musk E. An integrated brain-machine interface platform with thousands of channels. J Med Internet Res. (2019) 21:e16194. doi: 10.2196/preprints.16194

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Cook MJ, O'Brien TJ, Berkovic SF, Murphy M, Morokoff A, Fabinyi G, et al. Prediction of seizure likelihood with a long-term, implanted seizure advisory system in patients with drug-resistant epilepsy: a first-in-man study. Lancet Neurol. (2013) 12:563–71. doi: 10.1016/S1474-4422(13)70075-9

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Weisdorf S, Duun-Henriksen J, Kjeldsen MJ, Poulsen FR, Gangstad SW, Kjær TW. Ultra-long-term subcutaneous home monitoring of epilepsy–490 days of EEG from nine patients. Epilepsia. (2019) 60:2204–14. doi: 10.1111/epi.16360

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Höller Y, Kutil R, Klaffenböck L, Thomschewski A, Höller PM, Bathke AC, et al. High-frequency oscillations in epilepsy and surgical outcome. A meta-analysis. Front Hum Neurosci. (2015) 9:574. doi: 10.3389/fnhum.2015.00574

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Roehri N, Pizzo F, Lagarde S, Lambert I, Nica A, McGonigal A, et al. High-frequency oscillations are not better biomarkers of epileptogenic tissues than spikes. Ann Neurol. (2018) 83:84–97. doi: 10.1002/ana.25124

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Jacobs J, Wu JY, Perucca P, Zelmann R, Mader M, Dubeau F, et al. Removing high-frequency oscillations: a prospective multicenter study on seizure outcome. Neurology. (2018) 91:e1040–52. doi: 10.1212/WNL.0000000000006158

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Boccaletti S, Latora V, Moreno Y, Chavez M, Hwang DU. Complex networks: structure and dynamics. Phys Rep. (2006) 424:175–308. doi: 10.1016/j.physrep.2005.10.009

CrossRef Full Text | Google Scholar

24. Stam CJ, Reijneveld JC. Graph theoretical analysis of complex networks in the brain. Nonlinear Biomed Phys. (2007) 1:3. doi: 10.1186/1753-4631-1-3

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Casdagli M, Iasemidis L, Savit RS, L RG, Roper SN, Sackellares JC. Non-linearity in invasive EEG recordings from patients with temporal lobe epilepsy. Electroencephalogr Clin Neurophysiol. (1997) 102:98–105. doi: 10.1016/S0921-884X(96)95195-4

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Andrzejak RG, Mormann F, Widman G, Kreuz T, Elger CE, Lehnertz K. Improved spatial characterization of the epileptic brain by focusing on nonlinearity. Epilepsy Res. (2006) 69:30–44. doi: 10.1016/j.eplepsyres.2005.12.004

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Andrzejak RG, Schindler K, Rummel C. Nonrandomness, nonlinear dependence, and nonstationarity of electroencephalographic recordings from epilepsy patients. Phys Rev E Stat Nonlin Soft Matter Phys. (2012) 86:046206. doi: 10.1103/PhysRevE.86.046206

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Anvari M, Tabar MRR, Peinke J, Lehnertz K. Disentangling the stochastic behavior of complex time series. Sci Rep. (2016) 6:35435. doi: 10.1038/srep35435

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Rizzi M, Weissberg I, Milikovsky DZ, Friedman A. Following a potential epileptogenic insult, prolonged high rates of nonlinear dynamical regimes of intermittency type is the hallmark of epileptogenesis. Sci Rep. (2016) 6:31129. doi: 10.1038/srep31129

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Andrzejak RG, Chicharro D, Lehnertz K, Mormann F. Using bivariate signal analysis to characterize the epileptic focus: the benefit of surrogates. Phys Rev E Stat Nonlin Soft Matter Phys. (2011) 83:046203. doi: 10.1103/PhysRevE.83.046203

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Gotman J. How would you like your epileptic network? Linear, nonlinear, virtual? Epilepsy Curr. (2020) 20:80–82. doi: 10.1177/1535759720904161

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Müler M, Caporro M, Gast H, Pollo C, Wiest R, Schindler K, et al. Linear and nonlinear interrelations show fundamentally distinct network structure in preictal intracranial EEG of epilepsy patients. Hum Brain Mapp. (2020) 41:467–83. doi: 10.1002/hbm.24816

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Westmoreland BF. Epileptiform electroencephalographic patterns. Mayo Clin Proc. (1996) 71:501–11. doi: 10.4065/71.5.501

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Rummel C, Abela E, Andrzejak RG, Hauf M, Pollo C, Müller M, et al. Resected brain tissue, seizure onset zone and quantitative EEG measures: towards prediction of post-surgical seizure control. PLoS ONE. (2015) 10:e0141023. doi: 10.1371/journal.pone.0141023

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Cook MJ, Varsavsky A, Himes D, Leyde K, Berkovic S, O'Brien T, et al. The dynamics of the epileptic brain reveal long memory processes. Front Neurol. (2014) 5:217. doi: 10.3389/fneur.2014.00217

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Cook MJ, Karoly PJ, Freestone DR, Himes D, Leyde K, Berkovic S, et al. Human focal seizures are characterized by populations of fixed duration and interval. Epilepsia. (2016) 57:359–68. doi: 10.1111/epi.13291

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Baud MO, Kleen JK, Mirro EA, Andrechak JC, King-Stephens D, Chang EF, et al. Multi-day rhythms modulate seizure risk in epilepsy. Nat Commun. (2018) 9:88. doi: 10.1038/s41467-017-02577-y

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Kuhlmann L, Lehnertz K, Richardson MP, Schelter B, Zaveri HP. Seizure prediction – ready for a new era. Nat Rev Neurol. (2018) 14:618–30. doi: 10.1038/s41582-018-0055-2

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Rings T, von Wrede R, Lehnertz K. Precursors of seizures due to specific spatial-temporal modifications of evolving large-scale epileptic brain networks. Sci Rep. (2019) 9:10623. doi: 10.1038/s41598-019-47092-w

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Kuhnert MT, Elger CE, Lehnertz K. Long-term variability of global statistical properties of epileptic brain networks. Chaos. (2010) 20:043126. doi: 10.1063/1.3504998

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Rummel C, Abela E, Müller M, Hauf M, Scheidegger O, Wiest R, et al. Uniform approach to linear and nonlinear interrelation patterns in multivariate time series. Phys Rev E Stat Nonlin Soft Matter Phys. (2011) 83:066215. doi: 10.1103/PhysRevE.83.066215

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Müller M, Schindler K, Goodfellow M, Pollo C, Rummel C, Steimer A. Evaluating resective surgery targets in epilepsy patients: a comparison of quantitative EEG methods. J Neurosci Methods. (2018) 305:54–66. doi: 10.1016/j.jneumeth.2018.04.021

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Laiou P, Avramidis E, Lopes M, Abela E, Müller M, Akman O, et al. Quantification and selection of ictogenic zones in epilepsy surgery. Front Neurol. (2019) 10:1045. doi: 10.3389/fneur.2019.01045

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Schreiber T, Schmitz A. Surrogate time series. Physica D Nonlin Phenomena. (2000) 142:346–82. doi: 10.1016/S0167-2789(00)00043-9

CrossRef Full Text | Google Scholar

45. Hu DK, Mower A, Shrey DW, Lopour BA. Effect of interictal epileptiform discharges on EEG-based functional connectivity networks. Clin Neurophysiol. (2020) 5:1087–98. doi: 10.1016/j.clinph.2020.02.014

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Rummel C. Quantification of intra- and inter-cluster relations in nonstationary and noisy data. Phys Rev E Stat Nonlin Soft Matter Phys. (2008) 77:016708. doi: 10.1103/PhysRevE.77.016708

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Blauwblomme T, Jiruska P, Huberfeld G. Mechanisms of ictogenesis. Int Rev Neurobiol. (2014) 114:155–85. doi: 10.1016/B978-0-12-418693-4.00007-8

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Curtis MD, Avanzini G. Interictal spikes in focal epileptogenesis. Progr Neurobiol. (2001) 63:541–67. doi: 10.1016/S0301-0082(00)00026-5

PubMed Abstract | CrossRef Full Text | Google Scholar

49. de Curtis M, Gnatkovsky V. Reevaluating the mechanisms of focal ictogenesis: the role of low-voltage fast activity. Epilepsia. (2009) 50:2514–25. doi: 10.1111/j.1528-1167.2009.02249.x

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Wang Y, Sinha N, Schroeder GM, Ramaraju S, McEvoy AW, Miserocchi A, et al. Interictal intracranial electroencephalography for predicting surgical success: the importance of space and time. Epilepsia. (2020) 61:1417–26. doi: 10.1111/epi.16580

PubMed Abstract | CrossRef Full Text | Google Scholar

51. Obeid I, Picone J. The temple university hospital EEG data corpus. Front Neurosci. (2016) 10:196. doi: 10.3389/fnins.2016.00196

PubMed Abstract | CrossRef Full Text | Google Scholar