The association between the neutrophil-to-lymphocyte ratio, platelet-to-lymphocyte ratio, and lymphocyte-to-monocyte ratio and delirium in ischemic stroke patients

Background: Delirium is a severe neuropsychiatric symptom following acute ischemic stroke (IS) and is associated with poor outcomes. Systemic inflammation and immune dysregulation are believed to contribute to the pathophysiology of delirium. The neutrophil-to-lymphocyte ratio (NLR), platelet-to-lymphocyte ratio (PLR), and lymphocyte-to-monocyte ratio (LMR) are widely recognized as convenient and reliable biomarkers of systemic inflammation. However, their association with delirium after IS remains unclear.

Methods: In this study, we identified IS patients requiring ICU admission from the Medical Information Mart for Intensive Care (MIMIC)-IV database. We employed multivariable logistic regression and restricted cubic splines (RCS) to assess the association between the NLR, PLR, and LMR and delirium. Two-sample Mendelian randomization (MR) analysis was performed to further explore their causal relationship at the genetic level.

Results: A total of 1,436 patients with IS were included in this study, of whom 214 (14.9%) had delirium. In the multivariate logistic regression analysis, after adjustment for confounders, the patients in the highest quartile of the NLR (odds ratio [OR] 2.080, 95% confidence interval [CI], 1.282–3.375) and LMR (OR 0.503, 95% CI 0.317–0.798) and the patients in the second quartile of the PLR (OR 1.574, 95% CI 1.019–2.431) were significantly associated with delirium. The RCS function showed a progressive increase in the risk of delirium with higher NLR and PLR and lower LMR. In the MR analysis, only the PLR was negatively associated with the risk of delirium.

Conclusion: The observational studies found significant associations between the NLR, PLR, and LMR and delirium. However, the MR analysis only demonstrated a potential protective causal relationship between the PLR and delirium. Further prospective studies are needed to validate their association and to elucidate the underlying mechanisms.

1 Introduction

According to the World Health Organization, stroke is the second leading cause of death and the third leading cause of disability worldwide (1). As one of the most devastating neurological disorders, stroke imposes a significant economic and healthcare burden on society (2). Stroke can be divided into two major categories based on neuropathology: ischemic stroke (IS) and hemorrhagic stroke (3). Among them, IS is characterized by the interruption of cerebral blood supply due to various causes, resulting in corresponding neurological impairment, and it accounts for approximately 70–80% of all strokes (4). Intracranial atherosclerotic plaques and large artery stenosis are identified in 45–62% of patients with IS (5), while microvascular thrombosis and small artery occlusion are responsible for 25% of cases (5) and cardiogenic embolism is attributed to IS in 22% of cases (6, 7). While the neurological function of the majority of IS patients improves after intravenous thrombolysis or mechanical thrombectomy, some still suffer from varying degrees of complications, such as delirium and speech disorders (8).

Delirium is a severe neuropsychiatric symptom commonly observed in elderly patients following acute illness or surgery, characterized by a rapid decline in cognitive function, impaired attention, and disturbance of consciousness (9, 10). It is a common complication in hospitalized patients, with an incidence of approximately 15 to 27% in patients with stroke (11–14). Delirium has been associated with prolonged hospital stays (15), increased in-hospital mortality (16), permanent cognitive impairment (17, 18), and dementia (19) in patients with IS.

However, the pathophysiological mechanisms underlying delirium remain unclear, and there are currently no effective intervention measures available in clinical practice. Therefore, it is particularly important to identify modifiable risk factors and establish intervention strategies. Previous observational studies have suggested that inflammatory factors may be one of the precipitating factors for delirium (20, 21). Among the various inflammatory markers, the neutrophil-to-lymphocyte ratio (NLR), platelet-to-lymphocyte ratio (PLR), and lymphocyte-to-monocyte ratio (LMR) have emerged as convenient and reliable indicators of systemic inflammation. The NLR reflects the balance between the innate immune response (mediated by neutrophils) and adaptive immunity (mediated by lymphocytes), serving as a proxy for immune system dysregulation. Similarly, the PLR captures the interplay between inflammation and thrombophilia, while the LMR provides insight into the severity of systemic inflammation through its negative correlation with inflammatory activity. These readily available biomarkers have shown promise in predicting outcomes in various inflammatory and cardiovascular conditions (22), yet their roles in delirium following IS remain underexplored.

In this study, we specifically focused on ischemic stroke (IS) patients requiring ICU admission. We aimed to investigate the relationship between the NLR, PLR, and LMR and post-IS delirium, with the goal of identifying potential biomarkers for the early prediction of delirium. To achieve this, we combined observational data from the Medical Information Mart for Intensive Care (MIMIC)-IV database with Mendelian randomization (MR) methods. MR is an emerging epidemiological approach that uses genetic variables as proxies to assess the effects of exposures (e.g., NLR, PLR, and LMR) on specific outcomes (e.g., delirium) (23). Compared to observational studies, this approach can avoid the effects of confounding and reverse causality and can better simulate the results of randomized controlled trials. In this study, we combined observational data from the MIMIC-IV database with MR methods to investigate the causal effects of the NLR, PLR, and LMR on delirium after IS.

2 Methods

2.1 Overall study design

This study consisted of two phases. In the first phase, we performed multivariable logistic regression analysis using data from the MIMIC-IV v2.2 database to explore the association between the NLR, PLR, and LMR and delirium after IS. Before using the MIMIC database, we obtained institutional review board approval from Beth Israel Deaconess Medical Center and the Massachusetts Institute of Technology (record ID: 12299215).

In the second phase, we further investigated the causal relationship between the NLR, PLR, and LMR and delirium using summary statistics from genome-wide association studies (GWAS) through MR analysis. Based on the law of independent assortment, genetic variations selected as instrumental variables (IVs) for the exposure are randomly allocated to gametes during meiosis (24). Due to the random assignment of genotypes, MR analysis using single nucleotide polymorphisms (SNPs) as genetic IVs can effectively mimic the design of a randomized controlled trial (RCT), enabling the estimation of causal effects between the exposure and outcome without being influenced by traditional confounders (25), such as sex, since genetic variations do not systematically vary with demographic factors. The GWAS data for delirium were obtained from the ninth version of the FinnGen database, which includes 3,039 cases and 356,660 controls (ID: F5_DELIRIUM). According to the database, all cases of delirium were classified as conditions not caused by alcohol or other psychoactive substances. The GWAS data for the NLR, PLR, and LMR were obtained from a large-scale analysis of leukocyte-related genetic data, as published in the study by Zhou et al. (26). This study utilized summary statistics derived from the UK Biobank, and the datasets are publicly available. The specific details of the GWAS data are as follows: the GWAS summary statistics for the NLR were obtained from the study by Zhou et al. (26). The analysis included 5,973 SNPs and was conducted on a population of European ancestry, as described by the authors. The GWAS summary statistics for the PLR were sourced from the same study (26), involving 15,473 SNPs and based on a cohort of European ancestry. The data for the LMR were also obtained from the study by Zhou et al. (26), with analysis including 4,606 SNPs, conducted on a population of European ancestry, as described by the original research group. To obtain effective causal estimates, the IVs used in MR analysis must meet the following criteria: (1) Genetic variations must be strongly associated with the levels of the NLR, PLR, and LMR; (2) Genetic variations should not be influenced by confounding factors; (3) Genetic variations should affect delirium only through their influence on the NLR, PLR, and LMR. In addition, these summary statistic data come from different sample databases of European ancestry, and the probability of sample overlap is small. The original research for these GWAS studies had already received ethics committee approval (27, 28). Therefore, this study did not require additional ethical review.

2.2 Data sources and patient selection

This study used data from the MIMIC-IV (version 2.2) database, a comprehensive clinical database containing records of patients admitted to the intensive care units (ICUs) of Beth Israel Deaconess Medical Center. The MIMIC-IV database was developed and is managed by the MIT Laboratory for Computational Physiology. Patients diagnosed with IS according to the International Classification of Diseases, 9th and 10th revisions, were included in this study. The primary outcome of this study was to determine the presence of delirium during hospitalization. The assessment of delirium was primarily conducted using the widely adopted clinical tool, the Confusion Assessment Method for the ICU (CAM-ICU). The CAM-ICU consists of four features: Feature 1: acute changes or fluctuations in mental status; Feature 2: inattention; Feature 3: disorganized thinking; and Feature 4: altered level of consciousness (LOC) (29, 30). A patient was considered CAM-ICU-positive and diagnosed with delirium when Features 1 and 2 were both present, along with either Feature 3 or Feature 4 (29). The exclusion criteria were as follows: (1) patients who were admitted to the ICU multiple times for IS, with only the record of the first admission being extracted; (2) patients without sufficient data from the NLR, PLR, and LMR examinations on the first day of their stay in the ICU; (3) patients with a history of schizophrenia; and (4) patients with dementia, infections, trauma, allergies, or neoplasms. Ultimately, a total of 1,436 patients were enrolled in this study and categorized into two groups based on the presence or absence of delirium (Figure 1).

Figure 1

Figure 1. Flow diagram of patient selection.

2.3 Data extraction

These data were extracted from the MIMIC-IV database using Structured Query Language (SQL) and PostgreSQL (version 13.7.2) software. The dataset included the following information: (1) demographic characteristics: age, sex, ethnicity, body mass index (BMI), and adverse lifestyle factors; (2) comorbidities: diabetes, hypertension, heart disease, chronic kidney disease, chronic lung disease, chronic liver disease, peripheral vascular disease, and obstructive sleep apnea; (3) laboratory parameters: hemoglobin, platelet count, serum creatinine, fasting blood glucose, serum albumin, white blood cell count, lymphocyte count, monocyte count, and neutrophil count; (4) admission severity scores: the Sequential Organ Failure Assessment (SOFA) score and Glasgow Coma Scale (GCS) score; (5) admission vital signs: heart rate, mean arterial pressure (MAP), respiratory rate, and peripheral oxygen saturation (SpO₂); (6) use of benzodiazepines; (7) and outcomes: the primary outcome was the onset of delirium, and the secondary outcomes included length of hospital stay and in-hospital mortality. The NLR, PLR, and LMR were calculated using the following formulas: (1) NLR = neutrophil count/lymphocyte count, (2) PLR = platelet count/lymphocyte count, and (3) LMR = lymphocyte count/monocyte count. The ICD-9 and ICD-10 codes used to identify IS and delirium are shown in Supplementary Table S1.

2.4 Statistical analysis

For the observational studies using MIMIC-IV data, categorical variables were presented as numbers (n) and percentages (%), while continuous variables were presented as median (interquartile range, IQR) due to their non-normal distribution. The chi-squared test was used to compare the categorical variables between the groups. For the continuous variables, the Kruskal–Wallis test was applied to compare differences between the groups. Multivariable logistic regression analysis was used to assess the associations between the NLR, PLR, and LMR and delirium. To reduce bias caused by confounders, we adjusted for age, sex, ethnicity, BMI, and alcohol abuse in Model 1. Model 2 was adjusted for age, sex, ethnicity, BMI, alcohol abuse, SOFA score, GCS score, and comorbidities, including hypertension, diabetes mellitus, chronic pulmonary disease, chronic liver disease, chronic renal disease, congestive heart failure, myocardial infarction, peripheral vascular disease, and obstructive sleep apnea. Model 3 was further adjusted for benzodiazepine use and laboratory results, including hemoglobin levels, glucose, albumin, and creatinine, based on Model 2. The results were presented as odds ratios (ORs) with 95% confidence intervals (CIs), and statistical significance was determined at a p < 0.05.

In addition, we performed a two-sample MR analysis to re-evaluate the causal relationships between the NLR, PLR, LMR, and delirium. First, we selected single nucleotide polymorphisms (SNPs) that were strongly associated with the NLR, PLR, and LMR, with a p < 5 × 10^–8. In addition, to exclude linkage disequilibrium (LD), we set R² to <0.001 (clumping window size = 10,000 kb). Then, the SNPs with a minor allele frequency (MAF) ≤ 0.01 were removed. Finally, we harmonized the alleles of the exposures and outcomes to eliminate ambiguous SNPs with inconsistent alleles and intermediate allele frequencies. For the MR analysis, the inverse-variance weighted (IVW) method was used as the primary analytical approach. The MR-Egger, weighted median, and weighted mode methods were used to validate the IVW results. MR-Pleiotropy RESidual Sum and Outlier (MR-PRESSO) was used to identify potential outliers and horizontal pleiotropy. Cochrane’s Q test and MR-Egger intercept were used to assess heterogeneity and directional pleiotropy, respectively (31, 32). Sensitivity analyses were also performed using leave-one-out analysis. All analyses were two-sided and were performed using the TwoSampleMR and MR-PRESSO packages in R software (version 4.0.2).

3 Results

3.1 Baseline characteristics

In this study, a total of 1,436 patients with IS were included for analysis, and the selection process is shown in Figure 1. The median age of the included patients was 71 years (IQR: 59–81 years), and 52.4% were male. Among them, 14.9% experienced delirium during hospitalization and 224 patients (15.6%) had fatal events. Table 1 shows the baseline characteristics of the patients, stratified according to the presence or absence of delirium. Compared to the patients without delirium, those with delirium tended to have more comorbidities, higher SOFA scores on admission, a higher proportion of benzodiazepine use, poorer nutritional status, unstable vital signs, and longer hospital stays. The NLR values were significantly higher in the patients with delirium compared to the patients without delirium (7.23 vs. 5.65, p = 0.000), while the LMR values were significantly lower in the delirium group (2.22 vs. 2.72, p = 0.000). There was no significant difference in the PLR values between the two groups.

Table 1

Table 1. Baseline characteristics (n = 1,436).

3.2 The association between the NLR, PLR, and LMR, and delirium

Table 2 shows the association of the NLR, PLR, and LMR with the incidence of delirium after IS. In the univariate logistic regression analysis, an increase in the NLR was independently associated with an increased risk of delirium (OR 1.032, 95% CI 1.016–1.049). After adjustment for age, sex, ethnicity, BMI, and lifestyle factors in Model 1, this association remained significant (OR 1.034, 95% CI 1.017–1.050). The relationship between the elevated NLR and delirium risk persisted after adjustment for comorbidities, SOFA score, and GCS score in Model 2 (OR 1.023, 95% CI 1.006–1.041), as well as for laboratory results and benzodiazepine use in Model 3 (OR 1.022, 95% CI 1.004–1.039). However, in the univariate logistic regression analysis, as well as in models 1, 2, and 3, there was no significant association between an increase in the PLR and the risk of delirium (all p > 0.05). In the regression analysis between the LMR and delirium, the univariate logistic regression showed that an increase in the LMR was independently associated with a decreased risk of delirium (OR 0.891, 95% CI 0.831–0.956). This significant association remained after adjustment for confounders in Model 1, Model 2, and Model 3 (all p < 0.05). In addition, when the NLR was categorized as a nominal variable, the patients in the second, third, and highest quartiles of the NLR were significantly associated with an increased risk of delirium across all three logistic regression models (Model 1, Model 2, and Model 3) compared to those in the lowest quartile (Table 3). Similarly, the patients in the second quartile of the PLR were significantly associated with a higher risk of delirium compared to the patients in the lowest quartile: Model 2 (OR 1.579, 95% CI 1.026–2.429) and Model 3 (OR 1.574, 95% CI 1.019–2.431). Lastly, the patients in the highest quartile of the LMR were negatively associated with an increased risk of delirium compared to the patients in the lowest quartile: Model 1 (OR 0.430, 95% CI 0.276–0.672), Model 2 (OR 0.489, 95% CI 0.310–0.773), and Model 3 (OR 0.503, 95% CI 0.317–0.798). For the NLR and LMR, as well as the PLR in models 2 and 3, the restricted cubic spline (RCS) models showed that they were non-linearly correlated with delirium (all non-linear p < 0.05) (Figure 2).

Table 2

Table 2. Multivariable logistic regression analysis of the associations between the NLR, PLR, and LMR and delirium after ischemic stroke.

Table 3

Table 3. Odds ratio (95% CI) for delirium across the quartiles of the NLR, PLR, and LMR.

Figure 2

Figure 2. Restricted spline curves showing the relationship between the NLR, PLR, and LMR and delirium in the patients with ischemic stroke. The red bold line denotes the OR, while the shaded area represents the 95% CI. No covariates were adjusted in (A–C). Age, sex, ethnicity, BMI, and lifestyle factors were adjusted in Model 1 (D–F). Model 2 was further adjusted for comorbidities, SOFA score, and GCS score based on Model 1 (G–I). In Model 3, laboratory results and benzodiazepine use were adjusted based on Model 2 (J–L). BMI, body mass index; SOFA, Sequential Organ Failure Assessment; GCS, Glasgow Coma Scale.

3.3 Subgroup analysis

In addition, subgroup analyses were conducted to assess the associations between the NLR, PLR, and LMR and delirium across different populations. The participants were stratified based on sex, congestive heart failure, chronic pulmonary disease, hypertension, and diabetes. The results are presented in Figure 3.

Figure 3

Figure 3. Subgroup analyses of the association between the NLR, PLR, and LMR and delirium. (A) NLR; (B) PLR; (C) LMR. NLR, neutrophil-to-lymphocyte ratio; PLR, platelet-to-lymphocyte ratio; LMR, lymphocyte-to-monocyte ratio; OR, odds ratios; CI, confidence interval.

The positive correlation between the NLR and delirium was particularly pronounced in the female patients (OR 1.05, 95% CI 1.02–1.07) and the participants with hypertension (OR 1.04, 95% CI 1.02–1.06). This association remained consistent across the subgroups with and without congestive heart failure, chronic pulmonary disease, and diabetes, highlighting potential sex-specific and comorbidity-dependent influences. However, there was a positive association between the PLR and delirium in those with hypertension (OR 1.00, 95% CI 1.00–1.00) and a negative association in those without hypertension (OR 1.00, 95% CI 0.99–1.00). In contrast, the LMR showed a significant negative association with delirium, particularly in the female participants (OR 0.81, 95% CI 0.71–0.92), participants without congestive heart failure (OR 0.85, 95% CI 0.77–0.94), participants without chronic pulmonary disease (OR 0.89, 95% CI 0.82–0.96), participants with hypertension (OR 0.89, 95% CI 0.82–0.96), and those with and without diabetes. The interaction test suggested that the relationship between the LMR and delirium was affected by sex (male/female) (Pp < 0.05). Female individuals may serve as effect modifiers (Figure 3).

Moreover, in the sex-stratified analysis of comorbidities, the positive correlation between the NLR and delirium was more pronounced in the female participants across all comorbidity subgroups, including those with and without congestive heart failure, chronic pulmonary disease, and diabetes, as well as those with hypertension (Figure 4). The interaction test suggested that sex played a pivotal role in modulating these relationships, with female individuals acting as potential effect modifiers (p < 0.05). Similarly, the negative association between the PLR and delirium was more evident in the female participants, with and without hypertension. Furthermore, the LMR showed a significant protective effect against delirium in the female participants across all comorbidity subgroups, including those with congestive heart failure, chronic pulmonary disease, diabetes, and hypertension.

Figure 4

Figure 4. Sex-stratified analysis of comorbidities in the association between the NLR, PLR, and LMR and delirium. (A) NLR; (B) PLR; (C) LMR. NLR, neutrophil-to-lymphocyte ratio; PLR, platelet-to-lymphocyte ratio; LMR, lymphocyte-to-monocyte ratio; OR, odds ratios; CI, confidence interval.

3.4 Causal association between the NLR, PLR, and LMR and delirium in MR

In the MR study, we screened a total of 59, 85 and 44 SNPs, to investigate the causal relationship between the NLR, PLR, and LMR and delirium. Detailed information about these SNPs can be found in Supplementary Table S2. The results showed that an increase in the genetically predicted PLR was causally associated with a decreased risk of delirium (IVW: OR 0.996, 95% CI 0.994–0.999, p = 0.040). This finding was further validated by MR-PRESSO (OR 0.997, 95% CI 0.994–0.999, p = 0.036) (Table 4). On the other hand, no significant association was observed between an increase in the genetically predicted NLR (IVW: OR 0.799, 95% CI 0.628–1.017, p = 0.068) and LMR (IVW: OR 1.033, 95% CI 0.942–1.133, p = 0.489) and the risk of delirium. These results indicated a lack of evidence for a causal relationship between the NLR and LMR levels and the risk of delirium. In addition, the intercept obtained from the MR-Egger regression was not significant (Supplementary Table S3), suggesting that there was no significant horizontal pleiotropy in the MR analysis. The results of the MR-PRESSO heterogeneity test further confirmed the accuracy of the MR-Egger regression. In the absence of heterogeneity and pleiotropy, the abovementioned MR results were considered reliable.

Table 4

Table 4. The results of the MR study investigating the effects of the NLR, PLR, and LMR on delirium.

In the sensitivity analysis, the funnel plots (Supplementary Figure S1) showing the results of the MR analysis displayed a symmetrical shape, indicating no heterogeneity in the results. The leave-one-out analysis revealed a significant association between the NLR and delirium when the SNPs “rs61839660,” “rs77919370,” “rs4760,” “rs11975539,” “rs6981399,” and “rs4792711” were excluded from the model (Supplementary Figure S2). However, this association was no longer significant when these SNPs were included. In the analysis of the association between the PLR and delirium, excluding the SNPs “rs6713632,” “rs4907923,” “rs1982151,” “rs7003580,” “rs1076415,” “rs10794175,” “rs11953668,” “rs10843375,” “rs80230005,” “rs536327,” and “rs6679677” resulted in no significant association between the PLR and delirium, whereas the association became significant when these SNPs were included (Supplementary Figure S3). The association between the LMR and delirium was not influenced by any single SNP (Supplementary Figure S4).

4 Discussion

In this study, we investigated the association between the NLR, PLR, and LMR and the occurrence of delirium in IS patients admitted to the ICU. We found that 214 (14.9%) of the 1,436 IS patients developed delirium. High levels of the NLR and PLR were associated with an increased risk of delirium during hospitalization, and low levels of the LMR were associated with an increased risk of delirium during hospitalization. This association persisted after adjusting for age, sex, ethnicity, comorbidities, and poor lifestyle factors. However, after further adjustment for other study outcomes, vital signs, and disease severity scores, the association between the high NLR and PLR values and the risk of delirium disappeared. In contrast, the increased risk of delirium associated with the low LMR values remained. Furthermore, in the MR analysis, we found no causal relationship between either the NLR or LMR and the occurrence of delirium, while a significant relationship between the PLR and delirium persisted.

Delirium is an acute state of cerebral dysfunction characterized by sudden, fluctuating disturbances in consciousness and impaired attention (33–35). Increasing evidence suggests a strong association between delirium and adverse patient outcomes. Patients with delirium face a 10-fold increased risk of death, a 3–5-fold increased risk of complications, longer hospital stays, and a greater need for post-discharge care and support measures (33). In addition, patients with delirium are more likely to experience poor functional and cognitive recovery. The pathophysiological mechanisms underlying delirium are not fully understood (36). Currently, there are no effective medications or interventions for the treatment of delirium. Therefore, early identification of patients at risk of delirium is important (36).

Numerous studies have examined the association between inflammatory cytokines and delirium, with some showing that levels of inflammatory mediators such as plasma interleukin (IL)-6, C-reactive protein (CRP), IL-8, IL-10, and tumor necrosis factor (TNF)-α are significantly elevated in patients with delirium (14). However, the relationship between the NLR, PLR, and LMR and delirium in IS patients remains underexplored.

Compared to cytokines and acute-phase reactants, neutrophils, lymphocytes, and platelet counts are routinely measured as part of the hospital admission laboratory test for the majority of patients, with low acquisition costs and minimal requirements (37). In addition, these blood cell ratios reflect the dynamic balance between pro-inflammatory and anti-inflammatory immune responses, which are believed to contribute directly to the pathophysiology of delirium via neuroinflammatory mechanisms. As a result, they may offer greater clinical utility in predicting delirium after IS, facilitating early intervention to reduce its incidence.

In various inflammatory conditions, the NLR has been positively correlated with IL-6 levels, which supports its potential as a reliable marker of systemic inflammation (38–40). Neutrophils, as key pro-inflammatory immune cells, are elevated in response to acute inflammatory processes, indicating an intense inflammatory response. Conversely, lymphocytes typically play a suppressive role in inflammation, and their reduced numbers may indicate immune suppression. Thus, an elevated NLR value is generally associated with increased inflammation or immune system dysregulation. Similarly, the PLR has been associated with elevated levels of TNF-α in patients experiencing acute inflammation, underscoring its role as an inflammatory marker (41, 42). In addition to their traditional role in coagulation, platelets also contribute to immune responses by promoting pro-inflammatory pathways. The elevation of platelets and the corresponding decrease in lymphocytes observed in an increased PLR may signal a systemic inflammatory response, further associating this ratio with acute or chronic inflammatory conditions. Monocytes, which play a critical role in immune modulation, particularly in chronic inflammation, contribute to the regulation of immune responses. A low LMR value generally reflects a state of immune suppression, while a higher LMR value suggests a more balanced immune environment or a dominant anti-inflammatory response. This suggests that the LMR could offer insights into immune balance in IS patients, potentially providing prognostic value in predicting delirium risk.

These findings indicate that the NLR, PLR, and LMR may reflect similar inflammatory processes as traditional markers, reinforcing their clinical utility in predicting delirium after IS. Although IL-6 and TNF-α are well-established markers of inflammation, the NLR, PLR, and LMR offer a more accessible and cost-effective alternative for the assessment of inflammatory status in acute conditions such as IS (43). Moreover, the NLR, PLR, and LMR have demonstrated strong prognostic value in a variety of diseases, including IS (44), cardiovascular events (45), and cancer (46).

The pathophysiological mechanisms underlying delirium remain unclear. One hypothesis is that neuroinflammation-induced metabolic dysregulation contributes to the occurrence of delirium. Infection, surgery, or trauma can trigger acute peripheral inflammation in the body, leading to increased adherence and activation of leukocytes on the endothelial cells of cerebral blood vessels. This results in the significant release of free radicals and enzymes, which, in turn, disrupt the integrity of the endothelial cell membrane and intercellular adhesion, increase the permeability of the blood–brain barrier, cause perivascular edema, reduce perfusion, and increase the oxygen diffusion distance, ultimately leading to neuronal hypoxia and the manifestation of delirium (47–49). In addition, the elderly or those with impaired cognitive function are more susceptible to delirium (50). This increased susceptibility may be due to age-related changes in neurotransmitters, decreased cerebral blood flow, neuronal loss, and reduced intracellular signaling capacity. When stimulated by factors such as peripheral inflammation, cognitive decline is further exacerbated, leading to the manifestation of delirium.

Our findings highlight the relevance of systemic inflammatory biomarkers—NLR, PLR, and LMR—in the context of delirium. In the occurrence and prognosis of delirium, variations in these indicators reflect the body’s inflammatory response, immune dysfunction, cytokine release, and other related processes.

In addition to the overall analysis, we performed subgroup analyses based on sex and comorbidities, which provided further insights into the relationship between the inflammatory markers and delirium. Our subgroup analysis suggested that sex may influence the association between the inflammatory markers and delirium. Specifically, the positive correlation between the NLR and delirium was more pronounced in the female patients, while the LMR appeared to offer a protective effect in this group. These findings imply that sex may modulate the inflammatory response and its impact on delirium. One possible explanation for these sex-related differences is the role of estrogen in modulating immune responses, including leukocyte proliferation and antibody production, which may result in a stronger immune response in female individuals, thereby influencing the relationship between inflammation and delirium (51).

Moreover, our subgroup analysis highlighted the role of comorbidities (such as hypertension, congestive heart failure, and chronic pulmonary disease) in modulating the relationship between the inflammatory markers and delirium. In the hypertensive patients, we observed a significant positive correlation between the NLR and delirium, along with a negative correlation between the LMR and delirium. Hypertension is a well-established risk factor for IS, exerting chronic effects on vascular health and the blood–brain barrier integrity through increased shear stress, endothelial dysfunction, and large artery stiffness (52, 53). In hypertensive individuals, the increase in systemic inflammation may further exacerbate endothelial dysfunction, leading to an elevated risk of delirium. In this context, the hypertensive state may amplify the inflammatory response, thereby strengthening the association between the inflammatory markers and delirium in this subgroup.

The present study has several strengths. First, the cohort size was larger than in previous studies. Second, we evaluated the overall systemic inflammation of the patients using the NLR, PLR, and LMR as inflammatory markers, which are simple and easily obtainable methods. Third, we adjusted for more confounders than previous studies, including laboratory indicators and vital signs, to determine the association between the NLR, PLR, and LMR and delirium. Finally, we performed an MR analysis to investigate the causal relationship between these markers and delirium at the genetic level.

However, there are also some limitations to this study. First, the definition of delirium in the FinnGen database (ID: F5_DELIRIUM) encompasses various etiologies and is not limited to IS-related delirium. This phenotypic heterogeneity could have potentially diluted the specificity of our findings in the MR analysis. Second, IS patients often present with acute and severe neurological impairments, which may hinder the recognition of delirium, potentially leading to an underestimation of its true prevalence. Third, the NLR, PLR, and LMR data used in the MR analysis were derived from secondary GWAS analyses, which might have introduced errors or biases. The inconsistency between our observational study and the MR study suggests that the NLR, PLR, and LMR remain valuable predictors of post-IS delirium and require further rigorous exploration, such as prospective studies.

5 Conclusion

In conclusion, our observational study showed that the NLR, PLR, and LMR were significantly associated with the occurrence of delirium and that the inclusion of the NLR, PLR, and LMR in the delirium prediction model may improve its predictive accuracy. However, further MR analysis revealed a causal relationship only between the PLR and the risk of delirium. Therefore, well-designed prospective cohort studies are warranted to validate the associations between the NLR, PLR, and LMR and delirium, as well as to explore the underlying mechanisms.

Data availability statement

The original contributions presented in this study are included in the article and its Supplementary Material. Further inquiries regarding the data can be directed to the corresponding authors.

Ethics statement

This study used publicly available data. No new data were collected, and we did not require additional ethical approval.

Author contributions

PW: Formal analysis, Investigation, Writing – original draft. JH: Data curation, Methodology, Writing – original draft. LX: Methodology, Project administration, Validation, Visualization, Writing – review & editing. RH: Funding acquisition, Project administration, Resources, Supervision, Writing – review & editing.

Funding

The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This study was supported by the Joint project of the Chongqing Health Commission and Science and Technology Bureau (2023GGXM003) and the Program for Innovation Research Group of Universities in Chongqing (CXQT19012).

Acknowledgments

We gratefully acknowledge all the authors who participated in the GWAS study and the MIMIC-IV cohort.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2024.1456742/full#supplementary-material

Supplementary Figure S1 | Funnel plots of the MR analyses for the association of NLR, PLR, and LMR and delirium. (A) Causal effect of NLR on delirium; (B) Causal effect of PLR on delirium; (C) Causal effect of LMR on delirium. SNP, single nucleotide polymorphisms; MR, Mendelian randomization.

Supplementary Figure S2 | Leave-one-out plots of the MR analyses (IVW model) for the association of NLR and delirium.

Supplementary Figure S3 | Leave-one-out plots of the MR analyses (IVW model) for the association of PLR and delirium.

Supplementary Figure S4 | Leave-one-out plots of the MR analyses (IVW model) for the association of LMR and delirium.

References

1. Miao, R, Li, J, Meng, C, Li, Y, Tang, H, Wang, J, et al. Diet-derived circulating antioxidants and risk of stroke: a Mendelian randomization study. Oxidative Med Cell Longev. (2022) 2022:1–9. doi: 10.1155/2022/6457318

PubMed Abstract | Crossref Full Text | Google Scholar

2. O'Donnell, MJ, Chin, SL, Rangarajan, S, Xavier, D, Liu, L, Zhang, H, et al. Global and regional effects of potentially modifiable risk factors associated with acute stroke in 32 countries (INTERSTROKE): a case-control study. Lancet. (2016) 388:761–75. doi: 10.1016/S0140-6736(16)30506-2

PubMed Abstract | Crossref Full Text | Google Scholar

3. Song, J, Chen, W, and Ye, W. Stroke and the risk of gastrointestinal disorders: a Mendelian randomization study. Front Neurol. (2023) 14:1131250. doi: 10.3389/fneur.2023.1131250

PubMed Abstract | Crossref Full Text | Google Scholar

4. Pandian, JD, Gall, SL, Kate, MP, Silva, GS, Akinyemi, RO, Ovbiagele, BI, et al. Prevention of stroke: a global perspective. Lancet. (2018) 392:1269–78. doi: 10.1016/S0140-6736(18)31269-8

Crossref Full Text | Google Scholar

5. Hoshino, T, Sissani, L, Labreuche, J, Ducrocq, G, Lavallée, PC, Meseguer, E, et al. Prevalence of systemic atherosclerosis burdens and overlapping stroke etiologies and their associations with long-term vascular prognosis in stroke with intracranial atherosclerotic disease. JAMA Neurol. (2018) 75:203–11. doi: 10.1001/jamaneurol.2017.3960

PubMed Abstract | Crossref Full Text | Google Scholar

6. Ornello, R, Degan, D, Tiseo, C, Di Carmine, C, Perciballi, L, Pistoia, F, et al. Distribution and temporal trends from 1993 to 2015 of ischemic stroke subtypes: a systematic review and Meta-analysis. Stroke. (2018) 49:814–9. doi: 10.1161/STROKEAHA.117.020031

PubMed Abstract | Crossref Full Text | Google Scholar

7. Meng, C, Deng, P, Miao, R, Tang, H, Li, Y, Wang, J, et al. Gut microbiome and risk of ischaemic stroke: a comprehensive Mendelian randomization study. Eur J Prev Cardiol. (2023) 30:613–20. doi: 10.1093/eurjpc/zwad052

PubMed Abstract | Crossref Full Text | Google Scholar

8. Hacke, W, Kaste, M, Bluhmki, E, Brozman, M, Dávalos, A, Guidetti, D, et al. Thrombolysis with alteplase 3 to 4.5 hours after acute ischemic stroke. N Engl J Med. (2008) 359:1317–29. doi: 10.1056/NEJMoa0804656

PubMed Abstract | Crossref Full Text | Google Scholar

9. Wilson, JE, Mart, MF, Cunningham, C, Shehabi, Y, Girard, TD, MacLullich, AMJ, et al. Delirium. Nat Rev Dis Primers. (2020) 6:90. doi: 10.1038/s41572-020-00223-4

PubMed Abstract | Crossref Full Text | Google Scholar

10. Li, J, Wang, J, Yang, M, Wang, G, and Xu, P. The relationship between major depression and delirium: a two-sample Mendelian randomization analysis. J Affect Disord. (2023) 338:69–73. doi: 10.1016/j.jad.2023.05.046

PubMed Abstract | Crossref Full Text | Google Scholar

11. Qu, J, Chen, Y, Luo, G, Zhong, H, Xiao, W, and Yin, H. Delirium in the acute phase of ischemic stroke: incidence, risk factors, and effects on functional outcome. J Stroke Cerebrovasc Dis. (2018) 27:2641–7. doi: 10.1016/j.jstrokecerebrovasdis.2018.05.034

PubMed Abstract | Crossref Full Text | Google Scholar

12. Pasinska, P, Kowalska, K, Klimiec, E, Szyper-Maciejowska, A, Wilk, A, and Klimkowicz-Mrowiec, A. Frequency and predictors of post-stroke delirium in PRospective observational POLIsh study (PROPOLIS). J Neurol. (2018) 265:863–70. doi: 10.1007/s00415-018-8782-2

PubMed Abstract | Crossref Full Text | Google Scholar

13. Kotfis, K, Bott-Olejnik, M, Szylińska, A, Listewnik, M, and Rotter, I. Characteristics, risk factors and outcome of early-onset delirium in elderly patients with first ever acute ischemic stroke - a prospective observational cohort study. Clin Interv Aging. (2019) 14:1771–82. doi: 10.2147/CIA.S227755

PubMed Abstract | Crossref Full Text | Google Scholar

14. Guldolf, K, Vandervorst, F, Gens, R, Ourtani, A, Scheinok, T, and De Raedt, S. Neutrophil-to-lymphocyte ratio predicts delirium after stroke. Age Ageing. (2021) 50:1626–32. doi: 10.1093/ageing/afab133

PubMed Abstract | Crossref Full Text | Google Scholar

15. Zhang, M, Zhang, X, Gao, L, Yue, J, and Jiang, X. Incidence, predictors and health outcomes of delirium in very old hospitalized patients: a prospective cohort study. BMC Geriatr. (2022) 22:262. doi: 10.1186/s12877-022-02932-9

PubMed Abstract | Crossref Full Text | Google Scholar

16. Tieges, Z, Quinn, T, MacKenzie, L, Davis, D, Muniz-Terrera, G, MacLullich, AMJ, et al. Association between components of the delirium syndrome and outcomes in hospitalised adults: a systematic review and meta-analysis. BMC Geriatr. (2021) 21:162. doi: 10.1186/s12877-021-02095-z

PubMed Abstract | Crossref Full Text | Google Scholar

17. Goldberg, TE, Chen, C, Wang, Y, Jung, E, Swanson, A, Ing, C, et al. Association of delirium with long-term cognitive decline: a meta-analysis. JAMA Neurol. (2020) 77:1373–81. doi: 10.1001/jamaneurol.2020.2273

PubMed Abstract | Crossref Full Text | Google Scholar

18. Yu, M, Li, Y, Li, B, and Ge, Q. Inflammatory biomarkers and delirium: a Mendelian randomization study. Front Aging Neurosci. (2023) 15:1221272. doi: 10.3389/fnagi.2023.1221272

PubMed Abstract | Crossref Full Text | Google Scholar

19. Silverstein, JH, and Deiner, SG. Perioperative delirium and its relationship to dementia. Prog Neuro-Psychopharmacol Biol Psychiatry. (2013) 43:108–15. doi: 10.1016/j.pnpbp.2012.11.005

PubMed Abstract | Crossref Full Text | Google Scholar

20. Klimiec, E, Dziedzic, T, Kowalska, K, Slowik, A, and Klimkowicz-Mrowiec, A. Knowns and unknowns about delirium in stroke: a review. Cogn Behav Neurol. (2016) 29:174–89. doi: 10.1097/WNN.0000000000000110

PubMed Abstract | Crossref Full Text | Google Scholar

21. Maldonado, JR. Delirium pathophysiology: an updated hypothesis of the etiology of acute brain failure. Int J Geriatr Psychiatry. (2018) 33:1428–57. doi: 10.1002/gps.4823

PubMed Abstract | Crossref Full Text | Google Scholar

22. Kim, S, Eliot, M, Koestler, DC, Wu, WC, and Kelsey, KT. Association of neutrophil-to-lymphocyte ratio with mortality and cardiovascular disease in the Jackson heart study and modification by the Duffy antigen variant. JAMA Cardiol. (2018) 3:455–62. doi: 10.1001/jamacardio.2018.1042

PubMed Abstract | Crossref Full Text | Google Scholar

23. Jung, SY, Papp, JC, Sobel, EM, and Zhang, ZF. Mendelian randomization study: the association between metabolic pathways and colorectal cancer risk. Front Oncol. (2020) 10:1005. doi: 10.3389/fonc.2020.01005

PubMed Abstract | Crossref Full Text | Google Scholar

24. Luo, M, Cai, J, Luo, S, Hong, X, Xu, L, Lin, H, et al. Causal effects of gut microbiota on the risk of chronic kidney disease: a Mendelian randomization study. Front Cell Infect Microbiol. (2023) 13:1142140. doi: 10.3389/fcimb.2023.1142140

PubMed Abstract | Crossref Full Text | Google Scholar

25. Carter, P, Yuan, S, Kar, S, Vithayathil, M, Mason, AM, Burgess, S, et al. Coffee consumption and cancer risk: a Mendelian randomisation study. Clin Nutr. (2022) 41:2113–23. doi: 10.1016/j.clnu.2022.08.019

PubMed Abstract | Crossref Full Text | Google Scholar

26. Zhou, G, Ren, X, Tang, Z, Li, W, Chen, W, He, Y, et al. Exploring the association and causal effect between white blood cells and psoriasis using large-scale population data. Front Immunol. (2023) 14:1043380. doi: 10.3389/fimmu.2023.1043380

PubMed Abstract | Crossref Full Text | Google Scholar

27. Okbay, A, Wu, Y, Wang, N, Jayashankar, H, and Bennett, M. Polygenic prediction of educational attainment within and between families from genome-wide association analyses in 3 million individuals. Nat Genet. (2022) 54:437–49. doi: 10.1038/s41588-022-01016-z

PubMed Abstract | Crossref Full Text | Google Scholar

28. Yu, H, Wan, X, Yang, M, Xie, J, Xu, K, Wang, J, et al. A large-scale causal analysis of gut microbiota and delirium: a Mendelian randomization study. J Affect Disord. (2023) 329:64–71. doi: 10.1016/j.jad.2023.02.078

PubMed Abstract | Crossref Full Text | Google Scholar

29. Qian, X, Sheng, Y, Jiang, Y, and Xu, Y. Associations of serum lactate and lactate clearance with delirium in the early stage of ICU: a retrospective cohort study of the MIMIC-IV database. Front Neurol. (2024) 15:1371827. doi: 10.3389/fneur.2024.1371827

PubMed Abstract | Crossref Full Text | Google Scholar

30. Ely, EW, Inouye, SK, Bernard, GR, Gordon, S, Francis, J, May, L, et al. Delirium in mechanically ventilated patients: validity and reliability of the confusion assessment method for the intensive care unit (CAM-ICU). JAMA. (2001) 286:2703–10. doi: 10.1001/jama.286.21.2703

Crossref Full Text | Google Scholar

31. Qian, Y, Ye, D, Huang, H, Wu, DJH, Zhuang, Y, Jiang, X, et al. Coffee consumption and risk of stroke: a Mendelian randomization study. Ann Neurol. (2020) 87:525–32. doi: 10.1002/ana.25693

PubMed Abstract | Crossref Full Text | Google Scholar

32. Wang, M, Jian, Z, Yuan, C, Jin, X, Li, H, and Wang, K. Coffee consumption and prostate Cancer risk: Results from National Health and nutrition examination survey 1999-2010 and Mendelian randomization analyses. Nutrients. (2021) 13:2317. doi: 10.3390/nu13072317

Crossref Full Text | Google Scholar

33. Marcantonio, ER. Delirium in Hospitalized Older Adults. N Engl J Med. (2017) 377:1456–66. doi: 10.1056/NEJMcp1605501

PubMed Abstract | Crossref Full Text | Google Scholar

34. Maldonado, JR. Acute brain failure: pathophysiology, diagnosis, management, and sequelae of delirium. Crit Care Clin. (2017) 33:461–519. doi: 10.1016/j.ccc.2017.03.013

PubMed Abstract | Crossref Full Text | Google Scholar

35. The American Geriatrics Society. National Institute on Aging bedside-to-bench conference: research agenda on delirium in older adults. J Am Geriatr Soc. (2015) 63:843–52. doi: 10.1111/jgs.13406

Crossref Full Text | Google Scholar

36. Mattison, MLP. Delirium. Ann Intern Med. (2020) 173:ITC49–64. doi: 10.7326/AITC202010060

PubMed Abstract | Crossref Full Text | Google Scholar

37. Fernández-Jiménez, E, Muñoz-Sanjose, A, Mediavilla, R, Martínez-Alés, G, Louzao, II, Andreo, J, et al. Prospective analysis between neutrophil-to-lymphocyte ratio on admission and development of delirium among older hospitalized patients with COVID-19. Front Aging Neurosci. (2021) 13:764334. doi: 10.3389/fnagi.2021.764334

PubMed Abstract | Crossref Full Text | Google Scholar

38. Hao, Y, Qi, Z, Ding, Y, Yu, X, Pang, L, and Zhao, T. Effect of interventional therapy on IL-1β, IL-6, and neutrophil-lymphocyte ratio (NLR) levels and outcomes in patients with ischemic cerebrovascular disease. Med Sci Monit. (2019) 25:610–7. doi: 10.12659/MSM.912064

PubMed Abstract | Crossref Full Text | Google Scholar

39. Yuan, C, Pan, Y, and Ning, Y. Predictive value of IL-6 combined with NLR in inflammation and Cancer. Cancer Investig. (2021) 39:489–504. doi: 10.1080/07357907.2020.1858852

PubMed Abstract | Crossref Full Text | Google Scholar

40. Făgărășan, I, Rusu, A, Comșa, H, Simu, TD, Vulturar, DM, and Todea, DA. IL-6 and neutrophil/lymphocyte ratio as markers of ICU admittance in SARS-CoV-2 patients with diabetes. Int J Mol Sci. (2023) 24:4908. doi: 10.3390/ijms241914908

PubMed Abstract | Crossref Full Text | Google Scholar

41. Park, JW, Chang, HJ, Yeo, HY, Han, N, Kim, BC, Kong, SY, et al. The relationships between systemic cytokine profiles and inflammatory markers in colorectal cancer and the prognostic significance of these parameters. Br J Cancer. (2020) 123:610–8. doi: 10.1038/s41416-020-0924-5

PubMed Abstract | Crossref Full Text | Google Scholar

42. Wang, C, Wu, R, Yao, D, Yu, Z, and Shen, X. Comparison of dietary inflammatory index and inflammatory biomarkers between vegetarians and omnivores in Chinese population. Sci Rep. (2024) 14:19593. doi: 10.1038/s41598-024-69168-y

PubMed Abstract | Crossref Full Text | Google Scholar

43. Gong, P, Liu, Y, Gong, Y, Chen, G, Zhang, X, Wang, S, et al. The association of neutrophil to lymphocyte ratio, platelet to lymphocyte ratio, and lymphocyte to monocyte ratio with post-thrombolysis early neurological outcomes in patients with acute ischemic stroke. J Neuroinflammation. (2021) 18:51. doi: 10.1186/s12974-021-02090-6

PubMed Abstract | Crossref Full Text | Google Scholar

44. Zhang, Y, Xing, Z, Zhou, K, and Jiang, S. The predictive role of systemic inflammation response index (SIRI) in the prognosis of stroke patients. Clin Interv Aging. (2021) 16:1997–2007. doi: 10.2147/CIA.S339221

PubMed Abstract | Crossref Full Text | Google Scholar

45. Zhou, E, Wu, J, Zhou, X, and Yin, Y. The neutrophil-lymphocyte ratio predicts all-cause and cardiovascular mortality among U.S. adults with rheumatoid arthritis: results from NHANES 1999-2020. Front Immunol. (2023) 14:1309835. doi: 10.3389/fimmu.2023.1309835

PubMed Abstract | Crossref Full Text | Google Scholar

46. Yamamoto, T, Kawada, K, and Obama, K. Inflammation-related biomarkers for the prediction of prognosis in colorectal Cancer patients. Int J Mol Sci. (2021) 22:8002. doi: 10.3390/ijms22158002

PubMed Abstract | Crossref Full Text | Google Scholar

47. He, F, Yin, F, Peng, J, Deng, X, Wu, L, and Zhang, C. Molecular mechanism for change in permeability in brain microvascular endothelial cells induced by LPS. Zhong Nan Da Xue Xue Bao Yi Xue Ban. (2010) 35:1129–37. doi: 10.3969/j.issn.1672-7347.2010.11.003

PubMed Abstract | Crossref Full Text | Google Scholar

48. Bogatcheva, NV, Sergeeva, MG, Dudek, SM, and Verin, AD. Arachidonic acid cascade in endothelial pathobiology. Microvasc Res. (2005) 69:107–27. doi: 10.1016/j.mvr.2005.01.007

PubMed Abstract | Crossref Full Text | Google Scholar

49. Maldonado, JR. Neuropathogenesis of delirium: review of current etiologic theories and common pathways. Am J Geriatr Psychiatry. (2013) 21:1190–222. doi: 10.1016/j.jagp.2013.09.005

PubMed Abstract | Crossref Full Text | Google Scholar

50. Maldonado, JR. Delirium in the acute care setting: characteristics, diagnosis and treatment. Crit Care Clin. (2008) 24:657–722, vii. doi: 10.1016/j.ccc.2008.05.008

PubMed Abstract | Crossref Full Text | Google Scholar

51. Chakraborty, B, Byemerwa, J, Krebs, T, Lim, F, Chang, CY, and McDonnell, DP. Estrogen receptor signaling in the immune system. Endocr Rev. (2023) 44:117–41. doi: 10.1210/endrev/bnac017

PubMed Abstract | Crossref Full Text | Google Scholar

52. Cipolla, MJ, Liebeskind, DS, and Chan, SL. The importance of comorbidities in ischemic stroke: impact of hypertension on the cerebral circulation. J Cereb Blood Flow Metab. (2018) 38:2129–49. doi: 10.1177/0271678X18800589

PubMed Abstract | Crossref Full Text | Google Scholar

53. Gorgui, J, Gorshkov, M, Khan, N, and Daskalopoulou, SS. Hypertension as a risk factor for ischemic stroke in women. Can J Cardiol. (2014) 30:774–82. doi: 10.1016/j.cjca.2014.01.007

PubMed Abstract | Crossref Full Text | Google Scholar