Post-COVID-19 cognitive symptoms in patients assisted by a teleassistance service: a retrospective cohort study

Introduction: Four years after the onset of the COVID-19 pandemic, the frequency of long-term post-COVID-19 cognitive symptoms is a matter of concern given the impact it may have on the work and quality of life of affected people.

Objective: To evaluate the incidence of post-acute COVID-19 cognitive symptoms, as well as the associated risk factors.

Methods: Retrospective cohort, including outpatients with laboratory-confirmed COVID-19 and who were assisted by a public telehealth service provided by the Telehealth Network of Minas Gerais (TNMG), during the acute phase of the disease, between December/2020 and March/2022. Data were collected through a structured questionnaire, applied via phone calls, regarding the persistence of COVID-19 symptoms after 12 weeks of the disease. Cognitive symptoms were defined as any of the following: memory loss, problems concentrating, word finding difficulties, and difficulty thinking clearly.

Results: From 630 patients who responded to the questionnaire, 23.7% presented cognitive symptoms at 12 weeks after infection. These patients had a higher median age (33 [IQR 25–46] vs. 30 [IQR 24–42] years-old, p = 0.042) with a higher prevalence in the female sex (80.5% vs. 62.2%, p < 0.001) when compared to those who did not present cognitive symptoms, as well as a lower prevalence of smoking (8.7% vs. 16.2%, p = 0.024). Furthermore, patients with persistent cognitive symptoms were more likely to have been infected during the second wave of COVID-19 rather than the third (31.0% vs. 21.3%, p = 0.014). Patients who needed to seek in-person care during the acute phase of the disease were more likely to report post-acute cognitive symptoms (21.5% vs. 9.3%, p < 0,001). In multivariate logistic regression analysis, cognitive symptoms were associated with female sex (OR 2.24, CI 95% 1.41–3.57), fatigue (OR 2.33, CI 95% 1.19–4.56), depression (OR 5.37, CI 95% 2.19–13.15) and the need for seek in-person care during acute COVID-19 (OR 2.23, CI 95% 1.30–3.81).

Conclusion: In this retrospective cohort of patients with mostly mild COVID-19, cognitive symptoms were present in 23.7% of patients with COVID-19 at 12 weeks after infection. Female sex, fatigue, depression and the need to seek in-person care during acute COVID-19 were the risk factors independently associated with this condition.

Introduction

With the emergence of the coronavirus disease 2019 (COVID-19), the world has been forced to face a health crisis that has lasted for over 4 years. Although current case numbers have drastically reduced, COVID-19 is still a global health issue, with a high social burden, especially in developing countries (1). In this context, the COVID-19 pandemic represented a great stimulus for the development of telehealth services worldwide, as the pandemic necessitated innovative solutions to provide quality health care while preserving protective social distancing measures – in place at the time – for the safety and of both health care providers and the general population.

Currently, increasing attention is directed towards the burden associated with symptoms that persist beyond the acute phase of the infection (2). According to the National Institute for Health and Care Excellence (NICE), post-COVID-19 syndrome refers to signs and symptoms that develop during or after an infection consistent with COVID-19, continue for at least 12 weeks and are not explained by an alternative diagnosis (3). Post-COVID-19 syndrome may occur regardless of acute COVID-19 severity, although those who present severe COVID-19 or have several comorbidities are more prone to develop the syndrome (1, 4).

The symptoms of post-COVID-19 syndrome are heterogeneous, and may include, but are not limited to, fatigue, difficulty breathing, “brain fog,” insomnia, joint pain, and cardiac issues (5). “Brain fog” is an umbrella term for the presence of cognitive symptoms, including mental fatigue, impaired concentration and memory that may impact daily activities. Fatigue and cognitive impairment, including “brain fog,” are among the most common and debilitating long-term effects of COVID-19 (6). An Israeli cohort examined the long-term clinical outcomes in over 1.9 million people with mild COVID-19, and observed that unvaccinated SARS-CoV-2 infected patients had a higher risk for concentration and memory impairment hazard ratio 1.85, 95% confidence interval [CI] 1.58 to 2.17 – between 1 and 6 months after acute infection; and hazard ratio 12.8, 95% confidence interval 9.6 to 16.1 – 6 months to a year after COVID-19 diagnosis, when compared to uninfected people (7). Other neurological and neuropsychiatric symptoms such as anxiety and depression are also common components of the post-COVID-19 syndrome (8).

The increase in the incidence of cognitive symptoms is a matter of concern given the impact they may have on the work and quality of life of the affected patients. However, little is known about its determinants, especially in patients who have had mild cases of SARS-CoV-2 infection (9). Therefore, this study aimed to evaluate the incidence of cognitive symptoms in post-COVID syndrome, as well as the associated risk factors, in COVID-19 outpatients assisted by a Brazilian public telehealth service. The study is innovative for including patients attended via a telehealth service during acute COVID-19.

Methods

Study design and eligibility

This is a retrospective cohort, which included staff and students from a public university, the Universidade Federal de Minas Gerais (UFMG), located in Belo Horizonte, the capital of Minas Gerais state, in Southeast Brazil. The cohort consisted of a convenience sample of consecutive individuals who had laboratory-confirmed COVID-19 (by real-time reverse transcription-polymerase chain reaction assay [RT-PCR] or antigen testing), between December 01, 2020 and March 31, 2022. All patients had been followed during the acute phase of the disease by TeleCOVID-MG, a public telehealth service provided by the Telehealth Network of Minas Gerais (TNMG) (10). The study period comprehended patients who had acute COVID-19 during the second or third pandemic waves in Brazil. According to previous evidence, the second wave lasted from August 11, 2020, to December 25, 2021, with delta and gamma variants as dominant; and the third wave lasted from December 26, 2021, to May 5, 2022, marked predominantly by the omicron variant.

TeleCOVID-MG

TeleCOVID-MG was a public structured multilevel teleconsultation and telemonitoring program, developed by the TNMG, to assist patients with respiratory tract symptoms during the COVID-19 pandemic. This service was maintained in operation between May 2020 and March 2023. The TNMG represents a partnership between seven public Brazilian Universities, with a coordinating hub at the Telehealth Center at the University Hospital/UFMG (10). It is one of the largest telehealth services in Brazil and Latin America. TeleCOVID-MG was first implemented in two Brazilian medium-sized cities and then, in December 2020, it was expanded to assist students, faculty, and technical-administrative staff from UFMG, as well as healthcare professionals from UFMG’s University Hospital. Universidade Federal de Minas Gerais is a public federal university with more than 40,000 students (undergraduate and postgraduate), and the staff has more than 10,000 people, including professors and administrative staff.

For the UFMG students and staff, access to the TeleCOVID-MG service was done through an online symptom auto-verification application developed by the pandemic committee of the university. Upon identification of any flu-like symptoms, patients were referred to a chatbot, a computer program that collected name, Brazilian identification number, telephone number, warning signs or any comorbidities which increased the risk of worse outcomes (11). In cases of suspicion of flu-like syndrome and according to the severity of the symptoms, the patient was assisted by a nurse or a physician through a phone call teleconsultation. At the end of the teleconsultation, the patient was advised to keep domiciliary isolation or to seek an onsite evaluation at the primary care center or at the emergency department, in case of warning signs such as fever for more than 3 days, signs of hemodynamic instability, decompensation of the underlying disease or any other critical clinical condition identified by the health professional. In addition, patients received a request for an RT-PCR laboratory test to identify SARS-CoV-2, which could be performed at the university itself or at the reference laboratory of the patient’s preference. Positive antigen tests performed in duly accredited services were also accepted as laboratory confirmation for COVID-19 (12).

Data collection

For the present study, data was collected through two main steps. In the first step, we obtained information on the COVID-19 acute phase, while in the second we assessed post-acute COVID-19 symptoms. Data regarding the acute phase of COVID-19 was obtained from the TeleCOVID-MG database including: age, sex, COVID-19 acute symptoms, the date on which the laboratory exam was performed, comorbidities, and if the patient was vaccinated for COVID-19 before the laboratory confirmation. With regards to the post-acute disease stage, data was collected at least 6 months after the diagnostic laboratory test, through the application of a structured questionnaire, developed exclusively for this study. The questionnaire was based on clinical protocols for the management of post-COVID syndrome, by the Brazilian Ministry of Health and the health department of Belo Horizonte (13, 14), as well as on previously validated tools, including Generalized Anxiety Disorder–7 (GAD-7) (15), Posttraumatic Stress Disorder Checklist for DSM-5 (PCL-5) (16), Patient Health Questionnaire-9 (PHQ-9) (17), Chalder’s Fatigue Scale (18, 19), New York Health Association’s functional scale (20), and Charlson’s comorbidity index (21).

The questionnaire was developed using Google Forms^® and was composed of 82 questions, divided into twelve sections: researcher identification, demographic characteristics, respiratory manifestations, neuromusculoskeletal disorders, physical fatigue, mental fatigue, neuro-cognitive manifestations, other manifestations, comorbidities, life habits (smoking and physical activity), work impact and post-COVID-19 functioning (Supplementary file 1). The questionnaire investigated symptoms at different time points after the COVID-19 acute phase (up to 1 month, up to 3 months, up to 6 months, more than 6 months). For the present analysis, the occurrence of cognitive symptoms for at least 12 weeks was evaluated.

The constructs used to assess cognitive functions, namely memory problems, concentration problems, difficulties for thinking clearly and word finding difficulties, were obtained from the Chalder Fatigue Scale (18, 19). This is a previously validated instrument widely applied to measure physical and mental fatigue in patients with chronic fatigue syndrome (18, 19). Emerging literature has linked post-COVID cognitive symptoms to myalgic encephalomyelitis/chronic fatigue syndrome (ME/CFS) in which the Chalder Fatigue Scale is frequently used (22). The question “Did you have difficulties in thinking clearly?” was adapted from the Chalder Fatigue Scale’s question: “Do you think as clearly as usual?.” Similarly, the question “Did you have word finding difficulties?” was adapted from the Chalder Fatigue Scale: “Do you find it more difficult to find the correct word?.” The question “Did you present memory loss?” was adapted from “Is your memory as good as usual?” and “Did you present alterations in concentration?” from “Do you have difficulty concentrating?” [(18), Supplementary file 1].

In the present study, the presence of cognitive symptoms in post-COVID-19 syndrome was considered if the patient had a positive response to at least one of the four questions related to the occurrence of memory problems, concentration problems, difficulties for thinking clearly and word finding difficulties. Other cognitive studies on post COVID have used a similar approach (23–25).

Regarding the presence of cognitive symptoms, patients were divided into two groups: those who presented at least one of the four cognitive symptoms (memory problems, concentration problems, word finding difficulties and difficulty in thinking clearly) and those who had no cognitive symptoms lasting at least 12 weeks from the onset of COVID-19 symptoms.

To evaluate neuropsychiatric manifestations, questions based on previously validated instruments (GAD-7 for anxiety, PCL-5 for post-traumatic disorder and PHQ-9 for depression) were included in the questionnaire (15–17).The PHQ-2, which is an abbreviated form of the PHQ-9, was used as a criterion for the occurrence of depression. In other words, the positive answer to the two PHQ-2 questions defined the occurrence of depression in the present study (questions 49 and 54 of Supplementary file 1).

Patients who practiced physical activities regularly in accordance with World Health Organization recommendations (at least 150 min of physical exercise at moderate intensity or 75 at vigorous intensity, weekly) were considered non-sedentary (26).

In order to assess the possible impact caused by the post-COVID-19 syndrome, patients were asked about eventual loss of ability to carry out their daily tasks, the need to leave work longer than the expected period of isolation (for the acute phase of COVID-19), as well as the need for any restrictions after returning to work, such as reducing the workload or adapting the activity carried out. Finally, survey participants rated themselves on a post-COVID functional status scale (13, 27).

The questionnaire was applied through phone calls by a team of eight trained researchers who were supervised by a senior researcher. A data collection protocol (Supplementary file 2) was created in order to standardize the collection and all team members were previously trained on the study protocol.

The questionnaire was applied from December 2021 to November 2022. Each participant received a single phone call at least 6 months after laboratory confirmation of COVID-19. The protocol for patient inclusion in the study involved four contact attempts, including two phone calls, one in the morning and one in the afternoon, and two standardized text messages through an app (Whatsapp^®). This message consisted of a short text presenting the project, and the individual was inquired about the best time for the telephone call. If the participant initially did not understand any question, as per the study protocol the researcher should explain the question, according to the collection manual. As the questionnaires were filled out, the senior researcher audited the responses. Periodic audits were performed weekly, in order to increase quality data and to reduce biases. Incorrect data were reported to applicators and corrected. Whenever necessary, researchers underwent refresher training before applying new questionnaires.

During the study period, 11,585 patients were treated by TeleCOVID-MG. Of these, 1,575 had a positive laboratory test (RT-PCR or antigen test) for SARS-CoV-2. All patients with positive tests in this period would be able to participate in the research through the application of the questionnaire on post-COVID symptoms. However, in 888 cases contact was unsuccessful, of these 873 patients did not answer calls, and 15 phone numbers were wrong. Among 687 patients who answered the call, 13 refused to participate in the study, and 44 were excluded by missing data in all questions concerning cognitive symptoms. In the end, a total of 630 participants were included in this study (Figure 1).

Figure 1

Figure 1. Flowchart of patients included in the study.

Statistical analysis

Descriptive analysis of the variables regarding the presence or absence of cognitive symptoms, patient characteristics and impact on daily activities was performed. Participants were categorized into three age groups: 17–40, 41–60, and >60 years (7). The wave the patient was infected was defined based on the date of the laboratory examination. Exams taken between November 8, 2020 and December 25, 2021 corresponded to the second wave; while the tests carried out between December 26, 2021 and March 31, 2022 referred to the third wave of COVID-19 in Brazil (28).

For the purpose of the current analysis, the number of comorbidities was defined according to a modified Charlson comorbidity index including: chronic cardiac disease, chronic respiratory disease (excluding asthma), chronic renal disease, liver disease, dementia, chronic neurological conditions, connective tissue disease, diabetes mellitus, HIV and malignancy (21). Obesity was not included in the original modified Charlson comorbidity index, but we opted to include it due to its probable association with adverse outcomes in patients with COVID-19.

The statistical method to analyze the association between patient characteristics and the occurrence of cognitive symptoms (each symptom separately and also at least one of the four symptoms) was based on Bursac et al.’s proposal (29). The authors suggest starting the selection of variables through an univate analysis with a cutoff p-value of 0.25, but we opted to be more strict and 0.20 as the cutoff point.

The selected characteristics (possible predictors of cognitive symptoms) were then included in multivariate logistic regression models. Variables that were considered possible consequences of cognitive symptoms or other post-COVID-19 manifestations (such as loss of ability to perform daily tasks, absence from work longer than the usual period of isolation, restrictions on return to work and functional status post-COVID) were not tested in the multivariate models (Supplementary Table S1). A backward stepwise method was used to define significant characteristics, considering a cutoff point of 5%. The impact of significant characteristics was estimated using odds ratios (OR) and 95% confidence intervals. Deviance, Pearson, Hosmer-Lemeshow indicated well-adjusted final models.

As the presence of depression is linked to cognitive impairments, and a higher frequency of cognitive symptoms not always confirmed by the objective assessment (30), a subanalysis was conducted, excluding patients with depression.

Ethics statement

This study was approved by the Brazilian National Commission for Research Ethics (Comissão Nacional de Ética em Pesquisa on number CAAE 30350820.5.1001.0008), and it was conducted in accordance with the Declaration of Helsinki. All patients gave informed consent to participate in the study.

Results

Study population

Of the 630 post-COVID-19 patients included in the study, 149 (23.7%) reported cognitive symptoms at least 12 weeks after COVID-19 infection. The main characteristics of the study’s population stratified by the presence of cognitive symptoms are presented in Table 1 and Supplementary Table S2.

Table 1

Table 1. Baseline characteristics of the study cohort at 12 weeks after acute COVID-19 onset.

The median age and proportion of women were higher in the group of patients with cognitive symptoms when compared to those without cognitive symptoms (33 [interquartile range (IQR) 25–46] vs. 30 [IQR 24–42] years-old, p = 0.042; 80.5% vs. 62.2%, p < 0.001). With regards to race, education, being a healthcare worker or intern, vaccination status, pregnancy, and physical activity, there were no statistically significant differences between groups. As for the presence of comorbidities, the vast majority of the sample (85.7%) did not present any comorbidity, and, among those who reported at least one comorbidity, there was no difference between groups. Patients with cognitive symptoms had a higher frequency of symptoms related to depression (11.4% vs. 1.7, p < 0.001), lower frequency of smoking (8.7% vs. 16.0%, p = 0.026), higher frequency of infection during the second wave (32.2 vs. 22.2%, p = 0.014) and sought in-person care more frequently (21.5% vs. 9.1%, p < 0.001), when compared to those without cognitive symptoms. When assessing patients per COVID-19 wave (Supplementary Table S3), there was a higher frequency of cognitive symptoms in the second wave.

Patients with cognitive symptoms reported a higher frequency of starting treatment for psychiatric diseases (21.5% vs. 10.4%, p < 0.001), greater loss of ability to perform day-to-day tasks (16.8% vs. 1.2%, p < 0.001), needed to be absent from work activities for a longer period (7.4% vs. 1.7%, p < 0.001) and reported having more limitations than patients without cognitive symptoms (44.3% vs. 9.6% to no impairment, 20.1% vs. 1.7% to mild impairment, p < 0.001).

Predictive factors for cognitive symptoms

The most prevalent cognitive symptoms in the sample studied were memory loss (17.5%), followed by word finding difficulties (16.2%), concentration problems (15.9%), and difficulty in thinking clearly (9.5%), as shown in Table 2. Among affected patients (n = 149), 23.5% had all four symptoms, 24.8% had three symptoms, 29.5% had two symptoms, and 22.1% had only one symptom.

Table 2

Table 2. Incidence of cognitive symptoms in the study population.

In the multivariate analysis, a statistically significant association was observed between the occurrence of cognitive symptoms in post-COVID-19 syndrome and depression (OR 5.37 [95% IC 2.19–13.15]), as well as the presence of fatigue (OR 2.33 [95% CI 1.19–4.56]), female sex (OR 2.24 [95% CI 1.41–3.57]) and the need to seek in-person care in the acute phase of COVID-19 infection (OR 2.23 [95% CI 1.30–3.81]) (Table 3). In regards to each symptom, the same factors were associated (depression, fatigue, sex and need to seek in-person care), except for “thinking clearly,” which showed no significant association with the need to seek in-person care, but had a significant association with COVID-19 acute infection during the second wave (OR 1.92 [95% CI 1.08–3.41]).

Table 3

Table 3. Predictors of cognitive symptoms according to the multivariate analysis (n = 630).

In the subanalysis excluding patients with depressive symptoms (n = 25), results were similar to the previous model (Supplementary Table S4).

Discussion

The present study found that cognitive symptoms are a prominent feature of post-COVID-19 syndrome, with a prevalence of 23.7%. Female sex OR 2.27 (95% CI [1.41–3.57]), fatigue OR 2.33 (95% CI [1.19–4.56]), depression OR 5.37 (95% CI [2.19–13.15]) and the need to seek in-person care in the acute phase of COVID-19 infection OR 2.23 (IC 95% [1.30–3.81]) were associated with cognitive symptoms. To the best of our knowledge, this is the first study in Latin America to address post-COVID-19 cognitive symptoms.

From the total sample, 12.1% required to seek in-person care and only 0.5% required hospitalization, which confirms that overall this is a cohort of mild cases of COVID-19. Even though the cases were mostly mild and patients were young, post-COVID-19 cognitive symptoms were reported by almost a quarter of them. This number corroborates previous studies in which cognitive symptoms, especially memory impairment, were highly prevalent among patients who had COVID-19 (6, 8, 31, 32). A recent systematic review, including data from 10,530 patients (59% women, average age 52 years, 51% who were hospitalized and 3% were admitted to an intensive care unit), has shown that cognitive symptoms were present in roughly one-third of patients at 12 or more weeks after the onset of COVID-19: brain fog (32, 10–54%), memory issues (28, 22–35%), attention disorder (22, 7–36%) (8). Interestingly, the prevalence of cognitive symptoms did not change significantly between mid-term (3 to 6 months) and long-term follow-up (6 or more months post-infection, lower than 5% change).

Women were 2.24 (95% CI 1.41–3.57) times more likely to have at least one of the four cognitive symptoms than men, with greater chances of having concentration problems (OR 2.66 [95% CI 1.48–4.78]), memory loss (2.09 [95% CI 1.24–3.54]), and word finding difficulties (2.24 [95% CI 1.29–3.87]). These findings are in line with two large cohorts in Iran and Norway, which also observed female sex as a risk factor for post-COVID brain fog (OR 1.4 [95% CI 1.06–1.90] and RR 2.0 [95% IC 1.3–3.2], respectively) (33, 34), and a recent Polish study with 303 outpatients, 47% of them healthcare professionals, and a median age similar to the present study. In this Polish study, 12 weeks after acute COVID-19, women reported problems with writing, reading, counting (17.0 vs. 5.1%) and communication of thoughts in a way that others can understand (34.3 vs. 20.7%) more often than men (35).

Patients who experienced fatigue were 2.33 (95% CI 1.19–4.56) times more likely to have cognitive symptoms than those who did not experience it, with 3.23 (95% CI [1.26–8.31]) greater chance of having concentration problems; 3.87 (95% CI 1.52–9.89) greater chance of reporting memory problems, 2.35 (95% CI 1.04–5.31) greater chance of reporting word finding difficulties and 3.40 (95% CI 1.03–11.25) greater chance of having difficulty thinking clearly. A systematic review and meta-analysis that included almost 50,000 patients from various countries and settings, with different levels of severity, observed that approximately one-third of subjects experienced persistent post-COVID-19 fatigue and more than one-fifth of subjects exhibited cognitive impairment 12 or more weeks after confirming the diagnosis of acute COVID-19 (8).

Regarding the severity of disease, patients who needed to seek in-person care were 2.23 (95% CI 1.30–3.81) times more likely to have cognitive symptoms than those who did not, with 2.87 (CI 95% 1.02–4.43) greater chance of having concentration problems; 2.39 (CI 95% 1.36–4.20) greater chance of reporting memory problems and 2.82 (95% CI 1.61–4.95) greater chance of reporting word finding difficulties. This result agrees with the trend that the more severe the acute infection, the greater the chance of developing cognitive symptoms as a feature of the post-COVID-19 syndrome (36). In an ongoing cohort study that followed more than 70,000 adult participants during the COVID-19 pandemic, there was a higher prevalence of cognitive symptoms among individuals with moderate/severe COVID-19 when compared to mild cases (RR 1.9 [95% CI 1.3–2.9]) (34). In addition, a recent North American study of 89 patients hospitalized during acute SARS-CoV-2 infection with 6 months of follow-up found that having developed pneumonia after COVID-19 is a risk factor for cognitive symptoms (OR 1.69 [95% CI 1.16–2.46]) (37).

As for medical comorbidities, the presence of one or more of them did not influence the occurrence of persistent cognitive symptoms. This could be due to the nature of the sample, which consisted of patients with mild COVID-19. A systematic review and meta-analysis including 677,045 COVID-19 survivors demonstrated that underlying comorbidities may be a predisposing factor for the development of long-term COVID-19 symptoms (38). However, studies specifically assessing comorbidities as a risk factor for post-COVID-19 cognitive symptoms are needed.

Although the pathophysiology underlying post-COVID-19 cognitive symptoms is not understood, there are interesting. In the acute phase of the disease, the SARS-CoV-2 virus can penetrate the blood–brain barrier directly through the olfactory nerve, and viral proliferation can benefit from areas of cerebral hypoxia, increasing the central nervous system (CNS) viral load over time and affecting mitochondrial function. As brain tissues have high metabolic demand, this can lead to cognitive impairment (39). Another theory suggests that the impairment is not caused by a direct viral aggression to the CNS but by an overreaction of the immune system’s response to the infection (40). It is also possible that those symptoms are the consequence of acute phase damage since the severity of acute disease is associated with cognitive symptoms (32). Finally, it is important to highlight that the inflammation and oxidative stress due to the SARS-CoV-2 infection may lead to neuropathological processes, such as cortical and hippocampal atrophy and small vessel disease, which could contribute to post-COVID-19 cognitive dysfunction symptoms (6). It should be noted that while symptoms related to concentration, language and memory have relatively well-defined neuroanatomical correlates, “thinking clearly” cannot be easily mapped into some brain neural circuitry or structure (41).

Cognitive subdomains such as memory and concentration are significantly impaired during and between episodes in individuals with depression (42). Multiple interacting neurobiological mechanisms (e.g., neuroinflammation and endothelial dysfunction) are implicated as subservient to cognitive deficits in depressive episodes (42, 43). In the current study, patients who experienced depression were 5.37 (95% CI [2.19–13.15]) times more likely to have cognitive symptoms than those who did not experience it, with 6.72 (95% CI [2.81–16.09]) greater chance of having concentration problems; 3.76 (95% CI [1.60–8.87]) greater chance of reporting memory problems, 2.81 (95% CI [1.18–6.70]) greater chance of reporting word finding difficulties, and 4.12 (95% CI [1.66–10.22]) greater chance of having difficulty thinking clearly. These results agree with a large longitudinal analysis of 1,733 consecutive patients with laboratory-confirmed COVID-19, in which 23% of patients reported concomitant symptoms of anxiety/depression 6 months after acute SARS-CoV-2 infection (44).

A systematic review that analyzed eight studies found considerable rates of depressive symptoms and clinically significant depression in post-COVID-19 syndrome. The frequency of depressive symptoms more than 12 weeks after a SARS-CoV-2 infection ranged between 11 and 28%. Two separate studies investigated the association between depression and neurocognitive functioning in post-COVID-19 syndrome and found that patients with depression tended to perform worse on neurocognitive tests compared to those without depression. Baseline markers of systemic inflammation and its change over time have been shown to predict depressive symptoms at three months of post-discharge follow-up. However, it remains to be seen whether the high frequency of depression among individuals with post-COVID-19 syndrome is a long-term consequence of the viral infection or a result of social, economic, and spatial factors (45). In the subanalysis excluding patients with depressive symptoms, we have obtained very similar results to the analysis using the full sample size. In other words, despite the overlap, post-COVID cognitive symptoms are not always linked to depression.

No association was found between the occurrence of persistent cognitive symptoms and the patient’s vaccination status, although the impact of vaccination on post-COVID-19 syndrome differs across studies. An exploratory, observational single-center cohort study of patients hospitalized for COVID-19 demonstrated that vaccinated patients have a lower risk of developing impaired concentration (OR 0.49 [95% CI 0.24–0.98]) (37). Similarly, other studies also evidenced that vaccination reduces the risk of post-COVID outcomes (46) as reported in a meta-analysis, in which people who received two doses of vaccine were significantly less likely to develop this condition than unvaccinated people (47). With the widespread dissemination of COVID-19 vaccination, however, the evolving landscape necessitates further in-depth study. As vaccination reduces COVID-19 severity (48), there might be a positive impact in reducing persistent cognitive symptoms.

When evaluating cognitive symptoms, it is also important to understand their impact on patients’ daily activities, more specifically in their professional lives, but evidence of this impact is still scarce. In the present study, patients with cognitive symptoms reported a higher frequency of time away from work longer than the usual period of isolation, fourteen, ten or seven days since the onset of symptoms, depending on the protocol used and the moment of the pandemic (7.0 vs. 1.7%), and restrictions on returning to work (4.0 vs. 1.4%, p < 0.001 for both). A previous study focused on evaluating the quality of life at work in 300 patients before COVID-19 up to over 12 weeks post-acute infection. Only 44.67% of patients presented a normal quality of life at work after 12 weeks of COVID-19 diagnosis, and the authors observed that memory and focus impairment after 12 weeks of COVID-19 diagnosis was a predictor of poor quality of life at work (49).

The main limitation of this study is its reliance on the report of cognitive symptoms and the lack of formal neuropsychological assessment. Self-report measures can be influenced by different factors, including mood status. For instance, depression is associated with cognitive complaints not necessarily confirmed by objective assessment (50). Nevertheless, we have confirmed a high frequency, even in patients without depressive symptoms. Other validated self-report measures of cognition could have been also used, for example, the Subjective Cognitive Decline Questionnaire (SCD-Q) (54). However, we opted for not including it to avoid the research questionnaire being extremely long, which could compromise data quality and response rate (51). The “temporal report” and recall biases may also be seen as potential limitations of the study. Additionally, other authors have already highlighted differences in the definitions of cognitive dysfunction, “brain fog,” memory issues and attention disorder (8). Conversely, the homogeneity of the studied population constitutes a strength of this study.

Future studies must map specific cognitive deficits (e.g., attention, memory, executive function) using quantitative neuropsychological tests. Additionally, future studies are needed to better understand the underlying mechanisms of post-COVID cognitive symptoms, so effective therapeutic approaches can be developed in order to improve quality of life and to mitigate disease burden.

Conclusion

In this retrospective cohort of patients with mostly mild COVID-19, we have demonstrated that cognitive symptoms were common components of post-COVID-19 syndrome, present in 23.7% of patients at 12 weeks after acute COVID-19. Female sex, fatigue, depression and the need to seek in-person care during acute COVID-19 were independently associated with a higher risk for this condition.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

This study was approved by the Brazilian National Commission for Research Ethics (Comissão Nacional de Ética em Pesquisa on number CAAE 30350820.5.1001.0008), and it was conducted in accordance with the Declaration of Helsinki. All patients gave informed consent to participate in the study.

Author contributions

LB: Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Project administration. TC: Writing – original draft, Writing – review & editing, Data curation. BF: Writing – original draft, Data curation, Writing – review & editing. TP: Writing – original draft, Writing – review & editing, Data curation. DP: Writing – original draft, Writing – review & editing, Data curation. TF: Writing – original draft, Writing – review & editing, Data curation. LK: Writing – original draft, Writing – review & editing, Data curation. CO: Writing – original draft, Writing – review & editing, Conceptualization. AT: Writing – original draft, Writing – review & editing, Conceptualization. MM: Writing – original draft, Writing – review & editing, Conceptualization, Data curation, Project administration.

Glossary

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was supported in part by Minas Gerais State Agency for Research and Development (Fundação de Amparo à Pesquisa do Estado de Minas Gerais – FAPEMIG) [grant number APQ-01154-21], National Institute of Science and Technology for Health Technology Assessment (Instituto de Avaliação de Tecnologias em Saúde – IATS)/National Council for Scientific and Technological Development (Conselho Nacional de Desenvolvimento Científico e Tecnológico – CNPq) [grant number 465518/2014-1], CAPES Foundation (Coordenação de Aperfeiçoamento de Pessoal de Nível Superior) [grant number 88887.507149/2020-00] and Brazilian Ministry of Education (MEC—27849*8). TP received a scholarship from Pró-Reitoria de Extensão from Universidade Federal de Minas Gerais (PROEX UFMG). BF and DP received scholarships from FAPEMIG though the Pró-Reitoria de Pesquisa Universidade Federal de Minas Gerais (editais PROBIC 09/2021 and 04/2023, respectively). MM was supported in part by CNPq [grant number 310561/2021-3]. The funding bodies played no role in the design of the study and collection, analysis, and interpretation of data and in writing the manuscript. LB and MM had full access to all the data in the study and had responsibility for the decision to submit for publication.

Acknowledgments

We would like to thank the staff who helped with data collection.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2024.1282067/full#supplementary-material

References

1. Global Burden of Disease Long COVID Collaborators. Estimated global proportions of individuals with persistent fatigue, cognitive, and respiratory symptom clusters following symptomatic COVID-19 in 2020 and 2021. JAMA. (2022) 328:1604–15. doi: 10.1001/jama.2022.18931

PubMed Abstract | Crossref Full Text | Google Scholar

2. Halpin, S, O’Connor, R, and Sivan, M. Long COVID and chronic COVID syndromes. J Med Virol. (2021) 93:1242–3. doi: 10.1002/jmv.26587

PubMed Abstract | Crossref Full Text | Google Scholar

3. National Institute for Health and Care Excellence. COVID-19 rapid guideline: managing the long-term effects of COVID-19. National Institute for Health and Care Excellence. (2020) Available at: https://www.nice.org.uk/guidance/ng188/resources/COVID19-rapid-guideline-managing-the-longterm-effects-of-COVID19-pdf-51035515742 (Accessed November 15, 2021)

Google Scholar

4. Maley, JH, Sampsel, S, Abramoff, BA, Herman, E, Neerukonda, KV, and Mikkelsen, ME. Consensus methodology for the development of postacute sequelae of SARS-CoV-2 guidance statements. PM R. (2021) 13:1021–6. doi: 10.1002/pmrj.12670

PubMed Abstract | Crossref Full Text | Google Scholar

5. Pfaff, ER, Madlock-Brown, C, Baratta, JM, Bhatia, A, Davis, H, Girvin, A, et al. Coding long COVID: characterizing a new disease through an ICD-10 lens. BMC Med. (2023) 21:58. doi: 10.1186/s12916-023-02737-6

PubMed Abstract | Crossref Full Text | Google Scholar

6. Ceban, F, Ling, S, Lui, LMW, Lee, Y, Gill, H, Teopiz, KM, et al. Fatigue and cognitive impairment in post-COVID-19 syndrome: A systematic review and meta-analysis. Brain Behav Immun. (2022) 101:93–135. doi: 10.1016/j.bbi.2021.12.020

PubMed Abstract | Crossref Full Text | Google Scholar

7. Mizrahi, B, Sudry, T, Flaks-Manov, N, Yehezkelli, Y, Kalkstein, N, Akiva, P, et al. Long COVID outcomes at one year after mild SARS-CoV-2 infection: nationwide cohort study. BMJ. (2023) 380:e072529. doi: 10.1136/bmj-2022-072529

PubMed Abstract | Crossref Full Text | Google Scholar

8. Premraj, L, Kannapadi, NV, Briggs, J, Seal, SM, Battaglini, D, Fanning, J, et al. Mid and long-term neurological and neuropsychiatric manifestations of post-COVID-19 syndrome: a meta-analysis. J Neurol Sci. (2022) 434:120162. doi: 10.1016/j.jns.2022.120162

PubMed Abstract | Crossref Full Text | Google Scholar

9. Nouraeinejad, A. Brain fog as a long-term sequela of COVID-19. SN Compr Clin Med. (2023) 5:9. doi: 10.1007/s42399-022-01352-5

PubMed Abstract | Crossref Full Text | Google Scholar

10. Soriano Marcolino, M, Minelli Figueira, R, Pereira Afonso Dos Santos, J, Silva Cardoso, C, Luiz Ribeiro, A, and Alkmim, MB. The experience of a sustainable large scale Brazilian telehealth network. Telemed J E Health. (2016) 22:899–908. doi: 10.1089/tmj.2015.0234

PubMed Abstract | Crossref Full Text | Google Scholar

11. Chagas, BA, Pagano, AS, Prates, RO, Praes, EC, Ferreguetti, K, Vaz, H, et al. Evaluating user experience with a Chatbot designed as a public health response to the COVID-19 pandemic in Brazil: mixed methods study. JMIR Hum Factors. (2023) 10:e43135. doi: 10.2196/43135

PubMed Abstract | Crossref Full Text | Google Scholar

12. Marcolino, MS, Diniz, CS, Chagas, BA, Mendes, MS, Prates, R, Pagano, A, et al. Synchronous teleconsultation and monitoring service targeting COVID-19: leveraging insights for postpandemic health care. JMIR Med Inform. (2022) 10:e37591. doi: 10.2196/37591

PubMed Abstract | Crossref Full Text | Google Scholar

13. Belo Horizonte City Hall. Guide for Post-COVID-19 Management. (2021). Available at: https://prefeitura.pbh.gov.br/sites/default/files/estrutura-de-governo/saude/2021/guia_manejo_pos-COVID-21-09-2021_0.pdf (Accessed November 15, 2021)

Google Scholar

14. Brazil. Ministry of Health. Manual for assessment and management of post-COVID conditions in Primary Health Care/Ministry of Health, Federal University of Rio Grande do Sul. – Brasília: Ministry of Health. (2022) Available at: https://bvsms.saude.gov.br/bvs/publicacoes/manual_avalia%C3%A7%C3%A3o_manejo_condi%C3%A7%C3%B5es_COVID.pdf (Accessed March 10, 2022)

Google Scholar

15. Spitzer, RL, Kroenke, K, Williams, JBW, and Löwe, B. A brief measure for assessing generalized anxiety disorder: the GAD-7. Arch Intern Med. (2006) 166:1092–7. doi: 10.1001/archinte.166.10.1092

Crossref Full Text | Google Scholar

16. Osório, FL, Silva, TDAD, Santos, RGD, Chagas, MHN, Chagas, NMS, Sanches, RF, et al. Posttraumatic stress disorder checklist for DSM-5 (PCL-5): transcultural adaptation of the Brazilian version. Arch Clin Psychiatry. (2017) 44:10–9. doi: 10.1590/0101-60830000000107

Crossref Full Text | Google Scholar

17. Levis, B, Sun, Y, He, C, Wu, Y, Krishnan, A, Bhandari, PM, et al. Accuracy of the PHQ-2 alone and in combination with the PHQ-9 for screening to detect major depression: systematic review and meta-analysis. JAMA. (2020) 323:2290–300. doi: 10.1001/jama.2020.6504

PubMed Abstract | Crossref Full Text | Google Scholar

18. Chalder, T, Berelowitz, G, Pawlikowska, T, Watts, L, Wessely, S, Wright, D, et al. Development of a fatigue scale. J Psychosom Res. (1993) 37:147–53. doi: 10.1016/0022-3999(93)90081-p

Crossref Full Text | Google Scholar

19. Jing, MJ, Lin, WQ, Wang, Q, Wang, JJ, Tang, J, Jiang, ES, et al. Reliability and construct validity of two versions of Chalder fatigue scale among the general population in mainland China. Int J Environ Res Public Health. (2016) 13:147. doi: 10.3390/ijerph13010147

PubMed Abstract | Crossref Full Text | Google Scholar

20. American Heart Association. Classes of Heart Failure. (2017). Available at: https://www.heart.org/en/health-topics/heart-failure/what-is-heart-failure/classes-of-heart-failure (Accessed November 15, 2021)

Google Scholar

21. Knight, SR, Ho, A, Pius, R, Buchan, I, Carson, G, Drake, TM, et al. Risk stratification of patients admitted to hospital with COVID-19 using the ISARIC WHO clinical characterisation protocol: development and validation of the 4C mortality score. BMJ. (2020) 370:m3339. doi: 10.1136/bmj.m3339

PubMed Abstract | Crossref Full Text | Google Scholar

22. Komaroff, AL, and Lipkin, WI. ME/CFS and long COVID share similar symptoms and biological abnormalities: road map to the literature. Front Med. (2023) 10:1187163. doi: 10.3389/fmed.2023.1187163

PubMed Abstract | Crossref Full Text | Google Scholar

23. Evans, RA, McAuley, H, Harrison, EM, Shikotra, A, Singapuri, A, Sereno, M, et al. Physical, cognitive, and mental health impacts of COVID-19 after hospitalisation (PHOSP-COVID): a UK multicentre, prospective cohort study. Lancet Respir Med. (2021) 9:1275–87. doi: 10.1016/S2213-2600(21)00383-0

PubMed Abstract | Crossref Full Text | Google Scholar

24. Jayasekera, MMPT, De Silva, NL, Edirisinghe, EMDT, Samarawickrama, T, Sirimanna, SWDRC, Govindapala, BGDS, et al. A prospective cohort study on post COVID syndrome from a tertiary care Centre in Sri Lanka. Sci Rep. (2023) 13:15569. doi: 10.1038/s41598-023-42350-4

PubMed Abstract | Crossref Full Text | Google Scholar

25. Jennings, G, Monaghan, A, Xue, F, Duggan, E, and Romero-Ortuño, R. Comprehensive clinical characterisation of brain fog in adults reporting long COVID symptoms. J Clin Med. (2022) 11:3440. doi: 10.3390/jcm11123440

PubMed Abstract | Crossref Full Text | Google Scholar

26. Bull, FC, Al-Ansari, SS, Biddle, S, Borodulin, K, Buman, MP, Cardon, G, et al. World Health Organization 2020 guidelines on physical activity and sedentary behaviour. Br J Sports Med. (2020) 54:1451–62. doi: 10.1136/bjsports-2020-102955

PubMed Abstract | Crossref Full Text | Google Scholar

27. Klok, FA, Boon, GJAM, Barco, S, Endres, M, Geelhoed, JJM, Knauss, S, et al. The post-COVID-19 functional status scale: a tool to measure functional status over time after COVID-19. Eur Respir J. (2020) 56:2001494. doi: 10.1183/13993003.01494-2020

PubMed Abstract | Crossref Full Text | Google Scholar

28. Moura, EC, Cortez-Escalante, J, Cavalcante, FV, Barreto, ICHC, Sanchez, MN, and Santos, LMP. COVID-19: temporal evolution and immunization in the three epidemiological waves, Brazil, 2020–2022. Rev Saude Publica. (2022) 56:105. doi: 10.11606/s1518-8787.2022056004907

PubMed Abstract | Crossref Full Text | Google Scholar

29. Bursac, Z, Gauss, CH, Williams, DK, and Hosmer, DW. Purposeful selection of variables in logistic regression. Source Code Biol Med. (2008) 3:17. doi: 10.1186/1751-0473-3-17

PubMed Abstract | Crossref Full Text | Google Scholar

30. Moretta, P, Ambrosino, P, Lanzillo, A, Marcuccio, L, Fuschillo, S, Papa, A, et al. Cognitive impairment in convalescent COVID-19 patients undergoing multidisciplinary rehabilitation: the association with the clinical and functional status. Healthcare. (2022) 10:e480. doi: 10.3390/healthcare10030480

PubMed Abstract | Crossref Full Text | Google Scholar

31. Pilotto, A, Cristillo, V, Cotti Piccinelli, S, Zoppi, N, Bonzi, G, Sattin, D, et al. Long-term neurological manifestations of COVID-19: prevalence and predictive factors. Neurol Sci. (2021) 42:4903–7. doi: 10.1007/s10072-021-05586-4

PubMed Abstract | Crossref Full Text | Google Scholar

32. Carod-Artal, FJ. Post-COVID-19 syndrome: epidemiology, diagnostic criteria and pathogenic mechanisms involved. Rev Neurol. (2021) 72:384–96. doi: 10.33588/rn.7211.2021230

PubMed Abstract | Crossref Full Text | Google Scholar

33. Asadi-Pooya, AA, Akbari, A, Emami, A, Lotfi, M, Rostamihosseinkhani, M, Nemati, H, et al. Long COVID syndrome-associated brain fog. J Med Virol. (2022) 94:979–84. doi: 10.1002/jmv.27404

PubMed Abstract | Crossref Full Text | Google Scholar

34. Caspersen, IH, Magnus, P, and Trogstad, L. Excess risk and clusters of symptoms after COVID-19 in a large Norwegian cohort. Eur J Epidemiol. (2022) 37:539–48. doi: 10.1007/s10654-022-00847-8

PubMed Abstract | Crossref Full Text | Google Scholar

35. Tronson, NC. Focus on females: a less biased approach for studying strategies and mechanisms of memory. Curr Opin Behav Sci. (2018) 23:92–7. doi: 10.1016/j.cobeha.2018.04.005

PubMed Abstract | Crossref Full Text | Google Scholar

36. Chudzik, M, Babicki, M, Kapusta, J, Kałuzińska-Kołat, Ż, Kołat, D, Jankowski, P, et al. Long-COVID clinical features and risk factors: a retrospective analysis of patients from the STOP-COVID registry of the PoLoCOV study. Viruses. (2022) 14:1755. doi: 10.3390/v14081755

PubMed Abstract | Crossref Full Text | Google Scholar

37. Ch'en, PY, Gold, LS, Lu, Q, Ye, T, Andrews, JS, and Patel, P. Exploring risk factors for persistent neurocognitive sequelae after hospitalization for COVID-19. Ann Clin Transl Neurol. (2023) 10:1200–8. doi: 10.1002/acn3.51801

PubMed Abstract | Crossref Full Text | Google Scholar

38. Yellumahanthi, DK, Barnett, B, Barnett, S, and Yellumahanthi, S. COVID-19 infection: its lingering symptoms in adults. Cureus. (2022) 14:e24736. doi: 10.7759/cureus.24736

PubMed Abstract | Crossref Full Text | Google Scholar

39. Stefano, GB, Ptacek, R, Ptackova, H, Martin, A, and Kream, RM. Selective neuronal mitochondrial targeting in SARS-CoV-2 infection affects cognitive processes to induce brain fog and results in behavioral changes that favor viral survival. Med Sci Monit. (2021) 27:e930886. doi: 10.12659/MSM.930886

PubMed Abstract | Crossref Full Text | Google Scholar

40. Sklinda, K, Górecki, A, Dorobek, M, Walecki, J, Modrzyńska, A, and Mruk, B. Ischaemic background of brain fog in long-haul COVID-19 - a nuclear magnetic resonance spectroscopy-based metabonomic analysis. Pol J Radiol. (2021) 86:e654–60. doi: 10.5114/pjr.2021.111100

PubMed Abstract | Crossref Full Text | Google Scholar

41. Resende, EPF, Hornberger, M, Guimarães, HC, Gambogi, LB, Mariano, LI, Teixeira, AL, et al. Different patterns of gray matter atrophy in behavioral variant frontotemporal dementia with and without episodic memory impairment. Int J Geriatr Psychiatry. (2021) 36:1848–57. doi: 10.1002/gps.5503

PubMed Abstract | Crossref Full Text | Google Scholar

42. Pan, Z, Park, C, Brietzke, E, Zuckerman, H, Rong, C, Mansur, RB, et al. Cognitive impairment in major depressive disorder. CNS Spectr. (2019) 24:22–9. doi: 10.1017/S1092852918001207

Crossref Full Text | Google Scholar

43. Moretta, P, Maniscalco, M, Papa, A, Lanzillo, A, Trojano, L, and Ambrosino, P. Cognitive impairment and endothelial dysfunction in convalescent COVID-19 patients undergoing rehabilitation. Eur J Clin Investig. (2022) 52:e13726. doi: 10.1111/eci.13726

Crossref Full Text | Google Scholar

44. Huang, C, Huang, L, Wang, Y, Li, X, Ren, L, Gu, X, et al. 6-month consequences of COVID-19 in patients discharged from hospital: a cohort study. Lancet. (2021) 397:220–32. doi: 10.1016/S0140-6736(20)32656-8

PubMed Abstract | Crossref Full Text | Google Scholar

45. Renaud-Charest, O, Lui, LMW, Eskander, S, Ceban, F, Ho, R, Di Vincenzo, JD, et al. Onset and frequency of depression in post-COVID-19 syndrome: a systematic review. J Psychiatr Res. (2021) 144:129–37. doi: 10.1016/j.jpsychires.2021.09.054

PubMed Abstract | Crossref Full Text | Google Scholar

46. Català, M, Mercadé-Besora, N, Kolde, R, Trinh, NTH, Roel, E, Burn, E, et al., The effectiveness of COVID-19 vaccines to prevent long COVID symptoms: Staggered cohort analyses of data from the UK, Spain, and Estonia. (2023)

Google Scholar

47. Watanabe, A, Iwagami, M, Yasuhara, J, Takagi, H, and Kuno, T. Protective effect of COVID-19 vaccination against long COVID syndrome: a systematic review and meta-analysis. Vaccine. (2023) 41:1783–90. doi: 10.1016/j.vaccine.2023.02.008

PubMed Abstract | Crossref Full Text | Google Scholar

48. Jiesisibieke, ZL, Liu, WY, Yang, YP, Chien, CW, and Tung, TH. Effectiveness and safety of COVID-19 vaccinations: an umbrella Meta-analysis. Int J Public Health. (2023) 68:1605526. doi: 10.3389/ijph.2023.1605526

PubMed Abstract | Crossref Full Text | Google Scholar

49. Chatys-Bogacka, Z, Mazurkiewicz, I, Slowik, J, Bociaga-Jasik, M, Dzieza-Grudnik, A, Slowik, A, et al. Brain fog and quality of life at work in non-hospitalized patients after COVID-19. Int J Environ Res Public Health. (2022) 19:12816. doi: 10.3390/ijerph191912816

PubMed Abstract | Crossref Full Text | Google Scholar

50. Rami, L, Mollica, MA, García-Sanchez, C, Saldaña, J, Sanchez, B, Sala, I, et al. The subjective cognitive decline questionnaire (SCD-Q): a validation study. J Alzheimers Dis. (2014) 41:453–66. doi: 10.3233/JAD-132027

PubMed Abstract | Crossref Full Text | Google Scholar

51. Lavrakas, PJ. Encyclopedia of survey research methods. Thousand Oaks, CA: Sage Publications, Inc. (2008). 1 p.

Google Scholar