Erin Smith1,2,3,4,5*
Eric A. Storch6
Ipsit Vahia7,8
Stephen T. C. Wong9
Helen Lavretsky10
Jeffrey L. Cummings11
Harris A. Eyre1,2,4,5,6,12- 1The PRODEO Institute, San Francisco, CA, United States
- 2Organisation for Economic Co-operation and Development (OECD), Paris, France
- 3Department of Neurology & Neurological Sciences, Stanford University, Stanford, CA, United States
- 4Global Brain Health Institute, University of California, San Francisco, San Francisco, CA, United States
- 5Global Brain Health Institute, Trinity College Dublin, Dublin, Ireland
- 6Department of Psychiatry and Behavioral Sciences, Baylor College of Medicine, Houston, TX, United States
- 7Division of Geriatric Psychiatry, McLean Hospital, Boston, MA, United States
- 8Division of Geriatric Psychiatry, Harvard Medical School, Boston, MA, United States
- 9Systems Medicine and Biomedical Engineering Houston Methodist, Houston, TX, United States
- 10Semel Institute for Neuroscience and Human Behavior, University of California, Los Angeles, Los Angeles, CA, United States
- 11Chambers-Grundy Center for Transformative Neuroscience, Department of Brain Health, School of Integrated Health Sciences, University of Nevada, Las Vegas (UNLV), Las Vegas, NV, United States
- 12IMPACT, The Institute for Mental and Physical Health and Clinical Translation, Deakin University, Geelong, VIC, Australia
Affective computing (also referred to as artificial emotion intelligence or emotion AI) is the study and development of systems and devices that can recognize, interpret, process, and simulate emotion or other affective phenomena. With the rapid growth in the aging population around the world, affective computing has immense potential to benefit the treatment and care of late-life mood and cognitive disorders. For late-life depression, affective computing ranging from vocal biomarkers to facial expressions to social media behavioral analysis can be used to address inadequacies of current screening and diagnostic approaches, mitigate loneliness and isolation, provide more personalized treatment approaches, and detect risk of suicide. Similarly, for Alzheimer's disease, eye movement analysis, vocal biomarkers, and driving and behavior can provide objective biomarkers for early identification and monitoring, allow more comprehensive understanding of daily life and disease fluctuations, and facilitate an understanding of behavioral and psychological symptoms such as agitation. To optimize the utility of affective computing while mitigating potential risks and ensure responsible development, ethical development of affective computing applications for late-life mood and cognitive disorders is needed.
Introduction
Between 2019 and 2050, the number of people aged 65 years or over in the world will increase from 703 million to 1.5 billion people (1). With the rapid growth in the aging population around the world, improving the standard of care in late-life mood and cognitive conditions is of the utmost importance. Late-life mood and cognitive conditions are characterized by their complexity, multisystemic nature and broad societal impact, hence making them poorly suited to siloed approaches of thinking and innovation (2). Issues such as overlapping symptoms, comorbidities, and misdiagnosis among mental health and neurological disorders represents only a small subset of the challenges facing late-life mood and cognitive conditions (2). For example, psychiatric symptoms often occur during prodromal stages of neurodegenerative diseases (3–5). Neurodegenerative diseases are often misclassified as psychiatric disease, which can cause the patient to receive delayed, inappropriate treatment and experience more distress (4). Relatedly, many neurodegenerative conditions have clinical and neuropathological overlap, which can cause difficulty with accurate diagnosis and treatment (6). Comorbidities also present further challenges. As an example, 61% of people with dementia are estimated to have three or more comorbid diagnoses (7), which often remain over or under-treated and negatively affect the physical and psychological well-being of people with dementia (8). As the burden of late-life mood and cognitive disorders continues to rise, improved prevention, diagnosis, and treatment options are urgently needed (9, 10). To solve the unprecedented complexities and challenges associated with late-life mood and cognitive conditions, new technologies and approaches to care are needed. Affective computing has immense potential to benefit the treatment and care of late-life mood and cognitive disorders. Affective computing (also referred to as artificial emotion intelligence or emotion AI) is the study and development of systems and devices that can recognize, interpret, process, and simulate emotion or other affective phenomena (11, 12). It is a transdisciplinary field that combines engineering and computer science with psychology, cognitive science, neuroscience, sociology, education, psychophysiology, value-centered design, ethics, and more.
In this paper, we aim to provide an overview of common affective computing modalities with applications for late-life mood and cognitive disorders. We additionally explore specific applications for late-life depression and Alzheimer's disease (AD). Lastly, we discuss ethical implications and analyze key challenges that must be overcome to ensure ethical development of affective computing for late-life mood and cognitive conditions.
Methods
Publications were collected in September 2021 from four databases: PubMed, PsycINFO, OvidSP, and Web of Science. To construct the search protocol, the research question was structured in terms of the following topics: Affective Computing, digital phenotyping, late-life depression, dementia, Alzheimer's disease, and ethical issues. Synonyms and main terms for these topics were selected to construct the search codes. After an initial search, publication titles and abstracts were screened according to year of publication, publication in English language, and of peer-reviewed type. A total of 150 articles were utilized in this review.
Results
Affective Computing and Late-Life Depression
Depression is the leading cause of disability worldwide and a major contributor to the overall global burden of disease (13). The scope and burden of late-life depression is significant and expected to rise in the 21st century (14). Depression is estimated to affect 29% of elderly Europeans (15) and 30.6% of elderly Chinese (16). When compared to younger patients with depression, older adults with depression typically have more medical and neurologic comorbidities and display more cognitive impairment (17). Data suggest that 1 in 10 cases of dementia world-wide can be attributed to depression (18). Neurotoxicity due amyloid and tau protein aggregation may represent a pathophysiological cascade which, along with vascular compromise, may predispose individuals to late-life depression (17). Table 1 highlights clinical challenges in late-life depression and ways in which affective computing may be beneficial.
Table 1. Affective computing applications for clinical challenges in late-life depression.
Williamson et al. (23) examined changes in motor output in people with depression from vocal acoustics and facial movements (23). Using the 4th International Audio/Video Emotion Challenge (AVEC), which consists of a read passage and free-response speech segment from subjects with varying depression levels according to their self-reported Beck depression inventory assessment, they developed a multimodal analysis pipeline that leverages complementary information in audio and video signals including structure and timing features for estimating depression severity. Using the identified features of changes in coordination, movement, and timing of vocal and facial movements, the developed algorithm was able to predict the Beck depression inventory ratings from the AVEC test set with a root-mean-square error of 8.12 and mean absolute error of 6.31 (23).
De Choudhury et al. (36) used behavioral attributes from social media to characterize severity of depression at a population level by developing a social media depression index (SMDI) (73). Using crowdsourcing techniques, they built a corpus of over 69K Twitter postings shared by individuals diagnosed with clinical depression that was measured using the Center for Epidemiologic Studies Depression Scale (CES-D) screening test. By analyzing behavioral features including emotional expression, linguistic style, user engagement, and egocentric social network properties, they built a model that can predict if a post is indicative of depression with an accuracy of more than 70% and precision of 0.82. Lastly, they developed the SMDI metric. The SMDI metric leverages the prediction model to predict posts indicative of depression on Twitter and helps characterize the levels of depression in populations. The geographical, demographic, and seasonal patterns of depression given by SMDI confirm known clinical characteristics of depression and are highly correlated with depression statistics reported by the Centers for Disease Control and Prevention (CDC) (73).
Mundt et al. (42) used vocal acoustic biomarkers to predict depression severity and treatment response (42). One hundred five adults with depression were recruited into a 4-week, randomized, double-blind, place-controlled clinical trial. Speech samples were collected at baseline and study end point using an automated telephone system. Clinician-rated and patient-reported measures of depression severity and treatment response were collected. Results from the study replicated and supported findings from prior studies. More severe depression produced longer recordings with more pause time, more variable pause lengths, a greater percentage of pause time, smaller speech/pause ratios, and slower speaking rates. Speech pause times were found to shorten with clinical improvement following treatment, and depressed patients who did not improve clinically were found to have smaller vocal acoustic changes and/or changes that were directionally opposite to treatment responders.
Affective Computing and Alzheimer's Disease
Alzheimer's disease (AD) is the most common neurodegenerative disorder and largest cause of dementia in the world with rapidly growing personal, societal, economic, and medical implications. In the United States alone, over 6.2 million people suffer from AD that costs the healthcare system more than $355 billion annually, not including the value of informal caregiving (74). At a global level, there are more than 35 million people currently living with AD, and by 2050 the number is expected to more than triple, exceeding 115 million people (75). There is growing interest in detecting AD during prodromal stages because (a) the likelihood of reversing anatomic and physiologic changes (such as neuronal death) likely decreased dramatically as the disease advances (61, 76), (b) there is a growing body of evidence that cognitive, sensory, and motor changes may precede clinical manifestation of AD by 10–20 years (74, 76), and (c) aducanumab, the only approved disease-modifying therapy is recommended only for patients with early AD (77). Table 2 highlights clinical challenges in AD and ways in which affective computing may be beneficial.
Table 2. Affective computing applications for clinical challenges in AD.
Ahmed et al. (83) examined connected speech as a marker of disease progression in autopsy-proven AD (83). Samples of connected speech were obtained from 15 patients who were part of a longitudinal cohort study in whom AD was diagnosed during life and later confirmed at post-mortem. The study analyzed spoken discourse over the course from MCI to mild AD dementia to moderate AD dementia. Samples were analyzed using measures of syntactic complexity, lexical content, speech production, fluency, and semantic content. Subtle changes were found in spoken language that were detectable in MCI stages and enabled monitoring progression through successive clinical stages of AD. Language biomarkers could help identify prodromal AD and provide a way to monitor disease in therapeutic trials (83).
Bayat et al. (113) evaluated the ability of in-vehicle GPS data loggers and driving behavior to distinguish cognitively normal older drivers with preclinical AD from those without preclinical AD using machine learning algorithms (113). For 1 year, 139 subjects (64 with preclinical AD; 75 without preclinical AD, as determined by cerebrospinal fluid biomarkers) were monitored while they drove with a commercial in-vehicle GPS data logger. Random Forest models were trained on the GPS data. The receiver operating curve (ROC) area under the curve (AUC) for predicting preclinical AD from driving features alone was 0.82, with the addition of age alone increased to 0.94, and with the additions of age and APOE ε4 status increased to 0.96 (113).
Gills et al. (95) developed and validated a short digital eye-tracking assessment that predicts cognitive status among adults (95). Fifty-five adults (11 with MCI and 44 cognitively normal) were tested on two occasions. During the first visit, participants underwent a brief eye-tracking based visual paired-comparison (VPC), Montreal Cognitive Assessment (MoCA), Digital Symbol Coding test (DSC), and NIH Toolbox Cognitive Battery (NIHTB-CB). During the second visit, participants underwent VPC, DSC, NIHTB-CB, and dual-task (DT). VPC reliably predicted cognitive status while demonstrating high test-retest reliability and displayed significant associations with gold standard cognitive assessments (95). Eye-tracking based VPC may provide a useful, brief, and scalable screening tool for cognitive impairment (95).
Ethical Implications of Affective Computing in Healthcare
New approaches are needed to address the technical, scientific, philosophical, and ethical challenges associated with affective computing applications in healthcare (114). One key challenge is ensuring that the tools account for sex, gender, racial, ethnic, and culture-based differences. For example, two-thirds of AD patients worldwide are women (115, 116). Women have a higher lifetime risk of stroke than men, and women are twice as likely to be diagnosed with depression and anxiety disorders, and migraines (116–118). These are all specific risks factors for developing dementia.
Given the higher rate of depression and AD among women research is needed to understand if and how sex and gender-based differences affect disease manifestations that may result in the need for different digital biomarkers and machine learning affective computing approaches for males vs. females. It is critical to ensure that the machine learning algorithms capture symptoms that may be more common or different in females.
We must ensure that algorithmic fairness with affective computing does not stop at merely accounting for sex and gender-based differences. Data sets and algorithms used for affective computing must also include bias reduction measures that account for ethnic, racial, geographical, cultural, and other human biases (119). As one example, facial analysis algorithms are often trained on datasets that are predominately comprised of lighter-skinned males and may fail to detect female faces and people of different races and ethnicities (120). Algorithmic and human bias must be addressed to ensure greater fairness, transparency, and accountability in the development of affective computing applications.
The various ways affective computing can be leveraged in healthcare also have different ethical implications. For example, current affective computing technologies typically leverage passive or active data collection. In the context of affective computing for healthcare, passive data collection may entail information continuously collected from smartphone usage, driving, or social media, whereas active data collection may involve specific vocal or facial expression assessments within a clinical setting. Different ethical and practical considerations arise for passive vs. active data collection and the setting in which data is conducted (e.g., at home using everyday technological devices or via a specific assessment in-clinic). These challenges must be addressed.
Other ethical considerations that will need to be addressed and mitigated with the advent of affective computing applications in healthcare are (a) addressing privacy and data security concerns; b) determining what it means for people and society to know they have a disorder 10–20 years before clinical manifestations (such as with AD or PD); (c) navigating how to update affective computing models after deployment to make them optimally adaptive and effective; and (d) considering what privacy means when a doctor or friend or family member could tell if you have a certain health condition or are suicidal.
The 2021 “Ethics and governance of artificial intelligence for health: WHO guidance” report by the World Health Organization (WHO) highlights key ethical principles for the use of artificial intelligence in healthcare, which include protecting autonomy; promoting human well-being, human safety and the public interest; ensuring transparency, explainability and intelligibility; fostering responsibility and accountability; ensuring inclusiveness and equity; and promoting artificial intelligence that is responsive and sustainable (121). These ethical principles must guide the development of affective computing applications for late-life mood and cognitive conditions to ensure human rights are upheld and that patient and community interests do not become subordinate to the powerful commercial interest of technology companies or the interests of governments in surveillance and social control (121). Questions related to data security, privacy, and ownership must also be addressed; how CAN these concerns be navigated on a global scale? Initiatives such as the National Institute of Health Bridge to Artificial Intelligence (Bridge2AI) program brings together technologists, biomedical experts, social scientists, and humanists to develop ethical data sets and tools and will help navigate these pressing questions related to affective computing applications in brain health (122).
By addressing the ethical implications and challenges during early stages of technology development, the introduction of affective computing technology in healthcare can bring a new era of health that is marked by a proactive, personalized, and preventative approach to care. Affective computing has the possibility to be applied globally, such as through the ubiquity of smartphones, and can help address challenges related to health equity. Affective computing technology has the potential to improve early detection and screening, disease severity and progression monitoring, treatment efficacy monitoring, and the quality of life for people around the world with a myriad of different neurological health conditions.
Conclusion
Affective computing can address challenges associated with late-life mood and cognitive conditions, including depression and AD. Affective computing technologies- ranging from vocal dynamics to facial expressions to social media usage to driving behavior- can provide objective biomarkers and tools for early detection, monitoring treatment response, tracking disease progression, and more comprehensively understanding the daily life of patients. To leverage affective computing to increase global brain health equity and a precision medicine approach to care, efforts are needed to ensure ethical development of affective computing for late-life mood and cognitive conditions that account for algorithmic and human bias. With these safeguards affective computing can become a major tool of care of late life affective and cognitive disorders.
Author Contributions
ES led the manuscript development and wrote the initial review. HE, JC, HL, SW, IV, and EAS contributed edits, additional examples, and ideas. All authors contributed to the idea development, writing, editing of this manuscript, and approved the submitted version.
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
1. UN. World Population Ageing. (2019). United Nations Department of Economic and Social Affairs (2019).
2. Smith E, Au R, Mossé M, Lavretsky H, Forbes M, Eyre HA. Rebooting late-life mental health innovation and entrepreneurship with convergence science. Am J Geriatr Psychiatry. (2020) 28:591–6. doi: 10.1016/j.jagp.2020.03.003
3. Fujishiro H, Nakamura S, Sato K, Iseki E. Prodromal dementia with L ewy bodies. Geriatr Gerontol Int. (2015) 15:817–26. doi: 10.1111/ggi.12466
4. Woolley JD, Khan BK, Murthy NK, Miller BL, Rankin KP. The diagnostic challenge of psychiatric symptoms in neurodegenerative disease: rates of and risk factors for prior psychiatric diagnosis in patients with early neurodegenerative disease. J Clin Psychiatry. (2011) 72: 126–33. doi: 10.4088/JCP.10m06382oli
5. Pellicano C, Benincasa D, Pisani V, Buttarelli FR, Giovannelli M, Pontieri FE. Prodromal non-motor symptoms of Parkinson's disease. Neuropsychiatr Dis Treat. (2007) 3:145. doi: 10.2147/nedt.2007.3.1.145
6. Das S, Zhang Z, Ang LC. Clinicopathological overlap of neurodegenerative diseases: a comprehensive review. J Clin Neurosci. (2020) 78:30–3. doi: 10.1016/j.jocn.2020.04.088
8. Fox C, Smith T, Maidment I, Hebding J, Madzima T, Cheater F, et al. The importance of detecting and managing comorbidities in people with dementia? Age Ageing. (2014) 43:741–3. doi: 10.1093/ageing/afu101
9. Eyre H, Baune B, Lavretsky H. Clinical advances in geriatric psychiatry: a focus on prevention of mood and cognitive disorders. Psychiatr Clin North Am. (2015) 38:495–514. doi: 10.1016/j.psc.2015.05.002
10. Richardson S, Sinha A, Vahia I, Dawson W, Kaye J, Reynolds III CF, et al. Brain health living labs. Am J Geriatr Psychiatry. (2020) 29:698–703. doi: 10.1016/j.jagp.2020.11.010
14. Forbes M, Rego T, Lavretsky H. III CFR. Convergence Mental Health Across the Life Span: Advances in Precision Geriatric Psychiatry, Convergence MEntal Health: A Transdisciplinary Approach to Innovation. Oxford: Oxford University Press (2020).
15. Horackova K, Kopecek M, Machu V, Kagstrom A, Aarsland D, Motlova LB, et al. Prevalence of late-life depression and gap in mental health service use across European regions. Eur Psychiatry. (2019) 57:19–25. doi: 10.1016/j.eurpsy.2018.12.002
16. Zhong BL, Xu YM, Xie WX, Liu XJ, Huang ZW. Depressive symptoms in elderly chinese primary care patients: prevalence and sociodemographic and clinical correlates. J Geriatr Psychiatry Neurol. (2019) 32:312–8. doi: 10.1177/0891988719862620
17. Eyre HA. Siddarth P. van Dyk K, St Cyr N, Baune BT, Barrio JR, Small GW, Lavretsky H. Neural correlates of apathy in late-life depression: a pilot [(18) F]FDDNP positron emission tomography study. Psychogeriatrics. (2017) 17:186–93. doi: 10.1111/psyg.12213
18. Norton S, Matthews FE, Barnes DE, Yaffe K, Brayne C. Potential for primary prevention of Alzheimer's disease: an analysis of population-based data. Lancet Neurol. (2014) 13:788–94. doi: 10.1016/S1474-4422(14)70136-X
19. Williamson JR, Quatieri TF, Helfer BS, Horwitz R, Yu B, Mehta DD. Vocal biomarkers of depression based on motor incoordination. In: Proceedings of the 3rd ACM International Workshop on Audio/Visual Emotion Challenge (Barcelona). (2013). p. 41–8.
20. Cummins N, Epps J, Breakspear M, Goecke R. An investigation of depressed speech detection: Features and normalization. In: Twelfth Annual Conference of the International Speech Communication Association (Florence). (2011).
21. Trevino AC, Quatieri TF, Malyska N. Phonologically-based biomarkers for major depressive disorder. EURASIP J Adv Signal Process. (2011) 2011:1–18. doi: 10.1186/1687-6180-2011-42
22. Scherer S, Stratou G, Gratch J, Morency LP. Investigating Voice Quality as a Speaker-Independent Indicator of Depression and PTSD. Lyon: Interspeech (2013). p. 847–51.
23. Williamson JR, Quatieri TF, Helfer BS, Ciccarelli G, Mehta DD. Vocal and facial biomarkers of depression based on motor incoordination and timing. In: Proceedings of the 4th International Workshop on Audio/Visual Emotion Challenge (Orlando, FL). (2014). p. 65–72.
24. Cohn JF, Kruez TS, Matthews I, Yang Y, Nguyen MH, Padilla MT, et al. Detecting depression from facial actions and vocal prosody (2009). In: 3rd International Conference on Affective Computing and Intelligent Interaction and Workshops, IEEE (Amsterdam). (2009). p. 1–7. doi: 10.1109/ACII.2009.5349358
25. Maddage NC, Senaratne R, Low LS, Lech M, Allen N. Video-based detection of the clinical depression in adolescents. Annu Int Conf IEEE Eng Med Biol Soc. (2009) 2009:3723–6. doi: 10.1109/IEMBS.2009.5334815
26. Stratou G, Scherer S, Gratch J, Morency LP. Automatic nonverbal behavior indicators of depression and PTSD: Exploring gender differences, (2013). In: Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE (Geneva). (2013). p. 147–52.
27. Joshi J, Dhall A, Goecke R, Breakspear M, Parker G. Neural-net classification for spatio-temporal descriptor based depression analysis. In: Proceedings of the 21st International Conference on Pattern Recognition (ICPR2012), IEEE (Tsukuba). (2012). p. 2634–8.
28. Pampouchidou A, Simos PG, Marias K, Meriaudeau F, Yang F, Pediaditis M, et al. Automatic assessment of depression based on visual cues: a systematic review. IEEE Trans Affect Comput. (2017) 10:445–70. doi: 10.1109/TAFFC.2017.2724035
29. Cohn JF, Cummins N, Epps J, Goecke R, Joshi J, Scherer S. Multimodal assessment of depression from behavioral signals. The Handbook of Multimodal-Multisensor Interfaces: Signal Processing, Architectures, and Detection of Emotion and Cognition-Volume 2 (New York, NY: ACM Books). (2018). p. 375–417.
30. Dibeklioglu H, Hammal Z, Cohn JF. Dynamic Multimodal Measurement of Depression Severity Using Deep Autoencoding. IEEE J Biomed Health Inform. (2018) 22:525–36. doi: 10.1109/JBHI.2017.2676878
31. Joshi J, Goecke R, Parker G, Breakspear M. Can body expressions contribute to automatic depression analysis? (2013). In: 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition (FG), IEEE (Shanghai). (2013). p. 1–7. doi: 10.1109/FG.2013.6553796
32. Alghowinem S, Goecke R, Wagner M, Parkerx G, Breakspear M. Head pose and movement analysis as an indicator of depression, (2013). In: Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE (Geneva). (2013). p. 283–288.
33. Alghowinem S, Goecke R, Wagner M, Parker G, Breakspear M. Eye movement analysis for depression detection, (2013). In: IEEE International Conference on Image Processing, IEEE (Beijing). (2013). p. 4220–4.
34. Vesel C, Rashidisabet H, Zulueta J, Stange JP, Duffecy J, Hussain F, et al. Effects of mood and aging on keystroke dynamics metadata and their diurnal patterns in a large open-science sample: a BiAffect iOS study. J Am Med Inf Assoc. (2020) 27:1007–18. doi: 10.1093/jamia/ocaa057
35. Islam MR, Kabir MA, Ahmed A, Kamal ARM, Wang H, Ulhaq A. Depression detection from social network data using machine learning techniques. Health Inf Sci Syst. (2018) 6:8. doi: 10.1007/s13755-018-0046-0
36. De Choudhury M, Gamon M, Counts S, Horvitz E. Predicting depression via social media. In: Proceedings of the International AAAI Conference on Web and Social Media (Montreal, QC). (2013).
37. Cacheda F, Fernandez D, Novoa FJ, Carneiro V. Early detection of depression: social network analysis and random forest techniques. J Med Internet Res. (2019) 21:e12554. doi: 10.2196/12554
38. Aalbers G, McNally RJ, Heeren A, De Wit S, Fried EI. Social media and depression symptoms: a network perspective. J Exp Psychol: Gen. (2019) 148:1454. doi: 10.1037/xge0000528
39. Song H, You J, Chung JW, Park JC. “Feature Attention Network: Interpretable Depression Detection from Social Media,” In: Proceedings of the 32nd Pacific Asia Conference on Language, Information and Computation, China (2018).
40. Guntuku SC, Yaden DB, Kern ML, Ungar LH, Eichstaedt JC. Detecting depression and mental illness on social media: an integrative review. Curr Opin Behav Sci. (2017) 18:43–9. doi: 10.1016/j.cobeha.2017.07.005
41. Randall N, Bennett CC, Šabanović S, Nagata S, Eldridge L, Collins S, et al. More than just friends: in-home use and design recommendations for sensing socially assistive robots (SARs) by older adults with depression. Paladyn, J Behav Rob. (2019) 10:237–55. doi: 10.1515/pjbr-2019-0020
42. Mundt JC, Vogel AP, Feltner DE, Lenderking WR. Vocal acoustic biomarkers of depression severity and treatment response. Biol Psychiatry. (2012) 72:580–7. doi: 10.1016/j.biopsych.2012.03.015
43. Yang Y, Fairbairn C, Cohn JF. Detecting depression severity from vocal prosody. IEEE Trans Affect Comput. (2013) 4:142–50. doi: 10.1109/T-AFFC.2012.38
44. Cummins N, Sethu V, Epps J, Schnieder S, Krajewski J. Analysis of acoustic space variability in speech affected by depression. Speech Commun. (2015) 75:27–49. doi: 10.1016/j.specom.2015.09.003
45. Scherer S, Hammal Z, Yang Y, Morency LP, Cohn JF. Dyadic behavior analysis in depression severity assessment interviews. Proc ACM Int Conf Multimodal Interact. (2014) 2014:112–9. doi: 10.1145/2663204.2663238
46. Dibeklioglu H, Hammal Z, Yang Y, Cohn JF. Multimodal detection of depression in clinical interviews. In: Proceedings of the (2015). ACM on International Conference on Multimodal Interaction. (2015). p. 307–10. doi: 10.1145/2818346.2820776
47. Joshi J, Dhall A, Goecke R, Cohn JF. Relative body parts movement for automatic depression analysis, (2013). In: Humaine Association Conference on Affective Computing and Intelligent Interaction, IEEE (Geneva). (2013). p. 492–7.
48. Sweeney JA, Strojwas MH, Mann JJ, Thase ME. Prefrontal and cerebellar abnormalities in major depression: evidence from oculomotor studies. Biol Psychiatry. (1998) 43:584–94. doi: 10.1016/S0006-3223(97)00485-X
49. Mastoras RE, Iakovakis D, Hadjidimitriou S, Charisis V, Kassie S, Alsaadi T, et al. Touchscreen typing pattern analysis for remote detection of the depressive tendency. Sci Rep. (2019) 9:1–12. doi: 10.1038/s41598-019-50002-9
50. Bettmann JE, Anstadt G, Casselman B, Ganesh K. Young adult depression and anxiety linked to social media use: assessment and treatment. Clin Soc Work J. (2021) 49:368–79. doi: 10.1007/s10615-020-00752-1
51. Doryab A, Villalba DK, Chikersal P, Dutcher JM, Tumminia M, Liu X, et al. Identifying behavioral phenotypes of loneliness and social isolation with passive sensing: statistical analysis, data mining and machine learning of smartphone and fitbit data. JMIR Mhealth Uhealth. (2019) 7:e13209. doi: 10.2196/13209
52. Arnold AJ, Winkielman P. Smile (but only deliberately) though your heart is aching: Loneliness is associated with impaired spontaneous smile mimicry. Soc Neurosci. (2021) 16:26–38. doi: 10.1080/17470919.2020.1809516
53. Kanai R, Bahrami B, Duchaine B, Janik A, Banissy MJ, Rees G. Brain structure links loneliness to social perception. Curr Biol. (2012) 22:1975–9. doi: 10.1016/j.cub.2012.08.045
54. Hunt MG, Marx R, Lipson C, Young J. No more FOMO: Limiting social media decreases loneliness and depression. J Soc Clin Psychol. (2018) 37:751–68. doi: 10.1521/jscp.2018.37.10.751
55. Scoglio AA, Reilly ED, Gorman JA, Drebing CE. Use of social robots in mental health and well-being research: systematic review. J Med Internet Res. (2019) 21:e13322. doi: 10.2196/13322
56. Chen SC, Jones C, Moyle W. Social robots for depression in older adults: a systematic review. J Nurs Scholarsh. (2018) 50:612–22. doi: 10.1111/jnu.12423
57. Beuscher LM, Fan J, Sarkar N, Dietrich MS, Newhouse PA, Miller KF, et al. Socially assistive robots: measuring older adults' perceptions. J Gerontol Nurs. (2017) 43:35–43. doi: 10.3928/00989134-20170707-04
58. Zhao N, Zhang Z, Wang Y, Wang J, Li B, Zhu T, et al. See your mental state from your walk: Recognizing anxiety and depression through Kinect-recorded gait data. PLoS ONE. (2019) 14:e0216591. doi: 10.1371/journal.pone.0216591
59. Armstrong T, Olatunji BO. Eye tracking of attention in the affective disorders: a meta-analytic review and synthesis. Clin Psychol Rev. (2012) 32:704–23. doi: 10.1016/j.cpr.2012.09.004
60. Yu B, Quatieri TF, Williamson JR, Mundt JC. Cognitive impairment prediction in the elderly based on vocal biomarkers. In: Sixteenth Annual Conference of the International Speech Communication Association (Dresden). (2015).
61. Kourtis LC, Regele OB, Wright JM, Jones GB. Digital biomarkers for Alzheimer's disease: the mobile/wearable devices opportunity. NPJ Digit Med. (2019) 2:1–9. doi: 10.1038/s41746-019-0084-2
62. Schröter A, Mergl R, Bürger K, Hampel H, Möller H-J, Hegerl U. Kinematic analysis of handwriting movements in patients with Alzheimer's disease, mild cognitive impairment, depression and healthy subjects. Dement Geriatr Cogn Disord. (2003) 15:132–42. doi: 10.1159/000068484
63. Hutton JT, Nagel J, Loewenson RB. Eye tracking dysfunction in Alzheimer-type dementia. Neurology. (1984) 34:99–99. doi: 10.1212/WNL.34.1.99
64. Zhu J, Wang Z, Gong T, Zeng S, Li X, Hu B, et al. An improved classification model for depression detection using EEG and eye tracking data. IEEE Trans Nanobioscience. (2020) 19:527–37. doi: 10.1109/TNB.2020.2990690
65. Ntracha A, Iakovakis D, Hadjidimitriou S, Charisis VS, Tsolaki M, Hadjileontiadis LJ. Detection of mild cognitive impairment through natural language and touchscreen typing processing. Front Digit Health. (2020) 19. doi: 10.3389/fdgth.2020.567158
66. Fraser KC, Rudzicz F, Hirst G. Detecting late-life depression in Alzheimer's disease through analysis of speech and language. In: Proceedings of the Third Workshop on Computational Linguistics and Clinical Psychology (San Diego, CA). (2016). p. 1–11.
67. Stasak B, Epps J, Schatten HT, Miller IW, Provost EM, Armey MF. Read speech voice quality and disfluency in individuals with recent suicidal ideation or suicide attempt. Speech Commun. (2021) 132:10–20. doi: 10.1016/j.specom.2021.05.004
68. Amico F, Healy G, Arvaneh M, Kearney D, Mohedano E, Roddy D, et al. Multimodal validation of facial expression detection software for real-time monitoring of affect in patients with suicidal intent. Eur Psychiatry. (2016) 33:S596. doi: 10.1016/j.eurpsy.2016.01.2225
69. Huckvale K, Venkatesh S, Christensen H. Toward clinical digital phenotyping: a timely opportunity to consider purpose, quality, and safety. NPJ Digit Med. (2019) 2:1–11. doi: 10.1038/s41746-019-0166-1
70. Vannoy S, Gable S, Brodt M, Cadet M, Andrews B, Maloney M. An application of eye tracking technology to detect attention bias for suicide related stimuli (2016).
71. De Choudhury M, Kiciman E, Dredze M, Coppersmith G, Kumar M. Discovering shifts to suicidal ideation from mental health content in social media. In: Proceedings of the (2016). CHI conference on human factors in computing systems (San Jose, CA). (2016). p. 2098–110.
72. Coppersmith G, Leary R, Crutchley P, Fine A. Natural language processing of social media as screening for suicide risk. Biomed Inform Insights. (2018) 10:1–11. doi: 10.1177/1178222618792860
73. De Choudhury M, Counts S, Horvitz E. Social media as a measurement tool of depression in populations. In: Proceedings of the 5th annual ACM web science conference (New York, NY). (2013). p. 47–56.
75. Population Reference Bureau: Dementia Cases Expected to Triple by 2050 as World Population Ages (2012).
76. Sperling R, Mormino E, Johnson K. The evolution of preclinical Alzheimer's disease: implications for prevention trials. Neuron. (2014) 84:608–22. doi: 10.1016/j.neuron.2014.10.038
77. Cummings J, Aisen P, Apostolova L, Atri A, Salloway S, Weiner M. Aducanumab: Appropriate use recommendations. J Prev Alzheimers Dis. (2021) 1–13. doi: 10.14283/jpad.2021.41
78. König A, Satt A, Sorin A, Hoory R, Toledo-Ronen O, Derreumaux A, et al. Automatic speech analysis for the assessment of patients with predementia and Alzheimer's disease. Alzheimers Dement (Amst). (2015) 1:112–24. doi: 10.1016/j.dadm.2014.11.012
79. Martínez-Sánchez F, Meilán JJG, Carro J, Ivanova OA. Prototype for the voice analysis diagnosis of Alzheimer's disease. J Alzheimers Dis. (2018) 64:473–81. doi: 10.3233/JAD-180037
80. Fagherazzi G, Fischer A, Ismael M, Despotovic V. Voice for health: the use of vocal biomarkers from research to clinical practice. Digit Biomarkers. (2021) 5:78–88. doi: 10.1159/000515346
81. Pistono A, Jucla M, Barbeau EJ, Saint-Aubert L, Lemesle B, Calvet B, et al. Pauses during autobiographical discourse reflect episodic memory processes in early Alzheimer's disease. J Alzheimers Dis. (2016) 50:687–98. doi: 10.3233/JAD-150408
82. Yeung A, Iaboni A, Rochon E, Lavoie M, Santiago C, Yancheva M, et al. Correlating natural language processing and automated speech analysis with clinician assessment to quantify speech-language changes in mild cognitive impairment and Alzheimer's dementia. Alzheimers Res Ther. (2021) 13:1–10. doi: 10.1186/s13195-021-00848-x
83. Ahmed SA. Haigh MF, de Jager CA, Garrard P. Connected speech as a marker of disease progression in autopsy-proven Alzheimer's disease. Brain. (2013) 136:3727–37. doi: 10.1093/brain/awt269
84. Seidl U, Lueken U, Thomann PA, Kruse A, Schröder J. Facial expression in Alzheimer's disease: impact of cognitive deficits and neuropsychiatric symptoms. Am J Alzheimers Dis Other Demen. (2012) 27:100–6. doi: 10.1177/1533317512440495
85. Burton KW, Kaszniak AW. Emotional experience and facial expression in Alzheimer's disease. Aging Neuropsychol Cogn. (2006) 13:636–51. doi: 10.1080/13825580600735085
86. Verghese J, Robbins M, Holtzer R, Zimmerman M, Wang C, Xue X, et al. Gait dysfunction in mild cognitive impairment syndromes. J Am Geriatr Soc. (2008) 56:1244–51. doi: 10.1111/j.1532-5415.2008.01758.x
87. Lyons BE, Austin D, Seelye A, Petersen J, Yeargers J, Riley T, et al. Pervasive computing technologies to continuously assess Alzheimer's disease progression and intervention efficacy. Front Aging Neurosci. (2015) 7:102. doi: 10.3389/fnagi.2015.00232
88. Ellis RJ, Ng YS, Zhu S, Tan DM, Anderson B, Schlaug G, et al. A validated smartphone-based assessment of gait and gait variability in Parkinson's disease. PLoS ONE. (2015) 10:e0141694. doi: 10.1371/journal.pone.0141694
89. Buracchio T, Dodge HH, Howieson D, Wasserman D, Kaye J. The trajectory of gait speed preceding mild cognitive impairment. Arch Neurol. (2010) 67:980–6. doi: 10.1001/archneurol.2010.159
90. Albers MW, Gilmore GC, Kaye J, Murphy C, Wingfield A, Bennett DA, et al. At the interface of sensory and motor dysfunctions and Alzheimer's disease. Alzheimers Demen. (2015) 11:70–98. doi: 10.1016/j.jalz.2014.04.514
91. Crutcher MD, Calhoun-Haney R, Manzanares CM, Lah JJ, Levey AI, Zola SM. Eye tracking during a visual paired comparison task as a predictor of early dementia. Am J Alzheimers Dis Other Demen. (2009) 24:258–66. doi: 10.1177/1533317509332093
92. Zola SM, Manzanares C, Clopton P, Lah J, Levey A, A. behavioral task predicts conversion to mild cognitive impairment and Alzheimer's disease. Am J Alzheimers Dis Other Demen. (2013) 28:179–84. doi: 10.1177/1533317512470484
93. Fernández G, Manes F, Politi LE, Orozco D, Schumacher M, Castro L, et al. Patients with mild Alzheimer's disease fail when using their working memory: evidence from the eye tracking technique. J Alzheimer's Dis. (2016) 50:827–38. doi: 10.3233/JAD-150265
94. Ladas A, Frantzidis C, Bamidis P, Vivas AB. Eye blink rate as a biological marker of mild cognitive impairment. Int J Psychophysiol. (2014) 93:12–6. doi: 10.1016/j.ijpsycho.2013.07.010
95. Gills JL, Bott NT, Madero EN, Glenn JM, Gray MA. short digital eye-tracking assessment predicts cognitive status among adults. GeroScience. (2021) 43:297–308. doi: 10.1007/s11357-020-00254-5
96. Rabinowitz I, Lavner Y. Association between finger tapping, attention, memory, and cognitive diagnosis in elderly patients. Percept Mot Skills. (2014) 119:259–78. doi: 10.2466/10.22.PMS.119c12z3
97. Austin D, Jimison H, Hayes T, Mattek N, Kaye J, Pavel M. Measuring motor speed through typing: a surrogate for the finger tapping test. Behav Res Methods. (2011) 43:903–9. doi: 10.3758/s13428-011-0100-1
98. Stringer G, Couth S, Brown L, Montaldi D, Gledson A, Mellor J, et al. Can you detect early dementia from an email? A proof of principle study of daily computer use to detect cognitive and functional decline. Int J Geriat Psychiatry. (2018) 33:867–74. doi: 10.1002/gps.4863
99. Gras LZ, Kanaan SF, McDowd JM, Colgrove YM, Burns J, Pohl PS. Balance and gait of adults with very mild Alzheimer disease. J Geriatr Phys Ther. (2015) 38:1–7. doi: 10.1519/JPT.0000000000000020
100. Wittwer JE, Webster KE, Menz HB, A. longitudinal study of measures of walking in people with Alzheimer's disease. Gait Posture. (2010) 32:113–7. doi: 10.1016/j.gaitpost.2010.04.001
101. Maquet D, Lekeu F, Warzee E, Gillain S, Wojtasik V, Salmon E, et al. Gait analysis in elderly adult patients with mild cognitive impairment and patients with mild Alzheimer's disease: simple versus dual task: a preliminary report. Clin Physiol Funct Imaging. (2010) 30:51–6. doi: 10.1111/j.1475-097X.2009.00903.x
102. Freitas Pereira ML, Camargo ZA, Aprahamian I, Forlenza OV. Eye movement analysis and cognitive processing: Detecting indicators of conversion to Alzheimer's disease. Neuropsychiatr Dis Treat. (2014) 10:1273–85. doi: 10.2147/NDT.S55371
103. Coubard OA. What do we know about eye movements in Alzheimer's disease? The past 37 years and future directions. Fut Med. (2016) 10:677–80. doi: 10.2217/bmm-2016-0095
104. Husebo BS, Heintz HL, Berge LI, Owoyemi P, Rahman AT, Vahia IV. Sensing technology to monitor behavioral and psychological symptoms and to assess treatment response in people with dementia. A systematic review. Front Pharmacol. (2020) 10:1699. doi: 10.3389/fphar.2019.01699
105. Au-Yeung TM, Miller L, Beattie Z, May R, Cray HV, Kabelac Z, et al. Monitoring behaviors of patients with late-stage dementia using passive environmental sensing approaches: a case series. Am J Geriat Psychiatry. (2021) 30:1–11. doi: 10.1016/j.jagp.2021.04.008
106. Martínez-Nicolás I, Llorente TE, Martínez-Sánchez F, Meilán JJG. Ten years of research on automatic voice and speech analysis of people with Alzheimer's Disease and mild cognitive impairment: a systematic review article. Front Psychol. (2021) 12:645. doi: 10.3389/fpsyg.2021.620251
107. Martínez F. Acoustic analysis of speech and voice disorders in patients with Lewy Body Diseases (2019).
108. Smith KM, Williamson JR, Quatieri TF. Vocal markers of motor, cognitive, and depressive symptoms in Parkinson's disease, (2017). In: Seventh International Conference on Affective Computing and Intelligent Interaction (ACII), IEEE (San Antonio, TX). (2017). p. 71–78.
109. Williamson JR, Quatieri TF, Smith KM. Vocal Markers of Motor, Cognitive, and Depressive Symptoms in Parkinson's Disease. Lexington, MA: MIT Lincoln Laboratory Lexington United States (2017).
110. Coates L, Shi J, Rochester L, Del Din S, Pantall A. Entropy of real-world gait in Parkinson's disease determined from wearable sensors as a digital marker of altered ambulatory behavior. Sensors. (2020) 20:2631. doi: 10.3390/s20092631
111. Mosimann UP, Müri RM, Burn DJ, Felblinger J, O'Brien JT, McKeith IG. Saccadic eye movement changes in Parkinson's disease dementia and dementia with Lewy bodies. Brain. (2005) 128:1267–76. doi: 10.1093/brain/awh484
112. Adams WR. High-accuracy detection of early Parkinson's Disease using multiple characteristics of finger movement while typing. PLoS ONE. (2017) 12:e0188226. doi: 10.1371/journal.pone.0188226
113. Bayat S, Babulal GM, Schindler SE, Fagan AM, Morris JC, Mihailidis A, et al. GPS driving: a digital biomarker for preclinical Alzheimer disease. Alzheimer's Res Ther. (2021) 13:1–9. doi: 10.1186/s13195-021-00852-1
114. Picard RW. Affective computing: from laughter to IEEE. IEEE Trans Affect Comput. (2010) 1:11–7. doi: 10.1109/T-AFFC.2010.10
115. Alzheimer's disease facts and figures. Alzheimers Dement. (2016) 12:459–509. doi: 10.1016/j.jalz.2016.03.001
117. Organization WH. World health statistics 2016: monitoring health for the SDGs sustainable development goals. World Health Organization (2016).
118. Chuang CS, Lin CL, Lin MC, Sung FC, Kao CH. Migraine and risk of dementia: a nationwide retrospective cohort study. Neuroepidemiology. (2013) 41:139–45. doi: 10.1159/000353559
119. Lee NT, Resnick P, Barton G. Algorithmic Bias Detection and Mitigation: Best Practices and Policies to Reduce Consumer Harms. Washington, DC, USA:Brookings Institute (2019).
120. Buolamwini J, Gebru T. Gender shades: Intersectional accuracy disparities in commercial gender classification. In: Conference on fairness, accountability and transparency, PMLR (New York, NY). (2018). p. 77–91.
Keywords: affective computing, late-life depression, dementia, Alzheimer's disease, digital phenotyping
Citation: Smith E, Storch EA, Vahia I, Wong STC, Lavretsky H, Cummings JL and Eyre HA (2021) Affective Computing for Late-Life Mood and Cognitive Disorders. Front. Psychiatry 12:782183. doi: 10.3389/fpsyt.2021.782183
Received: 23 September 2021; Accepted: 29 November 2021;
Published: 23 December 2021.
Edited by:
Valeria Manera, Université Côte d'Azur, FranceReviewed by:
Maribel Pino, Broca Hospital (APHP), FranceHatice Kose, Istanbul Technical University, Turkey
Copyright © 2021 Smith, Storch, Vahia, Wong, Lavretsky, Cummings and Eyre. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Erin Smith, erin.smith@gbhi.org