AUTHOR=Huang Yating , Zhai Dengyue , Song Jingze , Rao Xuanheng , Sun Xiao , Tang Jin TITLE=Mental states and personality based on real-time physical activity and facial expression recognition JOURNAL=Frontiers in Psychiatry VOLUME=13 YEAR=2023 URL=https://www.frontiersin.org/journals/psychiatry/articles/10.3389/fpsyt.2022.1019043 DOI=10.3389/fpsyt.2022.1019043 ISSN=1664-0640 ABSTRACT=Introduction

To explore a quick and non-invasive way to measure individual psychological states, this study developed interview-based scales, and multi-modal information was collected from 172 participants.

Methods

We developed the Interview Psychological Symptom Inventory (IPSI) which eventually retained 53 items with nine main factors. All of them performed well in terms of reliability and validity. We used optimized convolutional neural networks and original detection algorithms for the recognition of individual facial expressions and physical activity based on Russell's circumplex model and the five factor model.

Results

We found that there was a significant correlation between the developed scale and the participants' scores on each factor in the Symptom Checklist-90 (SCL-90) and Big Five Inventory (BFI-2) [r = (−0.257, 0.632), p < 0.01]. Among the multi-modal data, the arousal of facial expressions was significantly correlated with the interval of validity (p < 0.01), valence was significantly correlated with IPSI and SCL-90, and physical activity was significantly correlated with gender, age, and factors of the scales.

Discussion

Our research demonstrates that mental health can be monitored and assessed remotely by collecting and analyzing multimodal data from individuals captured by digital tools.