Benjamin Strenge
Thomas Schack- Neurocognition and Action Research Group, Faculty of Psychology and Sports Science, Center for Cognitive Interaction Technology (CITEC), Bielefeld University, Bielefeld, Germany
Cognitive assistance systems aim at compensating shortcomings of natural cognition concerning specific activities. Notable progress has been made regarding data acquisition, analysis, and the exploration of technical means for supporting human action selection and execution. The related challenges and potential solutions can be associated to four largely independent questions: What actions should be executed, when this must or should be done, whether assistance is needed for a specific action, and if so, how the action should be supported. A broad range of technological and methodical approaches can be taken for tackling each of these issues, including recent advances and new challenges in the automatized analysis of task-related mental representation structures.
1. Introduction
In the 2020s, pandemics, wars, and climate change pose enormous challenges to human civilization. At the same time, many people's daily lives are often dominated by less far-reaching but equally non-trivial questions such as: What food should I buy, and how and in what order can I then prepare the available ingredients in order to adhere to a particular diet plan and achieve the relevant environmental, health, and athletic goals? Other common types of issues at work and at home revolve around even more narrowly focused questions such as: How can I assemble the new piece of furniture as efficiently and safely as possible? In such situations, it would be desirable to always have an expert on hand to accompany you and help out with appropriate hints and advice—or an appropriately “intelligent" technical assistance system, such as the fictional Tony Stark's J. A. R. V. I. S., which compensates for shortcomings and error-proneness of human cognitive systems. Arguably, the current state of science and technology is still fairly far from this vision, and yet substantial progress has been made in relevant sub-areas in recent years. On this basis, we aim to classify in the following where we currently stand and which essential challenges still have to be solved from our perspective in the future in order to offer cognitive assistance suitable for everyday use through technical systems that leads to better selection and error-free, pleasant execution of human actions.
2. The four primary independent issues
In general, the challenges to be solved in the area of cognitive action assistance can be roughly assigned to answering four independent questions:
1. What actions should be executed?
2. When must or should this be done?
3. Is assistance needed for a specific action? And if so:
4. How should the action be supported?
Figure 1 provides an overview about these issues and potential approaches that have been investigated by different researchers over the past years.
Figure 1. Fundamental issues and potential approaches to cognitive assistance for human action support.
By its very nature, the answer to question 1 (“What actions should be executed?”) is often highly context-dependent and individual. For example, the selection of "correct" actions in the area of nutrition is a highly complex, multi-layered issue that depends strongly on individual preferences, priorities and life circumstances (e.g., Franz et al., 2014; Cecil and Barton, 2020), and the relevant scientific body of knowledge is constantly evolving (Mozaffarian et al., 2018; Ridgway et al., 2019). In such cases, it is conceivable in principle that a technical system will inquire about the corresponding preferences and priorities of the user and then derive suitable suggestions for action on the basis of certain rules or heuristics. Depending on the complexity of the respective area, typically only a rough approximation to the theoretical optimum can be achieved. The assembly of a piece of furniture, on the other hand, can be specified largely independent of context as a simple sequence of action steps. For such activities, it is relatively easy and unambiguous to determine which action is necessary in which step. In any case, addressing question 1 commonly requires more or less extensive authoring for each activity to be supported, in the context of which the rules, criteria, or sequences are explicitly identified and formalized in cooperation with domain experts, and possibly a more or less extensive database must be made available to the system (e.g., the nutritional values of various ingredients and dishes, or the tools that can be used for certain assembly steps).
The treatment of question 2 (“When must or should this be done?”) could again be divided into a) the determination of an optimal sequence of different actions when performing several activities in parallel and b) the recognition of the current state. The search for an optimal temporal ordering of actions is addressed by theoretical and practical computer science in the context of scheduling algorithms under various conditions, but unfortunately many variants of this problem have turned out to be NP-complete, i.e., practically intractable (Ullman, 1975). Thus, in many cases, a technical assistance system can only find and propose approximately optimal solutions. Nevertheless, considering the properties and limitations of human working memory and cognitive bottlenecks in attention and executive functions (Anderson et al., 2004; Borst et al., 2010; Salvucci and Taatgen, 2010), any help in multitasking is likely to be welcome. Recognition of the current activity state is obviously necessary for a cognitive assistance system to know when to be proactive. In this respect, impressive progress has been observed for years in the fields of computer vision and action recognition facilitated by machine learning techniques (e.g., Baccouche et al., 2011; Schröder and Ritter, 2017; Abdulazeem et al., 2021), but overall these have so far typically still been limited to specific, well-defined applications and require prior recording of, or access to, huge amounts of data. For the foreseeable future, therefore, technical systems are likely to fall short of the power of human cognition in this area. A less elegant but technically simpler and much more robust approach is to ask for users' conformation that they executed an action or otherwise initiated or recognized a relevant change of the activity state. Other possibilities lie in the use of environmental sensors and other external data sources that can provide information on the current activity status. For example, if a user shall be assisted while operating a complex industrial machine, that machine may already be connected to suitable external or built-in sensors for gathering process status information and make them available to the assistance system.
The answer to question 3 (“Is assistance needed for a specific action?”) can be approached in a static or dynamic way, or by a combination of these two approaches. For the static estimation of whether assistance is needed for certain actions, on the one hand, statistics on error frequency or the generally expected need for assistance can be used if they are available. However, since task-related prior knowledge and relevant expertise can differ greatly between individuals, such approaches can only serve as very rough heuristics. In contrast, a more precise assessment can be obtained on the basis of an individual task-related structural-dimensional analysis of mental representations (SDA-M) (Schack, 2012), whose current status and perspectives are outlined in more detail in the following sections. Another, also complementary feasible way to find out when assistance is needed could be found in the detection of signals on the user, for example by means of portable electroencephalography (EEG), electrocardiography (ECG), and eye tracking (ET) systems, or by measuring electrodermal activity (EDA). During activity execution, confusion or a lack of crucial information can trigger an acute stress response, which can be measured, for example, as a reduced heart rate variability (e.g., Camm et al., 1996; Szakonyi et al., 2021), increased skin conductance (Critchley, 2002), or decreased pupil dilation (Henckens et al., 2009), thus providing an indication that assistance is needed.
The handling of the fourth and last question (“How should the action be supported?”) again depends strongly on the field of application and the complexity of the activities to be assisted. Portable devices are generally advantageous if the activity is not performed exclusively in a stationary position (e.g., sitting or standing at a fixed workspace). According to our perspective, wearables such as spatial computing smart glasses, which can augment reality by displaying arbitrary virtual elements and helpful instructions directly where the action needs to be performed in the real three-dimensional space, are particularly suitable to support a wide range of activities effectively and comprehensibly. But also simpler 2D head-up displays (HUDs), headsets, or "ambient devices" placed at fixed positions in the environment of the activity performance (e.g., using projectors) can offer assistance functions while users can freely perform the tasks without having to hold the assistance device in their hands. Complementary to the usual considerations concerning usability and user experience in the context of interaction design, an issue of particular importance for effective action assistance is proper attention guidance, especially when using wearable devices with limited fields of view (Renner and Pfeiffer, 2017a,b,c; Renner et al., 2018).
3. Recent advances and successes of SDA-M-based approaches
Structural-dimensional analysis of mental representations (SDA-M) is a method that originated in cognitive psychology and has later also been established in sports science, cognitive robotics, and human-technology interaction. It is based on the cognitive action architecture approach (CAA-A) by Schack (2004). The CAA-A postulates that the control of human movements is based on mental representation units, the so-called basic action concepts (BACs), and their structural composition in relation to one another (Schack and Frank, 2021). Within the hierarchical cognitive architecture of skilled action, the level of mental representations that uses BACs as a means is linked to the highest regulatory level of mental control, which intentionally controls overarching strategies, as well as to lower levels of sensorimotor representation and control that utilize and automatize functional systems and basic reflexes. Accordingly, BACs connect goal-directed functional and perceptual aspects of actions to sensory effects of movements. The individual strengths of associations between BACs of an activity in long-term memory can be analyzed with SDA-M software tools based on a special semi-automatic survey procedure (the so-called "split procedure"). These data can then be visualized via hierarchical clustering algorithms in the form of dendrograms to allow appropriately trained experts to assess the mental representational structure and identify expectable problems in action execution (e.g., Heinen et al., 2002; Schack, 2004; Schack and Hackfort, 2007; Vogel, 2016). In recent years, this procedure has been advanced for use in the cognitive assistance systems ADAMAAS (Essig et al., 2016) and AVIKOM (Neumann et al., 2020) by automating the diagnosis step. For this purpose, the Correct Action Selection Probability Analysis (CASPA) algorithm has been created, which is based on approaches from the cognitive architecture ACT-R by Anderson et al. (2004) and estimates for each individual action from a sequence of actions the individual probability whether a user will be able to select a correct subsequent action after completing the action on his/her own, or will need assistance in doing so (Strenge et al., 2019). Empirical studies indicate that the majority of all action errors that actually occurred could be correctly predicted in this way (Strenge et al., 2020; Strenge and Schack, 2021). Cognitive assistance systems could use this information to proactively prevent human action errors in many cases through timely intervention and appropriate support.
4. Specific challenges of SDA-M-based approaches
A fundamental limitation regarding the applicability of current SDA-M-based approaches for cognitive assistance is that the prediction of error probabilities is currently only possible for predefined action sequences that satisfy some additional criteria (for details see Strenge et al., 2019). This is less problematic in many application domains than it might seem at first glance, because, as Sun (2004, p. 345) noted, "human everyday activities are mostly sequential." Another issue in practical use is the time required for data collection (the "split procedure") by users, since this increases quadratically with the incorporated number of mental representation elements (e.g., actions of an action sequence). Ongoing research aims to investigate whether this issue can be mitigated by sampling from a limited subsequence of actions and using this sample to derive an estimate of an individual's general task-related expertise. Furthermore, it is so far largely unclear how stable the captured mental representational structures are over time. Learning processes induced by practice lead to changes in mental representational structures such that the previously recorded SDA-M data no longer reflect the current state (Frank et al., 2013, 2016; Schack et al., 2014). Therefore, adequate test periods must be defined to measure and reflect task-relevant learning periods in order to always have sufficiently up-to-date information for meaningful cognitive assistance. Conversely, a dynamic adjustment of the extent of assistance to promote learning processes in line with the principle of learning facilitation in ISO 9241-110 is certainly desirable. Neumann et al. (2021) developed experimental approaches for tackling this issue.
5. Discussion
Overall, this perspective on which current issues concerning cognitive assistance systems are especially important, as well as the entailed considerations, should be regarded as a mostly subjective one that was derived to a large extent from research results and lessons learned in the context of two research projects on mobile cognitive assistance systems funded by the German Ministry of Education and Research (BMBF): Project ADAMAAS, which was conducted from 2015 to 2018, and project AVIKOM that started in 2019 and was scheduled to finish by the end of 2022. However, the scope and focus of these projects was narrower than what has been addressed here. Most of these further aspects could be related to what had been considered as “nice-to-have” functionality that did not make it into the research prototypes, or visions for the near future conceived by fellow researchers and partner companies. Future cognitive assistance systems may embrace these visions and solve the connected challenges or explore completely different innovative ways to support human activities and lead to better action selection. Regardless of the technological and methodological tools, it is hoped that sustainable and thriving future assistance systems will not only help out with limited, short-lived everyday problems, but also help their users, perhaps indirectly and subliminally, by choosing appropriate actions, to contribute to overcoming the great challenges of our time—the sustainable preservation of a habitable planet and functioning social structures.
Data availability statement
The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
Written informed consent was obtained from the individual(s) for the publication of any identifiable images or data included in this article.
Author contributions
BS wrote the primary draft of the manuscript. All authors contributed to manuscript revision, read, and approved the submitted version.
Funding
This article was partially based on insights gained from research that has been funded by the German Federal Ministry of Education and Research (BMBF) and the European Social Fund (ESF) in the frame of project AVIKOM. We acknowledge support for the publication costs by the Open Access Publication Fund of Bielefeld University and the Deutsche Forschungsgemeinschaft (DFG).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher's note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
References
Abdulazeem, Y., Balaha, H. M., Bahgat, W. M., and Badawy, M. (2021). Human action recognition based on transfer learning approach. IEEE Access 9, 82058–82069. doi: 10.1109/ACCESS.2021.3086668
Anderson, J. R., Bothell, D., Byrne, M. D., Douglass, S., Lebiere, C., and Qin, Y. (2004). An integrated theory of the mind. Psychol. Rev. 111, 1036–1060. doi: 10.1037/0033-295X.111.4.1036
Baccouche, M., Mamalet, F., Wolf, C., Garcia, C., and Baskurt, A. (2011). “Sequential deep learning for human action recognition,” in Human Behavior Understanding, eds A. A. Salah and B. Lepri (Berlin: Springer), 29–39.
Borst, J. P., Taatgen, N. A., and Van Rijn, H. (2010). The problem state: a cognitive bottleneck in multitasking. J. Exp. Psychol. Learn. Mem. Cogn. 36, 363–382. doi: 10.1037/a0018106
Camm, A. J., Malik, M., Bigger, J. T., Breithardt, G., Cerutti, S., Cohen, R. J., et al. (1996). Heart rate variability. Standards of measurement, physiological interpretation, and clinical use. task force of the european society of cardiology and the north american society of pacing and electrophysiology. Eur. Heart J. 17, 354–381. doi: 10.1093/oxfordjournals.eurheartj.a014868
Cecil, J. E., and Barton, K. L. (2020). Inter-individual differences in the nutrition response: from research to recommendations. Proc. Nutr. Soc. 79, 171–173. doi: 10.1017/S0029665119001198
Critchley, H. D. (2002). Electrodermal responses: what happens in the brain. Neuroscientist 8, 132–142. doi: 10.1177/107385840200800209
Essig, K., Strenge, B., and Schack, T. (2016). “ADAMAAS-towards smart glasses for mobile and personalized action assistance,” in Proceedings of the 9th ACM International Conference on PErvasive Technologies Related to Assistive Environments, Vol. 46 (New York, NY: ACM), 1–46.
Frank, C., Land, W. M., and Schack, T. (2013). Mental representation and learning: the influence of practice on the development of mental representation structure in complex action. Psychol. Sport Exerc. 14, 353–361. doi: 10.1016/j.psychsport.2012.12.001
Frank, C., Land, W. M., and Schack, T. (2016). Perceptual-cognitive changes during motor learning: the influence of mental and physical practice on mental representation, gaze behavior, and performance of a complex action. Front. Psychol. 6, 1981. doi: 10.3389/fpsyg.2015.01981
Franz, M. J., Boucher, J. L., and Evert, A. B. (2014). Evidence-based diabetes nutrition therapy recommendations are effective: the key is individualization. Diabetes Metab. Syndrome Obesity 7, 65–72. doi: 10.2147/DMSO.S45140
Heinen, T., Schwaiger, J., and Schack, T. (2002). “Optimising gymnastics training with cognitive methods,” in Proceedings of 7th annual Congress of the European College of Sport Science (Athens), 608.
Henckens, M. J., Hermans, E. J., Pu, Z., Joëls, M., and Fernández, G. (2009). Stressed memories: how acute stress affects memory formation in humans. J. Neurosci. 29, 10111–10119. doi: 10.1523/JNEUROSCI.1184-09.2009
Mozaffarian, D., Rosenberg, I., and Uauy, R. (2018). History of modern nutrition science–implications for current research, dietary guidelines, and food policy. BMJ 361, k2392. doi: 10.1136/bmj.k2392
Neumann, A., Strenge, B., Schalkwijk, L., Essig, K., and Schack, T. (2021). Facilitating workers' task proficiency with subtle decay of contextual AR-based assistance derived from unconscious memory structures. Information 12, 1–12. doi: 10.3390/info12010017
Neumann, A., Strenge, B., Uhlich, J., Schlicher, K., Maier, G. W., Schalkwijk, L., et al. (2020). “AVIKOM: towards a mobile audiovisual cognitive assistance system for modern manufacturing and logistics,” in Proceedings of the 13th ACM International Conference on PErvasive Technologies Related to Assistive Environments (New York, NY: ACM), 1–8.
Renner, P., Blattgerste, J., and Pfeiffer, T. (2018). “A path-based attention guiding technique for assembly environments with target occlusions,” in 2018 IEEE Conference on Virtual Reality and 3D User Interfaces (VR) (Tuebingen; Reutlingen: IEEE), 671–672.
Renner, P., and Pfeiffer, T. (2017a). “Attention guiding techniques using peripheral vision and eye tracking for feedback in augmented-reality-based assistance systems,” in 2017 IEEE Symposium on 3D User Interfaces (3DUI) (Los Angeles, CA: IEEE), 186–194.
Renner, P., and Pfeiffer, T. (2017b). “Augmented reality assistance in the central field-of-view outperforms peripheral displays for order picking: results from a virtual reality simulation study,” in 2017 IEEE International Symposium on Mixed and Augmented Reality (ISMAR-Adjunct) (Nantes: IEEE).
Renner, P., and Pfeiffer, T. (2017c). “Evaluation of attention guiding techniques for augmented reality-based assistance in picking and assembly tasks,” in Proceedings of the 22Nd International Conference on Intelligent User Interfaces Companion, IUI '17 Companion (New York, NY: ACM), 89–92.
Ridgway, E., Baker, P., Woods, J., and Lawrence, M. (2019). Historical developments and paradigm shifts in public health nutrition science, guidance and policy actions: a narrative review. Nutrients 11, 531. doi: 10.3390/nu11030531
Schack, T. (2004). The cognitive architecture of complex movement. Int. J. Sport Exerc. Psychol. 2, 403–438. doi: 10.1080/1612197X.2004.9671753
Schack, T. (2012). “Measuring mental representations,” in Measurement in Sport and Exercise Psychology, eds G. Tenenbaum, R. C. Eklund, and A. Kamata (Champaign, IL: Human Kinetics), 203–214.
Schack, T., Essig, K., Frank, C., and Koester, D. (2014). Mental representation and motor imagery training. Front. Hum. Neurosci. 8, 328. doi: 10.3389/fnhum.2014.00328
Schack, T., and Frank, C. (2021). Mental representation and the cognitive architecture of skilled action. Rev. Philos. Psychol. 12, 527–546. doi: 10.1007/s13164-020-00485-7
Schack, T., and Hackfort, D. (2007). An action theory approach to applied sport psychology. Handbook Sport Psychol. 3, 332–351. doi: 10.1002/9781118270011.ch15
Schröder, M., and Ritter, H. (2017). Deep learning for action recognition in augmented reality assistance systems. ACM SIGGRAPH 2017 Posters 75, 1–75. doi: 10.1145/3102163.3102191
Strenge, B., Koester, D., and Schack, T. (2020). Cognitive interaction technology in sport-improving performance by individualized diagnostics and error prediction. Front. Psychol. 11, 597913. doi: 10.3389/fpsyg.2020.597913
Strenge, B., and Schack, T. (2021). Empirical relationships between algorithmic SDA-M-based memory assessments and human errors in manual assembly tasks. Sci. Rep. 11, 1–12. doi: 10.1038/s41598-021-88921-1
Strenge, B., Vogel, L., and Schack, T. (2019). Computational assessment of long-term memory structures from SDA-M related to action sequences. PLoS ONE 14, e0212414. doi: 10.1371/journal.pone.0212414
Sun, R. (2004). Desiderata for cognitive architectures. Philos. Psychol. 17, 341–373. doi: 10.1080/0951508042000286721
Szakonyi, B., Vassányi, I., Schumacher, E., and Kósa, I. (2021). Efficient methods for acute stress detection using heart rate variability data from ambient assisted living sensors. Biomed. Eng. Online 20, 73. doi: 10.1186/s12938-021-00911-6
Ullman, J. D. (1975). NP-complete scheduling problems. J. Comput. Syst. Sci. 10, 384–393. doi: 10.1016/S0022-0000(75)80008-0
Keywords: assistance systems, human augmentation, human enhancement, SDA-M, mental representation structures, sustainability
Citation: Strenge B and Schack T (2023) Cognitive assistance for action selection: Challenges and approaches. Front. Psychol. 13:1031858. doi: 10.3389/fpsyg.2022.1031858
Received: 30 September 2022; Accepted: 07 November 2022;
Published: 04 January 2023.
Edited by:
Ulrich Hoffrage, Université de Lausanne, SwitzerlandReviewed by:
Nataniel Boiangin, Barry University, United StatesCopyright © 2023 Strenge and Schack. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Benjamin Strenge, 
benjamin.strenge@uni-bielefeld.de