A Systematic Multidisciplinary Process for User Engagement and Sensor Evaluation: Development of a Digital Toolkit for Assessment of Movement in Children With Cerebral Palsy

Objectives: To describe and critique a systematic multidisciplinary approach to user engagement, and selection and evaluation of sensor technologies for development of a sensor-based Digital Toolkit for assessment of movement in children with cerebral palsy (CP).

Methods: A sequential process was employed comprising three steps: Step 1: define user requirements, by identifying domains of interest; Step 2: map domains of interest to potential sensor technologies; and Step 3: evaluate and select appropriate sensors to be incorporated into the Digital Toolkit. The process employed a combination of principles from frameworks based in either healthcare or technology design.

Results: A broad range of domains were ranked as important by clinicians, patients and families, and industry users. These directly informed the device selection and evaluation process that resulted in three sensor-based technologies being agreed for inclusion in the Digital Toolkit, for use in a future research study.

Conclusion: This report demonstrates a systematic approach to user engagement and device selection and evaluation during the development of a sensor-based solution to a healthcare problem. It also provides a narrative on the benefits of employing a multidisciplinary approach throughout the process. This work uses previous frameworks for evaluating sensor technologies and expands on the methods used for user engagement.

Overview

Digital health technologies represent a new means of addressing modern healthcare challenges (1). For example, wearable sensor technologies can generate in-depth physiological and performance measurements outside of the laboratory environment, thereby, providing insight into real-world user behaviours. This can help patients and clinicians evaluate therapies and monitor progress over time. When thoughtfully developed and implemented, digital solutions can augment the role of the health care professional and enhance patient participation in health care decisions.

The following report describes a systematic multidisciplinary approach to user engagement, and selection and evaluation of sensor technologies for development of a sensor-based Digital Toolkit for assessment of movement in children with cerebral palsy (CP).

Background

Cerebral palsy (CP) is a neurodevelopmental condition caused by injury to the developing brain. It is the most frequent cause of physical disability among children and has a life-long impact on the individual and their use of healthcare (2, 3). People with CP experience disordered posture and movement that in turn causes limitations in activities (e.g., walking) (4, 5). This can lead to physical inactivity, increased risk of developing cardiovascular disease, and limited participation and social integration in the home and community (3, 6, 7).

Rehabilitative and assistive devices, such as mobility and postural aids, can help restore or replace the loss of activity caused by a disability such as CP (8). However, a large proportion of interventions for children with CP have low or inconclusive evidence supporting their effectiveness (9, 10). As assistive devices (such as wheelchairs, walking frames, and communication devices) form a large part of standard care for children with CP, a systematic, objective, and disciplined approach to measuring clinical outcome is needed when prescribing them (10). To date, specialised equipment and technology have been vastly under-researched. This is potentially due to benefits, such as independent mobility, being easily observable yet difficult to objectively quantify outside of the clinic or laboratory environment. Some simple clinical tests allow for a quick overview of a patient's condition, but do not afford more in-depth evaluation of individual impairments or components of activities that may, or may not, be amenable to change. Nevertheless, considering device abandonment issues and associated costs, extensive efficacy research is warranted at both an individual and a population level. Moreover, prescribing assistive technology may raise expectations of good outcomes by the patient and family and give rise to an overinflated perception of high-quality expert care. Thus, it is essential to know if the interventions are working, to prevent device abandonment, false hopes, and unnecessary effort (10). In addition, The Royal Academy of Engineering has called for improved methodologies to obtain evidence for not only safety and performance of medical devices but also for efficacy (11).

To this end, a Digital Toolkit of sensor-based technologies is currently being developed for assessment of movement in children with CP, in relation to rehabilitative and assistive devices. The Digital Toolkit will provide a means of integrating and visualising data from commercially available, wearable sensors to provide a platform for data-driven real-world evaluation of effect of physical therapies, and rehabilitative and assistive devices. To meet the needs of users and provide valuable insights, a process of user engagement and device evaluation has been used to inform the development of the Digital Toolkit.

Engagement with users (e.g., industry, clinicians, patients, and families) throughout the technology development pipeline is essential in ensuring that a product that adds significant value is translated into practice and ultimately delivers impact (12, 13). There is no universally accepted framework for engaging with users in the development of digital health technologies. However, in health and social care research in the UK, meaningful involvement of users through Patient, and Public Involvement (PPI) has increased in recent years and is now well-established (14, 15). From a technology perspective, two research groups have recently proposed processes for user engagement and technology selection and evaluation (12, 16). Further to this, there are established principles for user-centred design [ISO 9241-210:2019(en)], however, these tend to focus on how a user engages with computer-based technology and have limited value in guiding selection of sensors for digital health (17).

It is good practice to disseminate and critically reflect on the approaches used for user engagement in healthcare. Principles of the Short Form Guidance for Reporting Involvement of Patients and the Public checklist (GRIPP2), developed to improve reporting of patient and public involvement (PPI) in research, have been embedded across this report (18).

The aim of this report is to document and critique a systematic multidisciplinary approach to user engagement, and selection and evaluation of sensor technologies for development of a sensor-based Digital Toolkit for assessment of movement in children with cerebral palsy (CP).

Materials and Methods

A sequential process was employed comprising three steps:

Step 1: Define user requirements, by identifying domains of interest;

Step 2: Map domains of interest to potential sensor technologies; and

Step 3: Evaluate and select appropriate sensors to be incorporated into the Digital Toolkit.

The process employed a combination of principles from frameworks based in either healthcare or technology design (12, 14, 16).

Scope of the project: Inform the development of a Digital Toolkit to aid in-depth objective evaluation of effectiveness of mobility devices in children with CP.

Step 1: Define User Requirements

1a. Identification of Potential Domains of Interest

The Core Set for CP, derived from the World Health Organisation's International Classification of Functioning, Disability and Health Children and Youth Version - ICF-CY (WHO 2007), was used to identify potential domains of interest to target with sensor technologies (19, 20). The ICF-CY is a framework for describing and organising information on functioning and disability. The framework covers body structures (anatomical parts of the body such as organs, limbs and their components) and functions (physiological functions of body systems), and activity (the execution of a task or action by an individual) and participation (involvement in a life situation). The Core Set for CP is a list of 135 ICF-CY domains that are considered most important for describing the functional profile of individuals with CP aged 0 to 18 years. Through discussion, the research team compiled a short-list of domains from the Core Set which were of potential relevance to the Digital Toolkit (Table 1). Whilst the aim of the Digital Toolkit is to aid assessment of movement in children with CP in relation to rehabilitative and assistive devices, it was agreed that the current project would focus on mobility devices. Selection criteria were based on the research team's consideration of the potential of the ICF-CY domain to either:

a) demonstrate an immediate change, i.e., a change that may be observed during use of a mobility device;

b) demonstrate a change in response to sustained use, i.e., a change that may be observed when the device is used as part of a training program over a period of time; or

c) domains that may have a bearing on the child's ability to use a mobility device.

TABLE 1

Table 1. Short list of domains of interest compiled from ICF-CY core set for CP.

The short-listed domains and their ICF-CY standard definitions were compiled. To aid communication with non-clinical users, “lay” definitions were developed by the research team.

1b. Rapid Review

A list of outcome measures reported as being used in children with CP and relevant to each ICF-CY short-listed domain, was generated through a series of rapid literature reviews. Each review used key words elicited from the ICF-CY domain definitions. Medline and Embase were searched from inception using key words specific to each domain. For example, for the domain “B455 Exercise Tolerance Functions” the following key terms were used: (“exercise tolerance” OR “respiratory capacity” OR “cardiovascular capacity” OR “physical endurance” OR “aerobic capacity” OR “stamina” OR “fatigability” OR “fitness” OR “exercise test” OR “cardiorespiratory” OR “cardiopulmonary”) AND “cerebral palsy.”

A matrix was produced detailing included ICF-CY domains (code, category name, sub-domains, standard definitions), measurement instruments used in clinical practice or research in CP for each domain, any reported technological/sensor-based measurement solutions, references to relevant literature on clinimetric properties, and links to websites where instruments could be obtained.

1c. User Engagement

The domains and their associated definitions, short-listed in Step 1a, were presented in separate deliberative workshops to three stakeholder groups (21). The stakeholder groups were: (i) children with CP and their families; (ii) clinicians; and (iii) industry. The workshops were facilitated by both the clinical and computer science researchers. The team adhered to the principles of patient and public involvement (PPI) as proposed by NIHR (14). The aim of the workshops was to obtain a broad range of perspectives from end-users on the relative importance of the short-listed domains.

Workshops With Children and Families

Two deliberative workshops were held with families of children with CP. Workshops were advertised via social media (Twitter and Facebook) and using two community mailing lists (Northern Ireland Cerebral Palsy Register Community Mailing List, James Leckey Design Families Communications Email List). Two workshops were convened at a central location with accessible parking at different times (one afternoon, one evening) in an effort to accommodate work, childcare, and access arrangements. Travel expenses of workshop participants were reimbursed and a gift card to the value of £20 was provided in acknowledgement of their time and contributions.

At each workshop the research team explained the project to attendees: an overview of the short-listed domains and rapid review (described above in Steps 1a and 1b) was presented and a video demonstration of a range of potential sensors was shown. Following group discussion, each adult was provided with printed copies of the short-listed domains and their lay definitions and was asked to rank order the domains from most (ranked 1) to least important, in their opinion, with respect to measuring the effects of mobility devices. Discussions were not audio-recorded but researchers took anonymised notes during the workshop. Attendee rank order lists were collated at the end of each workshop.

Workshop With Healthcare Professionals

A deliberative workshop was held for physiotherapists and technical instructors involved in the care of children with CP in school or community settings. After a short presentation outlining the findings of the rapid review (Step 1b), attendees were provided with the short-list of ICF-CY domains and definitions and asked to rank the domains in order of priority (1 being highest priority) without referring to others in the group. The four highest ranked domains were presented back to the group and time allowed for discussion of the results. The group were then asked to consider the measurement instruments (highlighted by the rapid review in Step 1b) associated with the four highest ranked domains and asked to confirm if they were aware of the instruments, were they currently available to them, and had they used them. They were also asked for their opinions on whether the instrument would be useful in a research environment and/or clinical practice. The group was also asked if they were aware of any alternative potential measurement instruments or technology solutions.

Workshop With Industry

The research team met with industry specialists (designers, engineers, research and development management, and therapists involved in manufacture of supportive equipment for children with CP) to present the findings of the ICF-CY domain short-listing and rapid review (Steps 1a and 1b). The short-listed domains were iteratively discussed and industry priorities in relation to domains of interest were noted.

1d. Comparison and Agreement of User Requirements

The four highest ranked domains from the family and healthcare profession stakeholder groups were compared and integrated with the findings from the industry workshop. Final domains of interest across stakeholder groups were agreed within the research team. Workshop attendees were provided with a written summary of findings and the opportunity to provide further feedback.

Step 2: Device Search

The prioritised domains from Step 1 were systematically mapped to feasible sensor technological solutions based on the process proposed by Caulfield et al. (12). To achieve this, two researchers independently searched online using key terms related to each domain. Initially, researchers sought to identify technology-based solutions based on the following criteria:

i) is the system commercially available,

ii) is the solution portable and could it be used in a community setting,

iii) does the system allow for integration i.e., access to data in its rawest form/access to the device through an application programming interface (API) or software development kit (SDK), and

iv) has the device been used with children with CP or other movement disorders.

Whilst non-contact wireless solutions show promise in monitoring of vital signs, they currently suffer from a number of technical challenges limiting their use in clinical and community settings. That, coupled with a lack of a commercially available solution meant that they were not considered for inclusion in the Digital Toolkit. Further, for domains that could not be mapped to objective technology-based solutions (e.g., health-related quality of life), the team identified alternative validated tools, such as questionnaires, that could be used to capture these elements.

Step 3: Device Evaluation

Once relevant solutions were identified, researchers undertook a detailed technical evaluation to ensure the technologies could be successfully integrated. Criteria for consideration were based on a combination of those proposed by Caulfield et al. (12) and Booth et al. (16) and are presented in Supplementary Material. The technical evaluation expanded on the criteria specified above (Step 2, Device Selection) to further refine the selection process. Additional information assessed at this device evaluation step included device dimensions, sensors included, battery life, connectivity requirements and data access, as well as the availability of established software for data collection and interpretation. The team also investigated developer support for integration such as the availability of an SDK or API. Information on regulatory compliance and scientific evidence, specifically, if the device had been clinically validated and if it had been used in a CP or paediatric population was sought. Additionally, purchase information such as device availability in the target country, cost, and the availability of technical support was determined. Finally, human factors such as comfort and usability, as well as the availability of sizes suitable for a paediatric population, were considered. Essentially, we sought to identify established commercial systems that had been validated in the population and setting of interest, that allowed integration and had established software for analysis. Researchers searched online repositories (IEEE Explore, Web of Science and Google Scholar) for device names along with search terms such as “children,” “youth,” “paediatric,” and “cerebral palsy” to identify validation studies and studies reporting use of the with the target population.

Results

Step 1: Define User Requirements

1a. Identification of Potential Domains of Interest

Eleven impairment and eight activity domains that were likely to be amenable to change through use of a mobility device were shortlisted by the research team (Table 1). Three additional domains were also identified that are not included in the ICF-CY Core Set for CP but would potentially be amenable to change through use of a mobility device: health related quality of life, self-concept, and ease of care-giving. Whilst these domains are not included in the ICF-CY or core sets, they are referred to within the ICF-CY section “future directions for ICF.”

1b. Rapid Review

No sensor-based technological measurement solutions were identified from the literature retrieved. For example, the ICF-CY domain “B455 Exercise Tolerance Functions” mapped to various “low-tech” solutions such as timed walk tests (1 minute walk test, 2 minute walk test, 6 minute walk test), set-distance walking test (10 m walk test), shuttle tests (10 m shuttle test, 10 × 5 sprint test) and “high-tech” laboratory tests using online gas analysis.

1c. User Engagement

Workshops With Children and Families

Sixteen families responded to the initial advert, however, due to family and work commitments nine families were unable to attend the scheduled workshops. Seven families (nine adults in total) thus attended one of two scheduled family workshops. In total three grandparents and six parents of children ranging in age from <2 to 15 years attended. Four children also attended with their families. Mean score and rank for each domain is presented in Table 2. Of the 11 short-listed ICF-CY impairment (body function) domains, families ranked the following as most important: (1) muscle tone functions, (2) sensation of pain, (3) mobility of joint functions, and (4) muscle power functions. Of the eight short-listed ICF-CY activity and participation domains, families ranked (1) maintaining a body position, (2) moving around using equipment, (3) walking (short distances), and (4) moving objects with lower extremities as the most important. In addition, the groups unanimously agreed that it was important to consider health-related quality of life, self-esteem and ease of care when evaluating movement in children with CP.

TABLE 2

Table 2. Results of workshops.

Workshop With Healthcare Professionals

Fourteen clinicians with a wide range of experience were present at the workshop (11 physiotherapists and three technical instructors). The following four ICF-CY impairment (body function) domains, of the eleven short-listed, were ranked as most important by clinicians: (1) exercise tolerance functions, (2) mobility of joint functions, (3) sensation of pain, (4) control of voluntary movement functions. The top ranked ICF-CY activity and participation domains, of the eight short-listed, were: (1) changing basic body position, (2) moving around using equipment, (3) transferring oneself, and in joint fourth, moving around, and moving around in different locations. All healthcare professionals agreed that it was important to consider health-related quality of life, self-esteem and ease of care when evaluating movement in children with CP.

With regards to knowledge and use of measurement instruments related to the top ranked ICF-CY impairment (body function) domains, healthcare professionals were most familiar with the 6 minute walk test (exercise tolerance), goniometry (mobility of joint) and visual analogue scales (pain). The group considered that these instruments were useful in both clinical and research settings.

For measurement of control of voluntary movement, a small number of healthcare professionals had an appreciation of laboratory-based measurements such as electromyography and kinematic measures, however, use of these in their area was limited. Instead, a subjective description of control of voluntary movement was reported to be the norm in clinical practice in community healthcare settings.

No additional instruments or sensor-based technological solutions were suggested for any of the domains identified as important.

Workshop With Industry

A workshop was held with four industry specialists working in a commercial enterprise concerned with design and manufacture of mobility device products. Professional backgrounds of participants were therapy, biomedical engineering and research and development. The priorities of the industry workshop participants were primarily focused on physiological measures aligned with the ICF-CY impairment level, such as discerning muscle activation patterns and ascertaining the “smoothness” of movement. No additional domains of interest were identified by industry specialists.

1d. Comparison and Agreement of User Requirements

The engagement process identified similarities and differences between groups of stakeholders regarding domains they considered important to measure. Two top-ranked ICF-CY impairment domains were common to both families and clinicians: mobility of joint functions and sensation of pain. Families' top-ranked ICF-CY impairment domain was muscle tone, which was ranked 7th (of 11) by clinicians, whereas, clinicians' top-ranked domain was exercise tolerance, which families ranked 9th (of 11). Differences in perceived priorities for measurement between families and clinicians were also evident for the ICF-CY activity and participation domains, as shown in Table 2: only “moving around using equipment” was common to both respondent groups. Strong agreement between clinicians and families was evident in relation to the importance of health-related quality of life, ease of care, and self-concept. Domains rated least important were broadly consistent between stakeholder groups and included weight maintenance, respiratory function, and involuntary movements. These domains were thus removed from subsequent steps in the development of the Digital Toolkit given their lower perceived importance.

A high-level summary of findings was communicated directly to all workshop attendees and circulated via the Northern Ireland Cerebral Palsy Community Mailing List to disseminate the findings of the workshops to the wider community. No further feedback or suggestions were received.

Step 2: Device Search

The research team successfully mapped the prioritised clinical domains in Step 1 (mobility of joint functions, muscle tone, muscle power, control of voluntary movement, exercise tolerance, and moving around using equipment), to potential instrumented measurements (Table 3), with the exception of the “sensation of pain” domain. Instrumented techniques commonly used to measure joint mobility, voluntary movement, muscle tone, muscle power and moving around, included three dimensional (3D) motion capture and surface electromyography (sEMG) (24). Methods to measure exercise tolerance included vital sign monitors for heart rate (HR) and respiration rate (RR) (23).

TABLE 3

Table 3. Mapping of instrumented physiological measures from Step 3.

With the underlying instrumented measurements identified, the research team then began to search for suitable sensor technologies that met the requirements highlighted previously in Step 2 of the Methods Section. Findings, by ICF-CY domain, are summarised below.

Exercise Tolerance

Wearable vital signs monitors have become increasingly common and accessible in recent years. Many commercial fitness and lifestyle monitors, such as Fitbit and Apple Watch, are now capable of recording information on a user's heart rate and activity. The research team chose to focus on chest worn vital signs monitors rather than wrist or patch-based monitors for two reasons. Firstly, heart rate measurements from chest strap-based monitors have been shown to be more accurate compared to optical wrist-based monitors (29). Secondly, a chest strap was deemed more practical for participants, particularly considering the locations of other sensors that the participant would be required to wear during the study. Chest based monitors come in two main formats, chest straps or garment integrated (30).

Joint Mobility and Moving Around

The device search identified motion capture (MoCap) as a useful tool for objectively quantifying human movement. It is often used with individuals with CP in a clinical or laboratory setting to analyse walking patterns (31). However, whilst optical based MoCap is considered the gold standard, most optical MoCap systems are costly, cumbersome, suffer from optical occlusion, and require a dedicated laboratory (31). More recently wearable MoCap systems based on inertial measurement unit (IMU) have come onto the market. These systems employ sensors fusion and provide information on degrees of freedom, force, and moment to estimate joint moments and powers.

Muscle Power, Muscle Tone, and Control of Voluntary Movement

Surface electromyography (sEMG) was identified as a non-invasive means of investigating and evaluating skeletal muscle activity. Contemporary sEMG systems consist of wireless solutions and generally come in garment-based or fully wireless sensor forms, thus meet our “portability” requirement (32). Garment-based solutions tend to focus on specific areas rather than targeted muscles and are aimed more at consumers rather than clinical research markets. A number of systems that have been used in clinical research were identified. These solutions allow for real-time feedback and can be used outside of a laboratory setting. Many of these solutions also incorporated IMU affording the potential for kinematic data capture in tandem with sEMG. Whilst typically more accurate, the clinical systems were considerably more expensive than consumer systems.

Step 3: Device Evaluation

With a range of suitable systems identified from the device search in Step 2, the research team set out to technically evaluate each of the devices. Technical information was gleaned from manufacturer websites, with information on clinical validation and use of the device in the target population coming from academic publications. The results of the technical evaluation are summarised in Supplementary Material. With the explosion of wearable technologies now available on the market, there was ample choice of instrumented solutions available for review.

Vital signs monitors typically captured information on heart rate, respiration rate and activity. Some devices captured additional clinical information including heart rate variability and temperature. Devices ranged in price from approximately £200–£1,000. Battery life for these devices typically lasted at least 24 h and up to 65 h, depending on the sampling and data transmission rates. These devices usually communicated over standard transmission protocols such as Bluetooth and provided good access to raw data.

EMG solutions were generally the most expensive instrumented element of the Digital Toolkit. System prices ranged from £12,000 to £20,000. The price was typically associated with the substantial number of sensors required to measure the whole body (typically 16 sensors). Nodes generally included EMG in combination with other sensors to measure movement such as IMU. EMG systems were generally wireless, transmitting data over proprietary data transmission. Battery life was typically 6–10 h depending on the sampling rate and number of sensors.

For motion capture systems, there was a wide range of solutions available, from those that use one or two sensors to track specific measures of gait, to full body motion capture solutions that use up to 24 sensor nodes. These solutions typically incorporated IMU sensors with the option to add other sensing technologies such as force sensors, EMG, and electro-goniometers. Depending on the sampling rate and the number of sensors, battery life lasted between 3 and 18 h. Data was typically transmitted over Wi-Fi or proprietary wireless data transmission. Traditional camera-based systems were also available. However, these tended to be less portable, susceptible to occlusion and limited to a set measurement volume.

Across all three application areas, some of the solutions identified were targeted more toward consumer markets, rather than research. These solutions tended to be lower cost, however, are less likely to have been clinically validated. The most established market was that of vital signs monitoring. Devices in this category were of reasonable cost, all those identified had been clinically validated and many had been used with the target population. Motion capture systems are also very well-established with many solutions commercially available. Whilst the cost of these systems is still high, particularly for research focused systems, this is likely due to the technical complexity of these solutions which incorporate multiple sensors and require complex algorithms for data processing.

Summary of Results

Through combination of the aforementioned steps, the multidisciplinary team developed a Digital Toolkit consisting of IMU/ motion capture (Xsens) to evaluate joint mobility, ECG and Respiratory Inductive Plethysmography (Hexoskin) to evaluate exercise tolerance, and surface EMG (DELSYS) to explore muscle power, control of voluntary movement and muscle tone (see Figure 1).

FIGURE 1

Figure 1. Overview of user-prioritised domains of interest and selected sensor technology.

Hexoskin was chosen as an exercise tolerance measurement solution, over other vital signs solutions, as the system met selection criteria in terms of measurement requirements (ECG, HR, and RR), interoperability, access to raw data and portability, and was clinically validated with a paediatric population. The Zephyr Bio Harness was considered an appropriate alternative. Nonetheless the Hexoskin was selected due to its lower cost and garment-based form factor which was deemed more practical for this particular population by the research team (See Supplementary Material). Delsys was selected as the EMG solution, potentially capturing data relating to muscle power, control of voluntary movement and muscle tone, as again it met all the relevant selection criteria around measurement requirements, interoperability, access to raw data and portability. The solution was also clinically validated, medically certified and used with people with CP. Compared to the other solutions investigated, Delsys was found to have slightly lower cost and good availability of software tools (Supplementary Material). Finally, for joint mobility and movement, Xsens was selected as the most appropriate solution. Many of the solutions in this category were targeted at motion capture and were not clinically validated. Xsens had been used with people with CP and had a lower price compared to the other solutions evaluated (Supplementary Material).

Other user-prioritised domains, such as pain, activity and participation, health-related quality of life, ease of care and self-concept, could not be mapped to sensor technology. These are nonetheless important to users and will be captured through validated questionnaires.

Discussion

The process reported here provides detailed insights to support decision-making when developing the Digital Toolkit, a sensor-based solution for evaluating movement in children with CP, that will be used in future clinical research projects. Through rapid review of the current evidence base and engagement with healthcare professionals, children and families, and industry, a broad range of perspectives were obtained on the domains that should be considered. This was followed by a systematic selection and evaluation process for sensor technologies that addressed the prioritised domains.

Use of pre-identified core outcome sets for CP (19), further, refined by deliberative workshops with stakeholders, allowed the research team to quickly identify meaningful domains of importance for potential inclusion in our Digital Toolkit, such as sensation of pain, exercise tolerance, joint mobility, muscle power, muscle tone, and control of voluntary movement. Interestingly, although families and clinicians had differing views on the relative importance of a number of prioritised domains, there was significant agreement between stakeholders on domains considered to be less important. These included weight maintenance, respiratory function, involuntary movement, gait pattern, and muscle endurance. This is perhaps unsurprising as therapeutic walking devices typically aim to provide opportunities for upright mobility, weight-bearing through the legs and changes in posture, as opposed to delivering exercise at an intensity sufficient to build muscle endurance or alter respiratory function and weight. Clinicians and families also agreed that outcomes related to health-related quality of life, ease of care, and self-concept were important to consider when evaluating movement related to mobility devices. Although, these outcomes do not correspond to domains listed in the ICF-CY core set for CP, they are important to consider when introducing new devices or interventions in line with family-centred care principles. Findings from the deliberative workshops with stakeholders directly informed the subsequent device search and device evaluation and ensured that the Digital Toolkit will be useful and relevant to families, clinicians, and industry.

When mapping sensor technologies in Step 2 of the process reported here, we identified commercially available, instrumented measures for all the prioritised domains, except for sensation of pain. In the 2020 updated definition of pain, the International Association for the Study of Pain acknowledge the nuances and complexity of pain, and the difficulties in assessment of this sensory and emotional experience (33). Even though pain is very common, given that it is a personal experience influenced to varying degrees by biological, psychological, and social factors, the lack of sensor-based measurement solutions is not surprising. Our device search also highlighted that for some domains, such as muscle power and control of voluntary movement, instrumented measures are not yet well-established and remain subject to ongoing research and validation (24), whereas, devices such as sEMG and MoCap are used more commonly with our population of interest. A potential challenge with some of the identified devices was the ability to collect data in “real-world” settings, i.e., outside of a laboratory environment.

A key requirement in developing our Digital Toolkit was portability as in future studies we plan to evaluate the effects of mobility devices in “real-world” settings, rather than in a laboratory environment. Wearable technologies, particularly garment-based solutions, were noted as having huge potential in the monitoring of movement outside of the laboratory setting. Garment-based solutions which incorporate textile electrodes are available for vital signs monitoring and EMG. Benefits include comfort (over the application and removal of traditional adhesive sensors). However, incorporating sensors within a garment also raises challenges for researchers in relation to the need for multiple sizes (particularly when working with children), cleaning regimens, and loss of precision in sensor placement. The issue of sizing was also apparent in vital signs monitoring devices, such as chest straps, with few paediatric sized products being available, especially for younger children.

Variation in measurement precision was noted when evaluating various vital signs monitors, EMG systems and 3D motion capture systems. Typically, devices could be categorised as (1) research/clinical grade, or (2) consumer grade solutions. For example, for sEMG, full body solutions such as Delsys have been clinically validated and used in a range of populations in a research setting, whereas, the more “consumer grade” Myontec Mbody 3 EMG shorts do not appear to be clinically validated. A similar situation was identified for 3D motion capture when comparing the Xsens Awinda system to the lower cost Shadow MoCap. Whilst these consumer grade systems are less costly (i.e., ~£12,000 for Xsens vs. £900 for Myontec) it is difficult to accurately compare data quality and precision without a formal validation study. Given that our Digital Toolkit will be used with children with CP, where changes in outcome measurements can be small yet clinically meaningful, we selected only solutions which had been clinically validated or used extensively in research as specified in our Device Evaluation selection criteria (Methods, Step 3).

From an interoperability perspective many of the devices were found to have either an SDK or API. This allows good access to data in its rawest form whilst also allowing for integration of multiple systems. Integration and synchronisation will be one of the key technical challenges with data processing associated with our Digital Toolkit as it will necessitate combination of data from a number of systems that have different data transmission rates, protocols and data formats. Therefore, when selecting the final set of devices for inclusion in the Digital Toolkit, it was important to consider them together as a system. Selecting a research platform like Shimmer, which provides a number of add on boards to collect a range of physiological and kinematic parameters such as ECG and EMG, may provide flexibility to create a bespoke integrated solution on a single platform. Doing so may, however, have a disadvantage in that technical development and software tools that have already been developed and validated through extensive research cannot be used. Therefore, the research team decided to select established commercial systems (Methods, Step 2, Device Selection) that had been validated and provided established software for analysis (Methods, Step 3, Device Evaluation).

Review of the device evaluation matrix and consideration of the practicalities of use, measurement precision, interoperability and cost of the various devices, facilitated selection of a final set of sensors for inclusion in the Digital Toolkit that met the priorities of families, clinicians and industry partners. These included the Hexoskin Classic Pro shirt for vital signs monitoring, the Delsys Avanti Trigno for EMG and the Xsens MTw Awinda for 3D motion capture. Combination of these three solutions provide instrumented measures covering all the clinical domains of interest. These products also allowed for integration and synchronisation and were flexible enough to be configured to measure the muscle groups and movements required when evaluating mobility devices. Furthermore, the sensors are wireless, comfortable and can be used outside of a laboratory setting. All the solutions selected have also been clinically validated and used in a paediatric population.

Reflections and Critical Perspective

Comparison to Other Frameworks

To the authors' knowledge, this is the first reported use of previously published frameworks by Booth et al. (16) and Caulfield et al. (12) in the selection and evaluation of sensor technologies for ambulatory human physiological measurement. The work presented combines the criteria for device selection and evaluation from both frameworks and expands upon the frameworks in relation to end-user involvement. The framework presented by Booth et al. (16) assumes that measurement domains have been decided upon and that the research team has performed a device search. It focuses on selecting and managing sensor technology within a research study. The framework presented by Caulfield provides considerations for the device search and criteria for selection of devices, however, the user involvement is limited, and the focus is mainly on technical and human factors (12). A summary of the user involvement in the current study, as per the GRIPP2, is reported in Table 4. The workflow presented in the current report is summarised in Table 5, which also compares items between the frameworks. In relation to defining requirements, the current work expands on the previous frameworks by increasing the focus on identification of the priorities for measurement through engagement with end users.

TABLE 4

Table 4. GRIPP2 short form.

TABLE 5

Table 5. Comparison of frameworks.

Strengths

The multidisciplinary approach utilised throughout the described process was integral for developing a common sense of purpose within the team. Both clinical researchers and computer scientists met regularly to discuss and prioritise tasks, share work in-progress and were involved in decision-making steps. Co-facilitation of the deliberative workshops with end users ensured that all members of the team shared a common vision and language with families and healthcare professionals.

Workshops were tailored according to the stakeholder group. For example, in the healthcare professional workshop, individuals were asked to consider their responses independently to avoid deferential effects and inhibition of responses within a group of mixed seniority, whereas, the patient and family workshops took a more informal and collaborative approach in which discussion was encouraged (34). This allowed the research team to gain more insight into the differing perspectives within the group. The rapid review and user engagement methods described expand on the processes described by Caulfield et al. (12) and Booth et al. (16) which have a greater focus on the later phases of device selection and evaluation (see Table 5).

Limitations

Rapid reviews have the advantage of being time efficient however they have some inherent limitations in comparison to a systematic review process. In the case of this report, one reviewer performed study selection and data extraction, and older articles, non-English articles and grey literature were not considered.

With respect to the deliberative workshops, many families showed an interest and motivation to engage. However, despite having various time slots available, some families were unable to attend due to work and family commitments. This would suggest that creative strategies to involve families and individuals with CP would be of benefit. Strategies that do not involve travel, that remove the need to arrange time away from work, or the need to arrange childcare may open opportunities to others.

Further, user engagement after the communication of the results of the workshops could have been used to confirm the domains of interest. However, by including all four top ranked domains from both the family and healthcare professional workshops it is likely that most user requirements have been captured.

From a technical perspective, the sensors selected incur a substantial purchasing cost with the estimated total price exceeding £25,000. Whilst this represents a significant outlay, and may preclude roll out of the Digital Toolkit in routine clinical practice, it is hoped that findings from development of the Digital Toolkit and collection and interpretation of data will inform future research directions such as investigation of a minimum set of sensors that can be utilised and development of a lower cost solution for longitudinal monitoring in a community setting.

Finally, in relation to scope of purpose, the development of the Digital Toolkit was focussed on how best to assess the effect of mobility devices in children with CP. If the scope of purpose were to be expanded, e.g., to other rehabilitative and assistive devices, repetition of all three steps in the reported process should be undertaken.

Future Work

Further evaluation of the Digital Toolkit is planned. Future work will focus on feasibility of measuring the chosen domains with the selected sensor technologies in a group of children with CP. In brief, the objectives will be to assess fidelity and safety of testing procedures, to record any technical issues with the sensors, to explore the sensors' ability to discern between two movement tasks, and to use qualitative methods to gain an understanding of the perceptions and preferences of children and families with CP in relation to the sensor technologies employed.

Conclusion

This report demonstrates a systematic approach to user engagement, and device selection and evaluation, during the development of a sensor-based solution to a healthcare problem, building on previous frameworks. It also provides a narrative on the benefits of employing a multidisciplinary approach throughout the process. The approach resulted in the selection of valid, reliable, commercially available, interoperable sensors that address the priorities of families, healthcare professionals, industry partners, and academic researchers when evaluating the effects of mobility devices on movement of children with CP.

Data Availability Statement

The original contributions presented in the study are included in the article/Supplementary Material, further inquiries can be directed to the corresponding author/s.

Ethics Statement

The Research Governance Office in Queen's University Belfast confirmed that ethical approval was not necessary for the presented project.

Author's Note

In this article we present a systematic process to guide the multidisciplinary team through user engagement and sensor device selection and evaluation. Our approach builds on those previously reported in the literature and expands on the user engagement phase. It also provides a ‘use case' where clinical researchers and computer science researchers collaborate to develop a Digital Toolkit for the assessment of movement in children with cerebral palsy.

This report provides much needed guidance on how to facilitate multidisciplinary collaboration and has an emphasis on patient experience and user engagement in the field of Digital Health.

Author Contributions

All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.

Funding

This work was supported by Invest Northern Ireland under the Competence Centre Programme Grant [RD1014267].

Conflict of Interest

LF is an employee of Sunrise Medical and contributed to the conception of the project but was not involved in the user engagement or device search and evaluation.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

Paediatric physiotherapy team members and the children, parents, and grandparents that contributed to the workshops.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fdgth.2021.692112/full#supplementary-material

References

1. The Topol Review. Preparing the Healthcare Workforce to Deliver the Digital Future. An Independent Report on behalf of the Secretary of State for Health and Social Care. (2019).

2. Beckers LWME, Schnackers ML, Janssen-Potten YJ, Kleijnen J, and Steenbergen B. Feasibility and effect of home-based therapy programmes for children with cerebral palsy: a protocol for a systematic review. BMJ open. (2017) 7:e013687. doi: 10.1136/bmjopen-2016-013687

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Ryan JM. Activity capacity and activity performance: time to be specific? Dev MedChild Neurol. (2016) 58:659–660 doi: 10.1111/dmcn.13089

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Kerr C, Parkes J, Stevenson M, Cosgrove A, and McDowell B. Energy efficiency in gait, activity, participation, and health status in children with cerebral palsy. Dev Med Child Neurol. (2008) 50:204–10 doi: 10.1111/j.1469-8749.2008.02030.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Kerr C, McDowell B, Parkes J, Stevenson M, and Cosgrove A. Age-related changes in energy efficiency of gait, activity, and participation in children with cerebral palsy. Dev Med Child Neurol. (2011) 53:61–7 doi: 10.1111/j.1469-8749.2010.03795.x

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Ryan JM^a, Hensey O, McLoughlin B, Lyons A, and Gormley J. Reduced moderate to vigorous physical activity and increased sedentary behaviour are associated with elevated blood pressure values in children with cerebral palsy. Phys Ther. (2014) 94:1144–53 doi: 10.2522/ptj.20130499

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Ryan JM, Crowley VE, Hensey O, Broderick JM, McGahey A, and Gormley J. Habitual physical activity and cardiometabolic risk factors in adults with cerebral palsy. Res Dev Disabil. (2014) 35:1995–2002 doi: 10.1016/j.ridd.2014.03.051

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Bickenbach J. The world report on disability. Disabil Soc. (2011) 26:655–8 doi: 10.1080/09687599.2011.589198

CrossRef Full Text | Google Scholar

9. Novak I, Morgan C, Fahey M, Finch-Edmondson M, Galea C, Hines A, et al. State of the evidence traffic lights 2019: systematic review of interventions for preventing and treating children with cerebral palsy. Curr Neurol Neurosci Rep. (2020) 20:3 doi: 10.1007/s11910-020-1022-z

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Novak I, McIntyre S, Morgan C, Campbell L, Dark L, Morton N, et al. A systematic review of interventions for children with cerebral palsy: state of the evidence. Dev Med Child Neurol. (2013) 55:885–910. doi: 10.1111/dmcn.12246

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Royal Academy of Engineering. Establishing High-Level Evidence for the Safety and Efficacy of Medical Devices and Systems. (2013). Available online at: https://www.raeng.org.uk/publications/reports/establishing_high_level_evidence (accessed May 13, 2020).

12. Caulfield B, Reginatto B, and Slevin P. Not all sensors are created equal: a framework for evaluating performance measurement technologies. NPJ Digit Med. (2019) 2:7 doi: 10.1038/s41746-019-0082-4

PubMed Abstract | CrossRef Full Text | Google Scholar

13. BBSRC EPSRC MRC. Technology Touching Life Consultation. (2015). Available online at: https://mrc.ukri.org/publications/browse/technology-touching-life-summary-of-consultation-2015/ (accessed June 17, 2020).

14. NIHR. Better Public Involvement for Better Health and Social Care Research. (2019). Available online at: https://www.invo.org.uk/wp-content/uploads/2019/11/UK-standards-for-public-involvement-v6.pdf (accessed March 23, 2020).

15. Price A, Schroter S, Snow R, Hicks M, Harmston R, Staniszewska S, et al. Frequency of reporting on patient and public involvement (PPI) in research studies published in a general medical journal: a descriptive study. BMJ Open. (2018) 8:e020452 doi: 10.1136/bmjopen-2017-020452

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Booth BM, Mundnich K, Feng T, Nadarajan A, Falk TH, Villatte JL, et al. Multimodal human and environmental sensing for longitudinal studies in naturalistic setting: framework for sensor selection, deployment, and management. J Med Internet Res. (2019) 21:e12832 doi: 10.2196/12832

CrossRef Full Text | Google Scholar

17. International Organisation for Standardisation. Ergonomics of Human-System Interaction – Part 210: Human-Centred Design for Interactive Systems (ISO 9241-210:2019). (2019).

Google Scholar

18. Staniszewska S, Brett J, Simera I, Seers K, Mockford C, Goodlad S, et al. GRIPP2 reporting checklists: tools to improve reporting of patient and public involvement in research. BMJ. (2017) 358:j3453 doi: 10.1136/bmj.j3453

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Schiariti V, Selb M, Cieza A, and O'Donnell M. International classification of functioning, disability and health core sets for children and youth with cerebral palsy: a consensus meeting. Dev Med Child Neurol. (2015) 57:149–58 doi: 10.1111/dmcn.12551

PubMed Abstract | CrossRef Full Text | Google Scholar

20. World Health Organisation. International Classification of Functioning, Disability Health: Children & Youth Version: ICF-CY. World Health Organisation (2007). Available online at: https://apps.who.int/iris/handle/10665/43737 (accessed May 02, 2019).

Google Scholar

21. National Consumer Council. Deliberative public engagement: nine principles. (2008). Available online at: https://www.involve.org.uk/resources/publications/practical-guidance/deliberative-public-engagement-nine-principles (accessed May 06, 2021).

22. Jiang M, Mieronski R, Syrjälä E, Anzanpour A, Terävä V, Rahmani AM, et al. Acute pain intensity monitoring with the classification of multiple physiological parameters. J Clin Monitor Comput. (2019) 33:493–507. doi: 10.1007/s10877-018-0174-8

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Verschuren O, Ketelaar M, Keefer D, Wright V, Butler J, Ada L, et al. Identification of a core set of exercise tests for children and adolescents with cerebral palsy: a Delphi survey of researchers and clinicians. Dev Med Child Neurol. (2011) 53:449–56 doi: 10.1111/j.1469-8749.2010.03899.x

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Haberfehlner H, Goudriaan M, Bonouvrié LA, Jansma EP, Harlaar J, Vermeulen RJ, et al. Instrumented assessment of motor function in dyskinetic cerebral palsy: a systematic review. J NeuroEng Rehabil. (2020) 17:39 doi: 10.1186/s12984-020-00658-6

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Thongpanja S, Phinyomark A, Phukpattaranont P, and Limsakul C. Mean and median frequency of EMG signal determine muscle force based on time-dependent power spectrum. Elektronika Ir Elektrotechnika. (2013) 19:51–6 doi: 10.5755/j01.eee.19.3.3697

CrossRef Full Text | Google Scholar

26. Guimaraes AC, Herzog W, Allinger TL, and Zhang YT. The EMG-force relationship of the cat soleus muscle and its association with contractile conditions during locomotion. J Experimental Biol. (1995) 198:975–87 doi: 10.1242/jeb.198.4.975

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Kuriki HU, de Azevedo FM, Takahashi LSO, Mello EM, de Varia Negrão Filho R, and Alves N. The relationship between electromyography and muscle force. In: Schwartz M, editor. EMG Methods for Evaluating Muscle and Nerve Function. IntechOpen (2012). Available online at: https://www.intechopen.com/books/emg-methods-for-evaluating-muscle-and-nerve-function/the-relationship-between-electromyography-and-muscle-force (accessed August 19, 2020).

Google Scholar

28. Michelsen JS, Lund MC, Alkjær T, Finni T, Nielsen JB, and Lorentzen J. Wearable electromyography recordings during daily life activities in children with cerebral palsy. Dev Med Child Neurol. (2020) 62:714–22 doi: 10.1111/dmcn.14466

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Ge Z, Prasad PWC, Costadopoulos N, Alsadoon A, Singh AK, and Elchouemi A. Evaluating the accuracy of wearable heart rate monitors. In: 2016 2nd International Conference on Advances in Computing, Communication, & Automation (ICACCA)(Fall). Piscataway, NJ: IEEE (2016). p. 1–6. doi: 10.1109/ICACCAF.2016.7748986

CrossRef Full Text | Google Scholar

30. Soon S, Svavarsdottir H, Downey C, and Jayne DG. Wearable devices for remote vital signs monitoring in the outpatient setting: an overview of the field. BMJ Innovat. (2020) 6:55–71. doi: 10.1136/bmjinnov-2019-000354

CrossRef Full Text | Google Scholar

31. Behboodi A, Zahradka N, Wright H, Alesi J, and Lee SCK. Real-time detection of seven phases of gait in children with cerebral palsy using two gyroscopes. Sensors. (2019) 19:2517. doi: 10.3390/s19112517

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Côté-Allard U, Gagnon-Turcotte G, Laviolette F, and Gosselin B. A low-cost, wireless, 3-D-printed custom armband for sEMG hand gesture recognition. Sensors. (2019) 19:2811 doi: 10.3390/s19122811

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Raja SN, Carr DB, Cohen M, Finnerup NB, Flor H, Gibson S, et al. The revised international association for the study of pain definition of pain: concepts, challenges, and compromises. PAIN. (2020) 161:1976–82 doi: 10.1097/j.pain.0000000000001939

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Hofmeyer AT, Scott CM. Moral geography of focus groups with participants who have pre-existing relationships in the workplace. Int J Qual Methods. (2007) 6:69–79. doi: 10.1177/160940690700600207

CrossRef Full Text | Google Scholar