Usability-Focused Development and Usage of NeoTree-Beta, an App for Newborn Care in a Low-Resource Neonatal Unit, Malawi

Background: Neonatal mortality is high in low-resource settings. NeoTree is a digital intervention for neonatal healthcare professionals (HCPs) aiming to achieve data-driven quality improvement and improved neonatal survival in low-resource hospitals. Optimising usability with end-users could help digital health interventions succeed beyond pilot stages in low-resource settings. Usability is the quality of a user's experience when interacting with an intervention, encompassing their effectiveness, efficiency, and overall satisfaction.

Objective: To evaluate the usability and usage of NeoTree beta-app and conduct Agile usability-focused intervention development.

Method: A real-world pilot of NeoTree beta-app was conducted over 6 months at Kamuzu Central Hospital neonatal unit, Malawi. Prior to deployment, think-aloud interviews were conducted to guide nurses through the app whilst voicing their thoughts aloud (n = 6). System Usability Scale (SUS) scores were collected before the implementation of NeoTree into usual clinical care and 6 months after implementation (n = 8 and 8). During the pilot, real-world user-feedback and user-data were gathered. Feedback notes were subjected to thematic analysis within an Agile “product backlog.” For usage, number of users, user-cadre, proportion of admissions/outcomes recorded digitally, and median app-completion times were calculated.

Results: Twelve overarching usability themes generated 57 app adjustments, 39 (68%) from think aloud analysis and 18 (32%) from the real-world testing. A total of 21 usability themes/issues with corresponding app features were produced and added to the app. Six themes relating to data collection included exhaustiveness of data schema, prevention of errors, ease of progression, efficiency of data entry using shortcuts, navigation of user interface (UI), and relevancy of content. Six themes relating to the clinical care included cohesion with ward process, embedded education, locally coherent language, adaptability of user-interface to available resources, and printout design to facilitate handover. SUS scores were above average (88.1 and 89.4 at 1 and 6 months, respectively). Ninety-three different HCPs of 5 cadres, recorded 1,323 admissions and 1,197 outcomes over 6 months. NeoTree achieved 100% digital coverage of sick neonates admitted. Median completion times were 16 and 8 min for admissions and outcomes, respectively.

Conclusions: This study demonstrates optimisation of a digital health app in a low-resource setting and could inform other similar usability studies apps in similar settings.

Introduction

Usability can be defined as “the quality of a user's experience when interacting with a product, encompassing effectiveness, efficiency, and overall satisfaction of the user” (1). Usability is crucial to the success of interactive healthcare applications (2) beyond pilot stages, for efficiency, avoiding staff burnout (3), patient safety, and quality of care.

The usability of digital health interventions can be measured quantitatively and qualitatively. Quantitative measures include the System Usability Scale (SUS) created by John Brooke in 1986 (4), the industry standard for the quantitative assessment of usability (3), technologies (5), and products (6). SUS is a 10-question assessment proven reliable (7) and valid in high resource settings (8, 9). Scores of more than 68 are considered acceptable or above average. Although widely used and recognised, SUS does not explore human idiosyncrasies or delineate why particular usability issues occur during a user's experience (10).

User experience is termed as “UX” and Neilson's landscape of UX research methodologies (11) for understanding these issues better, includes scripted “think-aloud” usability studies, where an early prototype is tested in theoretical conditions before being released to the real world. Participants are asked to vocalise their thoughts and experiences while user-testing an intervention (12) generating qualitative feedback regarding the “what” but also the “why” of usability issues. This in combination with SUS has been shown to be more effective at explaining usability compared with SUS alone (2, 12). User Centred Design (UCD) and Agile are both established UX philosophies involving the user-focused iterative cycles of testing and refinement. While UCD is designer-led, centred on the end-user achieving their goals, Agile is more the remit of developers, focusing on the early and continuous delivery of working software to bring commercial value as early as possible (13). An Agile “product backlog” can structure the prioritisation of the smallest functional increments of work (“User-Stories”) within larger bodies of work (“Epics”), which can then be efficiently worked on simultaneously by different “scrum-teams.” Agile-UXD approaches are now being applied more frequently to the development of health apps for the patients' own self-management of diabetes (14), paediatric concussion (15), weight management (16), and smoking cessation (17). Reports of their use for health apps in low-resource hospital settings, where usability may be particularly important, are less common. Preliminary examples of Agile development in low resource settings include the development of a smart phone app for perinatal monitoring by Guatemalan midwives (18).

The NeoTree (Figure 1) is a recently developed, neonatal digital-health system, centred around an app which supports Health Care Professionals (HCP) working in low-resource neonatal units (20, 21). NeoTree aims to improve neonatal care and reduce newborn mortality by bringing evidence-based guidelines, algorithms, and digital data capture to the bedside. Within the NeoTree app, nurses complete digital forms, documenting admission, and discharge details for each sick newborn admitted to a neonatal unit. The app asks them to complete one field/question per page of the app. Figure 2 maps out the sections of the digital admission form that users must complete as they progress through the app, such as the embedded emergency clinical algorithm. Figure 3 shows a screen flow of the first nine pages of the app, demonstrating how a user is prompted to resuscitate a newborn who is not breathing and how to manage various newborn emergencies should they occur during admission. In this way, as the user completes the app, they receive decision support, prompts, and management support on how to manage a sick neonate according to best practise and guidance. Table 1 summarises the functions, aim, and goal of NeoTree to date.

FIGURE 1

Figure 1. NeoTree system. Copyright 2021 Mgusha et al (19).

FIGURE 2

Figure 2. Map of NeoTree App.

FIGURE 3

Figure 3. Example NeoTree screen flow (Screens 1–9; emergency triage and vital signs).

TABLE 1

Table 1. Neotree functions and purpose overview.

Prior to being implemented in Malawi and Zimbabwe central hospitals (19, 22), the NeoTree app was piloted in Zomba Central Hospital (ZCH), Malawi (20). This 1-month pilot included some preliminary usability testing which resulted in early adjustments to the app and produced the beta-app, “Minimum Viable Product 1” (MVP1), the most basic working version that can be safely launched and tested (23). Although mean SUS score for this version was 86.1 (24) (relatively high), this was in the context of a small preliminary phased pilot in a district general hospital (ZCH). Since usability varies with context (4) and NeoTree was now to be implemented permanently, in a new tertiary referral centre (Kamuzu Central Hospital (KCH), Malawi), the app now required further user-testing and refinement.

The overall aim of this study was therefore to investigate the usability of the NeoTree β-app (MVP1) from both qualitative and quantitative perspectives. Objectives were to first conduct qualitative (UXD/Agile) usability testing of MVP1 and apply themes to the usability-focused development of MVP2. Second, to conduct a quantitative usability testing of MVP1 using SUS. Third, to evaluate the usage of the NeoTree β-app (MVP2) during the first 6 months of deployment in a Malawian tertiary neonatal unit.

Methods

Setting

Malawi is a low-income country in sub-Saharan Africa with a high-neonatal mortality of 20 per thousand live births (25). This study was conducted in Ethel Mutharika Neonatal Unit at KCH, Lilongwe which is one of four Central hospitals in Malawi. Central hospitals in Malawi are tertiary referral centres providing the highest level of medical care. The neonatal unit admits 5–10 sick or small neonates each day with a capacity of 80-beds, serving a population of ~5 million people (26). The layout of the unit includes four separate wards: “High-risk” for unstable neonates requiring respiratory support, “Low-risk” for stable infants requiring feeding support, “Isolation” for patients requiring isolation, and a ward for Kangaroo Mother Care. There are 10 permanent neonatal staff providing care in the unit, such as continuous positive airways pressure, phototherapy, nasal cannula oxygen, and intravenous and oral medications. Surgery is available for common neonatal surgical problems including some congenital anomalies.

Study-Design

This was a mixed methods study that included theoretical think-aloud (12) usability interviews prior to app deployment, a real-world usability pilot over 6 months, SUS scores collected before and after app deployment (baseline and end-line), and usage metrics exported from the app at the end of the pilot (Figure 4).

FIGURE 4

Figure 4. Study design.

Participants and Recruitment

Think-aloud interviews were conducted with 6 neonatal nursing staff prior to app deployment, because 85% of usability insights are revealed by user-testing with five participants (27, 28). Participants were recruited via WhatsApp using a purposive sampling method. Inclusion criteria were employment as a permanent member of neonatal nursing staff and no previous use of NeoTree app prior to the study. Exclusion criteria were any temporary staff or cadres not responsible for admitting neonates (e.g., clinicians). Information sheets were provided to potential participants in hardcopy or via WhatsApp. For the real-world pilot, participants were a convenience sample of any HCPs using the NeoTree during the first 6 months of rollout. For SUS we recruited 8 neonatal nurses [as sample sizes of 8 for SUS are 75% reliable (7)]. Inclusion criteria were that participants were permanent members of neonatal nursing staff, working at KCH. For baseline SUS1, they must have used the NeoTree app on just one occasion and for end-line SUS2 they must have used NeoTree during the real-world pilot. Exclusion criteria were any member of staff being paid by the NeoTree research team. For usage, participants were any HCP that used NeoTree during the real-world pilot.

Procedure

First think-aloud interviews were conducted with each participant as they “user-tested” the NeoTree β-app (MVP1) (Figure 5). Hardware are shown in Figure 6. Interviews were conducted by the first Author in English (the working language of Malawi) according to an adapted UX topic guide (Supplementary Material 1). After completing consent forms, each participant was assigned a participant number and asked to sit in the centre of the room, holding and looking at a tablet device containing the NeoTree app. Interviews were conducted in the absence of a sick neonatal patient; hence a senior neonatal nurse (PD) provided an imaginary patient scenario. Together, first author and neonatal nurse guided participants through a digital admission and discharge form, and how to print hard copies. This allowed participants to progress through the app, in a timely manner without struggling to think up example patient data. Whilst a degree of walking-through was required for Malawian participants, unused to theoretical testing, prompts were limited to non-leading questions, encouraging users to think-aloud and verbalise their thoughts as much as possible. A second researcher (THB) typed participants' voiced thoughts into an excel spreadsheet with timestamps, whilst video recording the session, making sure to capture both the participants hands and face as they used the app and the clock on the videoing device. Both the first author and second researcher could also voice observations or comments during the sessions, but they allowed participants to speak uninterrupted first, so as not to bias their responses. All three facilitators checked through the spreadsheet of usability notes for sense at the end of each session, adding any feedback that might have been missed. Upon leaving each usability interview, participants were issued with a training manual (to support their ongoing use of NeoTree), a certificate of attendance, and an appropriate cash remuneration (29).

FIGURE 5

Figure 5. App usability interviews.

FIGURE 6

Figure 6. NeoTree hardware.

Second, after making suggested changes from think-aloud sessions to the app, the real-world pilot was conducted. Six tablet devices containing the NeoTree app were deployed on the ward (at month 0: Figure 4) with chargers, extension leads, network router, and Wi-Fi printer. Users were informed that NeoTree would be a part of usual care and should be used for every baby at the point of admission and discharge, a policy supported by the heads of department (MC and YM). New users that had not attended the usability sessions were given induction training on the ward by a NeoTree ambassador and issued with a certificate and a training manual. User and patient data collected during the first 6 months of app deployment, were exported pseudo-anonymised from the tablets on a weekly basis to a cloud database, where admission and outcome forms were linked by NeoTree identification number. Three “NeoTree Ambassadors” already working in full time rolls (Nurse, Clinical Officer, and Paediatrician) at the hospital were paid a supplement to their usual salaries to support activities, provide technical support, supervision, and system maintenance. During the first 6 months of roll out, all users (permanent and temporary) were asked to report ad-hoc feedback regarding their experience using NeoTree either to the ambassadors or to the first author who recorded any feedback notes in a notebook and spreadsheet.

Thirdly, participants were asked to complete SUS score cards (4) (Supplementary Material 2), in the absence of the research team without signatures to maintain anonymity. Score cards included a Chichewa translation (the most frequently spoken Malawian language) as a helpful but not essential addition to ease understanding. These had been translated and back translated to check consistency.

Finally, four measures of usage were calculated (Table 2). For “user-count” and “cadre,” signatures and cadres recorded by HCPs at the end of NeoTree forms were exported. To provide a denominator for “coverage,” access to aggregate data from ward logbooks was granted by KCH's Health Management Information System department. For “completion-time,” users were asked to record in a ward notebook the start- and end-time of each NeoTree session they completed (displayed on session tabs in the app home page). These data were then copied from the logbook into an excel spreadsheet by an ambassador.

TABLE 2

Table 2. Usage metrics.

Analysis

Analysis of the think aloud interviews combined Braun and Clarke's five thematic analysis phases (30) with the elements of Agile methodology (23, 31, 32) (Figure 7 adapted from Braun and Clarke). First, feedback notes (with timestamps) were transferred into the product backlog where the team could familiarise themselves with the whole dataset, re-consulting the videos where necessary. Second, the first author generated codes phrased as stories using the template “As a (Type of user) I want/need (some goal) so that (some reason) (32).” This approach not only captured why a piece of data was useful but also linked it to different user-roles (32, 33). For example, “As a neonatal nurse, I need Microcephaly in the head-shape dropdown list, so that I can accurately document head shape during an admission.” Third, similar codes were grouped together and assigned theme names which were then coded as “editor” (Epic 1) or “developer” (Epic 2) and prioritised within the backlog according to the clinical need/frequency of theme. App changes were then made in the order of the backlog, either configured by the first author (via the editor) or hard coded by the developers. Changes were only made if they (A) aligned with the best practice and (B) were feasible within team capacity. Overarching themes were then generated by grouping together similar themes under broader headings. In the fourth phase of analysis, themes and overarching themes were reviewed to check responses/themes that were correctly grouped together. Finally, theme names were defined with IT experts, to check that they embodied the correct meaning of the data and made sense to both a technical and research audience.

FIGURE 7

Figure 7. Rapid Agile analysis (Adapted from Braun and Clarke (30).

Usability feedback notes from the real-world pilot were analysed within the same backlog following the same 5-step process. Previous theme labels from the think-aloud interviews were used again where appropriate and corresponding changes implemented in the app iteratively as the pilot continued. Nurses were asked to briefly pause the use of app on the ward when changes were configured via the editor, to avoid crashes in the software. Median SUS [with interquartile range (IQR)] and mean SUS [with standard deviation (SD)] were calculated with 95% confidence intervals (CIs) for each time point. Median SUS scores were compared using Mann–Whitney U-test (if non-parametric) or mean SUS scores were compared using a t-test (if normally distributed). Finally, Microsoft Power-BI and Excel were used to analyse usage metrics as per Table 2.

Results

Six participants attended usability interviews, eight completed SUS1, and eight completed SUS2 (participant details are included in Table 3). Those attending usability interviews included four women and two men, aged 28–43 years. Exact numbers of participants who gave ad-hoc feedback in the real-world pilot were not recorded.

TABLE 3

Table 3. Demographic details and professional and training experience of participants attending think aloud app usability sessions (age and gender are excluded for confidentiality).

Qualitative Usability

Overall, both think-aloud interviews and real-world pilot generated a total of 57 usability stories, 39 (68%) from think aloud analysis (Tables 4, 5), and 18 (32%) from the real-world testing (Table 6). In total, 21 usability themes/issues with corresponding app features were produced and features were added to the app (Tables 4–6). Themes were organised under 12 usability themes under two headings “Usability as an Electronic Medical Record (EMR)” and “Usability relating to clinical care.”

TABLE 4

Table 4. Usability as an electronic medical record—Data collection feature set.

TABLE 5

Table 5. Usability relating to clinical care—clinical care feature set.

TABLE 6

Table 6. Usability of the NeoTree beta app – iterative changes made during real-world pilot.

Six overarching themes related to usability as an EMR (i.e., the data collection function of NeoTree) were generated (Table 4—themes 1–6) with the most common being “exhaustiveness of data schema.” Example of adjustments included changing the field type, e.g., the “reason for admission” field was changed from single choice to multiple-choice because sick newborns often present with more than one problem. Other additions included important missing fields (e.g., a question regarding whether chlorhexidine had been applied to the umbilicus after birth) and relevant missing options in dropdown lists. Other key changes to the EMR included improved field validations, e.g., restricting the patient identification number to certain digits (prevention of errors to support data integrity), making non-essential fields optional or adding “unknown” options (ease of progression through app), advanced field logic to automatically calculate the age of the baby (efficiency of data entry using shortcuts/calculations), removing irrelevant confusing form content (relevancy of content), and adding instructions and signposts to improve navigation (navigation of user interface [UI]). Real-world testing revealed problems with entering babies brought in dead and prompted making the name field optional for these cases as usually they are not given a name.

Six overarching themes relating to the clinical usability of NeoTree were generated, such as confidentiality of identifiable information, cohesion with usual ward processes, embedded education, locally coherent clinical language, adaptability of UI according to available resources, and printout design to facilitate handover (Table 5—themes 7–12). Key examples of changes made to improve the clinical usability of NeoTree included adding a page with common management options and a specific page to facilitate the pathway for surgical babies with gastroschisis/exomphalos (cohesion with usual ward process). Other examples of clinical usability improvements included adding an explanation of the time cut-off between early and late neonatal sepsis and a picture of “strong distal flexion” from the Thompson score for neonatal encephalopathy (embedded education) and the ability for nurses to turn-off the head circumference question when tape-measures were not available (adaptability of UI to available resources). Again, real-world testing resulted in more nuanced iterative refinements, e.g., the addition of the surgeons' contact details to the new gastroschisis page.

SUS

Mean SUS1 prior to the app adjustments was high at 88.1 and SUS2 rose marginally to 89.4 (Table 7). Both scores remained in the 80–90 range with no statistically significant difference between sample means (p = 0.389).

TABLE 7

Table 7. SUS responses and scores.

User-Count

Ninety-three different individuals used NeoTree ß-app during the first 6 months of implementation including 9 permanent staff and multiple temporary staff and students. The 9 permanent staff working on the unit had used the NeoTree app on average 96 times (range 51–195) by the end of the first 6 months.

User-Cadre

Out of 1,323 admissions, cadre data were available for 1,181 admissions which were completed by five different cadres: Nurse Midwife technicians (417, 35%), Nursing Officers (498, 42%), Nursing Students (260, 22%), Medical Students (1, 0.08%), and Clinical Officers (5, 0.42%). Hence, almost all admissions were completed by nursing cadres, one-fifth by students.

Coverage

The NeoTree app captured 1,323 digital admissions compared with 1,298 paper admissions logged by the ward-clerk (25 more admissions via the app = 102% of paper admissions). The NeoTree captured 1,197 digital outcomes compared with 1,180 paper outcomes logged by the ward-clerk (17 more outcomes via the app = 101% of paper outcomes). If “coverage” is the proportion of paper-recorded patient-events captured digitally by NeoTree, then coverage was 100% for both admissions and outcomes with NeoTree events slightly exceeding those logged on paper. Matched outcomes were achieved for 1,197 (90% of NeoTree admissions) leaving 10% with no digital outcome in the database.

Completion Time

The median completion time for NeoTree admission and outcome forms during the last 2 months of the study was 16 (IQR 11.21) and 8 (IQR 5.12) minutes respectively.

Discussion

In this study, we have demonstrated usability-focused optimisation of an app for neonatal HCPs in theoretical and real-world conditions. Rapid Agile analysis allowed abundant, detailed, context-specific, user-led refinements to be applied rapidly and iteratively, expediting deployment, and optimising usability. A new MVP2 was produced, and this was used widely on a low-resource neonatal ward, achieving high usability scores and coverage with short completion times. Multiple usability priorities relating to NeoTree as an EMR, and a clinical aid were uncovered and acted on. Similar bedside apps for low-resource settings may also benefit from this approach to prevent later implementation problems.

Results of this 2019 study can be directly compared with the previous smaller 1-month pilot in 2017 (Table 8). This larger 6-month study included more users (93 vs. 46), achieved higher SUS (89.4 vs. 86.1), higher coverage (100 vs. 70%), and shorter completion times (16 vs. 37 min for an admission). Together these might suggest that the MVP2 produced in this study was more usable than the previous MVP1. However, it is difficult to know the contribution of other factors in this new facility setting, such as the policy to use NeoTree on every baby, and the employment of NeoTree champions to provide technical support. Some of the usability themes and features from the pilot were similar to those of this study, however, this analysis was more sophisticated organising and sorting themes more clearly. Occasionally the same features were added in this study as were previously added (e.g., adding unknown options and making questions non-compulsory to ease progression through the app), but this time they were added to all parts of a now more complex app. Optimising usability in this way without compromising data integrity was more of a challenge here, as we approached the minimum dataset required for each baby. Complete data were now also crucial for quality improvement via new features of the system (diagnostic algorithm and connected dashboard). This trade-off between usability and data integrity could be a relevant consideration for other apps that combine data capture with quality improvement.

TABLE 8

Table 8. Comparison of findings with previous pilot.

Expected usability themes produced in this study relating to NeoTree's data capture function, included “Exhaustiveness of data schema.” An exhaustive digital form that includes all relevant data fields and all possible list options is likely to reduce user frustration and forced errors, while providing a structured, standardised, and systematic user experience. Similar themes from other app studies include that the mPneumonia app allowed users to systematically capture all signs and symptoms of pneumonia (34), a Dutch digital intake tool produced a better overall picture of patients (35) and an Australian EMR prevented nurses from missing out important data inputs (36). The “relevancy of content” theme was also expected because this is consistent with lean product development which recommends that efficient systems should minimise waste (37). This in turn aligns with data protection requirements that data systems should only collect the minimum patient data needed to successfully accomplish a given task (38). For these reasons and to minimise possible harm from the burden of data collection (39), our study supports that adding new fields to clinical apps for low resource settings, should be kept to a minimum and recommends the creation of national minimum datasets already underway in Malawi.

Expected “clinical usability' improvements included adding extra pages to digitally replicate the original paper forms” (theme “cohesion with usual ward process”) aiming to reduce deviation from norms and increase buy-in. This theme corroborates a review of EMR-powered clinical decision support systems, which concluded that integrating new technologies into clinical work practices is essential for successful implementation (40). Moreover, adding educational images to improve clinical usability, was helpful for an app for illiterate midwives in Guatemala, substantiating that picture content may improve usability in low resource settings where there may be reduced literacy, or multiple languages spoken. In addition, “locally coherent clinical language” was true for apps in high resource settings where usability themes included “clarity of wording” (41) and “clear and unambiguous text feedback” (42), highlighting the importance of clear, context-specific written communication for health apps across settings.

Although SUS scores for NeoTree ß were expected to be high (following relatively high scores in the pilot), the lack of statistically significant rise in SUS after adjustments might be explained by four of the same staff completing both SUS1 and SUS2, or a possible inflation of baseline scores and therefore less room for improvement. High baseline SUS scores could have been due to a more competent group of staff using the app this time, or initial excitement and enthusiasm for a new intervention. Indeed, the perceived newness and modernity of the mPneumonia app in Ghana improved acceptance and was associated with the perception of improved care (34). Overall, NeoTree's SUS scores have been consistently higher than those of UK Emergency Department (ED) data systems, which recently scored below average (median SUS 53) (3) suggesting NeoTree may be easier to use than these. However, the UK study compared a much larger array of digital systems, while in Malawi NeoTree is the first such system, hence with no comparisons available subjective SUS scores might be inflated.

While think aloud feedback and SUS scores are both the subjective measures of usability, our objective usage statistics were encouragingly high after refinements were made, adding weight to the qualitative aspects of the study, and demonstrating that NeoTree MVP2 was used widely. The proficiency or efficiency of use, may be indicated by shorter completion times compared with the Zomba pilot (20), but these may also be attributable to more practice time over a 6-month period. In this study similar to the pilot, a significant proportion of admissions (one-fifth) were completed by nursing students which may reflect the apps educational advantages (20) or an ongoing reliance on students for service provision, in a resource-limited setting. The problem of staff shortages endures across African healthcare settings (43–46), and needs more consideration by Governments and NGOs as new technologies are introduced.

Limitations

A limitation of the study might be the reliance on the subjective verbal expressions of a small number of users to guide and shape the application as per Agile. Objective log data of user errors (such as, completing a field and then deleting it or navigating to a page and then navigating back) might strengthen the study. However, log data analytics were not feasible within team capacity in a preliminary phase of the project. Despite the small sample size, qualitative data were rich, prompting abundant app adjustments. Log data analytics could potentially be made available in the future to strengthen follow on usability studies, with perhaps a more structured task analysis, but these fail to capture human idiosyncrasies and require large samples for statistical relevance. Agile methods allow lean iterative development at accelerated velocity, focusing on usability and outcomes with only a few participants, and are mainstream in industrial organisations. Increasingly academic publications are employing Agile methods. For example, a Kenyan mixed-method study combined qualitative interviews with human centred design workshops to engage users in designing and tailoring an intervention to fit the low-resource context (47).

Other limitations include that the interviews were not transcribed verbatim, and this might have limited the qualitative analysis by allowing bias during summarisation or interpretation of participant responses. However, the interview dialogue was relatively sparse and usability notes often related to observed interactions with the UI (such as clicking the wrong button), hence it was decided that transcribing the videos would not add significantly to the richness of data. Completion times were recorded manually from the app home page that may have been affected by human error. These data are amenable to automation in future studies.

Conclusion

This study show-cases usability-focused development of a digital app for neonatal HCPs in a low-resource neonatal unit, combining Agile methods with thematic analysis to expedite deployment and allow iterative user centric refinement during real-world use. System usability remained high after refinements were made and usage was high with 93 neonatal nurses using the app on 1,323 babies. Coverage of admissions and discharges exceeded numbers recorded on paper and completion times were shorter than a previous pilots. Results suggest that NeoTree was now tailored enough for successful use on a permanent basis and the large number of qualitative usability insights and subsequent refinements, promotes the importance of usability for digital intervention development alongside acceptability and feasibility. This study could inform the optimisation, uptake, and implementation of similar apps for other low-resource hospitals as they digitalise over the coming decades.

Data Availability Statement

The data analysed in this study are part of an MDRes dissertation at UCL. For more information please contact the corresponding author.

Ethics Statement

Ethical approval for this study was confirmed with the University of Malawi, College of Medicine Research Council (P.02/19/2613) and the UCL Research Ethics Board (Ref. 6681/001).

Author Contributions

CC conceptualised and conducted the study, analysed the data, and wrote the manuscript. MC is Malawi principle investigator, advised on study design, assisted with ethical approval, supported app implementation, and revised the manuscript. YM was chief NeoTree Ambassador overseeing KCH NeoTree activities and edited the manuscript. PD was also NeoTree Ambassador, facilitated usability interviews, contributed to analysis, and supervised the real-world pilot. TH-B was project manager, facilitated usability interviews, contributed to analysis, and oversaw NeoTree logistics. CN contributed to the conceptualisation and study design, analysis and edited the manuscript. YS advised on data analysis. DN contributed to data cleaning and quantitative analysis. KG contributed to analysis by reviewing usability themes. ML, FL, and MH supervised CC in her MD thesis, advising on study design and methodology, supervising CC in the field (remotely), and editing the manuscript. MH is principal investigator and contributed to the conceptualisation of the study. All authors contributed to the article and approved the submitted version.

Funding

This study was completed as part of an MDRes thesis for CC which was funded partly by a clinical fellow post at UCL, the Naughton Cliffe Matthews grant (PI MH) and partly by charity donations to the NeoTree charity. This research was funded in part by the Wellcome Trust [215742_Z_19_Z]. For the purpose of open access, author has applied a CC BY public copyright license to any Author Accepted manuscript version arising from this submission.

Conflict of Interest

CN was employed by Spinspire Consultants. KG acquired her Agile certification from ICAgile but was not employed by ICAgile at the time of this work. YS was employed by Snowplow Analytics.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We would like to thank the infants, mothers, guardians, families, and parents who welcomed NeoTree at KCH. Thank you to clinicians Prince Magawa, Bentry Tembo, and the late Dr. Kazembe who championed NeoTree implementation at KCH and to all the staff at KCH who used NeoTree willingly. Thank you to the doctors at Baylor College who supported recruitment of NeoTree ambassadors. Thank you to MC for her vital leadership and support at KCH, and in aligning us with the Ministry of Health and other stakeholders. Thank you to the HMIS department at KCH for providing logbooks for comparison data. Thank you to T-HB for providing project management and logistics expertise for no fee, and to Gerry Douglass for providing accommodation and technical guidance from the Institute of Global Health, Lilongwe. Thank you to Matteo Giaconne, Farah Shair, and Jane Minto for their technical expertise on the tech team.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2022.793314/full#supplementary-material

Abbreviations

App, Application; ED, Emergency Department; HCH, Harare Central Hospital; HCP, Health Care Professional; KCH, Kamuzu Central Hospital; MCL, Multiple Choice List; MVP, Minimum Viable Product; SUS, System Usability Score; UCD, User Centred Design; UX, User Experience; UI, User Interface; ZCH, Zomba Central Hospital.

References

1. Home, | Usability,.gov. Available online at: https://www.usability.gov/ (accessed January 9, 2020).

2. Jaspers MWM. A comparison of usability methods for testing interactive health technologies: methodological aspects and empirical evidence. Int J Med Inform. (2009) 78:340–53. doi: 10.1016/j.ijmedinf.2008.10.002

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Bloom BM, Pott J, Thomas S, Gaunt DR, Hughes TC. Usability of electronic health record systems in UK EDs. Emerg Med J. (2021) 38:410–5. doi: 10.1136/emermed-2020-210401

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Brooke J. SUS - A quick and dirty usability scale. Usability Eval Ind. (1996) 189:4–7.

Google Scholar

5. Bangor A, Kortum PT, Miller JT. An empirical evaluation of the system usability scale. Intl J Hum Computer Interact. (2008) 24:574–94. doi: 10.1080/10447310802205776

CrossRef Full Text | Google Scholar

6. Kortum PT, Bangor A. Usability ratings for everyday products measured with the system usability scale. Int J Hum Comput Interact. (2013) 29:67–76. doi: 10.1080/10447318.2012.681221

CrossRef Full Text | Google Scholar

7. Tullis TS, Stetson JN. A Comparison of Questionnaires for Assessing Website Usability ABSTRACT: Introduction. Usability Prof Assoc Conf (2004) 1–12.

Google Scholar

8. Martins AI, Rosa AF, Queirós A, Silva A, Rocha NP. European Portuguese validation of the System Usability Scale (SUS). In: Procedia Computer Science. Sankt Augustin: Elsevier B.V. (2015). p. 293–300. doi: 10.1016/j.procs.2015.09.273

CrossRef Full Text | Google Scholar

9. Peres SC, Pham T, Phillips R. Validation of the system usability scale (sus): sus in the wild. In: Proceedings of the Human Factors and Ergonomics Society. Los Angeles, CA: SAGE PublicationsSage CA (2013). p. 192–6.

Google Scholar

10. Sauro J. Measuring Usability With The System Usability Scale (SUS). Userfocus (2016). Available online at: https://www.userfocus.co.uk/articles/measuring-usability-with-the-SUS.html (accessed May 4, 2021).

PubMed Abstract

11. Rohrer C. When to Use Which User-Experience Research Methods. Nielsen Norman Group (2014). p. 1–7. Available online at: http://www.nngroup.com/articles/which-ux-research-methods/ (accessed October 05, 2019).

Google Scholar

12. Broekhuis M, van Velsen L, Hermens H. Assessing usability of eHealth technology: a comparison of usability benchmarking instruments. Int J Med Inform. (2019) 128:24–31. doi: 10.1016/j.ijmedinf.2019.05.001

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Agile and User-Centered Design | ThoughtWorks. Available online at: https://www.thoughtworks.com/insights/blog/agile-and-user-centered-design-0 (accessed April 20, 2021).

14. Modave F, Bian J, Rosenberg E, Mendoza T, Liang Z, Bhosale R, et al. DiaFit: the development of a smart App for patients with type 2 diabetes and obesity. JMIR Diabetes. (2016) 1:e5. doi: 10.2196/diabetes.6662

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Sandhu H, Wilson K, Reed N, Mihailidis A. A mobile phone app for the self-management of pediatric concussion: development and usability testing. J Med Internet Res. (2019) 21:e12135. doi: 10.2196/12135

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Rivera J, McPherson AC, Hamilton J, Birken C, Coons M, Peters M, et al. User-centered design of a mobile app for weight and health management in adolescents with complex health needs: qualitative study. J Med Internet Res. (2018) 2:e7. doi: 10.2196/formative.8248

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Vilardaga R, Rizo J, Zeng E, Kientz JA, Ries R, Otis C, et al. User-centered design of learn to quit, a smoking cessation smartphone app for people with serious mental illness. J Med Internet Res. (2018) 6:e2. doi: 10.2196/games.8881

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Martinez B, Hall-Clifford R, Coyote E, Stroux L, Valderrama CE, Aaron C, et al. Agile development of a smartphone app for perinatal monitoring in a resource-constrained setting. J Health Inform Dev Ctries. (2017) 11:1–2. Available online at:http://www.jhidc.org/index.php/jhidc/article/view/158/212

PubMed Abstract | Google Scholar

19. Mgusha Y, Nkhoma DB, Chiume M, Gundo B, Gundo R, Shair F, et al. Admissions to a low-resource neonatal unit in malawi using a mobile app and dashboard: a 1-year digital perinatal outcome audit. Front Digit Heal. (2021) 3:761128. doi: 10.3389/fdgth.2021.761128

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Crehan C, Kesler E, Nambiar B, Dube Q, Lufesi N, Giaccone M, et al. The NeoTree application: developing an integrated mHealth solution to improve quality of newborn care and survival in a district hospital in Malawi. BMJ Glob Heal. (2019) 4:860. doi: 10.1136/bmjgh-2018-000860

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Crehan C, Kesler E, Chikomoni IA, Sun K, Dube Q, Lakhanpaul M, et al. Admissions to a low resource neonatal unit in Malawi using a mobile App: a digital perinatal outcome audit. JMIR mHealth uHealth. (2020) 8:e16485. doi: 10.2196/16485

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Gannon H, Chimhuya S, Chimhini G, Neal SR, Shaw LP, Crehan C, et al. Electronic application to improve management of infections in low-income neonatal units: pilot implementation of the NeoTree beta app in a public sector hospital in Zimbabwe. BMJ Open Qual. (2021) 10:e001043. doi: 10.1136/bmjoq-2020-001043

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Agile Alliance. Available online at: https://www.agilealliance.org/ (accessed May 3, 2021).

Google Scholar

24. Bangor A, Kortum P, Miller J. Determining what individual SUS Scores mean: adding an adjective rating scale. J Usability Stud. (2009) 4:114–23.

Google Scholar

25. World Bank. World Bank Data, Under five years and Neonatal mortality trends. (2018). Available online at: https://data.worldbank.org/indicator (accessed July 12, 2018).

26. Yardley L, Morrison L, Bradbury K, Muller I. The person-based approach to intervention development: application to digital health-related behavior change interventions. J Med Internet Res. (2015) 17:e30. doi: 10.2196/jmir.4055

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Why You Only Need to Test with 5 Users. Available online at: https://www.nngroup.com/articles/why-you-only-need-to-test-with-5-users/ (accessed September 22, 2020).

Google Scholar

28. What sample size do you really need for UX research? | BetterUX Blog. Available online at: https://www.userzoom.com/blog/what-sample-size-do-you-really-need-for-ux-research/ (accessed September 22, 2020).

29. Gordon SB, Chinula L, Chilima B, Mwapasa V, Dadabhai S, Mlombe Y. A Malawi guideline for research study participant remuneration [version 2; referees: 2 approved] (2018). Available online at: https://doi.org/10.12688/wellcomeopenres.14668.1 (accessed September 23, 2020).

PubMed Abstract | Google Scholar

30. Braun V, Clarke V. Using thematic analysis in psychology. Qual Res Psychol. (2006) 3:77–101. doi: 10.1191/1478088706qp063oa

CrossRef Full Text | Google Scholar

31. A Quick Introduction to Agile UX Design. Available online at: https://www.webfx.com/blog/web-design/agile-ux-design/ (accessed May 26, 2020).

32. User Stories and User Story Examples by Mike Cohn. Available online at: https://www.mountaingoatsoftware.com/agile/user-stories [accessed October 6, 2002).

33. Landis-Lewis Z, Kononowech J, Scott WJ, Hogikyan RV, Carpenter JG, Periyakoil VS, et al. Designing clinical practice feedback reports: three steps illustrated in Veterans Health Affairs long-term care facilities and programs. Implement Sci. (2020). 15:7. doi: 10.1186/s13012-019-0950-y

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Ginsburg AS, Agyemang CT, Ambler G, Delarosa J, Brunette W, Levari S, et al. MPneumonia, an innovation for diagnosing and treating childhood pneumonia in low-resource settings: a feasibility, usability and acceptability study in Ghana. PLoS ONE. (2016) 11:e0165201. doi: 10.1371/journal.pone.0165201

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Van Leeuwen LM, Pronk M, Merkus P, Goverts ST, Anema JR, Kramer SE. Barriers to and enablers of the implementation of an ICF-based intake tool in clinical otology and audiology practice—a qualitative pre-implementation study. PLoS ONE. (2018) 13:e0208797. doi: 10.1371/journal.pone.0208797

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Jedwab RM, Hutchinson AM, Manias E, Calvo RA, Dobroff N, Glozier N, et al. Nurse motivation, engagement and well-being before an electronic medical record system implementation: a mixed methods study. Int J Environ Res Public Health. (2021) 18:1–23. doi: 10.3390/ijerph18052726

PubMed Abstract | CrossRef Full Text | Google Scholar

37. León HCM, Farris JA. Lean product development research: current state and future directions. Eng Manag J. (2011) 23:29–51. doi: 10.1080/10429247.2011.11431885

CrossRef Full Text | Google Scholar

38. GDPR compliance: Meeting EU rules for data privacy. Available online at: https://try.imperva.com/gdprcompliance/?utm_source=google&utm_medium=paidsearch&utm_campaign=jm-fy20q3-nonbrand&utm_content=gs_1771154349_70079615918_349103544887_gdpr&gclid=CjwKCAiA9bmABhBbEiwASb35V8IIdxeuMbdN8lhJD0vIrOK859MlNSxXVAA33YIYiLQWU2fGLKpDDBoCY3oQAvD_BwE (accessed January 25, 2021).

39. Stevenson AG, Tooke L, Edwards EM, Mangiza M, Horn D, Heys M, et al. The use of data in resource limited settings to improve quality of care. Semin Fetal Neonatal Med. (2021) 26:101204. doi: 10.1016/j.siny.2021.101204

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Ali SM, Giordano R, Lakhani S, Walker DM. A review of randomized controlled trials of medical record powered clinical decision support system to improve quality of diabetes care. Int J Med Inform. (2016). 87:91–100. doi: 10.1016/j.ijmedinf.2015.12.017

PubMed Abstract | CrossRef Full Text | Google Scholar

41. Harte R, Quinlan LR, Glynn L, Rodríguez-Molinero A, Baker PM, Scharf T, et al. Human-centered design study: enhancing the usability of a mobile phone app in an integrated falls risk detection system for use by older adult users. JMIR mHealth uHealth. (2017) 5:e71. doi: 10.2196/mhealth.7046

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Murray D, Sedgeworth C, Kinnear M, Diack L. Evaluation of an electronic paediatric intensive care unit (PICU) medication reconciliation (MR) form. Arch Dis Child. (2016) 101:e2.72–2. doi: 10.1136/archdischild-2016-311535.74

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Crehan C, Huq T, Kesler E, Dube Q, Lakhanpaul M, Heys M. A qualitative study of healthcare workers perceived barriers and facilitators to neonatal care in a central hospital in Malawi (Poster Abstract 073). Arch Dis Child. (2018) 103:A30. doi: 10.1136/goshabs.73

CrossRef Full Text | Google Scholar

44. Shao AF, Rambaud-Althaus C, Swai N, Kahama-Maro J, Genton B, D'Acremont V, et al. Can smartphones and tablets improve the management of childhood illness in Tanzania? A qualitative study from a primary health care worker's perspective. BMC Health Serv Res. (2015) 15:1–12. doi: 10.1186/s12913-015-0805-4

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Liese B, Blanchet N, Dussault G. Background Paper The Human Resource Crisis in Health Services In Sub-Saharan Africa. (2003). Available online at: http://www3.who.int/whosis (accessed November 11, 2020).

PubMed Abstract | Google Scholar

46. Mathauer I, Imhoff I. Health worker motivation in Africa: the role of non-financial incentives and human resource management tools. Hum Resour Health. (2006) 4:24. doi: 10.1186/1478-4491-4-24

PubMed Abstract | CrossRef Full Text

47. Triplett N, Munson S, Mbwayo A, Mutavi T, Weiner B, Collins P, et al. Applying human-centered design to maximize acceptability, feasibility, and usability of mobile technology supervision in Kenya: a mixed methods pilot study protocol. Implement Sci Commun. (2021) 2:2. doi: 10.1186/s43058-020-00102-9

PubMed Abstract | CrossRef Full Text | Google Scholar