Multilevel impacts of a pediatric early warning system in resource-limited pediatric oncology hospitals

Background: Pediatric Early Warning Systems (PEWS) reduce clinical deterioration, improve interdisciplinary communication, and provide cost savings; however, little is known about how these impacts are achieved or related. This study evaluates the multi-level impacts of PEWS in resource-limited pediatric oncology centers.

Methods: We conducted 71 semi-structured interviews including physicians (45%), nurses (45%), and administrators (10%) from 5 resource-limited pediatric oncology centers in 4 Latin American countries. Interviews were conducted in Spanish, transcribed, and translated into English. A code book was developed using a priori and inductively derived codes. Transcripts were independently coded by 2 coders, achieving a kappa of 0.8-0.9. Thematic content analysis explored perceived impacts of PEWS at the level of the patient, clinician, healthcare team, and institution.

Results: PEWS improved the quality of attention for patients, reducing morbidity and mortality. Clinicians felt more knowledgeable, confident, and empowered providing patient care, resulting in greater job satisfaction. PEWS affected team dynamics by improving interdisciplinary (ward and intensive care unit) and interprofessional (physicians and nurses) relationships and communication. This ultimately led to institutional culture change with emphasis on patient safety, collaboration with other centers, and receipt of institutional awards. Together, these impacts led to hospital-wide support of ongoing PEWS use.

Conclusions: In resource-limited hospitals, PEWS use results in multi-level positive impacts on patients, clinicians, teams, and institutions, creating a feedback loop that further supports ongoing PEWS use. These findings can guide advocacy for PEWS to various stakeholders, improve PEWS effectiveness, and inform assessment of other interventions to improve childhood cancer outcomes.

Introduction

Hospitalized pediatric oncology patients are at high risk for clinical deterioration, particularly in resource-limited settings (1, 2). Pediatric Early Warning Systems (PEWS) are bedside assessment tools associated with an action algorithm used for early identification of patients at risk for deterioration (3), and have been validated to predict clinical deterioration in hospitals of all resource levels (4–7).

The impacts of PEWS have been demonstrated across multiple levels of hospital care. PEWS decrease clinical deterioration events and pediatric intensive care unit (PICU) utilization (8), and improve the perceived quality of care (9). PEWS have also been shown to foster nursing empowerment and increase confidence in recognizing and managing clinical deterioration (10), improve interdisciplinary communication and relationships (11, 12), and lead to cost savings (13).

While many positive impacts of PEWS have been identified in resource-limited settings, little is known about how these effects are achieved or interrelated. This study explores hospital staff perceptions of the multilevel impacts of PEWS, how they are achieved, and the process by which they facilitate and augment one another.

Methods

This is a secondary analysis of a study designed to identify barriers and enablers to PEWS implementation, and study methods have been previously described in detail (14). The study was approved by the St. Jude institutional review board as a minimal risk and thereby exempt study. Additional approvals were obtained by participating facilities as required. Written participant consent was waived on account of the study’s exempt status; each participant provided verbal consent prior to the start of their interview. The Consolidated Criteria for Reporting of Qualitative Research (COREQ) guidelines were used to maintain rigor of qualitative reporting (15).

Hospital and participant selection

Escala de Valoraciόn de Alerta Temprana (EVAT) is a Spanish-language PEWS validated in pediatric oncology patients (4). Proyecto EVAT is an international collaborative led by St. Jude Children’s Research Hospital (St. Jude) to support PEWS implementation in resource-limited hospitals providing pediatric oncology care in Latin America (16, 17).

Five Proyecto EVAT centers which completed PEWS implementation prior to March 2020 were selected to participate in the study with representation from Mexico, Central America, and South America. Center characteristics are described in Supplemental Table 1. Each center selected a study lead who identified 10-15 participants involved in PEWS implementation, including PEWS implementation leaders, hospital administrators, and staff indirectly involved in utilizing PEWS.

Data collection

An interview guide (Supplemental Figure 1) was designed to identify barriers and enablers to PEWS implementation at participating centers (14). The guide was translated to Spanish, iteratively revised for relevance and clarity, pilot tested with three individuals from hospitals not participating in this study but demonstrative of the target population, and modified based on feedback. Interviews were conducted in participants’ native language (Spanish) by bilingual members of the research team (PE, SG) via a video conferencing platform (WebEx) from June to August of 2020. The interviewers were previously unknown to participants, not affiliated with their hospital, and not involved in PEWS implementation. Audio recordings of the interviews were professionally transcribed, translated to English, and de-identified (removing all names and other identifiers) prior to analysis.

Analysis

A codebook was established using a priori (18) along with inductively-derived codes defined by two authors (AA, GF) through iterative review of nine transcripts. Two authors (AA, GF) independently coded each transcript using MAXQDA software (VERBI GMBH, Berlin, Germany), achieving a kappa of 0.8-0.9. Incongruities in coding were resolved by a third author (DG) serving as an arbitrator.

Three “outcomes” codes were identified to describe perceived impacts of PEWS at the level of the patient, individual, and institution (Supplemental Table 2). Individual outcomes were subsequently split into impacts on the clinician and healthcare team. Thematic content analysis explored participant perceptions of these multilevel impacts of PEWS at their centers. Codes were examined independently and concurrently with constant comparative analysis of transcripts by site, participant role (e.g., clinician vs. non-clinician and nurse vs. physician), and center characteristics (e.g., presence or absence of a dedicated PICU).

Results

Seventy-one interviews were conducted at 5 pediatric oncology centers in Latin America (see Table 1 for participant characteristics). Content analysis revealed perceived benefits of PEWS for patients, clinicians, team dynamics, and institutions. Figure 1 summarizes these multilevel effects and their interplay, which is further described below.

TABLE 1

Table 1 Characteristics of interview participants.

FIGURE 1

Figure 1 PEWS Cycle of Reinforcement. This figure describes staff perception of the impact of PEWS use on patients, clinicians, teams, and institutions. The benefits at each of these levels facilitate, augment, and reinforce the positive outcomes at the other levels and support ongoing PEWS use.

Patient

Participants at all centers described similar benefits of PEWS for patients including higher-quality patient attention, earlier detection of deterioration, and a reduction in morbidity and mortality (Table 2).

TABLE 2

Table 2 Patient outcomes.

Using PEWS required frequent and focused patient assessments leading to increased situational staff awareness and more individualized attention: “we’re not only applying a routine on that patient, there was more specific care depending on their current situation,” (Nurse, Xalapa). Staff explained that, prior to PEWS, deteriorating patients would go undetected for hours as their bed was the last discussed on rounds or because vital signs were checked only once or twice a shift: “before when a child got critical in the [ward] no one was aware of him” (Ward Physician, San Luis Potosi). With the PEWS algorithm, patients were monitored at appropriate intervals based on their clinical status (Supplemental Figure 2). Thus, staff were better able to track the condition of each patient and focus their attention and resources where they were most needed: “they enter the service and the first thing they do is check the sheet and see if someone has a red or yellow [PEWS] so they can start to work on that patient” (Nurse, Lima); “now, the detection of the child is done on time” (Ward Physician, San Luis Potosi).

Greater situational awareness facilitated early detection of clinical deterioration and increased opportunities for prevention of critical illness: “if you see a patient who doesn’t look that bad but he has a yellow [PEWS], it makes you act before; you prevent a bigger complication and it doesn’t depend on what you see but it’s something more objective” (Ward Physician, Lima). As a result, deteriorating patients were identified and treated earlier, leading to fewer unplanned PICU transfers. Patients who did need PICU care were transferred earlier and required fewer interventions: “The patient doesn’t need to go to intensive therapy to get better. In case the patient goes to intensive therapy … he won’t stay too long or need a tube,” (Nurse, San Luis Potosi).

These improvements led to a perceived reduction in the morbidity and mortality of hospitalized patients: “the mortality was highly reduced,” (Ward Physician, San Luis Potosi). Additionally, with early detection, a patient transfer to the PICU was no longer synonymous with death: “Before [PEWS], children with cancer would go to the ICU and it was considered a child with no opportunity, that child should die. Once [PEWS] came, our children began to get out and we started saying a child with cancer doesn’t die, we just transfer him too late” (Nurse, Lima).

Clinician

In addition to improving patient care, PEWS use led to multiple benefits for clinicians, including reduced nursing workload, improved job satisfaction, increased knowledge, and empowerment (Table 3).

TABLE 3

Table 3 Clinician outcomes.

While nurses initially perceived PEWS use as increasing their workload, with continued use, it became part of their workflow, and ultimately, the reduction in deterioration events and earlier PICU transfers due to PEWS was felt to decrease nursing workload as they were caring for fewer critical patients: “with the implementation of [PEWS] we have patients that stay only a few days at the hospital, less patients at ICU, etc. So they see results, they see less work for them” (Administrator, Xalapa). Additionally, some centers leveraged the initial increase in nursing workload to advocate for a reduction in the nurse-to-patient ratio: “Without [PEWS] we couldn’t justify the need of a nurse to take care of 6 children, they used to take care of 10 or 12 before,” (Nurse, INEN).

As a result of using PEWS to positively impact the care of their patients, clinicians experienced greater job satisfaction: “you can intervene your patients early and avoid ICU or even death; that gives you great satisfaction,” (Nurse, San Luis Potosi). Staff members across all disciplines found this to be motivating: “they see that their work is represented in a patient who is discharged in very good conditions, that makes their effort worthy,” (Ward Physician, Lima).

Additionally, many staff members, especially nurses, found that PEWS and the accompanied trainings expanded their knowledge-base: “we used to take signs without knowing what was normal … now with [PEWS], we know how different it is to have a bradycardia or an asymptomatic bradycardia, it changes a lot,” (Nurse, Cuenca). As a result, they were better able to monitor their patients and felt more confident speaking up when necessary to raise an alarm.

As staff gained knowledge and confidence, they reported increasing feelings of empowerment: “little by little the nurses found out they could go beyond with their work in the service, more than just give medicine, prepare chemotherapies, the fact that they could evaluate a patient … makes them feel more educated, with more power for decision,” (Ward Physician, Xalapa). PEWS empowered nurses to take a more active role in patient assessment and management, and physicians felt empowered to contribute to ongoing improvements in patient care: “there is motivation from the resident part knowing the supervision is higher in the entire service” (Ward Physician, San Luis Potosi).

Team

In addition to benefits for patients and clinicians, PEWS led to benefits for the interprofessional team including better communication between healthcare providers and improved interprofessional (e.g., nurses and doctors) and interdisciplinary (e.g., ward and ICU) team dynamics (Table 4).

TABLE 4

Table 4 Team outcomes.

Participants explained that prior to the introduction of PEWS, nurses and physicians did not have common terminology to discuss patient status, resulting in ineffective communication: “Before [PEWS] … we would come to the doctor and say I can see the patient is getting worse, the patient is not well, but it was subjective; the doctor would say maybe you’re just seeing him that way and maybe you’re wrong,” (Nurse, Cuenca). PEWS use provided teams a common language, improving interprofessional and interdisciplinary communication: “with the [PEWS] score, it was so easy with everyone talking about the same thing to detect a patient when he needed to be transferred to the ICU or when they could treat him in the [ward] … now we all speak the same language” (Ward Physician, San Luis Potosi).

Improvements in team dynamics not only enhanced communication but also improved interprofessional and interdisciplinary relationships. Prior to implementation of PEWS, collaboration between professions was minimal; after implementation, nurses and doctors interacted more frequently and the “relationship between them improved a lot” (Ward Physician, Cuenca), “not only professionally, but friendly” (Ward Physician, Lima). PEWS diminished hierarchies between physicians and nurses, and as a result, nurses felt their input was welcome and valued in ways that it was not previously: “that we would all talk the same language and that the nurse would have voice and vote in the evaluation of the patient, it’s been one of the biggest and most successful projects,” (Nurse, San Salvador). Similarly, the use of PEWS facilitated better relationships between the ward and PICU teams: “before it was like we must transfer him to the ICU, I’m scared they won’t accept him, but not anymore … they are more sensitized, more accessible,” (Ward Physician, Lima). Following PEWS implementation, PICU transfers were less chaotic as PICU clinicians were more willing to evaluate and admit patients earlier in the course of illness.

Institution

Perceived institutional impacts of PEWS included cost reduction, a change in hospital culture emphasizing high-quality patient care, receipt of institutional awards, and opportunities for collaboration with other hospitals (Table 5).

TABLE 5

Table 5 Institutional outcomes.

PEWS use reduced hospital costs by decreasing inpatient days and resource utilization: “we are spending less; patients arrive in ICU on time and they don’t need a ventilator, vasopressors, they stay in ICU only one or two days … compared to the times when it was too late for them, they would stay a lot of days in ICU, they needed ventilator, expensive medicine … the before and after is remarkable,” (Nurse, Cuenca).

Additionally, participants at all sites noted that PEWS implementation altered hospital culture to increase emphasis on patient-centered care: “the culture changed, the culture for the whole medical staff to see the patient in a comprehensive way,” (ICU Physician, San Salvador). Staff experience using PEWS demonstrated that clinical deterioration was largely preventable, leading to a hospital-wide focus on patient safety: “[Clinicians] see that his life is in danger or that he could get critical … they visualize that it is very important to be on alert with that patient so he won’t have risks, and applying [PEWS] on all our patients has influenced a lot as part of their safety” (Ward Physician, Xalapa). Quality improvement projects became more common as staff were inspired to explore other strategies to improve patient care: “A lot of us have started to get involved in other quality improvement projects that maybe didn’t exist before [PEWS], but it has helped us and pushed us to work … to motivate ourselves as professionals to keep looking for alternatives for our patients,” (Ward Physician, Lima).

Centers were further motivated by receiving awards honoring their PEWS program from entities such as the Ministry of Health: “all we wanted was to implement [PEWS] and try to give quality to our patients, but it has been recognized by the Ministry, so that’s an achievement bigger than we expected,” (Nurse, San Salvador). Additionally, participation in Proyecto EVAT provided new opportunities to collaborate with other hospitals: “We’re able to visit other countries, know the realities of other people, share experiences, share situations,” (Nurse, Lima).

Cycle of reinforcement

PEWS implementation led to benefits for patients, clinicians, teams, and institutions initiating a feedback loop that reinforced ongoing PEWS use (Figure 1). Recognition of the patient-level benefits of PEWS led to increased buy-in as clinicians were motivated by opportunities to directly improve patient outcomes: “It was the motivation of seeing the children who could have had a fatal ending return to the [ward] in a better condition” (Nurse, Cuenca). Greater job satisfaction and empowerment among staff led to improved interdisciplinary and interprofessional relationships and communication. Hierarchical barriers were reduced and the interprofessional team functioned more cohesively: “we gained friendship and fellowship, which reinforced our work” (Ward Physician, Cuenca). As relations improved, so did the work environment, facilitating a change in hospital culture with implications for staff’s wellbeing and patient safety. Additionally, observed reduction in resource utilization and mortality galvanized support for PEWS among hospital leadership: “Even in the administration field, we can see that if there’s a better response to the patient’s need before he gets critical, this reduces spending and reduces the probability to go to the ICU or get critical or even die” (Ward Physician, San Salvador). This encouraged leadership support for ongoing staff training and expansion of PEWS within the hospital. Over time, PEWS became embedded in the hospital’s culture and workflow, further reinforcing its continued use: “The hospital has accepted [PEWS] as part of the staff’s work, so they give us the sheets, they open the doors for the training” (Nurse, Lima).

Discussion

Our study demonstrates multiple benefits of PEWS implementation for patients, clinicians, healthcare teams, and institutions and the ways in which these benefits modulate and reinforce one another. Similar to prior studies, we found that staff perceived PEWS to reduce adverse events (8), improve quality-of-care (9), increase staff knowledge, confidence, and empowerment (10), improve interprofessional and interdisciplinary communication (11), and reduce hospital costs (13). Our study, however, additionally demonstrates that improvements in patient outcomes increase staff motivation and job satisfaction, and better interpersonal relationships foster an improved work environment leading to changes in hospital culture including increased emphasis on patient-centered care, patient safety, and quality improvement. The use of qualitative methods allowed for this in-depth exploration of the interplay between the multi-level impacts of PEWS and development of an explanatory model for these impacts as understood by staff directly engaged in PEWS use. Our findings can be used to advocate for PEWS implementation to stakeholders at various levels within an institution by focusing on the outcomes most relevant to them.

Implementation and improvement research is important in resource-limited settings where contextual and infrastructural challenges make implementing evidence-based practices more difficult (14). Correct use of any evidence-based practice is integral to assuring impact, and quality of use must be measured and iteratively improved over time. Process evaluation is a strategy to identify and address gaps at each level of an intervention to maximize implementation success and address barriers to successful use (19, 20). Understanding how PEWS impacts are interrelated helps explain how they are achieved. Our study revealed a cycle of reinforcement which outlines the mechanism by which multilevel outcomes contribute to PEWS success. This process evaluation helps identify critical components of effective quality improvement interventions in settings of all resource-levels, creating a framework for implementation and continuous monitoring (19). Furthermore, understanding impacts relevant to different stakeholders can help address specific barriers and inform targeted and contextually-appropriate strategies to improve intervention adoption and use. This approach can help inform the assessment of PEWS and other clinical interventions.

The cycle of reinforcement identified in our study provides a model to promote the sustainability and expansion (scale) of effective quality improvement interventions. Prior work suggests that an institution’s capacity to sustain an evidence-based practice such as PEWS increases with time (21), and the cycle of reinforcement described in this work identifies a potential mechanism to explain this finding. Furthermore, participants identified components of the clinical capacity for sustainability framework (22), which describes an organization’s capacity to sustain evidence-based interventions across seven domains, including engaged stakeholders, outcomes and effectiveness, implementation and training, and workflow integration (23), as important outcomes of PEWS implementation. More work is needed to prospectively evaluate whether these factors contribute to the maintenance of high-quality PEWS use over time and explore possible strategies to promote sustainability of the multi-level benefits of PEWS use.

Our study has several limitations. Key stakeholder interviews have a risk of social desirability bias (24); however, we attempted to mitigate this by using interviewers previously unknown to participants and not involved in PEWS implementation and by explaining the process of interview de-identification to participants. In addition, interview questions were designed to explore barriers and enablers to PEWS implementation rather than its impacts; identified themes regarding PEWS outcomes were largely spontaneously reported by participants, minimizing bias. All data were collected in Spanish with analysis conducted in English, potentially influencing the interpretation of original statements. To minimize inaccuracies, a professional service was used for translation and 20% of transcripts were audited by a bilingual team member (SG) to confirm accuracy. Finally, this study was conducted in one region (Latin America) among pediatric oncology centers, potentially limiting generalizability of study findings to other regions and patient populations. However, diversity of participating hospitals and similarities between our findings and prior literature on PEWS supports the applicability of these findings to other settings.

This study uniquely describes the interplay between the multilevel impacts of PEWS implementation in resource-limited settings. Benefits at the level of the patient, clinician, team, and institution create a cycle of reinforcement that amplifies impact and supports ongoing PEWS use. These findings can guide advocacy for PEWS to different stakeholders, improve PEWS implementation and efficacy, and inform the implementation and evaluation of other quality improvement initiatives to reduce disparities in childhood cancer outcomes globally.

Data availability statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics statement

The studies involving human participants were reviewed and approved by St. Jude Children’s Research Hospital Institutional Review Board. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

AA and DG developed the idea. MP-T, SG, PE, HM-T, AGR, MA, CB, RDC, CH, SJ, JJL, AM, EMo, EPe, and EPi collected the data. AA and DG provided supervision. EMi, GF, and AA conducted the data analyses. EMi, AA, and DG drafted manuscript and prepared the tables and figures. EMi, GF, and AA had full access to all the data in the study and take responsibility for the integrity of the data and the accuracy of the analysis. All authors contributed to the interpretation of the findings, the editing of the article, and the approval of the final submitted version.

Funding

This study was funded by the American Lebanese-Syrian Associated Charities (ALSAC). EMi was supported by grant R25CA23944 from the National Cancer Institute. These funders were not involved in the design or conduct of the study; collection, management, analysis, or interpretation of the data; preparation, review, or approval of the manuscript; or decision to submit the manuscript for publication.

Acknowledgments

We thank the PEWS implementation teams at all Proyecto EVAT centers, including those who participated in this study, as well as the Proyecto EVAT Steering Committee for oversight of this work.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fonc.2022.1018224/full#supplementary-material

References

1. Bhakta N, Force LM, Allemani C, Atun R, Bray F, Coleman MP, et al. Childhood cancer burden: A review of global estimates. Lancet Oncol (2019) 20(1):e42–53. doi: 10.1016/S1470-2045(18)30761-7

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Agulnik A, Cárdenas A, Carrillo AK, Bulsara P, Garza M, Carreras YA. Clinical and organizational risk factors for mortality during deterioration events among pediatric oncology patients in Latin America: A multicenter prospective cohort. Cancer (2021) 127(10):1668–78. doi: 10.1002/cncr.33411

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Brown SR, Martinez Garcia D, Agulnik A. Scoping review of pediatric early warning systems (PEWS) in resource-limited and humanitarian settings. Front Pediatr (2019) 6:410. doi: 10.3389/fped.2018.00410

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Agulnik A, Méndez Aceituno A, Mora Robles LN, Forbes PW, Soberanis Vasquez DJ, Mack R, et al. Validation of a pediatric early warning system for hospitalized pediatric oncology patients in a resource-limited setting. Cancer (2017) 123(24):4903–13. doi: 10.1002/cncr.30951

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Parshuram CS, Duncan HP, Joffe AR, Farrell CA, Lacroix JR, Middaugh KL, et al. Multicentre validation of the bedside paediatric early warning system score: a severity of illness score to detect evolving critical illness in hospitalised children. Crit Care (2011) 15(4):R184. doi: 10.1186/cc10337

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Elencwajg M, Grisolía NA, Meregalli C, Montecuco MA, Montiel MV, Rodríguez GM, et al. Usefulness of an early warning score as an early predictor of clinical deterioration in hospitalized children. Arch Argent Pediatr (2020) 118(6):399–404. doi: 10.5546/aap.2020.eng.399

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Agulnik A, Forbes PW, Stenquist N, Rodriguez-Galindo C, Kleinman M. Validation of a pediatric early warning score in hospitalized pediatric oncology and hematopoietic stem cell transplant patients. Pediatr Crit Care Med (2016) 17(4):e146–53. doi: 10.1097/PCC.0000000000000662

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Agulnik A, Mora Robles LN, Forbes PW, Soberanis Vasquez DJ, Mack R, Antillon-Klussmann F, Kleinman M, et al. Improved outcomes after successful implementation of a pediatric early warning system (PEWS) in a resource-limited pediatric oncology hospital. Cancer (2017) 123(15):2965–74. doi: 10.1002/cncr.30664

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Garza M, Graetz DE, Kaye EC, Ferrara G, Rodriguez M, Soberanis Vásquez DJ, et al. Impact of PEWS on perceived quality of care during deterioration in children with cancer hospitalized in different resource-settings. Front Oncol (2021) 11:660051. doi: 10.3389/fonc.2021.660051

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Graetz DE, Giannars E, Kaye EC, Garza M, Ferrara G, Rodriguez M, et al. Clinician emotions surrounding pediatric oncology patient deterioration. Front Oncol (2021) 11:626457. doi: 10.3389/fonc.2021.626457

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Graetz D, Kaye EC, Garza M, Ferrara G, Rodriguez M, Soberanis Vásquez DJ, et al. Qualitative study of pediatric early warning systems’ impact on interdisciplinary communication in two pediatric oncology hospitals with varying resources. JCO Glob Oncol (2020) 6:1079–86. doi: 10.1200/GO.20.00163

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Demmel KM, Williams L, Flesch L. Implementation of the pediatric early warning scoring system on a pediatric hematology/oncology unit. J Pediatr Oncol Nurs (2010) 27(4):229–40. doi: 10.1177/1043454209358410

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Agulnik A, Antillon-Klussmann F, Soberanis Vasquez DJ, Arango R, Moran E, Lopez V, et al. Cost-benefit analysis of implementing a pediatric early warning system at a pediatric oncology hospital in a low-middle income country. Cancer (2019) 125(22):4052–8. doi: 10.1002/cncr.32436

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Agulnik A, Ferrara G, Puerto-Torres M, Gillipelli SR, Elish P, Muniz-Talavera H, et al. Assessment of barriers and enablers to implementation of a pediatric early warning system in resource-limited settings. JAMA Netw Open (2022) 5(3):e221547. doi: 10.1001/jamanetworkopen.2022.1547

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Tong A, Sainsbury P, Craig J. Consolidated criteria for reporting qualitative research (COREQ): A 32-item checklist for interviews and focus groups. Int J Qual Health Care (2007) 19(6):349–57. doi: 10.1093/intqhc/mzm042

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Agulnik A, Garza M, Gonzalez-Ruiz A, Soberanis D, Rivera J, Martinez A, et al. Successful implementation of a pediatric early warning system (PEWS) in 10 resource-limited pediatric oncology centers in Latin America and the Caribbean. Pediatr Blood Cancer (2019) 66:S512–3. doi: 10.1002/pbc.27989

CrossRef Full Text | Google Scholar

17. St. Jude Global. Available at: https://www.stjude.org/global.html (Accessed March 24, 2022).

Google Scholar

18. The consolidated framework for implementation research – technical assistance for users of the CFIR framework. Available at: https://cfirguide.org/ (Accessed March 24, 2022).

Google Scholar

19. Hulscher MEJL. Process evaluation on quality improvement interventions. Qual Saf Health Care (2003) 12(1):40–6. doi: 10.1136/qhc.12.1.40

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Ojo T, Kabasele L, Boyd B, Enechukwu S, Ryan N, Gyamfi J, et al. The role of implementation science in advancing resource generation for health interventions in low- and middle-income countries. Health Serv Insights (2021) 14:1178632921999652. doi: 10.1177/1178632921999652

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Agulnik A, Malone S, Puerto-Torres M, Gonzalez-Ruiz A, Vedaraju Y, Wang H, et al. Reliability and validity of a Spanish-language measure assessing clinical capacity to sustain paediatric early warning systems (PEWS) in resource-limited hospitals. BMJ Open (2021) 11(10), e053116. doi: 10.1136/bmjopen-2021-053116

PubMed Abstract | CrossRef Full Text | Google Scholar

22. CSAT. PSAT/CSAT. Available at: https://www.sustaintool.org/csat/ (Accessed April 18, 2022).

Google Scholar

23. Malone S, Prewitt K, Hackett R, Lin JC, McKay V, Walsh-Bailey C, et al. The clinical sustainability assessment tool: measuring organizational capacity to promote sustainability in healthcare. Implement Sci Commun (2021) 2(1):77. doi: 10.1186/s43058-021-00181-2

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Grimm P. Social desirability bias. In: Wiley International encyclopedia of marketing edsSheth J., Malhotra N.John Wiley & Sons, Ltd (2010). doi: 10.1002/9781444316568.wiem02057

CrossRef Full Text | Google Scholar