Protocol-driven primary care and community linkage to reduce all-cause mortality in rural Zambia: a stepped-wedge cluster randomized trial

Introduction: While tremendous progress has been made in recent years to improve the health of people living in low- and middle-income countries (LMIC), significant challenges remain. Chief among these are poor health systems, which are often ill-equipped to respond to current challenges. It remains unclear whether intensive intervention at the health system level will result in improved outcomes, as there have been few rigorously designed comparative studies. We present results of a complex health system intervention that was implemented in Zambia using a cluster randomized design.

Methods: BHOMA was a complex health system intervention comprising intensive clinical training and quality improvement measures, support for commodities procurement, improved community outreach, and district level management support. The intervention was introduced as a stepped wedge cluster-randomized trial in 42 predominately rural health centers and their surrounding communities in Lusaka Province, Zambia. Baseline survey was conducted between January–May 2011, mid-line survey was conducted February–November, 2013 and Endline survey, February–November 2015.The primary outcome was all-cause mortality among those between 28 days and 60 years of age and assessed through community-based mortality surveys. Secondary outcomes included post-neonatal under-five mortality and service coverage scores. Service coverage scores were calculated across five domains (child preventative services; child treatment services; family planning; maternal health services, and adult health services). We fit Cox proportional hazards model with shared frailty at the cluster level for the primary analysis. Mortality rates were age-standardized using the WHO World Standard Population.

Results: Mortality declined substantially from 3.9 per 1,000 person-years in the pre-intervention period, to 1.5 per 1,000 person-years in the post intervention period. When we compared intervention and control periods, there were 174 deaths in 49,230 person years (age-standardized rate = 4.4 per 1,000 person-years) in the control phase and 277 deaths in 74,519 person years (age-standardized rate = 4.6 per 1,000 person-years) in the intervention phase. Overall, there was no evidence for an effect of the intervention in minimally-adjusted [hazard ratio (HR) = 1.18; 95% confidence interval (CI): 0.88, 1.56; value of p = 0.265], or adjusted (HR = 1.12; 95% CI: 0.84, 1.49; value of p = 0.443) analyses.

Coverage scores that showed some evidence of changing with time since the cluster joined the intervention were: an increasing proportion of children sleeping under insecticide treated bed-net (value of p < 0.001); an increasing proportion of febrile children who received appropriate anti-malarial drugs (value of p = 0.039); and an increasing proportion of ever hypertensive adults with currently controlled hypertension (value of p = 0.047). No adjustments were made for multiple-testing and the overall coverage score showed no statistical evidence for a change over time (value of p = 0.308).

Conclusion: We noted an overall reduction in post-neonatal under 60 mortality in the study communities during the period of our study, but this could not be attributed to the BHOMA intervention. Some improvements in service coverage scores were observed.

Clinical Trial Registration: clinicaltrials.gov, Identifier NCT01942278.

Introduction

There has been unprecedented progress in global health in the past 20 years mostly driven by external global health actors (1). However, progress has not been uniform, especially in lower- and middle-income countries (LMICs) where the poorest sectors of the population often face the worst outcomes (2). While globally mortality rates have decreased across all age groups over the past five decades, with the largest improvements occurring among children younger than 5 years, the current rates remain unacceptably high. In 2015, about 5.9 million children died before reaching their 5^th birthday; half of these resided in sub-Saharan Africa (3). Similarly, deaths among younger adults continues to increase (4). If current trends continue, more than 44 million lives will have been lost to avoidable death by 2030 (5).

The agenda to develop resilient health systems has become urgent as it has been seen as a key driver to achieving equitable and sustainable health outcomes for all. The UN Sustainable Development Goals (SDG) are ambitious and include targets to radically reduce maternal mortality, to end preventable child mortality, to end deaths from TB, AIDS and malaria, to curtail the burden of non-communicable diseases, and to achieve universal healthcare coverage (6). The services required to achieve these SDGs can only be delivered through a transformation of primary health care delivery. In many LMICs, primary health systems face inadequate financial and human resources and are overwhelmed by the needs of the ever-growing populations they serve. It is clear that system strengthening is needed, but it is unclear how best to do this approach daunting task. Zambia exemplifies the challenges facing LMIC primary health care systems (7).

The Better Health through Mentoring and Assessment (BHOMA) project was a complex public health intervention funded by the Doris Duke Charitable Foundation. BHOMA introduced a protocol-driven primary care and community linkage intervention, which aimed to improve clinical care at the primary care level and increase community demand through improved confidence in the health system. The hypothesis behind the intervention was that a systematic investment in improved clinical care would immediately improve outcomes and ultimately lead to better utilization at community level. Details of the multi-level BHOMA intervention have been published elsewhere (8). Development of the intervention was consultative and addressed most urgent local priorities, including reducing under-five and adult mortality. The primary goal was to reduce all-cause mortality in individuals between 28 days and 60 years old within the study area. We excluded neonates because the interventions necessary to improve survival in this population have been extensively studied. In this paper, we report the primary results of the BHOMA intervention focusing on post-neonatal adult mortality (PN-U60M), post neonatal under-five mortality (PN-U5M), and on service coverage.

Methods

The BHOMA intervention

BHOMA was conducted in 3 predominately rural districts within the Lusaka Province of Zambia. Details of the complex health systems intervention its underlying theory of change have been described in detail in prior publications (7–22). The intervention was based on the idea that the patient-provider interaction is critical to service quality and community trust, and that all aspects of an effective health system should be organized around ensuring that this encounter is effective. Toward this end, BHOMA provided support for establishment of clear clinical management protocols, improved medical record keeping, intensive clinical mentorship, on-the-job training and iterative quality improvement (QI), bolstering the commodities supply chain, and innovative linkages between clinic and community. We briefly recount its salient features here.

Clinical care protocols

We began by convening technical personnel within the Zambian Ministry of health to review existing guidelines for clinical care in primary settings. In general these guidelines comprised local adaptations of the WHO’s Integrated Management of Adolescent and Adult Illnesses (23), Integrated Management of Childhood Illness (24), and Emergency Obstetric and Newborn Care (25). From these guidelines we created simple, step-by-step care protocols that were supported by job aids, including checklists, wall charts, and a system of 7 clinical forms to guide clinicians through standardized patient evaluation and management. The project employed a new cadre of lay workers, known as “clinic support workers” – 2 or 3 individuals per facility – to support care delivery. These staff members assisted with patient navigation and check-in, obtained and documented vital signs, and maintained the clinic’s paper medical record system by ensuring form completion and organized filing.

Support for improved record keeping

The project established an organized medical records system of patient charts that were kept on-site for each patient. At the conclusion of each patient visit, these paper records were transcribed into a comprehensive, purpose-built electronic system by dedicated data entry technicians supported by the study (CommCare® https://www.dimagi.com/commcare/). The system—which used low voltage touch screen computers networked through local cellular providers—included detailed reporting functionality to monitor a set of performance indicators and clinical outcomes. These reports were available on demand to clinic leadership and comprised a key aspect of the quality improvement (QI) intervention (#3, below). At the time of registration, the electronic system assigned each patient to a specific community health worker (CHW) based upon where they live. It also kept track of open/unresolved cases and alerted CHWs through linked cell phones to patients who missed follow-up appointments so they could be traced at home (13).

Training and quality improvement

We formed 6 quality improvement (QI) teams, each comprising a senior clinician, pharmacist, and data technician to guide an intensive effort in refresher training and implementation of the new protocols. An initial 1-month visit to each facility included an assessment of current staffing and equipment needs, strengthening of commodities tracking and requisition procedures, strategizing for improvement in patient flow, training in the clinical protocols, and implementation of improved record keeping. After the initial implementation phase, the QI teams returned to the facility monthly for 3 visits and then quarterly to conduct structured reviews of medical records (focusing on the accuracy of diagnosis and management) and review performance metrics with the facility’s clinical officers, nurses, midwives, and other staff to develop specific goals and plans for improvement.

Support for commodity procurement

BHOMA provided targeted support for essential supplies and equipment needed by site-level clinicians to deliver quality care. This began with a pre-implementation assessment of each participating facility, followed by targeted equipment procurement. Part of the QI teams’ mandate was to strengthen site-level forecasting and ordering.

Improved community linkages

BHOMA worked to increase linkages between each facility and the community it served through active patient referral and follow-up. We recruited more than 200 CHWs who participated in 4 weeks of initial implementation training at their respective facilities. Each CHW was trained in recognition of danger signs, including when and how to refer individuals who met specific criteria to health facilities. We also trained them in how to dispense ferrous sulfate and folic acid for treatment of anemia, prescribe oral rehydration salts for child diarrhea, and give anti-malarial medication.

CHWs were each assigned to a specific zone within the catchment geography of their respective clinic. They made quarterly visits to all households in their zone where they interviewed household members and referred those in need of care. In the home, the electronic system system—installed on their mobile phones—guided the CHWs through a series of questions to which they keyed in responses. If a referral was made, the appointment was registered electronically at the corresponding clinic and not resolved in the CWH’s phone until the patient had presented. All household information, including interval patient outcomes, was transferred back to the clinic servers via mobile phone at the end of each home visit. Quarterly statistics on new pregnancies, illnesses, and deaths were aggregated and provided to neighborhood health committees to facilitate local solutions to health care access.

The CHWs also made ad hoc visits to patient homes when a clinic appointment had been missed. This was facilitated by the electronic health record system, which sent an alert to the mobile phone of the corresponding CWH. The system kept a running tally of each CHW’s open cases and prioritized those who were diagnosed with specified danger signs at their clinic visit.

District level support

Continuously throughout the project—and thus not evaluated by the randomized intervention roll-out—we provided support at the district level in governance, management, site communication, and commodities forecasting. The project supported improved implementation of existing district management tools, including the Health Management Information System (HMIS), and the District Integrated Logistics and Supplies Assessment Tool (DILSAT). We also placed a dedicated pharmacy technician at each District Health Office to strengthen the district’s ordering and supply system for equipment, supplies, diagnostics, and drugs. To ensure early ownership and integration into routine districts functions, the implementation teams worked closely with a designated district clinical lead, who was involved in most trainings and quality improvement visits.

Intervention implementation phases

The intervention was introduced at the site level through structured on-site training and mentorship. During this implementation phase, the QI team spent 4 weeks training staff in a variety of quality-related skills and competences (Table 1). During the first 2 weeks, the QI teams conducted training on diagnosis and management of common illnesses and introduced relevant protocols for clinical case management. In week 2, we trained on patient triage and record keeping using standardized forms. In week 4, we incorporated pregnancy care. All members of staff at target health facility (clinical officers, nurses, midwives, clinic support workers, and CHWs) were engaged in the site-level training.

TABLE 1

Table 1. BHOMA site-level implementation activities.

To assess the effectiveness of our intervention, we conducted three population-based household surveys (community surveys)—one at baseline and two at follow-up. Within each cluster households to be surveyed were randomly selected through a satellite mapping exercise where the catchment area of each facility was geographically outlined. In each cluster, squares of 900 m² were marked within a 3.8 km of the health facility. Computer-generated randomization was used to determine which squares would be visited and the order of visitation. All households in randomly selected squares where the survey was started were visited until the sample size was reached.

Trial design

The BHOMA intervention was delivered in a cluster-randomized fashion (8). The trial—formally categorized as a stepped wedge trial of incomplete design with an implementation period—is registered at clinicaltrials.gov (NCT01942278). Our primary unit of analysis was an implementation cluster, comprising one primary health care facility and the population that it served. All clusters were situated within three rural districts of Lusaka Province, Zambia (Figure 1). In these districts a total of 52 health facilities existed at the start of the trial and were initially eligible for inclusion. Ten clusters were excluded—three military facilities, six pilot sites, and one facility with no trained health care workers (Figure 2). The population estimates of the clusters varied from 1,501 to 44,658 people.

FIGURE 1

Figure 1. Map showing BHOMA intervention districts in Zambia.

FIGURE 2

Figure 2. Schematic showing stepped-wedge trial design, with timing of community and health facility surveys.

Randomization and masking

Forty-two clusters were randomly assigned to start the intervention in seven steps, each step having six clusters, 3 months apart. Randomization of the clusters into the seven steps was stratified by district. At each step, three clusters from district 1, two clusters from district 2 and one cluster from district 3 were randomly assigned to start the intervention. Randomization was done by a statistician (JL) based in London and not involved in implementation, who generated a random implementation schedule using Stata version 15 (StataCorp LLC; College Station, TX). The order of roll out of the intervention could not be blinded.

Outcomes assessment

Outcomes were measured in community and facility level surveys conducted at three time points during the study (Figure 1). Our pre-specified primary outcome—derived from community survey—was age-standardized mortality among individuals aged > 28 days and < 60 years (PN-U60M). Secondary outcomes included post-neonatal under-five mortality (PN-U5M).

The primary outcome of mortality was measured in a random sample of households (120 in the first survey and 300 in surveys 2 and 3). To estimate deaths occurring only within the cluster and to provide monthly mortality rates to allow for analysis of implementation timing in the stepped wedge roll-out, mortality rates were calculated using a retrospective household enumeration. The head of each household (or, if not available, another adult member) was asked to enumerate every person who had stayed the previous night in that household and then to go back month-by-month. We looked back 12 months for survey 1 and 24 months for surveys 2 and 3, identifying those that had been present or absent in each month. The respondent was also asked to include any individuals who had been present in the household for at least 1 month but were no longer present, either because they had left or because they had died. We collected the cause (if known) and date of all reported deaths.

The secondary outcome of PN-U5M was assessed in two ways: first by restricting the enumeration above to individuals under 5 years old and second from a separate birth history collected from women using the standard instrument employed by the Demographic and Health Survey.¹

For each facility, Coverage scores were constructed using the methods of Victora et al. (26) We adopted 2 or 3 domains for each coverage score and calculated the coverage gap. A full list of indicators and their definitions is available in the Appendix (Supplementary Table S1). Information on the coverage indicators was collected from individual questionnaires administered to all available adults in survey 1 and in 120 randomly selected households from the 300 per cluster in surveys 2 and 3. Questionnaires were constructed to align with Demographic and Health Survey tools where possible (27).

Sample size

The sample size was based on standard formula for parallel cluster randomized trials, adjusted for the design effect of a stepped wedge design (based on the number of steps) (28). The plan was to conduct three surveys of the 42 clusters, each recruiting 150 households per cluster, of whom 6 members were assumed aged <60 years, among whom deaths within the household in the last 12 months would be ascertained. Assuming a mortality rate for those age < 60 years of 20/1000 person years and a coefficient of variation of 0.3 there would be at least 90% power to detect a 35% reduction in mortality. Assuming a post-neonatal child mortality rate of 35/1,000 person years (corresponding to under-5 mortality of 168/1,000) and that each participating household would have an average of two children under 5 years of age, there would be at least 90% power to detect a 35% reduction in the secondary outcome of post-neonatal under-5 mortality. After completion of the baseline survey, we observed mortality rates were lower than anticipated. We thus doubled the survey sample size to 300 households per cluster in survey rounds 2 and 3.

Statistical analysis

The primary outcome of all-cause mortality in those aged under 60 years was analyzed by constructing retrospective mortality cohorts, analyzed using a Cox proportional hazards model with shared frailty at the cluster level. The variable for intervention phase was coded as a quantitative variable with values: 0 for the control and intervention start-up phases; ½ for the intervention partial implementation phase; and 1 for the intervention full implementation phase. Sensitivity analyses were: using a Poisson model with a random effect for clustering and a categorical variable for time step; the same Poisson model, but restricted to the same time period as the Cox model; the Cox model excluding trauma deaths; Cox model excluding any persons who had not started the retrospective cohort in the community (i.e., excluding in-migrants); Cox model excluding the 3-month intervention start-up and partial implementation phases. Minimally adjusted models adjusted for district as the stratifying variable, age category (as age-adjusted rates were pre-specified) and calendar time (implicitly in the Cox model or explicitly in the Poisson model). Adjusted models also adjusted for baseline mortality rate at the cluster-level as this showed some imbalance at baseline. Mortality rates were age-standardized using the WHO World Standard Population.

Under-five mortality rates were analyzed firstly using detailed birth histories collected from all mothers in the surveys, which were used to construct retrospective cohorts. These were analyzed using Cox models as for mortality rates for those <60 years. Neonatal deaths were excluded, as were those who were alive and no longer living with the mother. Sensitivity analyses were: excluding the three-month intervention start-up and partial implementation phases; and using the primary analysis for mortality rates for those aged < 60 years, restricted to those aged under 5 years old.

Coverage scores were calculated across five domains (under-fives prevention; under-fives treatment; family planning; maternal health and adult health). Summary measures were calculated for each domain by averaging all scores within that domain; overall coverage was calculated by averaging across the five domains. As all clusters were in the control phase during the baseline survey and intervention phase at subsequent surveys, the time since the cluster joined the intervention phase was used as the exposure variable in 1 year categories. Since the order of clusters joining the intervention phase was randomized, the time since the cluster joined the intervention was also a randomized exposure variable. The coverage scores were measured at two surveys and so analysis accounted for this repeat measurement using generalized estimating equations in linear regression models.

All analyses were performed using Stata SE 15 (StataCorp, College Station, TX, United States). The trial was registered with clinicaltrials.gov, number NCT01942278.

Ethical statement

Permission for the study was sought at community level by traditional leaders such as chiefs and “head men” or counselors in urban areas. The Ministry of Health and district health management teams gave permission for their facilities to be included in the study. For all households randomly selected to take part in the outcome assessment survey, the head of the house or another responsible adult were asked to provide written informed consent for enumeration and collection of household characteristics. Each adult (18 years and older) was asked to complete an individual level questionnaire and to give individual written informed consent. All versions of the protocol were reviewed and approved by the ethics committees of the University of Zambia, London School of Hygiene and Tropical medicine, University of Alabama at Birmingham and subsequently the University of North Carolina, Chapel Hill. The Ministry of Health also reviewed the protocol and gave permission for the study to be undertaken.

Role of the funder

The Doris Duke Charitable Foundation (DDCF) funded the project, was involved in its planning, and monitored its progress. DDCF was not involved in implementation of the intervention or the surveys used to measure study outcomes. Neither was it involved in the interpretation of study results, the preparation of this manuscript, or the decision to submit it for publication.

Results

Data was collected at three time points: Baseline survey was conducted between January–May 2011, mid-line survey was conducted February–November, 2013 and Endline survey, February–November 2015.

Across the three surveys, 30,472 were randomly selected for enumeration. 29,486 households were enumerated, 711 households were absent and 275 households refused consent (Figure 3). The 29,486 households enumerated 138,430 members, giving total person years of 49,240 in the control phase and 74,530 in the intervention phase. No clusters left or joined the study during its duration. Individuals recorded in the household census in the baseline survey were 50% male and 47.5% were aged 0–14 years, 21.4% were aged 15–24 years and 31.1% were aged 25–59 years (Table 2). The population was relatively stable with 84.7% having been in the household every month in the year prior to the survey. Reasonable baseline balance was observed on key characteristics (Table 2), although the weighted average of age-standardized mortality was 3.9 per 1,000 person-years in the control phase and 5.0 per 1,000 person-years in the intervention phase. Some imbalance in the opposite direction was observed when restricted to those under 5 years old, but this estimate was based on only 19 recorded deaths.

FIGURE 3

Figure 3. Consort diagram modified for cluster and stepped wedge.

TABLE 2

Table 2. Baseline characteristics of population enumerated in the control and intervention phases of the study.

Over the period of interest (steps one to seven), the number of deaths recorded per cluster and per intervention phase varied from zero to 17 (Supplementary Table S1). Mortality declined substantially from 3.9 per 1,000 person-years in the pre-intervention period (2011), to 1.5 per 1,000 person-years in the post intervention period (2015; Figure 4). When restricted to the formal trial period, there were 174 deaths over 49,230 person years of follow-up (age-standardized rate = 4.4 per 1,000 person-years) in the control phase compared to 277 deaths over 74,519 person years (age-standardized rate = 4.6 per 1,000 person-years) in the intervention phase (Table 3). There was no clear pattern in mortality hazard observed across the steps (Supplementary Table S2). Overall, there was no evidence for an effect of the intervention in minimally-adjusted (hazard ratio [HR] = 1.18; 95% confidence interval [CI]: 0.88, 1.56; value of p = 0.265), or adjusted (HR = 1.12; 95% CI: 0.84, 1.49; value of p = 0.443) analyses (Table 3). These results remained consistent across a range of sensitivity analyses (Table 4). There was a trend observed for an intervention effect in a planned sensitivity analysis comparing the phase of >1 year full implementation to the control phase (aHR = 0.69; 95% CI: 0.44, 1.07; value of p = 0.093; Table 4).

FIGURE 4

Figure 4. Age-standardized mortality rates by time-step and intervention phase. py, person years; CI, confidence interval.

TABLE 3

Table 3. Analyses of mortality rates by control versus intervention phase.

TABLE 4

Table 4. Analysis of mortality rates by the different phases of implementation.

Whether calculated from the birth cohorts or by restricting the primary analysis to those under-5 years, there was no evidence for an impact on mortality among under-fives (Table 4). However, in one sensitivity analysis there was weak evidence for a negative impact of the intervention on mortality among under-fives (aHR = 2.21; 95% CI: 1.01, 4.84; value of p = 0.048; Table 5).

TABLE 5

Table 5. Analyses of mortality rates among those aged under five years by control versus intervention phase.

TABLE 6

Table 6. Analysis of coverage by the duration of full implementation.

Some, but not all, coverage scores showed statistical evidence of improvement as a result of the intervention, including: an increasing proportion of children sleeping under insecticide treated bed-net (value of p < 0.001); an increasing proportion of febrile children who received appropriate anti-malarial drugs (value of p = 0.039); and an increasing proportion of ever hypertensive adults with currently controlled hypertension (value of p = 0.047; Supplementary Tables S3 and S4). No adjustments were made for multiple-testing and the overall coverage score showed no statistical evidence for a change over time (value of p = 0.308) (Table 6).

Discussion

Our study findings indicate mixed results for the BHOMA health system intervention. While we observed a general decline in post-neonatal under 60 mortality (from 3.9 to 1.5 per 1,000 person-years) over the period of surveillance, we were unable to directly attribute this reduction to our intervention. We similarly did not observe an effect on post-neonatal under 5 mortality. We did, however, observe significant improvement in facility-level coverage scores for a number of key indicators.

Our findings are consistent with prior work in Rwanda and Ghana (29, 30), where health system interventions have been associated with substantial improvement in process indicators. Most prior studies have employed “before and after” comparisons, which require caution in their interpretation because potential bias may be introduced through secular trends. We attempted to mitigate this bias through a cluster randomized stepped wedge design, but cannot completely rule out such an effect.

Another issue to consider when interpreting these findings is the complex and dynamic nature of the intervention that was implemented (11, 12). Our intervention attempted simultaneous intervention upon multiple “building blocks” (7, 11) of the health system, some of which occurred at a level above that of the randomization. For instance, the efforts we made to strengthen management capacity at the district level could not be evaluated through the stepped wedge randomization, and may have biased our outcomes toward the null. Furthermore, the complexity and scope of our multi-faceted intervention may have invited opportunistic adaptation at the district or provincial level with some “leakage” of benefits to study sites prior to formal intervention. For example, our work at the district level led to shared benefit across the control sites thus potentially diluting expected or desired effect (14). We also supported the district planning and supply chains, which could have benefited control sites, even before we introduced the intervention (14, 30). Trials are embedded within an already existing health system which tend to be hierarchical, with primary centers being supervised by districts which are in turn are supervised by provincial health teams who report to the national level. This is true for the Zambian health system. This hierarchical interdependence could potentially have introduced an element of complexity which statistical models in this analysis may not account for (31).

We used retrospective assessment of child survival via birth history analysis, which has potential bias relating to event omission or event displacement recall biases. In addition, there are well-described cultural beliefs in Zambia that prevent people from talking about stillbirth and child death (32). This phenomenon could have contributed to under reporting particularly for infants (31).

An important limitation for our study is inherent in its stepped-wedged design (33). As in our study, stepped-wedge cluster randomized trials are often conducted on a large scale and are therefore sensitive to external challenges that cannot directly be influenced by the researchers such as changes in the local context (33). Our study communities experienced several competing interventions during the study period (7, 11, 14). These could have impacted on the outcome measures we intended (7) (33)

Our intervention may also have been dependent on the temporal nature of the intervention; it may have taken time for the improvements to fully manifest. Delay in achieving the desired effect could result in loss of power for the stepped wedge design, especially if the intervention is not fully effective in the time interval in which the evaluation was conducted 34. Such delays can be caused by a slower than expected intervention rollout or by intrinsic lag between introduction of the intervention and its effect on the outcome. Our trial had two challenges that fit well with this observation. Firstly, we had a 6-month delay in introducing the 5th of the 7 clusters due to administrative issues with funding. A second challenge was related to the outcome we aimed to improve. Improvements in all-cause mortality may have taken more than the 3 months we used per implementation step to manifest. Perhaps longer implementation steps and longer follow-up time would have yielded different results 34. The observation that a few short-term outcomes in our study, such as coverage scores, showed significant improvement across the steps, may support this argument. We have similarly reported positive health systems impact of the BHOMA intervention in the corresponding publication.

Our findings have public health and research significance, first we utilized a gold standard randomized approach to evaluate this multi-faceted intervention. Despite the challenges outlined, this is one of the few complex health system interventions that have employed both a randomized design and mortality outcomes to evaluate efficacy. The fact that our chosen outcome measure of mortality was not demonstrated, but instead demonstrated a change in the process indicators such as coverage and health system indicators (Mutale, BHOMA paper 2), indicate that allowing longer term follow-up is crucial to making definitive conclusion. It will be important for future studies to consider varying periods of follow-up time to in order to estimate minimum time required to establish effect of similar large health system interventions on mortality. Our study had a very short follow-up time and is therefore not able to estimate such lead time. Other modern approaches such as modeling the period and effect size should be considered in future studies.

Finally, the study clearly demonstrate the importance of carefully choosing indicators and allowing sufficient follow-up time when evaluating complex health system interventions. Our findings may be generalizable to other settings but we caution that follow-up time need to be sufficient and that chosen indicators must be realistic within the study time frame in order to demonstrate meaningful health system impact.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material, further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving human participants were reviewed and approved by University of Zambia Bioethics Committee. The patients/participants provided their written informed consent to participate in this study.

Author contributions

WM, HA, JL, SB, RC, MT, AS, NC, and JS participated in the design and implementation of the intervention. JL and SB conducted the analysis. WM drafted the manuscript. All authors contributed to the article and approved the submitted version.

Funding

This work was supported by Doris Duke Charitable Foundation (DDCF).

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2023.1214066/full#supplementary-material

Footnotes

1. ^https://dhsprogram.com

References

1. Marchal, B, Cavalli, A, and Kegels, G. Global health actors claim to support health system strengthening: is this reality or rhetoric? PLoS Med. (2009) 6:e1000059–9. doi: 10.1371/journal.pmed.1000059

CrossRef Full Text | Google Scholar

2. Collaborators, GS. Measuring progress and projecting attainment on the basis of past trends of the health-related sustainable development goals in 188 countries: an analysis from the global burden of disease study 2016. Lancet. (2017) 390:1423–59. doi: 10.1016/S0140-6736(17)32336-X

CrossRef Full Text | Google Scholar

3. Liu, L, Oza, S, Hogan, D, Chu, Y, Perin, J, Zhu, J, et al. Global, regional, and national causes of under-5 mortality in 2000–15: an updated systematic analysis with implications for the sustainable development goals. Lancet. (2016) 388:3027–35. doi: 10.1016/S0140-6736(16)31593-8

CrossRef Full Text | Google Scholar

4. Collaborators, GBDM. Global, regional, and national under-5 mortality, adult mortality, age-specific mortality, and life expectancy, 1970-2016: a systematic analysis for the global burden of disease study 2016. Lancet. (2017) 390:1084–150. doi: 10.1016/S0140-6736(17)31833-0

CrossRef Full Text | Google Scholar

5. Secretary-General. Global sustainable development report 2019: The future is now – Science for achieving sustainable development. New York: United Nations publication issued by the Department of Economic and Social Affairs (2019).

Google Scholar

6. Kruk, ME, Gage, AD, Arsenault, C, Jordan, K, Leslie, HH, Roder-DeWan, S, et al. High-quality health systems in the sustainable development goals era: time for a revolution. Lancet Glob Health. (2018) 6:e1196–252. doi: 10.1016/S2214-109X(18)30386-3

CrossRef Full Text | Google Scholar

7. Mutale, W, Ayles, H, Bond, V, Chintu, N, Chilengi, R, Mwanamwenge, MT, et al. Application of systems thinking: 12-month postintervention evaluation of a complex health system intervention in Zambia: the case of the BHOMA. J Eval Clin Pract. (2017) 23:439–52. doi: 10.1111/jep.12354

CrossRef Full Text | Google Scholar

8. Stringer, JS, Chisembele-Taylor, A, Chibwesha, CJ, Chi, HF, Ayles, H, Manda, H, et al. Protocol-driven primary care and community linkages to improve population health in rural Zambia: the better health outcomes through mentoring and assessment (BHOMA) project. BMC Health Serv Res. (2013) 13:S7. doi: 10.1186/1472-6963-13-S2-S7

CrossRef Full Text | Google Scholar

9. Hensen, B, Lewis, JJ, Schaap, A, Tembo, M, Vera-Hernández, M, Mutale, W, et al. Frequency of HIV-testing and factors associated with multiple lifetime HIV-testing among a rural population of Zambian men. BMC Public Health. (2015) 15:960. doi: 10.1186/s12889-015-2259-3

CrossRef Full Text | Google Scholar

10. Mutale, W, Ayles, H, Bond, V, Mwanamwenge, MT, and Balabanova, D. Measuring health workers' motivation in rural health facilities: baseline results from three study districts in Zambia. Hum Resour Health. (2013) 11:8. doi: 10.1186/1478-4491-11-8

CrossRef Full Text | Google Scholar

11. Mutale, W, Balabanova, D, Chintu, N, Mwanamwenge, MT, and Ayles, H. Application of system thinking concepts in health system strengthening in low-income settings: a proposed conceptual framework for the evaluation of a complex health system intervention: the case of the BHOMA intervention in Zambia. J Eval Clin Pract. (2016) 22:112–21. doi: 10.1111/jep.12160

CrossRef Full Text | Google Scholar

12. Mutale, W, Bond, V, Mwanamwenge, MT, Mlewa, S, Balabanova, D, Spicer, N, et al. Systems thinking in practice: the current status of the six WHO building blocks for health system strengthening in three BHOMA intervention districts of Zambia: a baseline qualitative study. BMC Health Serv Res. (2013) 13:291. doi: 10.1186/1472-6963-13-291

CrossRef Full Text | Google Scholar

13. Mutale, W, Chintu, N, Amoroso, C, Awoonor-Williams, K, Phillips, J, Baynes, C, et al. Improving health information systems for decision making across five sub-Saharan African countries: implementation strategies from the African health initiative. BMC Health Serv Res. (2013) 13:S9. doi: 10.1186/1472-6963-13-S2-S9

CrossRef Full Text | Google Scholar

14. Mutale, W, Cleary, S, Olivier, J, Chilengi, R, and Gilson, L. Implementing large-scale health system strengthening interventions: experience from the better health outcomes through mentoring and assessments (BHOMA) project in Zambia. BMC Health Serv Res. (2018) 18:795. doi: 10.1186/s12913-018-3619-3

CrossRef Full Text | Google Scholar

15. Mutale, W, Godfrey-Fausset, P, Mwanamwenge, MT, Kasese, N, Chintu, N, Balabanova, D, et al. Measuring health system strengthening: application of the balanced scorecard approach to rank the baseline performance of three rural districts in Zambia. PLoS One. (2013) 8:e58650. doi: 10.1371/journal.pone.0058650

CrossRef Full Text | Google Scholar

16. Mutale, W, Masoso, C, Mwanza, B, Chirwa, C, Mwaba, L, Siwale, Z, et al. Exploring community participation in project design: application of the community conversation approach to improve maternal and newborn health in Zambia. BMC Public Health. (2017) 17:277. doi: 10.1186/s12889-017-4187-x

CrossRef Full Text | Google Scholar

17. Mutale, W, Mwanamwenge, MT, Balabanova, D, Spicer, N, and Ayles, H. Measuring governance at health facility level: developing and validation of simple governance tool in Zambia. BMC Int Health Hum Rights. (2013) 13:34. doi: 10.1186/1472-698X-13-34

CrossRef Full Text | Google Scholar

18. Mutale, W, Stringer, J, Chintu, N, Chilengi, R, Mwanamwenge, MT, Kasese, N, et al. Application of balanced scorecard in the evaluation of a complex health system intervention: 12 months post intervention findings from the BHOMA intervention: a cluster randomised trial in Zambia. PLoS One. (2014) 9:e93977. doi: 10.1371/journal.pone.0093977

CrossRef Full Text | Google Scholar

19. Mwanza, M, Zulu, J, Topp, SM, Musonda, P, Mutale, W, and Chilengi, R. Use of lot quality assurance sampling surveys to evaluate community health worker performance in rural Zambia: a case of Luangwa district. BMC Health Serv Res. (2017) 17:279. doi: 10.1186/s12913-017-2229-9

CrossRef Full Text | Google Scholar

20. Schuttner, L, Sindano, N, Theis, M, Zue, C, Joseph, J, Chilengi, R, et al. A mobile phone-based, community health worker program for referral, follow-up, and service outreach in rural Zambia: outcomes and overview. Telemed J E Health. (2014) 20:721–8. doi: 10.1089/tmj.2013.0240

CrossRef Full Text | Google Scholar

21. Yan, LD, Chi, BH, Sindano, N, Bosomprah, S, Stringer, JSA, and Chilengi, R. Prevalence of hypertension and its treatment among adults presenting to primary health clinics in rural Zambia: analysis of an observational database. BMC Public Health. (2015) 15:933. doi: 10.1186/s12889-015-2258-4

CrossRef Full Text | Google Scholar

22. Yan, LD, Chirwa, C, Chi, BH, Bosomprah, S, Sindano, N, Mwanza, M, et al. Hypertension management in rural primary care facilities in Zambia: a mixed methods study. BMC Health Serv Res. (2017) 17:111. doi: 10.1186/s12913-017-2063-0

CrossRef Full Text | Google Scholar

23. Vasan, A, Ellner, A, Lawn, SD, Gove, S, Anatole, M, Gupta, N, et al. Integrated care as a means to improve primary care delivery for adults and adolescents in the developing world: a critical analysis of integrated Management of Adolescent and Adult Illness (IMAI). BMC Med. (2014) 12:6. doi: 10.1186/1741-7015-12-6

CrossRef Full Text | Google Scholar

24. Daelmans, B, Seck, A, Nsona, H, Wilson, S, and Young, M. Integrated community case Management of Childhood Illness: what have we learned? Am J Trop Med Hyg. (2016) 94:571–3. doi: 10.4269/ajtmh.94-3intro2

CrossRef Full Text | Google Scholar

25. Otolorin, E, Gomez, P, Currie, S, Thapa, K, and Dao, B. Essential basic and emergency obstetric and newborn care: from education and training to service delivery and quality of care. Int J Gynecol Obstet. (2015) 130:S46–53. doi: 10.1016/j.ijgo.2015.03.007

CrossRef Full Text | Google Scholar

26. Victora, CG, Fenn, B, Bryce, J, and Kirkwood, BR. Co-coverage of preventive interventions and implications for child-survival strategies: evidence from national surveys. Lancet. (2005) 366:1460–6. doi: 10.1016/S0140-6736(05)67599-X

CrossRef Full Text | Google Scholar

27. Bakibinga, P, Matanda, D, Kisia, L, and Mutombo, N. Factors associated with use of injectables, long-acting and permanent contraceptive methods (iLAPMs) among married women in Zambia: analysis of demographic and health surveys, 1992-2014. Reprod Health. (2019) 16:78–8. doi: 10.1186/s12978-019-0741-6

CrossRef Full Text | Google Scholar

28. Hayes, RJ, and Moulton, LH. Cluster randomised trials, second edition. Boca Raton, Florida, United states of America: CRC Press (2017).

Google Scholar

29. Thomson, DR, Amoroso, C, Atwood, S, Bonds, MH, Rwabukwisi, FC, Drobac, P, et al. Impact of a health system strengthening intervention on maternal and child health outputs and outcomes in rural Rwanda 2005-2010. BMJ Glob Health. (2018) 3:e000674–4. doi: 10.1136/bmjgh-2017-000674

CrossRef Full Text | Google Scholar

30. Garchitorena, A, Miller, AC, Cordier, LF, Rabeza, VR, Randriamanambintsoa, M, Razanadrakato, HTR, et al. Early changes in intervention coverage and mortality rates following the implementation of an integrated health system intervention in Madagascar. BMJ Glob Health. (2018) 3:e000762. doi: 10.1136/bmjgh-2018-000762

CrossRef Full Text | Google Scholar

31. Bawah, AA, Awoonor-Williams, JK, Asuming, PO, Jackson, EF, Boyer, CB, Kanmiki, EW, et al. The child survival impact of the Ghana essential health interventions program: a health systems strengthening plausibility trial in northern Ghana. PLoS One. (2019) 14:e0218025. doi: 10.1371/journal.pone.0218025

CrossRef Full Text | Google Scholar

32. Haws, RA, Mashasi, I, Mrisho, M, Schellenberg, JA, Darmstadt, GL, and Winch, PJ. “These are not good things for other people to know”: how rural Tanzanian women’s experiences of pregnancy loss and early neonatal death may impact survey data quality. Soc Sci Med. (2010) 71:1764–72. doi: 10.1016/j.socscimed.2010.03.051

CrossRef Full Text | Google Scholar

33. Eichner, FA, Groenwold, RHH, Grobbee, DE, and Oude Rengerink, K. Systematic review showed that stepped-wedge cluster randomized trials often did not reach their planned sample size. J Clin Epidemiol. (2019) 107:89–100. doi: 10.1016/j.jclinepi.2018.11.013

CrossRef Full Text | Google Scholar

34. Hughes, JP, Granston, TS, and Heagerty, PJ. Current issues in the design and analysis of stepped wedge trials. Contemp Clin Trials. (2015) 45:55–60. doi: 10.1016/j.cct.2015.07.006

CrossRef Full Text | Google Scholar