Development and validation of risk-adjusted quality indicators for the long-term outcome of acute sepsis care in German hospitals based on health claims data

Background: Methods for assessing long-term outcome quality of acute care for sepsis are lacking. We investigated a method for measuring long-term outcome quality based on health claims data in Germany.

Materials and methods: Analyses were based on data of the largest German health insurer, covering 32% of the population. Cases (aged 15 years and older) with ICD-10-codes for severe sepsis or septic shock according to sepsis-1-definitions hospitalized in 2014 were included. Short-term outcome was assessed by 90-day mortality; long-term outcome was assessed by a composite endpoint defined by 1-year mortality or increased dependency on chronic care. Risk factors were identified by logistic regressions with backward selection. Hierarchical generalized linear models were used to correct for clustering of cases in hospitals. Predictive validity of the models was assessed by internal validation using bootstrap-sampling. Risk-standardized mortality rates (RSMR) were calculated with and without reliability adjustment and their univariate and bivariate distributions were described.

Results: Among 35,552 included patients, 53.2% died within 90 days after admission; 39.8% of 90-day survivors died within the first year or had an increased dependency on chronic care. Both risk-models showed a sufficient predictive validity regarding discrimination [AUC = 0.748 (95% CI: 0.742; 0.752) for 90-day mortality; AUC = 0.675 (95% CI: 0.665; 0.685) for the 1-year composite outcome, respectively], calibration (Brier Score of 0.203 and 0.220; calibration slope of 1.094 and 0.978), and explained variance (R² = 0.242 and R² = 0.111). Because of a small case-volume per hospital, applying reliability adjustment to the RSMR led to a great decrease in variability across hospitals [from median (1st quartile, 3rd quartile) 54.2% (44.3%, 65.5%) to 53.2% (50.7%, 55.9%) for 90-day mortality; from 39.2% (27.8%, 51.1%) to 39.9% (39.5%, 40.4%) for the 1-year composite endpoint]. There was no substantial correlation between the two endpoints at hospital level (observed rates: ρ = 0, p = 0.99; RSMR: ρ = 0.017, p = 0.56; reliability-adjusted RSMR: ρ = 0.067; p = 0.026).

Conclusion: Quality assurance and epidemiological surveillance of sepsis care should include indicators of long-term mortality and morbidity. Claims-based risk-adjustment models for quality indicators of acute sepsis care showed satisfactory predictive validity. To increase reliability of measurement, data sources should cover the full population and hospitals need to improve ICD-10-coding of sepsis.

1. Introduction

Sepsis is the final pathway to death from infectious diseases (1) and affects an estimated 49 million patients per year worldwide, of whom 11 million die (2). It is considered as one of the leading causes of preventable deaths in hospitals (3). One-sixth of sepsis survivors experience severe persistent physical disability or cognitive impairment, and one-third die during the following year after the acute disease (4). Acknowledging deficits of care, the World Health Assembly adopted the sepsis resolution WHA70.7 in May 2017, which urges WHO member states to improve prevention, diagnosis and management of sepsis (5).

Measuring and comparing performance of health care providers are a central part of quality improvement (6). For this purpose, administrative health data can be used for performance measurement with the advantage of covering all ICD-coded cases with data readily available, at minimal time and costs (7). Performance measures need to account for differences in the mix of important patient attributes across hospitals by adequate risk-adjustment models (8, 9). Several risk-adjusted quality indicators on sepsis care based on administrative health data have been presented in the literature (10–13). Such indicators have been used to assess and compare hospital performance as well as to evaluate effects of voluntary and mandated quality improvement programs (14, 15). Existing administrative data-based indicators on the quality of sepsis care share two shortcomings. First, they only used in-hospital or 30-day post-discharge mortality as outcomes, although short-term case fatality is increasingly regarded to be inadequate as sole metric for the outcome of sepsis patients (4, 16). Improved quality of care should ideally reduce short-term mortality, but also long-term mortality and morbidity resulting in a higher proportion of patients with full recovery. Second, risk-adjustment models were based on hospital discharge data solely. Therefore, pre-existing conditions were defined only based on ICD-coding during the hospital stay, which may result in bias based on incomplete coding as well as a failure to distinguish conditions present-on-admission from complications (7, 17, 18).

To overcome these shortcomings, we developed risk-adjusted quality indicators based on longitudinal health claims data incorporating long-term outcomes of sepsis care as well as pre-existing conditions coded before hospital admission.

2. Materials and methods

2.1. Data source

This is a secondary analysis of health care claims data provided for the SEPFROK study (19). This cohort study was based on nationwide anonymous administrative health claims data of the largest German health insurance, the “Allgemeine Ortskrankenkasse” (AOK), which covers approximately 32% of the German population. Data were provided by the Research Institute of the Local Health Care Funds (WIdO). Health insurance is mandatory in Germany; residents can select any insurer and enroll without restriction. Within the AOK data, hospitals were identified by a unique institutional identifier (IK: “Institutionskennzeichen”). More than one hospital site of the same institution might use the same IK, but typically, these sites are organizationally linked and mutually dependent. Based on the AOK health care claims data, the WIdO already provides the quality assurance using health claims data “Qualitätssicherung mit Routinedaten” (QSR, quality assurance using routine data) (20). Indicators are reported to participating hospitals and are part of a web-based information portal to support patients in selecting a hospital. Sepsis is not yet part of the set of quality indicators.

2.2. Study population

The SEPFROK study included patients aged ≥ 15 years with an inpatient hospitalization (discharged January 1, 2013, to December 31, 2014) with an explicit International Statistical Classification of Diseases and Related Health Problems, Tenth Revision, German Modification (ICD-10-GM) code for sepsis as primary or secondary discharge diagnoses (Supplementary material 1–Definition of variables). The first hospitalization with sepsis within this period was defined as the index hospitalization and included in the analyses. Patients with a diagnosis of sepsis in the 2 years preceding the index hospitalization were excluded. Since SEPFROK included a 5-year-look-back period and a 3-year-follow-up, patients who were not continuously insured from January 1, 2009, through their respective 3-year follow-up period after the index hospitalization (or until death) were excluded (Supplementary Figure 1).

For this secondary analysis, we included patients with index hospitalization with severe sepsis or septic shock defined by ICD-Codes R65.1 and R57.2 in 2014.

2.3. Outcomes

We included short- and long-term endpoints. 90-day mortality after hospital discharge was chosen as short-term endpoint. As long-term outcome, we defined a composite (binary) outcome of 1-year mortality and increase in the dependency on chronic care during the year after hospital discharge from index hospitalization to address the competing risk they represent (21). The increase in dependency on nursing care was defined by an increase in nursing care level or a new transition to a long-term nursing home, which both are recorded with high reliability in claims data and thus can serve as objective measure of a relevant increase of morbidity and decrease of functioning. In Germany, nursing care levels are defined on graded care needs and entitle patients to long-term care insurance benefits. Care can be provided by informal or formal caregivers or in nursing homes (see Supplementary material 1: Definition of variables for details). For the analysis of the composite endpoint only 90-day survivors were included.

2.4. Model derivation

2.4.1. Risk factors

Based on clinical reasoning and existing research, candidate variables were chosen among patient demographics, pre-existing comorbidities, pre-existing conditions and treatments, clinical characteristics of the infection, hospital admission type and specific treatments during the index hospitalization (10–12, 22–24). Detailed definitions of risk factors are given in detail in the Supplementary material 1.

2.4.1.1. Patient demographics

Patient demographics included gender and age. To allow for non-linear effects of age, quadratic, and cubic polynomials were included. Age was transformed by mean-centering and standardization to decades [age_t = (age–70)/10].

2.4.1.2. Comorbidities

Comorbidities were assessed in a period of 12 months prior to hospitalization and were defined by the categories of the Charlson Comorbidity Index (CCI) and the Elixhauser Comorbidity Index (ECI) (25, 26) based on a German adaptation of a previously developed ICD-10 coding algorithm (27, 28). If a CCI and an ECI category assessed the same comorbidity, the ECI category was included. An additional indicator variable for presence of leukemia was also included.

2.4.1.3. Pre-existing conditions and treatments

Pre-existing conditions and treatments included the prior dependency on immobility, nursing care, mechanical ventilation, renal replacement therapy, palliative care, which were defined by procedures and general medical measures [OPS: Operationen und Prozedurenschlüssel (German Procedure Classification)] and ICD-10-GM codes, were assessed in a period of 12 months prior to the index hospitalization. The cumulative length of previous hospital stays during the 1-year period before the index hospitalization was categorized as follows: “0 day,” “1 day,” “> 1 day and < 6 days,” “≥ 6 days and ≤ 10 days”, and “> 10 days.”

2.4.1.4. Clinical characteristics of the infection

Clinical characteristics of the infection included “focus of infection” defined by presence of specific ICD-10-GM codes, and presence of an explicit sepsis code as a primary diagnosis. “Focus of infection”-codes were derived from the literature (29–31) and clinical knowledge. A primary diagnosis of sepsis was defined if an explicit sepsis code (A40.–A41., R57.2) was present as primary diagnosis. Finally, infection by multi-resistant pathogens was defined by presence of an ICD-10-GM and OPS-code for the presence and treatment of multi-drug resistant pathogens during the index stay.

2.4.1.5. Hospital admission type

Hospital admission type was categorized as “emergency admission,” “referral by physician” or “transfer from another hospital.”

2.4.1.6. Specific treatments during the index hospitalization

Specific treatments during the index hospitalization not related to sepsis care but associated with increased risk of death were also included and defined by OPS-codes (chemotherapy, stroke treatment).

2.4.2. Model development

Two risk-models were developed–one for each specified endpoint. Risk factors were first selected from the set of candidate variables by a logistic regression model with backward elimination for each endpoint. Because of the large sample size, the criterion to exclude variables from the model was set p > 0.01. Since patients with septic shock or sepsis as primary diagnosis are a distinctive subgroup with a special importance for quality measurement, we aimed to make the model suitable also for comparing the endpoints within the subgroups of cases with or without septic shock and with or without sepsis as primary diagnosis. Risk factors might have different effects within these respective subgroups, which can be modeled by statistical interaction effects. Therefore, interaction effects of the selected predictors with the presence of a diagnosis of septic shock or sepsis as primary diagnosis were also included and backward-selected in a second modeling step. Since observed outcomes are expected to be correlated within-hospitals, these models were then refitted by hierarchical generalized linear models (HGLMs) with binomial errors, a logit link and a random intercept for the hospitals (9, 32).

2.4.3. Model validation

We did not conduct a validation in external cohorts, since this model is not intended for use in external cohorts. If such a model is intended to be used in a quality assurance program, like QSR, a recalculation on a yearly base would be necessary. Therefore, we conducted an internal validation with correction for over-fitting using two bootstrap approaches. First, following advice by Harrell et al. (33), two hundred bootstrap replications were done by sampling over the hospitals. In each bootstrap step, the variable selection and the re-fitting by HGLM was repeated within the bootstrap sample. The following validation measures were calculated: the area under the curve (AUC) as a measure of discrimination, the squared Pearson correlation (R²) as measure of explained variation (34, 35), and the Brier Score and the calibration slope as measures of calibration. These validation measures were calculated in the bootstrap sample on the one hand and in the original sample on the other hand. The difference of these two values is the optimism. The corrected performance is the difference of the validation measure in the original sample and the averaged optimism, respectively. The second approach was similar, but estimated validation measures by prediction on the out-of-bag samples in each bootstrap step and then taking their mean (36). To visualize calibration, the distribution of observed mortality across deciles of predicted mortality was plotted.

2.4.4. Calculation of risk-and reliability-adjusted indicators

Risk-adjusted endpoints per hospital were calculated as risk-standardized mortality rates (RSMR). Note that the expression RSMR will also be used when referring to the composite endpoint. Two methods were used to calculate RSMR. The first method was based on the standard logistic regression approach, in which the RSMR results from the ratio of observed mortality to mortality predicted from the logistic regression model, multiplied with the unadjusted rate in the full sample (37, 38). Low number of cases per hospital cause unreliability in the estimation of the RSMR, which results in higher rates of randomly extreme values among hospitals with small case numbers. Reliability adjustment by shrinkage estimators was repeatedly proposed to achieve more stable estimates (39–41). To implement this, we applied the methodology used for the quality indicators of the Centers for Medicare and Medicaid Service’s as a second method (35, 38, 42). Here, reliability-adjusted RSMR are obtained as the ratio of predicted to expected mortality obtained from the HGLM, multiplied by the unadjusted rate of the full sample (38). Confidence limits (95% CI) were calculated by a large sample approximation for the RSMR (37, 38), and by a bootstrap approach, as described by Normand et al. (42), for the reliability-adjusted RSMR. The distribution of the observed rates, RSMR and reliability-adjusted RSMR was analyzed by descriptive statistics and graphics. The bivariate relationships between the 90-day mortality and the composite endpoint of 1-year mortality or increased dependency on nursing care was analyzed by scatterplot and calculation of Pearson’s ρ for unadjusted rates, RSMR and reliability-adjusted RSMR, respectively. Hospitals with a CI not overlapping with the unadjusted rate of the full sample are regarded as showing a deviation from the average. We used cross-tabulation to describe the distribution of hospitals with a significant deviation in RSMR or reliability-adjusted RSMR for both endpoints. The data analysis for this paper was generated using SAS software, version 9.4 (Copyright© 2002–2012 SAS Institute Inc., SAS and all other SAS Institute Inc., product or service names are registered trademarks or trademarks of SAS Institute Inc., Cary, NC, USA) and with R, version 4.1.1 [R Core Team, 2020 (43). R: A language and environment for statistical computing. R Foundation for Statistical Computing, Vienna, Austria. URL https://www.R-project.org/].

3. Results

Table 1 provides an overview of the study sample (and Supplementary Figure 1 is the corresponding flowchart). Among 35,522 patients who were hospitalized with sepsis or septic shock in 2014, 18,884 (53.2%) died within the first 90 days after admission. Of 16,638 90-day survivors, 3,632 (21.8%) died within the first year. Increase in the dependency on nursing care affected 4,316 (25.9%) of 90-day survivors. Death or increase in the dependency on nursing care (composite endpoint) occurred in 6,639 (39.8%) of 90-day survivors. The mean age of the total sample at hospital cases was 73.96 ± 12.28 years (90-day survivors: 71.58 ± 13.02 years), 45% were males (90-day survivors: 44.7%). Patients of the total sample were treated in 1,174 hospitals. Since part of hospitals had no 90-day survivors, the 1-year composite endpoint was analyzed for 1,105 hospitals only.

TABLE 1

Table 1. Sample of cases with severe sepsis or septic shock.

3.1. Risk-adjustment models

After backward selection 29 of 58 initial risk factors and 12 of 136 initial interaction effects with septic shock or sepsis as primary diagnosis were identified for the endpoint 90-day mortality. Higher age, emergency admission or transfer from another hospital, and septic shock were associated with increased risk of death. In general, indicators of preexisting morbidity–like pre-sepsis comorbid illness, duration of previous hospital stays, or pre-existing treatments–were associated with an increased risk of death. Exemption were depression, complicated hypertension, obesity, and pre-existing long-term ventilation, which showed protective effects (Table 2). For the 1-year composite endpoint (1-year mortality or increased dependency on nursing care), 27 risk factors and 10 of 112 initial interaction effects with septic shock or sepsis as primary diagnosis were identified. Again, higher age, emergency admission or transfer from another hospital, and septic shock were associated with increased risk. Indicators of pre-sepsis morbidity were all associated with increased risk, with the exemption of pre-existing dependency on chronic care, which showed a protective effect (Table 2).

TABLE 2

Table 2. Coefficients estimates of the risk-adjustment model for 90-days mortality and 1-year composite endpoint of mortality or increased dependency on chronic care.

3.2. Validation

Uncorrected and corrected estimates of the validity measures are given in Supplementary material 2: Internal validation, Supplementary Tables 1, 2. The approach of Harrel et al. resulted in more conservative estimates and yielded a corrected discrimination of AUC = 0.748 [95% CI: 0.742; 0.752], an explained variance of R² = 0.242, a Brier Score of 0.203 and a calibration slope of 1.094 for the model for 90-day-mortality. The model for the 1-year composite endpoint showed an estimated discrimination of AUC = 0.675 [95% CI: 0.665; 0.685], an explained variance of R² = 0.111, a Brier Score of 0.220 and a calibration slope of 0.978. Calibration of both models was good (observed rate in lowest risk-decile and highest decile: 0.17–0.87 and 0.13–0.68, respectively, Figures 1A, B).

FIGURE 1

Figure 1. Calibration of the risk-adjustment models. (A) 90-day mortality. (B) 1-year composite endpoint (1-year mortality or increased dependency on chronic care).

3.3. Distribution of indicators across hospitals

There was a large variation in numbers of cases across hospitals. Across 1,174 hospitals, the number of cases with coded sepsis per hospital for 90-day mortality ranged from 1 to 745, 25th, 50th, and 75th percentile of 7, 17, and 36, respectively. Across 1,105 hospitals, the number of 90-day survivors per hospital for the composite 1-year endpoint ranged from 1 to 334, 25th, 50th, and 75th percentile of 3, 8, and 18, respectively.

Figure 2 presents the distribution of the two endpoints–90-day mortality and the 1-year composite endpoint per hospital–with their observed values, their risk-adjusted values (RSMR), and their risk-and reliability adjusted values (reliability-adjusted RSMR). While the variability of the RSMR was comparable to the observed values (panel A vs. C, and panel B vs. D, respectively), the implementation of reliability adjustment let to a strong reduction in variability across hospitals (panels E and F).

FIGURE 2

Figure 2. Distribution of observed, risk-standardized and risk-and reliability adjusted endpoints per hospital. (A) Observed 90-day mortality, range from 0 to 100%, 25th, 50th, and 75th percentile of 44.0, 54.6, and 66.7%, respectively. (B) Observed rate for composite outcome of 1-year-mortality or increased dependency on chronic care, range from 0 to 100%, 25th, 50th, and 75th percentile 26.9, 39.3, and 50.0%, respectively. (C) Risk-standardized rate (RSMR) for 90-day mortality without reliability adjustment, range from 0 to 100%, 25th, 50th, and 75th percentiles of 44.3, 54.2, and 65.5%, respectively. (D) RSMR for the 1-year composite endpoint without reliability adjustment, range from 0 to 100%, 25th, 50th, and 75th percentiles of 27.8, 39.2, and 51.1%, respectively. (E) Reliability-adjusted RSMR for 90-day mortality, range from 35.9 to 68.1%, 25th, 50th, and 75th percentiles of 50.7, 53.2, and 55.9%, respectively. (F) Reliability-adjusted RSMR for 1-year composite outcome, range from 35.8 to 43.3%, 25th, 50th, and 75th percentile of 39.5, 39.9, and 40.4%, respectively.

Reliability adjustment also had an effect regarding the proportion of hospitals, which showed a deviation of the RSMR from average (Table 3). Without reliability adjustment, 7.6% of hospitals showed a RSMR on 90-day mortality with the lower 95% confidence limit above the average; with reliability adjustment, this was true for only 1.2%. Regarding the 1-year composite endpoint, 3.5% of hospitals showed an RSMR above average without reliability adjustment, while none showed a RSMR above average with reliability adjustment.

TABLE 3

Table 3. Proportion of hospitals showing significant deviation of risk-adjusted outcomes from average.

The relationships between the different rates for 90-day mortality and the 1-year composite endpoint are depicted in Figure 3. There were no substantial correlations observed between the two endpoints (observed rates: ρ = 0, p = 0.99; RSMR: ρ = 0.017, p = 0.56; reliability-adjusted RSMR: ρ = 0.067; p = 0.026).

FIGURE 3

Figure 3. Relationship between 90-day mortality and the 1-year composite endpoint. Each point represents one hospital. The red line represents the linear relationship based on least-squares estimation. One-year composite endpoint defined by death or increased dependency on chronic care during 1 year after discharge. (A) Relationship between observed rates. (B) Relationship between risk-standardized rates (RSMR). (C) Relationship between reliability-adjusted RSMR.

4. Discussion

4.1. Summary of results

This is the first study on the development of health claims based risk-adjusted quality indicators for acute sepsis care, which incorporate both 90-day mortality as well as long-term outcomes on mortality and functional dependency. The risk-adjustment models relied on pre-existing conditions actually measured before hospitalization and showed a decent predictive validity. There was no evidence for correlation between short-term and long-term outcomes at the level of hospitals.

4.2. Interpretation

Regarding 90-day mortality, the predictive validity of our model was comparable to previously reported administrative data based models on short-term mortality after acute treatment of sepsis (10–12). Even risk-adjustment models, which were based on clinical data or a mix of clinical and administrative data, did–with AUCs between 0.75 and 0.78–not achieve relevantly higher discrimination (22, 24, 44). The effects of risk factors in the model are similar to previously reported risk-models for sepsis mortality (10–12, 22). Protective effects of some comorbidities on short-term mortality–namely obesity, depression, and hypertension–have also been shown in these studies as well as in studies on the Elixhauser comorbidity index conducted among representative samples of hospital patients (26, 45, 46). It has been argued that these seemingly protective effects reflect a bias in coding, where relatively healthy patients without severe comorbidities have a higher chance of having these milder comorbidities coded compared to more severely ill patients. We found that also pre-existing long-term mechanical ventilation had a protective effect on risk of 90-day mortality. This might be due to the intense ongoing monitoring of these patients, which might have allowed early detection and adequate treatment of sepsis and thereby prevention of acute deterioration and death (47).

To our knowledge, no other risk-adjustment model on long-term outcomes of sepsis care based on administrative data exists. Moreover, prediction scores for long-term outcomes based on clinical data are also lacking (48, 49). Based on an ICU-registry, Shankar-Hari et al. developed a prognostic score for the composite endpoint of 1-year mortality or re-hospitalization, which showed an AUC of 0.68–comparable to the AUC of 0.675 estimated for our model on 1-year mortality or increased dependency on chronic care. As expected, we found that older patients with comorbid conditions tended to have a higher risk of long-term mortality or dependency on chronic care after having survived 90 days post-discharge. Pre-existing long-term mechanical ventilation increased the risk of long-term mortality or care dependency, likewise because these patients have a reduced survival time in general. The protective effect of pre-existing dependency on chronic care might simply indicate that the risk to develop dependency on chronic care for the first time is higher compared to switching to a higher degree of dependency, if a patient was already receiving chronic care.

Patients with sepsis can suffer from a broad spectrum of clinical sequelae in the areas of physical disability, cognitive impairment, mental health impairment, recurrent infection and sepsis, exacerbation of chronic conditions, all of which decrease overall functioning and quality of life (4, 19). Since concrete sequelae can be highly variable across patients and their measurement in administrative data is dependent on validity of ICD-coding (19), we selected the increase in the dependency on chronic care as an objective indicator of cumulative, overall functioning. It has to be acknowledged that long-term outcomes of acute care cannot solely be attributed to the initially treating hospital, since they are also influenced by other health-care providers responsible for the further treatment as well as other factors (50). Little is known on how to enhance long-term recovery of survivors during acute in-patient care on the ICU or the ward (4, 51, 52). Because of this, attributing long-term outcomes of sepsis care to the initial hospitalization is especially problematic. This might explain why 90-day mortality and the composite 1-year endpoint did not correlate in our study. Therefore, short-term mortality may serve as the better indicator of quality of acute sepsis care. On the other hand, a reduction in short-term mortality after changes in treatment regiments might come at the cost of an increase in long-term mortality or worsening of other patient-centered outcomes (4, 16). Including indicators of long-term sequelae to a measurement of the quality of care can therefore help to interpret differences between providers as well as changes across time both in quality assurance and population surveillance (16).

The problem of reliability adjustment has been extensively discussed in methodological literature (35, 38–42), but is currently not applied in prominent voluntary performance measurement programs in Germany or the methodology of the mandatory quality indicators for German hospitals (20, 53, 54). When case numbers are small, it is hard to tell if extremely high or low outcome rates are due to chance or true differences in quality of care (39). Since there typically is a time-lag between data collection and report of quality indicators, the validity of a quality indicator to predict future performance is important. Shrinkage estimators shrink the estimate of the rate toward the average rate of the population, with the amount of shrinkage negatively proportional to the number of cases. This shrunken estimator has been shown to be a better predictor for future performance compared to classical methods based on logistic regression (38, 39). In our study, reliability adjustment resulted in a great reduction in variability in endpoints between hospitals compared to raw endpoints as well as compared to non-reliability-adjusted RSMR. This effect was stronger for the 1-year-composite endpoint, given the smaller case numbers per hospital. Shrunken estimators are especially relevant for quality indicators used in public reporting or pay-for-performance systems, where wrongfully assigning low-volume hospitals a below-average performance due to unreliable estimates would result in unwarranted negative financial consequences. Nevertheless, if the aim is the identification of possible shortcomings of care for further investigation in continuous quality improvement programs or by peer-reviews, it might be important not to miss possible signals and therefore to also take non-reliability-adjusted estimates into consideration–especially if case-numbers per hospital tend to be low (55).

4.3. Practical usage of the presented methodology

The developed methodology could be used for the purposes of quality measurement and between-hospital comparisons. It could also be useful in population surveillance to monitor the changes in mortality and morbidity on a population level across time in Germany (19, 56). It can be applied to data of the same structure and content, i.e., health claims data of the AOK or of other German public health care insurers, which all obtain largely the same information collected for administrative purposes. The German Institute for Quality and Efficiency in Health Care has recently been instructed to develop mandatory quality indicators for acute sepsis care by the Joint Federal Committee–the institution responsible for quality assurance of health care in Germany (57). Since combined data of all public health insurers are used to calculate mandatory quality indicators of German hospitals, the problem of low case volume and lacking reliability of estimates might be overcome in this context. Thus, the presented methodology could be applied both in voluntary performance measurement in the context of the QSR-program as well as in the context of mandatory quality assurance for German hospitals. For any purpose, the risk-adjustment models should be recalculated using the respective current population and reference period to allow adequate comparisons. For this reason, we refrained from validating the reported model in a separate external cohort, since the methodology is not intended to be used this way.

4.4. Strengths and limitations

Our study has several strengths. It presents the first claims-based risk-adjusted quality indicators for long-term outcome of acute sepsis care, which include risk factors measured before the index hospitalization. This is an important improvement compared to previous claims-based models, which only relied on information documented during the hospital stay (10–12). Operationalization of variables was done in a rigorous process, based on a multiprofessional panel of experts who care for patients with sepsis (19). Based on the large sample size, complex risk-adjustment models with decent predictive validity were derived.

Our study also has several limitations, mostly associated with general shortcomings of administrative health data (7). These data are limited in content and provide no information on vital signs, microbiological results or medication during the hospital stay. Therefore, several known risk factors for short-term and long-term patient outcomes, like presenting signs and symptoms or severity of initial critical illness could not be assessed (49, 51). On the other hand, predictive validity was comparable to models incorporating clinical data (22, 24, 44, 49), and previous studies found no relevant differences between risk-standardized rates based solely on administrative data compared to administrative data enhanced by clinical information (11, 35). Administrative data are also limited in accuracy and completeness of coding of information (7). This results in possible information biases affecting the case selection, as well as identification of risk factors and outcomes. Lacking accuracy and mostly undercoding has been described for the ICD-coding of sepsis (58, 59). This can have huge effects on the validity of performance measures and provider comparisons, especially if the quality of coding varies across hospitals (7, 60). Based on our data, in average only one third of German patients coded with sepsis per hospital and year were included, which–together with the problem of undercoding of sepsis–results in decreased reliability of estimates and reduced power to detect hospitals, which are outliers in risk-adjusted endpoints. Our study was based on data gathered before the introduction of the new Sepsis-3 definitions (1). Therefore, we included patients with ICD-codes for sepsis with organ dysfunction or septic shock according to sepsis-1/sepsis-2 definitions, since severe sepsis according to old sepsis definitions shows a high overlap with the new definitions (61). We used a composite outcome of mortality and increase in long-term morbidity. While this allows to address the competing risk between both endpoints (21), it leads to difficulties in interpreting differences in hospital performance. Multistate models provide a comprehensive methodology to address competing risks between outcomes (62). But since there does not yet exist a method to obtain hospital-specific estimates of risk-adjusted quality indicators from such models, further methodological development is needed to apply them in the context of provider comparisons.

5. Conclusion

We presented a methodology for claims-based risk-adjusted quality indicators on short-term and long-term mortality and morbidity after acute sepsis care. Beside the limitations of administrative health data, this methodology could provide a valuable tool in assessing and monitoring outcome quality achieved by German hospitals caring for patients with sepsis. Future studies should recalculate the risk-adjustment models based on current data incorporating the new Sepsis-3 definitions and may embrace multi state modeling to address mortality and morbidity jointly. To increase reliability and validity of measures of outcome quality, data sources should cover the full population and hospitals need to improve ICD-10-coding of sepsis.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: The data used in this study cannot be made available in the manuscript, the Supplementary material, or in a public repository due to German data protection laws (Bundesdatenschutzgesetz). Therefore, they are stored on a secure drive in the Wissenschaftliches Institut der AOK to facilitate replication of the results. Generally, access to data of statutory health insurance funds for research purposes is possible only under the conditions defined in the German Social Law (SGB V §287). Requests for data access can be sent as a formal proposal, specifying the recipient and purpose of the data transfer, to the appropriate data protection agency. Access to the data used in this study can only be provided to external parties under the conditions of the cooperation contract of this research project and after written approval by the AOK. For assistance in obtaining access to the data, please contact d2lkb0B3aWRvLmJ2LmFvay5kZQ==.

Ethics statement

The studies involving human participants were reviewed and approved by Ethik-Kommission der Friedrich-Schiller-Universität Jena. Written informed consent for participation was not required for this study in accordance with the national legislation and the institutional requirements.

Author contributions

DS, CF-S, and MS: concept and design. CF-S, NR, MS, CG, and LW: acquisition of data. LW, DS, NR, and PS: statistical analysis and planning. LW and DS: drafting of the manuscript. DS: supervision. CG, AS, and PS: administrative support. LW: conduction and had full access to all of the data in the study and takes responsibility for the integrity of the data and the accuracy of the data analysis. All authors contributed to interpretation of results and critical revision of the manuscript for important intellectual content. All authors contributed to the article and approved the submitted version.

Funding

This study was funded by the grant 01VSF17010 from the German Innovations Fund of the Federal Joint Committee. We acknowledge support by the German Research Foundation Project-Nr. 512648189 and the Open Access Publication Fund of the Thüringer Universitäts- und Landesbibliothek Jena.

Conflict of interest

CF-S was funded by grants from the German Federal Ministry of Education and Research, outside the submitted work and the German Innovations Fund of the Federal Joint Committee in Germany (G-BA), inside and outside the submitted work. DS leads the coordinating bureau of the German Quality Network sepsis–a quality initiative to improve hospital care for sepsis–and was partly funded via this project from 2015 to 2022. KR was shareholder with less of 0.5% of InflaRx NV a Jena/Germany based Biotech Company that evaluates an immunmodulatory approach for the adjunctive treatment of COVID-19. PS was funded by grants from the German Federal Ministry of Education and Research, outside the submitted work and the German Innovations Fund of the Federal Joint Committee in Germany (G-BA) outside the submitted work. AS was funded by grants from the German Federal Ministry of Education and Research, outside the submitted work.

The remaining authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2022.1069042/full#supplementary-material

Abbreviations

AOK, “Allgemeine Ortskrankenkasse,” German health insurance; AUC, area under the curve; CCI, Charlson Comorbidity Index; ECI, Elixhauser Comorbidity Index; HGLM, hierarchical generalized linear models; ICD, international classification of diseases; ICD-10-GM, international statistical classification of diseases and related health problems–German modification–10th revision; IK, “Institutionskennzeichen,” unique institutional identifier for hospitals; OPS, Operationen und Prozedurenschlüssel [German procedure classification]; QSR, “Qualitätssicherung mit Routinedaten,” quality assurance using routine data; RSMR, risk standardized mortality rate; SEPFROK, initial study, “Sepsis: Folgeerkrankungen, Risikofaktoren, Versorgung, und Kosten”; USA, United States of America; WIdO, Research Institute of the Local Health Care Funds.

References

1. Singer M, Deutschman C, Seymour C, Shankar-Hari M, Annane D, Bauer M, et al. The third international consensus definitions for sepsis and septic shock (sepsis-3). J Am Med Assoc. (2016) 315:801–10. doi: 10.1001/jama.2016.0287

PubMed Abstract | CrossRef Full Text | Google Scholar

2. Rudd K, Johnson S, Agesa K, Shackelford K, Tsoi D, Kievlan D, et al. Global, regional, and national sepsis incidence and mortality, 1990–2017: analysis for the global burden of disease study. Lancet. (2020) 395:200–11. doi: 10.1016/S0140-6736(19)32989-7

CrossRef Full Text | Google Scholar

3. Goodwin A, Srivastava V, Shotton H, Protopapa K, Butt A, Mason M. Just Say Sepsis! A Review of the Process of Care Received by Patients With Sepsis. (2015). Available online at: http://www.ncepod.org.uk/2015report2/downloads/JustSaySepsis_FullReport.pdf (accessed December 22, 2022).

Google Scholar

4. Prescott H, Angus D. Enhancing recovery from sepsis a review. J Am Med Assoc. (2018) 319:62–75. doi: 10.1001/jama.2017.17687

PubMed Abstract | CrossRef Full Text | Google Scholar

5. World Health Assembly Executive Board. EB140.R5 Improving the Prevention, Diagnosis and Management of Sepsis. (2017). Available online at: http://apps.who.int/gb/ebwha/pdf_files/EB140/B140_R5-en.pdf (accessed December 27, 2019).

Google Scholar

6. Brook R, McGlynn E, Cleary P. Quality of health care .2. Measuring quality of care. N Engl J Med. (1996) 335:966–70. doi: 10.1056/NEJM199609263351311

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Iezzoni L. Assessing quality using administrative data. Ann Intern Med. (1997) 127:666–74. doi: 10.7326/0003-4819-127-8_Part_2-199710151-00048

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Iezzoni L. Risk adjustment. In: Smith P, Mossialos E, Papanicolas I, Leatherman S editors. Performance Measurement for Health System Improvement: Experiences, Challenges and Prospects. New York, NY: Cambridge University Press (2009). p. 251–85.

Google Scholar

9. Krumholz H, Brindis R, Brush J, Cohen D, Epstein A, Furie K, et al. Standards for statistical models used for public reporting of health outcomes – an American Heart association scientific statement from the quality of care and outcomes research interdisciplinary writing group - cosponsored by the council on epidemiology and prevention and the stroke council - endorsed by the American college of cardiology foundation. Circulation. (2006) 113:456–62. doi: 10.1161/CIRCULATIONAHA.105.170769

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Ford D, Goodwin A, Simpson A, Johnson E, Nadig N, Simpson KNA. Severe sepsis mortality prediction model and score for use with administrative data. Crit Care Med. (2016) 44:319–27. doi: 10.1097/CCM.0000000000001392

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Darby J, Davis B, Barbash I, Kahn J. An administrative model for benchmarking hospitals on their 30-day sepsis mortality. BMC Health Serv Res. (2019) 19:221. doi: 10.1186/s12913-019-4037-x

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Schwarzkopf D, Fleischmann-Struzek C, Rüddel H, Reinhart K, Thomas-Rüddel DO. A risk-model for hospital mortality among patients with severe sepsis or septic shock based on German national administrative claims data. PLoS One. (2018) 13:e0194371. doi: 10.1371/journal.pone.0194371

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Hu J, Yo C, Lee H, Su C, Su M, Huang A, et al. Risk-standardized sepsis mortality map of the United States. Digit Health. (2022) 8:20552076211072400. doi: 10.1177/20552076211072400

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Kahn J, Davis B, Yabes J, Chang C, Chong D, Hershey T, et al. Association between state-mandated protocolized sepsis care and in-hospital mortality among adults with sepsis. JAMA. (2019) 322:240–50. doi: 10.1001/jama.2019.9021

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Schwarzkopf D, Rueddel H, Gruendling M, Putensen C, Reinhart K. The German quality network sepsis: study protocol for the evaluation of a quality collaborative on decreasing sepsis-related mortality in a quasi-experimental difference-in-differences design. Implement Sci. (2018) 13:15. doi: 10.1186/s13012-017-0706-5

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Iwashyna T, Angus D. Declining case fatality rates for severe sepsis good data bring good news with ambiguous implications. JAMA. (2014) 311:1295–7. doi: 10.1001/jama.2014.2639

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Lawthers A, McCarthy E, Davis R, Peterson L, Palmer R, Iezzoni L. Identification of in-hospital complications from claims data: is it valid? Med Care. (2000) 38:785–95. doi: 10.1097/00005650-200008000-00003

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Quan H, Parsons G, Ghali W. Validity of information on comorbidity derived rom ICD-9-CCM administrative data. Med Care. (2002) 40:675–85. doi: 10.1097/00005650-200208000-00007

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Fleischmann-Struzek C, Rose N, Freytag A, Spoden M, Prescott H, Schettler A, et al. Epidemiology and costs of postsepsis morbidity, nursing care dependency, and mortality in Germany, 2013 to 2017. JAMA Netw Open. (2021) 4:e2134290. doi: 10.1001/jamanetworkopen.2021.34290

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Wissenschaftliches Institut der Aok (WIdO). QSR-Verfahren – Indikatorenhandbuch - Verfahrensjahr 2021. Berlin: Wissenschaftliches Institut der AOK (WIdO) (2021).

Google Scholar

21. Manja V, AlBashir S, Guyatt G. Criteria for use of composite end points for competing risks-a systematic survey of the literature with recommendations. J Clin Epidemiol. (2017) 82:4–11. doi: 10.1016/j.jclinepi.2016.12.001

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Lagu T, Lindenauer P, Rothberg M, Nathanson B, Pekow P, Steingrub J, et al. Development and validation of a model that uses enhanced administrative data to predict mortality in patients with sepsis. Crit Care Med. (2011) 39:2425–30. doi: 10.1097/CCM.0b013e31822572e3

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Mohr N, Harland K, Shane D, Ahmed A, Fuller B, Torner J. Inter-hospital transfer is associated with increased mortality and costs in severe sepsis and septic shock: an instrumental variables approach. J Crit Care. (2016) 36:187–94. doi: 10.1016/j.jcrc.2016.07.016

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Osborn T, Phillips G, Lemeshow S, Townsend S, Schorr C, Levy M, et al. Sepsis severity score: an internationally derived scoring system from the surviving sepsis campaign database. Crit Care Med. (2014) 42:1969–76. doi: 10.1097/CCM.0000000000000416

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Charlson M, Pompei P, Ales K, Mackenzie C. A new method of classifying prognostic co-morbidity in longitudinal-studies - development and validation. J Chronic Dis. (1987) 40:373–83. doi: 10.1016/0021-9681(87)90171-8

CrossRef Full Text | Google Scholar

26. Elixhauser A, Steiner C, Harris D, Coffey R. Comorbidity measures for use with administrative data. Med Care. (1998) 36:8–27. doi: 10.1097/00005650-199801000-00004

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Quan H, Sundararajan V, Halfon P, Fong A, Burnand B, Luthi J, et al. Coding algorithms for defining comorbidities in ICD-9-CM and ICD-10 administrative data. Med Care. (2005) 43:1130–9. doi: 10.1097/01.mlr.0000182534.19832.83

CrossRef Full Text | Google Scholar

28. Hagn S. Vergleich Verschiedener Komorbiditäts-Scores in Routinedaten der Stationären Versorgung. München: Institut für Medizinische Informationsverarbeitung (2014).

Google Scholar

29. Angus D, Linde-Zwirble W, Lidicker J, Clermont G, Carcillo J, Pinsky M. Epidemiology of severe sepsis in the United States: analysis of incidence, outcome, and associated costs of care. Crit Care Med. (2001) 29:1303–10. doi: 10.1097/00003246-200107000-00002

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Wang H, Shapiro N, Angus D, Yealy D. National estimates of severe sepsis in United States emergency departments. Crit Care Med. (2007) 35:1928–36. doi: 10.1097/01.CCM.0000277043.85378.C1

CrossRef Full Text | Google Scholar

31. Wilhelms S, Huss F, Granath G, Sjoberg F. Assessment of incidence of severe sepsis in Sweden using different ways of abstracting International classification of diseases codes: difficulties with methods and interpretation of results. Crit Care Med. (2010) 38:1442–9. doi: 10.1097/CCM.0b013e3181de4406

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Normand S, Glickman M, Gatsonis C. Statistical methods for profiling providers of medical care: Issues and applications. J Am Stat Assoc. (1997) 92:803–14. doi: 10.1080/01621459.1997.10474036

CrossRef Full Text | Google Scholar

33. Harrell F Jr, Lee K, Mark D. Multivariable prognostic models: issues in developing models, evaluating assumptions and adequacy, and measuring and reducing errors. Stat Med. (1996) 15:361–87. doi: 10.1002/(SICI)1097-0258(19960229)15:4<361::AID-SIM168>3.0.CO;2-4

CrossRef Full Text | Google Scholar

34. Mittlbock M, Schemper M. Explained variation for logistic regression. Stat Med. (1996) 15:1987–97. doi: 10.1002/(SICI)1097-0258(19961015)15:19<1987::AID-SIM318>3.0.CO;2-9

CrossRef Full Text | Google Scholar

35. Bratzler D, Normand S, Wang Y, O’Donnell W, Metersky M, Han L, et al. An administrative claims model for profiling hospital 30-day mortality rates for pneumonia patients. PLoS One. (2011) 6:e17401. doi: 10.1371/journal.pone.0017401

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Kuhn M, Johnson K. Applied Predictive Modeling. New York, NY: Springer (2013). doi: 10.1007/978-1-4614-6849-3

CrossRef Full Text | Google Scholar

37. DeLong E, Peterson ED, DeLong D, Muhlbaier L, Hackett S, Mark D. Comparing risk-adjustment methods for provider profiling. Stat Med. (1997) 16:2645–64. doi: 10.1002/(SICI)1097-0258(19971215)16:23<2645::AID-SIM696>3.0.CO;2-D

CrossRef Full Text | Google Scholar

38. Shahian D, Torchiana D, Shemin R, Rawn J, Normand S. Massachusetts cardiac surgery repory card: Implications of statistical methodology. Ann Thorac Surg. (2005) 80:2106–13. doi: 10.1016/j.athoracsur.2005.06.078

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Dimick J, Staiger D, Birkmeyer J. Ranking hospitals on surgical mortality: the importance of reliability adjustment. Health Serv Res. (2010) 45:1614–29. doi: 10.1111/j.1475-6773.2010.01158.x

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Goldstein H, Spiegelhalter D. League tables and their limitations: statistical issues in comparisons of institutional performance. J R Stat Soc Ser Stat Soc. (1996) 159:385–409. doi: 10.2307/2983325

CrossRef Full Text | Google Scholar

41. MacKenzie T, Grunkemeier G, Grunwald G, O’Malley A, Bohn C, Wu Y, et al. A primer on using shrinkage to compare in-hospital mortality between centers. Ann Thorac Surg. (2015) 99:757–61. doi: 10.1016/j.athoracsur.2014.11.039

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Normand S, Wang Y, Krumholz H. Assessing surrogacy of data sources for institutional comparisons. Health Servic Outcomes Res Methodol. (2007) 7:79–96. doi: 10.1007/s10742-006-0018-8

CrossRef Full Text | Google Scholar

43. R Core Team. R: A Language and Environment for Statistical Computing. Vienna: R Foundation for Statistical Computing (2020).

Google Scholar

44. Phillips G, Osborn T, Terry K, Gesten F, Levy M, Lemeshow S. The New York sepsis severity score: development of a risk-adjusted severity model for sepsis. Crit Care Med. (2018) 46:674–83. doi: 10.1097/CCM.0000000000002824

PubMed Abstract | CrossRef Full Text | Google Scholar

45. van Walraven C, Austin P, Jennings A, Quan H, Forster AJA. Modification of the elixhauser comorbidity measures into a point system for hospital death using administrative data. Med Care. (2009) 47:626–33. doi: 10.1097/MLR.0b013e31819432e5

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Thompson N, Fan Y, Dalton J, Jehi L, Rosenbaum B, Vadera S, et al. A new elixhauser-based comorbidity summary measure to predict in-hospital mortality. Med Care. (2015) 53:374–9. doi: 10.1097/MLR.0000000000000326

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Evans L, Rhodes A, Alhazzani W, Antonelli M, Coopersmith C, French C, et al. Surviving sepsis campaign: international guidelines for management of sepsis and septic shock 2021. Intensive Care Med. (2021) 47:1181–247.

Google Scholar

48. Shankar-Hari M, Saha R, Wilson J, Prescott H, Harrison D, Rowan K, et al. Rate and risk factors for rehospitalisation in sepsis survivors: systematic review and meta-analysis. Intensive Care Med. (2020) 46:619–36. doi: 10.1007/s00134-019-05908-3

PubMed Abstract | CrossRef Full Text | Google Scholar

49. Shankar-Hari M, Rubenfeld G, Ferrando-Vivas P, Harrison D, Rowan K. Development, validation, and clinical utility assessment of a prognostic score for 1-year unplanned rehospitalization or death of adult sepsis survivors. JAMA Netw Open. (2020) 3:e2013580. doi: 10.1001/jamanetworkopen.2020.13580

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Quentin W, Partanen V, Brownwood I, Klazinga N. Measuring healthcare quality. In: Busse R, Klazinga N, Panteli D, Quentin W editors. Improving Healthcare Quality in Europe: Characteristics, Effectiveness and Implementation of Different Strategies. Health Policy Series. 53. Copenhagen: European Observatory on Health Systems and Policies (2019).

Google Scholar

51. Prescott H, Costa D. Improving long-term outcomes after sepsis. Crit Care Clin. (2018) 34:175–88. doi: 10.1016/j.ccc.2017.08.013

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Prescott H, Iwashyna T, Blackwood B, Calandra T, Chlan L, Choong K, et al. Understanding and enhancing sepsis survivorship. Priorities for research and practice. Am J Respir Crit Care Med. (2019) 200:972–81. doi: 10.1164/rccm.201812-2383CP

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Iqtig. Methodische Grundlagen: Entwurf für Version 2.0. Berlin: IQTIG (2021).

Google Scholar

54. Nimptsch U, Mansky T. G-IQI Version 5.0 – Messen, um zu verbessern. 2 ed. In: Eberlein-Gonska M, Martin J, Zacher J editors. Handbuch IQM: Kosequent transparent – Qualität mit Routinedaten!. Berlin: Medizinisch Wissenschaftliche Verlagsgesellschaft (2017).

Google Scholar

55. Mansky T, Nimptsch U. Hospital quality measurement – what matters? Z Evid Fortbild Qual Gesundhwes. (2014) 108:487–94. doi: 10.1016/j.zefq.2014.09.023

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Fleischmann-Struzek C, Mikolajetz A, Schwarzkopf D, Cohen J, Hartog C, Pletz M, et al. Challenges in assessing the burden of sepsis and understanding the inequalities of sepsis outcomes between national health systems: secular trends in sepsis and infection incidence and mortality in Germany. Intensive Care Med. (2018) 44:1826–35. doi: 10.1007/s00134-018-5377-4

PubMed Abstract | CrossRef Full Text | Google Scholar

57. IQTIG. Qualitätssicherungsverfahren Diagnostik, Therapie und Nachsorge der Sepsis. Konzeptstudie. Berlin: IQTIG (2019).

Google Scholar

58. Jolley R, Sawka K, Yergens D, Quan H, Jette N, Doig C. Validity of administrative data in recording sepsis: a systematic review. Crit Care. (2015) 19:12. doi: 10.1186/s13054-015-0847-3

PubMed Abstract | CrossRef Full Text | Google Scholar

59. Fleischmann-Struzek C, Thomas-Rüddel D, Schettler A, Schwarzkopf D, Stacke A, Seymour C, et al. Comparing the validity of different ICD coding abstraction strategies for sepsis case identification in German claims data. PLoS One. (2018) 13:e0198847. doi: 10.1371/journal.pone.0198847

PubMed Abstract | CrossRef Full Text | Google Scholar

60. Walkey A, Shieh M, Liu V, Lindenauer P. Mortality measures to profile hospital performance for patients with septic shock. Crit Care Med. (2018) 46:1247–54. doi: 10.1097/CCM.0000000000003184

PubMed Abstract | CrossRef Full Text | Google Scholar

61. Shankar-Hari M, Harrison D, Rubenfeld G, Rowan K. Epidemiology of sepsis and septic shock in critical care units: comparison between sepsis-2 and sepsis-3 populations using a national critical care database. Br J Anaesth. (2017) 119:626–36. doi: 10.1093/bja/aex234

PubMed Abstract | CrossRef Full Text | Google Scholar

62. Arefian H, Hagel S, Heublein S, Rissner F, Scherag A, Brunkhorst F, et al. Extra length of stay and costs because of health care–associated infections at a German university hospital. Am J Infect Control. (2016) 44:160–6. doi: 10.1016/j.ajic.2015.09.005

PubMed Abstract | CrossRef Full Text | Google Scholar