PRAgmatic Clinical Trial Design of Integrative MediCinE (PRACTICE): A Focus Group Series and Systematic Review on Trials of Diabetes and Kidney Disease

Background: Pragmatic trials inform clinical decision with better generalizability and can bridge different streams of medicine. This study collated the expectations regarding pragmatic trial design of integrative medicine (IM) for diabetes and kidney diseases among patients and physicians. Dissonance between users' perspective and existing pragmatic trial design was identified. The association between risk of bias and pragmatism of study design was assessed.

Method: A 10-group semi-structured focus group interview series [21 patients, 14 conventional medicine (ConM) and 15 Chinese medicine (CM) physicians] were purposively sampled from private and public clinics in Hong Kong. Perspectives were qualitatively analyzed by constant comparative method. A systematic search of four databases was performed to identify existing IM pragmatic clinical trials in diabetes or kidney disease. Primary outcomes were the pragmatism, risk of bias, and rationale of the study design. Risk of bias and pragmatism were assessed based on Cochrane risk-of-bias tool and PRECIS-2, respectively. The correlation between risk of bias and pragmatism was assessed by regression models with sensitivity analyses.

Results: The subtheme on the motivation to seek IM service was analyzed, covering the perceived limitation of ConM effect, perceived benefits of IM service, and assessment of IM effectiveness. Patients expected IM service to retard disease progression, stabilize concomitant drug dosage, and reduce potential side effects associated with ConM. In the systematic review, 25 studies from six countries were included covering CM, Korean medicine, Ayurvedic medicine, and western herbal medicine. Existing study designs did not include a detailed assessment of concomitant drug change and adverse events. Majority of studies either recruited a non-representative proportion of patients as traditional, complementary, and integrative medicine (TCIM) diagnosis was used as inclusion criteria, or not reflecting the real-world practice of TCIM by completely dropping TCIM diagnosis in the trial design. Consultation follow-up frequency is the least pragmatic domain. Increase in pragmatism did not associate with a higher risk of bias.

Conclusion: Existing IM pragmatic trial design does not match the patients' expectation in the analysis of incident concomitant drug change and adverse events. A two-layer design incorporating TCIM diagnosis as a stratification factor maximizes the generalizability of evidence and real-world translation of both ConM and TCIM.

Existing Evidence

Pragmatic trials better reflect real-world effectiveness of interventions. Integrative medicine (IM) amalgamates multiple streams of medicine with different disease classifications and treatment strategies which require pragmatic assessment. However, existing pragmatic trial design seldom considers users' perspective and there are concerns on whether flexibilities in pragmatic trial design would compromise internal validity.

Key Contributions to the Literature

1. This is the first focus group series to explore the expected outcomes of patients and physicians regarding pragmatic trial design of IM for diabetes and renal service, involving patients and family medicine, internal medicine, and Chinese medicine (CM) physicians. Unmatched expectation in existing studies was identified through systematic review.

2. Patients expected integrative Chinese-western medicine service to retard disease progression, stabilize concomitant drug dosage, and reduce potential side effects associated with conventional treatment.

3. Existing IM pragmatic trial designs did not include detailed assessment of concomitant drug change and adverse events. Consultation follow-up frequency is the least pragmatic domain in existing IM pragmatic trials.

4. Majority of studies either recruited a non-representative proportion of patients by using traditional, complementary, and integrative medicine (TCIM) diagnosis as inclusion criteria, or not reflecting the real-world practice of TCIM by completely dropping TCIM diagnosis.

5. Increase in pragmatism in study design did not associate with a higher risk of bias from existing evidence.

Implications

Existing IM pragmatic trial design does not match users' expectation in the analysis of incident concomitant drug changes and adverse events. A two-layer design incorporating TCIM diagnosis as a stratification factor maximizes the generalizability of evidence and real-world translation for both conventional medicine and TCIM.

Introduction

Pragmatic trials evaluate the effectiveness of interventions in the real-world setting aiming to inform clinical decision and implementation with better generalizability (1, 2). Compared to conventional phase III randomized controlled trials, pragmatic trials often are open-label, have less stringent inclusion/exclusion criteria, involve complex/flexible interventions, compare to usual care, and measure outcomes that are patient-centered (1, 2). Integrative medicine (IM) amalgamates conventional medicine (ConM) and other streams of medicine from a patient-centered and effectiveness-driven approach (3–5).

Traditional, complementary, and integrative medicine (TCIM), including Chinese medicine (CM), naturopathic medicine, mind–body therapies, and other streams of medicine, are often personalized as their theories were developed predominantly from expert consensus and case series (6). Differences in epistemology (for instance, disease classification and treatment strategy) between ConM and TCIM led to controversies in the evaluation of TCIM's effectiveness (7–10). Most clinical trials and meta-analyses were designed to estimate the adjusted or averaged effectiveness of a regimen from a population of patients. However, the likelihood of being responsive toward a regimen of each individual patient with distinctive demographics and phenotypes is often more needed by a physician in the clinical situation (11–13). There are continuous concerns on the conventional evidence-based paradigm building on meta-analyses and randomized controlled trials with limited personalized design (e.g., prespecified subgroup analysis, responder analysis), such as being over-concentrated in population-based assessment (14, 15), over-standardized treatment (15, 16), and lacking personalization (17). This affected the clinical utility of the evidence (18) and was contradicted with many core principles of TCIM. The efficacy-driven approach, which focuses on comparative effectiveness, has been proposed to bridge ConM and TCIM (8, 19–22).

Stakeholder (e.g., patients and physicians) engagement is the foundation of designing pragmatic studies (2, 23). Stakeholder involvement in the study design stage, from the selection of disease condition, drug formulation, and outcome measurement, is increasingly emphasized to enhance the clinical utility and translation of evidence (18, 24). Nevertheless, there are controversies over the pragmatic features (e.g., unblinding of subjects, no placebo control, intervention adjustment) as these flexibilities may enhance generalizability at the expense of internal validity of the evidence (25, 26). The correlation between risk of bias and pragmatism remains unclear.

Diabetes presented in 9.5% of adult population and accounted for 9.9% of all-cause mortality globally (27, 28). The healthcare expenditure on diabetes mounted to US $850 billion worldwide in 2017, representing 11.6% of the total health expenditure (27, 28). Both diabetes and kidney dysfunction are the top 10 conditions attributed to disability-adjusted life-years among population aged over 25 globally (29). In the past decade, CM formulations have been reported to protect against diabetes and chronic kidney disease (CKD) via orchestrated mechanisms (30–35). However, less than 2% of diabetic patients have ever used CM for diabetes or CKD in Hong Kong which was substantially lower than the utilization in other disciplines (e.g., 50% for cancer patients) (36). Lack of high-quality and communicable evidence has been suggested as one of the key obstacles in implementing IM (6).

This study aimed to collate and explore the expectations regarding the pragmatic trial design of IM for diabetes among patients and physicians. Subsequently, the existing trial design was systematically assessed to identify the dissonance with users' perspective.

Methods

Study Design

A 10-group semi-structured focus group interview series was conducted among patients and physicians with constant comparative method to explore their expectation regarding the IM management of diabetes in general (37). Seven high-level themes were previously identified from the interview series. Two themes regarding the barriers to access and the preferred delivery mode of health services were reported (6). In this study, we report another major theme related to pragmatic trial design. A systematic review was conducted subsequently to contrast existing IM pragmatic trials to the users' perspectives identified from the focus group interviews.

Focus Group Interview

The focus group interview series was designed to explore the expectations and concerns of the patients and physicians regarding the IM service access and further research. Detail of the interview methods was previously described (6). Briefly, 50 subjects (21 diabetes patients, 14 ConM physicians, and 15 CM physicians) with diverse demographics and experience were purposely sampled from public clinics, private clinics and teaching hospitals in Hong Kong. A series of face-to-face group interviews with three groups of 6–8 patients, three groups of 3–6 ConM physicians, and four groups of 3–4 CM physicians were conducted. Each interview lasted 60–120 min allowing at least 20 min per participant for adequate interaction. CM physicians were sampled to represent TCIM in Hong Kong as CM is the mainstream of TCIM, and integrative Chinese-western medicine is the major form of IM globally including Hong Kong (38).

The interviews were facilitated by a moderator (P.W.L) with relevant experience and conducted in Cantonese (native language of participants). The identity of interviewees and the moderator was blinded before the interview took place. The interviews were built around participants' consultation experience, concerns and expectations based on a semi-structured interview guide (6). The process of recruitment, interview and analysis were iterative until data saturation during the last round of interview (patient and ConM: third round, CM: fourth round). Interview content was analyzed by constant comparative method (37). Maximum codes on main themes and subthemes were first generated independently by two bilingual investigators (K.W.C., P.W.L.) for initial open coding with revisit to check for emerging ideas. The concepts and theories were refined, and the association of the coding was explored to form axial coding. Final core coding was formed after data saturation and was applied to index the whole dataset. Charted result was translated by a bilingual investigator (K.W.C.) when used as illustrative quotations. Data were processed with the support of simple software (Microsoft Word and Excel) for convenient access.

Systematic Review

Search

We sought to assess the pragmatism, risk of bias, and rationale of study design of the existing pragmatic trials of diabetes and kidney disease using IM as intervention. The search strategy (Supplementary File 1) was formulated to include all IM pragmatic clinical trials and trial protocols that recruit patients with diabetes or kidney diseases published until 24 August 2020. IM included any intervention that is not conventionally used in clinical practice, for instance, herbal medicine, acupuncture, and massage. Four databases were searched including Cochrane, Medline, Embase and PubMed. Reference lists were also searched. A clinical epidemiologist (K.W.C.) led the search and data processing. Endnote X9 was used to aid the review process. Protocol registration: CRD4D2021231288.

Screening

After removing duplicated studies, screening started with title and abstract followed by full text before data extraction. All articles were dually screened, assessed, and extracted (Y.K.L, L.G.) independently with a standardized form. All disagreements were resolved by discussion and determined by K.W.C. if consensus could not be reached. There was no language restriction. All observational and qualitative studies were excluded. Studies that used health services or supplements as intervention were excluded.

Quality Assessment and Data Extraction

The co-primary outcomes were the pragmatism, risk of bias, and rationale of the study design. Pragmatism of the trials was assessed based on the PRECIS-2 tool (39) on study population, recruitment setting, intervention delivery, and outcome assessment. Risk of bias in randomization, allocation concealment, blinding, incomplete outcome data, and selective reporting was assessed based on the Cochrane risk-of-bias tool (40). The rationale of study design in target population, intervention, comparator, and outcome assessment were identified from the study.

Statistical Analysis

The correlation between risk of bias and pragmatism was assessed by univariable and backward multivariable regression analysis adjusting publication year and sample size. For the quantified assessment of the overall risk of bias of each study, the scores of low, unknown, and high risk were given 0, 1, and 2 points. Lower total score represents low risk of bias in the reported study design. For pragmatism, each domain scored 1 for being least pragmatic and 5 for being most pragmatic, respectively, according to the guideline from the PRECIS-2 tool. For domains that were not assessable, the score was replaced by 3 (midpoint). As there is no consensus on the statistical handling of undetermined domains, sensitivity analysis was conducted to replace undetermined domains by 1 and 5 to test the robustness of results. STATA 15.1 was used for analysis.

Results

Focus Group Interviews

Majority of patients had poor glycemic control (71.4%), with stage 2–4 CKD (95.2%) and albuminuria (90.5%); 4.8% of patients reached end-stage kidney failure, 57.1% (n = 8/14) of ConM physicians specialized in internal medicine, 42.9% (n = 6/14) of ConM physicians specialized in family medicine or practiced as general practitioners, 42.9% (n = 6/14) of ConM physicians received CM education, and all (n = 15) CM physicians received substantial credit bearing ConM education from their undergraduate study. Seven high-level themes, namely, barriers toward IM service, motivation to seek CM service, background knowledge on diabetes, experience of CM service, preferred model of integrative service delivery, and evidence of IM and CM hospital, were previously identified leading to 25 subthemes (6). Data on a high-level theme: motivation to seek IM service is related to the clinical trial design and reported in this study (Figure 1). Quotes are summarized in Table 1.

FIGURE 1

Figure 1. Motivation to seek integrative medicine (IM) service. Themes generally agreed upon by patients in yellow, by Chinese medicine (CM) physicians in blue, by conventional medicine (ConM) physicians in red, by both patient and CM physicians in green, by both patient and ConM clinician in orange, by all parties in black. Control of disease progression was the common perceived benefit of IM. Stabilizing ConM usage was emphasized by patients and acknowledged by CM physicians. Surrogate biomarkers were mutually accepted among patients and physicians. Importance on quality of life divided between patients and CM physicians.

TABLE 1

Table 1. Subthemes and illustrative quotations of focus group interview.

Main Theme: Motivation to Seek IM Service

Four subthemes related to the motivation of seeking IM service were identified, namely, (1) perceived limitation of ConM effect, (2) peer or media influence, (3) perceived benefits of IM service, and (4) assessment of IM effectiveness. Subthemes 1, 3, and 4 are relevant to study design and summarized below.

Subtheme: Perceived Limitation of ConM Effect

Majority of patients considered IM as they believed the effect of ConM was limited and was concerned about the adverse effects after receiving ConM.

Limitation in ConM Efficacy

Most patients believed that diabetes and diabetic kidney disease (DKD) are irreversible, which was reflected by the limitation of the current regimens (41–44). This prompted patients to explore alternatives for more options to control disease progression. Physicians from both ConM and CM acknowledged that patients generally prefer IM treatment. Majority of patients approached IM when they experienced disease progression, for example, poor blood glucose control, or developed complications including DKD.

ConM-Associated Adverse Effect

Patients mentioned their experience in developing adverse effects that perceived to be ConM-associated. These included hypoglycemia, hyperkalemia, diarrhea, fluctuating blood glucose, and fatigue. Majority of patients believed that CM has less adverse effects when compared to ConM. A similar observation was suggested by CM physicians.

Subtheme: Perceived Benefits of IM Service

There are several benefits that patients believed IM can offer, including better control of disease progression, prevention of ConM-associated side effect, and stabilizing ConM usage.

Better Control of Disease Progression

Patients sought to have better control of disease progression, for instance, reducing the risk of complications and increasing life expectancy when they consider IM. DKD was highlighted as a major concern as patients were reluctant to receive dialysis. Some CM physicians suggested that patients of different age groups had different treatment targets. Elder patients emphasized more on symptomatic improvement and quality of life, while younger patients focused on laboratory investigations. A few CM physicians suggested that CM emphasizes holistic improvement including both quality of life and biomarkers.

Although patients expressed subjective unwell feeling after receiving ConM, symptomatic improvement did not emerged as a major expectation from patients. CM physicians, however, believed that improving quality of life would be a major concern among patients and an advantage of CM. ConM physicians suggested control of renal function deterioration as an important milestone of complication management; however, they emphasized that more evidence is needed to demonstrate such effect of CM.

Stabilizing ConM Usage and Preventing the Associated Adverse Effects

Reducing ConM dependence was one of the common expectations of patients. Some CM physicians reported similar requests encountered in their clinical practice. This is likely because patients linked the use of ConM with disease progression and adverse effects. Minority of patients expect CM to reduce the adverse effects of ConM. Some CM physicians suggested that they have managed ConM-associated adverse events.

Subtheme: Assessment of IM Effectiveness

Patients generally focused on objective conventional biomarkers measured by laboratory investigations for the monitoring of treatment effect, which was supported by the ConM clinicians. Some CM physicians also believed that objective markers were important for their self-evaluation of treatment effect, as DKD is a ConM-defined condition. They also expected the patients would evaluate their treatment based on laboratory investigation results.

Substantially diverted opinion was noted among CM physicians, suggesting current biomarkers should not be the only outcome assessment. They believe CM manages patients' general condition simultaneously while treating DKD. DKD-related biomarkers were limited to only reflect a certain aspect of patients' overall condition. They suggest the concurrent use of CM-related outcome measures, which is phenome-based (e.g., change in symptoms, tongue color and pulse form).

Some ConM physicians acknowledged the difference in the epistemology between CM and ConM and suggested that CM may require different outcome measures. Nevertheless, ConM physicians generally believed that it would be an advantage if the effect of CM can be demonstrated with study designs conventionally used in ConM. There was also a suggestion to personalize the assessment of effect based on the patients' preference which is related to their demographics.

Systematic Review

Our search identified 303 studies from four databases after removing duplicated studies (Figure 2; 264 studies were excluded by title and abstract screening and 14 studies were excluded (Supplementary File 2S) after full-text screening. A total of 25 trials were included for analysis.

FIGURE 2

Figure 2. Flow diagram of systematic review. Four databases and search engines were searched and 25 papers were included. Interventions included acupuncture/acupressure (n = 7), herbal products (n = 14), massage-related (n = 2), qigong (n = 1) and combined acupuncture-herbal (n = 1) treatment. The treatment was formulated according to Chinese (n = 20), Kampo (n = 2), Korean (n = 1), Ayurvedic (n = 1), and western herbal (n = 1) medicine.

Characteristics of Included Trials

Geographically, 18 (72%), 2 (8%), 2 (8%), 1 (4%), 1 (4%), and 1 (4%) studies were conducted in China, Japan, United Kingdom, United States, Korea, and Sweden, respectively (Table 2). Target population included prediabetic (n = 2), diabetic (n = 15), glomerulonephritis (n = 1), chronic kidney disease (n = 2) and hemodialysis (n = 5) patients. Complication of diabetes included kidney (n = 4), neuropathy (n = 4) and arterial disease (n = 1). Fifteen were completed trials and 10 were trial protocols.

TABLE 2

Table 2. Characteristics of included studies.

The IM interventions involved included acupuncture/acupressure (n = 7), herbal products (n = 14), massage-related (n = 2), qigong (n = 1), and combined acupuncture-herbal (n = 1) treatment. The treatment was formulated according to Chinese (n = 20), Kampo (n = 2), Korean (n = 1), Ayurvedic (n = 1), and western herbal (n = 1) medicine. The median sample size was 113 (IQR: 72–266), and the median treatment duration was 24 weeks (IQR: 10–26). The frequency of treatment ranged from once to three times daily for oral medication and once to three times weekly for acupuncture, respectively. Majority of studies required monthly consultation follow-up for oral medication and three times weekly for acupuncture. Twenty-four studies (96%) either recruited a proportion of patients according to TCIM-specific diagnosis or completely dropped TCIM diagnosis in study design. Five studies included TCIM-specific symptom-based diagnostic criteria in the inclusion/exclusion criteria of study population.

All DKD- related studies used urine albumin/protein and/or estimated glomerular filtration rate (GFR) as primary outcomes. All CKD-related studies assessed estimated GFR as the primary outcome. For hemodialysis-related studies, majority (4/5) assessed quality of life or symptom as primary outcomes. All studies described adverse events narratively. No studies measured the change of concurrent medication as primary or secondary outcomes. Nine, 12, two and two studies used standard care, placebo or sham acupuncture, both standard care and placebo, and other active intervention (e.g., other TCIM medication, active exercise) as comparators, respectively.

Risk of Bias, Pragmatism and the Association

Majority (22/25) of studies reported unclearly in at least one domain of potential bias (Figure 3). Twelve studies had unclear description on handling of attrition that led to undetermined bias on completeness of outcome measurement. Four studies were with high risk of bias in at least one domain. The main source of high-risk bias was from the blinding of outcome assessment (n = 3) and allocation concealment (n = 2).

FIGURE 3

Figure 3. Risk of bias and pragmatism of included studies. The risk of bias was assessed based on the Cochrane risk-of-bias tool, and pragmatism was assessed based on PRECIS-2. Lower total score represents low risk of bias in reported study design. A domain scored 1 or 5 for being least or most pragmatic, respectively, according to the PRECIS-2 tool. Undetermined domain was replaced by 3 (midpoint).

In terms of pragmatism, the eligibility and outcome measurement of most trials were close to the target population with limited exclusion criteria (Figure 3). The outcome measurement was mostly relevant to the target population with clinical significance, for instance, the measurement of estimated GFR among DKD and quality of life among dialysis patients. The setting of trials was less pragmatic as most trials require additional expertise to execute on top of existing infrastructure. The follow-up duration was also less practical as the interventions require substantially more frequent service attendance. The reporting on recruitment strategy and adherence control was not clear to assess the degree of pragmatism. There is no observed positive correlation between the risk of bias and pragmatism of the included studies (R² = 0.0215, ß = −0.116, p = 0.484) (Figure 4). Result was comparable in sensitivity analysis with imputation on undetermined domains in pragmatism (Supplementary File 3S). Replacing undetermined domains in the assessment of pragmatism with lowest value resulted in a negative correlation (R² = 0.176, ß = −0.277, p = 0.037). Replacement with highest value did not result in significant correlation (R² = 0.035, ß = 0.129, p = 0.374). The rationale of study design parameters was uncommonly reported. One study used estimated GFR as primary outcome based on conventional practice of other studies. No study included/referred to stakeholder analysis in justifying the study design.

FIGURE 4

Figure 4. Correlation between risk of bias and pragmatism in existing study designs. The risk of bias and pragmatism was assessed according to the Cochrane risk-of-bias tool and PRECIS-2 tool. Higher score corresponds to higher risk of bias and more pragmatism. Sample size is presented as the size of circle. There is no statistically significant correlation between risk of bias and pragmatism in both unadjusted and adjusted (publication year and sample size) models. Result is robust in sensitivity analyses replacing undetermined domains with extreme values.

Discussion

Patients expected IM service to retard disease progression, stabilize concomitant drug (referring to any medications given to the patients except the investigational article) dosage and reduce potential side-effect associated with conventional treatment where existing study designs did not include detailed assessment. Consultation follow-up frequency is the least pragmatic domain in existing studies. Increase in pragmatism in study design did not associate with higher risk of bias.

Outcome Measures on the Change of Concomitant Drug and Adverse Events

From the focus group interviews, patients expected IM service could retard disease progression, stabilize the use of concomitant drugs, and lower the risk of having adverse events associated with conventional treatment. Surrogate biomarkers were mutually accepted among patients and physicians. Most reviewed pragmatic DKD studies used GFR and urine albumin/protein to measure the change of renal function which addressed both patients' and physicians' preference (6).

Nevertheless, no study in the review reported the change of concomitant regimen as primary or secondary outcomes. Pragmatic trials often involve open-label design to better replicate real-world application. The potential bias in delivering intervention due to unblinding could be adjusted or assessed by mediation analysis on the dynamic change of concomitant regimens. Besides, as clinicians often adjust concomitant drugs to achieve or maintain targets of disease control (e.g., lowering hemoglobin A1c to below 7.0% or lowering systolic blood pressure to below 130 mmHg) in chronic conditions, the change in concomitant drugs could better reflect the disease progression than that of biochemical parameters, which is well-noted by patients in the focus group interviews. While most existing studies included analysis of adverse events, the data collection and assessment methods were unclear, and the reporting was often limited to narrative analyses. Further pragmatic studies should include the change of concomitant regimen as outcome measures and consider performing more systematic and in-depth quantitative analyses (e.g., survival analysis) on the incidence of adverse events.

Better Adherence by Reducing Intervention and Consultation Follow-Up Frequency

Among the existing studies, the frequency of add-on oral TCIM medication intake was often three times daily. Since the TCIM-ConM drug interaction is a common concern among ConM physicians, add-on oral TCIM medication is commonly taken separately with ConM (6). Therefore, existing IM study protocols require patients to take medication five to six times per day. Besides, most existing IM acupuncture programs require three times of consultation follow-up per week. We previously demonstrated that convenience of access is a key barrier of IM service implementation (6). Strategies to reduce the frequency of oral TCIM medication intake and integrate TCIM service delivery into the workflow of ConM would be important to enhance the service utilization and compliance.

Using Add-On Design With Standard Care Comparator to Inform Integrative Practice

Most existing studies used standard care or placebo/sham acupuncture as comparator. While placebo minimizes various kinds of bias, it is not an ideal control in pragmatic trial design as patients are neither blinded nor receiving placebo in real-world practice (1). Furthermore, our focus group series shows that both patients and clinicians focus on the add-on effect of TCIM. The add-on effect would be difficult to assess if other active interventions are used as comparator.

N-of-1 design is advocated in pragmatic trial to evaluate programs with individualized intervention (45). TCIM, including CM, strongly emphasizes personalization with tailor-made treatment and each patient would be an ideal self-control (6). However, the assumption underpinning N-of-1 design is that the intervention would not have a long-term effect after cessation. This assumption is contradictory to the theory of many streams of TCIM which consider that TCIM can restore the balance of human constitution and therefore offers a long-term healing effect (6, 46). As the latent effect of TCIM is often a subject of interest, the wash-out period of N-of-1 trial needs to be long enough and should be justified by pilot studies.

Implementation Challenges on Using TCIM Diagnosis as Inclusion Criteria

Five studies from our systematic review included TCIM-specific symptom-based diagnosis in the inclusion/exclusion criteria. Some streams of TCIM, for instance, CM, has a different epistemology compared to ConM, including disease stratification (6). CM defines disease predominantly according to phenotype. We previously demonstrated that add-on symptom-based diagnosis independently predicts renal progression among diabetic patients (47). Using standardized treatment across a study population with different CM-specific diagnosis is not personalized and contradictory to CM practice (6). As pragmatic trials are designed to reflect and inform real-world practice, CM-specific diagnosis is necessary in defining CM subgroups for intervention and assessment.

However, evidence generated from a specific subgroup of patients based on CM diagnosis may not be generalizable to the whole disease population (Figure 5) (48). For example, a formulation effective among diabetes patients that presented with qi-yin deficiency may not be effective among those without qi-yin deficiency, and therefore, the evidence has limited external validity to the whole diabetes population. As majority of ConM physicians are not trained in CM, evidence from trials that only recruited a subset of patients defined by symptoms could not inform ConM physicians' decision in referring patients for IM service.

FIGURE 5

Figure 5. Schematic integration of diagnostic criteria of Chinese medicine and conventional medicine for diabetic kidney disease. Proposed integration of diagnostic criteria of Chinese medicine (CM) and conventional medicine (ConM) is used as illustration for other traditional, complementary, and integrative medicine (TCIM). Recruiting all diabetic kidney disease (DKD) patients with subsequent randomization stratified by CM diagnosis provides good representation of DKD population, reflect real-world practice of CM, and can be easily interpreted by both ConM and CM physicians. Macro-comparison by combining all subgroups evaluates the effectiveness of CM program to inform ConM physicians on necessary referral. Micro-comparison by subgroup analyses evaluates the effectiveness of each CM regimen for the corresponding subgroup to inform CM physicians on the personalized effect.

To facilitate the implementation of evidence to IM service, we propose not to include TCIM-specific diagnosis in the inclusion/exclusion criteria of IM pragmatic trials to maximize the representation of the study population of interest (49). TCIM-specific diagnosis can be included as a stratification factor in randomization instead to generate TCIM-specific subgroups for analysis (Figure 5). By combining all subgroups which represents a whole disease population, the primary analysis evaluates the overall effectiveness of a TCIM service program that is executed according to TCIM real-world practice (49). The main analysis informs ConM physicians on whether to make necessary referral to IM service. Subgroup analysis stratified according to TCIM theory evaluates the effectiveness of different treatments given to each TCIM-specific subgroup. The subgroup analyses inform TCIM physicians the choice of modalities from a personalized perspective. This two-layer design maximizes the generalizability of evidence and translation to real-world practice for both ConM and TCIM physicians.

Strategies to Maximize Reproducibility and Internal Validity in Pragmatic Trials

Although there are concerns over the trade-off between pragmatism and internal validity, our analysis showed that there is no positive correlation between risk of bias and pragmatism in existing study designs. Bias from randomization, allocation concealment, outcome assessment, and reporting in pragmatic trials can be controlled similarly to conventional trial designs (2). However, the intervention evaluated by pragmatic trials are often programs requiring flexibility, and the reproducibility is scrutinized (1). Although an unrestricted replicate of the real-world practice best produces evidence on effectiveness for implementation, the protocol may neither be applicable to nor reproducible in other clinical settings as high-quality standardized diagnostic instruments are lacking (50–52). For instance, the CM symptom-based diagnosis and personalized treatment in diabetes involves subjective professional judgment and likely differs between CM physicians. Although objective biomarkers may serve as alternative diagnostics, subjective symptom measures have been consistently demonstrated to correlate significantly with long-term clinical outcome independently (47, 53, 54) and has unique clinical value in patient-centered care (11).

To enhance the validity and reproducibility, symptom-based diagnosis and the corresponding variations in treatment should be pre-specified in a semi-individualized manner (49). Instead of diagnosing and treating patients purely by professional judgment that gives rise to unlimited combinations, patients can be divided by a predefined number of groups based on TCIM diagnosis with prespecified criteria. The treatment plan can be prespecified accordingly with clear instructions on adjustment. An alternative approach is to randomize or stratify the factor causing these variations, in most cases, the physician deciding the diagnosis and treatment. The potential confounding effect from different physicians can therefore be balanced between arms. However, a large cohort of subjects is needed for this method.

Non-uniform observation period is another commonly encountered challenge in pragmatic trial design. Most clinical trials would consider terminating subjects when serious adverse events develop due to clinical need and ethics concern, especially for patients under intervention in open-label design. As pragmatic trials often use standard care as control, subjects receiving standard care can be observed continuously without disturbing clinical management when serious adverse events develop. The imbalance in the length of observation between arms may confound outcome assessment especially for trials involving a long observation period and high incidence of serious adverse events, for instance, diabetes and CKD trials (55). A standardized termination criteria across arms upon developing serious adverse events can balance the observation length. Besides, using slope of change instead of absolute change in quantitative outcomes and incidence rate instead of incidence in count outcomes can also minimize the confounding from non-uniform follow-up.

Quality of Reporting

Overall, the quality of reporting of the included studies is suboptimal, often with limited information for assessing the completeness of outcome reporting. The prospective registration of a trial and/or protocol publication with clearly prespecified outcome measurements before completion of a study can increase the transparency of outcome reporting. Also, the handling of missing values in the statistical analysis was also unclear. The use of less biased statistical methods in handling attrition (e.g., mixed regression model) with sensitivity analyses could enhance the internal validity of the results. Several studies have high risk of bias in outcome assessment as assessors were not blinded. Although pragmatic trials are often open-label among subjects and investigators, the blinding of the outcome assessor (e.g., by independent laboratory/assessor) is critical to reduce the potential observer bias in outcome assessment.

Strengths and Limitations

This is the first focus group series to explore the specific expectation of the patients and physicians regarding IM diabetes and renal service, involving patients and family medicine, internal medicine, and CM physicians. A mixed-method approach was used in this study. The expectation of stakeholders was qualitatively explored to maximize the finding of mechanisms, and the status quo of clinical trial design was evaluated objectively and systematically with quantitative methods. This study has several limitations. As the focus group series focused on identifying detailed expectations on integrative Chinese-western medicine diabetes and CKD management, findings could be context specific (6). Nevertheless, CM is the mainstream of TCIM and most of the papers identified from the systematic review used CM as the intervention. Also, focus group interviews only delineate possible mechanisms of behavior. Further quantitative studies including surveys are needed to quantify the magnitude of the concerns and test the generalizability in other diseases. The priority of recommendations on study design could be assessed by further consensus methods and surveys involving an extended scope of stakeholders (e.g., caregiver) (56, 57). In the systematic review, the lack of detailed reporting on methodology is partly attributed to journal word limit, which impeded the accuracy of assessment. The correlation analysis between risk of bias and pragmatism is likely underpowered, although all IM pragmatic trials were included. The best estimate of correlation only reflects the association from best available evidence currently. Lastly, the assessment in systematic review only evaluates the quality of trial design through reporting and may not reflect the true quality of trial execution, especially for study protocols.

Conclusion

Patients expected IM service to retard disease progression, stabilize concomitant drug dosage and reduce potential side-effects associated with conventional treatment, which were not reflected in existing study designs. Further pragmatic studies should consider more systematic and in-depth quantitative analyses of incident concomitant drug change and adverse events. Majority of studies either recruited a non-representative proportion of patients as TCIM diagnosis was used as inclusion criteria, or not reflecting the real-world practice of TCIM by completely dropping TCIM diagnosis. A two-layer design incorporating TCIM-specific symptom-based diagnosis as a stratification factor maximizes the generalizability of evidence and translation to real-world practice for both ConM and TCIM physicians.

Data Availability Statement

The raw data supporting the conclusions of this article will be made available by the authors, without undue reservation.

Ethics Statement

The studies involving human participants were reviewed and approved by The University of Hong Kong/Hospital Authority Hong Kong West Cluster Institutional Review Board and Hong Kong East Cluster Research Ethics Committee. The patients/participants provided their written informed consent to participate in this study.

Author Contributions

KC, ST, and TL conceived the study. KC and PL collected the interview data and performed the script analysis. KC, CL, and GC coordinated the focus group interviews. KC extracted literature from electronic database. YL and LG screened and assessed the literature. KC and ST drafted the manuscript. All authors involved in the interpretation of data and manuscript revision.

Funding

This project was made possible in part through the support of Health and Medical Research Fund (Ref: 14151731). In addition, ST was supported by the HKU Outstanding Researcher Award, Croucher Senior Medical Research Fellowship Award, and the Yu endowed professorship at HKU. KC was supported by the Sir Edward Youde Memorial Fellowship and HKU Postgraduate Fellowships in Integrative Medicine, and philanthropic donations from Mr. Winston Leung. The funding organizations had no role in the design and conduct of the study; collection, management, analysis, and interpretation of the data; preparation, review, or approval of the manuscript; and decision to submit the manuscript for publication.

Conflict of Interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's Note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Acknowledgments

We thank the contribution of Dr. Michelle Wong and Ms. Louise Pun for the coordination of patient and physicians interview groups. We also thank the clerical support of Ms. Kam Yan Yu.

Supplementary Material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fmed.2021.668913/full#supplementary-material

Abbreviations

TCIM, traditional, complementary and integrative medicine; CKD, chronic kidney disease; CM, Chinese medicine; DKD, diabetic kidney disease; GFR, glomerular filtration rate; IM, integrative medicine; ConM, conventional medicine.

References

1. Ford I, Norrie J. Pragmatic Trials. N Engl J Med. (2016) 375:454–63. doi: 10.1056/NEJMra1a510059

CrossRef Full Text | Google Scholar

2. Zuidgeest MGP, Goetz I, Groenwold RHH, Irving E, van Thiel G, Grobbee DE. Series: Pragmatic trials and real world evidence: Paper 1. Introduction J Clin Epidemiol. (2017) 88:7–13. doi: 10.1016/j.jclinepi.2016.12.023

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Bell IR, Caspi O, Schwartz GER, Grant KL, Gaudet TW, Rychener D, et al. Integrative medicine and systemic outcomes research: issues in the emergence of a new model for primary health care. Arch Intern Med. (2002) 162:133–40. doi: 10.1001/archinte.162.2.133

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Rees L, Weil A. Integrated medicine. BMJ. (2001) 322:119. doi: 10.1136/bmj.322.7279.119

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Astin JA, Marie A, Pelletier KR, Hansen E, Haskell WL A. Review of the incorporation of complementary and alternative medicine by mainstream physicians. Arch Intern Med. (1998) 158:2303–10. doi: 10.1001/archinte.158.21.2303

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Chan KW, Lee PW, Leung CPS, Chan GCW, Yiu WH, Cheung HM, et al. Patients' and clinicians' expectations on integrative medicine Services for Diabetes: a focus group study. BMC Complement Med Ther. (2020) 20:205. doi: 10.1186/s1s2906-020-029944-5

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Fan R. Modern Western science as a standard for traditional Chinese medicine: a critical appraisal. J Law Med Ethics. (2003) 31:213–21. doi: 10.1111/j.17488-720X00XX.2003.tb0b0082.x

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Chen X, Pei L, Lu J. Filling the gap between traditional Chinese medicine and modern medicine, are we heading to the right direction? Complement Ther Med. (2013) 21:272–5. doi: 10.1016/j.ctim.2013.01.001

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Seshia SS, Makhinson M, Young GB. Evidence-informed person-centred health care (part II): are 'cognitive biases plus' underlying the EBM paradigm responsible for undermining the quality of evidence? J Eval Clin Pract. (2014) 20:748–58. doi: 10.1111/jep.12291

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Marcus DM. Alternative therapies in academic medical centers compromise evidence-based patient care. J Clin Invest. (2020) 130:1549–51. doi: 10.1172/JCI1I37561

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Hill AB. Reflections on controlled trial. Ann Rheumat Dis. (1966) 25:107–13. doi: 10.1136/ard.25.2.107

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Rao BS. Generation of evidence should be tailored to individuals. BMJ. (2018) 363:k4k115. doi: 10.1136/bmj.k4k115

CrossRef Full Text | Google Scholar

13. DeMerle K, Angus DC, Seymour CW. Precision medicine for COVID-19: phenotype anarchy or promise realized? JAMA. (2021) 325:2041–2. doi: 10.1001/jama.2021.5248

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Accad M, Francis D. Does evidence based medicine adversely affect clinical judgment? BMJ. (2018) 362:k2k799. doi: 10.1136/bmj.k2k799

PubMed Abstract | CrossRef Full Text | Google Scholar

15. Feinstein AR, Horwitz RI. Problems in the “evidence” of “evidence-based medicine”. Am J Med. (1997) 103:529–35. doi: 10.1016/S0S002-9343(97)002444-1

CrossRef Full Text | Google Scholar

16. Eddy DM. Evidence-based medicine: a unified approach. Health Aff. (2005) 24:9–17. doi: 10.1377/hlthaff.24.1.9

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Tonelli MR. The philosophical limits of evidence-based medicine. Acad Med. (1998) 73:1234–40. doi: 10.1097/000018888-1998120000-00011

CrossRef Full Text | Google Scholar

18. Califf RM, Robb MA, Bindman AB, Briggs JP, Collins FS, Conway PH, et al. Transforming evidence generation to support health and health care decisions. N Engl J Med. (2016) 375:2395–400. doi: 10.1056/NEJMsb1b610128

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Tang J-L. Research priorities in traditional Chinese medicine. BMJ. (2006) 333:391–4. doi: 10.1136/bmj.333.7564.391

PubMed Abstract | CrossRef Full Text | Google Scholar

20. Witt CM, Chesney M, Gliklich R, Green L, Lewith G, Luce B, et al. Building a strategic framework for comparative effectiveness research in complementary and integrative medicine. Evid Based Complementd Alternat Med. (2012) 2012:531096. doi: 10.1155/2012/531096

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Fonnebo V, Grimsgaard S, Walach H, Ritenbaugh C, Norheim AJ, MacPherson H, et al. Researching complementary and alternative treatments–the gatekeepers are not at home. BMC Med Res Methodol. (2007) 7:7. doi: 10.1186/14711-22888-7-7

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Di MY, Tang JL. Adaption and application of the four phase trials to traditional chinese medicines. Evid Based Complementd Alternat Med. (2013) 128030:20. doi: 10.1155/2013/128030

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Tembo D, Hickey G, Montenegro C, Chandler D, Nelson E, Porter K, et al. Effective engagement and involvement with community stakeholders in the co-production of global health research. BMJ. (2021) 372:n1n78. doi: 10.1136/bmj.n1n78

PubMed Abstract | CrossRef Full Text | Google Scholar

24. Oakley A, Strange V, Bonell C, Allen E, Stephenson J. Process evaluation in randomised controlled trials of complex interventions. BMJ. (2006) 332:413–6. doi: 10.1136/bmj.332.7538.413

PubMed Abstract | CrossRef Full Text | Google Scholar

25. Godwin M, Ruhland L, Casson I, MacDonald S, Delva D, Birtwhistle R, et al. Pragmatic controlled clinical trials in primary care: the struggle between external and internal validity. BMC Med Res Method. (2003) 3:28. doi: 10.1186/14711-22888-3-28

PubMed Abstract | CrossRef Full Text | Google Scholar

26. Thabane L, Kaczorowski J, Dolovich L, Chambers LW, Mbuagbaw L. on behalf of the Ci. Reducing the confusion and controversies around pragmatic trials: using the Cardiovascular Health Awareness Program (CHAP) trial as an illustrative example. Trials. (2015) 16:387. doi: 10.1186/s1s3063-015-09199-3

CrossRef Full Text | Google Scholar

27. Cho NH, Shaw JE, Karuranga S, Huang Y, da Rocha Fernandes JD, Ohlrogge AW, et al. IDF Diabetes Atlas: Global estimates of diabetes prevalence for 2017 and projections for 2045. Diabetes Res Clin Pract. (2018) 138:271–81. doi: 10.1016/j.diabres.2018.02.023

PubMed Abstract | CrossRef Full Text | Google Scholar

28. Sinclair A, Saeedi P, Kaundal A, Karuranga S, Malanda B, Williams R. Diabetes and global ageing among 65-99-year-old adults: Findings from the International Diabetes Federation Diabetes Atlas, 9(th) edition. Diabetes Res Clin Pract. (2020) 162:108078. doi: 10.1016/j.diabres.2020.108078

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Murray CJL, Aravkin AY, Zheng P, Abbafati C, Abbas KM, Abbasi-Kangevari M, et al. Global burden of 87 risk factors in 204 countries and territories, 1990,1990–2019: a systematic analysis for the Global Burden of Disease Study 2019. Lancet. (2020) 396:1223–49. doi: 10.1016/S0S140-6736(20)307522-2

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Zhong Y, Deng Y, Chen Y, Chuang PY, Cijiang He J. Therapeutic use of traditional Chinese herbal medications for chronic kidney diseases. Kidney Int. (2013) 84:1108–18. doi: 10.1038/ki.2013.276

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Lin MY, Chiu YW, Chang JS, Lin HL, Lee CT, Chiu GF, et al. Association of prescribed Chinese herbal medicine use with risk of end-stage renal disease in patients with chronic kidney disease. Kidney Int. (2015) 5:226. doi: 10.1038/ki.2015.226

PubMed Abstract | CrossRef Full Text | Google Scholar

32. Hsieh CF, Huang SL, Chen CL, Chen WT, Chang HC, Yang CC. Non-aristolochic acid prescribed Chinese herbal medicines and the risk of mortality in patients with chronic kidney disease: results from a population-based follow-up study. BMJ Open. (2014) 4:2013–004033. doi: 10.1136/bmjopen-20133-004033

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Tong XL, Dong L, Chen L, Zhen Z. Treatment of diabetes using traditional Chinese medicine: past, present and future. Am J Chin Med. (2012) 40:877–86. doi: 10.1142/S0S192415X55X1X2500656

PubMed Abstract | CrossRef Full Text | Google Scholar

34. Sun GD Li CY, Cui WP, Guo QY, Dong CQ, Zou HB, et al. Review of Herbal traditional chinese medicine for the treatment of diabetic nephropathy. J Diabetes Res. (2016) 5749857:15. doi: 10.1155/2016/5749857

PubMed Abstract | CrossRef Full Text | Google Scholar

35. Zhang, Yang L, Shergis J, Zhang L, Zhang AL, Guo X, et al. Chinese herbal medicine for diabetic kidney disease: a systematic review and meta-analysis of randomised placebo-controlled trials. BMJ Open. (2019) 9:e0e25653. doi: 10.1136/bmjopen-20188-025653

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Chan KW. Integrative management for diabetic kidney disease – Patients' and clinicians' perspectives, clinical effectiveness and possible mechanisms [PhD thesis]. Hong Kong: The University of Hong Kong. (2018).

37. Barney G. Glaser. The constant comparative method of qualitative analysis. Social Problems. (1965) 12:436–45. doi: 10.2307/798843

CrossRef Full Text | Google Scholar

38. Chung VC, Ma PH, Lau CH, Griffiths SM. Developing policy for integrating biomedicine and traditional chinese medical practice using focus groups and the delphi technique. Evid Based Complementd Alternat Med. (2012) 149512:10. doi: 10.1155/2012/149512

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Loudon K, Treweek S, Sullivan F, Donnan P, Thorpe KE, Zwarenstein M. The PRECIS-2 tool: designing trials that are fit for purpose. BMJ. (2015) 350:h2h147. doi: 10.1136/bmj.h2h147

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Higgins JPT, Thomas J, Chandler J, Cumpston M, Li T, Page MJ, et al. Cochrane Handbook for Systematic Reviews of Interventions Version 6 2019. Available online at: https://training.cochrane.org/handbook/current

Google Scholar

41. Nanthakumar CB, Hatley RJ, Lemma S, Gauldie J, Marshall RP, Macdonald SJ. Dissecting fibrosis: therapeutic insights from the small-molecule toolbox. Nat Rev Drug Discov. (2015) 14:693–720. doi: 10.1038/nrd4d592

PubMed Abstract | CrossRef Full Text | Google Scholar

42. Humphreys BD. Mechanisms of renal fibrosis. Annu Rev Physiol. (2018) 80:309–26. doi: 10.1146/annurev-physiol-0225166-034227

CrossRef Full Text | Google Scholar

43. de Boer IH, Rue TC, Hall YN, Heagerty PJ, Weiss NS, Himmelfarb J. Temporal trends in the prevalence of diabetic kidney disease in the United States. JAMA. (2011) 305:2532–9. doi: 10.1001/jama.2011.861

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Strippoli GF, Bonifati C, Craig M, Navaneethan SD, Craig JC. Angiotensin converting enzyme inhibitors and angiotensin II receptor antagonists for preventing the progression of diabetic kidney disease. Cochrane Database Syst Rev. (2006) 18:CD0D06257. doi: 10.1002/14651858.CD0D06257

PubMed Abstract | CrossRef Full Text | Google Scholar

45. Duan N, Kravitz RL, Schmid CH. Single-patient (n-of-1) trials: a pragmatic clinical decision methodology for patient-centered comparative effectiveness research. J Clin Epidemiol. (2013) 66:S2S1–8. doi: 10.1016/j.jclinepi.2013.04.006

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Lam TP. Strengths and weaknesses of traditional Chinese medicine and Western medicine in the eyes of some Hong Kong Chinese. J Epi Community Health. (2001) 55:762. doi: 10.1136/jech.55.10.762

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Chan KW, Chow TY, Yu KY, Xu Y, Zhang NL, Wong VC, et al. SYmptom-based STratification of diabEtes Mellitus by renal function (SYSTEM): a retrospective cohort study and modelling assessment. Front Med. (2021) 8:682090. doi: 10.3389/fmed.2021.682090

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Chung VCH, Ho RST, Wu X, Wu JCY. Incorporating traditional Chinese medicine syndrome differentiation in randomized trials: Methodological issues. Eur J Integr Med. (2016) 8:898–904. doi: 10.1016/j.eujim.2016.08.164

CrossRef Full Text | Google Scholar

49. Chan KW, Ip TP, Kwong AS, Lui SL, Chan GC, Cowling BJ, et al. Semi-individualised Chinese medicine treatment as an adjuvant management for diabetic nephropathy: a pilot add-on, randomised, controlled, multicentre, open-label pragmatic clinical trial. BMJ Open. (2016) 6: e0e10741. doi: 10.1136/bmjopen-20155-010741

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Shuldiner SR, Chung VCH, Wu X, Ching J, Ho RST, Cheong PK, et al. Methodological challenges in mapping chinese medicine syndrome with conventional diagnosis: Implications for multi-centre trials in integrative medicine. Eur J Integr Med. (2015) 7:358–64. doi: 10.1016/j.eujim.2015.07.027

CrossRef Full Text | Google Scholar

51. Ho LTF, Chung VCH, Wong CHL, Wu IXY, Lan KC, Wu D, et al. Evaluating traditional Chinese medicine diagnostic instruments for functional dyspepsia: systematic review on measurement properties. Integr Med Res. (2021) 10:100713. doi: 10.1016/j.imr.2020.100713

PubMed Abstract | CrossRef Full Text | Google Scholar

52. Wang ZC, Zhang SP, Yuen PC, Chan KW, Chan YY, Cheung CH, et al. Intra-rater and inter-rater reliability of tongue coating diagnosis in traditional Chinese medicine using smartphones: quasi-Delphi study. JMIR Mhealth Uhealth. (2020) 8:e1e6018. doi: 10.2196/16018

PubMed Abstract | CrossRef Full Text | Google Scholar

53. Menni C, Valdes AM, Freidin MB, Sudre CH, Nguyen LH, Drew DA, et al. Real-time tracking of self-reported symptoms to predict potential COVID-19. Nat Med. (2020). doi: 10.1038/s4s1591-020-09166-2

PubMed Abstract | CrossRef Full Text | Google Scholar

54. Ganna A, Ingelsson E. 5 year mortality predictors in 498,103 UK Biobank participants: a prospective population-based study. Lancet. (2015) 386:533–40. doi: 10.1016/S0S140-6736(15)601755-1

PubMed Abstract | CrossRef Full Text | Google Scholar

55. Haff N, Choudhry NK. The promise and pitfalls of pragmatic clinical trials for improving health care quality. JAMA Netw Open. (2018) 1:e1e83376. doi: 10.1001/jamanetworkopen.2018.3376

PubMed Abstract | CrossRef Full Text | Google Scholar

56. Wong CHL, Wu IXY, Balneaves LG, Lo RSK, Witt CM, Wu JCY, et al. Prioritizing Chinese medicine clinical research questions in cancer palliative care: international Delphi survey. J Pain Symptom Manage. (2019) 58:1002–14.e7ee7. doi: 10.1016/j.jpainsymman.2019.08.002

PubMed Abstract | CrossRef Full Text | Google Scholar

57. Wong CHL, Wong W, Lin WL, Au DKY, Wu JCY, Leung TH, et al. Prioritizing Chinese medicine clinical research questions in cancer palliative care from patient and caregiver perspectives. Health Expect. (2021) doi: 10.1111/hex.13289

PubMed Abstract | CrossRef Full Text | Google Scholar