Development of self-administered questionnaire on barriers, prescription practices, and guideline adherence of osteoporosis management among tertiary care clinicians: content validity and reliability analysis

Objective: This study outlined the development of the barriers, prescribing practices, and guideline adherence for osteoporosis management according to the Clinicians’ Osteoporosis Questionnaire (COQ) followed by an assessment of the content validity index and reliability test.

Methods: The development of the COQ was performed in two stages. Stage I involved the development of the COQ, and stage II involved judgmental evidence and quantification of the questionnaire. Five panel experts related to the study area and five clinicians participated in the validity of the COQ assessment. Fifty clinicians took part in the reliability test evaluation by filling out the questionnaire twice at 2-week intervals. The content validity index (CVI) and content validity ratio (CVR) were analyzed using Microsoft Excel, while Cohen’s kappa statistic was used to determine the test–retest reliability using SPSS version 29.

Results: Forty items and three domains, namely, barriers, prescribing practices, and guideline adherence for osteoporosis management, were identified in the COQ (version 4.0). The scale-level CVI (S-CVI/Ave) for every domain was above 0.9, which is considered acceptable. The CVRs for all the items were above 0.7, except for two items in the barrier domain and two items in the guideline adherence domain. Two items were revised to improve the clarity of the item, and other items were retained based on consensus among the expert panel. Between the test and retest, the reliability of individual items ranged from moderate to almost perfect for the barrier domain (k = 0.42–0.86), prescribing practice domain (k = 0.79–0.87), and guideline adherence domain (k = 0.46–1). None of the items had “fair” or “poor” agreement. Thus, the 40-item COQ (version 4.0) was finalized following the content and face validity analysis.

Conclusions: Through an iterative process, the development and assessment of the COQ showed a high degree of content validity and reliability in measuring the barriers, prescribing practices, and guideline adherence among clinicians managing osteoporosis. Future studies should aim to further validate this instrument across different populations and settings, as well as explore methods to enhance its reliability and validity.

Introduction

Osteoporosis is a chronic degenerative bone disease characterized by reduced bone density that increases susceptibility to fragility fractures, leading to substantial morbidity, mortality, and socioeconomic burden (1, 37, 38). The Asian Federation of Osteoporosis Societies has projected that there will be a 3.5-fold increase in osteoporotic hip fractures by 2050, from 6,000 to almost 21,000 fractures each year, costing over USD125 million (MYR540 million) annually in healthcare expenditure (2). Malaysia will have the highest expected rate of increase among countries in Southeast Asia due to the rapidly increasing aging population (2, 39). Thus, future fracture prevention strategies, which include optimizing osteoporosis management in clinical settings, are key to reducing this burden.

Osteoporosis is preventable and treatable, yet it remains underdiagnosed, undertreated, and undermanaged globally (3). The silent nature of this disease poses a significant challenge for clinicians to detect it early (4). Consequently, fragility fractures following mild falls or impacts are typically the first presenting symptom among patients with osteoporosis in tertiary care settings (4). Despite this, there is a paucity of studies focusing on the management of osteoporosis by tertiary clinicians. Most of the literature primarily focuses on primary care physicians’ role in managing osteoporosis (5–8). However, the unique challenges and opportunities in tertiary care settings warrant dedicated research attention, as tertiary care providers often serve as frontline clinicians for patients with fragility fractures. Moreover, compared to primary care settings, tertiary clinicians are well positioned to diagnose the underlying cause of fractures, as tertiary centers are usually well equipped with the necessary facilities, including the assessment of bone mineral density (BMD) using dual-energy X-ray absorptiometry (DXA), which is the gold standard for diagnosing osteoporosis (7). Following diagnosis, these patients also had greater opportunity for earlier initiation of anti-osteoporosis medication and provided interventions for fall prevention. The study by Boff et al. (9) highlighted that the most concerning gap in current practice among clinicians managing osteoporosis is that they tend to treat the fracture solely without further diagnosing the underlying cause of the fracture. As a result, they miss the opportunity to initiate antiresorptive drugs as early as possible. These drugs have been proven to be effective in increasing bone density and preventing further fractures (10). This gap in treatment is globally acknowledged as a crisis in osteoporosis patient care, which underscores the seriousness of this condition (9, 11, 12). While the occurrence of a previous fragility fracture almost doubles a patient’s future fracture risk if anti-osteoporosis medications are not started promptly, this condition will lead to a vicious cycle of recurrent fractures, often resulting in disability and premature death (13).

The effective management of osteoporosis among clinicians can be hindered by several factors. For example, barriers to osteoporosis care can significantly influence the efficacy of management strategies. These barriers may encompass a lack of awareness of current guidelines, variations in available pharmacotherapy, poor communication within the care team, and patient-related factors (3). Prescribing practices also play a pivotal role. The timely initiation of anti-osteoporotic medications, along with related vitamins and supplements, and the assessment of medication adherence could greatly enhance patient outcomes (10). In terms of guideline adherence, it is important to highlight measures for preventing osteoporosis and secondary fractures, appropriate diagnosis, and treatment follow-up to bridge the gap in osteoporosis management (8). However, it is noteworthy that previous studies have not documented in detail the validation and reliability of their questionnaires. Therefore, this study aimed to develop a valid and reliable self-administered questionnaire that can effectively capture the barriers, prescription practices, and adherence to guidelines among clinicians and will serve as a valuable tool for understanding and improving osteoporosis management in tertiary care settings, ultimately reducing the burden of this treatable condition. This study outlines the process involved in the development of the Clinicians’ Osteoporosis Questionnaire (COQ), as well as the methods employed to ascertain its content validity and test–retest reliability.

Methods

Study design

This cross-sectional study was conducted at the outpatient orthopedic department of tertiary healthcare institutions in Malaysia. The development, validation, and reliability assessment of the COQ were performed in a two-stage process, as outlined in Figure 1. Stage I involved instrument design and development, and Stage II involved obtaining judgmental evidence and quantifying the COQ, which included an analysis of content validity, face validity, and reliability. The panels of experts were invited to assess content validity through a judgmental sampling method based on their expertise, specialty, and credibility. For face validity, respondents were chosen using purposive sampling among orthopedic clinicians managing osteoporosis.

Figure 1

Figure 1. Summary of the development, validity, and reliability of the COQ. COQ, Clinicians’ Osteoporosis Questionnaire.

Stage I: development of the COQ

Item generation

The items for the COQ were generated through a thorough literature review that evaluated barriers, prescription practices, and guideline adherence to osteoporosis; the latest version of Malaysia’s Clinical Practice Guideline (CPG) of Osteoporosis 2022; and a focus group discussion conducted among osteoporosis subject matter experts. Given the limited research involving tertiary care clinicians, the items were derived from studies conducted on primary care physicians, supplemented with insights relevant to tertiary care settings, which include the following:

i. Literature review:

● Malaysian Primary Care Study (Tay et al., 2022): Identified knowledge, attitudes, practices, and barriers in osteoporosis management among primary care doctors.

● Singapore Primary Care Study (Derek et al., 2023): Evaluated osteoporosis guideline utilization and barriers to care among primary care physicians.

● Osteoporosis Medication Study (Yi et al., 2020): Assessed factors contributing to the initiation of osteoporosis medications post-hip fracture, as well as compliance and persistence with these medications.

ii. Clinical Practice Guidelines:

● The latest Malaysia CPG for Osteoporosis 2022 provided a framework for identifying essential practices and barriers relevant to osteoporosis management.

iii. Focus group discussion:

● Conducted with osteoporosis expert panel to identify additional barriers and practices specific to tertiary care settings.

Based on the results of the extensive literature review, the Osteoporosis CPG, and the discussion with experts, a conceptual framework was constructed, and a preliminary version of the COQ consisting of three domains (barriers, prescribing practices, and osteoporosis guideline adherence) was created.

Stage II: judgmental evidence and quantification of the COQ

This stage included three evaluations. The first was an assessment by expert panels, which focused on the content validity of each item in the COQ, specifically on relevance, clarity, and essentiality. Second, the face validity was evaluated via interviews and focus groups among orthopedic clinicians managing osteoporosis. Finally, the reliability of the questionnaire was evaluated among 50 orthopedic clinicians in a test–retest study.

Content validity

Expert panel

The number of experts recommended for content validation purposes can range from 2 to 20 individuals (14). To ensure that the consensus among experts represents their professional knowledge and comprehension of the subject matter rather than mere coincidence, it is recommended that a minimum of five individuals review the instruments. In this study, the content validation was determined by the following five members of the expert panels:

● one medical lecturer in bone and osteoporosis research,

● one medical lecturer in pharmacoepidemiology, and

● three orthopedic clinicians specializing in osteoporosis and fracture liaison services.

Experts were invited to participate in the content validity survey, which was sent via email. The email included a cover letter, the COQ, and an evaluation sheet asking for experts’ input on several aspects of each question in the tool, including 1) the importance of each question (relevance), 2) the clarity of each question (in terms of wording), 3) the necessity of each question (essentiality), and 4) recommendations for improvement of each question. A 4-point Likert scale was used for relevance, while a 3-point Likert scale was used for clarity and essentiality. The relevance scale includes 1 = not relevant, 2 = somewhat relevant, 3 = quite relevant, and 4 = very relevant, where the question is considered valid if the ratings are 2 and 3, while a rating of 1 is considered invalid. The clarity scale was 1= not clear, 2 = item needs some revision, and 3 = very clear. For the essentiality, it was 1 = not essential, 2 = useful but not essential, and 3 = essential.

Content validity evaluation

The COQ underwent thorough testing to confirm its validity. Validity refers to the degree to which an instrument accurately measures what it is supposed to measure (14). Consequently, several iterations were performed on the COQ to ensure that the survey was unambiguous and well defined and covered topics of importance to clinicians managing osteoporosis. In this study, content validity was quantitatively measured using the content validity index (CVI) for relevance and the content validity ratio (CVR) for the essentiality of each question. The calculation is summarized in Figure 2.

Figure 2

Figure 2. Summary of the quantitative analysis of content validity.

Content validity index

The CVI is a commonly used method for assessing content validity in newly developed instruments (15). It can be measured at both the item level (I-CVI) and the scale level (S-CVI). The 4-point Likert scale was used to rate each item (1 = not relevant, 2 = needs some revision, 3 = needs minor revision, and 4 = very relevant). The I-CVI is determined by dividing the number of experts who rated an item as 3 or 4 by the total number of experts (16). The I-CVI can range from 0 to 1. If the I-CVI is greater than 0.79, the item is considered relevant. If it falls between 0.70 and 0.79, the item requires revisions. An item with an I-CVI value below 0.70 was eliminated. When there are more than five experts, an I-CVI value of 0.78 is considered acceptable (16, 17).

The Scale-Content Validity Index (S-CVI) indicates the average score of the overall content validity scale. An S-CVI value of 0.8 or higher is generally considered acceptable (17). There are two methods for calculating the S-CVI:

i. Universal Agreement (UA) among experts (S-CVI/UA) was calculated as the number of items on a scale that all experts rated as 3 or 4 (representing universal agreement) divided by the total number of items. The number of experts involved greatly affected the S-CVI/UA, in which the likelihood of obtaining a lower S-CVI is greater when there are more experts. This is because a UA score of 1 is assigned only when there is complete consensus among all experts on an item. If there was any disagreement, the UA score was set to 0.

ii. The average CVI (S-CVI/Ave) is the total I-CVI score of all the items divided by the total number of items (the average I-CVI score for all the items of all the experts).

Content validity ratio

The CVR is a measure of the essentiality of each item. The experts rated each item on a 3-point scale (1 = not essential, 2 = useful but not essential, and 3 = essential). Ratings of 1 and 2 signify “not essential” content, while ratings of 3 signify “essential” content (18). The CVR is calculated by subtracting half the total number of experts from the number of experts who rated an item as “essential” and then dividing this result by half the total number of experts (CVR = (ne(N/2)) (19, 20). The CVR ranges from −1 to 1, with a higher score indicating a higher level of agreement among panel members. A higher score signifies greater consensus among the expert panels regarding the necessity of the items. The minimum acceptable CVR for the six panels of experts involved was 0.99 (27). However, a CVR value of at least 0.78 is generally acceptable; if <0.78, the individual item should be revised or removed (28).

Face validity evaluation

Face validity can be explored both qualitatively (via a cognitive interview or a focus group discussion between researchers and a few respondents from the target population) and quantitatively (via feedback), which is measured on a Likert-scale survey. A cognitive interview is a method that helps individuals comprehend and respond to survey questions (21). It aims to gain insights into the thought process behind their responses and identifies potential issues. The question-and-answer model, which includes understanding, memory retrieval, decision-making, and response selection, predicts how individuals determine the level of detail needed for their answers. However, focus groups, which are informal discussions on a specific topic, were used in this study to gather participants’ understanding, opinions, and perspectives of the items in the questionnaire (22). In this study, face validity was qualitatively measured using a pilot group of five orthopedic medical officers who filled out the questionnaire and were encouraged to give comments, either written or oral, regarding the questionnaire. All the necessary information was noted.

Reliability test

The reliability of the questionnaire was evaluated in a test–retest study involving a sample of 50 orthopedic clinicians from tertiary care institutions in Malaysia. Participants were asked to complete the questionnaire on two separate occasions at 2-week intervals. The responses from the two administrations were compared, and Cohen’s kappa statistic was used to determine the test–retest reliability of each questionnaire item.

Cohen’s kappa statistic is a measure of the level of agreement between two raters who each classify items into mutually exclusive categories (23, 40). the study was extensively utilized in reliability analysis: It accounts for the possibility of agreement occurring by chance. There are two types of kappa—unweighted and weighted:

i. Unweighted kappa: This type treats all disagreements equally. For example, if we have a scale from 1 to 5, a disagreement between 2 and 1 is treated the same as a disagreement between 5 and 1.

ii. Weighted kappa: This version of kappa is used for measuring agreement on ordered variables, where certain disagreements (e.g., lowest versus highest) can be weighted as more or less important (23). For example, on a scale from 1 to 5, a disagreement between 2 and 1 could be considered less serious than a disagreement between 5 and 1. In both cases, a higher kappa value indicates a higher level of agreement.

The strength of agreement was interpreted in line with the reference values provided by Landis and Koch (23). According to the scale, a Cohen’s k coefficient of less than 0.20 indicates poor agreement, 0.21–0.40 indicates fair agreement, 0.41–0.60 indicates moderate agreement, 0.61–0.80 indicates substantial agreement, and 0.81–1.00 represents almost perfect agreement. In this study, the unweighted k statistic was used to evaluate nominal variables, whereas the quadratic weighted k statistic was used to evaluate ordinal variables.

Statistical analysis

For the content validity calculation, Microsoft Excel was used for data entry and tabulation of the CVI and CVR. The Cohen’s kappa test for the reliability of the questionnaire was conducted using IBM SPSS Statistics for Windows (version 29.0).

Results

Stage I: COQ revisions and development

From the COQ conceptual framework, three domains were identified: barriers, prescribing practices, and guideline adherence to osteoporosis management. The COQ underwent four rounds of revisions from the expert panels. Table 1 summarizes the major amendments.

Table 1

Table 1. Summary of major amendments of the COQ revision.

COQ revision results

In the first round, the expert panels evaluated the items for relevance and clarity. The second domain was initially regarded as “practice” but was changed to “prescribing practice” following discussions. The response format for prescribing practice was also changed from a 3-point scale to a 4-point Likert scale. The research team found that the items in the third domain (guideline adherence) were more suited to assessing knowledge rather than adherence to guidelines. As a result, all items in the third domain were eliminated, leaving out 26 of the initial 60 items (COQ version 1.0).

During the second round, 12 new items were developed from the latest version of Malaysia’s Osteoporosis CPG 2022. A dichotomous answer format (yes or no) was selected for the guideline adherence domain. Consequently, the total number of items increased to 38, up from the initial 26 in the second version (COQ version 2.0).

The third revision round took place after a pilot test of the 38-item questionnaire was conducted among 20 orthopedic medical officers involved in osteoporosis management at Hospital Canselor Tuanku Muhriz (HCTM), Cheras, Malaysia. The pilot test analysis led to the removal of one item from the barrier domain, two items from the prescribing practice domain, and the addition of five items to the guideline adherence domain, resulting in a total of 40 items in the questionnaire (COQ version 3.0).

For the fourth revision round, a repilot test was conducted with the 40-item questionnaire among 50 orthopedic clinicians managing osteoporosis. The analysis following the repilot test was satisfactory. A minor amendment was made where items were rearranged according to the subdomain in the guideline adherence domain. The final version of the questionnaire (COQ version 4.0), consisting of 40 items, was used in this study’s analysis. The summary of each domain, with a total of 40 items, is presented in Table 2.

Table 2

Table 2. The domain, subdomain, and number of items of the COQ (version 4.0).

COQ item development results

The domain concerning perceived barriers, which includes 15 items, was further categorized into three subdomains: barriers to knowledge of disease, facility-related barriers, and patient-related barriers. Out of 15 items, only six items were adopted from the study by Li et al. (7) and Choong et al. (8) (lack of doctor–patient time, inadequate knowledge, inaccessibility of pharmacotherapy at your practice, worry about side effects of the anti-osteoporotic medication, patients’ lower socioeconomic status, and patients’ refusal of screening). The remaining nine items were added to this study following the update from the latest version of the Osteoporosis CPG and the results from the expert recommendations of the subjects. This included the “silent” nature of the disease, which made it difficult to detect early, difficulty interpreting DXA scan results, inadequate knowledge of the different variations of calcium supplements and vitamin D, lack of choices of anti-osteoporotic medication at your hospital, lack of variation of calcium supplements and vitamin D at your hospital, difficulty integrating with another department (e.g., gynecology, geriatric, endocrinology, and primary care), inadequate staff to provide a better postfracture care program such as the Fracture Liaison Service (FLS), patients’ inadequate knowledge of osteoporosis, patient refusal to start anti-osteoporotic medication, and patient non-adherence to anti-osteoporotic medication. A 3-point Likert scale was used to measure the barriers to osteoporosis management in the barrier domain.

The prescribing practice comprised eight items with two subdomains, namely, practice on prescribing anti-osteoporosis medication (bone mineral consideration, history of fracture consideration, calcium, and vitamin D prescription) and practice on follow-up of the prescribed medication (compliance assessment, consideration to switching to other anti-osteoporosis medication if no improvement, scheduling DXA scans to monitor osteoporosis treatment, and recommendation of drug therapy when necessary). The study by Li et al. (7) served as a reference, but no items were directly adopted due to different contexts, as their study focused on primary care physicians. Consequently, the eight items in this domain were newly developed based on the most recent osteoporosis CPG, a literature review, and expert consensus. The response options employ an ordinal scale following the approach of Li et al. (7), with a 4-point Likert scale [never (0%), occasional (1%–50%), often (51%–99%), and always (100%)] to indicate the prescribing practice in osteoporosis management.

For the guideline adherence domain, 17 items were developed with three subdomains, including preventive measures for osteoporosis (eight items), diagnosis (four items), and treatment (seven items). The treatment subdomain differed from the prescribing practice domain, as the latter primarily emphasizes existing prescribing practices, which may or may not be included in the current guidelines. Conversely, the treatment subdomain focused mainly on treatment recommendations, as outlined in the latest Malaysia’s CPG, Osteoporosis, 2022 (24). This implies that while the prescribing practice domain reflects the prescribing practices that are currently being implemented (which could vary from the guidelines), the treatment subdomain strictly adheres to and focuses on the recommendations provided in the most recent guidelines. According to the study by Choong et al. (8), the research team found that the items regarding guideline utilization were too general, as there were only three questions: whether the participants read the latest guidelines, self-reported good knowledge, and self-reported good guideline utilization. Therefore, the research team decided to extract crucial practices from the latest osteoporosis CPG and finalized 17 items with a dichotomous answer format (yes or no) to measure the agreement with the statement.

Stage II: diagnostic evidence and quantification of the COQ

Content validity analysis

Based on the expert panel’s judgment, all content validity (CVI and CVR) was calculated from the fourth version (three domains, 40 items) of the COQ and is shown in Table 3 (barrier domain), Table 4 (prescribing practice domain), and Table 5 (guideline adherence domain).

Table 3

Table 3. Summary items (I-CVI, CVR, and clarity) that were retained and revised for the barrier domain of the COQ (version 4.0).

Table 4

Table 4. The summary items (I-CVI, CVR, and clarity) that were retained and revised for the prescribing practice domain of the COQ (version 4.0).

Table 5

Table 5. The summary items (I-CVI, CVR, and clarity) that were retained and revised for the guideline adherence domain of the COQ (version 4.0).

The results showed that the I-CVI values, which indicate the relevance of individual items of all 40 items in the COQ (version 4.0), were beyond the acceptability cut-off point of 0.78. All items demonstrated an I-CVI value of 1, except for two items, one from the barrier domain (B13) and another from the guideline adherence domain (G6), both of which had an I-CVI of 0.8. In terms of the overall relevance of the questionnaire, represented by the S-CVI values, the results also exceeded the cut-off point of 0.8, with the S-CVI/UA (overall) being 0.95 and the S-CVI/Ave (overall) being 0.99. The S-CVI/UA and S-CVI/Ave per domain were also calculated and showed acceptable values (>0.8).

The CVR calculations indicated that four items (B3, B8, B13, and G6) out of the 40 items in the COQ did not meet the acceptable value of 0.99. This value was set based on the evaluations of a panel consisting of five experts as suggested by Lawshe et al. Each of these items had a CVR value of 0.6. While non-essential items can potentially be removed, in this instance, they were retained based on the recommendations and consensus of the expert panels. A detailed summary of the items across all domains is presented in Table 6.

Table 6

Table 6. The summary of the items for all domains in COQ (4.0) following the content validity analysis.

The clarity of the fourth version of the COQ was evaluated by five raters using a 3-point Likert scale, where 1 signifies “not clear”, 2 represents “somewhat clear”, and 3 denotes “very clear”. The mean clarity scores for individual items were calculated, and the results varied between 2.60 and 3.00. A total of 34 items, accounting for 85% of the items, were classified as “very clear”, with an average score of 3.00. Five items received an average clarity score of 2.80, while one scored 2.60. The overall clarity mean score for the fourth version of the COQ was determined to be 2.97.

Face validity analysis

The five-member pilot group qualitatively confirmed the face validity of the COQ. All participants in the group unanimously agreed on the relevance, essentiality, and clarity of the items included in the questionnaire, which they found to be straightforward and easy to comprehend.

Reliability test analysis

Test–retest reliability

Of the 60 initial respondents, 83% (50) of the clinicians successfully completed the questionnaire on two separate occasions at 2-week intervals. Most of the respondents were male doctors (70%, 35), and the mean age of the respondents was 33.76 (4.35) years. Most clinicians who participated in the survey were of Malay ethnicity, represented by 80% (40). The detailed characteristics of the clinicians who participated in the survey are presented in Table 7.

Table 7

Table 7. Characteristics of the study participants.

The reliability of the three domains of the questionnaire was assessed, and the results are presented in Tables 8–10. The overall kappa agreement ranged from 0.42 to 1, with no items having kappa values less than 0.40, indicating no “fair” or “poor” agreement in the barrier domain; the kappa agreement values ranged from 0.42 to 0.86. Seven items demonstrated substantial agreement, five items showed moderate agreement, and three items exhibited almost perfect agreement. The item pertaining to the limited choices of anti-osteoporotic medication at the hospital had the highest agreement (k = 0.86). In contrast, the item regarding patients’ inadequate knowledge of osteoporosis had the lowest agreement (k = 0.42). The percentage of agreement for this domain ranges from 70.0% to 94.0%.

Table 8

Table 8. Cohen’s kappa (k) coefficient for the barrier domain.

Table 9

Table 9. Cohen’s kappa (k) coefficient for the prescribing practice domain.

Table 10

Table 10. Cohen’s kappa (k) coefficient for the guideline adherence domain.

In the prescribing practice domain, the kappa agreement values ranged from 0.79 to 0.87. Most items (5) displayed almost perfect agreement, and three items showed substantial agreement. The item related to assessing patient compliance with osteoporosis medication during follow-up had the highest agreement (k = 0.86), while the item about prescribing calcium and vitamin D to all patients with fragility fractures had the lowest agreement (k = 0.79). The percentage of agreement for this domain varied from 82.0% to 90.0%.

In the guideline adherence domain, the kappa agreement values ranged from 0.46 to 1. Twelve items demonstrated substantial agreement, three items showed almost perfect agreement, and one item exhibited moderate agreement. The item about advising patients on optimizing calcium and vitamin D intake had the highest agreement (k = 0.86), while the item concerning identifying risk factors for osteoporosis in advanced-aged patients (>60 years old) had the lowest agreement (k = 0.79). The percentage of agreement for this domain ranged from 82.0% to 100.0%.

Discussions

This study aimed to assess the content validation and reliability of a newly developed questionnaire on barriers, prescribing practices, and guideline adherence among clinicians managing osteoporosis (COQ). An approach suggested by Khosla and Shane (11) that has been widely used mainly in medical and nursing research was employed in this study. Content validity is an important quality indicator of an instrument’s validity. The researchers developed the COQ containing a total of 60 items in four domains at the first stage and ultimately retained 40 items for postface validity and pilot test analysis in the COQ version 4.0.

In accordance with the guidelines proposed by Khosla and Shane (11), which had been extensively utilized in this study area, a panel of five experts was selectively assembled to oversee the questionnaire development process and make necessary adjustments to the instrument. The panel comprised experts with a robust background in bone research, pharmacoepidemiology, and osteoporosis management. A medical lecturer with extensive experience in bone research was included, providing valuable insights into the development and refinement of the questionnaire. Additionally, three orthopedic surgeons were recruited, two of whom specialized in postosteoporotic fracture management and Fracture Liaison Service implementation. Their expertise allowed for the identification of gaps in current practice, providing a unique perspective that would otherwise be unattainable. In addition, an expert in clinical pharmacoepidemiology participated, providing guidance on question selection and formulation, ensuring item relevance and accuracy, and assisting in choosing appropriate answer options. This strategic recruitment not only strengthens our research methodology but also enriches the validity and reliability of our findings while shedding light on areas for improvement in current practice. While the panel’s expertise significantly contributes to the development and refinement of the instrument, it is crucial to note the potential bias due to the subjective nature of the panel expert’s feedback (15). This subjectivity could introduce a degree of bias into the process, which may influence the interpretation and revision of items in the instrument. To mitigate this risk, achieving consensus among panel experts is of paramount importance because we can harness their collective wisdom while minimizing the impact of individual biases, thereby ensuring the overall validity and reliability of the instrument (25).

Discussion on CVI

In this study, the CVI was utilized as a key tool for refining the COQ (version 4.0). This approach is consistent with methodologies used in other studies. The CVI, supplemented by feedback and comments from the expert panel, served as a critical judgment tool to enhance the content validity indices. Both the item-level content validity index (I-CVI) and the scale-level content validity index (S-CVI) were considered in this process. The S-CVI provides an average score of the overall content validity scale, offering a comprehensive view of the instrument’s validity (14, 18). An I-CVI of 0.78 or higher is considered excellent. The I-CVIs of all the items in the COQ ranged from 0.80 to 1.00, with only two items having an I-CVI of 0.8, indicating high content validity. There were differing opinions among the panel regarding two items (both with I-CVI: 0.8): “Patient’s refusal of screening” in the barrier domain and “Do you provide interventions for fall prevention?” from the guideline adherence domain. These differences could be attributed to the diverse backgrounds and experiences of the experts (26). For instance, an expert with a clinical background may view these items as crucial elements in osteoporosis management, as they witnessed the situations in their clinical practice, while a non-clinical expert may not have had the same experiences, hence considering the items irrelevant. The minimum acceptable S-CVI ranges from 0.80 to 0.90 (17). In this study, two values were calculated: the Universal Agreement (S-CVI/UA = 0.95) and the average (S-CVI/Ave = 0.99). Both scores indicated excellent overall content validity of the COQ. This suggests that the individual items developed were important and relevant for measuring barriers, prescribing practices, and guideline adherence among clinicians managing osteoporosis.

The CVR was used to assess the quality of the items in the COQ. A total of 36 items achieved an excellent CVR value of 1, indicating that most of the raters considered these items to be essential (14). This resulted in an overall content validity score of CVR = 0.96. However, four items received a CVR of 0.6, suggesting that they were not considered essential by some of the panels. These items included three from the barrier domain [“Difficulty in interpreting DXA scan results”, “Difficulty integrating with another department (e.g., gynecology, geriatric, endocrinology, and primary care)”, and “Patients’ refusal of screening”] and one from the guideline adherence domain (“Do you provide interventions for fall prevention?”).

The differing opinions among the panel regarding these four items could be attributed to the interpretation of available evidence, as some experts may interpret the data to suggest that these items are significant barriers or crucial for guideline adherence in osteoporosis management, while others may not see a strong enough link. These differing perspectives underscore the complexity of developing a questionnaire and highlight the importance of achieving consensus among panel experts (27, 28).

The clarity of the COQ appears to be straightforward and comprehensible, as evidenced by 85% of the items being rated as “very clear”. This clarity is crucial because it allows respondents to answer the questionnaire with ease, which in turn ensures the reliability and validity of their responses (29). The qualitative validation of the face validity of the COQ in the pilot group further supported this conclusion. The unanimous agreement among the participants on the relevance, essentiality, and clarity of the items in the questionnaire indicates that the instrument is well designed. However, it is important to note that while these initial results are promising, they are not definitive. As a useful initial step, face validity is a subjective measure and does not guarantee overall validity because it is based on apparent relevance, not on a rigorous statistical examination of the measurement properties of the scale (30). Therefore, further validation studies using more robust statistical methods are necessary to confirm these initial findings and to assess other forms of validity (e.g., construct validity and criterion validity) and reliability (31, 32). The COQ is expected to offer valuable insights and reliable sources for clinicians managing osteoporosis in tertiary care settings that could lead to more effective management strategies and prevention of fractures, which in turn improve patient outcomes.

Reliability testing

In this study, the reliability of the questionnaire was assessed using test–retest reliability analysis. For the Likert scale items in the barriers and prescribing practice domains, a quadratic weighted kappa was employed. The quadratic weighted kappa is an extension of Cohen’s kappa that considers the severity of disagreements, making it particularly useful for ordinal scales where disagreements between categories that are farther apart are more serious than disagreements between adjacent categories (33). For the guideline adherence domain, Cohen’s kappa coefficient was used instead of the intraclass correlation coefficient (ICC) due to the dichotomous nature of the questionnaire responses (32). Cohen’s kappa statistic is a type of reliability coefficient that measures the degree of agreement between two different evaluations in questionnaires with dichotomous variables (34). The results showed that the kappa values were almost perfect for the prescribing practice domain (0.82), while the barriers and guideline adherence domains showed substantial agreement (0.66 and 0.74, respectively). The overall average kappa coefficient of the questionnaire was 0.73, indicating substantial agreement.

The substantial agreement in the barrier domain suggests that clinicians are facing similar challenges in their practice. The lack of choices of anti-osteoporotic medication at hospitals, for instance, could be a systemic issue that needs to be addressed at a policy level. This shared recognition of barriers could serve as a call for healthcare administrators and policymakers to improve the availability of medication options in tertiary care settings (8). In the prescribing practice domain, most items displayed almost perfect agreement and a high level of consistency among clinicians in their views on prescribing practices. This could be interpreted as a collective agreement on best practices for prescribing medication for osteoporosis, reflecting a standardized approach to treatment. This uniformity in prescribing practices is beneficial because it ensures that patients receive consistent care regardless of their healthcare provider (9). The substantial agreement in the guideline adherence domain reflects a common consensus among clinicians about the importance of adhering to guidelines in osteoporosis management. The high level of agreement in advising patients on optimizing calcium and vitamin D intake highlights the universal acknowledgment of this aspect in managing osteoporosis (35). This shared understanding underscores the importance of guideline adherence in ensuring effective and standardized care for osteoporosis patients. The absence of any “fair” or “poor” agreement across all domains further highlights the reliability of the questionnaire, with most items demonstrating substantial to almost perfect agreement. However, it is worth noting that even though the kappa values were generally high, there were variations within each domain. This suggests that while the questionnaire is generally reliable, there may be specific items that could benefit from further refinement to improve their clarity and ensure that they are accurately capturing the constructs they are intended to measure. In addition, direct comparison of these results with those of prior studies was difficult due to differences in the context of the questions and type of response alternatives used in the questionnaire. Moreover, the repeatability of most questionnaires used in previous studies related to clinicians managing osteoporosis has not been well statistically documented.

Application of COQ in clinical settings

The COQ involves identifying barriers, prescribing practices, and guideline adherence among clinicians managing osteoporosis. By systematically assessing these domains, applying the COQ in clinical settings can reveal critical gaps and challenges faced by healthcare providers, enabling the development of targeted interventions. For instance, identifying barriers such as inadequate knowledge of osteoporosis guidelines and medications, difficulty in interpreting DXA scan results, and patient-related factors like non-adherence to medication can help design educational programs tailored to clinicians’ needs. Similarly, understanding prescribing practices and adherence to guidelines can highlight areas where clinicians deviate from recommended protocols, facilitating the identification of gaps in practice and addressing areas needing improvement.

The legitimacy of the COQ as an effective tool in osteoporosis management is underscored by our recent study comparing barriers and guideline adherence between standard tertiary care and FLS-accredited hospitals (36). The COQ successfully highlighted important gaps and significant differences in guideline adherence between these settings. FLS settings were more likely to initiate timely treatment, monitor patients effectively using BMD assessments, and consider anabolic agents for high-risk patients. These insights demonstrate that the COQ is effective not only in identifying critical areas needing improvement but also in driving the development of interventions that enhance osteoporosis management practices and promote evidence-based policies.

Moreover, the COQ serves as a self-audit tool for clinicians. By simply reading and answering the straightforward and informative questions in the COQ, clinicians can self-assess how well they are practicing and adhering to osteoporosis guidelines. This immediate feedback can help clinicians identify areas for improvement in their clinical practice, making the COQ a practical and beneficial tool for continuous self-improvement and adherence to best practices.

As the COQ targets clinicians in tertiary centers, who are at the frontline of managing fragility fracture patients, it specifically addresses a prioritized group. This approach may improve discharge care plans for osteoporotic fracture patients as tertiary clinicians become more aware of and adhere to guidelines for secondary fracture prevention. These comprehensive care plans can then be continued in primary care settings, enhancing continuity of care. The integration with primary care also helps reduce the burden caused by the lack of time between doctors and patients in tertiary hospitals.

Targeted interventions and policy changes

Based on COQ findings, educational programs can be designed to address knowledge gaps among clinicians. For example, workshops or online courses can be developed to improve understanding of current osteoporosis guidelines and effective use of DXA scans.

If the COQ identifies systemic issues such as limited medication options or insufficient time for patient interactions, healthcare institutions can strategically revise policies to elevate bone health as a national priority, addressing gaps not currently listed in the National Health Agenda 2016–2025 (36). Such policy revisions could lead to allocating additional resources, ensuring a broader availability of osteoporosis medications, and granting clinicians more time for comprehensive patient consultations. Additionally, this approach could promote the wider adoption of FLS in hospitals, further enhancing the standard of care for osteoporosis management.

Recognizing barriers to patient adherence can lead to developing comprehensive adherence programs, including patient education, regular follow-ups, and the application of reminder systems to enhance medication compliance.

Impact on patient outcomes, clinician education, and healthcare resource allocation

Implementing the COQ in clinical settings has the potential to improve patient outcomes significantly. By ensuring that clinicians are well-informed about the latest guidelines and have access to necessary resources, patients are more likely to be diagnosed and receive timely and appropriate treatment and fall prevention measures, reducing the risk of fractures and other complications associated with osteoporosis. Clinician education is another critical area where the COQ can substantially impact. Continuous professional development based on COQ findings ensures clinicians stay updated with the latest advancements and best practices in osteoporosis management, ultimately leading to higher quality of care. Furthermore, the COQ serves as critical evidence-based support for healthcare resource allocation. By identifying resource-lacking areas, the COQ enables healthcare institutions to allocate funds more efficiently, ensuring that critical needs are met and resources are used effectively to improve overall patient care.

Integration with clinical workflows and electronic health records

Integrating the COQ with clinical workflows and electronic health record (EHR) systems makes it easier to use in real-world settings. Electronic integration allows for smooth data collection, analysis, and reporting, helping clinicians incorporate COQ assessments into their routine practice.

For example, the COQ can be added to EHRs for regular assessments, with automated prompts and reminders to ensure consistent use. The following outlines how the COQ can be effectively embedded into EHR systems: in the barrier domain, questions such as “Inadequate knowledge of current osteoporosis guidelines and medications” can be embedded into EHRs. When a clinician fills out a patient’s EHR, they could be prompted to answer questions about their knowledge and comfort level with current guidelines. If a clinician indicates a lack of knowledge, the system can automatically provide links to educational resources or suggest participation in upcoming workshops. In the prescribing practices domain, questions such as “I assess osteoporosis medication compliance among patients during follow-up” can be embedded into EHRs to prompt clinicians during patient visits. This can ensure that medication adherence is regularly checked and recorded in the patient’s health records. The EHR can also generate alerts if a patient is due for a follow-up DXA scan or if there are inconsistencies in medication adherence, allowing the clinician to address these issues promptly. In the guideline adherence domain, questions like “Do you measure BMD in clinically diagnosed osteoporosis?” can be integrated into EHRs to ensure clinicians adhere to guidelines. The EHR can include checklists that clinicians must complete during patient consultations, ensuring all necessary diagnostic and treatment steps are followed according to the latest guidelines. This can standardize care and ensure no critical steps are overlooked.

Additionally, EHR integration can enable real-time data analysis, providing immediate feedback to clinicians and administrators​. This can help identify trends and patterns, allowing for quick adjustments to management strategies and policies to improve patient outcomes​.

In summary, the COQ provides a strong framework for improving osteoporosis management in clinical settings. Its use can lead to targeted interventions, better clinician education, efficient healthcare resource allocation, and easy integration with clinical workflows and EHR systems, ultimately improving patient care and outcomes. Additionally, as a self-audit tool, it helps clinicians continually evaluate and improve their adherence to osteoporosis management guidelines. By targeting clinicians in tertiary centers, the COQ enhances secondary fragility fracture prevention in primary care settings and reduces time constraints in tertiary hospitals.

Strengths and limitations

To the best of our knowledge, this is the first study on the development of a questionnaire that is specifically designed for clinicians managing osteoporosis in a tertiary care setting, as well as an in-depth discussion of the content validity and reliability of the questionnaire. However, this study has several limitations. While this study focused on content validity and test–retest reliability, other forms of validity (such as construct or criterion validity) were not assessed. These additional measures could provide a more comprehensive understanding of the questionnaire’s validity. The feedback from the pilot group, although valuable, represents a small sample size. To ensure broader applicability of the questionnaire, it would be beneficial to expand the evaluation to include a larger and more diverse group of participants. This would provide additional insights and help ensure that the questionnaire is generalizable across different contexts. Future research should aim to address these limitations by assessing other forms of validity, using larger and more diverse samples, and refining the questionnaire design. Additionally, factors such as potential biases in responses, cultural and geographical variations in osteoporosis management practices, and external validation of the questionnaire could also be considered to further enhance the robustness and applicability of the newly developed questionnaire, thereby contributing to better management of osteoporosis in various clinical settings.

Conclusion

This study provides valuable insights into the content validity and reliability of the COQ and demonstrates its effectiveness in assessing barriers, prescribing practices, and guideline adherence among clinicians in tertiary care settings. Future research should aim to further validate this tool across diverse populations and settings and explore ways to enhance its reliability and validity. The COQ is a promising tool for practical application at various levels, potentially enhancing osteoporotic patient care and outcomes.

Data availability statement

The original contributions presented in the study are included in the article/supplementary material. Further inquiries can be directed to the corresponding author.

Ethics statement

The studies involving humans were approved by Research Ethics Committee, Universit Kebangsaan Malaysia (FF-2022-355); National Medical Research Register, Ministry of Health Malaysia (RSCH ID-22-03398-043). The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.

Author contributions

NKM: Formal analysis, Investigation, Validation, Writing – original draft, Writing – review & editing. IM: Conceptualization, Methodology, Project administration, Resources, Supervision, Writing – review & editing. SM: Conceptualization, Methodology, Resources, Supervision, Validation, Writing – review & editing. JL: Conceptualization, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing. NK: Conceptualization, Methodology, Project administration, Resources, Supervision, Validation, Writing – review & editing. NM: Conceptualization, Funding acquisition, Investigation, Methodology, Project administration, Resources, Supervision, Validation, Visualization, Writing – review & editing.

Funding

The author(s) declare financial support was received for the research, authorship, and/or publication of this article. This study was financially supported by a Fundamental Grant from the Faculty of Medicine, Universiti Kebangsaan Malaysia (FF-2022-355). We would like to express our gratitude to Universiti Kebangsaan Malaysia for the financial support in this research.

Acknowledgments

Our heartfelt thanks also go to the expert panel involved in this study for their invaluable participation, contribution, and commitment. We would particularly like to express our sincere appreciation to Dr. Adam Bujang for his expertise and guidance, which were instrumental in the success of this work. Additionally, the authors utilized ChatGPT-4 (OpenAI) for language editing and improving the clarity of the manuscript. The AI tool was used solely for linguistic purposes and did not contribute to the conceptual or analytical aspects of the study. Finally, we extend our appreciation to the Director General of Health Malaysia for granting us permission to publish this manuscript.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Abbreviations

COQ, Clinicians’ Osteoporosis Questionnaire; CVI, content validity index; CVR, content validity ratio; I-CVI, item-level content validity index; S-CVI, scale-level content validity index; UA, Universal Agreement; BMD, bone mineral density; DXA, dual-energy X-ray absorptiometry; CPG, Clinical Practice Guideline.

References

1. Thambiah SC, Yeap SS. Osteoporosis in South-East Asian countries. Clin Biochem Rev. (2020) 41:29–40. doi: 10.33176/AACB-19-00034

PubMed Abstract | Crossref Full Text | Google Scholar

2. Cheung CL, Ang SB, Chadha M, Chow ES, Chung YS, Hew FL, et al. An updated hip fracture projection in Asia: The Asian Federation of Osteoporosis Societies study. Osteoporos. Sarcopenia. (2018) 4:16–21. doi: 10.1016/j.afos.2018.03.003

PubMed Abstract | Crossref Full Text | Google Scholar

3. Bennett MJ, Center JR, Perry L. Exploring barriers and opportunities to improve osteoporosis care across the acute-to-primary care interface: a qualitative study. Osteoporos. Int. (2023) 34:1249–62. doi: 10.1007/s00198-023-06748-0

PubMed Abstract | Crossref Full Text | Google Scholar

4. Fuggle NR, Curtis B, Clynes M, Zhang J, Ward K, Javaid MK, et al. The treatment gap: The missed opportunities for osteoporosis therapy. Bone. (2021) 144:115833. doi: 10.1016/j.bone.2020.115833

PubMed Abstract | Crossref Full Text | Google Scholar

5. Fogelman Y, Goldshtein I, Segal E, Ish-Shalom S. Managing osteoporosis: A survey of knowledge, attitudes and practices among primary care physicians in Israel. PLoS One. (2016) 11:e0160661. doi: 10.1371/journal.pone.0160661

PubMed Abstract | Crossref Full Text | Google Scholar

6. Naik-Panvelkar P, Norman S, Elgebaly Z, Elliott J, Pollack A, Thistlethwaite J, et al. Osteoporosis management in Australian general practice: An analysis of current osteoporosis treatment patterns and gaps in practice. BMC Fam. Pract. (2020) 21:1–13. doi: 10.1186/s12875-020-01103-2

PubMed Abstract | Crossref Full Text | Google Scholar

7. Tay CL, Ng WL, Beh HC, Lim EC, Hussin N. Screening and management of osteoporosis : a survey of knowledge, attitude and practice among primary care physicians in Malaysia. Arch Osteoporos. (2022) 17:72. doi: 10.1007/s11657-022-01111-y

PubMed Abstract | Crossref Full Text | Google Scholar

8. Choong DS, Tan NC, Koh YLE, Leong CK, Sankari U, Koh KH. Osteoporosis management by primary care physicians in Singapore: a survey on osteoporosis guidelines utilisation and barriers to care. Arch Osteoporos. (2023) 18:1–14. doi: 10.1007/s11657-023-01283-1

Crossref Full Text | Google Scholar

9. LeBoff MS, Greenspan SL, Insogna KL, Lewiecki EM, Saag KG, Singer AJ, et al. The clinician’s guide to prevention and treatment of osteoporosis. Osteoporos Int. (2022) 33(10):2049–102. doi: 10.1007/s00198-021-05900-y

PubMed Abstract | Crossref Full Text | Google Scholar

10. Ayub N, Faraj M, Ghatan S, Reijers JAA, Napoli N, Oei L. The treatment gap in osteoporosis. J Clin Med. (2021) 10:1–13. doi: 10.3390/jcm10133002

Crossref Full Text | Google Scholar

11. Khosla S, Shane E. A crisis in the treatment of osteoporosis. J Bone Miner. Res. (2016) 31:1485–7. doi: 10.1002/jbmr.2888

PubMed Abstract | Crossref Full Text | Google Scholar

12. Roux C, Briot K. Osteoporosis in 2017: Addressing the crisis in the treatment of osteoporosis. Nat Rev Rheumatol. (2018) 14:67–8. doi: 10.1038/nrrheum.2017.218

PubMed Abstract | Crossref Full Text | Google Scholar

13. Mitchell RM, Jewell P, Javaid MK, McKean D, Ostlere SJ. Reporting of vertebral fragility fractures: can radiologists help reduce the number of hip fractures? Arch Osteoporos. (2017) 12:71. doi: 10.1007/s11657-017-0363-y

PubMed Abstract | Crossref Full Text | Google Scholar

14. Yusoff MSB. ABC of content validation and content validity index calculation. Educ Med J. (2019) 11:49–54. doi: 10.21315/eimj2019.11.2.6

Crossref Full Text | Google Scholar

15. Zamanzadeh V, Ghahramanian A, Rassouli M, Abbaszadeh A, Alavi-Majd H, Nikanfar A-R. Design and Implementation Content Validity Study: Development of an instrument for measuring Patient-Centered Communication. J Caring Sci. (2015) 4:165–78. doi: 10.15171/jcs.2015.017

PubMed Abstract | Crossref Full Text | Google Scholar

16. Lynn MR. Determination and quantification of content validity. Nurs Res. (1986) 35:382–6. http://ijoh.tums.ac.ir/index.php/ijoh/article/view/26.

PubMed Abstract | Google Scholar

17. Polit DF, Beck CT. The content validity index: Are you sure you know what’s being reported? Critique and recommendations. Res Innursing&Health. (2006) 29:488–95. doi: 10.1002/nur.20147

Crossref Full Text | Google Scholar

18. Rodrigues IB, Adachi JD, Beattie KA, MacDermid JC. Development and validation of a new tool to measure the facilitators, barriers and preferences to exercise in people with osteoporosis. BMC Musculoskelet Disord. (2017) 18:1–9. doi: 10.1186/s12891-017-1914-5

PubMed Abstract | Crossref Full Text | Google Scholar

19. Lawshe CH. A quantitative approach to content validity”.Personnel Psychology. Pers. Psychol. (1975) 28:563–75. doi: 10.1111/j.1744-6570.1975.tb01393.x

Crossref Full Text | Google Scholar

20. Dalawi I, Isa MR, Chen XW, Azhar ZI, Aimran N. Development of the Malay Language of understanding, attitude, practice and health literacy questionnaire on COVID-19 (MUAPHQ C-19): content validity & face validity analysis. BMC Public Health. (2023) 23:1–13. doi: 10.1186/s12889-023-16044-5

PubMed Abstract | Crossref Full Text | Google Scholar

21. Collins D. An overview of cognitive methods Pretesting survey instruments. Quality. (2009) 12:229–38. doi: 10.1023/a:1023254226592

Crossref Full Text | Google Scholar

22. Wilkinson S. Focus group methodology: A review. Int J Soc Res Methodol. (1998) 1:181–203. doi: 10.1080/13645579.1998.10846874

Crossref Full Text | Google Scholar

23. Landis JR, Koch GG. Landis amd Koch1977_agreement of categorical data. Biometrics. (1977) 33:159–74. doi: 10.2307/2529310

PubMed Abstract | Crossref Full Text | Google Scholar

24. Ministry of Health Malaysia. CPG: Management of Osteoporosis Clinical Practice Guidelines 2022 3rd Edition (2022). Available online at: https://www.moh.gov.my/index.php/pages/view/3962?mid=1570 (accessed on August 30, 2022).

Google Scholar

25. Niederberger M, Köberich S. Coming to consensus: The Delphi technique. Eur J Cardiovasc Nurs. (2021) 20:692–5. doi: 10.1093/eurjcn/zvab059

PubMed Abstract | Crossref Full Text | Google Scholar

26. Åkesson KE, McGuigan FEA, Akesson KE, McGuigan FEA. Closing the osteoporosis care gap. Curr Osteoporos. Rep. (2021) 19:58–65. doi: 10.1007/s11914-020-00644-w

PubMed Abstract | Crossref Full Text | Google Scholar

27. Ricci L, Lanfranchi JB, Lemetayer F, Rotonda C, Guillemin F, Coste J, et al. Qualitative methods used to generate questionnaire items: A systematic review. Qual. Health Res. (2019) 29(1):149–56. doi: 10.1177/1049732318783186

PubMed Abstract | Crossref Full Text | Google Scholar

28. Solans-Domènech M, MV Pons J, Adam P, Grau J, Aymerich M. Development and validation of a questionnaire to measure research impact. Res Eval. (2019) 28:253–62. doi: 10.1093/reseval/rvz007

Crossref Full Text | Google Scholar

29. Gehlbach H, Brinkworth ME. Measure twice, cut down error: A process for enhancing the validity of survey scales. Rev Gen Psychol. (2011) 15:380–7. doi: 10.1037/a0025704

Crossref Full Text | Google Scholar

30. Moores KL, Jones GL, Radley SC. Development of an instrument to measure face validity, feasibility and utility of patient questionnaire use during health care: The QQ-10. Int J Qual. Heal Care. (2012) 24:517–24. doi: 10.1093/intqhc/mzs051

Crossref Full Text | Google Scholar

31. Tsang S, Royse CF, Terkawi AS. Guidelines for developing, translating, and validating a questionnaire in perioperative and pain medicine. Saudi J Anaesth. (2017) 11:S80–9. doi: 10.4103/sja.SJA_203_17

PubMed Abstract | Crossref Full Text | Google Scholar

32. Sim J, Wright CC. The Kappa Statistic in Reliability Studies : Use, Interpretation, and. Phys Ther. (2005) 85:257–68. http://www.ncbi.nlm.nih.gov/pubmed/15733050.

PubMed Abstract | Google Scholar

33. Kvålseth TO. An alternative interpretation of the linearly weighted kappa coefficients for ordinal data. Psychometrika. (2018) 83:618–27. doi: 10.1007/s11336-018-9621-1

Crossref Full Text | Google Scholar

34. Azraii AB, Ramli AS, Ismail Z, Abdul-Razak S, Badlishah-Sham SF, Mohd-Kasim NA, et al. Validity and reliability of an adapted questionnaire measuring knowledge, awareness and practice regarding familial hypercholesterolaemia among primary care physicians in Malaysia. BMC Cardiovasc Disord. (2021) 21:1–17. doi: 10.1186/s12872-020-01845-y

PubMed Abstract | Crossref Full Text | Google Scholar

35. Kilim HP, Rosen H. Optimizing calcium and vitamin D intake through diet and supplements. Cleve. Clin J Med. (2018) 85:543–50. doi: 10.3949/ccjm.85a.17106

PubMed Abstract | Crossref Full Text | Google Scholar

36. Muhamad Jamil NK, Naina Mohamed I, Mokhtar SA, Leong JF, Kamudin NAF, Muhammad N. Barriers to osteoporosis management and adherence to Clinical Practice Guideline: a comparative study between tertiary East Coast hospitals and a Fracture Liaison Services (FLS)-accredited hospital in Malaysia. Arch Osteoporos. (2024) 19:1–10. doi: 10.1007/s11657-024-01407-1

Crossref Full Text | Google Scholar

37. Abd Kahar NS, Kuan CS, Singh DKA, Mokhtar SA. Risk factors associated with fragility fracture among older adults with fragility fracture: a systematic review. Malays J Med Health Sci. (2022) 18:318–26. doi: 10.47836/mjmhs18.s15.44

Crossref Full Text | Google Scholar

38. Choo YW, Mohd Tahir NA, Mohamed Said MS. Budget impact of increasing uptake of denosumab for the treatment of postmenopausal osteoporosis in Malaysia. Arch Osteoporos. (2023) 18:145. doi: 10.1007/s11657-023-01358-z

PubMed Abstract | Crossref Full Text | Google Scholar

39. Ibrahim N, Sharkawi Ahmad M, Zulfarina MS, Zaris SNASM, Nor Azlin ZA, Naina Mohamed I, et al. Physical function assessment of older adults with lower body fractures at 3 months post-discharge from hospital. Ther Clin Risk Manag. (2019) 15:201–10. doi: 10.2147/TCRM.S189748

PubMed Abstract | Crossref Full Text | Google Scholar

40. Zulfarina MS, Sharif R, Sharkawi AM, Tg Abu Bakar Sidik TMI, Mokhtar SA, Shuid AN, et al. Reliability of self-administered questionnaire on dietary supplement consumption in Malaysian adolescents. Nutrients. (2020) 12(9):2853. doi: 10.3390/nu12092853

PubMed Abstract | Crossref Full Text | Google Scholar