Assessing the quality of electronic medical records in academic hospitals: A multi-center study in Iran

Objective: The present study aimed to assess the quality of electronic medical records (EMR) retrieved from hospital information systems (HIS) of three educational hospitals in Mashhad, Iran.

Methods: In this multi-center, cross-sectional study, inpatient electronic records collected from three academic hospitals were categorized into five data groups, namely demographics (D); care handler (CH), indicating the doers of the medical actions; diagnosis and treatment (DT); administrative and financial (AF); and laboratory and Para clinic (LP). Next, we asked 25 physicians from the three academic hospitals to determine data elements of medical research and education value (called research and educational data) in every group. Flowingly, the quality of the five data groups (completeness * accuracy) was reported for entire sampled data and those specified as research and educational data, based on the exact concordance between electronic medical records and corresponding paper records. HISRA, standing for HIS recording ability, was also assessed compared to data elements of standard paper forms.

Results: For entire data, HISRA was 58.5%. In all hospitals, the highest data quality (more than 90%) belongs to D and AF data groups, and the lowest quality goes to CH and DT groups (less than 50%, and 60%, respectively). For research and educational data, HISRA was 47%, and the quality of D and AF data groups were the highest (nearly 100%), while CH and DT stood around 50% and 60% in order. The quality of the LP data group was almost 85% in all hospitals but hospital C (well over 30%). Total data quality for the hospitals was almost less than 70%.

Conclusions: The low quality of electronic medical records was mostly a result of incompleteness, while the accuracy was relatively good. Results showed that the HIS application development mainly focused on administrative and financial aspects rather than academic and clinical goals.

Introduction

A hospital information system (HIS) is an integrated, computer-assisted system that commonly reflects all hospital operations dimensions. It is used to store and retrieve information related to clinical, administrative, financial, and legal tasks to meet the needs of all authorized system users in the hospital (1). HIS is a potentially rich source usually used for research and educational purposes besides the quality of care improvement and managerial activities. A HIS with high-quality data can improve the quality of health care, monitor patient condition, and therapeutic response, reduce the frequency of errors, avoid adverse drug interactions, and reduce hospital costs (2). In recent years, there has been a growing interest in researching electronic medical records (EMRs) collected in HIS (3). Medical studies, epidemiological investigations, and analysis of disease progression are conducted based on EMRs collected from HIS. Furthermore, patients’ EMRs can be used to train medical students and increase their clinical knowledge (4).

The validity of medical research and epidemiological studies done using EMRs strongly relies on the quality of recorded data (5). If the quality of data and reports recorded in HIS are not sufficiently high, the investment return is not guaranteed, and the health care system will not achieve the predefined objectives. Besides, high-quality EMRs can be considered a major educational tool in health care education (6) as it offers acknowledged benefits to medical students in academic environments (7). Therefore, the quality of patient EMRs is considered highly important in any hospital and should be frequently assessed to determine defective processes (8).

So far, studies have been done on HIS in Iran as a developing country, mainly focused on user satisfaction, financial issues (hospital costs, for example), and managerial and administrative functions (9). To our knowledge, a few studies have been performed to assess the HIS data quality by this study method.

Therefore, this study assessed the quality of electronic medical records especially those that possess educational and research value in terms of completeness, accuracy, and quality (completeness * accuracy), in academic hospitals in Mashhad, Iran. We also present the hospital information system recording ability (HISRA; the potential of the system to record the five necessary data groups) of the HIS.

Materials and methods

In this study, we assessed completeness and accuracy as the measures of data quality. We used “Data Source Agreement” as the strategy of assessment and paper medical records were regarded as the assessment “standard” to which the EMRs were compared (3).

Study location and sampling

We selected three academic hospitals, a 920-bed general hospital (A), a 144-bed pediatric hospital (B), and a 60-bed ophthalmic hospital (C) in Mashhad, which is Iran's second-largest city with about 3 million inhabitants. In these three hospitals, data were mainly recorded in the form of hard copies and some electronic recordings were performed in parallel.

All hospitals use the same version of HIS designed in compliance with the international standard for the exchange of medical information (HL7). The HIS software was launched in 2001 in all hospitals belonging to Mashhad University of Medical Sciences. It is connected to the three ancillary subsystems (laboratory, pharmacy, and radiology) and to the databases including SNOMED, ICD.9.cm, ICD.10, and California tariff that feed patients' EMR. Furthermore, the system records some clinical documentation including medication administration, physician orders, and clinical and surgical procedures.

The study was conducted within 90 days, from September 2019 through November 2019. Over this period, 150, 100, and 50 cases were selected in the three hospitals, A, B, and C, respectively. We used stratified random sampling as the number of selected cases in each hospital was proportionate to that hospital's daily inpatient discharges. We included only inpatient discharges for them both paper and electronic records were available.

We carried out a retrospective comparison between a sample of electronic records and hard copies to examine the quality of electronic medical records.

Categorization of data elements and evaluation of HISRA

To collect data, we considered a checklist of entire data elements of a standard paper record. In the checklist, the data elements were categorized into five groups, namely demographics (D), care handler (CH); diagnosis and treatment (DT); administrative and financial (AF); and laboratory and Para clinic (LP).

HISRA, standing for Hospital information software recording ability, indicates the potential of the hospital information software to capture all data values from the paper records. For each data group, it was calculated as the ratio of the number of available data fields that can be filled in both paper charts and the HIS software, to the total number of data elements, within the paper charts.

Specification of research and educational data elements

To specify data elements that serve medical research and educational objectives in particular, the checklist was sent to 25 physicians from the studied hospitals, who were all faculty members of University of Medical Sciences, Mashhad, Iran; they were asked to determine whether a given data element is “Necessary “or “Not necessary” concerning research and educational importance. Then, the content validity ratio (CVR) was calculated for each data element according to the collected answers. For a group size of 25 (the number of physicians included in this experiment), the critical number and critical CVR were calculated as 18 and 0.44, respectively. The data elements with CVR ≥ 0.44 were selected as elements with research and educational importance and called “research and educational data” (10).

Data collection, completeness, and accuracy

For every sampled discharge, a checklist was completed by comparing the electronic and paper medical records. In the checklist, the presence or absence of each entry in the paper and the electronic records was determined to evaluate data completeness. In addition, the exact concordance between the two values recorded for a specific element in electronic and paper medical records was considered to measure data accuracy.

Completeness

“Completeness” was defined as the existence of an entry for a given element in the electronic medical record that is present in the paper record, regardless of its accuracy. Completeness was calculated as the ratio of the number of data entries found in patients' electronic records to the total number of data entries that existed in patients' paper records.

Accuracy

“Accuracy” was defined as a strict concordance between electronic data value and the corresponding value in the paper record. Accuracy was calculated as the ratio of the number of accurate data values to the total number of data values (11, 12).

Data analysis

Finally, for entire data together with the research and educational data, the quality of each data group was calculated by multiplying accuracy by completeness (11, 12). The hospitals were statistically compared using P-value (<0.05), derived from the Pearson Chi-square test, to examine whether hospitals' difference considering data completeness and accuracy is statistically significant or the difference is by chance.

Results

Data elements with research and educational importance

Based on their educational and research value, data elements were specified as “essential” and “not essential” by 25 physician selected from the three hospitals; then CVR was calculated for each data element. Table 1 shows the data elements that meet the criteria of CVR ≥ 0.44, and were selected as elements with research and educational importance. In this table, data elements of the same data group and the same CVR are shown together in a row.

TABLE 1

Table 1. Content validity ratio (CVR) for research and educational data elements.

HISRA of the HIS software

Concerning entire data elements, HISRA was 58.5%, while it was 47% for research and educational data elements (Tables 2, 3). Among entire data elements, HISRA of CH, and DT groups (75% and 43%, respectively) were lower than those of D, LP and AF groups (100%, 100%, and 91%, respectively).

TABLE 2

Table 2. HISRA of the HIS concerning entire data elements.

TABLE 3

Table 3. HISRA of the HIS concerning research and educational data elements.

Similarly, for research and educational data elements, HISRA of CH, and DT groups (75% and 39%, in order) were far lower than those of D, LP and AF groups (100%).

Data quality in hospitals A, B and C

In total, 300 discharged cases were assessed from September 2020 to November 2020. Table 4 shows the completeness, accuracy, and quality of five data groups in the three hospitals for entire data elements. Similarly, Table 5 shows them for research and educational data.

TABLE 4

Table 4. Completeness, accuracy and quality of data groups concerning entire data elements.

TABLE 5

Table 5. Completeness, accuracy and quality of research and educational data elements.

Concerning entire data, the total data quality of hospital B (70%) is higher than hospitals A and C (54% and 69%), and the quality of the D data group (more than 97%) is considerably higher than the other groups in the three hospitals.

Concerning research and educational data elements, hospital B has better total data quality compared to the others (70% vs. 53 and 66%). Moreover, the D data group has the highest quality (100%) in the three hospitals. It is equal to the AF in hospital C and followed by the LP data group in hospitals B and A (95 and 89%, in order).

Comparison of completeness and accuracy among the three hospitals

Table 6 shows the probability values of completeness and accuracy for entire data, which differ significantly among the three hospitals. The completeness of data groups but D were significantly different when comparing the three hospitals. There were significant differences in terms of accuracy of CH, DT, and LP data groups among the three hospitals (Table 7).

TABLE 6

Table 6. Comparison of completeness and accuracy for entire data values.

TABLE 7

Table 7. Comparison of completeness and accuracy for research and educational data values.

Discussion

We assessed the data quality of the HIS used in the hospitals belonging to the Mashhad University of Medical Sciences. To this, we performed a retrospective comparison between a sample of electronic and paper medical records in three academic hospitals. We calculated HISRA and five data groups' quality (completeness * accuracy), namely demographics, care handler, diagnosis and treatment; administrative and financial; and laboratory and Para clinic.

Assessment of the HIS Software used in these three hospitals indicated low HISRA for entire data elements (58.5%). It was even lower (47%) when considering research and educational data elements. The diagnosis and treatment data group had the poorest HISRA (less than 45%) among the five data groups, while HISRA for demographics, laboratory and Para clinic; and administrative and financial data groups was over 90%. Poor HISRA observed for the diagnosis and treatment data group was due to the lack of corresponding data elements in the HIS software to capture some essential data such as physical examinations; detailed nurse notes; vital signs control; fluid absorption/excretion; pre-operation; post-operation, and anesthesia care, and their timelines. HIS software flaw also has increased inaccuracy of the diagnosis and treatment data group due to the lack of data validity checks in open-text fields.

For care handler; and diagnostic and treatment data, although the accuracy was relatively good (over 80% in all hospitals), incompleteness reduced the overall data quality to less than 70% for both entire data; and research and educational data. We observed that policies and regulations flaw has led to reduced completeness of diagnostic and treatment data in the three hospitals because caregivers are not required to record medical history, disease progress received consultations, and physician orders completely in electronic records. Poor policies and regulations also resulted in reduced completeness of the care handler group because clinical staff did not have to capture such information within the HIS. In the care handler group, inaccuracy commonly resulted from the wrong timestamps. Because the timestamp was created automatically by the computer and for a late data entry, which was not done at the point of medical action, it reflects the time of computerized data entry, not the real-time of the medical procedure. Moreover, EMRs sometimes included the user who entered the data instead of the doer of medical procedures.

According to our observations, the quality of laboratory and para clinic data in hospital B was the highest (over 90%). It was relatively good (over 80%) in hospital A, and unexpectedly poor in hospital C. This happened because, in the hospital C, a laboratory or para clinic report is recorded in both electronic and paper records, provided that it was done by the hospital's laboratory or para clinic section. Otherwise, it is held only in paper records. Whereas, in this hospital, it is common to refer patients to outside laboratories to do some physician-ordered tests. Similarly, the three hospitals, capture the medicine administration in the patient's EMR when the ordered medicine has been provided by the hospital's pharmacy. Otherwise, it was recorded only in the paper charts as this procedure flaw has increased inconsistency between paper and electronic records and has led to low completeness of laboratory and Para clinic data in hospital C and low completeness of diagnosis and treatment data in all three hospitals.

Several studies investigated the quality of electronic medical data used for research purposes; they employed various assessment strategies to report data quality. We found studies that referred to “data availability” as “completeness” and assessed it by three major strategies, (i) comparing with a gold standard, (ii) checking the presence or absence of data value, and (iii) comparing with data from other data sources.

We also found studies that defined “agreement among data elements” as “concordance, consistency, validity, correctness or accuracy”. Considering the publications that we found, data agreement was assessed by two main strategies, including (i) Data Element Agreement, comparing the data with those within the EMR, and (ii) comparing the data with those from other sources that may be a gold standard. Other used data sources include paper records (11–18) patient-reported data (19–21) or physician-reported data (22). Some studies reported the overall completeness and accuracy of the electronic data, comparing paper records as the “gold standard” (11–14, 23). Some others compared electronic medical data to paper records as the “data source”(15–17).

Compared to the mentioned studies, our study had a higher level of granularity of the elements measured and benefited from extensive assessment. Although most studies assessed data elements that serve the purpose of doing a determined study in the future (diagnosis and procedure codes related to a specific disease, for instance) (11–14, 16, 17), our study did not focus on the suitability of data elements for determining research. We tried to identify and assess essential data for research and education. We also expanded the assessment to entire recorded data (even financial ones) to investigate how to improve HIS to collect effective records. Our definitions of completeness, and accuracy, and the protocol we used to measure them were consistent with some previous studies. However, it was quite different from that employed by Whitelaw et al. (14) and Logan et al. (21) and also was somewhat different from the definition of Prins et al. (13).

In light of our findings, the following suggestions are offered to improve the HIS software used in these hospitals and as instructions to be given to the medical staff:

Given the low HISRA of the software, particularly in terms of diagnosis and treatment data, the HIS used in these hospitals needs to improve to record physical examinations, pre-operation, post-operation, anesthesia care, detailed nurse notes, vital signs, fluid absorption and excretion, and their time trends. Additionally, in these hospitals, a great amount of data are entered in an open text and non-structural form. By adding data-entry fields, software capabilities can facilitate input validation checking, which may increase the completeness and accuracy of the recording process. Considering the incompleteness of the care handler data group, the medical staff should be instructed to record the doers of the medical action. Moreover, digital signature and audit logging should be implemented to assure care handler data accuracy.

Hospitals should set strict rules assuring the medical staff records the patient's history, physician's orders, disease progress notes, and consultations. HIS also must be enabled to record prescriptions and medications together with clinical information received from other medical centers outside the hospital to track medication and refill histories. This trend will certainly increase the completeness of the diagnosis and treatment; and laboratory and Para clinic data.

Since we frequently observed inaccuracy in date and time in the diagnosis and treatment data group, HIS must be redesigned to record the date and time for each entry based on the actual date and time. HIS must also provide the documenter with the possibility of manually entering the date and time of late entries.

If it is necessary to have both electronic and paper medical records in parallel, it is recommended to omit the hand-written paper records and print the computer-generated EMR. This procedure ensures legibility, completeness, validity, and precision of the paper record by utilizing software controls such as “input masks”. It also increases the concordance between electronic and paper records.

Studies that use the “Data Source Agreement” strategy to assess the quality of electronic data prefer to compare data with a valid and reliable external data source. We used patients' paper records as the assessment standard even though they might have been illegible, incomplete, or unreliable; they were the only available documentation usually used in addition to electronic data by medical care providers in the mentioned hospitals. Thus, the absence of a reliable external data source formed a limitation of the present study. Furthermore, the result was not influenced by the transformation of the electronic record or the paper record, or both (because data values often had been recorded in the form of open text). However, accuracy assessment was influenced by the personal impression of the investigators as well as the performance of the persons who entered the data in to the EMRs.

Future work

As a possible future work, one could investigate the degree to which the data deviate from a truly “random sample” of the patients' data. For example, if normal laboratory values are more likely to be in the computer and abnormal values are more likely to be on paper, this could have a pronounced impact, and bias, on research that attempts to use only one or the other.

Conclusions

The findings demonstrated that the HISRA of the system was not good in general, and it was poor for diagnosis and treatment data in particular. It also showed the low quality of registering the care handler; and diagnosis and treatment data groups.

The high quality of administrative and financial data groups imply that in the three hospitals, the development of the software and data entry policies focus on managerial, administrative, and financial applications rather than potential academic and clinical applications. Thereby, it is suggested that the clinical and diagnostic aspects of those EMRs should be addressed. Easier data entry methods, better training of personnel, and changes in the HIS software can probably solve many obstacles. Furthermore, regulations and policies need to be revised to serve the data quality improvement.

Data availability statement

The data analyzed in this study is subject to the following licenses/restrictions: The data for this study was obtained from databases in three hospitals that belong to Mashhad University of Medical sciences. Both forms of electronic and paper records are protected severely from general access by confidentiality and privacy regulations, and in addition to the care providers, researchers affiliated with the University have access to anonymized records for a definite time. None of the authors had access to any identifying patient information. The data were anonymized at the point of extraction from the systems and paper forms. The researcher signed a commitment to maintaining the confidentiality of information which do not allow them to keep, disclose or reuse the information, for another study, for example, or other purposes after the end of the study. Requests to access these datasets should be directed to the corresponding author.

Author contributions

HZ: Performed the analysis; Wrote the paper. HZ designed the study, collected the data, performed the analytic calculations, and wrote the manuscript with support from authors SE and MRH. Both SE and MRH conceived of the original idea, verified the analytical methods and supervised the findings of this work. SD contributed to the final version of the manuscript. All authors contributed to the article and approved the submitted version.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher's note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

References

1. Reference.MD [updated 6/6/2012; cited 2014 Friday, 04-Jul-2014]. Available from: http://www.reference.md/files/D006/mD006751.html (Accessed February 9, 2018).

2. Wagner MM, Hogan WR. The accuracy of medication data in an outpatient electronic medical record. J Am Med Inform Assoc. (1996) 3(3):234–44. doi: 10.1136/jamia.1996.96310637

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Weiskopf NG, Chunhua W. Methods and dimensions of electronic health record data quality assessment: enabling reuse for clinical research. J Am Med Inform Assoc. (2013) 20(1):144–51. doi: 10.1136/amiajnl-2011-000681

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Ahmadi M, Rezaei H, Shahmoradi L. Electronic health record: structure, content, and evaluation. Tehran: Jafari Publication (2008). 4–8.

5. Haynes K, Forde KA, Schinnar R, Wong P, Strom BL, Lewis JD. Cancer incidence in the health improvement network. Pharmacoepidemiol Drug Saf. (2009) 18(8):730–6. doi: 10.1002/pds.1774

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Kushniruk A, Borycki E, Joe R, Otto T, Armstrong B, Ho K. Integrating electronic health records into medical education: Considerations, challenges, and future directions. New York: Springer (2012). doi: 10.1007/978-1-4614-3495-5_2 (Accessed February 16, 2018).

7. Tierney MJM, Pageler NMM, Kahana MM, Pantaleoni JLM, Longhurst CAMM. Medical education in the electronic medical record (EMR) era: benefits, challenges, and future directions. Acad Med. (2013) 88(6):748–5. doi: 10.1097/ACM.0b013e3182905ceb

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Sadoughi F, Aminpour F. A review on the evaluation methods of health information systems. Iran J Med Educ. (2011) 10(5):1077–86.

Google Scholar

9. Moghaddasi H, Asadi F, Hossaini A, Mohammadpour A. Hospital information system in Iran: findings from a systematic literature review. Hakim Research Journal. (2013) 16(3):8.

Google Scholar

10. Brinkman W-P. Design of a questionnaire instrument, handbook of Mobile technology research methods. New York: Delft University of Technology, Nova Publishers (2009). p. 31–57.

11. Ricketts D, Patterson M, Newey M, Hitchin D, Fowler S. Markers of data quality in computer audit: the Manchester Orthopaedic Database. Ann R Coll Surg Engl. (1993) 75(6):393–6. PMID: 8285541; PMCID: PMC2498017.8285541

PubMed Abstract | Google Scholar

12. Barrie JL, Marsh DR. Quality of data in the Manchester orthopaedic database. BMJ Clinical Research. (1992) 304(6820):4. doi: 10.1136/bmj.304.6820.159

CrossRef Full Text | Google Scholar

13. Prins H, Kruisinga FH, Büller HA, Zwetsloot-Schonk JHM. Availability and accuracy of electronic patient data for medical practice assessment. In: Hasmann A, Blobel B, Dudeck J, Engelbrecht R, Gell G, Prokosch HU, editors. Medical infobahn for Europe. Proceedings of MIE2000 and GMDS2000. Amsterdam: IOS (2000). p. 484–8.

14. Whitelaw FG, Nevin SL, Milne RM, Taylor RJ, Taylor MW, Watt AH. Completeness and accuracy of morbidity and repeat prescribing records held on general practice computers in Scotland. Br J Gen Pract. (1996) 46(404):181–6. PMID: 8731627; PMCID: PMC12395818731627

PubMed Abstract | Google Scholar

15. Mikkelsen G, Aasly J. Concordance of information in parallel electronic and paper based patient records. Int J Med Inform. (2001) 63:123–31. doi: 10.1016/S1386-5056(01)00152-6

PubMed Abstract | CrossRef Full Text | Google Scholar

16. Jick H, Terris BZ, Derby LE, Jick SS. Further validation of information recorded ona general practitioner based computerized data resource in the United Kingdom. Pharmacoepidemiol Drug Saf. (1992) 1:347–9. doi: 10.1002/pds.2630010607

CrossRef Full Text | Google Scholar

17. Stausberg J, Koch D, Ingenerf J, Betzler M. Comparing paper-based with electronic patient records: lessons learned during a study on diagnosis and procedure codes. J Am Med Inform Assoc. (2003) 10:470–7. doi: 10.1197/jamia.M1290

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Neal RD, Heywood PL, Morley S. Real world data e retrieval and validation of consultation data from four general practices. Fam Pract. (1996) 13:455–61. doi: 10.1093/fampra/13.5.455

PubMed Abstract | CrossRef Full Text | Google Scholar

19. Pakhomov SV, Jacobsen SJ, Chute CG, Roger VL. Agreement between patientreported symptoms and their documentation in the medical record. Am J Manag Care. (2008) 14:530–9. PMID: 18690769; PMCID: PMC258150918690769

PubMed Abstract | Google Scholar

20. Pringle M, Ward P, Chilvers C. Assessment of the completeness and accuracy of computer medical records in four practices committed to recording data on computer. Br J Gen Pract. (1995) 45:537–41. PMID: 7492423; PMCID: PMC12394057492423

PubMed Abstract | Google Scholar

21. Logan JR, Gormann PN, Middleton B. Measuring the quality of medical records: a method for comparing completeness and correctness of clinical encounter data. In: Bakken S, editor. A medical odyssey: visions of the future and lessons from the past. Proc AMIA 2001 annu symp (2001). p. 408–12.

22. Conroy MB, Majchrzak NE, Silverman CB, Chang Y, Regan S, Schneider LI, et al. Measuring provider adherence to tobacco treatment guidelines: a comparison of electronic medical record review, patient survey, and provider survey. Nicotine Tob Res. (2005) 7(Suppl. 1):S35–43. doi: 10.1080/14622200500078089

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Hogan WR, Wagner MM. Accuracy of data in computer-based patient records. J Am Med Inform Assoc. (1997) 4:342–55. doi: 10.1136/jamia.1997.0040342

PubMed Abstract | CrossRef Full Text | Google Scholar