Experience with Quality Assurance in Two Store-and-Forward Telemedicine Networks

Despite the increasing use of telemedicine around the world, little has been done to incorporate quality assurance (QA) into these operations. The purpose of the present study was to examine the feasibility of QA in store-and-forward teleconsulting using a previously published framework. During a 2-year study period, we examined the feasibility of using QA tools in two mature telemedicine networks [Médecins Sans Frontières (MSF) and New Zealand Teledermatology (NZT)]. The tools included performance reporting to assess trends, automated follow-up of patients to obtain outcomes data, automated surveying of referrers to obtain user feedback, and retrospective assessment of randomly selected cases to assess quality. In addition, the senior case coordinators in each network were responsible for identifying potential adverse events from email reports received from users. During the study period, there were 149 responses to the patient follow-up questions relating to the 1241 MSF cases (i.e., 12% of cases), and there were 271 responses to the follow-up questions relating to the 639 NZT cases (i.e., 42% of cases). The collection of user feedback reports was combined with the collection of patient follow-up data, thus producing the same response rates. The outcomes data suggested that the telemedicine advice proved useful for the referring doctor in the majority of cases and was likely to benefit the patient. The user feedback was overwhelmingly positive, over 90% of referrers in the two networks finding the advice received to be of educational benefit. The feedback also suggested that the teleconsultation had provided cost savings in about 20% of cases, either to the patient/family, or to the hospital/clinic treating the patient. Various problems were detected by regular monitoring, and certain adverse events were identified from email reports by the users. A single aberrant quality reading was detected by using a process control chart. The present study demonstrates that a QA program is feasible in store-and-forward telemedicine, and shows that it was useful in two different networks, because certain problems were detected (and then solved) that would not have been identified until much later. It seems likely that QA could be used much more widely in telemedicine generally to benefit patient care.

Introduction

Telemedicine can be broadly defined as any healthcare activity carried out at a distance. One common application of telemedicine concerns long-distance clinical consultation, where the doctor being consulted may be located in a different city or country from the patient. This reduces travel time and costs, and improves access to specialist healthcare. The use of telemedicine has grown in recent years as equipment has become more sophisticated, telecommunication networks have become more widespread and reliable, and the need for efficient use of healthcare resources has become more pressing. The assumption underlying the present study is that in any healthcare activity that has become routine procedure (i.e., it is not research), some sort of quality assurance (QA) will be desirable, perhaps mandatory. QA is the systematic process of checking regularly to see whether a product or service continues to meet specified requirements (1). Such a process will increase both customer confidence and the provider’s credibility, and will improve the efficiency of the work being carried out. The important features of a QA program have been summarized as (1)

1. a focus on the organization’s mission and the customer’s needs

2. a systematic approach to improvement, e.g., using the plan-do-check-act cycle, which offers a scientific method for continuous process improvement

3. stimulation for the development of human resources

4. facilitation of long-term thinking

5. the commitment of every participant.

Thus, a QA program will enable the people responsible for the service to satisfy themselves that its quality is being maintained. It will also allow them to identify problems at an early stage, with a view to solving them.

Quality assurance has been little used in telemedicine, apart from in image-based areas, such as radiology (2), retinal screening (3), or cytology (4). Liddy et al. added routine collection of user feedback data in a store-and-forward telemedicine network – this might be regarded as informal QA (5). Some guidance on routine audit of teledermatology services has also been published (6). Although QA is uncommon in telemedicine at present, there is no reason in principle why it cannot be applied more widely. In particular, QA of consultations taking place by telemedicine is an area of interest, whether such consultations occur in real time (e.g., by video) or asynchronously (e.g., by email).

The Collegium Telemedicus system is a general purpose system for managing telemedicine consultations (7). It is aimed at organizations which deliver services in low-resource settings, although its application is wider than that. Organizations which have successfully established telemedicine networks using the Collegium system may wish to conduct a program of QA; the Collegium system, therefore, includes a number of tools for this purpose. To date, there has been no published evidence relating to their use.

QA in Telemedicine

We have previously proposed a framework for assessing telemedicine networks which provide teleconsultation services to doctors (8). The framework includes performance indicators for each of the three user groups (i.e., referrers, experts, and case coordinators). In addition, the framework covers the societal perspective, based on information about clinical- and cost effectiveness and integration with the conventional health care system, see Table 1.

TABLE 1

Table 1. Performance measurement framework for assessing telemedicine networks that provide teleconsultation services to doctors (8).

This framework represents a basic method of performance ­measurement with which teleconsultation networks can be measured and compared with one and other. However, QA requires long-term monitoring, rather than intermittent snapshots. Thus, a QA program depends on being able to measure appropriate performance indices but also requires methods for interpreting the resulting time-series data. In the Collegium system, the tools for QA include

1. performance reporting (on demand and automated) to assess trends

2. automated follow-up of patients to obtain outcomes data

3. automated surveying of referrers to obtain user feedback

4. retrospective assessment of randomly selected cases to assess quality.

These tools allow the majority of the indicators in the framework to be monitored, either directly or indirectly, see Table 1. The interpretation itself can be done in two ways. Graphs of performance statistics are useful for detecting long-term trends, but a more powerful method of obtaining early warning of adverse changes is by use of control charts that allow the sources of variation to be identified (9). In particular, common-cause variation (the usual, random variation which occurs in any process) can be differentiated from special-cause variation (which may indicate a change in the process requiring further investigation).

Aim

The aim of the present study was to investigate the feasibility of using these tools in two mature telemedicine networks, and to review the results obtained. The hypothesis was that appropriate QA tools can be usefully applied to telemedicine networks, producing results that will assist clinical care and effective management.

Materials and Methods

Study Design

The feasibility of QA was examined in two Collegium networks over a period of 2 years, from August 2013 to July 2015 inclusive. The experience was reviewed by two telemedicine researchers at the end of the study period. Ethics permission was not required, because patient consent to access the data had been obtained and the work was a retrospective chart review conducted by the staff of the organizations concerned in accordance with their research policies (10).

Networks

The Médecins Sans Frontières (MSF) network provides ­teleconsultations across a wide range of medical, surgical, and allied health disciplines, for health care workers in low-resource settings (11). During the study period, 122 referrers consulted 200 specialists about a total of 1241 cases. The New Zealand Teledermatology (NZT) network provides teledermatology consultations for general practitioners in certain health regions of New Zealand (12). During the study period, 57 referrers consulted four dermatologists about a total of 639 cases. The two networks, therefore, differed in size, workload, client base, and setting, thus providing contrasting conditions for the study.

QA Program

The QA program implemented in the two networks was based on regular monitoring of

1. performance indicators, which were made available to case coordinators on demand via the system’s web interface. The indicators included the case referral rate (all cases, and in selected specialties), the time to allocation to a specialist, and the time to first response. Each month, summary reports of selected performance indicators were also sent automatically to the case coordinators by email

2. outcomes data. Information on patient outcomes was solicited from all referrers automatically. The system requested completion of a short progress report containing questions about the patient (13).

3. user feedback. Information about the management of the case, any benefits from the consultation, and the referrer’s opinion of the service generally was solicited from all referrers automatically. The system requested completion of a short progress report containing user-feedback questions (13).

4. overall teleconsultation quality, which was assessed each month by a QA review panel. We have previously described the measurement of teleconsultation quality, i.e., the “output” from the network (14). A panel of appropriately qualified observers responds to a questionnaire relating to a randomly chosen past case. The review panels were open to all network coordinators. The answers to the questionnaire allow two different dimensions of quality to be assessed: the quality of the process itself and the outcome, defined as the value of the response to three of the four parties concerned, i.e., the patient, the referring doctor and the organization. It is not practicable to estimate the value to society by this technique.

In addition, the senior case-coordinators in each network were responsible for identifying potential adverse events from email reports received from users.

There were four case coordinators on the NZT network and 19 coordinators on the MSF network. Not all coordinators were active for the whole study period. All coordinators participated in some monitoring activities (e.g., receiving email reports and notifications of completed user feedback surveys) and one senior coordinator in each network participated in all monitoring activities and in adverse incident detection.

Results

Regular Monitoring

1. Performance indicators. Monthly summary reports were sent to the coordinators, see Figure 1. In addition, plots were produced on demand, see Figure 2. These allowed the detection of trends in important network performance indices. For example, there was a trend to reduced allocation delay in the MSF network during the study (Figure 3). Although gratifying, the trend was not significant (P = 0.22).

2. Outcomes data. During the study period, there were 149 responses to the follow-up questions relating to the 1241 MSF cases (i.e., 12% of cases). During the study period, there were 271 responses to the follow-up questions relating to the 639 NZT cases (i.e., 42% of cases). The responses are summarized in Appendix 1. Referrers provided useful follow-up information on the cases, such as “Patient had a suspicious finding in the chest X ray which was inconsistent with the clinical course. But this was well clarified by the radiology expert.” (MSF) and “Mother says treatment cream is magic and is very happy” (NZT).

3. User feedback. The collection of user feedback reports was combined with the collection of patient follow-up data, thus producing the same response rates. The responses are summarized in Appendix 2. Referrers made very positive comments about the telemedicine service, such as “It is great to be able to have expert advice in a very short time. It helps a lot to evaluate better and to make the right decisions for unknown diseases/symptoms” (MSF) and “I think it is an excellent service. It helps to educate us GPs, if we can get advice on what a lesion is, then next time we see it we’ll know how to manage it better and therefore make fewer referrals to dermatologists” (NZT).

4. Overall teleconsultation quality. Monthly case reviews were carried out in both networks. One network used a single observer, and the second used a panel of observers. In the MSF network, a total of 123 assessments were carried out by the review panel and there were 222 non-responses (see Table 2). The process control chart for the Grand Quality Score indicated that there was a problem near the end of the study period, where special-cause variation was detected (MSF case 1751 fell outside the control limits), see Figure 4. This was investigated and remedial action was taken, see Table 3.

FIGURE 1

Figure 1. Monthly email reports to the coordinators on the two networks.

FIGURE 2

Figure 2. Performance graph (produced on demand).

FIGURE 3

Figure 3. Trend in the average delay in allocation of new cases (MSF network). The solid line is the linear regression. In month 8, there were 23 referrals. The apparent outlier in allocation delay at month 8 was caused by a single case in which an image dataset had to be sent by post, because the Internet connection at the field site in question did not allow a very large file to be uploaded.

TABLE 2

Table 2. Number of assessments conducted by the QA review panel in the MSF network.

FIGURE 4

Figure 4. Process control chart for the Grand Quality Score in the MSF network.

TABLE 3

Table 3. Examples of problems detected during the QA program.

Problems Detected

During the 2-year study period, a number of problems were detected by regular monitoring. Some examples from the two networks are provided in Table 3.

In addition, various adverse events were identified in each network. For example, in the MSF network, there were several telemedicine cases for which appropriate specialists were not available, and new specialists had to be recruited at short notice. In the NZT network, there were instances of duplicate case ­numbering (data subsequently amalgamated), misunderstandings about image uploading (help documentation updated), difficulties in using the system with a mobile device (mobile interface developed), missing images (systems error corrected by IT staff), large files not ­uploading (file transfer problem ­corrected), and difficulty in finding a specific case (free-text keyword search function added).

Discussion

In the present study, the feasibility of conducting a comprehensive QA program was examined in two relatively stable and mature telemedicine networks. Thus, the nature of the study was operational research, rather than clinical research, and the methodology employed was able to take advantage of existing systems. We are not aware of previous similar work.

Outcomes Data

The limited outcomes data available for MSF cases suggested that the telemedicine advice proved useful for the referring doctor in the majority of cases and was likely to benefit the patient (Appendix 1). The percentage of requests for follow-up information that were fulfilled in the two networks was much lower in the MSF network, presumably because of the high turnover of field staff, and patients being discharged and, therefore, lost to follow up. Conversely, most referrers in the NZT network were stable general practitioners, who closely followed up their patients. There may also have been a response bias whereby referrers with positive experiences tended to respond to requests for follow-up information, whereas those with negative experiences did not.

User Feedback

The user feedback was overwhelmingly positive (Appendix 2). It also suggested that the teleconsultation had provided cost savings in about 20% of cases, either to the patient or family, or to the hospital/clinic treating the patient. The comments provided by the referrers suggested that the cost savings to the patient or family were mainly due to avoided journeys, i.e., that telemedicine removed the need for a trip to hospital for a specialist appointment; this was the case in both networks. The comments provided by the referrers suggested that the cost savings to the hospital or clinic were also mainly due to avoiding unnecessary referrals.

Comparison Between the Two Networks

Three important questions concerning patient follow-up and five important questions concerning user feedback were compared between the two networks. In all eight questions, the proportion of responders answering “yes” was higher in the NZT network than in the MSF network, Table 4. However, the differences were not significant in three of the questions. The largest differences were in the question about whether the eventual outcome would be beneficial for the patient (75% of NZT responders vs. 51% of MSF responders) and whether the referrer was able to follow the advice provided (94% of NZT referrers vs. 70% of MSF referrers). This probably reflects the different settings of the two networks. For example, the NZT network deals with a single specialty (dermatology), whereas the MSF network deals with a very wide range of medical, surgical, nursing, and allied health matters. In addition, MSF referrers operate in resource-constrained settings, which often limit their diagnostic and therapeutic options.

TABLE 4

Table 4. Summary responses – comparison between networks.

Control Charts

Over a 2-year study period, one aberrant quality reading was detected. Further investigation found that in providing a response to the referrer, various problems had occurred that reduced the quality of the resulting consultation (see Table 3).

Points Arising

There could be better integration of the outcomes data with the performance indices. Further software development is required in order to allow ready visualization of outcomes data.

Strengths and Weaknesses

The present study was conducted in two operational telemedicine networks, and was therefore, based on real, not simulated data – this represents a strength. The results represent a validation of the service being delivered, demonstrating its value to the patients, users, and organizations involved. Audit and quality improvement of this kind are aligned with contemporary thinking about the importance of overseeing service function and safety. These are all strengths.

However, the judgment about the value of the QA activities as a whole was subjective. That is, it was not feasible to conduct a study in which a telemedicine network using QA was compared with a control telemedicine network that did not use QA. In conducting the QA program, several problems came to light, and were dealt with, which might otherwise have gone unnoticed. But it is hard to prove this. What we can say is that the MSF network had operated for 4 years before the QA program was introduced, and in this period, there was obviously no formal monitoring of performance data. Thus, the QA process enforced a discipline of monitoring, leading to the early detection of problems.

Future Work

The present study shows that QA is feasible in the two networks studied, and also suggests that it is useful. The next stage must be the organizational adoption of a comprehensive QA program for telemedicine by the organizations concerned. Such a program would include procedures to review reports, monitor cases/users and check data, as described in the present paper. In addition, a comprehensive QA program would investigate protocol compliance, to identify protocol violations. For example, anecdotal reports from several of the Collegium networks suggests that problems are often caused unintentionally by users who upload files in proprietary formats (despite automatically generated warnings in the software, and the provision of user documentation explaining that the intended recipient of a file in a non-open source format may be unable to read it). There have also been problems caused by referrers who submit the same case more than once. Finally, a comprehensive QA program would include the measurement of service availability, e.g., web site performance.

Conclusion

The present study demonstrates that a QA program is feasible in store-and-forward telemedicine. Experience over the first 2 years in two different networks indicates that QA is also useful, because certain problems were detected (and then solved) that would not have been identified until much later. The QA tools available via the Collegium system worked well and provided the foundation for a comprehensive QA program. QA elements examined in the two different networks produced similar findings. Thus, the hypothesis that QA can be applied effectively in a teleconsulting network was supported. It seems likely that QA could be used much more widely in telemedicine generally.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank Will Wu for software support. We are grateful to the members of the QA review panels and to other colleagues concerned with the day-to-day operation of the Collegium networks.

References

1. Busari JO. Comparative analysis of quality assurance in health care delivery and higher medical education. Adv Med Educ Pract (2012) 3:121–7. doi: 10.2147/AMEP.S38166

PubMed Abstract | CrossRef Full Text | Google Scholar

2. McLean ID, Heggie JC, Herley J, Thomson FJ, Grewal RK. Interim recommendations for a digital mammography quality assurance program. Australas Phys Eng Sci Med (2007) 30(2):65–100. doi:10.1007/BF03178412

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Erginay A, Chabouis A, Viens-Bitker C, Robert N, Lecleire-Collet A, Massin P. OPHDIAT: quality-assurance programme plan and performance of the network. Diabetes Metab (2008) 34(3):235–42. doi:10.1016/j.diabet.2008.01.004

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Harnden P, Coleman D, Moss S, Kodikara S, Griffin NR, Melia J. Evaluation of the use of digital images for a national prostate core external quality assurance scheme. Histopathology (2011) 59(4):703–9. doi:10.1111/j.1365-2559.2011.03987.x

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Liddy C, Afkham A, Drosinis P, Joschko J, Keely E. Impact of and satisfaction with a new econsult service: a mixed methods study of primary care providers. J Am Board Fam Med (2015) 28(3):394–403. doi:10.3122/jabfm.2015.03.140255

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Primary Care Commissioning. Quality Standards for Teledermatology Using ‘Store and Forward’ Images (2013). Available from: http://www.pcds.org.uk/article/quality-standards-for-teledermatology

Google Scholar

7. Wootton R, Wu WI, Bonnardot L. Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach. J Telemed Telecare (2013) 19(7):411–7. doi:10.1177/1357633X13506511

PubMed Abstract | CrossRef Full Text | Google Scholar

8. Wootton R, Vladzymyrskyy A, Zolfo M, Bonnardot L. Experience with low-cost telemedicine in three different settings. Recommendations based on a proposed framework for network performance evaluation. Glob Health Action (2011) 4:7214. doi:10.3402/gha.v4i0.7214

PubMed Abstract | CrossRef Full Text | Google Scholar

9. Anhøj J, Olesen AV. Run charts revisited: a simulation study of run chart rules for detection of non-random variation in health care processes. PLoS One (2014) 9(11):e113825. doi:10.1371/journal.pone.0113825

PubMed Abstract | CrossRef Full Text | Google Scholar

10. MSF Ethics Review Board. MSF Research Ethics Framework – Guidance Document. (2013). Available from: http://fieldresearch.msf.org/msf/handle/10144/305288

Google Scholar

11. Bonnardot L, Liu J, Wootton E, Amoros I, Olson D, Wong S, et al. The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system. Front Public Health (2014) 2:126. doi:10.3389/fpubh.2014.00126

PubMed Abstract | CrossRef Full Text | Google Scholar

12. McGoey ST, Oakley A, Rademaker M. Waikato teledermatology: a pilot project for improving access in New Zealand. J Telemed Telecare (2015) 21(7):414–9. doi:10.1177/1357633X15583216

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Wootton R, Liu J, Bonnardot L. Quality assurance of teleconsultations in a store-and-forward telemedicine network – obtaining patient follow-up data and user feedback. Front Public Health (2014) 2:247. doi:10.3389/fpubh.2014.00247

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Wootton R, Liu J, Bonnardot L. Assessing the quality of teleconsultations in a store-and-forward telemedicine network. Front Public Health (2014) 2:82. doi:10.3389/fpubh.2014.00211

CrossRef Full Text | Google Scholar

Appendix 1

APPENDIX 1

Summary of patient follow-up.

FIGURE S1

Figure S1. Word frequencies* in the free-text responses to the invitation to comment about why the advice was helpful in the case being followed up. *See http://tagcrowd.com/.

Appendix 2

APPENDIX 2

Summary of referrer feedback.

FIGURE S2

Figure S2. Word frequencies* in the free-text responses to the invitation to comment about the telemedicine service.*See http://tagcrowd.com/.