User Feedback on the MSF Tele-Expertise Service After a 4-Year Pilot Trial – A Comprehensive Analysis

We surveyed all users of the Médecins Sans Frontières (MSF) tele-expertise service, approximately four years after it began operation. The survey contained 50 questions and was sent to 294 referrers and 254 specialists. There were 163 responses (response rate 30%). There were no significant differences between the responses from French and English users, so the responses were combined for subsequent analysis. Most of the responders were doctors (133 of 157 who answered that question), and most had completed field missions for MSF, i.e., both specialists and referrers. The majority stated that the system was user friendly and that they found it self-explanatory (i.e., they did not need to be shown how to use it). Almost all the referrers found that the telemedicine advice that they received was helpful, changed diagnosis and management, and/or reassured the patient. Similar feedback came from the specialists, who also felt that there was educational value for the field doctor. Although there was general satisfaction with the service, the survey identified various problems. The main concerns of the referrers were the lack of promotion of the system at headquarters’ level, and the main concerns of the specialists were the lack of feedback about patient follow-up. Nonetheless, both referrers and specialists recognized the benefits of telemedicine in improving patient management, providing education, and reducing isolation in the field.

Introduction

In 2009, Médecins Sans Frontières (Doctors Without Borders, MSF) began a pilot trial of a store-and-forward medical tele-expertise service. “Tele-expertise” is defined in the French Public Health Code (1), as one of the five main areas of telemedicine (teleconsultation, tele-assistance, tele-monitoring, medical emergency call center, and tele-expertise). The MSF system uses electronic message exchange between field clinicians working in resource-limited settings without access to specialized medical expertise, and specialists working in relatively well-resourced settings who offer pro bono specialist advice to their field colleagues. In collaboration with the Swinfen Charitable Trust (2), three separate tele-expertise networks (French, English, and Spanish) were established initially and then combined into a single multilingual network in the late 2013, using web-based technology based on the Collegium Telemedicus system (3).

The MSF tele-expertise service provides specialized advice to field clinicians across a very wide range of medical and surgical specialties. At present more than 350 MSF field clinicians have access to the system, and there are 300 volunteer specialists from all over the world who provide tele-expertise; the median delay in providing the first specialist response to the referrer is less than 6 h (4).

To ensure that the allocation of resources is appropriate in health care, pilot projects require careful evaluation. In the clinical field, such an evaluation can also be considered as a necessary reflective practice, carried out in order to improve both the efficiency of the system and provider practice (5). We therefore surveyed the users of the MSF pilot tele-expertise service to obtain their feedback in terms of satisfaction and benefit, to draw conclusions about the usefulness of the service, and to identify potential areas of improvement. In a previous paper, we presented the main results of the survey (4). The present study provides the raw data and a further and more detailed analysis. In order to understand the results and put them in perspective, we compared them with the (limited) data which has been published on other, similar telemedicine networks (6, 7).

Materials and Methods

We carried out a survey of all users with an account on the tele-expertise network in January 2014. The questions were established after conducting a literature search combined with qualitative data collection:

1. in-depth interviews with three referrers and three specialists;

2. participant observation performed by two of the authors during several field missions in MSF settings (Democratic Republic of the Congo, Central African Republic, Yemen, Somalia, Syria, and Haiti).

The survey contained 50 questions. They were closed-ended, multiple choice and scale type questions, and open-ended questions.

The survey was tested on three referrers and three specialists, in English and in French. After the pilot testing, the survey was sent separately to all referrers (French and English) and specialists (French and English) with accounts on the system, regardless of whether they were active (i.e., irrespective of whether they had logged in and submitted cases or answered them). The questions for referrers and specialists were almost identical. Versions of the survey were made available in French and English. Web-based survey software (https://www.surveymonkey.com/) was used to collect the data. Two reminders were sent by email to the users, after 1 and 3 weeks. Data collection was closed after 1 month.

Ethics permission to conduct the survey was not required because patient consent to access the data had been obtained previously, and the work was a retrospective chart review conducted by the organization’s staff in accordance with its research policies (8).

Survey responses were examined with the usual methods for quantitative analysis, while the results of the open-ended questions were processed in a qualitative way based on a systematic content analysis.

Differences Between Language Groups

Responses were received to both the English and French versions of the survey. To examine potential differences between the two language groups, the English and French responses were compared using chi-squared tests for certain key variables:

1. Age. The respondents’ ages were recorded in five age categories. Since the age categories were ordered, a chi-squared test for trend was carried out.

2. Qualifications. The respondents’ qualifications were recorded in three categories. These were unordered categories, so Fisher’s exact test was used with P-values calculated according to the Freeman–Halton extension (9, 10).

3. User-friendliness of the system interface. The respondent’s opinion about the user-friendliness of the system was categorized as yes/no, so Fisher’s exact test was used.

4. Value of telemedicine. The respondent’s opinions about whether telemedicine improved patient management was recorded in three unordered categories, so Fisher’s exact test was used with P-values calculated according to the Freeman–Halton extension (9, 10).

Differences Between Referrers and Specialists

Responses from referrers and specialists were compared, irrespective of language, using chi-squared tests. Where multiple ordered responses were possible, the chi-squared test for trend was employed.

Results

Users and Response Rate

A total of 294 referrers and 254 specialists had accounts on the MSF network. From an examination of the system database, a total of 104 referrers (35%) had submitted cases and 120 specialists (47%) had replied to queries, i.e., 224 of the 548 account holders could be considered as to be active users (41%).

The survey was sent to all referrers and specialists. Of the 548 people who were sent the survey, 163 (30%) responded: the group details are summarized in Table 1. The survey was completed reasonably promptly. Non-linear regression showed that during the first week, the numbers of responses received doubled every 1.8 days (R² = 0.94). Seventy percent of the questionnaires were completed within 6 days (Figure 1).

TABLE 1

Table 1. Responses from referrers and specialists.

FIGURE 1

Figure 1. Cumulative number of responses to the survey (n = 163).

By matching answers to some questions (such as whether they found the system easy to use and how long it took to write a referral or answer a case), the survey showed that approximately 40 of the 64 referrers who responded (62%) and 70 of 99 specialists who responded (70%) could be considered as active users. Although there were more referrers (54%) than specialists (46%) in the population sampled, specialists provided significantly more answers than referrers – 39 versus 22%. This was consistent when considering the individual item response rates.

Differences Between Main Categories

There were no significant differences between the responses from French and English users (Table 2), so the responses were combined for subsequent analysis. The survey questions were divided into five main categories (see Table 3):

TABLE 2

Table 2. Differences between the responses from those completing the French and English versions of the questionnaire.

TABLE 3

Table 3. Survey results.

General User Profile (Personal, Professional)

The average age of the specialists was significantly higher than that of the referrers (P < 0.01). The responders were multinational, predominantly European. Most responders were medical doctors, and the majority had previously completed one or more MSF field missions (including the specialists). Further details are provided in Table 3A.

IT Profile (Telemedicine Experience, Habits)

Most responders had access to email, usually via a laptop computer or a mobile phone. Three-quarters of the responders had used social media, such as Facebook and Twitter, for general communications, with a small difference (P = 0.04) between specialists (67%) and referrers (83%). Further details are provided in Table 3B.

Use of Telemedicine (Interface/Platform and Interaction)

Only about half of the responders (44% of the referrers and 55% of the specialists) had received a prior briefing about telemedicine. However, the majority (more than three-quarters) stated that the system was user friendly, and they found it self-explanatory (i.e., they did not need to be shown how to use it). The average time reported by referrers for writing a case was 14.7 min, and the average time reported by specialists for replying was 17.4 min.

Half of the referrers (51%) reported having connection problems, either sometimes or always, compared to 24% of specialist; this difference was significant (P < 0.01). Less than half of the referrers (45%) stated that they had provided feedback about the patient for the specialist; however, 92% of the specialists reported that they had not received any feedback about the patient. Further details are provided in Table 3C.

Outcomes (Satisfaction, Usefulness)

The majority of referrers (91%) stated that they found the advice received via telemedicine to be useful. The most common reasons for this were in providing reassurance for the referrer or patient, in changing the management of the patient and in changing the diagnosis. The majority of referrers (79%) felt that advice from the specialist improved their management of the patient. The referrers’ mean score for their satisfaction with telemedicine was 7.6 (on a scale from 1 = very unsatisfied to 10 = very satisfied).

The majority of specialists who provided an opinion (94%) thought that their advice had improved the management of the patient. The majority of specialists (99%) felt that there was educational benefit for the field doctor. The specialists’ mean score for their satisfaction with telemedicine was 6.6 (on a scale from 1 = very unsatisfied to 10 = very satisfied). Further details are provided in Table 3D.

Other (Referrals, Opinions)

Both referrers and specialists indicated the time in which it was desirable to receive (or provide) the response, by choosing from a series of time intervals. Using the midpoint of these time intervals to summarize the frequency distributions showed that the referrers felt that it was desirable to receive the specialist’s response within 9 h, and that the specialists thought that it was desirable to provide a response within 14 h, a slightly longer delay. Similarly, almost all responders felt that receiving feedback (specialists) or providing feedback (referrers) was necessary or obligatory; only 14% of referrers and 1% of specialists felt that it was optional. Further details are provided in Table 3E.

Free-Text Comments Analysis

The comments made by referrers and specialists in response to the open-ended questions are summarized in Tables 4 and 5, respectively. Both referrers and specialists used adjectives such as “excellent” and phrases expressing their gratitude that the system had been set up. It is worth noting that these comments emphasized the users’ involvement in the pilot and reflected their desire to improve the system where appropriate.

TABLE 4

Table 4. Comments from referrers in response to open-ended questions (numbers 42 and 49).

TABLE 5

Table 5. Comments from specialists in response to open-ended questions (numbers 39 and 50).

The main comments from referrers focused on the lack of support, promotion, and predeparture briefing at headquarters’ level, whereas the principal concern expressed by the specialists was the lack of feedback about patient follow-up. There were some negative comments made by referrers concerning the characteristics of the expert advice received (delayed, not field specific, etc.) and the platform itself (not enough interactive communication). Such remarks may explain the reluctance of some potential users to use the system (e.g., fear of not being able to implement the expert advice received).

Anticipating a possible loss of motivation, specialists expressed their frustration in not receiving cases, which can also be seen as another consequence of the lack of promotion and the low use of the system. However, the specialists provided constructive comments about the system. One pointed out the long-term benefit of such a network, which can continue at a health care facility even after MSF support in the field has ended.

Discussion

We conducted a comprehensive survey of the users of the MSF tele-expertise service during its first four years of operation. Our previous paper (4) summarized the main results, and the present paper provides a detailed description and analysis. There was high satisfaction with the service, among both referrers and specialists. All users recognized the benefits of providing access to specialist advice in low-resource settings where there is usually no alternative way of obtaining specialist expertise.

Although there was general satisfaction with the service, the survey identified some specific problems. For example, some referrers reported poor connectivity in the field, which affected their ability to communicate by email and to upload files via the telemedicine network. This problem is amenable to technical solution. However, the main concerns – raised by both referrers and specialists – were the lack of promotion of the system at headquarters’ level, and the lack of feedback about patient follow-up. These problems are less easy to address.

Another issue uncovered during our field participant observations was the use of uncontrolled social media for communicating confidential information in remote health care. The survey confirmed that most referrers and specialists used Facebook and/or Twitter for general communications, and personal communications from certain users confirmed that some MSF staff also used these media for obtaining second opinions about medical cases. For example, a Facebook group was created to provide access to a videolink and obtain feedback from expert friends. In contrast, the MSF tele-expertise service represents a safe and secure method of facilitating such consultations at a distance, and this aspect has been an important factor in convincing some reluctant headquarters’ stakeholders to adopt the system.

Limitations of the Study

The study had certain limitations. For example, some of the information solicited in the survey was based on the opinions of the responders, which may itself introduce bias. Despite the methodological weakness of seeking personal opinions, it is important to conduct such surveys in order to identify areas for service improvement.

About two-thirds of the responses were from active users. The results therefore provide a view from users with practical experience of the tele-expertise service. This was supplemented by the views of the inactive users, i.e., people who were registered account holders but at the time of the survey were still awaiting an opportunity to submit a case or provide an answer. Unfortunately, the detailed characteristics of the 385 non-responders are unknown, which represents a potential source of bias. This study limitation was a consequence of the strict anonymization, which made it impossible to link the answers in the survey to individual user profiles in the system database. This in turn prevented the follow-up of the non-responders, so that we were unable to hypothesize about their reasons for non-response. One way to improve the design of a future study would be send different surveys to users who had previously been identified as active or inactive, and perhaps to differentiate the latter into those who had never logged in and those who had yet to submit or answer a case.

Another weakness of the study was the low response rate (30%). In principle, this might have been improved by extending the period for which the survey was open, but the trajectory of the responses received (Figure 1) suggests that 99% of responses were received within 2 weeks, so that the cutoff at 30 days was, if anything, conservative. Nonetheless the response rate was lower than those observed in previous surveys of doctors in industrialized countries, which have been up to twice as high. On the other hand, a survey of the users of a telemedicine network in a low-resource setting had a response rate of 19% (7).

User Feedback

It is clear that the MSF tele-expertise service was easy to use and provided clinically useful diagnostic and management advice to clinicians in the field. Most referrers reported that the advice received via the service improved their management of the patient (30/33). Similarly, the majority of specialists who provided an answer felt that their advice had improved the management of the patient (30/32). However, a similar number of specialists did not know about the value of their advice – presumably these were specialists who had not yet answered a case. The lack of feedback about patient follow-up was the main concern expressed by the specialist. This feedback can be considered as an implied contract and is crucial in keeping our experts motivated. Well-motivated experts answer promptly, and the feedback allows them to improve and fine-tune their advice. For this purpose, the system now sends progress reports automatically (from the referrer to the specialist) that includes follow-up data.

The specialists were asked “Was the information (including any images) supplied by the referrer adequate?” Of those who responded, 33% said no. Poor quality information submitted in a referral is a common problem in store-and-forward telemedicine networks generally, and in particular it is very difficult to ensure that good quality images are supplied (11, 12). Methods for improving the quality of the information in referrals include user education and structured referral templates.

Specialist Workforce: Cornerstone of the System

Comments made by the volunteer specialists suggested that they were highly motivated, demonstrating a high level of user participation, despite the questionnaire being rather long (50 questions). Their comments suggest that being able to show solidarity with their colleagues in the field, together with the positive image of MSF worldwide, were important factors in recruiting and keeping them motivated. Indeed, by virtue of their commitment, they played a key role in the development and the implementation of the system. All the experts who were asked to join the MSF network accepted the invitation to do so, and after four years only one specialist decided to quit because of pressure of other work. It is clear that their support and the quality of their service were vital for the system development. Their answers and free-text comments demonstrate that they see the service as more than just a way of providing a second opinion. They are fully aware that they are also providing education, support, reassurance, and reducing the isolation of the field staff.

However, in their survey responses, some of them expressed frustration about not receiving enough cases. Part of the art in managing a telemedicine network is to maintain a large enough pool of volunteer specialists to be able to answer the range of queries occurring on the network, but to keep individual specialist workloads at a reasonable level. Too many cases are likely to lead to “consultant fatigue,” whereas too few cases may lead to specialists abandoning the network. It is worth noting that the majority of specialists considered that their status (i.e., acting as unpaid volunteers) was appropriate. In terms of desirable workload, three-quarters of them mentioned that they could routinely answer several cases per week.

Barriers to Service Adoption

Both the analysis of the free comments and the large number of inactive users among the account holders emphasize that the successful implementation of telemedicine is fraught with difficulties, including technical, cultural, legal, financial, organizational, and political barriers (13). It is clear from the literature on technology adoption that one reason for the reluctance to adopt telemedicine is that it is sometimes perceived as a threat (14) with the risk of managers losing control. In contrast, it should more properly be viewed as a complementary tool, and people should be invited to take full advantage of it, even if this necessitates changes in working practices.

The high proportion of inactive users among those registered to use the service (59%) could have several explanations, such as the rapid turnover of field staff or specialists still waiting to receive a case. In our experience, it is not unusual in store-and-forward telemedicine networks to have a substantial proportion of inactive users.

Studies of technology adoption show that there are three main reasons for under-use (15). The results of the survey suggest that service use may have been reduced for two of these reasons:

1. Lack of knowledge about the existence of the service. This seems plausible because half of potential users were not briefed before going to the field, and many users mentioned poor communication or lack of promotion of the system.

2. Lack of interest in the service by potential users. This also seems plausible because the field doctors often have very high workloads, so that access to specialized advice for non-critical cases may be considered as a luxury. In addition, the turnover of staff is high: field missions commonly last only a few months.

3. Non-delivery when the service is requested. This seems unlikely because published performance indicators show that in fact the service is delivered rapidly and at high quality (4).

Comparison with Other Networks

A recent review identified seven store-and-forward telemedicine networks which were reasonably well-established (i.e., had been in operation for periods of five years or more) and which delivered teleconsultations to health care staff in low-resource settings (16, 17). Some of these networks have conducted user surveys, the results of which can be used as a comparator for the present study. In particular, previous surveys have asked referrers to identify the benefits of telemedicine (2, 6, 7). The results are summarized in Table 6. It is clear that referrers find these telemedicine services helpful in managing patients and in other ways, e.g., providing reassurance for patient and doctor. Nonetheless, it must be recognized that this represents low-level evidence and formal studies of the clinical and cost-effectiveness of telemedicine in low-resource settings are still awaited.

TABLE 6

Table 6. Benefits identified by referrers.

Conclusion

The MSF tele-expertise service is highly regarded by field users, and the majority of specialists are satisfied with it. All users recognize the benefits of providing access to specialist advice in low-resource settings, where there is usually no other way of obtaining specialist expertise.

Both referrers and specialists recognize the benefits of telemedicine in terms of better patient management, the provision of education, and the reduction of isolation in the field. Although access to specialist advice might not be considered as a priority in low-resource settings, it is clear from the user feedback that the MSF tele-expertise service provides an answer to a real need. Much of the success of the service rests on the efforts of the volunteer specialists, who not only provide the benefits of their special expertise, but demonstrate solidarity toward their colleagues in the field in supporting their education, reducing their sense of isolation, and improving the management of their patients.

Although there was general satisfaction with the service, the survey identified some specific problems. Some of these could be solved relatively easily, such as poor connectivity in certain locations, while others, such as the lack of feedback on patients, are more difficult. Perhaps the main challenge for building a sustainable service lies in the political dimension, since improving the adoption of the system requires a strong organizational commitment. The evidence base for the tele-expertise service now seems to be irrefutable, so it is to be hoped that further diffusion will occur more rapidly.

Conflict of Interest Statement

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Acknowledgments

We thank all the referrers and specialists who took part of the study. We are particularly grateful for support from Isabel Amoros, Eric Comte, Sophie Delaigue, Bertrand Draguez, Kathrin Gottwald, Jarred Halton, Annette Heinzelmann, Jean-Paul Jemmy, Daniel Martinez Garcia, David Olson, Pedro Pablo Pablo, Jean Rigal, Micaela Serafini, Leslie Shanks, Saskia Spijker, Raghu Venugopal, Sidney Wong, and Will Wu.

References

1. Code de la Santé Publique – Article R6316-1. Code de la Santé Publique. (2015). Available from: http://www.legifrance.gouv.fr

Google Scholar

2. Patterson V, Wootton R. A web-based telemedicine system for low-resource settings 13 years on: insights from referrers and specialists. Glob Health Action (2013) 6:21465. doi: 10.3402/gha.v6i0.21465

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Wootton R, Wu W-I, Bonnardot L. Nucleating the development of telemedicine to support healthcare workers in resource-limited settings: a new approach. J Telemed Telecare (2013) 19(7):411–7. doi:10.1177/1357633X13506511

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Bonnardot L, Liu J, Wootton E, Amoros I, Olson D, Wong S, et al. The development of a multilingual tool for facilitating the primary-specialty care interface in low resource settings: the MSF tele-expertise system. Front Public Health (2014) 2:126. doi:10.3389/fpubh.2014.00126

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Mann K, Gordon J, MacLeod A. Reflection and reflective practice in health professions education: a systematic review. Adv Health Sci Educ (2007) 14(4):595–621. doi:10.1007/s10459-007-9090-2

CrossRef Full Text | Google Scholar

6. Wootton R, Youngberry K, Swinfen P, Swinfen R. Prospective case review of a global e-health system for doctors in developing countries. J Telemed Telecare (2004) 10(Suppl 1):94–6. doi:10.1258/1357633042614177

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Zolfo M, Bateganya MH, Adetifa IM, Colebunders R, Lynen L. A telemedicine service for HIV/AIDS physicians working in developing countries. J Telemed Telecare (2011) 17(2):65–70. doi:10.1258/jtt.2010.100308

CrossRef Full Text | Google Scholar

8. MSF Ethics Review Board. MSF Research Ethics Framework – Guidance Document. (2015). Available from: http://fieldresearch.msf.org/msf/handle/10144/305288

Google Scholar

9. Soper DS. Fisher’s Exact Test Calculator for a 2x3 Contingency Table [Software]. (2015). Available from: http://www.danielsoper.com/statcalc

Google Scholar

10. Freeman GH, Halton JH. Note on an exact treatment of contingency, goodness of fit and other problems of significance. Biometrika (1951) 38:141–9. doi:10.2307/2332323

CrossRef Full Text | Google Scholar

11. Jakowenko J, Wootton R. An analysis of the images attached to referral messages in an email-based telemedicine system for developing countries. J Telemed Telecare (2006) 12(Suppl 3):S3:49–53. doi:10.1258/135763306779380066

CrossRef Full Text | Google Scholar

12. Delaigue S, Morand JJ, Olson D, Wootton R, Bonnardot L. Teledermatology in low-resource settings: the MSF experience with a multilingual tele-expertise platform. Front Public Health (2014) 2:233. doi:10.3389/fpubh.2014.00233

PubMed Abstract | CrossRef Full Text | Google Scholar

13. LeRouge C, Garfield MJ. Crossing the telemedicine chasm: have the U.S. barriers to widespread adoption of telemedicine been significantly reduced? Int J Environ Res Public Health (2013) 10(12):6472–84. doi:10.3390/ijerph10126472

PubMed Abstract | CrossRef Full Text | Google Scholar

14. World Health Organization. Telemedicine: Opportunities and Developments in Member States: Report on the Second Global Survey on eHealth 2009. World Health Organization (2011). Available from: http://www.who.int/goe/publications/ehealth_series_vol2/en/

Google Scholar

15. Bradol J-H, Vidal C, editors. Medical Innovations in Humanitarian Situations. (2015). Available from: http://www.doctorswithoutborders.org/sites/usa/files/msf-medical-innvoations-book.pdf

Google Scholar

16. Wootton R, Geissbuhler A, Jethwani K, Kovarik C, Person DA, Vladzymyrskyy A, et al. Comparative performance of seven long-running telemedicine networks delivering humanitarian services. J Telemed Telecare (2012) 18(6):305–11. doi:10.1258/jtt.2012.120315

PubMed Abstract | CrossRef Full Text | Google Scholar

17. Wootton R, Geissbuhler A, Jethwani K, Kovarik C, Person DA, Vladzymyrskyy A, et al. Long-running telemedicine networks delivering humanitarian services: experience, performance and scientific output. Bull World Health Organ (2012) 90(5):341D–7D. doi:10.2471/BLT.11.099143

PubMed Abstract | CrossRef Full Text | Google Scholar