Ísis de Siqueira Silva1*†
Cícera Renata Diniz Vieira Silva2†
Claudia Santos Martiniano3
Aguinaldo José de Araújo1
Renan Cabral de Figueirêdo4Luís Velez Lapão5,6
Renan Cipriano Moioli7Ewerton William Gomes Brito4
Severina Alice da Costa Uchôa8- 1Department of Public Health, Postgraduate Program in Collective Health, Federal University of Rio Grande do Norte, Natal, Brazil
- 2Federal University of Campina Grande, Campina Grande, Brazil
- 3Department of Nursing, State University of Paraíba, Campina Grande, Brazil
- 4Department of Public Health, Federal University of Rio Grande do Norte, Natal, Brazil
- 5IDeas Laboratory, UNIDEMI, Nova School of Science and Technology, Caparica, Portugal
- 6LASI, Laboratório Associado de Sistemas Inteligentes, WHO Collaborating Centre for Health Workforce Policy and Planing, Nova Instituto de Higiene e Medicina Tropical, Lisbon, Portugal
- 7Bioinformatics Multidisciplinary Environment, Digital Metropolis Institute, Federal University of Rio Grande do Norte, Natal, Brazil
- 8Department of Public Health, Postgraduate Program in Collective Health, Postgraduate Program in Family Health, Postgraduate Program in Health Sciences, Federal University of Rio Grande do Norte, Natal, Brazil
Introduction: The implementation of technologies in Primary Health Care with monitoring and evaluation of the quality of health care is fundamental to direct the access and quality of health care in the context of the Sustainable Development Goals. Our objective was to develop a model for evaluating digital health interventions in primary care, considering its impact on first contact, longitudinality, integrality and coordination in health.
Methodology: This is an exploratory methodological study of a qualitative nature. This study seeks to explore strategic actors’ perceptions of an evaluation model, and was developed in a process between June 2021 and March 2024. The following stages were followed: Identification of previous models available in the literature, model development, model validation, model update. We performed a nominal group consensus technique online with seven experts. Stages taken to define the design of the model: sending the elaborated model, together with an electronic form with 18 subjective questions, such as brainstorming strategy, for recording impressions, judgment of agreement and suggestions; workshop for discussion by videoconference, at the time the objectives and the model diagram were presented, followed by debate with clarification of doubts and suggestions for clarification of the step-by-step design. After incorporating the suggestions, the model agreed upon in the workshop was subjected to another analysis by the same experts, sent in an online Google document, in which possible inconsistencies between the proposed model and the agreed one were verified, as well as the final agreement on the recommended proposal. At the end of this stage, with synchronous and asynchronous remote procedures, consensus was reached.
Results: The proposed evaluation model presents as primary categories, structure, process and result. The structure encompasses four indicators, namely, employees; financial resources; infrastructure resources; and regulatory and strategic resources. The process is organized into three dimensions, namely: technical dimension, organizational dimension and relational dimension. The results will be evaluated in products; intermediate results; and impacts (short, medium and long term). The results will be measured by the seven pillars of quality: efficacy, effectiveness, efficiency, optimization, acceptability, legitimacy, equity.
Conclusion: This study achieved the goal of developing a model to evaluate digital health interventions in Primary Health Care, helping to identify adequate and useful evaluation methods filling the gap of the lack of quality evaluation standards in the brazilian Digital Health Strategy. It presents an important difference in relation to models from different countries, as it considers the impacts on Primary Health Care quality attributes such as first contact care, longitudinality, integrality and coordination of care. The model will be used by managers and health professionals of Primary Health Care in a case study in Brazil to evaluate the quality of health care. It is expected that the proposed model may be used in other evaluation studies and countries through contextual adaptations.
1 Introduction
The advancement of digital technologies has motivated the accelerated development of information systems, applications, decision support systems based on artificial intelligence, machine learning, and internet of things, among others (1, 2). In order to improve health care, the use of digital technologies in this sector has expanded, such as wearable devices, computerized decision support systems, and telehealth, which increase the technical performance and satisfaction of health professionals, showing the potential to reduce the direct and indirect costs of health services and improve the access and quality of care provided (3).
The implementation of digital interventions in health has fostered the formulation of the concept of digital health, which is evolving and is characterized as a broad field that encompasses various Information and Communication Technologies (ICTs) applied to healthcare. For instance, artificial intelligence is being used to improve diagnostics and personalized treatments; data analytics tools are assisting in predicting health trends and managing patient data; blockchain ensures secure sharing of medical records; health information systems enable the efficient management of patient information across different levels of care; the internet of things, with integrated sensors allows for real-time monitoring of patients’ vital signs; teleconsultations facilitate remote medical consultations, and telemonitoring helps in managing chronic conditions from a distance (4–6).
Within the scope of Primary Health Care (PHC), defined as the first point of contact of users with the health care network, which offers comprehensive and accessible care (7), technologies can strengthen the communication and continuity of primary care (8, 9). Digital health infrastructure and its integration into PHC services vary between countries, influenced by the economic situation, health priority and technological advances in the region (10). In line with health, the so-called digital transformation process can create opportunities to expand services in the health sector, provided that there is adequate infrastructure and the necessary training, both for professionals and patients (11).
Digital health offers numerous benefits for improving the quality of life and well-being of populations. Firstly, it facilitates access to health information and medical services, particularly in remote regions, through mobile technologies and digital platforms. Additionally, it enables the management and monitoring of chronic conditions via applications and tracking devices, allowing for continuous and personalized patient follow-up. Furthermore, digital health contributes to the efficiency of health systems by reducing operational costs and optimizing resource allocation. Digital interventions also promote patient engagement, encouraging healthy behaviors and increasing adherence to prescribed treatments (4).
The use of digital health presents, in addition to benefits, significant challenges for the quality of care. This phenomenon is exacerbated by the existence of a technological gap that separates countries with advanced digital ecosystems and technological support conducive to the implementation of ICTs in health, from those that have not yet reached sufficient maturity (12). In less technologically developed countries, the lack of adequate digital infrastructure, such as reliable internet networks and technological equipment, and insufficient technical support, such as the absence of professionals trained in, digital technologies hinder both access to and the quality of digital care at the first point of contact most people have with the health system, as well as the integration and sharing of health information between different primary care units and other levels of the health system, such as hospitals and specialized clinics, which are the central roles of PHC. Considering these aspects, in many cases, the challenges focus on the coordination and continuity of Health Promotion Care, which is often limited and fragmented (10, 12).
The implementation of digital technologies in PHC with monitoring and evaluation is essential to direct the quality of health care in the context of the Sustainable Development Goals. The evaluation process is considered as the exercise of measuring, understanding and judging the effects of an intervention, in order to support decision-making, as well as allowing the process of continuous improvement (13, 14). In this context, political and academic agents can act, aiming to guide the improvement of health practices, toward a more favorable condition (15).
Beginning in 2013, Brazil implemented the Health Information System for Primary Care, initiating the digitalization process of PHC (16). The lack of standardization in health data collection and processing procedures spurred the need for a National Health Information and Informatics Policy to guide ICTs actions across the entire Brazilian healthcare system (17). Advancing the implementation of ICTs in healthcare, the digital health strategy was approved in 2017 (18), based on the National eHealth Strategy Toolkit published by World Health Organization (WHO) in 2012 (19). To align the principles and guidelines of the Unified Health System (SUS) with the e-government policy, the Ministry of Health has launched programs to incentivize digitalization, including the Conecte SUS program (20), which aims to integrate citizens health information into an app, and the Program to Support the Informatization and Qualification of Primary Health Care Data (21), which aims to financially support the digitalization of PHC.
With advancements in healthcare technologies, the National Health Data Network (22) was created to enable interoperability and integrate data generated and stored across different information systems. The Health Strategy for Brazil 2028 (23) aims to systematize and consolidate the work carried out over the past decade, as outlined in several key documents, particularly the National Health Information and Informatics Policy (17), published in 2015 and revised in 2020, and the e-Health Strategy for Brazil (18). Corroborating the expansion of digital health, in 2023, the Secretariat of Information and Digital Health was established to support the Ministry of Health in formulating policies and strategies for the implementation of digital health (24).
Brazil has expanded its digital health portfolio by investing in data security (25), information systems, promoting interoperability, and incorporating technologies (22, 26). In this context, the SUS Digital Program and the National Digital Health Maturity Index, based on quantitative indicators of digital maturity, were established (26). The strengthening of digital health programs and regulatory frameworks encourages states and municipalities to implement digital strategies within the scope of SUS. Therefore, alongside the implementation of ICTs, promoting the evaluation of digital health is essential, as it is expected to help identify barriers and facilitators, supporting its growth and development. An adequate evaluation requires a model that thoroughly comprehends the object of evaluation. Developing this model involves planning data collection, verifying available resources, designing activities, and specifying changes and outcomes (27, 28).
The expansion of the application of ICTs in health transcends geographical borders, and different countries have been concerned with evaluating the implementation of digital health services. Some frameworks models are already available, such as the Global Digital Health Index (29), The Global Digital Health Monitor (30), and the Healthcare Information and Management Systems Society (31). These models present relevant criteria for the evaluation of digital health, such as governance, infrastructure, development of computerization systems, training for the use of digital health, privacy and data security, telehealth, remote care and user satisfaction, among others. However, these models do not demonstrate the articulation between resources, work processes through digital means and their influence on results. Even considering the results of technologies, they do not assess their impact on the strengthening of health systems in general and in particular on the essential attributes of PHC, the focus of this study.
Although the frameworks mentioned have the potential to be applied in different scenarios, in the case of Brazil, which has the largest universal public health system in the world (32), there is a need for a scientifically validated evaluation model that, at the same time, contemplates domains of technology evaluation and the impact on PHC as a specific basis of SUS in the context of the Brazilian digital health strategy. That said, this study aims to develop a model for evaluating digital health interventions in primary care, considering their impact on first contact, longitudinality, integrality and coordination in health. In this perspective, it will be useful for managers, health professionals and patients to assess the quality of care mediated by digital health services. The results of this study can be applied in countries with a universal health model, which adopt primary health care, and especially for low- and middle-income countries, which are expanding the use of digital health applications.
2 Methodology
This is an exploratory methodological study that involves the development and validation of tools and methods (33). The present study was qualitative in nature, and its stages are described in Figure 1. This study seeks to explore perceptions of strategic actors (experts) about the model developed between June 2021 and March 2024.
Figure 1. Study steps.
2.1 Identification of previous models available in the literature
A scoping review was developed to identify frameworks used to evaluate digital health worldwide during the pandemic. Furthermore, it was the basis for the Brazilian evaluation model. The frameworks are summarized in Table 1.
Table 1. Frameworks to evaluate digital health identified in the scoping review (9).
2.2 Model development
The researchers developed the first version of the model to evaluate Brazilian digital health strategies between September 2021 and January 2022. The integration of Donabedian’s theoretical model (34) and the W. K. Kellogg Foundation’s logic model (35) was adopted to evaluate the quality of care through digital means in a complementary and comprehensive manner. Donabedian’s tripartite model, comprising structure, process, and outcomes, offers simplicity and flexibility, making it widely applicable in various scenarios of health care quality evaluation. Meanwhile, the Kellogg Foundation’s logic model provides a versatile and organized framework, useful for project planning and implementation by incorporating components such as resources, activities, outputs, outcomes, and impacts. The combination of these two models allows for a more detailed analysis, with Donabedian’s model offering a classic and solid foundation for assessing the quality of digital health strategies, while the Kellogg logic model adds greater analytical depth, covering not only immediate outcomes but also long-term impacts. This integration was adapted by the authors and is represented in Figure 2, connecting the technical, organizational, and relational dimensions of digital tools, which are crucial elements to be assessed and improved in PHC.
Figure 2. Integration of Donabedian’s model with the Kellogg Foundation’s logic model.
2.3 Model validation
The validation step was a formative moment with broad participation and interaction among members, contributing to the exchange of information and the maturing of opinions. These opinions were based on criticism and systematized ideas, facilitating decision-making with group consensus (36). We performed a nominal group consensus technique (37, 38) online (39) with seven experts, from February to March 2022. The experts were defined based on scientific publications in the areas of health evaluation, digital health, experience in the use of information systems and/or ICTs in primary care, or the development of digital health technologies.
The methodology followed a structured process, comprising four key steps: group preparation, brainstorming, nominal group discussion, and post-group refinement.
Step 1–Group preparation: Experts were selected based on their scientific publications in health evaluation, digital health, large experience in information systems, other applications ICTs in primary care, or the development of digital health technologies. Before the nominal group session, participants received the evaluation model (Appendix 1) via email, along with a link to a Google Forms® questionnaire containing guiding questions for the next step, the brainstorming. They could use a pseudonym to ensure confidentiality. This remote format replaced the traditional in-person silent idea generation phase, offering participants more time for reflection.
Step 2–Brainstormin: Participants had 10 days to submit their ideas using the electronic form, focusing on identifying gaps in the proposed evaluation model (Appendix 2). All responses were organized in a Microsoft Excel® spreadsheet and shared with the research team for a preliminary analysis, setting the stage for the upcoming nominal group discussion.
Step 3–Nominal group discussion: The nominal group discussion was conducted via videoconference using Google Meet®. This method promoted social interaction, democratic discussion, and time efficiency (40). One researcher facilitated the session, ensuring equal time for all participants to present their evaluations on the model components. The themes that emerged during the nominal group are organized in Appendix 3. The experts then clarified and discussed the relative importance of each idea, with a second researcher recording the key points and organizing them into thematic categories.
Step 4–Post-group refinement: Following the discussion, the research team incorporated the experts’ feedback into the evaluation model. The revised model was shared with the experts for final verification, ensuring alignment with the group’s suggestions. This iterative process, combining both synchronous and asynchronous inputs, led to a consensus of the model (Appendix 4).
2.4 Model update
An additional consultation was necessary due to the end of the global COVID-19 emergency, which directly impacted the practice of care and health and the use of digital health. Between January and March 2024, stakeholders with expertise in primary health care, digital health management and governance were invited to review the design of the digital health model proposed in this research, in order to identify inaccuracies and point out needs for content improvement. The stakeholders were intentionally defined according to their work in health evaluation, digital health, experience in the use of information systems and ICTs in primary care or in the development of digital health technologies, and the invitation was sent by e-mail. Six invitations were sent, among which five accepted to answer the questions, and one accepted, but did not answer the form. The five stakeholders submitted their analyses through an online form that contained the following guiding questions: “What do you consider to be the weaknesses of this model?”; “What do you consider to be the potential of this design?”; “What do you think can be modified?”; “What dimensions of governance need changes?” (Appendix 5).
The responses received were analyzed through thematic analysis and the relevant updates were incorporated into the final structure of the evaluation model.
2.5 Ethics and dissemination of the results
The present study was approved by the research ethics committee of the Onofre Lopes University Hospital of the Federal University of Rio Grande do Norte (CAAE 48655521.9.0000.5292). We followed the recommendations of Resolutions 466/12 (41) and 510/16 (42) of the National Health Council and the guidelines proposed by the General Law of Data and Personal Protection number 13.709/2018 (25). All experts signed by e-mail the informed consent form and authorized voice and image recording.
3 Results
Before presenting the evaluation model, it is necessary to understand the political context of digital health in Brazil, a summary of the advances in digital health is available in Box 1. The country has stood out as a reference, within Latin America, in the implementation of ICTs in health. During a G20 meeting, a WHO representative highlighted the importance of countries being the protagonists of their own digital transformation and reinforced the Organization’s support, including through Global Digital Health Index, to sustainably improve the health of their populations (43).
BOX 1. Political-institutional context of digital health in Brazil.
To propose a model to evaluate digital health in the Brazilian PHC, proper understanding and articulation of the model and previous knowledge about it were needed. This knowledge was regarding context, history of digital health, delimitation of the evaluative research, and qualifying indicators of PHC. The involvement of experts in validation has increased the accuracy of the product developed.
During the nominal group discussions, the judges raised potential topics at the intersection of health and technology that required refinement in the initial model (Appendix 3). Among the categories that emerged from the experts’ statements, factors influencing the implementation of digital health were highlighted, such as robust infrastructure and investment in data technologies. Additionally, the issue of healthcare professionals’ training was debated, bringing to light digital health literacy and user engagement. Digital health has the potential to increase the efficiency of PHC and improve universal coverage, but it faces challenges such as data interoperability and the need for effective governance. Considering the political-institutional context and ensuring data ethics and security are also essential aspects for successful implementation. The arguments presented by the judges with expertise in the field were crucial for the model’s development.
The model to evaluate digital health in PHC (Figure 3) comprises different aspects (collaborators, financial, infrastructure, and normative and strategic resources) to allow the execution of activities across the technical (scope of actions and accuracy), organizational (organization of care and management), and relational (interpersonal) dimensions. Given the actions proposed for the implementation strategy, results will be short- (products), medium- (intermediate results), and long-term (impacts).
Figure 3. Model to evaluate digital health in Primary Health Care.
As presented in Figure 3, the digital health evaluation model in primary health care incorporated elements from Donabedian’s model (34) and the Kellogg Foundation (35).
The structure component incorporates resource elements encompasses four indicators: employees, characterized by family health teams, managers and also by Information Technology professionals; financial resources, characterized by the financing of digital health in PHC; infrastructure resources, characterized by the adequacy of the physical space to perform health care mediated by digital health, computerization (availability of equipment, information systems, internet, applications, electronic devices); and normative and strategic resources, characterized by normative regulation of digital health, protocols, guidelines, data storage, and partnership networks.
The process component includes activity elements is organized into three dimensions, namely:
The technical dimension that includes the actions of prevention, promotion, protection, diagnosis, treatment, rehabilitation, harm reduction and palliative care, developed in the sphere of PHC; qualification of professionals and users to consciously use digital health, promoting digital health literacy for health professionals, caregivers and patients, health literacy for professionals develop health technologies, technical support for technologies in PHC, and data security.
The organizational dimension, which includes the sphere of care (access, comprehensiveness, longitudinality, coordination, ordering and surveillance), governance (governance of systems, technological solutions for PHC demands; encouragement of innovations in managerial, care and educational technologies, with a view to adapting work processes; monitoring the use of resources for digital health; strategic partnerships).
The relational dimension is focus on the interaction of health professionals with their patients, and the participation of both in the process of choosing the use of technologies, empowerment of users in health decisions, and strengthening of bonds should be evaluated.
Quality care must maximize the well-being of patients, according to Donabedian (34, 51) the results verify the effects produced in health care, positive results should improve the quality of health care. And the quality applied to health care can be measured by the 7 pillars of quality: efficacy (adequate resources), effectiveness (ensuring that digital care achieves the same results in everyday practice); efficiency (good use of available resources); optimization (improvement of the cost–benefit ratio between the amount spent and the result achieved); acceptability (adaptation and acceptance of technologies by users and health professionals); legitimacy of the ethical and legal compliance of digital care; and equity (equal access and quality in digital health services for all people, regardless of their individual characteristics).
The results component is organized into: Products (work process mediated by technologies, implementation of ICTs, training for professionals and managers, governance system and involvement of participants); Intermediate results (digital inclusion, trust in technologies, and the satisfaction of stakeholders with practices that comply with regulatory and ethical standards, digital health literacy, clear effects of the benefits of the use of ICTs in PHC, improvements in health access, expansion of computerization, interoperability, investment and efficiency of resources for digital PHC, and strengthening of communication between users and health professionals at different levels of care networks); and Impacts, according to the Kellogg Foundation (35), can be broader and long-term, addressing the effects of actions on the population’s health. The model presented in this study includes the following impact indicators: potential and sustainable use of managerial, care, and educational digital technologies used in PHC; institutionalization of evaluation for the continuous improvement of digital health quality in PHC; and strengthening and enhancing the effectiveness of PHC. Another important impact of digital health use in PHC is the evaluation of patient satisfaction, which will be assessed with indicators that consider: patient participation in the choice and preference of digital strategies used in health care, satisfaction with treatment and outcomes (respect for patient dignity and privacy; patient empowerment and participation); quality of communication; access to health services (availability of consultations and adequate resources to meet patient needs, geographic proximity of services, accessibility); and patient experience in digitally-mediated care, including health professionals’ empathy, waiting time, ease of communication, and responsiveness to patient concerns.
4 Discussion
4.1 Principal findings
This study presents an evaluation model of primary health care mediated by digital strategies; its application will be useful for patients, health professionals, managers and technology developers to know the potentialities and limitations when using ICTs in health care, and how these tools interfere with the quality of primary care. The organization of a specific model for PHC will help identify patterns and predict trends, allowing more informed and personalized decision-making, providing insights for the implementation of digital health in health care with a focus on primary care, providing continuous quality improvement.
Encouraging the adoption of digital health has contributed to the diversification of digital strategies in health and computerization of health services (52). The inclusion of technologies is modifying the way of providing health care, generating and storing data (11). Thus, the impact on the quality of health care, from the perspective of the digital age, has aroused the interest of researchers in the area (53–55). An investigation focused on the analysis of digital maturity concluded that greater maturity was associated with the maintenance of the patient’s health record, the screening of patient experience data, the monitoring of the patient’s journey and the mitigation of clinical risk (56).
The evaluation model proposed in this study has a theoretical and methodological foundation aiming at advancing and institutionalizing evaluation processes within the Brazilian PHC. This methodological development of the model was qualified using frameworks similar to worldwide evaluations and validated using a collaborative and participatory strategy. The model was based on a formative perspective, intending to provide information to adjust and overcome issues indicated in the evaluation. Thus, the social contribution of this study relies on this model promoting participatory and comprehensive evaluations centered on the stakeholders and engaging the responders (57).
The various evaluation frameworks mapped out exhibit both similarities and significant differences compared to the evaluation model proposed in this study, which focuses on the quality of services based on two recognized theoretical models (34, 35). Our model offers a holistic and structured assessment of the quality of digital health services, in this sense, it does not focus on just one dimension, but demonstrates the interconnection between the structure and its processes, how these influence the results and impacts on health, considering both the organizational, technical and relational dimensions of care, and especially their impacts on the quality of PHC, emphasizing the implementation and sustainability of technologies over time. In contrast, the mapped models (Table 1), such as MAST (58), CSIRO (59), Health Optimum Telemedicine Acceptance Questionnaire (60), and MOMENTUM (61), concentrate on the evaluation of telemedicine. The CHEATS (62) framework assesses information communication technologies in health, detailing technical, educational, and social aspects. The Khoja-Durrani-Scott (KDS) (63) Framework for e-Health Evaluation provides a structured framework for the overall assessment of digital health services quality, with specific indicators for the stages of the e-health lifecycle, also focusing on the implementation and sustainability of technologies over time. The Digital Health Monitor (30) compares the maturity of digital health across different countries, offering a global overview. The RE-AIM (64) is an evaluation and implementation model not exclusively intended for digital health, as is NoMAD (65), an instrument specialized in the normalization of complex interventions within clinical practice.
The healthcare profile within PHC requires a tailored model for evaluating the quality of care mediated by digital strategies. This is justified because PHC offers philosophical support, as a strategy for organizing and reorganizing health systems (66–70), and has some essential attributes, namely: first contact: Offering users accessibility and use of health services, being the preferred gateway; longitudinality: it comprises the regular provision of care and its use over time, regardless of the presence of specific problems related to health or the type of problem; comprehensiveness or integrality: is one of the pillars in construction of the SUS enshrined in the Federal Constitution of 1988 and has four dimensions: primacy of promotion and prevention actions, attention to the three levels of complexity of medical care, articulation of promotion, protection and prevention actions and a comprehensive approach to the individual and families (68); coordination of care: articulation between the various health services and actions, so that they are synchronized and aimed at achieving a common objective, regardless of where they are provided; orientation to the community: it refers to the understanding that the health needs of individuals, families and the population are directly related to the social context (70); centrality in the family: achieved with the consideration of the context and family dynamics to better evaluate how to respond to the needs of each member and with the knowledge of the members and their health problems (69); cultural competence: recognition of the distinct needs of population groups, their ethnic, racial and cultural characteristics, understanding their representations of the health-disease processes (69).
Understanding the nuances of PHC in contemporary practice is critical for effective integration of digital technologies (71). That said, the inclusion of digital health strategies needs to take into account the attributes of PHC (9). Healthcare professionals must be prepared to address the ethical, security, and privacy challenges associated with the use of these technologies, while taking advantage of the benefits they provide to improve the quality and affordability of healthcare. In addition to the analysis of digital health, our study focuses on care itself, evaluating the structure, process and result of actions in primary care. The maturity of digital health will be one of the elements analyzed, within the scope of the quality of health care, but acting as a complement, and not the final target of the evaluation.
One critical aspect that warrants further attention is eHealth literacy, a concept essential for enhancing health outcomes in primary care settings. eHealth literacy refers to the ability of individuals to seek, understand, and use health information from electronic sources effectively. Given the increasing integration of digital health tools in primary care, ensuring that both patients and healthcare professionals possess adequate eHealth literacy is crucial for the success of these interventions. Studies have shown that higher levels of eHealth literacy are associated with better self-management of chronic diseases, improved patient engagement, and more effective use of telemedicine services (72, 73). Moreover, addressing eHealth literacy can help reduce health inequities by ensuring that vulnerable populations—such as older adults and those with lower education levels—can benefit from digital health innovations. As digital tools continue to evolve, it is essential that primary care incorporates strategies to improve eHealth literacy, which can, in turn, lead to more equitable health outcomes (74, 75). Thus, integrating eHealth literacy initiatives into primary care is not only a matter of access but also of optimizing the efficacy and reach of digital health strategies.
The model proposed for evaluating digital health in Brazilian Primary Health Care, which integrates Donabedian’s health quality framework and the Kellogg Foundation’s logic model, has similarities and differences to some studies in the literature. The clearest difference is its practical application to evaluate primary care services that use digital strategies in health care in a country in the Global South. Compared to published studies (76, 77), it can be said that all three studies emphasize barriers that must be overcome for the sustainable implementation of digital health. However, our approach stands out for integrating the logic of the Kellogg Foundation and Donabedian’s theory, providing a more dynamic analysis of interventions. The study by Willis et al. (78) also recognizes the importance of engaging healthcare professionals. However, the proposition we present goes further by incorporating contextual aspects and feedback, promoting local adaptations that are essential for the effectiveness of digital initiatives. We present a clearer framework for assessing the sustainability and long-term impact of digital interventions, which has led to a validated matrix of indicators (79). In this way, the proposed modeling, while in line with the concerns of previous studies, also broadens and updates the discussion on evaluating digital health in PHC.
Assessing differences in national contexts in the digitalization of health is crucial, as cultural and infrastructure differences directly impact the success of actions (9, 11). Quality monitoring contributes to the development and design of new technologies, which should focus on the needs of local communities, breaking the mold of coloniality in digital health (80). Digital health strategies implemented in a sustainable way in PHC and with guaranteed quality improvement can help increase service coverage and assist in health care (81).
A model for evaluating ICT care proposed/starting from the global South strengthens the chorus of the emancipation process of these countries, given the power that big techs have over technologies (80). The results of the application of this model can be used to guide governments in the implementation of public digital health policies aimed at emerging countries, countries with PHC, discuss health data security, technologies customized to the realities of each country, guiding about the weaknesses and strengths of the use of technologies in PHC. Countries in the global South will benefit from this model, and will be able to apply it in health quality assessment within technologies and in the development of health data regulatory infrastructures, health information systems, applications and programs.
4.2 Limitations and potentialities of this study
This study had a limitation related to the difficulty in terms of responding to the forms sent out, which prolonged the time intended for data collection, since it adopted virtual and asynchronous data collection, where repeated contacts were made with the judges when the validation forms were not responded to in a certain period. On the other hand, the technique allowed access to geographically distant judges and responses at more opportune times.
Despite the strong theoretical consistency of the instrument, its content was qualitatively validated for use in the context of Primary Health Care in Brazil, a country with continental dimensions and different realities, which limits its degree of reproducibility. However, it can be applied with contextual adaptations, maintaining the addressed analytical dimensions. The scalability of the model is associated with its structure based on two international models (Donabedian and the Kellogg logic model), which enhance its replicability in different contexts. For the application of this framework in other countries, it is important for stakeholders in the interested countries to update it. This update is necessary to capture social determinants of health, cultural factors, and aspects related to the organization of each country’s health system. To achieve this, it is crucial to invite health workers, technology developers, and users of health systems to update the indicators according to the country’s reality.
The potential of this model is related to its application for practical evaluation in health services, based on the organization of an indicator matrix, followed by validated instruments, applicable in various data collection methods such as interviews and questionnaires.
5 Conclusion
Evaluating the use of digital health in the quality of primary care is opportune to direct continuous education and training of health professionals; promote transparency in decision-making processes and encourage the participation of all stakeholders; ethical and effective use of digital technologies, data privacy, preparing professionals to deal with the challenges and make the most of the opportunities offered by the digital age. The model presented in this article considers the characteristics of realities of Brazilian health services.
This study achieved the goal of developing a model to evaluate digital health in PHC, helping to identify adequate and useful evaluation methods filling the gap of the lack quality indicators of health care mediated by digital strategies. It presents an important difference in relation to models mapped, as it considers the impacts on PHC quality attributes such as first contact care, longitudinality, integrality and coordination of care. The framework will be used by managers and health professionals of PHC in a case study in Brazil to evaluate the quality of digital health normatively. The results of this evaluation, based on the model, could influence the development of digital technologies, the improvement of ICTs used in PHC, public policies, regulations aimed at digital health and the transformation of the health work process. Also, the Model to evaluate digital health in Primary Health Care may be used in other evaluation studies and countries through contextual adaptations.
Data availability statement
The original contributions presented in the study are included in the article/Supplementary material, further inquiries can be directed to the corresponding author.
Ethics statement
The present study was approved by the research ethics committee of the Onofre Lopes University Hospital of the Federal University of Rio Grande do Norte (CAAE 48655521.9.0000.5292). We followed the recommendations of Resolutions 466/12 (46) and 510/16 (47) of the National Health Council and the guidelines proposed by the General Law of Data and Personal Protection number 13.709/2018 (23). All experts signed the informed consent form and authorized voice and image recording. The studies were conducted in accordance with the local legislation and institutional requirements. The participants provided their written informed consent to participate in this study.
Author contributions
ÍS: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. CS: Conceptualization, Data curation, Formal analysis, Investigation, Methodology, Resources, Software, Validation, Visualization, Writing – original draft, Writing – review & editing. CM: Validation, Visualization, Writing – review & editing, Data curation, Resources. AA: Data curation, Resources, Visualization, Writing – review & editing. RF: Formal analysis, Visualization, Writing – review & editing. LL: Formal analysis, Writing – review & editing, Validation, Visualization. RM: Validation, Writing – review & editing, Conceptualization, Methodology. EB: Formal analysis, Validation, Visualization, Writing – review & editing. SU: Conceptualization, Data curation, Formal analysis, Funding acquisition, Investigation, Methodology, Project administration, Resources, Software, Supervision, Validation, Visualization, Writing – original draft, Writing – review & editing.
Funding
The author(s) declare that financial support was received for the research, authorship, and/or publication of this article. This study was partly financed by the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior – Brasil (CAPES) – Finance Code 001.
Acknowledgments
The authors thank the stakeholders and judges who contributed to the development of this study, the Postgraduate Program in Collective Health at the Federal University of Rio Grande do Norte, and the Coordenação de Aperfeiçoamento de Pessoal de Nível Superior (CAPES). Moioli RC was supported by CNPq (305401/2023-8). Uchôa SAdC was supported by research background of UFRN (edital n°06/2022-PROPESQ). We would like to acknowledge the use of the ChatGPT artificial intelligence model (version October 2024, developed by OpenAI) to support the translation of the Supplementary material.
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.
Supplementary material
The Supplementary material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fpubh.2024.1443862/full#supplementary-material
Abbreviations
CAPES, Coordenação de Aperfeiçoamento Pessoal de Nível Superior; CSIRO, Commonwealth Scientific and Industrial Research Organization; DHS, Digital Health Strategy; GDHI, Global Digital Health Index; ICTs, Information and Communication Technologies; INMSD, National Digital Health Maturity Index; MAST, Model for Assessment of Telemedicine; MS, Ministry of Health; NHDN, National Health Data Network; NoMAD, Normalization Measure Development; PHC, Primary Health Care; PNIIS, National Policy on Health Information and Informatics; RE-AIM, Reach, efficacy, adoption, implementation and maintenance; SEIDIGI, Secretariat of Information and Digital Health; SISAB, System for Primary Care; SUS, Unified Health System; WHO, World Health Organization.
References
1. Renu, N . Technological advancement in the era of COVID-19. SAGE Open Med. (2021) 9:20503121211000912. doi: 10.1177/20503121211000912
2. Watts, T . (2023). The role of technology in the future and its impact on society. The Times of India Apr 14. Available at: https://timesofindia.indiatimes.com/readersblog/amitosh/the-role-of-technology-in-the-future-and-its-impact-on-society-52565/ (Accessed March 3, 2024)
3. Nascimento, IJB, Abdulazeem, HM, Vasanthan, LT, Martinez, EZ, Zucoloto, ML, Ostengaard, L, et al. The global effect of digital health technologies on health workers’ competencies and health workplace: an umbrella review of systematic reviews and lexical-based and sentence-based meta-analysis. Lancet Digit Health. (2023) 5:e534–44. doi: 10.1016/S2589-7500(23)00092-4
4. World Health Organization (WHO) . Classification of digital health interventions v 1.0. Geneva: World Health Organization (2018).
6. Silva, CRDV, Lopes, RH, Martiniano, CS, Silva, IS, and Uchôa, SAC. Conceito de saúde digital na atenção primária à saúde (2020-2022): um estudo baseado no método evolucionário de rodgers. Bol Conjuntura. (2024) 17:432–54. doi: 10.5281/zenodo.10565467
7. Organização Pan-americana de Saúde . (2023) O que é atenção primária à saúde?. Brasília: OPAS. Available online at: https://www.paho.org/pt/topicos/atencao-primaria-saude#:~:text=O%20que%20%C3%A9%20aten%C3%A7%C3%A3o%20prim%C3%A1ria,ao%20longo%20de%20sua%20vida (Accessed November 23, 2023)
8. Lapão, LV, Peyroteo, M, Maia, M, Seixas, J, Gregório, J, Mira da Silva, M, et al. Implementation of digital monitoring services during the COVID-19 pandemic for patients with chronic diseases: design science approach. J Med Internet Res. (2021) 23:e24181. doi: 10.2196/24181
9. Silva, CRDV, Lopes, RH, Bay, OG Jr, Martiniano, CS, Fuentealba-Torres, M, Arcêncio, RA, et al. Digital health opportunities to improve primary health care in the context of COVID-19: scoping review. JMIR Hum Factors. (2022) 9:e35380. doi: 10.2196/35380
10. Erku, D, Khatri, R, Endalamaw, A, Wolka, E, Nigatu, F, Zewdie, A, et al. Digital health interventions to improve access to and quality of primary health care services: a scoping review. Int J Environ Res Public Health. (2023) 20:e6854. doi: 10.3390/ijerph20196854
11. Stoumpos, AI, Kitsios, F, and Talias, MA. Digital transformation in healthcare: technology acceptance and its applications. Int J Environ Res Public Health. (2023) 20:e3407. doi: 10.3390/ijerph20043407
12. Nakayama, LF, Binotti, WW, Link Woite, N, Fernandes, CO, Alfonso, PG, Celi, LA, et al. The digital divide in Brazil and barriers to telehealth and equal digital health care: analysis of internet access using publicly available data. J Med Internet Res. (2023) 25:e42483. doi: 10.2196/42483
13. Silva, LMV . Conceitos, abordagens e estratégias para a avaliação em saúde In: ZMA Hartz and LMV Silva, editors. Avaliação em saúde: dos modelos teóricos à prática na avaliação de programas e sistemas de saúde. Salvador: EDUFBA (2005). 15–39.
14. Figueiró, AC, Frias, PG, and Navarro, LM. Avaliação em saúde: conceitos básicos para prática nas instituições In: I Samico, E Felisberto, AC Figueiró, and PG Frias, editors. Avaliação em saúde: bases conceituais e operacionais. Rio de Janeiro: MedBook (2010). 1–13.
15. Furtado, JP, and Silva, LMV. The evaluation of health programs and services in Brazil as a space for knowledge and practice. Cad Saúde Pública. (2014) 30:2643–55. doi: 10.1590/0102-311X00187113
16. Ministério da Saúde (Brasil) . (2013). Portaria n° 1.412, de 10 de julho de. Institui o Sistema de Informação em Saúde para a Atenção Básica (SISAB). Available online at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2013/prt1412_10_07_2013.html (Accessed November 23, 2023)
17. Ministério da Saúde (Brasil) . (2015). Portaria n° 589, de 20 de maio de 2015. Institui a Política Nacional de Informação e Informática em Saúde (PNIIS). Brasília, DF: MS. Available online at: http://bvsms.saude.gov.br/bvs/saudelegis/gm/2015/prt0589_20_05_2015.html (Accessed November 23, 2023)
18. Ministério da Saúde (Brasil) . Comitê Gestor da Estratégia e-Saúde. Estratégia e-Saúde para o Brasil. Brasília, DF: Ministério da Saúde (2017).
19. World Health Organization (WHO) . National eHealth strategy toolkit. Geneva: World Health Organization (2012).
20. Ministério da Saúde (Brasil) . (2021). Conecte SUS Pilot Project Final Report: analysis of progress made between October/2019 and June/2020. Brasília, DF: Ministério da Saúde. Available online at: https://bvsms.saude.gov.br/bvs/publicacoes/conectesus_pilot_project_final_report.pdf (Accessed January 24, 2023)
21. Ministério da Saúde (Brasil) . (2019). Portaria n° 2.983, de 11 de novembro. Institui o Programa de Apoio à Informatização e Qualificação dos Dados da Atenção Primária à Saúde – Informatiza APS. Diário Oficial da União. Available online at: https://www.in.gov.br/en/web/dou/-/portaria-n-2.983-de-11-de-novembro-de-2019-227652196 (Accessed March 2, 2023)
22. Ministério da Saúde (Brasil) . (2020). Portaria n° 1.434, de 28 de maio de 2020. Institui o Programa Conecte SUS e altera a Portaria de Consolidação n° 1/GM/MS, de 28 de setembro de 2017, para instituir a Rede Nacional de Dados em Saúde e dispor sobre a adoção de padrões de interoperabilidade em saúde. Diário Oficial da União. Available online at: https://www.in.gov.br/en/web/dou/-/portaria-n-1.434-de-28-de-maio-de-2020-259143327 (Accessed January 24, 2024)
23. Ministério da Saúde (Brasil) . Secretaria-Executiva. Estratégia de Saúde Digital para o Brasil 2020–2028. Brasília, DF: Ministério da Saúde (2020).
24. Ministério da Saúde (Brasil) . (2023). Sobre a SEIDIGI. Brasília, DF: Ministério da Saúde. Available online at: https://www.gov.br/saude/pt-br/composicao/seidigi#:~:text=Sobre%20a%20SEIDIGI,a%20gest%C3%A3o%20da%20sa%C3%BAde%20digital (Accessed March 2, 2024)
25. Brasil . (2018). Lei n° 13.709, de 13 de agosto de 2018. Lei Geral de Proteção de Dados Pessoais (LGPD). Brasília, DF: Ministério da Saúde. Available online at: https://www.planalto.gov.br/ccivil_03/_ato2015-2018/2018/lei/l13709.htm (Accessed September 22, 2022)
26. Ministério da Saúde (Brasil) . (2024). Portaria GM/MS n° 3.232, de 1° de março de 2024. Institui o Programa SUS digital. Diário Oficial da União. Available online at: http://www5.saude.ba.gov.br/portalcib/images/arquivos/Portarias/2024/08_MARCO_2024/PT_3232_1_MARCO_2024.pdf (Accessed April 22, 2024)
27. Donida, B, Costa, CA, and Scherer, JN. Making the COVID-19 pandemic a driver for digital health: Brazilian strategies. JMIR Public Health Surveill. (2021) 7:e28643. doi: 10.2196/28643
28. Varge Maldonado, JMS, Marques, AB, and Cruz, A. Telemedicine: challenges to dissemination in Brazil [Telemedicina: desafios à sua difusão no Brasil]. Cad Saúde Pública. (2016) 32:e00155615. doi: 10.1590/0102-311X00155615
29. Measure Evalution . (2018). Global digital health index (GDHI). North Carolina: Measure Evalution. Available online at: https://www.measureevaluation.org/his-strengthening-resource-center/his-tools/tool%3FToolId=67.html. (Accessed January 23, 2024)
30. Global Digital Health Monitor . (2024) World maps: State of Digital Health around the world today. Available online at: https://digitalhealthmonitor.org/. (Accessed May 22, 2024)
31. Healthcare Information and Management Systems Society . (2024). Digital Health Transformation Save lives and improve outcomes with tech. Chicago: HIMSS. Available online at: https://www.himss.org/what-we-do-solutions/digital-health-indicator. (Accessed May 10, 2024)
32. Ministério da Saúde (Brasil) . (2022). Maior sistema público de saúde do mundo, SUS completa 31 anos. Brasília, DF: Ministério da Saúde. Available online at: https://www.gov.br/saude/pt-br/assuntos/noticias/2021/setembro/maior-sistema-publico-de-saude-do-mundo-sus-completa-31-anos#:~:text=Garantido%20no%20artigo%20196%20da,para%20qualquer%20atendimento%20de%20sa%C3%BAde (Accessed September 22, 2023)
34. Donabedian, A . Evaluating the quality of medical care. Milbank Q. (2005) 83:691–729. doi: 10.1111/j.1468-0009.2005.00397.x
35. WK Kellogg Foundation . Using logic models to bring together planning, evaluation, and action: Logic model development guide. Battle Creek: WK Kellogg Foundation (2004).
36. Bezerra, TCA, Falcão, MLP, Goes, PSA, and Felisberto, E. Evaluation of professional training programs in health: indicator construction and validation. Trab educ saúde. (2016) 14:445–72. doi: 10.1590/1981-7746-sip00111
37. Jones, J, and Hunter, D. Qualitative research: consensus methods for medical and health services research. BMJ. (1995) 311:e376:376–80. doi: 10.1136/bmj.311.7001.376
38. Uchôa, SAC, Gondim, GMM, Barreto, MA, Rocha, NSPD, and Rocha, PDM. Utilizando técnicas de consenso: potencialidades e limites na avaliação de informações em saúde In: Z Hartz, E Felisberto, and LM Vieira-da-Silva, editors. Meta-avaliação da atenção básica à saúde: teoria e prática. Rio de Janeiro: Editora Fiocruz (2008). 253–82.
39. Cisneros-Caicedo, AJ, Guevara-García, AF, Urdánigo-Cedeño, JJ, and Garcés-Bravo, JE. Técnicas e Instrumentos para la Recolección de Datos que Apoyan a la Investigación Científica en Tiempo de Pandemia. Dom Cienc. (2022) 8:1165–85. doi: 10.23857/dc.v8i1.2546
40. Gusmão, VCL, Flausino, TGC, Couto, DS, Abraão, LM, Felix, AMS, Ciofi-Silva, CL, et al. Adapting the nominal group technique to a virtual version: an experience report. Rev Esc Enferm USP. (2024) 58:e20230298. doi: 10.1590/1980-220X-REEUSP2023-0298en
41. Conselho Nacional de Saúde (Brasil) . (2012). Resolução n° 466 de 12 de dezembro de 2012. Brasília: Conselho Nacional de Saúde. Available online at: https://conselho.saude.gov.br/resolucoes/2012/Reso466.pdf (Accessed September 20, 2022)
42. Conselho Nacional de Saúde (Brasil) . (2016). Resolução n° 510 de 07 de abril de 2016. Brasília: Conselho Nacional de Saúde. Available online at: https://bvsms.saude.gov.br/bvs/saudelegis/cns/2016/res0510_07_04_2016.html (Accessed September 20, 2022)
43. Organização Pan-americana de Saúde (OPAS) . (2023). Tecnologias devem garantir inclusão e equidade, reforçam OPAS e Ministério da Saúde do Brasil em Simpósio para fortalecer transformação digital e sistemas de informação. Brasília, DF: OPAS. Available online at: https://www.paho.org/pt/noticias/28-3-2023-tecnologias-devem-garantir-inclusao-e-equidade-reforcam-opas-e-ministerio-da (Accessed November 23, 2023)
44. Ministério da Saúde (Brasil) (2011). Portaria n° 2.546, de 27 de Outubro de. Redefine e amplia o Programa Telessaúde Brasil, que passa a ser denominado Programa Nacional Telessaúde Brasil Redes (Telessaúde Brasil Redes). Brasília, DF: Ministério da Saúde. Available online at: https://bvsms.saude.gov.br/bvs/saudelegis/gm/2011/prt2546_27_10_2011.html (Accessed November 23, 2023)
45. Ministério da Saúde (Brasil) . (2020). Lei n° 13.989, de 15 de abril de 2020. Dispõe sobre o uso da telemedicina durante a crise causada pelo coronavírus (SARS-CoV-2). Brasília, DF: Ministério da Saúde. Available online at: https://www.planalto.gov.br/ccivil_03/_ato2019-2022/2020/Lei/L13989.htm#:~:text=Disp%C3%B5e%20sobre%20o%20uso%20da,SARS%2DCoV%2D2).&text=O%20PRESIDENTE%20DA%20REP%C3%9ABLICA%20Fa%C3%A7o,SARS%2DCoV%2D2 (Accessed November 23, 2023)
46. Ministério da Saúde, Conselho Nacional de Secretários de Saúde – CONASS . Conselho Nacional de Secretarias Municipais de Saúde – CONASEMS (Brasil). Guia Orientador para o enfrentamento da pandemia covid-19 na Rede de Atenção à Saúde 4ª edição. Brasília, DF: CONASS (2021).
47. Ministério da Gestão e da Inovação em Serviços Públicos (Brasil) . (2022). Legislação Governança Digital. Brasília, DF: Ministério da Gestão e da Inovação em Serviços Públicos. Available online at: https://www.gov.br/governodigital/pt-br/legislacao/governanca-digital#:~:text=Decreto%20n%C2%BA%2010.782%2C%20de%2030,Brasileira%20para%20a%20Transforma%C3%A7%C3%A3o%20Digital (Accessed November 23, 2023)
48. Brasil . (2022). Lei n° 14.510, de 27 de dezembro de 2022. Brasília, DF: Presidência da República. Available online at: https://legislacao.presidencia.gov.br/atos/?tipo=LEI&numero=14510&ano=2022&ato=42aETQU9kMZpWT56d (Accessed November 23, 2023)
49. Brasil . (2023) Ministério da Saúde. Meu SUS Digital. Brasília, DF: Ministério da Saúde. Available online at: https://www.gov.br/saude/pt-br/composicao/seidigi/meususdigital (Accessed November 23, 2023)
50. Brasil . (2024). Ministério da Saúde. Portaria GM/MS n° 3.564, de 18 de abril. Altera a Portaria de Consolidação GM/MS n° 1, de 28 de setembro de 2017, para instituir o Laboratório Inova SUS Digital, do Ministério da Saúde Digital. Brasília, DF: Ministério da Saúde. Available online at: https://www.in.gov.br/web/dou/-/portaria-gm/ms-n-3.564-de-18-de-abril-de-2024-555679304 (Accessed November 23, 2023)
52. Modolo, L, Carvalho, S, and Dias, T. Digital health issues for the SUS: “mobile health” and the algorithmic automation of medical knowledge-power. Saude soc. (2023) 32:2024. doi: 10.1590/S0104-12902023220245pt
53. Gu, W, Shu, L, Chen, W, Wang, J, Wu, D, Ai, Z, et al. Evaluation of Chinese healthcare organizations' innovative performance in the digital health era. Front Public Health. (2023) 11:1141757. doi: 10.3389/fpubh.2023.1141757
54. Andreoni, G, Caiani, EG, and Castaldini, N. Digital health services through patient empowerment: classification, current state and preliminary impact assessment by health pod systems. Appl Sci. (2022) 12:359. doi: 10.3390/app12010359
55. Debelle, H, Packer, E, Beales, E, Bailey, HGB, Mc Ardle, R, Brown, P, et al. Feasibility and usability of a digital health technology system to monitor mobility and assess medication adherence in mild-to-moderate Parkinson's disease. Front Neurol. (2023) 14:1111260. doi: 10.3389/fneur.2023.1111260
56. Woods, L, Dendere, R, Eden, R, Grantham, B, Krivit, J, Pearce, A, et al. Perceived impact of digital health maturity on patient experience, population health, health care costs, and provider experience: mixed methods case study. J Med Internet Res. (2023) 25:e45868. doi: 10.2196/45868
57. Furtado, JP . A constructivist method for health evaluation. Ciênc Saúde Coletiva. (2001) 6:165–81. doi: 10.1590/S1413-81232001000100014
58. Kidholm, K, Ekeland, AG, Jensen, LK, Rasmussen, J, Pedersen, CD, Bowes, A, et al. A model for assessment of telemedicine applications: mast. Int J Technol Assess Health Care. (2012) 28:44–51. doi: 10.1017/S0266462311000638
59. Nepal, S, Li, J, Jang-Jaccard, J, and Alem, L. A framework for telehealth program evaluation. Telemed J E Health. (2014) 20:393–404. doi: 10.1089/tmj.2013.0093
60. European Comission . Regions of Europe working together for HEALTH. European Comission: Brussells, Belgium (2011).
61. Christiansen, EK, Henriksen, E, Jensen, LK, Lange, M, Lapão, L, Kaye, R, et al. (2014) European momentum for mainstreaming telemedicine deployment in daily practice (Grant agreement no 297320). Available online at: http://www.telemedicine-momentum.eu/wp-content/uploads/2014/12/D3.2_v13_Momentum_ConsolidatedBlueprint.pdf (Accessed January 24, 2023)
62. Shaw, NT . CHEATS: a generic information communication technology (ICT) evaluation framework. Comput Biol Med. (2002) 32:209–20. doi: 10.1016/s0010-4825(02)00016-1
63. Khoja, S, Durrani, H, Scott, RE, Sajwani, A, and Piryani, U. Conceptual framework for development of comprehensive e-health evaluation tool. Telemed J E Health. (2013) 19:48–53. doi: 10.1089/tmj.2012.0073
64. Glasgow, RE, Vogt, TM, and Boles, SM. Evaluating the public health impact of health promotion interventions: the RE-AIM framework. Am J Public Health. (1999) 89:1322–7. doi: 10.2105/ajph.89.9.1322
65. Finch, TL, Girling, M, May, CR, Mair, FS, Murray, E, Treweek, S, et al. Improving the normalization of complex interventions: part 2-validation of the NoMAD instrument for assessing implementation work based on normalization process theory (NPT). BMC Med Res Methodol. (2018) 18:e135. doi: 10.1186/s12874-018-0591-x
66. Starfield, B . Atenção primária: equilíbrio entre necessidades de saúde, serviços e tecnologia. Brasília, DF: UNESCO, Ministério da Saúde (2002).
67. Oliveira, MAC, and Pereira, IC. Primary health care essential attributes and the family health strategy. Rev Bras Enferm. (2013) 66:158–64. doi: 10.1590/S0034-71672013000700020
68. Giovanella, LL, Costa, LV, Carvalho, AI, and Conill, EM. Sistemas municipais de saúde e a diretriz da integralidade da atenção: critérios para avaliação. Saúde debate. (2002) 26:37–61.
69. Giovanella, L, and Mendonça, MHM. Atenção Primária à Saúde In: L Giovanell, S Escorel, L LVC, and AI Carvalho, editors. Políticas e Sistema de Saúde no Brasil. Rio de Janeiro: Fiocruz (2009). 575–625.
70. Conselho Nacional de Secretário de Saúde (CONASS) . Atenção Primária e Promoção da Saúde. Brasília, DF: CONASS (2011).
71. Chen, S, Niu, M, and Ngai, CSB. What is the next step of ICT development? The changes of ICT use in promoting elderly healthcare access: a systematic literature review. Heliyon. (2024) 10:e25197. doi: 10.1016/j.heliyon.2024.e25197
72. Stellefson, M, Paige, SR, Alber, JM, Chaney, BH, Chaney, D, Apperson, A, et al. Association between health literacy, electronic health literacy, disease-specific knowledge, and health-related quality of life among adults with chronic obstructive pulmonary disease: cross-sectional study. J Med Internet Res. (2019) 21:e12165. doi: 10.2196/12165
73. Wong, DK, and Cheung, MK. Online health information seeking and eHealth literacy among patients attending a primary Care Clinic in Hong Kong: a cross-sectional survey. J Med Internet Res. (2019) 21:e10831. doi: 10.2196/10831
74. Xu, RH, Zhou, LM, Wong, EL, and Wang, D. The association between Patients' eHealth literacy and satisfaction with shared decision-making and well-being: multicenter cross-sectional study. J Med Internet Res. (2021) 23:e26721. doi: 10.2196/26721
75. Xu, RH, Bao, XL, Shi, LS, and Wang, D. Associations between eHealth literacy, mental health-seeking attitude, and mental wellbeing among young electronic media users in China during the COVID-19 pandemic. Front Public Health. (2023) 11:1139786. doi: 10.3389/fpubh.2023.1139786
76. Neve, G, Fyfe, M, Hayhoe, B, and Kumar, S. Digital health in primary care: risks and recommendations. Br J Gen Pract. (2020) 70:609–10. doi: 10.3399/bjgp20X713837
77. Pagliari, C . Digital health and primary care: past, pandemic and prospects. J Glob Health. (2021) 11:01005. doi: 10.7189/jogh.11.01005
78. Willis, VC, Thomas Craig, KJ, Jabbarpour, Y, Scheufele, EL, Arriaga, YE, Ajinkya, M, et al. Digital health interventions to enhance prevention in primary care: scoping review. JMIR Med Inform. (2022) 10:e33518. doi: 10.2196/33518
79. de Figueirêdo, RC, de Siqueira, SÍ, de Araújo, AJ, Silva, CRDV, Martiniano, CS, Brito, EWG, et al. Preparation and validation of the instrument "QualiAPS digital-Brazil" for assessing digital health care in primary health care: a required tool. Front Public Health. (2024) 12:1304148. doi: 10.3389/fpubh.2024.1304148
80. Sekalala, S, and Chatikobo, T. Colonialism in the new digital health agenda. BMJ Glob Health. (2024) 9:e014131. doi: 10.1136/bmjgh-2023-014131
81. World Health Organization . (2019). Primary health care on the road to universal health coverage: 2019 monitoring report. Available online at: https://reliefweb.int/report/world/primary-health-care-road-universal-health-coverage-2019-monitoring-report?gclid=CjwKCAjw69moBhBgEiwAUFCx2CRt64GlQuM6IhHXsO-wKbVNmjO2pBx0UsxO0NFstSkxDjTfxRAdZBoCXgoQAvD_BwE (Accessed December 20, 2023)
Keywords: health evaluation, evaluation models, health care quality, access, evaluation, digital health, Primary Health Care
Citation: Silva ÍdS, Silva CRDV, Martiniano CS, Araújo AJd, Figueirêdo RCd, Lapão LV, Moioli RC, Brito EWG and Uchôa SAdC (2024) Digital health and quality of care in Primary Health Care: an evaluation model. Front. Public Health. 12:1443862. doi: 10.3389/fpubh.2024.1443862
Edited by:
Frank Chen, City University of Hong Kong, Hong Kong SAR, ChinaReviewed by:
Richard Xu, Hong Kong Polytechnic University, Hong Kong SAR, ChinaAnn D. Bagchi, Rutgers, The State University of New Jersey, United States
Copyright © 2024 Silva, Silva, Martiniano, Araújo, Figueirêdo, Lapão, Moioli, Brito and Uchôa. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Ísis de Siqueira Silva, isis1998.siqueira.silva@gmail.com
†These authors have contributed equally to this work