Overview of global publications on machine learning in diabetic retinopathy from 2011 to 2021: Bibliometric analysis

Purpose: To comprehensively analyze and discuss the publications on machine learning (ML) in diabetic retinopathy (DR) following a bibliometric approach.

Methods: The global publications on ML in DR from 2011 to 2021 were retrieved from the Web of Science Core Collection (WoSCC) database. We analyzed the publication and citation trend over time and identified highly-cited articles, prolific countries, institutions, journals and the most relevant research domains. VOSviewer and Wordcloud are used to visualize the mainstream research topics and evolution of subtopics in the form of co-occurrence maps of keywords.

Results: By analyzing a total of 1147 relevant publications, this study found a rapid increase in the number of annual publications, with an average growth rate of 42.68%. India and China were the most productive countries. IEEE Access was the most productive journal in this field. In addition, some notable common points were found in the highly-cited articles. The keywords analysis showed that “diabetic retinopathy”, “classification”, and “fundus images” were the most frequent keywords for the entire period, as automatic diagnosis of DR was always the mainstream topic in the relevant field. The evolution of keywords highlighted some breakthroughs, including “deep learning” and “optical coherence tomography”, indicating the advance in technologies and changes in the research attention.

Conclusions: As new research topics have emerged and evolved, studies are becoming increasingly diverse and extensive. Multiple modalities of medical data, new ML techniques and constantly optimized algorithms are the future trends in this multidisciplinary field.

Introduction

Diabetic retinopathy (DR), as one of the characterized microvascular complications of diabetes mellitus, has already become the leading cause of vision loss in the worldwide working-age population (1). Most patients with early-stage DR appear normal without any visual disruptions, however, the potential pathological changes, such as microvascular damage and neurodegeneration, are progressing (2, 3). Severe DR can cause visual impairment and finally lead to irreversible blindness, seriously affecting the quality of life. To prevent or manage DR, screening, early detection and intervention are crucial (4). In clinical practice, fundus examinations are recommended during the process of screening, diagnosis and follow-up of DR. The mainstream examinations include digital retinal photography, optical coherence tomography (invasive technologies such as fundus fluorescein photography are less common), etc. (1, 5, 6). Ophthalmologists can diagnose DR based on the typical lesions (e.g., exudates, microaneurysms) that appeared in the digital images (7). In addition to forming the basis of clinical diagnosis, the massive medical data from examinations has significant value for academic research.

With the development of artificial intelligence (AI) technologies, machine learning (ML), as an advanced field of AI, has gradually intertwined with various aspects of modern medicine (8). Machine learning focuses on enabling computers to automatically learn from the data of different modalities without being explicitly programmed (9). ML is a general name including many technological terms, such as deep learning (DL), supervised learning or neural networks. The implementation of ML in medicine is usually related to disease detection, survival prediction and risk evaluation, and so on (10–12). When compared to other medical specialties, ophthalmology features a wide application of imaging techniques with abundant data resources and an urgent need for computer-aided diagnosis due to the shortage of ophthalmologists (13). This leads to the emergence and rapid development of ML in ophthalmology. DR is one of the widely researched diseases in this field because of its increasing prevalence and the high risk of blindness in severe cases. Automatic DR grading/identification, automatic DR lesion detection and other related achievements have been reported in various conferences or journal articles (14, 15). Moreover, a number of review articles discuss the overall development of ML techniques in DR (4, 16). Thus far, however, no bibliometric analysis has been conducted on this topic.

The bibliometric analysis uses mathematical and statistical methodologies to obtain quantifiable and objective data from intangible features of the literature (17, 18). It has been applied in numerous topics and disciplines. To our knowledge, this is the bibliometric study focused on the literature related to ML in DR. To search for as much relevant literature as possible, we prepared a keyword list based on related books and articles. However, the search based on these keywords leads to the retrieval of documents with diversified purposes, study design and topics, or some irrelevant records. Thereby we generated the inclusion criteria and manually screened all the retrieved documents to confirm that included articles focused on ML in DR. Moreover, as the topics of included documents are relatively diverse and impractical to summarize one by one, we utilized VOSviewer and Wordcloud to visualize these topics in the form of co-occurrence maps of keywords. In addition, we also interpret our results based on the overall progress of ML techniques and the status of DR during 2011-2021 to make our analysis reasonable.

This paper has three objectives: first, to summarize the publication trend and identify the outstanding achievements; second, to reveal the contributions of countries/institutions/journals and visualize the collaboration networks; third, to uncover the mainstream topic and study the evolution of subtopics in this area.

Methods

Search strategy

All of the reference data used in this study were collected from the Web of Science Core Collection (WoSCC), which incorporates articles in over 20,000 high-quality peer-reviewed scholarly journals published worldwide in addition to a large number of proceedings papers (a single set of 28 criteria was made to evaluate journals). To search for the relevant data, a set of DR-related keywords and a set of ML-related keywords were prepared based on relevant literature and books (1, 19). Specific keywords were shown in Table 1. We searched for documents containing at least one DR-related keyword and one ML-related keyword in the “topic” of records (including title, abstract, author and keywords), for example, the documents that include both “diabetic retinopathy” and “machine learning”. As many state-of-the-art achievements in computer science involving machine learning technologies would publish in conference proceedings besides journal articles, the scope of document types included journal articles, proceedings papers and reviews. The timespan was from 2011 to 2021. The last search was conducted on September 24, 2021. A total of 2960 retrieved documents from the WoSCC were prepared for the following screening (Figure 1).

TABLE 1

Table 1 List of the search keywords.

FIGURE 1

Figure 1 Flowchart of search, screening and bibliometric analysis.

Screening strategy

As we searched based on a relatively big set of keywords, some irrelevant documents may also be retrieved. Three authors (A.S., K.J. and L.L.) made the inclusion criteria by reviewing the first 500 documents (primary screening). The practical inclusion criteria: 1. Involves ML technologies. 2. Involves DR, including: (1) Studies focused on DR; (2) Studies focused on the characterized clinical features of DR, for example, microaneurysms and exudates detection (20); (3) Studies focused on multiple diseases and DR was included (21); (4) Studies focused on a topic that is beneficial to the various clinical scenarios of DR, for example, blood-vessel segmentation in the fundus images is beneficial for the following diagnosis of DR (22). After carefully reviewing the 2960 retrieved documents (secondary screening by A.S. and K.J.), 1147 documents were included for the bibliometric analysis (Figure 1).

Bibliometric analysis

Bibliometric analysis was conducted on 1147 documents for obtaining insights into the current trends and topics on ML for DR. In this study, we conducted a trend analysis of publications and citations, publication pattern and collaboration analysis, research domains and targeted sources analysis, as well as the analysis of the keywords.

The analytic tool of the WoSCC database, and Microsoft Excel were used for data mining and representation. The summarized data included publication count and citation count of years/countries/institutions/journals/research domains. Self-citations were included. The Hirsch-index (H-index) was used originally to reflect the academic impact of a researcher, which describes that a researcher has published h number of articles, and each of the h articles has at least h times of citations (23, 24). Currently, the H-index is commonly used for assessing the academic influence of countries/institutions/journals in the bibliometric analysis (25). The growth rate of publications was calculated as follows:

$$Growth rate=\left(\sqrt[t_2-t_1]{p_2÷p_1}-1\right)\times 100$$

where t₁: First year; t₂: Last year; p₁: Publication count of the first year; p₂: Publication count of the last year.

Visualized analysis

VOSviewer and Wordcloud were also applied to visualize the collaboration of countries/institutions, co-occurrence of keywords and evolutions of hotspots in the target field. VOSviewer is an analytical tool for constructing and displaying bibliometric maps in an easy-to-interpret manner. To use VOSviewer, we first exported the entire record and cited references of included documents in plain text form, and then the data were imported into VOSviwer (version 1.6.17) (26). By adjusting the options of types and other parameters (type of analysis: co-occurrence; unit of analysis: author keywords; counting method: full counting; the size variation of items, labels and lines between two items were also adjusted for the best presentation), we generated the primary bibliometric maps. Based on the observation of these maps, we generated a text file of the thesaurus to avoid the appearance of synonyms (e.g., “automated detection” & “automatic detection”) in the keyword map. At last, certain meaningless keywords (e.g., “level”) were also deleted to generate the final diagram. To use Wordcloud, we loaded the Wordcloud Python package. The data of title, abstract and keywords were exported and stored as 5 text documents corresponding to 5 periods of time. The thesaurus file was also applied so that synonyms will be regarded as the same word/phrase. After deleting the meaningless characters in the files (e.g., “TI”, the abbreviation before each title), we generated 5 diagrams, representing the 5 studied periods.

Results

Trend analysis of publications and citations

Figure 2 plots the annual trends of publications and citations on machine learning in diabetic retinopathy. We included 1147 articles for the analysis in this study (658 journal articles, 449 proceedings and 40 reviews). From 2017, the annual publication number exceeded 100, and the last 5 years (2017-2021) contributed 78.12% (896/1147) of all articles. The average growth rate from 2011 to 2020 was 42.68%. Polynomial regression analysis was conducted to model the publication and citation trends (2021 was excluded because of incomplete indexing). The estimated models of y1 = 2.6591x2 - 3.2742x + 15.433 and y2 = 102.63x2 - 670.7x + 911.78 indicate changes in the quantities of publications and citations with time, respectively. Both the results of the growth rate and polynomial regression model demonstrate the significant and rapid increase in publications and citations, indicating that machine learning in diabetic retinopathy keeps gaining researchers’ attention and the field is generally at the growth phase. The detailed publication number of different article types and study designs (retrospective/prospective) during 2011-2021 were shown in Table 2.

FIGURE 2

Figure 2 Trend analysis of publications and citations.

TABLE 2

Table 2 The yearly publication count.

Moreover, we listed the top 10 articles ranked by annual citation count in Table 3. Of these 10 articles, all were journals, and 6 articles were published in the last 5 years. The earliest article was by Wang et al. in 2015 (33), introducing a new retinal blood-vessel segmentation method that was beneficial to the screening of DR. The most impactful article was by Gulshan et al. in 2016 (14). They developed a deep learning algorithm to identify referable/non-referable DR and DME, which was a milestone in this field. The algorithm achieved fairly high performance with the area under the receiver operating curve above 0.99 in 2 publicly available datasets (EyePACS and Messidor-2). It is noteworthy that the medical device called IDx-DR mentioned in the pragmatic trial by Abramoff et al. is the first device authorized for marketing by the FDA to automatically detect DR based on fundus images without the need for the interpretation of an additional specialist. The year 2016 and 2017 witnessed 7 of the top 10 articles in this field. The top 10 articles ranked by total citation count were also listed in Table 3. Three journals and one review published before 2015 were newly on the list.

TABLE 3

Table 3 Top articles ranked by annual citations and total citations.

Publication pattern and collaboration analysis

Overall, 58 countries contributed to the publications on this topic. The top 10 countries ranked by total publication output accounted for 92.50% (1061/1147) of all included studies and were listed in Table 4. India published the most documents (350/1147), accounting for 30.51% of all included studies. China was the second leading country (222/1147, 19.35%), followed by the USA (161/1147, 14.04%). It is worth noting that the USA ranked 1st in terms of citation count while it ranked 3rd in the publication count, and the citation ranks of Singapore, Malaysia, and Australia were also higher than their publication ranks. Institutions with at least 15 documents were also listed in Table 4 ranked by total publications. National University of Singapore is the most prolific institution (26/1147, 2.27%), followed by National University of Sciences Technology, Pakistan (25/1147, 2.18%) and Singapore National Eye Center (24/1147, 2.09%). However, the publication of reviews cannot directly indicate the active research of a certain institution. Therefore, we also calculate the number of publications except for the reviews. National University of Sciences Technology was the most active institution in research (23/1147, 2.01%), followed by Sun Yat-Sen University (22/1147, 1.92%). There were 3 institutions from Singapore with the highest overall H-index (11, 14, 14). Three Chinese institutions, 2 Indian institutions, 2 American institutions and 2 Pakistan institutions were listed in Table 4. Figure 3 demonstrates the collaboration networks of countries (documents ≥5, 36 countries were included) and institutions (documents ≥5, 65 institutions met the criteria, 16 institutions had no connections to other institutions and were excluded, hence 49 institutions were included).

TABLE 4

Table 4 Top countries and institutions ranked by publication count.

FIGURE 3

Figure 3 Collaboration maps between countries and institutions.(A) Highly contributed countries. (B) Highly contributed institutions. Circle size represents the publication count; circle color represents average citations; links represent the collaboration.

Research domains and targeted sources

Table 5 shows the 10 most common research domains that the included documents belong to. Computer Science (537/1147, 46.82%), Engineering (527/1147, 45.95%) and Radiology Nuclear Medicine Medical Imaging (128/1147, 11.16%) were the 3 main research domains.

TABLE 5

Table 5 The most related research domains ranked by publication count.

Journals with H-index ≥5 and publications ≥10 were listed in Table 6, ranked by publication count. We also referred to the Journal Citation Reports (JCR)(2020) to demonstrate the academic impact of these journals. IEEE Access was the journal with the most articles published (36, 3.14%) while IEEE Transactions on Medical Imaging was the most impactful among all included journals at the time of analysis. The top 12 journals, which only accounted for 5.31% of 226 journals that have published articles in this field, published 29.23% of all journal articles and reviews (204/689). Seven journals ranked “Q1” in JCR, two journals ranked “Q2” and one journal ranked “Q3”. As for the conference documents, only the International Conference on Medical Imaging Computer-Aided Diagnosis published more than 5 documents (7/1147, 0.61%).

TABLE 6

Table 6 Most productive journals ranked by publication count.

Keywords analysis

To obtain a deeper understanding of research topics and how they are interconnected, we visualized the hotspots of included studies by conducting a keyword co-occurrence analysis using VOSviewer (Figure 4). For the total of 2088 automatically identified keywords, 84 keywords occurred at least 10 times, which were shown in Figure 4. This map of keywords illustrates the hotspots related to machine learning in diabetic retinopathy. All included keywords were divided into 3 clusters, indicated by red, green and blue colors, representing the ML techniques (e.g., “deep learning”, “convolutional neural networks”, etc.), applications of ML techniques (e.g., “classification”, “segmentation”, etc.) and the DR-related diseases, clinical features and medical data (e.g., “microaneurysms”, “fundus images”, etc.), respectively. From Figure 4, we can identify the hot topics represented by strongly linked keywords and the weakly-explored subareas between 2 relatively isolated keywords.

FIGURE 4

Figure 4 The co-occurrence map of keywords; reveals 3 clusters (in 3 colors): ML techniques; applications of ML techniques; relevant diseases, clinical features and medical data. Circle size represents the frequency of occurrence; links represent the co-occurrence.

To understand when these hotspots emerged and how they evolved, we divided the documents into 5 groups by publication time:1.2014-2015; 2.2016-2017; 3.2018-2019; 4.2020-2021; 5.2011-2013 (the only 3-year group, considering that in the first 3 years, the academic output was relatively small when compared with other periods). Five corresponding maps of keywords were conducted by Wordcloud (Figure 5). Each map includes the top 30 keywords ranked by the frequency of occurrence. The size of the font represents the frequency (the more frequently-occurred, the bigger scale). “Diabetic retinopathy” was the most dominant keyword for the entire period. Other frequent keywords included “classification”, “fundus image”, “deep learning”, indicating that most studies in this field focused on applying the classification ability of ML techniques into DR based on the medical images. Figure 5A displays the top keywords identified during 2011-2013, where the dominant keywords besides “diabetic retinopathy” were “microaneurysm”, “exudate”, and “blood vessel” (ranked 1st to 15^th, red color), whereas “neural network”, “diagnosis” and “database” were less dominant (ranked 16th to 30^th, green color). In 2014-2015, “detection”, “segmentation”, and “support vector machine” were more dominant, while “blindness”, “image processing”, and “vessel segmentation” were less dominant (Figure 5B). In 2016-2017, “detection”, “neural network” and “diabetic macular edema” were more dominant; “deep learning”, “convolutional neural network”, and “support vector machine” were less dominant (Figure 5C). In 2018-2019, “deep learning”, “optical coherence tomography” and “dataset” were more dominant; “exudate”, “microaneurysm” and “grading” were less dominant (Figure 5D). In 2020-2021, “deep learning”, “convolutional neural network” and “grading” were more dominant; “lesion”, “blood vessel” and “exudate” were less dominant (Figure 5E). The detailed frequency of keywords was listed in Supplementary Table 1.

FIGURE 5

Figure 5 Cloud maps of keywords in 5 periods (the top 30 most frequently-occurred keywords in each map, red keywords are more dominant, green keywords are less dominant): (A) during periods 2011-2013; (B) during periods 2014-2015; (C) during periods 2016-2017; (D) during periods 2018-2019; (E) during periods 2020-2021.

Discussion

Trend analysis of publications and citations

From 2011 to 2020, the number of publications grew from 10 to 245 and the overall growth rate reached 42.68%, indicating significant growth in research interests in this field. In addition, the rapid expansion of the annual citations reflected the increasing impact of related publications. On the one hand, this growing trend is due to the breakthroughs in AI technology and its wide application in medicine: in 2012, a well-trained deep convolutional neural network won the ImageNet challenge (40); in 2014, the generative adversarial network was invented (41). As a subarea of ML, DL was gradually applied to various domains of medicine, including radiology, pathology, dermatology, ophthalmology, and so on (42). On the other hand, multiple public ophthalmic datasets were set up around 2010, which accelerated the development of relevant research. For example, Kaggle EyePACS (2015) consists of over 80000 annotated fundus images with DR staging; the Messidor dataset (2008) consists of 1200 fundus images accompanied with medical diagnosis. These public large-scale datasets have created a great opportunity for academic groups worldwide to test and benchmark their models/systems/algorithms. Furthermore, the establishment of recognized DR grading standards (e.g., ICDRSS scale) (7) also promoted the comparison of diagnostic ability between different models or between man and machine. In general, there is still a distance between the current ML in DR and the clinical practice as most studies are in silico and aim to optimize algorithms and propose new techniques based on recognized public datasets or local private datasets. The prospective studies in this field mainly focus on the real-world viability test, clinical validation of algorithms/software and human-machine comparison (43–45). However, as machine learning becomes mature in this area, the number and proportion of real-world-oriented studies are increasing.

Table 3 shows that the most impactful articles were published after 2015. After Gulshan et al. published the most impactful in 2016 and received widespread attention from ophthalmic researchers, many DL-based studies have sprung up, which is also consistent with the publication trend and the development of technologies and databases. Some common points of impactful articles were found out: 1. Published by influential journals (e.g. JAMA - IF:56.27; Ophthalmology, IOVS – the top journals of ophthalmology); 2. New techniques (e.g. deep learning in 2016, 2017); 3. Excellent results (e.g., great performance of algorithms with an area under the receiver operating curve > 0.99); 4. Involved in multiple tasks (e.g. automatic grading of DR severity or detection of multiple diseases including DR). These articles led the developing trend in this field and many articles were based on these achievements.

Publication pattern and collaboration analysis

Researchers all over the world have contributed to the field of ML in DR. The publication pattern reveals that India and China have been the most productive countries. The two densely populated developing countries contributed to nearly half of the relevant publications, which is uncommon in other bibliometric studies on the topic of AI technologies in medicine (17, 25), as developed countries such as the USA or England are usually the main force. In addition, there are 5 developing countries in the top 10 countries ranked by publication count, all with considerable academic output. However, in terms of the H-index and citations of different countries, developed countries performed relatively better compared to developing countries. This can be explained by differences in social medical resources and technologies between countries. With the global epidemic of diabetes, the prevalence of DR is also rising predominantly, especially in densely populated countries like India and China (46, 47). There is a clear but unmet need to comprehensively screen DR in the diabetic population in the rural area of these developing countries due to the disproportionally low ophthalmic population (13). Developing countries are urgently calling for a cost-effective way to manage DR. Therefore, the automatic system based on ML is widely explored by academic groups from developing countries. As for developed countries, institutions and researchers benefit from technological breakthroughs and the mature ophthalmic system. Researchers are more likely to publish impactful articles. The National University of Singapore is the most productive institution and most publications also belong to Singapore National Eye Center. The two institutions tend to publish articles that push forward the clinical application of ML techniques in DR, including the clinical validation of DL systems based on the Singapore National Diabetic Retinopathy Screening Program or other multiethnic DR screening data and reviews that discussed the current status of AI techniques in the real-word DR screening (48, 49). By analyzing the top institutions (documents ≥ 20), we found that most studies from National University of Sciences Technology Pakistan, Indian Institute of Technology System and Northeastern University China are ML technique-oriented. Most studies from two Singaporean institutions are medicine-oriented. Researchers from Sun Yat-Sen University published both technique-based studies and clinical validation studies as they collaborated a lot with hospitals and computer science laboratories. From the perspective of citations, those medicine-oriented and pragmatic studies are more popular than technique-oriented studies.

The collaboration analysis also revealed that productive countries/institutions have more options for international collaborations. In addition, the nodes in the middle of Figure 3A tend to appear yellow, indicating that countries/institutions with more external collaborations have a greater chance of publishing impactful articles (i.e., high average citations).

Research domains and targeted sources

As included documents are mainly related to computer techniques and imaging systems, journals that specialized in these domains were productive in this field. On the one hand, the advancement of computer science and engineering accelerated the pace of applying AI technologies in medicine. On the other hand, imaging techniques are commonly used in ophthalmology and produce lots of valuable data on DR patients, which is useful for developing ML algorithms. The impact factor (mostly around five) and the JCR rank of the twelve “core journals” indicate the overall impact and quality of relevant publications. Only Computers in Biology and Medicine and Biomedical Optics Express have published impactful articles, as shown in Table 3. Impactful journals such as JAMA and Ophthalmology are not shown due to the publication count.

Keywords analysis

The frequently occurred keywords in the literature always indicate the research hotspots. The co-occurrence of several keywords represents the widely discussed topic containing several basic components. By dividing the relevant literature by time, the emergence and evolution of keywords can be visualized on the word clouds. Keywords analysis reveals the mainstream topics in the field, the research focuses on different periods and the subareas that are currently popular or remained to be explored.

Overall, the application of machine learning techniques in diabetic retinopathy is extensive and diverse, while most documents aim to diagnose DR automatically. “Diabetic retinopathy” is the most dominant keyword for the whole period, along with other frequent keywords such as “classification”, “segmentation” and “fundus images”. Thus, fundus images are the most commonly used data for research. Classification and segmentation are the tasks for ML or the processing steps for the data. Some keywords relating to DR lesions (e.g., “microaneurysms”, “exudates”) are also dominant in Figure 5, as many documents focus on detecting characterized lesions of DR to mimic the diagnostic process of ophthalmologists. A tiny microaneurysm can be the key to distinguishing between diseases and normality, thereby the automatic detection of these lesions makes the diagnosis of ML algorithms reasonable (20). Some keywords of ML techniques were prominent in the keywords co-occurrence analysis (e.g., “deep learning”, “support vector machine”, “convolutional neural network”), representing the popular tool applied in DR. By linking up the keywords, the mainstream concepts are immediately visible, for example, the diagnosis of “diabetic retinopathy” based on the “automated detection” of “exudates” in “retinal images” by “deep learning”.

However, both techniques and clinical focuses change over time. From 2011 to 2021, the evolution of topics mainly focused on computer methods, clinical tasks and data modalities. First, “deep learning” and “convolutional neural network” appeared in 2016-2017 for the first time and subsequently became larger in the word cloud, indicating that deep learning and related techniques gained increased research attention, which was consistent with the publication time of the paper by Gulshan et al. and the overall development of DL techniques. The traditional technique “support vector machine” became less popular in this field due to the remarkable performance of DL in feature extraction and representation. Second, the keywords of DR features (e.g., exudate, microaneurysm) became less frequent, indicating that simple lesion-detection algorithms were gradually dismissed. Many comprehensive DR grading systems and multi-disease diagnosis systems have sprung up recently as the keyword “grading” gradually become frequent (21). Third, due to the limited information offered by digital fundus images, the data from new imaging techniques such as optical coherence tomography, gradually emerged in this field (Figures 5C–E). Other imaging techniques like fundus fluorescein angiography were also considered but not shown in the word cloud, which needs to be further studied (50). Moreover, the ML algorithms usually focused on the simple data modality while doctors would refer to different types of examination data and complaints of patients. As the keyword “dataset” and “database” has become much more dominant from 2011 to 2021, the integration of multi-modal data from different sources might be the future direction for automated diagnosis. In addition, we found that although the keyword “patient” was less prominent from 2011 to 2021, the frequency rank kept rising. From a clinical point of view, patients are always the main components of all relevant studies. With the ML techniques in DR getting matured, more researchers designed studies that better reflect the real-world effectiveness of AI systems. These studies not only included the existing datasets but also test their algorithm/software in broader patient groups. To utilize AI as tools in real clinical settings, the algorithms in this field are constantly optimized in both the techniques (from “support vector machine” to “deep learning”) and the capacity of dealing with more complex conditions which mimic the clinical settings (e.g., grading DR based on multi-modal data).

This study is the first bibliometric analysis of ML in DR and aims to provide a holistic view of the relevant research. The results discussed in this study are objective, quantifiable and macroscopical, which would be suitable for any researchers interested in this field to get familiar with the basic knowledge structure (e.g., the mainstream topic, the outstanding achievements, the emerging trends, global publication pattern, and relevant research domains, etc.) and can help them find potential collaborators and develop relevant studies. Moreover, the change in publication trends and keywords from 2011 to 2021 indicated the potential directions of further studies in this field, including the incorporation of optimized ML techniques, multi-modal data, real-world-oriented study design, etc.

Limitations

This study has some limitations. First, we only used reference data from a single database (WoSCC) and the results of the bibliometric analysis may not be as robust as studies that collect data from multiple databases due to some unpredictable bias when we search for documents in WoSCC. However, WoSCC is a well-indexed database that represents one of the largest multidisciplinary collections of indexed published literature. Moreover, the list of keywords may not be comprehensive enough to retrieve all related documents even if we referred to the relevant literature and books. Second, although some meaningless keywords were deleted in the figure conducted by VOSviewer and Wordcloud, not all keywords are informative enough in the figure, such as “system”. These general keywords occur frequently but do not refer to any deeper subfields, therefore, these keywords cannot be analyzed. Finally, like other bibliometric analyses, this study didn’t focus on the content of every single article; the uniqueness and novelty of most articles were ignored and only top articles were analyzed. Third, the emerging novel topics discussed in this study may have stagnations to practice as the co-occurrence maps of keywords are based on frequency. A breakthrough was reflected on these maps only when it gradually became recognized in the research community and also it takes time for researchers to cite these articles (to be listed as top articles in the bibliometric analysis).

Conclusions

In this study, we provided a comprehensive overview of all retrieved articles on ML in DR following a bibliometric approach for the first time. It’s a growing research area and has been studied by researchers from multiple countries and institutes. As new topics have emerged and evolved since 2011, studies in this field are becoming more diverse and extensive. Real-world-oriented studies with multi-modal data and optimized ML techniques are the further directions as clinical application is the ultimate goal in this field. Further studies can focus on larger research fields (e.g., AI techniques in ophthalmology) and the integration of data from multiple databases.

Data availability statement

The original contributions presented in the study are included in the article/Supplementary Material. Further inquiries can be directed to the corresponding authors.

Author contributions

Design of the work: AS, KJ, WZ and JY. Collection and screening of the data for the work: AS, KJ and LL. Analysis of the data: YL. Drafting the work: AS. Revising of the manuscript: KJ, LL, WZ and JY. All authors contributed to the article and approved the submitted version.

Funding

This work was financially supported by the National Key Research and Development Program of China (grant number 2019YFC0118401), Zhejiang Provincial Key Research and Development Plan (grant number 2019C03020), Natural Science Foundation of Zhejiang Province (grant number LQ21H120002), Medical and Health Science and Technology Program of Zhejiang Province (grant number 2021RC064), the Natural Science Foundation of China (grant number 81670888), ZJU-BIOMIND Medical Artificial Intelligence Research.

Conflict of interest

The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.

Publisher’s note

All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article, or claim that may be made by its manufacturer, is not guaranteed or endorsed by the publisher.

Supplementary material

The Supplementary Material for this article can be found online at: https://www.frontiersin.org/articles/10.3389/fendo.2022.1032144/full#supplementary-material

References

1. Wong TY, Cheung CMG, Larsen M, Sharma S, Simó R. Diabetic retinopathy. Nat Rev Dis Primers (2016) 17(2):16012. doi: 10.1038/nrdp.2016.12

CrossRef Full Text | Google Scholar

2. Jenkins AJ, Joglekar M v., Hardikar AA, Keech AC, O’Neal DN, Januszewski AS. Biomarkers in diabetic retinopathy. Rev Diabetic Stud (2015) 12(1-2):159–95. doi: 10.1900/RDS.2015.12.159

PubMed Abstract | CrossRef Full Text | Google Scholar

3. Lieth E, Gardner TW, Barber AJ, Antonetti DA. Retinal neurodegeneration: Early pathology in diabetes. Clin Exp Ophthalmol (2000) 28(1):3–8. doi: 10.1046/j.1442-9071.2000.00222.x

PubMed Abstract | CrossRef Full Text | Google Scholar

4. Tsiknakis N, Theodoropoulos D, Manikis G, Ktistakis E, Boutsora O, Berto A, et al. Deep learning for diabetic retinopathy detection and classification based on fundus images: A review. Comput Biol Med (2021) 135:104599. doi: 10.1016/j.compbiomed.2021.104599

PubMed Abstract | CrossRef Full Text | Google Scholar

5. Sim DA, Keane PA, Rajendram R, Karampelas M, Selvam S, Powner MB, et al. Patterns of peripheral retinal and central macula ischemia in diabetic retinopathy as evaluated by ultra-widefield fluorescein angiography. Am J Ophthalmol (2014) 158(1):144–53. doi: 10.1016/j.ajo.2014.03.009

PubMed Abstract | CrossRef Full Text | Google Scholar

6. Chung YR, Kim YH, Ha SJ, Byeon HE, Cho CH, Kim JH, et al. Role of inflammation in classification of diabetic macular edema by optical coherence tomography. J Diabetes Res (2019) 2019:8164250. doi: 10.1155/2019/8164250

PubMed Abstract | CrossRef Full Text | Google Scholar

7. Wilkinson CP, Ferris FL, Klein RE, Lee PP, Agardh CD, Davis M, et al. Proposed international clinical diabetic retinopathy and diabetic macular edema disease severity scales. Ophthalmology (2003) 110(9):1677–82. doi: 10.1016/S0161-6420(03)00475-5

PubMed Abstract | CrossRef Full Text | Google Scholar

8. El-Alfy ESM, Mohammed SA. A review of machine learning for big data analytics: Bibliometric approach. Technol Anal Strateg Manag (2020) 32(8):984–1005. doi: 10.1080/09537325.2020.1732912

CrossRef Full Text | Google Scholar

9. Deo RC. Machine learning in medicine. Circulation (2015) 132(20):1920–30. doi: 10.1161/CIRCULATIONAHA.115.001593

PubMed Abstract | CrossRef Full Text | Google Scholar

10. Montazeri M, Montazeri M, Montazeri M, Beigzadeh A. Machine learning models in breast cancer survival prediction. Technol Health Care (2016) 24(1):31–42. doi: 10.3233/THC-151071

PubMed Abstract | CrossRef Full Text | Google Scholar

11. Li W, Yang Y, Zhang K, Long E, He L, Zhang L, et al. Dense anatomical annotation of slit-lamp images improves the performance of deep learning for the diagnosis of ophthalmic disorders. Nat BioMed Eng (2020) 4(8):767–77. doi: 10.1038/s41551-020-0577-y

PubMed Abstract | CrossRef Full Text | Google Scholar

12. Santilli AML, Jamzad A, Sedghi A, Kaufmann M, Logan K, Wallis J, et al. Domain adaptation and self-supervised learning for surgical margin detection. Int J Comput Assist Radiol Surg (2021) 16(5):861–69. doi: 10.1007/s11548-021-02381-6

PubMed Abstract | CrossRef Full Text | Google Scholar

13. Resnikoff S, Lansingh VC, Washburn L, Felch W, Gauthier TM, Taylor HR, et al. Estimated number of ophthalmologists worldwide (International council of ophthalmology update): Will we meet the needs? Br J Ophthalmol (2020) 104(4):588–92. doi: 10.1136/bjophthalmol-2019-314336

PubMed Abstract | CrossRef Full Text | Google Scholar

14. Gulshan V, Peng L, Coram M, Stumpe MC, Wu D, Narayanaswamy A, et al. Development and validation of a deep learning algorithm for detection of diabetic retinopathy in retinal fundus photographs. JAMA - J Am Med Assoc (2016) 316(22):2402–10. doi: 10.1001/jama.2016.17216

CrossRef Full Text | Google Scholar

15. Levenkova A, Sowmya A, Kalloniatis M, Ly A, Ho A. (2017). Automatic detection of diabetic retinopathy features in ultra-wide field retinal images, in: Conference on Medical Imaging - Computer-Aided Diagnosis. Proceedings of SPIE. Orlando, FL: International Society for Optics and Photonics p. 101341. doi: 10.1117/12.2253980

CrossRef Full Text | Google Scholar

16. Raman R, Srinivasan S, Virmani S, Sivaprasad S, Rao C, Rajalakshmi R. Fundus photograph-based deep learning algorithms in detecting diabetic retinopathy. Eye (Basingstoke) (2019) 33(1):97–109. doi: 10.1038/s41433-018-0269-y. Nature Publishing Group; Eye (Basingstoke).

CrossRef Full Text | Google Scholar

17. Guo Y, Hao Z, Zhao S, Gong J, Yang F. Artificial intelligence in health care: Bibliometric analysis. J Med Internet Res (2020) 22(7):e18228. doi: 10.2196/18228

PubMed Abstract | CrossRef Full Text | Google Scholar

18. Wallin JA. Bibliometric methods: Pitfalls and possibilities. Basic Clin Pharmacol Toxicol (2005) 97(5):261–75. doi: 10.1111/j.1742-7843.2005.pto_139.x

PubMed Abstract | CrossRef Full Text | Google Scholar

19. WIPO. WIPO technology trends 2019: Artificial intelligence. Geneva: World Intellectual Property Organization (2019).

Google Scholar

20. Usman Akram M, Khalid S, Tariq A, Khan SA, Azam F. Detection and classification of retinal lesions for grading of diabetic retinopathy. Comput Biol Med (2014) 45(1):161–71. doi: 10.1016/j.compbiomed.2013.11.014

PubMed Abstract | CrossRef Full Text | Google Scholar

21. Cen LP, Ji J, Lin JW, Ju ST, Lin HJ, Li TP, et al. Automatic detection of 39 fundus diseases and conditions in retinal photographs using deep neural networks. Nat Commun (2021) 12(1):4828. doi: 10.1038/s41467-021-25138-w

PubMed Abstract | CrossRef Full Text | Google Scholar

22. Orlando JI, Blaschko M. LNCS 8673 - learning fully-connected CRFs for blood vessel segmentation in retinal images. Med Image Comput Comput Assist Interv (2014) 17(1):634–41. doi: 10.1007/978-3-319-10404-1_79

PubMed Abstract | CrossRef Full Text | Google Scholar

23. Sierra JC, Haynes SN, Eysenck MW, Buela-Casal G, Hirsch JE. The meaning of the h-index. Int J Clin Health Psychol (2014) 14(2):161–4. doi: 10.1016/S1697-2600(14)70050-X

CrossRef Full Text | Google Scholar

24. Manyangu G, Dineen B, Geoghegan R, Flaherty G. Descriptive bibliometric analysis of global publications in lifestyle-based preventive cardiology. Eur J Prev Cardiol (2019) 28(12):1303–14. doi: 10.1177/2047487319854827

CrossRef Full Text | Google Scholar

25. Chen X, Tao X, Wang FL, Xie H. Global research on artificial intelligence-enhanced human electroencephalogram analysis. Neural Computing Appl (2021) 34(14):11295–333. doi: 10.1007/s00521-020-05588-x

CrossRef Full Text | Google Scholar

26. van Eck NJ, Waltman L. Software survey: VOSviewer, a computer program for bibliometric mapping. Scientometrics (2010) 84(2):523–38. doi: 10.1007/s11192-009-0146-3

PubMed Abstract | CrossRef Full Text | Google Scholar

27. Ting DSW, Cheung CYL, Lim G, Tan GSW, Quang ND, Gan A, et al. Development and validation of a deep learning system for diabetic retinopathy and related eye diseases using retinal images from multiethnic populations with diabetes. JAMA - J Am Med Assoc (2017) 318(22):2211–23. doi: 10.1001/jama.2017.18152

CrossRef Full Text | Google Scholar

28. Gargeya R, Leng T. Automated identification of diabetic retinopathy using deep learning. Ophthalmology (2017) 124(7):962–9. doi: 10.1016/j.ophtha.2017.02.008

PubMed Abstract | CrossRef Full Text | Google Scholar

29. Abràmoff MD, Lou Y, Erginay A, Clarida W, Amelon R, Folk JC, et al. Improved automated detection of diabetic retinopathy on a publicly available dataset through integration of deep learning. Invest Ophthalmol Vis Sci (2016) 57(13):5200–6. doi: 10.1167/iovs.16-19964

PubMed Abstract | CrossRef Full Text | Google Scholar

30. Krause J, Gulshan V, Rahimy E, Karth P, Widner K, Corrado GS, et al. Grader variability and the importance of reference standards for evaluating machine learning models for diabetic retinopathy. Ophthalmology (2018) 125(8):1264–72. doi: 10.1016/j.ophtha.2018.01.034

PubMed Abstract | CrossRef Full Text | Google Scholar

31. Pratt H, Coenen F, Broadbent DM, Harding SP, Zheng Y. (2016). Convolutional neural networks for diabetic retinopathy, in: 20th Conference on Medical Image Understanding and Analysis (MIUA) Loughborough Univ, Loughborough, ENGLAND: ELSEVIER SCIENCE BV, , Vol. 90. pp. 200–5. doi: 10.1016/j.procs.2016.07.014

CrossRef Full Text | Google Scholar

32. Quellec G, Charrière K, Boudi Y, Cochener B, Lamard M. Deep image mining for diabetic retinopathy screening. Med Image Anal (2017) 39:178–93. doi: 10.1016/j.media.2017.04.012

PubMed Abstract | CrossRef Full Text | Google Scholar

33. Wang S, Yin Y, Cao G, Wei B, Zheng Y, Yang G. Hierarchical retinal blood vessel segmentation based on feature and ensemble learning. Neurocomputing (2015) 149:708–17. doi: 10.1016/j.neucom.2014.07.059

CrossRef Full Text | Google Scholar

34. Gadekallu TR, Khare N, Bhattacharya S, Singh S, Maddikunta PKR, Ra IH, et al. Early detection of diabetic retinopathy using pca-firefly based deep learning model. Electron (Switzerland) (2020) 9(2). doi: 10.3390/electronics9020274

CrossRef Full Text | Google Scholar

35. Karri SPK, Chakraborty D, Chatterjee J. Transfer learning based classification of optical coherence tomography images with diabetic macular edema and dry age-related macular degeneration. BioMed Opt Express (2017) 8(2):579–92. doi: 10.1364/BOE.8.000579

PubMed Abstract | CrossRef Full Text | Google Scholar

36. Mookiah MRK, Acharya UR, Chua CK, Lim CM, Ng EYK, Laude A. Computer-aided diagnosis of diabetic retinopathy: A review. Comput Biol Med (2013) 43(12):2136–55. doi: 10.1016/j.compbiomed.2013.10.007

PubMed Abstract | CrossRef Full Text | Google Scholar

37. Antal B, Hajdu A. An ensemble-based system for microaneurysm detection and diabetic retinopathy grading. IEEE Trans BioMed Eng (2012) 59(6):1720–6. doi: 10.1109/TBME.2012.2193126

PubMed Abstract | CrossRef Full Text | Google Scholar

38. Srinivasan PP, Kim LA, Mettu PS, Cousins SW, Comer GM, Izatt JA, et al. Fully automated detection of diabetic macular edema and dry age-related macular degeneration from optical coherence tomography images. BioMed Opt Express (2014) 5(10):3568. doi: 10.1364/BOE.5.003568

PubMed Abstract | CrossRef Full Text | Google Scholar

39. Zhang X, Thibault G, Decencière E, Marcotegui B, Laÿ B, Danno R, et al. Exudate detection in color retinal images for mass screening of diabetic retinopathy. Med Image Anal (2014) 18(7):1026–43. doi: 10.1016/j.media.2014.05.004

PubMed Abstract | CrossRef Full Text | Google Scholar

40. Krizhevsky A, Sutskever I, Hinton GE. ImageNet classification with deep convolutional neural networks. Communication ACM (2017) 60(6):84–90. doi: 10.1145/3065386

CrossRef Full Text | Google Scholar

41. Goodfellow IJ, Pouget-Abadie J, Mirza M, Xu B, Warde-Farley D, Ozair S, et al. Generative adversarial networks. Adv Neural Inf Process Systems (2014) 3:2672–80. doi: 10.1145/3422622

CrossRef Full Text | Google Scholar

42. Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nat Med (2019) 25(1):44–56. doi: 10.1038/s41591-018-0300-7

PubMed Abstract | CrossRef Full Text | Google Scholar

43. Heydon P, Egan C, Bolter L, Chambers R, Anderson J, Aldington S, et al. Prospective evaluation of an artificial intelligence-enabled algorithm for automated diabetic retinopathy screening of 30 000 patients clinical science. Br J Ophthalmol (2021) 105(5):723–8. doi: 10.1136/bjophthalmol-2020-316594

PubMed Abstract | CrossRef Full Text | Google Scholar

44. Zhang Y, Shi J, Peng Y, Zhao Z, Zheng Q, Wang Z, et al. Artificial intelligence-enabled screening for diabetic retinopathy: A real-world, multicenter and prospective study. BMJ Open Diabetes Res Care (2020) 8(1). doi: 10.1136/bmjdrc-2020-001596

CrossRef Full Text | Google Scholar

45. Gulshan V, Rajan RP, Widner K, Wu D, Wubbels P, Rhodes T, et al. Performance of a deep-learning algorithm vs manual grading for detecting diabetic retinopathy in India. JAMA Ophthalmol (2019) 137(9):987–93. doi: 10.1001/jamaophthalmol.2019.2004

PubMed Abstract | CrossRef Full Text | Google Scholar

46. Raman R, Ganesan S, Pal SS, Kulothungan V, Sharma T. Prevalence and risk factors for diabetic retinopathy in rural india. Sankara nethralaya diabetic retinopathy epidemiology and molecular genetic study III (SN-DREAMS III), report no 2. BMJ Open Diabetes Res Care (2014) 2(1):e000005. doi: 10.1136/bmjdrc-2013-000005

PubMed Abstract | CrossRef Full Text | Google Scholar

47. Wang FH, Liang YB, Zhang F, Wang JJ, Wei Wb, Tao QS, et al. Prevalence of diabetic retinopathy in rural China: The handan eye study. Ophthalmology (2009) 116(3):461–7. doi: 10.1016/j.ophtha.2008.10.003

PubMed Abstract | CrossRef Full Text | Google Scholar

48. Grzybowski A, Brona P, Lim G, Ruamviboonsuk P, Tan GSW, Abramoff M, et al. Artificial intelligence for diabetic retinopathy screening: A review. Eye (Basingstoke) (2020) 34:451–60. doi: 10.1038/s41433-019-0566-0

CrossRef Full Text | Google Scholar

49. Bellemo V, Lim ZW, Lim G, Nguyen QD, Xie Y, Yip MYT, et al. Articles artificial intelligence using deep learning to screen for referable and vision-threatening diabetic retinopathy in Africa: A clinical validation study. Lancet Digital Health (2019) 1(1):E35–44. doi: 10.1016/S2589-7500(19)30004-4

PubMed Abstract | CrossRef Full Text | Google Scholar

50. Jin K, Pan X, You K, Wu J, Liu Z, Cao J, et al. Automatic detection of non-perfusion areas in diabetic macular edema from fundus fluorescein angiography for decision making using deep learning. Sci Rep (2020) 10(1). doi: 10.1038/s41598-020-71622-6

CrossRef Full Text | Google Scholar