Nicolas Duchateau
Andrew P. King
Mathieu De Craene- 1CREATIS, CNRS UMR 5220, INSERM U1206, Université, Lyon, France
- 2School of Biomedical Engineering and Imaging Sciences, King's College London, London, United Kingdom
- 3Philips Research Paris, Suresnes, France
Information about myocardial motion and deformation is key to differentiate normal and abnormal conditions. With the advent of approaches relying on data rather than pre-conceived models, machine learning could either improve the robustness of motion quantification or reveal patterns of motion and deformation (rather than single parameters) that differentiate pathologies. We review machine learning strategies for extracting motion-related descriptors and analyzing such features among populations, keeping in mind constraints specific to the cardiac application.
1. Introduction
1.1. Myocardial Motion and Deformation Analysis: What For?
Pump efficiency can discriminate failing from healthy hearts, as quantified by volume and ejection fraction. Clinicians are well aware of the limitations of these simple measurements to face the complexity of heart disease, and recommend finer markers of cardiac mechanical dysfunction (1). Myocardial motion (displacement or velocity) and deformation (strain or strain rate) are richer descriptors of (ab)normal cardiac function (2, 3). They can provide characteristic spatiotemporal signatures for disease at each location of the myocardium and each instant of the cardiac cycle. They are often projected onto anatomically-relevant directions to facilitate interpretations (4). Interestingly, they can be estimated from routine modalities such as echocardiography and magnetic resonance (MR) (5), and have therefore been thoroughly investigated for a wide range of applications.
1.2. Machine Learning for Myocardial Motion and Deformation Analysis: What For?
Machine learning builds upon models whose optimal parameters are learnt from a set of samples representative of the studied population. This data-driven approach is more flexible than traditional methods (e.g., variational), as demonstrated for myocardial segmentation (6, 7), and has strong potential for the analysis of complex descriptors such as myocardial motion and deformation. In essence, machine learning seeks to learn data representations (either explicit or hidden) for better solving a supervised problem or for characterizing the data distribution. This often involves dimensionality reduction to facilitate the analysis of high-dimensional descriptors, and requires navigating between the low-dimensional/latent space and high-dimensional/original space for better interpretation.
1.3. Which Data Approach for Learning?
Over the years, researchers have gained detailed knowledge of the complexity of cardiac mechanics, and proposed physiologically-relevant motion and deformation descriptors, from global strain in a single anatomical direction to richer representations such as 3D+t vector or tensor fields. Most approaches decompose the analysis into two steps (Figure 1A): the extraction of motion/deformation descriptors from image sequences, followed by their analysis over a population of interest. Machine learning can address both parts, and we discuss these topics separately (sections 2 and 3). Deep neural networks (8) may address the two parts in Figure 1A, but also enable the analysis of population data directly from the image sequences by looking for image features not necessarily interpretable or visualizable, but optimal to answer the clinical question of interest (Figure 1B). We specifically comment on this strategy, which is more recent and preliminary, in section 4.7.
Figure 1. Two possible approaches for analyzing myocardial motion and deformation from image sequences using machine learning: (A) extraction of descriptors followed by their analysis, and (B) both parts addressed at once.
2. Motion and Deformation Estimation
Traditionally, myocardial motion fields have been estimated from images using standard image registration techniques such as optical flow (9), free-form deformation (10), or block matching (11). Naturally, this depends on the algorithm ability to catch motion-related structures, which strongly varies with the imaging modality. Tags and speckles can directly be tracked within the myocardium in tagged MR and 2D/3D echography (within the limits of tag fading, speckles temporal consistency, and out-of-plane motion), contrary to cine MR where algorithms tend to approximate motion from endocardial/epicardial contour tracking. A dedicated review (5) details the standards for spatial and temporal resolution and the influence of imaging parameters on the estimation of myocardial deformation.
Approaches based on neural networks challenge the variational formulation of motion estimation, as shown on video image sequences with the FlowNet2 convolutional neural network (CNN) architecture (12) that focuses on optical flow. Similar approaches have been applied to cardiac imaging (13, 14), but raise several methodological questions. First, the generalization ability of the trained networks to estimate a wide range of deformations at multiple scales still needs to be verified. This is critical for specific disease traits of lower prevalence. Furthermore, robustness to a variety of routine clinical imaging conditions (different image qualities, fields of view, devices, etc.) needs to be established. Second, supervised CNN-based motion estimators such as FlowNet2 do not embed any regularization, and are therefore sensitive to imaging noise if it differs from the training database. This not the case for unsupervised approaches like (13), which use an intensity-based loss, combined with a regularization term as in classical image registration. Finally, motion features can boost segmentation performances (15–17), as looking at several frames improves the manual segmentation of physicians. Further details are given in a review dedicated to deep learning for motion estimation in medical imaging (18).
Statistical models learnt from data can act as regularizers for tracking algorithms. (19) used dictionary learning as a sparse basis for cardiac motion fields to feed the regularization. Within deep learning, auto-encoders can encode spatial transformations into a low-dimensional space and provide powerful projection and reconstruction operators to connect with the tracking in the original image space (20).
Additional constraints specific to the cardiac application can provide more plausible registration outputs, such as invertibility (the myocardium does not fold) and incompressibility, as investigated for the diffeomorphic LogDemons (21) and free-form-deformation algorithms (22). Temporal consistency has been enforced through 4D representations of motion (23, 24), for multiple pairwise transformations simultaneously (25), or for intra/inter-subject mappings (26). Motion and deformation estimation with machine learning should also consider these aspects for better consistency and robustness.
3. Motion and Deformation Analysis
3.1. Before the Analysis: Data Normalization
Cardiac image data often need to be normalized in terms of anatomy, frame rate or cycle phases, before any statistical or machine learning analysis.
Image sequences can be registered using a 4D transformation model based on e.g., free-form deformation (10) or demons (26). This approach quantifies the spatiotemporal differences between the image sequences, analyzed statistically afterwards through deformation-based morphometry methods.
Motion or deformation descriptors (or any other data) from a given individual can also be transported to a reference template (generally, a central case at end-diastole). This involves local reorientation of the motion/deformation fields (27, 28), adjusted to the addressed clinical question (29). Temporal differences between sequences can also be normalized by resampling before the motion extraction [e.g., piece-wise linear interpolation (30)]. Recent approaches transport the whole subject-specific trajectory instead of the descriptors of interest, with specific computational considerations (31, 32). Automatically estimating multiple templates across the sequence may also be well adapted to the cardiac circular/periodic dynamics (33).
In both strategies, existing data correspondences facilitate the normalization. Spatial alignment can rely on anatomical landmarks (apex, valve ring, etc.) or point-to-point correspondences obtained from model-based tracking of the anatomy. Temporal alignment can use physiologically-relevant instants, such as the maximum contraction (10) or QRS and valve events (28).
3.2. Learning From Motion and Deformation Data
Machine learning can benefit a wide range of clinical problems. Unsupervised approaches learn a data representation that uncovers useful insights into the data distribution, but without explicit reference to a particular clinical question. Clustering and dimensionality reduction techniques fall into this category. Supervised approaches train a model for a specific task, and labels/annotations are provided as supervision. For example, diagnosing disease may involve binary labels for supervision (disease/healthy) and the task would be to predict these labels from the motion data. The type of labels determines the task addressed by the model: categorical labels mean classification, whereas discrete or continuous labels imply regression. Supervised approaches also involve learning a (lower dimension) representation of the data that facilitates the classification/regression, but this representation can be formed in an unsupervised or supervised way, as described below.
3.2.1. Unsupervised Learning
Unsupervised motion and deformation analysis shares objectives with statistical atlases, regarding how to characterize variability across a population. Pioneering works directly applied a principal component analysis (PCA) on myocardial displacements at each spatiotemporal location (34) over a healthy population, later extended through the estimation of local abnormalities in the myocardial velocities of a given subject compared to a reference population (28, 35). However, these analyses consider each spatial location and temporal instant independently from the others. The statistical analysis can also consider the motion patterns over the entire cardiac cycle as high-dimensional objects, as simply demonstrated through a PCA on temporal strain traces concatenated over the heart segments (36, 37). This approach reminds earlier work on Active Appearance Motion Models (38), which statistically analyzed both displacement and image intensity information over the entire cardiac cycle.
More advanced strategies estimate a low-dimensional space that encodes the high-dimensional myocardial motion/deformation data and navigate through this space, although this requires specific care. Myocardial shapes across a population can be considered as originating from one or several references under the action of a transformation such as a diffeomorphic warping. In this case, the space of myocardial shapes is related to the (known) non-linear high-dimensional space of diffeomorphic transformations. This space is a manifold, and known tools exist to perform statistics on such transformations and therefore on myocardial shapes while preserving this data structure (39, 40). Myocardial motion/deformation patterns may also be considered as originating from a non-linear high-dimensional manifold, but in this case the manifold is unknown. Machine learning allows estimating this space from data, and can overcome the limitations of linear techniques such as PCA that ignore this known structure. A general framework (41) groups the vast variety of existing manifold learning techniques. A graph is built across high-dimensional samples to approximate the manifold, and diagonalization, and dimensionality reduction processes provide a low-dimensional space that encodes the data. Techniques generally differ on how input samples are related within the graph, either locally (e.g., distance between neighbors, or local structure variations expressed in the graph Laplacian) or globally (e.g., geodesic distance). These techniques improve the statistical analysis of myocardial motion and deformation patterns. They can represent the continuum of disease from normality while preserving the data structure (42). The unsupervised representation of populations is particularly interesting when existing labels are not fully trusted, as in heart failure with preserved ejection fraction (43, 44) or when a supervised formulation of the clinical problem is uncertain, such as outcome from cardiac resynchronization therapy (45).
Nonetheless, these techniques normally lack explicit mappings between the high-dimensional and low-dimensional spaces, which are typically approximated using out-of-sample reconstruction/regression (46) and are therefore inexact. Deep learning auto-encoders explicitly address this by simultaneously learning how to encode and decode high-dimensional data with a limited number of parameters while minimizing the reconstruction error. However, this also requires constraining the distribution of samples in the latent space so that a statistical analysis can still be performed on it afterwards, as in variational auto-encoders (47). These techniques are promising for the analysis of myocardial motion and deformation and start being used in cardiac imaging for segmentation (48, 49) or segmentation-based biomarkers (50).
3.2.2. Supervised Learning
As noted above, designing a supervised learning model traditionally consists of two steps (Figure 1A). First, the input data are transformed to a new representation that facilitates the task performance. Second, a classification or regression model is trained to predict the label given the new representation. More recent techniques such as deep learning combine these two steps: the representation is learnt and optimized during the model training (Figure 1B). Below, we first summarize works using supervised learning in the traditional way and then we briefly review more recent deep learning approaches.
The new data representation can be estimated using knowledge of the labels (supervised way) or without such knowledge (unsupervised). In other words, although the final classification or regression model is supervised, the transformation to a new representation can be unsupervised. Examples include the dimensionality reduction methods reviewed in section 3.2.1, such as PCA (51–53) or non-linear manifold learning (53, 54). The use of hand-crafted features such as volumes/diameters/strains (55) and radius/thickness (56, 57) also falls into this category, although one could argue that knowledge of the task was also used to design these features. A supervised approach was taken in Dawes et al. (58), in which supervised PCA was used to find the principal components of displacement data related to survival.
Classification or regression come once the new representation is obtained. Many classification algorithms have been used, including support vector machines (SVM) (55, 59), random forests (55), variants of dictionary learning (59–61) and ridge logistic regression (57). Regression applications rely on svm (62) and multiscale kernel regression (54).
Recent research has increasingly focused on deep learning for both classification and regression from dynamic imaging data. In these approaches, the activations of intermediate network layers can stand as a transformed representation formed in a supervised way. Inputs to these models are commonly dynamic image intensity data, but segmentation data has also been used (63). For classification, variants of auto-encoders have been a common architecture choice. An auto-encoder is a deep learning-based dimensionality reduction technique, and classification can be performed in the low-dimensional latent space learnt without supervision (53), or in a supervised way by including classification accuracy into the loss function (48, 63, 64). Auto-encoders are attractive as they allow examining the classification features in the original image space, leading to more interpretable analyses. CNNs have also been proposed for classification (65), and a challenge on automated diagnosis was recently organized (7). Regression tasks such as estimating volume and/or ejection fraction may also involve CNNs (66), as tested on the recent Kaggle Challenge data1. Variational auto-encoders have also been used to perform regression in the latent space (50).
A wide set of classification applications involved myocardial motion or deformation, including identifying abnormal wall motion (59, 61), predicting therapy response (67) and survival (58, 64), and diagnosing myocardial infarct (16, 60, 65, 68) or pathology (7, 48, 57, 63). Regression applications aimed at localizing myocardial infarct (54), grading myocardial motion defects (62), and estimating volumes (66).
Detecting some form of abnormality is a common theme for supervised learning applications, for which two main strategies exist. In the first one, the transformed representation only involves healthy subjects: the distribution of samples in the low-dimensional space therefore represents healthy variations, and subsequent subjects who fall away from the healthy distribution are considered abnormal, as investigated on myocardial velocities (28, 35) and shapes (69). The other strategy learns a low-dimensional representation from both healthy and pathological subjects, where supervised classification can be applied afterwards (70).
4. Specificities of the Clinical Context
4.1. Physiological Consistency
Learning algorithms utilize a low-dimensional representation of the high-dimensional motion/deformation data, where the population variability is either rendered through diagonalization according to inter-subject distances, or correlated to labels of interest. Transforming to and from this representation involves interpolation between samples. Regularizing the low-dimensional space ensures smoother interpolation and generates new samples that are physiologically plausible (49, 71). In both of these works, the low-dimensional space produced by the encoding part of a CNN was regularized to map smoothly to a set of input shapes, labeled images, or slice locations. This notion of joint projection from the image and label space is also inherently present in more classical manifold learning techniques such as partial least squares. Similar notions need to be extended to motion fields, whilst mapping similar pathological conditions to close locations in the latent space.
4.2. Spatiotemporal Analysis
Most learning techniques consider high-dimensional inputs as high-dimensional column vectors or a set of patches, and disregard the spatiotemporal characteristics of motion and deformation. Few works explicitly addressed this issue for the statistical analysis of populations. A bilinear statistical model was used on cardiac shapes (72) to distinguish inter-subject variations from individual heart dynamics. (73, 74) explicitly addressed the problem through spatiotemporal tensor decomposition. Duchateau et al. (75) tuned up the contributions of the spatial, temporal, and magnitude dimensions to analyze changes in deformation patterns through registration. Jia et al. (31) and Guigui et al. (32) transported temporal trajectories without explicitly extracting motion or deformation descriptors beforehand. These strategies, limited to variability analyses, pave the ground for better considering spatiotemporal aspects with machine learning.
4.3. Interpretability
Many tasks may benefit from somehow “interpretable” learnt models, i.e., a user should have ways to inspect the input data characteristics that led to the output prediction or representation. The recent trend toward more complex learning models (such as deep learning) has raised the interest for this property, since these models are generally harder to interpret than simpler ones. One approach consists in defining a simpler model that is “locally similar” to the global complex model (i.e., it has similar performance for similar inputs) (76). For deep learning based approaches, “saliency maps” can be produced, which show which parts of the input data were important in producing the output. Alternatively, regression or autoencoders can be used to reconstruct cases from the low-dimensional latent space and examine features in the original-high dimensional space, with clear benefits for interpretability as demonstrated in Clough et al. (48), Puyol-Anton et al. (53), Biffi et al. (63), and Bello et al. (64).
4.4. Database Size and Heterogeneity
Traditionally, difficulties in accessing and reliably annotating databases of medical images have led to smaller databases in medical imaging compared to computer vision applications. Recent initiatives such as the UK Biobank project2 (77) now provide large-scale annotated imaging databases, fuelling a rise in more data-intensive methods such as deep learning. Figure 2 illustrates this high increase over recent years for the studies reported in this paper. The impact of these large databases is high: reporting reference ranges for cardiac functional biomarkers is now possible with much greater confidence (78, 79), in addition to detecting effects otherwise hidden with smaller databases, as shown for genome data (77). Data heterogeneity is also crucial when choosing or curating a database for a specific task, i.e., the database should include sufficient subjects to cover a range of values for the output label and guarantee the model generalizability. More pathology-focused databases such as those in the Cardiac Atlas Project3 (80) have an important role to play in this respect.
Figure 2. Database sizes (left) and distribution of imaging modalities, application purposes, and target populations for the studies cited in this paper that use machine learning for myocardial motion or deformation analysis.
4.5. Validation and Standardization Initiatives
As analyzing the tracking output is sensitive to processing errors, in particular for multi-centric data, tracking algorithms should be benchmarked to prevent bias due to different manufacturers or settings/practices. To ensure reproducibility of clinical decision-making from these data, standardization initiatives arose from academic, clinical, and industrial actors of cardiac imaging. Strain estimation was compared across vendors for synthetic and real images (81). Outputs were consistent regarding the differentiation between pathological and healthy regions, and the identification of ambiguous zones. However, statistically significant differences among vendors were reported, including differences around 15% for the biggest scars. These differences call for benchmarks on more realistic datasets (both regarding geometry and image quality), obtained e.g., from simulation frameworks that mix image formation and biomechanical models with real images (82).
Complementary standardization of imaging are also investigated through deep learning, for the control of e.g., the full coverage of the ventricles (83), the view/plane (84, 85), and the image quality in general (78) or due to motion-related artifacts (86).
4.6. Multiple Modalities/Descriptors
Most studies only consider a single type of motion or deformation descriptor at once from a single acquisition and a single modality, unlike clinical reasoning, which repeats acquisitions in the same or different modality and uses different types of measurements and descriptors. Recent works addressed these limitations within the framework of manifold learning. (30) enforced the complementarity of multimodal acquisitions (tagged MR and 3D echocardiography) using canonical correlation analysis and partial least squares methods. (87) used a similar strategy to better relate myocardial shape and deformation descriptors. Puyol-Anton et al. (70) investigated multi-view linear discriminant analysis for classification purposes. Finally, the more generic framework of multiple kernel learning allows reducing the dimensionality and examining the weights attributed to each descriptor. It was applied to supervised (67) and unsupervised (43–45, 88) problems, to investigate multiple descriptors among which motion-based ones, which could come from different modalities or different views of a single modality.
4.7. Complexity of the Models and Data Descriptors
Machine learning relies on models whose complexities should be adjusted to the question being answered. Researchers should keep in mind that such models only provide an approximation of reality, and try to minimize this error (e.g., by refining the model, adding more data or relevant descriptors, or estimating uncertainties). We strongly recommend to start with simple data descriptors and models, and carefully benchmark the retained methods against simpler models or even standard statistics.
Deep learning approaches allow circumventing the design of hand-crafted features (Figure 1B), and therefore go beyond a substantial limitation of standard machine learning. They mainly have been used for supervised problems and avoiding segmentation. The ACDC challenge (7) included a diagnosis challenge not necessarily requiring segmentation, although all participants opted for segmentation-based diagnosis. Regression-based estimation of cardiac parameters directly from images was proposed in (66, 89, 90), and may also strengthen the segmentation-based estimation of such parameters (91). However, as already pointed out, this direct strategy may also limit interpretability, and therefore transfer to clinical practice.
5. Conclusion
Machine learning offers wide possibilities to automate processing, and notably extract and analyze myocardial motion and deformation. Driven by advances in cardiac segmentation and large databases collection, there is potential for substantially improving the characterization of the cardiac function and impacting clinical practice. Changes cover the automation of time-consuming and user-dependent tasks such as feature extraction, higher performance on supervised problems such as (earlier) diagnosis, prognosis, and risk stratification, and new unsupervised data representations for knowledge discovery such as clustering or phenotyping. Nonetheless, motion and deformation are rich but complex high-dimensional data. Efforts need to be continued to reduce uncertainties, approximations, and crucial misinterpretations along the analysis pipeline, from careful problem definition, compliance with the mathematical and physiological data properties, algorithms benchmarking/validation/testing, and health actors' education.
Author Contributions
All authors listed have made a substantial, direct and intellectual contribution to the work, and approved it for publication.
Funding
ND was partially supported by the LABEX PRIMES of Université de Lyon (ANR-11-LABX-0063) and the MIC-MAC JCJC project (ANR-19-CE45-0005). AK was partially supported by the Wellcome EPSRC Centre for Medical Engineering at Kings College London (WT 203148/Z/16/Z) and the UK EPSRC (EP/P001009/1 and EP/R005516/1). MD was partially supported by the European Union Horizon 2020 Programme for Research and Innovation (CardioFunXion, H2020-14-MSCA-ITN-642676).
Conflict of Interest
MD was employed by Philips Research Paris. ND and AK have research publications and/or projects with researchers and engineers from private companies, but this did not influence the contents of this review.
Acknowledgments
The articles discussed in this review were selected by querying PubMed over the last 10 years with the terms (myocardial [OR] cardiac) [AND] learning [AND] (motion [OR] deformation), complemented by the authors' knowledge, and examining the publication profile of the authors of the already selected articles. Papers not using spatial or temporal motion or deformation patterns but single measurements such as peak values or timings, and papers addressing cardiac respiratory motion were removed from this selection, although we acknowledge their importance for the more complete analysis of cardiac function.
Footnotes
1. ^Available online at: https://www.kaggle.com/c/second-annual-data-science-bowl/data
2. ^Available online at: https://www.ukbiobank.ac.uk/
3. ^Available online at: https://www.cardiacatlas.org/
References
1. Cikes M, Solomon S. Beyond ejection fraction: an integrative approach for assessment of cardiac structure and function in heart failure. Eur Heart J. (2016) 37:1642–50. doi: 10.1093/eurheartj/ehv510
2. Bijnens B, Cikes M, Claus P, Sutherland G. Velocity and deformation imaging for the assessment of myocardial dysfunction. Eur J Echocardiogr. (2009) 10:216–26. doi: 10.1093/ejechocard/jen323
3. Bijnens B, Cikes M, Butakoff C, Sitges M, Crispi F. Myocardial motion and deformation: what does it tell us and how does it relate to function? Fetal Diagn Ther. (2012) 32:5–16. doi: 10.1159/000335649
4. D'hooge J, Heimdal A, Jamal F, Kukulski T, Bijnens B, Rademakers F, et al. Regional strain and strain rate measurements by cardiac ultrasound: principles, implementation and limitations. Eur J Echocardiogr. (2000) 1:154–70. doi: 10.1053/euje.2000.0031
5. Amzulescu M, De Craene M, Langet H, Pasquet A, Vancraeynest D, Pouleur A, et al. Myocardial strain imaging: review of general principles, validation, and sources of discrepancies. Eur Heart J Cardiovasc Imaging. (2019) 20:605–19. doi: 10.1093/ehjci/jez041
6. Petitjean C, Dacher J. A review of segmentation methods in short axis cardiac MR images. Med Image Anal. (2011) 15:169–84. doi: 10.1016/j.media.2010.12.004
7. Bernard O, Lalande A, Zotti C, Cervenansky F, Yang X, Heng P, et al. Deep learning techniques for automatic MRI cardiac multi-structures segmentation and diagnosis: is the problem solved? IEEE Trans Med Imaging. (2018) 37:2514–25. doi: 10.1109/TMI.2018.2837502
8. Litjens G, Ciompi F, Wolterink J, de Vos B, Leiner T, Teuwen J, et al. State-of-the-art deep learning in cardiovascular image analysis. JACC Cardiovasc Imaging. (2019) 12:1549–65. doi: 10.1016/j.jcmg.2019.06.009
9. McLeod K, Prakosa A, Mansi T, Sermesant M, Pennec X. An incompressible log-domain demons algorithm for tracking heart tissue. In: International Workshop on Statistical Atlases and Computational Models of the Heart. Toronto, ON (2011). p. 55–67. doi: 10.1007/978-3-642-28326-0-6
10. Perperidis D, Mohiaddin R, Rueckert D. Spatio-temporal free-form registration of cardiac MR image sequences. Med Image Anal. (2005) 9:441–56. doi: 10.1016/j.media.2005.05.004
11. Perrot V, Garcia D. Back to basics in ultrasound velocimetry: tracking speckles by using a standard PIV algorithm. In: 2018 IEEE International Ultrasonics Symposium (IUS). Kobe (2018) p. 206–12. doi: 10.1109/ULTSYM.2018.8579665
12. Ilg E, Mayer N, Saikia T, Keuper M, Dosovitskiy A, Brox T. Flownet 2.0: evolution of optical flow estimation with deep networks. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR). Honolulu, HI (2017) p. 2462–70. doi: 10.1109/CVPR.2017.179
13. Morales MA, Izquierdo-Garcia D, Aganj I, Kalpathy-Cramer J, Rosen BR, Catana C. Implementation and validation of a three-dimensional cardiac motion estimation network. Radiol Artif Intell. (2019) 1:e180080. doi: 10.1148/ryai.2019180080
14. Østvik A, Smistad E, Espeland T, Rye Berg E, Lovstakken L. Automatic myocardial strain imaging in echocardiography using deep learning. Deep Learning in Medical Image Analysis and Multimodal Learning for Clinical Decision Support. DLMIA 2018, ML-CDS 2018. Lecture Notes in Computer Science. Cham: Springer (2018). p. 309–16. doi: 10.1007/978-3-030-00889-5-35
15. Qin C, Bai W, Schlemper J, Petersen S, Piechnik S, Neubauer S, et al. Joint learning of motion estimation and segmentation for cardiac MR image sequences. First MICCAI Workshop, DART 2019, and First International Workshop, MIL3ID 2019, Shenzhen, Held in Conjunction with MICCAI 2019. Shenzhen (2018) 11071:472–80. doi: 10.1007/978-3-030-00934-2-53
16. Xu C, Xu L, Gao Z, Zhao S, Zhang H, Zhang Y, et al. Direct delineation of myocardial infarction without contrast agents using a joint motion feature learning architecture. Med Image Anal. (2018) 50:82–94. doi: 10.1016/j.media.2018.09.001
17. Yan W, Wang Y, van der Geest R, Tao Q. Cine MRI analysis by deep learning of optical flow: adding the temporal dimension. Comput Biol Med. (2019) 111:103356. doi: 10.1016/j.compbiomed.2019.103356
18. Haskins G, Kruger U, Yan P. Deep learning in medical image registration: a survey. arXiv preprint arXiv:190302026 (2019).
19. Ouzir N, Basarab A, Liebgott H, Harbaoui B, Tourneret JY. Motion estimation in echocardiography using sparse representation and dictionary learning. IEEE Trans Image Proc. (2017) 27:64–77. doi: 10.1109/TIP.2017.2753406
20. Krebs J, e Delingette H, Mailhé B, Ayache N, Mansi T. Learning a probabilistic model for diffeomorphic registration. IEEE Trans Med Imaging. (2019) 38:2165–76. doi: 10.1109/TMI.2019.2897112
21. Mansi T, Pennec X, Sermesant M, Delingette H, Ayache N. iLogDemons: a demons-based registration algorithm for tracking incompressible elastic biological tissues. Int J Comput Vis. (2011) 92:92–111. doi: 10.1007/s11263-010-0405-z
22. Heyde B, Alessandrini M, Hermans J, Barbosa D, Claus P, D'hooge J. Anatomical image registration using volume conservation to assess cardiac deformation from 3D ultrasound recordings. IEEE Trans Med Imaging. (2015) 35:501–11. doi: 10.1109/TMI.2015.2479556
23. Shi W, Jantsch M, Aljabar P, Pizarro L, Bai W, Wang H, et al. Temporal sparse free-form deformations. Med Image Anal. (2013) 17:779–89. doi: 10.1016/j.media.2013.04.010
24. De Craene M, Piella G, Camara O, Duchateau N, Silva E, Doltra A, et al. Temporal diffeomorphic free-form deformation: application to motion and strain estimation from 3D echocardiography. Med Image Anal. (2012) 16:427–50. doi: 10.1016/j.media.2011.10.006
25. De Craene M, Camara O, Bijnens BH, Frangi AF. Large diffeomorphic FFD registration for motion and strain quantification from 3D-US sequences. Functional Imaging and Modeling of the Heart. FIMH 2009. Lecture Notes in Computer Science Berlin; Heidelberg: Springer (2009) p. 437–46. doi: 10.1007/978-3-642-01932-6-47
26. Peyrat J, Delingette H, Sermesant M, Xu C, Ayache N. Registration of 4D cardiac CT sequences under trajectory constraints with multichannel diffeomorphic demons. IEEE Trans Med Imaging. (2010) 29:1351–68. doi: 10.1109/TMI.2009.2038908
27. Rao A, Chandrashekara R, Sanchez-Ortiz G, Mohiaddin R, Aljabar P, Hajnal J, et al. Spatial transformation of motion and deformation fields using nonrigid registration. IEEE Trans Med Imaging. (2004) 23:1065–76. doi: 10.1109/TMI.2004.828681
28. Duchateau N, De Craene M, Piella G, Silva E, Doltra A, Sitges M, et al. A spatiotemporal statistical atlas of motion for the quantification of abnormalities in myocardial tissue velocities. Med Image Anal. (2011) 15:316–28. doi: 10.1016/j.media.2010.12.006
29. Duchateau N, De Craene M, Pennec X, Merino B, Sitges M, Bijnens B. Which reorientation framework for the atlas-based comparison of motion from cardiac image sequences? Spatio-temporal Image Analysis for Longitudinal and Time-Series Image Data. STIA 2012. Lecture Notes in Computer Science. Berlin; Heidelberg: Springer (2012) 7570:25–37. doi: 10.1007/978-3-642-33555-6-3
30. Puyol-Anton E, Sinclair M, Gerber B, Amzulescu M, Langet H, De Craene M, et al. A multimodal spatiotemporal cardiac motion atlas from MR and ultrasound data. Med Image Anal. (2017) 40:96–110. doi: 10.1016/j.media.2017.06.002
31. Jia S, Duchateau N, Moceri P, Sermesant M, Pennec X. Parallel transport of surface deformations from pole ladder to symmetrical extension. Shape in Medical Imaging. ShapeMI 2018. Lecture Notes in Computer Science. Cham:Springer (2018). p. 116–24. doi: 10.1007/978-3-030-04747-4-11
32. Guigui N, Jia S, Sermesant M, Pennec X. Symmetric algorithmic components for shape analysis with diffeomorphisms. Processinds GSI LNCS. (2019). p. 759–68. doi: 10.1007/978-3-030-26980-7-79
33. Rohe M, Sermesant M, Pennec X. Low-dimensional representation of cardiac motion using Barycentric Subspaces: a new group-wise paradigm for estimation, analysis, and reconstruction. Med Image Anal. (2018) 45:1–12. doi: 10.1016/j.media.2017.12.008
34. Rougon N, Petitjean C, Preteux F. Building and using a statistical 3D motion atlas for analyzing myocardial contraction in MRI. In: Proceedings SPIE Medical Imaging. San Diego, CA (2004) 5370:253–64. doi: 10.1117/12.535609
35. De Craene M, Duchateau N, Tobon-Gomez C, Ghafaryasl B, Piella G, Rhode K, et al. SPM to the heart: mapping of 4D continuous velocities for motion abnormality quantification. In: 2012 9th IEEE International Symposium on Biomedical Imaging (ISBI). IEEE: Barcelona. (2012) p. 454–7. doi: 10.1109/ISBI.2012.6235582
36. Tabassian M, Alessandrini M, Herbots L, Mirea O, Pagourelias E, Jasaityte R, et al. Machine learning of the spatio-temporal characteristics of echocardiographic deformation curves for infarct classification. Int J Cardiovasc Imaging. (2017) 33:1159–67. doi: 10.1007/s10554-017-1108-0
37. Tabassian M, Sunderji I, Erdei T, Sanchez-Martinez S, Degiovanni A, Marino P, et al. Diagnosis of heart failure with preserved ejection fraction: machine learning of spatiotemporal variations in left ventricular deformation. J Am Soc Echocardiogr. (2018) 31:1272–84.e9. doi: 10.1016/j.echo.2018.07.013
38. Bosch J, Mitchell S, Lelieveldt B, Nijland F, Kamp O, Sonka M, et al. Automatic segmentation of echocardiographic sequences by active appearance motion models. IEEE Trans Med Imaging. (2002) 21:1374–83. doi: 10.1109/TMI.2002.806427
39. Arsigny V, Commowick O, Pennec X, Ayache N. A log-Euclidean framework for statistics on diffeomorphisms. Med Image Comput Comput Assist Interv. (2006) 9(Pt 1):924–31. doi: 10.1007/11866565-113
40. Pennec X. Intrinsic statistics on Riemannian manifolds: basic tools for geometric measurements. J Math Imaging Vis. (2006) 25:127. doi: 10.1007/s10851-006-6228-4
41. Yan S, Xu D, Zhang B, Zhang H, Yang Q, Lin S. Graph embedding and extensions: a general framework for dimensionality reduction. IEEE Trans Pattern Anal Mach Intell. (2007) 29:40–51. doi: 10.1109/TPAMI.2007.250598
42. Duchateau N, De Craene M, Piella G, Frangi A. Constrained manifold learning for the characterization of pathological deviations from normality. Med Image Anal. (2012) 16:1532–49. doi: 10.1016/j.media.2012.07.003
43. Sanchez-Martinez S, Duchateau N, Erdei T, Fraser A, Bijnens B, Piella G. Characterization of myocardial motion patterns by unsupervised multiple kernel learning. Med Image Anal. (2017) 35:70–82. doi: 10.1016/j.media.2016.06.007
44. Sanchez-Martinez S, Duchateau N, Erdei T, Kunszt G, Aakhus S, Degiovanni A, et al. Machine learning analysis of left ventricular function to characterize heart failure with preserved ejection fraction. Circ Cardiovasc Imaging. (2018) 11:e007138. doi: 10.1161/CIRCIMAGING.117.007138
45. Cikes M, Sanchez-Martinez S, Claggett B, Duchateau N, Piella G, Butakoff C, et al. Machine learning-based phenogrouping in heart failure to identify responders to cardiac resynchronization therapy. Eur J Heart Fail. (2019) 21:74–85. doi: 10.1002/ejhf.1333
46. Bengio Y, Paiement J, Vincent P, Delalleau O, Le Roux N, Ouimet M. Out-of-sample extensions for LLE, Isomap, MDS, Eigenmaps, and Spectral Clustering. Advances in Neural Information Processing Systems. Vancouver, BC (2003) p. 177–84.
48. Clough J, Oksuz I, Puyol-Anton E, Ruijsink B, King A, Schnabel J. Global and local interpretability for cardiac MRI classification. In: Proceedings Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Shenzhen (2019). p. 656–64. doi: 10.1007/978-3-030-32251-9-72
49. Painchaud N, Skandarani Y, Judge T, Bernard O, Lalande A, Jodoin P. Cardiac MRI segmentation with strong anatomical guarantees. In: Proceedings Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Shenzhen (2019). p. 632–40. doi: 10.1007/978-3-030-32245-8-70
50. Puyol-Anton E, Ruijsink B, Clough J, Oksuz I, Rueckert D, Razavi R, et al. Assessing the impact of blood pressure on cardiac function using interpretable biomarkers and variational autoencoders. In: Proceedings Statistical Atlases and Computational Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Shenzhen (2019).
51. Bai W, Shi W, de Marvao A, Dawes T, O'Regan D, Cook S, et al. A bi-ventricular cardiac atlas built from 1000+ high resolution MR images of healthy subjects and an analysis of shape and motion. Med Image Anal. (2015) 26:133–45. doi: 10.1016/j.media.2015.08.009
52. Piras P, Teresi L, Puddu PE, Torromeo C, Young AA, Suinesiaputra A, et al. Morphologically normalized left ventricular motion indicators from MRI feature tracking characterize myocardial infarction. Sci Rep. (2017) 7:12259. doi: 10.1038/s41598-017-12539-5
53. Puyol-Anton E, Ruijsink B, Langet H, De Craene M, Piro, Schnabel J, et al. Learning associations between clinical information and motion-based descriptors using a large scale MR-derived cardiac motion atlas. In: Proceedings Statistical Atlases and Computational Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Shenzhen (2019). p. 94–102. doi: 10.1007/978-3-030-12029-0-11
54. Duchateau N, De Craene M, Allain P, Saloux E, Sermesant M. Infarct localization from myocardial deformation: prediction and uncertainty quantification by regression from a low-dimensional space. IEEE Trans Med Imaging. (2016) 35:2340–52. doi: 10.1109/TMI.2016.2562181
55. Narula S, Shameer K, Omar A, Dudley J, Sengupta P. Machine-learning algorithms to automate morphological and functional assessments in 2D echocardiography. J Am Coll Cardiol. (2016) 68:2287–95. doi: 10.1016/j.jacc.2016.08.062
56. Bleton H, Margeta J, Lombaert H, Delingette H, Ayache N. Myocardial infarct localization using neighbourhood approximation forests. In: Proceedings Statistical Atlases and Computational Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Munich (2015). p. 108–16. doi: 10.1007/978-3-319-28712-6-12
57. Zheng Q, Delingette H, Ayache N. Explainable cardiac pathology classification on cine MRI with motion characterization by semi-supervised learning of apparent flow. Med Image Anal. (2019) 56:80–95. doi: 10.1016/j.media.2019.06.001
58. Dawes T, de Marvao A, Shi W, Fletcher T, Watson G, Wharton J, et al. Machine learning of three-dimensional right ventricular motion enables outcome prediction in pulmonary hypertension: a cardiac MR imaging study. Radiology. (2017) 283:381–90. doi: 10.1148/radiol.2016161315
59. Mantilla J, Garreau M, Bellanger J, Paredes J. Machine learning techniques for LV wall motion classification based on spatio-temporal profiles from cardiac cine MRI. In: ProICMLA '13 Proceedings of the 2013 12th International Conference on Machine Learning and Applications. Washington, DC (2013). p. 167–72. doi: 10.1109/ICMLA.2013.36
60. Peressutti D, Bai W, Shi W, Tobon-Gomez C, Jackson T, Sohal M, et al. Towards left ventricular scar localisation using local motion descriptors. In: Proceedings Statistical Atlases and Computational Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Athens (2016). p. 30–9. doi: 10.1007/978-3-319-28712-6-4
61. Mantilla J, Paredes J, Bellanger J, Donal E, Leclercq C, Medina R, et al. Classification of LV wall motion in cardiac MRI using kernel Dictionary Learning with a parametric approach. 2015 37th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC). Milan (2015) p. 7292–5. doi: 10.1109/EMBC.2015.7320075
62. Marin T, Kalayeh M, Pretorius P, Wernick M, Yang Y, Brankov J. Numerical observer for cardiac motion assessment using machine learning. In: Proceedings SPIE Medical Imaging. Lake Buena Vista, FL (2011). p. 79660G. doi: 10.1117/12.878186
63. Biffi C, Oktay O, Tarroni G, Bai W, de Marvao A, Doumou G, et al. Learning interpretable anatomical features through deep generative models: application to cardiac remodeling. In: Proceedings Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Granada (2018). p. 464–71. doi: 10.1007/978-3-030-00934-2-52
64. Bello GA, Dawes TJW, Duan J, Biffi C, de Marvao A, Howard LSGE, et al. Deep-learning cardiac motion analysis for human survival prediction. Nat Mach Intell. (2019) 1:95–104. doi: 10.1038/s42256-019-0019-2
65. Kusunose K, Abe T, Haga A, Fukuda D, Yamada H, Harada M, et al. A deep learning approach for assessment of regional wall motion abnormality from echocardiographic images. JACC Cardiovasc Imaging. (2019). doi: 10.1016/j.jcmg.2019.02.024. [Epub ahead of print].
66. Luo G, Dong S, Wang K, Zuo W, Cao S, Zhang H. Multi-views fusion CNN for left ventricular volumes estimation on cardiac MR images. IEEE Trans Biomed Eng. (2018) 65:1924–34. doi: 10.1109/TBME.2017.2762762
67. Peressutti D, Sinclair M, Bai W, Jackson T, Ruijsink J, Nordsletten D, et al. A framework for combining a motion atlas with non-motion information to learn clinically useful biomarkers: application to cardiac resynchronisation therapy response prediction. Med Image Anal. (2017) 35:669–84. doi: 10.1016/j.media.2016.10.002
68. Suinesiaputra A, Ablin P, Alba X, Alessandrini M, Allen J, Bai W, et al. Statistical shape modeling of the left ventricle: myocardial infarct classification challenge. IEEE J Biomed Health Inform. (2018) 22:503–15. doi: 10.1109/JBHI.2017.2652449
69. Zhang X, Cowan B, Bluemke D, Finn J, Fonseca C, Kadish A, et al. Atlas-based quantification of cardiac remodeling due to myocardial infarction. PLoS ONE. (2014) 9:e110243. doi: 10.1371/journal.pone.0110243
70. Puyol-Anton E, Ruijsink B, Gerber B, Amzulescu MS, Langet H, De Craene M, et al. Regional multi-view learning for cardiac motion analysis: application to identification of dilated cardiomyopathy patients. IEEE Trans Biomed Eng. (2019) 66:956–66. doi: 10.1109/TBME.2018.2865669
71. Oktay O, Ferrante E, Kamnitsas K, Heinrich M, Bai W, Caballero J, et al. Anatomically constrained neural networks (ACNNs): application to cardiac image enhancement and segmentation. IEEE Trans Med Imaging. (2017) 37:384–95. doi: 10.1109/TMI.2017.2743464
72. Hoogendoorn C, Sukno F, Ordas S, Frangi A. Bilinear models for spatio-temporal point distribution analysis: application to extrapolation of left ventricular, biventricular and whole heart cardiac dynamics. Int J Comput Vis. (2009) 85:237–52. doi: 10.1007/s11263-009-0212-6
73. McLeod K, Sermesant M, Beerbaum P, Pennec X. Spatio-temporal tensor decomposition of a polyaffine motion model for a better analysis of pathological left ventricular dynamics. IEEE Trans Med Imaging. (2015) 34:1562–75. doi: 10.1109/TMI.2015.2405579
74. McLeod K, Tondel K, Calvet L, Sermesant M, Pennec X. Cardiac motion evolution model for analysis of functional changes using tensor decomposition and cross-sectional data. IEEE Trans Biomed Eng. (2018) 65:2769–80. doi: 10.1109/TBME.2018.2816519
75. Duchateau N, Giraldeau G, Gabrielli L, Fernandez-Armenta J, Penela D, Evertz R, et al. Quantification of local changes in myocardial motion by diffeomorphic registration via currents: application to paced hypertrophic obstructive cardiomyopathy in 2D echocardiographic sequences. Med Image Anal. (2015) 19:203–19. doi: 10.1016/j.media.2014.10.005
76. Ribeiro M, Singh S, Guestrin C. “Why Should I Trust You?” explaining the predictions of any classifier. In: Proceedings of the 2016 Conference of the North American Chapter of the Association for Computational Linguistics: Demonstrations. San Diego: CA (2016) p. 135–44. doi: 10.18653/v1/N16-3020
77. Littlejohns T, Sudlow C, Allen N, Collins R. UK Biobank: opportunities for cardiovascular research. Eur Heart J. (2019) 40:1158–66. doi: 10.1093/eurheartj/ehx254
78. Ruijsink B, Puyol-Anton E, Oksuz I, Sinclair M, Bai W, Schnabel J, et al. Fully automated, quality-controlled cardiac analysis from CMR: validation and large-scale application to characterize cardiac function. JACC Cardiovasc Imaging. (2019). doi: 10.1016/j.jcmg.2019.05.030. [Epub ahead of print].
79. Attar R, Pereanez M, Gooya A, Alba X, Zhang L, de Vila M, et al. Quantitative CMR population imaging on 20,000 subjects of the UK Biobank imaging study: LV/RV quantification pipeline and its evaluation. Med Image Anal. (2019) 56:26–42. doi: 10.1016/j.media.2019.05.006
80. Fonseca C, Backhaus M, Bluemke D, Britten R, Chung J, Cowan B, et al. The Cardiac Atlas Project - An imaging database for computational modeling and statistical atlases of the heart. Bioinformatics. (2011) 27:2288–95. doi: 10.1093/bioinformatics/btr360
81. D'hooge J, Barbosa D, Gao H, Claus P, Prater D, Hamilton J, et al. Two-dimensional speckle tracking echocardiography: standardization efforts based on synthetic ultrasound data. Eur Heart J Cardiovasc Imaging. (2015) 17:693–701. doi: 10.1093/ehjci/jev197
82. Alessandrini M, Heyde B, Queirós S, Cygan S, Zontak M, Somphone O, et al. Detailed evaluation of five 3D speckle tracking algorithms using synthetic echocardiographic recordings. IEEE Trans Med Imaging. (2016) 35:1915–26. doi: 10.1109/TMI.2016.2537848
83. Zhang L, Gooya A, Pereanez M, Dong B, Piechnik S, Neubauer S, et al. Automatic assessment of full left ventricular coverage in cardiac cine magnetic resonance imaging with fisher discriminative 3D CNN. IEEE Trans Biomed Eng. (2018) 66:1975–86. doi: 10.1109/TBME.2018.2881952
84. Margeta J, Criminisi A, Cabrera Lozoya R, Lee D, Ayache N. Fine-tuned convolutional neural nets for cardiac MRI acquisition plane recognition. Comput Methods Biomech Biomed Engin. (2017) 5:339–49. doi: 10.1080/21681163.2015.1061448
85. Østvik A, Smistad E, Aase S, Haugen B, Lovstakken L. Real-time standard view classification in transthoracic echocardiography using convolutional neural networks. Ultrasound Med Biol. (2019) 45:374–84. doi: 10.1016/j.ultrasmedbio.2018.07.024
86. Oksuz I, Ruijsink B, Puyol-Anton E, Clough J, Cruz G, Bustin A, et al. Automatic CNN-based detection of cardiac MR motion artefacts using k-space data augmentation and curriculum learning. Med Image Anal. (2019) 55:136–47. doi: 10.1016/j.media.2019.04.009
87. Di Folco M, Clarysse P, Moceri P, Duchateau N. Learning interactions between cardiac shape and deformation: application to pulmonary hypertension. In: Proceedings Statistical Atlases and Computational Modelling of the Heart-Medical Image Computing and Computer Assisted Intervention Lecture Notes in Computer Science. Shenzhen (2019).
88. Nogueira M, De Craene M, Sanchez-Martinez S, Chowdhury D, Bijnens B, Piella G. Analysis of nonstandardized stress echocardiography sequences using multiview dimensionality reduction. Med Image Anal. (2019) 60:101594. doi: 10.1016/j.media.2019.101594
89. Zhen X, Wang Z, Islam A, Bhaduri M, Chan I, Li S. Multi-scale deep networks and regression forests for direct bi-ventricular volume estimation. Med Image Anal. (2016) 30:120–9. doi: 10.1016/j.media.2015.07.003
90. Xue W, Brahm G, Pandey S, Leung S, Li S. Full left ventricle quantification via deep multitask relationships learning. Med Image Anal. (2018) 43:54–65. doi: 10.1016/j.media.2017.09.005
Keywords: machine learning, computer-aided diagnosis, myocardial motion, myocardial strain, cardiac imaging
Citation: Duchateau N, King AP and De Craene M (2020) Machine Learning Approaches for Myocardial Motion and Deformation Analysis. Front. Cardiovasc. Med. 6:190. doi: 10.3389/fcvm.2019.00190
Received: 14 October 2019; Accepted: 12 December 2019;
Published: 09 January 2020.
Edited by:
Steffen Erhard Petersen, Queen Mary University of London, United KingdomReviewed by:
Antonio De Marvao, Imperial College London, United KingdomWenjia Bai, Imperial College London, United Kingdom
Alireza Sojoudi, University of Calgary, Canada
Copyright © 2020 Duchateau, King and De Craene. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Nicolas Duchateau, bmljb2xhcy5kdWNoYXRlYXUmI3gwMDA0MDtjcmVhdGlzLmluc2EtbHlvbi5mcg==