Abstract
Modern facilities for high-throughput phenotyping provide plant scientists with a large amount of multi-modal image data. Combination of different image modalities is advantageous for image segmentation, quantitative trait derivation, and assessment of a more accurate and extended plant phenotype. However, visible light (VIS), fluorescence (FLU), and near-infrared (NIR) images taken with different cameras from different view points in different spatial resolutions exhibit not only relative geometrical transformations but also considerable structural differences that hamper a straightforward alignment and combined analysis of multi-modal image data. Conventional techniques of image registration are predominantly tailored to detection of relative geometrical transformations between two otherwise identical images, and become less accurate when applied to partially similar optical scenes. Here, we focus on a relatively new technical problem of FLU/VIS plant image registration. We present a framework for automated alignment of FLU/VIS plant images which is based on extension of the phase correlation (PC) approach − a frequency domain technique for image alignment, which relies on detection of a phase shift between two Fourier-space transforms. Primarily tailored to detection of affine image transformations between two structurally identical images, PC is known to be sensitive to structural image distortions. We investigate effects of image preprocessing and scaling on accuracy of image registration and suggest an integrative algorithmic scheme which allows to overcome shortcomings of conventional single-step PC by application to non-identical multi-modal images. Our experimental tests with FLU/VIS images of different plant species taken on different phenotyping facilities at different developmental stages, including difficult cases such as small plant shoots of non-specific shape and non-uniformly moving leaves, demonstrate improved performance of our extended PC approach within the scope of high-throughput plant phenotyping.
1. Introduction
In recent years, plant phenotyping became an indispensable analytical tool in quantitative plant sciences. Modern multi-camera systems such as LemnaTec-Scanalyzer3D (LemnaTec GmbH, Aachen, Germany) enable acquisition of large amount of multi-modal image data, including visible light (VIS), fluorescence (FLU), and near-infrared (NIR) images. To derive reliable quantitative traits of plant morphology, development and functions from large amount of multi-modal image data, efficient algorithmic solutions for detection and quantification of plant structures are required (Minervini et al., 2015).
Quantitative analysis of plant images begins with image segmentation which aims to identify image regions corresponding to whole plant or particular plant organs. Reliability of phenotypic plant traits essentially depends on accuracy and robustness of image segmentation algorithms. Straightforward segmentation of VIS plant images by means of global thresholding is often hampered by a number of natural and technical reasons including variable plant coloring, inhomogeneous illumination, shadows and reflections in plant and background regions. Differently from VIS, intensity of FLU images strongly correlates with chlorophyll content of plant structures which provides a natural contrast to chlorophyll-free background regions. Higher contrast between intensity of plant and background regions makes fluorescent images to a natural reference for detection and segmentation of plant structures. Once appropriately aligned, the binary mask of segmented FLU images can be applied for segmentation of VIS images. Such a segmentation-via-registration scheme has a considerable advantage of being generic and avoids diverse difficulties by the segmentation of structurally more complex and variable VIS images, see Figure 1.
Figure 1
Two images of different modalities may, in general, differ by a relative affine transformation (i.e., translation, rotation and scaling), but also structurally. For example, contours of walls, carriers and other light reflecting/absorbing objects in VIS images are typically not present in FLU images, see Figure 2. Consequently, alignment of multi-modal images is associated with the problem of finding correspondences between two structurally non-identical images that exhibit only partial similarities.
Figure 2
A broad spectrum of methods for image registration has been previously developed in context of biomedical and geographic imaging (Zitova and Flusser, 2003; Xiong and Zhang, 2010; Lahat et al., 2015; Brock et al., 2017; Goshtasby, 2017). To establish correspondences between two images, manually or automatically generated landmarks (spatial feature-points), intensity information or frequency-domain features were used. The frequency-space based techniques such as Fourier-Mellin phase correlation (PC) rely on the Fourier-shift theorem, which enables detection of a spatial shift in Cartesian or polar systems of coordinates from the phase-shift of their Fourier transforms (Kuglin and Hines, 1975; Reddy and Chatterji, 1996; Wolberg and Zokai, 2000). From previous works (Stone et al., 2001; Foroosh et al., 2002; Argyriou and Vlachos, 2006), it is known that PC is surprisingly robust with respect to noise, but becomes less accurate in presence of multiple structurally similar patterns or considerable structural distortions such as non-rigid image transformations (e.g., deformation, non-uniform motion, etc.). Requirements of additional pre-processing steps by applying PC for registration of non-identical and multi-modal images were reported in Wisetphanichkij and Dejhan (2005), Wang et al. (2013), Gladilin and Eils (2015), and Almonacid-Caballer et al. (2017).
Applications of image registration techniques in context of plant image analysis are still relatively scarce (De Vylder et al., 2012; Raza et al., 2015). Structural differences between multi-modal plant images and presence of non-uniform image motion due to uncorrelated movements of leaves make alignment of multi-modal plant images a challenging task. Here, we are concerned with investigation of diverse facets of multi-modal plant image alignment and suggest extensions to the conventional single-step PC approach for improved robustness and accuracy of FLU/VIS image registration.
2. Methods
2.1. Image acquisition
Time-series of VIS and FLU top-/side-view images of developing maize, wheat and arabidopsis shoots were acquired from high-throughput experiments performed over more than 2 weeks using LemnaTec-Scanalyzer3D high-throughput phenotypic platforms (LemnaTec GmbH, Aachen, Germany). In the highest expansion stage, the LemnaTec Scanalyzer3D consists of three measuring boxes, each equipped with one (or more) different sensor system. Following a measuring plan, plants are moved automatically from the greenhouse to the measuring facility where they are successively transported from one measuring box (e.g., VIS) to the next one (e.g., FLU). Corresponding VIS and FLU images are therefore taken within few seconds one after another, which are required to move the plants from the VIS to the FLU measuring box, respectively. Table 1 summarizes image data modalities and formats used in this study.
Table 1
| Species, views | # Plants | # Days | # Angles | # VIS/FLU pairs | VIS size | FLU size |
|---|---|---|---|---|---|---|
| Arabidopsis, top | 4 | 20 | 1 | 80 | 2,056 × 2,454 | 1,234 × 1,624 |
| Wheat, side | 4 | 47 | 3 | 564 | 1,234 × 1,624 | 1,234 × 1,624 |
| Maize, side | 6 | 22 | 4 | 526 | 2,056 × 2,454 | 1,038 × 1,390 |
An overview of image data used in this study including three different experiments of three different species, each taken in visible light and fluorescence, obtained by three different LemnaTec high-throughput phenotyping facilities for large, intermediate size, and small plants at the IPK Gatersleben.
2.2. Image preprocessing
To increase the robustness of PC calculation, FLU images are uniformly pre-scaled to the height of VIS images prior to affine PC registration. In order to assess effects of structural differences between VIS and FLU images on accuracy and robustness of PC registration, evaluation tests were carried out with original as well as manually segmented images. Manual segmentation was performed using variable cut-off thresholds for different background regions followed by a subsequent manual removal of remaining structural artifacts. Since PC is known to rely on edge information, edge images were generated using color-edge algorithm (Henriques, 2010) and used in addition to grayscale images for finding global affine transformations. Furthermore, image scaling and cropping was introduced to investigate effects of absolute and relative image size on accuracy and robustness of PC registration. In cropped images, the crop-mask was defined by the dimension of the bounding box of all manually segmented plant structures for a particular day of experiment, i.e., the developmental stage of the plant. No further preprocessing steps were applied with exception of Arabidopsis images, where blue-dominant pixels were removed to eliminate the blue mat used for improvement of contrast in top view images of small plants.
2.3. Affine image alignment using fourier-mellin phase correlation
Phase correlation between each two images is computed as Fourier inverse of the normalized cross-power spectrum (CPS):
where
are the complex Fourier transforms of the images A and B and
is the so-called cross-power spectrum (CPS). According to the Fourier shift theorem, relative displacement (Δx, Δy) in the Cartesian system or coordinates (or, alternatively, scaling and rotation in the polar system of coordinates) between two otherwise structurally identical images, i.e.,
leads to phase-shift in the frequency domain
where and is N×M are the image dimensions. As a consequence, the cross power spectrum between two identical images with a relative shift in the Cartesian system of coordinates (or scaled/rotated in the polar system of coordinates) describes the phase-shifts of the Fourier transform in the frequency domain:
For two identical images with the relative spatial displacement (Δx, Δy), the inverse Fourier integral of (6) represents a N×M map exhibiting a single singularity at the point (x = Δx, y = Δy)
This means that the maximum peak of phase correlation between two identical images yields the relative image translation in the Cartesian system of coordinates, or their relative scaling and rotation in polar coordinates, see examples in Figure 3.
Figure 3
Calculation of affine image transformations from Fourier-Mellin phase correlation was performed using a modified version of the MATLAB imregcorr routine which in addition to the affine transformation matrix returns the height of the maximum PC peak. For assessment of reliability of image transformation, a fixed threshold of H > 0.03 was used as suggested in Reddy and Chatterji (1996). Transformations obtained with H < 0.03 typically indicate a failure of PC registration, for example, due to low and missing structural similarities between two images.
2.4. Evaluation of image registration
To evaluate the results of image registration two criterions for characterization of algorithmic robustness and accuracy are introduced.
2.4.1. Success rate of image registration
The success rate (SR) of image registration is calculated as the ratio between the number of successfully performed image registrations (ns) divided by the total number of registered image pairs (n):
Thereby, the criterion of successful image alignment was defined by the minimum admissible height of the maximum PC peak (H > 0.03) as suggested by Reddy and Chatterji (1996) as well as reasonable bounds of image translation, rotation and scaling. Geometrical transformations that do not match these criterions were treated as failure of PC registration.
2.4.2. Overlap ration of registered image regions
The second criterion is constructed to quantify the overlap ratio (OR) between the area of plant regions in VIS images that are covered by the registered FLU image (ar) and the total area of manually segmented plant regions (a):
While SR serves as a criterion indicating that PC routine succeed in producing some reasonable transformation, OR describes the accuracy of successful transformations.
3. Results
3.1. Single-step PC registration of full-size images
First, PC registration of original, full-size FLU and VIS images of maize, wheat and arabidopsis shoots was performed using the conventional single-step PC approach. Thereby, eight different preprocessing variants including
Gray-scale version of unprocessed full-size VIS/FLU images
Color-edges version of unprocessed full-size VIS/FLU images
Gray-scale version of unprocessed full-size and adaptively cropped VIS/FLU images
Color-edges version of unprocessed full-size and adaptively cropped VIS/FLU images
Gray-scale version of manually segmented full-size VIS/FLU images
Color-edges version of manually segmented full-size VIS/FLU images
Gray-scale version of manually segmented full-size and adaptively cropped VIS/FLU images
Color-edges version of manually segmented full-size and adaptively cropped VIS/FLU images
were compared. To assess the performance of PC registration for different preprocessing conditions, cumulative statistics of successful image alignment was calculated for all days of each experiment. As one can see from Figure 4A, manual segmentation significantly improves the success rate of PC registration. Surprisingly, cropping of plant regions does not always improve and sometimes even worsens the PC performance. This rather unexpected result could be traced back to higher probability of misalignment of partially similar plant structures with the larger relative size in relationship to the size of (cropped) image. This was, in particular, observed in juvenile arabidopsis plants with only a few similar leaves. The relationship between the size of plant structures and the image size has, in turn, an impact on their spectral representation, i.e., different weights of lower and higher frequencies, which, in the case of partially similar, blurry and/or repetitive pattern can lead to maximization of PC peak related to locally optimal alignment. An example of such a case is shown in Figures 4B–F. We found that image downscaling can help to avoid such misalignments and to enhance the PC peak corresponding to globally optimal image registration, cf. Figure 4E vs. Figure 4F.
Figure 4
3.2. Effects of downscaling on robustness of PC image alignment
In order to systematically analyzed the effects of image downscaling on robustness of PC registration, tests with downscaled images in the range of scaling factors between [0.1, 1.0] and the step-size 0.02 were performed. Plots in Figure 4G show the success rate (Equation 8) and the overlap ratio (Equation 9) as a function of scale factor. As one can see, downscaling improves both accuracy and robustness of PC registration. However, the robust algorithmic performance is achieved in the range of intermediate scaling factors [0.3, 0.6] that probably correspond to the optimal degree of image smoothing. Detailed analysis of geometrical transformations calculated for differently scaled images reveals that they correspond to optimal registration of some but not all leaves. Consequently, all components of the affine transformation matrix that stand for the relative image scaling, translation, and rotation undergo variations, see Figure 5A. This sort of locally-optimal alignment is particularly evident for plants exhibiting a non-uniform motion, for example, due to uncorrelated leaf movements that occur, for example, shortly after abrupt stop of carriers, e.g., after movements or rotations.
Figure 5
3.3. Integration of multiple PC registrations into a single mask
Since downscaling of images with different scaling factors results in slightly different geometrical transformations that tend to be locally- but not globally-optimal, integration of a series of PC registrations into a single registration mask was introduced. Figure 5B shows examples of single-step locally-optimal image alignments followed by their integration into a single integrated mask (see the right raw). Using the iterative PC strategy, the overlap ratio of 100% between integrated FLU mask and VIS regions was achieved for all images of three different experiments with arabidopsis, wheat, and maize shoots.
3.4. Dependency of PC performance on plant growth
As the accuracy of PC registration is essentially dependent on unique spectral characteristics of target plant structures, a reduced PC performance was observed for young plant shoots exhibiting redundant shapes (e.g., thin vertical lines, blobs, etc.). Similar to the problem of multiple similar leaves, non-specific shape of plant shoots causes ambiguity and inaccuracy of the PC image alignment. Figure 6A summarizes success rates of the PC registration calculated for three age/growth phases of arabidopsis, wheat and maize phenotyping experiments including young, intermediate stage and adult plant shoots. As one can see, success rate of the PC registration of wheat and maize shoots gradually improves with the plant age (i.e., phase of experiment). Figure 6B gives examples of successful and failed image registration of young and adult maize shoots. From certain views (here, for example, the rotation degree 45°), young maize shoots exhibit a non-specific shape (“thin vertical line”) similar to some non-plant structures (e.g., boundaries of carriers, background markers, etc.). Obviously, it is the combination of several factors (i.e., optical plant appearance (shape/size) at certain developmental stages from certain views, and the presence of non-plant background structures) which causes dependency of the PC performance on plant age/growth in our setup.
Figure 6
4. Conclusion
Here, we approached the problem of multi-modal plant image registration using the Fourier-Mellin phase correlation technique. We began this explorative study with assumption that FLU/VIS image registration can be performed using a global affine image transformation. Our investigations showed, however, that structural differences and non-uniform image motion between FLU/VIS plant images require substantial extensions of the conventional single-step PC approach. Our experimental tests with large amount of different plant images confirm previous observations that PC registration of multi-modal non-identical images is sensitive to structural noise and ambiguous image content which can be caused by repetitive self-similar plant structures, combination of young shoots with non-specific shape and background structures, image blurring or non-uniform motion due to frequently observed inertial leaf movements. Some of these problems can be avoided by optimization of the optical scene and the measurement protocol. For example, homogenization and elimination of complexity of background regions as well as longer relaxation times after relocation of plants from VIS to FLU chambers will certainly be helpful. We demonstrate that the accuracy of PC registration can be improved when PC is applied to appropriately preprocessed and downscaled images that exhibit higher degree of structural similarity such as color-edge and background-filtered images. In contrast, cropping of target regions may be counterproductive as it enhances spectral differences between non-identical images and makes phase correlation rather noisy. Strictly speaking, non-uniform image motion represents a non-rigid image transformation which goes beyond the scope of applicability of the affine PC-based registration. To overcome this limitation, we introduced an extension to the conventional single-step PC which is based on integration of a series of locally-optimal PC registrations resulting from alignment of differently scaled images. Suggested iterative scheme for calculation of an integrated registration mask turned out to provide a significantly better overlap between registered FLU and VIS images in the case of non-uniform leaf motion. The disadvantage of the present algorithmic implementation consists in computationally inefficient search for different locally-optimal image transformations in the scale space. Alternative algorithmic approaches are required for a more efficient detection of the relevant peaks of a noisy phase correlation. In summary, our extended PC scheme represents a promising approach to fully automated alignment and segmentation of optically complex and heterogeneous multi-modal plant images suitable for application within the scope of high-throughput plant image analysis and phenotyping.
Statements
Author contributions
MH and EG conceived, designed and performed the computational experiments, analyzed the data, wrote the paper, prepared figures and tables, and reviewed drafts of the paper. AJ and KN executed the laboratory experiments, acquired image data, co-wrote the paper, and reviewed drafts of the paper. TA co-conceptualized the project, and reviewed drafts of the paper.
Funding
This work was performed within the German Plant-Phenotyping Network (DPPN) which is funded by the German Federal Ministry of Education and Research (BMBF) (project identification number: 031A053).
Conflict of interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
References
1
Almonacid-CaballerJ.Pardo-PascualJ. E.RiuzL. A. (2017). Evaluating fourier cross-correlation sub-pixel registration in landsat images. Remote Sens.9:1051. 10.3390/rs9101051
2
ArgyriouV.VlachosT. (2006). A study of sub-pixel motion estimation using phase correlation, in Proc. of British Machine Vision Conference (Edinburgh, UK), 387–396.
3
BrockK. K.MuticS.McNuttT. R.LiH.KesslerM. L. (2017). Use of image registration and fusion algorithms and techniques in radiotherapy: report of the AAPM Radiation Therapy Committee Task Group No. 132. Med. Phys.44, e43–e76. 10.1002/mp.12256
4
De VylderJ.DouterloigneK.PrinceG.Van Der StraetenD.PhilipsW. (2012). A non-rigid registration method for multispectral imaging of plants, in Proc. of SPIE Sensing for Agriculture and Food Quality and Safety IV, Vol. 8369 (Baltimore, MD), 6.
5
ForooshH.ZerubiaJ. B.BerthodM. (2002). Extension of phase correlation to subpixel registration. IEEE Trans. Image Process.11, 188–200. 10.1109/83.988953
6
GladilinE.EilsR. (2015). On the role of spatial phase and phase correlation in vision, illusion, and cognition. Front. Comput. Neurosci.9:45. 10.3389/fncom.2015.00045
7
GoshtasbyA. A. (2017). Theory and Applications of Image Registration. Hoboken, NJ: John Wiley & Sons.
8
HenriquesJ. F. (2010). COLOREDGES: Edges of a Color Image by the Max Gradient Method. Available online at: https://de.mathworks.com/matlabcentral/fileexchange/28114
9
KuglinC. D.HinesD. C. (1975). The phase correlation image alignment method, in Proc. of Intl. Conf. on Cybernetics and Society, Vol. 1 (New York, NY), 163–165.
10
LahatD.AdaliT.JuttenC. (2015). Multimodal data fusion: an overview of methods. Proc. IEEE103, 1449–1477. 10.1109/JPROC.2015.2460697
11
MinerviniM.ScharrH.TsaftarisS. A. (2015). Image analysis: the new bottleneck in plant phenotyping. IEEE Signal Process. Mag.32, 126–131. 10.1109/MSP.2015.2405111
12
RazaS.SanchezV.PrinceG.ClarksonJ. P.RajpootN. M. (2015). Registration of thermal and visible light images of diseased plants using silhouette extraction in the wavelet domain. Pat. Recogn.48, 2119–2128. 10.1016/j.patcog.2015.01.027
13
ReddyB. S.ChatterjiB. N. (1996). An FFT-based technique for translation, rotation, and scale-invariant image registration. IEEE Trans. Image Process.5, 1266–1271.
14
StoneH. S.OrchardM. T.ChangE. C.MartucciS. A. (2001). A fast direct Fourier-based algorithm for subpixel registration of images. IEEE Trans. Geosci. Remote Sens., 39, 2235–2243. 10.1109/36.957286
15
WangJ.XuZ.ZhangJ. (2013). Image registration with hyperspectral data based on Fourier-Mellin transform. Int. J. Signal. Process. Syst.1, 107–110. 10.12720/ijsps.1.1.107-110
16
WisetphanichkijS.DejhanK. (2005). Fast Fourier transform technique and affine transform estimation-based high precision image registration method. GESTS Intl. Trans. Comp. Sci. Eng.20:179.
17
WolbergG.ZokaiS. (2000). Robust image registration using Log-Polar transform, in Proc. of IEEE Intl. Conf. on Image Process., Vol. 1 (Vancouver, BC), 493–496.
18
XiongZ.ZhangJ. (2010). A critical review of image registration methods. Intl. J. Image Data Fusion1, 137–158. 10.1080/19479831003802790
19
ZitovaB.FlusserJ. (2003). Image registration methods: a survey. Image Vis. Comput.21, 977–1000. 10.1016/S0262-8856(03)00137-9
Summary
Keywords
high-throughput plant phenotyping, automated image analysis, multi-modal image registration, affine transformations, non-uniform motion, Fourier-Mellin phase correlation
Citation
Henke M, Junker A, Neumann K, Altmann T and Gladilin E (2018) Automated Alignment of Multi-Modal Plant Images Using Integrative Phase Correlation Approach. Front. Plant Sci. 9:1519. doi: 10.3389/fpls.2018.01519
Received
23 May 2018
Accepted
27 September 2018
Published
16 October 2018
Volume
9 - 2018
Edited by
Stefano Santabarbara, Consiglio Nazionale Delle Ricerche (CNR), Italy
Reviewed by
Shigeichi Kumazaki, Kyoto University, Japan; Jin Chen, University of Kentucky, United States; Victor Sanchez, University of Warwick, United Kingdom
Updates
Copyright
© 2018 Henke, Junker, Neumann, Altmann and Gladilin.
This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Michael Henke mhenke@ipk-gatersleben.deEvgeny Gladilin gladilin@ipk-gatersleben.de
This article was submitted to Plant Physiology, a section of the journal Frontiers in Plant Science
Disclaimer
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations, or those of the publisher, the editors and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.