Hongwei Chen
Dong Li1 Introduction
Data centers (DCs) are computing structures housing a large number of information and communications technology devices installed for processing, storing, and transmitting information (Khalaj and Halgamuge, 2017). In recent years, the size and number of DCs have been growing dramatically (Zhang, et al., 2018). According to statistics, as of the end of 2017, there were 285,000 DCs in use in China, and the equipment in DCs consumed more than 120 billion kWh of electricity annually, accounting for 2% of China’s total electricity consumption. Based on this trend, it is predicted that by 2022, China’s DCs will consume more than 250 billion kWh of electricity (Gong et al., 2022). Cooling systems consume 30–50% of the total DC energy consumption (Zhang et al., 2014), and failure to reduce cooling system energy consumption will undoubtedly result in a huge energy waste for the growing DC demand.
In the meantime, with the rapid development of new high-performance chips, the heat flux of the servers is increasing. If the heat of the servers cannot be released in time, the computing efficiency and stability of the chip will be affected and even cause the failure of electronic equipment (Murshed and Castro, 2017). Traditional air cooling can no longer meet the cooling needs of DCs; thus, it is necessary to adopt efficient cooling technology to ensure the safe operation of electronic equipment. Liquid cooling technology uses liquid as a refrigerant to transfer heat through direct or indirect contact with the heating element. Compared with air, the volumetric heat capacity of water is about 3500 times higher and the thermal conductivity is 25 times higher (Zimmermann et al., 2012), so for high-density servers, liquid cooling has become the obvious choice for DCs.
2 Current Research Status of Liquid-Cooled DCs
2.1 Classification of Liquid Cooling Technology
At present, the liquid cooling technology for the servers in DCs mainly includes immersion liquid cooling, cold plate liquid cooling, and spray liquid cooling. The difference lies mainly in the different ways of heat dissipation. Immersion liquid cooling is to completely soak the server in the cooling liquid and take away the heat of the heating element by single- or two-phase cooling. The cold plate liquid cooling is to connect the liquid cooling plate with the high-power heating element of the server, and other low-power heating elements are assisted by air cooling to achieve the overall cooling of the server. The spray liquid cooling is used to spray the cooling liquid on the surface of the heating element for heat dissipation. There are three main types of liquid cooling solutions that are discussed in the following subsections.
2.1.1 Immersion Liquid Cooling
According to whether the coolant undergoes phase change, the immersion liquid cooling is generally divided into single- and two-phase immersion liquid cooling. As shown in Figure 1A for single-phase immersion liquid cooling, the low-temperature coolant is sent to the heat exchanger by the circulation pump after absorbing the heat from the heating elements of the server using sensible heat. Then, it re-enters the liquid cooling tank to cool the server when the coolant is cooled by warm water in the heat exchanger. Coolant control is relatively simple in single-phase liquid cooling, and coolant loss is low with good sealing, thereby eliminating the demand for frequent coolant replenishment. In the two-phase immersion liquid cooling system, the server is immersed in a liquid cooling tank containing low-boiling-point cooling liquid. As shown in Figure 1B, when the ambient heat reaches certain conditions, the cooling liquid will use latent heat to absorb the heat and produce a boiling phase change to cool down the equipment. The cooling liquid vapor is condensed by the condenser tube to the liquid state and then returned to the liquid cooling tank (Xie et al., 2022).
FIGURE 1. Schematic diagram of liquid cooling technology for DCs: (A) single-phase immersion liquid cooling, (B) two-phase immersion liquid cooling, (C) cold plate liquid cooling, and (D) spray liquid cooling.
2.1.2 Cold Plate Liquid Cooling
Generally speaking, the cold plate liquid cooling mostly solves the heat dissipation of the devices with high heat generation in the server, such as the central processing unit (CPU), while other devices with low heat generation, such as hard disks and main memory, still rely on air cooling. The cold plate liquid cooling mainly transfers the heat from the heating element of the server to the circulating coolant indirectly through the liquid cooling plate. The circulation process is shown in Figure 1C. First, a manifold is arranged on the liquid cooling cabinet to convey the coolant to each liquid cooling computing node. Second, the inlet and outlet are connected with quick connectors, and the main connector is connected to the built-in or external coolant distribution unit (CDU). Third, the CDU is connected with the outdoor cooling unit, thus completing the liquid cooling cycle of the whole cabinet. Compared with traditional air-cooled servers, this technology significantly improves the energy utilization efficiency of DCs, with lower noise, and the installation and maintenance are basically the same as that of air-cooled servers (Kheirabadi and Groulx, 2016).
2.1.3 Spray Liquid Cooling
Spray liquid cooling is a liquid cooling technology in which the server is modified to deploy the corresponding spray device, and the coolant is directly sprayed on the surface of the heating element through the nozzle to absorb heat and is then discharged. Because it is a targeted cooling technology, compared with immersion liquid cooling, the required dose of coolant is lower, and the implementation is more economical. As shown in Figure 1D, the sprayed liquid cooling system mainly includes a liquid cooling cabinet, heat exchanger, circulation pump, and outdoor cooling unit. The liquid cooling cabinet is connected to the heat exchanger through piping, i.e., the heat of the heat-generating components of the server in the cabinet is absorbed by the coolant and cooled down in the heat exchanger. Sprayed liquid cooling has the features of high device integration, high energy efficiency, and low noise. Compared with air-cooled DCs, the total energy consumption of sprayed liquid-cooled DCs can be reduced by 25.8% (Kandasamy et al., 2022).
2.2 Application of Liquid Cooling Technology
Among the three liquid cooling methods, cold plate liquid cooling is the most widely used. In the cold plate liquid cooling system, the CDU plays a crucial role, not only by providing circulating coolant for the system but also by transporting heat from the cabinet cooling system to the outdoor cooling unit, and the temperature of the coolant can be flexibly regulated to meet different cooling requirements. Since CDU failure may lead to liquid cooling system paralysis, the equipment in CDU should be considered for redundant setup, and the circulating pump as the core component is generally used for one use and one standby. The current research on cold plate liquid cooling focuses on strengthening the heat transfer technology and designing efficient micro-channel heat sinks for heat-generating components (Kheirabadi and Groulx., 2016). There have been many application cases of cold plate liquid cooling. For example, IBM Power 575 and 775 supercomputers, using liquid-cooled cold plates connected directly to the processor and air-cooled auxiliary components, have achieved better energy-saving effect (Ellsworth et al., 2012). In addition, Beijing water-cooled supercomputing center uses a room temperature water cooling technology to solve the memory cooling problem of the DC, which reduces the DC energy consumption and makes it reach the highest energy-saving level (Wang et al., 2021). Furthermore, the all-liquid cooling solution from Huawei, which uses liquid technology, has better realized the high-density deployment of information technology (IT) equipment and improved the energy use efficiency of DCs (Wang et al., 2019), etc.
Immersion liquid cooling not only has high heat transfer efficiency but also avoids local hot spots. It is currently the most effective technical means to solve the problem of high-performance server heating, which can increase the power of a single cabinet to 100 kW or even more than 200 kW. Research on immersion systems is observed to be growing in the 21st century. In 2009, Green Revolution Cooling rebooted the concept of open bath immersion cooling by bringing commercial bath immersion systems to the high-performance computing industry. Moreover, the Open Compute Project officially embraced a new project under the Rack & Power as part of Advanced Cooling Solutions in 2018, while the first documented industry standard for immersion systems was presented at the Open Compute Project summit in San Jose in 2019. Midas Green Technologies designed and built the first immersive DC during the same year (Pambudi et al., 2022). In China, Sugon has launched an immersion cooling server, which can reduce the power usage effectiveness (PUE) of the DC room to 1.05 and reduce energy consumption by more than 30% compared to traditional air-cooled DC. In addition, Alibaba’s “Kirin” server does not require air conditioning, fans, or other accessories, and it can be deployed anywhere on the ground, thereby saving more than 75% of space. In the research on cooling media, the U.S. company 3M has developed a coolant called Novec, which has a lower boiling point than pure water, fluorinated liquid, and mineral oil (Zhao et al., 2021).
Nozzle design is the key issue in the design of spray liquid cooling system, as the nozzle manufacturing process and nozzle structure affect the spray parameters and cooling performance (Chen et al., 2022). The heat transfer mechanism in the spray cooling process is more complex, while the theoretical basis of the research is relatively small and is still at the laboratory stage. Kandasamy et al. developed a lab-scale spray-cooled rack with PF5060 coolant using phase change heat transfer, which can make the temperature distribution on key components such as CPU and graphics processing unit more uniform compared with the cold plate liquid cooling method, and the total energy consumption of spray-cooled DCs is reduced by 25.8% compared with air-cooled DCs (Kandasamy et al., 2022). Due to the limited application scenarios of sprayed liquid cooling technology, only a small number of DCs have adopted this technology, such as the Shanghai Big Data Proving Ground, which uses modular containerized technology and is combined with sprayed liquid cooling technology to achieve an IT load of 284 kW and control the PUE of IT equipment within a stable range, which not only reduces the cost of building a DC but also improves the energy utilization efficiency of the DC. Spray cooling has great potential for application in DCs and thus merits further research to optimize its cooling performance for its application prospects (Wang et al., 2021).
3 Opportunities and Challenges
At present, DC liquid cooling technology is still in the preliminary development stage, and it still faces many problems and challenges. The following points illustrate the development direction of future DC liquid cooling technology application research:
1) Because cold plate liquid cooling has lower requirements for the transformation and adaptation of traditional air-cooled servers, the technology is relatively mature and will represent the mainstream of liquid cooling development in the next few years. Immersion liquid cooling is currently only applicable to new DCs due to its special heat dissipation method, while spray liquid cooling systems are more complex and have the risk of coolant evaporating and escaping, and still need long-term research at the laboratory stage.
2) Liquid-cooled DCs use higher temperature coolants, which makes heat recovery possible for liquid-cooled systems. The coupling of liquid cooling technology and heat recovery will further improve the energy utilization efficiency of DCs, which can achieve broader energy saving and emission reduction by using waste heat for district heating, providing domestic hot water, etc.
3) In the cold plate liquid cooling system, there is often an uneven distribution of the flow of individual servers in a single cabinet, which may lead to local hot spots or wasted energy consumption of individual servers; thus, it is also crucial to study the flow characteristics and optimal design research of each parallel branch pipeline of servers.
4 Conclusion
The application of liquid cooling technology has greatly reduced the energy consumption of DCs, which is an important way to improve the energy efficiency of DCs and one of the future development trends. This study introduced the cooling methods and system components of three liquid cooling technologies and briefly analyzed their applications, pointing out the development direction of liquid cooling technology application research. In the future, the industry should actively pay attention to the changes in liquid cooling technology, actively promote its development, and lay the foundation for the realization of high-performance DC construction.
Author Contributions
DL contributed to the conception of the study. HC wrote the first draft of the manuscript. All authors revised the manuscript and read and approved the submitted version.
Funding
This work was supported by the National Natural Science Foundation of China (No. 52006107) and the Natural Science Foundation of Jiangsu Province (BK20200720).
Conflict of Interest
The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest.
Publisher’s Note
All claims expressed in this article are solely those of the authors and do not necessarily represent those of their affiliated organizations or those of the publisher, the editors, and the reviewers. Any product that may be evaluated in this article or claim that may be made by its manufacturer is not guaranteed or endorsed by the publisher.
References
Chen, H., Ruan, X.-h., Peng, Y.-h., Wang, Y.-l., and Yu, C.-k. (2022). Application Status and Prospect of Spray Cooling in Electronics and Energy Conversion Industries. Sustain. Energy Technol. Assessments 52. doi:10.1016/j.seta.2022.102181
Ellsworth, M. J., Goth, G. F., Zoodsma, R. J., Arvelo, A., Campbell, L. A., and Anderl, W. J. (2012). An Overview of the IBM Power 775 Supercomputer Water Cooling System. J. Electron. Packag. 134, 020906. doi:10.1115/1.4006140
Gong, Y., Zhou, F., Ma, G., and Liu, S. (2022). Advancements on Mechanically Driven Two-phase Cooling Loop Systems for Data Center Free Cooling. Int. J. Refrig. 138, 84–96. in press. doi:10.1016/j.ijrefrig.2022.03.007
Habibi Khalaj, A., and Halgamuge, S. K. (2017). A Review on Efficient Thermal Management of Air- and Liquid-Cooled Data Centers: From Chip to the Cooling System. Appl. Energy 205, 1165–1188. doi:10.1016/j.apenergy.2017.08.037
Kandasamy, R., Ho, J. Y., Liu, P., Wong, T. N., Toh, K. C., and Chua, S. J. (2022). Two-phase Spray Cooling for High Ambient Temperature Data Centers: Evaluation of System Performance. Appl. Energy 305, 117816. doi:10.1016/j.apenergy.2021.117816
Kheirabadi, A. C., and Groulx, D. (2016). Cooling of Server Electronics: A Design Review of Existing Technology. Appl. Therm. Eng. 105, 622–638. doi:10.1016/j.applthermaleng.2016.03.056
Pambudi, N. A., Sarifudin, A., Firdaus, R. A., Ulfa, D. K., Gandidi, I. M., and Romadhon, R. (2022). The Immersion Cooling Technology: Current and Future Development in Energy Saving. Alexandria Eng. J. 61 (12), 9509–9527. doi:10.1016/j.aej.2022.02.059
Sohel Murshed, S. M., and Nieto de Castro, C. A. (2017). A Critical Review of Traditional and Emerging Techniques and Fluids for Electronics Cooling. Renew. Sustain. Energy Rev. 78, 821–833. doi:10.1016/j.rser.2017.04.112
Wang, J. Y., Zhou, B. Y., Zhang, F., Shi, X., Zeng, N., and Liu, Z. Y. (2019). Data Center Energy Consumption Models and Energy Efficient Algorithms. J. Comput. Res. Dev. 56 (8), 1587–1603. doi:10.7544/issn1000-1239.2019.20180574
Wang, Y. S., Zhang, Q., Sun, C., Zhuang, Z. Y., Huang, Z. X., and Zhai, T. Y. (2021). Analysis on the Development of Liquid Cooling Data Center Technology. Electr. Power Inf. Commun. Technol. 19 (12), 69–74. doi:10.16543/j.2095-641x.electric.power.ict.2021.12.010
Xie, L. N., Xing, Y. P., and Lan, B. (2022). Investigation on Key Problems of Liquid Coolant of Immersion Liquid Cooling in Data Center. Inf. Commun. Technol. Policy 48 (3), 40–46. doi:10.12267/j.issn.2096-5931.2022.03.007
Zhang, H., Shao, S., Xu, H., Zou, H., and Tian, C. (2014). Free Cooling of Data Centers: A Review. Renew. Sustain. Energy Rev. 35, 171–182. doi:10.1016/j.rser.2014.04.017
Zhang, K., Zhang, Y., Liu, J., and Niu, X. (2018). Recent Advancements on Thermal Management and Evaluation for Data Centers. Appl. Therm. Eng. 142, 215–231. doi:10.1016/j.applthermaleng.2018.07.004
Zhao, T. T., Wang, L. Y., Zhang, X. Y., Sun, R. F., Xuan, C. B., Geng, W. G., et al. (2021). Research Progress of Immersion Phase-Change Cooling for Data Centers. Huadian Technol. 43 (10), 68–72. doi:10.3969/j.issn.1674-1951.2021.10.008
Keywords: data center, liquid cooling, immersion, cold plate, spray
Citation: Chen H and Li D (2022) Current Status and Challenges for Liquid-Cooled Data Centers. Front. Energy Res. 10:952680. doi: 10.3389/fenrg.2022.952680
Received: 25 May 2022; Accepted: 03 June 2022;
Published: 05 July 2022.
Edited by:
Chengbin Zhang, Southeast University, ChinaReviewed by:
Lingzi Wang, Xi’an Jiaotong University, ChinaCopyright © 2022 Chen and Li. This is an open-access article distributed under the terms of the Creative Commons Attribution License (CC BY). The use, distribution or reproduction in other forums is permitted, provided the original author(s) and the copyright owner(s) are credited and that the original publication in this journal is cited, in accordance with accepted academic practice. No use, distribution or reproduction is permitted which does not comply with these terms.
*Correspondence: Dong Li, NjIwODRAbmpudS5lZHUuY24=