Computer system dependency has been increased in the modern world and that has encouraged the rapid growth of data centers in leading business units like banking, education, transportation, social media and many more. Data center is a facility that incorporates an organisation's IT operations and equipment, as well as where it stores, processes, and manages the data. To fulfill the demands of data storage and data processing, corresponding increase in power density of servers are needed. The data center energy efficiency largely depends on the thermal management of servers. Currently, air cooling is the most widely used thermal management technique in data centers. But air cooling has started to reach its limitations due to high powered microprocessors and packaging. Therefore, industries are looking for single-phase immersion cooling using different dielectric fluids which reduces operational and cooling costs by enhancing the thermal management of servers. Form factor study of 3rd generation open compute server is another area of research in which impact of form factor (geometry of different Open Rack Units) on maximum junction temperature and thermal resistance at the server level is documented. This work is to provide an insight to increase the rack density by reducing form factor of an existing server. This work could open to more heat load per rack. A computational study is conducted in operational range of temperatures and the thermal efficiency has been optimized. A parametric study is conducted by changing the velocities and inlet temperatures of cooling liquid for different heights of the open compute 3rd generation server. The comparative study was carried out for white mineral oil and synthetic fluid (EC100). The results show an enhancement in thermal management for synthetic fluid when compared to mineral oil for the same inlet temperatures. This study clearly indicates that the single-phase immersion cooling is efficient and capable to accommodate high thermal mass.
In today's networking world, utilization of servers and data center has been increasing significantly. Increasing demands of processing and storage of data causes corresponding increase in power density of servers. The data center energy efficiency largely depends on thermal management of servers. Currently, air cooling is the most widely used thermal management technology in data centers. However, air cooling is starting to reach its limits due to very high-powered microprocessors and packaging. To overcome the limitations of air cooling in data centers, operators are moving towards immersion cooling using different dielectric fluids. Thermal shadowing is the effect in which temperature of a cooling medium increases by carrying heat from one server and results in decreasing its heat carrying capacity due to a reduction in the temperature difference between the maximum junction temperature of successive heat sinks and incoming fluid. Thermal Shadowing is a challenge for both air and low velocity oil flow cooling and as such, both air and low velocity dielectric flow cooling technologies need to be optimized to get high energy efficiency. In this study, the impact of Thermal Shadowing between different Dielectric Fluids is compared. The results of dielectric fluids, Mineral Oil and Synthetic Fluid EC100 are compared. The heat sink is a critical part for cooling effectiveness at server level. This work also provides an efficient range of operation for heat sink with computational modelling of third generation open compute server. Optimization of heat sink can allow to cool high-power density servers effectively. A parametric study is conducted, and the thermal efficiency has been optimized.
Complete immersion of servers in synthetic dielectric fluids is rapidly becoming a popular technique to minimize the energy consumed by data centers for cooling purposes. In general, immersion cooling offers noteworthy advantages over conventional air-cooling methods as synthetic dielectric fluids have high heat dissipation capacities which are roughly about 1200 times greater than air. Other advantages of dielectric fluid immersion cooling include even temperature profile on chips, reduction in noise and addressing reliability and operational enhancements like whisker formation and electrochemical migration. Nevertheless, lack of data published and availability of long-term reliability data on immersion cooling is insufficient which makes most of data centers operators reluctant to implement this technique. The first part of this paper will compare thermal performance of single-phase oil immersion cooled HP ProLiant DL160G6 server against air cooled server using computational fluid dynamics on 6 SigmaET®. Focus of the study are major components of the server like Central Processing Unit (CPU), Dual in Line Memory Module (DIMM), Input/output Hub (IOH) chip, Input controller Hub (ICH) and Baseboard Management Controller (BMC). The second part of this paper focuses on thermal performance optimization of oil immersion cooled servers by varying oil and its inlet temperature and volumetric flow rate.
The Datacom facility which comprises rooms or closets used for communication, computers/servers or electronic equipment requires cooling unit which consumes 31% (23% HVAC cooling + 8% HVAC fans) of overall energy. The ASHRAE TC9.9 subcommittee, on Mission Critical Facilities, Data Centers, Technology Spaces, and Electronic Equipment, has suited the data center's executives by permitting brief period outings of the environmental conditions outside the prescribed temperature-humidity range, into passable extents A1-A3. To comprehend the expanding server densities and the required cooling vitality costs, data center operators are falling back on cost cutting measures. For instance, they are not firmly controlling the temperature and humidity levels as per ASHRAE recommended envelope and as a rule turning to airside economizers with the related danger of bringing particulate and gaseous contaminants into their data centers. This dissertation is a first attempt at addressing this challenge by characterizing contamination found in a real-world data center to expand ASHRAE envelope. In situ studies of contaminants found in data centers using air-side economization and corresponding ASHRAE Envelope Expansion in Airside Economization is the main objective of this PhD Dissertation. This study serves several purposes: 1.) Cumulative corrosion damage study for the correlation of equipment reliability to levels of airborne corrosive contaminants and the study of the degree of reliability degradation, when the equipment is operated, outside the recommended range,in the allowable temperature-humidity range in geographies with high levels of gaseous and particulate contamination. 2.) Experimental description of information technology equipment reliability exposed to a data center using airside economizer operating in recommended and allowable ASHRAE envelopes in an ANSI/ISA classified G2 environment to estimate the end of the life of the components and to determine the free air cooling hours for the site. The study took place at the modular data center which uses air-side economizer located at Dallas Industrial area which falls under ISA 71.04-2013 severity level G2. 3.) The servers were removed, and qualitative study of cumulative corrosion damage was carried out. The particulate contaminants were collected from different locations of a server and material characterization was performed using Scanning Electron Microscopy (SEM), Energy Dispersive Spectrometer (EDS) and Fourier Transform Infrared Spectroscopy (FTIR). The analysis from these results helps to explain the impact of the contaminants on IT equipment reliability. 4.) To develop a precise and cost-effective technique to measure deliquescent relative humidity of particulate contaminants found in a data center utilizing airside economization. To develop an experimental technique to measure the DRH of dust particles by logging the leakage current versus %RH for the particulate matter dispensed on an interdigitated comb coupon. To validate this methodology, the DRH of pure salts like MgCl2, NH4NO3 and NaCl is determined and their results are then compared with their published values.This methodology can therefore be implemented to help lay a modus operandi of establishing the limiting value or an effective relative humidity envelope to be maintained at a real-world data center facility for its continuous and reliable operation at its respective location. 5.) The interdigitated comb coupon is lodged with a dust solution in the form of slurry taken from various servers in an actual modular Data Center. The results obtained, and the methodology used in this study can pioneer in standardizing the limiting %RH values for various operating conditions in real world Data Centers, and 6.) This dissertation also examines the flow path of contaminants in high density data centers to determine the most vulnerable location using the computational techniques. Full submersion of servers in dielectric oils offers an opportunity for significant cooling energy savings and increased power densities for data centers. The enhanced thermal properties of oil can lead to considerable savings in both the upfront and operating costs over traditional air cooling methods. Despite recent findings showing the improved cooling efficiency and cost savings of cooling fluids, this technique is still not widely adopted. Many uncertainties and concerns persist regarding the non-thermal aspects of a single-phase immersion cooled data center. This study serves multiple purposes: 1.) To measure the thermal performance of a single-phase immersion cooled server at extreme temperatures for prolonged time. Thermal overstress experiment was performed on a fully immersed server and its cooling system components. This work explores the performance of a server and other components like pump including flow rate drop, starting trouble and other potential issues under extreme climatic conditions. 2.) The dissertation presents impact of form factor on a maximum junction temperature and thermal resistance at the server level. This work is to provide an insight to increase the rack density by reducing form factor of an existing server. The heat sink is a critical part for cooling effectiveness at server level. This work is to provide an efficient range of operation for heat sink with numerical and computational modelling of a third-generation open compute server for immersion cooling application. A parametric study is conducted,and the thermal efficiency has been optimized for mineral oil and EC 100. 3.) This research reviews the changes in physical and chemical properties of information technology (IT) equipment and compatibility of materials like polyvinyl chloride (PVC), printed circuit board (PCB) and switching devices with mineral oil and EC 100 to characterize the interconnect reliability of materials. For the first time, material compatibility is being tested. Accelerated thermal degradation testing of printed circuit board, passive components and optical fibers for single phase immersion cooling systems is a significant part of this study. The study proposes a testing methodology which can be adopted by all for evaluating the reliability of electronic packages and components when immersed in a dielectric fluid. The study indicates the effect of mineral oil and EC 100 on IT equipment reliability and reliability enhancements for immersion cooled data centers and, 4.) This dissertation also includes Cup Burner Experiment asper ISO 14520/NFPA 2001 standard to determine the minimum design concentration of fire extinguishing agent for the class B hazard of heavy mineral oil and the class C hazard of electronic equipment as a part of the safety concerns for oil cooled data centers.
Air cooling is predominant cooling technique employed in most of the data centers. As the demand for High performance computing (HPC) which deploy large concentration of high-end servers (30 kW to 200 kW per rack) is increasing, it is becoming challenging to cool the systems using air cooling. Liquid cooling has significant advantages over air cooling techniques due to higher heat capacities of fluids. Liquid immersion cooling using dielectric and non-corrosive mineral oils is one of the potential alternative to air cooling methods for high density data centers. In this work, we consider a third generation open compute server optimized for air cooling and find optimal heat sinks for immersion cooling. It is possible to use low profile heatsinks due to the thermal mass of the liquid vs air and thus reduce the server profile and potentially increase the server density in a rack. CFD is used to design the optimal heatsink.
Liquid immersion cooling of servers in synthetic dielectric fluids is an emerging technology which offers significant cooling energy saving and increased power densities for data centers. A noteworthy advantage of using immersion cooling is high heat dissipation capacity which is roughly 1200 times greater than air. Other advantages of dielectric fluid immersion cooling include high rack density, better server performance, even temperature profile, reduction in noise. The enhanced thermal properties of oil lead to the considerable saving of both upfront and operating cost over traditional methods. In this study, a server is completely submerged in a synthetic dielectric fluid. Experiments were conducted to observe the effects of varying the volumetric flow rate and oil inlet temperature on thermal performance and power consumption of the server. Various parameters like total server power consumption, the temperature of all heat generating components like Central Processing Unit (CPU), Dual in Line Memory Module (DIMM), input/output hub (IOH) chip, Platform Controller Hub (PCH), Network Interface Controller (NIC) will be measured at steady state. Since this is an air-cooled server, the results obtained from the experiments will help in proposing better heat removal strategies like heat sink optimization, better ducting and server architecture. Assessment will also be made on the effect of thermal shadowing caused by the two CPUs on the nearby componentslike DIMMs and PCH.
Rising energy demands in data centers have constantly made thermal engineers to think and come up with innovative cooling solutions in data centers. It is of utmost importance to have control over the environmental impact of Data Centers. In 2011, the Open Compute Project was started which aimed at sharing energy efficient practices for data centers. The hardware and electrical specifications of the first open compute server -Freedom was shared on open compute project's website. It was a vanity free design and its components were custom designed. It was deployed in one of the data centers in Prineville, Oregon and within first few months of operation, considerable savings in every and cost were observed. Since then, many open compute servers have been introduced for applications like - compute, storage, etc. The open compute servers which were being introduced mainly had a 2 socket architecture. Yosemite Open compute server was introduced for serving heavy compute workloads. It provided significant improvement in performance per watt as compared to previous generations open compute servers. Yosemite Open Compute Server has a system on a chip architecture and has 4CPU's (1 CPU per sled). This study involves optimization of Yosemite Open Compute server to improve its cooling performance using. CFD tools are very useful for thermal modeling of these servers and predict their efficiency. A commercially available CFD tool has been used to do the thermal modeling of the server and its optimization has been done to improve the cooling performance of the server. The model of the improved design has been compared to the existing design to show the impact of air flow optimization on the cooling performance of the server. The air flow characteristics and utilization of the fans have been significantly improved in the improved design.
As the world is rapidly transitioning into a digital era, blockchain technology has high potential deployments in many organizations' digital transformation plans. Blockchain technology, which is the main driver of cryptocurrency as many other digital technologies, requires sophisticated computing hardware to run it, mainly Application specific Integrated Circuit (ASIC). One of the challenges that faces cryptocurrency miners, who are an integral pillar of the network, is efficiency of their hardware. Efficiency is crucial to miners since it dictates how fast and cost-effective they can run their rigs to generate reasonable profit. With high computing scenarios (or what is known as hash rates in the cryptocurrency community) the hardware capability, despite the existence of air-cooling fans, could be limited due to high temperatures. Computing hardware in a datacenter, cryptocurrency mining rig or even in a gaming console produces a tremendous amount of heat that is essentially cooled by forced air through fans. The excessive heat in the ASIC especially in a high hash rate scenario can affect performance and potentially develop micro cracks in the hash boards and lead to failure and disruptive service. Air cooling by fans is energy consuming, noisy and requires air conditioning to remove heat from the ambient. Moreover, air is considered a good insulator and poor thermal conductor. Alternatively, recent developments in dielectric fluids that have superior thermal characteristics over air show a promising future for cooling computing hardware more efficiently. In this thesis, the thermal performance of a dielectric liquid named Bitcool-888 is explored via numerical simulation to study its effectiveness in cooling ASIC miners. This method is known as one-phase liquid immersion cooling. The ASIC miner is one of four immersed in the dielectric fluid that is being circulated in a system of tank, piping, pumps and a heat exchanger. The objective of the simulation is to find the optimum flow rate and inlet temperature that ensures adequate cooling of the ASIC miners at their peak computing power and to compare the performance of air and dielectric liquid. 32 simulations were conducted via ANSYS Icepak to study the dielectric liquid cooling capability for various power densities (1000 W and 1250 W per hash board), inlet temperatures (30 °C, 35 °C, 40 °C and 45 °C) and inlet flow rates (4 lpm, 10 lpm, 15 lpm and 20 lpm). Air performance was examined for one power density (1000 W per hash board), various inlet velocities (2 m/s, 3 m/s, 4 m/s and 5 m/s) and at 30 °C and 35 °C inlet temperatures. The results show that the dielectric liquid is better than air in cooling the ASIC miner, even at high power density, low flow rates and high inlet temperatures. The maximum hash board temperatures particularly were lower in the dielectric liquid case. Air showed higher maximum hash board temperatures in both 30 °C and 35 °C inlet temperatures cases with low to medium velocities.
A data center is a centralized facility that we use for housing the computer systems and its related components such as high-end servers, redundant data connection and security controls. The next radical change in the thermal management of data centers is to shift from conventional cooling methods like air-cooling to direct liquid cooling (DLC) to deal with high thermal mass. The past few years have consistently seen wider adoption of direct liquid cooling because of its simplicity and high heat dissipation capacity. Passive single phase engineered fluid immersion cooling has several other benefits like better server performance, even temperature profile and higher rack densities.This report provides an overview of the considerations of using single-phase dielectric fluid to cool a server based on experiments conducted at extreme conditions in an environmental chamber. The server was placed in the environmental chamber ranging from extremely low temperatures at -20°C to 20°C and varying humidity for extended durations. Thermal overstress experiment was performed on a fully immersed server and its cooling system components. This work explores the effects of low temperature on the performance of a server and other components like pump including flow rate drop and starting trouble under extreme climatic conditions. The possibility of connector seals observing reduced performance upon accelerated temperature cycling is addressed. Throttling limit for the CPU along with power draw over a range of different temperatures was recorded. Similar observations were recorded for pump. Dependence of pump performance on operating temperature determines the flow rate and operating temperature relationship. Pumping power consumption is directly related to the operating cost of a data center.This research can be expanded by performing similar experiments at elevated temperatures to establish an operating temperature envelope in order to get the optimum performance of a direct liquid cooled high-density server.
The growth in the computing and data industry demands high-performance data centers where the thermal management issues are of greater concern. A cooling system paradigm using the immersion cooling technology is a more reliable and effective method. In immersion cooling, servers are directly submerged into a dielectric fluid. A comparative study on the reliability of the passive components cooled by air, mineral oil and EC-100 was carried out. The Accelerated Thermal Cycling test based on ATC JEDEC is applicable only for air cooling. The ASTM standards D 3455 with some suitable modification was adopted to test the material compatibility with Immersion cooling. The experiment is designed to operate at an elevated temperature of 45oC and relative humidity of 35%. For every 72-hour time interval 3 samples each of thick film resistor, an electrolytic capacitor, polymer capacitor and transistors were taken out and the experiment was continued for the remaining samples. The study focuses on analyzing the change in the electrical properties, and the formation of cracks in the microstructural level of the passive components. The electrical properties such as resistance, capacitance etc. are measured using the multimeter and the microstructure study is done using the Scanning Electron Microscope (SEM). This experiment will provide with trend data of the effect of thermal aging on the electrical property of the passive components aged by air, mineral oil and synthetic fluid. This trend data provides with the comparative performance study of the passive components aged by air, mineral oil and synthetic fluid and, provides the advantages and disadvantages of each medium.
