We’ve been talking about cold plates. We introduced the thermal resistance of cold plates in “Why Drive 1.5?” and then got into the details of how microchannel cold plates work in “The Skinny on Skived Cold Plates.” In both cases, our starting point is this plot, which I’ll repeat here to save you the 3.5 s it would take you to go back and look at those other posts (which you should do anyway! I think they were pretty good 😉 ).

We talked about how we could improve the Convective Resistance portion of the resistance with good engineering of the fins/enhancements and good fluids engineering. But improving the Convective Resistance only draws more attention to the Caloric Resistance. The Caloric Resistance is a First Law of Thermodynamics thing and that law is one that is notoriously hard to break.

But, what if there were a way to eliminate the caloric heat rise entirely? Oh, and possibly cut the convective thermal resistance significantly at the same time? Well, according to several things I’ve seen in my feeds, there’s this one weird trick…

Indeed, this is the promise of two-phase DLC. No caloric heat rise because heat is absorbed via transforming liquid to vapor (phase change) instead of warming the liquid. And all of those bubbles must be good for something! Like improving the convective heat transfer, right?

Let me just cut the suspense and let you know what I believe the data show:

1. It is possible for a two-phase cold plate to have (much) better thermal performance than a single-phase cold plate

2. On the other hand, the data from cold plates in real-world systems show that, in practice, it is difficult to design and operate a two-phase cold plate with better performance than a single-phase cold plate in a realistic data center cooling environment.

3. Two-phase cold plates actually have to be higher performing than single phase cold plates because the real thermal challenge for two-phase systems is in the CDU.

Now, getting to these conclusions will require looking into two-phase flow and heat transfer, and this will take two or three posts at least, since these are much more complex topics than single-phase flow and heat transfer.

Subscribe now

So before we invest all of that effort, let’s step back and look at why we would want to do this anyway. Is there a requirement, from a thermal performance perspective, for two-phase cold plates in the future? I argue that the answer is no. Water is a fantastic heat transfer medium and, if we can continue to support 1.5 lpm/kW, I believe we can efficiently remove the heat from any package we expect to see through 2030 and probably beyond. And I believe that there is room for additional innovation in cold plate design that will further improve thermal resistance. In this respect, PG25 was a long runway.

Raw thermal performance isn’t the only factor here, though. In this post, we will take a look at the broader business impacts of single- and two-phase cooling. While it is not possible to say at this moment that there is a clear advantage for two-phase DLC, the technology has enough potential benefits that I believe we will see some durable level of adoption once commercial systems arrive that can be deployed, serviced and operated at scale.

Share

An Important Disclosure

At this point, it is important (to me, anyway) to state that I have a potential financial interest in the success of two-phase DLC. I was one of the founders of Ebullient, Inc., and one of many inventors of that two-phase DLC system and devices. While I have not been an officer in the company for many years, I still have an ownership stake and, should it be revived, I could see some positive financial benefit from those shares.

I believe, and hope you will find, that my personal experience with this topic makes the information that I’m sharing more useful. But I also dislike reading a 5 page whitepaper only to realize that it was an advertisement the whole time! (I’m getting smarter, really, but it does still happen…) This post is not an advertisement, as I think you will see. But you now know my biases, and I encourage you to be critical of them and what I present on this topic.

Caveats

This post discusses several points where two-phase DLC may be advantageous to single-phase systems and vice-versa. (Green is better and Red is less good in the tables.) This list is surely not comprehensive, and we may need to revisit it later. The comparisons are based as much as possible on hard numbers, but hard numbers are hard to come by for emerging technologies. You should consider these comparisons more like taking a look at the two different approaches, eyes squinting, and noting what strikes you as the most impactful at the moment. This analysis is mostly subjective and you should disagree with it whenever your experience tells you to do so.

In addition, even if this were a 100% objective comparison, the actual business impacts of the differences that are identified will vary widely from organization to organization. While everyone might agree with what I’ve identified as a relative weakness or risk with single-phase DLC, for instance, the monetary value of that risk may be insignificant when all of the strengths are accounted for. But the same analysis may be quite different for another organization.

Anyone who uses this analysis to make a definitive statement that one DLC technology is superior to the other is misusing it.

Summary

The comparison presented here suggests the following:

• Two-phase DLC Cold plates have the potential to cool very heat-dense processors and possibly to cool some processors that single-phase designs will struggle with while maintaining 45 °C TCS supply temperatures

• However, these clearly superior systems don’t exist commercially at scale today.1 A great deal of work needs to be done before scalable, high-performance two-phase DLC systems can meet even a fraction of the total DLC system demand.

• Two-phase CDUs require a much higher approach temperature difference (ATD… see this post) than single-phase CDUs for the same size and heat load, partially or completely negating any cold plate performance advantage.

• Once available, two-phase DLC systems may directly address other key pain points for single-phase systems:

  • Volume of servers occupied by hoses and tubes

  • Weight of cold plate loops and manifolds

  • Coolant maintenance

  • Time to deploy (time to first token)

  • Risk of coolant leaks

    Subscribe now

TCS Hardware

For a useful comparison of the thermal performance of a number of different cooling technologies using specific existing hardware from 2020 to 2022, please see Curtis, et al.2 The approach temperature difference (ATD) is discussed in “Data Center Chillers… Who Needs ‘em”. Other aspects of the single-phase cold plate thermal challenge are discussed in “The Skinny on Skived Cold Plates”.

We will discuss the two-phase DLC CDU in a future post and get into the specifics on the condensation challenges. But you can see that condensation is generally more challenging by looking at a household air-conditioner. The evaporator fits in a small space taking up just a fraction of a closet, or a small unit hanging near the ceiling, while the condenser sitting outside is quite large in comparison. Or if you get the chance to see a steam-cycle power plant, the condenser section is very large compared with the boiler.

Condensation has lower heat transfer coefficients, often by a factor of 4 or so, compared with evaporation because 1) the refrigerant is at higher pressure and flows more slowly and 2) the condensation makes the two-phase mixture continuously more dense, decelerating it as it moves through the condenser. Also, you are having to move heat through a slow moving layer of liquid instead of generating bubbles next to the hot wall. More to say on this, but hopefully the concept that the two-phase CDU is going to need to be larger to match the single-phase CDU in performance makes some sense.

Figure 2 comes from the “Introduction to Data Center Cooling” post. I want to emphasize here that, at a high level, the basic components of the TCS don’t change between single- and two-phase DLC systems. I expect that, overall, the initial cost (CapEx) of a two-phase DLC system will be very similar to that of a single-phase one and that the remaining systems, the FWS and HRS, will see little or no change in most cases.

I plan to discuss the physics of two-phase cold plate thermal performance in future posts. In addition, we’ll need to have a few more posts to discuss CDUs and how they work. Please Subscribe to the Ebullient Insights Substack to be sure you don’t miss these gripping discussions!

Subscribe now

The OpEx angle

As noted in Table 1, the thermal performance from chip package to FWS is about the same for both approaches as far as existing (2026) technology goes. Thus, we expect the overall energy consumption to be similar and that the same high level analysis presented in “Data Center Chillers… Who Needs ‘em?” will apply to either single- or two-phase DLC implementations. (There may be different temperature steps at the cold plate and CDU, but the overall downhill ride will be about the same.)

There is much more to OpEx than energy costs, however. Table 2 presents two additional key areas: Deployment and Maintenance.

For a more complete summary of single-phase DLC TCS deployment considerations, this ASHRAE Tech Bulletin is a must-read! 2026 Technical Bulletin - TCS Coolant Integrity and System Readiness Best Practices

Note that the issues discussed in Table 2 for single-phase DLC increase, generally non-linearly, with TCS loop size. This is an argument against large, data-hall sized TCS piping systems. Other considerations are discussed in this ASHRAE Tech Bulletin from ASHRAE TC 9.9.

Two-phase DLC systems are likely to be deployed in rack- and row-based configurations. ASHRAE Standards 15 and 343 deal with the safety of refrigeration systems, and it will generally be easier for data centers to comply with them if the total system size is limited to either a rack or a small row of racks.

System Design Considerations

Since two-phase DLC uses a dielectric refrigerant as the coolant, more materials are available for use in the TCS. In addition, the lower flow rates of liquid coolant supply may enable additional benefits.

Figure 3 provides a visual comparison of the possible tube sizes that could be deployed in TCS systems using PG25 for single-phase DLC and two different refrigerants for two-phase DLC. For the PG25 examples, two different velocities are assumed. ASHRAE and OCP are jointly performing studies to determine the maximum safe velocities that should be used in single-phase DLC systems. Right now, the general guidance is pointing to acceptable velocities of about 3 m/s in copper or brass tubes, pipes and passages, while approx. 6 m/s appears to be safe in stainless steel and peroxide-cured EPDM hoses. We are seeing more use of stainless steel tubing in servers, so I wanted to provide a reasonable comparison for current and near future single-phase DLC designs incorporating both 3 m/s and 6 m/s. The flow rates assume 1.5 lpm/kW, so a 3 kW package would be cooled with 4.5 lpm, for example. The diameters shown are inner diameters only, since a variety of different tubes and hoses are used in practice.

The “package” tube sizes assume cooling of a 3 kW or 5 kW cold plate with one supply and one return tube. The “server” sizes assume the supply and return tubing for the entire cold plate system inside the specified server.

The two-phase tube sizes assume 6 m/s for the liquid velocity and 24 m/s for the two-phase (mostly vapor) velocity. Two different refrigerants are shown to demonstrate the difference between a low (R-1233zd(E)) and medium (R-515B) pressure coolant selection. The properties of these refrigerants are from EES4 assuming T_sat = 55 °C for both. The saturation pressure for R-1233zd(E) is 340 kPa or 49.3 psi (absolute) and for R-515B is 1121 kPa or 162.6 psi (absolute).

Figure 3 is not intended to provide design guidance or in any way provide a specification for server liquid cooling design. It is only an illustration of the impact of cooling technology on the volume of tubes and hoses that may be required. I believe that the values I have used are not unreasonable, but system designers should verify with the best available guidance and design standards.

It is interesting to note the impact of pressure on the two-phase return tube sizes. Because R-1233zd(E) vapor has a much lower density at 55 °C, it requires a larger cross-sectional area to keep the velocity at or below 24 m/s. The highly observant reader may notice that the R-515B supply sizes are larger than the R-1233zd(E) supply lines. This is because the heat of vaporization of R-1233zd(E) at 55 °C is larger than that of R-515B and thus a higher flow rate of R-515B is required. These are the sorts of details we have to get into to design two-phase DLC systems.

Subscribe now

Supply Chain Issues

These are just the most critical high-level items. Opening up the CDU exposes key supply chains for pumps, control valves, heat exchangers, etc. QDs are a critical concern for two-phase DLC at the moment. In addition, two-phase DLC will require some high pressure equivalent of the fully-open double valved connector, like the Danfoss FD-83, Stäubli TDU24/50, +GF+ Quick Connect Valve 700 or others.

Single-phase DLC CDU standards for method of test and certification are published as ASHRAE Standard 127-2020 Addendum B and AHRI Standard 1360. Work is ongoing (August 2026) in ASHRAE to extend this standard to two-phase CDUs.

Share

Risks

It is important to state that risks do not prevent a technology from being deployed, but they must be managed, and this usually carries some cost that appears as an Operational Expense.

For more information

I’ve summarized these findings above, so I’ll just add some recommendations for further reading here.

Your first stop for data center thermal management information should be the ASHRAE Datacom Encyclopedia. The subscription is cheap and it is updated several times a year by industry experts.

While you’re at it, read the other ASHRAE TC 9.9 publications, or better yet, join the committee and contribute. ASHRAE Technical Committee 9.9 website.

Subscribe now

In “Why Drive 1.5?” we began to dive pretty deep into how cold plates actually work and where the thermal resistance hides in them. In this post, we’re going even deeper.

When Direct Liquid Cooling (DLC) came on the scene, processors were generally under 150 W and a dense rack was 30 to 40 kW. Now, just because the total power was low doesn’t mean there weren’t challenges. As we have discussed already more than once, the thermal resistance, or the difficulty of moving dissipated heat from a device is directly related to both the amount of heat and the area through which we move it. One hundred watts generated on a 2 cm x 2 cm piece of silicon has a heat density (heat flux, or heat per unit area) of 25 W/cm², where that same amount of heat generated on a 1 cm x 1 cm piece of silicon has a heat flux of 100 W/cm². It will generally be about 4x harder to move the 100 W with the 100 W/cm² heat density than that with the 25 W/cm². Some of those processors back in the 2000’s in fact had heat flux densities approaching 100 W/cm² and liquid cooling was needed in many cases.

Not so long ago, those high heat fluxes tended to be niche applications: High Performance Computing (HPC, think Supercomputers) or special purpose computing for specific applications. That is no longer the case. Not only have average heat fluxes begun to approach 200 W/cm², but the cm² part has gotten pretty large and the total heat load per processor package is climbing past 1500 W. Thinking about 5000 W processors is not only not crazy, it is necessary.

Cold plate designs have advanced significantly in the last few years. The thought of cooling a 10 kW server with 45 °C TCS supply was considered unserious even a few years ago. Now it is both serious and essentially a product requirement. This has only happened through drastic improvements in design for DLC systems. However, review Figure 2 from “Why Drive 1.5?”, reproduced here as Figure 1. It is apparent that there is a pretty big resistance in that “Convective” layer. That is good, because this is the layer where good engineering and innovative fluid flow design can make some progress. The caloric heat rise is a First Law of Thermodynamics thing, so there’s no changing that. The conduction heat loss is challenging because we need some thickness of copper to withstand the forces on the cold plate. Thermal interface materials have improved incredibly in the past decade, but seem to be approaching limits to how much better they can be engineered.

The goal of this post is to create a simplified, but realistic, model of a cold plate so that we can understand why they are designed like they are. This will require that we get into some of the fundamentals of fluid flow, heat conduction and convective heat transfer. I have done my best to take an “intuitive” approach instead of highly mathematical. But I learned not to trust “intuitive” whenever I read it in a text book. Please let me know if you have a better understanding of cold plates when you’ve read it all. If not, I want to know and do a better job.

Leave a comment

It is important to understand that this example does not represent any specific product, nor does it exactly model every nuance and innovation in modern cold plates. It does, however, capture the most important thermodynamic impacts of single-phase cold plate heat transfer, which will enable us to draw some important conclusions as we continue to discuss all of the components of data center cooling.

If you want to get a high-level idea of how cold plate thermal resistance impacts the energy efficiency of the entire data center, please take a look at the early post, “Data Center Chillers… Who needs ‘em?”

A tour of the cold plate

Our cold plate model will be based on a micro-channel, or skived-fin design that has served the industry well for many years. The basic idea is shown in Figure 2.

In this diagram, coolant enters on the left, then flows through the fins, and then out on the right. The processor heat enters the cold plate from the “Top of Processor,” which could be basically the silicon itself in some cases, but most frequently it is a protective cover called the lid or Integrated Heat Spreader (IHS). This IHS is typically nickel-plated copper, or something similar, approximately 2 mm thick.

A thermal interface material (TIM) is placed between the Top of Processor and the cold plate base to act as a coupling between the two surfaces. It is necessary due to small variations in surface roughness and/or flatness that would otherwise prevent a direct conduction path for heat between the processor and the cold plate. In addition, it serves to minimize the impedance to thermal energy transfer between the two surfaces at the molecular level. We will look into TIMs at more depth in a later post.

Finally, the heat passes through the base of the cold plate, and into the thin fins or other area-increasing enhancement structure. The base of GPU/CPU cold plates is most frequently made of copper and is between 1 and 3 mm thick. The combined structure of the base + enhancements + lid must be strong and stiff in order to permit the cold plate to be compressed against the processor sufficiently to fully enable the TIM to make good thermal contact with both surfaces. This force frequently ranges from 150 lbf (660 N) to maybe 600 lbf (2700 N).

Why micro-channels?

Data Center Cold plates have been dominated by the skived microchannel geometry for several years. While new geometries are beginning to appear as the cost falls for their manufacture, skiving remains a reliable, fast and affordable way to manufacture high performance cold plates.

If you look at the volume of liquid-cooled server deployments now and projected in 2027, we will, as an industry, be approaching the deployment of nearly 1 million (1 Mega, 1 and 6 zeros, an excrement-ton… it’s a lot however you say it!) cold plates a month by the end of 2027 or by early 2028. It is hard to describe the scale of growth in this industry, but that’s equivalent to several years-worth of cold plates in the before-times (before 2023, say).

So, the emphasis on reliable, fast and affordable cannot be overstated. We have to scale.

What is skiving? If you search it up, you will find videos and explanations that are worth taking a look at.1 Basically, think of taking a really heavy duty knife and peeling a very thin layer of copper from the surface of a plate, then gently nudging it into a straight vertical position. This process can be performed at scale, with excellent quality and repeatability. It is one of the enabling factors behind the rise of high performance cold plates.

Historically, the limit to fin thickness and gaps (often the same dimension, though they don’t have to be) has been about 100 microns. The current state of the art is pushing beyond this to 50 microns.2 Fin lengths (i.e., height above the base) can range from 3 mm to about 7 mm for these very thin fins and gaps, meaning that very high aspect ratio channels can be formed.

But why do this in the first place? What benefit do we get from these microscopic fins and gaps?

It has been known for a long time that the convective thermal resistance between the solid walls of a channel and the fluid flowing between them is proportional to the gap width. We can derive this with information we’ve discussed in previous posts. For example, we know that convection heat transfer is described by Newton’s Law of Cooling,

and that we define the convective thermal resistance as

We will show below that the convection coefficient, h, is approximately

where g is the gap between the fins and k_liq is the thermal conductivity of the liquid.

If we substitute this h into the resistance equation, we get

So, if we can make g really small, we can make the thermal resistance really small. Fins with a 100 micron gap will have, in theory, half the convective resistance as fins with a 200 micron gap.

Of course, nothing in life is actually that simple. But the trend is absolutely observed in real life and this is what is behind the drive to thinner and thinner channels in cold plates.

To show why this relationship between gap width and thermal resistance exists, we’re going to have to go really deep into the behavior of fluids in the fins.

Share

Laminar flow between parallel plates (or walls or fins)

We first need to define some important dimensions. Figure 3 gives a simplified diagram of a pair of fins and associated dimensions.

• L = fin length, the height of the fin above the base

• W = fin width. This is the dimension along the flow. If the base is 30 mm wide and the fin extends along the entire width, we say that the Fin W = 30 mm.

• th = fin thickness, a traditional label.

• g = channel gap, the separation between neighboring fins.

For the purposes of this discussion, we label the x-direction as along the width (i.e., in the direction of flow in the channel), extending into the page in Figure 3. The y-direction is normal to the fin face, from left to right in Figure 3. The z-direction is along the fin length, L, or vertical in Figure 3.

When a liquid like water with 25% Propylene Glycol (PG25) flows between the fins in a cold plate, its velocity will be relatively low and it will flow smoothly; generally without mixing or vortices (spinning flow). This kind of flow is called laminar flow, coming from the word “lamina” or layer. You can imagine the liquid flowing in very thin layers, each layer sliding over the one below it. Except the layer next to the fin, or wall of the channel. That layer will be “stuck” to the solid surface (this is called the no-slip boundary condition in fluid mechanics). This construct is imaginary, so please don’t think that actual liquids divide themselves cleanly into layers. Still, it is helpful for visualizing the behavior.

Figure 4 is a kind of busy diagram that describes laminar flow in the gaps between the fins that is pushed through the gap by a uniform pressure on one of the open ends of the gap. To understand this diagram, imagine that you are looking down at flow between two of the fins. First, you can see that on either side, there is a thick black line labeled “WALL.” “Wall” is common terminology in engineering fluid mechanics for the solid boundary of a fluid flow. Here it is synonymous with the face of the fins. Again, imagine the liquid as a lot of very thin layers, each flowing parallel to the walls. The velocity of the fluid within each layer as a function of distance from the wall is given by the blue curve.

As noted above, the velocity of the layers next to the walls is zero. The layer of liquid right next to the wall layer will slide over it with some resistance, because the molecules in the moving layer are actually vibrating due to thermal energy, and they collide with the molecules in the layer stuck along the wall. This interaction creates a resistance, or “stickiness,” that we define as viscosity. With each layer of fluid, the impact of the wall is “felt” less and less, so the layers can speed up. The layer at the exact center of the channel will have the maximum velocity.

In this type of flow between two parallel walls, the maximum velocity at the center will be 1.5x the average velocity. In the diagram, the average velocity has been set to 1 m/s. This is probably much higher than any velocities in a cold plate fin, but it makes the diagram cleaner.

We can derive this velocity behavior from fundamental physics. One way to write it is

where u is the velocity in the flow direction (the x-direction in Figure 3), u_avg is the average velocity across the gap, y is the position across the channel (in the y-direction in Figure 3) and g is the gap width.3

As we’ve noted in Figure 4, about 70% of the liquid mass flows in the middle half of the channel. The left quarter width of the channel (between y/g = 0 and 0.25) carries about 15% of the flow while the right-most quarter width of the channel (between y/g = 0.75 and 1) carries the final 15% of the flow. This will be pretty important later on.

Subscribe now

Heat Transfer to Laminar Flow

Let’s return to our analogy of the liquid essentially flowing in a very large number of very thin sheets. The fluid molecules in each sheet, on average, do not mix with molecules in neighboring sheets (this is not strictly true, but is a useful model). Thus, if there is any transfer of momentum or heat, it must occur by thermal fluctuations of the molecules causing neighboring molecules to randomly collide with each other and exchange momentum (organized movement of mass) and thermal energy (random molecular vibrational or rotational energy). This is the fundamental process of diffusion and is the physical process behind conduction heat transfer.

All of this is to say that when we speak about convection of heat in the liquid, really what we are saying is that the heat is conducted perpendicular to the wall (and the flow) through the slower moving liquid near the walls to a point where the bulk fluid motion near the center of the channel carries the thermal energy in fluid molecules in the direction of flow at a much greater rate than the conduction of heat perpendicular to the wall and further into the liquid.

A reasonable estimate of the region where the bulk fluid motion dominates the thermal transport is the center 50% of the channel where 70% of the liquid mass flows, as shown in Figure 4. This would mean that the heat transfer into the fluid primarily occurs by conduction in the ¼-width of the channel next to each wall. So, this is the region where most of the temperature difference occurs. We can show this conceptually in Figure 5.

Making a model based on this physical description

As noted above, we can model convection heat transfer using Newton’s Law of Cooling,

But this must also be equal to the heat conduction into the fluid from each side

Since these two equations must be equal, we can write

And since

we can write

Note that this result can also be derived very formally from the laws of thermodynamics and mechanics, and we arrive at a very slightly different value (for constant heat flux at the walls) of

I refer you to the Nellis and Klein text for the details.4

Share

Skived fins in the real world

Let’s apply this to “real” examples. The thermal conductivity of PG25 is 0.494 W/m-°C at 50 °C.5

Let’s start with a gap between fins of 100 microns. We can then estimate the heat transfer coefficient (h) using our model and find it to be approximately 19,760 W/m²-°C.

If the gap is 200 microns, then the heat transfer coefficient is approximately 9,880 W/m²-°C. And so on.

Now, let’s assume that we have a 3 cm x 3 cm region covered by fins that are 3 mm tall, 100 microns thick with 100 micron gaps. With these dimensions, each fin requires 200 microns, so we will have 0.03/0.0002 = 150 fins. The total surface area of these 150 fins available for heat transfer will be 300x the area of one side of one fin, which is 0.03 x 0.003 = 0.00009 m². So the total fin surface area would be 300 x 0.00009 = 0.027 m².

Then, the convective thermal resistance would be estimated to be

assuming that the fins are 100% efficient. This is not a good assumption, actually. The higher the heat transfer coefficient, the more heat is pulled out of the fin near to the base and the less useful is the rest of the fin area. The fin efficiency is often between 50% and 90%. But we will start here for sake of comparison and not further complicate things.

Doing the same estimate for 200 micron fins and gaps, we arrive at 75 fins, 0.0135 m², and a thermal resistance of

This is 4x worse than the 100 micron case.

While these values of thermal resistance are not actually seen in the real world (they are too optimistic) for reasons related to the fin efficiency, complex 3-dimensional flow effects and other less understood impacts, the benefit of using thinner fins and gaps is clear. The cold plate industry has seen this and has used it to great advantage.

Subscribe now

Summary

We now have a basic understanding of how most current (2026) single-phase cold plates transfer heat from the processors to the coolant. Although the model was presented in terms of relatively simple behaviors that are hopefully clear and easy to understand, the model is supported by fundamental, first principles physics.

Note that this model does not indicate any dependence of the convective heat transfer on flow rate. This is fundamentally true if the coolant is truly laminar and steady. However, it is not exactly what we see in cold plate testing.

Also, as noted above, the heat transfer coefficients that the model generates are much higher than measurements in actual cold plates would suggest. Interestingly, for such a theoretically simple flow, this discrepancy is not very well understood. However, it seems to be strongly related to the much more complex 3D nature of the flow in these very narrow, tall gaps. The geometry also means that the liquid itself has pretty large 3D variations in temperature. These temperature variations can have a significant impact on the value of important properties, like viscosity, which can feed back to the heat transfer and temperature behavior.

So, we can use our knowledge and models to guide us, since the trends they show are also seen on the test bench. But ultimately we will depend on testing and models of these test data to design and deploy cold plates.

Future posts will investigate what happens when we go two-phase with a refrigerant as the coolant in this kind of cold plate geometry, as well as what other kinds of cold plate geometries may offer some improvements over this basic design and why.

Share

Leave a comment

Depending on how deep you are into liquid cooling deployments, you either have strong feelings about a recommendation of 1.5 lpm/kW cold plate flow rate, or you think it doesn’t really matter anyway, just keep the dang chip cool, or you don’t know what we are talking about. As a matter of transparency, I am in the first group. And I have been for a while, at least since the times when 30 kW was a dense rack! But if you don’t really care, I hope you will soon. And if you don’t know why someone would spend hours and hours thinking and writing about this topic, I hope you will enjoy learning about it in this post.

Cold plates are the devices that mount to hot electronic devices, like GPUs and CPUs, for removing the heat using a flowing liquid. They are not a new invention, and have been used for many decades not only in computers, but also for power electronics, lasers, imaging chips, and much more. As your intuition tells you, we can make the electronic devices cooler at a given heat load if we flow more liquid through the cold plate. But how much coolant flow should we use? Is there a right answer and/or a wrong answer? Sort of and/or sort of not.

In general, we don’t like to waste energy or over size pumps to flow too much liquid through cold plates. And at some point, there are definitely diminishing returns to increasing flow rate. So, there is sort of a right answer on the high side of flow rate through a cold plate in that there is some point where most people would ask, “Is that really necessary?”

On the other hand, clearly (?) having liquid that is not flowing at all does not feel right. So there logically seems to be a low end to the flow rate where we would say the cold plate can’t do its job.

So we have a range between too low and unnecessarily high. Is there an optimal flow rate? I would argue that there is not, but I could probably think of at least 5 others who would love to argue about that for argument’s sake. On the other hand, is there a flow rate that makes sense in most situations, even if it is not necessarily a mathematical optimum? Yes, I believe that this is the case.

But how and why did 1.5 lpm/kW become a common recommendation? I don’t know exactly how, to be honest, so we won’t go there. I’m pretty sure it was some combination of lab testing, and other forms of liquid testing in restaurants, bars and pubs after ASHRAE and OCP meetings. But as to why: I hope to convince you that 1.5 lpm/kW is a really logical flow rate for cold plates cooled with 25% Propylene Glycol - 75%(ish) water solutions (PG25 from here on out).

As a preview, the thermal resistance (see this post for a summary) of a cold plate is basically the sum of three resistances: 1) conduction through the thermal interface material (TIM) and solid base that mounts to the electronic package or device, 2) the convection of the thermal energy into the PG25 (including the fin efficiency or other factor to account for enhanced solid structures on the base), and 3) the caloric heat rise of the PG25 as it absorbs the heat.

The flow of PG25 doesn’t have much impact on the base conduction, but the convective thermal resistance is inversely related to the flow rate - in other words, as the flow rate increases, the convective thermal resistance decreases. And the PG25 liquid increases in temperature from cold plate inlet to outlet by about 10 °C for any heat load when the flow rate is set to 1.5 lpm/kW.

The answer to the question of what is the best cold plate flow rate is not just about these thermal resistances, though. It is equally about managing system variations in flow rates, variations in system water temperature, fouling due to impurities, corrosion or biology, and transient thermal behaviors that may occur when the heat load is suddenly turned on or off.

Giving away the conclusion here:

• We need thermal margin to manage coolant flow and temperature variations that we know occur even under normal operating conditions

• We also need thermal margin to manage fouling and power transients

• Below 1.5 lpm/kW, we lose thermal margin non-linearly as flow rate decreases

• Above 1.5 lpm/kW, we gain thermal margin at a pretty slow, nearly linear fashion as flow rate increases.

• Thus, 1.5 lpm/kW is a good target flow rate for providing strong thermal performance with healthy margins for reliability while not specifying an unreasonably large flow rate.

This will be, I promise, a high level treatment of these topics. I promise this because diving too deeply will exceed the 10 minutes or so you have set aside to read this (and I thank you for that!). But I also promise more detailed posts on these liquid cooling complications. So subscribe to the Ebullient Insights substack to be sure not to miss them!

Subscribe now

What is 1.5 lpm/kW?

When we say that we are providing 1.5 lpm/kW of coolant to a cold plate (or any other heat transfer device), we are referring to the fact that for every 1000 W of heat removal, the device will receive 1.5 lpm. So,

• A 300 W device will receive 0.45 lpm (450 ml/min)

• A 500 W device will receive 0.5 lpm (500 ml/min)

• An 800 W device will receive 1.2 lpm

• A 2000 W device will receive 3 lpm

We also extend this thinking to racks and CDUs. A 220 kW rack, where 200 kW is liquid cooled, will receive 300 lpm of coolant. A 2.5 MW CDU needs to produce 3,750 lpm of TCS (Technology Cooling System, or Secondary) flow to cool all the devices attached to it if we specify 1.5 lpm/kW.

Where does the thermal resistance of a cold plate come from?

Cold plates can have a huge number of different designs. We will discuss, as usual, an over-simplified, but not unreasonable, approximation of cold plates you would find in 2025-26 high end servers (see Figure 1). These cold plates have a copper base that provides structure and a very flat, very smooth surface to interface with the processor. A thermal interface material (TIM) is required to thermally couple the cold plate to the processor package. Not using a TIM will result in very high total thermal resistance. Inside the cold plate, the copper base usually has some form of complex structure, often with features on the mini- or micro-scale (10’s to 100’s of microns, or millionths of a meter; for those with hair, a human hair is typically about 80 to 100 microns in diameter; for those without – or even with! – hair, a red blood cell is about 10 microns in diameter). This complex structure enhances the effectiveness of the heat transfer to the liquid flowing through the cold plate. A lid of some sort will be brazed or welded onto the top side of the cold plate to keep the liquid where it is supposed to be (most of the time, thankfully).

So, starting at an electronic device package, heat must travel through a TIM (more on that in the future), then through the base of the cold plate, then into the liquid coolant. The liquid, for virtually every system in use in 2026, will remain in the liquid phase (i.e., not boil or experience phase change), and thus it must increase in temperature as it absorbs the heat.

Overall, the thermal resistance is a summation of these three segments of the thermal journey:

Conduction through the base + convection into the liquid + caloric heat rise of the liquid

Altogether, the thermal resistance as a function of flow rate looks something like the curve in Figure 2.

Share

Conduction resistance through the base

Conduction of heat in a solid can be described as thermal energy being transferred from hot to cold across a specified distance through a certain area. In simplified form:

Thermal energy transfer (in Watts) = Conductance of the material x Area of transfer x (T_hot - T_cold) / Distance of travel

In engineering textbooks, this is known as Fourier’s Law and is written

For this 1-dimentional case, we showed in a previous post how this can be written like Ohm’s Law if we think of the energy transfer as current, the temperature difference as a voltage, or potential, difference, and the remaining terms grouped into a conduction thermal resistance.

where

and k = the thermal conductance of the material, A = the area through which the thermal energy is transferred and Δx = the distance over which the thermal energy is transferred.

To make this less esoteric, let’s think about our copper cold plate. The base might be 2 mm thick, so Δx = 0.002 m. The thermal conductivity of copper at room temperature is about 395 W/m-°C. So the conduction resistance per unit area is 0.002/395 = 0.0000050 m²-°C/W. But that’s obnoxious. So we can multiply by 100² cm² in every m² to get 0.05 cm²-°C/W.

Why would we want to use something with these crazy units (cm²-°C/W)? Well, it lets us compare different materials without worrying about the area through which the heat travels. If the cold plate surface is 2 cm x 2 cm (4 cm²), then the thermal resistance would be 0.05/4 = 0.0125 °C/W. This means that to move 100 W of heat through 2 mm of copper, we would require 100 x 0.0125 = 1.25 °C of temperature difference.

Computer processors are much more likely to be something like 5 cm x 5 cm = 25 cm². So the thermal resistance of that cold plate base would be 0.05/25 = 0.002 °C/W. So to move 100 W of heat through that area only requires 0.2 °C of temperature difference. But we more likely will be worried about 1000 W of dissipated heat, which will need 2 °C of temperature difference.

As you can see, just presenting the thermal resistance in °C/W doesn’t really tell you how good a material or a system is at transferring heat. Referring to the resistance per unit area gives you a better apples-to-apples comparison (even if it probably won’t be perfect). I.e., a 2 mm thick copper plate will always have about 0.05 cm²-°C/W thermal resistance, even if the baby in the family has a thermal resistance of 0.0124 °C/W while the oldest sibling has 0.002 °C/W.

We won’t repeat this whole discussion for the TIM. It is quite similar for our purposes, though there can be some specific, important differences that we will discuss in a future post.

We will assume, as shown in Figure 2, that this resistance is constant for all flow conditions.

Subscribe now

Convective resistance

Next, we have to move the heat from the base to the liquid. At a high level, we can think of this rather simply as

where h is known as the convection coefficient with units of W/m²-°C, A is again the area across which we are transferring the heat, T_base is some average temperature of the cold plate base and T_avg,liq is the average temperature of the liquid flowing through the cold plate. This equation describes observed behavior in nature, held up by some underlying first-principles physics, but mostly we use it as an engineering approximation and Newton gets the credit for it. And, again, I’m sure I have set off a lot of angry muttering at screens, since it is such an oversimplification that purists and practitioners alike will argue over an infinite number of beverages because we clearly don’t have any real problems in life to solve or anything.

In fact, the statement as written is kind of useless since we can’t easily measure the local temperature of the liquid or the base, and the average liquid temperature is a function of both flow rate and heat load. So we simplify this even further and write

where R_conv = 1/hA in units of °C/W or 1/h in units of m²-°C/W. In fact, this convective thermal resistance also incorporates the impact of the enhanced area of all the fancy features on the base (like the fins in the sketch in Figure 1), so it is almost never something derived from first principles but rather found through experiment or, if you’re lucky, CFD simulation.

As you might think (though it isn’t actually obvious in many cases), the convective resistance is a function of the liquid flow rate. If you go back and look at Figure 2, I have made a very approximate attempt to capture what we see in “real” cold plates. The convective resistance is somewhat sensitive to flow at low flow rates, but decreases very slowly with coolant flow at higher flow rates.

Caloric heat rise

The caloric heat rise is just the formal description of the fact that the temperature of a fluid must increase as it absorbs thermal energy. This fact comes from the First Law of Thermodynamics. When we apply the First Law to a cold plate we find that

where

is the mass flow of liquid through the cold plate in kg/s and c_p is the specific heat of the liquid in J/kg-°C. The mass flow rate [kg/s] is the volumetric flow rate in m³/s times the density of the liquid in kg/m³. Or, to use more common units, the volumetric flow rate in liters/min x 1/60 min/s x 1/1000 m³/liter or lpm/60000 x density in kg/m³. If that wasn’t painful enough, we remind everyone that the specific heat is the amount of thermal energy that causes the temperature of 1 kg of a substance to rise 1 °C.

As a specific example, the density of water with 25% Propylene Glycol (PG25) at 45 °C is about 1007 kg/m³ and its specific heat is about 3980 J/kg-°C. If T_liq,out - T_liq,in is 10 °C and the flow rate is 1.5 lpm, then

In other words, we can see that if we always use 1.5 lpm of PG25 coolant for every 1 kW of heat load, the temperature rise will be about 10 °C, no matter what the actual heat load.

Now, as we noted above when talking about convection, the temperature we care about is closer to the average temperature of the liquid, not the inlet or the outlet. And the average will be

So we can approximate the impact of the caloric heat rise as a caloric resistance

This is an approximation like everything else here, but it proves useful and not so far off experimental data.

Looking at Figure 2 again, we see that the caloric resistance has a pretty dramatic impact on the thermal resistance. This impact is very strong at lower flow rates, but it remains significant at higher flow rates as well.

Putting it all together

Now that we have a basic understanding of the thermal resistance of cold plates with PG25, we can begin to understand the reasoning behind the very common 1.5 lpm/kW recommendation for DLC with PG25.

Before getting into the details, it is very important to again emphasize that the example I’ve presented here is artificial and does not exactly represent any existing product. (For those who may be thinking that I’m shilling for the vast 1.5 lpm/kW conspiracy, I modeled the convection resistance through 100 micron fins and gaps using a modified Sieder and Tate correlation.1) This example should capture the overall behavior and trends of state of the art cold plates in 2025 to 2026. But the actual thermal resistance may be pretty different for many reasons that we will go into in a future post.

Figure 2 is reproduced below as Figure 3 with several details noted. On the right hand side, I’ve noted the thermal resistances corresponding to various TCS supply temperatures. First, let’s assume that the processor power is 1500 W and the critical temperature of the processor is 85 °C. If we want to use a TCS supply of 45 °C, then the total cold plate thermal resistance must be

We see that, for this example cold plate, we achieve 0.027 °C/W at 1.5 lpm/kW. Again, to emphasize, this exact thermal resistance reflects this example only. However, it is not unreasonable to assume that a cold plate may have been designed to this specification given the generally accepted target flow of 1.5 lpm/kW. Clearly, this fact doesn’t justify the use of the 1.5 lpm/kW as that would be a circular argument.

We also see that it could be acceptable to operate at 0.9 lpm/kW if the TCS supply is 35 °C or below. Or even 0.55 lpm/kW if the TCS supply is 25 °C or below.

Share

TCS Flow Variations

The challenge with the lower flow rates is that we lose thermal safety margin for any variations in the system. For example, we should assume that the TCS flow rate will experience fluctuations of +/- 10%. One common reason for this is that Coolant Distribution Units (CDUs) have multiple redundant pumps to ensure reliable operation even if a pump fails. These pumps are continually rotated in and out of service to ensure that they are in good condition when needed. During pump service rotation, it is common for there to be temporary fluctuations of 10% (or more) in TCS flow rate.

If the nominal flow rate is 1.5 lpm/kW and there is a drop of 10% of the flow, it is noted in Figure 3 that there will be a thermal resistance penalty of 0.0009 °C/W. This doesn’t seem very large. But when we multiply by 1500 W, it translates to 1.4 °C. The critical chip temperature may rise to 86.4 °C during this fluctuation.

If the fluctuation is 20%, the thermal resistance increases by 0.002 °C/W, which means that the critical chip temperature may increase by 3.1 °C to 88.1 °C.

At 1 lpm/kW nominal flow rate, the impact is greater. A 10% decrease in TCS flow will cause an increase of 1.9 °C, while a 20% decrease will cause the chip temperature to rise by 4.1 °C.

Finally, at 0.7 lpm/kW, we find impacts of 2.5 °C and 5.5 °C for flow variations of 10% and 20%, respectively.

In other words, as we lower the flow rate, even if the TCS supply temperature is low enough to permit nominally safe operation, we are at much greater risk of a thermal fault and throttling of the processors, even in “normal” operation. This is a result of the very non-linear behavior of the cold plate thermal resistance as the flow rate decreases.

TCS Temperature Variations

Modern AI data centers experience wide power and heat load swings with regularity. This can cause temperature variations on the TCS, FWS and HRS loops as the control systems cannot generally act fast enough to keep the temperature fluctuations to zero. It is not unreasonable to plan on TCS temperature variations of 1 to 2 °C under normal conditions. A system operating at 1.5 lpm/kW will have several more degrees of margin to absorb these fluctuations compared with one operating at 1.0 lpm/kW.

Fouling

Over time, it is inevitable that the cold plate will accumulate debris, corrosion products, small amounts of algae, etc. Assuming a 5 to 10% degradation in the cold plate performance is not unreasonable, though the degradation factor for any specific cold plate should be verified with the cold plate manufacturer. We would much rather start with the lower thermal resistance at 1.5 lpm/kW than the 1 lpm/kW value to ensure that the computing system will still be functioning well after a few years of continuous operation.

Processor Power transients

It is generally well known that some compute work loads, especially for artificial intelligence, have very large, rapid transitions from an idle condition to high power. Many times these transitions are accompanied by significant transient power spikes. While very short, the heat generated must still be managed to prevent overheating at the processors. The thermal system has a time constant similar to an electric circuit, i.e., 𝛕 = RC. In the above example, operating at 1.5 lpm/kW will allow a 15% shorter time constant than at 1.0 lpm/kW because the thermal resistance is about 15% smaller. This means that short power transients are handled more effectively at the higher flow rate.

Summary

We have spent nearly 3000 words now learning where the thermal resistance in cold plates comes from and how the coolant flow rate directly impacts the thermal resistance. We can see that below 1.5 lpm/kW, the risks of overheating the processors increase non-linearly due to system level variations that we don’t have any control over. And we also see from Figure 3 that using flow above 1.5 lpm/kW doesn’t really add a lot more thermal margin.

While this example does not represent any cold plate in particular, and the exact thermal resistance curve will vary from cold plate to cold plate, the general trends are pretty consistent.

• We need thermal margin to manage coolant flow and temperature variations that we know occur even under normal operating conditions

• We also need thermal margin to manage fouling and power transients

• Below 1.5 lpm/kW, we lose thermal margin non-linearly as flow rate decreases

• Above 1.5 lpm/kW, we gain thermal margin at a pretty slow, nearly linear fashion as flow rate increases.

• Thus, 1.5 lpm/kW is a good target flow rate for providing good thermal performance with strong margins for reliability while not specifying an unreasonably large flow rate.

Subscribe now

Further Reading

Your first stop for data center thermal management information should be the ASHRAE Datacom Encyclopedia. The subscription is cheap and it is updated several times a year by industry experts.

While you’re at it, read the other ASHRAE TC 9.9 publications, or better yet, join the committee and contribute. ASHRAE Technical Committee 9.9 website.

I consider the thermodynamics text by Sanford Klein and Greg Nellis to be the best one out there for learning how to think about engineering thermodynamics. It is rigorous in theory, but has a strong focus on how to solve real world problems using a variety of software tools and hand calculations.

Klein, S. and Nellis, G., 2012, Thermodynamics. Cambridge University Press, New York, NY. www.cambridge.org/kleinandnellis

Independently, I really like the approach to heat transfer taken by the same authors in their Heat Transfer text. I’m partial to this because the way they present the topic is similar to how I liked to teach it during my years in academia. Or it is probably better to say that I approached teaching the subject the way that they present it. Really cannot recommend this book more highly. Very solid theoretically with lots and lots of solved examples and practical instruction on how to solve complex real problems with available software tools.

Nellis, G. and Klein, S., 2009, Heat Transfer. Cambridge University Press, New York, NY. www.cambridge.org/nellisandklein

The last post gave an overview of data center cooling and some background as to why we frequently use three separate liquid loops to collect, transport and reject the heat from the high-powered compute common to AI and HPC data centers. These three loops are the Technology Cooling System (TCS), the Facility Water System (FWS) and the Heat Rejection System (HRS). Figure 1 is shown again here to refresh your memory.

This post is going to explain why some HRSes require refrigerated chillers and others don’t, and begin to show how the design of each component in the thermal system impacts the final data center cooling design. We will also begin to see why designing servers to accept 45 °C cooling water is significant.

This discussion uses some examples that do not exactly represent any real data center, but that use reasonable temperatures and heat values for 2026. The intent is to demonstrate how these systems interact and the impact of design choices; the intent is not to say that this is exactly what should be expected in any given data center.

Share

Moving the heat - the best case

The Second Law of Thermodynamics requires that heat only “flows downhill,” or from hot to cold. Thus, the temperatures must go from hottest at the cold plate on the hot electronics to coolest in the surroundings, unless we have an intervention to use the First Law of Thermodynamics to go against nature and move heat from cold to hot. Figure 2 visualizes the ideal heat flow. It starts with the electronic device dissipating heat at, say, 80 °C. The heat is transferred to TCS coolant at 40 °C in the cold plate. The TCS then transfers the heat to the FWS coolant in the CDU at 35 °C. The FWS to HRS heat exchanger is usually designed to have a very small temperature difference, so let’s say that the HRS coolant temperature is 34 °C. Now, this heat can be transferred to the surrounding air using a radiator with a fan on it, a dry cooler or fluid cooler, with about a 10 °C temperature difference. So, in this case, heat can flow downhill the entire way from the electronic device to the surroundings if the ambient temperature is about 24 °C.

(To reinforce this point, these temperatures are not unreasonable, but are definitely an over-simplified example. There are many variations on data center cooling systems and many great engineers who know how to design systems optimized for their environments.)

When we slide down this more ideal Second Law of Thermodynamics path from heat source to heat sink, we will generally pay the lowest “energy price” to move the heat.1 Very approximately, let’s estimate what it takes to move 1 MW of heat down this thermal hill. The pumps in the CDUs “cost” about 10 kW of electricity for every 1 MW of heat moved. Similarly for the FWS and HRS pumps. So that’s 3x 10 kW, or 30 kW in pump power. The fans on the dry coolers might cost about 80 kW of electricity for every 1 MW of heat. Altogether, we could use something like 110 kW of electricity to move 1 MW of heat. That’s a pretty good deal! From a Coefficient of Performance (COP) perspective, that is a COP of about 9. It is a cooling partial PUE of about 1.11 for those curious about that.

As a reminder, the COP can be broadly defined as

To be intentionally pedantic, energy and power are not the same. Power is the rate at which energy is transferred:

We will often interchange concepts based on energy with expressions using power. This is generally acceptable if we are considering the same time intervals for all of the quantities involved. We will do our best to be careful when it is important not to confuse these concepts. For example, you don’t pay for power. You pay for energy, which is power (kW) used during a time interval (one hour, for instance, or kWh).

And, by the way, no water is consumed in this process! The water is entirely contained in the pipes and heat exchangers; it is just circulated around the TCS, FWS and HRS. It is never used up, evaporated to the environment or dumped down a drain.2

Thermal Resistance

There is actually a whole lot going on in this example. Much of it comes down to a concept of thermal resistance, which is an over-simplification of the physics that allows us to make broad, generally quick estimates to compare options before doing the hard, more expensive detailed work to validate the design.3 Generally, thermal resistance is defined as

or

which is related to the more familiar version of electrical resistance when you think of it as

This concept of thermal resistance assumes a lot of things that are not always quite true. Some of these are:

• One-dimensional heat transfer

• Uniform temperatures on each side of the resistance (really restating the above)

• Uniform heat flux (heat transfer per unit area)

• Steady state (the heat transfer and temperatures are all constant and not changing in time)

Even though these things are not always true, thermal resistance is still very useful for helping us analyze really complex problems quickly and with sufficient accuracy to eliminate options that won’t lead to a viable solution.

Let’s see how this works. In the example above, we assumed that the electronic device temperature (at some important, known location) is 80 °C and that the liquid entering the cold plate, or heat exchanger attached to the electronic device, is 40 °C. Thus, T = 80 - 40 = 40 °C. If the liquid removes 1000 W from the electronic device, then the thermal resistance is

Some readers may judge this as a bad thermal resistance, others as a good thermal resistance, and others as ¯\_(シ)_/¯. The last response is probably right. We can’t really judge the thermal resistance without knowing a lot more details. Which, I promise, we will dive into in later posts. For now, we accept this as the thermal resistance from the electronic device to the liquid inlet to the cold plate. And that each electronic device is dissipating 1000 W of heat that has to be removed to keep it from overheating.4

The next, and, in my opinion, morally ambiguous, definition we have to deal with this the Approach Temperature Difference (ATD), which is where we come up with the “Resistance” values for the different pieces of equipment between the TCS, FWS, HRS, etc. The ATD is defined as:

ATD = Temperature of fluid leaving the device to cool some downstream device - Temperature of coolant fluid entering the device from some upstream device

That definition was written as broadly as possible to make it general. Which possibly makes it useless for understanding. So as a concrete example, let’s look at the Coolant Distribution Unit (CDU) in Figure 1 and Figure 3, below.

The CDU takes as input the FWS supply. In the example of Figure 2, that supply temperature is 35 °C. The CDU supplies coolant to the TCS at 40 °C. Thus,

ATD_CDU = TCS_Sup - FWS_Sup = 40 °C - 35 °C = 5 °C.

So, now we have some common understanding of what we mean by resistance in data center cooling. We’ll also have a lot more to discuss about what these things actually mean in real life in later posts. Including why the ATD is really ambiguous and sometimes very misleading and unhelpful. But for now, we know enough to continue.

Moving the heat - a moderately good case

The example above seems great. But it assumes that the ambient temperature outside the data center is 24 °C. In many locations, the ambient temperature will be much warmer than this for at least some fraction of the year. On a summer afternoon, there are many parts of the world where the temperature will be equal to or exceed 40 °C. What then?

In the last example, the temperature of the coolant entering the servers needs to be about 40 °C, which we will now assume to be our ambient temperature. But there was an additional temperature “cost” of 16 °C due to thermal resistances in the CDUs, heat exchangers and heat rejection equipment. So, we actually need to make the heat go “uphill” by 16 °C so that we can have 40 °C coolant enter the servers and still reject heat to 40 °C air outside the data center. Typically, we will use a vapor-compression cycle, as in a refrigerator or air-conditioner, to move heat from a colder region to a warmer region.

The vapor-compression cycle takes energy as an input and moves heat from cold to hot as an output. The COP of a vapor compression cycle is defined as the ratio of the heat energy moved to the required energy input and can range from, say, 2 in a very challenging environment to 6 or even 10 when the temperature difference between the cold and hot sides is pretty small.

To illustrate, we will keep the example the same, but replace the dry cooler, which is just a radiator with a fan, like in the cooling system of an internal-combustion engine in many cars, with an air-cooled refrigerated chiller. The temperature of the HRS coolant leaving the chiller will be 34 °C. If the thermal resistance of the condenser, or heat rejection system of the chiller, is also 10 °C, like the dry cooler, then the chiller must raise the temperature of the refrigerant flowing through the condenser to at least 50 °C (10 °C above the ambient). Note that 50 - 34 = 16 °C “uphill”, as promised.5

Given the high temperature of the HRS supply and the relatively low temperature of the outside air, we estimate the chiller COP to be about 6. To move 1 MW of heat, then, we require 167 kW of power input. Together with the pumps used in the TCS, FWS and HRS loops, we need about 197 kW to provide the same 1 MW of cooling as the example above where the ambient was 24 °C. That’s an increase of 80% in cooling power in this second case just because we can’t slide down the temperature hill all the way from the electronic devices to the ambient.

Wrapping up

• Moving heat “downhill,” from hot to cold, requires the least power input for a given heat load

• Thermal resistance tells us the thermal energy potential (temperature difference) required to move a given amount of power

• Due to a combination of thermal resistance and the local climate where we need to reject the heat, we sometimes have to move heat “uphill,” from cold to hot

• In the above examples, moving heat downhill only used 44% less power to cool. This is directionally true in real life, even if the exact numbers are not.

• Increasing the TCS supply water temperature makes it easier (and less expensive) to move heat to the ambient around a data center. This is one of the main motivations for engineering lower thermal resistances.

What about air cooling and how do those technologies compare? What about other forms of heat rejection, like cooling towers? I thought cold plates were supposed to be super great heat transfer devices, but they caused the biggest temperature difference in the path from device to ambient. What’s up with that? What about cost? Just because a system is energy efficient, is it really the most economical system, all things considered?

These are some of the many questions we will discuss in upcoming posts.

Data Centers. Surprisingly controversial when, in fact, a significant portion of humanity would struggle to get through a day without interacting with one. By number, the vast majority of data centers are relatively small, consume kilowatts to megawatts of electrical power, and handle the mundane tasks that keep us on time, up to date, and sometimes overly attached to our phones. And despite all of the talk about new technologies, most of these data centers are cooled much the same way we cool other buildings: cool down air, blow that air across the hot things, take the resulting warm air and cool it down again. But higher powered computers benefit from different cooling strategies, and we’re going to try to explain what those strategies are and, though not in this first piece, what the benefits are for each.

It is important to begin with the observation that, in obedience to the First Law of Thermodynamics, every watt of power that enters a data center has to be removed from the data center. In this case, on average, every watt of electricity coming in is converted to heat. Sometimes that is a very quick, direct conversion, like resistive losses in copper conductors. At other times it is more round-about, converting electricity to motion to moving air and then to heat by friction of that moving air with itself and other surfaces in the room. If we don’t remove the thermal energy as fast as it enters the data center, the thermal energy will be stored in the mass of materials inside and the temperature will rise until bad things happen. If we remove heat faster than energy is delivered, the average thermal energy stored in the data center will decrease and the temperature will fall. The goal, then, is to match the rate of heat removal with the rate of energy delivered so that the thermal energy in the data center remains relatively constant at an acceptable, predetermined set point.

Removing the heat can be thought of as a three-step process. First, we must collect it from the places where it is generated. Last, we must reject the heat, generally to the environment directly surrounding the data center, though there are many creative ways to send the heat to more useful ends. In the middle, we must move the heat between the collection point and the rejection point. So three steps: Collection, Transport, Rejection. We will walk through some typical ways data centers implement each of these steps.

Share

A Data Center Cooling Overview

Let’s take a look at a data center with IT equipment that uses both Direct Liquid Cooling (DLC) and air cooling. There are many definitions of DLC, actually, so to be specific, we are discussing the kind where a liquid is pumped to a cold plate, or heat exchanger, mounted directly to the electronic device or devices that need to be cooled. The diagram in Figure 1 shows the key components of a data center cooling system. Please be aware that this is just a simplified example. There are many variations and optimizations in the field.

We have divided the data center into three major sections. The “white space”, the “gray space” and outside the building. The white space is where the racks of compute and networking are installed. The tiles used for raised floors are often white, or mostly white, hence this is often literally a white space. The gray space roughly refers to the areas of mechanical and electrical support equipment. Besides the large pumps and heat exchanger shown in Figure 1, this space will typically house the medium voltage switch gear, UPS batteries, and even the backup generators. The concrete floors and, often, the paint on the electrical equipment, are gray and thus the logic of “gray space.” The heat rejection equipment is placed outside the data center since its purpose is to reject the heat to the surroundings.

Why might a data center deploy three loops like this? As with most things engineering, it comes down to optimizing cost, maintenance and performance.

The TCS

The Technology Cooling System (TCS) is the equipment in the data center that actually cools the IT equipment (ITE), as highlighted in Figure 2. This equipment can be the air handlers and fans in the case of air cooling. For liquid cooling, it is the pumped piping loop that operates in close proximity to the equipment cooled with DLC. It includes the cold plates that are mounted to the electronic devices, the hoses and piping in the servers, the manifolds that distribute flow to all of the servers in the rack, the hoses that connect the manifold to the in-rack CDU or row piping, and the in-rack or in-row coolant distribution unit (CDU). It is composed of special materials that won’t react in any way with the special, highly filtered and treated coolant that flows from the CDU, through the cold plates and back again.

The TCS cools the IT equipment. This equipment needs to be available 24/7/365. Any downtime is costly. The cold plates that cool the devices almost always have intricate passageways that enhance heat transfer and make it possible to cool very high-powered packages with warm coolant (45 °C/ 113 °F). These passageways are often on the order of 100 microns (0.004 in.) in width.

Thus, the TCS coolant is carefully treated to prevent corrosion and biological growth of any sort. It is filtered to prevent particles from clogging the small passageways. Still, the chemical treatment of a water-based coolant is often minimized in order to retain as much as possible the great heat transfer properties of water. And in a small, but growing number of cases, the TCS coolant is a dielectric or refrigerant for two-phase heat transfer; very different from the water-based coolant used in the FWS and HRS.

The piping, hoses, valves and fittings of the TCS are carefully selected from a small number of materials that will not corrode or react with the TCS coolant. This often forces the majority of the pipe work to be made of stainless steel, copper, and specialized elastomers. If the cost of these materials were not enough, each component needs to be finished to minimize roughness, and joints must be specially designed to avoid places where particles or biology may accumulate. Of course, the system mustn’t leak, so there is extensive testing and verification of every component.

For all of these reasons, the TCS is typically expensive relative to other piping systems of a similar size. It is generally recommended that a data center deploy many smaller TCSes connected to the same FWS in order to optimize the expense, minimize the chance of contamination as servers are added and removed, and to minimize the blast radius, or affected zone, if there is a problem.

The FWS

The Facility Water System (FWS) is a pumped piping loop that typically operates within the data center walls and provides cool water to the devices that actually manage the data center IT equipment heat. This is highlighted in Figure 3. These IT cooling devices can be air cooling equipment like Computer Room Air Handlers (CRAHs), Computer Room Air-Conditioners (CRACs), Rear Door Heat Exchangers (RDHX) or In-Row Coolers (IRC). Where DLC is implemented, the FWS provides cooling water to Coolant Distribution Units (CDUs) either in the rows of compute racks or installed in the compute racks themselves.

Every system in the data center is mission critical and is essential to data center uptime. But the critical components connected to the FWS are typically heat exchangers with passages that are at least 10x larger than those inside cold plates. Thus, filtering requirements can be somewhat relaxed. These heat exchangers can be oversized a bit so that the FWS coolant can often use higher concentrations of corrosion inhibitors and biocides, permitting a wider range of materials. In addition, the FWS may actually operate at very low temperatures and may need to deploy higher concentrations of anti-freeze for freeze protection.

So, while coolant quality remains important, the FWS piping can often use cheaper materials than those used in the TCS. Even when extensive redundancy is built into the FWS, it is often much more economical to build out a larger FWS piping loop connected to several smaller TCSes, rather than one large FWS connected to one large TCS.

Subscribe now

The HRS

The Heat Rejection System (HRS) comprises the pumps, piping and equipment used to reject the data center heat to the surroundings. This equipment may include cooling towers, refrigerated chillers, and dry coolers, or combinations of all of these. Figure 4 highlights some example HRS equipment.

Dry Coolers are essentially radiators with fans attached. The radiators are generally made with tubing that makes a serpentine pattern through a large set of plates (fins), which are attached to the tubes to add surface area. Air flows through the plates and removes heat from the coolant flowing through the tubes.

There are a number of designs for Cooling Towers, but conceptually, they spray water into a volume of plastic “packing” that has a lot of gaps so that air can flow upward while water flows downward. This allows the water to form a thin film on the packing with a large surface area. Air is drawn upward by big fans through the packing over this thin water film, causing some of the water to evaporate. The water leaving the cooling tower is cooled through this evaporation process down to the local dewpoint temperature. The dewpoint temperature is typically much lower than the outside temperature (dry bulb temperature), though the higher the humidity, the closer the dew point is to the dry bulb temperature. Evaporation is a very energy efficient way to cool water, but clearly some water is consumed in the process, which is a concern in many localities.

Mats of porous material can be mounted to a dry cooler; they can be sprayed with water only during those hours when the dry bulb temperature is high. This can allow for a more optimal balance between cooling efficiency and water usage.

Refrigerated chillers employ vapor-compression cycles, like a home air-conditioner, to reject the data center heat. They are generally used where the outside temperature is frequently quite warm and dry coolers are not a viable option for the temperature requirements of the FWS and TCS. Refrigeration systems don’t usually consume water, but they often use a great deal more energy than either dry coolers or cooling towers.

There are many advantages in having a third coolant loop for the heat rejection equipment. First, this allows some standardization of the FWS and TCS piping without concern for the heat rejection design, which may vary due to local climate, space and other constraints. In some cases, the coolant may come directly from a cooling tower whose water must be treated for direct exposure to the ambient and would often not have chemistry that would be compatible with other FWS components. In addition, the coolant flow rates may want to be different in the HRS than in the FWS to meet equipment operational constraints.

Three > one + one + one (often, anyway)

Three separate cooling systems can have several advantages over one.

• IT equipment can be standardized into scalable units or pods with identical TCSes. This makes it easier to design, sell and deploy liquid cooled IT into a wide variety of data centers

• The FWS can be designed independently of the HRS, allowing for the same or very similar data center designs to be used in a wide range of climates and locations.

• Cost is often optimized by minimizing the extent of the much more expensive TCS.

• IT blast radius is minimized by using smaller TCS on larger FWS networks.

On the other hand, there are some creative proposals for eliminating one or more of these systems in a data center. Many data centers are deploying large, data hall scale TCSes despite the advantages to smaller systems laid out above. In the end, each data center will evaluate what works best for them in the optimization of cost, performance and maintenance.

Further reading

Your first stop for data center thermal management information should be the ASHRAE Datacom Encyclopedia. The subscription is cheap and it is updated several times a year by industry experts.

While you’re at it, read the other ASHRAE TC 9.9 publications, or better yet, join the committee and contribute. ASHRAE Technical Committee 9.9 website.

A lot of good information is being produced by the industry-wide collaboration at the Open Compute Project:

• https://www.opencompute.org/community/data-center-facility

• https://www.opencompute.org/community/rack-and-power

• https://www.opencompute.org/community/cooling-environments