In the second part of this series we take a closer look at the specific design, build and operation of cryptocurrency data centres. As we saw in part one of this series, there is now an increasing trend towards data centre capacity that is designed and operated to support a specific set of applications. For example some public cloud service provider sites achieve high levels of energy and cost efficiency through the use of infrastructure that is heavily tailored to support custom applications running on custom hardware. Many cryptocurrency sites follow a similar model; in fact some facilities are so tailored to the single purpose of cryptocurrency mining that they more closely resemble warehouses or other industrial buildings than traditional data centres. They may not always be elegant but they are usually highly efficient, very cost effective and fit for purpose.
IT equipment
Historically Bitcoin and other kinds of cryptocurrency have been mined using conventional hardware including PCs. However over the recent past there has been a considerable escalation in the development of specific IT hardware to accelerate the speed and efficiency of mining operations. This has seen mining move from equipment based around central processing units (CPUs), to graphical processing units (GPUs), to field programmable gate arrays (FPGAs) and most recently to application-specific integrated circuit (ASICs). ASIC-based systems have now proliferated as they provide competitive processing speeds at lower power requirements than FPGA or GPU-based systems.
But although the efficiency and power of cryptocurrency mining hardware has improved the escalation in the required compute means that mining operations often still require high-rack power densities and specialist cooling equipment. Using ASIC-based systems typical rack densities can easily exceed 20kW compared to less than 5kW per rack for typical enterprise facilities. More power-dense sites can incur higher costs because of the additional cooling, power distribution and other requirements. But this is offset to some degree by the fact that most mining operations are more tolerant of downtime and don’t require the high-levels of redundant mechanical and electrical (M&E) systems of conventional mission-critical sites. This reduces the M&E fitout cost for a dedicated cryptocurrency site to 40% or less of the $6.5 million per MW of a more typical Tier III (certification scheme operated by data centre resiliency specialist Uptime Institute) colocation or enterprise site.
Dedicated versus colocation
Given the design and build costs for dedicated cryptocurrency sites, multi-tenant datacentre (MTDCs) operators offer a low capital alternative to dedicated sites. However, historically most MTDCs have been built out to the lower density requirements of more typical enterprise workloads; this could make some MTDCs either not suitable or more costly for cryptocurrency mining operations. According to estimates from high-performance compute (HPC) specialist Allied Control (now owned by Bitcoin company BitFury) a cryptocurrency MTDC tenant requiring 1.25MW of IT capacity could spend more than $3.5 million annually on colocation costs. However this will vary significantly depending on the specification and location of the MTDC.
Location, location, location
Given these costs, cryptocurrency miners are increasingly looking to use colocation operators in regions with low-energy costs such as the Nordics. A number of MTDC operators in this region have cryptocurrency clients and some have even developed specific services to appeal to these customers. For example, Iceland-based operator Verne Global has a power distribution and resiliency service called Power Direct that has optional UPS coverage and is designed for cryptocurrency and other high-density users. Verne says the lower resiliency requirements and related power costs means Power Direct is up to 80% cheaper than some rival MTDC capacity. New cryptocurrency sites are also being built out in China, which accounts for a large percentage of existing capacity, Georgia and other areas with low-energy costs.
Dedicated sites: opportunities and challenges
Future-tech has completed four designs for cryptocurrency sites this year and expects to see more in the future. For those mining operations that opt for dedicated sites, cryptocurrency data centres enjoy some benefits compared to conventional sites in terms of flexible designs, connectivity and other factors. These include:
Resiliency – In locations such as the Nordics where there is usually a highly stable grid there may be no requirement for the usual back-up generators or UPSs required in mission critical sites. If an individual mining unit fails, it can be relatively easily replaced. Even if the entire site is down for a period, while inconvenient, the cost of a short interruption is fairly negligible. This has led some cryptocurrency sites to be described as ‘Tier 0’ to reflect that they don’t meet the lowest levels of redundancy on Uptime’s Tier I to IV framework. Some sites may also be built out without expensive fire-suppression systems or other safety equipment.
Latency – Another benefit enjoyed by cryptocurrency sites is that they have little or no latency requirements. For that reason they can be built outside of key data centre locations such as London, Paris, or Frankfurt and take advantage of the low-energy costs, cheaper real-estate and free-air cooling in less developed markets.
Free air-cooling – Iceland and other regions in Northern Europe are not only attractive because of low energy costs and stable grids but also because of temperate climates. This allows approaches such as direct and in-direct free air-cooling with little or no requirement for costly mechanical cooling which can account for up to 40% of facility capital costs. The fact that cryptocurrency hardware is replaced so frequently means that it is less likely to be in place long enough to suffer from the effects of air-based contaminants associated with some direct free air-cooling facilities. Some cryptocurrency operators take free-air cooling to an extreme and will simply open up the doors of a building to let in outside air directly with no filtration at all. One of Future-tech’s clients is converting an internal louvred plant area, intended for future phase M&E equipment, to provide a short term crypto focussed area for a specific end user deployment.
Management – The relative simplicity and homogeneity of cryptocurrency hardware and facilities M&E means that a small pool of staff with specific skills will probably be sufficient for operational management.
However cryptocurrency sites also face some specific challenges associated with power density and cooling requirements:
Equipment replacement – The hardware refresh rate has some benefits but the requirement for competitive mining sites to replace IT hardware, in some cases every six months, can create additional costs and complexity when it comes to operating large-scale operations.
Power density – Depending on the density of the deployment miners may need to deploy extreme forms of high-density cooling which has direct costs but also benefits. This can include technologies such as direct liquid cooling (DLC) where the processor or other components are in direct contact with, or full immersed in, a liquid coolant. Future-tech has seen growing demand for the high-density cooling technology and undertaken a number of M&E designs to support DLC systems. Some immersion liquid cooling suppliers have also developed containerized, modular DLC data centre designs. As with other types of prefabricated modular data centre (PFM) these modules can also be deployed in a relatively short time, 8 to 10 weeks – compared to some traditional data centre designs.
Speed to market – The competitive nature of cryptocurrency mining means there is often considerable pressure to bring new capacity online quickly. This can present challenges using conventional design and build processes. PFM designs offer a way to bring new capacity online but innovative engineering services companies, such as Future-tech, have also developed ways to reduce traditional build times.
Future direction
The development of future data centre capacity is obviously dependent on the continued growth and adoption of cryptocurrencies. However it is likely to mirror some of the trends in the rest of the data centre industry including more widespread use of PFM designs but also more workloads migrating to specialist cloud mining services. However there is also the potential that some mining capacity may migrate back to highly distributed networks of specialist desktop systems. Whatever the future, cryptocurrencies look set to continue to be a source of innovation and disruption.
If you would like to learn more about workload specific data centre design or discuss the impact of cryptocurrency-optimized infrastructure in your data centre contact us on info@future-tech.co.uk