IDC predicts that the Global Datasphere will grow from 33 Zettabytes (ZB) in 2018, to 175 ZB by 2025. By 2025, half of the world’s stored data will reside in public cloud environments.

The upshot is that this massive amount of data will need compression.

For companies like Microsoft, the ability to store and process data efficiently is core to the cloud’s value proposition, and its Azure solution relies on an algorithm and optimized hardware implementation to manage this challenge.

In a blog post, Kushagra Vaid, General Manager of Azure Hardware Infrastructure, says that by engineering innovation at the systems level, the company has been able to achieve higher compression ratios, high throughput and lower latency than the other algorithms currently available, and it’s now going to open source that technology.

‘Project Zipline’ is Microsoft’s compression algorithm, that yields “double high” compression ratios versus the commonly used Zlib-L4 6KB model. Now, this algorithm, along with hardware design specifications and Verilog source code for register transfer language (RTL) will be available publicly.

“Over time, we anticipate Project Zipline compression technology will make its way into several market segments and usage models such as network data processing, smart SSDs, archival systems, cloud appliances, general purpose microprocessor, IoT, and edge devices,” said Vaid.

The technology is optimized for a large variety of datasets, and the release of RTL allows hardware vendors to use the reference design to produce hardware chips to allow the highest compression, lowest cost, and lowest power out of the algorithm.

It’s available to the Open Compute Project (OCP) ecosystem, so they can contribute to it, and create further benefit for the entire ecosystem, including Azure and its customers.

Vaid anticipates Project Zipline compression technology will make its way into several market segments and usage models such as network data processing, smart SSDs, archival systems, cloud appliances, general purpose microprocessor, IoT, and edge devices.

To put things in perspective, today, more than five billion consumers interact with data every day – by 2025, that number will be 6 billion, or 75 percent of the world’s population.

In 2025, meanwhile, each connected person will have at least one data interaction every 18 seconds. Many of these interactions are because of the billions of IoT devices connected across the globe, which are expected to create over 90 ZB of data in 2025.