Big Data: Scaling Digital Innovation With Responsible Data Democratization

One of the big business challenges to which big data can deliver solutions is the scaling of digital innovation. It’s a problem that depends on a kind of growth that doesn’t simply depend on more data, but on the intelligent development of frameworks in which more data can be handled in such a way that the scaling is natural and managed, rather than more data simply resulting in more pressure, more stress, and eventually a failure point within the innovation pipeline.

In his keynote speech at Big Data London 2022, Lathik Ravindran, General Manager Enterprise Data and Analytics at Microsoft, addressed the issues of scaled innovation and data democratization, described ways in which to avoid pitfalls, create focused harmony, and ultimately develop a “Digital Virtuous Circle” that balances technology considerations and business investments.

The Three Pillars of Data Post-Primalism

“I’ve never seen a subject as contentious as data,” he announced, suggesting he’d never heard of Brexit, the “stolen election” of 2020, Meghan Markle, or the skin color of Disney mermaids. “It brings out the primal instinct in people.”

But rather than allowing the audience to descend into anarchy, with data scientists bludgeoning CFOs to death with their prediction models, Ravindran was all about the peace, love and harmony. “To get beyond that primalism, you have to start with strong data leadership. And data leadership has three constituent parts:

1. Technology
2. Data culture and literacy, and
3. Change management.

Get that right and then you get the credibility, and the collaborative approach, and the strength even to call out your leadership if they’re not delivering the things you need.”

There’s a tendency in many technology and data-driven businesses towards expecting innovation through data siloing, data hording, and particularly the notion of individual advancement irrespective of team structures. Ravindran was having none of that.

This Is Not Political Science

“Data strategy is not a political science,” he told the room. “It’s a team sport. And a lot of the big conversations these days are a complete distraction. Centralization vs decentralization? That’s a political conversation. Close it down and do it now.”

He took a moment for an explanatory sidebar of this fairly nuclear statement. “There was a long debate between a tech team and a customer experience team over the building of a data product. If IT built it, it was centralized. If an external team built it, it was decentralized. But if you step back from that polarizing debate, the fact was that there was probably only going to be one copy of the product, and it would be used by the company as a whole – so you tell me what difference the centralization question ultimately made? Data leadership is about being able to look at the bigger picture, and having the strategic strength to step aside when necessary for the greater good, rather than getting entrenched in turf wars between teams.”

Next, he made a point that, while built on the initial nuclear premise of team play, was much easier for the room to grasp. “You need to invest in the data fundamentals, in data culture and data literacy. Adopt the idea that data is an organizational asset, rather than an individual asset or a team asset. If you think that way, strategically rather than being restricted to a narrow problem-focus, you enable the broader responsible democratization of data towards organizational goals.”

Make Friends With Technology

His advice on data literacy was a call to put out fires. “To deliver good data leadership, you have to address the primal fear that automation will lead to job destruction. You need to educate your teams and organizations on the ways in which AI will help staff, the ways in which will enrich the process. Show them how technology is going to transform their roles – for the better.”

On data architectures, Ravindran was prescriptively non-proscriptive. “Stop arguing about which architecture is best,” he advised. “Modern data architectures are not mutually exclusive, they’re collectively transformative. Our own architecture is a mixture of a lot of different architectures – whichever is best at a particular task.” There is of course a slight whiff of multinational budget privilege in this outlook. It’s undeniably right, but it’s right in the same way that lottery winners can have a garage full of luxury sports cars to meet their driving needs on any given day. Companies or departments with tight budgets may not find themselves with the Microsoft luxury of building a whole umbrella of interconnected architectures. But the message was clear – if you can, do. Don’t unnecessarily restrict your architecture options, but get the benefits of using the “best-for” option for each major task.

The Vista Philosophy

“Good data leadership emphasizes iterative progression over perfection and completeness,” he added. “Get started – the chances are you’ll be forever improving, iteration by iteration.” The irony of this advice coming from the company that launched both Windows 8 and Windows Vista on the world was widely appreciated, but Ravindran’s point was quickly underlined. “If you start, with an iterative process, you can evolve constantly. If you wait for completeness or perfection before you start, you’ll never start. Shift the mindset of your organization from perfection to progression. We went from one or two IT teams to teams all over the world.”

And finally, when it comes to aligning data and digital innovation, Ravindran had considerations for both data officers and data citizens (those who use data for their companies). For data officers, his advice was simple but emphatic. “Democratize your data responsibility, but do it as a champion of enablement. That idea of the data as an organizational asset means whenever possible, you should be letting people and teams that could use the data productively have access to it.”

And for those data citizens who wanted access to data to do things that perhaps weren’t within their daily remit, his advice was simpler still. “Do it. Differentiate in responsible domain innovation. Work with your data officer to get the access to the data you need. If you have permission to do it, just do it.”