Why Big Tech is quietly collaborating on open source AI

• Big tech companies have traditionally been private with their source codes, libraries, and methodologies
• But there are advantages for all in teaming up on AI technology 

Artificial intelligence (AI) is a hype phrase that comes with a lot of baggage: will its potential ever be realized; will it enhance humans, or make them obsolete; is it really that revolutionary?

But one area of the debate that is often overlooked — and is one of the more positive aspects of modern innovation, in fact — is the way big tech companies like Google, Amazon, Facebook, and Microsoft are working together to help progress AI. These companies have been the focus of much criticism over the last few years, consolidating their influence and dominating specific parts of our lives but when it comes to AI, something is different. That something is open source.

The sheer number of open source tools available to developers — from libraries to frameworks, IDEs, data lakes, streaming, model serving, and inference solutions, and even the recent end-to-end tool aggregator, Kubeflow — means businesses can now harness all the knowledge they have accumulated over the years. Open source has become that common ground for companies that would traditionally be seen as competitors. They are already working closer than many suspect, and it is the unique combination of AI and open source that will foster better relationships moving forward.

Why open source?

The benefits of open source technology to IT development have been well documented over the years. The first open-source program, the launch of Netscape Navigator in 1998, is said to be one of the initial starting points for this trend. The strategy Netscape chose was to emphasize the business potential of sharing the software’s source code. As with science, if all researchers kept their methods secret, progress and innovation would take place much more slowly. With developers racing to deliver “the next big thing”, secure and easy-to-deploy software frameworks are essential to support this.

However, there are numerous barriers for businesses looking to develop successful AI-based technologies. It is no secret that AI and machine learning development can be an expensive process. Not only that, but development requires significant computing power and data sets in place to build and train an advanced model. The open-source community offers potential solutions to these challenges by encouraging collaboration as well as expertise and resource sharing. For example, open-source software allows IT teams to access frameworks, data sets, workflows, and software models in the public domain and reduces training costs. At the same time, the open-source community is always monitoring the code for flaws and vulnerabilities — adding an extra layer of security and making such concerns a common responsibility.

In another front, the rise of ops methodologies has greatly increased the efficiency of developers to bring solutions into production. Kubernetes is the open-source platform that automates the deployment and management of containerized applications and has become a mainstream technology for enterprises to get into DevOps. It’s now being extended to Machine Learning ops (MLops), allowing complicated AI workloads to be kept up to date.

Big tech, bigger surprises

It will come as little surprise that big tech companies have traditionally been private with their source codes, libraries, and methodologies. So, what makes AI the differentiator for these giants to begin revealing methods from the core of their businesses and unleashing their own open-source APIs?

Essentially, the progression of AI remains paramount, and big tech companies have made leaps and bounds in developing technology over the past few years. Open source allows any developer or IT team to facilitate cheaper, faster, more flexible, and secure deployment. Development through open-source helps accelerate the adoption of numerous frameworks and software solutions through support from a large community of contributors. So, with big tech embracing open source, their work can then be further developed, explored, adapted, and improved. Looking to the future where AI is expected to be ingrained into everyday life, open-sourcing AI will foster innovation and reach maturity even sooner.

Google is one company leading the way, having made its popular machine learning framework, TensorFlow, open to the public. This subsequently led to the creation of TensorFlow Extended (TFX) which matured into Kubeflow, an open-source project designed to enable using machine learning pipelines to orchestrate complicated workflows running on Kubernetes, all based on Google’s internal method.

Meanwhile, Facebook has open-sourced DeepFocus, its AI-powered framework for rendering natural, realistic focus effects in virtual reality (VR), and the Microsoft Cognitive Toolkit has also embraced open source with the ultimate aim of training deep learning algorithms to function like the human brain.

It’s also important to remember that all big tech companies are made up of people. In particular, the machine learning field has a long history of being very open and cooperative. Those running the ML labs inside of Google, Facebook, and Microsoft have been pioneering the field for decades and have always worked transparently and cooperatively. A key reason we have seen such major advancement is precisely that effective cooperation.

It all comes down to trust

Ultimately, trust is the key factor here. Big tech companies have seen the trust in them dwindle because of the consolidation of personal data they hold, and the perceived notion of the unnecessary power and influence it allows them to hold. But now, these companies are becoming completely open as it relates to the development of this technology, which will form the backbone of our future.

Even over the past six months which have been typified by massive world change because of COVID-19, tech collaborations have emerged, such as Apple and Google jointly working on a contact tracing solution.

Going forward, these ongoing relationships will likely spark a new era whereby big tech and even public sector organizations work together to foster innovation and help overcome crises. And with such established tech companies betting so heavily on the openness of AI, its development will continue to transform and flourish in the near future.

The article was contributed by Rui Vasconcelos, Product Manager, Canonical (the company behind Ubuntu)