How much should we trust explainable AI?

Researchers argue that logical explanations from providers can't be universally trusted.
13 October 2020

Can we trust what we’re told is in the black box? Source: Shutterstock

  • Researchers argue that an AI system’s explanations may not actually be as transparent or truthful as previously assumed
  • The researchers expose how logical explanations from a provider can always be misconstrued, which can be near impossible to detect for an isolated user due to the sheer space and scope for a possible attack

As machine learning and artificial intelligence (AI) applications play a growing role in the services we use, knowingly or not, there is a greater emphasis than ever on the need to be able to understand how these systems are making their decisions.

If a bank refused a mortgage or loan application based on an algorithm, for example, the customer should be entitled to know why.

Explainable AI is now a principle of ethical artificial intelligence. The UK’s Information Commissioner’s Office (ICO) even put forward a regulation that businesses and other organizations are required to explain decisions made by AI or face multimillion-dollar fines. 

At the root of this drive is trust: by being able to access and understand an algorithm’s decision-making process, society will more openly accept the technology.

In an attempt to better understand how explainable AI may actually impact users’ trust in AI, scientists at the French National Center for Scientific Research and Bretagne Atlantique Research Center carried out a study on AI systems that provide explanations for their actions or predictions.

The paper published in Nature Machine Intelligence, argues that an AI system’s explanations may not actually be as transparent or truthful as previously assumed.

“This paper originates from our desire to explore an intuitive gap,” said Erwan Le Merrer and Gilles Trédan, two researchers who carried out the study, in an interview with TechXplore.

“As interacting humans, we are used to not always trusting provided explanations, yet as computer scientists, we continuously hear that explainability is paramount for the acceptance of AI by the general public. While we recognize the benefits of AI explainability in some contexts, we wanted to make a point on its limits from the user perspective.”

Bouncer problem

In contrary to popular opinion, the researchers discovered that AI explanations may not have as much value as previously believed. While the study states that explanations might have value in local contexts like providing useful feedback to developers who are trying to debug a system, they aren’t as useful in remote contexts seen in AI system’s trained and managed by a specific service provider, where decisions are delivered to users via a third party.

The paper sought to expose how logical explanations from providers can’t be universally trusted.

Drawing a comparison with bouncers outside of nightclubs refusing a guest entry, researchers said remote service providers could lie to users about the reasoning behind an AI’s predictions or actions.

The work in question dismantles the widespread belief that explanations will increase users’ trust in AI systems, and rather concludes the opposite.

Through the use of practical examples of how the explainability of AI systems in remote contexts could be affected by the “the bouncer problem”, it’s hoped that their work will inspire future studies, as well as the development of alternative solutions such as; in-premises white-box algorithm auditing and cryptographic approaches, for increasing people’s trust in AI.

“We plan to continue studying AI systems from the users’ perspective, particularly exploring this question: What can regular users discover/learn/understand/infer about the AI systems that shape a growing part of their life?” Added Le Merrer and Trédan.