• Decision-makers tend to favour AI’s judgement.
• Poorer outcomes from AI than employees’ intuition.
• Employees go against better judgement to keep ‘bonuses’.

New research from the Frankfurt School of Finance and Management and ESMT Berlin has found that workers are penalised for making their own decisions and ignoring AI recommendations, even if a decision is justified and results in better outcomes.

When employees move away from an AI’s judgement, researchers discovered managers lower bonus payments, indicating over-confidence in AI and algorithm-based recommendations.

Researcher Prof. Dr. Mirko Kremer, Professor of Supply Chain Management at the Frankfurt School, spoke of the importance of human knowledge over a reliance on AI.

“While businesses across industries are racing to implement AI algorithms to enhance efficiency and decision-making, it’s important to remember that these systems should not be relied on completely.”

“[…] Human expertise remains essential to navigate complexities, make nuanced judgements, and adapt to unexpected situations that AI might miss – especially in sectors like healthcare, aviation or autonomous vehicles, where the implications of blindly following AI advice can have huge implications.”

The researchers were Prof. Dr. Kremer, alongside Francis De Véricourt, Academic Director of the Institute for Deep Tech Innovation (DEEP) at ESMT Berlin, and Hossein Nikpayam, a postdoc in Operations Management at ESMT Berlin and Frankfurt School. They examined how different establishments use AI-based systems to enhance and speed up work processes. The researchers identified unintended side effects, like potential negative impacts on workers or workflows that arise from an over-reliance on AI and algorithms to handle tasks.

AI’s judgement not always optimum

The study took the form of a lab experiment in which an AI model made recommendations before human decision-makers decided whether to opt for an AI-suggested proposal or use their own judgement.

Managers were tasked with evaluating their decision-makers’ choices and were asked to assign bonuses based on those decisions. In some scenarios, the managers were given either full access to all recommendations, one recommendation, or no recommendations at all.

The study revealed decision-makers depended heavily on the AI’s recommendations, even when not optimal and would likely lead to poorer results. Managers preferred decision-makers to follow AI advice, prioritising adherence to AI guidance rather than the success of the actual outcomes, highlighting a significant bias towards a trust in AI systems.

Such behaviour from managers could result in employees ignoring their own intuition and expertise, even when they judge their opinions to be superior to those of AI systems, in order to avoid the possibility of censure. This creates a cycle where human creativity and thinking are suppressed, and the overall quality of decisions declines.

The researchers’ conclusions contained two messages. The first, that transparency is important. Managers should have a clear understanding of how AI generates its recommendations. This allows them to evaluate the AI’s advice objectively, to make more balanced, informed decisions.

Secondly, there should be more trust in human knowledge and expertise. Organisations should prioritise and value human skills, expertise, and intuition alongside any AI input. The researchers suggest managers should create a culture where employees are not afraid to challenge or even ignore AI recommendations when those suggestions conflict with the employee’s judgement.

A better understanding of how an AI generates its decisions is required, the researchers state, and clearer communication is important for employees to confidently convey their concerns.