Is this the (worrying) future of hybrid work technology?

A patent filed by Microsoft hints at the normalization of employee monitoring, tailored to a world of hybrid work.
1 December 2020

Diagram accompanying Microsoft’s patent application. Source: Microsoft

Zoom leveraged the boom in videoconferencing uptake early on in 2020, and competing products — such as Teams, Meet, and Slack — quickly answered in a bid to differentiate from the current market-leader and entice users back to them.

Fundamentally, though, these tools don’t differ greatly from one another. They offer an audio-visual feed between users, in real-time and that serves the perfunctory purpose.

Hybrid working is ‘here to stay’, however — and with Grand View Research predicting the video-conferencing market could reach a worth of US$8.6 billion by 2027, growing by nearly 10% annually — many businesses are racing to innovate, transform, and lead the new virtual meeting ‘experience’.

  • Nvidia, for one, is developing artificial intelligence software that dramatically reduces bandwidth consumption to increase call quality. The technology works by replacing traditional, data-heavy video frames with neural data — it’s unfavorable been compared to “deepfakes in real-time”.
  • Zoom, meanwhile, launched a videoconferencing hardware-as-a-service, offering “easy access to hardware” which it said was “critical for offices, distance learning, telehealth, and more.”
  • Much noise has been made about the role of mixed reality technologies in making distanced interactions and connections more natural and engaging in 360-degree environments, complete with whiteboards, and more. HTC Vive recently launched an XR videoconferencing suite which is designed to “remove the physical distance” from remote working.

All these advances are geared at solving certain technical challenges that affect the quality of our colleague interactions, and subsequently, our organization’s net productivity in a new era of distribution working. But a newly-filed patent by Microsoft portends more worrying developments, where our reliance on technology to keep in touch with each other is exploited to monitor and benchmark employee behavior and performance.

The rise of employee monitoring?

As reported by the BBC, the tech giant has filed a patent for a system to monitor employees’ body language and facial expressions during work meetings, in order assign events a ‘quality score’. The filing suggests it could be used in both real-world meetings, in rooms filled with sensors, or in virtual catchups.

The filing comes after Microsoft released a controversial productivity score tool, enabling managers to keep track of individual workers’ use of Microsoft Office 365 software — including Teams, Outlook, and Excel. Critics have called the functionality — which isn’t enabled by default — an employee surveillance tool.

Microsoft’s “meeting insight computer system”, as it’s outlined in the patent, could record which invitees actually attend the meeting, body language, and facial expressions, the time participants spent contributing to the meeting, and speech patterns “consistent with boredom and fatigue”.

It also suggests that employees’ smartphones could be used to monitor whether they were engaged in other tasks such as texting or browsing the internet.

All data could be combined to determine meeting efficiency, participants’ sentiment, and how comfortable the meeting was — providing it with an “overall quality score”.

“Many organizations are plagued by overly long, poorly attended, and recurring meetings that could be modified and/or avoided if more information regarding meeting quality was available,” the patent said.

People analytics

The worry is that this type of technology would also be used to monitor employees on the basis that there is a ‘normal’ way of working, which could have a hugely negative impact on the culture of trust within the business.

Of course, tech companies file patents all the time — it doesn’t necessarily mean a product will come into fruition, but they do frequently indicate the direction one might eventually take.

The pandemic of 2020 has ushered in a new appreciation for how technology can enable better work and productivity in business, keeping the wheels turning even in the most challenging of circumstances. But it has also given employers more license to deploy employee monitoring tools.

‘People analytics’ has long been a controversial concept — e-commerce giant Amazon is perhaps the most infamous user, using software to monitor the length of breaks taken or products picked by warehouse employees, for example.

A report by The Guardian last year found UK businesses were using the Isaak system to analyze staff behavior by extracting data on email transactions, edits on files, and interactions that are being logged in the system.

Some 130,000 workers across five law firms, a training company called Smarter Not Harder, and London estate agency JBrown, among others, were being ranked by the tool, giving bosses a snapshot of whether these workers are “collaborative”, “influencers” or “change-makers”.

Fearing the unfair judgment of the algorithm is the central issue, and critics say that every action (or inaction) being logged could lead to workers not wanting to take breaks or spend time in creative thought. Any time not typing, moving the mouse, or interacting with the computer could be interpreted as time-wasting.

“If performance targets are being fine-tuned by AI and your progress towards them being measured by AI, that will only multiply the pressure,” said Ursula Huws, a professor of labor and globalization at the University of Hertfordshire, at the time.