Did Amazon Go automate cashier jobs, or relocate them?

As automation profoundly changes the retail environment, some ethical issues are being swept under the rug.
18 October 2019

Amazon Go AI has not yet been perfected. Source: Shutterstock

When Amazon debuted its automated convenience store in Seattle in early 2018, I was impressed with the consumer experience but also aware of the long-term ripple effects. At a time when the self-checkout machine market was expected to grow, due in part to labor shortages and lowered kiosk prices, Amazon leapfrogged over that technology and many others, such as mobile payments.

They went from self-checkout to no-checkout. The company had, in effect, done the exact same thing that allowed it to dominate the e-commerce side of things: They made shopping easier. To use industry lingo, they reduced the friction of transactions. But how?

Amazon Go’s Humans-in-the-Loop

It’s now 2019 Q4 and the AI technologies deployed in its Amazon Go stores haven’t yet been fully perfected. I recently learned that the company still relies on “humans-in-the-loop” who confirm purchases whenever the AI system is uncertain. This work takes place in multiple locations and Amazon did not deny the possibility that some of this work is done overseas, far away from the actual brick-and-mortar stores.

The company maintains that these interventions are infrequent. However, there’s no publicly disclosed or external data to verify that. Therefore, it isn’t unreasonable to suggest that some cashiers haven’t been automated; they’ve been outsourced and their work has been technologically augmented. That may be a transitional state but it fundamentally changes the nature of the conversation.

The political conversation

Currently, some US Presidential candidates are anticipating change and offering up plans to mitigate the effects of technological unemployment. It’s more difficult to do that when there is limited transparency around the state of progress and the operational efficiencies of solutions on the market. There are even differing postures on the recent past. In the recent CNN/New York Times debate, Andrew Yang and Sen. Elizabeth Warren disagreed over the impact that automation has already produced.

Many AI systems are still learning. Amazon Alexa relies on a surprising number of human workers. A Bloomberg investigation revealed that thousands of contractors and employees around the world, bound by NDAs, are listening to audio clips in order to further improve the tech. In 2018, Google unveiled its Duplex system, a form of AI that can sound natural while booking appointments over the phone. The company declared that its system would be able to operate fully autonomously for the majority of tasks, but acknowledged the need for real-time supervised training and human intervention on unusually complex tasks.

The New York Times dabbled around with the service earlier this year and found what it characterized as “a heavy reliance on humans” who sometimes initiated and handled the calls entirely. I told Forbes that this human involvement shouldn’t be misinterpreted as a marketing deception or technological inadequacy. Rather, it’s a natural part of AI progression and is consistent with the philosophy that early stage AI has the potential to serve as an augmentation of the workforce.

But retail is somewhat different. We’re already used to the idea of overseas call centers, which were largely enabled by telecommunication expansions and cost-saving strategies in the 1990s. Google Duplex feels like more of the same. Partly automating and outsourcing the work of a neighborhood cashier is new.

Various automation technologies could disrupt the retail labor force, disproportionately impacting women, who account for 73 percent of cashier jobs. But there are also other sociopolitical implications.

Why cash still matters in automated retail

I’ve shopped at Amazon Go on multiple occasions, in both Seattle and San Francisco. Healthy eating options abound. I can scan a QR code, walk in, grab whatever I want, and then leave. Regardless of what is happening behind-the-scenes, the process allows customers to glide through transactions…

But not all customers. Amazon Go is largely geared toward identifiable Amazon customers with banking cards and smartphones. The checkout is gone but there is a wealth-enabled check-in. According to an FDIC national survey, administered in partnership with the U.S. Census Bureau, approximately 8.4 million households didn’t have bank accounts in 2017.

To prevent discrimination at the point of sale, some states have passed laws requiring all retailers to accept cash. Amazon Go has responded to these local mandates by accepting cash at many of its locations but the process for that is more complicated and time-consuming. It involves signaling for the attention of a store associate in order to clear the turnstile, doing so again at checkout time, and then having items rung up in a conspicuous way, with a cart that is wheeled out from the back of the store.

As automation profoundly alters the retail environment, some of these ethical issues are being swept under the rug. Now, an Israeli computer vision startup is going head-to-head with Amazon Go and emphasizing this aspect of the commerce conversation.

“We don’t think that all the interactions in the seamless checkout world need to be an opt-in, checked-in, identified experience like Amazon Go has.”

Ran Peled, VP of marketing at Trigo Vision, outlined several causes for concern as he spoke with me on the phone from Tel Aviv, starting with the importance of cash in a modern, increasingly digitized world. His company recently announced a partnership with Tesco, the British grocer, which included both an equity investment by Tesco in Trigo and a commercial agreement.

“We don’t think that all the interactions in the seamless checkout world need to be an opt-in, checked-in, identified experience like Amazon Go has,” said Peled. When I asked him about the problem of food deserts, he said that a cashless, automated store could just create a tiny food desert in the middle of an affluent area. He said that Amazon Go’s method of cash acceptance “makes the grocery experience even worse” and “it’s also a bit humiliating.”

Ran Peled thinks that stores should offer different options to consumers, in a well-integrated and seamless way. He told me his preferred version of a cash-based experience: “You enter the store, unidentified, you take the products that you want, and then at the end, you are presented with your shopping list and you pay whatever way you want. And you can completely be unidentified. You can pay cash and exit the store and it’s like you’ve never been there.”

However, the opted-in, more automated experience allows for better data collection, in addition to the cost savings from reduced labor. It might also encourage stronger customer loyalty.

Of course, Amazon Go could resolve Peled’s criticisms, simply by integrating cash acceptance more skillfully and automatically. But they haven’t. He says this is one of the indicators that “the man in the loop is still very dominant.”

It could be. But there could be a technical explanation for some of those manual ring-ups. A customer without an identified smartphone isn’t providing the same level of signals.

The mysterious receipt delays

Ran Peled also pointed to the unpredictable delays with the digital delivery of Amazon Go’s itemized receipts, though this conceivably could be influenced by other technical factors not necessarily related to a human reviewer.

When asked for his interpretation, an established AI researcher told me that the 20-minute receipt delays are most likely not spent in the AI algorithm. The current inference time for image recognition, he said, is less than 300ms per image when running on a reasonable GPU. He said that the particulars of Amazon’s image processing pipeline could multiply that by 2-10 but that would still add up to seconds, not minutes.

And what if the system is just really, really busy? If there’s increased demand, with more incoming images getting queued up, the system might request extra cloud GPUs for processing, which could take a while to spin up. This would be consistent with multi-minute delays during peak demand hours, and no delays in off-peak hours. There could also theoretically be network congestion.

It’s also possible that the company is employing humans for labeling services not only to train the next versions of machine learning classifiers but to meet a high standard. They could be going for impressively high precision, as opposed to lower levels of precision with frequent corrections.

But why are we being left to generously speculate? Amazon is being scant with information that is directly relevant to a global, political conversation concerning automation and jobs. Other vendors have also figured out how to automate retail (Standard Cognition has raised US$86.1 million) and some of them are being more transparent about the way their tech works. Many other factors, such as pricing and inventory, will determine the outcome of this competition.

“I don’t think that they want to reveal either that it’s still being tried and tested, and second, that it might not be as advanced as other companies might think otherwise,” said Peled. He suggested that Amazon Go relies on a probabilistic model with established thresholds and indicated that data is sent to overseas centers in the event of uncertainty.

“They just send a lot of 2D pictures to a center in India and or somewhere else in the Far East,” he said.

When I emailed back and forth with Amazon about this, an Amazon spokesperson provided the following on-the-record statement: “As we’ve shared in the past, when our machine vision system isn’t sure about an action such as which product was selected, it asks a trained associate for confirmation. This happens a small fraction of the time.”

Amazon did not deny that some of this work is being done in overseas centers, despite being directly asked. It’s worth noting that tech companies can employ humans-in-the-loop while still concealing the identifying details of their customers.

Amazon Go is largely geared toward identifiable Amazon customers with banking cards and smartphones

Amazon Go is largely geared toward identifiable Amazon customers with banking cards and smartphones. Source: Shutterstock

The best-case scenario

As AI continues to improve, there will be a reconfiguration of technologies and society. Alongside the social risks and technological controversies, there’s an opportunity to solve some deeply entrenched problems. We may lose certain types of jobs, but can we increase access to nutritious foods and improve public health? Will automated grocers, with reduced labor costs and condensed retail footprints, eventually pop up in underserved communities?

The USDA concluded that 17.7 percent of the US population has limited access to a supermarket or grocery store within a distance of 0.5 and 10 miles. However, its economic research of 2010-2015 also noted: “These findings suggest that income and resource constraints may be greater barriers to accessing healthy food retailers than proximity.”

Over the period analyzed, there were declining levels of vehicle access across all US housing units, not just the low-income and low-access areas. Sarah Jo Peterson, an urban planner, wrote: “Indeed, the high showing of states with stable or declining numbers of households, many with large rural populations, indicates that car-free living isn’t just an urban hipster trend.”

When considering these factors, it becomes apparent that one technological solution isn’t enough here. The introduction of autonomous vehicles or even good old-fashioned public transportation service upgrades could also facilitate access to healthy food.

But even that reconfiguration wouldn’t ameliorate the problem of insufficient incomes. Although President Trump has claimed credit for unemployment rates being at a 50-year low, worker earnings have actually edged lower. This negative relationship between unemployment and earnings growth could be attributed to various causes, such as the weakening of unions. According to the Phillips curve, wages should reflect the heightened demand for labor and rise accordingly. Some labor economists have suggested that the perceived risk of a recession, prompted in part by President Trump’s trade wars, made employers more stingy and job seekers less demanding. Those light wallets mean that consumers will still be drawn to cheap, unhealthy options. Birmingham, Alabama and other cities have actually passed laws to curb dollar store expansions and make proper grocery stores more viable.

In the best-case scenario, the automation of grocery retail would not only increase the number of accessible locations but also positively affect the inventory. For example, a bodega could replace its cashier with an on-site sushi chef to draw people in. It would also be fascinating to see if this technology could reinvigorate the concept of a “food cooperative,” which saw its heyday in the 1970s. But it would be naive to oversimplify the nature of this problem or to overlook the multitude of solutions, both technological and societal, that may be required. Similarly, it’s important to note that some of these solutions could create new problems.