Snap AR app hints at interactive digital cities of future

• Snap unveiled its AR Local Lens feature City Painter this week
• The feature enabled multiple app users to paint a street in London 
• The technology is a glimpse of a future of interactive digital twin cities 

Augmented reality (AR) is one of those rare areas of technology that shows promise as a highly-practical solution across multiple use cases, while simultaneously capturing our imaginations. It can help aerospace engineers map wiring on Dreamliners, but it can also provide escapism, reinventing the physical world around us in real-time, in ways that would otherwise be impossible. 

Snap gave us a taste of the second this week, demonstrating how AR can be used to bring people together in a novel way: the maker of social media app Snapchat this week launched Local Lenses, a new feature that enables the large-scale, collaborative use of AR in public spaces. 

The first Local Lens is called City Painter. Available earlier this week on London’s Carnaby Street, Snapchat’s geotargeted AR feature allowed users to turn their smartphones into spray cans and plaster the buildings around them with brightly-colored paint. 

Users of the app could see fellow City Painters in action around them, and battle with them to daub red-brick walls with their color first. 

Snap unveiled the first-stage technology last year as Landmarkers, which enabled the Snapchat camera to “understand” and interact with certain well-known buildings. App users could put filters over buildings like Buckingham Palace, New York’s Flatiron, and the Taj Mahal, for example. 

Local Lenses builds on this, making it possible to augment larger areas, or “entire city blocks”, said the firm. 

“By taking in information from 360-degree images, and Community Snaps, we’re able to build up a digital representation of the physical world, and view an augmented experience from different viewpoints.

“Combining this with 3D reconstruction, machine learning, and distributed cloud compute, we can now map whole city blocks.”

However, while 3D data of global landmarks can be obtained easily, mapping a single, not-always-that-interesting corner of London in its entirety is a lot more complex: “It’s very unlikely that people are going to be taking photos of every single part of a street, just because, generally, there’s not necessarily something interesting on every single part of any street,” Snap London’s senior manager of research engineering, told Mashable. 

Snap had to 3D map Carnaby Street so that users could paint it from any angle, and that meant acquiring visual data through public Snaps of the area, combined with 360-degree camera imagery taken by the Snap team, whose HQ was located nearby.

The second challenge, however, is making this experience synchronous and consistent between multiple users. 

“We have a single shared reality,” said Pan. “And when you do something to this world, someone else can see that result almost instantly. Those changes also persist if everyone leaves the experience and new people pop up the next day – they can see the space that’s been altered by yourself and others.”

The end result is a fun, communal experience. But it hints at a world where entire cities will have multiple digital versions — or digital twins — of themselves accessible to anyone through a device. 

This opens doors for groundbreaking, new immersive experiences in gaming, advertising, and social media. 

“We envisage doing new things that couldn’t be done before, for example, AR tours of neighbourhoods guided by famous people who used to live there, leaving notes, photos and videos for your friends to discover later, or playing immersive AR games together with your friends, altering the shared digital space,” Pan told Mashable.

“We’re also coming up to Halloween, in the future, we could imagine a total transformation of Carnaby Street to make it feel spooky.”