Man from Mars: The Cosmic Origins of Google's Project Tango

Project Tango, an effort from Google’s Advanced Technology and Projects Group (ATAP) to imbue a smartphone with the ability to build complex, accurate 3D models of the physical world, has the potential to change the interactions between humans and machines. This is the stuff of science-fiction, so it’s not too surprising that the groundwork for Tango actually started in outer space.

Mars to be precise.

The 3D mapping and tracking technology that powers the Project Tango prototype smartphones comes in large part thanks to the work done in the Multiple Autonomous Robotic Systems (or MARS) laboratory of Professor Stergios Roumeliotis at the University of Minnesota in Minneapolis. About ten years ago, Roumeliotis worked with NASA on perfecting the technology to help robots land on Mars, utilizing a system called vision-aided inertial navigations system, or VINS. But when the funding that NASA had earmarked for further VINS research was reallocated, Roumeliotis needed to find a new application for the 3D mapping tech.

A few years later, three of Roumeliotis’s students at the U of M—Joel Hesch, Dimitris Kottas, and Sean Bowman—found a new use for VINS that would refocus the MARS Lab’s sights from the red planet to the palm of your hand. The trio of students ported the VINS system to a laptop connected to a camera and an inertial measurement unit—a gizmo containing three accelerometers and three gyroscopes, which also happens to come standard in just about every smartphone on the market.

Using VINS, the laptop could render a 3D map of an indoor environment in real-time. Suddenly, plenty of Earth-based applications for the tech sprang to mind for Roumeliotis and his students. The lab wrote a proposal to the National Science Foundation that looked to use mobile devices to map public places like airports, universities, malls—anyplace that GPS wasn’t much of a help. The MARS Lab moved its sights away from the stars and focused on the ground right beneath our feet.

Down to Earth

Now, the work of Roumeliotis and his current and former students provides the foundation for ATAP’s Project Tango. According to an article in the Twin Cities’ Star Tribune, Google has provided a $1.35 million grant to the University of Minnesota for the MARS Lab to continue its research into fast and accurate 3D localization and mapping with a mobile phone. And based on the results of that ongoing research, it seems like money well-spent.

For Roumeliotis, the transition from working with NASA-ready equipment to the tech that fits in your pocket has been dramatic, at least where the cost of research materials is concerned.

“We started working with IMUs that cost tens of hundreds or thousands of dollars, and now we’re down to whatever’s on your cell phone,” he said to me during a visit to his lab a few weeks ago. “A one dollar IMU.”

Roumeliotis and Kottas ran me through a tech demo of their contribution to Tango. Their 3D mapping app lives inside of a standard Samsung Galaxy S4 smartphone, and it’s remarkably easy to use. Once activated, the user holds up the phone and walks around, just like one would when shooting a video. As the user walks, the app traces a line that’s super-imposed over the running video on the screen.

“This is trajectory,” said Roumeliotis, pointing to the line, then moving his finger to the triangle. “And this pyramid tells you where the camera is pointing.” But despite appearances, it’s not just a pyramid and a line.

“This is full-scale 3D in places where you don’t have GPS. It’s extremely difficult, and people have been trying to do this for decades, with no luck. And the processing used to take a beefy desktop to do the number crunching, and still it wouldn’t be real time. And now we can run it on an ARM processor, no problem.”

Two Cameras Are Better Than One

After getting a good look at the app running on the Samsung smartphone, I had the chance to try out the Project Tango prototype handset from ATAP. I was pleased to see that the prototype also offers the same “line-and-pyramid” found in the app running on the Samsung Galaxy, showing how the MARS Lab’s work has been implemented.

The prototype device itself is thick, presumably to accommodate a bigger-than-average battery. It also seems to sport two cameras—I say ‘seems’ because Roumeliotis wanted to be sure that he didn’t break any NDAs during our conversation. “We’ve always been on the conservative side about how much information we reveal,” he said.

Even without the professor’s confirmation, it seems logical that a 3D-mapping smartphone might be aided by two cameras working together rather than just one. The brains’ ability to capture stereoscopic imagery and depth information is helped quite a bit by the fact that we’ve got two eyes on our heads. A single-camera device could still do the job, but, to paraphrase the old saying, “two cameras are better than one.”

Google’s app provides a side-by-side view of the actual environment as the camera captures it next to a 3D-rendered environment that’s pieced together as you walk. It seemed like the same app as the one demonstrated on the Samsung, but taken to the next level.

“We are partners in the same project, but they have their own independent work,” explained Roumeliotis on the ways the two apps diverge. At the MARS Lab, they’re working with the kind of equipment you can buy in a store today. The Project Tango prototype built by ATAP, however, has a bit more under the hood, like a chip provided by advanced computer vision company Movidius.

“We work with regular, simpler phones. We don’t have a Movidius chip in there, and we do other things to make it work without it,” he said.

“Two Major Problems”

I was suitably impressed with the brief time I spent with VINS and Project Tango. I walked along the halls and various rooms in the Walter Library on the University of Minnesota campus; as I walked, the Tango prototype built a detailed, working model of my travels in real-time. So if the tech works, then why can’t we expect to find one of these handsets in a Verizon store next to an HTC One M8?

“There are two major problems that we’re trying to deal with,” explains Roumeliotis.

“One of them is robustness. You want to be able to take the phone and—if you shake it like crazy, or if you cover the camera, or if you drop it all of a sudden on the floor and you pick it up again—you want it to continue doing the localization and mapping.

“Imagine you go to the city,” he continued, “and you’re at an airport and you want to find out where you are. You want to go to Starbucks, right? As you walk, your phone slips from your hand and falls down. Assuming it doesn’t break, you want to be able to pick up from where you were and continue. This is a problem—that’s one example.

“And the other [problem we’re tackling] is to be able to do large-scale localization and mapping. So we want to be able to push the boundaries of what can be done nowadays on cell phones in terms of how large a space you can map. My personal dream would be able to go to the Mall of America and map the whole place.”

And since the Mall of America boasts about being able to fit seven Yankee Stadiums within its walls, no one could accuse Roumeliotis of not dreaming big.

Full Circle

As the MARS Lab continues its research in concert with Google ATAP’s Project Tango, Roumeliotis has high hopes for how the work will be used—applications like photo-realistic 3D mapping, assistance for the visually impaired, more immersive training programs for new employees, and augmented reality and games.

And, funnily enough, Project Tango will have some applications more in line with its star-bound origins after all. A recent post on PC World reports that two Project Tango handsets will be launched on the upcoming Orbital 2 mission, scheduled to head up to the International Space Station this May. It’s only fitting that technology originally designed to guide robots landing on the surface of Mars should finally find its way into space.

But that isn’t the only aspect of Project Tango that’s come full circle. One of the best parts of the job for Roumeliotis seems to be working with his former PhD students—first with Joel Hesch, and later on with Esha Nerurkar, who are currently two of the integral members of the ATAP team under Johnny Lee.

“It was definitely very exciting,” said Roumeliotis, recalling how he felt when he learned he’d get to work with Hesch again.

“He’s my boss now,” he smiled. “I used to boss him around.”