Mapping the World In Real Time (On a Budget) – Built In

Carmeras engineers want real-time updates to the maps theyre building for autonomous vehicles. They also want to map every road in the world.

Unfortunately, doing both isnt possible.

Carmera builds its maps using LiDAR, a sensing method that measures distances using pulsating lasers, as well as RGB color scans, GPS data and inertial measurements. But those high-precision instruments only accomplish half the job. The maps still need to be maintained, and this startup found a distinct way to do it: sticking mobile phones onto vehicle fleets and letting them collect video data as they travel around.

Voila: crowdsourced maps.

But all that data has to be processed through edge or cloud computing. For a handful of cities, its doable. For the entire earth, it gets a little pricey.

It would take hundreds of millions of dollars to process once, Ethan Sorrelgreen, Carmeras chief product officer, said. And youd have to process it every day.

As Carmera works to usher in the adoption of autonomous vehicles, Sorrelgreens team performs an ongoing balancing act of speed, scale and cost.

Autonomous vehicles rely on a combination of cameras and sensors to navigate. Adding a base map, like the ones Carmera generates, helps a vehicle better understand its location, anticipate the road ahead and plan its course.

A vehicle driving behind a large truck, for example, may not be able to see an approaching stoplight. By referencing a base map, it could still know the stoplight is there. Base maps could also make autonomous vehicles less vulnerable to bad actors who try to trick the cars cameras.

Both functions are a draw for companies working on self-driving technology. Carmeras customers include Voyage, a driverless ride-hailing service operating in retirement communities, and Toyota.

Signing a large original equipment manufacturerlike Toyota was a big win for a startup like Carmera. But working with OEMs came with problems, as well. Specifically: The cost of processing an outrageous amount of data.

Analysis paralysis kills you when youre a startup.

Theres this general attitude that cloud processing is free, and you basically can process as much data as you want to, Sorrelgreen said. When we were working with smaller mobility-as-a-service companies, this wasnt as much of an issue. But when we started working with major automakers and looking at continent-scale or world-scale maps, our existing paradigm for processing data needed to change.

Sorrelgreen, Carmera CTO Justin Day and a couple of senior engineers sat down in a room to decide what to do. Their goal, he said, was to walk out with a potential solution as quickly as possible.

Analysis paralysis kills you when youre a startup, Sorrelgreen said. Implement something and then make it better, as opposed to trying to come up with the perfect idea up front.

Two weeks later, they had a plan. Within a quarter, they were using three critical questions to help build the most accurate maps possible at the lowest cost.

Why arent self-driving cars here yet?The Big Questions Keeping Driverless Cars Off the Road

At first, the engineers at Carmera tried to build theirown hardware to mount on fleets and process video data. But after six months of product development, they made an important discovery.

We realized we were just building a mobile phone, Sorrelgreen said.

With that realization, they shifted gears and started looking for the best cellphone for their purposes. A more advanced model would make data processing easier, but it would also drive the cost up. They decided to save money on the hardware side.

We have to get really, really smart about how we break the problem down, because we cant just throw the latest and greatest technology at it.

We dont use the latest Galaxy mobile phone to do data processing, Sorrelgreen said. Were using five-year-old, little Motoralas that are very, very old in terms of todays technology, and that means were using processors that are nowhere near as fast as what you can get.

Given that limitation, the team is continually experimenting with how best to allocate the phones processing power. In other words, how robust can their modeling get before it requires an expensive hardware upgrade?

We try to keep the phones super simple so we can deploy them on more cars, Carmera senior engineer Alana Ohno said. But weve played around with attaching more machine learning onto the phone.

Not all the data the companys cameras collect is worthwhile to process, and building a reliable computer vision model on an oldfangled mobile phone is no easy task.

We have to get really, really smart about how we break the problem down, because we cant just throw the latest and greatest technology at it, Sorrelgreen said.

The model, for instance, must be able to identify pertinent objects, like orange traffic cones that indicate road work. But running an object classifier algorithm in real time on the phones would be impossible. So the team created a few layers of algorithms to help determine what gets processed on the phone itself and what gets sent to the cloud for further analysis.

For example, if a phone has four cores, one may process a thread of key frames in the video data, hunting for the color orange. Once it finds the color, it sends that frame to a second core, which runs a rudimentary object classifier. If the algorithm determines that video data may contain objects of value like traffic cones the data is uploaded to the cloud.

Other threads look through video for diamond-shaped road signs or color combinations that could be traffic lights. But these naive classifiers even the one that hunts for the color orange dont always get it right.

Once it found us a Tropicana orange juice delivery truck, Sorrelgreen said. There was a lot of orange, but they were actual oranges.

That, however, isnt a bad thing. Carmeras developers optimized the algorithms for false positives. A fresh map is a safe map, Sorrelgreen said, and reviewing too many details is far better than missing an important one especially when your clients are major automakers.

The cloud processing system in charge of reviewing those details is known at Carmera as the cortex, and aptly so. It acts like a brain, sifting through sensory data from the mobile phones and determining whats important enough to receive further review, and what that review should look like. If video from one phone indicates a roadblock, the cortex may request a few additional frames from other vehicles in the same area. And sending single frames, instead of more video, saves processing power.

With machine learning models, you can never really test with 100 percent accuracy.

Like any other brain, the cortex uses a variety of variables to make its decisions. Data from a rarely-traveled road carries more weight than that from a busy city street, for instance, so the system collects observations each time a car passes through a less-traversedarea.

An observations potential impact on traffic plays a role, as well. If a car-camera picks up a concrete barrier, that obstruction is likely to be there for a while, and the cortex knows it doesnt need to collect data on that area again right away. Traffic cones, however, could be gone in a day, so the system will check in more frequently.

Lastly, the system prioritizes detection over clearance. That means discovering new impactful events is more important than noticing when old ones are cleared away.

Carmera built the cortex to be intelligent. Nonetheless, it still needs some help from human brains. Before new data is integrated into the companys maps, it gets passed to an operations team member for final review.

The work the human does is meant to be very, very simple, but its also really important because with machine learning models, you can never really test with 100 percent accuracy, Ohno said.

Are you getting the most from your data team?How Data Scientists and Engineers Can Work Better Together

As the Carmera team selected candidates for algorithmic review or details of value to map users some were common sense. The color orange, for example, is a good thing to watch for while youre driving. Others, however, came from customer feedback.

Its not that the customer is always right, but the customer always has more information.

Some companies, Ohno said, want to avoid roads with construction or police activity altogether because their sensors arent equipped to navigate those obstructions. Others want their maps to display every relevant detail, down to metal manhole covers.

Its not that the customer is always right, but the customer always has more information, and you should feed that into your development, Sorrelgreen said. Whether theyre right or not about whether you should be implementing something, theyre close to the problem and know what their needs are.

Going forward, Carmera will continue to tweak its algorithms daily, he said. Each new customer will come with its own needs and constraints, starting the product teams three-pronged approach back at step one.

Its a challenge, but Sorrelgreen isnt too worried. The company is preparing for demand at a global scale, he said, because Carmeras maps give self-driving vehicles an important safety feature that sensors do not: wisdom.

In order to give an autonomous vehicle a sort of intuition, you need to give it this rich, very detailed information from which to make those inferences.

Originally posted here:

Mapping the World In Real Time (On a Budget) - Built In