Optimizing Oculus Insight Controller Tracking to Work in Challenging Conditions (Like Near Holiday Lights)

Hi everyone, Andrew Melim back again to talk about our continued efforts to enhance and strengthen the Oculus Insight controller tracking system! If this is the first post you’re reading in the series, I highly recommend you read the first two articles on this topic, which provide a foundation of knowledge that I’ll reference in the article below.

Today we’re excited to share more information about a recently shipped update to improve controller tracking performance, especially in environments that are challenging for the tracking system, like those that include Christmas trees or holiday lights. Since our tracking system is based on detecting and geometrically matching against small clusters of infrared light, the holiday season introduces significant challenges with correctly tracking the controllers against a background containing several hundred light blobs that can be mistaken by the system as controllers.

Additional lights in the images present several challenges to the tracking algorithm.

With these challenges, controller tracking in environments near holiday lights can be extremely difficult and could previously cause errors unless users turned off the lights or moved to a different room. But we want people to be able to use Quest and Rift S in their homes no matter the season or environment, so we focused on optimizing our tracking system to find a solution.

We created two main approaches to tackle different aspects of this problem:

First, we developed a new algorithm that detects stationary 3D lights in the environment and uses their 3D locations to help reject these light blobs from even being segmented. This process of mapping external points is similar to the system behind the headset’s core Insight tracking system.

Developing an accurate map of these stationary outlier blobs requires the system to compute the underlying geometry of these points and triangulate their positions in 3D space. Internally the system develops a list of landmarks that it tracks during the play session. These landmarks store not only the location of the outlier blobs, but also an uncertainty value known as the covariance. This uncertainty allows the system to know how confident it can be when a landmark is stationary. Once the same landmark has been seen in the same location enough times, it’s promoted and used by the system to ignore the blob during tracking.

Within the side-by-side image below, you can see that our new algorithm (on the left) has identified the noise/background marked in pink, and controller blobs marked in green.

Once the system has mapped stationary landmarks, it projects them onto the camera views to get their pixel positions in each frame. Blobs detected in these positions (measured by an error metric within a specific threshold), are not used for blob to LED matching. After subtraction, the system gets a list of non-stationary blob indices, which are indices of blobs that are likely to be controller blobs. However, just keeping track of the stationary landmarks isn’t enough to ensure robust tracking. In the most challenging environments, we want the system to reject light sources that are obviously incorrect before having to process them at all.

So we conducted internal tests to analyze why tracking mismatches occur, and found that oftentimes, the mismatched controller blobs have wrong sizes given their matched pose (e.g. blob is too small when supposed pose is close). In summary, the apparent blob size in the image is mostly predictable based on the following:

Given these relationships, we experimented with different methods to model the above input signals to the output (actual segmented blob size). We found geometric modeling is fairly inaccurate due to refraction effects of the plastic, and that a machine learning method, specifically a small neural network, resulted in the best accuracy.

We trained an extremely small and targeted CNN (Convolutional Neural Network) using Caffe2 to validate blobs which were obviously too large or too small to be controller blobs. Once trained on specific internally-collected datasets, we found it was an effective way to significantly reduce the number of outliers when used in conjunction with the mapped landmarks.

With these changes, we’ve seen a notable improvement in challenging environments like those more common during the holidays. Quantitatively, the largest improvements are in significantly reducing the amount of data loss and nearly eliminating incorrect LED matching in the presence of dense external lights. We’ve found these changes have definitely improved the experience and we hope that for those new Quest and Rift S owners this holiday season, you’re able to enjoy VR even more seamlessly all year round.

We hope that you’ve enjoyed this blog series detailing our ongoing work to improve Oculus Insight controller tracking on Quest and Rift S. We will continue to share insights into how we’re enhancing the performance and capabilities of our tracking systems in the future, so be sure to keep an eye out for future posts!

- Andrew Melim