Medium Under The Hood Part 3 - Move Tool and Asynchronous Operations

Oculus Medium is an immersive VR experience, designed for Touch, that lets you sculpt, model, paint, and create tangible objects in a VR environment. Earlier this year, we released an update which included the Move Tool. In earlier posts, we discussed how the Move Tool was developed and some directions that did and didn't work out, as well as how we transform from a signed distance field to a triangle mesh and back.

In this last post of the Move Tool series, we’ll discuss how we schedule performance-intensive operations asynchronously. This is useful for any piece of software, from content creation apps to games, that need to maintain VR framerates while also running computationally heavy tasks.

We split the Move Tool operations into two phases: the first is the realtime preview, and the second is the more lengthy processing of that move operation.

In the first phase, the response time is critical. You need to be able to use subtle hand movement to pose the sculpture exactly as you intend. When sculpting, there's only a small difference between something that's in a lifelike pose and something that just looks wrong, and so Medium must allow you to move your hands and judge the result in realtime.

To enable this, the first phase offsets vertex positions in the vertex shader using the Move Tool's displacement field. This has little impact on performance as it’s a simple calculation of the displacement vector and an offset added to each vertex’s position. However, this phase does take some shortcuts that can impact the rendering quality (but not performance). For example, this phase doesn't recompute vertex normals of the mesh as it's being deformed, and we don't recalculate ambient occlusion.

In the second phase, we do the more intensive processing: we deform the triangle mesh, convert the deformed mesh back to a signed distance field, and regenerate the triangle mesh from the SDF. This effectively retopologizes the surface, allowing subsequent move operations to operate on an evenly tessellated mesh. As part of regenerating the mesh from SDF, we recalculate the vertex normals and ambient occlusion. After this phase, the sculpt renders correctly again.

The second phase can take much longer than a single 90 Hz frame, although we optimized this code to be as fast as possible. Small movements look and feel nearly instantaneous. Larger operations take up to a few seconds. The time is proportional to how much is being moved

When the second phase takes more than one second, we put up an animated Medium logo, indicating to you that the operation is taking some time and Medium won't accept any new commands. We call this the 'hourglass hands' because it's analogous to the Windows hourglass icon or the OSX spinner. In this phase, the application is still running at 90 Hz, including head tracking and hand movement.

You can imagine that you’re using the Move Tool to tell the computer a series of commands. When you squeeze the trigger down, you tell the computer "let me show you what I want you to do". When you release the trigger, you tell the computer "now generate the results". When you are showing the computer what you want it to do, it's critical that you're able to use your hands in VR in realtime. But when you tell the computer to generate the results, it's not too disruptive to wait for it for even a couple of seconds. As long as your sense of immersion isn't disturbed-- your eyes receive 90 Hz frames and your head and hand movement are accurately tracked-- this is acceptable and doesn't break the illusion of VR.

The sound design in Medium reinforces what's happening. We play an upbeat sound when you pull the trigger and begin a move operation, and play a downbeat when you release the trigger and finish the move. We play a final 'ding' sound when the second phase has completed. This cycle of sounds works well to communicate to you what the computer is doing. Sound is not often used in desktop content creation applications, but for virtual reality applications, sound is an important component to immersion.

Displaying the hourglass hands and blocking user input is a pattern borrowed from desktop GUI applications. It would be ideal if everything were fast enough that this wasn't necessary, but it's a practical way to handle lengthy operations in a VR application.

In this classic paper from 1968 "Response Time in Man-Computer Conversational Transactions", it's advised that operations taking 100 ms or less feel instantaneous, one second is the limit for the user's flow of thought to stay uninterrupted, and 10 seconds is the limit for keeping the user's attention. For a sense of presence in virtual reality, Medium needs to generate new frames closer to 10 ms. The Move Tool works well within these limits: the first phase runs at VR framerates, and the second phase either finishes within one second, or we display the hourglass hands if the operation takes longer than one second.

The VR simulation must generate a new frame every 90 Hz, or about 10 ms. However, many operations in Medium take much longer to compute than a sliver of a 10 ms frame. The answer is of course that we compute those lengthy operations on threads running asynchronously, but it's worth taking a close look at this. This is another pattern that's useful for any VR application.

When you add new clay in Medium, the sculpting operation can often have tens of milliseconds of latency before it completes and the results are seen. However, because new frames are rendered every 90 Hz, you see your head and hand move in real time. As long as your head and hands are moving in real time, you don't notice that the sculpting operation lags a bit behind. VR requires that you respond nearly immediately to head and hand movement, but it's still acceptable for the results of the application to run at a lower frequency.

Imagine that you're using your favorite desktop application: in the real world, your eyes receive a steady stream of photons from the real world, but the computer itself might be generating frames on the flat screen at a much lower frequency. This is what we simulate in Medium, and is the way we can put a demanding 3D content creation application into a realtime VR application.

This is analogous to a hardware mouse cursor on a flatscreen desktop. You expect that the mouse cursor be immediately responsive to your hand's movement, but you don't necessarily expect that the application runs at the same speed.

Under the hood, Medium almost operates as two different programs. One program does the sculpting and meshing work, and the other program does all of the rendering. The sculpting and meshing operations don't happen at VR framerates, but the rendering does.

There are some frames where the rendering does drop below 90 Hz. For those frames, Medium relies on asynchronous timewarp (ATW) and asynchronous spacewarp (ASW) to smoothly deliver a new image to the Rift.

In the video below, you can see Medium’s in-app profiler. The orange bar at the top is the time for each frame, which remains steady. During the video, there are several move operations, and each one takes several frames to complete. There are four CPU threads running, which are displayed in the blue/purple bars. The topmost thread is the main thread, which consistently produces frames. The last three threads are performing the move tool work, run asynchronously of the main thread, and take several frames to complete. They do not impact the overall frame time.

Medium uses a single CPU thread for rendering that produces OpenGL commands that the GPU consumes. The GPU renders a stereo image for display by the Rift headset. These two threads (for the sake of this discussion, we can consider the GPU a thread) produce an image every 10 ms.

The remaining CPU hardware threads perform sculpting operations and other lower priority work, such as video recording, loading assets in the background, etc. The sculpting operations work in parallel across all available threads, and we use Intel's ISPC compiler to generate SSE4 and AVX2 code for our particularly performance-intensive code.

The sculpting thread and main thread are connected to each other through a producer/consumer FIFO. As new mesh data is generated by the sculpting thread, it is placed into the FIFO. The main thread spends up to two milliseconds a frame copying data from that FIFO and preparing it for the GPU. Once two milliseconds have elapsed, we stop copying data from the FIFO, and proceed to rendering. We include that two milliseconds in our CPU frame time budget so that we avoid hitches when updating data. When the FIFO becomes full, the sculpting thread stalls, and waits for the main thread to consume data from the FIFO. This trades off some latency for tool operations for an even, consistent framerate.

We currently use the CPU to do the sculpting and meshing work. This work could be done much faster by the GPU, but the GPU is already occupied with rendering, especially on our minimum-spec machines. The amount of free time on the minimum spec GPU is actually quite small, if any. Thus, although the GPU could easily beat the CPU at this work, the CPU has more available resources. That said, we are investigating using the GPU for more of our sculpting work, but carefully watching that we don't impact the frame rate.

It is important that the latency of the asynchronous sculpting operations not grow too long. For example, it becomes frustrating if you squeeze the trigger to add some clay but it doesn't show up for half a second. When the latency grows too long, the application feels sluggish. This sluggishness can cause the feeling of physical fatigue, even if the application is displaying frames at 90 Hz. We're constantly tuning Medium's operations to be as fast as possible, but it’s not necessary that asynchronous operations complete within a 10 ms frame.

This is the last post in our three-part Move Tool series. Throughout the series, we covered the design and prototyping of the tool, we looked at how we use multiple surface representations within Medium and how we convert from mesh to SDF, and finally we discussed how the Move Tool uses a realtime preview phase followed by a longer processing phase and how the app maintains VR framerates while running computationally intensive tasks.

We hope that this is useful information that will help you design and develop your own VR applications!