Developer Perspective: Using Oculus Medium For 3D Production
Tyler Hurd
Today we hear from Tyler Hurd, VR developer/creator and recent addition to our Oculus Medium Team. He recently embarked on a journey to better understand how Medium can be integrated into the VR development process. From concepting and design to the Unity build, there were plenty of lessons to be learned from his recent project developing his VR latest title, Beach Body Bros.
Hi, I’m Tyler Hurd, I’ve made some fun VR things over the past few years, like BUTTS, Old Friend and more. For my last project I worked with Oculus Medium (version 2.25) with the purpose of informing future development of the 3D modeling tool, and I wanted to share a bit more about my process and learnings from this experience.
The Design
Medium is a sculpting tool so naturally an experience about bodybuilders made sense as a starting point, they’re sculptors of their own bodies after all, and I knew there was a well spring of hilarious ideas to pull from. From that starting point all the creative ideas and decision making just easily appeared. The design grew quickly and organically, much like your bulging muscles in this upcoming VR title, Beach Body Bros.
VR has the power to play with users’ perceptions and emotions, so I try to design with the user in mind as much as possible, make them the center of the universe and build the world around them. In my past projects I’ve found that users enjoy playing with the avatar they are embodying, and it’s best to make their role in the experience as clear as possible while keeping their engagement with the virtual world fun and constant. Naturally in a weightlifting environment there’s a lot of focus on self-image which is perfect for someone embodying a new avatar. There is also an intense amount of camaraderie and encouragement which is great for the user to feel like they’re engaging with the virtual world and the characters in it.
Here was the initial description:
In a world ruled by Bros, one feeble wimp has what it takes to become the ultimate Beach Body Bro, and that pint-sized little runt is you! Beach Body Bros takes you on a ridiculous musical journey through a world of pure muscle, intense strength, and stupid greatness. As you march along the sand your puny frail body struggles with the lightest of dumbbells, but with the support and blunt encouragement of a few wicked Beach Body Bros your body quickly beefs up until you’re transformed into the proudest of hulking meat specimens, glistening in the sun.”
Medium in Production
Historically this sort of process would require time and effort to develop the designs in 2D, followed by a lot of back and forth between the concept artist and the modeler to prove out the final art in 3D. This is especially true with stylized art because a lot of 2D techniques just don’t translate in 3D quickly and easily. Medium provided a number of improvements to this process by blurring the lines between concept art and modeling, all while accelerating this entire process.
All of the 3D art in Beach Body Bros was created by artist Razmig Mavlian using Medium. Coming from a 2D concept art background, Raz was apprehensive about learning traditional 3D modeling software but Medium was approachable enough that he was able to pick it up quickly, grew to love it and eventually found a place for it in his art practice.
Adding Medium to your art practice will get you to where you want to go faster than any one method alone. -Razmig Mavlian
Raz started with some loose 2D concept art to warm up and find the right style, he pulled inspiration from the 1980s Venice Beach bodybuilding scene and Retro Wave art, which led to some delightful results.
After warming up, Raz went right into Medium to design and iterate in 3D to find something that felt right. The iterations were much closer to the final results, not only were they already 3D but we could preview them in VR on the Rift to get a sense of what they will feel like in the headset. Creation time was cut down immensely; when the design was locked, Raz could just polish what he started (instead of handing off a 2D design to a modeler to start that back and forth process).
Once the main character and the avatar designs were finalized, Raz began sculpting everything else in the experience, the background models and props. It was so fast! He was sending me several sculpts from Medium in a day to review. He could design in 3D, make tweaks and have something polished and ready to go in very little time.
These beautiful sculpts still needed to be integrated into the experience in Unity, and because I was so unprepared for the speed at which they were ready, they started piling up and integration quickly became a bottleneck. This part of the production was not nearly as smooth, so I’d recommend you leave yourself some extra time for integration.
Using the Sculpts
Medium has export tools to get models into a usable state for real-time. It provides a triangulated mesh at a resolution of your choosing with either vertex colors, or a color map and a normal map. For props and less featured items, I exported to FBX with maps so I could keep the poly count low while retaining detail with the normal map.
This worked pretty well, but there were a few challenges. When exporting with maps you get a separate map per sculpt layer, and for performance in real-time the fewest maps possible is ideal. Combining maps in other software is a cumbersome step I didn’t want to add to the pipeline, so I ended up merging everything into a single layer in Medium before exporting. If objects were overlapping and merging sculpt layers would remove a clean seam, I would just separate the sculpt layers in 3D space using the move tool before merging, and bring them back together in Maya before the final export.
The exported texture maps also did not contain sufficient atlas padding, which can lead to visible seams if you down-res textures for optimization, or more importantly when mip-maps are visible. To correct this in Photoshop I made an action for creating the texture atlas padding. It goes like this: Ctrl+Select Layer, Invert Selection, Filter > Other > Minimum..., Deselect, Repeat. The "Minimum" operation bleeds the selection border by 1 pixel, so repeat for as many pixels as you want for your atlas padding. See the results on the sandcastle images below.
For rigging and animation you need complete control of your topology, thoughtfully laid quads are a requirement, so for the main characters I sought help from the wonderful 3D Artist Shawn Witt to take care of retopologizing by hand and transferring using a combination of Maya and Substance. Shawn exported a lightly decimated sculpt from Medium to bring into Maya as the guide to create the quad mesh using their Quad Draw tools. He then exported the raw sculpt from Medium as FBX with vertex colors and brought both the raw sculpt and the UV mapped quad mesh from Maya into Substance. From there he transferred the color and normal maps and painted roughness and metallic maps to create the beautiful final real-time mesh.
Although retopologizing by hand takes considerably more authoring time, it yields the best results, and it’s a more controlled traditional approach that most pipelines and tools are already designed for, so the rest of your pipeline becomes more manageable. This technique was also a necessity for the avatar’s muscle growth blend shape that Shawn made from Raz’s sculpts.
Conclusion
The goal of this project was to stress test Medium as a 3D modeling tool in a real-time production and it did really well. I’d appreciate features to combine objects and maps on export, a way to paint and export roughness and metallic maps in Medium, and proper atlas padding in the exported maps. It was really amazing to see how it enabled an artist like Raz to generate so many 3D models so fast, and the ability to preview the models in the Oculus Rift during the design phase eliminated unforeseen revisions during the modeling process. I’m excited to see how Medium evolves in the future and changes 3D modeling in production.
At Meta Connect, we shared new hardware, development paths, tools, and mixed reality technologies paving the way for greater developer success on Meta Horizon OS.
Meta Avatars SDK Update: Now Supporting Animated Full-bodied Avatars, Parametric Body Shapes, Custom Animations, and More
Meta Avatars SDK v29.6 enables your app to support full-bodied avatars, custom animations, attachables and more. Paired with an updated visual style, Meta Avatars SDK gives you the tools to create more engaging MR and VR experiences for Meta Horizon OS.
Unlock New Possibilities in Mixed Reality with Tools and Features from Meta Connect 2024
Explore announcements from Connect 2024 that reveal why building mixed reality experiences is more rewarding and easier to start than ever before. Browse the latest updates and launches.