Building Puzzling Places: How the Realities.io Team Brought Their World-Building App to Life

Puzzling Places⁠ is available today on the Oculus Quest Platform. Initially launched on App Lab in February, the team at Realities.io developed a unique and modern take on the puzzle pastime, combining disassembled photogrammetry 3D scans and immersive soundscape to create a calming VR experience that lets you build hyper-realistic places from around the world.

We spoke with the Puzzling Places team (Azad Balabanian: Puzzlemaker & Photogrammetry, Daniel Kraft: 3D Artist & Designer, Daniel Sproll: Producer & Developer, Pierre-Marie Blind: Audio Designer, Shahriar Sharabi: Programmer, Game Designer & Art Director) about the development of their 3D jigsaw puzzle game, technical challenges they encountered, and lessons learned along the way.

Daniel Sproll: The initial idea for Puzzling Places was what Bob Ross would call a happy little accident. We’ve been working on photogrammetry based VR experiences since 2016 and one day we had an import error on one of our meshes. For optimization purposes our photogrammetry assets are usually cut into pieces with a common pivot, but somehow for this import all the pieces were jumbled up. Shahriar joked we should just ship it like that and let users puzzle it together - and so the idea for Puzzling Places was born!

Azad Balabanian: One of our goals with Puzzling Places is to enable players to interact with unknown and beautiful places from around the world so we spend quite a lot of effort in finding diverse locations that could make for fun puzzles.

Every puzzle is made with a high resolution Photogrammetry 3D scan that was either captured by our scanning team or by one of our scanning partners. Typically, the places are captured using DSLR cameras for terrestrial scanning, drones, planes, and helicopters for aerial scanning, and sometimes LIDAR scanners for more complex locations. Each scan consists of hundreds to thousands of photographs which can generate 3D models with hundreds of millions of polygons with incredible detail.

Once the scanning and model processing is complete, we typically do quite a lot of “cleanup” of the models to fix Photogrammetry artefacts, add details that were not captured in the 3D scan, and create a “final crop” of the model that will give the puzzle the clean, smooth-edged border pieces that people are familiar with.

We then simplify and optimize the high resolution models so that they can run on the Quest and Quest 2 without any performance issues, as well as cut them into puzzle pieces. This step is not trivial by any means so we developed a robust processing pipeline to produce Quest-ready puzzles reliably.

Daniel Kraft: As our super detailed puzzles have a fairly high polycount and texture resolution for mobile we didn’t have much performance headroom left for the rest of the game and had to be quite thrifty there. One example for this is our game environment: We modeled the assets very lowpoly and made extensive use of vertex colors and UV channels to color them instead of using expensive image textures.

Shahriar: A lot of inspiration came from Chinese ink paintings and Japanese screen paintings. Also specifically Ukiyo-e prints of the Edo era and the woodblock prints of the 20th century. The main point there was the intriguing way they showcase depth using only a few overlapping layers and the serenity the images reach through simple to read compositions. Puzzling Places is a relaxing game and the typical strong depth queues you would see in VR games generates too much tension so we also tried to limit that by reducing the amount of depth cues and taking another lesson from Japanese woodblocks to add visual interest through gradients and texture. At the end we combine this with thin graphical lines to modernize the design and bind it to the photo realistic environments. Other lessons we learned from those paintings were simplifying the image by grouping elements and establishing a strong sense of rhythm. We similarly took concepts from films such as limited space and framing through diegetic elements which you find in puzzling places.

Pierre-Marie: Each puzzle comes with a unique handcrafted soundscape. While they have required a slightly different approach every time, they all follow the same design pillar: as the scene comes together, the player slowly witnesses the making of a soundscape from an initial bed of pure wind to a fully fledged ambiance. The idea was then to encapsulate the sonic essence of each puzzle with evocative sources: the joyful shouts of kids playing together in a street, the echoing bells of a flock of sheep in the mountains, the whistling of ropes and masts in a harbor on a windy day... These field recordings hopefully send back players to their own sonic memories. And akin to the visual environment of Puzzling Places, the sound ambience supports the soft and foggy feel of the experience to let the player's imagination own the scene completely.

Regarding music, a lot of puzzles feature musical textures. As the pacing of the game is so slow, it was important to create something that wasn't too repetitive or tiring, yet creating a unique atmosphere. Sometimes the music is linked to a particular achievement, like finishing the inside of a puzzle, other times it is linked to the global progression of the puzzle, sometimes it is a mix of both. The music in Puzzling Places takes many forms from long aeolian drones to light piano touches, or the chiming of a bell tower. The Armenia Pack also features beautiful interpretations from the Little Armenian Singers choir whose performance has given these puzzles a wondrous depth.

Shahriar: Lesson number one was to look at the general best practices of Android Development. Since the developer community there is bigger, you have more resources. On the CPU side, don’t call functions such as update/ tick if you don't need to. Avoid too many draw calls by combining objects that are not overlapping with respect to your viewer. And finally avoid overdraws or anything that puts pressure on the bandwidth like the plague. Simple transparent shaders that cover 10 percent of your screen can kill the frame rate.

Shahriar: That we have a very diverse player base. I could not have guessed that without the prototype. We have everything from people who are so into VR tech that they own every headset since the 2010s, and also people who are not that comfortable with using smartphones, yet alone joysticks. This knowledge greatly shaped how we designed the interactions in Puzzling Places.

Azad: Our SideQuest and App Lab releases were pivotal for setting up the groundwork for marketing and outreach strategies because we had already established a brand name with the Quest playerbase, a relationship with VR press and media organizations, as well as learned about what kind of marketing assets we needed and when.

This time around with the launch of Puzzling Places on the Oculus Store, we felt a lot more confident and prepared.

Over the past year and a half, starting with our SideQuest release, we’ve been building up an engaged player base that has been invested (literally) to help us finish the full game and release it on the Store. Of course, there is also a world of Quest owners out there that have no idea what Puzzling Places is, so our marketing strategy had to account for both types of players.

To our playerbase, we slowly started announcing and revealing aspects of the game to our playerbase, week by week, by showing the content that’s included in the game. Since they already know how fun puzzling in VR can be, their interest was more in knowing about the new puzzles as well as the game’s new features.

However, for Quest owners that don’t know about the game, the strategy was still to sell them on the idea that jigsaw puzzling with Photogrammetry 3D scans can be super fun, even if they’re not a jigsaw puzzler outside of VR. We’ve been active in the VR communities on Reddit, Twitter, Instagram, and Facebook so that’s been our main source of engagement with the broader Quest community. Our launch trailer is made to stoke the curiosity of new players as well as satisfy our hardcore playerbase who have played every puzzle we’ve made so far.

Shahriar: Find a core gameplay loop/ mechanic that is inherently fun on its own, everything else will follow.

Azad: Release a short but polished prototype of the core gameplay loop to the public as early as you can. Developing and iterating with feedback from the real world was one of the most important elements that has led Puzzling Places to be the game that it is today. SideQuest and App Lab can be incredible resources to give players something small and experimental without the overhead of releasing a feature-full game.

Daniel Sproll: Not everything in VR has to be the metaverse! Picking a single engaging mechanic or theme and really drilling down is incredibly fun and satisfying to work on. It opens up the space and flexibility to develop in an explorative and iterative way instead of getting tangled in something that sounded good on paper but just doesn’t pan out.

Daniel Sproll: We are already working on some super exciting new puzzle packs that will come out later this year! During our Patreon campaign we saw how much people enjoyed these regular updates, so our goal is to keep this up - we’ll be making puzzles as long as people want to play them!

On the feature side we are hard at work as well. Our first update should be out shortly after launch, adding additional polish and some small new features to keep everything neat and organized. Another feature we are of course particularly excited to look into is co-op multiplayer puzzling. Our first experiments here were super fun - but also showed us that there is still quite a bit of work to do.