ILMxLAB Q&A: How They Ensure Stable Performance with Crash Analytics

This past February we launched the Crash Analytics Dashboard. Since release, we have seen many developers start using the tool to identify and prioritize crashes that are negatively impacting the user experience. We recently sat down with two talented individuals from ILMxLAB, Alyssa Finley and Marissa Martinez-Hoadley. They work on Star Wars: Tales from the Galaxy’s Edge⁠, and we asked them about how they are using the Crash Analytics Dashboard at their studio.

Alyssa: I’m the Producer of Star Wars: Tales from the Galaxy’s Edge. It’s my job to balance priorities against time to make sure we make the best possible choices of how we spend development time.

Marissa: I’m the Quality Assurance Manager. I set the procedures, priorities, and quality standards for the Test Teams across all our products. I also support our development teams set up and maintain their devices and development environments. I test our products as well, and even help fix some bugs when I can.

Alyssa: We want to provide the stablest possible experience for our users. Basically, crashes are terrible for immersion, and super frustrating from a user POV. But sometimes they’re also really hard to track down. So any additional information about how frequently something happens, helps us prioritize our time to have maximum impact for our players.

Alyssa: Previously we would use the Quest bug reporting to collect information on crashes found locally, but we had no means of tracking what was happening live other than user reports to customer service.

Marissa: Yeah, initially the only crash data we had access to, was what QA could find. That was great for bugs that had solid, or very frequent, reproduction steps. But, for bugs that seemed to be extremely rare, we couldn’t get much insight into how much they would impact the players.

Alyssa: The Oculus team recommended we try it.

Alyssa: We did upload debug symbols.

Marissa: It’s helped us identify how much crashes we found to be difficult to reproduce, were impacting our players. The stack traces allowed us to match up the crashes between the Analytics Dashboard and the Bug records in our database. Very helpful indeed.

Marissa: Production, QA, and Engineering are the teams that use it the most. We use it to identify the various crashes that are happening to our end-users , and how often. From there we cross-reference it with our bug database, and either ask for QA to try and get repro steps, or assign it to an Engineer to fix.

Alyssa: It’s also very useful to see what crashes are happening on the dev team/QA side for builds that are in development. We can keep track of whether a crash is new, or is something that’s been around for a while. We have a separate store channel for our current development build, so it helps us keep track of the state of stability both locally and in the wild.

Alyssa: We had a crash happening in our test environment that happened daily under very specific circumstances. Because that was a performance test suite, not normal playthrough, we didn’t prioritize a fix. The analytics dashboard helped us identify that that crash was happening multiple times outside of perf testing, which helped us identify that we needed to prioritize a fix.

Marissa: The data has been great! We’ve been able to address a couple of our most frequently occurring crashes, which will be shipped in future releases.

Alyssa: We haven’t released a patch since Crash Analytics came out, but we’re looking forward to using it in future releases both to make sure the user experience is as good as possible at ship, and to assess the severity of any issues in the wild.

Alyssa: Just do it :)

Thank you to both Alyssa and Marissa for chatting with us! If you want to start using Oculus Crash Analytics for your application, here’s how:

Crash Analytics dashboard - available on the Developer Dashboard⁠ under an application’s Analytics section in a new tab labeled “Crash.” The overview page provides actionable information at a glance, including total number of crash events and active/affected users, and can be filtered by device model, app version, and date range.

Uploading Debug Symbols - You can upload debug symbols with your APK via the Oculus Platform Command Line Utility (CLI) or directly from Unity and Unreal engine integrations. For developers with existing applications, you can use the CLI to attach debug symbol files to a Quest app previously uploaded to a release channel.

Documentation - See our Crash Analytics documentation for more details.